AI in 2020

AI in 2020 is dramatically shaping human civilization. There is much to be predicted about the future and the following are few of the speculations cum predictions about the field.

Infinite Neural Networks

Neural networks are notoriously classified as Black Boxes, outputting accurate prediction given some data, with no real way to understand. Why?

A great step in this direction is the paper by Roman Novak et al. Titled Neural Tangents Fast and Easy Infinite Neural Networks in python (Google Brain & University of Cambridge). Neural Tangent is a high-level python Library designed to enable more research into infinitely wide neural networks. Neural Networks are no longer matrix multiplication tables that nobody understands. We are making progress towards a theoretical understanding of them.

Consider a network, Where the number of neurons in each layer is increased to infinity. Why? Because in mathematics it is usually easier to study concepts when they are stretched to an infinite limit. When this happens we can consider the network as a function drawn from what is called Gaussian Processes or GPs. For any given Dataset there are potentially infinitely many functions we can use to fit it to the data. GPs help solve this problem by assigning a probability value to each of these potential functions. In general, this enables us to understand a huge range of phenomenon in Deep Learning.

Explainable AI

There is also XAI, An explainability toolbox for machine learning by the institute for ethical AI in ML. It allows you to easily identify imbalances in data visualization correlation.

In terms of power consumption. We are beginning to see how much AI data centers contribute to carbon emission (Paper by Strubell et al) and that is not a sustainable long-term strategy for our planet. especially since the need for AI will increase server time.

Quantization and Data efficiency

Lately, Quantization is a word that is on everybody's mind. Think of quantization as an umbrella term that describes a set of techniques used to convert input values from a big set to output values in a smaller set. A tool called Graffitist enables you to do just that. It is a framework build on top of TensorFlow. To process low-level graph descriptions of neural networks into an efficient inference of fixed-point network. There is also TensorFlow light, and Pytorch is always getting speed gains from quantization.

Moving on to Data Efficiency, which is nowadays a super hot topic. My favorite paper on this topic is called practical "Deep Learning with Bayesian Principles". Here it is said how using Bayesian Statistics has the potential to address issues like representing uncertainty, by employing the data distribution and overfitting. Thus, helping the machine to learn with less data.

Reproducibility and Hardware

As per Reproducibility, Harvard and Google recently teamed up on writing a deep network for seismic prediction and no one could reproduce their results. Moreover, it was later shown that the task could be just as easily done using a logistic regression model.

Believed by many i.e, Most AI advances will be in hardware, not software. - NVIDIA is set to release their new 7nm Ampere GPUs(up to 50% faster than Turing) - Google is going to release the Tensor Processing Unit (4th Gen). - Intel GPUs

Multi-modal & Multi-task Learning

Multi-modal learning is learning that involves using varied types of training data together instead of just one type. we can see examples of this by looking at some of the biggest datasets released last year. Specifically the autonomous driving ones from Waymo & Baidu. Waymo's open datasets contain, not just millions of image frames from all of its driving but also related data like temperatures, pedestrian's information, Geographical data, etc.

Multi-task learning is a model able to perform multi-tasks. I personally think graph-based networks are going to be a huge help here a paper from just last month Titled " An Attention-based Graph Neural Network for Heterogeneous Structural Learning" points in this General Direction. Graph Network theory is still in its early stages, but the idea is to represent your network as a graph and have it solve graph-related problems.

Neural-Symbolic Architecture

Of late there has been many promoting Symbolic AI instead of deep learning, whereas rest are currently seen debating about this topic. The solution at this point is taking the Ideas from both factions. Symbolic AI basically means to encode a symbolic representation of some object or concept, using human-readable symbols. Neural Networks also do this but in their own internal representational language.

The idea behind neural symbolic AI is to combine the best of both worlds. Having Neural Networks learn discrete symbols and not just black-box representation and use them for processing.

A Generative Modelling Renaissance

We are going to experience an absolute renaissance in creativity because of advances in Generative Modelling Technology. Consider the Oldify, the AI that can turn Black&White pictures to color pictures. Also, how far we have come in AI-generated voices or the fact that deep fakes are getting photo-realistic totally indistinguishable from the real thing.

Right now leveraging these tools require a little bit of programming language. But soon there are going to be one click Apps for all of these powers and in the hands of creators. We are going to see all the new types of content emerge.

Finally, These advancements are by no means exhaustive. More breakthroughs are awaited in the field of AI. I personally believe everything in our universe can be represented as a mathematical function. The question is the accurate modeling and interpretation of the this function.

Thank you for your time.