Neural Networks

Neural networks are a special class of parameterized functions that can be used as building blocks in many different applications. Neural networks operate in layers. Each layer takes as input the output of the previous layer. It applies a linear transformation to and adds a constant bias, and then it passes the result through a non-linear function before giving it to the next layer. We say that we have a deep neural network when we have many such layers, say more than five. Despite being around for decades, neural networks have been recently revived in power by major advances in algorithms (e.g., back-propagation, stochastic gradient descent), network structures (e.g., relu activation functions, convolutional neural networks), computational resources (e.g., GPUs), and software (e.g., TensorFlow, PyTorch). Today, neural networks are used in many applications such as image classification, object detection, natural language processing, autonomous driving, and in many cases, they perform better than humans. We start with the basics of neural networks in the context of regression covering the mathematical structure of dense layers, the derivation of loss functions, and the minimization of the loss functions using stochastic gradient descent. Then, we discuss classification networks with convolutional layers, a network structure which is particularly successful in image classification tasks because it mimics the structure of the animal visual cortex. Finally, we discuss a class of techniques known as physics-informed neural networks, the goal of which is to bake known physics into the networks. Physics-informed neural networks are the state-of-the-art in scientific machine learning, and they have the potential to revolutionize the way we deal with data in engineering and scientific problems.