Neural Networks

When you think of the human brain, you may conjure up an image of a pink dense blob with grooves, split into two hemispheres. But if you were to zoom in on this blob, you could see that it is made up of a complex and layered system of interconnected brain cells (or neurons). It is amazing to think that a system of neurons powers our ability to process the world around us. So, what happens if we apply this idea to computing?

Neural networks are a method in artificial intelligence that is loosely modeled after the way the human brain works. This is a subset of machine learning techniques that teach computers to analyze data inspired by the processes of the human brain. They can identify issues, compare options, and deliver a conclusion.

In 1943, neurophysiologist Warren McCulloch and mathematician Walter Pitts published a paper discussing how the neurons in our brain may work.³ They demonstrated their thinking in a simple neural network through an electrical circuit. Donald Hebb took this idea further and explored how neural pathways functioned. In his 1949 book, The Organization of Behavior, Hebb argued that we strengthen our neural pathways every time we use them (similar to how training our muscles makes them stronger), which is how humans learn things.

The 1950s was a period when these theories gained traction in the field of artificial intelligence. Assuming that computer processes are analogous to human cognition, researchers attempted to translate the theories on computational systems.

In 1958, Frank Rosenblatt developed McCulloch and Pitts’ theory by introducing the idea of weights. He came up with the “perceptron” which was a simple input-output system modeled on McCulloch and Pitts’ neuron theory.⁴ An input is weighed, and the output returned is determined by whether the input meets a given weight threshold. A fascinating aspect of the perceptron is that the weight of the inputs would be adjusted and learned through trial and error, laying the groundwork for backpropagation.

Just a year after Rosenblatt’s work, Bernard Widrow and Marcian Hoff developed the first neural network that was successfully applied to a real-life problem. MADALINE, Multiple ADAptive LINear Elements, is a neural network designed to remove echoes on phone lines. 

In 1974, Paul Werbos highlighted the application of backpropagation within neural networks in his PhD thesis.⁵ And by the late 80s to 90s, backpropagation was integrated into neural networks to help train algorithms, forming the basis of the modern neural network we know of today.

Neural networks have the power to solve complex real-life problems. Because they can model and learn complex relationships as well as make generalizations, inferences, and predictions, neural networks can be leveraged to enhance human decision-making. These qualities have led to the widespread application of neural networks across different industries from fraud detection to voice recognition and even gaming! Over time, neural networks transformed into important and valuable decision-making tools.

In recent years, neural networks have been used as a research tool as well. Notably in the material sciences field, a type of neural network known as graph neural networks (GNNs) contributed to the discovery of new stable materials.⁶ Beyond enhancing decision-making, neural networks have the potential to facilitate innovation and scientific discovery.

The field of neural networks itself continues to advance. The reason neural networks are a useful tool is because massive networks have been built and established. But the downside of that is that they are more expensive, they take up more storage space, and they can take time to run. To tackle this, techniques for neural pruning i.e. removing the unnecessary weighted connected or neurons to compress the size of the overall network without sacrificing its accuracy. 

Another development is the method of transfer learning where previously trained models are reused on a new problem so that the neural network being built requires less training data. For example, a neural network that has been trained to identify food in a picture can be generalized to another neural network to identify drinks. This helps to resolve the issue of needing large amounts of data to train an accurate neural network just by simply reusing another network. 
Lastly, the field has been putting in their effort in being environmentally conscious. Energy-efficient neural networks have been created to perform with minimized power consumption and computational resources. This is particularly important for deploying neural networks on edge devices, mobile phones, and in data centers to minimize operational costs and environmental impact.

The analogy with the human brain suggests that both artificial and natural intelligence share common principles of learning. However, while making these comparisons help improve understanding, the dynamic and adaptable nature of the human brain is vastly more complex than neural networks, which are static once trained. 

For starters, how we learn, from a neuroscience perspective, is similar to how neural networks are trained—we can both learn from examples. For neural networks, this is achieved through the process of backpropagation where weighted connections between nodes are adjusted based on training data to close the gap between the desired and the actual output. In the human brain, long-term potentiation persistently strengthens the connections between neurons based on repeated experience. Notice the similarities? Both intelligence systems need training to learn which is represented through the quality of the connections between the nodes/neurons.

Next, both the natural and artificial intelligence systems organize information hierarchically. Recall how nodes in neural networks are layered. As we get deeper into each hidden layer, we observe the representation of increasingly complex features. Let’s take image recognition as an example. The first layer may detect edges in the image. The second layer may detect the eyes and the third may detect the eye color. We see the same sort of organization in the human brain where more basic inputs are processed in lower brain regions (e.g. brainstem and limbic system) and more complicated inputs are localized to higher brain areas (e.g. frontal cortex, parietal lobes, and occipital lobes).

While these overlaps between neural networks and the human brain highlight fascinating similarities in our learning processes, it is important to note that there are still significant differences. The analogy does not mean neural networks equal to the human brain. Rather, it serves as a recognition of how computer scientists drew inspiration from cognitive science and theories of human thinking in the development of artificial intelligence. So, studying the human brain can provide insights for developing more powerful and robust artificial intelligence systems.

We know that neural networks can function as a powerful decision-making tool for us. Despite this, numerous controversies surround the use of neural networks. Let’s take a look.

1. The Technical Issue: The Need for Huge Amounts of Data
Neural networks learn via training data to adjust and improve their accuracy. But to achieve the highest level of accuracy, you need to provide the system with millions, or better, billions of data points. This volume of data may not be accessible or available. As a result, we may end up sacrificing the accuracy of neural network outputs because of the limited training data we have. This brings to light issues with neural network reliability.

2. The Ethical Issue: Lack of Transparency
Neural networks are typically considered black box models. This means that the system's calculations to generate a result are so complicated that it’s challenging, if not impossible, for humans to understand. Because the process that led to the result remains unknown to us, how can we trust the produced output? It can be especially problematic when using neural networks as a decision-making tool in industries that call for accountability, trust, and transparency.
For instance, we mentioned previously that neural networks can be used to help with medical diagnoses. But given the high level of accountability and trust required in the healthcare field, do we need to limit the extent to which we use neural networks as a tool in medicine because of its inherent lack of transparency? This debate remains prevalent as AI technology booms.

3. The Legal Issue: Right to Ownership
Finally, a big question in the legal sphere is who owns the right to the results generated by neural networks? From a legal standpoint, AI is not recognized to have authorship rights.⁷ This can cause copyright complications that the law hasn’t fully dissected. As with the transparency issue, this debate is still ongoing.

The Use of Neural Networks in Flight Simulations

Simulations are like mock exams. They test the subject in a virtual environment. They are employed across multiple industries, primarily to replicate real-world environments and assess the performance or "readiness" of a test subject. However, developing simulations of any sort is a very lengthy process. Recently, neural networks have been proposed to help facilitate the development of flight simulations. Researchers have suggested exploiting a neural network’s ability to deal with extremely complex and nonlinear data rather than relying on engineers and physicists alone.⁸ Their study pointed towards the potential of integrating neural networks in simulation development with its results outperforming traditional modeling approaches.⁹

This example demonstrates that the symbiotic relationship between humans and computers allows us to maximize the information we have in front of us. If combining the expertise of humans and computers leads to us achieving more than we ever could independently, maybe we should be viewing AI as a partner rather than a rival. 

Deep Learning
Deep learning is a subfield of artificial intelligence. It bridges the gap between neuroscience and computer science by modeling human cognition into computer systems. Deep learning has come to revolutionize computer operations. Functions like facial recognition and objection detection are made possible because of deep learning. Despite its influence in the artificial intelligence field, deep learning is not without its controversies. Read more about it in this TDL article. 

Hebbian Learning
Introduced by Donald Hebb, in his influential book, The Organization of Behavior, the concept of Hebbian learning has laid important foundations vital for the development of neural networks. The theory argues that learning is represented in our brains as neurons activating and connecting. This creates a neural network that can be strengthened through repetition, eventually becoming intuitive. While there are some controversies surrounding this theory explaining a human trait can be applied to computers, it has significantly influenced the development of artificial neural networks.

1. Hardesty, L. (2017, April 14). Explained: Neural networks. MIT News; MIT. https://news.mit.edu/2017/explained-neural-networks-deep-learning-0414
2. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017, December 5). Attention Is All You Need. ArXiv.org. https://doi.org/10.48550/arXiv.1706.03762
3. McCulloch, W. S., & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics, 5(4), 115–133. https://doi.org/10.1007/bf02478259
4. Rosenblatt, F. (1958). The Perceptron: A Probabilistic Model For Information Storage and Organization in the Brain. Psychological Review, 65(6), 386–408. https://doi.org/10.1037/h0042519
5. Werbos, P. (1974). Beyond Regression: New Tools For Prediction and Analysis in the Behavioral Sciences. https://www.researchgate.net/publication/35657389_Beyond_regression_new_tools_for_prediction_and_analysis_in_the_behavioral_sciences
6. Merchant, A., Batzner, S., Schoenholz, S. S., Aykol, M., Cheon, G., & Cubuk, E. D. (2023). Scaling deep learning for materials discovery. Nature, 1–6. https://doi.org/10.1038/s41586-023-06735-9
7. Tserakhau, K. (2023, December 5). Copyright and Neural Networks: Current Issues. Revera. https://revera.legal/en/info-centr/news-and-analytical-materials/1290-avtorskoe-pravo-i-nejroseti-aktualnye-momenty/
8. Stachiw, T., Crain, A., & Ricciardi, J. (2022). A physics-based neural network for flight dynamics modelling and simulation. Advanced Modeling and Simulation in Engineering Sciences, 9(1). https://doi.org/10.1186/s40323-022-00227-7
9. See above.