Neural Networks

The Basic Idea

When you think of the human brain, you may conjure up an image of a pink dense blob with grooves, split into two hemispheres. But if you were to zoom in on this blob, you could see that it is made up of a complex and layered system of interconnected brain cells (or neurons). It is amazing to think that a system of neurons powers our ability to process the world around us. So, what happens if we apply this idea to computing?

Neural networks are a method in artificial intelligence that is loosely modeled after the way the human brain works. This is a subset of machine learning techniques that teach computers to analyze data inspired by the processes of the human brain. They can identify issues, compare options, and deliver a conclusion.

Image source: Artificial Neural Networks and its Applications, GeeksforGeeks (2023)

Neural networks are made up of input, output, and hidden layers of interconnected nodes. The input layer is where information is fed into the network and the output layer provides a response. The hidden layers are where the data processing occurs, performing complex transformations on the input data. Data typically moves in one direction in feedforward networks but can loop back in recurrent networks.1

The connections between the nodes are weighted. This means that the strength of the connections, or weights, determines the activation of nodes (the entire neural network is characterized by different levels of activation). Knowledge is stored in the network as patterns in these weights, not in the individual nodes. During training, weights are adjusted to minimize errors, a process known as backpropagation.

In conversation, neural networks and deep learning are terms used interchangeably—but this isn’t entirely correct. Though they are highly related concepts, the “deep” in deep learning refers to the depth of the layers in a neural network. So, as a rule of thumb, a neural network that contains more than three layers can be specifically referred to as a deep learning algorithm. The ones that contain three or fewer layers are simply known as basic neural networks.

Although neural networks are often compared analogously to the human brain, they also differ in many ways. Neural networks merely draw inspiration from human brain function. While they share some basic principles of computation and learning, neural networks are simplified AI models that do not at all fully capture the complexity, adaptability and cognitive capabilities of the human brain. 

Neural networks have become more popular over the years thanks to their advanced ability to learn and create complex patterns from unstructured data. Their use has proliferated across industries. For one, they can help with medical diagnoses by classifying medical images. They can also help create targeted marketing by filtering social media and analyzing market behaviors. Because they help computers make smart decisions with limited human intervention, neural networks are increasingly important in this current age of technology.

Key Terms

Deep Learning: A subset of machine learning that describes a neural network with many layers. The depth of layers allows for the learning of hierarchical representations of data.

Machine Learning: A branch of AI that allows computers to learn on their own by analyzing data, matching patterns, and making predictions. The goal of machine learning is to develop AI programs that can learn without explicit instructions.

Backpropagation: A method of training neural networks by adjusting the weight of connections between the network’s nodes gradually through calculated adjustments based on the previous errors. The aim is to reduce the difference between the actual output of the network and the desired output to increase its accuracy.

Long-Term Potentiation: A process of learning in neuroscience where connections between neurons (brain cells) strengthen through repeated and persistent stimulation. It is also considered a fundamental process in memory retention in the brain.

Convolutional Neural Networks (CNNs): A type of neural networks that is commonly used to process structured grid data such as images. They are particularly effective in processing visual data because they can automatically and adaptively learn spatial hierarchies of features, from low-level edges to high-level object representations.

Recurrent Neural Networks (RNNs): A type of neural networks designed to process sequential data. They work especially well with temporally or contextually dynamic tasks such as natural language processing (NLP), time-based forecasting, and speech recognition. 

Transformer Models: A class of neural network architecture first introduced in the 2017 seminal paper, Attention is All You Need, by Vaswani et al.2 They have revolutionized the field of natural language processing and are now widely used in various applications, including language translation, text generation, and more.

History

In 1943, neurophysiologist Warren McCulloch and mathematician Walter Pitts published a paper discussing how the neurons in our brain may work.3 They demonstrated their thinking in a simple neural network through an electrical circuit. Donald Hebb took this idea further and explored how neural pathways functioned. In his 1949 book, The Organization of Behavior, Hebb argued that we strengthen our neural pathways every time we use them (similar to how training our muscles makes them stronger), which is how humans learn things.

The 1950s was a period when these theories gained traction in the field of artificial intelligence. Assuming that computer processes are analogous to human cognition, researchers attempted to translate the theories on computational systems.

In 1958, Frank Rosenblatt developed McCulloch and Pitts’ theory by introducing the idea of weights. He came up with the “perceptron” which was a simple input-output system modeled on McCulloch and Pitts’ neuron theory.4 An input is weighed, and the output returned is determined by whether the input meets a given weight threshold. A fascinating aspect of the perceptron is that the weight of the inputs would be adjusted and learned through trial and error, laying the groundwork for backpropagation.

Just a year after Rosenblatt’s work, Bernard Widrow and Marcian Hoff developed the first neural network that was successfully applied to a real-life problem. MADALINE, Multiple ADAptive LINear Elements, is a neural network designed to remove echoes on phone lines. 

In 1974, Paul Werbos highlighted the application of backpropagation within neural networks in his PhD thesis.5 And by the late 80s to 90s, backpropagation was integrated into neural networks to help train algorithms, forming the basis of the modern neural network we know of today.

People

Warren McCulloch

An American neurophysiologist whose study on the human nervous system, alongside Walter Pitts, made significant contributions to the field of neuroscience and artificial intelligence. Their study, which led to the first mathematical model of a biological neuron, laid the groundwork for the development of artificial neural networks.

Walter Pitts

An American mathematician who developed the first mathematical model of a neural network with Warren McCulloch. The two successfully bridged the gap between neuroscience and computational systems.

Donald Hebb

A Canadian psychologist whose expertise in neuropsychology led to major developments in the field. His 1949 book, The Organization of Behavior, introduced his famous theory of Hebbian learning which discussed how the formation of neural networks contribute to learning. This theory has come to influence the development of artificial neural networks, particularly in the idea that neural pathways can be strengthened with practice.

Consequences

Neural networks have the power to solve complex real-life problems. Because they can model and learn complex relationships as well as make generalizations, inferences, and predictions, neural networks can be leveraged to enhance human decision-making. These qualities have led to the widespread application of neural networks across different industries from fraud detection to voice recognition and even gaming! Over time, neural networks transformed into important and valuable decision-making tools.

In recent years, neural networks have been used as a research tool as well. Notably in the material sciences field, a type of neural network known as graph neural networks (GNNs) contributed to the discovery of new stable materials.6 Beyond enhancing decision-making, neural networks have the potential to facilitate innovation and scientific discovery.

The field of neural networks itself continues to advance. The reason neural networks are a useful tool is because massive networks have been built and established. But the downside of that is that they are more expensive, they take up more storage space, and they can take time to run. To tackle this, techniques for neural pruning i.e. removing the unnecessary weighted connected or neurons to compress the size of the overall network without sacrificing its accuracy. 

Another development is the method of transfer learning where previously trained models are reused on a new problem so that the neural network being built requires less training data. For example, a neural network that has been trained to identify food in a picture can be generalized to another neural network to identify drinks. This helps to resolve the issue of needing large amounts of data to train an accurate neural network just by simply reusing another network. 
Lastly, the field has been putting in their effort in being environmentally conscious. Energy-efficient neural networks have been created to perform with minimized power consumption and computational resources. This is particularly important for deploying neural networks on edge devices, mobile phones, and in data centers to minimize operational costs and environmental impact.

The Human Element

The analogy with the human brain suggests that both artificial and natural intelligence share common principles of learning. However, while making these comparisons help improve understanding, the dynamic and adaptable nature of the human brain is vastly more complex than neural networks, which are static once trained. 

For starters, how we learn, from a neuroscience perspective, is similar to how neural networks are trained—we can both learn from examples. For neural networks, this is achieved through the process of backpropagation where weighted connections between nodes are adjusted based on training data to close the gap between the desired and the actual output. In the human brain, long-term potentiation persistently strengthens the connections between neurons based on repeated experience. Notice the similarities? Both intelligence systems need training to learn which is represented through the quality of the connections between the nodes/neurons.

Next, both the natural and artificial intelligence systems organize information hierarchically. Recall how nodes in neural networks are layered. As we get deeper into each hidden layer, we observe the representation of increasingly complex features. Let’s take image recognition as an example. The first layer may detect edges in the image. The second layer may detect the eyes and the third may detect the eye color. We see the same sort of organization in the human brain where more basic inputs are processed in lower brain regions (e.g. brainstem and limbic system) and more complicated inputs are localized to higher brain areas (e.g. frontal cortex, parietal lobes, and occipital lobes).

While these overlaps between neural networks and the human brain highlight fascinating similarities in our learning processes, it is important to note that there are still significant differences. The analogy does not mean neural networks equal to the human brain. Rather, it serves as a recognition of how computer scientists drew inspiration from cognitive science and theories of human thinking in the development of artificial intelligence. So, studying the human brain can provide insights for developing more powerful and robust artificial intelligence systems.

Controversies

We know that neural networks can function as a powerful decision-making tool for us. Despite this, numerous controversies surround the use of neural networks. Let’s take a look.

1. The Technical Issue: The Need for Huge Amounts of Data
Neural networks learn via training data to adjust and improve their accuracy. But to achieve the highest level of accuracy, you need to provide the system with millions, or better, billions of data points. This volume of data may not be accessible or available. As a result, we may end up sacrificing the accuracy of neural network outputs because of the limited training data we have. This brings to light issues with neural network reliability.

2. The Ethical Issue: Lack of Transparency
Neural networks are typically considered black box models. This means that the system's calculations to generate a result are so complicated that it’s challenging, if not impossible, for humans to understand. Because the process that led to the result remains unknown to us, how can we trust the produced output? It can be especially problematic when using neural networks as a decision-making tool in industries that call for accountability, trust, and transparency.
For instance, we mentioned previously that neural networks can be used to help with medical diagnoses. But given the high level of accountability and trust required in the healthcare field, do we need to limit the extent to which we use neural networks as a tool in medicine because of its inherent lack of transparency? This debate remains prevalent as AI technology booms.

3. The Legal Issue: Right to Ownership
Finally, a big question in the legal sphere is who owns the right to the results generated by neural networks? From a legal standpoint, AI is not recognized to have authorship rights.7 This can cause copyright complications that the law hasn’t fully dissected. As with the transparency issue, this debate is still ongoing.

Case Study

The Use of Neural Networks in Flight Simulations

Simulations are like mock exams. They test the subject in a virtual environment. They are employed across multiple industries, primarily to replicate real-world environments and assess the performance or "readiness" of a test subject. However, developing simulations of any sort is a very lengthy process. Recently, neural networks have been proposed to help facilitate the development of flight simulations. Researchers have suggested exploiting a neural network’s ability to deal with extremely complex and nonlinear data rather than relying on engineers and physicists alone.8 Their study pointed towards the potential of integrating neural networks in simulation development with its results outperforming traditional modeling approaches.9

This example demonstrates that the symbiotic relationship between humans and computers allows us to maximize the information we have in front of us. If combining the expertise of humans and computers leads to us achieving more than we ever could independently, maybe we should be viewing AI as a partner rather than a rival. 

Related TDL Content

Deep Learning
Deep learning is a subfield of artificial intelligence. It bridges the gap between neuroscience and computer science by modeling human cognition into computer systems. Deep learning has come to revolutionize computer operations. Functions like facial recognition and objection detection are made possible because of deep learning. Despite its influence in the artificial intelligence field, deep learning is not without its controversies. Read more about it in this TDL article. 

Hebbian Learning
Introduced by Donald Hebb, in his influential book, The Organization of Behavior, the concept of Hebbian learning has laid important foundations vital for the development of neural networks. The theory argues that learning is represented in our brains as neurons activating and connecting. This creates a neural network that can be strengthened through repetition, eventually becoming intuitive. While there are some controversies surrounding this theory explaining a human trait can be applied to computers, it has significantly influenced the development of artificial neural networks.

References

  1. Hardesty, L. (2017, April 14). Explained: Neural networks. MIT News; MIT. https://news.mit.edu/2017/explained-neural-networks-deep-learning-0414
  2. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017, December 5). Attention Is All You Need. ArXiv.org. https://doi.org/10.48550/arXiv.1706.03762
  3. McCulloch, W. S., & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics, 5(4), 115–133. https://doi.org/10.1007/bf02478259
  4. Rosenblatt, F. (1958). The Perceptron: A Probabilistic Model For Information Storage and Organization in the Brain. Psychological Review, 65(6), 386–408. https://doi.org/10.1037/h0042519
  5. Werbos, P. (1974). Beyond Regression: New Tools For Prediction and Analysis in the Behavioral Sciences. https://www.researchgate.net/publication/35657389_Beyond_regression_new_tools_for_prediction_and_analysis_in_the_behavioral_sciences
  6. Merchant, A., Batzner, S., Schoenholz, S. S., Aykol, M., Cheon, G., & Cubuk, E. D. (2023). Scaling deep learning for materials discovery. Nature, 1–6. https://doi.org/10.1038/s41586-023-06735-9
  7. Tserakhau, K. (2023, December 5). Copyright and Neural Networks: Current Issues. Revera. https://revera.legal/en/info-centr/news-and-analytical-materials/1290-avtorskoe-pravo-i-nejroseti-aktualnye-momenty/
  8. Stachiw, T., Crain, A., & Ricciardi, J. (2022). A physics-based neural network for flight dynamics modelling and simulation. Advanced Modeling and Simulation in Engineering Sciences, 9(1). https://doi.org/10.1186/s40323-022-00227-7
  9. See above. 

About the Author

Samantha Lau

Samantha Lau

Samantha graduated from the University of Toronto, majoring in psychology and criminology. During her undergraduate degree, she studied how mindfulness meditation impacted human memory which sparked her interest in cognition. Samantha is curious about the way behavioural science impacts design, particularly in the UX field. As she works to make behavioural science more accessible with The Decision Lab, she is preparing to start her Master of Behavioural and Decision Sciences degree at the University of Pennsylvania. In her free time, you can catch her at a concert or in a dance studio.

Read Next

Notes illustration

Eager to learn about how behavioral science can help your organization?