Supervised Learning

From the moment a child starts to talk, they are provided with labeled data to learn about the world around them and how to speak. Sounds a bit mathematical and boring? Well, not really. Take the alphabet for example; children learn to recognize the sounds of each letter by associating them with a picture of an apple (A) or a hat (H). When an adult shows them a picture of an apple, the child immediately knows the correct sound to make. 

AI algorithms also use labeled data to learn how to recognize patterns and make predictions about future inputs. Supervised learning is a type of machine learning that uses datasets labeled by a human to train computer algorithms to predict outcomes and recognize patterns.¹ The examples given to the algorithm are like pairs of questions and answers; the computer studies these pairs and learns to give the right answers when it’s asked similar questions it hasn't encountered before. Ultimately, the goal of supervised learning is to make predictions from data.²

How do you know if the algorithm is learning its data correctly? Well, the dataset is usually divided into two parts; a training set and a testing set. The training set is used to teach the model and the testing set is used to evaluate its performance with unseen data. This division of the dataset is important to make sure that the model is not learning the training set too well to the extent that it can’t perform on new data (this is called overfitting). 

Let’s look at a simple example. Imagine you want to teach a model to identify pictures of flowers. You start by providing the algorithm with a labeled data set that contains lots of pictures of different kinds of flowers and the corresponding name of each species (e.g. rose, petunia, sunflower). The algorithm then tries to define the characteristics that belong to each flower based on the labeled outputs (e.g. thorns or the colour yellow). Once this is done, you can test the model by showing it a flower picture and asking it to guess the correct species. If the model provides an incorrect answer, you continue training it and adjusting its parameters with more examples to improve accuracy. When the model is ready, it can use its existing knowledge to make predictions about unknown data.  

The whole process can be loosely compared to the student-teacher dynamic we see in schools. In many subjects, students are required to learn information provided to them by a teacher and then apply this knowledge to unseen questions on a test. If they don’t pass the test, the teacher simply goes back over the information again, but this time adjusting the way the material is taught to improve retention and understanding. 

One of the most crucial steps in supervised learning is, as the name suggests, human supervision in the form of feedback and corrections. Just like a teacher tells a child when their answer is or isn’t correct, humans give feedback to algorithms about prediction accuracy in the training process. We’ll talk more about that later. 

In today’s hyper-connected, digital society there is an increasing need for machines that can make quick and accurate predictions for us. In the world of AI, there are two main types of supervised learning:

1. Regression: In regression tasks, the target variable is continuous, meaning it can take on any value within a range. The goal is to predict a numerical value. Examples include predicting house prices based on features like size, location, and number of bedrooms, or predicting stock prices based on historical data.
2. Classification: In classification tasks, the target variable is categorical, meaning it falls into one of a limited number of classes or categories. The goal is to predict the class label of new instances based on their features. Examples include email spam detection (classifying emails as either spam or not spam), image recognition (classifying images into different categories such as cats, dogs, or cars), and sentiment analysis (classifying text as positive, negative, or neutral).

Target variable: Also known as the dependent variable or response variable, the target variable is the thing you're trying to predict or understand in a machine learning model. It's the outcome or result that you're interested in based on the data that you already have. 

Machine Learning: A subfield of AI that focuses on the development of algorithms and models that allow computers to learn from and make predictions or decisions based on data, without being explicitly programmed to do so. The term was first coined in 1959 by Arthur Samuel, an IBM employee and pioneer in the field of computer gaming and AI.   

Probabilistic Classifier: A type of machine learning tool that tells you how likely it is that a piece of data (input) belongs to each category. Instead of just saying, "This is a cat," it will say, "There is an 80% chance this is a cat and a 20% chance this is a dog." This helps us understand how sure the tool is about its prediction.

Unsupervised Learning: A type of machine learning where the model is trained on data that has no labels or predefined outcomes. The goal is to identify patterns, structures, or relationships in the data without any prior knowledge of what those patterns might be.

Reinforcement Learning: A type of machine learning where an agent learns to make decisions by interacting with an environment. The agent takes actions, observes the outcomes, and receives rewards or penalties. The goal is to learn a policy that maximizes cumulative rewards over time.

Supervised learning is one of the most common and well-established approaches to machine learning, along with its counterparts unsupervised learning and reinforcement learning. Although it feels like we’ve only been talking about AI for less than a decade, in reality, machine learning has been around for over 70 years.³

Early machine learning efforts were hampered by the lack of sheer computing power needed for complex calculations and old computers that could only execute commands and not follow them. Computers were also extremely expensive to run at this time, with one report stating that the monthly rent for one computer could be up to €150,000. 

However, in the 1980s American machine learning and artificial intelligence expert Gerald DeJong made a significant theoretical contribution to the development of machine learning in the form of his Explanation-Based Learning (EBL) method.⁴ This technique uses available domain knowledge to improve supervised learning in several ways including speed of learning, confidence of learning, and accuracy of learned information. Unlike other learning paradigms that require numerous examples to induce patterns, EBL focuses on understanding the underlying principles and structures that make the example valid, allowing for more efficient learning.

The rise of the internet in the 1990s was arguably a turning point in the history of machine learning. Almost overnight, there was a vast increase in accessible data through the World Wide Web which became publicly available on April 30, 1993. Computers were also becoming steadily more powerful and could therefore process the larger data sets needed for machine learning to thrive. 

One of the main goals of machine learning is to get computers to do human tasks to the same level of proficiency as us, or even better. The first time this was achieved was in May 1997 when IBM’s supercomputer ‘Deep Blue’ beat the reigning world chess champion Gary Kasparov, opening up a world of possibilities for future AI.⁵ DeJong’s EBL was actually one of the key technologies used in developing Deep Blue!

Eventually machine learning gave way to deep learning whose algorithms are loosely inspired by the connections of the human brain. While machine learning always needs a human to supervise, deep learning can learn from its own errors.⁶ The downside of this increased independence is that deep learning requires significantly more data and computational power than machine learning. 

Gerald DeJong: American machine learning and artificial intelligence expert and Professor Emeritus at the University of Illinois Urbana-Champaign. DeJong made significant contributions to the advancement of machine learning during the 1980s, mostly notably in his Explanation-Based Learning (EBL) method. 

The Human Element

If we look at supervised learning from a psychological or behavioral perspective, we find parallels with how humans learn. We typically learn through a combination of reinforcement (‘great job!’), error correction (‘that’s not quite right, have a look at this’), and adaptive feedback (‘the areas you should work on are X, Y, and Z’). In other words, we continually monitor our performance and actions through feedback in our environment, whether that be from a teacher, colleague, or a digital interface. 

When we align machine learning with human educational approaches, we create more effective, robust, and accurate AI systems. 

Imagine you’re starting to learn a new language. Its alphabet is very different from that of your mother tongue, and you find yourself struggling with the pronunciation. To help you along, your new teacher regularly corrects your pronunciation, a process known in education as ‘scaffolding.’ By giving you additional tailored support and feedback in the early stages of your new language learning, the teacher reduces your cognitive load so that you can learn faster and with more ease. 

After a few weeks of learning, you take a mini test which you pass with flying colors. This positive feedback boosts your confidence and your belief in your abilities, enabling you to move forward with more independence. As you gradually learn more and more in your new language, the need to receive pronunciation feedback from your teacher diminishes (the scaffolding is taken away) to the point where you can independently predict new words and sounds. 

The early feedback and correction provided to a computer algorithm by a human in supervised learning functions in very much the same way as the scenario above. Human feedback offers the algorithm a scaffold on which it can build its knowledge base quickly and efficiently. Once the algorithm has received sufficient inputs and corrections, the scaffold is removed and it can make predictions on its own. In other words, human feedback helps manage the cognitive load by focusing corrections on specific errors (like human pronunciation errors), thereby optimizing the learning process. However, the algorithm is not left entirely to its own devices once the scaffold is removed; humans are always needed to update, retrain, and evaluate the performance of the model and to make any necessary adjustments to ensure it adapts to changing conditions and new information. 

Likewise, just as positive reinforcement in humans encourages repeated desirable behavior, feedback in supervised learning reinforces correct predictions. This strengthens the algorithm's "confidence" in these predictions and makes it more likely to repeat similar actions in the future. When an algorithm receives feedback indicating an error, it adjusts its parameters to avoid the same mistake, mirroring negative reinforcement in human behavior.

However, even with the best supervision, algorithms can miss the mark. The terms overfitting and underfitting describe situations where an algorithm either learns the training data too well or doesn’t learn it well enough, respectively. In the case of overfitting, this means that the algorithm performs really well on the training data, but poorly on new data. Underfitting, on the other hand, happens when a model is too simple to capture the underlying patterns in the training data, leading to poor performance on new data. 

Overall, the benefits of supervised learning generally outweigh its disadvantages. It’s a simple process that yields a high level of accuracy in predictions and can be used across a wide range of applications. However, it is important to be aware of some of the drawbacks associated with supervised learning in order to decide whether it’s the best approach for the task at hand.⁷

First, although supervised learning is a type of artificial intelligence it doesn’t completely eradicate the need for human input. Experts and professionals are still needed to provide the supervised learning models with feedback and corrections to enhance their accuracy. And as we all know, experts tend to cost a lot of money. 

Second, while supervised learning undoubtedly saves us time on time-consuming and complex tasks such as medical image processing (see below), the initial training of the model takes a lot of time and effort. Add on top of this the fact that as the size of the data set increases, the time and resources needed to train the model increase exponentially, and you might want to consider other methods of machine learning. 

Finally, echoing the famous phrase ‘you are what you eat,’ a supervised learning model is only as good as what you put into it. In other words, the quality of the model depends heavily on the quality of the labels you give it, with inaccurate or inconsistent labelling significantly degrading the model’s performance.

Spam email filtering 

Spam emails. We all get them, we all hate them, but most of us don’t even get to read them thanks to the automatic filtering done by our email provider. 

But have you ever wondered how your email server categorizes your incoming mail into either legitimate mail or spam? What does it look for? 

Using large datasets of both spam and legitimate emails, supervised learning models can extract relevant information from each type such as the message body, the email subject, and sender information.⁸ The model then learns to identify common characteristics and patterns between these features and the corresponding labels (spam or legitimate). Once the algorithm has been trained, it can use this information to predict new, unseen emails and place them in the correct folder. 

Some of the algorithms used to predict whether an email is spam or not are called probabilistic classifiers; these tools not only tell you which category the email belongs to but the confidence level of this prediction.⁹ These algorithms have some rather fancy and intriguing names such as Naïve Bayes, Random Forest, and Decision Trees (do you sense a nature theme?). 

Medical Image Processing

 Interpreting medical images is crucial for accurately diagnosing a variety of diseases. Professionals such as pathologists, radiologists, physicians, and ophthalmologists depend on these images to diagnose conditions and create new treatment plans. Unfortunately, manual analysis of medical images by humans is often laborious and time-consuming, highlighting the need for more efficient automated techniques. 

Supervised learning has demonstrated remarkable effectiveness in the classification, detection, and segmentation of medical images, achieving performance levels comparable to those of human experts.¹⁰ In addition to saving medical professionals time and effort, automated medical image analysis can help accelerate the diagnostic process; people get treated earlier and, in the case of severe disease, may have a better prognosis.

So how does it work? There are three main tasks in medical image processing: 

Classification: Objects are categorized into groups or types based on specific features. This categorization is either binary (such as a benign or malignant cell) or multi-class (classifying a wound into multiple types or degrees). 

Detection: At this stage, a ‘bounding box’ (literally, a red box) is drawn around the region of objects that are of interest (such as abnormalities).

Segmentation: This is the technique of dividing the image into different parts or regions to isolate specific structures or areas of interest. This process helps identify and analyze distinct anatomical features, such as organs, tissues, or abnormal growths like tumors.

As medical imaging technology evolves and more labeled data becomes available, supervised learning models can be retrained to improve their accuracy and adapt to new types of medical images and conditions. However, this always needs human medical professionals to give the model feedback and corrections on its predictions and to update its training data. 

Artificial Intelligence Models

Supervised learning is just one of several artificial intelligence models that mimic human problem solving and decision-making. Each model has its advantages and disadvantages and is suited to different tasks and situations. Read this article to learn more about each model and some of the ethical controversies surrounding the wider world of artificial intelligence.

   Google Cloud. (n.d.). Supervised Learning. Google.  https://cloud.google.com/discover/what-is-supervised-learning

2.    Cornell University. (2022). Supervised Learning. Cornell CS.https://www.cs.cornell.edu/courses/cs4780/2022fa/lectures/lecturenote01_MLsetup.html

3.    Inveniam. (2023, December 29). A Brief History of Machine Learning. Inveniam. https://www.inveniam.fr/a-brief-history-of-machine-learning

4.    DeJong, G., & Lim, A. H. (2011). Explanation-Based Learning. In Sammut, C., Webb, G. I. (eds.) Encylopedia of Machine Learning. Springer: Boston, MA. https://doi.org/10.1007/978-0-387-30164-8_296

5.    IBM. Deep Blue. IBM Heritage. https://www.ibm.com/history/deep-blue

6.    Google Cloud. (n.d.). What’s the difference between deep learning, machine learning, and artificial intelligence. Google. https://cloud.google.com/discover/deep-learning-vs-machine-learning

7.    Bhavsar, H., & Ganatra, A. (2012). A Comparative Study of Training Algorithms for Supervised Machine Learning. International Journal of Soft Computing and Engineering, 2(4). 

8.    Shinde, S. (2024, February 24). What is Supervised Learning in Machine Learning? A Comprehensive Guide. Emeritus. https://emeritus.org/blog/ai-and-ml-supervised-learning/#:~:text=Image%20and%20speech%20recognition%2C%20recommendation,how%20supervised%20learning%20is%20used.

9.    Toma, T., Hassan, S., & Arifuzzaman. (2021). An Analysis of Supervised Machine Learning Algorithms for Spam Email Detection. 2021 International Conference on Automation, Control and Mechatronics for Industry 4.0 (ACMI), July 2021, Rajshahi, Bangladesh. 

10. Aljuaid, A., & Anwar, M. (2022). Survey of Supervised Learning for Medical Image Processing. SN Computer Science, 3(292).