Generative Adversarial Networks

Have you ever seen the show Is It Cake? If not, the entire three seasons can be explained as follows: host Mikey Day presents a panel of celebrity guest judges with a collection of objects—perhaps some shoes, rubber ducks, or bananas—and asks them to identify which is not like the others. Namely, which one is secretly a cleverly disguised cake? Behind the scenes, expert bakers compete for a cash prize, toiling for hours to create a hyperrealistic cake replica of an object that could easily fool someone into thinking it was the real deal. Judges, on the other hand, are motivated to suss out the clandestine cakes. 

Generative Adversarial Networks (GANs) are a type of machine learning framework, which, although more complex than this reality show gameplay, work in a somewhat similar way. Though the analogy is a major oversimplification, the core ideas are the same: in the game show, there is a baker and a judge, and in a GAN, there is a generator and a discriminator. The generator is like the baker, trying to create a convincing copy of something. In this case, the baker is creating fake items that are actually cakes, but a generator in an artificial neural network might generate fake images, hoping to make them look real. 

The discriminator is like the judge, who is an expert at telling what is cake and what isn’t (perhaps they’ve watched a lot of episodes of the show in preparation). In the GAN, the discriminator evaluates both real images from the training dataset and the fake images from the generator and tries to tell them apart. If it’s able to tell the difference, the discriminator gives feedback to the generator, helping it improve. 

If the judge guesses wrong, the baker has clearly done well, and gets to advance in the competition. If not, then the baker knows they have things to work on, as their cake replica wasn’t too convincing. Similarly, when the generator manages to fool the discriminator, it signifies that no change is needed to the model parameters, and it gets to proceed as is, whereas the discriminator will be penalized with significant updates, and vice versa when the generator loses.

Just as the baker and judge may continue to go through multiple rounds, learning from each other about how to discern real from cake, the generator and discriminator continuously play against each other to figure out what is real or fake data. The generator tries to fool the discriminator, and the discriminator tries to catch the generator. As they keep playing, they both get better and better. The discriminator gives feedback to the generator, which helps it improve its fakes. The process repeats until the training process has successfully converged, at which point the discriminator can no longer tell the difference between the real and fake data.¹ As the Is it Cake? bakers continue to hone their skills, the judges soon won’t stand a chance, and fooling them will be, well, a piece of cake.

Generator: In the context of AI image generation, this refers to the neural network within a Generative Adversarial Network (GAN) that is responsible for creating new, fake images based on the data it has been trained on. Essentially, it generates the "fake" part of the GAN system.¹ 

Discriminator: The network in the GAN that tells the difference between real data from the training set and fake data created by the generator. Through a feedback loop, the discriminator improves its ability to classify real vs. fake data, while the generator adapts to fool the discriminator.¹

Deep Learning: A subset of machine learning containing multilayered neural networks, structured similarly to the human brain. Just as humans learn from experience, AI systems use deep learning to analyze vast amounts of data to develop effective algorithms.

Machine Learning: A subset of artificial intelligence that uses statistical techniques to enable machines to learn from data and improve over time, loosely based on human learning. By analyzing data and using trial and error, ML algorithms can generate accurate predictions, content, recommendations, and decisions without explicitly programmed instructions.

Algorithms: Algorithms are step-by-step procedures or mathematical models used to process data and generate outcomes, such as predictions or classifications. In machine learning, algorithms learn patterns from data to improve performance on a given task, such as linear and logistic regression, decision trees and random forests, neural networks, and so on.
Neural Network: An interconnected, artificial model designed to mimic how neurons in the human brain interact with each other. It connects artificial nodes with edges, which resemble the way synapses in the brain connect neurons.

The history of machine learning frameworks like Generative Adversarial Networks dates back to the mid-20th century, when early researchers in the field of artificial intelligence sought to create machines that could simulate human cognition. In the 1950s and 60s, early AI models relied heavily on rule-based systems, which often followed explicit if-then statements, but these more simplistic strategies struggled with complex tasks that required adaptability and pattern recognition. 

The advent of machine learning in the 1980s and 90s marked a significant shift, as researchers began developing algorithms that could learn from experience rather than relying on pre-programmed rules. This type of AI allows a machine to learn from its training data, meaning that it’s possible for the computer to improve task performance without direct human intervention. An early breakthrough of deep learning, a subset of machine learning, was computer scientist Yann LeCun’s convolutional neural network that could recognize handwritten digits.
Deep learning continued to gain traction in the 2000s with advances in neural networks. Researchers like Yann LeCun, Geoffrey Hinton, and Yoshua Bengio—the so-called “Godfathers of AI”—refined techniques like backpropagation, advancing the capabilities of neural networks and enabling AI models to process vast amounts of data more efficiently. However, before GANs, early generative AI models relied on computationally demanding techniques that were often expensive and struggled with producing high-quality, realistic outputs. The models frequently produced blurry images or suffered from a lack of creative variation, so the challenge became finding a way to train AI to generate data that wasn’t just an approximation but truly indistinguishable from real-world examples. 

It was within this context that Generative Adversarial Networks were conceptually introduced in 2010, with data analyst Olli Niemitalo's blog. He proposed the idea of what would later be known as a conditional GAN, a type of neural network that uses data labels, though his exact model was never implemented. Around 2013, researchers developed an idea related to GANs to model animal behavior. They found that they could use dueling neural networks, in combination with historical animal study data, to predict future animal behavior, identify animals, and isolate relevant behavior patterns to better understand animal health.^1,4 

In 2014, American computer scientist Ian Goodfellow and his colleagues formalized the Generative Adversarial Network framework, defining a loss function and training procedure that enabled the stable learning of generative models. In the next few years, our understanding of GANs expanded, as did the quality of generators and discriminators. This included improvements to generating images with realistic textures and creating higher-resolution images, which was key for creating even more realistic outputs. Researchers elaborated on the theory and techniques of GANs. As scientists continued to study deep learning and neural networks, GANs were applied for more common-use practices, like facial recognition technology. In the last decade, many overviews of GANs have been published, further expanding our understanding of them and their applications.⁵ 

Today, Generative Adversarial Networks are integrated into countless tools, and you may have even used the technology without realizing it. From AI image generation tools like MidJourney or DALL-E to Photoshop's "Generative Fill,” GANs now excel at realistic image, video, and even music generation. These improvements in both generation and recognition have revolutionized not only how we create and engage with media but also transformed vital sectors like medical imaging and fraud detection. With so many groundbreaking applications, it’s hard to predict exactly how GANs will continue to shape our world in the future!

Olli Niemitalo

A European data scientist who is credited with the first original documentation of the idea for generative adversarial networks in his blog post in 2010.⁵  

Ian Goodfellow

An American computer scientist and the creator of GANs. He introduced the concept of GANs and their computational architecture in 2014 in his groundbreaking paper titled "Generative Adversarial Nets." His work has had a profound impact on deep learning and generative models.⁵ 

Yann LeCun

A researcher in the field of deep learning and neural networks, LeCun did not directly create GANs, but his foundational work on neural networks and unsupervised learning has contributed significantly to the development of models like GANs. He is also one of the pioneers in the field of AI and has been a driving force behind much of the progress in deep learning.²

Generative Adversarial Networks have revolutionized how we use the internet, the way we use data, how we understand art and music, and even the capacity of medical professionals to interpret results and conduct in-depth research. 

Image Generation and Improving Image Resolution 

Have you ever taken a photo, then looked back at it later and thought, “Dang, I wish that wasn’t so blurry” or, “This would be a great photo if I hadn’t accidentally gotten my thumb in the corner”? GANs can greatly improve image resolution by filling in details in blurry areas of an image, using what they’ve learned from their training about how similar photos look and filling in the blanks. You might have seen smartphone features advertised (or tried them yourself) that allow you to remove people from the background of photos, and seamlessly generate a new backdrop that makes it look like those people (or your thumb) were never even there.

But improving existing photos isn’t the end of their capabilities—GANs can create brand-new images that look like real photographs, including images of human faces. Although this introduces the possibility for misuse, the ability to create images of people who don’t actually exist can be beneficial for social science research. In behavioral science, GAN-generated images allow researchers to study human perception and emotional responses without the ethical concerns tied to real human subjects. For example, psychologists can use synthetic faces to study biases in facial recognition or the emotional impact of certain features, all without relying on real individuals.⁶

Teams in industries like law enforcement, the military, and emergency response sometimes use training simulations or tools like the Implicit Association Test (IAT) to better understand their biases and practice responses to various real-world situations in a controlled environment.⁷ These artificially generated images can be used to create diverse and inclusive datasets for such tests. Traditional datasets for most AI suffer from biases, leading to the underrepresentation of certain demographics. Thus, if GANs can generate synthetic faces and images that better reflect a broad spectrum of humanity, these images can be used in all sorts of training sets, helping to correct imbalances in AI training data for applications in everything from facial recognition to hiring algorithms.⁶ However, it’s important to note that the generation capacity of GANs is limited by their training datasets, which are impacted by the same biases, underscoring the challenge of trying to construct technology free from human bias. 

Generating New Music

Just as GANs can be used to generate images, they can also be used to create new music. When given a database of real songs, the system can study the different types of music and “learn” the qualities of arrangements that humans produce. Think of the generator as a musical composer (maybe an amateur who’s still learning what sounds good). The composer’s job is to create new music, which starts with random noise. The generator essentially throws out random sounds and tries to turn them into a song, using what it has learned from real music to compose something new. If the generator has enough musical styles to draw on, the music they create can be pieces of all different styles, emulating all sorts of emotions and tones.

Now, think of the discriminator as a music critic. The critic listens to both the real songs from the training data and the songs created by the generator. The critic’s job is to tell which songs are real and which are fake (a real critic would also probably try to decide which are good).

Together, the composer/generator tries to fool the music critic/discriminator by creating music that sounds like it could have been written by a human composer. The discriminator then gives feedback to the generator by pointing out what sounds fake, and the generator then uses this feedback to improve its music, making it sound more and more like the real thing. Over time, both the generator and discriminator work in harmony to get better at their jobs. The generator learns to create music that’s hard to distinguish from music produced by real people. GANs can even be used to improve existing music, filling in missing parts of a song or creating variations on a theme. Older music can be revived too, enhancing sounds or removing background noises that were picked up by recording equipment.⁸ 

Revolutionizing Medical Imaging 

The power of GANs to create and improve images extends far beyond creating fake faces or improving the quality of your Instagram posts. GANs can boost the quality of medical images by enhancing resolution and filling in missing details on scans like MRIs, X-rays, and CT scans, which can often be low-resolution. Sharpening these potentially blurry images using GAN-based image super-resolution methods can help doctors detect small tumors, lesions, or tears that might be missed in lower-quality images.⁹ 

Just as most general training datasets are often limited by their gender and racial makeup, medical datasets are often limited due to privacy concerns, ethical constraints, and patient data scarcity. GANs can generate synthetic medical images that maintain statistical properties of real images while ensuring patient privacy, giving systems even more to train on.9 This could not only help AI models learn how to detect diseases without having to rely on real patient data, but also create purposefully diverse datasets to improve AI fairness and reduce bias.

Although GAN technology has helped many industries, in the wrong hands, Generative Adversarial Networks create dangerous opportunities for political manipulation, targeted abuse, and financial fraud. As the technology evolves, so must countermeasures to protect consumers and businesses. Additionally, GANs face challenges in their development, sometimes running into roadblocks in the training process. 

Training Barriers

GANs can be notoriously difficult and unstable in their optimization process. The first problem that often arises relates to convergence. Remember that GANs involve two neural networks—the generator and the discriminator—competing against each other. Ideally, the generator improves over time by fooling the discriminator, and the discriminator becomes better at distinguishing real from fake data. However, this doesn’t always result in smooth, stable convergence, because the training process is delicate. If one player becomes too powerful or too weak compared to the other, convergence can be hindered.⁵ For example, if the discriminator becomes too good early on, the generator won’t receive meaningful feedback and will struggle to improve. Conversely, if the generator dominates, the discriminator might never learn effectively.

The other main problem is mode collapse, which happens when the generator produces a very limited variety of outputs, effectively "collapsing" to a few specific patterns or outputs. So, if the generator starts to figure out what’s working well, it might start producing more similar outputs. In the analogy of the cake game show, perhaps the baker realizes that contestants have a hard time figuring out what’s actually cake when they make cakes that look like fruit, so they start baking apples, blueberries, and so on. Mode collapse usually results in poor sample diversity, as the generator fails to capture the full data distribution, leading to a lack of variety in the generated outputs.⁵

Lastly, discriminator dominance happens when the discriminator becomes too good at distinguishing between real and fake data early on in the training process. If the discriminator's loss quickly converges toward zero, the generator has no signal to learn from. As a result, the generator doesn't receive any meaningful updates, essentially not gaining any feedback, preventing it from improving. If the discriminator is too powerful, it can make the learning process for the generator unstable or even stagnant. When the generator can barely improve, the training process can either stall or produce low-quality results.⁵

Fake Photos and Videos

Generative Adversarial Networks can be used to create realistic-looking fake photos and videos. While this can be a great thing, there’s also the possibility of detrimental applications. Any technology with the capability of producing convincing falsified images has the potential to be used to generate or spread misinformation. Already, we’ve seen tons of fake images and even videos of politicians in unlikely situations, shaking hands with everyone from major celebrities to heads of terrorist organizations. You’ve probably seen or read about the deep fakes that try to spread conspiracy theories like the idea that the moon landing never happened (or that the moon is made of cheese). While these types of images used to be created as a joke, as it was generally quite obvious when images were created artificially, GAN technology has improved so drastically that it can be almost impossible to tell what’s real.⁶ Especially when the stakes are as high as understanding the truth about a president or prime minister’s actions, the regulation and security around GAN-generated images and videos are incredibly important. 

With access to increasingly sophisticated image-generation technology based on GANs, it seems inevitable that people would use these tools to create pornographic images. In addition to all of the inaccurate or misleading images that can be used to interfere with political issues, explicit photos can have major ramifications on people’s personal lives. Although photos can be generated of anyone, fake explicit content with a real person’s face or voice (sometimes known as “revenge porn” or “sexual photoshopping”) disproportionately affects women, with recent studies finding that somewhere between 60-90% of content victimizes women.¹⁰ Men are also more likely than women to perpetrate image-based abuse, and so although deepfakes can be harmful to anyone, it’s important to emphasize the gendered nature of this type of violence.¹⁰ 

Financial Fraud

GANs have also introduced new risks in financial fraud, making identity theft, deepfake scams, and financial document forgery more sophisticated than ever. Imagine picking up the phone and hearing your parent or child’s voice saying they’re stuck in prison and need you to transfer them money to get them out. Scammers can use GANs to generate realistic synthetic voices by training on a person's speech patterns and trick people in fake phone calls. In 2019, fraudsters used AI-generated deepfake voices to impersonate a CEO and tricked a UK-based energy firm into wiring $243,000 to a fake supplier.¹¹

GANs can also create highly realistic synthetic identity documents like passports, driver’s licenses, and financial records. Criminals can then use these to apply for credit cards and loans that won’t be repaid, set up money laundering schemes with fake businesses and transactions, and bypass bank verifications. GANs can even be trained to generate fake financial reports or entire trading patterns. This means that fraudsters can manipulate the stock market by spreading false financial reports, alter balance sheets and audit trails to hide financial mismanagement, or commit insurance fraud using fake claim documents that look completely legitimate. In the cryptocurrency industry, scammers have used AI-generated whitepapers, fake investment performance reports, and deepfake promotional videos of fake CEOs to attract investors into fraudulent schemes, often scamming people out of significant amounts of money.¹¹ 

As preventative measures, many banks are now using AI to detect anomalies in transactions and biometric verification systems. Some governments are working on stricter regulations for AI-generated identity verification, and advances in AI forensics are helping spot manipulated audio, video, and documents.

Detecting Fake Signatures & Identity Fraud

Because Generative Adversarial Networks have been used to both attack and defend security systems, it’s unsurprising that there is a market for GAN-powered fraud detection. On the one hand, attackers have been known to generate highly realistic forged signatures that mimic a specific person’s handwriting. But if a GAN can generate a convincing forgery, the discriminator can also work to detect what’s legitimate and what’s not. 

For instance, systems might learn to look for fine-grain details that would otherwise be invisible to the untrained human eye, such as signature pressure sensitivity and stroke patterns, making it much harder for human fraudsters to bypass AI security checks. Some researchers have used GANs to create synthetic ID photos that appear completely real but don’t correspond to actual people, and then have used these images to train GANs to detect identity theft or deepfake-based fraud in banking. Banks and cybersecurity firms can thus use GANs to cultivate better AI-driven security systems.¹² 

AI-Painted Portrait Sold for $432,500

In 2018, a portrait called "Edmond de Belamy" was auctioned at Christie’s for $432,500, which far exceeded its original $7,000 estimate. Some might balk at paying this much for a relatively simple portrait, but those in the art world may not be too surprised… until they find out that the art was actually created by Obvious, a collective of French artists using AI. The portrait was one of the first major sales of AI-generated art. The French artists had trained a GAN on a dataset of 15,000 classical portraits from the 14th to 20th centuries.¹³

In their GAN, the generator was the AI artist, creating images that resembled historical portraits, while the discriminator was essentially an AI critic, evaluating whether the images looked like real or artificially generated paintings. The end result may challenge the very definition of art: who is the real artist, after all? The AI, the coders, or the dataset’s original artists? This work is likely just the beginning of significant AI artworks, because although some traditional artists reject AI-generated art, others have embraced it as a new medium, going so far as to use it as a central tool in their creative processes.¹⁴ It’s worth asking the many ethical and philosophical questions that arise: since the AI was trained on historical artwork, does it infringe on copyright? Can AI truly be creative, or is it just mimicking patterns, as a good GAN does?

Ethical AI 

If you’re interested in the many potential repercussions of artificial intelligence, both good and bad, this article delves into what it means to create ethical AI, the history of the concept, and some major case studies in the field. 

Neural Networks & Recurrent Neural Networks  

To understand artificial intelligence more deeply, it may help to dive into the definitions of recurrent and general neural networks, which are loosely modeled after the structure of the human brain. This subset of machine learning teaches computers to analyze data and can identify issues, compare options, and draw conclusions.

1. Creswell, A., White, T., Dumoulin, V., Arulkumaran, K., Sengupta, B., & Bharath, A.A. (2017). Generative Adversarial Networks: An Overview. IEEE Signal Processing Magazine, 35, 53-65. 
2. Giles, M. (2018, February 21). The GANfather: The man who’s given machines the gift of imagination. MIT Technology Review. https://www.technologyreview.com/2018/02/21/145289/the-ganfather-the-man-whos-given-machines-the-gift-of-imagination/
3. Chakraborty, Tanujit & K S, Ujjwal Reddy & Naik, Shraddha & Panja, Madhurima & Manvitha, Bayapureddy. (2024). Ten Years of Generative Adversarial Nets (GANs): A survey of the state-of-the-art. Machine Learning: Science and Technology. 5. 10.1088/2632-2153/ad1f77.  
4. Aggarwal, Alankrita & Mittal, Mamta & Battineni, Gopi. (2021). Generative adversarial network: An overview of theory and applications. International Journal of Information Management. 100004. 10.1016/j.jjimei.2020.100004. 
5. Asimopoulos, D. C., Nitsiou, M., Lazaridis, L., & Fragulis, G. F. (2021). Generative Adversarial Networks: a systematic review and applications. Department of Electrical and Computer Engineering, University of Western Macedonia. 
6. Bharath, K. (2024, October 17). Face generation with GANs. AI/ML Neural Network Computer Vision. https://www.digitalocean.com/community/tutorials/face-generation-with-dcgans 
7. Nosek, Brian & Greenwald, Anthony & Banaji, Mahzarin. (2005). Understanding and Using the Implicit Association Test: II. Method Variables and Construct Validity. Personality & social psychology bulletin. 31. 166-80. 10.1177/0146167204271418.  
8. Deupa, Abhishek & Saha, Garima & Sharma, Nikhil & Singh, Virendra. (2024). Generative Adversarial Network Based Music Generation. Far Western Review. 2. 1-25. 10.3126/fwr.v2i1.70491.  
9. Kazeminia, S., Baur, C., Kuijper, A., van Ginneken, B., Navab, N., Albarqouni, S., & Mukhopadhyay, A. (2020). GANs for medical image analysis. Artificial Intelligence in Medicine, 109, 101938. https://doi.org/10.1016/j.artmed.2020.101938 
10. Venema, A. E. (2024). Deepfakes as a security issue: Why gender matters. Women in International Security. 
11. Damiani, J. (2019, September 3). A voice deepfake was used to scam a CEO out of $243,000. Forbes. https://www.forbes.com
12. Bird, Jordan & Naser, Abdallah & Lotfi, Ahmad. (2023). Writer-independent signature verification; Evaluation of robotic and generative adversarial attacks. Information Sciences. 170-181. 10.1016/j.ins.2023.03.029.  
13. Christie’s Auction House (2018). AI Artwork Sells for $432,500. Christie’s Official Site
14. Gros, T. (2019). Can artificial intelligence create art? [Master’s thesis, HEC Paris]. ResearchGate. https://doi.org/10.13140/RG.2.2.10238.41287