What do you picture when you think about the future? If you imagine classic sci-fi images of technological advancement—flying cars, robot companions, and the like—you’re not alone. For decades, humans have associated forward progress with technological innovation. Even today, it’s not uncommon to think of computers as humanity’s saviors, finally allowing us to overcome biases, prejudices, and inequities through a gold standard of rationality and objectivity. 

As is the case with many sci-fi fantasies, the reality is much more complicated. While we rapidly advance toward unprecedented achievements in automation and artificial intelligence (AI), it’s more important than ever that we pay attention to how these systems perpetuate human biases and systematically produce harmful and discriminatory outcomes, also known as algorithmic bias.^1,2 These hidden skews can be embedded into many aspects of an algorithm’s design, training, or distribution, shaping who gets to benefit from technological innovation and who bears the cost.³ Can a robot be racist? The answer is: sometimes, yes. 

Algorithm: A step-by-step procedure, consisting of rules, instructions, or calculations, for solving problems and making decisions. Algorithms can be coded explicitly, or, in the case of machine learning, can be trained using large datasets. 

Black Box: Describes how the construction, function, and algorithms of technological and scientific systems are hidden. In the context of AI, the term “black box” not only refers to intentionally keeping an algorithm’s inner workings a secret, but also to how training and development models obscure underlying processes to the technology’s creators themselves.^1,4,5 

Harms of Allocation: Reinforcing existing inequities by unfairly withholding opportunities, services, or resources from marginalized groups.⁶ 

Harms of Representation: Skewed depictions that erase certain social groups or perpetuate negative stereotypes about them, shaping how users perceive and interact with the world around them.⁶ 

Machine Learning: A field within artificial intelligence that trains models to learn patterns from data without explicit programming. While often used interchangeably with AI, machine learning refers specifically to the training techniques used to create some AI systems. 

While algorithms (and by extension, algorithmic bias) are often associated with digital technology and AI today, they’ve actually been around for much longer, predating the earliest computers, electricity, and even paper. Historians can trace algorithms back to ancient Babylonian mathematicians in 2,500 BC, who carved iterative procedures for solving algebraic problems and numeric computations into clay tablets.⁹ 

Algorithms gained new potential near the end of the Industrial Revolution, with Ada Lovelace writing the first machine algorithm in 1840. Fast forward to 1936, Alan Turing invented the Turing machine, a theoretical model of algorithmic computation that laid the groundwork for modern computers. Within this period of innovation, numerous scientists and scholars reshaped the limits of algorithmic thinking, pushing the boundaries of what concepts and problems could be translated into generalized, step-by-step procedures. 

Another turning point for algorithms came a few years later, when John McCarthy coined the term “artificial intelligence” in a 1955 presentation.¹⁰ Widely recognized as the father of AI, McCarthy believed that machines could eventually simulate every aspect of human intelligence. In 1966, renowned MIT computer scientist Joseph Weizenbaum invented ELIZA, the world’s first chatbot.¹¹ An early, extremely simple predecessor for the likes of ChatGPT and Amazon’s Alexa, ELIZA mimicked an intelligent psychotherapist by applying a sequence of rules to generate responses from pre-programmed scripts. 

While Weizenbaum is considered a key figure in the development of AI technology, he is also known as one of AI’s most prominent skeptics.^11,12 His later work ardently criticized the idealistic views of AI held by McCarthy and others, arguing that AI algorithms reinforce systemic inequities rather than dismantling them. Weizenbaum observed that users easily perceived ELIZA as human, and worried that anthropomorphizing computers would create a dangerous illusion of objectivity that hides where (or who) algorithmic biases come from. Despite alienating himself from many of his peers, Weizenbaum was one of the first and most influential voices in highlighting algorithmic bias and how the vessel of AI especially obscures those biases. 

It didn’t take long for theoretical warnings to turn to real-life impacts. In the 1980s, St. George’s Hospital Medical School began using an admissions algorithm developed by Dr. Geoffrey Franglen, who wrote a computer program based on reviewers’ past decisions to make the screening process more efficient.¹³ A few years later, the U.K. Commission for Racial Equality found that the program docked up to 15 points from applicants with non-Caucasian names, and an average of three points from female applicants. Having been validated against human reviewers, the controversy clearly revealed how measuring success against human standards can reproduce human biases.

Over the past decade, attention toward algorithmic bias has skyrocketed. You may have heard some of these headlines yourself: Amazon’s (now discontinued) hiring algorithm boosting resumés from applicants it perceived as male, financial AI algorithms disproportionately denying mortgages to Black applicants, the list goes on.^14,15 In 2016, Dr. Joy Buolamwini went viral for her work exposing racial bias in facial recognition algorithms after she, a dark-skinned Black woman, had to wear a white mask for the facial recognition software to recognize her; the industry-standard program only recognized lighter skin tones, erasing many racial minorities.¹⁶
The global rise of the Black Lives Matter movement in 2020 brought racially biased surveillance and facial recognition technology further into the public spotlight.¹⁷ A bombshell study from the U.S. government revealed racial biases in a majority of facial recognition algorithms, finding higher rates of false positives for Asian and African American faces relative to Caucasian faces.¹⁸ Around this time, Harvard researchers Dr. Trishan Panch, Dr. Heather Mattie, and Dr. Rifat Atun first identified algorithmic bias in healthcare, raising awareness about discriminatory algorithms across fields. Overall, users and developers alike were taking a closer look at the hidden assumptions behind AI algorithms.² 

Scholars and scientists are still working to uncover the full extent of biases in algorithms, AI and otherwise. There are countless researchers spearheading the study of algorithmic bias and ethical AI today, including Joy Buolamwini, Kate Crawford, and Ruha Benjamin, to name a few.¹ While there’s a lot we still don’t know about algorithmic bias, attitudes are shifting; algorithms are increasingly understood as tools that are vulnerable to human biases and error, demanding our attention to ensure those biases aren’t reproduced at unprecedented scales.

Joseph Weizenbaum

The German-American computer scientist who created ELIZA, the world’s first chatbot. Despite his revolutionary contributions to AI, Weizenbaum later became an outspoken critic of AI and its idealistic promises, warning that computers reproduce oppressive power structures behind an illusion of progress, objectivity, and understanding.^11,12

Trishan Panch, Heather Mattie, and Rifat Atun

The team of Harvard researchers who first defined algorithmic bias in the context of healthcare. Dr. Panch and Dr. Mattie are co-directors of a multi-course program on responsible applications of AI in healthcare at the Harvard T.H. Chan School of Public Health.¹⁹ Dr. Atun is a Professor of Global Health Systems and is recognized as one of the world’s most influential and highly cited scientists. His research focuses on the adoption and distribution of innovations in health systems.²⁰

Joy Buolamwini

An AI researcher, best-selling author, artist, and activist whose work centers around algorithmic bias, specifically racial and gender inequities in AI technology. Dr. Buolamwini is perhaps most well-known for her thesis work on biased AI facial recognition technology with the MIT Media Lab, which is the focus of the Emmy-nominated documentary Coded Bias. Dr. Buolamwini also founded the Algorithmic Justice League, a non-profit organization that raises awareness about the social harms of AI and advocates for equitable and accountable AI.²¹ 

Kate Crawford

A leading author, artist, and scholar of AI and its social and material impacts. Named by TIME as one of the 100 Most Influential People in AI in 2023, Dr. Crawford has co-founded multiple research institutes investigating ethical and accountable AI systems.³² Her art, which also tackles the political and environmental dimensions of AI, has been featured in exhibitions across the world, including in the permanent collection of the Museum of Modern Art (MoMA) in New York.

The existence of algorithmic bias doesn’t necessarily mean we should scrap AI entirely—but we do have to reimagine how we build, develop, and use AI systems. It might seem counterintuitive, but algorithmic bias can help us improve these technologies by bringing human bias to the forefront—if we pay attention.

Deconstructing step by step

Using algorithmic bias as a lens can tell us a lot about how machine learning systems are created. Algorithmic bias can arise at many stages in the life cycle of a machine learning system—too many to cover all of them in this article. So, let’s take a brief tour through some key stages of development.

1. Defining the task: Before we even begin to create an algorithm, we need to decide what problem we want to solve—a choice that carries specific assumptions, costs, and benefits.²³ Let’s take facial recognition systems that identify a person’s gender.²² In defining the task, we weave in assumptions about gender binaries and appearance norms. We also design with a goal in mind: why is a gender identification system useful in the first place? This is not to say that these tasks cannot be useful in certain contexts, but being aware of these assumptions is one of the first steps to minimizing bias. 
2. Training the model: Machine learning models are trained on large datasets, where the algorithm “learns” to identify patterns and apply them to new data. Unrepresentative datasets are a huge contributor to algorithmic bias.²² If a facial recognition software is mostly trained on images of white and lighter-skinned people, it makes sense that the resulting model would struggle with darker skin tones—the algorithm can’t learn from material it isn’t given. To address this, some organizations are creating and using more diverse datasets, rather than scraping data from the web (where equal representation is not guaranteed).²² 
3. Evaluating the system: Biases can also emerge when we’re evaluating and refining models after initial training. If we create models to replicate human decisions, we also replicate the biases within those decisions (St. George’s Medical School certainly learned that the hard way). Algorithmic bias shows us that success isn’t a neutral standard; if we want to design systems that surpass human decision-making, we likely need to set standards that surpass humans as well. 

Using algorithms to expose human error

As technological reflections of human assumptions, biases, and decisions, could algorithmic biases help us learn more about ourselves? Oftentimes, it’s easier to pinpoint bias and unfairness when the culprit is an algorithm; we’re generally more reluctant to admit our own biases. Algorithms can also make biases clearer at the systemic level, yielding more outputs that can be compiled, compared, and analyzed for recurring patterns. 

In 2024, researchers at the Boston University Questrom School of Business tested how algorithms change our awareness of bias.²⁴ Participants were asked to rate Airbnb listings and Lyft drivers based on average star rating and the host’s name, some of which were commonly perceived to be “white-sounding” and others “African-American-sounding.” After, they were shown a series of ratings made by an algorithm and were asked to identify any racial biases. However, sometimes these ratings were actually those of the participants, unbeknownst to them. 

The researchers found that participants were more likely to perceive bias when they thought the ratings were decided by an algorithm or even another person. Participants were also more likely to correct an “algorithm’s” bias than their own (again, even if the “algorithm” was really themselves). The researchers theorize this is because we tend to give ourselves the benefit of the doubt, finding some explanation other than bias to explain the outcomes when we’re the ones making the decisions. Consequently, algorithms can actually help us see our own errors more clearly, allowing us to correct those tendencies in ourselves and our decision-making tools. 

It’s hard to overstate the repercussions of algorithmic bias—past, present, and future—especially considering the scale and ongoing expansion of AI and large data systems. But the road towards ethical AI is unclear; to address algorithmic bias, we still need to understand what makes bias in AI and machine learning so much more difficult to uncover, and ask whether “neutral” systems are even possible. 

Into the unknown

Originally coined during World War II as a name for recording devices in aircraft, the term “black box” has now come to describe any system or device whose internal mechanisms are hidden from the user. With black boxes, we can see the system's inputs and outputs, but the “magic,” everything that happens in between, is a mystery.^{1, 4, 5} For example, we can train AI models to distinguish cat pictures from dog pictures; we put in a picture, something happens, and the model outputs a designation, cat or dog. But why is this a problem?
Without access to an algorithm’s logic—the factors considered (or not considered), transformations performed, and equivalences made—it can be incredibly difficult to locate and correct algorithmic bias. This is even more true of machine learning systems, as their training model hides algorithms from the creators themselves. For instance, if a resumé screening program prioritizes mostly male applicants, how do we know what goes into those decisions, if gender bias is involved, or how to fix it? Even explicitly coded algorithms are often hidden from users, classified as proprietary knowledge, limiting options for transparency, fairness, and accountability.⁵ 

The black box problem is still a topic of contention today. Some researchers argue that “opening” the black box is not necessary to advance more equitable AI—we can regulate models while protecting trade secrets by monitoring outputs alone.²⁵ Another common solution for biased machine learning models involves using more representative datasets to minimize bias in the training process, like using face images with diverse skin tones to train facial recognition software.²³ Other researchers remain skeptical, arguing that black boxes reinforce power asymmetries, obscure inequities, and pose critical ethical concerns.^4,5 

We also don’t know if it’s possible to open the machine learning black box, although research continues to bring us closer.²⁶ In an age when AI innovation is outpacing understanding, data remains a hot commodity, and as privacy and ethics come into the spotlight, black boxes may hold the key to a fairer future. 

Magic machines

AI algorithms can seem like magic, essentially giving us super speed and the ability to see into the future (though we’re sadly not quite at teleportation yet). However, AI’s magical qualities might actually make it more dangerous, concealing human decisions behind a veil of ease and detachment. In his early work on ELIZA, Joseph Weizenbaum warned of this very phenomenon; for him, ELIZA’s “intelligence” was simply an illusion that would “dazzle” and fool users. Weizenbaum believed that seeing ELIZA and any computer program as magic would prevent users from critically analyzing its outputs and inner workings, allowing bias to slip past unquestioned. 

Many AI scholars continue to share this sentiment today.²⁷ Whether it be algorithmic bias, invisible labor, or significant environmental impacts, the awe-inspiring nature of AI and its abilities can lead us to forget the very real resources and human actions that went into it. As much as we are drawn to fantasies and mysticism, letting go of the magic might actually bring us closer to fairer and more equitable AI systems.

A bug or a feature?

In addition to practical solutions, algorithmic bias raises many big philosophical and ethical questions. For one, what does a “neutral” technology even look like? Many experts argue there is no such thing, and that some tasks just shouldn’t be done by machine learning algorithms to begin with.^6,24 Across the world, some researchers and policymakers are advocating for strengthened regulatory frameworks that limit AI applications in high-stakes contexts, like determining medical treatments or in criminal justice. The jury’s still out (no pun intended) on whether some tasks should be completely off-limits due to their sensitive nature, or whether we can eventually make our systems fair “enough” to use in any application. 

Pitfalls of predicting recidivism

One of the most widely referenced case studies in algorithmic bias is the Correctional Offender Management Profiling for Alternative Sanctions, also known as COMPAS. Developed in the 1990s, COMPAS is a risk assessment program used in criminal courts across the U.S., designed to predict whether defendants convicted of crimes would re-offend.²⁸ Based on defendant responses to a questionnaire, the algorithm would score Risk of Recidivism and Risk of Violent Recidivism on a scale from one to ten. Judges use these scores as key factors when deciding a defendant’s sentence, with potentially life-changing implications. 

In 2016, journalists from ProPublica published an analysis exposing significant racial biases in COMPAS’s outcomes.²⁹ Comparing COMPAS’s predictions to actual re-offenses, the researchers found that Black defendants were twice as likely to be mislabelled as high risk for violent recidivism than white defendants. Even when controlling for factors such as prior crimes, later actual recidivism, age, and gender, black defendants were 45% more likely to be assigned higher risk scores than white defendants.

This analysis is far from the only critique of COMPAS’s algorithm, which remains black-boxed as a trade secret. Numerous other studies have also yielded evidence of racial discrimination, in addition to age and gender-based discrimination.³⁰ Even further, a 2018 study found that COMPAS predictions were no more accurate than the predictions of regular people without any criminal justice or legal background.

Controversy surrounds the controversy—some analyses and critiques have received pushback from COMPAS’s creators and other academics. Despite these multiple layers of controversy and the high stakes, COMPAS is still used in jurisdictions across the U.S. today. While further research would help clarify the extent and impact of algorithmic bias in COMPAS and similar programs, the lack of transparency surrounding COMPAS’s inner workings raises concerns for many about accountability and fairness. To this day, COMPAS exemplifies many of the complex realities of algorithmic bias, including how difficult it can be to identify and its drastic potential consequences. 

Medical algorithms and costs of care

Healthcare is emerging as a significant playing field for AI applications, and for many good reasons. Healthcare systems around the world have historically struggled with administrative burden, and the implications of data-informed predictions for preventative care are significant. Simultaneously, algorithmic bias in medical applications can be the difference between life and death, potentially hurting our most vulnerable populations. 

In 2019, researchers examined potential disparities in an industry-standard commercial prediction algorithm.³¹ Aimed at supporting resource allocation, the algorithm selects patients with complex health needs to receive additional support, including resources and greater medical attention. The study’s results indicated that the algorithm consistently underestimated the sickness of Black patients compared to white patients, potentially reducing the number of Black patients receiving extra support by half.

Digging deeper, the source of this bias reveals critical design missteps. The algorithm used healthcare cost as a proxy for illness, failing to account for the fact that Black patients have historically had unequal access to healthcare and thus lower amounts have been spent on their care. By assuming that healthcare spending is equal and proportionate to illness severity across demographics, the algorithm replicates racial biases already present in the healthcare system. As a result, Black patients likely continue to receive less medical attention, creating a cycle of inequity. 

AI algorithms at work: How to use AI to help overcome historical biases 

AI and human decision-makers are imperfect, presenting unique strengths and weaknesses when it comes to bias and rationality. In this article, Turney McKee and Ariel LaFayette explore how strategically combining the two can create smarter, fairer futures for all. 

The potential and pitfalls of AI in healthcare 

In recent years, AI has gained significant traction in healthcare applications, with its potential to ease administrative burdens and process more data than any single person could ever dream of. But with great power comes great responsibility; in this article, Sophie Cleff dives deeper into algorithmic bias in healthcare—how it works, who it affects, and ways we can minimize its consequences. 

1. Walker, R., Dillard-Wright, J., & Iradukunda, F. (2023). Algorithmic bias in artificial intelligence is a problem-And the root issue is power. Nursing Outlook, 71(5), 102023. https://doi.org/10.1016/j.outlook.2023.102023
2. Panch, T., Mattie, H., & Atun, R. (2019). Artificial intelligence and algorithmic bias: implications for health systems. Journal of global health, 9(2), 010318. https://doi.org/10.7189/jogh.09.020318
3. OECD (2023), OECD Digital Education Outlook 2023: Towards an Effective Digital Education Ecosystem, OECD Publishing, Paris, https://doi.org/10.1787/c74f03de-en.
4.  Benjamin, R. (2019). Race after technology : abolitionist tools for the new Jim code. Polity. https://www.overdrive.com/search?q=4372806
5. Pasquale, F. (2015). The black box society : the secret algorithms that control money and information. Harvard University Press. https://doi.org/10.4159/harvard.9780674736061
6. Crawford, K. (2017). The Trouble with Bias. NeurIPS. 
7. Langston, J. (2015). Who’s a CEO? google image results can shift gender biases. UW News. https://www.washington.edu/news/2015/04/09/whos-a-ceo-google-image-results-can-shift-gender-biases/ 
8. boyd, danah, & Elish, M. C. (2018, November 26). Don’t believe every AI you see. New America. https://www.newamerica.org/pit/blog/dont-believe-every-ai-you-see/ 
9. Barbin, É. (1999). A history of algorithms: from the pebble to the microchip (Vol. 23). J. L. Chabert (Ed.). Berlin: Springer.
10. Mucci, T. (n.d.). The history of Artificial Intelligence. IBM. https://www.ibm.com/think/topics/history-of-artificial-intelligence 
11. Tarnoff, B. (2023, July 25). Weizenbaum’s nightmares: How the inventor of the first chatbot turned against AI. The Guardian. https://www.theguardian.com/technology/2023/jul/25/joseph-weizenbaum-inventor-eliza-chatbot-turned-against-artificial-intelligence-ai
12. Loeb, Z. (2021). The lamp and the lighthouse: Joseph Weizenbaum, contextualizing the critic. Interdisciplinary Science Reviews, 46(1-2), 19–35. https://doi.org/10.1080/03080188.2020.1840218
13. Schwartz, O. (2019, April 15). Untold history of AI: Algorithmic bias was born in the 1980s. IEEE Spectrum. https://spectrum.ieee.org/untold-history-of-ai-the-birth-of-machine-bias 
14. Dastin, J. (2018, October). Insight - Amazon scraps secret AI recruiting tool that showed bias against women. Reuters. https://www.reuters.com/article/idUSKCN1MK0AG/ 
15. Martinez, E., & Kirchner, L. (2021, August 25). The secret bias hidden in mortgage-approval algorithms. AP News. https://apnews.com/article/lifestyle-technology-business-race-and-ethnicity-mortgages-2d3d40d5751f933a88c1e17063657586 
16. Buolamwini, J. (2016). How I’m fighting bias in algorithms. TED. https://www.ted.com/talks/joy_buolamwini_how_i_m_fighting_bias_in_algorithms?language=en 
17. Fowler, G. (2020). Black Lives Matter could change facial recognition forever — if Big Tech doesn’t stand in the way. The Washington Post. https://www.washingtonpost.com/technology/2020/06/12/facial-recognition-ban/ 
18. National Institute of Standards and Technology. (2019, December 19). NIST study evaluates effects of race, age, sex on face recognition software. NIST. https://www.nist.gov/news-events/news/2019/12/nist-study-evaluates-effects-race-age-sex-face-recognition-software 
19. Mitchell, T. (2021, March). Algorithmic bias in health care exacerbates social inequities-how to prevent it. Harvard T.H. Chan School of Public Health. https://hsph.harvard.edu/exec-ed/news/algorithmic-bias-in-health-care-exacerbates-social-inequities-how-to-prevent-it/ 
20. Rifat Atun. Harvard T.H. Chan School of Public Health. (n.d.). https://hsph.harvard.edu/profile/rifat-atun/ 
21. Buolamwini, J. (n.d.). About. Poet of Code. https://poetofcode.com/about/ 
22. Cramer, H., Holstein, K., Vaughan, J., III, H., Dudík, M., Wallach, H., Reddy, S., Garcia-Gathright, J. (2019). Translation Tutorial: Challenges of incorporating algorithmic fairness into industry practice.. FAT* 2019 . 
23. Heilweil, R. (2020, February 18). Why algorithms can be racist and sexist. Vox. https://www.vox.com/recode/2020/2/18/21121286/algorithms-bias-discrimination-facial-recognition-transparency 
24. Cao, J. (2024, April 8). Can the bias in algorithms help us see our own?. Boston University. https://www.bu.edu/articles/2024/can-the-bias-in-algorithms-help-us-see-our-own/ 
25. Bembeneck, E., Nissan, R., & Obermeyer, Z. (2022, March 9). To stop algorithmic bias, we first have to define it. Brookings. https://www.brookings.edu/articles/to-stop-algorithmic-bias-we-first-have-to-define-it/ 
26. Perrigo, B. (2024, May 21). Artificial Intelligence is a “black box.” maybe not for long. Time. https://time.com/6980210/anthropic-interpretability-ai-safety-research/ 
27. Elish, M. c. (2018, January 17). Don’t Call AI Magic. Medium. https://medium.com/datasociety-points/dont-call-ai-magic-142da16db408 
28. Taylor, A. (n.d.). Data and discretion: Why we should exercise caution around using the Compas algorithm in court. Stanford Rewired. https://stanfordrewired.com/post/data-and-discretion 
29. Larson, J., Angwin, J., Kirchner, L., & Mattu, S. (2016, May 23). How we analyzed the compas recidivism algorithm. ProPublica. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm 
30. Engel, C., Linhardt, L., & Schubert, M. (2024). Code is law: how COMPAS affects the way the judiciary handles the risk of recidivism Code is law: how COMPAS affects the way the judiciary.. Artificial Intelligence and Law, 33(2), 383–404. https://doi.org/10.1007/s10506-024-09389-8
31. Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science (New York, N.Y.), 366(6464), 447–453. https://doi.org/10.1126/science.aax2342
32. Kate Crawford. (n.d.). https://katecrawford.net/