brain with stethoscope

The potential and pitfalls of AI in healthcare

read time - icon

0 min read

Aug 06, 2024

Artificial intelligence (AI) is making waves in all kinds of sectors, and healthcare is no exception. The use of algorithms and chatbots in medicine holds immense promise, from easing the burden on healthcare workers to improving patient outcomes and accessibility. However, the path to fully realizing this potential is paved with serious equity considerations that cannot be ignored. This piece delves into both the promise and the challenges of AI in healthcare, underscoring the importance of ethical and equitable implementation.

The Potential 

Helping healthcare workers

Hospitals are facing a growing workforce crisis, with many healthcare professionals facing burnout. There are many different causes, but one important factor that is rarely taken into account is the administrative workload of physicians and nurses (such as charting, uploading electronic medical records, and tracking down lab results). All of these tasks can be overwhelming and leave physicians with less time to actually care for patients. This impacts not only healthcare workers’ job satisfaction but also the quality of care patients receive.1

The strategic use of AI technology has the potential to alleviate this burden by streamlining administrative tasks, thus improving the quality of life for healthcare workers and freeing up more time for them to actually practice medicine. While AI may not yet be powerful enough to understand all the nuances a trained physician can, it can reliably automate or semi-automate chart review by summarizing patient information or generating clinical reports.1,2,3 This not only reduces administrative workload but can also lower costs for patients and improve accuracy in billing, which can help prevent issues like insurance fraud.3

Improving patient outcomes

But AI’s potential extends well beyond administrative work—it has the potential to improve healthcare outcomes.4 For instance, its ability to interpret large amounts of data across many parameters can aid in making diagnoses. One AI model has even demonstrated an ability to calculate risk for breast cancer that is superior to traditional methods.4,5 

The ability to synthesize large amounts of data can also be useful in designing treatments that are more tailored to individual patients, which could improve health-related outcomes and allow for more efficient use of resources. AI has even outperformed human doctors in treatment design when trained on a specialized condition.6 These advancements suggest a future where AI could enhance diagnostic and treatment processes beyond current human capabilities.

Granting greater accessibility

People around the world struggle to access healthcare. AI, particularly chatbots, promises greater accessibility by providing real-time information and advice. This is especially valuable for people in healthcare deserts where access to medical professionals is limited.7,8 Chatbots could also support patients who may be reluctant to seek medical advice for a stigmatized condition such as an STD, substance abuse, or mental health issues.8,9,10 Check out some of TDL’s previous research on technology to deliver mental health services, including a personalized mental health app and a digital mental health platform for students.

The prohibitive cost of healthcare is an additional barrier to many people seeking help. AI can reduce healthcare costs by replacing expensive imaging techniques and private consultations or eliminating the need to travel to a medical facility.11 While AI is not a replacement for human doctors, it could serve as a valuable supplement to lower barriers to healthcare access.

The Pitfalls 

Bias in, bias out

Although AI in healthcare holds lots of promise, there are warning signs that discourage implementation in its current state. For one, AI has to be trained on existing healthcare data to learn how to make decisions. In the instance of breast cancer diagnosis, an AI might be shown real mammograms along with the patient diagnosis to understand the relationship between the two and translate that into predictions for future data. 

But, if the dataset that an AI is trained on is biased, the AI will make biased decisions. A dataset might be biased because it does not include data from a balanced sample of participants (e.g., men and women) or because the results of the data have been based on biased human decision-making.12,13 For demographic groups who historically and currently experience worse healthcare outcomes as a result of bias, such as gender or racial minorities, the training of AI on past care decisions could mean the reproduction of—or at worst, amplification of—existing systematic disparities.14,15,16,17

This issue is not unique to machine learning. Prediction methods and guidelines in medicine have historically been built on health data collected disproportionately from white people and men. This often means that these methods are less accurate for women and people of color.18,19 For AI to be beneficial for everyone, it needs to be able to predict reality—not just replicate biased human decision-making.

Data privacy concerns

Another significant issue that plagues the entire AI industry is data privacy. Many AI technologies are owned by private companies, and the implementation of AI in healthcare would require access to patients’ private health information.20 This raises concerns about privacy breaches or potential conflicts of interest between private companies and patients. Ensuring the protection of sensitive health data is crucial as AI becomes more integrated into healthcare.

Reality check

Some machine learning algorithms have already been implemented in healthcare and there is evidence that the consequences are not just theoretical. A landmark study demonstrated that an algorithm used on millions of people a year in the U.S. to identify which patients would benefit the most from an extensive care program was racially biased. The algorithm used dollars spent on healthcare as a proxy for illness severity, leading it to conclude that white patients (who spent more on their healthcare) were sicker and in need of more care than Black patients, even if their risk scores were actually the same.12

The Solutions

Testing is best

The pitfalls of AI in healthcare don’t mean it should be abandoned. Rigorous testing can compare AI's decisions with human decision-making and real outcomes to ensure accuracy and minimize any bias.16  Software to test the accuracy and bias of algorithms already exists and more are in development. Consistent use of these tools could help prevent premature implementation of a biased AI.21

Regulation and legislation

Legislation often struggles to keep pace with innovation and AI in healthcare is no exception. Developing regulations and standards that promote equity in AI development and use in healthcare is essential.22 These regulations could include penalties for companies who develop or use AI that fails to meet an ethical standard to encourage accountability.18

Regulations also need to balance the inclusion of relevant sociodemographic factors like gender, race, or income in datasets used by AI, while preventing the misuse of this data and protecting individual privacy.18 In general, legislation needs to encourage the protection of patient data and the need for consent.23

Meaningful collaboration

Bias in AI is not limited to training datasets; it litters the entire tech space, influencing who in society has a say in creating machine learning algorithms and what tools will be implemented in our healthcare system. Collaborations among private developers, the public sector, academia, and citizen co-designers are necessary to build equitable AI systems and ensure that these technologies serve everyone fairly.18

The bottom line

Anyone working in healthcare takes an oath to minimize harm; preventing the perpetuation of existing inequities with equitable and accurate AI can be part of that. But equity cannot be an afterthought—it is not good enough to implement biased technology and say that it will be fixed later.22 New technologies must be designed and implemented with ethics and human rights as central considerations, and with AI in healthcare, we have a golden opportunity to get it right the first time.24

References

  1. Scheinker, D., & Bohn, R. E. (2022). Improving Healthcare Productivity by Using Technology Strategically. Health Management, Policy, and Innovation. https://hmpi.org/2022/10/07/improving-healthcare-productivity-by-using-technology-strategically/
  2. Agatstein, K. (2023, December). Chart Review Is Dead; long live chart review: How artificial intelligence will make human review of medical records obsolete, One day. Population health management. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10698762/
  3. The benefits of AI in Healthcare. IBM. (2023, July 11). https://www.ibm.com/blog/the-benefits-of-ai-in-healthcare/
  4. AI for Health Care: Concepts and Applications. Harvard. (n.d.). https://www.hsph.harvard.edu/ecpe/programs/ai-for-health-care-concepts-and-applications/?gclid=cjwkcajwx46tbhbheiwara_djpivzt43u3rvveax4p7j3gjz4rknn-99z-4adr_3cniltlqubnkrdbocqi0qavd_bwe
  5. Young, L. (2021, September 30). AI can predict cancer risk through mammograms. University of Hawaiʻi at Mānoa. https://manoa.hawaii.edu/news/article.php?aId=11568
  6. Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C., & Faisal, A. A. (2018, October 22). The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care. Nature News. https://www.nature.com/articles/s41591-018-0213-5
  7. Potts, C., Ennis, E., Bond, R. B., Mulvenna, M. D., McTear, M. F., Boyd, K., Broderick, T., Malcolm, M., Kuosmanen, L., Nieminen, H., Vartiainen, A. K., Kostenius, C., Cahill, B., Vakaloudis, A., McConvey, G., & O’Neill, S. (2021). Chatbots to support mental wellbeing of people living in rural areas: Can user groups contribute to co-design?. Journal of technology in behavioral science. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8450556/
  8. Benavides-Vaello, S., Strode, A., & Sheeran, B. C. (2013, January). Using technology in the delivery of Mental Health and Substance Abuse Treatment in rural communities: A Review. The journal of behavioral health services & research. https://pubmed.ncbi.nlm.nih.gov/23093443/
  9. Habicht, J., Viswanathan, S., Carrington, B., Hauser, T. U., Harper, R., & Rollwage, M. (2024, February 5). Closing the accessibility gap to mental health treatment with a personalized self-referral chatbot. Nature News. https://www.nature.com/articles/s41591-023-02766-x
  10. Branley-Bell, D., Brown, R., Coventry, L., & Sillence, E. (2023, September 11). Chatbots for embarrassing and stigmatizing conditions: Could chatbots encourage users to seek medical advice?. Frontiers. https://www.frontiersin.org/articles/10.3389/fcomm.2023.1275127/full
  11. Cutler, D. M. (2023, July 6). What artificial intelligence means for health care. JAMA Health Forum. https://jamanetwork.com/journals/jama-health-forum/fullarticle/2807176
  12. Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019, October). Dissecting racial bias in an algorithm used to manage the health of populations | science. Science. https://www.science.org/doi/10.1126/science.aax2342
  13. Seyyed-Kalantari, L., Zhang, H., McDermott, M. B. A., Chen, I. Y., & Ghassemi, M. (2021, December). Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature medicine. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8674135/
  14. Benjamin, R. (2025, October 25). Assessing risk, automating racism. Science. https://www.science.org/doi/10.1126/science.aaz3873
  15. McCoy, L. G., Banja, J. D., Ghassemi, M., & Celi, L. A. (2020, November). Ensuring machine learning for healthcare works for all. BMJ health & care informatics. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7689076/
  16. Ledford, H. (2019, October 24). Millions of black people affected by racial bias in health-care algorithms. Nature News. https://www.nature.com/articles/d41586-019-03228-6
  17. M;, C. I. P. (2019, February). Can ai help reduce disparities in general medical and Mental Health Care?. AMA journal of ethics. https://pubmed.ncbi.nlm.nih.gov/30794127/
  18. Igoe, K. J. (2021, March 12). Algorithmic bias in health care exacerbates social inequities - how to prevent it. Harvard. https://www.hsph.harvard.edu/ecpe/how-to-prevent-algorithmic-bias-in-health-care/
  19. Zucker, I., & Prendergast, B. J. (2020, June 5). Sex differences in pharmacokinetics predict adverse drug reactions in women. Biology of sex differences. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7275616/
  20. Frank, E., & Olaoye, G. O. (2024, February). Privacy and data protection in AI-enabled healthcare systems. Research Gate. https://www.researchgate.net/publication/378287462_Privacy_and_data_protection_in_AI-enabled_healthcare_systems
  21. Carlos, J., & Demil, G. (n.d.). How do you measure AI accuracy?. LinkedIn. https://www.linkedin.com/advice/3/how-do-you-measure-ai-accuracy-skills-information-technology
  22. Wawira Gichoya, J., McCoy, L. G., Celi, L. A., & Ghassemi, M. (2021, April). Equity in essence: A call for operationalising fairness in machine learning for Healthcare. BMJ health & care informatics. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8733939/
  23. Murdoch, B. (2021, September 15). Privacy and artificial intelligence: Challenges for protecting health information in a new era - BMC medical ethics. BioMed Central. https://bmcmedethics.biomedcentral.com/articles/10.1186/s12910-021-00687-3
  24. World Health Organization. (2021). Ethics and governance of Artificial Intelligence for Health. World Health Organization. https://www.who.int/publications/i/item/9789240029200
  25. NHS. (2019, October). Artificial Intelligence: How to get it right. NHS England. https://transform.england.nhs.uk/media/documents/NHSX_AI_report.pdf

About the Author

A person is smiling, standing in front of a wall with horizontal wooden slats. They wear a dark, patterned shirt and a necklace with a pendant.

Sophie Cleff

Sophie is an Associate at The Decision Lab. She holds a Bachelor of Science in Microbiology and Immunology from McGill University. She is passionate about applying her research background to interdisciplinary problems, especially related to public health. Before joining The Decision Lab, Sophie worked with the Montreal Children’s Hospital and Translating Emergency Knowledge for Kids (TREKK) to increase the quality, safety, and integrity of research in pediatric medicine. In her free time, she enjoys crocheting and playing the guitar.

About us

We are the leading applied research & innovation consultancy

Our insights are leveraged by the most ambitious organizations

Image

I was blown away with their application and translation of behavioral science into practice. They took a very complex ecosystem and created a series of interventions using an innovative mix of the latest research and creative client co-creation. I was so impressed at the final product they created, which was hugely comprehensive despite the large scope of the client being of the world's most far-reaching and best known consumer brands. I'm excited to see what we can create together in the future.

Heather McKee

BEHAVIORAL SCIENTIST

GLOBAL COFFEEHOUSE CHAIN PROJECT

OUR CLIENT SUCCESS

$0M

Annual Revenue Increase

By launching a behavioral science practice at the core of the organization, we helped one of the largest insurers in North America realize $30M increase in annual revenue.

0%

Increase in Monthly Users

By redesigning North America's first national digital platform for mental health, we achieved a 52% lift in monthly users and an 83% improvement on clinical assessment.

0%

Reduction In Design Time

By designing a new process and getting buy-in from the C-Suite team, we helped one of the largest smartphone manufacturers in the world reduce software design time by 75%.

0%

Reduction in Client Drop-Off

By implementing targeted nudges based on proactive interventions, we reduced drop-off rates for 450,000 clients belonging to USA's oldest debt consolidation organizations by 46%

Read Next

A group of people in a modern meeting room, some seated and working at a large table with laptops and cameras, while others stand and converse. The atmosphere is casual, with natural light filtering through large windows in the background.
Insight

Why Teams Make Bad Decisions

Sometimes, the best way to avoid group decision-making failures is not to make decisions as a group at all.

Notes illustration

Eager to learn about how behavioral science can help your organization?