The Delphi Method

The Delphi method is a technique used in group decision-making and some forms of qualitative research. It involves gathering a panel of experts, having them complete a survey or questionnaire individually, and sharing these anonymised answers within the panel to allow for feedback and debate. Each expert is presented with the questions again, and the process is repeated. It is expected that all opinions will eventually converge around a general consensus.

The Delphi method has been used extensively in forecasting, especially in business and technology. It’s also a commonly used tool in public policy: when policymakers use panels of experts to inform decisions around issues like healthcare, education and climate change, they frequently make use of the Delphi method.

Delphi Method – A research/decision-making technique in which individual experts give their opinion on a particular subject, then respond to the opinions of others.  Afterward, they evaluate their decision again before presenting it for the final time. In most cases, an alignment of opinions arises surrounding a general consensus.

Wisdom of Crowds – The theory that decisions made by groups are usually of better quality than those made by individuals.

Consensus – A general agreement reached by members of a group.

Although the name was inspired by the ‘Oracle of Delphi’, a high priestess of Ancient Greece, The Delphi method has its roots in military warfare. The method was developed in the United States at the onset of the Cold War, as a way to predict the role that technology would play in future combative events.¹ So, unlike the vast majority of theories and methods we encounter in behavioral science, the Delphi method didn’t really come about by way of academic research.

In 1944, General Henry Arnold commissioned a report for the United States Air Corps on technological capabilities that might be deployed by the military in the future. After much trial-and-error of conventional approaches to forecasting, including quantitative models and trend extrapolation, it became apparent that a novel technique was required when forecasting situations that involve yet-to-be-determined parameters. As a result, the American public policy think-tank RAND Corporation, led by Norman Dalkey and Olaf Helmer, developed the Delphi method. In its early applications, the technique was used to investigate the probability and potential effects of future attacks on the United States. Experts made estimates, discussed them, and then estimated again, in a process that would be known as ‘Estimate-Talk-Estimate’. The idea was that opinions would eventually start to converge around the same repeated estimations.

Since then, the Delphi method has been deployed in a wide range of settings, including business, government, medicine, and science.² While there is great variation in how the technique is used, the general structure of Estimate-Talk-Estimate defines the Delphi method. In policy-making, the Policy Delphi³ is used to generate the most divergent political views on how a major policy issue should be addressed. It’s also been influential in the development of direct democracy and stakeholder engagement, as policymakers increasingly try to involve a wide range of experts in their decision-making.

That said, the Delphi method was not the first technique to advocate for the leveraging of groups in decision-making. Scientists and mathematicians had long observed the benefits of using groups instead of individuals to make decisions. In 1907, Francis Galton (a cousin of Charles Darwin) observed how the average of all the entries in a county fair ‘guess the weight of the ox’ competition proved incredibly accurate – more so than the guesses of most individuals, even farmers and so-called cattle experts.⁴ The concept was developed and is known today  as the Wisdom of Crowds.

Once declassified by the US government, the Delphi method rose in prominence and was adopted by several fields. By 1980, a range of papers and books on the subject had been published, especially in the social sciences and public policy.

In recent years, the Delphi method has developed from its initial application as a forecasting tool, to wider use in qualitative research and stakeholder consultation. Scaled-down formats appear in business settings today; for example, during face-to-face meetings during which participants give initial opinions, offer feedback to the group, and then re-confirm their final viewpoints. This variation is sometimes used as an alternative to brainstorming in strategy meetings.

Other variations include the Policy Delphi³, which seeks to consult the most extreme opinions around an important public issue, giving governments an idea of some arguments they need to consider when drafting new policy. An example of this is the ‘Citizen Assembly’ consultation tool that’s been successfully deployed in countries like Canada, Ireland and the Netherlands, amongst others.⁶

Finally, the Argument Delphi7 is more process-driven, used to foster continued discussion and evaluate relevant arguments surrounding a topic, instead of aiming for consensus. Through its many variations and use in more and more fields, the Delphi method remains more influential now than ever.

The main argument against the Delphi method is that it strips group-decision making of the group context. Real-time discussion and debate doesn’t happen because experts give their opinions and receive feedback without interacting with the other members of the panel. The influence of contextual factors that come with being in a group – for example, social pressure -is not as prominent  in the Delphi method as in conventional group decision-making (committees or meetings). Experts don’t get the opportunity to debate with one another because the Delphi method calls for ‘controlled feedback’, which is generally not interactive.⁸

Another problem arises when participants either misunderstand the questions being asked, or the items used in a questionnaire invoke a variety of interpretations. Open-ended questions are a particular challenge here, because they make it difficult for facilitators to identify patterns in opinions. Challenges also arise when the ‘experts’ involved possess varying levels of expertise. Unfortunately, the Delphi method does not account for psychological forces such as the imposter syndrome, or the role of cognitive biases.

United States Air Corps: Cold War Scenario Forecasting

As we’ve mentioned, the Delphi method was initially developed for the US Air Corps in an effort to predict the likelihood of Soviet attacks during the Cold War. Dalkey, Helmer and their team at RAND Corporation (known as Project RAND at the time) sought to use expert advice to predict how the Soviets would select and attack US military targets. They were specifically looking for estimations of the number of atomic missiles that the Soviets would be expected to use in their attacks, and the impact these would have on US resources. In essence, they were trying to predict how Soviet military strategists would act towards the US in the future.

To do this, the team at RAND Corporation  involved a range of experts and collected their estimations through what became the Delphi method. Their goal was to “collect the most reliable consensus of opinion of a group of experts by a series of intensive questionnaires interspersed with controlled opinion feedback.’’²

Healthcare – Clinical Guidelines

The Delphi method has proven to be particularly useful in the development of clinical guidelines for certain diseases and illnesses. What’s especially beneficial is that it allows for a broad range of expert opinions to be considered; in fact, most clinical research has evolved to extend this ‘expert’ title to patients, caregivers, (e.g. parents) and patient advocacy groups. As Natalie Street from the Centers for Disease Control and Prevention observes, “Clinicians are experts, but the patients with the disease are also experts. They’re the ones living with it on a daily basis.” ⁹ The anonymity of the Delphi method allows patients to share their voice, without fear of being challenged, or at worst, ridiculed, by medical experts. Patients and caregivers  can offer their opinions in a controlled environment, and the chance of the Reactive Devaluation Bias occurring is much lower. Finally, the very fact that patients are involved in the consultation phase greatly increases the likelihood that they will follow the guidelines themselves (see: Ikea effect.)

Group Decision Making: How to Be Effective and Efficient

Yasmine Kalkstein discusses how groups can avoid groupthink and make meetings more effective and efficient.

COVID-19 and the Science of Risk Perception

How the ‘Wisdom of Crowds’ can improve fact-checking around COVID-19 risk.

1. RAND Corp. (2021). Delphi Method. Retrieved 11 February 2021, from https://www.rand.org/topics/delphi-method.html
2. Gupta, U. G., & Clarke, R. E. (1996). Theory and applications of the Delphi technique: A bibliography (1975–1994). Technological forecasting and social change, 53(2), 185-211.
3. Franklin, K. K., & Hart, J. K. (2007). Idea generation and exploration: Benefits and limitations of the policy Delphi research method. Innovative Higher Education, 31(4), 237-246.
4. The Real Wisdom of the Crowds. (2021). Retrieved 11 February 2021, from https://www.nationalgeographic.com/science/phenomena/2013/01/31/the-real-wisdom-of-the-crowds/
5. Surowiecki, J. (2005). The wisdom of crowds. Anchor.
6. Beekers, E. (2021). Are citizens’ assemblies the future of participation? – CitizenLab’s Blog. Retrieved 15 March 2021, from https://www.citizenlab.co/blog/civic-engagement/are-citizens-assemblies-the-future-of-participation/
7. Mukherjee, N., Huge, J., Sutherland, W. J., McNeill, J., Van Opstal, M., Dahdouh‐Guebas, F., & Koedam, N. (2015). The Delphi technique in ecology and biological conservation: applications and guidelines. Methods in Ecology and Evolution, 6(9), 1097-1109.
8. Rowe, G., & Wright, G. (1999). The Delphi technique as a forecasting tool: issues and analysis. International journal of forecasting, 15(4), 353-375.
9. RAND Corporation. (2021). Giving Patients a Voice in Medical Guidelines. Retrieved 11 February 2021, from https://www.rand.org/blog/rand-review/2019/12/giving-patients-a-voice-in-medical-guidelines.html