Delphi Method-Introduction-Verbatim Edit

  •  DELPHOI Master’s Degree Programme, FutuS-2 Futures Research Methods Otavan Opisto, February 9, 2012 Anita Rubin
  • 2. Knowledge interests Knowledge Interests in Research Technical knowledge interest (Delphi) Practical, Hermeneutical and/or interpretative knowledge interest(SSM, Delphi) Intuition (CLA, Delphi, SSM Critical knowledge Interest (CLA) Study problem Source: P. Kyrö, 30 May, 2001 10.11.2014 2
  • 3. Definition A technique of structuring a communication process of a group in order to help to understand and deal with the future development of a complex problem. (Linstone & Turoff 1975 • A panel formed by invited experts evaluates the future of a specific topic /course of development. • Often replaces and/-or supplements the work of expert groups  creates material for scenario working (e.g. the variables in a FAR table). • Especially useful in exploring large, fundamental issues and aims, development endeavours, or problems which are socially sensitive (not to be approached with the help of more ordinary methods). • Most beneficial as a method when the topic under study is open. 10.11.2014 4
  • 4. Ancient history The Greek myth tells that in Delphoi, people worshipped Gaia, the Earth goddess. She had a temple called Pytho and it was guarded by a huge snake dragon, Python. Apollo, Son of Zeus, killed Goddess Gaia’s dragon and made himself the ruler of the place. He then changed himself into a dolphin in order to lure priests for his sailor oracles. In the course of time, the site of the dolphin, Delphoi, became the center of the future. There Apollo’s future prophesies were mediated by Pythias, young and drugged female oracles, whose blurred messages were translated by priests. Throughout ancient Europe, Delphoi was famous for its oracles. As answers to the questions stated by warriors, kings and politicians, priestess Pythia dictated blurry prophesies in an ecstatic condition, caused by herbs and psychoactive drugs. Apollo’s priests then interpreted an formulated the prophesies into poetic form. 10.11.2014 5
  • 5. As a technique, Delphi was first introduced by Olaf Helmer at the Rand corporation (USA) in the 1950’s and 1960’s. The Delphoi technique is often also called the Delphi technique (method). While Delphi is a specific method to produce new information, it is also a group of similar techniques to evaluate information and knowledge already existing  tacit knowledge. Delphi has also been used in e.g. finding and defining new technologies and innovations and evaluating the speed of their expansion (e.g. large studies of technology assessment involving thousands of panelists in Japan). 6 More history
  • 6. …and still some more history Originally the aim of a Delphi process was to determine the future development of some specific topic by the experts in the field. The idea was to gain a consensus, and this was done by pressing the opinions through the panels as many times as needed for that. The method was strongly criticised in the 1970’s from • leaning on the median in the opinion as the most competent estimation; • causing a bandwagon effect on some panelists, and • allowing some experts to use anonymity to shape the general opinion towards his/her direction by using the process.  The method was nearly forgotten until the beginning of the 90’s. Now the method widely in use  new principles were established. 10.11.2014 7
  • 7. Characteristics The meaning of a modern Delphi study is to create a mixture of expert knowledge and tacit knowledge. The aim is no more to gain a consensus, but to especially dig out new and fresh viewpoints and ideas and to make the others reflect on those  many alternative and well-grounded ideas of the future development. Today Delphi is used to help decision-making and/or to build scenarios on a more profound basis. Six special features: • anonymity • iteration of arguments • communicativeness • well-founded feedback • specified expertise • learning with dialogue
  • 8. Anonymity The experts in the panel usually do not know who the other panel members are. They answer to the questionnaire or survey program alone. The advantages of minimising ordinary group dynamics  independency from roles  imposition of authority and strong opinion leaders;  authenticity of opinions;  new ideas also from not so well-known/respected experts;  the shy and the quiet who usually do not bring forth their ideas and opinions, can safely do so;  easier for experts to change their opinion (without the fear of ”losing face” or credibility among colleagues).
  • 9. Iteration of arguments At first the researcher forms a series of questions or claims and then gives them to the experts to answer. The experts are asked not only to cross out the option which best corresponds with their opinion, but also give reasons why they think the way they think. Then the answers are analysed and the researcher formulates a new questionnaire where he/she especially concentrates on those answers which a) differ from the general way of thinking; b) include aspects which sound new, fresh, original, or show marks of new knowledge; c) open new and different viewpoints or lines for further study; d) need further clarification; e) are interesting.
  • 10. …/… Communicativeness The panelists are given a chance to comment on each others’ ideas and opinions, question them and give more information.  Dialogue between a large number of panelists from different (but tangenting9 fields of expertise;  Experts can get new and fresh information;  Possibility to change one’s mind in the light of new information;  Possibility to develop the original topic with the help of expert discussion. 10.11.2014 11
  • 11. 12
  • 12. Well-grounded feedback In addition to multiple-choice answers (Likert), the panelists are usually also asked to give grounds to their opinion – why they pick up just the alternative they crossed out. The reasons for that might be very different among the panelists who choose the same alternative. Feedback consists of analysed results with a special emphasis on diverging opinions and the reasons/arguments behind those (asked esp. from those original panelists who gave an out-of-the line answer). • Both statistical and interpretative data  challenge for the researcher (e.g. how to prepare valid and relevant theses for the next round, how to analyse the results for reporting; how to make scenarios?); • The expert opinions are not accepted as such  reasoned views (in the light of latest information) are expected. • Each opinion can be corrected and changed, based on argumentation and feedback from the others.
  • 13. Specified expertise Expertise is the crucial factor in order to make a Delphi process successful  Osmo Kuusi. The choice is a systematic process to be carried out e.g. with a table in which the topic under study is opened from the point of view of (1) experts’ knowledge competences and (2) interests and background. Therefore the expert should  be on the top level of his/her field (work, interests, background, etc.);  have knowledge and an open mind to other (nearby) fields;  be able to see interconnections between national/global, past/present/future courses of development;  be creative and imaginative, i.e., be able to view the topic from also unusual and un-orthodox angles;  be able to see wholes there where others see details;  be motivated, courageous, open-minded and honest (to accept also opinions and ideas which differ from those of him-/herself.
  • 14. Expert panel The ideal size of the panel is 20 to 50 members (no more than 70) (see later some exceptions). The panel members can also be found by asking those panelists who have already been invited to name other experts in the field. Who are the experts?  Depends on the knowledge interests of the study: researchers, decision-makers, entrepreneurs, citizen activists, teachers of the field topic, key figures in society dealing with the study problem, etc.  People who usually are accepted as experts by the other experts in the field;  Leading figures in their field of expertise;  Experts from the tangential fields.
  • 15. 17 A good expert panelist • Represents an important field of expertise; • Represents an expert view point which would stay unnoticed if he/she wouldn’t participate; • ”Catalyses” the other experts; • Is interested in the topic and willing to reason; • Has the guts to take a stance on the topic/theme in a futures-oriented way.
  • 16. Learning At its best, a Delphi study is a profound learning and development process to the panelists. A panelist’s way of thinking may change, when (1) the level of knowledge grows along the Delphi rounds, and (2) his/her own knowledge and thinking is critically weighed by the other panelists.  Deeply rooted basic beliefs get dusted, aired and if they are good, pollinated;  The knowledge of the members of a group creates new levels of knowledge which may become more than just the sum of individual panelists’ knowledge  benefit for the field under this specific Delphi study. Jovi Schnell: Tacit knowledge adrift on the Myth of Progress
  • 17. 19 The original Delphi In the beginning, the meaning of a Delphi process was with the help of experts, to define the future of some specific phenomenon. The aim was to gain consensus as the result of the process. This consensus was reached by moulding and circulating the opinions in the panel so many times that in the end the panelists all thought the same way. (The problems of the original Delphi were discussed earlier.)
  • 18. Characteristics of Delphi today Today, the information gained from a successful Delphi is a systemic combination of tacit knowledge and expert knowledge  visionary knowledge. The information gained in modern Delphis is needed especially for the production of such views, ideas and grounds which are needed for decision-making and planning. The aim is therefore no more to gain a consensus but especially bring into light new and fresh viewpoints, knowledge and ideas and make other experts to react to them  new and well-grounded 20 ideas for possible futures or future states of the topic under study.
  • 19. • The goal is to reach a consensus among the experts. • Traditionally large panels. • The meaning is to gain unanimous and precision forecasts and to utilize the deviant opinions and ideas in decision-making. • The consensus can be aimed at only in the last round  benefiting from the marginal opinions. • Useful especially when it is necessary to find/create a consensus in a small unit (group, society, organisation, firm, etc.) in order to build shared goals (a tool in strategy work). 21 Consensus Delphi (Classical Delphi)
  • 20. Policy Delphi • The goal is to find/create well-grounded ideas and views of the future(s); • Aims at finding the probable and desirable futures/future states; • Aims also at a set of complementary arguments (e.g. net-based so-called 22 Dialogue Delphi); • Special attention to divergent views so that as many relevant thing/idea/factor/course of development/trend etc. can be considered in decision-making; • Rather small panels (usually 10-45 panelists); • A method for social/societal learning; • A special case: Argumentative Delphi (Argument Delphi) where the panelists are made to comment on their own claims/opinions  the goal is the variety of opinions.
  • 21. Other Delphi processes: Trend Delphi • The starting point is a single trend which is presented as a 23 line/curve. • The panelists are asked to continue the line/curve and then give grounds to the course of development they drew by listing their presumptions and those factors of uncertainty which might change the line/curve. • In the end, each panelist is asked to vote about the different alternative lines/curves presented by the panelists.
  • 22. Delphi barometers • Argument Delphi • The same panel is asked to repeat their answers at certain intervals (e.g. a year, a quarter year)  motivation. • The first (and perhaps the second) round are carried out in order to test the set of questions. • The idea is to see how opinions on the topics/themes of the questionnaire develop in time. • Futures barometer of learning 2030
  • 23. • Online questionnaire • Does not include sequential rounds; • Based on qualitative analysis and dialogue; • In the given time frame, the panelists can visit the questionnaire, read the comments and discussion by the other panelists and then react and change their opinion as often as they wish; • More normative and communcative than the ordinary Delphi. Real-time Delphi
  • 24. • Quantitative • Large panels (often 26 Survey Delphi hundreds of respondents)  computer-based surveys • The level of expertise is not very important. • Often explores the possible future courses of technology (e.g. Japan)
  • 25. The phases of a Delphi process 1 A Delphi process can be divided into 11 phases: 1. Definition of the study problem and the goals of the study; 2. Invitation of the research group for planning and execution 27 of the study; 3. Collecting and analysis of necessary background information; 4. Selection and invitation of the actual expert panel and sharing of information about the study process  motivation; 5. Construction of the study questionnaire, testing and correcting it for the first round …/…
  • 26. The phases of a Delphi process 2 6. Execution of the first study round (either via mail oras an 28 electric questionnaire or face-to-face interview/questionnaire); 7. Analysis of the results of the first round; ) 8. Based on the results of the first round, construction of the second round questionnaire (argumentative round); 9. Testing, execution and analysis of the results of the 2. round questionnaire; 10. Third round: construction of the questionnaire based on the previous rounds, execution and analysis of the answers; 11. (possible extra rounds) and reporting of the results.
  • 27. To be noticed: Experts The choice of experts can be made carelessly, too roughly, or in an imprecise way  the validity and reliability of the research suffer. To be asked: • Is this expert really an expert in the exact field/topic needed in the Delphi?  knowledge competences, involvement, level of interest, etc. • As a group, are they representative enough?  too large or too specified group opinions only mean vague generalisations • Is each relevant field represented in an equal way? • Do they know the development in the near-by fields  if 29 not, some new experts are needed
  • 28. To be noticed: Set of questions/theses Check the accuracy, adequacy, validity and correctness of each question. The less motivated or frustrated experts might leave out from the next round. Why? • Unclear, ambiguous or too difficult set of questions, or • Too wide a questionnaire (takes too much time to answer)  use test panelists!
  • 29. To be noticed: Analysis of the rounds A summary of analysis (between rounds) which is incorrect, badly written or clings to irrelevant details inevitably affects the results of the next round: Coding and analysis incomplete:  in the next round, the panelists concentrate on irrelevant topics or exciting but useless little details from the point of view of the actual research  the validity of the study suffers. At worst, a badly executed Delphi only strenghtens conventional, narrow and superficial thinking, and it does not work as a learning process for the participants.
  • 30. To be noticed: Reporting The researcher cannot concentrate on the essential  analysis and thereafter the contents of the report become slanted. The researher gains results which do not please the customer  dishonesty in reporting the drawbacks of the study process. ”Secret” study results: lacking possibility for the scientific community to evaluate them. However, sometimes it is justifiable not to open all the results to the public (e.g. when the topic is socially delicate, the panelists might become endangered, etc.) The researcher should have a wide understanding of causal relations; especially the knowledge of social theory is important. 32
  • 31. To be noticed: Opinions Figure out how much of the opinions of experts is genuine and how much is mere repetition of general knowledge/opinions. Make sure already in the beginning that each expert gives his/her response as a private individual, not as the representative of his/her organisation, reference group, stakeholder, business firm etc., unless this specifically is the priority. The further in the future the questions take the panelists, the less accurate their opinions are. 33
  • 32. 34 No knowledge is more important than the knowledge of what not yet is, but could be. Mihail Bahtin[1]


  1. Reference1
Rubin, A. (2014, November 10). Delphi method - introduction. Retrieved March 23, 2015, from
Was this page helpful?

The poll was created at 14:08 on March 28, 2015, and so far 0 people voted.

Ad blocker interference detected!

Wikia is a free-to-use site that makes money from advertising. We have a modified experience for viewers using ad blockers

Wikia is not accessible if you’ve made further modifications. Remove the custom ad blocker rule(s) and the page will load as expected.