Delphi methodology in healthcare research: How to decide its appropriateness

Abstract

The Delphi technique is a systematic process of forecasting using the collective opinion of panel members. The structured method of developing consensus among panel members using Delphi methodology has gained acceptance in diverse fields of medicine. The Delphi methods assumed a pivotal role in the last few decades to develop best practice guidance using collective intelligence where research is limited, ethically/logistically difficult or evidence is conflicting. However, the attempts to assess the quality standard of Delphi studies have reported significant variance, and details of the process followed are usually unclear. We recommend systematic quality tools for evaluation of Delphi methodology; identification of problem area of research, selection of panel, anonymity of panelists, controlled feedback, iterative Delphi rounds, consensus criteria, analysis of consensus, closing criteria, and stability of the results. Based on these nine qualitative evaluation points, we assessed the quality of Delphi studies in the medical field related to coronavirus disease 2019. There was inconsistency in reporting vital elements of Delphi methods such as identification of panel members, defining consensus, closing criteria for rounds, and presenting the results. We propose our evaluation points for researchers, medical journal editorial boards, and reviewers to evaluate the quality of the Delphi methods in healthcare research.

Key Words: Delphi studies; Quality tools for methodology; Research methods; Delphi technique; Consensus; Expert panel; Coronavirus disease 2019; SARS-CoV-2

Core Tip: There are no standard quality parameters to evaluate Delphi methods in healthcare research. Delphi methods’ vital elements include anonymity, iteration, controlled feedback, and statistical stability of consensus. Published studies have used modified versions of Delphi, and details on methods like expert panel selection, defining consensus, or closing criteria for Delphi rounds are not explicit. We suggest quality assessment tools for readers and researchers for a systematic assessment of Delphi studies.

INTRODUCTION

This review used the “Delphi study” for the published studies that used Delphi methodology. “Delphi rounds” is used for the survey questionnaire rounds to develop iterative discussion among panel members. “Delphi process” is used for the steps of Delphi methods in research.

The term “Delphi” originated from ancient Greek mythology and was believed to be the precinct of Pythia (a major oracle), where prophecies were made to dictate and direct vital state affairs. In its literal sense, Delphi methods can be defined as a structured technique to modulate a group communication process effectively in allowing a group of individuals, as a whole, to deal with a complex problem[1]. The Delphi method was initially developed for business forecasting using an expert panel’s interactive discussion, assuming collective judgments are more valuable than individuals.

Possibly the first application of Delphi methodology was during the cold war in the 1950s by the United States army. They used it for their military project RAND to develop consensus among experts using repeated rounds of anonymous feedback, forecasting future enemy attacks[2]. After this first application, it has been used in many other academic domains like finance, economics, development planning, and healthcare, where group forecasting makes sense in the absence of accurate tested data. In modern times, this forecasting tool has evolved into a statistical methodology to collate individual opinions and converge them into statistically generated consensus with collective intelligence. A constant theme is observed across all domains with vital elements like anonymity, iteration, controlled feedback, and group response (or consensus)[3].

The anonymity of individual members in a Delphi study removes the inherent bias like dominance and group conformity (defined as groupthink) observed with face-to-face group meetings. The primary purpose of the Delphi technique is to generate a reliable consensus opinion of a group of experts by an iterative process of questionnaire interspersed with controlled feedback[2]. After initial slow acceptance in healthcare, it is now a widely used method to generate group consensus, develop qualitative practice points, or identify future research areas. In healthcare, the Delphi process had been used in diverse areas: (1) Evaluate current knowledge; (2) Resolving controversy in management[4]; (3) Formulating theoretical or methodological guidelines[5,6]; (4) Developing assessment tools and indicators[7,8]; and (5) Formulating recommendations for action and prioritizing measures[9].

The Delphi methods from its inception have undergone modifications to structure effective and faster consensus. The modified Delphi does not have a standard criterion, but in principle, a steering group facilitates the group communication process effectively. There are no set standards for reporting Delphi studies in healthcare research, unlike other qualitative and quantitative clinical research tools. There are also no validated quality parameters to evaluate Delphi studies. In a recent meta-analysis of Delphi studies in healthcare research, many studies were found to be of questionable quality[10]. The protocol design, the definition of consensus, and closing criteria were not set a priori and vary widely in Delphi studies. There have been attempts to identify quality parameters to conduct and evaluate Delphi studies[10-12]. The guidance on conducting and reporting of Delphi studies (CREDES) is a popular tool, developed for Delphi studies on palliative care. The authors acknowledged significant variation in the reporting and methodology of Delphi studies and proposed CREDES standards for reporting and conducting such studies[12]. However, these tools are neither been validated in other fields of medicine nor universally accepted for the conduct of Delphi studies. The discrepancy in conduct and transparency of reporting may overshadow the consensus recommendations generated by Delphi studies. There is an urgent need of simple tools for systematic assessment of the quality of Delphi studies. Like other statistical research studies, readers must consider if the methodology has been followed appropriately for the key elements of Delphi technique. This article recommends critical appraisal of a Delphi study in healthcare sequentially by nine qualitative evaluation points in a four-step methodological process (Figure 1).

Figure 1 Stepwise quality assessment of Delphi studies.

PROBLEM AREA

The Delphi study is practical in problematic areas where either statistical model-based evidence is not available, knowledge is uncertain and incomplete, and human expert judgment is better than individual opinion[1]. The emerging disease or conditions in healthcare often simulates such areas, where either standard research pathways cannot be adopted or become impractical. Various approaches can identify these problem areas: (1) Extensive systematic literature search; (2) Group discussion among a defined steering group; and (3) Open-ended discussion rounds among panel members.

The process of identifying problem areas and its communication among all participating panel members should be explicit and must be done before the final survey rounds to achieve consensus.

Evaluation point

The criteria used to identify the problem area and process followed should be documented. The systematic search of the literature must mention period, keywords, and database included in the search.

PANEL MEMBERS

The members who participate in the anonymous voting process of the Delphi survey are called panelists. The panel member selection is undoubtedly the most crucial aspect of Delphi research studies[13]. The methods used for the identification and selection of panel members are discrepant in published Delphi studies. There are no standard criteria used for the definition of panel members[10]. The readers should consider the following issues while evaluating the Delphi study: Homogeneity of panel, labelling panel members as an ‘expert’, and size of the panel.

Homogeneity of the panel

A diverse panel helps to achieve a broader perspective and generalization of consensus. The homogenous group, on the other hand, may be more reliable in a particular study objective. The homogenous panel is suitable when resolving unsettled issues of a focused problem like management of acute respiratory distress syndrome, while the heterogeneous panel is appropriate in a broader situation like when studying the impact of mental illness. The methodology should represent the process followed for achieving homogeneity in the study.

Expert panel

The labelling of panel members as ‘experts’ is most contentious. The expert can be defined as someone with knowledge and experience on a particular subject matter; however, it is practically difficult to measure experience quantitatively. Despite its controversy, the experts are commonly used in the Delphi studies for panel members without a uniform selection criterion. The common goal behind using experts is to increase the qualitative strength of recommendations or consensus. The readers must evaluate the criteria for expert panel selection. Panel selection should adhere to a predefined criteria[4,6,14].

Size

There is no standard size of the panel members and varies from 10 to 1000 (typically between 10-100) in published studies. However, due to data management difficulties and logistic issues (rounds of the survey), a panel with three-digit sample size is unusual[10,15]. Generally, a double-digit number close to 30-50 is considered optimum in concluding rounds for a homogenous Delphi[4,14]. Appropriate size depends on the complexity of the problem, homogeneity (or heterogeneity) of the panel, and availability of the resources. Apart from panel members with knowledge, some studies recruit members from diverse academic and practice backgrounds or involve end-users in the process[16].

Generalizability of Delphi results requires an appropriate panel size, diverse representation of members from different specialties, and geographical distribution.

The electronic Delphi survey (also called e-Delphi) helps in the global representation of panel members, saves time, and fastens the survey rounds using technology without physical voting. This process involves selecting experts after research for eligibility on the world wide web; further email invitations to participate in the project can be sent. The acceptance rate among experts can be low, and researchers usually consider this higher attrition rate during the invitation process.

Evaluation point

The selection of a panel or voting members in a Delphi study should be based on objective and predefined criteria and related to the problem under study.

DELPHI ROUNDS

The strength of Delphi process is anonymity of panelist in the survey rounds, controlled feedback and iterative discussions. Anonymous survey rounds have advantages over face-to-face or group encounters in reducing dominance and group conformity. Participants feel more comfortable in providing anonymous opinions on uncertain, unsettled issues. The interpretation of items may sometimes become a critical issue in anonymous Delphi rounds and may affect the consensus process.

The “controlled feedback” is another classic characteristic of the Delphi study. It is termed as “controlled” because moderator decides about feedback provisions based on responses to the items and open comments. After each of the survey rounds, obtained data are analyzed and presented in an easily interpretable format to all the panel experts. It can include simple charts and statistics showing the stability of responses. Statistics usually include the measurement of central tendencies with dispersion, percentage, and frequency of distribution[17]. Even anonymous comments can be incorporated as a part of the feedback. Sometimes individual feedback along with group responses are also provided. Controlled feedback gives insight to the individual member about the trend and one can change its response if needed. Panel members should clear their position if they have an extreme choice of response in a particular situation.

Analysis of successive iterative rounds provides an opportunity to evaluate data for consensus and interspersed stability among the two successive rounds. The repetitive and interactive survey rounds are useful for gathering qualitative information, improving framing of the statements for panel members, and achieving consensus.

Evaluation point

The Delphi survey should be assessed for iterative discussions and controlled feedback while maintaining a strict anonymity of the panel members and their responses.

CLOSING CRITERIA

As Delphi is a method to generate consensus of individual panel member opinion on unsettled critical issues, the consensus and closing criteria vary widely among the studies[10,12,15]. The definition of consensus used in published Delphi studies is discrepant.

Consensus

Traditionally, a consensus is considered as the primary outcome of the Delphi study. However, its understanding is quite confusing among various studies. Consensus can mean a group opinion, solidarity towards a sentiment, or sometimes absolute alignment of the opinion of experts[18]. Hence, various measures have been used to define consensus. A meta-analysis[10] to evaluate quality of published Delphi studies found 73% of the studies reported a consensus method, and only 68% did so in an advanced declared protocol. It was even observed that some studies declare achieving consensus but do not provide the process to reach the consensus and its definition[10,12,15]. The definition of consensus used in published Delphi studies is discrepant[12,19]. The consensus definition used commonly is the percentage of agreement based on a predefined cut-off, central tendency, or a combination of both. However, percentage agreement varies widely from 50%-97% and is selected arbitrarily[10,12].

Closing criteria

The conventional design of the Delphi study had at least four rounds. However, the essence of good Delphi surveys is an iterative process and controlled feedback to generate consensus. The closing criteria in most of the Delphi studies include consensus achieved after a prefixed (usually two) rounds[10,12]. The stability of the responses or consensus cannot be checked with two rounds of Delphi. Any change in the items or controlled feedback may alter the response of panelists. However, these responses may not be stable and hence a fixed number of rounds without assessment of the stability of the results is a compromise on statistical robustness. The invention of “modified Delphi” arbitrarily uses two-three rounds of survey decided a priori as a closing criterion. The “modified” term in Delphi studies is, however, discrepant and without any universal accepted criterion. The only common thing in modified Delphi methodology is the active effort of the steering group in generating consensus. The steering group performed a systematic search of the literature in the problem area and, instead of open-end, initial Delphi rounds are focused on achieving consensus among panelists. The group also review the results after each round and items that reached consensus are dropped for the next rounds, but the items that are consistently not achieving consensus despite controlled feedback can also be dropped[20]. However, this active participation of the steering group can cause bias through opinion of members.

Stability

Understanding the stability of responses is even more confusing than consensus, and the stability of the consensus is rarely used in Delphi studies as a closing criterion. Classically, consensus or a pre-fixed round of surveys served as a closing criterion. It comes with an inherent risk that a significant change in responses occurred in the last round, affecting the stability of the results or consensus. Hence some authors believed that achieving a consensus is meaningless with unstable responses[1,21-23]. The stability of the results is thus considered the necessary criterion. Stability is defined as the consistency of responses between successive rounds of a study[21]. The researchers believe that specific results of two separate rounds for a particular question can occur by chance, which can be decreased by obtaining statistically significant stability (or variance) of the responses[10]. In other words, consensus can be there in unstable responses, and stability can be there without consensus, and hence achieving response stability should be an appropriate closing criterion. However, every effort to achieve consensus should be made[21,23]. Therefore, a hierarchical stopping criterion should be adopted as a closing criterion for Delphi (Figure 2).

Figure 2 Stability assessment for Delphi rounds.

Evaluation point

The criteria for stopping the Delphi rounds based on consensus or stability should be identified a priori. The alternative plans and method to drop items should be defined if consensus is used as a stopping criterion of Delphi rounds. Stability of the responses is important for statistical stability of the consensus.

EVALUATION OF RECENT DELPHI STUDIES

We used our nine qualitative evaluation points to assess the quality of recent Delphi studies on coronavirus disease 2019 (COVID-19).

Search strategy and selection criteria

A systematic search of the literature was conducted from PubMed and MEDLINE databases between January 1, 2020 and December 31, 2020. We used a combination of keywords, “Delphi technique” OR “Delphi study” OR “Delphi” AND “COVID-19” OR “SARS-CoV-2”. We excluded search results that have non-human study subjects, non-English literature, and alternative medicine.

Included studies

Fifty-two Delphi studies were assessed as per the inclusion criteria, and 34 (67.3%) studies[24-57], were finally analyzed using nine evaluation points (Table 1). The data on medical specialty, geographical location, the purpose of the study, conclusion format, number of experts, and Delphi rounds were collected for each study (Table 2). The study methods were scrutinized using nine qualitative evaluation points on a 3-point scale, “yes”, “no”, and “not clear” (Table 2).

Table 1 Evaluation of Delphi studies on coronavirus disease 2019 that were published in 2020 on nine qualitative evaluation points.

No.	Ref.	Medicine field	Geographical location (Country/Continent)	Aim or purpose	Guidance format
1	Vitacca et al[24]	Rehabilitation	Italy, Europe	Consensus on pulmonary rehabilitation in patients with COVID-19 after discharge from acute care.	Recommendations from experts’ panel.
2	Mikuls et al[25]	Rheumatology	USA, North America	Guidance to rheumatology providers on the management of adult rheumatic diseases during COVID-19 pandemic.	77 initial guidance statements converted to 25 final guidance statements.
3	Greenhalgh et al[26]	Primary health	UK, Europe	To develop early warning score for patients with suspected COVID-19 who need escalation to next level of care.	Development of software for early warning score in COVID-19 patients.
4	Lamb et al[27]	Respiratory medicine and critical care medicine	USA, NA	Guidance to physicians on the preparation, timing, and technique of tracheostomy in COVID-19 patients.	Eight recommendations.
5	Welsh Surgical Research Initiative (WSRI) Collaborative[28]	General surgery	Global	Identify the needs of the global OR workforce during COVID-19.	Statements, predominantly standardization of OR pathways, OR staffing, and preoperative screening or diagnosis.
6	Eibensteiner et al[29]	Nephrology	Europe	To gather expert knowledge and experience to guide the care of children with chronic kidney disease during the COVID-19 pandemic.	Qualitative expert statements and answers.
7	Bhandari et al[30]	Gastroenterology	Global	Guidance on how to resume endoscopy services during COVID-19.	Best practice recommendations to aid the safe resumption of endoscopy services globally in the era of COVID-19.
8	Guckenberger et al[31]	Radiotherapy	NA and Europe	To develop practice recommendations pertaining to safe radiotherapy for lung cancer patients during COVID-19 pandemic.	Consensus recommendations in common clinical scenarios of radiotherapy for lung cancer.
9	Aj et al[32]	General surgery	NA, Europe and Australia	Validation of international COVID-19 surgical guidance during COVID-19 pandemic.	Area of consensus and contentious areas from previous guidelines.
10	Gelfand et al[33]	Dermatology	NA	Guidance on the management of psoriatic disease during the COVID-19 pandemic.	22 guidance statements.
11	Allan et al[34]	Surgery	Global	Guidance on surgery and OR practices during COVID-19 pandemic.	Development of research priorities in discipline of surgery related to COVID-19.
12	Shanbehzadeh et al[35]	Medical informatics and public health	Iran, Middle east	Development of minimum data set for COVID-19 surveillance system.	Conceptual COVID-19 surveillance model.
13	Bergman et al[36]	Long-term nursing care	NA	Consensus guidance statements focusing on essential family caregivers and visitors in nursing homes during COVID-19 pandemic.	Recommendations for visitors in long term nursing homes.
14	Daigle et al[37]	Ophthalmology	Canada	Risk stratifying for oculofacial plastic and orbital surgeries in context of transmission of SARS-CoV-2.	Risk based algorithm for oculoplastic surgeries and recommendations for appropriate PPE.
15	Sorbello et al[38]	Anaesthesia	Europe	Review of available evidence and scientific publications about barrier-enclosure systems for airway management in suspected/confirmed COVID-19 patients.	Recommendation on enclosure barrier systems.
16	Jheon et al[39]	Cardiovascular and thoracic surgery	Asia	Thoracic cancer surgery during COVID-19 pandemic.	Recommendations on timing, approach, type of surgery, and postoperative requirements.
17	Olmos-Gómez et al[40]	Behavioural sciences	Spain, Europe	To know the impact of learning environments and psychological factors.	Future research priorities.
18	Sawhney et al[41]	Gastroenterology	Global	Study to emphasize patient-important outcomes while considering procedural timing.	Recommendations on procedural timing for common indications for advanced endoscopy during COVID-19.
19	Sciubba et al[42]	Neurosurgery	USA	Study to device scoring system to help with triaging surgical patients during the COVID-19 pandemic.	Scoring system to triage spinal surgery cases during COVID-19 pandemic.
20	Errett et al[43]	Environmental health science	USA	Study to develop an Environmental Health Sciences COVID-19 research agenda.	To validate, find limitations, and identify future research priorities.
21	Arezzo et al[44]	Minimal access surgery	Global	To study and provide recommendations for recovery plan in minimally invasive surgery amid COVID-19 pandemic.	Framework for resumption of surgery with focus on minimally invasive surgeries following COVID-19 pandemic.
22	Dashash et al[45]	Healthcare education	Syria	To identify essential competencies required for approaching patients with COVID-19.	Core competency points for health care professionals to prepare them for COVID-19 pandemic.
23	Ramalho et al[46]	Psychiatry	Global	To create a practical and clinically useful protocol for mental health care to be applied in the pandemic.	Consensus protocol for use of telemedicine in psychiatry consults during COVID-19 pandemic.
24	Saldarriaga Rivera et al[47]	Rheumatology	Columbia, SA	To produce recommendations for patients with rheumatological diseases receiving immunomodulatory and immunosuppressive therapies.	Recommendations for pharmacological management of patients with rheumatic diseases during COVID-19 pandemic.
25	Tchouaket Nguemeleu et al[48]	Public health	Canada, NA	Study for development and validation of a time and motion guide to assess the costs of prevention and control interventions for nosocomial infections.	Development and validation of a new instrument for systematic assessment of costs relating to the human and material resources used in nosocomial infection prevention and control.
26	Santana et al[49]	Nursing	Brazil, SA	To develop an adaptable acceptable nursing protocol during the pandemic.	Protocol for nurse managers to cope with pandemic.
27	Tang et al[50]	Oncology	China	To develop a risk model based on the experience of recently resumed activities in many cancer hospitals in China to reduce nosocomial transmission of SARS-CoV-2.	Risk model development on the basis of experience from recently resumed cancer hospital.
28	Jiménez-Rodríguez et al[51]	Public health	Spain	Develop recommendations for telemedicine in video consultations during COVID-19.	Consensus recommendations for healthcare professionals for proper management of video consultation.
29	Reina Ortiz et al[52]	Public health	Ecuador	Development of bio-safety measures to reduce cross-transmission of SARS-CoV-2.	Biosafety-at-home flyer for high-risk group and health care workers to reduce the risk of cross-transmission.
30	Douillet et al[53]	Internal medicine	France and Belgium	Identify reliable criteria for hospitalization or outpatient management in mild cases of COVID-19.	Development of toolkit “HOME-CoV rule”, a decision-making support mechanism for clinicians to target patients with suspected or confirmed COVID-19 requiring hospitalization.
31	Richez et al[54]	Rheumatology	France	Management of anti-inflammatory agents and disease-modifying-anti-rheumatic-drugs for rheumatological patients during COVID-19.	Recommendations to rheumatologists on management.
32	Yalçınkaya et al[55]	Physiotherapy and rehabilitation medicine	Turkey	Recommendations for the management of spasticity in Cerebral palsy children during COVID-19 pandemic.	Consensus recommendations for spasticity management in cerebral palsy children.
33	Tanasijevic et al[56]	Haemato-oncology	USA	To identify minimum hemoglobin for safe transfusion in myelodysplastic syndrome during COVID-19 pandemic.	Recommendations for lowest value of hemoglobin for which transfusions can safely forgo.
34	Alarcón et al[57]	Dentistry	Latin America	Education and practice in implant Dentistry during COVID-19 pandemic.	Consensus recommendations.

COVID-19: Coronavirus disease 2019; SARS-CoV-2: Severe acute respiratory syndrome coronavirus 2; OR: Operating room.

Table 2 Basic information of the Delphi studies included for evaluation.

No.	Ref.	Identification of problem area	Selection of panel members	Anonymity of panellist	Controlled feedback	Iterative rounds	Consensus Criteria	Analysis of consensus	Closing criteria	Group stability	Number of rounds	Number of experts
1	Vitacca et al[24]	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	2	20
2	Mikuls et al [25]	Yes	Yes	Not clear	Yes	Yes	Yes	Yes	Yes	No	2	14
3	Greenhalgh et al[26]	Yes	Not clear	Yes	Yes	Yes	Not clear	Yes	Yes	No	4	72
4	Lamb et al[27]	Yes	Yes	No	No	No	Yes	Yes	Yes	No	1	13
5	Welsh Surgical Research Initiative (WSRI) Collaborative[28]	Yes	No	Yes	No	No	Yes	Yes	Yes	No	1	339
6	Eibensteiner et al[29]	Yes	Yes	Not clear	Yes	Yes	No	No	No	No	4	13
z	Bhandari et al[30]	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	2	34
8	Guckenberger et al[31]	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	3	32
9	Aj et al[32]	Yes	No	Yes	Yes	No	Yes	Yes	Yes	No	1	339
10	Gelfand et al[33]	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes	No	2	18
11	Allan et al[34]	No	No	Yes	No	Yes	Yes	Yes	No	No	3	213
12	Shanbehzadeh et al[35]	Yes	Yes	No	No	Yes	Yes	Yes	Yes	No	2	40
13	Bergman et al[36]	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	No	2	21
14	Daigle et al[37]	Yes	Yes	Not clear	Yes	Yes	Yes	Yes	Yes	No	2	18
15	Sorbello et al[38]	Yes	Not clear	Not clear	Not clear	Not clear	Not clear	Not clear	Not clear	Not clear	-	0
16	Jheon et al[39]	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	No	2	26
17	Olmos-Gómez et al[40]	Yes	Yes	Not clear	Not clear	Yes	Yes	Yes	Yes	Yes	3	441
18	Sawhney et al[41]	Not clear	Not clear	Not clear	Yes	Yes	Yes	Yes	No	Not clear	3	14
19	Sciubba et al[42]	Not clear	Not clear	No	Not clear	Yes	No	Yes	No	Not clear	3	16
20	Errett et al[43]	Not clear	Not clear	Yes	Not clear	Yes	Yes	Yes	Yes	No	3	28
21	Arezzo et al[44]	Yes	Not clear	Yes	Yes	No	Yes	Yes	Yes	No	2	55
22	Dashash et al[45]	Yes	Not clear	Not clear	Yes	No	Yes	Yes	No	No	3	20
23	Ramalho et al[46]	Yes	Not clear	Not clear	Yes	Yes	Yes	Yes	Not clear	No	2	16
24	Saldarriaga Rivera et al[47]	Yes	No	Not clear	Yes	Yes	Yes	Yes	Not clear	No	3	11
25	Tchouaket Nguemeleu et al[48]	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes	2	18
26	Santana et al[49]	No	Not clear	No	Yes	Yes	Yes	Yes	No	Not clear	4	6
27	Tang et al[50]	Yes	Yes	Not clear	No	Not clear	No	No	No	No	1	83
28	Jiménez-Rodríguez et al[51]	No	Yes	Not clear	Yes	Yes	Yes	Yes	Yes	No	3	16
29	Reina Ortiz et al[52]	No	Yes	Not clear	Yes	Yes	No	No	Not clear	No	2	12
30	Douillet et al[53]	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Not clear	4	51
31	Richez et al[54]	Yes	Yes	No	Yes	No	No	No	Not clear	No	2	10
32	Yılmaz Yalçınkaya et al[55]	Yes	Yes	Not clear	No	No	Yes	Yes	No	No	1	60
33	Tanasijevic et al[56]	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	No	3	13
34	Alarcón et al[57]	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	No	3	197

Summary of Delphi studies assessment

COVID-19 is a new disease coined by World Health Organization in February 2020. The exponential growth of the COVID-19 pandemic disrupted public health, healthcare, and the global economy in an unprecedented manner. The absence of quality evidence on pathophysiology, infection transmission or control, and management of COVID-19 made researchers deploying Delphi methodology for consensus recommendations in various medicinal fields affected by COVID-19. We used our evaluation points for the quality assessment of Delphi technique in 34 selected studies that met the inclusion criteria. The studies from various fields of medicine were included in this analysis. Most of the studies (60%) were done in Europe or North America. The median of 20 (interquartile range-41) experts participated in two (interquartile range-1) Delphi rounds (Table 1).

No single study met all nine evaluation points for quality assessment (Table 1). The systematic identification of the problem area was explicitly declared in 28 (79.41%) studies. The anonymity of panelist was missing in nine (26.47%) studies and not disclosed clearly in another 13 (38.24%) articles. The confidentiality in the identity of panelists was breached in few studies either for video/audio conference or in the final round to generate consensus on the items[25,27,29]. The consensus based on the percentage of agreement and consensus analysis was mentioned in 27 (79.41%) studies. The assessment for stability of the results or consensus was missing in many of the studies with only two studies mentioning this in their methodology.

This systematic evaluation of Delphi studies in medical fields highlights the variations in the research methodology used. There was no single study that could score in all the nine evaluation points.

STRENGTH AND LIMITATIONS OF THE ASSESSMENT

We assessed our evaluation points in a wide variety of Delphi studies across various medical fields. These evaluation points are a focused qualitative tool set to assess any Delphi study on a 3-point scale. The evaluation points can be used by the readers, journal editors, and reviewers to assess the quality of the Delphi methodology.

The limitations of this assessment are the inclusion of only English language published studies in the medical field. The evaluation points were qualitative and did not assess the reporting method of the results.

Despite the limitations mentioned above, the nine evaluation points can rapidly assess the quality of Delphi studies and, thus, the creditability of scientific research presented through them.

CONCLUSION

There are no standard quality parameters to evaluate Delphi methods in healthcare research. The vital elements of Delphi methodology include anonymity, iteration, controlled feedback, and statistical stability of consensus. The published studies have used modified Delphi, and details on methods like expert panel, consensus, or closing criteria are not explicit. We suggest tools for readers and researchers for a systematic assessment of the quality of the Delphi studies.