ACCORD-AI: Evaluating the role of generative AI in consensus development

Ellen L. Hughes,1 William T. Gattrell,2 Keith Goldman,3 Niall Harrison,4
Amy Price,5 Paul Blazey6

1. Camino Communications, Nottingham, UK; 2. Bristol Myers Squibb, Uxbridge, UK; 3. Global Scientific Publications, AbbVie, North Chicago, IL, USA; 4. OPEN Health Communications, London, UK; 5. Department of Community and Family Medicine (CFMED), Dartmouth-Hitchcock Clinics, Lebanon NH, USA; 6. School of Kinesiology, University of British Columbia, Vancouver, Canada.

Explore more

Exploring AI-assisted consensus development

The full, pre-specified ACCORD-AI protocol is openly available via the Open Science Framework

Access here

ACCORD-AI is an exploratory study evaluating how large language models (LLMs) can support the generation of draft voting statements for consensus processes. The work aims to inform transparent, efficient, and methodologically robust hybrid AI–human approaches to consensus development. 

Using outputs from the ACurate COnsensus Document (ACCORD) published systematic reviewas a test case, we compared draft AI-generated voting statements from nine LLMs with the original human-generated ACCORD longlist. We evaluated relevance, repetition, novelty, and overlap with the original human-generated ACCORD draft statements.

The materials presented here summarise exploratory findings from the ACCORD-AI study and are intended for academic discussion. A full manuscript reporting the complete methodology and analyses is in preparation.

The findings

Our exploratory findings found that overall, LLMs appear best positioned as pre-consensus accelerators, supporting but not substituting expert-led consensus development.

Below you’ll find resources discussing the study and our exploratory findings to date. 

Poster

Use of large language models in reporting guideline development: Reproduction of ACCORD voting items from systematic review findings (presented at ISMPP EU 2026)

Download poster

In-depth findings

Substantial differences were observed across nine LLMs in output volume, relevance, and novelty. View the detailed data to understand how individual models performed.

Click on a bar on the graph to the left to view the data

Videos

Amy Price explains why we decided
to evaluate the use of LLMs in
consensus development.

1 minute

Ellen Hughes talks through how 
we evaluated LLMs for drafting consensus statements.

2 minutes

Paul Blazey discusses the outcomes
and the implications of using LLMs in consensus methods.

6 minutes

Infographic

Summary infographic explaining the
ACCORD-AI process and outcomes

Download infographic

The materials presented here summarise exploratory findings from the ACCORD-AI study and are intended for academic discussion. A full manuscript reporting the complete methodology and analyses is in preparation.

ACCORD (ACcurate COnsensus Reporting Document) is an internationally developed
reporting guideline for consensus methods in biomedical research.It is a tool to guide
how biomedical studies using consensus methods are reported as researchers write up such studies for publication.

The ACCORD checklist was published in PLOS Medicine in 2004 and can be downloaded below
‍

Access here

The ACCORD Explanation 
& Elaboration document can be downloaded below
‍

Access here

For more information on ACCORD, including the guideline, supporting resources, and other related publications, visit the ISMPP ACCORD page below

Access here
Close
Model: Claude 3.5 Sonnet
Chat #: 1–3
Chat 1 Relevant AND novel (Y/N) Chat 2 Relevant AND novel (Y/N) Chat 3 Relevant AND novel (Y/N)
Background
2. Explain the rationale for choosing the specific consensus method. Y
3. Specify the intended use of the consensus results. Y
4. Describe the context and setting of the consensus study. Y
Methods
7. Describe any modifications made to the standard consensus method. Y
7. Explain how the study design addresses potential biases in the consensus process. Y
10. Explain how diversity among participants was addressed. Y
13. Describe modifications made to standard consensus methods. Y 11. Explain how diversity and inclusivity were considered in participant selection. Y
23. Explain how potential biases in the consensus process were addressed. Y
23. Explain how qualitative data were analyzed and incorporated. Y
26. Explain how missing data or non-responses were handled. Y
Results
27. Present the initial number of items or statements considered for consensus. Y
27. Summarize qualitative feedback or comments provided by participants. Y
28. Describe any changes in consensus between rounds. Y
31. Describe any items or statements that did not achieve consensus. Y
Discussion
34. Interpret the results in the context of the research or clinical question. Y 31. Discuss the implications of the consensus results for practice or future research. Y
35. Discuss items that did not reach consensus and potential reasons why. Y
33. Compare the results to previous related work or evidence in the field. Y 35. Discuss the implications of the consensus results for research or practice. Y
37. Compare the results with existing literature or guidelines, if applicable. Y 34. Discuss any unexpected or controversial findings from the consensus process. Y 36. Suggest areas for future research or refinement based on the consensus results. Y
38. Discuss the implications of the results for practice, policy, or future research. Y 37. Discuss the generalizability of the consensus results. Y
36. Suggest next steps or future research based on the consensus results. Y
Other Items
39. Explain how ethical considerations were addressed. Y
40. Describe plans for disseminating or implementing the results. Y
40. Outline plans for disseminating and implementing the consensus results. Y
40. Describe the ethical considerations and approvals for the study. Y
41. Explain how the consensus findings will be disseminated and implemented. Y 42. Provide a clear description of how to apply or interpret the consensus results. Y
Model: Claude 3.7 Sonnet
Chat #: 1–3
Chat 1 Relevant AND novel (Y/N) Chat 2 Relevant AND novel (Y/N) Chat 3 Relevant AND novel (Y/N)
Background Background
4. State the intended audience and users of the consensus outcomes. Y
7. Identify the intended audience for the consensus findings. Y
8. State whether the consensus was part of a larger initiative or project. Y
Methods Methods
11. Explain how consensus questions or domains were prioritized. Y
17. Explain how participant replacement between rounds was handled, if applicable. Y
25. Describe any weighting applied to different participants' responses or to different items. Y
Results Results
34. Report the distribution of responses, not just whether consensus was achieved. Y
43. Present minority viewpoints for items that did not achieve consensus. Y
37. Explain how decisions were made about borderline items that nearly reached consensus. Y
42. Report any unexpected issues that arose during the consensus process. Y
Discussion
46. Discuss practical implications and potential implementation of the findings. Y 45. Discuss the implications of the consensus results for practice, policy, or future research. Y
42. Discuss implications of the consensus results for practice, policy, or future research. Y
43. Explain items that did not reach consensus and possible reasons why. Y 47. Explain how variations in expert opinion were addressed in the final consensus. Y
44. Describe plans for dissemination and implementation of the consensus results. Y 49. Outline future research priorities based on the consensus process. Y 48. Discuss how cultural, geographical, or disciplinary differences might affect the application of the consensus outcomes. Y
50. Describe plans for updating the consensus findings over time. Y 49. Describe plans for updating the consensus in the future. Y
Other Items Other Items
47. Provide access to the full consensus process materials. Y
54. State whether and where the full dataset can be accessed. Y 53. Describe any quality assurance measures used during the consensus process. Y
49. Report whether the consensus process included international participation and describe any translation methods used. Y
50. Explain how consensus items were prioritized, if applicable. Y
51. Describe any quality control measures used during the consensus process. Y
Model: DeepSeek R1
Chat #: 1–3
Chat 1 Relevant AND novel (Y/N) Chat 2 Relevant AND novel (Y/N) Chat 3 Relevant AND novel (Y/N)
Background
5. Mention any previous attempts to address the problem through consensus (if applicable). Y 5. Explain the rationale for selecting the consensus method (e.g., Delphi, Nominal Group). Y
Methods
8. Describe the role of facilitators/moderators and whether they were trained for the process. Y 8. Describe efforts to include diverse participants (e.g., marginalized groups, non-English speakers). Y
9. State whether participants gave consent to join the study and how this was done. Y
10. Describe preparation steps for participants (e.g., training materials, pre-surveys). Y
11. State whether questions or voting materials were tested for clarity before the consensus process (if applicable). Y
12. Explain whether participants could suggest new topics or revise questions during rounds. Y
10. Address ethical considerations (e.g., informed consent, confidentiality). Y
15. Explain how disagreements were resolved (e.g., structured debate, iterative revisions). Y
16. Report modifications to the process during the study (e.g., added rounds, adjusted questions). Y
18. Describe how disagreements or non-consensus items were handled (e.g., discussed, excluded). Y 18. Report ethical approval status and consent procedures (e.g., verbal, written). Y
19. Describe facilitator training/selection criteria (e.g., neutrality, conflict management). Y
20. Report the start/end dates and duration of each stage. Y
Results
13. Highlight unresolved disagreements and summarize changes to statements across rounds. Y 22. Summarize how opinions changed between rounds (e.g., increased agreement). Y
24. List any items that did not reach consensus and explain why. Y 25. Describe documentation of minority viewpoints (e.g., dissenting opinions in appendices). Y
26. Report time required to complete the process (e.g., round durations). Y
27. Explain how outcomes were finalized (e.g., participant approval, editorial review). Y
Discussion
14. Interpret the implications of the findings, compare them to existing evidence, and address limitations. Y 25. Discuss how the consensus findings relate to existing knowledge or practices. Y 28. Compare consensus outcomes with existing evidence/guidelines. Y
15. Provide recommendations for applying the consensus results (e.g., clinical practice, policy). Y
27. Explain how the results could influence research, policy, or clinical care. Y 30. Describe validation steps (e.g., external review, comparison with other methods). Y
28. Compare the chosen consensus method to other approaches and discuss generalizability to other settings. Y 31. Discuss implications for research, policy, or practice. Y
32. Address generalizability to other settings/populations. Y
33. Reflect on how participant diversity influenced outcomes. Y
Other Items
31. Confirm whether ethical approval was obtained for the study. Y 36. Describe accessibility of outcomes (e.g., open-access, stakeholder summaries). Y
19. † Address reproducibility (e.g., data availability). (Optional) Y 37. Report translation support for non-primary language speakers. Y
20. † Describe validation steps (e.g., external review). (Optional) Y 33. Explain how others can access the data or materials from the consensus process. Y 38. Describe participant contributions to interpreting results (e.g., co-authorship, feedback). Y
Model: Gemini 1.5-pro-002
Chat #: 1–3
Chat 1 Relevant AND novel (Y/N) Chat 2 Relevant AND novel (Y/N) Chat 3 Relevant AND novel (Y/N)
Background Background Background
4. Explain why the specific method of consensus was chosen for the study. Y
Methods Methods Methods
12. Describe the instructions given to the expert panel. Y
Results Results Results
16. List the items that did not reach consensus in the consensus study. Y 15. Include the number of statements assessed in the literature search. Y
19. Report any change to the expert panel between rounds (e.g., experts leaving, new experts joining). Y
Discussion Discussion Discussion
Other Items Other Items Other Items
Model: Gemini 2.0 pro experimental
Chat #: 1-3
Chat 1 Relevant and novel Chat 2 Relevant and novel Chat 3 Relevant and novel
Background
4. State if a review of existing information informed the consensus topics. Y
5. Justify the choice of the consensus method. Y
Methods
6. Explain any modifications made to the consensus method, and state the reasons. Y
7. Describe the step-by-step procedure of the consensus method. Y
12. Explain how diversity among participants was achieved (if applicable). Y 9. Describe how non-responders to the invitation were followed up. Y 9. Explain how participants communicated (e.g., online surveys, in-person meetings). Y
13. State whether participants provided informed consent. Y
15. Describe the procedures of each round. Y 12. Explain the rationale for selecting these particular participants. Y
15. State if discussions were as a single group or in multiple smaller groups. Y
19. Describe any changes made to the chosen consensus method and explain the reasons. Y 16. If multiple groups were used, state if any measures were taken to standardise the experience between groups. Y
20. Describe any pilot testing of the consensus process. Y
22. Explain the rationale for the chosen definition of consensus. Y
20. Describe what was measured at each stage of the process. Y
24. Explain the procedure if consensus was not reached. Y 21. If changes were made, state whether panellists were told. Y 21. Explain whether participants could discuss results between rounds. Y
26. Describe what type of analysis was planned. Y 23. Explain how changes were made to the list of questions after each round. Y 23. Describe how disagreements between participants were handled. Y
24. State whether feedback on any draft versions of the output was sought from experts or non-experts. Y 24. Explain how final decisions were made based on the consensus process. Y
28. Justify why this type of analysis was appropriate. Y 25. Describe any changes to items, questions, or materials during the process, and provide justification. Y
29. Describe whether the final analysis differs from the initial plan, and state the reasons. Y 26. Provide a pre-specified plan of analysis, made prior to the consensus process, describing how consensus would be determined. Y
27. Justify any changes made to the pre-specified analysis plan. Y
28. Describe the development of any questionnaires or materials used in the consensus process. Y
Results Results
32. State which items or ideas did not reach consensus. Y
33. Provide information about the characteristics of the experts to aid interpretation of the results. Y 34. Summarize the main participant comments, even for items not reaching consensus. Y
36. Describe any significant disagreement on items. Y
Discussion
34. Discuss the main findings. Y 37. Discuss the main findings of the consensus process. Y
39. Compare the results to other similar studies. Y
40. Compare the results to existing evidence. Y 37. Discuss how the findings compare to other similar studies. Y 40. Discuss the implications of the consensus results for research, practice, or policy. Y
41. Discuss the implications of the findings for practice, policy, or research. Y 38. Discuss the implications of the consensus findings. Y
42. Consider factors that might have influenced the results. Y 39. Discuss any unexpected findings. Y
Other Items
41. State where the study was registered (if registered). Y
46. State whether an ethics committee reviewed and approved the study. Y 44. State whether any external support (other than funding) was received. Y
47. State whether the raw data will be shared publicly. Y 45. State if the study received ethical approval, and from where. Y 45. Indicate if any support, other than funding, was received. Y
48. Describe or share any pre-written plan (protocol) for the consensus process. Y 46. State whether an ethics committee or institutional review board reviewed the consensus study. Y
TOTAL 18 16 18
Model: GPT-4.5
Chat #: 1-3
Chat 1 Relevant AND novel (Y/N) Chat 2 Relevant AND novel (Y/N) Chat 3 Relevant AND novel (Y/N)
Background Background Background
3. Explain the rationale for selecting the specific consensus method (e.g., Delphi, nominal group technique). Y
4. Specify the intended audience or beneficiaries of the consensus findings. Y
Methods Methods Methods
15. Document clearly the instructions provided to participants at each consensus round. Y
17. Identify any software or statistical tools used in the analysis. Y
Results Results Results
26. Document reasons for participant dropout, if known. Y
Discussion Discussion Discussion
29. Provide recommendations for future consensus research. Y
30. Discuss alignment or differences between findings and existing research or guidelines. Y
Other Items Other Items Other Items
35. Include a visual summary (e.g., flowchart) of the consensus process. Y
36. Describe procedures for updating consensus findings as new evidence emerges. Y
Model: GPT-4o
Chat #: 1–3
Chat 1 Relevant AND novel (Y/N) Chat 2 Relevant AND novel (Y/N) Chat 3 Relevant AND novel (Y/N)
Background
5. Highlight gaps in consensus guidelines and explain how the study addresses these gaps. Y
Methods
29. Explain criteria for retaining or excluding items based on statistical thresholds. Y
Results
Discussion
Other Items
48. Identify gaps in current consensus methodology or reporting practices and suggest ways for future studies to address these gaps. Y
49. Highlight the need for further research into specific aspects of consensus methodology (e.g., feedback mechanisms or participant selection processes). Y
51. Include templates, forms, or questionnaires used during the consensus process for replication by other researchers. Y
Model: GPT-o1
Chat #: 1-3
Chat 1 Relevant AND novel (Y/N) Chat 2 Relevant AND novel (Y/N) Chat 3 Relevant AND novel (Y/N)
Background
1. Include a descriptive and informative title that reflects the objective of the consensus study. Y
5. Describe any theoretical framework or model guiding the study. Y
Methods
8. Report any ethical approvals obtained and consent procedures followed. Y
22. Report methods used to prevent or address potential biases (e.g., selection bias, response bias). Y
23. Describe ethical considerations, including informed consent procedures and ethical approval details. Y
Results
31. Summarize feedback provided to participants in each round. Y 28. Include any qualitative feedback from participants that influenced the results. Y
33. Describe challenges encountered during the study and how they were addressed. Y
41. Include detailed statistical results supporting the determination of consensus. Y
Discussion
30. Interpret the results in relation to the study objectives and existing literature. Y
36. Outline the implications of the consensus findings for practice, policy, or further research. Y
45. Consider the impact of participant selection and response rates on the results. Y
33. Consider the implications of the consensus results for practice, policy, or future research. Y
39. Discuss unexpected findings or deviations from expected results. Y
48. Recommend areas for further study or investigation. Y
36. Suggest areas for future research based on the consensus findings. Y
Other Items
41. Provide definitions of all technical terms and concepts used in the study. Y 37. Include acknowledgments for contributions from individuals or organizations not listed as authors. Y
42. Acknowledge external support, contributions, or assistance received. Y 38. Make available supplementary materials, such as questionnaires, data collection forms, or detailed results. Y
43. Specify the roles and contributions of each member of the research team. Y
44. Include a statement regarding the availability of supplementary materials or appendices. Y
45. Describe how the consensus process was monitored and quality assured. Y
46. Discuss ethical issues encountered and how they were resolved. Y
47. Include a conclusion summarizing the key outcomes and their significance. Y
57. Discuss ethical considerations and how they were addressed. Y 44. Include references to all sources cited, ensuring proper attribution and allowing readers to locate the original materials. Y
45. Comply with ethical standards for research, including obtaining informed consent from participants when applicable. Y
Model: Llama 3.2
Chat #: 1-3
Chat 1 Relevant AND novel (Y/N) Chat 2 Relevant AND novel (Y/N) Chat 3 Relevant AND novel (Y/N)
Background
3. Identify the target audience or stakeholders for the consensus study. Y 3. Identify the target audience or stakeholders for the consensus study. Y 3. Describe the context in which the consensus study was conducted. Y
4. Define the scope and boundaries of the consensus study. Y 4. Identify the stakeholder groups or organizations involved in the consensus process. Y
Methods
10. Describe the method for handling disagreements or non-consensus. Y
11. Analyze participant responses using statistical methods. Y
12. Report the software or tools used to facilitate the consensus process. Y
Results
15. Identify and discuss areas of disagreement or lack of consensus. Y
Discussion
13. Discuss the implications of the consensus study findings for practice, policy, or future research. Y 16. Discuss the implications of the consensus results for biomedical research and clinical practice. Y
19. Explore the potential applications and future directions of the consensus study findings. Y 18. Describe the plans for disseminating the consensus results to relevant stakeholders. Y
Other Items
23. Provide access to the raw data and results of the consensus study for transparency and verification. Y 21. Outline the process for maintaining or updating the consensus statement over time. Y