Evidence Prime Webinar: Why GRADE? 15 reasons to start using GRADE for your guideline development

‍
The second webinar of Evidence Prime has been another huge success.
Thank you all for your active participation!

We would like to especially thank our guest speaker, Holger J. Schunemann,a professor in the Department of Health Research Methods, Evidence, and Impact and of Medicine at McMaster University, widely considered the birthplace of evidence-based healthcare. He is also the director of Cochrane Canada, co-chair of the GRADE working group, and co-director of the WHO Collaborating Center for Infectious Diseases, Research Methods, and Recommendations.

If you want to read more about our speakers and the event, click here: Webinar info.‍

If you would like to learn more about GRADE Working Group, visit: https://www.gradeworkinggroup.org/

The theme "Why GRADE?" has sparked a great deal of interest. During the event, we welcomed a very active audience of almost 250 participants from all over the world, representing various fields of expertise and nearly 600 registrants who received the video recording after the event.

We talked about why to use GRADE for your guideline development and how the GRADEpro tool can help you create guidelines according to the best practices of GRADE.

To read more about our tool GRADEpro and test it for free, check https://www.gradepro.org/. 
To learn more about PanelVoice and other features, visit our constantly expanding Knowledge base.
To use our solutions in your organization, contact us at gradepro@evidenceprime.com, sales@evidenceprime.com.  

We received a lot of fantastic feedback during the webinar and plenty of questions, not all of which we could answer during the sessions due to the time limit. Therefore, we have compiled a summary of our Q&A session to answer all your inquiries and solve your doubts.

‍

Q&A Session

‍

GRADE Methodology

Can GRADE be used to assess evidence from case reports and short case series? If not, are you aware of any method we can use to evaluate evidence of medication efficacy in rare diseases?
The question was answered LIVE

Yeah, a great question. There are two things to distinguish: one is that a few tools out there have been published about assessing the quality of the case reports or case series. That is similar to the risk of bias tool that we can use to assess the quality (a term that we use) of randomized trials or non-randomized studies. So, with that in mind assessing the risk of bias of limitations of case reports in case series, it becomes pretty evident that they will be at high risk of bias and typically end up as very low certainty of evidence. And so, case reports and case series can be used in the same way that other types of research can be utilized. The other thing that might be relevant here is the work we've done around expert evidence and how to assess single cases. Again this will typically lead to very low certainty of the evidence, I should say, but GRADE is equally applicable there. 

Rare diseases - I also refer you to some literature that we've conducted, happy to disseminate this: how to use GRADE in the context of rare diseases?: 

• McMaster RARE-Bestpractices clinical practice guideline on diagnosis and management of the catastrophic antiphospholipid syndrome
• Methodology for the development of the NHF-McMaster Guideline on Care Models for Haemophilia Management
• GRADE notes: How to use GRADE when there is "no" evidence? A case study of the expert evidence approach
• Distinguishing opinion from evidence in guidelines

Great information about GRADE. I have a question about whether GRADE can be applied to systematic qualitative reviews or scoping reviews?
The question was answered LIVE

Ah, great question! I have big problems with the term "scoping reviews" and assessing any certainty based on that. I look at scoping reviews in the way that it basically gives you a "lay of the land," and you shouldn't be when you are not systematic. So unless you identify a systematic review that you find credible, you shouldn't be rating the certainty in this, but qualitative evidence absolutely! I should have mentioned this probably under prognosis, and this is an excellent reminder for me to add it under that particular slide. 

GRADE CERQual (Confidence in the Evidence from Reviews of Qualitative research) is created for qualitative evidence, and it may make enormous contributions to understanding how much trust to put in qualitative research. GRADE CERQual is the answer to the qualitative research, and I refer you to those publications again also available on this website: Cochrane Training: GRADE approach  

Is it reasonable for a group to say they are "using GRADE methodology" if they only address the certainty of the evidence on benefits and harms and do not take it to the next step of developing recommendations based on the other factors?
The question was answered LIVE 

It all depends on how the group presents it. If a group says: we use the GRADE to assess the certainty of the evidence of our systematic reviews, and they indeed did it, then that's perfectly fine.

Now, they shouldn't say: we use GRADE to develop our recommendations if they didn't use GRADE, and what I will refer you to, the easiest way of doing this is to go to the criteria for using GRADE. So, in other words, we have put together a minimal set of criteria that groups should be following to claim that they use GRADE.

I will also say that GRADE has never gone into the policing business. That's not our task. So we are not going out there and saying: this group uses GRADE, and this group didn't use GRADE. First of all, because we wouldn't have the resources, but also, I don't think it's in the spirit of GRADE.

What is the difference between grading the evidence itself vs. grading the strength of the recommendation based on the result of the evidence?
The question was answered LIVE

GRADE's recommendations are interpretable. 

One aspect relates to addressing the certainty of the underlying evidence. So, for instance, the certainty that we have in an intervention effect or in the values that people assign to outcomes is one part of the work. The other part of the work relates to balancing the desirable and undesirable consequences, where the certainty of the evidence is one factor to arrive at a recommendation.

GRADE makes at least two types of recommendations in the recommendation context: strong or conditional recommendations - they're also called weak recommendations, but we've moved towards that terminology (conditional).

The strong recommendations are typically a result of moderate or high certainty evidence. Typically, I say, when the balance of the desirable and undesirable consequences is clear. Conditional recommendations are a result of when things are not as clear. We've provided interpretation aids for patients, or people affected by a condition, the clinicians, policymakers, and researchers, what these strong recommendations mean. So, for instance, when there is a strong recommendation for additional research, unless they are based on very low certainty evidence, as an exception, it is unlikely going to have a significant impact on this recommendation, as an example.

Can we create a recommendation based on just one study with low quality?

Yes. This is well described in the GRADE methodology.

If there are disagreements in the author team related to any domains, e.g., a few team members feel a value is low, and the rest feel it's moderate based on the evidence - how do we come to a consensus - is there any standardized approach?
The question was answered LIVE

Yeah, it's a great, great question. So, there are a couple of ways of dealing with this. We don't necessarily have a standardized approach, but we utilized one approach similar to systematic reviews. It is to use them in independent arbitrage. In other words, think of inclusion and exclusion of an article in a systematic review: different opinions, you get a third senior investigator to help resolve the issue. That is one of the approaches that we would certainly suggest, however really important is the transparency of GRADE. Most domains result from judgment (I'm now talking about the evidence domain's certainty).

So most importantly, if you encounter a scenario like this, it is to describe in a footnote that there were alternate views. For instance, whether or not to rate down for indirectness by those who rated the certainty. Then you can say through arbitration, which a third senior investigator is doing,  providing that the final judgment, that the final decision was set, but it would have been reasonable to not weigh down for indirectness because of x y and z. That provides the transparency we are looking for, addressing the issue.

The other thing is, if that's related to actual judgments about a criterion on the Evidence to Decision (EtD) frameworks, what we say is: if consensus doesn't form, we often revert to voting to get to a final answer on a criterion in EtD.

How may we use GRADE on Consensus recommendations in clinical guidelines?

Consensus is always needed in guidelines. Therefore, nearly all recommendations are based on consensus.
See the following text for clarification:

• Methods for Development of the European Commission Initiative on Breast Cancer Guidelines: Recommendations in the Era of Guideline Transparency
• Idiopathic pulmonary fibrosis – clinical management guided by the evidence-based GRADE approach: what arguments can be made against transparency in guideline development?
• Distinguishing opinion from evidence in guidelines

How to address different opinions when judging the strength of recommendations when the certainty is conditional? Examples of how much to grade down or improve the grading of the certainty of evidence when evaluating the risk of bias and heterogeneity?

Moderation by a third reviewer. The Cochrane Handbook on GRADE provides guidance for how much to rate down.

Should we only include more experienced graders, i.e., more expertise in the topic?

Training in GRADE is necessary. The degree of which may differ by topic.

Could you provide more specific information on how to judge the GRADE, especially imprecision of results (e.g., are there any threshold to downgrade based on sample size, confidence interval)?

Currently, the new guidance is being published. The Journal of Clinical Epidemiology (JCE) GRADE series, two papers, numbers 34 and 35, are now under review. The two papers discuss that issue in detail. They relate to decision thresholds and have a calculator.

How do you incorporate Minimally Important Difference (MID) thresholds and imprecision in GRADE?

The MID corresponds to the threshold of going from trivial or none to small effects when using the GRADE EtDs. However, using the MID alone does not permit using the GRADE EtDs entirely.

Can we use words and phrases such as "maybe" / "can be" in the final recommendations, based on the relative strengths or weaknesses of evidence?
The question was answered LIVE

Maybe, can be?

So, GRADE, in one of the papers in the series, has provided some suggestions for standard wordings for the recommendations, and "may" is, for instance, one of the terms that we've utilized, or suggested for conditional recommendations, "can" is probably not very helpful. From Rick Schiffman and colleagues from quite a few years ago, there is a paper about what phrasing to utilize in recommendations. GRADE has adopted many of these suggestions. So, for instance, one of the tips is to avoid the term "to consider an option" because, by the time you look at an option, you're already considering it. It doesn't provide the type of guidance we need. "Should" is typically a word that we utilize in strong recommendations, and "may" is usually a word that we use in conditional recommendations. See the publications on the GRADE training site maintained by Cochrane.

Has GRADE been used in animal health research?
The question was answered LIVE

Another great question! Yes, indeed. Two things are relevant. 

One is in GRADE guidance for animal research and how it leads to translational research: Facilitating healthcare decisions by assessing the certainty in the evidence from preclinical animal studies.

The second aspect is that much work is ongoing in the environmental health project group. That in part deals with, for instance, mechanistic evidence and how that obviously can come from in vitro and in vivo studies. That includes animal research and how to utilize that information, for instance, by informing judgments about indirectness. 

Has GRADE been used in Correctional Settings, Social Justice?

I have not heard of that, but we have been thinking about social sciences. Nothing speaks against using this approach.

As the leader of one of the 10 Systematic reviews on health hazards from exposure to radiofrequency electromagnetic fields, I am particularly interested in the development of the GRADE approach to assess the certainty of evidence in evidence-based hazard assessments. In our systematic review protocol (see Lagorio et al. Environ Int 2021), we envisaged following the NTP-OHAT GRADE-based approach. Has there been further development of GRADE in this research area?

There are many articles about exposure assessment by GRADE. Further work is coming out. I suggest joining the GRADE project group on environmental health.

Do the GRADE criteria differ depending on the type of clinical question?

Not in terms of domains to assess the certainty of the evidence. Users of GRADE EtDs may select the criteria for decision-making based on context and perspective.
For criteria about using GRADE, go to Criteria for applying GRADE.

During the pandemic, there were some contrasting guidelines by CDC and WHO. For example, masks for kids less than five years old. Any thoughts or comments?

I suggest using the following source: COVID19 Recommendations maps. An article on diverging recommendations and how they come about is also being published by Z. Nasir et al. The exact population and other factors were common reasons when a review was done.

Does using GRADE differ in the context of an overview of reviews versus other contexts, such as guideline development? If so, how do the different contexts influence its implementation?

GRADE use depends on contexts. E.g., for decision making. Also, for the certainty of evidence assessment. See:

• The GRADE Working Group clarifies the construct of certainty of evidence
• Defining ranges for certainty ratings of diagnostic accuracy: a GRADE concept paper
• GRADE approach to drawing conclusions from a network meta-analysis using a partially contextualised framework

Where does the GRADE Working Group stand with using NMA in the GRADE method, and when is the Group planning to add the NMA results functionalities to be incorporated on the GRADEPro website?

We have published several articles on NMA:

• GRADE guidelines 33: Addressing imprecision in a network meta-analysis
• Cochrane - other publications by GRADE Working Group

Are you familiar with those? We have a Multiple Intervention Comparison module in GRADEpro (NMA is only solving a minuscule part of the problem of multiple interventions).

Regarding GRADEpro, both the NMA Evidence Table and Multi Intervention EtD, which utilizes NMA as evidence for effect judgment, will appear in GRADEpro in 2022

Do you foresee that appraisal of the certainty of evidence can be automized, e.g., with artificial intelligence?‍

I would certainly think that this is going to happen. However, I think we are still 5 - 10 years away. Nevertheless, we are making some progress with imprecision ratings and will make that available soon.

GRADEpro

Introductory information on GradePro would have been helpful. For example, who can use the tool, in which situations, teaching, as an exercise, or developing a national guideline?

The free accounts are available for everyone. The features and limits of these accounts are listed here: GRADEpro pricing. For more advanced projects, when those limits would not be sufficient, you can procure a Team or Enterprise subscription, depending on the size of your team and the scope of your project.

For introductory information, you can visit our Knowledgebase, which is currently being expanded.

How are different stakeholders involved and become acquainted with the tool? How do people find GRADEpro, and how do they get to know the tool - i.e., do we offer workshops, training sessions, or just this kind of webinars? How do they find out about GRADEpro - via GRADE WG?

Anyone can create an account at GRADEpro. We advertise our tool in the scientific community. With the prominent example of GRADE Working Group, many research groups utilize GRADE for the evidence assessment and use GRADEpro in their works. You can find our software cited in numerous papers.

Grade Working Group organizes GRADE workshops, more information you can find here.
‍INGUIDE also is organizing informative training.

GRADEpro itself offers a Knowledgebase, which is currently being expanded. In addition, some future webinars will also serve as GRADEpro instructions if you want to learn more. Subscribe here to stay up-to-date!

What software is available to produce outcome summary tables, or can we directly generate them in GRADE?

GRADEpro allows you to generate five different formats of summary tables. All you need to do is populate the outcomes with the data from your meta-analysis. You can also generate GRADEpro tables from within RevMan Web if you use this software.

Are any of the features on GRADEpro you mentioned NOT available in the free subscription?

PanelVoice mentioned in the presentation is available only in the team and enterprise license.

Is there a chance to try the GRADE software?

Anyone can create a free account on GRADEpro to try its features out. Many of the features are available for free. You can find out which ones at: GRADEpro pricing. You can also contact our Sales Team at sales@evidenceprime.com to set up a meeting or a demo for your team or organization.

I'd be interested to know more about the use of multiple comparisons on the software. The last time I looked, it was limited to single comparisons. Was there any development in this?

The currently available tool is Multi-Comparisons, which allows for grouping together several pairwise comparisons and comparing them in terms of various Evidence to Decision criteria. There will also be Evidence to Decision tables allowing for comparing multiple interventions and the Network Meta-analysis tables coming later this year.

Can we use GRADEpro during any systematic review by a group of authors and give guideline recommendations? Or can it be only used by a specifically trained panel of experts/larger societies? ‍

GRADEpro provides many tips and helpful materials. These are, e.g., the GIN&McMaster Guideline Checklist, explanations of tables' elements available through information icons, the GRADE Handbook, and our Knowledgebase, which is currently being expanded.

Joining GRADE Working Group - training, certifications, workshops

Thank you for the excellent webinar. I am interested in joining the Guidelines Working Group. Please advise how to become a GRADE Working Group member?
The question was answered LIVE‍

The link to the GRADE membership - it's just subscribing to our listserv to become a member of the group. You can also sign up for project groups. So you should sign up with your email, which means that you will receive the emails of the GRADE Working Group. We use that listserv exceptionally sparingly. In other words, we are not using it for any unnecessary information. Instead, it informs you about the meetings and the project groups. We try to keep it this way only to get the key information relevant to you.

Do you provide any formal training or certification to interested people in joining your team? Do you run Workshops?
The question was answered LIVE

Joining our team - The GRADE Working Group does training and workshops for GRADE. We do have the training program for INGUIDE, which includes GRADE. 

In terms of joining our team, it's often learning by doing. We provide a set of resources if somebody is interested in working with us on a guideline. Then we typically encourage people to participate in one of the  GRADE workshops, where the team members get acquainted with the approach. There is a lot of learning by doing. Colleagues in a team working on a particular guideline support the new members. That's the approach, but again, to be a methodologist for guidelines, because GRADE only covers certain aspects of GRADE - we've developed that INGUIDE program that I mentioned a bit earlier.

Other webinars

Is it possible to get the link for the recording of the previous and this webinar?

‍‍You can contact us at webinars@evidenceprime.com to receive the previous webinar recording link: Artificial Intelligence in Evidence-Based Healthcare: What to expect?

To stay up to date with future events, follow us on our social media channels and SUBSCRIBE to our newsletter: Newsletter Subscriptions (evidenceprime.com)

‍

Quotes and feedback

An excellent presentation. Thank you very much for this information and synthesis.
Neat and concise presentations. Thanks.
INGUIDE is Excellent! Highly recommended.
Thank you, Holger - A sweet capsule of GRADE!
GRADE is indeed fun!

If you have any more questions, comments, or suggestions for the following topics that would be of your interest, please do not hesitate to contact us at webinars@evidenceprime.com

Evidence Prime team

‍

‍

Our next webinar is touching on the topic widely requested by GRADEpro Community. To stay up-to-date - Subscribe: Newsletter Subscriptions (evidenceprime.com)

‍

‍

Follow us on our social media for more updates:

Evidence Prime:

Laser AI:

‍

‍

GRADEpro:

‍

‍