Guidelines Development Workshop-Session 4: Understand how to determine the strength and direction of a recommendation based on the evidence, review examples of recommendation statements and good practice statements

Session 4: Understand how to determine the strength and direction of a recommendation based on the evidence, review examples of recommendation statements and good practice statements

Back to course

Video Transcription

So, yes, so at this point we are going to take it home. So we are going to take the evidence and go from the evidence to recommendations statements.  And I have the slide up, you know, that's, you know, has decision making on it. And so we are sort of at the last stage of some of the crucial decision making process  that goes into the guideline development process. But, you know, all along this process decisions have been made.  You know, decisions about the clinical issues that are important to provide guidance on and make recommendations about, key questions, you know, the inclusion exclusion criteria,  decisions about how we rate the quality of the evidence. And so now we are here at the decision about can we make the recommendation that we intended  to make when we were thinking about those clinical areas that are important to provide guidance on. So this is the sort of final stages of our decision making process.  I think, you know, one of the things to bring up here that you see about the grade system is its strength in providing us with decision aids.  So, you know, as we are going through this process of identifying our priorities, identifying the evidence, assessing that evidence, and then going to making recommendations based  on that evidence, we have had criteria to follow that help us in that decision making process to make it a structured process, as John talked about, and a transparent process.  So when we come to, you know, the recommendations, you know, it can be sort of challenging process  to begin, but fortunately, you know, grade sort of allows us or limits our choices in a way. So we are going to be making recommendations starting with, well, what direction is our  recommendation going to be in? Are we going to recommend for or against? And then we're going to open up to more categories that are going to help us determine how strongly  we're going to recommend for or against. And here we have sort of a continuum of categories where we could strongly recommend something,  we could strongly recommend against something, or we will have a weak for or a weak against. And I'm going to talk about, you know, how we make these decisions about the strength  of the recommendation. We do have criteria to follow, domains that we can think about.  You know, it's oftentimes with recommendations that we're sort of in this middle of the continuum, you know, we don't often see evidence that's on either extreme of the continuum.  But we do sometimes, and we're going to talk about that evidence. So I just want to point out that here, when we're thinking about the recommendations,  we're first thinking about, is it going to be a recommendation for or against, and then we're looking at our continuum of how strongly we recommend that recommendation, how strong  that recommendation is. So let's just talk about what we mean about the strength of the recommendation, like what is that sort of, you know, the broader aspect of that.  So when Gray is talking about the strength of the recommendation, what it's referring to is the extent to which a guideline panel is confident about the desirable effects of  an intervention, and the desirable effects of an intervention outweigh the undesirable effects. So do the benefits outweigh the harms, and then vice versa.  So if we have a strong recommendation, then the panel is highly confident that the benefits do outweigh the harms.  If we have a weak recommendation, which is sometimes referred to as a conditional recommendation or a discretional recommendation, then the panel is less confident.  And we're going to break this down even further into how we can sort of anchor our thinking about what would be strong and what would be weak.  And so Gray provides us with domains to think about. One of those domains is balancing, as I mentioned, the benefits versus the harms, so the desirable and undesirable effects.  So the larger the difference between the desirable and undesirable effects, the more likely we're going to have a strong benefit or a strong recommendation.  The smaller the difference, then the more likely we will have a weaker recommendation. And you're going to see examples of this as I go through my presentation.  So right now, I just kind of want to get those concepts out there that we are thinking about as we go into the examples.  And then we're going to look at the confidence of the magnitude of the estimate. So the evidence, the grade profiles and the evidence that John was showing in his  presentation, we are going to take that evidence and that becomes a part of the overall strength of the recommendation.  So how confident are we that the effects that we're seeing from the evidence are the true effects of the intervention?  So we're looking at the overall quality of the estimate based on the risk of bias, based on the consistency, the directness and the precision of that evidence.  But then there's other things that we need to consider when we are thinking about the strength of the recommendation.  And that is the values and the preferences of the patient population for whom this recommendation is going to apply.  So what do the patients value? What do the patients want? And we're going to talk about this in more detail in just a moment.  So when we're looking at the confidence of the evidence or the magnitude of the estimates, if we have higher quality evidence, we're more likely to make a strong recommendation.  If we have weaker quality evidence, we're more likely to make a weak recommendation or lower quality evidence.  And when we're thinking about values and preferences of our patient population, again, we're thinking about is there variability?  So really, the strength of our recommendation lies on the amount of uncertainty we have. So if we have less uncertainty about the balance between the benefits of the  harm, we're going to have a strong recommendation. If we have less uncertainty about the quality of the evidence, we'll have a stronger  recommendation. And the same with the values and preferences of our patient population. Other things to consider when we're looking at the strength of a  recommendation are resources.  So, you know, is this and not all guideline PALs, you know, think about this or need to think about resources. But, you know, is this intervention that costs a lot?  Is it an intervention that's accessible to all patients or not? You know, you could have a situation where there is a large difference between benefits  and harms and it's in the direction of benefits. You know, we have high quality evidence, but this is something that is going to cost a  lot. This is something that maybe not all patients are going to have access to.  So we have to think again, do we want to make a strong recommendation in those circumstances? And so, you know, this is kind of like a pros and cons list in a sense.  You know, you're looking at, you know, the pros of this intervention versus the cons or the harms and then thinking about, you know, how strongly those benefits and harms are.  And, you know, we can break down some of these domains and really dive into them. So, you know, the first thing is looking at the balance between the benefits and harms.  And when we're thinking about the balance between the benefits and harms, you know, we're  just looking at the evidence to see, you know, as John had said, we're looking at all of the  outcomes, all of our critical outcomes, some of which are measuring success or benefit of that intervention, some of which are showing us, you know, harms.  And so do the benefits outweigh those harm outcomes?  And when we're looking at this, we're thinking about, you know, how important are those benefits? And, you know, most of these are going to be critical outcomes.  So if they're a critical outcome, they're, you know, we're making, we've made the decision that it's, you know, it's an important benefit.  But in thinking about that importance, are we avoiding death or avoiding, you know, a lesser benefit or reduction in, you know, something like a rash?  So, you know, when we're looking at the benefits, we also want to think about how important these benefits are to our patients.  And when we also need to look at the importance of the harms.  And I just want to take a moment to talk about the importance of harms for a couple of reasons. So, you know, it's been my experience working with guideline panels that, you know,  when we're at that decision about, you know, what's a critical outcome or not, you know, adverse events sort of always get, you know, everything is a critical outcome.  Well, there's a couple of problems with that, you know, because harms sometimes are not always reported well in studies.  The evidence for harms can be of low quality.  And as John was saying, you know, the quality of our evidence is as strong as the weakest link. So, you know, choosing harms, you know, it's about choosing those harms that are  going to have an impact on the decision making process.  So, you know, if it's a critical harm, it needs to have an impact on that decision making process. And some of that impact is going to come in the terms of how important those  harms are to patients and to clinicians. So we need to, you know, always think about the types of harms that we are selecting as critical outcomes.  So I just want to give some examples, you know, of sort of the distinction of benefits versus harm, and these examples are coming from the grade handbook.  So, you know, I mentioned when there's a large difference between desirable and undesirable  effects, we'll have a higher likelihood of making a strong recommendation. So, you know, one example is that of a low dose aspirin on reductions of death and  recurrent myocardial infarction. So, you know, the benefits of low dose aspirin are pretty high, whereas the consequences  are typically mild side effects and it's a low cost intervention. So in this case, we would probably we would make a strong recommendation for low dose  aspirin. But then we might have another situation where the difference between the benefits and harms is not so distinct.  And so, like, for instance, if we have anticoagulant therapy for atrial fibrillation, you know, and the benefit is whether it reduces stroke.  And in this situation, we are seeing that it's a low benefit in the reduction of stroke. But the tradeoff is that we have a risk of bleeding.  So the risk of bleeding is, you know, somewhat uncertain. So we have this, you know, we don't have this distinction between a high benefit and a low risk of harms.  So then we have uncertainty. And so, as I said, when we have areas where we are uncertain, we are probably going to make a low recommendation or weak recommendation.  So one of the other domains when we're determining the strength of the evidence is looking at the confidence of the magnitude of the estimate.  So this is the evidence that John was showing us. And a strong recommendation is going to be based on high or at least moderate quality  confidence in the estimate of the effects of our critical outcomes.  And we have situations, you know, where we will have high confidence in some of our critical outcomes, but low confidence in the other critical outcomes.  So these others may be long term harms.  So when we have that uncertainty, we are going to have a weak recommendation.  Again, we could have a large difference between the balance of desirable versus undesirable outcomes and low confidence in the estimates of those effects.  So we could see a big difference.  But the quality of that evidence is low. So, again, we have uncertainty and then we're going to have a weak recommendation.  And so, you know, when we're thinking about the grade system and we have the symbols that grade uses in its evidence summary tables, we have, you know, the symbols for high  quality evidence, moderate quality evidence. And when we see this type of evidence, it's more likely to lead to a strong recommendation.  When we have situations of low quality evidence or very low quality evidence, we are going to have weak recommendations.  But there's often exceptions to these domains.  So and sometimes we might have low quality evidence suggesting a benefit in a life threatening situation and the evidence regarding harms can be low or high.  So when we think about John's example of the parachute, where the benefit is, you know, the benefit is there, maybe the evidence, because this is not going to come from randomized  control trials, it's going to come from observation. But nonetheless, there is an obvious benefit.  You know, we're likely to make a strong recommendation in those situations.  You know, again, you know, we're bringing in there's there's, you know, judgment here.  And, you know, that judgment is important and things have to make clinical sense. So you would not tell someone not to wear a parachute when jumping out of a plane because  the evidence is low quality.  And, you know, on the flip side, we could have a low quality evidence suggesting a benefit and high quality evidence suggesting a harm.  So here, you know, we would not necessarily because we know that the harm could be quite substantial and that's high quality evidence, just because we have low evidence on the  benefit side, we would not say not to do that. So, again, here's another situation where we have high quality evidence suggests modest  benefits and low to very low quality evidence suggesting the possibility of a catastrophic harm.  So we are thinking we're, you know, about the importance of these harms, the importance of these benefits.  And does the recommendation make clinical sense?  So our judgments have to come into play.  So here are some of those other considerations, and, you know, you can see how these considerations kind of flows through all of our decision making processes when we're  thinking about how important a benefit is or harm is. So, you know, patients values and preferences are not just sort of an isolatory category that you think of alone.  It kind of flows through all of our decision making process, but we do have to give them, you know, consideration in and of itself.  So, you know, we're determining the strength of our evidence. You know, ideally, it's nice to have actual evidence about patients preferences and  values. Maybe we have, you know, published evidence in the form of qualitative studies looking at satisfaction data or interview evidence about patients' perceptions about  treatment. But oftentimes we don't have that. So, you know, our interpretation of values and preferences of patients may come in the  form of expert opinion and what we know about what our patients want. But, you know, this is where it's important to have those patients represented in the guideline process.  So either as part of the guideline group, you know, or as a representative, or I've seen guideline panels who have done focus groups with patients  to have their input incorporated into the guideline. And it's during those times that you can ask about the preferences and values of the patients.  But when we're thinking about it in terms of how do we rate the strength of our recommendation,  again, we're looking at, is there uncertainty? So we're thinking about, is this something that patients would really want and they really value?  And so if we have that in addition to a big difference between the benefits and harms, and we have high quality evidence, and we know this is something that patients want,  then that's a strong recommendation.  If we have some doubts, maybe some patients would want it, maybe some patients wouldn't, maybe those harms are important to some patients,  but other patients, that's just a minor inconvenience. We have some uncertainty. This might lead to a weak recommendation.  And then we also have to think about equity and health disparities in our consideration of patient values, and again, resources and cost.  And I made the example of you could have a situation of a intervention that there's a lot of benefits, the evidence is strong, but it's costly and it's not accessible.  And a strong recommendation would not be warranted, and a strong recommendation in that circumstance could actually lead to some health disparities,  if not all patients have access to that care.  So I want to move on to some examples  showing how we would rate the strength of the evidence,  given the evidence that we have and the questions that are being addressed and the recommendations that might want to be made out of this.  The first example is looking at pulmonary rehabilitation among high-risk patients with COPD. So this example is coming from a Cochran review,  and I have the reference, I think, on the next slide.  And it's also an example that is used in the grade handbook, but I'm also pulling from the Cochran review for the summary of evidence table.  So the question here is, what is the impact of pulmonary rehabilitation compared to usual care on patients with COPD who experienced a recent exacerbation?  So we have our population, and those are participants with COPD who had experienced a recent exacerbation of their illness.  And the setting here varies. So some of these patients, because they had an exacerbation, they may be in the hospital, they may be in an outpatient setting,  they may be in home-based setting. So the rehabilitation can start in any one of those settings.  The intervention is pulmonary rehabilitation and the preparation is usual care. And there was no indication of a timeframe.  So here is the grade summary of findings table.  And so in looking at this table, the critical outcomes were hospital readmission.  And the other critical outcome was health-related quality of life. And this was being measured by the St. George Respiratory Questionnaire. And in that measure,  lower scores indicate higher quality of life. So just looking at this table,  we can see that there is a reduction and quite a large reduction in hospital readmission.  If we look at the confidence intervals surrounding the odds ratio that John described, we see that the confidence intervals indicate that this is a significant reduction.  So we have a precise finding. This is based on a meta-analysis. This is based on the pooled results of eight randomized control trials.  It was downgraded to moderate, and that is because, and I didn't put the reasons here, just issues with the risk of bias and heterogeneity.  Because we're looking at a program like pulmonary rehabilitation, there's various ways that this program is delivered.  So there's not just one way of doing pulmonary rehabilitation. So you're going to see heterogeneity between the studies and how the intervention is delivered.  If we look at the St. George Respiratory Questionnaire findings,  we are seeing that there is a decrease in this overall score of close to eight units. So that's a pretty large increase to say, in quality of life,  even though we're looking at lower scores being better. So in this particular instrument, a four-unit change is typically considered to be a clinically significant change.  So here we're seeing nearly a double of that. So this, again, is based on a meta-analysis of eight RCTs.  We have a few more patients in this analysis,  and we are looking at high-quality evidence. So you can pretty much see that when we're thinking  about the balance between the benefits and harms, that there's definitely a benefit in pulmonary rehabilitation. It reduces hospital readmission, it improves quality of life,  and those benefits are large, and they're important benefits.  There were no meta-analysis that were done on adverse events, and adverse events in this situation were probably not reported,  but there were five studies that reported on it, or not reported well, I should say, and four of them reported no adverse events, and one study reported a serious adverse event  in which a patient felt unwell, but those symptoms resolved quickly. So it was just temporary serious adverse event. So here, we see that the benefits definitely outweigh the harms.  And so when we are thinking about our strength of evidence,  the confidence in the quality of the evidence is high to moderate.  Thinking about patient values and preferences, most patients would probably value these benefits. You're not going into the hospital, your quality of life improves,  and there are inconveniences to pulmonary rehabilitation.  You have to go to it. It can be somewhat strenuous for some patients, but these inconveniences are likely not considered enough that it would prevent patients from going  and receiving pulmonary rehabilitation. So the strength of our recommendation here is going to be strong, and we are likely, we are going to recommend  pulmonary rehabilitation for patients with exacerbation of COPD.  So the next slide, and I'm gonna talk about language of the recommendation in just a moment, but before I do that, I just wanna show you, this is a GRADE evidence profile table.  So GRADE also has tools that will help you in this evidence to decision phase of the guideline development process, and they have evidence to decision tools.  And this is part of GRADE Pro. It's part of an extended version of GRADE Pro. So I don't know that it's freely available. You might have to pay, but nonetheless,  you know, it's a nice tool because it takes you through those donations. And, you know, it asks for your judgment about, you know, the balance between the benefits and the harms.  And, you know, here, yes, the benefits outweigh the harms. It allows you to put in your rationale or your judgment for why. And then there's other, you know,  there may be some other things that you might wanna consider in those judgments, such as subgroup findings, et cetera. So this, again, is a way of making  your decision-making process structured and transparent.  And it goes down through all of the different domains.  But let's talk a little bit about language. So when we have strong recommendations,  we can make our language, the strength of the recommendation is indicated in the language of the recommendation. So a strong recommendation allows us  to say something like, we recommend. And whereas a weak recommendation is not going to allow us to use that kind of language. But when we're making these recommendation statements,  we need them to be clear. We need them to give direction.  So a recommendation is about, are we recommending for, are we recommending against? And we need to have, we need to be able to tell clinicians the direction of that recommendation.  What is the action? What should they do?  So we have the language of we recommend or we suggest depending on the strength of that recommendation. So in grade, if you have a strong recommendation,  you can say, as I said, something like we recommend. If you have a weak recommendation, you would say something like we suggest because we have uncertainty.  And so the language of the recommendation is capturing that uncertainty in the use of the suggest term instead of the we recommend.  But whether it's weak or a strong recommendation, we still need to provide direction and action for clinicians to follow. And so grade has, helps, it does have some tools  that will help you translate and gives you examples of the kind of language you would use for your recommendation statement. But just in general about recommendation statements,  they need to be more than just statements of fact or a definition. So this example is not an example of a recommendation. So we wouldn't say something like  markedly elevated serum inflammatory markers, especially erythrocyte sedimentation, suggestive osteomyelitis is in suspected cases.  That doesn't really tell us much about what we should do. It's a fact.  And recommendations, they need to have that direction. They need to tell clinicians what they should do or not do. And so they can't be triggerless or imperative recommendations.  So something like the risk and benefits of tamoxifen should be given careful consideration in the decision-making process. Well, okay, but what do I do? You know, what do I do  after I've given it careful consideration?  So a statement like that doesn't necessarily give direction. It doesn't give action.  And it should not include contradictions. So, you know, in a statement like inpatients given PMRT, we suggest that adequately treating chest wall is mandatory.  So, you know, we're suggesting it, but yet it's mandatory. So those are two contradictory terms because as we know, a weak recommendation in grade would be a suggest,  but here we're making it mandatory as if everybody should do this, which would be a stronger recommendation.  So we need our language to be clearly stated in the recommendation statement, and we need to give direction  for the action that clinicians should be making. So, as I said, you know, a strong recommendation in grade would start with re-recommend,  or, you know, there's other language that could be used. Clinicians should, clinicians should not, you know, clinicians should do, should not do. When we have weak recommendations,  we have uncertainty and that's expressed in our language. So we would have, we suggest, or clinicians might.  And, you know, or you could say something, you know, a conditional recommendation and make your qualifiers for it. So we recommend a qualified recommendation based on,  and here's our reasoning for that statement.  So let's move on to another example. So this is looking at lopinifer or rotinifer  for patients with COVID-19. So the question here is what is the impact of lopinifer or rotinifer or slash rotinifer on patients with COVID-19,  regardless of severity or duration of disease? So our population, our patients with COVID, regardless of their severity.  This is any setting and the intervention is lopinifer or rotinifer, and the comparison is usual care. And this is actually taken from a World Health Organization recent guidelines  on COVID-19. So this is a recommendation taken exactly from their guidelines. So I have the link here.  So this is the grade evidence summary table for with the evidence that's addressing that key question.  We've got four studies,  three studies for some of the outcomes. So the critical outcomes here are mortality,  mechanical ventilation, diarrhea, nausea, and vomiting.  And if we look over at the relative effect here,  we're seeing uncertainty. We've got large, wide confidence intervals.  We are looking at a ratio outcome or estimate.  So the neutrality level would be one. So here we're seeing, there's not much impact on mortality with these interventions.  There's not much difference between the intervention and the usual care. And if we move on to mechanical ventilation, it's similar. Actually, there's sort of a slight increase  maybe with these interventions  with patients being on mechanical ventilation. But there's a lot of uncertainty here.  And the strength of the evidence or the quality of the evidence is moderate. So this is evidence that's suggestive that we're not seeing an impact of these interventions.  But we look at the harms, there's a different story.  So there's definitely evidence that these interventions lead to an increase almost four times of diarrhea.  And even though the odds ratio was not reported, you can see that there's an increase in nausea and vomiting. And this is coming from moderate strength of evidence.  So here we have a situation where we,  and we're looking at the balance between the benefits of harm. The benefits are just really minimal to no benefit  and reducing mortality or need for mechanical ventilation. But there are, it's an association with a higher risk of gastrointestinal distress. And this is in a group of patients  that are already compromised and very ill. So these harms are important harms.  So here's a situation where the harms are outweighing the benefits. And given that we have moderate quality evidence  and thinking about our patients' preferences and values, I would say there's little uncertainty that most patients would not wanna receive this  given the lack of benefits and the increase of harms.  So this is a situation where the strength of the recommendation is strong.  And we would say something, we recommend not to use organ affair. And this is what was in the World Health Organization's guideline. So these are pretty straightforward examples  of a strong for or a strong against.  But like I said, we're usually not in that arena.  We're usually in sort of this world of uncertainty. And I think this is an example that is gonna be familiar because we have talked about this throughout our presentation.  So here we have our nutrient composition in obese patients with metabolic syndrome example. So our question is for obese patients with metabolic syndrome, what is,  and I can't see because I'm in the way here, whoops.  What is the impact of dietary nutrition composition on weight loss and cardiovascular disease? So those were our critical outcomes.  Our patients are participants with metabolic syndrome. It's an outpatient setting. And if you remember, we're comparing a low carb diet to a high carb diet.  And here is the evidence, the grade evidence profile table that John showed you and went through all of the domains of quality  in terms of rating the evidence. And we came to an overall rating of low to very low for the cardiovascular disease. We saw that there is some slight difference  in weight loss favoring the low carb diet.  And there is also some improvement in cardiovascular disease as measured by the surrogate outcomes for the low carb diet. But there's a lot of uncertainty  because of the width of the confidence intervals. So, it's difficult to really say that the benefits  outweigh the harms here.  There is some indication of benefit. There's some weight loss and there's some improvement with cardiovascular disease as in both studies he had mentioned  that most of the measures, although indirect were statistically significant.  You know, harms, well, you know, there are no harms reported, but we could look at harms as maybe there's a burden in following a calorie restricted diet  or a diet in which, you know, you're changing the carb amount.  So, there's no clear distinction here. There's uncertainty. So, this is a situation where we would say, okay, the benefits kind of outweigh the harms.  But not enough that we are probably going to be able to make a strong recommendation because the confidence in our estimates are very low quality.  And that's being driven by the cardiovascular disease markers and patient values and preferences. It's somewhat uncertain. Some patients might value a minimal weight loss  or some increase in their blood pressure.  Most probably wouldn't, you know, for some people that weight loss might encourage them to at least continue with the diet. Others, it might be a discouragement  because they want to see a larger weight loss.  So, here's a situation where we are going to make a weak recommendation. And our language here would be, we suggest offering obese patients with metabolic syndrome, a low carb diet.  You know, it's something that the evidence is not suggesting that you wouldn't recommend it.  So, you don't want to take it out of the toolbox of what can be recommended, but it's not strong evidence that you would definitely say,  we recommend low carb diets for all patients.  So, here's some other examples where things are not, are even less uncertain.  So, there are situations where we may not make a recommendation. And, you know, this is when the confidence in the estimates are so low. So, maybe all of your outcomes, you know,  have very little quality evidence. And the trade-offs between the benefits and harms are so closely balanced that you can't make a decision one way or the other.  And there's a lot of uncertainty about patient preferences and values. So, in this situation, you may choose not to make a recommendation because the evidence is insufficient.  You know, making recommendations, it's not a guessing game. And if there's a lot of uncertainty, it may mean that, you know, that evidence isn't sufficient  and these other considerations don't provide a clear path for you to make a recommendation one way or the other.  So, here, you know, we would have a statement saying, the evidence is insufficient to make a recommendation about treatment X for patients with metabolic disease.  And then this is because, and this is the transparent part, is, you know, explaining why that evidence is uncertain.  And then, you know, we have other situations where we might not make a recommendation. And this is when we have no evidence.  And, you know, we've said this in previous presentations, you know, sometimes no evidence is evidence and it's evidence of the needs of future research.  So, in those circumstances, you know, a lot of guidelines will have a future research needs section and when you're identifying gaps in the evidence  that don't allow you to make a recommendation that you had intended to make, you know, that's the area where you can, you know, have these future research needs.  In some situations, though, you might not need evidence. You know, maybe, you know, what you were recommending  is all, what you were trying to recommend is already something that's commonly practiced. And so GRADE allows for good practice statements.  And so you would make these good practice statements, you know, in situations where what you're recommended  is already commonly accepted.  And you need to think about, do we need to make that message?  If you are making good practice statements, you want to be sure that these statements are going to lead to a large positive outcome. So if there's uncertainty about something,  you would not make a good practice statement.  And these are statements that are not graded. And, you know, oftentimes these are referring to, you know, standard of care.  And, you know, one of, you know, good practice statements that come up in some of the guidelines that I've worked on, or, you know, are not necessarily  in the good practice statements, but maybe in algorithms or maybe in discussion is shared decision-making. You know, that's just, you know, something that you should be doing.  We don't need evidence to say that that's something that you should be doing. So these are, those types of statements would be considered good practice statements.  But, you know, grade suggests using them sparingly.  And, you know, they're not, I mean, obviously you want to be able to make a recommendation that's based on evidence and based on the ability to weigh the benefits versus the harms  in a more standardized way.  So that's it.  You know, we've gone through quite a few examples. We've talked about, you know, the different categories of making recommendations and grade provides us with a decision guide  for making those in terms of the direction of our recommendation being strong for or strong against.  You know, we have criteria that help us to determine what the strength of that recommendation is going to be. And that criteria is being able to balance  the benefits versus the harms. It's our confidence in the magnitude of the estimates or our confidence in the quality of the evidence.  And then we also have to take into consideration other domains such as patient's values and preferences and resources.  You know, and a strong recommendation reflects a large difference between the benefits and harms and vice versa. And a strong recommendation would have high  to moderate quality and there would be high certainty of patient preferences and values. So we don't have uncertainty here. When we have a weak or conditional recommendation,  this is reflecting uncertainty in one of the domains of the decision-making process. So, you know, whether there's uncertainty about the benefits or the harms  or the quality of the evidence or the patient values and preferences. And language is important.  If we are making a strong recommendation, we can start that recommendation by saying we recommend. If we're making a weak recommendation,  we can start that recommendation with we suggest.  So I guess before I'm gonna turn it over  to John to make some closing remarks, some overall summarization.  But before that- Why don't we first ask for questions? Yeah, that's what I was gonna say. Before I do that, you know, are there any questions that folks may have?  Yeah, Stacey, I really liked your presentation. Regarding the strength and language of recommendations, in my experience, there is often hesitancy  and sometimes even aversion among panel members to include weak or what they consider negative language or weak recommendations.  I really like how you talk about that the goal here should be that, you know, we're reflecting what the evidence shows and with the goal of transparency.  And when there's no evidence, you can talk about points in future research or discuss gaps in knowledge in the document. Are there any other words of wisdom you could provide  to guide panel members regarding what those goals are and why it's okay to include recommendations that might be considered weak recommendations or use words like not,  which sometimes panel members aren't comfortable with? Sure, yeah. I mean, it's been my experience, yes, that when you use the term weak,  people think that they're not doing their job,  you know, that weak is, you know, just has a bad connotation. And, you know, I do wanna say that, you know, GRADE allows you to use other terms  to reflect the uncertainty of the evidence. And really, you know, it's about the uncertainty that you're trying to express.  And, you know, at the end of the day, you want to be able to tell clinicians, you know, to give them some action, but you also wanna be able to express your uncertainty  about that action. And so these weak recommendations, you know, can be phrased differently to reflect that. So you could, you know, I think,  I forget now the phrases that you could say, but, you know, one is that you might recommend, you know, kind of captures that uncertainty.  Okay.  Thank you. Stacey. Go ahead, I'm sorry. I just wanted to know, so when you don't have sufficient evidence to make a recommendation and you're relying on consensus opinion,  in your experience, have you identified ways that the work group comes to that? Did they use some type of like modified Delphi process  to vote on how they're gonna make a recommendation in lieu of evidence?  Yeah, that's a good question, Janice. Yeah.  You know, there's often debate about, you know, especially when there's uncertainty, you know, what to recommend or if we can make a recommendation or not make a recommendation.  And, you know, usually you need to lay out some rules and engagement in those circumstances. You know, a good understanding of what, you know, grade allows you to do,  well, I don't wanna say allows you to do, but just a good understanding of the system and how you would come to these decisions is important.  So, you know, and then oftentimes, you know, things are put up to a vote. I haven't been in a situation where they used a full Delphi method,  but, you know, I have been in situations where voting has come into play. So, you know, and then just being able to facilitate these discussions in a thoughtful way  that everyone has heard. And yeah, so those have been my experiences.  Stacey, I did have a follow-up about the no recommendation situation.  Would the grade group recommend an explicit statement that there is no recommendation on this topic or would the grade group recommend that it just be silent about that topic?  I guess it would depend on the evidence.  Well, you know, we have no evidence.  I think if, you know, if you can make a research recommendation,  you would want to make that kind of statement as opposed to being silent.  But if there is no reason to make that, you know,  if it's unlikely that this is going to be followed up with research, then, you know, maybe it would be a circumstance where you would be silent.  And I don't know, John, if you've had other experiences.  No, not that I could think of. I just thought of that as you were talking. Yeah.  Okay, any other questions?  If not, I will do a couple minutes of wrap up. All right, so we are going to wrap up for a couple minutes here. So the first session,  Stacy talked a lot about key question development, the scope of a guideline.  There was that ugly purple creature, the scope creep, who we often see as an issue in our world of evidence review. A big goal of key question development is to avoid vagueness.  Vagueness allows for a misunderstandings of what the scope was and maybe some reshuffling later on that, you know, you have to tweak things and it just makes more time and money  to deal with the vagueness later.  Another key part of the engine driving all this is study inclusion criteria. What are the requirements of the studies you have in order to be considered as evidence?  And the big factors there are how relevant they are,  how good quality the evidence is, and the time and money you have in order to do the evidence review.  You may remember in the previous session, I had a continuum of evidence quality  where evidence reviewers and guideline panels are faced with hitting that sweet spot between super high quality, but ending up with very little versus allowing low quality,  but having to deal with a whole lot of studies. So there's hopefully a balance there.  I spoke mostly about the grade certainty of evidence ratings. These are going to be for each outcome. You're going to have a starting grade, downgrades, and then possibly upgrades,  but there will be an across outcome rating, which of course is the lowest rating amongst the critical outcomes. And then finally, just now, Stacey talked about the grade strength  of recommendation ratings. This is the other type of rating, and this focuses on the balance between desirable and undesirable outcomes.  So it's a judgment the panel is going to be making about how does that balance look.  Sometimes it's quite clear that the desirables outweigh the undesirables, might result in a strong for,  but if the undesirables outweigh the desirables very clearly, that could be a strong against. And furthermore, the strength of recommendation  is going to look at some non-evidentiary factors, such as patient preferences, equity, disparity, the cost of implementing a treatment, feasibility, et cetera.  All of the things that clinicians are intimately familiar with as they wrestle with these things day to day.  Next slide.  So ending up with a little bit of philosophy here, in our view, good evidence-based decision is going to join the evidence with opinion.  And so we've emphasized that there is subjectivity and opinion throughout.  Opinion informs what evidence to look for in the development of key questions, as well as what evidence to consider in the form of inclusion criteria.  It informs the judgments of criticality of outcomes, what's critical versus what's important, but not critical.  It informs how we interpret the evidence. When do we consider a study limitation serious versus not?  And of course, in the recommendation phase, it's going to inform the balancing of outcome, desirable, undesirable, the consideration of preferences and resources.  So all of this subjectivity is optimally dealt with by the use of two main things, being a structured process that people understand all the various steps,  and those steps are all taken consistently, as well as transparency, where you talk about why you made various downgrades, about why you were concerned about,  say, patient preferences or the balance. And so these two things are a great way, we think, to deal with all of this subjectivity  and help that optimal balance between evidence and opinion.  So any questions from anyone as we wrap up?  Okay, great. Well, thank you all for coming. I'm sure this has inspired a lot more questions as you go and good luck in your evidentiary journeys.  Thank you.

Video Summary

Thank you for summarizing the video content.