All right, well, thanks so much, Stacey. Those are really important aspects of this process. They really drive everything else, setting the scope, setting the inclusion criteria, etc. So for the next 45 minutes or so, I will be giving you an overview of GRADE. So this system came out in 2004. The GRADE Working Group is a group of probably 100 people around the world who have been making lots of suggestions, changes, additions to the initial system. It's been a very powerful and influential system in the world of evidence-based medicine. Lots of different societies are using it. There has been some negativity about it here and there, but for the most part, a lot of groups feel it is really helpful in helping them work through this process. GRADE has published a lot. This is a list. These two slides show a list of all of the articles that have appeared in the Journal of Clinical Epidemiology. There's a large series of somewhere between 25 and 30 articles. They sometimes skip a number. I'm not sure why. But there are also other articles not even in this journal. So it's a massive endeavor that GRADE Working Group has put together, and they are still evolving. For more resources, and of course, I'm only giving you a brief glimpse of numerous complexities in GRADE, but for more information, you can go to GRADE Working Group's website. There is a detailed handbook at that URL you see on your screen. In the big picture, GRADE does openly acknowledge that there is a lot of subjectivity in evidence-based medicine, and as a result of that, GRADE is a structured system to enhance consistency both within one guideline as well as between different guidelines. And furthermore, there is a lot of encouragement of extreme transparency for why certain decisions were made throughout the process. If you learn only one thing from this workshop, let it be this, that GRADE actually has two types of ratings. One rating is about the certainty of evidence about the outcomes, which I'll be going over. And the second rating is a strength of recommendation rating for the action that's being either recommended or disrecommended in the guideline. That will be discussed tomorrow. Now, these two ratings are related, but only indirectly so, and I remind you of my earlier mention of parachutes and headaches. You can imagine a situation in which we have such an obvious situation that we don't need evidence in order to make a recommendation, or the converse situation where we have some really well-done studies, but we still don't know what the clear recommendation is. So those two examples underscore the fact that the two types of ratings, certainty of evidence, strength of recommendation, are indirectly related. So what I'll be going over regarding that first type of rating, certainty of evidence, is various steps along the way. The first step of the guideline panel is to determine which outcomes are deemed critical for making a recommendation for or against in a given scenario. The second step is actually a multi-step process such that for each outcome, we're going to have an initial starting grade. We will decide if there are any downgrades from that. If there are not any downgrades, we decide if there's any upgrades. And then that single end outcome is therefore graded as high, moderate, low, or very low. Now, what that rating means is essentially one certainty in the evidence on that outcome. So once you're done with that outcome, you go back and do another outcome, starting grade, decide downgrade, decide upgrade, etc. So you're going to have some outcome ratings, each for a single outcome. And then you go to step three, where we look at all the outcomes together and we decide on the overall certainty of evidence by taking the lowest rating amongst the critical outcomes, and I'll go over that in some detail next. So I've provided a little guide right here so you can tell where we are in all these various steps with a little red arrow to indicate where each slide is. So first, determining which outcomes are critical. So as we all know, in medical situations, there's multiple patient outcomes of interest. It might be hospital stay, mortality, symptoms, quality of life, lab values, lots of different things out there that we might want to measure in our efforts to improve patients' conditions. So in a given clinical scenario, what we want to do is for each outcome, we want to think about how important is it when we decide how to manage patients, what treatments to administer, what tests to order, etc. And in the grade system, each outcome is based on the one of three categories. It's either called a critical outcome, important but not critical, or a not important outcome. And deciding this will be a group process. The panel will get together and make decisions about where each of the outcomes falls. Later on, when we are getting all these outcomes together and looking at the overall certainty of evidence, we're going to take the lowest grade among the critical outcomes. So the critical outcomes will be graded. The middle category of important but not critical, those will get a grade, but they will not go into the overall certainty rating. And that lowest level, the not important outcomes, those will not even receive a grade. So this example, and Stacy went through a few of her slides, were all about this example where if we have a guideline on treatments for obesity, among those, actually that's a mistake, should be metabolic syndrome. So what I've done here is I've listed out some outcomes that a panel might consider for this process, body weight, fasting, plasma, glucose, HbA1c, hypoglycemic events, as well as cardiovascular disease. And what we're going to do is decide which of these outcomes, might be more than one, are critical for making the decision about recommending a weight loss intervention in those with obesity and insulin resistance. So I went ahead and just my own assessment here, this is not at all anything official, I went ahead and said that body weight and cardiovascular disease would be the critical outcomes. That HbA1c and hypoglycemic events would be important but not critical. And then lastly, fasting, plasma, glucose would not be important. We will be returning to this table tomorrow when I have walked through the full example. So, we've decided our criticality of outcomes, and what we're now going to do is take one outcome at a time and walk through the grade process. The first step is the starting grade, this is actually the easiest step of all because it basically says, randomized trials start at high certainty. Non-randomized studies start at low certainty and that will include some controlled studies that didn't randomize or some uncontrolled studies. And then finally, any other sort of evidence such as some informal report of one's clinical experience, that will be categorized as very low at the outset. I will say, GRADE does make an exception for diagnostic accuracy studies, those can start at high under certain conditions. I do not have time to get into the details as to why. This is an example of a case where GRADE had a big discussion about what should we do with diagnostic accuracy studies. Are they always, should they always start at low and the final decision was well, they can start at high sometimes. If you remember that large list of papers that I had given you at the beginning, one of those papers is all about diagnostic accuracy. So now that we have, based on study design, we have an initial upgrade, starting grade, I will now go through all the various possible downgrades. Study limitations, inconsistency, indirectness, imprecision, and publication bias. So for each of these in parentheses, I've put the types of amounts of downgrade. For most of them, you can go either no downgrade or zero or negative one or negative two. The exception being for publication bias where it's either you downgrade one or not. So we're going to take these now one at a time and I'll walk you through what they all mean. First, study limitations. So as we all know, empirical studies are not perfect. There's often producing results that are somewhat inaccurate. It might be that a treatment as appearing in the study that it's true effectiveness is different than the study's results might suggest. It might actually work better than the study suggested, it might work worse than the study suggested. So when we're downgrading for limitations, it's really essentially our suspicions that the study results are not quite accurate. And there are many, many reasons for this. There's a few possible here. No concealment of allocation, lack of blinding of patients, caregivers, outcome assessors, incomplete data or data analysis, selected outcome reporting, any other limitations. I don't have time to get into a lot of the details here. I will say, and I have said before, there's a lot of subjectivity in evidence-based medicine. And this is probably number one on the list of how a study can have limitations, why we doubt the accuracy of its results. Assessments of study believability really probably are the largest source of subjectivity in this area. And so in my world, we often require at least two people to independently look at each study to assess its biases so that we have some level of consistency. So that's just a quick list of possible limitations of randomized trials. Here are some for non-randomized trials. Failure to develop and apply appropriate eligibility criteria. Perhaps a flawed measurement of exposure and outcome. Failure to adequately control confounding. And this, of course, would be the primary concern about a non-randomized study. Possibly that patients who got one treatment might have had a different prognosis than patients who got some other treatment. And perhaps the reason for the different outcomes was their initial prognosis rather than the actual treatments they got. So that would be called confounding. And then differential follow-up. Some studies might follow one group longer than another, which might result in different outcomes. But again, there's many layers to this onion. There's many different measurement tools for assessing study limitations, otherwise known as risk of bias. A lot to go over here. I don't have time for that. So the second possible downgrade in the GRADE system is for inconsistency. Now, this refers to the study results, the outcomes at six months after treatment, say. And it might be that one study found that one treatment was far better than another. And a study, a very similar study, might have found that in fact the treatments had very similar results. This would be an inconsistency. And the key question for an evidence reviewer would be, is that inconsistency large enough that it would affect the management of patients? If we were to go by this one study's result, how would we manage patients? Versus if we went with this other study's results, how would that result in patient management? And consistency can take many forms. It's typically focused on the point estimate. So, for example, when I say point estimate, I mean the size of the difference between groups. For example, in the headache one that I said earlier, I had a 25% rate with the drug versus a 25% rate of headache with the placebo versus 20% with the drug. So it lowered headaches by 5%. 5% is the point estimate of how large the difference is among groups. Also with the consistency, it's possible to look at other aspects, such as whether the confidence intervals do not show any overlap. One study had maybe confidence intervals around 5% and another one had it around, say, 1%. Sometimes, in some reports, consistency involves the statistical test for heterogeneity among studies. That's kind of a jargony term that people in the world of meta-analysis use, essentially quantifying whether different studies had different results. And finally, of course, tau and I squared is more jargon for you. I'm sure you're enjoying that, about measuring how much studies differed in their results. So I've gone over two ground down grades and the third one now is called indirectness. This is often one of the harder ones to wrap your head around, and it's one of the more complicated ones to deal with. And maybe one way to think of it is to think of it in steps. So we can have indirectness in many different ways. We can have indirectness of patients, interventions, comparisons, outcome measures, time points. When we're talking about indirect evidence with respect to the patients, we pretty much mean that the types of patients in the studies are somewhat different from the types of patients of interest for the key question. So, for example, if we had a study of prediabetes, but we wanted studies of actual diabetes, the prediabetes studies would be indirect evidence. It might even be excluded based on your inclusion criteria, but certainly if they were included, it would be considered indirect. Secondly, interventions. So this really involves the implementations of a given intervention. Maybe it's a device that's not the most recent iteration, it's not purchasable anymore. Maybe it's how you're delivering care in a specialized setting. With medications, of course, this is unlikely to be an issue given that that's a pretty standardized dosing schedule. But there are areas where actually implementing an intervention might be different than a study versus how the key question is demanding the implementation take place. Telehealth is a good example of that. Also, there's indirectness with respect to comparisons. There are some cases where we have two key interventions, but they're not actually being compared directly. They're both being compared to placebo, and yet we want to compare them for our key question. If we're relying only on placebo-controlled trials to make inferences about two competing medications, that would also be considered indirect evidence. And the fourth type of indirectness is outcome measures. This typically takes the form of trying to use a surrogate measure in place of the actual outcome. And you will see this later tomorrow when I discuss the use of lab values as a surrogate for cardiovascular disease. And finally, time points. Indirect time points basically means we really wanted longer-term data, but all you had was short-term data, so you included it. And you then might downgrade it for being only indirectly related to the long-term outcomes that you had wanted. Okay, turning now to imprecision. This is the fourth type of downgrade. Imprecision really refers more to the statistics, so the random error in a given study, which can be caused by many things, not just low N, which we all, of course, are aware. A study can be imprecise because of its small N, but also because large variability between patients. If some patients improve a lot and other patients only improve a little, that's going to result in imprecise results because there's no single patient experience that you can pin down. Random error is also being enlarged by small numbers of studies, as well as low event rates. Typically with random error in the grade system, you're going to be looking at the width of the confidence interval. Of course, that typically is a 95% confidence interval, conceptually, that I have 95% confidence that the true effect lies in this interval. A wider interval indicates less confidence, indicates less precision, so you might be more likely to downgrade for imprecision if you have a very wide confidence interval. Essentially, an evidence reviewer would be asking themselves, is this confidence interval wide enough to include effect sizes that are so different that they would lead to different decisions about management? So on one side of the confidence interval, it would lead me to recommend some intervention, but the other side would lead me to basically say it doesn't matter whether you implement the intervention. As an example, I've got a forest plot for you here. This is an example from one of the grade papers. The intervention here is corticosteroids to reduce hospital mortality in patients experiencing septic shock. We have here listed a group of nine studies with their publication dates, and we have in each column the rate of hospital mortality. In the Anand study, those who did get corticosteroids died at a rate of 95 out of 151, whereas those in the control group, and it's not clear what they got, but nevertheless, the control group died at a rate of 103 out of 149. Now when you compare those two rates, the data slightly favor the treatment group, so you have this one study confidence interval slightly to the left of the line of neutrality indicating a slight benefit of corticosteroids with a relative risk of 0.9 and it's competent. So this is just one study and the great group went ahead and meta-analyzed these nine studies and got a single overall estimate of how good it is to use corticosteroids in those experiencing septic shock and they found an overall relative risk of 0.88 with a confidence interval from 0.75 to 1.03. This was an example where the imprecision of that interval was enough to downgrade the evidence and as I said you would look at the two different sides of that confidence interval and ask does that lead me to different decisions if the confidence interval if it were sorry 0.75 that's a pretty surprisingly large effect that yes indeed we're going to recommend corticosteroids they reduce it by 25 percent but the other side of that interval is actually suggesting that corticosteroids don't work to reduce hospital mortality that would lead you to a different type of recommendation essentially don't bother and so the width of that confidence interval then makes us concerned about precision and so you would downgrade in this case for imprecision. Okay and then the last downgrade I'll discuss possible downgrade is this notion of publication so unfortunately there are studies that are never published you never see the light of day they're not findable in medline and the problem with that is that it suggests that maybe what is published is kind of a biased subset of what's been done in entirety and it's unclear sometimes where or why those other results weren't published it's possible that authors chose not to publish because they didn't like the results they didn't conform with their prior beliefs perhaps and so conceptually you can think of this as a tip of the iceberg situation where the published stuff is above the surface of the water that you can see and then who knows how large that iceberg is that would cause you some concern so if this is a concern publication bias you would then downgrade one level for that. Here is another slide to demonstrate publication bias this simply is showing what we have and I apologize for the for the difficulty of discriminating lines here but the lighter line is positive trial studies that found an effect and the darker line is negative trials studies that did not find an effect and what we have here is the amount of time it takes between when the study's finished and when it's published. So essentially the positive trials they're accelerating publication of them and so if you were to go in and do a search and say this year one you would find the positive trials but some of these negative trials would not have been accelerated and so you would miss them and so essentially this might give you pause about the possibility of publication bias. Here is another plot indicating publication bias this is something called a funnel plot. I won't get into great detail here but essentially if you look at this one on the right here you're seeing a pattern whereby the smaller studies those with higher standard error seem to be missing from the plot suggesting that the results of the smaller negative trials have been suppressed. So there are advanced methods to try to detect this to try to sort of fill in the results of these hypothetical negative trials and then adjust your point estimate based on that hypothetical process. You don't see this a lot this is a fairly advanced and often there's not enough data to even produce this sort of plot but it does show the notion that there do exist methods to try to detect publication bias. Okay so we've been through starting grade we've been through the five possible downgrades of the evidence and now I'll go through these upgrades. In the grade system these are only to be applied to non-randomized studies and the idea is that if there were no downgrades you're going to be considering whether to upgrade the evidence. Now actually grade was fairly unique at the time in 2004 introducing the notion of upgrading evidence in any way evidence was always considered to be either perfect or you would downgrade it. So grade really were unique in suggesting the possibility that there are cases that we can upgrade and I will be going over all three of the types of upgrades in the grade system magnitude of effect dose response association and the fairly lengthy blurb controlling for all possible confounders would increase the effect. I'll explain that in a minute. First magnitude of effect this one's pretty easy to understand. Here we might upgrade when the effect is so large that if there are biases they can't possibly explain that massive effect. We will be looking at both the point estimate and its confidence interval. Grade suggested some cutoffs for relative risk if the risk is between two and five maybe that's worth a plus one upgrade or if that relative risk is greater than five times more likely to happen than not you would get a plus two upgrade. An example of that comes from the world of sudden infant death syndrome. This is a quote from one of the grade papers where they found that front sleeping versus back sleeping you would get a odds ratio of 4.1. You were four times less likely to have SIDS if you're back sleeping resulting in back to sleep campaigns that reduced SIDS by 50 to 70 percent. This is a very large effect. These were not randomized studies but the feeling was that such a large effect could not have been due to any of the various biases or confounding in those studies. Here you would upgrade by one for this large effect. Secondly, dose response gradient. This is also fairly straightforward. It's just that when lower doses produce smaller effects than higher doses this would yield more confidence that there really is a cause and effect relationship. The higher the dose the bigger the effect. Now usually when we say dose we mean dose of medication but this concept can also apply to non-pharmacological interventions such as physical therapy. The high dose might be going five times a week versus a low dose of once a week. If we saw that physical therapy patients were recovering much faster or better with five times a week than once a week we might be more likely to upgrade the effectiveness of physical therapy based on this dose response gradient. The example in grayed here involves COX-2 inhibitors. In this case the concern is that they're causing cardiovascular events and the initial meta-analyses that came out several years ago were that with roflaproxib the relative risk of having these events was 1.33 if you're getting a lower dose, less than 25 milligrams a day, but if they did an analysis of those on a higher dose, more than 25 milligrams a day, they were getting quite a bit larger relative risk. So this basically increases our confidence that the medication itself is what's causing the cardiovascular events. So in this case we might well choose to upgrade one level due to the dose response gradient. And finally this rather confusing one about controlling for all possible confounders. The notion here is that we might have a non-randomized study that was actually biased against finding an effect and yet it found one anyway. So hypothetically if these studies had actually controlled for bias they would have found an even larger effect. The example in the example given by the gray group here concerns the use of condoms to prevent HIV. They found five observational studies that had a pretty large effect of 0.34, relative risk of 0.34. Essentially your risk of contracting HIV was one third the risk of someone who didn't use condoms. However these analyses of these five observational studies did not adjust for the fact that condom users were more likely to have multiple partners than non-condom users. And hypothetically had they controlled for that difference in partner choice they would have found something even larger than a one-third improvement. Maybe it would be 0.2 or even 0.1 in favor of condoms. So these kinds of examples are rather rare. I've actually never seen one in my work but it is part of the GRADE system that you could upgrade as a result of this scenario. So those are the various downgrades and upgrades. I return now to this list of outcomes and the judgment earlier, just the simple Treadwellian judgment of criticality. Remember we said that body weight and cardiovascular disease were critical. Hypoglycemic events in HbA1c were important but not critical. And then plasma glucose was not important. So let's suppose we've gone through the previous downgrades and upgrades. We've done them for each of these five outcomes. And let's suppose we ended up with a moderate grade for body weight. Plasma glucose did not even go through the process because it was deemed not important. We got a moderate for HbA1c, we got a very low for hypoglycemic events, and we got low for cardiovascular disease. The goal now is to come up with an overall certainty for these five outcomes. And if you remember, I said what we're going to do is we're going to focus only on the critical outcomes, what those grades were, and we're going to take the lowest of those, which of course is the one for cardiovascular disease, which is low. Now I will say, because I think it's a bit of extra time here, there are aspects of the grade system that allow you, after the fact, to edit your initial judgment of criticality. So you might at the end of the day decide that in fact plasma glucose was critical rather than not important. That would then go into the equation and would be one of the outcomes you would consider when looking across. So it's not that at the very beginning of the project when you said this was critical and this was not, that you're required to keep those judgments. You are allowed to edit those judgments later on. So in summary, your starting grade is going to be based on the study design. Randomized trials start at high, non-randomized trials will start at low, opinion will be starting at very low. There are five types of downgrades, study limitations, inconsistency, indirectness, imprecision, and publication bias. Actually, I'm remembering there's one thing I forgot to mention, which is that the wording of these is quite intentional. Inconsistency, indirectness, imprecision, they're worded negatively, so that it's only when we actually find evidence of inconsistency that we downgrade. That becomes important when we only have one study. Well, if we only have one study, there's actually no way you can be inconsistent with other studies. So with one study, you would actually never downgrade for inconsistency. The default assumption essentially is that there's not a problem, and you only downgrade when you see a problem. Upgrades, we have magnitude of effect, basically how large is the point estimate difference between groups. We have dose response association. Of course, when I give higher doses, do I get higher effects? And finally, the controlling for all plausible confounders would increase the effect. That's basically studies were biased against an effect and yet they found one anyway. That's the third type of upgrade. And across outcomes, once we've got all of our outcomes graded, we're going to take the lowest grade among them and take that as our overall certainty. So you might ask, John, this is a lot of information. How can we summarize all that for a guideline panel so that they can more easily digest it? Well, thankfully, the grade group has provided some tools to help that process. I'll go over a few of them now. So the first is GradePro. This is free software that accelerates the creation of grade tables. It reminds you to create footnotes to explain why you downgraded different things. But it has basically a column for each of the various downgrades and upgrades so that you know, oh, right, I've got to still assess and direct, so whatever you haven't yet done. So it really creates a structure to work with them. And there's the website for you. Grade does have some symbols that are kind of nice to use, kind of to see at a glance, the strength of evidence for various outcomes. And there's a couple of standardized tables that they recommend. This one is called the Evidence Profile, I believe. And here we see the given outcome. We have all of the different types of possible downgrades. These are RCTs, so there's no upgrades listed. We have the overall results. And we have the relative risks. It kind of summarizes all the stuff we looked at for this one outcome. And then in the next couple of rows, we have another outcome and all of its various things. So that's the Grade Evidence Profile. Getting up a bit more general is something called the Summary of Findings Table, where we still have all of the outcomes listed down the left. And we're given the relative risks and the absolute numbers of events per patient, etc. But we're not bothering with all the different downgrades. We're just saying, look, overall, the quality of evidence for these outcomes was, well, this is some really strong evidence, high, high, moderate, moderate. And if you want to see more details, you can just go back to this Evidence Profile, for example, for why the evidence on this hearing outcome was called moderate. And you should be able to transparently determine what downgrade it was that brought it to moderate. So that comes to the end of my planned slides for the day. And I wanted to encourage you all, before viewing the next presentation, which will be two additional hours, to consider going through a couple of articles. And we'll give you a background of what I'll be talking about in the next presentation. So let's suppose that we are reviewing the evidence on the following question. For obese patients with metabolic syndrome, what is the impact of dietary nutrient composition on weight loss and cardiovascular disease? So we went through the 2016 AACE obesity guideline, and we found two articles. They're both randomized trials. Two articles on this question. We will go ahead and send PDFs of those to AACE to be included with these presentations. And I encourage you all to read these two articles prior to watching the next presentation and get your own impressions of what was done well or not so well in these studies. Ask yourself the following three questions. How well designed and conducted were they? What aspects concern you about their reliability? How direct were these studies in answering this key question? Consider patients, interventions, comparators, and outcomes. And thirdly, how consistent and precise were these two studies? Did they have similar findings? Were they large enough to permit clear conclusions? And feel free to walk through all of the downgrades and upgrades yourself before you watch the next presentation and see if you end up agreeing with our assessments. And I believe that is the end of the talk today. So it's only 11.40. So does anyone have any questions for Stacey or I? Hi, John. This is Janice. I have a question with regard to inconsistency. So is that when you say you only have one study, but say you have multiple studies but they're not within a systematic review, how do you know if the results vary so broadly? You'd have to go from one study to the other to see the variance. So how then do you go and then go back and grade? Well, I mean, you're going to be looking at the two studies together, and you can have them side by side. And essentially, you're going to ask yourself, so I suppose I have study A and study B. You ask yourself, OK, if all I had were study A, what would my action be? And then separately, if all I had were study B, what would my action be? Now, if those two actions are different, that's when you would consider downgrading for inconsistency. Does that help? Yes. And then I actually had another question. You talked about you're combining the studies with all the outcomes, and you actually take the lowest one, which then represents certainty across all. Why the lowest versus perhaps the average? Yeah. Yeah, I think that's a question often wondered about the grade system. So I think it's the weakest link theory that if we have links in a chain, and they're all deemed critical links, the chain is only going to be as strong as the weakest link. So if you picture a train pulling a car, and it's got three links on it, the weakest link is what's going to break that chain. And so conceptually then, if I have three outcomes, all critical, and one is very low, and the other two are high, well, the chain is going to break because of that very low rating. Now, it may be, after realizing that, that the panel goes back and says, well, that one that was very low, did we really think it was critical? Maybe we didn't think it was all that critical. And so maybe it doesn't turn out to be a critical outcome for the decision. So there might be some back and forth on that. But yeah, short answer, it's the weakest link theory.

GRADE system

evidence-based medicine

certainty of evidence

strength of recommendation

grading evidence

GradePro software

grade symbols

grade tables