Defining the quality (outcomes) of our care: Part 2

In the second part of the Orthopedics Today Round Table on defining the quality of our care, we discuss how orthopedic surgeons can define and discuss outcomes, as well as what tools and questionnaires can be used. Click here to read part 1, which appeared in the June issue.

Richard J. Hawkins, MD
Moderator

Richard J. Hawkins, MD: Dr. Tokish, Would you comment on the idea of a ceiling and floor effect, which is important to understand?

John M. Tokish, MD: If the scores are too easy and a high percentage of patients max out the score, we call that a ceiling effect. Conversely, if a high percentage of patients score on the lower end of that we call that a floor effect. A working example of this might be the Simple Shoulder Test of 12 questions, which ask whether you can do certain things like put on a shirt, throw a ball or work overhead. You can imagine if you are asking a patient with a reverse shoulder arthroplasty that is probably a questionnaire that has great usefulness. But if you are asking a professional baseball pitcher those questions, most pitchers are going to score very high since they perform at such a high level. So, a Simple Shoulder Test in a Major League Baseball population is going to have such a high ceiling effect that it has little usefulness. We should have to match our outcomes measure to our population, their functional requirements, and expectations.

Hawkins: Let us define scores in different areas. For example, we will talk a little bit about a generic quality of life (QOL) score and why we might do that. Then we will talk about a joint-specific score, or some might say region specific, although they may be a little different. Dr. Mohtadi has already mentioned a more sophisticated score called disease-specific. Dr. Tokish, why measure a generic QOL score, example SF-12?

Tokish: One of the big advantages of a generic QOL score is that it allows us to compare results across diseases. This becomes important to agencies like government and health care payers. For example, if I can get a universal QOL score then I may be able to evaluate the treatment outcomes for diabetes as compared to outcomes for total hip replacement. If that QOL score is valid across those populations, then we can start talking about allocating resources and adoption of policies and processes that go to the bottom line question.

Hawkins: Not to belabor this, but it is important for the readers to understand these QOL scores. In the PROMIS program, they look at domains of physical functioning, pain and mental functioning. That is mostly what QOL scores do. The SF-36, SF-12, and now an SF-8, have been the standard over the years. Dr. Kuhn, what is your opinion about the SF- scoring systems in terms of QOL and should the general practice orthopedic surgeon collect the QOL in SF-8 or SF-12?

John E. Kuhn, MD: We do need to collect data on QOL because it allows us, as Dr. Tokish said, to compare different medical conditions. The way I think about these outcome measures is to consider them to be like different lenses on a microscope and the QOL measure is low power. It does not get very specific to the disease, or the patient’s interest, but it does kind of give you the big picture. Then you have your joint-specific score, which is a medium power lens, and finally you have a disease- or population-specific score which are like high power lenses, each level gives us different information.

Hawkins: Historically, the SF-36 has been utilized by many. I would guess the general orthopedic surgeon may not care too much about the SF-36, but it seems to be required. It seems to be a standard. So they would probably want to collect such a score if briefer.

Kuhn: The SF-36 is a little bit complicated. It has 36 questions, which is a difficult thing to have patients complete in clinic. They have shortened it to the SF-12. There are even some five-question scores that are validated and get at general health status.

Hawkins: There is an SF-8 that is official. The other score that is becoming popular in generic QOL is called the EQ-5D, the European Quality-5D, which is only five questions, three answers for each, and covers those domains of well-being and physical and mental function, and pain. Dr. Tokish, what is your opinion about the EQ-5D?

Tokish: It is popular now, especially in Europe. The utility of it is that it is one patients will often answer, because it is not difficult. We will get a high compliance rate with it. But, as Dr. Kuhn used the analogy of the microscope, this one is from the 35,000-foot level. For example, if you are comparing a patient who is on renal dialysis vs. a high-level athlete, then you are not beginning with a similar QOL and so asking those people once you get down to so few questions you may be destined to have different levels of answers no matter what the outcomes are with a particular procedure.

Hawkins: Dr. Mohtadi, do you think general orthopedic surgeons care much about a QOL score? Should they collect one if they are going to collect data on patients?

Nicholas G.H. Mohtadi MD, MSc, FRCSC: I need to clarify because the official term is a generic QOL score as opposed to a joint-specific or disease-specific because you can have joint-specific QOL questionnaires and disease-specific QOL questionnaires. What we have been talking is, strictly speaking, of generic QOL scores such as you mentioned, the SF-36, SF-12, SF-8, or EQ-5D. I think that we may be mandated to collect a generic QOL score because that is the only way governments or insurance companies can compare between practices, diseases and specialties. As long as we keep it simple then it should be reasonable for any practitioner to perhaps collect an EQ-5D or SF-8.

Hawkins: Thank you for defining these important terms. It is important for this Round Table discussion. We found out recently that the SF- and the EQ-5D are going to cost money. We have been chasing them down to find out how much. The Veterans’ Administration has taken right out of the SF-s what is called a VR36 and VR12. It is supposed to be in the public domain which means it is available to us at no cost. Dr. Mohtadi, do you differentiate between a joint-specific and a region-specific score?

Mohtadi: I think if we talked about a region-specific score one example would be the DASH, which is the Disability Arm Hand Shoulder score. Then a joint-specific would be the ASES Score.

Kuhn: Other joint-specific outcome measures might be a Harris Hip Score or Knee Society Score.

Hawkins: If we talk about a joint-specific score, there are many but let us talk about the shoulder, as an example. The ASES score, which is 10 functional questions and one pain question, is validated. The pain question is, "What is your pain today?" So we do not have a night pain question that often drives patients to surgery. We might add a Visual Analog Scale regarding night pain. Dr. Mohtadi , you are familiar in the knee with the IKDC or the KOOS score as joint-specific. Could you comment?

Mohtadi: The IKDC score was developed similar to the ASES score. It was a consensus amongst a variety of different surgeons from across the world, and it includes objective and subjective data. They have now developed a completely subjective component. The KOOS score is derived from a disease-specific score called the Western Ontario McMaster Osteoarthritis Index, and they have taken some of those questions and added a few others and modified it to create the KOOS.

Hawkins: Orthopedic surgeons love brevity. So how does one take a score like the DASH score (33 questions) and shorten it down to the QUICK DASH (11 questions)? If we reduce a score like that, do we still have to go through the same validation process regarding the psychometrics?

Mohtadi: Strictly speaking, yes, but there are some advantages and disadvantages to reducing the number of questions. The most obvious one is the practicality of using the questionnaire. The downside of going to the shorter questionnaire is you may be missing important information in specific patient subgroups.

Hawkins: If we talk about the general orthopedic practice, what are the guidelines we should advise a physician or practice in order to select some scores? Do you have a few rules that you say, “Well, this is how we should go about this?”

Tokish: The constant of benchmarking your own practice is exceedingly valuable. This doesn’t require detailed or terribly scientific methods. For example, one attempt at this was out of West Point where they founded the SANE score. You simply ask the patient, “Where are you on a zero-100 scale with regard to the condition that we are treating you for?” What percentage of normal is your knee, hip, shoulder, etc., and has been correlated with a number of the other validated outcome scores? If you do nothing other than to collect the QOL score and a SANE score that might be sufficient. If you employ a simple outcome system as simple as an SF-12 or a SANE score, or a simple region or joint specific score, such as ASES or KKDC, you will identify many areas to help direct improvements.

Hawkins: I have a paper in front of me that compares the IKDC and the SANE score, which you just alluded to, i.e., the SANE asks “What percentage of normal is your knee?” This was investigated following a knee injury in active female patients, and showed there was a significant correlation between the IKDC and the SANE. That is one question we ask a patient and we get a good score. That is something to keep in mind.

Dr. Mohtadi, what scores would you suggest for practicing orthopedic surgeons?

Mohtadi: I would insist they are patient-derived because that is the most important. But I would concur with what Dr. Tokish is saying in that you want to keep it as simple as possible, and I could not agree more that it allows an individual to benchmark their own practice. In fact, I think this is an obligation of a surgeon to measure how their patients are doing in some way, whether it is mandated or not.

Hawkins: If we advise those reading this Round Table, we might suggest the score should probably be 1) patient-derived; 2) simple and as straightforward as possible; 3) not time consuming to fill out; and 4) validated, which we have defined. Those would be some of the criteria. Dr. Kuhn, would you elaborate on expanding our scores to disease-specific, for example, the [Western Ontario Rotator Cuff Index] WORC?

Kuhn: Disease-specific tools are better for research purposes because what you are interested in is, “What is your patient’s concern?” If your patient is a baseball player, then you are going to want to know if he is going to be able to get back to playing sports at the same or a better level. Can you ask that with one question, or do you need to use the Kerlan-Jobe Upper Extremity Athlete Score? That is an example of a population-specific or disease-specific measure. I think that for most people in practice they probably do not have to get into that much detail unless they are studying the condition for research purposes.

Hawkins: Let us discuss physician-reported outcomes vs. physician involvement. For example, many years ago the Europeans agreed upon the Constant Score. The Constant Score involves some physician involvement such as a weight measurement of strength. Dr. Tokish, what do you think?

Tokish: The challenge with a Constant Score is that it is difficult or can be time intensive. You have to have strength measurements done so it requires physician time, input resources and there is some variability in how you perform that test. So maybe that is the reason it has not caught on in a fee-for-service environment like the United States. It also has the potential for a ceiling effect in certain populations and so that is a limitation.

The Europeans were far ahead of the Americans in establishing an agreed upon score. In the United States, an average outcome system you might have 10 or 11 shoulder scores on that system. The Europeans all sat down and they said, “This is the one system that we are going to do.” It would be great if we all had a QOL, as well as a region-specific as well as a disease-specific. We might add a fourth area which would be an activity-specific level, the baseball player vs. the laborer type scores. Every time you add a level of complexity you are adding cost, you are adding time, and you are adding effort and infrastructure and at some point you have to find a balance in all of these.

Hawkins: If we think about an elbow problem we might say, “Well, let us do one, simple EQ-5D and a Quick Dash.” However, the most common elbow score used, at least in North America, is the Mayo (MEPS). There are 12 questions, half are patient-derived, and half are physician-derived.

Dr. Kuhn, you have talked a little bit about a research approach at a center like yours at Vanderbilt looking at rotator cuff tears. You do that with a disease-specific WORC. We have talked about the regular orthopod trying to keep it as simple as possible and validated, as well as use a score that is standardized and agreed upon. Since we have a myriad of scores, who is going to decide or encourage us to agree upon some simple scores that most of us might use?

Kuhn: That is a difficult question because there are many different scores available. The question is which one should we use. Sometimes it can be determined based on what the scores tell us, like the SANE score that you talked about.

In my opinion, the subspecialty societies should take charge and make recommendations so the clinician has guidance. The ASES now has a committee trying to do exactly that, and I would suggest the Hip Society, the Knee Society, American Orthopaedic Society for Sports Medicine and some of the other subspecialty societies should guide their members in the orthopedic community at large as to what scores they recommend.

Hawkins: That is exactly right. The specialty societies are being charged by American Academy of Orthopaedic Surgeons to come up with some simplified form of what our general orthopedic surgeons can collect and, of course, what we can collect for research purposes.

Tokish: This is more than an academic question. The people who are most motivated to answer these questions are the payers. It all gets down to economics. The people who are paying the bills are going to simply ask one question: Which one is cheap and which one gets the patient back to work? I think that while those are important parts of the question, they certainly fall short of where we would hope to be as physicians in terms of optimizing patient QOL.

Hawkins: If I were to suggest something for shoulder I would say, “Let us pick a generic QOL score.” Let us think about an EQ-5D or SF-8. Then we might think about a joint-specific score. The ASES is validated and simple and its psychometrics tests out well. Similarly, the SST and Oxford score test well. The disease-specific measure that you have all suggested may be more for research. So maybe the general orthopedic surgeon might not need to consider disease-specific measures. How often do we collect?

Kuhn: If you read Michael Porter’s book about measuring value, he recommends collecting data through the cycle of care. This means you have to continue to collect data until your patient is essentially cured. Now, what does that mean? Well, there is nothing magical about 2 years. Some journals require 2-year follow-up for their publication. But if you have a fracture that has healed in 6 months and the patient is back in their routine, then why do you need to know that data on that patient for 2 years? I think for every different disorder there is going to be a different duration that you need to collect outcomes on your patients.

Hawkins: If we collect the simple scores I mentioned in the shoulder, then we might think about collecting them again at certain periods of time. If they are better with 3 months of our treatment, maybe that is enough or maybe 6 months are needed. If we are doing something academic, we might want to follow them for 1 year or 2 years.

One of the scores I think is helpful is what is called a Global Rating of Change (GROC). So we, as physicians, walk into the room after a total shoulder replacement and we say, “How are you doing? Are you better, worse or the same?” The patient says, “I am better.” “How much better are you — 25%, 50%, 75% or 100%?” “Doc, I am 100% better.” It is like the SANE score, the GROC is also a simple score and it is validated. You can follow that up with any patient at any time following treatment.

Mohtadi: We use that when we are trying to determine whether a patient has changed over time when we are assessing the reliability of a particular questionnaire. You can do it on paper. You put zero is no change and 100 is change to the better, and -100 is change to the worst and you can quantify that global rating of change.

Hawkins: Let us just talk about a couple of other examples. For example, for a total knee replacement, we would agree they should have a generic QOL score. We would encourage people to think about the SANE, the Simple Numeric Value One Question. “What percentage of normal is your knee?” Then in the arthroplasty literature it might be the Knee Society Score or the WOMAC.

Dr. Mohtadi , if you had to pick, on a sports knee score, what would you advise?

Mohtadi: It depends on the population, because you have alluded to the population that has a total joint replacement, and Dr. Tokish was talking about an active professional athlete population, and you can not have one questionnaire that addresses those two ends of the spectrum. So, it has to match the patients you are seeing or treating on a regular basis. Without that, it is not going to be as valuable.

The KOOS is a reasonable knee outcome questionnaire. The subjective part of the IKDC is also reasonable. But if you are dealing with ACL patients who are active, you might want to have something that is more disease-specific rather than those generic questionnaires, that is if you are interested in closely following ACL tears perhaps for research.

Hawkins: Do you think a general orthopedist would want to go so far as a disease-specific measure in his ACL patient or do you think he would want something simpler?

Mohtadi: Definitely simpler, unless they have a high volume of ACL patients.

Tokish: To your second point about when we should collect data points, I believe that is economically driven as well. In the military, if I do not get a Marine back to duty within 6 months to 7 months, he loses his job. For us, it is clear what our time frame is in terms of outcomes. If you are a professional or a collegiate athlete, probably that is around 2 years.

Hawkins: When we consider disease-specific collection, physicians must see the patient first before they can be put in a disease-specific category. The simple scores, QOL scores, SANE scores and ASES scores can be done in the waiting room before the physician sees the patient.

Do the electronic medical records we have experience with in North America allow us to do outcomes?

Mohtadi: Not directly. Unfortunately, because they were first used for scheduling and billing purposes and they are now slowly being adapted to collect outcomes.

Hawkins: So EPIC is a new system that is coming into our hospital at millions of dollars, and yet, they do not do outcomes. It may be important to integrate our scheduling system into a software system. Do we have to have a software program to help collect outcomes?

Tokish: Yes, we need a software program to collect outcomes. The amount of data is vast and you cannot rely on paper systems because it requires too much infrastructure and too much work, especially if you are trying to get it across a large population. The challenge with why outcomes are not imbedded into the larger medical records system is because there is no economic stimulus to do it.

“Did you check their hemoglobin A1C? Did you ensure that they are properly treated for their diabetes? Did you do a surgical timeout?” Those are of great value, but they do not get to those outcomes that you and I are particularly interested in when it comes to surgical, patient-reported, postoperative outcomes.

Hawkins: Software programs are available to allow outcomes to be collected. They cost money, and that is a concern. Another aspect is, what do we do about de-identifying patient data.

Mohtadi: This is standard in the research world. People collecting and analyzing data need to be able to do it without connecting the specific information to a specific patient, and so there are various ways to de-identify information, But, at some level, there has to be something that uses a patient number and a name, and then a patient record with information in it.

Hawkins: The other issue is consent. The way we have gotten around consent in our system is to include data collection in our “consent to treat” document. We say data will be collected and this data will be part of your treatment program. Subsequently, if we wish to perform research on the data, we do not have to go back to get the patients’ consent again and can do what is called an expedited IRB.

Tokish: Another challenge of outcomes is the confounding risk of patient psychological well-being and few of our scores consider that. For example, if you have a patient who is a worker’s comp patient then they are definitely at a risk for a higher failure rate and lower outcome scores. Beyond those considerations, there are certain measures called resilience and psychological well-being. You can imagine the patient with depression and how much that might impact upon a patient’s outcome, regardless of how great the surgical intervention is.

As we go forward in orthopedics, we are going to have to start thinking more translationally about how we bridge that gap so that we can truly measure “apples to apples” by taking some measure of resiliency and psychological well-being into overall outcomes measures in order to be valid.

Hawkins: Dr. Mohtadi , do you have anything that you want to complete the picture?

Mohtadi: We got a little confused when we started talking about QOL. We should talk about generic measures, joint- or region-specific, and disease-specific. I want to make sure that we have not confused people by using the term QOL as a specific category. It should be generic, joint or region and disease-specific.

Hawkins: If I may summarize our discussion, it is important to understand the psychometric properties with outcome scores. We have tried to keep this simple. As a general practicing orthopedic surgeon we may consider as simple approach as possible to collecting outcomes in our practice. It would be important that we collect a generic QOL score such as a SF- score or an EQ-5D. The SANE, which is one question, is a great simple question: “What percentage of normal is your part?” before treatment and after treatment. Then, we might have a joint-specific questionnaire, such as in the ASES score for the shoulder, IKDC for the knee and sports, or WOMAC for total knee replacements. For the elbow, hand and wrist, we could use the Quick Dash score.

If we are interested in research, then we might want to look at more robust scores such as the WORC, the Western Ontario Stability Index, or the Western Ontario Osteoarthritis Index based on the diagnosis of cuff, instability and arthritis.

Thank you to Drs. Mohtadi, Tokish and Kuhn for your contribution to this Round Table discussion.