Subjective measures present opportunities for improved care, but prove difficult to interpret

While patient-reported outcome measures are a valuable tool that can give orthopedists important information, the results of these measures can be called into question through a number of patient and surgeon variables. In addition, new health care legislation, and advanced surgical techniques and devices are changing the evolution of these measures.

William R. Beach, MD, an orthopedic surgeon at Tuckahoe Orthopaedics in Richmond, Va., said that he uses patient-reported outcome measures (PROMs) as a method to confirm whether a surgery was successful and advocates their use in published studies.

“Patient-specific outcomes, are the newer wave,” Beach told Orthopedics Today. “If you consider what is being reported in most articles in arthroscopy and sports medicine, they are objective rating scales. The subjective scales are and should be used beside them, not in place of them.”

Compared to other specialties, Gaurav Khanna, MD, noted that orthopedics has been slow to adapt to PROMs. However, he said that standard patient-reported outcome tests should be simple enough for patients to complete at home.

“Orthopedics is one of those fields where it sort of lends itself easily to patients reporting their own outcomes,” Khanna, an orthopedic surgeon and researcher in La Jolla, Calif., told Orthopedics Today. “Every field has an emphasis on learning how patients are doing after their procedures, but I think orthopedics, because it is based solely on motion, function and pain relief, lends itself easily to asking some basic questions and assessing how well the patient is doing postoperatively.”

William R. Beach, MD, said that questionnaires to obtain patient-reported outcomes should be detailed, but not onerous. Image: David Everette

In a 2011 study, Khanna assessed the accuracy of the patient-reported American Knee Society and Oxford Knee Society scores compared to clinician-assessed scores. Using the American Knee Society scale, he found that the mean patient-reported scores for knee pain were four points lower than clinician-assessed scores and mean patient-reported function subscores were 10 points lower than clinician-assessed scores. When the Oxford score was used, Khanna found no significant differences between the patient and clinician assessments.

Although patients self-reported worse outcomes on the take-home American Knee Society questionnaires, Khanna said one of the most interesting results came when patients returned for follow-up appointments.

“The most significant part of that was the patients typically reported a higher function — a better outcome — when they came back for their follow-up appointment as opposed to when they were filling out the forms in the privacy of their own home,” Khanna said. “Within 2 weeks, you would not expect such a dramatic increase, but many times they were reporting eight or 10 points higher on their pain and function scores using the American Knee Society Score.”

He concluded that the 2-week time period was too soon to see such dramatic change, but provided one theory as to why patients were reporting higher outcomes: they did not want to disappoint their surgeons.

Discuss in OrthoMind

Reducing patient bias

Reducing patient bias is also an issue with PROMs. This bias can be limited with proper administration of patient questionnaires, according to Karen M. Briggs, MBA, MPH, director of clinical research at the Steadman Philippon Research Institute. To limit bias, she suggests that questionnaires be administered digitally.

“When people collect outcomes on all their patients, [it] reduces the selection bias. These scores are being tested to make sure they work over different populations, so that again reduces the bias,” Briggs told Orthopedics Today. “It is important that patients fill out these forms on their own. These forms get filled out before they get to see the physician, so you do not have as much bias as the physician placing on how the patient how they are doing.”

A digital collection process also “eases the pain of data collection” and frees staff members to tend to other patients rather than collecting data, Briggs said. At her institute, patients fill out a form digitally and those outcomes are referenced by physicians during the first visit. The physicians fill out a digital form themselves, and the records are collected in a database where the patient file is kept indefinitely.

Some studies have indicated that gender, age and ethnicity may play a role in patient-reported outcomes.

In a study by Ageberg and colleagues, researchers analyzed 10,164 patients who underwent ACL reconstruction and filled out either the Knee Injury and Osteoarthritis Outcome Score or EuroQol questionnaires. The investigators found that female gender was a significant predicator of lower outcome scores. However, the authors were unable to conclude exactly why women were predisposed to reporting worse scores than men.

Similarly, a study conducted by Kenneth A. Egol, MD, and colleagues discovered that, of 443 patients treated for distal radius fractures, African-American and Latino patients reported worse outcome scores than Caucasian patients at almost all follow-up visits. However, African-American patients improved wrist function at 1-year follow-up when compared to either Caucasian or Latino patients at that time interval. The investigators were unable to conclude why ethnicity played a significant role in outcomes.

“We had some theoretical reasoning as to why we thought this might be — it is probably a multifactorial sociodemographic type of [situation], possibly a difference in patient expectations, possibly language or cultural differences, but we really do not know for sure,” Egol, who is professor and vice chairman of the of the Department of Orthopaedic Surgery at the Hospital for Joint Diseases Langone Medical Center, told Orthopedics Today.

“Orthopedics, because it is based solely on motion, function and pain relief, lends itself easily to asking some basic questions and assessing how well the patient is doing postoperatively.” — Gaurav Khanna, MD

Special populations

Obtaining patient-reported outcomes from children can prove challenging, as investigators must ensure that patients fully understand questions and correctly interpret their responses.

“For children, they have to be old enough to understand what the question means,” Beach said. “If someone explains the questionnaire to them, obviously that has to be consistent. I do not think that using the reported outcomes is particularly difficult if you have confidence in the way the study was set up or have confidence in what the study can tell you.”

Some surgeons use the Pediatric Outcomes Data Collection Instrument (PODCI) to get subjective data from children. In a study by Kunkel and colleagues, researchers concluded the instrument is a valid collection mechanism for children and adolescents, noting it was responsive to changes over time and was able to discriminate between populations in patients with hand and wrist injuries.

Beach noted that worker’s compensation status can also influence patient-reported outcomes. Such patients can be slow to recover in an effort to gain money from employers during their leave from work and may feel less satisfied with their care.

“Secondary gain is always a contributing factor to how you improve.” Beach said. “Everybody that knows about worker’s compensation knows that the faster you go back to work, the less time you are off, the better you will do. With worker’s compensation [patients], you do as good as you can, get them back to some kind of activity as soon as you can, if it is safe, and try to minimize the amount of time they are out of work.”

Various comorbidities have been shown in the literature to have a negative impact on PROMs. Tashjian and colleagues found that patients with rotator cuff tears had a mean of 2.07 comorbidities. The investigators found that a greater number of comorbidities negatively correlated with patient-reported outcomes. According to the study abstract, the authors concluded that factors such as diabetes, smoking, worker’s compensation, previous surgeries, revision surgery and duration of symptoms impacted patient-reported outcomes for preoperative baseline pain, function and general health status.

“There are so many different scores, I think the problem becomes trying to compare and contrast results of papers when different outcome measures are used.” — Kenneth A. Egol, MD

Gauge patient satisfaction

Patient satisfaction also plays a pivotal role in PROMs, with pain control and close patient monitoring listed at the top of concerns surgeons try to address with their patients. However, Briggs said that sometimes, such as in cases with patients suffering from arthritis, patient conditions can degrade and a case may become more about managing pain rather than remedying it entirely.

“As people lose joint space and become more degenerative, some procedures are less likely to provide relief of symptoms and increase satisfaction. Satisfaction is tightly linked to symptoms and function, which always relates to what kind of activity you do,” Briggs said.

Surgeons should reasonably set patient expectations before surgery, she said. While achieving good motion in the joint is important, establishing a timeline for returning to a desired activity can also increase a patient’s overall satisfaction.

Psychological factors

John Nyland, DPT, SCS, EdD, ATC, CSCS, FACSM, told Orthopedics Today that psychological factors have a lasting impact on patients’ desire to return to sport after ACL reconstruction. Patient factors such as level of commitment, altered motivation, fear of re-injury and kinesiophobia, curb nearly one-third of athletes’ returning to sports. Nyland uses positive reinforcement and group rehabilitation to help combat these psychological factors.

“Modeling or vicarious learning — seeing others perform sports-related movement tasks with success — can have a strong effect on improving one’s confidence and self-efficacy,” Nyland said. “This can be done through performing some rehabilitation activities in groups or having patients observe and discuss an educational movement video, so they have a better understanding of movement challenges and how they can be successfully performed following surgery.”

Nyland noted that no widely accepted standardized outcome measurement exists that evaluates combined concepts such as fear, anxiety, motivation and commitment level, making it difficult to directly compare studies in this area.

“We need to develop a concise survey that measures these factors. Even if normal knee range of motion, proprioception, strength and power are re-established, if the patient is afraid or has low self-efficacy, it is unlikely that they will return to their previous level of sports participation. When this happens, patients may be dissatisfied and outcome scores will be reduced despite the fact they had an otherwise successful ACL reconstruction and rehabilitation,” he said.

Use in studies

Within orthopedics, PROMs range from the general to the specific. Some, such as the SF-36, are widely used among subspecialties. Some measures, such as the Single Assessment Numeric Evaluation (SANE) score, rate a patient’s outcome on a 100-point scale using one question, while others such as the DASH score contains 30 questions to assess physical function and symptoms.

“There are so many different scores, I think the problem becomes trying to compare and contrast results of papers when different outcome measures are used,” Egol said. “I am not saying one [scoring system] is better than the other, but perhaps journals and societies that sponsor these research trials should have standardized scores that everyone should be utilizing so people can make comparisons between different studies.”

In addition, surgeons must decide which outcome measures to use in studies, taking into account various demographic factors and whether to give simple or complex questionnaires to a patient group. Another issue to consider is the ceiling effect — the higher the number of questions on a form, the more likely it is that patients will not pick extreme high or low answers.

Beach uses a custom “bubble” form that uses SANE, American Shoulder and Elbow scale (ASES) and EuroQol (EQ-5D) outcome measures to assess patients.

“The problem is if you make the questionnaires too long or complex, nobody finishes it. Patients will not take the time to do six pages of questions,” Beach said. “You want it to be as granular as you can be without making it onerous.”

The exception is in the military, where soldiers are a “captive audience” and can be asked to fill out detailed questionnaires. As soldiers constitute an active population, results from military studies are also “almost immediately applicable to everyone in the sports medicine community,” Beach said.

“People in private practice have less control over how much time [patients] are going to spend on a particular task like answering a questionnaire,” he added.

PROMs and reimbursement

A growing issue regarding PROMs stems from new health care legislation. The current administration hopes to use new policy guidelines and PROMs to gauge which physicians will receive Medicare reimbursement. Raj Rao, MD, professor and director of spine surgery at the Medical College of Wisconsin, said these new laws are making physicians uneasy, as it not only changes how they collect data, but how they interact with patients.

“I think the biggest immediate problem is a lot of apprehension — a lot of uncertainty — about exactly how the model is going to evolve… ” — Raj Rao, MD

“I think the biggest immediate problem is a lot of apprehension — a lot of uncertainty — about exactly how the model is going to evolve, and exactly what physicians and physician offices will need to do in order to document and determine patient-related outcomes in addition to their normal daily volume of work,” Rao told Orthopedics Today.

Although the details of these proposals are evolving, the new laws will reward physicians for implementing electronic health care recorders or be penalized for failing to do so. Although many physicians are already recording their patient data in this manner, there are some practices that may not have the time — or the finances — to make this change.

“These are huge undertakings, to change the way we practice medicine and prescribe medicine, [transitioning] from a paper system to an electronic system,” Rao said. “In reality, what has happened is these things have come down the pipe rapidly, and physician practices have been required to adapt or adopt these new practices in a somewhat hurried fashion without a clear understanding of what the ramifications of these various electronic processes might be.”

Future outcome scores

As new surgical techniques and implant types develop, older outcome measures, some of which were developed more than 25 years ago, will become outdated and new measurements will take their place. One example of a score that measures factors beyond traditional PROMs is the Forgotten Joint Score, or FJS-12, which takes its name from the ideal situation of a total hip or total knee replacement surgery that is so successful that patients “forget” that they have an implant.

“In our opinion, this is the ultimate goal to ensure maximum patient satisfaction,” Behrend and colleagues wrote in their study. “This new concept integrates a variety of variables such as pain, stiffness, function in activities of daily living, patients’ expectations, patients’ activity levels and psychosocial factors.”

In an initial validation study of the FSJ-12, the authors noted that it outperformed the WOMAC osteoarthritis index in several areas, including discriminatory power and combating the ceiling effect, despite having roughly half the number of questions as the WOMAC index.

Another new outcome measure comes from the Knee Society, replacing the Knee Society Scoring System created in 1989. This new system differs from the previous iteration through the creation of new measures to analyze the younger, more diverse population that now undergoes total knee arthroplasty (TKA).

In a study by Noble and colleagues, the survey was given to 497 patients and validated against the Knee Injury and Osteoarthritis Score and SF-12 scores. The authors wrote that the new system is a “valid instrument based on surgeon and patient-generated data, adapted to the diverse lifestyles of contemporary patients with TKA.” – by Jeff Craven

References:

• Ageberg E, Forssblad M, Herbertsson P, Roos EM. Sex differences in patient-reported outcomes after anterior cruciate ligament reconstruction: Data from the Swedish knee ligament register. Am J Sports Med. 2010:38(7):1334-1342.
• Behrend H, Giesinger K, Giesinger JM, Kuster MS. The forgotten joint as the ultimate goal in joint arthroplasty. Validation of a new patient-reported outcome measure. J Arthroplasty. 2011. Published online before print Oct. 12, 2011.
• Brand E, Nyland J. Patient outcomes following anterior cruciate ligament reconstruction: The influence of psychological factors. Orthopedics. 2009:32(5):335.
• Walsh MG, Davidovitch RI, Egol KA. Ethnic disparities in functional and mental recovery following fracture healing. Paper #34. Presented at the 24th Annual Meeting of the Orthopaedic Trauma Association. Oct. 16-18. 2008.
• Khanna G, Singh JA, Pomeroy DL, Gioe TJ. Comparison of patient-reported and clinician-assessed outcomes following total hip arthroplasty. J Bone Joint Surg Am. 2011:19; 93(20):e117(1)-(7).
• Kunkel S, Eismann E, Cornwall R. Utility of the pediatric outcomes data collection instrument for assessing acute hand and wrist injuries in children. J Pediatr Orthop. 2011:31(7):767-772.
• Noble PC, Scuderi GR, Brekke AC, et al. Development of a new knee society scoring system. Clin Orthop Relat Res. 2012:470(1):20-32.
• Tashjian RZ, Henn RF, Kang L, Green A. The effect of comorbidity on self-assessed function in patients with a chronic rotator cuff tear. J Bone Joint Surg Am. 2004;86(2):355-362.

• William R. Beach, MD, can be reached at Tuckahoe Orthopedics, 1501 Maple Ave., Richmond, VA 23226; 804-285-2300; email: beach@orv.com.
• Karen Briggs, MBA, MPH, can be reached at the Steadman Clinic Philippon Research Institute, 181 West Meadow Dr., Suite 1000, Vail, CO 81657; 970-479-9797; email: karen.briggs@sprivail.org.
• Kenneth A. Egol, MD, can be reached at the Hospital for Joint Diseases, New York University Longhorne Medical Center, 303 2nd Ave., New York, NY 10003; 212-598-6137; email: kenneth.egol@nyumc.org.
• Gaurav Khanna, MD, can be reached at the Scripps Clinic, Orthopedic Surgery, 10666 N. Torrey Pines, MS 116, La Jolla, CA 92037; 800-780-1277; email: khan0044@gmail.com.
• John Nyland, DPT, SCS, EdD, ATC, CSCS, FACSM, can be reached at the Division of Sports Medicine, Department of Orthopaedic Surgery, University of Louisville, 2301 South 3rd St., Louisville, KY 40292; 502-852-2782; email: john.nyland@louisville.edu.
• Raj Rao, MD, can be reached at the Department of Orthopaedic Surgery, Medical College of Wisconsin, 9200 W. Wisconsin Ave., Milwaukee, WI 53226; 414-805-7425.
• Disclosure: Beach, Briggs, Egol, Khanna, Nyland and Rao have no relevant financial disclosures.

How much weight should be given to patient-reported outcomes compared to clinician-assessed, objective outcomes?

Depends on the treatment objectives

Michael Suk

It is increasingly recognized that traditional clinician-based outcome measures need to be complemented by measures that focus on the patient’s concerns in order to evaluate interventions and identify whether one treatment is better than another. The amount of weight given to one form or another depends on the specific treatment objective.

Consider the most common symptom in orthopedics: pain. There is no clinical-based measure of this symptom, and yet, a common indication for many orthopedic procedures is “pain relief.” On the other hand, an equally important parameter, “function,” may be best objectively measured and reported through a clinician-based outcomes measure (e.g., 3-meter walk test).

It is clear that to obtain the best overall picture of a patient’s condition both clinician-based and patient-reported outcome measures are critical to assessing success. There is no generic formula to advise about relative importance of these in every situation. Their importance depends upon the condition in question, availability of valid measurement instruments and, most importantly, upon patient preferences for health outcomes. In the near future, incorporating the role of patient expectations will be the new frontier of outcomes research.

Michael Suk, MD, JD, MPH, FACS, is the chairman of the department of orthopedic surgery at Geisinger Health System in Danvile, Penn.
Disclosure: Suk has no relevant financial disclosures.

Both outcomes have pros and cons

Robert T. Trousdale

Both patient-reported outcomes and clinical assessment outcomes have strengths and weaknesses. Certainly, in the past, orthopedic surgeons focused more on clinically assessed outcomes. Over the last decade or so, patient-reported outcomes have come to the forefront and have become an important outcome tool to assess patient satisfaction after a certain intervention. Patient-reported outcomes have the strength in that they directly assess the patient’s perception of the outcome of the intervention. Certainly, this is an important outcome measure when the patient is the one receiving the outcome.

Patient outcomes can be influenced by many factors, not all of which may be directly related to the intervention given. For instance, if the patient undergoes a total knee replacement and has a pain-free knee with good range of motion but was unhappy with the hospital food or the parking situation in the hospital, it may affect the reported outcome after the intervention when in fact the actual intervention was successful. Hence, care should be taken when using patient-reported outcomes also, as there are a lot of variables that may affect their results.

Clinical-assessed outcomes have the strength of using the clinician’s knowledge and experience to choose what he or she feels are important. There are many validated outcome measures in all areas of orthopedic surgery, and many of those focus on what the surgeon feels are important parameters for the patient (in range of motion, functional improvement and pain relief). The problem with clinical-assessed objective outcomes is that they are, of course, influenced when given to the patient by other factors other than the direct outcome of the surgery. If patients socially like the clinician, they may answer the questions differently than if they feel they did not have a good relationship with the physician despite the success or lack of success of that intervention.

Hence, at the end of the day, both patient-reported outcomes and clinically assessed outcomes are important as long as one recognizes the strengths and weaknesses of both tools.

Robert T. Trousdale, MD, is an Orthopedics Today Editorial Board member and a professor of orthopedics at the Mayo Clinic in Rochester, Minn.
Disclosure: Trousdale receives royalties from DePuy, MAKO Surgical Corp. and Ortho Development.