Variations in quantifying patient reported outcome measures to estimate treatment effect

Abstract

In the practice of healthcare, patient-reported outcomes (PROs) and PRO measures (PROMs) are used as an attempt to observe the changes in complex clinical situations. They guide us in making decisions based on the evidence regarding patient care by recording the change in outcomes for a particular treatment to a given condition and finally to understand whether a patient will benefit from a particular treatment and to quantify the treatment effect. For any PROM to be usable in health care, we need it to be reliable, encapsulating the points of interest with the potential to detect any real change. Using structured outcome measures routinely in clinical practice helps the physician to understand the functional limitation of a patient that would otherwise not be clear in an office interview, and this allows the physician and patient to have a meaningful conversation as well as a customized plan for each patient. Having mentioned the rationale and the benefits of PROMs, understanding the quantification process is crucial before embarking on management decisions. A better interpretation of change needs to identify the treatment effect based on clinical relevance for a given condition. There are a multiple set of measurement indices to serve this effect and most of them are used interchangeably without clear demarcation on their differences. This article details the various quantification metrics used to evaluate the treatment effect using PROMs, their limitations and the scope of usage and implementation in clinical practice.

Key Words: Patient-reported outcome measures; Treatment effect; Minimal clinical important difference; Patient-accepted symptom state; Minimum detectable change; Orthopedics

Core Tip: In health care, patient-reported outcomes and patient-reported outcome measures (PROMs) help track changes in complex clinical situations. They provide evidence-based guidance for patient care by showing how a treatment affects a specific condition and if the patient benefits from it. For PROMs to be useful, they must be reliable and able to detect real changes. Regular use of structured outcome measures helps doctors understand a patient's limitations better than just an office interview. This allows for meaningful discussions and personalized treatment plans. Understanding how to measure treatment effects with PROMs is crucial, as there are many different metrics, often used interchangeably. This article explains these metrics, their limitations, and their practical use in healthcare.

INTRODUCTION

In the practice of healthcare, patient-reported outcomes (PROs) and PRO measures (PROMs) are used as an attempt to observe the changes in complex clinical situations. They guide us in making decision based on the evidence regarding patient care by recording the change in outcomes for a particular treatment to a given condition and finally to understand whether a patient will benefit from a particular treatment and to quantify the treatment effect[1]. According to the US Food and Drug Administration, PRO is any report coming directly from patients about a health condition and its treatment[2]. For any PROM to be usable in health care, we need it to be reliable, encapsulating the points of interest with a potential to detect any real change[3]. Using structured outcome measures routinely in clinical practice helps physicians to understand the functional limitation of patients, which would otherwise be not clear in an office interview, and this allows the physicians and patients to have a meaningful conversation as well as a customized plan for each patient. The importance of serially and routinely measuring outcomes is stressed by Codman, the father of modern-day outcome assessment[4] and the rationale behind collecting PROs is as follows: Better communication aids that also make the decision-making process shared between patients and providers; subjective assessment of health status and identification of treatment lacunae; quantifying the loss of function; distinguishing between problems due to physical, emotional and social reasons; identifying adverse effects of treatment methods; estimation of disease progression and treatment response; helping change treatment methods; and prognostication of disease course and treatment outcomes[5-7].

Having mentioned the rationale and the benefits of PROMs, understanding the quantification process is crucial before embarking on management decisions. Traditionally, statistical methods are used to measure the difference before and after an intervention. However, statistical significance may not relate to clinical improvement[8]. A better interpretation of change needs to identify the treatment effect based on clinical relevance for a given condition. Thresholds to measure the clinical relevance or significance of change can be of three types[8]. First, the minimum difference to understand the clinical relevance below which it cannot be distinguished from the random error; second, the difference between the scores pre- and postintervention, which can be perceived as good or bad by the patient; and finally, the difference that is perceived as clinically relevant or meaningful. There are several metrics to serve this purpose (Table 1)[8] and some of the most used are discussed in this review.

Table 1 Measurement indices to quantify patient-reported outcomes measures for clinical relevance.

Measure of change	Satisfaction threshold
Minimal clinically important difference	Patient acceptable symptom state
Minimal important change	Substantial clinical benefit
Minimal clinically important change
Minimal clinically important improvement
Minimal clinically significant difference
Minimal perceptible clinical improvement
Clinically important difference
Minimal important difference
Minimum detectable change
Minimal detectable difference
Smallest detectable difference

In a step towards understanding and standardizing the PROMs, Jaeschke et al[9] have described the concept called minimal clinically important difference (MCID) to aid in interpreting the questionnaire scores. Following MCID, other metrics to assess patients’ perception of treatment effect were developed for interpreting PROMs, called “the alphabet soup” by Tashjian[4], including patient-acceptable symptomatic state (PASS), substantial clinical benefit (SCB) and maximal outcome improvement (MOI). This review analyses these commonly used evaluation metrics of PROMs to aid better comprehension and implementation of these measures in clinical practice.

MCID

According to Jaeschke et al[9], MCID is defined as “the smallest difference in score in the domain of interest which patients perceive as beneficial and which would mandate, in the absence of troublesome side effects and excessive cost, a change in the patient’s management.” In this definition, three important things to note include patient perception; absence of excessive cost and troublesome side effects; and mandating change in management[10]. As shown in Table 1, there are multiple terms similar to MCID but they are different in their own definitions. MCID can be estimated for an outcome measure by various methods[11,12]. We discuss the commonly used methods to estimate MCID for a given PROM. Consensus method, anchor-based method, distribution-based method and a combination of anchor-based and distribution-based method[13].

Consensus method

In this method of assessment of MCID, an expert panel is convened, discussed and consensus is reached on the proposed MCID for the outcome of interest. The main problem with this method is that the patients’ perspective is not taken into consideration.

Anchor-based method

Outcome scores are compared with an independent, external face valid criterion called anchor to determine MCID of the particular outcome in question. Generally, anchor is a reliable and valid questionnaire for which patients respond based on their perspective[13]. Transition question, patient global impression change or patient global assessment of treatment effectiveness are some of the examples of the anchor questions. The threshold used to calculate MCID of an anchor question is minimally improved (it can be minimally deteriorated as well). Gum et al[14] have estimated MCID for back and leg pain for the numerical rating scale (NRS) 0–10 using better as threshold and concluded that a decrease in the score of ≥ 3.1 was considered as minimally improved[14]. The anchor-based method takes into account patients’ perspective, for which these metrics were designed in the first place, unlike purely statistical approaches. Second, this method cannot be used in conditions where most patients get better and the ones who are unchanged are minimal. Third, anchor questions do not take into consideration the variability in the sample. Finally, the idea of MCID will only help in understanding whether there is an understandable improvement but not if that improvement is meaningful to the patient[15-18].

Distribution-based method

Paradoxically, this method uses statistical means to measure MCID. One such statistical means is to use standard error measurement (SEM) as it reflects the lack of precision in the measurement. Thus, any value below SEM cannot be MCID as this does not show any real change. Alternatively, the minimal detectable change (MDC) is calculated, which by definition is the smallest change that can be detected beyond the measurement error. In this method, MCID is considered as the upper value of the 95%confidence interval (CI) of the average score in nonresponders for a specific intervention. Usually MCID is shown to be on average similar to either 1 SEM or half the standard deviation[19]. Gum et al[14] and Carreon et al[20] assessed MCID in lumbar fusion surgery[14,20]. This method is also not without limitations. First, one can define MCID only based on the hope that change in the score is not due to the measurement error. Second, patient perspective is not accounted in this method.

Combination of distribution- and anchor-based methods

In this approach, an anchor question is used to differentiate between the responders and nonresponders, and then uses MDC to calculate the MCID as described above and uses the upper value of the 95%CI in nonresponders as MCID. Using this method, the MCID of visual analog scale (VAS) (0–10) was 2.1 for neck pain but on validation by receiver-operator characteristics (ROC) analysis, the cutoff was 4.1. However, for arm pain, both the methods, MCID and ROC, resulted in 4.1 and 4.0 as cutoff points, respectively. This method of assessing is more complex compared to the previously described methods. However, it is advantageous compared to the pure anchor-based method that is vulnerable to the sample variability.

Although MCID is useful to compare the efficacy of treatment in clinical trials and determine the efficacy of treatment in individual patients to inform treatment effect, there are some notable pitfalls. These include variability in the metric based on the quality of the data used, method used, anchor type, definition of improvement, population demographics, and their perception of symptoms and functional limitations. The weaknesses of using MCID involves the lack of universal fixed attribute that can be used across different patient populations. There is no consensus on the method to calculate, leading to extreme variability. Ostelo et al[21] found that depending on the method used to calculate MCID, Oswestry Disability Score for low back pain, on a scale of 0–100 varied between 2 and 8.6. Taking these data, Wright et al[3] explained how this could be disastrous in clinical practice.

There is no single value for MCID for any specific outcome measure as it can be influenced by type of patient population and the method used to estimate. MCID, if reported as single point estimate rather than a confidence interval, can be problematic because it can risk misclassifying the outcomes in patients as not improved even when they were improved. Rossi et al[22] concluded that the calculation of MCID is not as important as it seems. They reported MCID to be "a low bar" and recommended scientific studies to not only provide MCID but also mention PASS and SCB to be meaningful to both the scientific community and more importantly to the patient for whom a meaningful improvement makes more sense than a minimum one, as illustrated in Figure 1.

Figure 1 Illustration of the various quantification indices in the context of numerical pain rating scale for knee osteoarthritis. MCID: Minimal clinically important difference; SCB: Substantial clinical benefit; PASS: Patient acceptable symptomatic state.

SCB

When assessing the clinical outcome of a patient, one does not reach a floor value like MCID but one would expect to see a substantial clinical improvement, which is the SCB; a concept introduced by Glassman et al[23]. SCB is the minimum amount of change in a PROM that allows a patient to feel sufficiently or substantially better after treatment. Generally, MCID is considered the lower limit of treatment effect and SCB is the upper limit of any meaningful outcomes of a treatment.

SCB is measured based on the anchor method as detailed for MCID. The commonly used anchor question is “Compared to the first evaluation, how is your physical condition now?” This question is usually answered using a Likert scale response[24]. Statistical analysis is done using various techniques to determine the SCB, but the most commonly used method is ROC curve analysis. SCB values are determined for every particular PROM and for every condition distinctively.

Depending on the type of anchor questions that are used, there can be an issue of recall bias in calculating SCB, as it is with MCID. Hubbard et al[24] used two anchor questions instead of one. While the first question was used to find out the improvement in the physical function since the first visit, the other was used to assess SCB. Similarly, Glassman et al[23] used five satisfaction statements in their study[23] to standardize the SCB determined. Although important, SCB is rarely reported in the published literature[25]. However, Wellington et al[26] showed that when patients were divided into those from different geographical locations or times, there was a high degree of variability in their SCB thresholds for total shoulder arthroplasty. Hence, SCB also suffers variability similar to MCID based on population characteristics.

PASS

Unlike MCID, which attempts to compare the pre- and postintervention scores for a given condition, PASS is a cross-sectional evaluation of how patients feel at a given point in time[27]. It is the magnitude of result that makes the patients feel fulfilled[4]. Several studies have proposed that overall improvement in health status is one of the crucial factors irrespective of the intervention for a given condition[28-30]. PASS is a holistic satisfaction score of the present health status of patients and not just related to the symptoms of a particular disease or intervention[31].

PASS cutoff point is estimated using an anchor question that has a binary response of yes or no. One of the most common question that is used is: "Taking into account your level of pain and also your functional impairment, if you were to remain for the next few months as you are today, would you consider that your current state is satisfactory?"[32]. Along with the anchor question, a 75th percentile method and ROC methods are used to reach a cutoff value for specific PROMs for a given condition[33,34]. MCID was called a low bar by Rossi et al[22]; however, PASS was called as “an ambitious target for disease management” by Maksymowych et al[35], thereby making it something to look forward to in PROMs[22,35] (Figure 1).

DISCUSSION

One of the most important thing that researchers and clinicians need to remember is that these metrics used to evaluate the treatment effect of interventions are not universal. They are usually specific for the condition that they are calculated for as well as the outcome measure that is used to calculate the change[36,37]. However, MCID does not differ with the treatment used when the same condition is assessed with the same outcome measure[13,38]. Katz et al[13] considered “improvement is improvement regardless of what produced it”. Farrar et al[38] demonstrated that pain intensity-NRS (PI-NRS), an 11-point pain measurement instrument (0-10), similar to MCID could be used among a host of conditions such as osteoarthritis, painful diabetic neuropathy, and low back pain. However, Stauffer et al[38,39] demonstrated that when a different version of VAS (0–100 mm) was used, MCID differed among different disease states such as knee and hip osteoarthritis and back pain.

Disease severity is another variable that influences these metrics. Patients with lower preoperative scores find it easier to achieve MCID and SCB, whereas those with higher scores were better off to reach PASS[13,40]. Patients with a severe disease state have more room to reach the clinically important or significantly better state but those with a low severity have not enough room. Therefore, in patients with low disease severity, it is difficult to define MCID or SCB as they do not have sufficient room to become better. Hence, one can consider patients with low health status or a severe disease with significant functional limitations to have a higher chance of achieving MCID or SCB after an intervention, as the room for improvement is higher but the chances of reaching the PASS threshold remain unpredictable[40,41].

Use of MCID, SCB and PASS is not practical in daily clinical practice, because each individual patient may not perceive the change in their health status in a similar manner. For example, if PASS threshold on 11-point PI-NRS is 3 for a specific condition, a patient in with PI-NRS 2.5 might still not be able to accept the present condition as satisfied[4]. However, Goh et al[27] described the PASS thresholds for multiple PROMs following unicompartmental knee arthroplasty, and recommended these thresholds as the target to treat the condition in future studies.

As already described, patients with higher functional scores or less severe disease status pre-intervention may not be able to reach MCID or SCB owing to the lack of enough room for improvement. In this regard, Berglund et al[42] have proposed a new metric called MOI[42]. MOI is a threshold for an outcome measure normalized to the maximum possible outcome for each patient who considers to have achieved a satisfactory result. Tashjian in his editorial commentary claimed MOI as the threshold that can be used at an individual patient level in daily practice[4]. MCID, SCB and PASS are more meaningful when discussing the outcome of a group of patients than an individual patient. These metrics can also be used to assess the sample size, power of a study etc., as well in the statistical aspect of the research that usually takes into consideration only the numbers, but with these metrics, we are introducing the aspect of patient perception of change or satisfaction.

Comparison of these metrics among different studies remains difficult as there is a lack of consensus on their assessment methods[43,44]. Depending on the type of method that is used to assess the threshold, the value of these metrics can and will be different, making it a priority for researchers to come to a consensus on their estimation methods. The type of anchor, number of questions and responses to be used, and identification of responses that are chosen as no, minimal or substantial change need to be ascertained because they have a significant impact on the evaluation metric that is calculated. Considering the amount of variability, achieving universal threshold for the PROMs does not seem to be on the horizon currently. One of the ways in which this variability could be managed is to define a range of measurement as threshold for these evaluation metrics rather than a single cutoff value[45]. Standardization of the methods to estimate these threshold ranges needs to be developed to aid in universal acceptability and ease of use in both research as well as in daily clinical practice[46,47]. Table 2 gives the list of common orthopedic PROMs for hip, knee and shoulder ailments and their MCID and PASS cutoff values[48-77].

Table 2 Characteristics of patient-reported outcomes measures and their quantification metric cut-off values, n (%).

Patient population	PROM	Value range	MCID	PASS
Hip
Arthroscopy patients[48,49]	mHHS	0-100	MDC 12	74
Measure of function in patients with hip disability[48,50-52]	HOS	0-100	MCID 9	87
Hip/groin disability[53-56]	HOOS	0-100	MDC 10	NR
Physically active patients with long standing hip/groin pain[57]	HAGOS	0-100	SDC 19	NR
Young (18-60 yr) active patients with hip disorders[58]	iHOT-12	0-100	NR	NR
Young (18-60 yr) active patients with hip disorders[59]	iHOT-33	0-100	MCID 6.1	NR
Young active nonarthritic patients with hip ailments[60]	NAHS	0-100	MDC 10	NR
Knee
Patients with knee pain[61,62]	IKDC-SKF	0-100	MCID 6.3 (6 months); MCID 16.7 (12 months)	75.9
Patients with knee injury or osteoarthritis[63]	KOOS	0-100	SDC 16.6	88.9
Patients with knee injury/pain[64]	Lysholm	0-100	MDC 8.9	NR
Patients with knee pain due to any condition[62,65]	Cincinnati	0-100	MCID 14 (6 months); MCID 26 (12 months)	NR
Patients with knee pain due to any condition[66]	WOMAC	0-100	MCID 11.5	NR
Patients with osteoarthritis knee[64]	Tegner activity	0-10	MDC 1	NR
Shoulder
Patients with shoulder instability[67]	WOSI	0-2100	MCID 220 (10.4)	NR
Patients with rotator cuff problems[68]	WORC	0-2100	MCID 245.25 (11.7)	NR
Patients with shoulder osteoarthritis[69]	WOOS	0-1900	NR	NR
Patients with shoulder pain due to instability, rotator cuff disease or arthritis[70]	ASES	0-100	MCID 6.4 MDC 9.7	NR
Patients with shoulder conditions including fracture, arthroplasty, cuff repair, adhesive capsulitis[71]	Constant	0-100	MCID 10.4	44
Patients with shoulder dysfunction[72]	SST	0-100	SDC 2.8	NR
Patients with upper limb disorders[73,74]	DASH	100-0	MCID 10.2; MDC 6.6-12.2	43
Patients with shoulder pathology from musculoskeletal, neurogenic, or other origin[75]	SPADI	0-100	MCID 8-13.2; MDC 18	41
Wrist and elbow
Patients with wrist conditions, ulnar impaction, tendonitis, arthritis, or nerve compression syndrome from forearm to hand[76]	DASH	100-0	MCID: 10-13.5; MDC: 9.3	NR
Patients with wrist conditions, ulnar impaction, tendonitis, arthritis, or nerve compression syndrome from forearm to hand[76]	PRWE	100-0	MCID: 14-17; MDC: 7.7	NR
Ankle
Patients with chronic ankle instability[76]	FAAM	0-84	MCID 8	NR
Patients with hallux valgus[77]	MOXFQ	100-0	MCID 12	NR

ASES: American Shoulder and Elbow Surgeons Score; DASH: Disabilities of the arm, shoulder and hand; FAAM: Functional Ankle Ability Measure; HAGOS: The Copenhagen Hip and Groin Outcome Score; HOOS: Hip dysfunction and osteoarthritis outcome score; HOS: Hip outcome score; iHOT-12: International Hip Outcome Tool-12; iHOT-33: International Hip Outcome Tool-33; IKDC-SKF: International Knee Documentation Committee Subjective Knee Form; KOOS: Knee injury and osteoarthritis outcome score; MCID: Minimal clinically important difference; MDC: Minimal detectable change; mHHS: Modified Harris Hip score; MIC: Minimal important change; MOXFQ: Manchester-Oxford Foot Questionnaire; NAHS: Nonarthritic hip score; NR: Not yet reported in the literature; PASS: Patient acceptable symptom state; PRWE: Patient-rated wrist evaluation; SDC: Smallest detectable change; SPADI: Shoulder Pain and Disability Index; SSS: Sport-specific subscore; SST: Simple shoulder test; WOMAC: Western Ontario and McMaster Universities Arthritis Index; WOOS: Western Ontario Osteoarthritis of the Shoulder Index; WORC: Western Ontario Rotator Cuff Index; WOSI: Western Ontario Shoulder Instability Index.

The findings of this study call for a unified approach in quantifying the PRO and its treatment effect measure for a given condition for the benefit of the readers and researchers. The concept of core outcome dataset (COD) is being developed to emphasize this concept[78,79]. However, they were not put into action as a standard practice due to the lack of necessary reporting guidelines. Authors suggest journals to facilitate the necessary COD for a given condition as a necessary publishing requirement. Although not possible for all study methods, studies of higher clinical impact such as randomized controlled trials should be mandated towards the same. Having tried to implement the COD concept and looking at its impracticality, the concept of minimum COD is now in development for various clinical conditions. The impracticality of the idea lies in the regional differences in the context of outcome measures utilized. The outcome measures and their treatment effect noted to be relevant in one part of the world may not be relevant to the other and making them mandatory only makes them impractical. Hence, the concept of minimum COD is in vogue to account for the regional, economic and cultural variations in outcome measurements[80]. Hence, the authors suggest that clinicians move towards a standard minimum COD for the condition with a standardized measure of treatment effect to make the reported results meaningful to the readers and researchers in the present and future.

CONCLUSION

There is substantial variability in the estimation of treatment effect through indices such as MCID, SCB or PASS for a given intervention and patient population that prevents their generalizability. Hence, researchers and clinicians must exercise caution while utilizing these indices with their patient population to estimate the treatment effect for any given intervention. The author suggests utilization of minimum COD for outcome selection and their recommended estimation of treatment effect for the given conditions to establish a standardized reporting method beneficial to global readers and researchers.