Machine learning for patient selection in corticosteroid decision making in knee osteoarthritis: A feasibility model

Abstract

BACKGROUND

Relieving pain is central to the early management of knee osteoarthritis, with a plethora of pharmacological agents licensed for this purpose. Intra-articular corticosteroid injections are a widely used option, albeit with variable efficacy.

AIM

To develop a machine learning (ML) model that predicts which patients will benefit from corticosteroid injections.

METHODS

Data from two prospective cohort studies [Osteoarthritis (OA) Initiative and Multicentre OA Study] was combined. The primary outcome was patient-reported pain score following corticosteroid injection, assessed using the Western Ontario and McMaster Universities OA pain scale, with significant change defined using minimally clinically important difference and meaningful within person change. A ML algorithm was developed, utilizing linear discriminant analysis, to predict symptomatic improvement, and examine the association between pain scores and patient factors by calculating the sensitivity, specificity, positive predictive value, negative predictive value, accuracy, and F2 score.

RESULTS

A total of 330 patients were included, with a mean age of 63.4 (SD: 8.3). The mean Western Ontario and McMaster Universities OA pain score was 5.2 (SD: 4.1), with only 25.5% of patients achieving significant improvement in pain following corticosteroid injection. The ML model generated an accuracy of 67.8% (95% confidence interval: 64.6%-70.9%), F1 score of 30.8%, and an area under the curve score of 0.60.

CONCLUSION

The model demonstrated feasibility to assist clinicians with decision-making in patient selection for corticosteroid injections. Further studies are required to improve the model prior to testing in clinical settings.

Key Words: Knee osteoarthritis; Machine learning; Predictive modelling; Corticosteroid injection; Patient selection

Core Tip: Historically, the efficacy of corticosteroid injections in knee osteoarthritis has been heavily debated, as patient responses can vary significantly. This study evaluates the feasibility of a machine learning model to identify which patients with knee osteoarthritis will benefit from corticosteroid injections. Data from two cohort studies were combined for analysis. The model generated an accuracy of 67.8% (95% confidence interval: 64.6%-70.9%), F1 score of 30.8%, and an area under the curve score of 0.60. These metrics demonstrate feasibility to assist clinicians with decision-making in patient selection for corticosteroid injections. Further research is required to improve the model prior to testing in clinical settings.

INTRODUCTION

Osteoarthritis (OA) is a degenerative joint disorder characteristic by progressive loss of articular cartilage, and a leading cause of pain and disability worldwide[1]. Knee OA represents the overwhelming majority of the disease burden[2]. Its presentation can vary, but the cardinal features are joint pain, impaired day-to-day functioning, and a diminished quality of life[3]. Factors influencing variations in symptoms among patients include age, gender, body habitus, genetic predisposition, muscle weakness, prior injuries or comorbidities, and job requirements[4]. Prior to end-stage disease, which usually requires knee replacement surgery, management involves a multidisciplinary approach using both non-pharmacological and pharmacological interventions to optimize symptom control and limit disease progression[3]. Non-pharmacological interventions such as patient education, weight loss, and muscle strengthening exercises form the foundational layer of management[3,5]. Whilst pharmacological agents are central to pain management, non-steroidal anti-inflammatory drugs, paracetamol, opioids, and antidepressants amongst the most frequently used agents[5,6].

Intra-articular corticosteroid injections are an alternative pharmacological method of managing the debilitating pain in knee OA, serving as the middle ground between oral analgesics and surgical intervention[5]. Commonly used in patients with knee OA, with seven million performed annually in the US alone, there remains much debate regarding its efficacy, particularly the extent of its benefits[7]. Raynauld et al[8] demonstrated improvements in night pain (P < 0.05) and stiffness (P < 0.05) at two-year follow-up in 34 patients receiving 40 mg triamcinolone acetate injections every three months compared to patients receiving placebo, supporting its role in long-term pain control. However, in a study by Jones et al[9], 59 patients with knee OA received a single 40 mg injection of methylprednisolone acetate. Whilst the treatment produced a significant improvement in visual analog scale pain scores at three weeks compared to baseline, with a median change of -2 points (interquartile range: -16.25 to 4.0, P < 0.05), these benefits were not sustained at eight-week follow-up. There was no change in median visual analog scale score compared to baseline (interquartile range: -14.5 to 8.0, P > 0.05), suggesting its benefits are short-lived[9]. Corticosteroids also have potential side-effects, including local infection, joint destruction, muscle atrophy, and tendon rupture[5,10].

Machine learning (ML) employs algorithms to analyze large datasets and identify patterns[11]. These algorithms can be used to assist clinical decision making, enabling a more personalized approach to patient care which translates into improved patient outcomes[12]. In the context of knee OA, its application has been described at various stages: To guide clinical decision-making, including predicting early diagnosis, disease progression, and the need for total knee replacement[13-15]. However, no studies have explored the use of corticosteroid injections in knee OA using ML models. This study, therefore, aims to develop a ML algorithm to identify which patients with knee OA will benefit from corticosteroid injections.

MATERIALS AND METHODS

Design

This study is a secondary analysis of the Multicentre OA Study (MOST) and the OA Initiative (OAI) datasets. MOST is a longitudinal, prospective observational study funded by the National Institute on Aging in the United States[16]. The aim of the MOST study was to evaluate factors that can affect prognosis in patients with knee OA. Patients between the ages of 50 to 79 were selected based on having pre-existing knee OA or being at high risk of knee OA[16]. The OAI is also a longitudinal, prospective observational study funded by the National Institute of Health and the private sector. Recruitment criteria included male or female patients between the ages of 45 to 79 with symptomatic knee OA or an increased risk of developing it during the study.

MOST data collection

Data collection began in 2003 and included 3026 patients at baseline[16]. Data collected include demographic data, medical history, joint symptoms, functional status, disability, mental health, physical performance, radiological findings, and medication/supplement history. Participants that were recruited were seen at one of the two centres for baseline clinical examinations, administration of self-reported health questionnaires, knee imaging (magnetic resonance imaging and X-ray), whole-body dual X-ray absorptiometry (DXA), and blood and urine specimens. Follow-up assessments occurred at 15 months, 30 months, 60 months, 72 months, and 84 months after recruitment with a mixture of telephone interviews and clinic visits[16].

OAI data collection

Data collection began in 2004 and included 4796 patients at baseline[17]. The OAI dataset consists of demographic data, joint symptoms, pain scores, functional status, mental health, physical performance, radiological findings, past treatments, and biological specimens. The OAI cohort underwent a comprehensive assessment schedule over an eight-year period, including in-clinic visits with biospecimen collection and imaging at baseline, 12 months, 24 months, 36 months, and 48 months. Mailed questionnaires and telephone interviews were conducted at 60 months and 84 months, while the 72-month visit included in-clinic biospecimen collection and imaging[17].

Inclusion and exclusion criteria

In this study, only patients who received intra-articular corticosteroid injections were included in the analysis. Data were collected on variables that can affect prognosis in knee OA. This included age, sex, ethnicity, body mass index (BMI)[18], comorbidities[19], X-ray findings (Kellgren Lawrence grade)[19,20], pain catastrophizing score[21,22], physical activity level score[23], presence of symptoms in the affected knee[19], baseline Western Ontario and McMaster Universities OA (WOMAC) index pain score[24], and whether patients had received corticosteroid injections in the past. Patients had multiple entries if they received treatment on more than one knee or received treatment multiple times throughout the data collection period as this represented a different event time point.

Primary outcome

The primary outcome in this study was patient-reported pain levels using the WOMAC index scale[24]. This scale evaluates pain severity during five activities of daily living: Walking, climbing stairs, sleep disturbance, sitting, lying, and standing. Knee pain in each of these situations is ranked on a 5-point scale ranging from “none” (0 points) to “extreme” (4 points), with a maximum overall score of 20 points[25,26]. Clinically significant change in pain scores was defined as any change considered clinically relevant across a population (minimally clinically important difference (MCID) or an individual person (meaningful within person change (MWPC))[27,28]. In this study, as established in the literature, a difference of five points in the WOMAC pain scale was used as the MCID, and a change of 12.5% for MWPC[28,29]. Treatment was considered successful if the patient achieved MCID or MWPC.

ML algorithm

A supervised classification algorithm was developed, utilizing linear discriminant analysis (LDA). LDA is a subspace method that involves projecting the original hyperdimensional data onto a lower-dimensional space in order to maximize the class separation, whilst also preserving key discriminative information between each class, allowing for optimal analysis of a dataset with a large number of variables[30]. In this analysis, age, presence symptoms, baseline WOMAC score, Kellgren-Lawrence grade, and physical activity scale for the elderly score were used as inputs in the ML algorithm. These variables were selected as they are established risk factors for the development and progression of knee OA[4]. It was implemented using an 80:20 test: Validation split. Model capabilities were determined using accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and F2 score. The experiment was repeated 100 times with the data “shuffled” each time, to ensure the reliability of results. The mean and confidence interval (CI) was then calculated for each metric. Furthermore, the receiver operating characteristic (ROC) curve was plotted to enhance robustness of results, both through visual inspection of the curve and the calculated area under the curve (AUC).

Ethics

For the MOST study, ethical approval was obtained from the Institutional Review Board at each of the core four sites (The University of Iowa, The University of Alabama at Birmingham, The University of California and Boston University)[16,31]. All participants provided informed consent. For the OAI study, ethical approval and informed consent was obtained from all the participants[17].

RESULTS

Cohort characteristics

The OAI and MOST enrolled 4796 patients and 3026 patients respectively, total 7822 patients. Data were collected at five different timepoints in the MOST study; however, only three of these time points included data on steroid injections, covering 9078 patients. Of these, 290 received a steroid injection, with 71 undergoing bilateral injections, giving a total of 361 entries. Forty-seven knees were removed due to missing radiographic data, leaving a total of 314 entries from the MOST study which met the inclusion criteria. In the OAI study, only one timepoint, with 4796 entries, included outcomes related to corticosteroid injection use. One hundred and five of these patients received a steroid injection, of which 18 were bilateral, making the total number of entries 123. Only 16 entries included all the data required for analysis. In total there were 330 entries from both datasets available for analysis.

The final cohort consisted of middle-aged patients (mean: 63.4 ± 8.3), who were predominantly male (70.6%), Caucasian (79.7%), and overweight (mean BMI: 32.3 ± 6.2). The majority of patients were classified as obese (55.5%), defined as a BMI greater than 30. The mean WOMAC pain score was 5.2 (SD: 4.1). The remaining baseline patient characteristics can be seen in Table 1. As seen in Table 2, there was no significant difference in the age, BMI, sex, and race of patients who achieved a clinically significant change in their WOMAC score compared to those who did not. Comorbidities were more common in patients who did not experience significant improvements in their WOMAC pain score compared to those that experienced significant improvements, but there was no statistical significance to this pattern. Following a similar pattern, there was no significant correlation between the Kellgren Lawrence grade, pain catastrophizing score, or physical activity scale for the elderly score between patients experiencing a significant or non-significant change in their WOMAC pain score.

Table 1 Baseline characteristics of patients included in final analysis.

Characteristics	Frequency	Percentage
Age (year), mean ± SD	63.4 ± 8.3	-
Gender
Male	233	70.6%
Female	97	29.4%
Race
White/caucasian	263	79.7%
Black/afro-caribbean	59	17.9%
Other	8	2.4%
BMI (kg/m2), mean ± SD	32.3 ± 6.2	-
BMI < 30	147	44.5%
BMI ≥ 30	183	55.5%
Number of comorbidities, mean ± SD	0.7 ± 1.2	-
Comorbidities ≤ 3	317	96.1%
Comorbidities > 3	13	3.9%
KLG
KLG < 2	69	20.9%
KLG ≥ 2	261	79.1%
PASE, mean ± SD	163.1 ± 91.6	-
CESD, mean ± SD	9.7 ± 9.1	-
Past steroid use
No	118	35.8%
Yes	38	11.5%
Unknown	174	52.7%
WOMAC pain score, mean ± SD	5.2 ± 4.1	-
Non-significant change	246	74.5%
Significant change	84	25.5%

BMI: Body mass index; PASE: Physical activity scale for the elderly; CESD: Center for epidemiological studies depression; KLG: Kellgren Lawrence grade; WOMAC: Western Ontario and McMaster Universities Osteoarthritis.

Table 2 Data showing the characteristics of patients that achieved significant and non-significant change in Western Ontario and McMaster Universities Osteoarthritis pain scores, n (%).

Characteristics	Meaningful clinical significant change (MCID or MWPC)		Pearson correlation coefficient	P value
	No	Yes
Age (year), mean ± SD	63.1 ± 7.8	64.3 ± 9.5	0.062	0.259
Gender			0.005	0.932
Male	174 (70.7)	59 (70.2)
Female	72 (29.3)	25 (29.8)
Race			-0.016	0.772
White/caucasian	199 (80.9)	64 (76.2)
Black/afro-caribbean	40 (16.3)	19 (22.6)
Other	7 (2.8)	1 (1.2)
BMI (kg/m2), mean ± SD	32.2 ± 6.2	32.4 ± 6.4	0.0150	0.786
BMI < 30	114 (46.3)	36 (42.9)
BMI ≥ 30	132 (53.7)	48 (57.1)
Number of comorbidities, mean ± SD	0.7 ± 1.1	0.7 ± 1.3	0.006	0.919
Comorbidities ≤ 3	239 (97.2)	78 (92.9)
Comorbidities > 3	7 (2.8)	6 (7.1)
KLG			0.034	0.537
KLG < 2	53 (21.5)	15 (17.9)
KLG ≥ 2	193 (78.5)	69 (82.1)
PASE, mean ± SD	159.8 ± 84.7	172.8 ± 109.5	0.062	0.262
CESD, mean ± SD	9.5 ± 8.6	10.2 ± 10.2	0.034	0.543
Past steroid use			-0.013	0.820
No	90 (36.6)	28 (33.3)
Yes	25 (10.2)	13 (15.5)
Unknown	131 (53.2)	43 (51.2)
WOMAC pain score, mean ± SD	5.0 ± 4.9	5.8 ± 4.1	0.084	0.128

MCID: Minimally clinically important difference; MWPC: Meaningful within person change; BMI: Body mass index; KLG: Kellgren Lawrence grade; PASE: Physical activity scale for the elderly; CESD: Center for epidemiological studies depression; WOMAC: Western Ontario and McMaster Universities Osteoarthritis.

Performance metrics of LDA model

The overall accuracy of the model was 67.8% (95%CI: 64.6%-70.9%) in identifying patients likely to benefit from intra-articular corticosteroid injections for knee OA. The LDA model had a specificity of 80.8% (95%CI: 77.2%-84.5%) and a sensitivity of 29.6% (95%CI: 23.6%-35.6%). The positive predictive value was 36.2% (95%CI: 28.5%-43.9%) and the negative predictive value 77.6% (95%CI: 74.7%-80.6%). The F1 score was 30.8% and the F2 score was 29.7% (Table 3).

Table 3 Sensitivity, specificity, positive predictive value, negative predictive value, accuracy, and F-scores of the linear discriminant analysis model.

Parameter	Mean	95%CI
Sensitivity	29.6%	23.6%-35.6%
Specificity	80.8%	77.2%-84.5%
Positive predictive value	36.2%	28.5%-43.9%
Negative predictive value	77.6%	74.7%-80.6%
Accuracy	67.8%	64.6%-70.9%
F1 score	30.8%	-
F2 score	29.7%	-

CI: Confidence interval.

ROC analysis

Figure 1 illustrates the ROC curve and the corresponding AUC. This quantifies the model’s ability to distinguish between patients who will likely benefit from corticosteroid injections and those who are not. The dashed red line represents an AUC score of 0.50, indicative of random guessing. The mean AUC of this study is 0.60 which is slightly higher and suggests moderate discriminatory ability.

Figure 1 A graph illustrating the receiver operating characteristic curve and area under the curve. AUC: Area under the curve.

DISCUSSION

In this study, only 26% of patients were observed to have a meaningful clinically important improvement in pain score following an intra-articular corticosteroid injection for knee OA. An LDA model was developed to identify these patients, with an accuracy of 67.8% (95%CI: 64.6%-60.9%), and an AUC of 0.6, suggesting a moderate ability to predict the outcome. This was reflected in the model’s low sensitivity of 29.6% (95%CI: 23.6%-35.6%). In contrast, the high specificity of 80.8% (95%CI: 77.2%-84.5%) suggests it was better at identifying patients who are unlikely to realize any benefits from the injection. Furthermore, the F1 score of 30.8% suggests a considerable imbalance between the model’s precision and recall ability.

While no other published studies have reported on the prediction of symptomatic improvement following intra-articular corticosteroid injections for knee OA, the application of ML has been discussed in a wide range of medical settings, some similar in their goals to our study[13,32-34]. For example, in 2019, Huber et al[35] compared multiple ML models in their ability to predict patient-reported outcomes in 30524 and 34110 patients undergoing hip and knee replacement surgery respectively. Huber et al[35] found the most successful model delivered an AUC score of 0.87, indicating a clinically acceptable ability to discriminate between positive and negative outcomes following knee replacement surgery. Another example is the retrospective study by Wang et al[36], who attempted to predict outcomes in 288 patients with cervical foraminal stenosis receiving epidural steroid injections for their radicular pain, using a deep learning algorithm that solely analyzed magnetic resonance imaging data. Their model demonstrated good efficacy, reporting an AUC score of 0.82. Both these studies reported higher AUC scores, which would be considered acceptable for clinical use, compared to our algorithm. A possible reason for our model’s lower performance relative to these models is the low number of entries in the dataset (330) and even fewer patients achieving clinically significant change in their WOMAC pain score (84), resulting in a small number of opportunities for the model to detect the already-challenging patterns[37]. The limited sample size likely led to overfitting, where the ML model memorizes patterns within the dataset too closely, and therefore fails to perform adequately on unseen data. This was exacerbated by the extensive variables included in the dataset which may have introduced irrelevant data, often termed as “noise”, hindering the LDA model from recognizing meaningful patterns, culminating in its restricted ability to discriminate between the study’s outcomes[37,38]. Permutation importance techniques are being increasingly utilized to determine the relevance of variables to the model’s predictive goal, enabling the selection of the most pertinent variables and subsequently improving the accuracy of their models[39]. Implementing these techniques in our model design could have helped exclude irrelevant variables from our dataset, thereby reducing noise and its impact on accuracy that follows. Ultimately, our model’s performance fell short of the clinically-acceptable AUC threshold of 0.80[40]. At present, it is not suitable for clinical use as a decision-making tool for selecting patients suitable for injections. Its moderate discriminatory ability may lead to a considerable amount of false-positive and false-negative outcomes. Further development of the model is required to enhance its discriminatory and predictive capabilities, aiming for a minimum AUC score of 0.9 to ensure exceptional accuracy.

Accepting these limitations, to the best of our knowledge, this study is the first to use ML to determine which knee OA patients will benefit from non-surgical management study, and has demonstrated the feasibility of this approach to assist in decision-making for patient selection regarding corticosteroid injection therapy for knee OA. This has the potential for clinical and economic significance, given that more than seven million corticosteroid injections are performed annually in the United States alone, with a traditional trial-and-error approach, rather than the personalized approach that would be facilitated by an improved version of our algorithm[12,41]. With our robust and reproductible methodology, this study provides a platform for future research to build on our findings, with the aim of developing a model that can accurately assist clinicians in decision making regarding the use of corticosteroid injections. It will also be crucial in incentivizing model development to examine various other management options for knee OA, such as hyaluronic acid and chondroitin sulphate supplements, both of which exhibit contentious efficacy[42].

Limitations

As discussed, the main limitation of our study is the small number of patients eligible for inclusion, and the even smaller number of those benefitting from a corticosteroid injection. The dataset may not have included the core variables that impact response to corticosteroids, further restricting the model’s overall performance. Moreover, neither MOST or OAI provided adequate information regarding when corticosteroid injections were administered during their respective study timelines. Given the minimum 12-month intervals between patient assessments, coupled with the short-term analgesic benefits of corticosteroid injections[9], some patients may have experienced a significant improvement in WOMAC score initially, but these effects may have not been sustained at follow-up. This introduces bias, limiting the validity of the datasets, and in turn hindering the predictive accuracy of our model. Incorporating a standardized approach, such as limiting injections to a maximum three doses per annum at minimum three-month intervals, as per current guidelines[43], combined with more frequent follow-up, would maximize analgesic effect, improve assessment of its effectiveness, and ultimately maximize model performance. Furthermore, incorporating objective assessments, such as monitoring patients’ ability to stand and walk at pre-specified intervals would provide a more comprehensive assessment of immediate and sustained therapeutic effectiveness.

CONCLUSION

This study represents the first attempt to predict which patients will benefit from intra-articular corticosteroid injections using a LDA model. The moderate discriminative ability of the model, secondary to the heterogeneity in data reporting, underscores a need for a larger and more refined training dataset. Albeit, despite the limitations, our model can serve as a valuable reference, paving the way for future research to build upon our findings.