External validation of EncephalApp Stroop test to screen minimal hepatic encephalopathy patients with nonalcoholic cirrhosis

Abstract

BACKGROUND

Neurocognitive impairment, including minimal hepatic encephalopathy (MHE) and overt hepatic encephalopathy, is one of the most common complications of all types of primary liver diseases, such as hepatitis B, biliary cholangitis, and autoimmune hepatitis. The EncephalApp Stroop test is a smartphone application-based test that is time-saving for MHE screening. However, neurocognitive impairment is different between alcoholic cirrhosis patients and nonalcoholic cirrhosis patients, so the cutoff value for MHE diagnosis might be inflated.

AIM

To validate the Stroop test in nonalcoholic cirrhosis patients.

METHODS

This external validation was performed at the National Center for Infectious Diseases (Beijing). Liver cirrhosis patients aged between 18 and 65 years who voluntarily enrolled in the study and provided signed informed consent were included. The Psychometric Hepatic Encephalopathy Score (PHES) test was used as the standard diagnostic criterion for MHE. The EncephalApp Stroop test was then performed on the iPad, including two sessions of tests (“off” and “on”) to measure patients’ ability to differentiate between numbers and letters. We assessed the performance of the EncephalApp Stroop test in terms of the area under the curve (AUC), sensitivity, specificity, positive predictive value, and negative predictive value, with the PHES as the standard criterion.

RESULTS

A total of 160 nonalcoholic cirrhosis patients were included in this validation study, including 87 (54.4%) patients without MHE and 73 (45.6%) patients with MHE. Taking the PHES as the gold standard, the EncephalApp Stroop test performed well for nonalcoholic liver cirrhosis patients in terms of “off” time [AUC: 0.85, 95% confidence interval (CI): 0.79-0.91] and “on + off” time (AUC: 0.85, 95%CI: 0.80-0.91); however, total runs of “off” session (AUC: 0.61, 95%CI: 0.52-0.69), total runs of “on” session (AUC: 0.57, 95%CI: 0.48-0.65), and “on – off” time (AUC: 0.54, 95%CI: 0.44-0.63) were comparatively low. The optimal cutoff points were “off” time > 101.93 seconds and “on + off” time > 205.86 seconds, with sensitivities of 0.84 and 0.90, specificities of 0.77 and 0.71, positive predictive values of 0.75 and 0.72, and false-positive values of 0.85 and 0.89, respectively.

CONCLUSION

Our results suggest that different cutoffs should be used for the EncephalApp Stroop tool for MHE screening between alcoholic and nonalcoholic living patients, which is a critical check before generalization to screen for neurocognitive impairment among the whole population of chronic liver diseases.

Key Words: Minimal hepatic encephalopathy; Nonalcoholic cirrhosis; EncephalApp Stroop test; Diagnosis; Screening

Core Tip: This study validated the EncephalApp Stroop test on screening minimal hepatic encephalopathy patients with nonalcoholic cirrhosis. The results showed that EncephalApp Stroop test was time-saving with good predictive performance on the validation dataset. This study concluded that we should use different cutoff value of EncephalApp Stroop tool on minimal hepatic encephalopathy screening between alcoholic and nonalcoholic patients, before widespread application of EncephalApp Stroop test in management of chronic liver diseases.

INTRODUCTION

Neurocognitive impairment in patients with chronic liver cirrhosis, including minimal hepatic encephalopathy (MHE) and overt hepatic encephalopathy, is one of the most common complications of all types of primary liver diseases, such as hepatitis B, biliary cholangitis, and autoimmune hepatitis, through a wide spectrum of stages of chronic liver diseases ranging from liver fibrosis and cirrhosis to cancer[1]. MHE is estimated to affect up to 60% of cirrhosis patients and adversely affects quality of life, driving ability, and job performance[2]. More importantly, MHE is associated with a greater risk of vehicle accidents, disease progression, hospitalization, and disease-related death[3-5]. MHE treatment is effective at improving quality of life and reducing mortality; therefore, early detection and treatment are highly recommended by the American/European Association for the Study of Liver Disease, including screening via neuropsychological tests such as the Psychometric Hepatic Encephalopathy Score (PHES)[6]. However, the PHES is a paper-pencil-based test that is time-consuming and biased by the patient’s educational background[7]. In contrast, the EncephalApp Stroop test is a smartphone application-based test that is timesaving for MHE screening[8]. However, neurocognitive impairment is different between alcoholic cirrhosis patients and nonalcoholic cirrhosis patients[9], so the cutoff value for MHE diagnosis might be inflated. The external validation of the EncephalApp Stroop test in nonalcoholic cirrhosis patients is crucial for ensuring the sensitivity and specificity of MHE diagnosis; however, this has not yet been reported. Thus, our study objective was to validate the Stroop test in patients with nonalcoholic cirrhosis.

MATERIALS AND METHODS

Study settings

This external validation was performed at both outpatient clinics and inpatient departments at Beijing Ditan Hospital of Capital Medical University, the National Center for Infectious Diseases (Beijing), prospectively from January 2022 to July 2023. The EncephalApp Stroop test was used to screen patients with nonalcoholic liver cirrhosis compared with the PHES. The study was reviewed and approved by the Ethics Committee of Beijing Ditan Hospital, and all participants provided informed consent. We reported this study according to the Strengthening the Reporting of Observational Studies in Epidemiology statement.

Patient inclusion and exclusion criteria

Patients were included in this validation study if they met all the following criteria at screening: (1) Having liver cirrhosis diagnosed by computed tomography scan or magnetic resonance imaging, confirmed by ultrasound elastography, or diagnosed by liver biopsy; (2) Aged between 18 and 65 years; and (3) Voluntarily signed informed consent. Patients were excluded from this validation study if they met any one of the following criteria at screening: (1) Had alcoholic liver diseases, diagnosed according to the guidelines of the American Association for the Study of Liver Diseases using the Diagnostic Criteria for Alcohol Use Disorders[10]; (2) Had other diseases that cause cognitive dysfunction, such as Parkinson’s disease[11] or Alzheimer’s disease[12]; (3) Were unable to finish the EncephalApp Stroop test; or (4) Were suffering from fever or were using lactulose, sedative drugs, or psychotropic drugs at screening.

Diagnosis of MHE via the PHES test

The PHES test, which has been thoroughly validated, was used as the gold standard for the diagnosis of MHE in this validation study. This PHES test was a paper-pencil-based test, including the following 5 sessions: (1) Number connection test-A: This test involves placing numbers in an arithmetic order from 1-25, time was measured in seconds; (2) Number connection test-B: This test involves linking numbers and letters in an alternating (1-A, 2-B, 3-C) pattern, time was measured in seconds; (3) Digit symbol test: This test involves linking symbols to corresponding numbers as quickly as possible; the time required to complete the test was 90 seconds, and the results are expressed in points; (4) Line tracing test (LTT): This test involves drawing a dash between 2 lines 5 mm apart without touching the borders and to do so as fast as possible; LTT results are expressed in terms of the time spent completing the test (LTTt seconds) and the error score (LTTe), and the calculation is LTT = (1 + LTTe/100) × LTTt; and (5) Serial dotting test: This test involves making dots at the center of 10 circles as fast as possible from left to right; the time was measured in seconds.

EncephalApp Stroop test procedure

The EncephalApp Stroop test was performed on an iPad in this validation study, which included two sessions-off” and “on” to test whether patients could tell the number and letters: (1) The “off” session involved asking patients to touch the color matched to the stimulus at the bottom of the iPad screen until 10 presentations were completed correctly. The color stimulus is randomly displayed. The results were recorded as time to finish and mistakes made during this session; and (2) In the “on” session, patients were asked to touch the corrected color in different words until 5 words were completed correctly. For example, when the text for the term “BLUE” is red, the patient should choose red instead of blue. The results were recorded as time to finish and mistakes made in this session. Finally, there were 6 parameters generated from this test, including on time (time to finish five correct colors), off time (time to finish five correct colors), off time + on time, on time - off time, run time for the on session (total number of runs to finish on session, including both correct runs and incorrect runs), and run time for the off session (total number of runs to finish off session, including both correct runs and incorrect runs).

Statistical analysis

Diagnostic test sample size estimation formula: N = U2α2 × P(1-P) δ2, (U2α2 = 1.96), P(sensitivity) = 0.9, P(specificity) = 0.8, δ(error) = 0.08. N(se): 54, N(sp): 96%. According to previous reports in the literature 12, the prevalence (prevalence) is calculated as 40% as follows: N(sp) = N(sp)/(1-prevalence) = 96/0.6 = 160. Sensitivity: N(se) = N(se)/prevalence = 54/0.4 = 135. The population characteristics of nonalcoholic liver cirrhosis patients without MHE and nonalcoholic liver cirrhosis patients with MHE were collected and assessed if the data were well balanced. We assessed the test performance via the area under the curve (AUC), sensitivity, specificity, positive predictive value, and negative predictive value, taking the PHES as the gold standard for diagnosis. The model for end-stage liver disease was also calculated using the serum bilirubin level, international normalized ratio, and serum creatinine level. Analyses were performed via R version 4.3.3 (R Foundation for Statistical Computing, Vienna, Austria), with bilateral P < 0.05 indicating statistical significance.

RESULTS

Patient characteristics

A total of 465 potential patients were screened, and ultimately 160 nonalcoholic cirrhosis patients were included in this validation study, including 87 (54.4%) patients without MHE and 73 (45.6%) patients with MHE. The patient selection flow diagram is shown in Figure 1. The baseline characteristics of the included participants are listed in Table 1. The MHE and non-MHE groups had similar characteristics, except that MHE patients were older than non-MHE patients (mean ± SD age, 56.7 ± 10.1 years vs 58.7 ± 11.0 years), the proportion of patients with Child-Pugh Grade B or C disease was greater for MHE patients [39 (53.4%) vs 24 (27.6%)], and the hemoglobin level in the MHE group was lower than that in the non-MHE group (mean ± SD hemoglobin, 113.3 ± 25.0 g/L vs 122.8 ± 25.8 g/L).

Figure 1 Flow diagram of patient selection.

Table 1 Description of the characteristics of patients in the validation data set and distinguished by minimal hepatic encephalopathy diagnosis, n (%).

Characteristics	All patients (n = 160)	Without MHE (n = 87)	With MHE (n = 73)
Age, mean ± SD, year1	52.3 ± 11.3	48.7 ± 11.0	56.7 ± 10.1
Sex
Female	52 (32.5)	28 (32.2)	24 (32.9)
Male	108 (67.5)	59 (67.8)	49 (67.1)
Primary liver disease
Hepatitis B	123 (76.8)	71 (81.6)	52 (71.2)
Hepatitis C	14 (8.8)	6 (6.9)	8 (11.0)
Autoimmune hepatitis	11 (6.9)	4 (4.6)	7 (9.6)
Others	12 (7.5)	6 (6.9)	6 (8.2)
Child-Pugh Score1
Grade A	97 (60.6)	63 (72.5)	34 (46.6)
Grade B	51 (31.9)	23 (26.4)	28 (38.4)
Grade C	12 (7.5)	1 (1.2)	11 (15.0)
MELD	11.4 (4.0)	10.9 (3.7)	12.0 (4.3)
ALT, U/L	46.2 (107.7)	49.8 (94.0)	42.0 (122.6)
AST, U/L	55.7 (130.3)	52.8 (75.9)	59.2 (175.0)
ALB, g/L	38.1 (29.5)	40.6 (39.7)	35.1 (5.0)
INR	1.3 (0.2)	1.3 (0.2)	1.4 (0.3)
K, μmol/L	3.9 (0.4)	3.8 (0.3)	3.9 (0.4)
Na, μmol/L	140.4 (3.4)	140.5 (2.9)	140.2 (3.8)
Cr, mL/min	69.4 (21.0)	67.1 (15.3)	72.1 (26.0)
WBC, 109/L	3.8 (1.9)	4.0 (2.1)	3.6 (1.6)
HGB, g/L1	118.5 (25.8)	122.8 (25.8)	113.3 (25.0)
PLT, 109/L	98.1 (82.0)	109.8 (99.2)	84.0 (52.1)

¹Significant differences were observed between minimal hepatic encephalopathy patients and non-minimal hepatic encephalopathy patients.

MHE: Minimal hepatic encephalopathy; MELD: Model for end-stage liver disease; ALT: Alanine transaminase; AST: Aspartate transferase; INR: International normalized ratio; ALB: Albumin; Cr: Creatinine; WBC: White blood cell; HGB: Hemoglobin; PLT: Platelet.

Validation of the EncephalApp Stroop test

We used the PHES as the gold standard, and in comparison, the EncephalApp Stroop test performed well for nonalcoholic liver cirrhosis patients in terms of the parameters of off time [AUC: 0.85, 95% confidence interval (CI): 0.79-0.91] and on + off time (AUC: 0.85, 95%CI: 0.80-0.91); however, the total number of runs off session (AUC: 0.61, 95%CI: 0.52-0.69), total number of runs on session (AUC: 0.57, 95%CI: 0.48-0.65), and on - off time (AUC: 0.54, 95%CI: 0.44-0.63) were comparatively low, as shown in Table 2. The optimal cutoff points were off time > 101.93 seconds and on + off time > 205.86 seconds, with sensitivities of 0.84 and 0.90, specificities of 0.77 and 0.71, positive predictive values of 0.75 and 0.72, and false-positive values of 0.85 and 0.89, respectively, as shown in Figure 2.

Figure 2 Receiver operating characteristic curve of the EncephalApp Stroop test in patients with nonalcoholic cirrhosis. A: Receiver operating characteristic (ROC) curve for off time; B: ROC curve for on time; C: ROC curve for off time + on time. ROC: Receiver operating characteristic.

Table 2 Performance of the EncephalApp Stroop test in the diagnosis of minimal hepatic encephalopathy in nonalcoholic liver cirrhotic patients.

Parameters	AUC (95%CI)	Optimal cutoff	Sensitivity	Specificity	Precision, PPV	NPV
Off time, second	0.85 (0.79-0.91)	101.93	0.84	0.77	0.75	0.85
On time, second	0.84 (0.78-0.90)	104.51	0.89	0.71	0.73	0.90
On + off time, second	0.85 (0.80-0.91)	205.86	0.90	0.71	0.72	0.89
On - off time, second	0.54 (0.44-0.63)	14.95	0.45	0.71	0.57	0.61
Total runs of off session	0.61 (0.52-0.69)	6.50	0.38	0.83	0.65	0.62
Total runs of on session	0.57 (0.48-0.65)	5.50	0.66	0.48	0.52	0.63

AUC: Area under the curve; CI: Confidence interval; PPV: Positive predictive value; NPV: Negative predictive value.

DISCUSSION

This study revealed that, compared with the standard criteria of the PHES, the Stroop-EncephalApp test generally has diagnostic value. The EncephalApp time was significantly different between patients with MHE and those without MHE. More specifically, the on + off time had the best diagnostic efficacy among all the parameters. The cutoff > 205.86 seconds of on + off time in all patients with cirrhosis is similar to that of the Stroop-CN model (204.992 seconds)[13], which was higher than the cutoff found in a previous study of a Chinese population[14]; a possible reason for the difference is that we included a larger proportion of patients with a Child-Pugh score of B. Several studies have shown that early diagnosis of MHE is very important for improving patients’ quality of life and long-term prognosis[15-17]. However, neurocognitive tests are not routinely administered in clinical practice[18]. Stroop[19], an American psychologist, published an article about the classic Stroop color-word test because humans name colors more slowly than words, causing interference with each other. The Stroop interference effect was initially revealed in the brain and has been used to screen patients with early impairment of neurocognitive function. With respect to MHE, cognitive impairment and decreased reaction speed are the main clinical manifestations. Therefore, Professor Jasmohan S Bajaj has applied the Stroop EncephalApp to screen and diagnose MHE in the United States in recent years. Stroop EncephalApp can be easily downloaded and used on mobile phones and iPads so that it is convenient and can be accessed easily. After the tester gives instructions to the patient, patients can complete the Stroop EncephalApp test by themselves[20]. The test takes only a few minutes, largely reducing the workload of clinicians and controlling the bias caused by the testers’ performance influence on patients. Moreover, our study also revealed that educational background scarcely affected the results of the Stroop EncephalApp test. Some patients with liver cirrhosis often rapidly develop encephalopathy after infection and fever[21]. Previous studies have shown that lactulose can affect the understanding of MHE[22,23], and psychotropic medications usually have side effects such as hallucinations and decreased reaction times; therefore, we excluded patients with fever and psychotropic medication. One published study investigated the associations between covert hepatic encephalopathy and related factors, such as age, sex, educational level, model of end-stage liver disease score, previous history of overt hepatic encephalopathy, and previous use of a smartphone and computer diagnosed with the EncephalApp Stroop test, and reported that a history of alcoholic hepatitis was positively correlated with the risk of diagnosis of covert hepatic encephalopathy with the EncephalApp Stroop test[14] and that alcohol abuse might affect cognitive function and the results of neuropsychological tests[24-27]. In addition, our study validated the Stroop-EncephalApp test for nonalcoholic cirrhosis in a Chinese population. This study has several limitations. First, this was a single-center study, and the sample size involved was limited because of the prevalence of MHE in the Chinese population. We expect that multicenter and large-sample validation studies will be conducted in the future. Second, we did not use an external dataset to cross-validate the results, and we will recruit external MHE patients to further confirm our conclusions.

CONCLUSION

In conclusion, the results of this study support the need for different cutoffs for the EncephalApp Stroop tool for MHE screening between alcoholic and nonalcoholic patients, which is critical before the tool can be generalized to the daily care of patients with chronic liver diseases. The EncephalApp Stroop test is timesaving and has good predictive performance on the validation dataset. Nevertheless, this validation study was limited by the use of single-center data in China and a lack of racial diversity. Further evaluation across countries and ethnicities is needed.

ACKNOWLEDGEMENTS

We would like to thank the participants in this study, Prof Yu-Yong Jiang (Beijing Ditan Hospital, Capital Medical University) and Prof. Hong-Bo Du (Dongzhimen Hospital, Beijing University of Chinese Medicine) for his suggestions regarding the clinical experience of minimal hepatic encephalopathy diagnosis.