Validation of the Rockall scoring system for outcomes from non-variceal upper gastrointestinal bleeding in a Canadian setting

Abstract

AIM: To validate the Rockall scoring system for predicting outcomes of rebleeding, and the need for a surgical procedure and death.

METHODS: We used data extracted from the Registry of Upper Gastrointestinal Bleeding and Endoscopy including information of 1869 patients with non-variceal upper gastrointestinal bleeding treated in Canadian hospitals. Risk scores were calculated and used to classify patients based on outcomes. For each outcome, we used χ² goodness-of-fit tests to assess the degree of calibration, and built receiver operating characteristic curves and calculated the area under the curve (AUC) to evaluate the discriminative ability of the scoring system.

RESULTS: For rebleeding, the χ² goodness-of-fit test indicated an acceptable fit for the model [χ² (8) = 12.83, P = 0.12]. For surgical procedures [χ² (8) = 5.3, P = 0.73] and death [χ² (8) = 3.78, P = 0.88], the tests showed solid correspondence between observed proportions and predicted probabilities. The AUC was 0.59 (95% CI: 0.55-0.62) for the outcome of rebleeding and 0.60 (95% CI: 0.54-0.67) for surgical procedures, representing a poor discriminative ability of the scoring system. For the outcome of death, the AUC was 0.73 (95% CI: 0.69-0.78), indicating an acceptable discriminative ability.

CONCLUSION: The Rockall scoring system provides an acceptable tool to predict death, but performs poorly for endpoints of rebleeding and surgical procedures.

Key Words: Upper gastrointestinal bleeding; Nonvariceal; Predictors; Rockall; Outcomes

INTRODUCTION

Upper gastrointestinal (UGI) bleeding is a common disorder affecting over 100 per 100 000 population yearly[1-7]. The most common etiologies include peptic ulcer disease, mucosal erosive disease and variceal bleeding[8-12]. Because there is an increasing concern for cost-containment without sacrificing clinical outcomes[13-15], there is room to implement emergent care for UGI bleeding with appropriate early discharge for subjects at low risk of rebleeding or death[16-20]. Although endoscopic findings can identify individuals at a high risk of rebleeding, overall mortality is often reflective of other factors such as age and comorbid conditions. In an effort to risk-stratify subjects with UGI bleeding, numerous scoring systems have been developed to predict bleeding recurrences, and the need for surgical procedures and death[17,20-28].

One instrument designed for that purpose is the Rockall scoring system[27,28]. The Rockall system has been shown to represent an accurate and valid predictor of rebleeding and death, performing better in the latter than in the former[27-29]. Rockall scores are designed to combine information such as the subject’s age, occurrence of shock assessed from systolic blood pressure readings and pulse rate, presence and severity of comorbid conditions, diagnosis and endoscopic stigmata of recent bleeding. Summing up the different levels of a point grading system assigned to each of the components yields a subject’s risk score bounded on a scale of 0 to 11, with 11 representing the highest risk. Results of previous investigations and validations of the scoring system have highlighted that those with a score of ≤ 2 are associated with a very low rate of bleeding recurrences and death and, therefore, can be reasonably managed as outpatients. This has the potential to result in a more appropriate management of subjects’ conditions based on their assessed risk of complications following the initial UGI bleeding. Further, managing low risk subjects as outpatients would free up scarce hospital resources for treating more serious cases.

Our objective was to validate the Rockall scoring system in the Canadian setting for the outcomes of rebleeding, the need for a surgical procedure and death, using data of 1869 patients with non-variceal UGI bleeding obtained from the registry of upper gastrointestinal bleeding and endoscopy (RUGBE)[30]. Additionally, we aimed to determine the mean length of hospital stay by levels of the Rockall score to compare current practice for subjects at a low risk of a serious event with an approach of managing their condition on an outpatient basis.

MATERIALS AND METHODS

The RUGBE initiative and data collection

A commercially available endoscopic reporting system (GI-Trac™, AD/MediTrac, Las Vegas, NV, USA) was linked to a project-specific patient registry. This software was distributed to 6 community and 12 university-affiliated health institutions across Canada, establishing a network, from which subjects were selected and source data collected. Research staff and monitors were trained at an initiation meeting and standardized definitions for all recorded variables were used. Information on all subjects was collected retrospectively from hospital records, denominalized and entered electronically in the reporting system. Data were then downloaded monthly (09/1999-12/2001) into the central repository and then, reviewed for internal logic and biological plausibility. All queries were resolved within one month of original data entry and 10% of all entries were audited quarterly for quality control.

Patient population

All subjects presenting with overt UGI bleeding or a history of hematemesis/coffee ground vomiting, melena, hematochezia, or a combination of any of the above within 24 h preceding admission were considered for the study. UGI bleeding was confirmed only if a member of the medical or nursing staff documented the presence of at least one of the following signs: (1) hematemesis; (2) melena; and/or, (3) bloody nasal gastric aspirates or black tarry material on rectal examination. Subjects were selected only if a UGI endoscopy was performed and a non-variceal source of bleeding was confirmed. A sequential time series sampling of eligible subjects was carried out at regular intervals to avoid a possible selection bias. An audit of all subjects presenting over a fixed time period at each institution was performed to further identify and prevent the possibility of a selection bias. The subset sampled constituted the entire dataset used in the study.

Study variables

Only the data for clinical and endoscopic variables necessary to build the Rockall risk scores and the outcome variables were extracted from the registry. Risk scores for each subject were calculated and used for risk stratification on the outcomes of rebleeding, the need for a surgical procedure and death. Standardized definitions for all outcomes were adopted according to adaptations of established definitions[31,32]. Continued bleeding following initial endoscopy was defined by the persistence of (a) spurting from an artery, (b) a bloody naso-gastric aspirate, (c) shock with a pulse greater than 100 beats per minute, a systolic blood pressure of below 100 mmHg, or both, or (d) the need for substantial replacement of blood and fluid volume (transfusion of greater than 3 units of blood within 4 h). Rebleeding was defined by recurrent vomiting of fresh blood, melena or both with either shock or a decrease in hemoglobin concentration of at least 2 g/L following initial successful treatment (modified from Daneshmend et al)[31,32]. Because the distinction is often blurred in practice, continued bleeding and rebleeding were subsequently combined within a single category termed ‘rebleeding’ for the purpose of this analysis.

Validation of the Rockall scoring system in the Canadian population

To validate the Rockall scoring system, we used χ² goodness-of-fit tests to assess the degree of calibration of each model (i.e. for outcomes of rebleeding, surgical procedures and death), and built receiver operating characteristic (ROC) curves based on a non-parametric technique as implemented in the statistical package STATA^® for Window^® for each outcome and calculated the area under the curve (AUC) along with 95% confidence intervals[33] to evaluate the discriminative ability of the scoring system. In our setting, a model has internal validity, or is well-calibrated, if it predicts the probability of experiencing an outcome that corresponds closely to the observed proportion of individuals with the outcome at each level of the Rockall risk score (i.e., from 0 to 11). The ROC curves plot the sensitivity of the Rockall score (true positive rate) versus 1-specificity (false positive rate) calculated for a series of different threshold values. The threshold values represent different levels of the Rockall scoring system, for which the rates of true positives (sensitivity) and true negatives (specificity) are calculated. The AUC is used to determine the ability of the scoring system to distinguish between individuals who experienced an outcome versus those who did not, over all possible threshold values. A test or risk scoring system with an AUC of 1 has a 100% sensitivity and 100% specificity, indicating that it would perfectly ‘discriminate’ between subjects experiencing the health event or not. A test with no better discriminative ability than what would otherwise be obtained by pure chance will have an AUC of 0.5, represented graphically by the area under a 45 degree line. The accepted statistical rule of thumb is that a test with an AUC of less than 0.7 has a poor discriminative ability; an AUC between 0.7 and 0.8 provides acceptable discrimination and a test with an AUC above 0.8 is considered to have an excellent discriminative ability[34].

Because the ROC curves are plotted over all possible threshold values, it is possible to identify the optimal risk score cut-off value, at which the test is most accurate. For each ROC curve, we identified the optimal threshold of the Rockall score by: (1) determining the pair of sensitivity and specificity associated with the point geometrically closest to the upper left corner of the graph; (2) calculating the Youden index[35] (i.e., J = sensitivity + specificity - 1) for each score level. The cut-off level associated with the highest J coefficient is the one that minimizes the sum of false negatives and false positives.

Mean lengths of hospital stay per level of the Rockall score were also evaluated. To test for significant differences in the distribution of length of hospital stay between risk score levels we used Kruskal-Wallis non-parametric analysis because of the usual skewedness observed in the distribution of that variable.

RESULTS

The population of 1869 subjects included in RUGBE had a mean age of 66 years [standard deviation (SD): 16.9, range: 7-105], and 62% were males. Fifty-six percent were diagnosed with peptic ulcer disease as the primary etiology for UGI bleeding (Table 1). The mean Rockall score was 4.8 (SD: 1.9, range: 0-10). Overall, 13% of subjects would be considered at a low risk (i.e., Rockall score ≤ 2) of experiencing rebleeding or death, while 8% of the population was classified as at a high risk (i.e., Rockall score ≥ 8). The distribution of subjects across levels of the Rockall score is reported in Table 2, as well as the rates of events for the three outcomes of rebleeding, surgical procedures and deaths, and the mean lengths of hospital stay. The results showed that the rates of events typically increased with higher risk levels expressed by the Rockall score. A cutoff score of 8 or greater for high risk persons was based on the same value used in the original analysis by Rockall[35]. Alternatively, in Table 2 we also show the same results for categories with a score of 2 or less for low risk, 3 to 5 for moderate risk and 6 or higher for high risk. The mean length of hospital stay also followed a similar trend with increasing levels of the risk scores. The distribution for the length of hospital stay was quite skewed as shown by the summary statistics on median and interquartile range. Results from the Kruskal-Wallis test confirmed this finding by showing a significant difference in the distribution of length of hospital stay between score levels [χ² (7) = 78.7, P = 0.0001]. Figure 1 provides the graphical representation of the trends for the three outcomes and the length of hospital stay.

Table 1 Endoscopic findings in the registry of upper gastrointestinal bleeding and endoscopy (RUGBE).

Peptic ulcer disease	55.5%
Esophagitis	8.2%
Mallory Weiss	4.4%
Dieulafoy	2.5%
Other	29.4%

Figure 1 Clinical outcomes by level of the Rockall risk scoring system.

Table 2 Observed outcomes of subjects by Rockall score (% of total within score level).

Rockall risk score	Distribution of subjects	Rebleeding	Surgical procedure	Deaths	bLength of hospital stay (d)
	n (%)	n (%)	n (%)	n (%)	Mean (SD)	Median (IQR)
≤ 2	240 (13)	21 (8.8)	6 (2.5)	0 (0)	3.6 (3.5)	2.9 (1.1-4.7)
3	205 (11)	18 (8.8)	5 (2.4)	3 (1.5)	4.4 (5.9)	3 (2-5.25)
4	359 (19)	49 (13.6)	11 (3.1)	11 (3.1)	5.7 (5.7)	4 (2.3-7)
5	435 (23)	63 (14.5)	17 (3.9)	20 (4.6)	5.9 (6.9)	4 (2.3-7)
6	290 (16)	31 (10.7)	12 (4.1)	24 (8.3)	6.7 (7.9)	4.5 (2.3-8)
7	195 (10)	39 (20)	15 (7.7)	18 (9.2)	6.6 (6.6)	4 (2.3-9)
≥ 8	145 (8)	37 (25.5)	9 (6.2)	24 (16.6)	7.4 (7.9)	5 (3-9)
Total	1869 (100)	258 (14)	75 (4.0)	100 (5.4)	5.7 (6.6)	4 (2-7)
Results for other risk score categories
≤ 2	240 (13)	21 (8.8)	6 (2.5)	0 (0)	3.6 (3.5)	2.9 (1.1-4.7)
3-5	999 (53)	130 (13)	33 (3.3)	34 (3.4)	5.6 (6.3)	4 (2-7)
≥ 6	630 (34)	107 (17)	36 (5.7)	66 (10.5)	7.2 (7.7)	5 (3-9)

11 (0.59%) and 28 (1.5%) values were missing for outcomes of surgical procedure and death, respectively; IQR: Interquartile range (25% centile-75% centile); ^bP = 0.0001, comparison between risk score levels in distribution of length of hospital stay [Kruskal-Wallis test: χ² (7) = 78.7].

Calibration of the Rockall scoring system

In Figure 2A we show the comparison of observed proportions and predicted probabilities for the outcome of rebleeding. For most levels of the Rockall score, the predicted probability was slightly lower than the observed proportion of events. The result of the χ² goodness-of-fit test indicated an acceptable fit for the model, although calibration could be improved to show better internal validity [χ² (8) = 12.83, P = 0.12]. Our findings from the corresponding analyses for the outcomes of surgical procedures and death (Figure 2B and 2C) showed a good fit for the models and thereby, good calibration as the measure of internal validity. For surgical procedures and death, the χ² goodness-of-fit test indicated solid correspondence between observed proportions and predicted probabilities [χ² (8) = 5.3, P = 0.73 for surgical procedures; χ² (8) = 3.78, P = 0.88 for death].

Figure 2 Predicted versus observed outcomes by Rockall risk core. A: Rebleeding; B: Surgical procedures; C: Death.

Overall, the predicted probabilities were closer in value to the observed proportions in our subject population for the outcomes of surgical procedures and death. The correspondence for rebleeding was acceptable, but not as strong as that for the two previous outcomes.

Discriminative ability of the Rockall scoring system

The ability of the Rockall scoring system to distinguish between individuals experiencing the events of rebleeding, surgical procedures and death (ROC curve) is illustrated in Figures 3-5, respectively. For rebleeding, the AUC was 0.59 (95% CI: 0.55-0.62) indicating a poor discriminative ability, or external validity of the Rockall scoring system. A similar result was found for the outcome of surgical procedures with an AUC of 0.60 (95% CI: 0.54-0.67). For the outcome of death, the AUC was higher at 0.73 (95% CI: 0.69-0.78), interpreted as an acceptable discriminative ability of the risk scoring system. The optimal cutoff Rockall scores were 6 for surgical procedures and death and 7 for rebleeding. This means that, at these threshold levels, the sum of false negatives and false positives is minimized or that the accuracy of the scoring system is highest.

Figure 3 Receiver operating characteristic (ROC) curve for outcome of rebleeding. Numbers along the curve indicate Rockall risk score cutoff values; The optimal threshold is a Rockall score of 7 (in bold).

Figure 4 Receiver operating characteristic (ROC) curve for outcome of surgical procedure. Numbers along the curve indicate Rockall risk score cutoff values; The optimal threshold is a Rockall score of 6 (in bold).

Figure 5 Receiver operating characteristic (ROC) curve for outcome of death. Numbers along the curve indicate Rockall risk score cutoff values; The optimal threshold is a Rockall score of 6 (in bold).

Overall, the internal and external validity of the Rockall risk scoring system was strongest for the outcome of death. With surgical procedures, the calibration achieved was high, but the scoring system had a poor discriminative ability. The level of calibration and discriminative ability were lowest for the outcome of rebleeding.

DISCUSSION

Several scoring systems have been developed to predict the clinical outcomes of gastrointestinal bleeding[17,21-25,27,28,35-40]. In 1987, Provenzale et al[28] tested various predictors of death from gastrointestinal bleeding and found that comorbid factors (i.e., liver and renal disease) and bleeding (i.e., hematochezia, short duration of bleeding, drop in hematocrit of 5% and hypotension) were the most valid. Subsequently, several other risk scoring systems have been developed, with some of them validated in different patient populations[30,37,40,41].

Risk scores have been most commonly used as an aid to clinical decision-making to identify subjects who can be efficiently managed as outpatients, rather than being unnecessarily admitted for a prolonged hospital stay. Blatchford[17] and Rockall[27] have developed such scoring systems to forecast: (1) subjects’ risk of rebleeding and death; (2) the need for early treatment of upper gastrointestinal bleeding. Although both scoring systems were designed for patients with UGI bleeding, the Blatchford scoring system does not incorporate information on endoscopic findings. This becomes an important limitation in circumstances where early endoscopic assessment is critical to optimal patient management. The Blatchford scoring system is still well-suited to the primary care setting when subjects need to be triaged to admission or outpatient management before an endoscopy is carried out.

When endoscopic information is available, the Rockall risk scoring system has been most widely applied to predict the risk of death and rebleeding. The system was originally developed to assess the risk of death, and its accuracy to forecast the risk of rebleeding has been shown to be relatively low in some validation studies[29]; however, in other studies the accuracy was relatively high. In two studies that assessed quality of care of a health care utilization[42-43], the risk of rebleeding appeared to correlate well with Rockall scores. In both studies there was considerable concern regarding excessive hospitalization of low risk Rockall patients since resources could be saved by early discharge.

Using data from a Canadian registry of subjects with non-variceal UGI bleeding, our objective was to test for the outcomes of rebleeding, surgical procedures and death: (1) the level of calibration of the Rockall scoring system as a measure of internal validity, and (2) the discriminative ability of the risk score for its generalizability to other populations. For that purpose, χ² goodness-of-fit tests for calibration and the area under the ROC curves for discriminative ability were used. Our results showed that the Rockall risk scoring system had acceptable performance for the outcome of death, but external validity and both internal and external validity were poor for surgical procedures and rebleeding, respectively. It is noteworthy that while subjects in the Vreeberg validation study[29] were from a different country (i.e. The Netherlands), the AUCs for the outcomes of death and rebleeding (0.73 and 0.61) resulted in almost identical numbers to ours (0.73 and 0.59). This adds weight to the conclusion that while acceptable to forecast the risk of death, the Rockall risk scoring system does not perform very well for the outcome of rebleeding. This study, performed in a Canadian setting, demonstrates that even with advanced endoscopic techniques, in a ‘real-life’ setting, the Rockall risk scoring system is acceptable for mortality prediction. Although the internal validity of the scoring system is high for the surgical procedures, its discriminative ability for this outcome is similar to that of rebleeding. We also found that the Rockall scoring system is in close agreement with length of hospital stay. This validation of the Rockall score is the first that has been done in a North American setting.

The Rockall risk scores are not widely used in Canada. However, it is clear that if endoscopic assessment could be expedient, a significant number of subjects (those with Rockall scores of ≤ 2) at very low risk of death or rebleeding, might be discharged earlier and managed as outpatients since their risk of mortality is low. Although we have not demonstrated validity with rebleeding, in the setting of low Rockall scores it is clear that mortality is low (and in other studies rebleeding as well). Even if rebleeding does not correlate well, the risk of death is extremely low with low Rockall scores and this would likely still support early discharge.

It is unclear why such a low percentage of patients have low Rockall scores. One possible explanation is that in the Canadian medical system, access to rapid hospital admission is relatively difficult and it is conceivable that some low risk patients would be managed as outpatients without coming through a hospital setting and therefore might not be recorded in the RUGBE database.

One the one hand, when these patients are admitted to hospitals, it seems unusual that they are not quickly discharged. One reason why subjects with low Rockall scores are not discharged quickly is that endoscopic assessment is often delayed. In RUGBE, 76% of patients had been investigated at 24 h[30]. The lengthy assessment period may contribute to the extended length of hospital stay.

On the other hand, subjects identified as high risk of death or rebleeding (Rockall score ≥ 8) may benefit from more intensive monitoring. This could be performed at an intensive care unit (ICU) or a ‘step-down’ unit rather than the usual medical/surgical wards. In Canada, the use of ICU beds for subjects with UGI bleeding has been shown to be less common than in some American centers[44]. In RUGBE, 22% of all patients were admitted to the ICU for investigation, therapy and monitoring[30] and even in the 145 patients with a Rockall score ≥ 8, only 52, or 36%, were sent to ICU. However, a large portion of them would have been classified as average risk and, therefore, unnecessarily monitored in the ICU. Other subjects with a high risk score were sent to medical/surgical wards instead of the ICU.

The strength of this study is in its ‘real-life’ evaluation of patients presenting to hospitals. The RUGBE database was a thorough one with internal validation that was collected retrospectively. This leads to the major weakness of the study, a retrospective evaluation. Although the RUGBE database was accurate, it was still retrospective with the associated weaknesses of a retrospective database. To device Rockall scores retrospectively will bring about occasional missing data and the inherent patient selection bias. Bias was limited in RUGBE (as best as possible) by having some sites receive virtually all of their non-variceal UGI bleeding patients over the specified time period.

This study has confirmed that the Rockall scoring system provides an acceptable tool to predict the risk of death, but performs poorly for endpoints of rebleeding and surgical procedures. Its cautious use for clinical decision-making purposes could still result in implementing more expedient care for low risk subjects, without sacrificing outcomes, and more efficient monitoring of high risk individuals.

ACKNOWLEDGMENTS

We wish to thank the The RUGBE investigators group, which includes: Alan Barkun, Carlo Fallone, and Gad Friedman, The McGill University Health Centre - the Montreal General and Royal Victoria Hospital sites, and the Sir Mortimer B Davis - Jewish General Hospital, Montréal, Québec; Raymond Lahaie, Georges Ghattas, and Judith Dorais, le centre hospitalier de l’université de Montréal, les pavillons hôpitaux St-Luc, Notre Dame, et Hôtel-Dieu, Montréal, Québec; Naoki Chiba, McMaster University, Hamilton, Ontario; David Armstrong and John Marshall, McMaster University & Hamilton Health Sciences, Hamilton, Ontario; Norman Marcon, St-Michael’s Hospital, Toronto, Ontario; Jonathon Love, the Queen Elizabeth II Health Sciences Centre, Halifax, Nova Scotia; Alan Cockeram, Saint John Regional Hospital, St John, New Brunswick; Franzjoseph Schweiger, Moncton Hospital, Moncton, New Brunswick; Jamie Gregor and John McDonald, London Health Sciences Centre, the University Hospital and Victoria Campuses, London, Ontario; Rob Enns, St Paul’s Hospital, Vancouver, British Columbia; Richard Fedorak, Bob Bailey, and Connie Switzer, the University of Alberta, Royal Alexander, and Grey Nuns Hospitals, Edmonton, Alberta.