Screening and evaluation of diabetic retinopathy via a deep learning network model: A prospective study

Abstract

BACKGROUND

Diabetic retinopathy (DR) is one of the most common serious complications in diabetic patients, and early screening and diagnosis are essential to prevent visual impairment. With the rapid development of deep learning technology, network models based on attention mechanisms have shown significant advantages in medical image analysis, which can improve the accuracy and efficiency of screening.

AIM

To evaluate the efficacy of an attention mechanism-based deep learning network model in screening for DR in natural and diabetic populations, as well as in screening with unilateral and bilateral fundus photography.

METHODS

From January 2023 to June 2024, a stratified multistage cluster sampling method was adopted to select a representative sample of permanent residents aged 18-70 years from our hospital. A total of 948 fundus images from 474 participants were included in the "deep learning model" system for scoring. The fundus images were graded via the early treatment of DR [Early Treatment Diabetic Retinopathy Study (ETDRS)] scoring system as the gold standard for the diagnosis of DR. With "DR to be referred (ETDRS > 31)" as the reference variable, a receiver operating characteristic curve was drawn to evaluate the area under the curve (AUC), sensitivity and specificity of the "deep learning model" to determine the screening efficiency of the system.

RESULTS

For each subject, in the natural population, the AUC of using the "deep learning model system" to screen "DR-requiring referral" was 0.941, and the sensitivity and specificity were 98.15% and 90.08%, respectively. The sensitivity and specificity of two-directional fundus photography were 100% and 86.91%, respectively. In the diabetic population, the AUC, sensitivity and specificity were 0.901, 98.08% and 82.10%, respectively, when "wise eye sugar net" unilateral fundus photography was used to screen for "DR-requiring referrals".

CONCLUSION

In both the natural population and the diabetic population, the deep learning model system has shown high sensitivity and specificity and can be used as an auxiliary means of DR screening.

Key Words: Diabetic retinopathy; Artificial intelligence; Stratified screening; Deep learning

Core Tip: Application value of a deep learning network model based on an attention mechanism in screening diabetic retinopathy (DR). By building and optimizing a deep learning model that incorporates attention mechanisms, we analyze its accuracy and efficiency in identifying features of DR and compare it with those of traditional methods. Research has focused on key performance indicators such as the sensitivity, specificity and accuracy of the model, as well as the contribution of attention mechanisms in the feature extraction process. Ultimately, the aim is to provide a more effective screening tool to improve the accuracy of clinical diagnosis and the possibility of early intervention.

INTRODUCTION

Diabetic retinopathy (DR) is a microvascular complication of diabetes that is currently the leading cause of blindness in adults[1-3]. In the Chinese population, the prevalence rates of DR, nonproliferative DR (NPDR) and proliferative retinopathy (NPDR) are 1.14%, 0.90% and 0.07%, respectively. In diabetic patients, the prevalence of DR reached 16.5%-48.3%. The International Diabetes Federation and the International Association of Ophthalmology believe that early screening and regular follow-up are the keys to the prevention and management of DR and recommend timely referral and treatment for patients with moderate NPDR and above[4-8].

In recent years, to promote the efficiency, accessibility and economic benefits of DR screening, artificial intelligence (AI) pattern recognition systems have emerged as auxiliary technologies for DR screening[9]. A deep learning approach is a type of AI technology that enables machines to learn features in images that can predict disease risk by identifying combinations of features layer by layer[10]. The deep learning model is used to identify DR and provide clinical diagnosis and treatment recommendations[11-13]. The screening efficacy of the system has not yet been proven in clinical practice.

Therefore, this study uses the Early Treatment Diabetic Retinopathy Study (ETDRS) score as the gold standard to evaluate the efficacy of the screening system for DR in both the natural population and the diabetic population. Moreover, the screening efficiency of single- and two-direction fundus photography was evaluated to provide suggestions for the clinical application of the system.

MATERIALS AND METHODS

Research subjects

From January 2023 to June 2024, a stratified multistage cluster sampling method was adopted to select permanent residents from our hospital. One to three days before the investigation, the residents' committee/village committee cadres and other relevant personnel accompanied the investigators, using standard language (local or national language) and investigation skills, making an appointment and signing an informed consent, and requiring all residents of the investigation unit (village or community) to participate in the investigation, if three appointments are unsuccessful, it is considered to have given up the investigation, and the survey objects need to be replaced when they give up the investigation.

Basic data collection

Demographic information, including name, sex, age and ethnicity, was collected via a questionnaire survey. Previous disease history and previous medication history. The physical examination results included the systolic blood pressure (SBP), diastolic blood pressure (DBP), heart rate, naked eye and corrected vision. The questionnaire survey was conducted by uniformly trained investigators face-to-face, and the participants completed the survey. Sampling recordings were made throughout the investigation, and the recording files of the subjects were saved separately.

Laboratory examination

All samples were stored at room temperature for no more than 4 h and then transported in the cold chain (2-8°C) to the central laboratory for fasting plasma glucose (FPG) and 2-hour-post glucose detection (2hPG) and glycated hemoglobin A1c (HbA1c). All testers receive unified training and can be tested only after passing the test. At the same time, there are quality control personnel on site to ensure that the test process and method are correct and that the record is accurate.

Fundus photography

Fundus images were collected via a Canon nondilatation-free digital fundus camera (model: CR-2 AF). Two fundus images (45° fundus image centered on the optic disc and 45° fundus image centered on the macula) were taken for each subject in each eye. If necessary, the compound topicamide was used for mydriasis. The images from the camera are transmitted to the computer terminal for storage in real time, and all the fundus photos are sent to our hospital for centralized reading, which is independently scored by two professional ophthalmologists, and the reading process is double blind.

When using AI to read. First, all 45° fundus photos centered on the optic disc were uploaded to the system for identification, and no clinical data from the study subjects were submitted to the AI terminal during this process. To further verify the reading efficiency of the deep learning model system, two directional fundus photos of the same eye were subsequently submitted to the "Smart Eye Sugar Net" system for reading.

Diagnostic criteria for diabetes

This includes newly diagnosed and already diagnosed diabetes patients. According to the 1999 World Health Organization standards, the diagnostic criteria for diabetes are FPG ≥ 7.0 mmol/L and/or 2hPG ≥ 11.1 mmol/L. Diabetes was diagnosed in people who self-reported a history of diabetes in the questionnaire.

DR diagnostic criteria

Two directional fundus image readings were taken, i.e., the same eye, and the "45° fundus image centered on the optic disc (Field 1)" and "45° fundus image centered on the macula (Field 2)" were evaluated simultaneously. ETDRS scoring is used as the gold standard. If the subject has DR and retinopathy grading results, reference is made to the eyes and fields with higher scores, and if one eye cannot be rated, the rating of the other eye is used. According to the recommendations of the International Diabetes Federation and Wisconsin study, "moderate NPDR and above (ETDRS > 31)" lesions were defined as referable DR (RDR) requiring referral, and "very mild NPDR and above (ETDRS ≥ 20)" were defined as different degrees of DR. "Severe NPDR and above (ETDRS > 51)" were defined as severe DR.

AI diagnostic criteria

DR was diagnosed when capillary hemangiomatoid swelling changes or more serious changes were found in fundus images (i.e., higher than the mild nonproliferative stage and the "deep learning model system" > 0). Diabetic macular edema was not included in the evaluation. When a unilateral fundus photograph is taken, the subject's rating is based on the eye with the highest score, and if one eye cannot be rated, the rating of the other eye is used. When reading with two-directional fundus photography, whether the subject has DR and DR grading results refers to the eyes and visual fields with higher scores.

Other definitions

Hypertension was defined as SBP ≥ 140 mmHg (1 mmHg = 0.133 kPa) or DBP ≥ 90 mmHg and a history of hypertension. Low vision is defined as the best corrected vision of less than 0.3 in both eyes. Abnormal visual acuity was defined as the best corrected visual acuity of the good eye being less than 1.0 in both eyes.

Statistical analysis

STATA/SE version 15.1 and MedCalc Statistical Software version 15.8 were used for data analysis. The missing values were replaced via the multiple interpolation method, and the extreme data distributed before 0.1% and after 99.9% were processed via the contraction method. The count data are expressed as frequencies/percentages, and the χ² test was used for comparisons between groups. The measurement data are expressed as mean ± SD values, and a t test was used for comparisons between groups. With the referable DR (RDR), different degrees of DR and severe DR as reference standards, receiver operating characteristic (ROC) curves were drawn, and indicators such as the area under the curve (AUC), sensitivity and specificity were calculated. The AUCs of different ROC curves were compared via the Hanley & McNeil method. P < 0.05 was considered to indicate statistical significance.

RESULTS

Comparison of general patient data

The 474 subjects were 50.04 ± 11.98 years old and included 164 males and 310 females. There were 279 patients (58.86%) with normal glucose metabolism, 117 patients (24.68%) with prediabetes and 78 patients (16.46%) with diabetes. Among the patients with diabetes, 331 (69.83%) were newly diagnosed with diabetes, and 143 (30.17%) had a history of diabetes. In the natural population, the prevalence of RDR was 0.60%, the prevalence of DR of different degrees was 3.9%, and the prevalence of severe DR was 0.04%. The prevalence of RDR was 3.86%, the prevalence of DR of different degrees was 12.46%, and the prevalence of severe DR was 0.22%. The prevalence of hypertension was 64.66% in the DR group and 45.15% in the non-DR group. Compared with those in the non-DR group, patients in the DR group were significantly older, and the levels of systolic blood pressure, DBP, FPG, 2hPG and HbA1c were significantly greater (P < 0.01, Table 1).

Table 1 Comparison of baseline data between non-diabetic retinopathy and diabetic retinopathy groups of study subjects.

Clinical indicators	Non-DR group (n = 255)	DR group (n = 219)	t/χ2 value	P value
Age (years, mean ± SD)	49.88 ± 12.05	54.21 ± 9.02	-6.62	< 0.01
Male [n (%)]	86 (33.73)	87 (39.73)	4.94	0.026
SBP (mmHg, mean ± SD)	132 ± 21	141 ± 23	-7.96	< 0.01
DBP (mmHg, mean ± SD)	81 ± 12	85 ± 13	-4.99	< 0.01
FPG (mmol/L, mean ± SD)	5.77 ± 1.35	7.70 ± 3.54	-23.56	< 0.01
2hpg (mmol/L, mean ± SD)	7.48 ± 3.43	11.52 ± 6.83	-20.30	< 0.01
HbA1c (%, mean ± SD)	5.57 ± 0.88	6.70 ± 2.02	-21.87	< 0.01
Hypertension [n (%)]	114 (44.71)	142 (64.84)	50.78	< 0.01

DR: Diabetic retinopathy; SBP: Systolic blood pressure; DBP: Diastolic blood pressure; FPG: Fasting plasma glucose; 2hpg: 2-hour-post glucose detection; HbA1c: Hemoglobin A1c.

Screening efficiency of deep learning model systems

The screening efficiency of single-fundus photographic readings of each eye: RDR, different degrees of DR, and severe DR were used as reference standards to draw ROC curves. In the natural population, the AUC of systematic screening for the RDR was 0.936, the sensitivity was 93.0%, and the specificity was 94.2%. The AUC of screening for different degrees of DR was 0.875, the sensitivity was 79.3%, and the specificity was 95.8% (Table 2).

Table 2 Diagnostic efficacy of artificial intelligence in screening diabetic retinopathy based on single direction fundus photography for each eye in natural population and diabetes population.

Different DR classifications	Natural population			People with diabetes
	AUC (95%CI)	Sensitivity (95%CI)	Specificity (95%CI)	AUC (95%CI)	Sensitivity (95%CI)	Specificity (95%CI)
RDR	0.936 (0.932-0.940)	93.0% (85.4%-97.4%)	94.2% (93.8%-94.6%)	0.911 (0.900-0.922)	94.0% (86.5%-98.0%)	88.3% (86.9%-89.5%)
Different degrees of DR	0.875 (0.870-0.880)	79.3% (75.3%-82.9%)	95.8% (95.4%-96.1%)	0.891 (0.878-0.903)	85.0% (79.9%-89.2%)	93.2% (92.1%-94.2%)
Severe DR	0.898 (0.893-0.902)	85.7% (42.1%-99.6%)	93.8% (93.4%-94.2%)	0.929 (0.918-0.938)	100.0% (47.8%-100.0%)	85.8% (84.3%-87.1%)

DR: Diabetic retinopathy; AUC: Area under the curve; RDR: Referable diabetic retinopathy.

In the diabetic population, the AUC of systematic screening for the RDR was 0.911, the sensitivity was 94.0%, and the specificity was 88.3%. The AUC of screening for different degrees of DR was 0.891, the sensitivity was 85.0%, and the specificity was 93.2%. The AUC for severe DR screening was 0.929, the sensitivity was 100.0%, and the specificity was 85.8% (Table 2).

Screening efficacy on the basis of single-fundus photographic readings of each subject: For each subject, the AUC, sensitivity and specificity of the system for screening the RDR in the natural population were 0.941, 98.2% and 90.1%, respectively. The AUC of screening for different degrees of DR was 0.881, the sensitivity was 83.7%, and the specificity was 92.5% (Table 3). In the diabetic population, the AUC, sensitivity and specificity of RDR screening were 0.901%, 98.1% and 82.1%, respectively. The AUC, sensitivity and specificity were 0.903, 91.6% and 89.0%, respectively. When screening for severe DR, the AUC was 0.896 (95%CI: 0.878-0.912), the sensitivity was 100.0%, and the specificity was 79.6% (Table 3).

Table 3 Diagnostic efficacy of artificial intelligence in screening diabetic retinopathy in natural population and diabetes population based on single orientation fundus photography of each subject.

Different DR classifications	Natural population			People with diabetes
	AUC (95%CI)	Sensitivity (95%CI)	Specificity (95%CI)	AUC (95%CI)	Sensitivity (95%CI)	Specificity (95%CI)
RDR	0.941 (0.936-0.946)	98.2% (90.1%-100.0%)	90.1% (89.4%-90.7%)	0.901 (0.884-0.916)	98.1% (89.7%-100.0%)	82.1% (79.9%-84.2%)
Different degrees of DR	0.881 (0.874-0.888)	83.7% (79.4%-87.4%)	92.5% (91.9%-93.1%)	0.903 (0.886-0.918)	91.6% (86.3%-95.3%)	89.0% (87.0%-90.7%)
Severe DR	0.948 (0.943-0.952)	100.0% (39.8%-100.0%)	89.6% (88.9%-90.2%)	0.896 (0.878-0.912)	100.0% (29.2%-100.0%)	79.6% (76.9%-81.3%)

DR: Diabetic retinopathy; AUC: Area under the curve; RDR: Referable diabetic retinopathy.

Screening efficacy on the basis of two-directional fundus photographic readings of each subject: The AUC for screening the RDR in the natural population was 0.935 (95%CI: 0.929-0.940), the sensitivity was 100% (95%CI: 93.4%-100.0%), and the specificity was 86.91% (95%CI: 86.2-87.6%). The AUC of screening for different degrees of DR was 0.899 (95%CI: 0.892-0.905), the sensitivity was 90.21% (95%CI: 86.5%-93.2%), and the specificity was 89.52% (95%CI: 88.8%-90.2%).

Subgroup analysis of hypertension, abnormal visual acuity and low visual acuity: A subgroup analysis was performed on the basis of whether the participants had hypertension. In people without hypertension, the AUC of systematic screening for the RDR was 0.965, which was greater than that in the general population and those with hypertension (P < 0.05). The sensitivity was 100.0%, and the specificity was 93.1% (Table 4). Vision loss due to fundus lesions often differs from refractive errors and cannot be corrected. Therefore, this study conducted a subgroup analysis of 4880 subjects who underwent visual acuity testing. The AUC of the system for screening DR in people with abnormal visual acuity was 0.923, which was lower than that in people with normal visual acuity, with a sensitivity of 93.8% and a specificity of 90.8% (Table 4).

Table 4 Diagnostic efficacy of artificial intelligence single directional fundus photography and image reading screening for diabetic retinopathy in different diabetic retinopathy classification populations.

Different DR classifications	AUC (95%CI)	Sensitivity (95%CI)	Specificity (95%CI)
RDR	0.941 (0.936-0.946)	98.2% (90.1%-100.0%)	90.1% (89.4%-90.7%)
RDR (non-hypertensive population)	0.965 (0.960-0.970)	100.0% (79.4%-100.0%)	93.1% (92.3%-93.8%)
RDR (hypertensive population)	0.920 (0.911-0.928)	97.4% (86.2%-99.9%)	86.6% (85.5%-87.6%)
RDR (normal vision population)	0.962 (0.952-0.969)	100.0% (78.2%-100.0%)	92.3% (91.1%-93.4%)
RDR (low vision population)	0.923 (0.912-0.933)	93.8% (69.8%-99.8%)	90.8% (89.7%-91.9%)
RDR (low vision group)	0.948 (0.908-0.975)	100.0% (29.2%-100.0%)	89.7% (84.5%-93.6%)
RDR (non-low vision group)	0.939 (0.932-0.946)	96.3% (81.0%-99.9%)	91.5% (90.6%-92.3%)

AUC: Area under the curve; RDR: Referable diabetic retinopathy.

DISCUSSION

Many newly diagnosed patients with type 2 diabetes are not screened for DR in a timely manner after diagnosis, with fewer than 50% being tested within 1 year of diagnosis and fewer than 60% being tested 3 times within 6 years of diagnosis[14-16]. Although screening for DR is included in universal health insurance, in developing countries, medical resources still do not meet the need for regular DR screening for all people with diabetes[17-20]. At present, the application of image classification, pattern recognition and machine learning technology has made DR screening intelligent. By learning tens of thousands of DR fundus images of different grades, the machine model based on AI deep learning can accurately identify whether the newly uploaded fundus images have DR changes[21-23]. As an auxiliary technology for DR screening, it can alleviate the current tight medical resources.

RDR is a key turning point in whether a patient with diabetes needs a referral[24]. In this study, the sensitivity of the system to screen each eye for RDR was greater than 93%. Compared with previous studies, the sensitivity of the deep learning model system in this study is similar to that of EyeArt, Bosch and other systems (EyeArt: 91.30%; Bosch: 91.18%), slightly lower than that of Google's AI system (96.87%), and higher than that of IDx's sensitivity (87.2%). Sensitivity is often a more critical indicator in screening, and the above conclusions suggest that the application of a deep learning model system can better identify RDR patients[25-27]. In addition, in this study, the system was able to accurately identify all patients with severe DR (100% sensitivity) with good reliability when the subjects were screened instead of each eye[28-30].

Although the specificity of the deep learning model system in screening for RDR was lower in the diabetic population than in the natural population (82.1% and 90.1%, respectively), the sensitivity was similar (98.2% and 98.1%, respectively)[31]. Therefore, screening for RDR only in patients with diabetes is recommended, and screening in the natural population is not necessary. Notably, the proportion of patients with hypertension in the diabetic population is greater than that in the natural population because hypertension may lead to misdiagnosis of non-DR fundus lesions by the deep learning model system, which may explain the decline in specificity[32]. However, retinopathy caused by hypertension is also a serious threat to patients' vision[33-35]. Therefore, we believe that even if screened retinopathy is not caused by diabetes but rather by high blood pressure or other fundus lesions, the results are still meaningful.

Currently, identifying mild lesions such as microhemangiomas is a challenge for machine learning[36-38]. In a natural population screening of subjects with "varying degrees of DR" via unilateral fundus photographic reading, the deep learning model system was less sensitive (83.7%). However, in this case, the deep learning model system uses only one-way fundus image reading. Since the results of the gold standard manual reading are based on two-directional fundus photo reading, we re-evaluated the accuracy of the deep learning model system's two-directional fundus photo reading, and the results showed that the sensitivity of the AI system in screening "different degrees of DR" exceeded 90%[39]. Even when screening for RDR, a two-directional fundus photographic reading can identify all patients with RDR. On the basis of the above discussion, we believe that it is better to use two-directional fundus photography for film reading during screening, but if the resources are limited, only unilateral fundus photography screening can yield a satisfactory RDR screening effect (sensitivity 98.2%).

For further algorithm optimization of the AI system for screening fundus lesions, this study suggests that adding algorithms for other fundus lesions, such as hypertensive fundus lesions, is very important[40-42]. In this study, the AUCAI of hypertensive people and corrected abnormal vision people was lower than that of normal people, suggesting that hypertensive fundus lesions affect the AI system's judgment of DR. In addition, reduced vision can affect the accuracy of the AI system in screening for DR[43]. Notably, in the low vision group (corrected vision < 0.3), the sensitivity of the AI system in screening for DR improved (100.0% and 96.3%, respectively), but its specificity was still lower than that of the nonlow vision group (89.8% and 91.5%, respectively)[44]. This may be because some other fundus lesions that cause vision loss (glaucoma, age-related macular degeneration, etc.) affect the accuracy of AI. In addition to the optimization of the algorithm, special emphasis is placed on the accurate collection of medical history and the necessary physical examination when screening primary care[45]. Not only can it help judge the screening results, but it can also be treated promptly if the patient has urgent hypertension or glaucoma symptoms.

CONCLUSION

In summary, the deep learning model system in this study has high sensitivity and specificity in screening the RDR in both the natural population and diabetic population and can be applied in clinical practice as an auxiliary means of DR screening to help shunt mild, moderate and severe DR patients. Moreover, to optimize the use of medical resources, screening for RDR only in the diabetic population and the use of unilateral fundus photographic readings are recommended.