Inter- and intra-rater reliability of diffusion tensor imaging parameters in the normal pediatric spinal cord

Abstract

AIM: To assess inter- and intra-rater reliability (agreement) between two region of interest (ROI) methods in pediatric spinal cord diffusion tensor imaging (DTI).

METHODS: Inner-Field-of-View DTI data previously acquired from ten pediatric healthy subjects (mean age = 12.10 years) was used to assess for reliability. ROIs were drawn by two neuroradiologists on each subject data twice within a 3-mo interval. ROIs were placed on axial B₀ maps along the cervical spine using free-hand and fixed-size ROIs. Agreement analyses for fractional anisotropy (FA), axial diffusivity, radial diffusivity and mean diffusivity were performed using intra-class-correlation (ICC) and Cronbach’s alpha statistical methods.

RESULTS: Inter- and intra-rater agreement between the two ROI methods showed moderate (ICC = 0.5) to strong (ICC = 0.84). There were significant differences between raters in the number of pixels selected using free-hand ROIs (P < 0.05). However, no significant differences were observed in DTI parameter values. FA showed highest variability in ICC values (0.10-0.87). Cronbach’s alpha showed moderate-high values for raters and ROI methods.

CONCLUSION: The study showed that high reproducibility in spinal cord DTI can be achieved, and demonstrated the importance of setting detailed methodology for post-processing DTI data, specifically the placement of ROIs.

Key Words: Diffusion tensor imaging; Reproducibility; Reliability; Spinal cord; Inter-rater; Intra-rater

Core tip: We tested the reliability of spinal cord diffusion tensor imaging (DTI) by assessing inter- and intra-rater agreement between two region of interest (ROI) selection methods. Results showed moderate to strong agreement between repeated measurements. There was a variation in DTI parameters at lower and upper spinal cord levels and significant differences between raters in the number of pixels they chose to outline ROIs. There were no significant differences in DTI parameter values derived from these ROIs. The study showed that strong reproducibility in spinal cord DTI can be achieved, and highlighted the importance of setting detailed methodology to standardize ROI drawing techniques.

INTRODUCTION

Diffusion tensor imaging (DTI) has become an important technique for evaluating the central nervous system noninvasively. The application of DTI in the spinal cord is technically limited by the small cross-sectional size of the spinal cord, cerebral spinal fluid pulsation (CSF), the presence of nearby vascular and osseous structures, respiratory and cardiac motion.

Despite these challenges, studies have successfully reported diffusion measures in the intact human spinal cord[1-9]. Several studies have investigated the clinical utility of DTI in assessing spinal cord pathology and have shown correlations between DTI and clinical scores[10-18]. While these studies used different data acquisition methods and post-processing software, they all shared one common technique to delineate the spinal cord: Manual drawing of regions of interests (ROIs) to calculate DTI parameters. Additionally, they all reported that ROI locations were confirmed using the conventional structural MR imaging sequences, and that care was taken to avoid inclusion of CSF. Interestingly, most of these studies reported that the variability seen in DTI parameters could be due to the user-dependent manual selection of ROIs.

Parameters derived from DTI can provide information about tissue properties which may have clinical significance. To be proven clinically reliable, the quantitative properties of spinal cord DTI need to be assessed by multiple reviewers. While extensive data is available on the reliability and reproducibility of brain DTI[19-24], to the best of our knowledge there is lack of research on the reliability of ROI placement to quantify spinal cord DTI measures. The goal of this study was to assess inter-rater (between) and intra-rater (within) agreement in pediatric spinal cord DTI, as well as agreement between two methods of drawing regions of interest.

MATERIALS AND METHODS

Subjects

This is a retrospective study. Ten pediatric subjects with a mean age of 12.10 years (age range 9 to 15) underwent cervical spinal cord DTI[25]. The subjects had no clinical or imaging evidence of spinal cord injury or pathology. Subjects and their parents provided written informed assent and consent of the IRB-approved protocol.

Imaging

The scans were performed using a 3T Siemens Verio MR scanner with a four-channel neck matrix and an eight-channel spine matrix coils. The protocol consisted of conventional sagittal Turbo Spin Echo T1- and T2-weighted scans, axial TSE T2 weighted scans as well as axial DTI acquisition with an inner Field-of-View single-shot EPI sequence with spatially 2D-selective RF excitations[26,27]. Anesthesia was not administered to the subjects. Neither cardiac nor respiratory gating was performed. The axial DTI images were acquired in the same anatomical location prescribed for the T2-weighted images to cover the cervical spinal cord (C1 to T1 levels). DTI scanning parameters included: 20 diffusion directions, b = 1000 s/mm², voxel size = 1.2 mm³× 1.2 mm³× 3 mm³, axial slices = 35-45 (depending on the subject’s height), TR = 6100-8000 ms, TE = 115 ms, number of averages = 3 and acquisition time = 7 min.

Image processing

Initially, the diffusion data sets were corrected for motion-induced artifacts using the Automated-Image-Registration where the target images (20 directional images) were aligned with the reference image (B₀) using a rigid registration algorithm and scaled-least-squares cost function[28]. Tensor estimation and placement of ROIs were performed on MedINRIA. Fractional anisotropy (FA), axial diffusivity (AD), radial diffusivity (RD) and mean diffusivity (MD) measures were calculated.

Placement of regions of interest

The methodology for ROI placement was devised by two board-certified neuroradiologists. ROIs were manually drawn on axial b₀ maps along the cervical spinal cord at each cervical intervertebral disk and mid vertebral body level. Two methods of selecting ROIs were examined in this study: (1) free-hand: Where the raters manually outlined the periphery of spinal cord tissue with consistent sparring of the outer margin of the spinal cord that represented approximately one voxel width to minimize volume averaging with the CSF, and (3) pre-set: where raters placed a rectangular 3 by 2 pixels region of interest in the center of the spinal cord (Figure 1).

Figure 1 B₀ images show how each rater outlined manual and fixed size regions of interest.

To asses for inter-rater variability, ROIs were drawn by two board-certified neuroradiologists (rater 1 and rater 2) who received training on using MedINRIA. To assess intra-rater variability, each rater performed ROI analysis on each subject data twice. The two raters worked independently and were blinded to each other’s results and were proficient in DTI post-processing. Data generated from the first set of ROIs was not available during the second assessment. Both neuroradiologists were asked to repeat the measurements after 3 mo.

Data analysis

For each ROI, DTI values were extracted from each voxel. The values of each DTI parameter (FA, AD, RD and MD) were averaged per cord level across all subjects. ROI selection was guided by axial TSE T2 and reconstructed sagittal and coronal b₀ maps.

Statistical analysis

Statistical analysis was performed using SAS, version 9.1, to test the inter- and intra-rater agreement between DTI data from the two methods of ROI drawing. This was assessed by (1) calculating the intra-class-correlation (ICC) coefficients[29,30] with 95%CI for each DTI parameter per spinal cord level and (2) calculating Cronbach’s alpha as a measure of test re-test for each rater. Descriptive statistics were calculated for FA, AD, RD, MD, and test re-test differences were compared using paired t tests, with P value ≤ 0.05 considered statistically significant.

RESULTS

Using B₀ images, ROIs were drawn manually and by placing a pre-set outline (Figure 1). Descriptive data for ROI drawing methods, raters and trials of spinal cord FA, AD, RD and MD are shown in Table 1. Overall, there were significant differences between raters in the number of pixels they chose to outline free-hand ROIs (P < 0.05). However, there were no significant differences in average inter- or intra-rater DTI parameter values. When two ROI drawing methods were compared to each other, significant differences were found in DTI parameters. Paired t tests were run for spinal cord levels C1 to T1. Data in Table 1 shows mean inter-observer and intra-observer DTI parameter values.

Table 1 Mean SD inter-rater and intra-rater values for fractional anisotropy, axial diffusivity (× 10^-3 mm²/s), radial diffusivity (×10^-3 mm²/s), and mean diffusivity (× 10^-3 mm²/s), for manually drawn regions of interest and fixed size regions of interest.

			Pixels	FA	AD	RD	MD
Freehand ROI	Rater 1	Trial 1	24	0.50 ± 0.13	1.35 ± 0.52	0.70 ± 0.54	0.92 ± 0.53
		Trial 2	24	0.47 ± 0.13	1.16 ± 0.18	0.52 ± 0.17	0.73 ± 0.18
	Rater 2	Trial 1	17	0.48 ± 0.12	1.53 ± 0.65	0.88 ± 0.64	1.10 ± 0.65
		Trial 2	14	0.52 ± 0.10	1.20 ± 0.16	0.48 ± 0.13	0.72 ± 0.14
Fixed ROI	Rater 1	Trial 1	6	0.54 ± 0.10	1.18 ± 0.19	0.46 ± 0.14	0.70 ± 0.16
		Trial 2	6	0.54 ± 0.10	1.18 ± 0.16	0.47 ± 0.13	0.71 ± 0.14
	Rater 2	Trial 1	6	0.53 ± 0.10	1.64 ± 0.58	0.91 ± 0.56	1.16 ± 0.56
		Trial 2	6	0.54 ± 0.10	1.19 ± 0.15	0.45 ± 0.11	0.70 ± 0.12

ROI: Region of interest; FA: Fractional anisotropy; AD: Axial diffusivity; RD: Radial diffusivity; MD: Mean diffusivity.

Overall agreement

The ICC coefficients for each DTI parameter as a function of spinal cord level are shown in Figure 2. The agreement was moderate to high among raters and ROI drawing methods. The ROI methods showed moderate (ICC = 0.5) to strong (ICC = 0.84) inter-rater and intra-rater agreement. Of the DTI parameters, FA showed the highest variability of ICC values (0.10-0.87). Low agreement was found in upper spinal cord levels C1 to mid-C3. RD showed slightly higher agreement values than FA (0.26-0.83). The agreement was lower in the pre-set ROIs for FA and RD. Figure 3 shows the Cronbach’s Alpha values evaluating test-retest agreement for each rater. There was a decrease in agreement at upper and lower spinal cord levels, especially in FA values. There was low and fluctuating agreement values in RD for rater 2 when placing pre-fixed size ROIs.

Figure 2 Intra-class-correlation values as a function of spinal cord level for fractional anisotropy, axial diffusivity, radial diffusivity and mean diffusivity. ROI: Region of interest; FA: Fractional anisotropy; AD: Axial diffusivity; RD: Radial diffusivity; MD: Mean diffusivity.

Figure 3 Cronbach’s alpha values as a function of spinal cord level for fractional anisotropy, axial diffusivity, radial diffusivity and mean diffusivity. ROI: Region of interest; FA: Fractional anisotropy; AD: Axial diffusivity; RD: Radial diffusivity; MD: Mean diffusivity.

DISCUSSION

This study sought to evaluate within and between rater agreements in DTI parameter values of the normal pediatric cervical spinal cord. Inter-rater reliability refers to the variability between raters and intra-rater reliability measures the agreement within raters across multiple trials.

There were significant differences between raters in the number of pixels they chose to outline using free-hand ROIs. There were no significant differences in DTI parameter values derived from these ROIs. Although one rater was more conservative in selecting the area of spinal cord tissue (16 pixels vs 24 pixels), the difference in the number of pixels may not have been large enough to detect difference in DTI parameter values.

When the free-hand and fixed-size ROI drawing methods were compared to each other, significant differences were found in DTI parameters. The large difference between the numbers of pixels selected to outline the spinal cord may reflect the differences found in DTI values. For example, rater 1 outlined 24 pixels using free-hand compared to 6 pixels when placing the pre-fixed ROI. A conservative outline may not be sensitive enough to detect physiological changes whereas a large ROI may include CSF. Thus consensus is needed to standardize ROI drawing techniques.

Overall agreement between and within raters was evaluated using the ICC coefficient and showed moderate to strong agreement between the repeated measurements (0.50-0.84). There was a variation in all DTI parameters at lower and upper spinal cord levels. Contributing factors to these differences could be signal drop-off at the edge of the neck coil. The lower cervical and upper thoracic levels were positioned at the extremities of the neck coil where signal-to-noise decreases, and accidental inclusion of CSF where image artifacts were prominent. Additionally, the lowest cervical levels (C4-C7) are the most sensitive to cardiac motion. Therefore some cardiac-related artifacts may have biased the selection or placement of ROIs. When examining DTI parameters individually, FA and RD showed the highest variability of ICC values. The agreement was lower in the pre-set ROIs, primarily in the upper and lower spinal cord levels (C1, C2, C6 and C7). The variations in these DTI parameters could be due to their relatively high sensitivity to CSF volume averaging. Furthermore, due to the decrease in cord volume between C1 and T1, consistent ROIs were difficult to maintain. This may have led to the lower agreement in the fixed size ROIs. At levels where the spinal cord cross section is small, the rectangular 3 × 2 ROI may have been too large and included artifact from surrounding CSF, while with larger cross sectional areas, the 3 × 2 pixel-ROIs were too small and did not include enough spinal cord tissue. It is important to note that the 3 × 2 pixel size ROI was chosen to be large enough to obtain sufficient data points but small enough to be potentially suitable for the atrophic spinal cord.

To examine if the raters showed any variability in their repeated measurements Cronbach’s alpha was used. The results showed that although one rater was more conservative in outlining spinal cord area and selected less pixels, strong test re-test agreement was seen for all DTI parameters. There were few data points with lower Cronbach’s alpha values in the lower and upper spinal cord levels, similar to the ICC analysis.

This study has some limitations. Only normally developed subjects were evaluated. A similar study in subjects with spinal cord injuries could provide different insights about the methodology of segmenting and measuring spinal cord tissue. Additionally, although both raters had extensive experience in DTI post-processing and agreed on the methodology used in this study, they may not have discussed all possible components in ROI drawing methods. There was a consensus of ROI size and shape taking into consideration the anatomical changes in cross sectional area at different levels of the spinal cord.

In conclusion, the usefulness of a non-invasive quantitative measurement depends on the validity, intra-rater and inter-rater reliability of the derived data for different conditions. This study demonstrates the importance of developing a robust method of DTI post-processing analysis, specifically ROI placement, to reduce operator variability and thereby develop accurate imaging biomarkers to examine spinal cord injury.

COMMENTS

Background

Diffusion tensor imaging (DTI) is a relatively new technique to examine the spinal cord in vivo. DTI offers a general understanding on structural connectivity of axonal white matter, and is believed to be a more sensitive measure in assessing damage to tracts in the spinal cord. It quantifies diffusion of water molecules in each voxel of an image in directions parallel and transverse to the plane of neuronal axons. The unique anisotropic characteristics of the spinal cord may allow DTI to localize white matter, separate white from gray matter and assess structural damage of the cord. To be proven clinically reliable, the quantitative properties of spinal cord DTI need to be assessed by multiple reviewers. The purpose of this study was to assess inter- and intra-rater reliability between two ROI methods in pediatric spinal cord DTI.

Research frontiers

Information extracted from DTI might allow analysis of the association between anatomical measures and degree of disability. This can be applied to various spinal cord diseases to assess viable tissue after injury and even guide surgical planning. Poor reproducibility limits the potential clinical application of any imaging technique. This study highlights the importance of assessing inter- and intra-rater agreement in the analysis of DTI data.

Innovations and breakthroughs

While extensive data is available on the reliability and reproducibility of brain DTI, to the best of our knowledge there is lack of research on the reliability of ROI placement to quantify spinal cord DTI measures. The goal of this study was to assess inter-rater (between) and intra-rater (within) agreement in pediatric spinal cord DTI, as well as agreement between two methods of drawing regions of interest.

Applications

DTI provides a means to measure tissue integrity following SCI and can allow an association between physical disability and spinal cord volume loss. Given the sensitivity of DTI to white matter integrity, it could and has been used in demyelinating conditions, traumatic spinal cord injury, Wallerian degeneration, and other conditions where white matter integrity in affected or lost.

Terminology

DTI is a MRI-based technique for non-invasively examining diffusion of water molecules in each voxel of an image in directions parallel and transverse to the plane of neuronal axons. The quantitative characteristics of DTI allows for the characterization of physical and biophysical properties of tissue. Fractional anisotropy is a unit-less index used to characterize the directionality of the fibers. Axial diffusivity is parameter derived from DTI calculations that reflects diffusivity of water molecules parallel to the spinal cord tracts. Radial diffusivity represents diffusivity perpendicular to the spinal cord tracts.

Peer-review

This study evaluated within and between rater agreements in DTI parameter values of the normal pediatric cervical spinal cord. Inter-rater reliability refers to the variability between raters and intra-rater reliability measures the agreement within raters across multiple trials. The data showed that high reproducibility in spinal cord DTI can be achieved, and demonstrated the importance of setting detailed methodology for post-processing DTI data, specifically the placement of ROIs.