Methods for improving colorectal cancer annotation efficiency for artificial intelligence-observer training

Table 1 Clinical and protocol details of training and test cases

	Training cases (n = 58)	Test cases (n = 20)
AJCC stage
Stage 1	0	5
Stage 2	15	5
Stage 3	14	7
Stage 4	29	3
T stage
T1	0	2
T2	0	4
T3	21	13
T4	37	1
Location
Right	39	17
Transverse	3	2
Left	16	1
CT slice thickness (mm)
7	0	1
5	29	17
3-4	25	0
2 or less	4	2
Contrast
IV+PO	27	18
IV	22	1
PO	4	1
None	5	0

AJCC: American Joint Commission on Cancer; CT: Computed tomography; IV: Intravenous; PO: Positive oral.

Table 2 Sensitivity and false positives/case for ensemble technique

	Single voter	2 voter	3 voter
Sensitivity	0.8	0.6	0.3
False positives/case	21.95	7.55	3.7

Table 3 Dice coefficient distribution for ensemble technique

Percentage of cases	Estimated dice coefficient
	0	0-0.25	0.25-0.5	> 0.5
Single voter	20	5	60	15
2 voter	40	35	20	5
3 voter	70	15	10	5

Table 4 Amount of time needed to annotate the tumor

Lesion size	Annotation time based on technique (Min:Sec ± min)
	Manual (n = 3 each)	AI-single voter (n = 3 each)	AI-2-voter (n = 3 each)	Skip-1 (n = 3 each)	Skip-2 (n = 3 each)
Large	22:09 ± 0.18	21:00 ± 0.23	20:29 ± 0.22	8:58 ± 1.22	5:34 ± 1.19
Medium	15:06 ± 0.4	10:37 ± 0.25	9:13 ± 0.15	4:58 ± 2.57	1:14 ± 1.38
Small	5:54 ± 0.07	6:26 ± 0.03	5:44 ± 0.02	2:23 ± 0.14	1:24 ± 0.28

AI: Artificial intelligence.