Evaluation of surgical training in the era of simulation

Table 1 Study characteristics assessing open surgical skills (n = 5)

Ref.	Year	No. of trainees	Tasks	Assessment tool
Acton et al[9]	2010	157 clerkship	Suturing	OSATS
Brydges et al[10]	2008	38 trainees	One-handed knot tying	Motion analysis (ROVIMAS) and GRS
Chipman et al[11]	2009	24 trainees PGY 1	Excision of skin lesion and wound closure	OSATS
Jensen et al[12]	2008	45 PGY 1-2	Excision of skin lesion and bowel anastomosis	Video-based OSATS and FPA (wound closure aesthetic quality and anastomotic leak pressure)
Olson et al[13]	2012	11 intern	Open laparotomy and bowel anastomosis	OSATS and survey

OSATS: Objective Structured Assessment of Technical Skills; ROVIMAS: RObotics VIdeo and Motion Assessment Software; GRS: Global Rating Score; FPA: Final product analysis.

Table 2 Study characteristics of studies assessing laparoscopic skills (n = 37)

Ref.	Year	No. of participants	Tasks	Assessment tool
Aggarwal et al[14]	2007	20 trainees	Laparoscopic cholecystectomy	Motion analysis and video-based GRS
Arora et al[15]	2011	25 surgeons	Laparoscopic cholecystectomy	OSATS
Bennett et al[16]	2011	70 students	Camera navigation	Box trainer
Botden et al[17]	2009	18 students	Laparoscopic suturing	ProMIS™, FPA using 5-point Likert Scale
Buzink et al[18]	2012	25 trainees6 experts	Diagnostic laparoscopy, laparoscopic cholecystectomy and laparoscopic appendicectomy	LapMentor
Cope et al[19]	2008	22 interns	6 tasks on MIST VR	MIST VR
Crochet et al[20]	2011	26 trainees	Laparoscopic cholecystectomy	VR Simulator
Ganai et al[21]	2007	19 students	Angled telescope navigation	VR Simulator
Grantchar-ov et al[22]	2009	37 residents	Basic laparoscopic task	MIST VR
Heinrich et al[23]	2007	17 experts	26 modules	LapMentor, LapSim, ProMIS™, Surgical SIM
Kanumuri et al[24]	2008	16 students	Laparoscopic suturing and knot tying	Video-based performance assessment tool on live porcine
Kolozsvari et al[25]	2012	63 residents	FLS tasks1	FLS scoring system
Kurashima et al[26]	2013	17 residents	Laparoscopic inguinal hernia repair	GOALS-GH
Langelotz et al[27]	2005	150 surgeons	Navigation, coordination, grasping, cutting and clipping	VR simulators
LeBlanc et al[28]	2010	29 surgeons	Laparoscopic sigmoid colectomy	ProMIS™ simulator, OSATS and operative error
Lehmann et al[29]	2012	36 surgeons	2 LapSim tasks	LapSim
Lehmann et al[30]	2013	105 surgeons	Lifting and Grasping, Fine dissection	LapSim
Loukas et al[31]	2011	25 trainees	Adhesiolysis, bowel suturing, laparoscopic cholecystectomy	LapVR
Loukas et al[32]	2011	20 trainees	Adhesiolysis, bowel suturing, laparoscopic cholecystectomy	LapVR
Loukas et al[33]	2012	44 novices	Peg transfer, cutting, knot tying	LapVR and video trainer
Lucas et al[34]	2008	32 students	Laparoscopic cholecystectomy	OSATS
Mansour et al[35]	2012	48 trainees	Peg transfer, clipping	VR simulators
Munz et al[36]	2007	20 novices	Intracorporeal knot tying	ICSAD and checklist
Munz et al[37]	2004	24 novices	Cutting a shape on a glove and clipping a rubber tube	Motion analysis and error score
Palter et al[38]	2012	25 residents	Laparoscopic right colectomy (live and simulator)	Video-based procedure-specific evaluation tool, modified OSATS global rating scale and LapSim
Palter et al[39]	2013	20 trainees	Clipping, and lifting and grasping, laparoscopic cholecystectomy (actual OR)	Video-based procedure-specific evaluation tool, modified OSATS global rating scale and LapSim
Panait et al[40]	2011	42 applicants	Navigation, coordination, grasping, cutting and clipping	LapSim
Rinewalt et al[41]	2012	20 residents	FLS tasks	GOALS
Rosenthal et al[42]	2006	20 students	Clip and cut cystic duct	Xitact LS500 Virtual Patient
Seymour et al[43]	2002	16 trainees	Laparoscopic cholecystectomy (OR)	Video-based operative error scoring system
Sharma et al[44]	2013	19 trainees	Laparoscopic cholecystectomy	LAP MentorTM
Stefanidis et al[45]	2013	42 novices	Laparoscopic suturing (OR)	GOALS, speed, accuracy and inadvertent injuries
Stelzer et al[46]	2009	23 interns	Peg transfer, intracorporeal knot tying in dry lab,	MISTELS scoring system
			running the bowel, intracorporeal knot tying in live porcine model	Video-based modified GOALS
Tanoue et al[47]	2010	194 surgeons	Lifting and grasping	LapSim
Torkington et al[48]	2001	13 trainees	MIST VR tasks	ICSAD and MIST VR
van Rijssen et al[49]	2012	162 trainees	Intracorporeal knot tying	OSATS and Motion Analysis Parameter (MAP)
Varas et al[50]	2012	25 residents	Laparoscopic jejunojejunostomy	OSATS, ICSAD, FPA

¹FLS tasks are peg transfer, pattern cut, endoloop placement, suture with an extracorporeal knot and suture with an intracorporeal knot. GRS: Global rating scale; OSATS: Objective Structured Assessment of Technical Skills; FPA: Final product analysis; MIST-VR: Minimally invasive surgical trainer-virtual reality; VR: Virtual reality; FLS: Fundamentals of laparoscopic surgery; GOALS: Global operative assessment of laparoscopic skills; GOALS-GH: Global Operative Assessment of Laparoscopic Skills-Groin Hernia; OR: Operating theatre; MISTELS: The McGill Inanimate System for Training and Evaluation of Laparoscopic Skills; ICSAD: Imperial College Surgical Assessment Device.

Table 3 Study characteristics of studies in assessment of open and laparoscopic skills (n = 4)

Ref.	Year	Number of participants	Tasks	Assessment tool
Beard et al[51]	2011	85 trainees	Mixed tasks (OR)	Procedure-based assessment, OSATS
Fernandez et al[52]	2012	30 PGY 1	Knot-tying, suturing, laparoscopic skills	OSATS, computer metric-based performance assessments
Mittal et al[53]	2012	60 residents	Basic skills(knot tying,wound closure, enterotomy,vascular anastomosis) and FLS	OSATS and FLS
Parent et al[54]	2010	28 interns	Wound closure and FLS tasks	Essential item checklist, economy of time, global competence, FLS system

OR: Operating theatre; OSATS: Objective Structured assessment of technical skills; FLS: Fundamentals of laparoscopic surgery.

Table 4 Characteristics of studies in assessment of endoscopic skills (n = 6)

Ref.	Year	Number of participants	Tasks	Assessment tool
Ende et al[55]	2012	28 residents	OGD	Simulator and observation
Götzberger et al[56]	2011	13 trainees	No mention in abstract	Simulator (5-point Likert scale)
Haycock et al[57]	2010	36 trainees	Colonoscopy (simulator and OR)	Direct Observation of Procedural Skills and Global Scores sheet
Haycock et al[58]	2009	28 trainees	Polypectomy, control of upper GI bleeding and oesophageal dilation and PEG insertion	Station-specific checklist and global score
Shirai et al[59]	2008	20 residents	OGD	11 items 5-grade scale
Van Sickle et al[60]	2011	41 trainees	Colonoscopy	GI Mentor II and GAGES

OGD: Oesophago-gastro-duodenoscopy; GI: Gastrointestinal; PEG: Percutaneous Endoscopic Gastroscopy; GAGES: Global Assessment of Gastrointestinal Endoscopic Skills.

Table 5 Characteristics of studies integrating skills assessment tools in a simulation-based curricula and selection process (n =12)

Ref.	Year	No. of participants	Tasks	Assessment tool
Open skills
Chipman et al[11]	2009	24 trainees	Excision of skin lesion and wound closure	OSATS
Olson et al[13]	2012	11 interns	Open laparotomy, bowel anastomosis	OSATS and survey
Laparoscopic skills
Buzink et al[18]	2012	25 trainees	Diagnostic laparoscopy, laparoscopic cholecystectomy and laparoscopic appendicectomy	LapMentor
		6 experts
Palter et al[39]	2013	20 trainees	Clipping, and lifting and grasping, Laparoscopic cholecystectomy (actual OR)	Video-based procedure-specific evaluation tool, modified OSATS global rating scale and LapSim tasks
Panait et al[40]	2011	42 applicants	Navigation, coordination, grasping, cutting and clipping	LapSim
Rinewalt et al[41]	2012	20 residents	FLS tasks	GOALS
van Rijssen et al[49]	2012	162 trainees	Intracorporeal knot tying	OSATS and Motion Analysis Parameter(MAP)
Varas et al[50]	2012	25 residents	Laparoscopic jejunojejunostomy	OSATS, ICSAD, FPA
Open and laparoscopic skills
Fernandez et al[52]	2012	30 PGY 1	Knot-tying, suturing, laparoscopic skills	OSATS, computer metric-based performance assessments
Mittal et al[53]	2012	60 residents	Basic skills(knot tying,wound closure, enterotomy, vascular anastomosis), FLS tasks1	OSATS and FLS score
Parent et al[54]	2010	28 interns	Wound closure, FLS tasks1	essential item checklist, economy of time, global competence, FLS score
Endoscopic skills
Van Sickle et al[60]	2011	41 trainees	Colonoscopy	GI Mentor II and GAGES

¹FLS tasks are peg transfer, pattern cut, endoloop placement, suture with an extracorporeal knot and suture with an intracorporeal knot. OSATS: Objective Structured Assessment of Technical Skills; FLS: Fundamentals of Laparoscopic Surgery; GOALS: Global Operative Assessment of Laparoscopic Skills; ICSAD: Imperial College Surgical Assessment Device; FPA: Final Product Analysis.