Prospective Study Open Access
Copyright ©The Author(s) 2017. Published by Baishideng Publishing Group Inc. All rights reserved.
World J Clin Pediatr. Feb 8, 2017; 6(1): 69-80
Published online Feb 8, 2017. doi: 10.5409/wjcp.v6.i1.69
Video recording of neonatal resuscitation: A feasibility study to inform widespread adoption
Sandesh Shivananda, Jennifer Twiss, Enas el-Gouhary, Salhab el-Helou, Connie Williams, Prashanth Murthy, Department of Pediatrics, McMaster University, Hamilton, ON L8N 3Z5, Canada
Gautham Suresh, Pediatrics-Newborn, Baylor College of Medicine, Houston, TX 77030, United States
Author contributions: Shivananda S coordinated and supervised data collection, carried out the initial analysis, and drafted the initial manuscript; Twiss J, el-Gouhary E, el-Helou S, Williams C and Murthy P took part in designing and data collection; Suresh G reviewed and revised the manuscript; Shivananda S conceptualized and designed the study, implemented the protocol, and reviewed the manuscript; all authors approved the final manuscript as submitted.
Supported by the Centre for Healthcare Optimization Research and Delivery (CHORD) at Hamilton Health Sciences with an aim of facilitating knowledge transfer initiatives from health care and supportive teams.
Institutional review board statement: The study was reviewed and approved by the institutional review boards of McMaster University and Faculty of Health Sciences Hamilton and Hamilton Health Sciences Quality and patient safety committee.
Clinical trial registration statement: This study was planned to assess feasibility (pilot or proof of concept) of wide scale adoption and study of video recording of neonatal resuscitation. We didn’t register this feasibility study in clinical trial registry.
Informed consent statement: A waiver of consent was obtained from ethics review board for this study.
Conflict-of-interest statement: The authors of this manuscript having no conflicts of interest to disclose.
Data sharing statement: All data supporting the study are provided in full in the results section of the manuscript, whereas video recording clips were deleted after review to ensure subject’s privacy and confidentiality. There is no additional data available.
Open-Access: This article is an open-access article which was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Correspondence to: Dr. Sandesh Shivananda, MD, MSc, FRCPC, Associate Professor, Department of Pediatrics, McMaster University, 1200 Main St. West, Hamilton, ON L8N 3Z5, Canada. sandesh@mcmaster.ca
Telephone: +1-905-5212100-73490 Fax: +1-905-5215007
Received: June 25, 2016
Peer-review started: June 28, 2016
First decision: September 5, 2016
Revised: October 12, 2016
Accepted: November 16, 2016
Article in press: November 17, 2016
Published online: February 8, 2017
Processing time: 220 Days and 14.9 Hours

Abstract
AIM

To determine the feasibility of introducing video recording (VR) of neonatal resuscitation (NR) in a perinatal centre.

METHODS

This was a prospective cohort quality improvement study on preterm infants and their caregivers. Based on evidence and experience of other centers using VR intervention, a contextually relevant implementation and evaluation strategy was designed in the planning phase. The components of intervention were pre-resuscitation team huddle, VR of NR and video debriefing (VD), all occurring on the same day. Various domains of feasibility and sustainability as well as feasibility criteria were predefined. Data for analysis was collected using quantitative and qualitative methods.

RESULTS

Seventy-one caregivers participated in VD of 14 NRs facilitated by six trained instructors. Ninety-one percent of caregivers perceived enhanced learning and patient safety and, 48 issues were identified related to policy, caregiver roles, and latent safety threats. Ninety percent of caregivers expressed their willingness to participate in VD activity and supported the idea of integrating it into a resuscitation team routine. Eighty-three percent and 50% of instructors expressed satisfaction with video review software and quality of audio VR. No issues about maintenance of infant or caregivers’ confidentiality and erasure of videos were reported. Criteria for feasibility were met (refusal rate of < 10%, VR performed on > 50% of occasions, and < 20% caregivers’ perceiving a negative impact on team performance). Necessary adaptations to enhance sustainability were identified.

CONCLUSION

VR of NR as a standard of care quality assurance activity to enhance caregivers’ learning and create opportunities that improve patient safety is feasible. Despite its complexity with inherent challenges in implementation, the intervention was acceptable, implementable, and potentially sustainable with adaptations.

Key Words: Video recording; Neonatal resuscitation; Delivery room; Feasibility; Perinatal centre

Core tip: Despite proven benefits video recording (VR) of neonatal resuscitation (NR) is not adopted by all perinatal centres. Major reasons include challenges in operationalization and sustainability. Understanding the enablers and mitigation strategies is crucial on making a decision on widespread adoption of VR of NR by hospitals. We conducted a feasibility analysis of introducing VR of NR in the delivery room. It was introduced as a standard of care quality assurance activity to enhance caregiver learning and address system issues that compromise patient safety. Our study results indicate that VR of NR was effective, acceptable, implementable, and potentially sustainable with adaptations.



INTRODUCTION

About 10% of newborns require assistance to begin breathing, and about 1% is critically ill and may need life-saving therapies such as ventilation, chest compression, and medications to support heart rate[1]. In perinatal centres, the goals of neonatal resuscitation (NR) program are to have a system in place to deliver effective, efficient and safe care during the neonatal transition or resuscitation at birth, and later while responding to cardiorespiratory events[2]. Relevant elements of care include policies that govern the provision of resuscitation services, availability of equipment and staff who are trained and competent to deliver consistent and reliable high-quality care[2]. Gaps in the delivery of high-quality care may arise out of team performance, adherence to best practices and presence of latent safety threats[3-5]. Poor or inconsistent team performance, in-turn could be secondary to the inevitability of having multiple teams with variable composition and lack of structured team training and reflective deliberate practice[6]. In this context, video debriefing (VD) of actual NR is believed to facilitate the acquisition, retention, and application of skills resulting in optimal team performance and outcomes[7]. Advantages of video recording (VR) include noninterference with resuscitation and collection of unalterable objective data[8]. Moreover, VR and VD of NR has been shown to facilitate individual learning, identify and address system issues leading to better team performance and patient safety[8-11]. Despite benefits and apparent feasibility, VR of actual NR is not adopted widely in all perinatal centres because it is a complex intervention[12] and its operationalization is challenging[9,13]. To date, there have been no reports of implementing VR of NR as a quality assurance standard of practice activity from a Canadian perinatal centre. We believed that testing the feasibility of this intervention on a small scale in the real world setting with contextual constraints would inform decisions on widespread adoption.

Objective

Conduct a feasibility analysis of introducing VR of NR in the delivery room, as a standard of care quality assurance activity to enhance caregiver learning and create opportunities that improve patient safety.

MATERIALS AND METHODS

This was a prospective cohort quality improvement (QI) study. All inborn preterm infants delivered at less than 33 wk of gestation from November 2013 to January 2014, as well as resuscitation team members, were included.

Setting

Approximately 225 infants of less than 33 wk gestation are born every year at McMaster University Medical Centre hospital. The L and D suite has eight delivery rooms (DR), two operating rooms (OR-A and OR-B) and three obstetric ICU beds. A 47-bedded level 3 regional NICU is located adjacent to L and D suite. All infants are delivered and resuscitated in the same room. Following resuscitation, all babies were transferred to a dedicated stabilization room adjacent to the delivery suite, before finally being transferred to NICU. A fellow, nurse, respiratory therapist and a nurse practitioner attended all births. A neonatologist participated in the resuscitation and stabilization of infants born at less than 26 wk. of gestation or when indicated for higher gestational age.

Planning of the intervention

In 2010, a large QI project aimed at improving practices during resuscitation in preterm infants was introduced in our center. Video recording of actual neonatal resuscitation as a quality assurance activity (NRQAA) was a project nested within the larger project. The QI team composed of two physicians, nurse practitioner and a nurse was formed to oversee the implementation of this project. We implemented NRQAA program in four phases.

Phase I (January 2010-July 2012): The QI sub-team initially did a literature review to understand the requirements (equipment, personnel, standard procedures), critical success factors and challenges in the implementation of VR of NR[4,7-11,13-15]. We obtained input from leads of two other centers with experience in the VR of NR. Following that, we developed the first draft of the NRQAA program and presented to nurses, physicians, respiratory therapists, nurse practitioners and managers on separate occasions. Concerns about workload, workflow and seeking approvals from multiple stakeholders were gathered. Concurrently, we reviewed policies, procedures, and guidelines, at the department, hospital, provincial (state) and national level before seeking approvals from all stakeholders. We contacted representatives of above organizations/authorities as necessary to facilitate approvals. A standardized operating manual providing complete details of the intended program, necessary approvals, and implementation process was prepared. Finally, we obtained approval of hospital’s quality of care and patient safety committee, represented by all stakeholders. Following approval, installation of camera, web server and necessary hardware and software happened in consultation with engineering, information and communication technology, infection control and obstetric teams. Finally testing and fine-tuning of audio VR, review of software, storage and erasure of videos were completed (Table 1).

Table 1 Overview of video recording of actual neonatal resuscitation as a quality assurance activity program and contextual details.
Overarching goalEnhance the likelihood of caregivers’ delivering effective, safe and high quality NR care
Specific goalFeasibility of introducing NRQAA program as a standard of care activity in a tertiary perinatal centre
Method of implementation Assessment of readinessQuality assurance activity. Not introduced as a research study or a teaching activity Although NRP certification of all caregivers, in-situ unadvertised mock code (2008), high fidelity simulation using SimnewB (2010) were occurring in the unit, training in team behaviors, crisis resource management and error prevention had not happened
Training in team behaviors and exposure to VRInterprofessional workshop in team behaviors, crisis resource management and error prevention (October 2011-January 2012), orientation of all new resuscitation team members and learners to team behaviors (January 2011 onwards), use of VD during mock resuscitations and training sessions (July 2011 onwards) were introduced. Pre resuscitation briefing of all anticipated high risk deliveries were introduced as a routine (September 2011). Team composition, configuration during resuscitation, member roles, anticipation of worst-case scenario and care planning were to be discussed by the neonatal fellow in briefing meetings. An expectation to complete the resuscitation and stabilization within 60 min of life was communicated to all members. T piece resuscitator for CPAP and PPV, Oxygen administration based on pulse oximetry reading and targeting saturation value appropriate for minute of life, and prophylactic CPAP for all < 33 wk s gestation infants were introduced as a part of larger QI initiative (January-July 2011)
Training instructors in debriefingOnly two out of 6 instructors had formal training in simulation and debriefing. These two instructors in-turn trained other instructors
TechnologyFixed IP video cameras with audio and video capturing capability and mounted on the roof/walls of the delivery rooms were used. They were wired to a web server placed in a room adjacent to NICU. VR was supposed to be turned on by the resuscitation team members (primarily by RTs) and stopped at the end of resuscitation. This video was automatically stored on the webserver and could be accessed or retrieved by instructors till its erasure
Securing resourcesFunding for installation of video camera, web server and storage were obtained from the hospital KT grant. All personnel in QI subcommittee contributed their non clinical time for the program
Consent from family and staffObtained waiver of consent as the project was introduced as a Quality assurance standard of care practice and not as a research study. Consent was required for use of video for non quality assurance activity such as teaching providers and learners beyond the NICU team members and for research
Information about NRQAA was to be provided for all care providers and parents
Medical record vs quality assurance record Data ownership, management and disclosure of errorVideo was considered, as quality of care documents as videos would offer any health benefit to patient would not be used for care and treatment of individual patient and those other records of resuscitation apart from video would be preserved in medical records
NRQAA committee was to oversee the NRQAA documents. No personal identifiers were collected. Any error was to be disclosed to the family as per the hospital policies
Video storage and securityVideos were directly stored on hospital web servers. They were accessible from a single computer located in a room adjacent to NICU. All VD was supposed to happen in the same room. The room was locked at all times and had swipe access. Access to VR was limited to instructors. All instructors had to sign a confidentiality and security statement after receiving training in accessing and reviewing videos. Any use of videos by instructors apart from quality assurance activity as well as sharing of access information and delegation was prohibited
Following video review the videos were erased from the server manually
Medico legal concernsNRQAA was not organized through QCIPA, as viewing of video by all team members or occasionally by parent would not have been possible. Thus an opportunity for collaborative learning and reflecting on one’s own performance would have been lost
Care providers were to understand that a video was subpoenable and parents had to consult hospital legal counsel and NRQAA committee before the release of the video
Risk of spoliation or intentional destruction of evidence allegation. Video erasure policyVideo destruction policy was defined with a caveat that any patients for whom there has been a report to hospital heath care liability insurance provider, a request for records, or involvement of a coroner, a professional college or any notice of any legal proceeding whatsoever involving the patient, that those videos be maintained as until any proceeding is finalized. We opted to delete the videos when videos are reviewed and debriefed or within 14 d of recording, whichever come first. We also informed care givers and parents that the videos will not be made available for any other reasons apart from those described above
Privacy of patient and staffThe video cameras were focused on the on the infant and not on caregivers. Caregivers’ hands were captured inadvertently during the process. All audio including caregivers’ conversation was captured during the VR. As per the Personal Heath Information protection Act, 2004 (PHIPA) NRQAA was to institute measures to ensure personal information is not inadvertently disclosed or accessed by inappropriate person through out the program course.
All learners while attending the VD activity were to sign a confidentiality agreement form. All NICU care providers were to abide with existing hospital confidential policy, which clearly prohibited the use of personal names or discussion performance issues outside the quality forums
Privacy office recommendationsManagement of access and transfer if any to be done by a person approved by NRQAA. The program lead is responsible for oversight of the process
Retention time to not exceed 14 d
Transfer and destruction log along with the signature of individual conducting transaction should be noted
Use encrypted USB key approved by hospital ICT team for any data transfer between NICU and hospital server
Ensure erasure process meets security requirements
Refusal from staff/familyRisk of refusal was proactively addressed by communicating the rationale for VR and attempting to minimize misconceptions among caregivers. An adequate lead-time and multiple forums to discuss concerns arising out of VR were provided. Similarly supervisors were encouraged to address concerns related to their respective professions and to support their colleagues during NRQAA
VR was supposed to be initiated by resuscitation team by turning on the switch as opposed to motion sensing/auto recording
All video reviews and VDs were supposed to be done by physicians during the feasibility period
Institutional supportAll stakeholders were informed and their support was obtained before launching the project, e.g., Quality of care and patient safety team, Information technology, Privacy, Obstetrics, Engineering, Infection control, Executives, legal council, risk management, REB and senior executives of the hospital
Support from professional bodiesSupport was obtained from Canadian Medical Protection Agency, Nursing association, heads of professional practice of nurses and respiratory therapy, nursing unions
Project managementProject timelines, committee members roles, training and evaluation were all defined by the program lead
Resource limitationsIn order to minimize cost of installation, instructors time and workload the following limitations were accepted apriori before the launch of the program
Video cameras were installed in three out of possible 13 delivery rooms. These three rooms had contributed to 60%-70% of all high-risk deliveries in 2007-2009. Obstetric staffs were informed to preferentially triage all less than 33 wk gestation laboring mothers to above three rooms
Video review was limited to first 10 min of life and scheduled VD to day deliveries on weekdays.
Superimposition of heart rate, SpO2, pressure and flow from pulse oximter and ventilator onto VR s were not done
Instructors did not have access to review the videos remotely
Resources for all instructors to take certification courses in debriefing was limited

Phase II (August 2013-October 2013): Six out of 11 neonatal attending volunteered to participate in NRQAA as instructors and joined the QI sub-team (NRQAA committee). All of them were NRP trainers and two of them had training and certification in debriefing. They were requested to sign up for a one-week block of facilitation and evaluation of NRQAA activity. The NRQAA committee met on two occasions to finalize the NQAA interventions, facilitator roles, and instruments. All instructors received 2 h of training on: (1) facilitating pre-resuscitation briefing, reviewing VR and debriefing videos; (2) accessing and using video review software on the web server; (3) maintaining privacy and confidentiality; and (4) use of instruments by using a simulated VR of NR. All instructors received a manual comprising of terms of reference, instruments and tools to facilitate the interventions and perform evaluations.

Phase III (November 2013-January 2014): Physicians assisted and completed all VR and VD assessments and facilitations.

Phase IV (February 2014-July 214): We conducted a survey of all resuscitation team members and instructors. The instructors also participated in a focus group to discuss the preliminary results and to identify factors critical for sustainability of the program.

Planning the study of intervention

We used the accepted frameworks to assess the effectiveness of NRQAA program and standard criteria for reporting feasibility[16-18].

Interventions

We introduced three interventions as part of NRQAA program: (1) facilitated pre-resuscitation briefing; (2) VR and review; and (3) facilitated VD (Figure 1).

Figure 1
Figure 1 Interventions and tools for assessment. 1: Assessment of nontechnical skills (23); 2: Assessment of team behaviors (3); 3: Assessment of crisis resource management (19); 4: Debriefing template (20); 5: Caregiver feedback form; 6: Instructor weekly report; 7: Instructor survey (22); 8: Caregiver survey; 9: Focus group (21). Circle: Completed/attended by instructors; Diamond: Completed by caregivers.

Facilitated pre-resuscitation briefing: Optimal team performance during NR depends on gathering pertinent perinatal history, understanding team member roles and case specific preparation[10]. The list of potential high-risk deliveries and the scheduled resuscitation team members for a particular day were almost always known at the beginning of the day. Thus a structured daily pre-resuscitation briefing at 930 was introduced in our center in September 2011. It was mandatory for all resuscitation team members and was led by a neonatal fellow. The neonatal fellow gathered all relevant history and identified case specific preparation before the briefing. During the huddle, member roles were assigned, case specific care plans were discussed, and contingency planning for worst-case scenario was done.

During phase III, a neonatologist instructor observed the briefing process, facilitated case specific preparation and care planning, provided feedback to fellow and documented any system issues identified. The instructor also prompted members to turn on VR during resuscitation of an infant less than 33 wk of gestation.

VR and review: All resuscitation team members were requested to activate the VR when they attended a resuscitation of less than 33-wk gestation infant in OR-B, ISR-1 and room 8. The VR button was supposed to be turned on, just before receiving the baby on the resuscitaire and turned off after 10 min of resuscitation. All members received orientation on NRQAA activity in their respective monthly meetings in October 2013. All members were reminded to press the VR button during the pre-resuscitation briefing at the beginning of the day.

We reviewed every VR. The focus of evaluation was team behaviors[3], leader’s crisis resource management skills[19] and documentation of debriefing points[20] and system issues[5]. The instructor contacted the parents of infants whose resuscitation was recorded and provided an information sheet on NRQAA. Resuscitation team members were contacted when necessary to provide feedback on their performance. The most responsible physician in NICU was informed of any system issues identified during the review of videos.

Facilitated VD: Facilitated VD happened every afternoon at 3 pm in a room adjacent to NICU. All videos were accessed and projected during the discussion. The instructors were supposed to review the videos and identify the debrief points before the debriefing. The goal of VD was to create a collaborative learning environment that allows the caregivers to reflect their performance, share their thoughts and emotions without fear, learn how to recognize and improve their deficiencies. VD session was limited to resuscitation team members who participated in the NR, wherever was done. Other caregivers were allowed to attend the meeting if there was no objection from the participants. The VD session was structured. At the outset, the trainers clarified the purpose of the debriefing, participants and facilitator’s role, confidentiality measures and the need for filling the evaluation form. Then the video was presented without interruption for 5-10 min. We used the “observation, advocacy and inquiry” format to initiate debriefing[20]. Debriefing points were usually kept to a maximum of three to ensure in-depth discussions. Selected parts of a video could be replayed as necessary. Finally, the debriefing session was summarized to distil the lessons learned for future use. All VR done after 4 pm and during night shift or weekends were reviewed but without a VD. On days when there was no VR, a VD session was adjourned. The instructor facilitated VD and documented any system issues identified during resuscitation.

Outcomes

The focus of feasibility analysis was to measure acceptability, demand, the usability of technology and instruments, adaptations, resource needs, unintended effects and limited efficacy[17,18] (Table 2). Limited efficacy was assessed using: (1) caregivers’ perception of VD on one’s learning and a likelihood of enhancing patient safety; and (2) ability of the program to create learning opportunities, identify latent safety threats and elicit solutions from caregivers during VD. Resource needs in initiation and maintenance of VD program were noted during the piloting. We surveyed instructors to determine the resource requirements for facilitating VD activity and its governance. Feasibility was defined apriori as less than 20% caregivers refusing to participate, the conduct VR and debriefing on more than 50% occasions, when resuscitated in delivery rooms with recording facilities, and perceived negative impact on team performance in fewer than 20% of caregivers. A decision on widespread adoption with confidence was based on likelihood of sustainability[21].

Table 2 Detailed matrix of outcomes and data collection.
Video reviewVideo debrief feedback evaluationEnd of pilot period caregiver surveyEnd of pilot period instructor surveyEnd of pilot period focus group
Limited efficacy testing Caregivers’ perception on learning and enhancing patient safety Impact on desired organizational outcome Create learning environment Enhance patient safetyXXXX
Acceptability and demand1 Management2 Caregivers Parents InstructorsXXXX
Usability analysis to assess team behaviors, debrief and identify system issues VR and video review software technology2 InstrumentsXXXX
Resource needs3 Initial MaintenanceX
Unintended adverse or beneficial effects3 Sustainability Feasibility criteria Adaptations before widespread adoptionXXXXX
XX
Methods of evaluation

We used the mixed method to assess the effectiveness of implementation and outcomes. Necessary data was collected using participation rates, surveys, feedback forms, focus group, participant observation and by analyzing the comments on feedback forms and surveys.

Instruments to facilitate interventions

Four validated instruments were used to measure team behaviors and system issues[3,5,19,20,22,23]. These tools were chosen to standardize the facilitation process and evaluation of performance by multiple instructors. We used caregiver feedback forms to assess the effectiveness of VD and a survey of caregivers and instructors at the end of feasibility period to determine the overall effectiveness of implementation of the intervention and the impact of interventions. Ease and satisfaction of using the instruments were measured by incorporating the USE Questionnaire tool in the instructor survey. We used the weekly reports completed by the instructors to document the frequency of all NRQAA interventions, identify the system issues and individuals who need moderate to significant improvement in crisis management skills. A combination of the focus group, comments in the survey form and logbook notes of program lead were used to identify the challenges, necessary adaptations to enhance sustainability. Finally, NHS sustainability model and guide was used to determine the likelihood of sustainability[21].

Surveys were designed indigenously to gather caregivers’ experience, perceived impact, intentions to continue and their preferences for modifying the program. Instructors’ survey had categories on assessing the instruments and workload in addition to above categories. Readability and its appropriateness in measuring the desired outcomes were evaluated by piloting the survey on five caregivers and two instructors. Based on their suggestions, a final draft of the survey was created. Caregivers’ and instructors’ survey had 15 and 32 questions respectively, required grading the response on a Likert scale, and took 10 and 20 min to complete respectively. At the end of each category, a section for “comments and suggestion” was provided. No personal identifiers were collected.

A focus group discussion was planned to gather input from instructors, managers, and leaders to assess feasibility, the likelihood of sustainability and to identify necessary adaptations in intervention[21]. The focus group was of one-hour duration and was moderated by the program lead. The moderator took the field notes and summarized the impressions of the team at the end of the session to confirm participants’ agreement with the records.

Sample size: A 3-mo time frame was based on convenience and availability of instructors to participate in this study.

Statistical analysis

Data collected from multiple sources were tabulated and presented as a percentage. Responses to surveys on a five-point Likert scale were condensed into three categories for simplicity and expressed as percentages. The program lead performed a content analysis of field notes, survey responses and minutes of the focus group. The themes and patterns emerging from the triangulation of results were recorded by the program lead and independently confirmed by another instructor.

This project was approved by the hospital’s quality of care and patient safety committee as a standard of care quality assurance activity. Since we intended to publish the study and create information sheets for parents and caregivers, a research ethics board approval was also obtained. For any use of VR outside the NRQAA activity, consent forms were created and REB approval was obtained.

RESULTS

Out of 50 high-risk NRs, 30 were performed in delivery rooms with VR facilities. VR and VD occurred in 18 (60%) and 14 (47%) instances. Seventy-one caregivers (31% physicians, 23% nurses, 24% respiratory therapists, 11% nurse practitioners and 11% others) and six instructors participated in VD. The median (range) gestational age and birth weight of neonates were 31 (24-32) wk and 1195 (570-2070) grams respectively. Procedural interventions captured during VR included CPAP (17), Mask PPV (10), intubation and positive pressure ventilation (4) chest compression (2), umbilical venous catheterization (2), epinephrine (2) and normal saline bolus (2). Team behavior event rates observed per resuscitation, median (range) included information sharing 3 (2-8), inquiry 2 (0-4), assertion 3 (0-6), teaching 2 (0-4), evaluation of plans 2 (0-4) and an overall rate of 12 (3-21). The median (range) events of crisis resource management skills observed on leader included Leadership 5 (1-7), problem-solving 5 (1-6), situational awareness 5 (1-7), resource utilization 5 (1-7), communication 4 (1-6) and an overall rate of 4 (1-6). Thirty-nine out 60 caregivers’ (65%) responded to the end of study survey. Four instructors, nursing director, medical director, and nurse manager took part in the focus group.

Limited efficacy

Ninty-one percent of caregivers reported increased ability to reflect individual’s performance and learning team behaviors. Similarly, 91% of caregivers perceived VD activity to enhance patient safety. Despite VR happening on 18 occasions, 48 issues related to resuscitation policy and procedures, team member roles and latent safety threats were identified during the study (Tables 3-6). Solutions to above issues were elicited or provided by caregivers or instructors in 21 (43%) issues. Instructors identified ten caregivers as requiring moderate to significant improvement in CRM skills and suggested further training. On eight occasions, caregivers’ recognized a deficiency in communication and sought instructors’ suggestions on learning those skills. Also, structured opportunities for creating team situational awareness, role clarification and infant specific contingency planning occurred during the pre-resuscitation huddle.

Table 3 Policy, caregiver roles and latent safety threat issues noted during: Pre-resuscitation briefing.
IssuesSolution
When do I call an attending for help during resuscitation?Whenever chest compression is initiated
Can I transfer the first twin from resuscitaire to a basinet and then receive the next twin on the same resuscitaire?No! Two separate resuscitaire should be kept ready
Why should I know the indication for a laboring mother receiving meropenam and opioids?To decide on appropriateness of using Naloxone, neonatal isolation and performing a septic work up
Where is the main surgical OR where a C-section is happening on a mother with placenta increta?To ensure resuscitation team members reach the OR in time
What special preparation is necessary?Higher room temperature, familiarization with the new environment and all necessary equipment should be ensured
What are the indications for admitting a newborn with fetal arrhythmia to NICU?Arrhythmia noted on connecting to a multi-channel monitor in stabilization room
How do I create beds for four less than 28 wk, anticipated high-risk deliveries?Efficient problem solving and triaging
What worst case scenario should I anticipate while attending a delivery in a mother with Spinal Muscular Atrophy, unexplained IUGR and non-reassuring fetal heat rateHypoplastic lung with difficulty in resuscitation
What is the role of learners (clerks, residents, others) during resuscitation?Team leader should assign roles on a case by case basis during the team huddle
Who is responsible for gathering all information on an anticipated high-risk delivery and case specific preparation?The expectation is that the neonatal fellow covering the Labor and Delivery unit is responsible for gathering information and case specific care planning. The dedicated resuscitation nurse is responsible for calling a team huddle before attending a high risk delivery
How should the family’s preference for resuscitating a 23 or 24-wk infant be documented in antenatal consults and handed over?Family’s preference for resuscitation should be documented in written and handed over at every shift. If family’s preferences change, the revised plans should be documented in written
Table 4 Policy, caregiver roles and latent safety threat issues noted during: Issues noted during video reviewing.
Potential problems/negative impact
Communication
Not-verbalizing the reasons for initiating an intervention. e.g., intubation, chest compression, etc.Lack of understanding the reasons behind an intervention, limits team members' ability to provide suggestions
Chest compression and PPV rhythm not verbalized “one and two and three and breathe”Lack of synchronization delays neonate’s response to resuscitation
Heart rate is not verbalized after auscultatingDelay in making a decision on initiation/non initiation of chest compression
Excessive reliance on non-verbal communication, e.g., asking for a suction catheter by “stretching hands” after inserting the laryngoscope orally, as opposed to a “verbal request”Delay in receiving suction catheter causes frustration in the intubator and delays the resuscitation efforts
Silencing alarms and not communicating the alarm to the team leaderLack of awareness impedes accurate decision making and timely initiation of interventions
Team members not communicating assertively, e.g., Considering a higher peak inspiratory pressure in a non-responding infantDelay in trouble shooting leading to ineffective resuscitation
Not sharing of relevant obstetric information with NR team during resuscitation of a depressed infant, e.g., MSL, abruption, MorphineDelay in considering appropriate interventions, e.g., ET suction, fluid bolus and Naloxone respectively
Leadership
Leader was totally passiveLeads to momentary assumption of role by another member. Often results in delayed decision making, team losing focus, excessive indulgence in unnecessary interventions, e.g., suctioning, and lack of assessment of response to interventions
Fixation error, e.g., Making decisions of intubation and chest compression in a nonresponsive infant without ensuring good seal during mask ventilationUnnecessary invasive interventions with a potential for adverse events
Lack of evaluation of plans during resuscitationPrevents team members ability to provide suggestions
Team members positioning/configuration
Hands free team leader standing at the head end and RRTs who are on one side of the infantLeader impedes effective delivery of mask ventilation
Initiating chest compression with the side walls upImpedes effective performance of chest compression
Technical
Ineffective seal around the mask during mask ventilationDelay in responding to resuscitation
Attempting nasal intubation while resuscitating an unresponsive infant with severe bradycardiaPotential delay in intubation
Not venting stomach after a prolonged mask PPVSecondary deterioration in SpO2 and heart rate
Not vigilant about FiO2 during resuscitation. Started 100% FiO2 only after 90 s of chest compressionDelay in response to resuscitation
Extubation while securing the ET tube as ET tube is not held firmly against the hard palate during tapingPotential for secondary deterioration or delay in resuscitation
Table 5 Policy, caregiver roles and latent safety threat issues noted during: System issues noted during video debriefing.
Suggestions/solutions
No response from NICU front desk when called for additional help by resuscitation team in infant stabilization roomAvoid unmanned NICU front desk all the time
Preterm infant on CPAP transferred directly to NICU as opposed to stabilization in infant stabilization room and then to NICUTransfer through stabilization room ensures that a ventilator and incubator is always ready for stabilization
Person attending resuscitation is different from the one who participated in team huddleCase specific preparation and management plans discussed during team huddle becomes redundant
Difficulty in paging the resuscitation team members as the composition of resuscitation team changed during a shiftDedicated resuscitation pagers to be carried by resuscitation team members as opposed to individual personal pagers
Infant stabilization room stocking was exhausted when 3 deliveries happened during a shift. Health care aides were replenishing stocks once a shiftHealth care aides will be called to replenish stocks when necessary
Delay in sending the blood samples from infant stabilization room to labTube system restored
Needle stick injury to a resuscitation team member while setting up the resuscitaireEducate all caregivers to remove sharps after the procedures
Fall and injury to foot while running to attend a pink code in labor and delivery unitEducate caregivers on taking precautions to avoid injury
Undue delay in starting a PIV in infant stabilization room due to non-availability of personnelEducate RN team members about creating a backup support to establish PIV in time
Who is the first responder (MD/NP) to attend labor and delivery calls during handover? (8-9 am and 5-6 pm)The day resuscitation team (MD/NP) members
Pending high-risk deliveries and family’s preference for resuscitation was not passed on to day team. Thus the day team was unclear about their roles when called to attend deliveryShould be an essential part of handover
Table 6 Policy, caregiver roles and latent safety threat issues noted during: Skills related questions posed by care givers during video debriefing.
How do I communicate assertively?
How do I develop leadership skills?
What do I do when a RRT/TT member/Resident asks for intubation when the fellow is almost about to intubate?
How do I provide constructive feedback to team members during resuscitation?
When should I be “hands-on” and “hands-off” during resuscitation?
How can I ensure that I get others input during a difficult resuscitation?
It is very difficult to maintain a global perspective during resuscitation. How do I maintain it?
How do I deal with a member passing sarcastic comments/gestures during resuscitation? “Wish you all the best”
Acceptability and demand

None of the caregivers refused to participate in a video-recording activity. Instructors did not report any untoward instances during their interaction with family, after VD event. Unit leaders and managers supported visible support in promoting caregiver participation and reviewing caregivers’ suggestions. Most caregivers (90%) and instructors (100%) expressed their willingness to participate in VD activity and supported the idea of integrating it into a resuscitation team routine. Apart from 78% of caregivers’ liking the debriefing experience, they appreciated the opportunity to discuss and provide suggestions on concerns (78%). Only 10% reported that VR made them over conscious, and it may have had a “negative impact” on their performance. Fifty-nine percent of caregivers said that VD on selected NRs, once every two weeks, as opposed to all NRs, was acceptable. Similarly, 53% felt that making the VD activity open to all NICU caregivers, as opposed to those who were involved in a particular resuscitation was acceptable.

Usability

The instructors expressed satisfaction with video-recording and video review software technology (83%). Apart from being easy to learn and use, they reported that the technology helped them to be more effective in supporting analysis and conduct of VD. However, only 50% of instructors expressed satisfaction with the quality of audio VR, as significant challenges in understanding the scenario, assessing team behaviors and quality of procedures and interpretation of response to resuscitation based on audible pulse tone and alarm sounds. All instructors expressed satisfaction with orientation, maintenance of confidentiality and templates used for a pre-resuscitation huddle, VR review, and VD.

Resource needs

Resource needs for facilitation; documentation and support of VD activity are provided in Table 7. The focus group identified that the following elements are crucial for the sustainability of the program. These include: (1) instructors’ team should be interdisciplinary with a representative from MD, RN, and RRT group; (2) Instructors should be enthusiastic, non-judgmental and possess background training in simulation, debriefing and QI; and (3) one of the instructors should take on additional role of governance.

Table 7 Resource needs for facilitating video-debriefing activity and ongoing maintenance of the program.
Time
Facilitation of team activities1
Facilitating a pre-resuscitation briefing15 min
Reviewing a resuscitation video30-60 min
Facilitating a VD60-90 min
Documentation of team activities-good practice1
Completing pre-resuscitation briefing template5 min
Completing a video review template5 min
Completing a video debrief template5 min
Completing a weekly reporting template5 min
Informing parents about VD activity15 min
Ongoing maintenance
Training instructors-once2 h
Training instructors to ensure reliable review and debriefing-once2 h
Scheduling instructors and booking rooms1-2 h/mo
Trouble shooting equipment, deleting videos and ensuring confidentiality of patients and caregivers1-2 h/mo
Reporting system issues to quality councils and ensuring appropriate training for candidates lacking skills1-2 h/mo
Addressing system issues and implementing solutionsVariable
Feasibility

The study results showed, no caregiver refusal to participate in VR; VR performed on 60% of occasions and less than 10% caregivers’ perceiving a negative impact on team performance. Thus criteria for the feasibility of VR and VD of NR in the delivery room intervention were met.

Sustainability

On completing the NHS sustainability and model guide during the focus group, a score of 32 was obtained for implementing the intervention in the current form. A score of less than 45 indicates the need for adaptations. Further discussions about changes in the intervention to ensure sustainability and continued accrual of benefits generated a list of adaptations (Table 8).

Table 8 Issues affecting sustainability and suggested adaptations.
ThemesIssue affecting sustainabilitySuggested adaptations
ResourcesSame day video-debriefing is resource intensive Transfer of ownership from project lead to unit leadership helps in buy-inConduct video-debriefing once every 2 wk on selected resuscitation recordings. Make the debriefing sessions open to all caregivers
Provide resources for scheduling instructors, maintaining technology, and compensate for instructors time and effort Consider VR all deliveries and team members to seek video-debriefing on selected cases by attending on service/call
Low rate of VRHesitation to voluntarily record and participate in VRChange from caregiver activated recording to motion sensor activated VR Link to certification/competence assessment (caregivers/learners)
System to remedy identified latent safety threats in real timeIdentified but unaddressed issues result in caregiver disengagementSet timelines for action
Support caregivers to take ownership on addressing issues
Communicate actions arising out of identified issues
Establish connections with Quality and Education committees for systematic training on frequently identified issues
Inability to assess impact of team actions during resuscitationLack of vitals (heart rate, SpO2) data on VRConsider superimposing vitals data on video-recording
Inconsistency in demonstrating team behaviors by caregiversLack of focused training in team behaviors and error preventionSustain video-debriefing activity for creating learning and self evaluation
Team behavior evaluation not mandatory for maintenance of professional accreditation or trainee certificationIntegrate demonstration of team behaviors during resuscitation into professional accreditation and certification requirements
Unintended adverse and beneficial effects

VR and review was helpful in assessing adoption of gentle resuscitation practices, adequacy of team preparation and documentation of facts on charts. Self-reflection of behaviors allowed caregivers to focus on improving their deficiencies and reinforcing their good practices. Thirty percent of caregivers reported that VR had a positive impact on their performance. No issues about maintenance of infant or caregivers’ confidentiality and erasure of videos were found. We did not receive any request for VR by parents and we did not take consent from parents for using VR for purposes other than NRQAA. Challenges observed during piloting are provided in Table 9.

Table 9 Challenges identified during video recording and debriefing program.
Challenges
Reminding care providers about team huddle, debriefing or turn-on the VR
Conflict with my other work (e.g., NICU service, etc.)
Reviewing videos in time
Engaging care providers during debriefing
Reviewing videos recorded in night
Providing feedback to caregivers who could not attend debriefing
Completing team huddle, video review and debrief templates in time
Completing the weekly report template
Interpreting the audio to assess team communication
Identifying debrief issues arising from videos for debriefing
Time for VD (60 min)
Informing parents about VD in a timely manner, once the VR has happened
Delay in implementing project on time
Reinforcing expectations, providing opportunities for learning especially with rapid turnover of caregivers and trainees in a tertiary centre
DISCUSSION

Our study has shown that it is feasible to adopt VR and VD of NR in DR as a standard of care quality assurance activity to enhance caregivers’ learning and create opportunities that improve patient safety. Despite its complexity and implementation challenges, the intervention was acceptable, implementable, and potentially sustainable with adaptations in a real world setting. To date, no such feasibility study has been reported.

Strengths of this study include methodology and potential for generalizability. Feasibility objectives and criteria, setting and constraints, intervention and implementation strategy were well defined. Validated instruments for evaluation and frameworks for design and implementation were used. Detailed results including lessons learned and mitigation strategies for challenges were found. We believe that such a comprehensive and rigorous approach may allow other centers in making informed decisions on the type of technology, scale of implementation that achieves objectives with limited resources, and adopting strategies that facilitate the application of intervention. Jo Rycroft-Malone has described that “sustained implementation of evidence into practice is a planned facilitated process involving interplay between individuals, evidence, and context to promote evidence-informed practice”[24]. A careful consideration of all above elements was considered in designing and implementing the intervention in our centre. Finally, clarity on the role of organization (enabling and empowering) and project leads (facilitate caregivers’ performing discrete practical tasks during NR) in facilitating implementation played a major role in successful implementation[17].

Based on apriori agreed upon criteria for feasibility, VR of NR in DR is feasible in a tertiary perinatal centre. Caregivers’ perceived enhanced learning and organization’s ability to create opportunities for learning and identify latent safety threats, observed in this study is consistent with other reports. We speculate that VR and VD of NR in DR creates opportunities that facilitate caregiver learning, promotes the interprofessional collaborative practice and creates a mechanism to address latent safety threats affecting patient safety. Integration of this intervention as a resuscitation team routine is likely to enhance caregivers’ delivery of effective, safe and high-quality NR.

Key factors that helped in operationalization of the project in our centre included management support, implementation as a standard of care QA activity as opposed to the research study and ensuring caregivers’ readiness through multiple educational sessions, team behavior workshops and acquaintance with VR.

Our study had limitations. Assessment of impact of VR and VD of NR in DR intervention on patient outcomes, cost effectiveness, and actual team performance were beyond the scope of the study. The results may not be generalizable where centers are planning on adopting a technology or implementation strategy different from ours. Sample size was low and duration of study was short to determine any impact on clinical outcomes. A multivariate regression model to identify major barriers and enablers of adoption of VR in NR could not be performed due to small sample size[25].

A pragmatic cluster randomized trial evaluating the impact of VR of NR on patient and organizational outcomes is warranted. In future, feedback to initiate immediate corrective actions during a NR is desirable[2,26]. Similarly, a provision for printing an objective performance report directly from the monitor that shows response to various actions during NR may eliminate the need for a debriefing instructor[2,26].

It is feasible to adopt VR and VD of NR in delivery room as a standard of care quality assurance activity to enhance caregivers’ learning and create opportunities that improve patient safety. Despite its complexity with inherent challenges in implementation, the intervention was acceptable, implementable, and potentially sustainable. The comprehensive approach and detailed results from our study may allow other centers in making informed decisions on the type of technology, scale of implementation that achieves objectives with limited resources, and adopting strategies that facilitate the application of VR of NR.

ACKNOWLEDGMENTS

We thank the NICU staff for participating and taking ownership during the implementation of the initiative and changing their practices during resuscitation and stabilization. We thank all the neonatologists for supporting this project. We are grateful to Drs. Fusch C and Thomas S for their suggestions on implementing the project. Finally we thank Dr. Thabane for his valuable suggestions on study design and reporting.

COMMENTS
Background

Despite proven benefits video recording (VR) of neonatal resuscitation (NR) is not adopted by all perinatal centres. Major reasons include challenges in operationalization and sustainability. Understanding the enablers and mitigation strategies is crucial on making a decision on widespread adoption of VR of NR by hospitals. The authors conducted a feasibility analysis of introducing VR of NR in the delivery room.

Research frontiers

VR of NR and debriefing has been shown to be an effective tool in enhancing resuscitation team learning and addressing latent safety threats that compromise the quality of resuscitation. In this study, there is a suggestion that a systematic approach to operationalization of this technology helps in widespread adoption in a perinatal centre.

Innovations and breakthroughs

There is evidence of benefits of VR of NR in improving the quality and safety of resuscitation. However there is a gap in literature in understanding and addressing challenges associated with operationalization. Despite its complexity with inherent challenges in implementation, VR of NR was acceptable, implementable, and potentially sustainable with adaptations in real world setting.

Applications

The current study provides a framework for implementation and sustainability for centres considering adopting VR of NR.

Peer-review

This is a well-written article investigating an interesting issue in the neonatal care.

Footnotes

Manuscript source: Invited manuscript

Specialty type: Pediatrics

Country of origin: Canada

Peer-review report classification

Grade A (Excellent): 0

Grade B (Very good): B

Grade C (Good): C

Grade D (Fair): 0

Grade E (Poor): 0

P- Reviewer: Sergi CM, Zhang ZH S- Editor: Ji FF L- Editor: A E- Editor: Lu YJ

References
1.  Kattwinkel J, Short J, Shavell L editors. Text book of neonatal resuscitation. 5th editors. Elk Grove Village: American Academy of Pediatrics 2006; .  [PubMed]  [DOI]  [Cited in This Article: ]
2.  Evaluating Resuscitation Quality. [accessed 2016 Jun 24].  Available from: http://www.resuscitationcentral.com/documentation/resuscitation-quality-cpr-ecc-guidelines/.  [PubMed]  [DOI]  [Cited in This Article: ]
3.  Thomas EJ, Sexton JB, Helmreich RL. Translating teamwork behaviours from aviation to healthcare: development of behavioural markers for neonatal resuscitation. Qual Saf Health Care. 2004;13 Suppl 1:i57-i64.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 1]  [Cited by in F6Publishing: 35]  [Article Influence: 1.8]  [Reference Citation Analysis (0)]
4.  Thomas EJ, Williams AL, Reichman EF, Lasky RE, Crandell S, Taggart WR. Team training in the neonatal resuscitation program for interns: teamwork and quality of resuscitations. Pediatrics. 2010;125:539-546.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 113]  [Cited by in F6Publishing: 116]  [Article Influence: 8.3]  [Reference Citation Analysis (0)]
5.  Wetzel EA, Lang TR, Pendergrass TL, Taylor RG, Geis GL. Identification of latent safety threats using high-fidelity simulation-based training with multidisciplinary neonatology teams. Jt Comm J Qual Patient Saf. 2013;39:268-273.  [PubMed]  [DOI]  [Cited in This Article: ]
6.  Cordero L, Hart BJ, Hardin R, Mahan JD, Nankervis CA. Deliberate practice improves pediatric residents’ skills and team behaviors during simulated neonatal resuscitation. Clin Pediatr (Phila). 2013;52:747-752.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 36]  [Cited by in F6Publishing: 39]  [Article Influence: 3.5]  [Reference Citation Analysis (0)]
7.  Weinstock P, Halamek LP. Teamwork during resuscitation. Pediatr Clin North Am. 2008;55:1011-1024.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 30]  [Cited by in F6Publishing: 30]  [Article Influence: 1.9]  [Reference Citation Analysis (0)]
8.  Gelbart B, Hiscock R, Barfield C. Assessment of neonatal resuscitation performance using video recording in a perinatal centre. J Paediatr Child Health. 2010;46:378-383.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 52]  [Cited by in F6Publishing: 52]  [Article Influence: 3.7]  [Reference Citation Analysis (0)]
9.  Gelbart B, Barfield C, Watkins A. Ethical and legal considerations in video recording neonatal resuscitations. J Med Ethics. 2009;35:120-124.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 39]  [Cited by in F6Publishing: 33]  [Article Influence: 2.2]  [Reference Citation Analysis (0)]
10.  Finer N, Rich W. Neonatal resuscitation for the preterm infant: evidence versus practice. J Perinatol. 2010;30 Suppl:S57-66.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 42]  [Cited by in F6Publishing: 39]  [Article Influence: 2.8]  [Reference Citation Analysis (0)]
11.  Carbine DN, Finer NN, Knodel E, Rich W. Video recording as a means of evaluating neonatal resuscitation performance. Pediatrics. 2000;106:654-658.  [PubMed]  [DOI]  [Cited in This Article: ]
12.  Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655.  [PubMed]  [DOI]  [Cited in This Article: ]
13.  O'Donnell CP, Kamlin CO, Davis PG, Morley CJ. Ethical and legal aspects of video recording neonatal resuscitation. Arch Dis Child Fetal Neonatal Ed. 2008;93:F82-F84.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 44]  [Cited by in F6Publishing: 48]  [Article Influence: 3.0]  [Reference Citation Analysis (0)]
14.  Nadler I, Sanderson PM, Van Dyken CR, Davis PG, Liley HG. Presenting video recordings of newborn resuscitations in debriefings for teamwork training. BMJ Qual Saf. 2011;20:163-169.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 58]  [Cited by in F6Publishing: 53]  [Article Influence: 4.1]  [Reference Citation Analysis (0)]
15.  Roberts KD, Leone TA, Edwards WH, Rich WD, Finer NN. Premedication for nonemergent neonatal intubations: a randomized, controlled trial comparing atropine and fentanyl to atropine, fentanyl, and mivacurium. Pediatrics. 2006;118:1583-1591.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 97]  [Cited by in F6Publishing: 91]  [Article Influence: 5.1]  [Reference Citation Analysis (0)]
16.  Kirkpatrick DL. Techniques for Evaluating Training Programs. VA: ASTD: Alexandria 1975; .  [PubMed]  [DOI]  [Cited in This Article: ]
17.  Stetler CB, Damschroder LJ, Helfrich CD, Hagedorn HJ. A Guide for applying a revised version of the PARIHS framework for implementation. Implement Sci. 2011;6:99.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 234]  [Cited by in F6Publishing: 259]  [Article Influence: 19.9]  [Reference Citation Analysis (0)]
18.  Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, Robson R, Thabane M, Giangregorio L, Goldsmith CH. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol. 2010;10:1.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 1672]  [Cited by in F6Publishing: 1672]  [Article Influence: 119.4]  [Reference Citation Analysis (0)]
19.  Kim J, Neilipovitz D, Cardinal P, Chiu M. A comparison of global rating scale and checklist scores in the validation of an evaluation tool to assess performance in the resuscitation of critically ill patients during simulated emergencies (abbreviated as “CRM simulator study IB”). Simul Healthc. 2009;4:6-16.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 94]  [Cited by in F6Publishing: 102]  [Article Influence: 6.8]  [Reference Citation Analysis (0)]
20.  Debriefing tool. Institute for Medical Simulation, Comprehensive instructor workshop manual handouts. Harvard: Centre for Medical Simulation 2012; 131-132.  [PubMed]  [DOI]  [Cited in This Article: ]
21.  Lund AM. Measuring usability with the USE questionnaire. Usability Interface. 2001;82:2016.  [PubMed]  [DOI]  [Cited in This Article: ]
22.  Fletcher G, Flin R, McGeorge P. Rating non-technical skills: developing a behavioural marker system for us in anaesthesia. Cogn Technol Work. 2004;6:165-171.  [PubMed]  [DOI]  [Cited in This Article: ]
23.  Rycroft-Malone J, Seers K, Chandler J, Hawkes CA, Crichton N, Allen C, Bullock I, Strunin L. The role of evidence, context, and facilitation in an implementation trial: implications for the development of the PARIHS framework. Implement Sci. 2013;8:28.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 171]  [Cited by in F6Publishing: 187]  [Article Influence: 17.0]  [Reference Citation Analysis (0)]
24.  Zhang Z. Model building strategy for logistic regression: purposeful selection. Ann Transl Med. 2016;4:111.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 215]  [Cited by in F6Publishing: 268]  [Article Influence: 33.5]  [Reference Citation Analysis (0)]
25.  Edelson DP, Litzinger B, Arora V, Walsh D, Kim S, Lauderdale DS, Vanden Hoek TL, Becker LB, Abella BS. Improving in-hospital cardiac arrest process and outcomes with performance debriefing. Arch Intern Med. 2008;168:1063-1069.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 345]  [Cited by in F6Publishing: 339]  [Article Influence: 21.2]  [Reference Citation Analysis (0)]