Traditional Poster Round
Background: : Simulated Neonatal Resuscitation Program (SimNRP) scenarios allow practice of advanced resuscitation based on the NRP algorithm including timed interventions, and cognitive and technical skills. Accurate instructor assessment of team performance is critical for simulation debriefing and education improvement.
Research Question: : To develop a novel software tool and to evaluate the accuracy of instructor assessment of learner performance during SimNRP scenarios using the tool versus gold standard video review.
Methodology: : This is the first iterative design cycle. Texas Children's Hospital NRP instructors (n=14) reviewed published Neonatal Resuscitation Performance Evaluation versions and the 7th edition NRP critical performance steps. A modified Delphi methodology with serial surveys determined initial skills and timed interventions to include in the software tool, NeoSTART.
Using NeoSTART, instructors recorded timed interventions during SimNRP scenarios. Team cognitive and technical skills were scored between the simulation and team debrief. Instructors completed usability and cognitive load surveys. Real-time instructor assessments were compared to video.
Results: : Seven time intervals between 2 interventions (Table 1), 7 cognitive skills, and 27 technical skills were included.
Baseline Instructor Assessment Accuracy for 23 SimNRP scenarios:
Although only 69% of time intervals were captured, time error was ≤ 15 seconds for 92% of intervals compared to video. For 22 SimNRP scenarios with skills captured, cognitive was more accurate than technical skill assessment (129/154 (84%) vs 201/334 (60%), p< 0.001) (Table 2).
Instructor surveys had a System Usability Scale score of 82.5 (> 90th percentile). The National Aeronautics and Space Administration Task Load Index cognitive load surveys showed low physical demand (11.1), and frustration (17.9) but still significant mental demand (41.4), temporal demand (48.9), and effort (43.2) with average performance (48.2).
Discussion/Conclusions: : Assessment of learner performance by instructors during SimNRP scenarios lacks accuracy compared to video. Significant mental and temporal demand may be associated with decreased accuracy. App improvement cycles will continue until skill scoring accuracy is ≥ 90% and ≥ 80% of timed intervals are ±15 seconds of video. App improvement cycles will simplify timed intervention event capture and identify causes of inaccurate skill assessments.