Introduction: Assessment plays a central role in competency-based medical education. The operating room (OR) is a demanding environment with many competing priorities placed on the primary surgeon, one being the assessment of resident performance. Previous studies show that assessment reliability decreases with increased task complexity. The objective was to explore the utility of video-based assessment of resident performance for ureteroscopy and laser lithotripsy.
Methods: Over a 6-month period, 15 ureteroscopy and laser lithotripsy cases performed by urology residents under staff surgeon guidance were captured on video using a 3-camera setup capturing the surgeon’s view, the endoscopic view, and a 360-degree view of the OR. Following each case, the staff surgeon assessed resident performance using the 11-item entrustment-based Ottawa Surgical Competency Operating Room Score Evaluation (O-SCORE), and evaluated his or her own cognitive load during the case using the Surgical Task Load Index (SURG-TLX). Two independent staff urologists reviewed each video using the same assessment tools. Raters’ results were compared those of the case surgeons.
Results: Over 15 cases, 630 minutes of video was collected of 7 residents and 5 staff surgeons. Six cases were completed by junior residents and 9 by senior residents. The interrater reliability between video assessors was highly correlated (k=0.82). Comparisons between intraoperative assessment, video-based assessment, and mean O-SCOREs were also highly correlated (r=0.7). Evaluation of O-SCORE domains for intraoperative performance showed a slight decrease in correlation between the intraoperative assessment and video-based assessment (r=0.68). All assessors (video and staff surgeons) expressed low cognitive load, with mean SURG-TLX scores of 5 (out of a possible 100) (SD ±1.8). Video-based assessors frequently rewound the video during critical steps of each case to evaluate resident performance.
Conclusions: We showed that video-based assessment of resident performance during ureteroscopy and laser lithotripsy using the O-SCORE is useful and feasible, with high interrater reliability among the case surgeon and independent video reviewers. Use of the SURG-TLX provided new insight into the cognitive load of both case surgeons and video assessors. Additional research with other types of surgical cases is required to further explore the use of video-based assessment in the OR. Source of
