Category: Education, Simulation & Virtual Reality

MP33-11 - Understanding entrustment-based assessment in the operating room for a urology training cohort

Sun, Sep 23
2:00 PM - 4:00 PM

Introduction & Objective : Entrustment-based assessments are increasingly being used for measuring residents’ performance in the operating room (OR). However, little is known regarding how staff surgeons come to their final entrustment decisions and whether their perceptions of their own required intraoperative involvement correlate with final entrustment scores. This study sought to evaluate the correlation between “observable entrustment of the resident” (defined as the frequency and level of involvement of the staff surgeon during the case) and the final entrustment-based assessment score. 


Methods : Over a 6-month period, 2 researchers directly observed 15 ureteroscopy and laser lithotripsy cases performed by a urology resident under guidance of a staff surgeon. Extensive independent field notes of every case were taken, outlining staff and resident interactions stepwise throughout the procedure. Immediately following each case, the staff surgeon assessed the resident using the 11-item validated entrustment-based Ottawa Surgical Competency Operating Room Score Evaluation (O-SCORE). The field notes were transcribed and coded, with the frequency of staff suggestions, directions, and takeover events collated for each case. Correlations were calculated to compare frequency of these events per case with the means of the total O-SCORE and subscores for the intraoperative domains.


Results : A total of 630 minutes of intraoperative observations were collected.  The study involved 7 residents and 5 staff surgeons. Six cases were completed by junior residents and 9 by senior residents. Results of the total O-SCORE correlated poorly with the observed number of takeovers by staff surgeons (r=0.056) and with intraoperative domain subscores (knowledge of steps, technical performance, visuospatial skills, and efficiency and flow) (r=0.198). The correlation was higher for subscores in cases where providing suggestions and directions was observed (r=0.294).


Conclusions : The measure of staff entrustment from the validated O-SCORE results correlated poorly with observed staff surgeon takeovers during resident-involved ureteroscopy and laser lithotripsy cases. Active staff participation such as taking over during a case may not be a reflection of resident ability to perform cases independently. Similarly, an O-SCORE indicating entrustment may not reflect the true ability for independent practice. Additional mixed methods analyses are being conducted, and a larger study is underway to evaluate entrustment and its role in competency-based assessment in multiple types of urological cases.

Yuding Wang

Department of Surgery, Division of Urology, McMaster University
Hamilton, Ontario, Canada

Edward D. Matsumoto

McMaster University
Hamilton, Ontario, Canada

Dana Russell

McMaster University
Hamilton, Ontario, Canada

Jen Hoogenes

Clinical Research Coordinator
McMaster University
Hamilton, Ontario, Canada

Jen Hoogenes, MS, MSc, PhD; Clinical Research Coordinator, Department of Surgery, Division of Urology, McMaster University in Hamilton, Ontario, Canada.

Kelly Dore

McMaster University
Hamilton, Ontario, Canada