Introduction: Automated performance metrics (APMs) objectively measure surgeon performance during a robot-assisted radical prostatectomy (RARP). Machine learning (ML) has shown that APMs, especially during the vesico-urethral anastomosis (VUA) of the RARP, are predictive of long-term outcomes such as continence recovery time. This study focuses on APMs during the VUA, on stitch versus sub-stitch levels, to distinguish surgeon experience.
Methods: During the VUA, APMs, recorded by a systems data recorder (Intuitive Surgical), were reported for each overall stitch (Ctotal) and its individual components: needle handling/targeting (C1), needle driving (C2), and suture cinching (C3) (Figure 1A). These metrics were organized into three datasets (GlobalSet [whole stitch], RowSet [independent sub-stitches], and ColumnSet [associated sub-stitches] (Figure 1B) and applied to three ML models (AdaBoost, Gradient Boosting, and Random Forest) in order to solve two classifications tasks: experts (= 100 cases) vs. novices (<100 cases); and ordinary-experts (OE; =100 but < 2000 cases) vs. super-experts (SE; = 2000 cases). Classification accuracy was determined using analysis of variance (ANOVA). Input features were evaluated for the stability of their importance to each classification task through a Jaccard index.
Results: From 68 VUAs, we analyzed 1,570 stitches broken down into 4,708 sub-stitches. ColumnSet, where sub-stitch APMs were provided as related to its sub-components, consistently produced the highest accuracy across all ML models (p<0.003). For both classification tasks, AdaBoost trained with ColumnSet best distinguished experts (n=8; median: 855 cases) vs. novices (n=9; median: 18 cases) and OE (n=5; median 168) vs. SE (n=3; median: 2000 cases) at an accuracy of 0.774 and 0.844, respectively. Stable feature importance scores highlighted Endowrist® articulation and needle handling/targeting (C1) APMs as most important for classification.
Conclusions: Surgeon performance measured by APMs on a granular sub-stitch level more accurately distinguishes expertise when compared to summary APMs over whole stitches. Wrist articulation and needle handling/targeting APMs were the most important features for accurate experience classification. Source of
Funding: This study was funded in part by an Intuitive Surgical Clinical Grant; Intuitive Surgical provided the systems data recorder. Research reported in this publication was supported in part by the National Institute Of Biomedical Imaging And Bioengineering of the National Institutes of Health under Award Number K23EB026493.