Category: Dissemination / Implementation

Symposium

Psychometric Assessment of Three Newly Developed Implementation Outcome Measures

Friday, November 17
12:45 PM - 1:45 PM
Location: Indigo Ballroom B, Level 2, Indigo Level

Keywords: Implementation | Measurement
Presentation Type: Symposium

Implementation outcome (IO) measures are essential for monitoring and evaluating the success of implementation efforts and comparing the effectiveness of implementation strategies. However, measures lack conceptual clarity and have questionable reliability and validity. We developed and psychometrically assessed 3 new IO measures:  acceptability, appropriateness, and feasibility. First, 36 implementation scientists and 27 mental health professionals assigned 31 items to the constructs, rating their confidence in assignments. We used the Wilcoxon one-sample signed rank test to assess substantive and discriminant content validity. Exploratory and confirmatory factor analysis (EFA and CFA) and Cronbach ɑ assessed the validity of our conceptual model. Next, 326 mental health counselors read one of six randomly assigned vignettes. Participants used 15 items to rate therapist’s perceptions of the acceptability, appropriateness, and feasibility of adopting an EBP. We used CFA and Cronbach ɑ to refine the scales, assess structural validity, and assess reliability. Analysis of variance (ANOVA) assessed known-groups validity. Finally, we randomly assigned half of the counselors to receive either the same vignette or the opposite vignette, and re-rate the IOs. Pearson correlation coefficients assessed test-retest reliability and linear regression assessed sensitivity to change. All but 5 items exhibited substantive and discriminant content validity. A trimmed CFA with 5 items per construct exhibited good model fit (CFI = 0.98, RMSEA= 0.08) and high factor loadings (0.79 to 0.94). The ɑ’s for 5-item scales were between .87-.89. Scale refinement based on measure-specific CFAs and Chronbach ɑ’s using vignette data produced 4-item scales (0.85 to 0.91). A 3-factor CFA exhibited good fit (CFI = 0.96, RMSEA = 0.08) and high factor loadings (0.75 to 0.89), indicating structural validity. ANOVA showed significant main effects, indicating known-groups validity. Test-retest reliability coefficients ranged from 0.73 to 0.88. Regression analysis indicated each measure was sensitive to change in both directions. The 3 new measures demonstrate promising psychometric properties.

Bryan Weiner

Professor
University of Washington

Presentation(s):

Send Email for Bryan Weiner


Assets

Psychometric Assessment of Three Newly Developed Implementation Outcome Measures



Attendees who have favorited this

Please enter your access key

The asset you are trying to access is locked. Please enter your access key to unlock.

Send Email for Psychometric Assessment of Three Newly Developed Implementation Outcome Measures