Measuring guideline-concordant care when recommendations require clinical judgment: a mixed methods approach

AI Summary

Exploratory sequential mixed methods developed 18 video vignettes from interviews, mapped to guideline indicators and paired with questions and a 0 to 3 scoring rubric.
Tool demonstrated acceptable internal consistency (Cronbach's alpha 0.74) and almost perfect inter-rater reliability (Cohen's kappa 0.92) in a national trial.
Provides a rigorous, context-sensitive and replicable method to evaluate guideline-concordant care when recommendations require nuanced clinical judgement.

JBI Evid Implement. 2026 May 18. doi: 10.1097/XEB.0000000000000603. Online ahead of print.

ABSTRACT

INTRODUCTION: General practitioners (GPs) face challenges in diagnosing and managing work-related mental health conditions. In Australia, this prompted the development of a national clinical guideline (the Guideline) to support high-quality care. However, measuring the implementation of recommendations that require clinical judgment remains methodologically challenging.

OBJECTIVE: This study describes an approach to developing and testing a guideline-specific tool to assess concordance with recommendations in the context of an implementation trial.

METHODS: We used an exploratory sequential mixed methods design. Semi-structured interviews were conducted with 20 individuals with lived experience of work-related mental health conditions. The interview data informed personas and case studies, which were converted into 18 video vignettes simulating GP-patient consultations. Each vignette was mapped to a checklist of clinical indicators from the Guideline and paired with one multiple-choice and two short-answer questions. A scoring rubric (0-3 points per vignette) was developed and pilot-tested with three groups: GPs familiar with the Guideline, GPs unfamiliar with it, and medical/research students. Vignette content and questions were refined for internal consistency before use in a national implementation trial.

RESULTS: Internal consistency was acceptable (Cronbach’s α = 0.74). In the IMPRovE trial, participants completed three vignettes per time point (score range: 0-9). Inter-rater reliability was almost perfect (Cohen’s κ = 0.92) from 873 responses across 97 GPs.

CONCLUSIONS: This study offers a rigorous, context-sensitive method for evaluating guideline-concordant care where recommendations require clinical judgment. Successfully implemented in a national trial, this approach offers a replicable method for evaluating guideline implementation where recommendations require nuanced clinical judgment.

IMPROVE TRIAL REGISTRATION: ACTRN12620001163998, November 2020.

SPANISH ABSTRACT: http://links.lww.com/IJEBH/A597.

PMID:42138289 | DOI:10.1097/XEB.0000000000000603

Document this CPD