Log In

Forgot Password?
Create New Account

Loading... please wait

Abstract Details

The Narrative Evaluation Quality Instrument: Development of a Tool to Assess the Assessor
Research Methodology, Education, and History
P6 - Poster Session 6 (12:00 PM-1:00 PM)
Recent work has shown that narrative-based evaluations can be used as a valid form of assessment, providing equal or better reliability than numeric scores with regard to neurology clerkship grading. However, reliability is contingent on assessors’ abilities to complete high quality narratives, and published work examining narrative quality is limited.
Determining the quality of narrative evaluations to assess medical student neurology clerkship performance remains a challenge. This study sought to develop a tool to comprehensively and systematically assess quality of student narrative evaluations.
The Narrative Evaluation Quality Instrument (NEQI) was created informed by existing literature to assess three components within clerkship narrative evaluations: (1) performance domains commented on, (2) specificity of comments, and (3) usefulness to trainee. In this retrospective study, five investigators scored 123 de-identified narrative evaluations from 53 unique assessors using the NEQI. Inter-rater reliability was estimated for each NEQI component arm and total NEQI score with intra-class correlation coefficients (ICC), and bootstrapped 95% confidence intervals (CI) were calculated.
A total of 615 NEQI scores were obtained, and the average overall NEQI score was 6.4 (SD = 2.9), with mean component arm scores of: 2.6 for performance domains (SD = 0.9), 1.8 for specificity (SD = 1.1), and 2.0 for usefulness (SD = 1.4). Assessors commented on an average of 4.7 (SD = 1.7) performance domains per narrative evaluation. Each component arm exhibited moderate reliability: performance domains ICC = 0.65 (95% CI: 0.58 – 0.72), specificity ICC = 0.69 (95% CI: 0.61 – 0.77), and usefulness ICC = 0.73 (95% CI: 0.66 – 0.80). Overall NEQI score exhibited good reliability (0.81; 95% CI: 0.77 – 0.86).
The NEQI is a novel, reliable tool to comprehensively assess the quality of narrative evaluation of neurology clerks, and will enhance the study of interventions seeking to improve clerkship evaluation.
Robert I. Thompson-Stone, MD, FAAN (University of Rochester)
Dr. Thompson-Stone has received personal compensation in the range of $5,000-$9,999 for serving as a Consultant for Conway Homer PC.
No disclosure on file
No disclosure on file
Melanie Braun, MD (University of Rochester Medical Center) Dr. Braun has nothing to disclose.
Justin Rosati, MD (University of Rochester Medical Center) Dr. Rosati has nothing to disclose.