Trainee report dashboard

Richard J. T. Gorniak, MD, Adam E. Flanders, MD and Richard E. Sharpe Jr, MD, MBA
Radiographics, Aug 29, 2013

trainee report feedback

  • Report construction important part of residency training

  • How to quantify reporting performance?
    • evaluation is subjective
    • criteria for good reports vary widely

  • Individualized report feedback from attendings occurs inconsistently

  • Trainees unlikely to consistently check their finalized reports for differences manually

trainee report feedback


Styles of feedback for reports:

  • Contemporaneous supervision
    • Daytime workflow, attending giving realtime feedback


  • Delayed feedback
    • On call workflow, attendings (sometimes) giving feedback hours later

dashboard components


  • Windows server running XAMPP, using Apache, MySQL, PHP,  Python, and ASP

  • Components:
    • Discrepancy Logger

    • Report Comparator

    • Report Difference Chart

discrepancy logger


  • Javascript plugin using PACS API (iSite 3.6, Philips)
    • button in PACS interface loads case metadata (time, date, modality, subspecialty, and accession) and allows comments, choosing RadPeer classification.
    • emails trainee directly and logs info in MySQL DB

report comparator


  • CRON job scripted in Perl
    • queries mirrored RIS for prelim reports, saves in MySQL DB
    • separate queries for finalized reports, matches and saves both versions in DB

  • Allow logging in to web service to view diffs
  • Authors' DB contains 1 million reports

report difference chart


  • Python script calculates Levenshtein distance between prelim and final reports
  • Normalizes distance for longer reports by calculating percentage

  • Allows visualization of data in various formats, e.g.
    • Trainees can then view the average # of changes between trainees globally and for each attending-trainee pair
    • Trainees ranked in chart so each trainee can track their performance relative to their peers
    • Color coded to highlight outliers

figures





  •  Average Lev dist % for trainee-attndg pair on top, avg % for particular study type on bottom
  • Trainees ordered by performance across top of chart, better performance to the right


  • Individual reports for each attending/trainee pair


Advantages

  • Streamlines feedback workflow, esp when not in close proximity

  • Encourages feedback by making process less onerous

  • Allows trainees to gauge relative performance
    • can drill down to specific poorly performing modalities

  • Possible use as objective measure of trainee competency
    • Showed decrease in Levenshtein percentage in cohort of residents over time

disadvantages

  • Many sources of differences between reports
      • Dictation software mistakes
      • Attending wording preferences
      • Mishearing attending during readout
      • Disagreement in interpretation
      • New clinical information following preliminary report

    • Makes assumption that more changed words equates with poorer report quality
      • cannot distinguish the clinical significance of a change, and a one word change may have high significance (e.g. inserting "no", or changing L3 --> L4)
    Made with Slides.com