Trainee report dashboard
Richard J. T. Gorniak, MD, Adam E. Flanders, MD and Richard E. Sharpe Jr, MD, MBA
Radiographics, Aug 29, 2013
trainee report feedback
- Report construction important part of residency training
- How to quantify reporting performance?
- evaluation is subjective
- criteria for good reports vary widely
- Individualized report feedback from attendings occurs inconsistently
- Trainees unlikely to consistently check their finalized reports for differences manually
trainee report feedback
Styles of feedback for reports:
- Contemporaneous supervision
- Daytime workflow, attending giving realtime feedback
- Delayed feedback
- On call workflow, attendings (sometimes) giving feedback hours later
dashboard components
- Windows server running XAMPP, using Apache, MySQL, PHP, Python, and ASP
- Components:
- Discrepancy Logger
- Report Comparator
- Report Difference Chart
discrepancy logger
- Javascript plugin using PACS API (iSite 3.6, Philips)
- button in PACS interface loads case metadata (time, date, modality, subspecialty, and accession) and allows comments, choosing RadPeer classification.
- emails trainee directly and logs info in MySQL DB
report comparator
- CRON job scripted in Perl
- queries mirrored RIS for prelim reports, saves in MySQL DB
- separate queries for finalized reports, matches and saves both versions in DB
- Allow logging in to web service to view diffs
- Authors' DB contains 1 million reports
report difference chart
-
Python script calculates Levenshtein distance between prelim and final reports
- Normalizes distance for longer reports by calculating percentage
- Allows visualization of data in various formats, e.g.
- Trainees can then view the average # of changes between trainees globally and for each attending-trainee pair
- Trainees ranked in chart so each trainee can track their performance relative to their peers
- Color coded to highlight outliers
figures
-
Average Lev dist % for trainee-attndg pair on top, avg % for particular study type on bottom
- Trainees ordered by performance across top of chart, better performance to the right
- Individual reports for each attending/trainee pair
Advantages
-
Streamlines feedback workflow, esp when not in close proximity
-
Encourages feedback by making process less onerous
-
Allows trainees to gauge relative performance
-
can drill down to specific poorly performing modalities
- Possible use as objective measure of trainee competency
- Showed decrease in Levenshtein percentage in cohort of residents over time
disadvantages
- Dictation software mistakes
- Attending wording preferences
- Mishearing attending during readout
- Disagreement in interpretation
- New clinical information following preliminary report
-
Makes assumption that more changed words equates with poorer report quality
- cannot distinguish the clinical significance of a change, and a one word change may have high significance (e.g. inserting "no", or changing L3 --> L4)
Trainee report dashboard
By Jason Hostetter
Trainee report dashboard
- 173