Ryerson IT Conference 2018

Ahmed Sagarwala

@amsagarwala • sagarwala@ryerson.ca

Custom-Built Blog Assessment and Participation Tracking Software

Overview

• Why I built this system?
• Where I started... and am heading.
• Expose setbacks in working with Twitter, D2L, and blogs.
• How lean methodology kept things moving.
• The benefits in automating course administration.
• Maybe a demo?

Guiding Principles

• Machines should not make the final decision.
• Students should learn how this system works.
• Gaming the system is a good thing. It often reveals a deep understanding and is a creative application. It's at the highest level of Bloom's taxonomy.
• Strategically use friction.

Think

• Does this system keep grading equitable?
• Do I focus on the right aspects of grading?
• What would you add/remove?
• The dangers of over-automation?
• Could it do _______?

Set the Stage

EID 100

Digital Skills & Innovation for the Global Economy

Image: Robert Laughter

EID100 Assessments

• 25% Modules (blog posts)
• 20% Group project
• 5%   Exam reviews
• 10% Online + In-class participation
• 50% Midterm and final exams*

1 Semester

2,982 Tweets

118 Students

1,070 Modules

834 Participation

1,052 Review

The 5 stages (weeks) of
course administration*

* based on the Kübler-Ross model

Image: Nik Shuliahin

Week 1: Denial

120 blog posts... I can handle it.

Image: Oziel Gómez

Week 2: Anger

Another 120 blog posts.

I need more TAs!!!

Image: Gabriel Matula

Image: Mosa Moseneke

Giphy: Drunk

Giphy: Walk Away

Down the Rabbit Hole

Adventures in Automation
Iteration #1

Giphy: Rabbit Hole, Disney

Pros

• Less time spent manually checking accounts
• Works well for <100 students
• Time invested in setup ∝ administration time
• Smooth transition

Cons

• Some reliability issues with picking up tweets
• Feedback loop (posting grades)
• Providing feedback was tedious
• Coordinating several loosely connected systems
• Reliance on 3rd-party services
• Students gamed the online participation algorithm
  during the final 2 weeks of class

Setback #1

IFTTT and Zapier cannot access private Twitter accounts.

• Manually check tweets
• Enter grades manually
• Students wait longer for grades

Setback #2

Twitter handles can be changed...

Several students changed usernames midway through.

• Track previous and current handles
• Manually reconcile marks
• Override code in cells

Iteration #2

Grading Dashboard

Don't rely on IFTTT & Zapier

Giphy: Power Up

Initiatives

1. Course registration via Google Forms
2. Grading dashboard for tweets
3. Pickup new tweets every 5 minutes
4. D2L export (CSV format)

4% improvement
on exams

Open Calais

• Developed by Thompson Reuters, free to use
• Classifies content
• Provides quick analysis of content
• Reveals depth vs. breadth of blog post
• Potential for data to be used to pre-grade

Pros

• No time spent manually checking accounts
• Works well for any number of students (scalable)
• No reliance on 3rd-party services
• Great for tracking online participation
• Private tweets are accessible!

Cons

• Time invested in setup > administration time
• Full reliance on system
• Lots of testing required + oversight

Setback #1

Students with multiple accounts mix things up

• Manually changed database content
• Support requests got in the way of grading
• Needed to develop workflow for dealing with issues:
  Simply solved by tweeting the same content with correct account used for the course

Setback #2

Encoding (utf8_bin vs. utf8_general_ci)

• Case-sensitivity caused lots of hard to solve issues
• #Module1 ≠ #module1 ≠ #MODULE1
• Usernames and emails weren't matching
• Students told me grades weren't adding up

Iteration #3

Level up

Automate everything!

Giphy: Viceland

Initiatives

1. Course registration via Google custom form
2. EID100 Bot
3. D2L class roster import
4. D2L export by exam review, participation, etc.
5. QR Codes!
6. Student report for auditing

4% improvement
on final exam

Writing improved within 2 weeks

• Associates Twitter user ID with email (not their handle)
• Not everyone reads the instructions :(
• Uses Twitter for future logins
• No credentials stored (security is simplified)

EID100 Bot

• Sends confirmations:
  • Course registrations
  • Receipts for modules
  • Grades + feedback
• All communication via direct messages to keep feedback private
• Writing improved for most students within 2-weeks
• Higher quality posts

Setbacks

• bit.ly links = no content
• Certain media types not loading (embeds)
• Misses some important checks
  • Plagiarism
  • Citations
• New feedback options need to be coded

Iteration #4

Coming soon!

Improve student engagement

Giphy: Future

Data Visualizations

Images: Emilia Zibaei, MDM Candidate

Turalt's Email Edge (YouTube)

If you're late

the machine will grade you.

Thanks to

• MDM: Alex Ferworn, Michael W. Carter, Dean Mactavish, Lissa Quaglia, Emilia Zibaei, Sam Legros
• Jaigris Hodson, Guy Hoskins
• Richard Adams, GCM
• Wendy Freeman, eLearning
• James Nadler & Creative Industries
• Sephanie Goetz & DMP
• Namir Ahmed & Fangmin Weng
• Gosha Zywno & LTO team
• School of Performance
• Linux & Apache Foundations

• Turalt
• Reuters
• Akindi
• Twitter
• Buffer
• IFTTT & Zapier
• Parlay Ideas
• PHP community
• Kahoot
• Giphy
• Slides.com
• Mozilla
• Swivl