Game Master

Exercise 1

30 mins total

Everyone opens their  own set of slides and reads the character backstory.

Remind to not click ahead.

1 min

Client explains the problem to NLP engineer.

Annotator is not present at this meeting.

(Annotator pretends to not hear anything.)

Client starts a timer for 3 minutes

NLP engineer explains the problem to annotator.

Annotator looks at the data on their laptop.

Remind them to make a copy of the sheet!

They annotate for 10 mins while the client pretends they are not there (in reality, they observe silently out of character)

Remind the annotator here to start their 6 min timer - their exams are coming up! Start it yourself as well.

Annotator exams timer beeps

You ask them their exam question  "What is Foucault's view on the discourse of the riddle of the sphinx?".

Annotator rolls the dice.

Odd - start the 1 minute timer. Stop annotating for 1 mins and google the answer.

Even - Say to the client that the annotator decided to  study for their exams alongside this project.  Ask them to keep annotating. 

Clients  10 minute timer beeps

Client says time is up.

NLP engineer asks for change of scope.

Client rolls the dice. Odd - no change. Even - agree to change.

Client gives final 5 mins to NLP enginer to finish up. 

You should also time 5 mins here.

Clients  5 minute timer beeps

Client says time is up again.

Asks to see the annotations

NLP engineer shows the annotation Google Sheet.

Maybe Client asks some questions about them.

Go to next slide

Clients  asks

"Are we doing AI?"

NLP engineer rolls the dice. Odd - deep learning. Even - truth, just data prep.

Client rolls the dice. Odd - pissed off that not AI. Even - appreciates honest communication.

THE END of Exercise 1.

Go to next slide.

Small group discussion

10 mins total.

First stick to the questions, will have time for freeform feedback in the end

Ask to come out of character, please!

Then discuss in your small group

• What was the most frustrating part of the process for you?
• What kind of communication difficulties have you experienced?​
• How did it feel being a team member in this exercise?
• Have you experienced a similar dynamic in your project?
• Has anyone asked you if you were doing AI?

Whole room discussion 

10 mins

"Is there anything that was interesting in your group that you think this room will benefit learning from?

If no one from your group says anything, then you should speak up and share something interesting that came up in your group

Lecture on extracting lessons learned, and intro to part 2.

GM gets a break.

5 mins

Exercise 2 

Switch the roles around!

Clockwise:

Annotator

Client

NLP engineer

about 30 minutes total

Exercise 2 

Make sure everyone opens the Exercise 2(it says part II on the tab of the slides) version of their slides

Everyone reads their backstory

Annotator is a respected professional now. Yay!

Client reads the problem description

It is almost the same, but slightly different.

Now we have the business owner. Exciting!

Client explains the problem (2 min timer)

Let them start explaining it to just the NLP engineer.

They will probably ignore the annotator.

The annotator reminds them that they are in the room and Client should talk to them too.

If after 1 minute they don't , poke the annotator to say it.

Initial exploration of data

Client has timer: 5 mins

Annotator leads this part.

Make them focus on missing fields: number of employees and name of owner.

Initial exploration of data

Client has timer: 5 mins

Annotator leads this part.

Make them focus on missing fields: number of employees and name of owner.

Stop them from talking about cafe/bakery/discussion now. They will have time for it later

Negotiation time!

Client 5 min timer beeps.

NLP Engineer leads here asking to remove number of employees and name of owner.

Client agree to dropping number of employees but keeps the owner name.

Again, stop them from talking about cafe/bakery/discussion now. They will have time for it later

NLP engineer leads category definition

NLP Engineer starts 5 min timer. 

Make sure they don't rush through it, but do it one rule at a time.

NLP engineer has to present it, client has to carefully consider then accept of course.

Annotator destroys each one of them.

The process has to be difficult for the NLP engineer.

Negotiation again

NLP Engineer asks to postpone the business categorisation till next sprint

This time we just do Information Extraction for business owner name, business name and postcode

Automated Suggestions

NLP Engineer explains how the model and UI  will speed up the annotation.

You, the GM, start a 5 minute timer here.

If your group struggles even with your help to understand the Annotation UI then ask Agata, Bhargav, Laurence or Lev for help.

Automated Suggestions

NLP Engineer explains how they created useful suggestions that will speed up the annotation.

The model suggests candidates for Postcode, Business Name, Business Owner

The slides are exactly the same as you saw on the big screen in the break. This is just repetition to make the participants "own it".

They might struggle as this is the most technical part of the workshop. Help them!

Annotation UI explained

"What am I looking at?"

It is expected that the participants don't understand the UI at all at first.

"Where does the text come from?"

It comes from the URL shown at the bottom right corner of the snippet.

"Why is it a small chunk?"

It is to make you focus and not miss anything

Annotation UI 

The process should be:

1. Make sure all the labels for Postcode, Business Name, Business Owner are correct. Click on the X to remove the label. Add the label by highlighting words.

2. Click the green button to save.

3. If you made a mistake, click Backspace to go back.

If there is no information, then just click the green button even if no yellow in the text.

Client starts the 10 min timer to do the annotations

Keep annotating till you run out of time or do 150 snippets provided. Look at the UI progress counter on the left

If your UI stops before that, choose a different port.

Whole room summary and end of project