Seeing and Hearing

 

an AI’s “Thinking” Process

Digital Media, Arts, and Technology ("DIGIT") @

Digital Media, Arts, and Technology ("DIGIT")

Digital Media, Arts, and Technology ("DIGIT") @

Digital Media, Arts, and Technology ("DIGIT") @

DigitAI Student Developers: Alexander C. Fisher, Hadleigh Jae Bills, Michael Simons

Faculty Mentor & Co-developer: Dr. Elisa Beshero-Bondar, Prof. of Digital Humanities

 

a project under development by 

Profs. Elisa Beshero-Bondar, Tommy Hartung, Joel Hunt, Lauren Liebe

 @

“AI: Mind the Gap” exhibit at the MIT museum

  • visitor draws a face on a touchscreen

  • witnesses light show indicating processing activities

  • a computer classifies the images and reveals text describing the sentiment expressed by the face drawing

  • designed to "slow down" the visitor, and educate about AI via an unfamilar interactive exhibit

 

 

 

💡Inspiration

💡Plan for library interactive exhibit

Signal chain: from neural network to exhibit experience Flow diagram: neural network logits pass through softmax to log-probs, which Ollama sends to our script; math.exp recovers probabilities; Shannon entropy measures spread; uncertainty score drives color, pause, and sound. Neural network raw scores for each token Logits unbounded real numbers Softmax probabilities summing to 1.0 Log-probs Ollama sends us these math.exp( ) recovers raw probabilities Shannon entropy spread of token candidates Uncertainty score normalized to 0–1 Color · pause · sound the exhibit experience inside the model logprob interface our computation exhibit output

How our project works

Walk-in exhibit space: illustrated system diagram Illustrated diagram: visitor figure triggers SLM processing on a laptop, MCP signals activate a stage spotlight, speaker, and projector within a walk-in campus library exhibit space. Walk-in exhibit space campus library VAR room visitor types a prompt Processing local computer small language model (SLM) Ollama · Qwen · Pleias logprob uncertainty signals token-by-token output minimal compute budget runs on lab hardware MCP Sensory outputs immersive experience Lighting Arduino kits · small projectors Sound campus studio theater equipment Projection signals → dramatic visual effects minimize: budget + computation · maximize: space + immersion
—— Uncertainty Monitor ——
"Describe the feeling of standing at the edge of a vast, dark ocean at night..."
● confident ● uncertain ● hesitating
Standing at the edge of a vast, dark ocean at night is like being enveloped by an endless expanse of mystery and silence.
HESITATION POINTS ◄
TOKENSCORETOP ALTERNATIVE
'is' 68.5% ',' 57.5%
'enveloped' 65.5% 'suspended' 22.7%
'endless' 76.8% 'infinite' 22.4%
Llama 3.2 (3B) · threshold 0.55
token-by-token uncertainty
0.55 is envlp endls each bar = one token · — — — threshold (0.55)

1. We select a small language model (SLM)

2. We give the SLM a "foggy" prompt that we think will yield many different possible outputs.

3. As the SLM generates its response, we retrieve uncertainty scores and their distribution spread for the top 5-10 next-tokens.

4. We re-calibrate these scores for visualization and sonification.