Interactions through non-verbal communication
2024
KArts, Seoul
This workshop will explore design, technology, and cross-cultural aspects, using p5.js and Teachable Machine AI.
The objective for this workshop is to apply Creative Code and Machine Learning Models to create forms informed by participants cultural context that change in response to user actions in a playful way.
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
This workshop will explore design, technology, and cross-cultural aspects, using p5.js and Teachable Machine AI.
The objective for this workshop is to apply Creative Code and Machine Learning Models to create forms informed by participants cultural context that change in response to user actions in a playful way.
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
Andreas Schlegel
Andreas’ practice stretches across various disciplines often informed by design, art, technology and computation.
He is currently a Senior Lecturer at the School of Design Communication at Lasalle, University of the Arts Singapore. He heads Lasalle's Media Lab, where many of his collaborative works have been shaped and presented as exhibitions, performance, workshop or on screen.
In this work he creates artifacts, tools and interfaces where art, design and technology meet in curious ways.
Who is
Design, Technology, Cultures.
web
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
What interests me in my work?
Interactivity
Exploratory character of code
Endless possibilities
Collaboration
Working across disciplines
Adaptability
Design, Technology, Cultures.
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
In today's technology-driven world, much of our communication takes place through digital devices such as our mobile phones or laptops.
Other scenarios such as autonomous assistive robots, smart home ecosystems, gesture-enabled entertainment systems, digital fitness coaches are less common, but are expected to become more prominent companions in our daily lives in the future.
In such cases, language and verbal communication as we know it may not be the main form of interaction after all? Instead, non-verbal communication may increase as we interact with these new systems within our cultural context. How do we need to design for such a future scenario?
Design, Technology, Cultures.
Workshop context
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
What defines a culture? How to define a culture? How to understand or get to know a culture better?
Through scientific measures, conversations, curiosity, play?
Thought process
Interactions through non-verbal communication
2024
KArts, Seoul
What defines a culture?
Search prompt: Culture
Search prompt: Cross-culture
Search prompt: Cultures
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
What defines a culture?
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
What defines a culture?
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
The Country Comparison Tool is an online tool based on Geert Hofstede's Cultural Dimensions Theory, a framework for cross-cultural psychology. It looks at 6 dimensions of national cultures:
Power Distance
Individualism
Motivation towards Achievement and Success
Uncertainty Avoidance
Long Term Orientation
Indulgence
How to define a culture? An example for a scientific approach, the Country Comparison Tool.
Hierarchy
Autonomy
Ambition
Stability
Persistence
Gratification
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
This dimension deals with the fact that all individuals in societies are not equal - it expresses the attitude of the culture towards these inequalities amongst us.
In Germany a direct and participative communication and meeting style is common, control is disliked and leadership is challenged to show expertise and best accepted when it’s based on it.
In Singapore power is centralized and managers rely on their bosses and on rules. Employees expect to be told what to do. Control is expected and attitude towards managers is formal. Communication is indirect and the information flow is selective.
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
The dimension Uncertainty Avoidance has to do with the way that a society deals with the fact that the future can never be known: should we try to control the future or just let it happen?
In Singapore people abide to many rules not because they have need for structure but because of a high Power Distance Index.
At 85 South Korea is one of the most uncertainty avoiding countries in the world. There is an emotional need for rules (even if the rules never seem to work), time is money, people have an inner urge to be busy and work hard, precision and punctuality are the norm, innovation may be resisted, security is an important element in individual motivation.
Rules: states the way things are or should be done
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
This dimension is defined as the extent to which people try to control their desires and impulses, based on the way they were raised. Relatively weak control is called "Indulgence" and relatively strong control is called "Restraint".
South Korean society is shown to be one of Restraint. in contrast to Indulgent societies, Restrained societies do not put much emphasis on leisure time and control the gratification of their desires.
German culture is Restrained in nature. Societies with a low score in this dimension have a tendency to cynicism and pessimism. their actions are restrained by social norms and feel that indulging themselves is somewhat wrong
Restrained: very calm and unemotional. Indulge: to allow yourself to have or do something that you like, especially something that is considered bad for you
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
How to define a culture?
Cultures can be defined by data that aims to find a generalised truth that (some may not agree with, but) often provides a general overview and insights into a culture.
Thought process
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
How to understand or get to know a culture better?
Through conversation, observation, curiosity, play?
Thought process
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
Can we consider cultures that go beyond human societies?
Can we consider cultures that emerge from technology, for example through machine learning and code?
Thought process
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
How can we bring technology and design into the conversation to play and interact with and through machines?
Thought process
Design, Technology, Cultures.
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
How can we use non-verbal communication to interact with machines, both within and across cultures?
Through physical presence, objects, poses, gestures?
Challenge
The System
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
p5js and Teachable Machine
Interactions through non-verbal communication
2024
KArts, Seoul
The System
The objective for this workshop is to apply Create Code (p5.js) and Teachable Machine Models to create forms informed by participants cultural context that change in response to user actions in a playful way.
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
We will use the p5js library to respond to trained classes detectable by Teachable Machine tool.
Find below the link to a template that we will use to interact with your own Teachable Machine model and to start with, use the testable model provided.
p5.js sketch
Duplicate and run the receive-sketch first from the link below, then press b to open and run the pose-send sketch to send data over.
Design, Technology, Cultures.
1
Interactions through non-verbal communication
2024
KArts, Seoul
A
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
B
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
C
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
D
Design, Technology, Cultures.
Image detection
Optional
Pose detection
Default template that receives data from the default Teachable Machine Pose Model, contains comments
Same as above but no comments only code
Sketch to load and capture data from a Teachable Machine Pose Model
Responds to the same data as above but with more appealing visuals
Interactions through non-verbal communication
2024
KArts, Seoul
Teachable Machine is a web-based tool that makes creating machine learning models fast, easy, and accessible to everyone.
For this workshop, we will look at training the pose model option, but the image model can also be used to recognize specific cultural artifacts, for example.
Teachable Machine
Let's try to train a model with the Teachable Machine tool. We can choose between 3 options: Image, Audio and Pose, you can find some linked tutorials below.
Design, Technology, Cultures.
2
E
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
Did you manage to run both exercises with p5js and Teachable Machine successfully?
Duplicate and run the receive-sketch first from the link below, then press b to open and run the pose-send sketch to send data over.
1
Let's try to train a model with the Teachable Machine tool. We can choose between 2 options: Pose and Image.
2
Recap Session 1
Interactions through non-verbal communication
2024
KArts, Seoul
Session 2
Design, Technology, Cultures.
Spoke about design, technology and culture
Tried out Teachable Machine Pose and Image models
Trained a custom Teachable Machine model using multiple classes
Tried p5js template sketches to use Teachable Machine model as input
Duplicated two p5js sketches to send and receive Teachable Machine data, and you made changes to code
✓
✓
✓
✓
✓
Tasks for Session 2
Interactions through non-verbal communication
2024
KArts, Seoul
Session 2
Design, Technology, Cultures.
Team up with 1 or 2 other workshop participants to form a group
Discuss and decide on a simple application for a pose or image model that addresses a cultural aspect of your choice
Decide on 2, 3 or 4 classes as input, preferably keep the class labels short
Train and test your model, then run it with your duplicated p5js sketches
Modify code to detect your model's class labels and code a visual response for each class
Design different visual behaviors for each class of your Teachable Machine model
Ask for feedback if unsure
Ask for help if unsure
Group work
Code work
Interactions through non-verbal communication
2024
KArts, Seoul
Default template that receives data from the default Teachable Machine Pose Model, contains comments
Same as above but no comments only code
Sketch to load and capture data from a Teachable Machine Pose Model
Responds to the same data as above but with more appealing visuals
Design, Technology, Cultures.
Image detection
Optional
Pose detection
List of prepared sketches to start from
Interactions through non-verbal communication
2024
KArts, Seoul
Session 2
Design, Technology, Cultures.
How can we use non-verbal communication to interact with machines, both within and across cultures?
Challenge
On a piece of paper, start by writing down 3 cultural aspects that are important to you and your group
Select one cultural aspect, how can it be expressed visually? Sketch this out on paper before moving on to code and the teachable machine tool
Discuss the classes you want to train
Discuss and decide on a simple application for a pose or image model that addresses a cultural aspect of your choice
Sketch an image for each class of your teachable machine model so that it is easier to visualise how to translate this into code
Session 3
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
meet at the exhibition space at 10am
미술원 B103 전시실(조형예술과)
Set up and test your project
Demonstrate and share project
Documentation
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
The Framework
Interactions through non-verbal communication
2024
KArts, Seoul
Design, Technology, Cultures.
Session 1
Session 2
Presentation
Interactions through non-verbal communication
2024
KArts, Seoul
scenario
train
behavior
iterate & test
functionality
narrative
experience
observe & discuss
The Framework
Design, Technology, Cultures.
Interactions through non-verbal communication
2024
KArts, Seoul
- Duplicate and run p5js sketches
- Train a model with Teachable Machine
- Manage to run your model together with the duplicated p5js sketches
When uncertain, please ask for help
Steps to take
The Framework
Conceptual and contextual considerations
Design, Technology, Cultures.
- Writte down and sketch out ideas for your work (Scenario, Train, Behavior)
- Change the p5js-receive-sketch to the needs of your idea
- test your creation
Scenario
Session 1
Interactions through non-verbal communication
2024
KArts, Seoul
A scenario is an imagined situation that describes how users might interact with a system. It helps to better understand what needs to be design, for whom, and should help as an outline for expected outcomes.
Each participant will define a scenario to help establish a cultural context they wish to work with, see examples on the right for consideration and guidance
- Non-verbal communication with machines
- Make the system respond to specific gestures commonly used in daily routines
- Create a virtual artifact that gives comfort or cheers up their audience
- Create a series of cultural symbols that appear on screen for certain actions performed
The Framework
Conceptual and contextual considerations
Then, draw out and write down some ideas on paper so we can discuss them further. Ideas should be kept simple, minimal but effective.
Design, Technology, Cultures.
3
Train
Session 1
Interactions through non-verbal communication
2024
KArts, Seoul
In order for the system to recognize gesture based inputs, we need to train it. In our case we will use Google's Teachable Machine.
Teachable Machine is a web-based tool that makes creating machine learning models fast, easy, and accessible to everyone.
Follow the link to the right or navigate downwards for more details on pose and gesture, then train your own model.
The Framework
Conceptual and contextual considerations
Design, Technology, Cultures.
4
Session 1
Interactions through non-verbal communication
2024
KArts, Seoul
Difference between Pose and Gesture
Pose
Poses are generally static, involving a particular stance or position held for a period of time and they can represent a broader range of attitudes, states, or identities rather than specific messages: A confident stance with hands on hips, a bow as a sign of respect, a slouched posture indicating disinterest or sadness.
Gesture
Gestures are typically dynamic, involving movement and are used to convey emotions, intentions, or specific messages: Waving goodbye, nodding for yes, shaking the head for no, thumbs up for approval.
Design, Technology, Cultures.
Behavior
Interactions through non-verbal communication
2024
KArts, Seoul
Based on the scenario defined earlier and the model trained, how will your idea translate visually on a screen?
Design two modes of behavior, one that represents the inactive mode when no interactions are happening and the other when the system responds to interactions detected.
Consider your coded and visual options based on your technical skills and understanding.
Session 1
Note down your behavior ideas on paper, this can be in bullet points format, keywords, full sentences and very simple drawings and scribbles.
The Framework
Conceptual and contextual considerations
Design, Technology, Cultures.
5
Iterate & Test
Interactions through non-verbal communication
2024
KArts, Seoul
After all the hard work and having establish your own responsive system, try and test it. Based on observations and conversations make changes to arrive at your final outcome.
Session 1 and Session 2
At this point you should have accomplished the following
- Duplicate p5js sketches
- Train a model with Teachable Machine
- Manage to run your model together with duplicated the p5js sketches
- Writte down and sketch out ideas for your work (Scenario, Train, Behavior)
- Change the p5js-receive-sketch to the needs of your ideas
The Framework
Conceptual and contextual considerations
Design, Technology, Cultures.
6