Who AM I

Developer, developer, developer!

at EXECOM

MICROSOFT STUDENT PARTNER

at Faculty of Technical Sciences

How can you contact me?

Follow Me on Twitter

@gdjonovic

How can you contact me?

SEND ME AN EMAIL

goran.djonovic@studentpartner.com

gdjonovic@execom.eu

first Lets

Explain kinect

Explain kinect

WHAT IS KINECT?

Explain kinect

YES, KINECT IS THE SENSOR

but what's in it?

how does it work?

Explain kinect

Explain Kinect

EXPLAIN KINECT

Depth data

EXPLAIN KInect

I don't care about tech spec I want to use it in an App

EXplain Kinect

USE THE KINECT SDK

but which one?

EXPLAIN KINECT

EXPLAIN KINECT

KINECT SKDs:

1. Kinect SDK 1.0
2. Kinect SDK Beta
3. Kinect SDK Beta 2
4. Kinect SDK 1.5 (1.5.1, 1.5.2)
   
5. Kinect SKD 1.6 (4 days ago)

... and some/many other hacks 

eXPLAIN KINECT

Kinect for xBox 360 SDKs (before 1.5) was

All about skeletal data

... and it was fun

Explain Kinect

EXPLAIN Kinect

Kinect for xBox 360 SDKs (before 1.5)

• Track two users
• On distance from 0.8 to 2 meters
  
• No gesture support
• No near mode
• No face recognition
• Academic use only

... and it was fun

EXPLAIN Kinect

Kinect SDK 1.5 (Kinect for Windows, February 21)

AND NOW IT'S COOL

EXPLAIN Kinect

Kinect SDK 1.5 (Kinect for Windows)

• Kinect studio (praise the lord)
• Human interface guidelines

• Face tracking SDK

• Samples and Doc (again praise the lord)
• Improved skeleton tracking 
• Performance and data quality enhancements
• The Joint Orientation (Hierarchical Rotation
  Absolute Orientation)
• Speech recognition improvement
  

explain KINECT

Explain Kinect

IS KINECT STILL MYSTERY?

And now, do you want to

Develop kinect apps

THE FIRST and MOST IMPORTANT STEP IN SOFTWARE development is?

IDEA, IDEA, IDEA, IDEA!

Plato: An archetype of which a corresponding being in phenomenal reality is an imperfect replica.
Kant: A concept of reason that is transcendent but nonempirical.
Hegel: Absolute truth; the complete and ultimate product of reason.

Guidelines

Be aware of technical abilities and limitations

Guidelines

Be aware of technical abilities and limitations

Guidelines

Avoid direct sun light

Guidelines

        Match your user/client needs and mindset.

...other

Guidelines

• Follow trends. Get to know what is already there 
• Work with people with from different backgrounds and cultures
• Any idea is potential gold. Don't trow it right away, polish it, work on it
  

while(true){

DEVELOP -> EVALUATE -> IMPROVE

}

Recent (THE BEST) kinect event

TANDEM WORKSHOP

in Budapest

30 highly motivated students from 13 different countries

8 days

7 teams

7 great ideas and countless more

 TANDEM WORKSHOP BEST IDEA

New interactions

• Use finger to point 
• Use legs to slide
• Use characteristic moves
• Suggestions

And of course...

Emotion analyzer

First there were hits/clicks, then we got likes, but now we can have true emotions.

ARE YOU INTERESTED IN HOW TO DO IT?  :)

Ok, Lets talk about code

FACE RECOGNITION

and analyzing

How to start

• Download Kinect SDK 1.5 and development toolkit

You will get

• FaceTrackData.dll
• FaceTrackLib.dll

C++ application can directly use the face tracking libraries

In the other case, you should use a wrapper

Starting projects

Use:

Initialize Sensor and face tracking

kinectSensor.ColorStream.Enable();
kinectSensor.DepthStream.Range = DepthRange.Near;
kinectSensor.DepthStream.Enable(rez80X60);
kinectSensor.SkeletonStream.EnableTrackingInNearRange = true;
kinectSensor.SkeletonStream.TrackingMode = SkeletonTrackingMode.Seated;
skeletonData = new Skeleton[6];
kinectSensor.Start();
faceTracker = new FaceTracker(kinectSensor);

Getting face frame

void kinectSensor_AllFramesReady(object sender, AllFramesReadyEventArgs e)
{
    //you must track face on every frame
}

using (ColorImageFrame colorImageFrame = e.OpenColorImageFrame())
{
    if (colorImageFrame == null)
        return;
    colorImageFrame.CopyPixelDataTo(colorPixelData);
}
 
using (DepthImageFrame depthImageFrame = e.OpenDepthImageFrame())
{
    if (depthImageFrame == null)
        return;
    depthImageFrame.CopyPixelDataTo(depthPixelData);
}
 
using (SkeletonFrame skeletonFrame = e.OpenSkeletonFrame())
{
    if (skeletonFrame == null)
        return;
    skeletonFrame.CopySkeletonDataTo(skeletonData);
}

var skeleton = skeletonData.FirstOrDefault(
s => s.TrackingState == SkeletonTrackingState.Tracked);
if (skeleton == null)
    return;
// only track face if all data are there


FaceTrackFrame faceFrame = faceTracker.Track(
kinectSensor.ColorStream.Format, colorPixelData,
kinectSensor.DepthStream.Format, depthPixelData,
skeleton);

Get3DShape()

This method will return 87 face  points in 3D

GetProjected3DShape()

returns the same collection but in a 2D referential corresponding to a 640×480 pixels image.

GetTriangles()

is a method that you can use in association with the Get3DShape method. It will return you a collection of triangles.

Get Animation Unit Coefficients()

returns the values describing the facial expressions.

Jaw lowerer

Lip Corner Depressor

Upper Lip raiser

Lip Stretcher

Outer Brow Raiser

Neutral face

AND NOW THE DEMO

Q&A

ANY QUESTIONS?

THANK YOU ALL