Let's
Change
the
game
As a service, we have decisions to make Every Seond
-
What shall we serve?
-
To whom?
-
when?
What shall we serve this customer?
Orange
or
banana
"Educated" Guess
I like orange so people must like orange too.
Let the customer decide
We have orange and banana, chose whichever you like.
What is orange? What is banana?
Why do I care?
Why gave me so many choices?
Customer:
A/B test!!!
Let's serve orange and banana to equal number of random people. three weeks later... It seems that there are more happy customers served with banana than those served with orange. Banana is awesome, let's serve only banana going forward.
Other drawback of a/B
Oops, I got my statistics wrong. It turns out it's just a coincidence that there are more happy customers served with banana. It's a misinterpretation of the result.
Let's study the customer, maybe she likes orange better when she is thirsty and banana better when she is hungry. We should observe cues and learn from her reactions. By doing so, we may figure out when we shall serve which.
Deep Q Learning
I noticed that the customer likes orange better after she just had banana, and doesn't like having four bananas in a roll. We can build a multi-course serving plan.
More Deep Q Learning
Is it too good to be possible?
What's new in this algorithm
-
deep Convolution Neural Network
-
Memory Pool
-
Stickiness to old models.
Proof of Concept
- Based on a simplified version of DQN
- Learns to play different genres preferred by the user when engaged with different activities.
- Learns to alternate between preferred genres to maximize variety of the music
some Example applications
-
Which recommendation algo/settings to use for this customer now
-
When do we prompt for social sharing, registration, social sharing, etc
-
Auto Equalizer settings
"songs on iheart just sound better!"
(the success of beats headphones) -
Multi-step user registration invitation
Recommendation Models
-
Cues:
-
Location, Time, Accelerometer Reading, Recent Play History
-
-
Actions:
-
which recommendation model to use
-
-
Rewards:
-
Thumbs up, Favoriting, Share, Listening Time
-
Thumbs down, Skips, Stops
-
Prompt Sharing
-
Cues:
-
User share history, recent user interaction
-
-
Actions:
-
To share or not to share
-
-
Rewards:
-
Choose to share
-
Dismissed, App closed
-
Auto equalizer settings
-
Cues:
Acoustic profile, Time, Location, Accelerometer Reading
-
Actions:
Which equalizer preset to use
-
Rewards:
Thumbs up, Fave, Share, Listening Time
Thumbs down, Skip, Volume Down
-
MULTI-STEP USER REGISTRATION INVITATION
A salesperson makes a conversation before asking you the question:
Do you want one?
Introducing Shifu
A production ready scala port of the original DQN algorithm from DeepMind
Shifu
Neural Network
Linear Algebra Lib
Agent
Akka Interface
DB
How Do we start?
A/B/C testing!
- A for strategy A
- B for strategy B
- C for Shifu driven decision between A and B
Get the Data in!
and then