Noria: dynamic, partially-stateful data-flow for high-performance web applications
Jon Gjengset, Malte Schwarzkopf,
Jonathan Behrens, Lara Timbo Araujo, Martin Ek, Eddie Kohler, M. Frans Kaashoek, Robert Morris
In Rust
Jon Gjengset's YouTube:
Background
| id | title |
|---|---|
| 1 | Peloton is the best DBMS |
| 2 | Terrier: Son of Peloton |
| story_id | user |
|---|---|
| 1 | Andy |
| 1 | Lin |
| 2 | Andy |
Story
Vote
SELECT Story.id, Story.title, count(*) FROM Story
JOIN Vote ON Story.id = Vote.story_id
GROUP BY Story.id, Story.title
COUNT
JOIN
What if we query multiple times.
SELECT Story.id, Story.title, count(*) FROM Story
JOIN Vote ON Story.id = Vote.story_id
GROUP BY Story.id, Story.title
Save
Query
Peloton ***, 2
Problems
- Consistency?
- Two clients simultaneously update cache
-
Thundering Herd
- After invaliding the cache
- the ton of queries go to MySQL
| id | title |
|---|---|
| 1 | Peloton is the best DBMS |
| 2 | Terrier: Son of Peloton |
| story_id | user |
|---|---|
| 1 | Andy |
| 1 | Lin |
| 2 | Andy |
Story
Vote
SELECT Story.id, Story.title, count(*) FROM Story
JOIN Vote ON Story.id = Vote.story_id
GROUP BY Story.id, Story.title
COUNT
JOIN
StoryWithVoteCount
Materialized View!
| story_id | count |
|---|---|
| 1 | 2 |
| 2 | 1 |
Operator is stateful
| id | title |
|---|---|
| 1 | Peloton is the best DBMS |
| 2 | Terrier: Son of Peloton |
| story_id | user |
|---|---|
| 1 | Andy |
| 1 | Lin |
| 2 | Andy |
Story
Vote
COUNT
JOIN
StoryWithVoteCount
| story_id | count |
|---|---|
| 1 | 2 |
| 2 | 1 |
| 2 | Chenyao |
|---|
| id | title | count |
|---|---|---|
| 1 | Peloton | 2 |
| 2 | Terrier | 1 |
2
2
Two Challenges
- Limit the size of its state and views
- Adapt the data-flow without downtime
Partially-stateful
data-flow
| id | title |
|---|---|
| 1 | Peloton is the best DBMS |
| 2 | Terrier: Son of Peloton |
| story_id | user |
|---|---|
| 1 | Andy |
| 1 | Lin |
| 2 | Andy |
Story
Vote
COUNT
JOIN
StoryWithVoteCount
| story_id | count |
|---|---|
| 1 | 2 |
| 2 |
WHERE id = 2
| id | title | count |
|---|---|---|
| 1 | Pelton | 2 |
| 2 |
| id | title |
|---|---|
| 1 | Peloton is the best DBMS |
| 2 | Terrier: Son of Peloton |
| story_id | user |
|---|---|
| 1 | Andy |
| 1 | Lin |
| 2 | Andy |
Story
Vote
COUNT
JOIN
StoryWithVoteCount
| story_id | count |
|---|---|
| 1 | 2 |
| 2 | 1 |
WHERE id = 2
| id | title | count |
|---|---|---|
| 1 | Pelton | 2 |
| 2 |
| id | title |
|---|---|
| 1 | Peloton is the best DBMS |
| 2 | Terrier: Son of Peloton |
| story_id | user |
|---|---|
| 1 | Andy |
| 1 | Lin |
| 2 | Andy |
Story
Vote
COUNT
JOIN
StoryWithVoteCount
| story_id | count |
|---|---|
| 1 | 2 |
| 2 | 1 |
WHERE id = 2
| id | title | count |
|---|---|---|
| 1 | Pelton | 2 |
| 2 | Terrier | 1 |
Dynamic data-flow
| id | title | autor |
|---|---|---|
| 1 | Peloton is the best DBMS | Andy |
| 2 | Terrier: Son of Peloton | Tianyu |
| 3 | Poker Night | Andy |
| story_id | user |
|---|---|
| 1 | Andy |
| 1 | Lin |
| 2 | Andy |
| 3 | Chenyao |
Story
Vote
COUNT
JOIN
StoryWithVoteCount
UserKarma
| user | karma |
|---|---|
| Andy | |
| Tianyu |
Just new an operator with empty states!
Lazy Approach!
| id | title | autor |
|---|---|---|
| 1 | Peloton is the best DBMS | Andy |
| 2 | Terrier: Son of Peloton | Tianyu |
| 3 | Poker Night | Andy |
| story_id | user |
|---|---|
| 1 | Andy |
| 1 | Lin |
| 2 | Andy |
| 3 | Chenyao |
Story
Vote
COUNT
JOIN
StoryWithVoteCount
UserKarma
| user | karma |
|---|---|
| Andy | |
| Tianyu |
2
evmap (Shadowing)
| id | title |
|---|---|
| 1 | Peloton is the best DBMS |
| 2 | Terrier: Son of Peloton |
| story_id | user |
|---|---|
| 1 | Andy |
| 1 | Lin |
| 2 | Andy |
Story
Vote
COUNT
JOIN
StoryWithVoteCount
Readers
Writer
root
Map
Copy #1
Map
Copy #2
A = 4
A = 4
A = 5
Readers
Writer
root
Map
Copy #1
Map
Copy #2
A = 4
A = 5
A = 5
Features&Implementation
- Wrote In Rust
- MySQL Protocol Compatible
- By MySQL adaptor
- RocksDB as BaseTable
- Eventually Consistent
- Distributed
- hash-partitioning each operator on a key assign to different instances.
-
all the instances hold same data-flow graph
- No Transactions Support (in paper)
- They claim the design is compatible with OCC
Evaluation
Question & Thoughts
- I don't know materialized view before
- many many people use memcached to address this (including me)
- So I think this is an amazing idea
