Deep dive into Meteor's reactivity model

...and why is DDP Router so exciting!

@radekmie

About me

@radekmie
  1. Meteor developer for over 10 years.
  2. Meteor Core Commiter (PRs).
    • Also Blaze (async support).
  3. Head of Engineering at aleno.

Everything is at radekmie.dev.

(Slides for this presentation too.)

@radekmie

Agenda

  1. What happens between collection.insert(...) and a rerun of collection.find(...).fetch()?
  2. Why cultofcoders:redis-oplog scales better than tailing oplog?
  3. Why changestream-to-redis makes it even better?
  4. Why I think DDP Router is exciting?
@radekmie

Client

Server

MongoDB

MongoDB

1. method

4. added

2. insert

3. oplog

DDP

  • Method simulation.
  • Optimistic insert.
  • Method execution.
  • Observe driver.
  • Observe driver.
  • "Mergebox" logic.
  • Processes all
    oplog entries.
  • Minimongo.
  • Tracker.

Oplog

@radekmie

Client

Server

MongoDB

1. method

5. added

2. insert

3. publish

DDP

MongoDB

Redis

Redis

4. subscribe

Redis
Oplog

  • Observe driver (Redis-based).
  • Some overhead
    from Redis inserts.
  • Processes only
    the published collections.
@radekmie

Client

Server

MongoDB

1. method

6. added

2. insert

DDP

MongoDB

Redis

Redis

Redis
Oplog
(external publisher)

changestream-to-redis

3. subscribe

4. publish

5. subscribe

  • Zero overhead
    from Redis inserts.
  • Some latency.
@radekmie

Client

Server

MongoDB

1. method

5. added

3. insert

DDP

MongoDB

DDP
Router

4. subscribe

DDP Router

2. method

  • Observe driver extracted into a dedicated service.
  • Less latency.
@radekmie

Comparison

Processes only published collections

Oplog

Redis Oplog

DDP Router

⚠️

✅

✅

See oplogExcludeCollections
and oplogIncludeCollections
for a way to manually select what to process and what to skip.

@radekmie

Comparison

Processes only published collections

Processes only published documents

Oplog

Redis Oplog

DDP Router

⚠️

✅

✅

❌

❌

⚠️

See "On Optimizing Meteor Publications" to learn how changestream-to-redis can use custom channels.

@radekmie

Comparison

Processes only published collections

Processes only published documents

Refetches full documents more often than you would think

Oplog

Redis Oplog

DDP Router

⚠️

✅

✅

❌

❌

⚠️

🔥

🔥

🔥

@radekmie

It was all a lie...

All the diagrams are missing one crucial
operation: fetching data from MongoDB.

Both oplog- and Redis-based observe drivers can handle only some operations directly. Everything else requires a full refetch.

It can be mitigated by using protectAgainstRaceConditions.
It may be worse with large documents.

@radekmie

...and we can't do better

users.find({}, {
  limit: 3,
  sort: { score: -1 },
})

User X (score: 42)

User Y (score: 37)

User Z (score: 21)

Scenario 1

User A (score: 69)

User should be added
to the result set.

✅

Scenario 2

User X (score: 1)

User should be removed
from the result set.

🔥

@radekmie

Takeaways

  1. Meteor reactivity model is a lot of pieces tied together to make the magic happen.
  2. My recommended approach to scaling would be:
    1. Oplog tailing (Meteor's default).
    2. Redis Oplog.
    3. Redis Oplog + changestream-to-redis.
  3. While DDP Router is not actively worked on right now, I believe this approach is a viable one for the future of Meteor.

Questions?

@radekmie