The New MongoDB Rust Driver

Rust NYC

July 22, 2015

Kevin Yeh & Sam Rossi

@kyeahokay | @saghm

What's on the menu?

• MongoDB Driver Architecture
• Borrowing and Concurrency
  • Client Architecture, Arc
  • Connection Pools, Locking
• Traits: Make developers happy
• Unsafety: Ignore rustc and feel good about it
• Recursive Macros: Make everyone happy
• Driver Status
• Building with Rust and MongoDB

Driver Architecture

Bson Encoding

A lightweight binary representation of JSON-like documents.  

• Easily traversable
• Efficiently encodable and decodable
• Supports additional types (BinData, Date)

Wire Protocol

BSON is sent to and from the server.

Each message starts with a header:

Borrowing & Concurrency

Struct Referencing: Looks Good?

Ownership

Rust guarantees speed and safety at runtime by enforcing ownership and lifetimes at compile time.

Since the Client object is borrowed, its lifetime must predictably last at least as long as the database that contains it.

Arc

Atomic Reference Count is your friend.

How do we pass an Arc of self?

Make poor design decisions.

Connection Pools

Traditional Capped Pool

Fixed-length arrays of locks and sockets.

Acquire socket lock, Use socket, Release socket lock.

Traditional Capped Pool

Two options: Mutable or immutable pool.

A mutable connection pool requires a mutex to guarantee thread safety.

• Ordered unlocking: If the pool is locked before the socket, it cannot be released before the socket.

An immutable connection pool requires explicit lock and socket initialization on creation.

• Poisoned socket locks: Permanently kills the connection and limits the pool size.

Separating Mutable and Immutable Components

Fixed-length locks, mutable sockets array.

Acquire lock, extract socket, lock socket, use socket.

Separating Mutable and Immutable Components

Sockets are lazily connected!

but...

• Doubles the number of locks per socket

• Requires some roundabout locking logic
• Lock array is pre-constructed and immutable

Poisoned locks still hinder pool functionality!

Hyper

Explicit extraction of lock-free sockets!

Hyper

The Good:

• Variable number of lazily-connected sockets

• Almost lock-free

• One master lock on the connection pool

The Bad:

The number of open sockets is uncapped.

Recycling vulnerability: Old sockets can be dropped as new ones are constantly made to take their place. 

Our Friend, the Condvar

Master pool lock with variable-length, lock-free sockets.

If no sockets are available and we are capped at the number of open sockets,  wait on the condition variable until we've been repopulated.

Thank you, Condvar.

Traits

High focus for Rust 1.0: Extendability

• Get core features down, provide traits for the community.
• Large portion of core rust features rely on generic traits.
• Flexible functional-style programming in a largely imperative language.  

Conversion Traits

Rust makes it easy to auto-convert types.

Encoder Traits

Encoding traits allow arbitrary structs to be converted.

BSON can be decoded into structs and vice versa without any matching involved.

Error Traits

All errors in rust are handled by Result<T, Err>.

Defining your own Error type is easy!

• Conversion traits automatically coerce different types of errors.
• Error traits provide easy access to coerced error information.

Unobtrusive error handling and control paths.

Dereferencing Traits

Dereferencing Traits

To provide concurrency and safety, Rust relies on a number of wrapping structs: Arc, Box, MutexGuard, Ref, etc. 

An auto-dereferencing system makes this as transparent and user-friendly as possible.

Iterator Traits

Iterate with control.

I/O Traits

Read and write anything.

FFI and Unsafety

When the rust compiler isn't enough.

ObjectId

How do I get the machine id?

How do I save a static generated id?

Driver Status

Donesies

• bson library

  • OID generation

  • stable macro

• connection strings

• wire protocol

• CRUD, commands, and cursors

• bulk writes

• connection pooling

• error handling

• automated testing suites

Up Next

• replica set failover

• server discovery and monitoring (SDAM)

• SCRAM-SHA-1 auth

• shard tagging, indexes, and other server commands