Kirk Haines

khaines@engineyard.com

RubyKaigi 2016

Web Server Concurrency

https://github.com/engineyard/rubykaigi2016-concurrency

Kirk Haines

• Rubyist since 2001

• First professional Ruby web app in 2002

• Dozens of sites and apps, and several web servers since

• Engine Yard since 2008

• Former Ruby 1.8.6 maintainer     (EOL announced at RubyKaigi 2011!!!)

khaines@engineyard.com

      @wyhaines

"A Shell Server for HTTP"

(slightly modified from the original)

http://info.cern.ch/hypertext/WWW/Provider/ShellScript.html

That's Terrifying

i.e. Don't Actually Do This

(super_simple.sh)

A (scary, limited, blocking, single threaded) web server!

Web Server Architecture

There is nothing special about web server architecture.

It is just server architecture.

A web server is nothing more than a server that receives HTTP requests and returns HTTP responses, typically through standard socket communications.

Basic TCP Server

Short Tangent

Simplest Ruby Web Server....

Short Tangent

Simplest Ruby Web Server....

Short Tangent

Simplest Ruby Web Server....

OK....That's kind of cheating.

Short Tangent

OK.... Server Architecture

• Setup

      • Command line arguments

      • Configuration

• Listen on socket(s)

• Enter main loop

      • Accept socket connection

      • Handle connection

Listen on Sockets

AKA Network Communications

A server needs a way to receive requests and to return responses.

The two most common options:

• Native Ruby Networking Support (TCPServer and friends)
• EventMachine

Listen on Sockets

Native Ruby Networking Support

Ruby has a rich set of networking libraries, making it easy to write TCP clients and servers.

A Simple Ruby Web Server

Scrawls

# gem install scrawls

( Scrawls is currently woefully incomplete; send me PRs! )

http://github.com/wyhaines/scrawls

Scrawls

Pluggable IO engines

Pluggable HTTP parsing

Shared core makes it easy to use to look at the impact of different concurrency options -- only the concurrency implementation changes between runs.

Scrawls

Scrawls

Pluggable IO engines

• Single Threaded
• Multiprocess
• Multithreaded (+ Multiprocess)
• Evented (pure Ruby implementation)

Main Loop

i.e. Concurrency Options

How a server handles concurrency is fundamental to it's design.

The simplest approach is this:

• Single threaded
• Blocking -- server waits until a request is handled before accepting and handling a new one​

Let's Try Some Things

Test server is an 8 core VM running Ubuntu 16.04

All examples ran with Ruby 2.3.1

Apache Bench (ab) used as a simple benchmarking tool

Single Threaded Example

scrawls --ioengine single​

    Single threaded, blocking IO

ab -n 100000 -c 1 http://127.0.0.1:8080/test.txt

Single Threaded Example

scrawls --ioengine single​

    Single threaded, blocking IO

3614 request/second?  

That's not too bad. Right?

Single Threaded Example

scrawls --ioengine single​

    Single threaded, blocking IO

The devil is in the details.

• Fast response (a small text file)
• Little network latency

Single Threaded Example

The real world is slow, though.

• Query databases
• Interact with microservices
• Generate content
• Assemble it all
• Return it over slow networks

This all takes time.
Web servers spend a lot of time waiting on other things.

Single Threaded Example

scrawls --ioengine single​ --app slow.rb

    Single threaded blocking IO, with app that takes one second to generate a response

ab -n 20 -c 1 http://127.0.0.1:8080/slowtest.txt

Main Loop

Blocking Single Threaded Server

How To Address This?

Concurrency!

Multiprocessing

Multithreading

Event Based

How To Address This?

Concurrency!

Concurrency: decomposability of a problem into order independent or partially ordered units.

i.e. chunks of work can happen independently of each other.

Where possible, we also like them to happen at the same time, or to at least look like they happen at the same time.

Main Loop

Multiprocessing

Just run a bunch of blocking servers, and have something else distribute and balance the load to them.

Load Balancer

Blocking Server

Blocking Server

Blocking Server

CONCURRENCY! WINNING!

Main Loop

Multiprocessing

Just run a bunch of blocking servers, and have something else distribute and balance the load to them.

Pros

Cons

• Simple to implement.
• Performance can still be quite good.

• Managing processes can be complex.
• Limited sharing of resources can be expensive.

Main Loop

Listen on a port, then fork.

A child processes share opened ports. OS load balances.

YMMV depending on OS.

Multiprocessing Simple Blocking Server

Main Loop

Multiprocessing Simple Blocking Server

scrawls --ioengine multiprocess --processes 8

    Single threaded blocking IO, across 8 processes

ab -n 100000 -c 100 http://127.0.0.1:8080/test.txt

Main Loop

Multiprocessing Simple Blocking Server

scrawls --ioengine multiprocess --processes 8 --app slow.rb

    Single threaded blocking IO, across 8 processes, slow app

ab -n 80 -c 8 http://127.0.0.1:8080/slowtest.txt

Main Loop

Multiprocessing Simple Blocking Server

scrawls --ioengine multiprocess --processes 32 --app slow.rb

    Single threaded blocking IO, across 8 processes, slow app

ab -n 320-c 32 http://127.0.0.1:8080/slowtest.txt

Main Loop

Multiprocessing Simple Blocking Server

Just like when shopping; more lines == get done more quickly

Main Loop

Multiprocessing Simple Blocking Server

But....it uses a lot of resources:

Main Loop

Multiprocessing Simple Blocking Server

With slow requests, multiprocessing with blocking servers still often feels like this.

Main Loop

Multiprocessing Simple Blocking Server

For most requests, each process still spends most of it's time waiting on something else to happen.

This can use a lot of RAM while still not being able to handle many slow requests at once because each slow request still blocks an entire process.

Fortunately.....

Main Loop

Ruby pre 2.0 was very copy-on-write unfriendly.

Multiprocessing consumed large amounts of RAM.

Modern Rubies are more resource friendly when forking.

Multiprocessing

Simple Blocking Server

Copy On Write

The Abbreviated Version

When a process is forked, the OS keeps an array of shared resources between the parent and the child.

If the child doesn't change anything, those shared resources don't have to be duplicated.

Pre-2.x MRI Rubies touched all objects during the mark phase of garbage collection, forcing the OS to make private copies. Forking was very expensive, as a result. Modern MRI Rubies behave better.

Main Loop

Multithreading

A thread is the smallest sequence of instructions that can be managed independently by the scheduler. Multiple threads will share one process's memory.

Pros

Cons

• Threading implementations vary a lot across Ruby implementations.
• Locking issues on shared resources can be complicated. i.e. Threads are difficult and it's easy to make mistakes!

• Easier to manage/load-balance in a single piece of software.
• Threads are lightweight, so resource usage is generally better.
• Can be very performant.

Main Loop

Multithreaded Server

Programming with threads can easily be a talk all by itself.  A few quick guides and tutorials:

• https://blog.engineyard.com/2013/ruby-concurrency
• http://www.csinaction.com/2014/10/10/multithreading-in-the-mri-ruby-interpreter/
• https://www.toptal.com/ruby/ruby-concurrency-and-parallelism-a-practical-primer
• http://www.tutorialspoint.com/ruby/ruby_multithreading.htm

Main Loop

Multithreaded Server

Simple, naive implementation - a new thread for every request, and assume everything else just works with this.

Main Loop

Multithreaded Server

 scrawls --ioengine multithread

   Multithreaded IO, single process

ab -n 100000 -c 100 http://127.0.0.1:8080/test.txt

Main Loop

Multithreaded Server

 scrawls --ioengine multithread

   Multithreaded IO, single process

ab -n 100000 -c 100 http://127.0.0.1:8080/test.txt

Requests per second: 2955.78

For fast requests, the overhead of a thread per request hurts speed on Ruby 2.3.1.

A more sophisticated implementation would use a fixed size pool of threads. The scrawls-ioengine-multithread gem doesn't support thread pools yet, though.

Main Loop

Multithreaded Server

 scrawls --ioengine multithread --app slow.rb

   Multithreaded IO, single process, slow app

ab -n 1000 -c 100 http://127.0.0.1:8080/slowtest.txt

Main Loop

Multithreaded Server

Slow requests scale pretty well with threads. Diminishing returns when thread count gets high when there is no thread pool, but it's not bad for such a trivial implementation.

Main Loop

Multithreaded Server

 scrawls --ioengine multithread --processes 8

   Multithreaded IO, multiple processes

ab -n 100000 -c 100 http://127.0.0.1:8080/test.txt

Main Loop

Multithreaded Server

 scrawls --ioengine multithread --processes 8

   Multithreaded IO, multiple processes

ab -n 100000 -c 100 http://127.0.0.1:8080/test.txt

Spreading requests across multiple processes, that are each multithreading, mitigates some of the losses from the Global Interpreter Lock in a single process on Ruby 2.3.1

Main Loop

Multithreaded Server

 scrawls --ioengine multithread --processes 8 --app slow.rb

   Multithreaded IO, multiple processes, slow app

ab -n 10000-c 100 http://127.0.0.1:8080/slowtest.txt

Main Loop

Event Driven Server

 "Event Driven" is a vague label, encompassing numerous patterns and feature sets. One of the most common of these patterns is the Reactor pattern.

The Reactor pattern describes a system that handles asynchronous events, but that does so with synchronous event callbacks.

Main Loop

Event Driven Server

Client/Server interactions are often slow, but most of that time is spent waiting on latencies. CPUs are fast. The rest of the world is pretty slow.

Main Loop

Event Driven Server

An event reactor just spins in a loop, waiting for something to happen - such as a network connection, or data to read or to write.

When it does, an event is triggered to deal with it.

Events block the reactor.

Main Loop

Event Driven Server

Pros

Cons

• Slow callbacks block the reactor.
• Callback structured often used with evented programming can be confusing and difficult to test and debug.

• Can be very fast and resource friendly.
• With an appropriate underlying event notification facility, can scale to thousands of simultaneous connections.

Main Loop

Event Driven Server

Like Threading, this could easily be a talk all by itself. A few resources for further reading:

• https://blog.engineyard.com/2011/what-are-events-why-might-you-care-and-how-can-eventmachine-help
• https://github.com/eventmachine/eventmachine
• http://www.bigfastblog.com/rubys-eventmachine-part-1-event-based-programming
• https://github.com/celluloid/celluloid-io/

Main Loop

Event Driven Server

Many ways to do it, including EventMachine, Celluloid.io, or even a simple pure ruby event framework (SimpleReactor).

Currently, there is a simple event-based IO Engine for Scrawls, which is based on SimpleReactor, a pure Ruby Reactor implementation.

Main Loop

Event Driven Server

 scrawls --ioengine simplereactor

   Evented IO, single processes

ab -n 10000-c 100 http://127.0.0.1:8080/test.txt

Main Loop

Event Driven Server

 scrawls --ioengine simplereactor --app slow.rb

   Evented IO, single processes, slow app

ab -n 20 -c 1 http://127.0.0.1:8080/slowtest.txt

Main Loop

Event Driven Server

It looks a lot like a blocking single threaded server.

However, where events shine is in dealing with high communications latencies.

They don't block on IO, so an efficient reactor can service large numbers of high latency connections efficiently.

Main Loop

Event Driven Server

Evented IO handling keeps things happy at high concurrencies, even across the country. For example, this is from an Engine Yard AWS Oregon instance talking to a non-AWS VM on the east coast.

Main Loop

Event Driven Server

And if your event notification framework is up to it, concurrencies can scale nicely.

Fast Responses

Main Loop

Event Driven Server - Hybridization!

Main Loop

Event Driven Server - Hybridization!

Evented IO can also mix well with threading, letting the slow stuff be slow, while not blocking the reactor from spinning and servicing things that are ready to be serviced.

Main Loop

Event Driven Server

Slow (1 second delayed) Responses

Main Loop

Event Driven Server

Reactor/Event IO combined with threading is a great combination, if you are willing to deal with the complexity of implementation.

Even pure Ruby + Ruby 2.3 is pretty fast

Parsing HTTP - A Quick Note

The very simple examples so far have cheated with this.  

HTTPRecognizer is being used to handle HTTP. It uses regular expressions. Going too far down that path will drive you to madness and sorrow.

Parsing HTTP

You can't parse [X]HTML with regex. Because HTML can't be parsed by regex. Regex is not a tool that can be used to correctly parse HTML. As I have answered in HTML-and-regex questions here so many times before, the use of regex will not allow you to consume HTML. Regular expressions are a tool that is insufficiently sophisticated to understand the constructs employed by HTML. HTML is not a regular language and hence cannot be parsed by regular expressions. Regex queries are not equipped to break down HTML into its meaningful parts. so many times but it is not getting to me. Even enhanced irregular regular expressions as used by Perl are not up to the task of parsing HTML. You will never make me crack. HTML is a language of sufficient complexity that it cannot be parsed by regular expressions. Even Jon Skeet cannot parse HTML using regular expressions. Every time you attempt to parse HTML with regular expressions, the unholy child weeps the blood of virgins, and Russian hackers pwn your webapp. Parsing HTML with regex summons tainted souls into the realm of the living. HTML and regex go together like love, marriage, and ritual infanticide. The <center> cannot hold it is too late. The force of regex and HTML together in the same conceptual space will destroy your mind like so much watery putty. If you parse HTML with regex you are giving in to Them and their blasphemous ways which doom us all to inhuman toil for the One whose Name cannot be expressed in the Basic Multilingual Plane, he comes. HTML-plus-regexp will liquify the n​erves of the sentient whilst you observe, your psyche withering in the onslaught of horror. Rege̿̔̉x-based HTML parsers are the cancer that is killing StackOverflow it is too late it is too late we cannot be saved the trangession of a chi͡ld ensures regex will consume all living tissue (except for HTML which it cannot, as previously prophesied) dear lord help us how can anyone survive this scourge using regex to parse HTML has doomed humanity to an eternity of dread torture and security holes using regex as a tool to process HTML establishes a breach between this world and the dread realm of c͒ͪo͛ͫrrupt entities (like SGML entities, but more corrupt) a mere glimpse of the world of reg​ex parsers for HTML will ins​tantly transport a programmer's consciousness into a world of ceaseless screaming, he comes, the pestilent slithy regex-infection wil​l devour your HT​ML parser, application and existence for all time like Visual Basic only worse he comes he comes do not fi​ght he com̡e̶s, ̕h̵i​s un̨ho͞ly radiańcé destro҉ying all enli̍̈́̂̈́ghtenment, HTML tags lea͠ki̧n͘g fr̶ǫm ̡yo​͟ur eye͢s̸ ̛l̕ik͏e liq​uid pain, the song of re̸gular exp​ression parsing will exti​nguish the voices of mor​tal man from the sp​here I can see it can you see ̲͚̖͔̙î̩́t̲͎̩̱͔́̋̀ it is beautiful t​he final snuffing of the lie​s of Man ALL IS LOŚ͖̩͇̗̪̏̈́T ALL I​S LOST the pon̷y he comes he c̶̮omes he comes the ich​or permeates all MY FACE MY FACE ᵒh god no NO NOO̼O​O NΘ stop the an​*̶͑̾̾​̅ͫ͏̙̤g͇̫͛͆̾ͫ̑͆l͖͉̗̩̳̟̍ͫͥͨe̠̅s ͎a̧͈͖r̽̾̈́͒͑e n​ot rè̑ͧ̌aͨl̘̝̙̃ͤ͂̾̆ ZA̡͊͠͝LGΌ ISͮ̂҉̯͈͕̹̘̱ TO͇̹̺ͅƝ̴ȳ̳ TH̘Ë͖́̉ ͠P̯͍̭O̚​N̐Y̡ H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝S̨̥̫͎̭ͯ̿̔̀ͅ

With Regular Expressions

http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454

Parsing HTTP

With Regular Expressions

For limited, specific pieces of information, you can get away with it. People may hate you, but you can do it. Future sanity is not guaranteed, however.

HTTP is so complicated that this approach is madness for most purposes. Use a real parser.

Scrawls Doesn't Have an HTTP Engine for a real parser, yet. Sorry.

TL;DR

Ruby Web Servers

A Quick Survey

• WEBrick
• Mongrel
• Thin
• Puma
• Passenger
• Unicorn
• ServerEngine
• Rainbows!
• Yahns
• Goliath
• Swiftiply

WEBrick

• Pure Ruby

• Thread based design

• Written in 2000 by Masayoshi Takahashi and Yuuzou Gotou.

• Ubiquitous, as it is bundled with Ruby itself.

• Flexible, fairly featureful, and easy to use.

• Fairly well documented.

• Fairly slow, but maybe fast enough...

ab -n 10000 -c 4 http://127.0.0.1:8080/test.txt

Requests per second:    450.30 [#/sec] (mean)

Mongrel

• Zed Shaw, 2006

• Ruby plus a C extension for parsing HTTP, built with Ragel

• Thread based

• Moderately fast for it's age

• EOL at 1.1.5 in a version that doesn't work with modern Rubies

• There is a 1.2.x version (gem install --pre mongrel) that does work with modern Rubies

• Completely unmaintained, but it is interesting code to look at and learn from. It inspired several other Ruby web servers.

Swiftiply

• My baby. May 2007

• Event Driven via EventMachine

• Original used regex for HTTP parsing....

• Structurally has support for real parsing, but needs work

• Intended to be a load balancing reverse proxy with a twist, with light web serving capabilities (static files, mostly)

• Very fast

• AFAIK, at least 100 production sites still use it

• I take pull requests. :)

Thin

• Marc-André Cournoyer, 2008

• Mongrel HTTP Parser

• Event Driven via EventMachine

• Rack interface

• Pretty fast. Still pretty commonly used

Goliath

• PostRank Labs, 2011

• Event Driven via EventMachine

• Completely asynchronous design, leveraging Ruby fibers to unwind callback complexity

• Performance and resource focused

• Niche usage; interesting use of Fibers

Puma

• Evan Phoenix, 2011

• Built on the bones of Mongrel

• Built from the ground up with concurrency in mind

    -- This means threaded with a thread pool

• Rack

• Runs on all major Ruby implementations (MRI, JRuby, Rubinius)

• If you use Thin, take a look at Puma

Passenger

• Phusion Passenger

• Heavy Rails world usage

• Directly integrates into Apache or Nginx

• New version includes a fast purpose built web server, Raptor

• Rack

• Commercial version has more multithreading/concurrency options than the single open source version does.

    -- Open source is single threaded multiprocess only

    -- Commercial version can be configured for multithreaded multiprocess concurrency.

Unicorn

• Built on Mongrel's bones by Eric Wong circa 2009

• fork() (i.e. built in multiprocessing) oriented

• Not the fastest, but it deals well with slow/variable requests

• Very mature, and very heavy utilization in the Rails world

• With modern copy-on-write friendly Rubies, can have reasonably nice resource utilization

Rainbows!

• Unicorn specifically tuned for those very big, very slow requests

Yahns

• Another in the Unicorn family

• Tuned for apps that receive very little traffic

• When idle, it is truly idle

• Very sensitive to app failures

ServerEngine

• Everything said about Unicorn is basically applicable here

• Works on Windows and JRuby, whereas Unicorn does not

• Multiprocess design

• Ritta Narita (@narittan) gave a nice talk about it yesterday

http://rubykaigi.org/2016/presentations/narittan.html