After PyCon PL

Warsaw, 22.11.2013

Agenda

Circus,
Vaurien,
Elastic search.

Circus

Circus is a Python program which can be used to monitor and control processes and sockets.

Circus can be driven via a command-line interface, a web interface or programmatically through its Python API.

Install and run Circus

$ sudo apt-get install libzmq-dev libevent-dev python-dev

$ pip install circus

$ pip install circus-web

./bin/circusd ./circus.ini

http://localhost:8080/

Sample configuration

[circus]
statsd = 1
httpd = 1

        

[watcher:appA]
cmd = php
args = test.php A 10000
numprocesses = 3

        

[watcher:appB]
cmd = php
args = test.php B 100000
numprocesses = 2

        

[watcher:appC]
cmd = php
args = test.php C 1000000
numprocesses = 1

CLI Tools

circus-top,
circusctl

Vaurien

Theory

Natural web server cycle:

Dev server,
Stage server - load testing, QA etc,
Production server - everything's smooth.

Practice

Something will break in production at
some point in time no matter what.

Lost connection to MySQL,
Timeout error,
Too many connections,
Too many open files,
Testing I/O is often ignored.

How to fix that?

App tested by end user and fixed in production?

No.

Manual testing?

No.

So, what's Vaurien?

Chaos Monkey for TCP connections.

Application ↔ Vaurien ↔ Backend

Vauries is defined by protocol and behavior.

Vaurien Design

Collection of protocols:

http,

memcache,

mysql,

redis,

smtp,

generic tcp

Collection of behaviors:

blackout,

delay,

dummy,

error,

hang

Example

MySQL with 5% hangs:

    $ pip install vaurien

$ vaurien --proxy 0.0.0.0:3307 --backend localhost:3306 

          --protocol mysql --behavior 5:hang

SSL SMTP proxy with 5% error rate and 10% delays:

$ vaurien --proxy 0.0.0.0:6565 --backend mail.example.com:465

          --protocol smtp --behavior 5:error,10:delay

Example #2

        [vaurien]
    
        backend = localhost:465

        proxy = localhost:6565

        protocol = smtp

        behavior = 5:error,10:delay

        [behavior:delay]
    
        sleep = 2
    
    $ vaurien --config vaurien.ini

Side note

Vaurien can be used in unit tests.

Python only.

Custom behavior

        from vaurien.behaviors import Behavior
    
        class MySuperBehavior(object):
        
            name = 'super'
        
            options = {}
    
            def on_before_handle(self, protocol, source, dest, to_backend):
        
                # do something here
    
                return True
    
             def on_after_handle(self, protocol, source, dest, to_backend):
    
                # do something else
    
                return True

Controling proxy with API

        $ curl -XGET http://localhost:8080/behavior
    
        {
    
          "behavior": "dummy"
    
        }

        $ curl -XPUT -d '{"sleep": 2, "name": "delay"}' http://localhost:8080/behavior -H "Content-Type: application/json"

{
    
  "status": "ok"
    
}

Elastic search

REST

HTTP

JSON

schemaless

distributed

analytics

real-time

scalable

open source

Lucene

Setup

$ wget elasticsearch.tar.gz
$ tar xzvf elasticsearch.tar.gz
$ bin/elasticsearch
$ curl localhost:9200

Basic usage - PUT

$ curl -XPUT "http://localhost:9200/movies/movie/1" -d'

{

    "title": "The Godfather",

    "director": "Francis Ford Coppola",

    "year": 1972

}'

{

        "ok":true,

    "_index": "movies",

        "_type": 
    "movie",

            "_id": 
        "1",

            "_version": 1 

}

Basic usage - GET

$ curl -XGET "http://localhost:9200/movies/movie/1"

{
   "_index": "movies",
   "_type": "movie",
   "_id": "1",
   "_version": 1,
   "exists": true,
   "_source": {
      "title": "The Godfather",
      "director": "Francis Ford Coppola",
      "year": 1972
   }
}

Basic usage - DELETE