After PyCon PL
Warsaw, 22.11.2013
Agenda
- Circus,
- Vaurien,
- Elastic search.
Circus
Circus is a Python program which can be used to monitor and control processes and sockets.
Circus can be driven via a command-line interface, a web interface or programmatically through its Python API.
Install and run Circus
$ sudo apt-get install libzmq-dev libevent-dev python-dev
$ pip install circus
$ pip install circus-web
./bin/circusd ./circus.ini
http://localhost:8080/
Sample configuration
[circus]
statsd = 1
httpd = 1
[watcher:appA]
cmd = php
args = test.php A 10000
numprocesses = 3
[watcher:appB]
cmd = php
args = test.php B 100000
numprocesses = 2
[watcher:appC]
cmd = php
args = test.php C 1000000
numprocesses = 1
CLI Tools
- circus-top,
- circusctl
Vaurien
Theory
Natural web server cycle:
- Dev server,
- Stage server - load testing, QA etc,
- Production server - everything's smooth.
Practice
Something will break in production at
some point in time no matter what.
some point in time no matter what.
- Lost connection to MySQL,
- Timeout error,
- Too many connections,
- Too many open files,
- Testing I/O is often ignored.
How to fix that?
App tested by end user and fixed in production?
No.
Manual testing?
No.
So, what's Vaurien?
Chaos Monkey for TCP connections.
Application ↔ Vaurien ↔ Backend
Vauries is defined by protocol and behavior.
Vaurien Design
Collection of protocols:
- http,
- memcache,
- mysql,
- redis,
- smtp,
- generic tcp
Collection of behaviors:
- blackout,
- delay,
- dummy,
- error,
- hang
Example
MySQL with 5% hangs:
$ pip install vaurien
$ vaurien --proxy 0.0.0.0:3307 --backend localhost:3306
--protocol mysql --behavior 5:hang
SSL SMTP proxy with 5% error rate and 10% delays:
$ vaurien --proxy 0.0.0.0:6565 --backend mail.example.com:465
--protocol smtp --behavior 5:error,10:delay
Example #2
[vaurien]
backend = localhost:465
proxy = localhost:6565
protocol = smtp
behavior = 5:error,10:delay
[behavior:delay]
sleep = 2
$ vaurien --config vaurien.ini
Side note
Vaurien can be used in unit tests.
Python only.
Custom behavior
from vaurien.behaviors import Behavior
class MySuperBehavior(object):
name = 'super'
options = {}
def on_before_handle(self, protocol, source, dest, to_backend):
# do something here
return True
def on_after_handle(self, protocol, source, dest, to_backend):
# do something else
return True
Controling proxy with API
$ curl -XGET http://localhost:8080/behavior
{
"behavior": "dummy"
}
$ curl -XPUT -d '{"sleep": 2, "name": "delay"}' http://localhost:8080/behavior -H "Content-Type: application/json"
{
"status": "ok"
}
Elastic search
REST
HTTP
JSON
schemaless
distributed
search
analytics
real-time
scalable
open source
Lucene
Setup
$ wget elasticsearch.tar.gz
$ tar xzvf elasticsearch.tar.gz
$ bin/elasticsearch
$ curl localhost:9200
Basic usage - PUT
$
curl -XPUT "http://localhost:9200/movies/movie/1" -d'
{
"title": "The Godfather",
"director": "Francis Ford Coppola",
"year": 1972
}'
{
"ok":true,
"_index": "movies",
"_type":
"movie",
"_id":
"1",
"_version": 1
}
Basic usage - GET
$ curl -XGET "http://localhost:9200/movies/movie/1"
{
"_index": "movies",
"_type": "movie",
"_id": "1",
"_version": 1,
"exists": true,
"_source": {
"title": "The Godfather",
"director": "Francis Ford Coppola",
"year": 1972
}
}
Basic usage - DELETE
$ curl -XGET "http://localhost:9200/movies/movie/1"
{
"ok": true,
"found": true,
"_index": "movies",
"_type": "movie",
"_id": "1",
"_version": 2
}
_search
- http://localhost:9200/_search
- http://localhost:9200/movies/_search
- http://localhost:9200/movies/movie/_search
$ curl -XGET "http://localhost:9200/_search"
Example search
$ curl -XPOST "http://localhost:9200/_search" -d'
{
"query": {
"query_string": {
"query": "kill"
}
}
}'
{
"query": {
"query_string": {
"query": "kill"
}
}
}'
Features
- Highlight matches,
- Synonyms,
- Simple relations (parent - child, nested docs),
- Aggregations,
- Auto completion (with typos!),
- Percolator (reverse search),
- Distributed, scalable, auto balanced
Extra
- Marteau,
- Funkload,
- Load,
- Ansible
Links
-
https://github.com/mozilla-services/circus
-
https://github.com/mozilla-services/vaurien
-
http://tinyurl.com/marketplace-test
- http://elasticsearch.org
Questions?
Thank you!
After PyCon PL
By Paweł Lewtak
After PyCon PL
- 616
