Sensible monitoring with Sensu
Avishai Ish-Shalom (@nukemberg)
In the beginning
God created The Servers and The Network
Now the dashboard was formless and empty, darkness was over the surface of the deep, and the Spirit of God was hovering over the servers.
And God said:
"Let there be Monitoring", and there was Monitoring
God saw that the monitoring was good, and he separated the green from the red.
And God said
“Let the data under the system be gathered to one place, and let a single process appear.” And it was so. God called the single process “server,” and the gathered data he called “checks” And God saw that it was good
Let's skip ahead a bit...
Now the cloud
was more crafty than any of the wild platforms the Lord God had made. He said to the sysadmin, “Did God really say, ‘You must configure everything manually’?”
The sysadmin said
to the cloud, “We may automate the servers in the system, but God did say, ‘You must not automate the monitoring that is in the middle of the system, and you must not touch it, or it will break.’”
“It will not certainly crash,”
the Cloud said to the Sysadmin. “For God knows that when you automate it your eyes will be opened, and you will be like Google, knowing good and evil.”
OK... jokes aside, what's wrong with Nagios?
Nagios was built for a static world of Pet servers
The Nagios model
- Checks executed on central server
- Host oriented model
- Every server needs to be configured
- File based configurations
- Scalability
- Configurations
- API
- Cluster aggregation
What's wrong?
A wild Sensu appears
Architecture
Features
- Auto client registration
- API
- Scalable
- Built for automation
- Aggregation
- Support for time-series data
Uchiwa
Mergable Json config files
{
"checks": {
"check_disk_usage": {
"command": "check-disk-usage.rb -w :::disk.warning|80::: -c :::disk.critical|90:::",
"subscribers": [
"production"
],
"interval": 60,
"environment": ":::environment|production:::"
}
}
}Check plugins
Compatible with Nagios
API
$ curl -s http://127.0.0.1:4567/clients | jq .
[
{
"timestamp": 1458625739,
"version": "0.25.0",
"socket": {
"port": 3030,
"bind": "127.0.0.1"
},
"subscriptions": [
"dev"
],
"environment": "development",
"address": "127.0.0.1",
"name": "client-01"
}
]Client (de)registration
- Auto register
- Deregister on clean shutdown (optional)
-
Deregister by API call (DELETE /clients/:id)
Integrations
- Notifications (Slack, PagerDuty)
- Time-series data (Graphite, InfluxDB)
- Events (ELK, GrayLog, Slack)
- CM (puppet, chef, ansible)
Checks collect data and create events
Handlers process the events passed on by Filters and optionally Mutate the event
Aggregates are pseudo checks that derive events from a group of checks
Checks are scheduled by the server and executed on the client
Stand-Alone checks are scheduled by the client
Sensu
- Auto client discovery
- Checks executed on clients
- Arbitrary aggregations
- API
- Mergeable json configs
Nagios
- Hosts must be configured
- Checks executed on server
- No aggregations*
- No API*
- Special config language
Cut the crap, let's have the bottom line
Pros
- Close to the Nagios model
- Compatible with Nagios plugins
- Scalable
- Automation friendly
- Modern
- Integration friendly
- Simple
Cons
- Close to the Nagios model
- Not powerful enough for serious app centric monitoring
- Barebones - no analytics, lame dashboard, no dependency graph, etc
Questions?
Sensible monitoring with Sensu
By Avishai Ish-Shalom
