Lee Calcote
Clouds, containers, functions, applications and their management.
Prometheus
Alert Manager & Long-term Storage
Lee Calcote
KubeCon + CloudNativeCon NA 2017
December, 4th 2017
Lee Calcote
clouds, containers, functions, applications and their management
Show of Hands
AlertManager
Prometheus
Alertmanager is an alert...
Purpose
ingester
grouper
de-duplicator
silencer
throttler
notifier
\ˈnō-mən-ˌklā-chər
a brief Prometheus AlertManager construct review
match alerts to their receiver and how often to notify
where and how to send alerts
- Silencers - matches alerts with specific labels and prevents them from being included in notifications.
- Inhibitors - suppress specific notifications when other specific alerts are already firing.
- Grouping - categorizes alerts of similar nature into a single notification.
\ˈnō-mən-ˌklā-chər
a brief Prometheus AlertManager construct review
Muting
Suppressing
Correlating
group_wait: 30s
group_by: ['alertname', 'cluster']
group_interval: 5mMultiple approaches to suppression
vs
vs
per route
global
via ui / api
Alerts
ALERT <alert name>
IF <PromQL vector expression>
FOR <duration>
LABELS { ... }
ANNOTATIONS { ... }Supports clients other than Prometheus
is notified when alerts transition state
a shared construct
Prometheus
AlertManager
inactive
firing
pending
state transition
inactive
firing
notifications
!
Notification Integrations
Notifying to Multiple Destinations
Use continue to advance to next receiver.
route:
receiver: email_webhook
receivers:
- name: email_webhook
email_configs:
- to: 'lee@example.io'
webhook_configs:
- url: <webhook url here>Use a receiver that goes to both destinations.
route:
receiver: ops-team-all # default
routes:
- match:
severity: page
receiver: ops-team-b
continue: true
- match:
severity: critical
receiver: ops-team-a
receivers:
- name: ops-team-all
email_configs:
- to: ops-team-all@example.io
- name: ops-team-a
email_configs:
- to: ops-team-a@example.io
- name: ops-team-b
email_configs:
- to: ops-team-b@example.ioor
api
Inhibitor
!
de-duplication
Dispatcher
Non-HA AlertManager Architecture
Silencer
| Dispatcher sorts incoming alerts into aggregation groups and assigns the correct notifiers to each. |
Alert Provider
UI
Silence Provider
store
de-duplication
subscribe
Router
batched alerts
notification pipeline
Notify Provider
| checks for previously sent notifications |
Retry
Retry
Maintenance Script
alerts
High Availability
being introduced in 0.5
I
gossip protocols.
built atop Weave Mesh
With HA, you no longer have to monitor the monitor.
Designed for an alert to be sent to all instances in the cluster.
All Prometheus instances send alerts to all Alertmanager instances.
Guarantees notifications to be sent at least once.
Understanding and Extending
Prometheus AlertManager
KubeCon EU 2017
Storage Adapters
Prometheus
“an ephemeral sliding window of recent data”
Prometheus stores sample data on disk in a highly optimized custom format based on flat files.
-
efficient
-
tunable
-
NOT scaleable
-
NOT durable
./data/01BKGV7JBM69T2G1BGBGM6KB12
./data/01BKGV7JBM69T2G1BGBGM6KB12/meta.json
./data/01BKGV7JBM69T2G1BGBGM6KB12/wal
./data/01BKGV7JBM69T2G1BGBGM6KB12/wal/000002
./data/01BKGV7JBM69T2G1BGBGM6KB12/wal/000001
./data/01BKGTZQ1SYQJTR4PB43C8PD98
./data/01BKGTZQ1SYQJTR4PB43C8PD98/meta.json
./data/01BKGTZQ1SYQJTR4PB43C8PD98/index
./data/01BKGTZQ1SYQJTR4PB43C8PD98/chunks
./data/01BKGTZQ1SYQJTR4PB43C8PD98/chunks/000001
./data/01BKGTZQ1SYQJTR4PB43C8PD98/tombstones
./data/01BKGTZQ1HHWHV8FBJXW1Y3W0K
./data/01BKGTZQ1HHWHV8FBJXW1Y3W0K/meta.json
./data/01BKGTZQ1HHWHV8FBJXW1Y3W0K/wal
./data/01BKGTZQ1HHWHV8FBJXW1Y3W0K/wal/000001
./data/01BKGV7JC0RY8A6MACW02A2PJD
./data/01BKGV7JC0RY8A6MACW02A2PJD/meta.json
./data/01BKGV7JC0RY8A6MACW02A2PJD/index
./data/01BKGV7JC0RY8A6MACW02A2PJD/chunks
./data/01BKGV7JC0RY8A6MACW02A2PJD/chunks/000001
./data/01BKGV7JC0RY8A6MACW02A2PJD/tombstonesPrometheus Local Storage
source: prometheus.io
Prometheus Remote Storage
The solution is a remote storage adapter.
3rd Party Storage
Adapter
Prometheus
custom protocol
write samples
request: matchers + time ranges
receive: series + samples
Adapter: an application that can receive batches of samples from Prometheus over HTTP and send them to some backend.
AppOptics Remote Storage Adapter
Send Prometheus data to AppOptics
prometheus2appoptics
- batching
- translation
POST /v1/measurements
POST /receive
1000 Samples per POST
appoptics
Prometheus
AppOptics Prometheus Storage Integration
Resources
IRC: #prometheus on irc.freenode.net
Mailing lists:
-
prometheus-users – discussing Prometheus usage and community support
-
prometheus-developers – contributing to Prometheus development
Prometheus repositories to file bugs and features requests
#
Lee Calcote
Thank you. Questions?
clouds, containers, infrastructure,
applications and their management
yes, we're hiring
KubeCon NA 2017: Prometheus AlertManager & Long-term Storage
By Lee Calcote
KubeCon NA 2017: Prometheus AlertManager & Long-term Storage
Presented at CloudNativeCon+KubeCon NA 2017. The challenge: durable, long-term storage in Prometheus. The answer: Remote Storage adapters.
