Pulse
Mozilla's Generic Notification System
The Problem
"Mozilla has a ton of different systems that are interconnected via polling, screen scraping, email, and other brittle methods. To make their lives easier community members often build tools on top of this house of cards, adding yet another level of scraping and polling. Many systems don't even export important data for others to scrape and use, preventing better tools from being written." — LegNeato
Events include
- Build, test results
- VCS commits
- Bug updates
- ...
The Solution:
AMQP-based pub/sub system
Pulse as AMQP pub/sub
Toward a formal definition
Messages MUST be UTF-8-encoded JSON.
All exchanges MUST be public.
All exchanges MUST be topic exchanges.
Messages SHOULD be short.
Messages SHOULD be published over confirm channels.
Messages MUST be delivered at least once.
Messages SHOULD be delivered at most once.
Following terms MUST be used:
- publisher (AMQP producer)
- subscriber (AMQP consumer)
- client ID (username)
- access token (password)
"At Mozilla, we like to make things difficult for ourselves"
— approximation of Jay Sullivan
RabbitMQ wasn't designed to be a public system:
- User creation must be performed by admins
- Durable queues are necessary but dangerous
- No system notifications
The solution:
middleware
(of course)
(or maybe it's management software)
PulseGuardian:
- Creates/edits/deletes users
- Monitors queues & sends warnings
- Deletes oversized queues
PulseGuardian user control
- Login via Persona
- Create Pulse users, tied to Persona account
- Can have multiple users for application separation
PulseGuardian user control
- Pulse users have restricted permissions:
- r/w over queue/<user>/.*, exchange/<user>/.*
- r-o over exchange/.*
PulseGuardian queue control
- Polls RabbitMQ for queue info
- Sends emails at warn_queue_size
- Deletes queue at del_queue_size (with email notice)
Future of Pulse
- VCS publisher
- Better support library (too much hard coding)
- Confirm channels