Pulse

Mozilla's Generic Notification System

The Problem

"Mozilla has a ton of different systems that are interconnected via polling, screen scraping, email, and other brittle methods. To make their lives easier community members often build tools on top of this house of cards, adding yet another level of scraping and polling. Many systems don't even export important data for others to scrape and use, preventing better tools from being written." — LegNeato

Events include

  • Build, test results
  • VCS commits
  • Bug updates
  • ...

The Solution:

AMQP-based pub/sub system

Pulse as AMQP pub/sub

Toward a formal definition

Messages MUST be UTF-8-encoded JSON.

All exchanges MUST be public.

All exchanges MUST be topic exchanges.

Messages SHOULD be short.

Messages SHOULD be published over confirm channels.

Messages MUST be delivered at least once.

Messages SHOULD be delivered at most once.

Following terms MUST be used:

- publisher (AMQP producer)

- subscriber (AMQP consumer)

- client ID (username)

- access token (password)

"At Mozilla, we like to make things difficult for ourselves"

— approximation of Jay Sullivan

RabbitMQ wasn't designed to be a public system:

  • User creation must be performed by admins
  • Durable queues are necessary but dangerous
  • No system notifications

The solution:
 middleware

(of course)

(or maybe it's management software)

PulseGuardian:

  • Creates/edits/deletes users
  • Monitors queues & sends warnings
  • Deletes oversized queues

PulseGuardian user control

  • Login via Persona
  • Create Pulse users, tied to Persona account
  • Can have multiple users for application separation

PulseGuardian user control

  • Pulse users have restricted permissions:
    • r/w over queue/<user>/.*, exchange/<user>/.*
    • r-o over exchange/.*

PulseGuardian queue control

  • Polls RabbitMQ for queue info
  • Sends emails at warn_queue_size
  • Deletes queue at del_queue_size (with email notice)

Future of Pulse

  • VCS publisher
  • Better support library (too much hard coding)
  • Confirm channels

Would you like to know more?

Made with Slides.com