https://slides.com/elpete/itb2022-cbq

https://slides.com/elpete/itb2022-cbq

What this talk is

  • An overview of cbq
  • Examples of problems that queues and jobs can solve
  • Overview of how to create new cbq protocols

What this talk isn't

  • How to use the ColdBox Async Manager
  • Overview of ColdBox Scheduled Tasks
  • A review of stable software — cbq is being released as an early preview

Who am I?

Utah

Ortus Solutions

qb, Quick, Hyper, lots of other modules

1 wife, 3 kids, 1 dog

Type 1 Diabetic

Theatre Nerd

What is cbq?

A protocol-based queueing system for ColdBox

A protocol-based queueing system for ColdBox

Can interact with different providers, like the ColdBox Async Engine, a database, Redis, or Rabbit MQ

A protocol-based queueing system for ColdBox

Sends a message to be consumed later.

It can be consumed by the same application or a completely different application, language, or service.

  • Defines a Job as the unit of work on a queue
  • Push a Job onto a queue connection to be worked later
  • Define multiple queues or named piles of work
  • Register worker pools to work the jobs passed to queues
  • Ability to switch between Queue Providers easily

What does cbq give me?

Why not just use...?

  • cfthread, runAsync, AsyncManager
  • Redis, Rabbit MQ, etc.
  • Homegrown queue framework

Why would I use cbq?

Sending Email

Preparing Large Spreadsheets

Video Processing

Background Uploads

Sequenced Jobs

Scheduled SMS Messages

Email Verification

Processing Payments

Cancelling Abandoned Orders

Send Monthly Invoices

Other reasons

  • Easier testing making sure the right jobs were queued
  • Sync in development, Rabbit in production
  • Adjust the worker pool scale dynamically
  • Retry, timeout, and backoff built in.

Definitions

Job

  • Does its work in the handle method
  • Serializes and deserializes itself to the queue protocol
  • Set instance data in the properties
  • Exist in the context of your application

Job

Queue

  • A named stack of jobs or messages to be delivered
  • A queue connection must have at least one queue which is usually default
  • A queue connection can have as many queues as desired

Queue

Queue Provider

  • How a queue connection connects to a backend like Redis, RabbitMQ, or a database
  • Can be used multiple times in a single application to define multiple queue connections with different configuration options

Queue Provider

Queue Connection

  • A named combination of Queue Provider and properties
  • Allows you to connect with multiple Database providers or multiple Redis providers

Queue Connection

Installation

box install cbq

Configuration

Module Settings

moduleSettings = {
    "cbq" : {
        // The path the custom config file to
        // register connections and worker pools
        "configPath" : "config.cbq",
        
        // Flag if workers should be registered.
        // If your application only pushes to the queues,
        // you can set this to false.
        "registerWorkers" : getSystemSetting( "CBQ_REGISTER_WORKERS", true ),
        
        // The interval to poll for changes to the worker pool scaling.
        // Defaults to 0 which turns off the scheduled scaling feature.
        "scaleInterval" : 0
    }
};

cbq Config File

component {

    function configure() {
        newConnection( "default" )
            .setProvider( "ColdBoxAsyncProvider@cbq" );

        newConnection( "database" )
            .setProvider( "DBProvider@cbq" )
            .setProperties( { "tableName": "custom_jobs_table" } );
    }

    function work() {
        newWorkerPool( "default" )
            .setTimeout( 5 )
            .setMaxAttempts( 5 )
            .setQuantity( 3 );
    }

}

Usage

Create a Job

// SendWelcomeEmailJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {
  
    property name="mailService" inject="MailService@cbmailservices";

    function handle() {
      	var user = getInstance( "User" )
            .findOrFail( getProperties().userId );
      
        variables.mailService
            .newMail(
                from = "no-reply@example.com",
                to = user.getEmail(),
                subject = "Welcome!",
                type = "html"
            )
            .setView( "/_emails/users/welcome" )
            .setBodyTokens( {
                "firstName" : user.getFirstName(),
                "lastName" : user.getLastName()
            } )
            .send();
    }

}

Per-Job Configuration

// SendWelcomeEmailJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {
  
    variables.connection = "emails";
    variables.maxAttempts = 3;
    variables.timeout = 10;

    function handle() {
      	// ...
    }

}

Dispatching Jobs

// handlers/Main.cfc
component {

    function create() {
        var user = createUser( rc );
      	
        getInstance( "SendWelcomeEmailJob" )
            .setProperties( { "userId": user.getId() } )
            .dispatch();
    }

}

Setting up Worker Pools

// config/cbq.cfc
component {

    function work() {
        newWorkerPool( "default" )
            .setTimeout( 30 )
            .setMaxAttempts( 5 )
            .setQuantity( 3 );
      
        newWorkerPool( "default" )
            .setQueue( "emails" )
            .setTimeout( 60 )
            .setMaxAttempts( 10 )
            .setQuantity( 10 );

    }

}
moduleSettings = {
    "cbq" : {        
        "registerWorkers" : true // this is also the default
    }
};

Demo

Bonus

Job Sequences

// FulfillOrderJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {

    function handle() {
        var productId = getProperties().productId;

      	processPayment( productId );

        getInstance( "SendProductLinkEmail" )
            .onConnection( "fulfillment" )
            .setProperties( {
                "productId" : productId,
                "userId" : getProperties().userId
            } )
            .dispatch();
    }

}

Delay the Current Job

// MonitorPendingCartJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {

    function handle() {
        var cartId = getProperties().cartId;
        var cart = getInstance( "Cart" ).findOrFail( cartId );

        if ( cart.isConfirmed() || cart.isCancelled() ) {
            return;
        }

        if ( abs( dateDiff( "n", cart.getModifiedDate(), now() ) ) > 60 ) {
            cart.cancel();
            return;
        }

        getInstance( "MonitorPendingCartJob" )
            .setProperties( { "cartId" : cartId } )
            .delay( 15 * 1000 )
            .dispatch();
    }

}

cbq

By Eric Peterson