https://slides.com/elpete/itb2022-cbq

What this talk is

An overview of cbq
Examples of problems that queues and jobs can solve
Overview of how to create new cbq protocols

What this talk isn't

How to use the ColdBox Async Manager
Overview of ColdBox Scheduled Tasks
A review of stable software — cbq is being released as an early preview

Who am I?

Utah

Ortus Solutions

qb, Quick, Hyper, lots of other modules

1 wife, 3 kids, 1 dog

Type 1 Diabetic

Theatre Nerd

What is cbq?

A protocol-based queueing system for ColdBox

Can interact with different providers, like the ColdBox Async Engine, a database, Redis, or Rabbit MQ

A protocol-based queueing system for ColdBox

Sends a message to be consumed later.

It can be consumed by the same application or a completely different application, language, or service.

Defines a Job as the unit of work on a queue
Push a Job onto a queue connection to be worked later
Define multiple queues or named piles of work
Register worker pools to work the jobs passed to queues
Ability to switch between Queue Providers easily

What does cbq give me?

Why not just use...?

cfthread, runAsync, AsyncManager
Redis, Rabbit MQ, etc.
Homegrown queue framework

Why would I use cbq?

Sending Email

Preparing Large Spreadsheets

Video Processing

Background Uploads

Sequenced Jobs

Scheduled SMS Messages

Email Verification

Processing Payments

Cancelling Abandoned Orders

Send Monthly Invoices

Other reasons

Easier testing making sure the right jobs were queued
Sync in development, Rabbit in production
Adjust the worker pool scale dynamically
Retry, timeout, and backoff built in.

Definitions

Job

Does its work in the handle method
Serializes and deserializes itself to the queue protocol
Set instance data in the properties
Exist in the context of your application

Job

Queue

A named stack of jobs or messages to be delivered
A queue connection must have at least one queue which is usually default
A queue connection can have as many queues as desired

Queue

Queue Provider

How a queue connection connects to a backend like Redis, RabbitMQ, or a database
Can be used multiple times in a single application to define multiple queue connections with different configuration options

Queue Provider

Queue Connection

A named combination of Queue Provider and properties
Allows you to connect with multiple Database providers or multiple Redis providers

Queue Connection

Installation

box install cbq

Configuration

Module Settings

moduleSettings = {
    "cbq" : {
        // The path the custom config file to
        // register connections and worker pools
        "configPath" : "config.cbq",
        
        // Flag if workers should be registered.
        // If your application only pushes to the queues,
        // you can set this to false.
        "registerWorkers" : getSystemSetting( "CBQ_REGISTER_WORKERS", true ),
        
        // The interval to poll for changes to the worker pool scaling.
        // Defaults to 0 which turns off the scheduled scaling feature.
        "scaleInterval" : 0
    }
};

cbq Config File

component {

    function configure() {
        newConnection( "default" )
            .setProvider( "ColdBoxAsyncProvider@cbq" );

        newConnection( "database" )
            .setProvider( "DBProvider@cbq" )
            .setProperties( { "tableName": "custom_jobs_table" } );
    }

    function work() {
        newWorkerPool( "default" )
            .setTimeout( 5 )
            .setMaxAttempts( 5 )
            .setQuantity( 3 );
    }

}

Usage

Create a Job

// SendWelcomeEmailJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {
  
    property name="mailService" inject="MailService@cbmailservices";

    function handle() {
      	var user = getInstance( "User" )
            .findOrFail( getProperties().userId );
      
        variables.mailService
            .newMail(
                from = "no-reply@example.com",
                to = user.getEmail(),
                subject = "Welcome!",
                type = "html"
            )
            .setView( "/_emails/users/welcome" )
            .setBodyTokens( {
                "firstName" : user.getFirstName(),
                "lastName" : user.getLastName()
            } )
            .send();
    }

}

Per-Job Configuration

// SendWelcomeEmailJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {
  
    variables.connection = "emails";
    variables.maxAttempts = 3;
    variables.timeout = 10;

    function handle() {
      	// ...
    }

}

Dispatching Jobs

// handlers/Main.cfc
component {

    function create() {
        var user = createUser( rc );
      	
        getInstance( "SendWelcomeEmailJob" )
            .setProperties( { "userId": user.getId() } )
            .dispatch();
    }

}

Setting up Worker Pools

// config/cbq.cfc
component {

    function work() {
        newWorkerPool( "default" )
            .setTimeout( 30 )
            .setMaxAttempts( 5 )
            .setQuantity( 3 );
      
        newWorkerPool( "default" )
            .setQueue( "emails" )
            .setTimeout( 60 )
            .setMaxAttempts( 10 )
            .setQuantity( 10 );

    }

}

moduleSettings = {
    "cbq" : {        
        "registerWorkers" : true // this is also the default
    }
};

Demo

Bonus

Job Sequences

// FulfillOrderJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {

    function handle() {
        var productId = getProperties().productId;

      	processPayment( productId );

        getInstance( "SendProductLinkEmail" )
            .onConnection( "fulfillment" )
            .setProperties( {
                "productId" : productId,
                "userId" : getProperties().userId
            } )
            .dispatch();
    }

}

Delay the Current Job

// MonitorPendingCartJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {

    function handle() {
        var cartId = getProperties().cartId;
        var cart = getInstance( "Cart" ).findOrFail( cartId );

        if ( cart.isConfirmed() || cart.isCancelled() ) {
            return;
        }

        if ( abs( dateDiff( "n", cart.getModifiedDate(), now() ) ) > 60 ) {
            cart.cancel();
            return;
        }

        getInstance( "MonitorPendingCartJob" )
            .setProperties( { "cartId" : cartId } )
            .delay( 15 * 1000 )
            .dispatch();
    }

}