https://slides.com/elpete/itb2024-cbq
What this talk is
- An overview of cbq
- Examples of problems that queues and jobs can solve
- Overview of how to create new cbq protocols
What this talk isn't
- How to use the ColdBox Async Manager
- Overview of ColdBox Scheduled Tasks
What is cbq?
A protocol-based queueing system for ColdBox
A protocol-based queueing system for ColdBox
Can interact with different providers, like the ColdBox Async Engine, a database, Redis, or Rabbit MQ
A protocol-based queueing system for ColdBox
A protocol-based queueing system for ColdBox
Sends a message to be consumed later.
It can be consumed by the same application or a completely different application, language, or service.
A protocol-based queueing system for ColdBox
- Defines a Job as the unit of work on a queue
- Push a Job onto a queue connection to be worked later
- Define multiple queues or named piles of work
- Register worker pools to work the jobs passed to queues
- Ability to switch between Queue Providers easily
What does cbq give me?
Why not just use...?
- cfthread, runAsync, AsyncManager
- Redis, Rabbit MQ, etc.
- Homegrown queue framework
Backend processing using message queues is a paradigm shift, and CBQ is a game changer. Asynchronous FIFO processing dynamically split across an unlimited number of servers allows one to break up any large processing load seamlessly. Batching, chains, and one-off jobs with built-in logging, automated intelligent retries, and the ability to use anything from databases to cloud-based services such as AWS SQS to manage the job queue allows me to build incredibly powerful, scalable, and resilient systems for my financial services clients. Highly recommended!
John Wilson
Synaptrix
Why would I use cbq?
Sending Email
Preparing Large Spreadsheets
Video Processing
Background Uploads
Sequenced Jobs
Scheduled SMS Messages
Email Verification
Processing Payments
Cancelling Abandoned Orders
Send Monthly Invoices
Other reasons
- Easier testing making sure the right jobs were queued
- Sync in development, Rabbit in production
- Adjust the worker pool scale dynamically
- Retry, timeout, and backoff built in.
Definitions
Job
- Does its work in the `handle` method
- Serializes and deserializes itself to the queue protocol
- Set instance data in the `properties`
-
Exist in the context of your application
Job
Queue
- A named stack of jobs or messages to be delivered
- A queue connection must have at least one queue which is usually `default`
- A queue connection can have as many queues as desired
Queue
Queue Provider
- How a queue connection connects to a backend like Redis, RabbitMQ, or a database
- Can be used multiple times in a single application to define multiple queue connections with different configuration options
Queue Provider
Queue Connection
- A named combination of Queue Provider and properties
- Allows you to connect with multiple Database providers or multiple Redis providers
Queue Connection
Worker Pool
- A group of workers for a Queue Connection
- Can optionally work a subset of queues
- Can optionally work queues in a specific order
- Can be scaled up or down as needed
Worker Pool
Installation
box install cbq
component {
this.javaSettings = {
loadPaths : [ expandPath( "./modules/cbq/lib" ) ],
loadColdFusionClassPath : true,
reloadOnChange : false
};
}Batch-specific Setup
Configuration
Module Settings
moduleSettings = {
"cbq" : {
// The path the custom config file to
// register connections and worker pools
"configPath" : "config.cbq",
// Flag if workers should be registered.
// If your application only pushes to the queues,
// you can set this to false.
"registerWorkers" : getSystemSetting( "CBQ_REGISTER_WORKERS", true ),
// The interval to poll for changes to the worker pool scaling.
// Defaults to 0 which turns off the scheduled scaling feature.
"scaleInterval" : 0,
// continued...
}
};moduleSettings = {
"cbq" : {
// The default amount of time, in seconds, to delay a job.
// Used if the connection and job doesn't define their own.
"defaultWorkerBackoff" : 0,
// The default amount of time, in seconds, to wait before timing out a job.
// Used if the connection and job doesn't define their own.
"defaultWorkerTimeout" : 60,
// The default amount of time, in seconds,
// to wait for tasks to complete before killing them when requesting a shutdown.
"defaultWorkerShutdownTimeout" : 60,
// The default amount of attempts to try before failing a job.
// Used if the connection and job doesn't define their own.
"defaultWorkerMaxAttempts" : 1,
// Flag to allow restricting Job interceptor execution using a `jobPattern` annotation.
"registerJobInterceptorRestrictionAspect" : false
// continued...
}
};moduleSettings = {
"cbq" : {
// Datasource information for tracking batches.
"batchRepositoryProperties" : {
"tableName" : "cbq_batches",
// `datasource` can also be a struct
"datasource" : "",
// The sibling `datasource` property overrides any defined datasource in queryOptions.
"queryOptions" : {},
// Cleanup options for the batch table
"cleanup" : {
"enabled" : false,
// A callback to configure the ColdBox Scheduled Task
"frequency" : ( task ) => { task.everyDay(); },
// A QueryBuilder instance to configure what batches to delete.
// Only completed or cancelled batches can be deleted.
"criteria" : ( qb, currentUnixTimestamp ) => {
var thirtyDaysAgo = currentUnixTimestamp - ( 60 * 60 * 24 * 30 );
qb.where( ( q ) => {
q.where( "cancelledDate", "<=", thirtyDaysAgo );
q.orWhere( "completedDate", "<=", thirtyDaysAgo );
} );
}
}
}, // continued...
}
};moduleSettings = {
"cbq" : {
// Flag to turn on logging failed jobs to a database table.
"logFailedJobs" : false,
// Configuration information for logging failed jobs.
"logFailedJobsProperties" : {
"tableName" : "cbq_failed_jobs",
// `datasource` can also be a struct.
"datasource" : "",
// The sibling `datasource` property overrides any defined datasource in `queryOptions`.
"queryOptions" : {},
"cleanup" : {
"enabled" : false,
"frequency" : ( task ) => { task.everyDay(); },
"criteria" : ( q, currentUnixTimestamp ) => {
var thirtyDaysAgo = currentUnixTimestamp - ( 60 * 60 * 24 * 30 );
q.where( "failedDate", "<=", thirtyDaysAgo );
}
}
}
}
};cbq Config File
component {
function configure() {
newConnection( "default" )
.setProvider( "ColdBoxAsyncProvider@cbq" );
newConnection( "database" )
.setProvider( "DBProvider@cbq" )
.setProperties( { "tableName": "custom_jobs_table" } );
newConnection( "external" )
.setProvider( "DBProvider@cbq" )
.setProperties( { "datasource": "external" } );
newWorkerPool( "default" )
.forConnection( "default" )
.setTimeout( 5 )
.setMaxAttempts( 5 );
newWorkerPool( "priority" )
.forConnection( "database" )
.onQueues( [ "priority", "*" ] )
.setQuantity( 3 );
}
}Usage
Create a Job
// SendWelcomeEmailJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {
property name="mailService" inject="provider:MailService@cbmailservices";
property name="userId";
function handle() {
var user = getInstance( "User" ).findOrFail( getUserId() );
variables.mailService
.newMail(
from = "no-reply@example.com",
to = user.getEmail(),
subject = "Welcome!",
type = "html"
)
.setView( "/_emails/users/welcome" )
.setBodyTokens( {
"firstName" : user.getFirstName(),
"lastName" : user.getLastName()
} )
.send();
}
}Per-Job Configuration
// SendWelcomeEmailJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {
variables.connection = "emails";
variables.maxAttempts = 3;
variables.timeout = 10;
function handle() {
// ...
}
}Dispatching Jobs
// handlers/Main.cfc
component {
function create() {
var user = createUser( rc );
getInstance( "SendWelcomeEmailJob" )
.setProperties( { "userId": user.getId() } )
.dispatch();
}
}Setting up Worker Pools
moduleSettings = {
"cbq" : {
"registerWorkers" : true // this is also the default
}
};// config/cbq.cfc
component {
function configure() {
newConnection( "default" ).setProvider( "DBProvider@cbq" );
newWorkerPool( "default" )
.forConnection( "default" )
.setTimeout( 30 )
.setMaxAttempts( 5 )
.setQuantity( 3 );
newWorkerPool( "default" )
.forConnection( "default" )
.onQueues( "emails" )
.setTimeout( 60 )
.setMaxAttempts( 10 )
.setQuantity( 10 );
}
}
cbq Model Helper
// handlers/Main.cfc
component {
property name="cbq" inject="cbq@cbq";
function create() {
var user = createUser( rc );
cbq.job( "SendWelcomeEmailJob", {
"userId": user.getId()
} ).dispatch();
}
}Job Chains
// FulfillOrderJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {
function handle() {
var productId = getProperties().productId;
processPayment( productId );
getInstance( "SendProductLinkEmail" )
.onConnection( "fulfillment" )
.setProperties( {
"productId" : productId,
"userId" : getProperties().userId
} )
.dispatch();
}
}// handlers/Main.cfc
component {
function create() {
getInstance( "FulfillOrderJob" )
.setProperties( { "productId": rc.productId } )
.chain( [
getInstance( "SendProductLinkEmail" )
.onConnection( "fulfillment" )
.setProperties( {
"productId" : rc.productId,
"userId" : auth().getUserId()
} )
] )
}
}// handlers/Main.cfc
component {
property name="cbq" inject="cbq@cbq";
function create() {
cbq.chain( [
cbq.job( "FulfillOrderJob", { "productId": rc.productId } ),
cbq.job( job = "SendProductLinkEmail", properties = {
"productId": rc.productId,
"userId": auth().getUserId()
}, connection = "fulfillment" ),
] ).dispatch();
}
}Job Batches
// handlers/Main.cfc
component {
property name="cbq" inject="cbq@cbq";
function create() {
cbq.batch( [
cbq.job( "ImportCsvJob", { "start": 1, "end": 100 } ),
cbq.job( "ImportCsvJob", { "start": 101, "end": 200 } ),
cbq.job( "ImportCsvJob", { "start": 201, "end": 300 } ),
cbq.job( "ImportCsvJob", { "start": 301, "end": 400 } ),
cbq.job( "ImportCsvJob", { "start": 401, "end": 500 } )
] ).then( cbq.job( "ImportCsvSuccessfulJob" ) )
.catch( cbq.job( "ImportCsvFailedJob" ) )
.finally( cbq.job( "ImportCsvCompletedJob" ) )
.dispatch();
}
}Demo
Bonus
Delay the Current Job
// MonitorPendingCartJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {
function handle() {
var cartId = getProperties().cartId;
var cart = getInstance( "Cart" ).findOrFail( cartId );
if ( cart.isConfirmed() || cart.isCancelled() ) {
return;
}
if ( abs( dateDiff( "n", cart.getModifiedDate(), now() ) ) > 60 ) {
cart.cancel();
return;
}
getInstance( "MonitorPendingCartJob" )
.setProperties( { "cartId" : cartId } )
.delay( 15 * 1000 )
.dispatch();
}
}// MonitorPendingCartJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {
function handle() {
var cartId = getProperties().cartId;
var cart = getInstance( "Cart" ).findOrFail( cartId );
if ( cart.isConfirmed() || cart.isCancelled() ) {
return;
}
if ( abs( dateDiff( "n", cart.getModifiedDate(), now() ) ) > 60 ) {
cart.cancel();
return;
}
this.release( 15 * 1000 );
}
}Using `release`
// MonitorPendingCartJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {
function handle() {
var cartId = getProperties().cartId;
var cart = getInstance( "Cart" ).findOrFail( cartId );
if ( cart.isConfirmed() || cart.isCancelled() ) {
return;
}
if ( abs( dateDiff( "n", cart.getModifiedDate(), now() ) ) > 60 ) {
cart.cancel();
return;
}
var delay = 2 ^ this.getCurrentAttempt() * 1000;
this.release( delay );
}
}Exponential Backoff
ITB 2024 — cbq
By Eric Peterson
