![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321067/SLIDES_SPEAKERS-keynote.001.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321069/SLIDES_SPEAKERS-keynote.002.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321068/SLIDES_SPEAKERS-keynote.003.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/9821487/cbq-L.png)
https://slides.com/elpete/itb2024-cbq
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
What this talk is
- An overview of cbq
- Examples of problems that queues and jobs can solve
- Overview of how to create new cbq protocols
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
What this talk isn't
- How to use the ColdBox Async Manager
- Overview of ColdBox Scheduled Tasks
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321072/SLIDES_SPEAKERS-keynote.004.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
What is cbq?
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
A protocol-based queueing system for ColdBox
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
A protocol-based queueing system for ColdBox
Can interact with different providers, like the ColdBox Async Engine, a database, Redis, or Rabbit MQ
A protocol-based queueing system for ColdBox
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
A protocol-based queueing system for ColdBox
Sends a message to be consumed later.
It can be consumed by the same application or a completely different application, language, or service.
A protocol-based queueing system for ColdBox
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
- Defines a Job as the unit of work on a queue
- Push a Job onto a queue connection to be worked later
- Define multiple queues or named piles of work
- Register worker pools to work the jobs passed to queues
- Ability to switch between Queue Providers easily
What does cbq give me?
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Why not just use...?
- cfthread, runAsync, AsyncManager
- Redis, Rabbit MQ, etc.
- Homegrown queue framework
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Backend processing using message queues is a paradigm shift, and CBQ is a game changer. Asynchronous FIFO processing dynamically split across an unlimited number of servers allows one to break up any large processing load seamlessly. Batching, chains, and one-off jobs with built-in logging, automated intelligent retries, and the ability to use anything from databases to cloud-based services such as AWS SQS to manage the job queue allows me to build incredibly powerful, scalable, and resilient systems for my financial services clients. Highly recommended!
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321603/391px-Quotation_Marks.svg.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321603/391px-Quotation_Marks.svg.png)
John Wilson
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11341129/IMG_1113.jpg)
Synaptrix
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Why would I use cbq?
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Sending Email
Preparing Large Spreadsheets
Video Processing
Background Uploads
Sequenced Jobs
Scheduled SMS Messages
Email Verification
Processing Payments
Cancelling Abandoned Orders
Send Monthly Invoices
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Other reasons
- Easier testing making sure the right jobs were queued
- Sync in development, Rabbit in production
- Adjust the worker pool scale dynamically
- Retry, timeout, and backoff built in.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Definitions
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Job
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
- Does its work in the `handle` method
- Serializes and deserializes itself to the queue protocol
- Set instance data in the `properties`
-
Exist in the context of your application
Job
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Queue
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
- A named stack of jobs or messages to be delivered
- A queue connection must have at least one queue which is usually `default`
- A queue connection can have as many queues as desired
Queue
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Queue Provider
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
- How a queue connection connects to a backend like Redis, RabbitMQ, or a database
- Can be used multiple times in a single application to define multiple queue connections with different configuration options
Queue Provider
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Queue Connection
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
- A named combination of Queue Provider and properties
- Allows you to connect with multiple Database providers or multiple Redis providers
Queue Connection
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Worker Pool
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
- A group of workers for a Queue Connection
- Can optionally work a subset of queues
- Can optionally work queues in a specific order
- Can be scaled up or down as needed
Worker Pool
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Installation
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
box install cbq
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
component {
this.javaSettings = {
loadPaths : [ expandPath( "./modules/cbq/lib" ) ],
loadColdFusionClassPath : true,
reloadOnChange : false
};
}
Batch-specific Setup
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Configuration
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Module Settings
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
moduleSettings = {
"cbq" : {
// The path the custom config file to
// register connections and worker pools
"configPath" : "config.cbq",
// Flag if workers should be registered.
// If your application only pushes to the queues,
// you can set this to false.
"registerWorkers" : getSystemSetting( "CBQ_REGISTER_WORKERS", true ),
// The interval to poll for changes to the worker pool scaling.
// Defaults to 0 which turns off the scheduled scaling feature.
"scaleInterval" : 0,
// continued...
}
};
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
moduleSettings = {
"cbq" : {
// The default amount of time, in seconds, to delay a job.
// Used if the connection and job doesn't define their own.
"defaultWorkerBackoff" : 0,
// The default amount of time, in seconds, to wait before timing out a job.
// Used if the connection and job doesn't define their own.
"defaultWorkerTimeout" : 60,
// The default amount of time, in seconds,
// to wait for tasks to complete before killing them when requesting a shutdown.
"defaultWorkerShutdownTimeout" : 60,
// The default amount of attempts to try before failing a job.
// Used if the connection and job doesn't define their own.
"defaultWorkerMaxAttempts" : 1,
// Flag to allow restricting Job interceptor execution using a `jobPattern` annotation.
"registerJobInterceptorRestrictionAspect" : false
// continued...
}
};
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
moduleSettings = {
"cbq" : {
// Datasource information for tracking batches.
"batchRepositoryProperties" : {
"tableName" : "cbq_batches",
// `datasource` can also be a struct
"datasource" : "",
// The sibling `datasource` property overrides any defined datasource in queryOptions.
"queryOptions" : {},
// Cleanup options for the batch table
"cleanup" : {
"enabled" : false,
// A callback to configure the ColdBox Scheduled Task
"frequency" : ( task ) => { task.everyDay(); },
// A QueryBuilder instance to configure what batches to delete.
// Only completed or cancelled batches can be deleted.
"criteria" : ( qb, currentUnixTimestamp ) => {
var thirtyDaysAgo = currentUnixTimestamp - ( 60 * 60 * 24 * 30 );
qb.where( ( q ) => {
q.where( "cancelledDate", "<=", thirtyDaysAgo );
q.orWhere( "completedDate", "<=", thirtyDaysAgo );
} );
}
}
}, // continued...
}
};
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
moduleSettings = {
"cbq" : {
// Flag to turn on logging failed jobs to a database table.
"logFailedJobs" : false,
// Configuration information for logging failed jobs.
"logFailedJobsProperties" : {
"tableName" : "cbq_failed_jobs",
// `datasource` can also be a struct.
"datasource" : "",
// The sibling `datasource` property overrides any defined datasource in `queryOptions`.
"queryOptions" : {},
"cleanup" : {
"enabled" : false,
"frequency" : ( task ) => { task.everyDay(); },
"criteria" : ( q, currentUnixTimestamp ) => {
var thirtyDaysAgo = currentUnixTimestamp - ( 60 * 60 * 24 * 30 );
q.where( "failedDate", "<=", thirtyDaysAgo );
}
}
}
}
};
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
cbq Config File
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
component {
function configure() {
newConnection( "default" )
.setProvider( "ColdBoxAsyncProvider@cbq" );
newConnection( "database" )
.setProvider( "DBProvider@cbq" )
.setProperties( { "tableName": "custom_jobs_table" } );
newConnection( "external" )
.setProvider( "DBProvider@cbq" )
.setProperties( { "datasource": "external" } );
newWorkerPool( "default" )
.forConnection( "default" )
.setTimeout( 5 )
.setMaxAttempts( 5 );
newWorkerPool( "priority" )
.forConnection( "database" )
.onQueues( [ "priority", "*" ] )
.setQuantity( 3 );
}
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Usage
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Create a Job
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
// SendWelcomeEmailJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {
property name="mailService" inject="provider:MailService@cbmailservices";
property name="userId";
function handle() {
var user = getInstance( "User" ).findOrFail( getUserId() );
variables.mailService
.newMail(
from = "no-reply@example.com",
to = user.getEmail(),
subject = "Welcome!",
type = "html"
)
.setView( "/_emails/users/welcome" )
.setBodyTokens( {
"firstName" : user.getFirstName(),
"lastName" : user.getLastName()
} )
.send();
}
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Per-Job Configuration
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
// SendWelcomeEmailJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {
variables.connection = "emails";
variables.maxAttempts = 3;
variables.timeout = 10;
function handle() {
// ...
}
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Dispatching Jobs
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
// handlers/Main.cfc
component {
function create() {
var user = createUser( rc );
getInstance( "SendWelcomeEmailJob" )
.setProperties( { "userId": user.getId() } )
.dispatch();
}
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Setting up Worker Pools
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
moduleSettings = {
"cbq" : {
"registerWorkers" : true // this is also the default
}
};
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
// config/cbq.cfc
component {
function configure() {
newConnection( "default" ).setProvider( "DBProvider@cbq" );
newWorkerPool( "default" )
.forConnection( "default" )
.setTimeout( 30 )
.setMaxAttempts( 5 )
.setQuantity( 3 );
newWorkerPool( "default" )
.forConnection( "default" )
.onQueues( "emails" )
.setTimeout( 60 )
.setMaxAttempts( 10 )
.setQuantity( 10 );
}
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
cbq Model Helper
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
// handlers/Main.cfc
component {
property name="cbq" inject="cbq@cbq";
function create() {
var user = createUser( rc );
cbq.job( "SendWelcomeEmailJob", {
"userId": user.getId()
} ).dispatch();
}
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Job Chains
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
// FulfillOrderJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {
function handle() {
var productId = getProperties().productId;
processPayment( productId );
getInstance( "SendProductLinkEmail" )
.onConnection( "fulfillment" )
.setProperties( {
"productId" : productId,
"userId" : getProperties().userId
} )
.dispatch();
}
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
// handlers/Main.cfc
component {
function create() {
getInstance( "FulfillOrderJob" )
.setProperties( { "productId": rc.productId } )
.chain( [
getInstance( "SendProductLinkEmail" )
.onConnection( "fulfillment" )
.setProperties( {
"productId" : rc.productId,
"userId" : auth().getUserId()
} )
] )
}
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
// handlers/Main.cfc
component {
property name="cbq" inject="cbq@cbq";
function create() {
cbq.chain( [
cbq.job( "FulfillOrderJob", { "productId": rc.productId } ),
cbq.job( job = "SendProductLinkEmail", properties = {
"productId": rc.productId,
"userId": auth().getUserId()
}, connection = "fulfillment" ),
] ).dispatch();
}
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Job Batches
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
// handlers/Main.cfc
component {
property name="cbq" inject="cbq@cbq";
function create() {
cbq.batch( [
cbq.job( "ImportCsvJob", { "start": 1, "end": 100 } ),
cbq.job( "ImportCsvJob", { "start": 101, "end": 200 } ),
cbq.job( "ImportCsvJob", { "start": 201, "end": 300 } ),
cbq.job( "ImportCsvJob", { "start": 301, "end": 400 } ),
cbq.job( "ImportCsvJob", { "start": 401, "end": 500 } )
] ).then( cbq.job( "ImportCsvSuccessfulJob" ) )
.catch( cbq.job( "ImportCsvFailedJob" ) )
.finally( cbq.job( "ImportCsvCompletedJob" ) )
.dispatch();
}
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Demo
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Bonus
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
Delay the Current Job
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
// MonitorPendingCartJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {
function handle() {
var cartId = getProperties().cartId;
var cart = getInstance( "Cart" ).findOrFail( cartId );
if ( cart.isConfirmed() || cart.isCancelled() ) {
return;
}
if ( abs( dateDiff( "n", cart.getModifiedDate(), now() ) ) > 60 ) {
cart.cancel();
return;
}
getInstance( "MonitorPendingCartJob" )
.setProperties( { "cartId" : cartId } )
.delay( 15 * 1000 )
.dispatch();
}
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
// MonitorPendingCartJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {
function handle() {
var cartId = getProperties().cartId;
var cart = getInstance( "Cart" ).findOrFail( cartId );
if ( cart.isConfirmed() || cart.isCancelled() ) {
return;
}
if ( abs( dateDiff( "n", cart.getModifiedDate(), now() ) ) > 60 ) {
cart.cancel();
return;
}
this.release( 15 * 1000 );
}
}
Using `release`
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321070/SLIDES_SPEAKERS-keynote.005.png)
// MonitorPendingCartJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {
function handle() {
var cartId = getProperties().cartId;
var cart = getInstance( "Cart" ).findOrFail( cartId );
if ( cart.isConfirmed() || cart.isCancelled() ) {
return;
}
if ( abs( dateDiff( "n", cart.getModifiedDate(), now() ) ) > 60 ) {
cart.cancel();
return;
}
var delay = 2 ^ this.getCurrentAttempt() * 1000;
this.release( delay );
}
}
Exponential Backoff
![](https://s3.amazonaws.com/media-p.slid.es/uploads/847439/images/11321246/SLIDES_SPEAKERS-keynote.007.png)
ITB 2024 — cbq
By Eric Peterson
ITB 2024 — cbq
- 118