Use Cases for Drupal Queues

BCO-DMO

                 Help researchers w. their data

  • Prepare data for reuse
  • Generate metadata standards
  • Improve discovery w. semantics
  • Submit data to archives

Metadata

Collated...

  • RDF
  • ISO 19115-2
  • Dublin Core
  • Datasets
  • Instruments
  • Measurements
  • People
  • Cruises
  • Funding
  • Projects

 

Stored.

  • Caches
  • Indexes

When metadata changes...

     what to do with cached docs?

When metadata changes...

     what to do with cached docs?

  1. Figure out what needs updating
  2. Re-generate
  3. Update stores

"...I'll just update right away!"

When metadata changes...

"...I'll just update right away!"

      meaning...

 

 When a Data Manager DID ANYTHING, they had to wait.

 

Finding all referenced entities

http://vignette1.wikia.nocookie.net/r2d/images/5/59/Yo-hurry-up.jpg/revision/latest?cb=20141011053551

How do I...
 

  remember the relevant info,

    so I can defer the work,

      and ensure the work is done?

 


 

Drupal Queue API

Drupal Queue API

  1. Remember the relevant info

/**
 * Implements hook_entity_update().
 */
function mymodule_entity_update($entity, $entity_type) {

  ...

  // Remember relevant info.
  $item = array(
    'entity_type' => $entity_type,
    'entity' => $entity,
    'op' => 'update',
   
  );

  ...
}

Drupal Queue API

  2. Defer the work

...

// Create a queue to store items. 
// 'TRUE' means get a reliable queue type
//  - guarantees every item executes at least once
$queue = DrupalQueue::get('myWorkQueue', TRUE);

// Remember relevant info.
$item = array(
  'entity_type' => $entity_type,
    'entity' => $entity,
  'op' => 'update',
);

// Put an item into the queue.
$queue->createItem($item);

...

Drupal Queue API

  3. Ensure the work gets done

Somehow the queue needs to be processed

Our solution was to:

  1. Setup a drush command to process the queue
     
  2. Setup a cron job to run the drush command

Drupal Queue API

  3. Ensure the work gets done:    DRUSH


// Get the queue. 
// 'TRUE' means get a reliable queue type.
$queue = DrupalQueue::get('myWorkQueue', TRUE);

// Iterate over the items.
while($item = $queue->claimItem()) {

  // Get the relevant information.
  $entity_type = $item['entity_type'];
  $entity = $item['entity'];
  $operation = $item['op'];

  $success = FALSE;
  // Do some work with the information you stored.
  // Specify if the task was successful $success = ???.

  // If successful, delete the item as it is no longer needed.
  if ($success) {
    $queue->deleteItem($item);
  }
}

Drupal Queue API

  3. Ensure the work gets done:    SUCCESS


// Iterate over the items.
while($item = $queue->claimItem()) {

...

  // If successful, delete the item as it is no longer needed.
  if ($success) {
    $queue->deleteItem($item);
  }
}

When an item is not deleted,

it stays in the queue for next time.

RESOURCES

 

http://rbayliss.net/drupal-queue-api
http://www.slideshare.net/philipnorton42/drupal-7-queues
http://atendesigngroup.com/blog/batch-api-cron-queues
https://api.drupal.org/api/examples/queue_example!queue_example.module/
https://api.drupal.org/api/drupal/modules!system!system.queue.inc/group/queue/7

 

or hit me up:

 

  • Twitter:    @ashep_15
  • Email:       adam@whoi.edu

Use Cases for Drupal Queues

By Adam Shepherd

Use Cases for Drupal Queues

  • 510