Browserless Drupal:

Render Unto Caesar, Don't Render Unto APIs

https://slides.com/lawrencemiller/renderuntocaesar

"Headless?"

So, What Are We Talking About Here?

I mean, isn't this just Web Services?

There's no "just" in "Web Services"

Ask anyone who wrote code before about 1998.  Seriously.

Chapter 1:

The Drupal Data Model

Doing things the

Drupal way:

• Nodes
• Views

Diverging from the Drupal Way

Nodes are spread out across LOTS of tables; writes are not cheap operations.  Sometimes a dedicated table is appropriate.

Of course we can add a pie chart to check the status.

(Using charts and charts_google)

YACM

/**
 * @param $node
 * @param $view_mode
 */
function uploader_batch_view_node_view($node, $view_mode)
{
  if ($node->type == "batch" && $view_mode == 'full') {
    // count the number of non-new status documents from this batch
    $non_new = db_query("select ds.status_text, count(*) from {documents} d INNER JOIN document_statuses AS ds on (d.status = ds.status_id) WHERE d.batch_id = $node->nid group by ds.status_text")->fetchAllKeyed();
    $node->content['batch_data'] = $non_new;
    return $node;

  }
}

/**
 * @param $batchID
 * @param string $limit
 * @return string
 */
function uploader_batch_view_render_documents($batchID, $limit = '10')
{
  $header_row = array(
    array('data' => 'Document ID', 'field' => 'd.document_id'),
    array('data' => 'Text', 'field' => 'd.document_text'),
    array('data' => 'Status', 'field' => 'ds.status_text'),
  );
  $query = db_select('documents', 'd')
    ->condition('batch_id', $batchID)
    ->fields('d', array('document_id','document_text'))
    ->fields('ds', array('status_text'));
  $query->join('document_statuses', 'ds', 'd.status = ds.status_id');
  $query = $query->extend('PagerDefault')->limit($limit);
  $query = $query->extend('TableSort')->orderByHeader($header_row);
  $result = $query->execute();
  foreach ($result as $row) {
    $table_rows[] = array($row->document_id, $row->document_text, $row->status_text);
  }
  return theme_table(array('header' => $header_row,
    'rows' => $table_rows, 'attributes' => array(), 'empty' => 'Data not found',
  ));
}

node--whatever.tpl.php

<article id="node-<?php print $node->nid; ?>" class="<?php print $classes; ?> clearfix"<?php print $attributes; ?>>

  <?php if ($user_picture || !$page || $display_submitted): ?>
    <header>
      <?php print $user_picture; ?>

      <?php print render($title_prefix); ?>
      <?php if (!$page): ?>
        <h2<?php print $title_attributes; ?>><a href="<?php print $node_url; ?>"><?php print $title; ?></a></h2>
      <?php endif; ?>
      <?php print render($title_suffix); ?>

      <?php if ($display_submitted): ?>

        <p class="submitted">
          <?php print $submitted; ?>
          <time pubdate datetime="<?php print $submitted_pubdate; ?>">
            <?php print $submitted_date; ?>
          </time>
        </p>

      <?php endif; ?>
    </header>
  <?php endif; ?>

  <div class="content"<?php print $content_attributes; ?>>
    <?php
    // We hide the comments, tags and links now so that we can render them later.
    hide($content['comments']);
    hide($content['links']);
    hide($content['field_tags']);

    $slice_colors = array(
      "New" => "blue",
      "Processing" => "yellow",
      "Accepted" => "green",
      "Rejected" => "red",
    );

    $foo = array_values($content['batch_data']);
    //dsm($foo);
    $ints = array();
    foreach ($foo as $f) {
      $ints[] = (int)$f;
    }

    $keys = array_keys($content['batch_data']);
    foreach ($keys as $k) {
      $slices[] = array("color" => $slice_colors[$k]);
    }

    $chart = array(
      '#type' => 'chart',
      '#title' => 'Batch',
      '#chart_type' => 'pie',
      '#slices' => $slices,
    );
    $chart['pie_data'] = array(
      '#type' => 'chart_data',
      '#title' => 'Hello, world!',
      '#labels' => array_keys($content['batch_data']),
      '#data' => $ints,
    );
    print render($chart);
    print render($content);
    print uploader_batch_view_render_documents($node->nid);
    print theme('pager', array('tags' => array()));
    ?>
  </div>
  <!-- /.content -->

  <?php if (!empty($content['field_tags']) || !empty($content['links'])): ?>
    <footer>
      <?php print render($content['field_tags']); ?>
      <?php print render($content['links']); ?>
    </footer>
  <?php endif; ?>

  <?php print render($content['comments']); ?>

</article><!-- /.node -->

Chapter 2:

REST

Modules required

• services
• rest_server

Services REST provides CRUD operations for:

• Nodes
• Users
• Taxonomy
• Comments
• Files
• System variables
• Views (with services_views)

Create a custom Resource if:

• you want to access objects other than those
• you created a custom table
• you need to adjust the HTTP Response codes

hook_services_resources

/**
 * Implements hook_services_resource()
 * @return array
 * @todo stop returning a 401 for malformed JSON.  Return 406 instead.
 */
function uploader_resource_services_resources() {
  return array(
    'uploader' => array(
      'operations' => array(
        'retrieve' => array(
          'help' => 'handles the RETRIEVE method (GET)',
          'callback' => '_uploader_resource_retrieve',
          'access callback' => '_uploader_resource_access',
          'access arguments' => array('view'),
          'access arguments append' => FALSE,
          'args' => array(
            array(
              'name' => 'batchID',
              'type' => 'int',
              'description' => 'batchID to check on',
              'source' => array('path' => '0'),
              'optional' => FALSE,
            ),
          ),
        ),
        'create' => array(
          'help' => 'handles the CREATE method of the uploader service',
          'callback' => '_uploader_resource_create',
          'access callback' => '_uploader_resource_access',
          'access arguments' => array('view'),
          'access arguments append' => FALSE,
          'args' => array(
            array(
              'name' => 'documents',
              'type' => 'struct',
              'description' => 'upload JSON payload',
              'source' => 'data',
              'optional' => FALSE,
            ),
          ),
        ),
      ),
    ),
  );
}

/**
 * Implements the REST retrieve callback:
 *
 * When the REST server receives a GET request on this resource, this callback
 * will fire.
 *
 * @param $batchID
 * @return array
 *
 */
function _uploader_resource_retrieve($batchID) {
  Global $base_url;
  $output = array();

  // for some reason is_int doesn't catch strings that include integers
  // but start with a non-integer, such as "abc11".

  $pattern = '/\D/';
  preg_match($pattern, $batchID, $matches);
  if ($matches[0]) {
    $output['Error'] = "Invalid Batch ID: $batchID";
  }
  else {
    $batch = node_load($batchID);
    if ($batch) {
      $created = date(DATE_RFC2822, $batch->created);
      $count = $batch->field_number_of_documents['und'][0]['value'];
      $status = $batch->field_status['und'][0]['value'];

      // convert numeric status into human-readable value
      $status_human = db_query("SELECT ds.status_text from {document_statuses} ds where ds.status_id = :sid", array(":sid" => $status))->fetchField();

      // count the number of non-new status documents from this batch
      $non_new = db_query("SELECT d.document_id from {documents} d WHERE d.batch_id=:bid AND d.status != :st", array(
        ":bid" => $batchID,
        ":st" => '1'
      ))->rowCount();

      $output['Batch ID'] = $batch->nid;
      $output['Location'] = $base_url . "/rest/documents/" . $batchID . ".json";
      $output['Upload Timestamp'] = $created;
      $output['Status'] = $status_human;
      $output['Comments'] = "$non_new / $count Processed";
    }
    else {
      drupal_add_http_header('Status', '500 Error');
      $output['Error'] = "Batch #$batchID not found.";
    }
  }
  return $output;

}

REST Retrieve

REST Create

/**
 * Implements the REST create hook:
 *
 * When the REST server receives a POST request on this resource, this callback
 * will fire.
 *
 * @param $documents
 * @return array
 *
 * @todo break the node saving process into a hook_insert (or hook_node_insert)?
 * @todo on success, instead of generating the output here, use the retrieve callback.
 */
function _uploader_resource_create($documents) {
  // Check to see if $documents is defined.  If the payload was valid JSON,
  // it will be.  Otherwise, it'll be empty?  Looks like malformed JSON
  if (is_null($documents)) {
    drupal_add_http_header('Status', '406 Not Acceptable');
  }
  Global $base_url;

  // create the batch, save it.  $batch becomes blessed as a node.

  $output = array();
  $batch = new stdClass();
  $batch->type = "batch";
  $batch->title = "Batch Upload";
  $batch->field_status['und'][0]['value'] = "1";

  if (isset($documents['documents'])) {
    $batch->field_number_of_documents['und'][0]['value'] = count($documents['documents']);
    $output['Status'] = count($documents['documents']) . " Documents Received";
  }
  else {
    $batch->field_number_of_documents['und'][0]['value'] = 0;
    $output['Status'] = "Error: No Documents Received";
  }
  node_save($batch);
  $location = $base_url . "/rest/documents/" . $batch->nid . ".json";
  try {
    drupal_add_http_header("Location", $location);
    drupal_add_http_header('Status', '202 Accepted');

    if (isset($documents['documents'])) {
      foreach ($documents['documents'] as $document) {
        // entity name
        $document['entity']['0']['name'] = htmlentities($document['entity']['0']['name']);
        // Certifying Agent
        $document['fields']['0']['value'][0] = htmlentities($document['fields']['0']['value'][0]);
        // Certification Number
        $document['fields']['1']['value'][0] = htmlentities($document['fields']['1']['value'][0]);
        // Regulator
        $document['fields']['2']['value'][0] = htmlentities($document['fields']['2']['value'][0]);
        $json_d = json_encode($document);
        $json_d = html_entity_decode($json_d);
        $json_d = stripslashes($json_d);
        $docid = db_insert('documents')
          ->fields(array(
            'document_text' => $json_d,
            'batch_id' => $batch->nid,
            'id' => $document['id'],
            'status' => '1',
          ))
          ->execute();
        // add an item to the queue to go do something with later
        // which will be implemented in another module
        $queue = DrupalQueue::get('uploader_documents');
        $queue->createItem(array(
          'document_id' => $docid,
          'document_text' => $json_d
        ));
      }
    }
  }
  catch
  (Exception $e) {
    drupal_add_http_header('Status', '500 Error');
    $output['Error'] = $e->getMessage();
  }

  $output['Batch ID'] = $batch->nid;
  $output['Location'] = $location;
  $output['Upload Timestamp'] = date(DATE_RFC2822);
  return $output;
}

REST "Clients"

• cURL
  • on the command line
  • underpins everything
• Guzzle
  • lightweight PHP library
  • install with Composer
  • https://github.com/guzzle/guzzle
• drupal_http_request
  • If you're reading this, you've already got it.
• Your browser
  • Chrome REST Console
  • Firefox POSTER
  • or, you know, the location bar

Chapter 3:

AMQP

What is AMQP?

There are innumerable ways to implement AMQP.

But all of them will involve decoupling individual components.

RabbitMQ

A well supported AMQP compliant message broker.

• Comes with an admin front-end!  Cunning!
• https://www.rabbitmq.com/

Exchanges

Queues

Routing Keys

• A label separating this message from other messages
• Only words and dots are allowed; you can string together long patterns like big.red.dog:
  • big.*.dog
  • fluffy.#

Bindings

Bindings

All finished!

AMQP Modules needed

• message_broker
• message_broker_amqp

• Your custom Module

Publisher

/**
 * @param $object
 * @return bool
 */
function query_publisher_send_request($object)
{
  $data = $object->value();
  $options = array();
  $options['routing_key'] = variable_get('query_publisher_mvpams_routing_key');
  $options['reply_to'] = variable_get('query_publisher_mvpams_reply_to');
  $options['correlation_id'] = $data->nid;

  query_publisher_publish($data->field_payload['und'][0]['value'], variable_get('query_publisher_mvpams_exchange'), $options);
  return TRUE;
}

/**
 * @param $message
 * @param $exchange
 * @param $options
 */
function query_publisher_publish($message, $exchange, $options)
{
  $broker = message_broker_get();
  try {
    $broker->sendMessage($message, $exchange, $options);
    watchdog('AMQP', 'Sent Query AMQP message to %exchange', array('%exchange' => $exchange), WATCHDOG_NOTICE);
  } catch (InvalidArgumentException $e) {
    watchdog('AMQP', 'AMQP Query Error (%code): %message', array(
        '%code' => $e->getCode(),
        '%message' => $e->getMessage()
      ),
      WATCHDOG_ERROR);
  }
}

Consumer

/**
 * Implements hook_message_broker_consumers
 * @param $self_name
 * @return array
 */
function query_consumer_message_broker_consumers($self_name) {
  $consumers = array();
  $consumers['Query Request'] = array(
    'queue' => variable_get('query_consumer_request_queue'),
    'callback' => 'query_consumer_request_handler',
  );
  return $consumers;
}

/**
 * Callback function for the request queue consumer.
 *
 * This function is responsible for taking a request for validation from ID
 * and creating a node with the information contained within.
 *
 * @param $request
 * @param $ack
 */
function query_consumer_request_handler($request, $ack) {
  watchdog('amqp',"Query Consumer activated.");
  $node = new stdClass;
  $node->type = "verification_query";
  $body = json_decode($request->body);
  $b_array = array();
  $b_array['canonical'] = array();
  $b_array['exact'] = array();
  foreach ($body->{'canonical'} as $can) {
    foreach (array_keys(get_object_vars($can)) as $key) {
      $b_array['canonical'][$key] = $can->$key;
    }
  }
  foreach ($body->{'exact'} as $ex) {
    foreach (array_keys(get_object_vars($ex)) as $key) {
      $b_array['exact'][$key] = $ex->$key;
    }
  }
  $node->title = $b_array['canonical']['entity.name'];
  $node->field_organization_id['und'][0]['value'] = $b_array['exact']['org id'];
  $node->field_country['und'][0]['value'] = $b_array['exact']['schema.countrycode'];
  $node->field_reply_to['und'][0]['value'] = $request->get("reply_to");
  $node->field_payload['und'][0]['value'] = $request->body;
  try {
    node_save($node);
    watchdog('amqp', 'Created Verification Query Node %nid', array('%nid' => $node->nid), WATCHDOG_INFO);
    $ack();
  }
  catch (Exception $e) {
    watchdog('amqp', 'Failed to create Verification Query: %error', array('%error' => $e->getMessage()), WATCHDOG_ERROR);
  }

}

Starting the Consumer

• message_broker_amqp includes a "drush cons" command
• shell script wrapper to restart drush cons if necessary
• pkill cron job if your behavior gets flaky
• Upstart if you wanna get fancy

drush cons

ldpm-laptop,[/path/to/drupal/sites/your.site],11:19 AM%> drush cons
Connected successfully to the AMQP message broker at turtle.rmq.cloudamqp.com:5672
Starting all consumers ...

(use "screen" to detach)

"drushalive.sh"

#!/bin/bash

ORGANIC="/var/www/afilias_d7/htdocs/sites/mvp.organic"
NGO="/var/www/afilias_d7/htdocs/sites/iddemo.info"

if [ $1 == "ORGANIC" ]; then
  cd $ORGANIC
else
  cd $NGO
fi

until drush cons; do
	echo "Drush crashed with exit code $?. Respawning.." >&2
	sleep 1
done

(use "screen" to detach)

Upstart: /etc/init/drushalive.conf

# drushalive - drushalive process script
description "Drush Alive Script for Organic"

start on starting tty1 and net-device-up IFACE!=lo
stop on starting rc RUNLEVEL=[016]

console output

respawn
respawn limit 10 20

chdir /var/www/afilias_d7/htdocs/sites/mvp.organic

 
# What to execute
script
    if [ -r /etc/default/drushalive ]; then
        . /etc/default/drushalive
    fi
	exec su -c '/usr/bin/drush cons' dpk >>/var/log/drush/drushorganic.log 2>&1
#   start-stop-daemon --start -c $RUN_AS --exec $DAEMON -- $DAEMON_OPTS
end script

pkill: if drush gets twitchy

[someuser@yourhost ~]$ crontab -l
*	*/2	*	*	*	/usr/bin/pkill -f drush.php

The combination of the keep-alive method (either the script or Upstart) and pkill should keep the consumer running all the time even if it errors out.

Thank you!

ldpm @ d.o

@ldpm

lmiller@afilias.info