Apachesolr API

Improving developer experience

Drupal Module

Shibin Das (D34dMan)

Agenda

Demonstrate how to use apachesolr_api module
Discuss thought process behind some strategy used
Talk about development roadmap

What it is...

Hides a lot of implementation details while querying solr.

It helps developer focus on business logic.

What it is not

Website builders and site admins should not need this module unless some other modules has dependency on apachesolr_api.

It does not expose any User interface or admin screen.

Assumptions

Target audience are Drupal Backend Developers.
The Developers has the knowledge of the following
- Drupal Hooks
- Drupal Theme functions
Knowledge of setting up of Apache Solr and its configuration is not required, but should have access to a working setup.
Knowledge of Apache Solr Queries.
Configurations are not stored in database, but in code.

Installation

Using Git

Using Drush

Using Composer

git clone --branch 7.x-1.x https://git.drupal.org/project/apachesolr_api.git

drush dl apachesolr_api

composer require drupal/apachesolr_api

Apachesolr API is available on Drupal.org

Depedencies

Apache Solr Search
https://www.drupal.org/project/apachesolr

Optional Depedencies

YAML Parser
https://www.drupal.org/project/yaml_parser

Supports

Panelizer
Flags

Configuration

Apache Solr Search has to be configured and able to communicate with a solr instance
Apache Solr API doesn't need any extra configuration
Create a custom module to consume the API
In the example that follows, we would be storing some configuration in yaml files. For this, Yaml Parser and its dependent library must be installed.

Helps with...

Query apachesolr
Search autocomplete
Search results
Add Panelizer content to document index
Add Flag Module's data to document index
Retrieve facets links
Apply filters
Pagination
Apply arbitrary filters and facets with custom queries

show me the code!

CODE


/**
 * Implements hook_apachesolr_api_config().
 */
function mymodule_apachesolr_api_config() {
  $data = [
    'pages' => [],
  ];
  return $data;
}

Initial Setup

File         : mymodule.module
Location     : sites/all/modules/custom/mymodule/mymodule.module


/**
 * Implements hook_apachesolr_api_config().
 */
function mymodule_apachesolr_api_config() {
  $data = [
    'pages' => [
      'push_grid_random' => [],
    ],
  ];
  return $data;
}


/**
 * Implements hook_apachesolr_api_config().
 */
function mymodule_apachesolr_api_config() {
  $data = [
    'pages' => [
      'push_grid_random' => [
        'apachesolr' => [
          'params' => [
            'fl' => 'entity_id id',
            'fq' => [
              'bundle:(article OR page)',
            ],
          ],
        ],
      ],
    ],
  ];
  return $data;
}


/**
 * Implements hook_apachesolr_api_config().
 */
function mymodule_apachesolr_api_config() {
  $data = [
    'pages' => [
      'push_grid_random' => [
        'apachesolr' => [
          'params' => [
            'fl' => 'entity_id id',
            'fq' => [
              'bundle:(article OR page)',
            ],
          ],
          'map' => [
            'nid' => 'entity_id',
          ],
        ],
      ],
    ],
  ];
  return $data;
}

CODE


/**
 * Implements hook_apachesolr_api_config().
 */
function mymodule_apachesolr_api_config() {
  $data = array();
  $path = drupal_get_path('module', 'mymodule');
  if ($path) {
    $filepath = $path . '/apachesolr_api_config.yaml';
    $realpath = drupal_realpath($filepath);
    $yaml = file_get_contents($filepath);
    $data = yaml_parser_parse_string($yaml);
  }
  return $data;
}

Initial Setup (using yaml configuration file)

File         : mymodule.module
Location     : sites/all/modules/custom/mymodule/mymodule.module

CODE

pages:
  push_grid_random:
    apachesolr:
      params:
        fl: "entity_id id"
        fq:
          - bundle:(article OR page)

Initial Setup (using yaml configuration file)

File         : apachesolr_api_config.yaml
Location     : sites/all/modules/custom/mymodule/apachesolr_api_config.yaml

CODE

# Use "pages" key to store various query.
pages:
  # A custom key that would be used to retrieve the results of a query later.
  push_grid_random:
    apachesolr:
      # The parameters of the solr query.
      params:
        fl: "entity_id"
        fq:
          - bundle:(article OR page)

Initial Setup (using yaml configuration file)

File         : apachesolr_api_config.yaml
Location     : sites/all/modules/custom/mymodule/apachesolr_api_config.yaml

CODE

/**
 * Get push grid as render array.
 */
function mymodule_get_push_grid() {
  $results = apachesolr_api_fetch('push_grid_random', '*', []);
  foreach ($results['rows'] as $item) {
    // The results can be rendered using a view mode.
    $node = node_load($item['entity_id']);
    if ($node) {
      $items[] = node_view($node, 'teaser');
    }
  }
  $content = [
    '#theme' => 'mymodule_push_grid',
    '#items' => $items,
  ];
}

Querying apachesolr and display data

File         : mymodule.module
Location     : sites/all/modules/custom/mymodule/mymodule.module

CODE

pages:
  push_grid_random:
    apachesolr:
      params:
        fl: "entity_id id label url"
        fq:
          - bundle:(article OR page)

Avoiding node_load()

File         : apachesolr_api_config.yaml
Location     : sites/all/modules/custom/mymodule/apachesolr_api_config.yaml

CODE

/**
 * Get push grid as render array.
 */
function mymodule_get_push_grid() {
  $results = apachesolr_api_fetch('push_grid_random', '*', []);
  $content = [
    '#theme' => 'mymodule_push_grid',
    '#items' => $results['rows'],
  ];
}

Avoiding node_load()

File         : mymodule.module
Location     : sites/all/modules/custom/mymodule/mymodule.module

CODE

pages:
  push_grid_random:
    apachesolr:
      params:
        fl: "entity_id id label url"
        fq:
          - bundle:(article OR page)
      map:
        nid: entity_id
        title: label
        link: url

Preprocessing variables for theme

File         : apachesolr_api_config.yaml
Location     : sites/all/modules/custom/mymodule/apachesolr_api_config.yaml

Search Autocomplete

CODE

pages:
  search_autocomplete:
    url: ajax/rmh-search/%
    menu_item:
      title: Search Suggestions
      page callback: 'mymodule_search_autocomplete_callback'
      page arguments:
        - 2
      access arguments:
        - access content
    apachesolr:
      fuzzy: TRUE
      fuzzy_weight: 1
      params:
        fl: "id, label, content, teaser, entity_id, bundle, entity_type, path, url"
        hl: TRUE
        hl.fl: "content, label"
        rows: 10
        hl.simple.pre: "<mark>"
        hl.simple.post: "</mark>"
        hl.simple.snippets: TRUE
        qf:
         - "label^20"
         - "content^5"
      map:
        label: label
        content: content
        url: path

Configuration for autocomplete callback

File         : apachesolr_api_config.yaml
Location     : sites/all/modules/custom/mymodule/apachesolr_api_config.yaml

CODE


/**
 * Autocomplete callback.
 */
function mymodule_search_autocomplete_callback($search_string) {
  $search_string = empty($search_string) ? "*" : $search_string;
  $results = apachesolr_api_fetch('search_autocomplete', $search_string, []);
  drupal_json_output($results['rows']);
  drupal_exit();
}

Autocomplete callback

File         : mymodule.module
Location     : sites/all/modules/custom/mymodule/mymodule.module

Search Result Page

CODE

pages:
  search_detail_page:
    url: search-results/%
    menu_item:
      title: Search results
      page callback: 'mymodule_search_details_page_callback'
      page arguments:
        - 1
      access arguments:
        - access content
    apachesolr:
      fuzzy: TRUE
      fuzzy_weight: 1
      params:
        fl: "id, label, content, teaser, entity_id, bundle, entity_type, path, url"
        hl: TRUE
        hl.fl: "content, label"
        rows: 10
        hl.snippets: 1
        hl.fragsize: 300
        hl.simple.pre: "<mark>"
        hl.simple.post: "</mark>"
        hl.simple.snippets: TRUE
        qf:
         - "label^20"
         - "content^5"
      map:
        label: label
        content: content
        url: path

Configuration for Search Results Page

File         : apachesolr_api_config.yaml
Location     : sites/all/modules/custom/mymodule/apachesolr_api_config.yaml

CODE


/**
 * Search details page callback.
 */
function mymodule_search_details_page_callback($search_string) {
  $search_string = empty($search_string) ? "*" : $search_string;
  $results = apachesolr_api_fetch('search_detail_page', $search_string, []);
  if ($results['meta']['total_rows']) {
    $output = theme('mymodule_search_results', $results);
  }
  else {
    $results['meta']['title'] = t("We're sorry, no results found");
    $output = theme('mymodule_search_results_empty', $results);
  }
  return $output;
}

Search Detail Page Callback

File         : mymodule.module
Location     : sites/all/modules/custom/mymodule/mymodule.module

Panelizer Integration

The Problem

Out of the box Apache Solr Search module can index nodes and their fields, but Panelizer pages won't be indexed

The Solution

As discussed in a blog post at previousnext, we could get the rendered output form page_manager, cleaning it using apachesolr clean text function, and setting it as a content of apachesolr document.

CODE

panelized content types:
  - story
  - landing_page
  - article_page

Panelizer Configuration

File         : apachesolr_api_config.yaml
Location     : sites/all/modules/custom/mymodule/apachesolr_api_config.yaml

Flag Integration

The Problem

Flag module does not store its data in normal Drupal fields. So the user interactions are not stored in the index.

The Solution

Similar to what we do for Panelizer content, inject the flag data into document while it gets build.

CODE

flag:
  save_for_later:
    entity:
      node:
        bundle:
         - module_intro
    flag_machine_name: save_for_later
  recommend:
    entity:
      node:
        bundle:
         - module_intro
    flag_machine_name: recommend

Flags Configuration

File         : apachesolr_api_config.yaml
Location     : sites/all/modules/custom/mymodule/apachesolr_api_config.yaml

Facets

CODE

pages:
  documentation_center:
    apachesolr:
      params:
        fl: "entity_id id"
        rows: 9
        facet: true
        facet.field: sm_field_media_collection
        facet.mincount: 1
        fq:
          - bundle:(media_pdf OR media_sample)
        sort: ds_changed desc, id desc
      map:
        nid: entity_id

Facets Configuration

File         : apachesolr_api_config.yaml
Location     : sites/all/modules/custom/mymodule/apachesolr_api_config.yaml

Filters

CODE

pages:
  documentation_center:
    apachesolr:
      params:
        fl: "entity_id id"
        rows: 9
        facet: true
        facet.field: sm_field_media_collection
        facet.mincount: 1
        fq:
          - bundle:(media_pdf OR media_sample)
        sort: ds_changed desc, id desc
      map:
        nid: entity_id

Filters configuration (Query)

File         : apachesolr_api_config.yaml
Location     : sites/all/modules/custom/mymodule/apachesolr_api_config.yaml

CODE

pages:
  documentation_center:
    apachesolr:
      filters:
        # Filter type. Single boolean query.
        query:
          # Custom identifier for the filter.       
          downloadable:
            # Dynamic query fragment that will be applied when the filter is active
            solr_query: bs_field_media_downloadable:true
            # the url query string to hold the filter status
            query_string: bif_dc_dwld
      params:
        fl: "entity_id id"
        rows: 9
        facet: true
        facet.field: sm_field_media_collection
        facet.mincount: 1
        fq:
          - bundle:(media_pdf OR media_sample)
        sort: ds_changed desc, id desc
      map:
        nid: entity_id

Filters configuration (Query)

File         : apachesolr_api_config.yaml
Location     : sites/all/modules/custom/mymodule/apachesolr_api_config.yaml

CODE


/**
 * Some page callback/panel that displays results.
 */
function mymodule_apachesolr_documentation_center_content() {

  ...

  $filters = drupal_get_query_parameters();
  $results = apachesolr_api_fetch('documentation_center', "*", [], $filters);

  $download_filter = $results['filters']['query']['downloadable'];

  // $download_filter is an array which contains status and url for the filter.

  ...
  
}

Query Filter

File         : mymodule.module
Location     : sites/all/modules/custom/mymodule/mymodule.module

CODE

pages:
  documentation_center:
    apachesolr:
      filters:
        # Collection of a field facet
        facet_fields:
          collections:
            solr_field: sm_field_media_collection
            query_string: bif_dc_cltn
            multiple_behaviour: OR
            field_type: taxonomy_term
      params:
        fl: "entity_id id"
        rows: 9
        fq:
          - bundle:(media_pdf OR media_sample)
        sort: ds_changed desc, id desc
      map:
        nid: entity_id

Filters configuration (Faceted)

File         : apachesolr_api_config.yaml
Location     : sites/all/modules/custom/mymodule/apachesolr_api_config.yaml

CODE


/**
 * Some page callback/panel that displays results.
 */
function mymodule_apachesolr_documentation_center_content() {

  ...

  $filters = drupal_get_query_parameters();
  $results = apachesolr_api_fetch('documentation_center', "*", [], $filters);
  
  foreach ($results['filters']['facet_field']['collections'] as $key => $filter) {
    // $filter contains status and url for the faceted filter.
  }

  ...
  
}

Query Filter

File         : mymodule.module
Location     : sites/all/modules/custom/mymodule/mymodule.module

CODE

pages:
  documentation_center:
    apachesolr:
      filters:
        # Single boolean query
        query:
          activated:
            solr_query: bs_field_media_activated_order:true
            query_string: bif_dc_ao
        # Collection of a field facet
        facet_fields:
          collections:
            solr_field: sm_field_media_collection
            query_string: bif_dc_cltn
            multiple_behaviour: OR
            field_type: taxonomy_term
      params:
        fl: "entity_id id"
        rows: 9
        facet: true
        facet.field: sm_field_media_collection
        facet.mincount: 1
        fq:
          - bundle:(media_pdf OR media_sample)
        sort: ds_changed desc, id desc
      map:
        nid: entity_id

Filters configuration (Faceted)

File         : apachesolr_api_config.yaml
Location     : sites/all/modules/custom/mymodule/apachesolr_api_config.yaml

CODE


/**
 * Some page callback/panel that displays results.
 */
function mymodule_apachesolr_documentation_center_content() {

  ...

  $filters = drupal_get_query_parameters();
  $results = apachesolr_api_fetch('documentation_center', "*", [], $filters);
  
  foreach ($results['filters']['facet_field']['collections'] as $key => $filter) {
    // $filter contains status and url for the faceted filter.
    // $filter also contains count and total count for the current query.
  }

  ...
  
}

Query Filter

File         : mymodule.module
Location     : sites/all/modules/custom/mymodule/mymodule.module

Pagination

Pagination Types

Basic Pagination
@todo
Cursor

CODE

pages:
  documentation_center:
    apachesolr:
      pagination:
        type: cursors
        query_string: bif_dc_csr
      filters:
        # Single boolean query
        query:
          activated:
            solr_query: bs_field_media_activated_order:true
            query_string: bif_dc_ao
      params:
        fl: "entity_id id"
        rows: 9
        facet: true
        facet.field: sm_field_media_collection
        facet.mincount: 1
        fq:
          - bundle:(media_pdf OR media_sample)
        sort: ds_changed desc, id desc
      map:
        nid: entity_id

Pagination Configuration

File         : apachesolr_api_config.yaml
Location     : sites/all/modules/custom/mymodule/apachesolr_api_config.yaml

CODE


/**
 * Some page callback/panel that displays results.
 */
function mymodule_apachesolr_documentation_center_content() {

  ...

  $filters = drupal_get_query_parameters();
  $results = apachesolr_api_fetch('documentation_center', "*", [], $filters);

  $next = $results['meta']['pagination']['next'];

  // $next contains the url to load more results as suitable for infinite scroll.

  ...
  
}

Pagination

File         : mymodule.module
Location     : sites/all/modules/custom/mymodule/mymodule.module

Dynamic Parameters

CODE

/**
 * Get push grid as render array.
 */
function mymodule_get_push_grid() {
  $params = [];
  global $user;
  // Add dynamic parameter to restrict results specific to current user as author.
  $params['fq'][] = 'is_uid:' . $user->uid;

  $results = apachesolr_api_fetch('push_grid_random', '*', $params);
  foreach ($results['rows'] as $item) {
    // The results can be rendered using a view mode.
    $node = node_load($item['nid']);
    if ($node) {
      $items[] = node_view($node, 'teaser');
    }
  }
  $content = [
    '#theme' => 'mymodule_push_grid',
    '#items' => $items,
  ];
}

Custom parameters - author is current user

File         : mymodule.module
Location     : sites/all/modules/custom/mymodule/mymodule.module

CODE

/**
 * Get push grid as render array.
 */
function mymodule_get_push_grid() {
  $seed = rand();
  $params = [
    'sort' => "random_$seed desc",
  ];
  $results = apachesolr_api_fetch('push_grid_random', '*', $params);
  foreach ($results['rows'] as $item) {
    // The results can be rendered using a view mode.
    $node = node_load($item['nid']);
    if ($node) {
      $items[] = node_view($node, 'teaser');
    }
  }
  $content = [
    '#theme' => 'mymodule_push_grid',
    '#items' => $items,
  ];
}

Custom parameters - randomise

File         : mymodule.module
Location     : sites/all/modules/custom/mymodule/mymodule.module

Roadmap

Support Basic Pagination
Support Sorting
Improve Flag integration
Performance improvement over facets and filtering
Port to Drupal 8

Apachesolr API

Agenda

What it is...

What it is not

Assumptions

Installation

Depedencies

Optional Depedencies

Supports

Configuration

Helps with...

show me the code!

CODE

CODE

CODE

CODE

CODE

CODE

CODE

CODE

Search Autocomplete

CODE

CODE

Search Result Page

CODE

CODE

Panelizer Integration

The Problem

The Solution

CODE

Flag Integration

The Problem

The Solution

CODE

Facets

CODE

Filters

CODE

CODE

CODE

CODE

CODE

CODE

CODE

Pagination

Pagination Types

CODE

CODE

Dynamic Parameters

CODE

CODE

Roadmap

Q & A

Thank You!