Feeds and the Feeds ecosystem

Hi. I'm Chris.


d.o & irc: twistor           

twitter: chrisleppanen

Feeds

Architecture

  1. Fetcher
  2. Parser
  3. Processor

     

It fetches!


  • HTTP
  • File/directory upload
  • SOAP :(
  • LDAP
  • Email
  • Databases
  • But wait, there's more.

It parses!


  • CSV
  • RSS/Atom
  • OPML
  • Sitemap files?
  • XML via XPath
  • XML via XSLT
  • JSON via JSONPath
  • KML
  • Pinterest
  • Slideshare
  • Shape files
  • ICCD
  • iCal
  • LDAP entries
  • OAI-PMH
  • Clients ?!
  • Twitter
  • Facebook
  • REGEX!!!
  • Wordpress
  • Sharepoint
  • Flickr
  • XLS
  • Excel
  • OpenERP
  • KML
  • Email headers
  • FFFFOUND!
  • OFX
  • Readability
  • Goutte
  • GDAL
  • Filemaker
  • Youtube

It processes!


  • Nodes
  • Users
  • Taxonomy terms
  • Data tables
  • Comments
  • Commerce products
  • Contacts
  • RedHen contacts
  • RedHen organizations
  • Locations
  • Arbitrary entities

Mapping!


Mapping


Will it do other things?


Yes.


  • Schedule periodic imports via cron
  • Lookup/create taxonomy terms
  • Pubsubhubbub
  • Attach to nodes for user created feeds
  • Expires old entries
  • Has support for most fields

I want it to do more.

Ok fine.

http://drupal.org/project/issues/feeds

There's probably an existing issue.

Maybe even a patch.

Someone might even respond to it.

Support for other modules should go in those modules.

Feeds Tamper





1.0 Woot!


Perfect and bug free.

Let's modify some data


  • String to UNIX timestamp
  • Unix timestamp to date
  • Keyword filter
  • Required filter
  • Encode/decode HTML entities
  • Make URLs absolute
  • Strip HTML tags
  • Explode a string into an array
  • Remove empty items
  • Implode an array into a string
  • Make a list unique
  • Format numbers
  • Convert to numbers
  • Math!
  • Hashes
  • Convert to/from country codes
  • Convert to/from US states
  • Entity Field Query finder
  • Rewrite
  • Set default value
  • Convert case
  • Convert to boolean
  • Find-replace
  • Find-replace with REGEX
  • Format a string
  • Pad a string
  • Trim a string
  • URL encode/decode

But that's not enough things!



I hear you.


Plugins!

Declare that you have plugins.





/**
 * Implements hook_ctools_plugin_directory().
 */
function my_module_ctools_plugin_directory($module, $plugin) {
  if ($module == 'feeds_tamper') {
    return 'feeds_tamper_plugins'; // The directory that contains said plugins.
  }
}

Weird global $plugin thingy.




<?php

/**
 * @file
 * Contains my super sweet Feeds Tamper plugin.
 */

$plugin = array(
  'form' => 'my_module_plugin_form',
  'callback' => 'my_module_callback',
  'name' => 'Super sweet tamper plugin',
  'multi' => 'loop',
  'category' => 'Other',
);

Add a form.




/**
 * Configuration form.
 */
function my_module_plugin_callback($importer, $element_key, $settings) {
  $form = array();

  $form['help'] = array(
    '#markup' => t('It does things to stuff.'),
  );

  return $form;
}

Add the callback.





/**
 * Modify things.
 */
function my_module_callback($result, $item_key, $element_key, &$field, $settings, $source) {

  $field = "You've been modified.";

}

All together now.

<?php

$plugin = array(
  'form' => 'my_module_plugin_form',
  'callback' => 'my_module_callback',
  'name' => 'Super sweet tamper plugin',
  'multi' => 'loop',
  'category' => 'Other',
);function my_module_plugin_callback($importer, $element_key, $settings) {
$form = array(); $form['help'] = array( '#markup' => t('It does things to stuff.'), ); return $form; }
function my_module_callback($result, $item_key, $element_key, &$field, $settings, $source) { $field = "You've been modified."; }

Let's do something.


Yay for live demos.

http://www.aperfectbrick.com

It has inspiration quotes and we need inspiration.

  1. Scrape a website's HTML
  2. Create some article nodes
  3. Assign some terms
  4. Import some images
  5. Make the article title UPPERCASE
  6. Get all past articles.

What we'll need:


  1. Feeds
  2. Feeds Tamper
  3. Feeds XPath Parser
  4. Feeds Crawler

Feeds XPath Parser


Feeds Crawler


Allows us to paginate through a site.

Has some modes:

  • URL patterns
  • XPath for the "next" link
  • Auto-mode for Atom rel="next" links




It's demo time

Entity Field Query finder

It's new and it's awesome.

Help!

The future and beyond...


  • Rapidly approaching a stable release.
  • 8.x - ported, but broken atm
  • 3.0 - backport from 8.x that uses the Entity API module exclusively.
  • New UI - Are there any designers in the audience?

Links

  • https://drupal.org/project/feeds
  • https://drupal.org/project/feeds_tamper
  • https://drupal.org/project/feeds_xpathparser
  • https://drupal.org/project/feeds_crawler
  • https://drupal.org/node/856644 (plugin list)

Questions?

Compliments?

Complaints?

Threats?


Feeds and the Feeds ecosystem

By Chris Leppanen

Feeds and the Feeds ecosystem

  • 1,293