MIP

Content Ingestion

What we had

with the previous one

  • simple configuration for a new roll-out
  • quite simple development for a new product
  • 8259 Python statements (according to Pylint)
  • manual category management (design decision)
  • sporadic crashes (very few during last months)
  • very few tests (hard regression analysis)

Products

  • iFortune
  • iMagazine
  • Fingerbooks (for premium contents)
  • 1.5
  • *TV (migration in progress)

What we have

with the new one

  • simple configuration for a new roll-out
  • simple development for a new product
  • 2246 Python statements (according to Pylint)
  • automatic category management (design decision)
  • no crashes
  • many tests (>80% coverage = easy regression analysis)

Products

  • PlayPlanet
  • Fingerbooks (self-pub and last mile)
  • Gamifive

Monitoring

Kibana shows data collected by:

  • log parsing for old ingestion
  • active push for new ingestion

Panels

For the moment a simple monitor for new ingestion.

In the future:

  • roll-out configuration
  • run scheduling configuration
  • EasyReplica integration via API

Errors

Fails and content defects are collected on a database (was: firsts were logged on file and seconds were emailed).

Benchmarks

Better overall performances:

  • shorter runs thanks to "delta" (was: Δ only on PlayPlanet)
  • different runs per day (was: about one run per day)

Content ingestion

By Giorgio Salluzzo

Content ingestion

  • 1,433