Migrations & Quack:

Looking back and looking forward

• History
• Current State
• Changes
• Going Forward

• History
• Current State
• Changes
• Going Forward

• Proof of Concept
  • Prima Migration
  • Used Multiple EC2 Instance & Queue, written in NodeJs

• Original Idea (Dev Library)
  • Harpersbazaar
  • Developed to use Lambda and two NodeJs Modules (MediaOsOps, MediaOsAPI)
  • MediaOsOps to develop, deploy, and execute functions on cloud
  • MediaOsAPI to interact with RAMS.

• History
• Current State
• Changes
• Going Forward

• Original Idea (Dev Library)
  • Harpersbazaar (ground work laid by Quack)
  • Issues:
    • Developers not familiar with NodeJS, setting up Node projects, NPM, and JS concepts like Promises.
    • Devops issues with Lambda and permissions not being able to reach databases on Dev IO machines. (locally running dbs) and DB load issues.
    • Multiple Lambda functions being created with no oversight 

• History
• Current State
• Changes
• Going Forward

• Original Idea (Dev Library)
• New Idea (Quack API)
  • Create Common Schema
  • Developers Post data and DB information to API 
  • No need to set up quack or learn javascript
  • No need to develop, deploy, run functions for Lambda
  • Can now develop in PHP or language of choice and still use the speed benefit
  • provide distinct separation of data preparation and processing
  • Still had issue with Devops.

• History
• Current State
• Changes
• Going Forward

• Original Idea (Dev Library)
• New Idea (Quack API)
• New Idea++ (Quack API & Devops + Feather SDK)
  • Quack would now create the DB and s3 buckets necessary for a migration
  • Develop Feather to preprocess data that would post to Quack
  • Add persistent storage first in DynamoDB, then MysqlDB to be able to check on jobs
  • Add Recipe Support

• History
• Current State
• Changes
• Going Forward

• New Idea++ (Quack API & Devops + Feather SDK)
  • ​Used for ElleUK (elleNL,elleES same time)
  • ISSUES:
    • ​Bugs in Quack
      • ​only one developer
    • Changing the way we do crops 
      • ​creating dragonfly
    • Using CS databases
    • Data issues
    • devops taking long time to sync
    • image names
    • body embeds not using short codes
    • developing as we migrate
    • Cost $

• History
• Current State
• Changes
• Going Forward

Proposed Process with Changes

Issue

• Creating all the crops for images prevents Quack from being real time.

Solution

• Develop dragonfly (an image processing service built on lamba) that creates images on the fly for QA. Process jobs in real time so when a job is posted to quack it is inserted into the DB and not return a job ID.

• History
• Current State
• Changes
• Going Forward

Proposed Process with Changes

Issue

• We need to run post migrations scripts to get the migration in a good place.

Solution

• Once the data is in a RAMS db, we can run generic scripts that work on any mediaOS site independent of the data and migration. 
  • Short code fixers (same for all)

• History
• Current State
• Changes
• Going Forward

Proposed Process with Changes

Issue

• Once we do a migration, we have no way to feel confident about it.

Solution

• Create a site health tool which checks 
  • All content is migrated
  • All redirects work
  • 404s
  • SEO (before/after)

• History
• Current State
• Changes
• Going Forward

Proposed Process with Changes

Issue

• Syncing content between different environments

Solution

• Only create all images after everything else passes QA. Create those images directly on prod s3 to prevent time delay in sync.
• we did this for elleuk after changing crops.

• History
• Current State
• Changes
• Going Forward

Proposed Process with Changes

Issue

• ??????

Solution

• brainstorm

• History
• Current State
• Changes
• Going Forward

• ElleNL -  proceed as we have been
• ElleES -   try new process?

Current Migrations