Migrations & Quack:
Looking back and looking forward
- History
- Current State
- Changes
- Going Forward
- History
- Current State
- Changes
- Going Forward
- Proof of Concept
- Prima Migration
- Used Multiple EC2 Instance & Queue, written in NodeJs
- Original Idea (Dev Library)
- Harpersbazaar
- Developed to use Lambda and two NodeJs Modules (MediaOsOps, MediaOsAPI)
- MediaOsOps to develop, deploy, and execute functions on cloud
- MediaOsAPI to interact with RAMS.
- History
- Current State
- Changes
- Going Forward
- Original Idea (Dev Library)
- Harpersbazaar (ground work laid by Quack)
- Issues:
- Developers not familiar with NodeJS, setting up Node projects, NPM, and JS concepts like Promises.
- Devops issues with Lambda and permissions not being able to reach databases on Dev IO machines. (locally running dbs) and DB load issues.
- Multiple Lambda functions being created with no oversight
- History
- Current State
- Changes
- Going Forward
Original Idea (Dev Library)- New Idea (Quack API)
- Create Common Schema
- Developers Post data and DB information to API
- No need to set up quack or learn javascript
- No need to develop, deploy, run functions for Lambda
- Can now develop in PHP or language of choice and still use the speed benefit
- provide distinct separation of data preparation and processing
- Still had issue with Devops.
- History
- Current State
- Changes
- Going Forward
Original Idea (Dev Library)New Idea (Quack API)- New Idea++ (Quack API & Devops + Feather SDK)
- Quack would now create the DB and s3 buckets necessary for a migration
- Develop Feather to preprocess data that would post to Quack
- Add persistent storage first in DynamoDB, then MysqlDB to be able to check on jobs
- Add Recipe Support
- History
- Current State
- Changes
- Going Forward
-
New Idea++ (Quack API & Devops + Feather SDK)
- Used for ElleUK (elleNL,elleES same time)
-
ISSUES:
-
Bugs in Quack
- only one developer
-
Changing the way we do crops
- creating dragonfly
- Using CS databases
- Data issues
- devops taking long time to sync
- image names
- body embeds not using short codes
- developing as we migrate
- Cost $
-
Bugs in Quack
- History
- Current State
- Changes
- Going Forward
Proposed Process with Changes
Issue
- Creating all the crops for images prevents Quack from being real time.
Solution
- Develop dragonfly (an image processing service built on lamba) that creates images on the fly for QA. Process jobs in real time so when a job is posted to quack it is inserted into the DB and not return a job ID.
- History
- Current State
- Changes
- Going Forward
Proposed Process with Changes
Issue
- We need to run post migrations scripts to get the migration in a good place.
Solution
- Once the data is in a RAMS db, we can run generic scripts that work on any mediaOS site independent of the data and migration.
- Short code fixers (same for all)
- History
- Current State
- Changes
- Going Forward
Proposed Process with Changes
Issue
- Once we do a migration, we have no way to feel confident about it.
Solution
- Create a site health tool which checks
- All content is migrated
- All redirects work
- 404s
- SEO (before/after)
- History
- Current State
- Changes
- Going Forward
Proposed Process with Changes
Issue
- Syncing content between different environments
Solution
- Only create all images after everything else passes QA. Create those images directly on prod s3 to prevent time delay in sync.
- we did this for elleuk after changing crops.
- History
- Current State
- Changes
- Going Forward
Proposed Process with Changes
Issue
- ??????
Solution
- brainstorm
- History
- Current State
- Changes
- Going Forward
- ElleNL - proceed as we have been
- ElleES - try new process?
Current Migrations
Quack - One year look back
By Daniel Nakhla
Quack - One year look back
- 601