Goals

Build on top of Englishtown
Highly performant
Scales horizontally (students from cross CN and US)
Scales proportionally (huge class)
Appealing interaction/modern look & feel
Responsive all the way down to mobile
Ship fast - prototype in 2m, production in 6m

Feature Highlights

topic-oriented platform
timeline-based class composer
social recording
self-enroll courses
mobile usabilites
real-time notifications

Facts

Limited dev resources (3 ppls, 8 month, 100k LOC)
Skinny team setup: QA(intern) Design(part time)
New DB (Cassandra)
Unstable Framework (TroopJS)
Aggressive PO (Bob)

Stats: front-end

Stats: server-side

Agenda

Efeckta integration
Cassandra storage
Rethink the UI workflow
API docs
Reactive rendering
Local development
Umbraco from Git
Pitfalls on web recording

Efecta integration

incorporate the self study layer in Camp, to make student experience exactly the same as in study plan, but...we're not the school team

camp

topic

N/A

week

task

step

activity

school

course

level

unit

lesson

step

activity

"course structure" revamped

camp

topic

N/A

week

task

step

activity

school

course

level

unit

lesson

step

activity

/school/e12/#school/2262345/385/550/2074/8861/2312/116439

school URL schema

camp

topic

N/A

week

task

step

activity

school

course

level

unit

lesson

step

activity

/camp/study#/study/YGyu5kxSRvyOgmxwaU9a2w/%3DRjO65LOSDulAgHXO1Kmig/c3%2BCxfd1TOaV4gCXdABIeA/IvltDfODS1usiBedyKPYRA

camp URL schema

/camp/study#/study/YGyu5kxSRvyOgmxwaU9a2w/%3DRjO65LOSDulAgHXO1Kmig/c3%2BCxfd1TOaV4gCXdABIeA/IvltDfODS1usiBedyKPYRA

camp url segment - base64 encoded UUID

/school/e12/#school/2262345/385/550/2074/8861/2312/116439

school url segment - uid

completely Routable UI

hash

#/study/YGyu.../

#/study/YGyu.../%3D.../

#/study/YGyu.../%3D.../n9W.../

#/study/...*.../eOrulp.../

UI state

which camp

which camp week

scroll into a camp task

open activity container of one task step

CAMP study page

resolve the UI puzzle

Efecta 13 is definitely a legacy application written in 4 years ago on troopjs 1.0 and is a bounch of jQuery plugin soup

challenges

Thanks to troop Efecta 13 is a requireJS application completely driven by hub events

clue

In case you don't know hub events

http://troopjs.com/docs/events.html#hub

A significant advantage of using hub is components can inject themselves in processing flow without modifying the host application, this allows for swapping out implementations of components without having to update all the dependent applications.

{lesson: '8861', step:'47198',activity:...}

{step:{...}, lesson: {}}

{id: "lesson!8861" …}

{id: "step!47198" …}

{id: "activity!142748" …}

Efecta I/O in troop events

load

load/result

(un)load/lesson

(un)load/step

(un)load/activity

step.1 - create a legacy sandbox

step.2 - bridge to sandbox events

step.3 - open a camp "step" in Efecta

step.4 - fulfill data requests from Efecta

step.last - sandbox the CSS

Cassandra storage

Cassandra is perfect for managing large amounts of data across multiple data centers and the cloud. Cassandra delivers continuous availability, linear scalability, and operational simplicity across many commodity servers with no single point of failure, along with a powerful data model designed for maximum flexibility and fast response times.

Why use Cassandra

Cross Data Center Camp Configuration
Cross Data Center Students
Cross Data Center Social Activity
Cross Data Center Teacher feedback
Cross Data Center Progress report
Approve and Support from T-team

Weakness of Cassandra & CQL

No Joins
No arbitrary Where statement
No arbitrary Order By statement
No Group By
A few operators are supported
Secondary Index

Data Model

Forget about RDBMS
Query First
No Secondary Index
Customized Secondary Index

Try Cassandra

CREATE TABLE Class_Tasks(
    Class_Id uuid, 
    Class_Section_Id uuid, 
    Class_Task_Id uuid,
    Task_Id uuid,
    Task_Order int, 
    Task_Type text, 
    Task_Sub_Type text,
    Name text,
    Description text,
    Start_Time text, 
    End_Time text, 
    Insert_Date timestamp,
    Update_Date timestamp,
    PRIMARY KEY(
        Class_Id,
        Class_Section_Id,
        Class_Task_Id
    )
);

CREATE TABLE Class_By_Task(
    Class_Task_Id uuid,
    Class_Section_Id uuid,
    Class_Id uuid,
    Insert_Date timestamp,
    Update_Date timestamp,
    PRIMARY KEY (
        Class_Task_Id    
    )      
);

Introduce Solr

The unique combination of Cassandra and DSE Search with Solr features robust full-text search, hit highlighting, and rich document (PDF, Microsoft Word, etc) handling.

Data Model with Solr

create table group_lesson_feedback(
    class_id uuid, 
    class_task_id uuid,
    comment text,
    student_status text, 
    teacher_member_id int,
    closed boolean,
    notified boolean,
    prop_ map<text, text>,
    insert_date timestamp,
    update_date timestamp,
    PRIMARY KEY(
        class_id,
        class_task_id            
    )    
);

e.g. the column value of student_status: 
[
    {
        "memberid": 1234,
        "absent": "true",
        "techissue": "false"
    },
    {
        "memberid": 56789,
        "absent": "false",
        "techissue": "false"
    }
]

e.g. the column value of prop_ map:
{
    'prop_name1': 'value1',
    'prop_name2': 'value2'
}

Use Solr with CQL

Generate Solr Config and Schema resources

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<schema name="autoSolrSchema" version="1.5">
<types>
<fieldType class="org.apache.solr.schema.TextField" name="TextField">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType class="org.apache.solr.schema.TrieIntField" name="TrieIntField"/>
<fieldType class="org.apache.solr.schema.UUIDField" name="UUIDField"/>
<fieldType class="org.apache.solr.schema.BoolField" name="BoolField"/>
<fieldType class="org.apache.solr.schema.TrieDateField" name="TrieDateField"/>
</types>
<fields>
<dynamicField indexed="true" multiValued="false" name="prop_*" stored="true" type="TextField"/>
<field indexed="true" multiValued="false" name="class_id" stored="true" type="UUIDField"/>
<field indexed="true" multiValued="false" name="class_task_id" stored="true" type="UUIDField"/>
<field indexed="true" multiValued="false" name="closed" stored="true" type="BoolField"/>
<field indexed="true" multiValued="false" name="notified" stored="true" type="BoolField"/>
<field indexed="true" multiValued="false" name="teacher_member_id" stored="true" type="TrieIntField"/>
</fields>
<uniqueKey>class_id</uniqueKey>
</schema>

Use Solr with CQL

--e.g. query by any column configured in solr schema   
select comment, student_status, closed, prop_ 
from group_lesson_feedback 
where solr_query = 'teacher_member_id:12345678';

--e.g. query by dynamic column "prop_"
select comment, student_status, closed, prop_ 
from group_lesson_feedback 
where solr_query =  'prop_name2:value2';

Key_Space configuration

Replication factor
Read Consistency Level (local_Quorm)
Write Consistency Level (local_One)

--e.g. Camp key_space on live
CREATE KEYSPACE IF NOT EXISTS Camp 
WITH REPLICATION = {
    'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy',
    'US1': 3,
    'CN1': 3
};

Tips

Use prepared statement
Escape single quote for concatenate statement
Don't use "select * from" statement
Avoid Alter column type on existing table
Ask for help from Teddy, Simon and Adrain

References

http://teddymaef.github.io/learncassandra/en/

http://docs.datastax.com/en/cassandra/2.1

API docs

There must be a formalized way of describing application APIs, before and after they are implemented, this reduce significantly the communication cost between front-back-end and reinforces API robustness and stablization.

we start from, documenting the model

sooner we realize for what we're doing, there's an existing standard...

Swagger!

the Swagger Schema allows the definition of data type expectations, this schema is based on the JSON Schema Specification Draft 4, and uses a predefined subset of it.

document a command

document a query

Applyied a few UI magics

You can even
try out each API live

kudos to @Teddy

Rethink the UI workflow

Rethink of the best practices to deliverUIs influenced by the movie producing industry.

#1

I'm waiting for the whole page to reload, just to test that little change.

#2

The server API required for testing this feature is not yet ready.

#3

My team have no idea of what the final UI looks until 3 weeks later.

If you have the same pain...

Evil

no separation of concerns

The “layers" of UX

Redefine "delivery"

KEYFRAMES

focus

informational structure
layout, styles
markup
typography, color, graphic...

design specs in "web format"

KEYFRAMES

ITERACTIVES

focus

user stories and interaction details
interaction flow boundaries
transition/animation

user interaction flow and state transitions

INTEGRATIONS

focus

server and environmental integrations

API data validity
other sever environment
authentication
architecture

Agileness gains

Reactive rendering

Trying to understand how to apply individual DOM manipulation smartly enough
to avoid re-rendering for each scenario,
is an super insufficient as well as error prune procedure

all front-end book will tell you:

dom mutation is the most expensive part of the browser, performance wise application should avoid unnecessary DOM updates to but only to carry the least changes required.

This is what usually called "DOM sculpting", but it's really hard to make it right all the time!

here is where we usually start from...
a single state UI

DOM patch#1 created to incoperate data#1

new patch#2 presenting data#2,

is based on patch#1

what if data#1 is removed,
or even comes after data#2

The Problem

The application states are implicitly DOM dependent
Every DOM mutations relies on the previous state thus cannot be ported
No "single source of truth"

Consequences

write spaghetti code that leads to error prune UI state
view logic code has to be changed once a while and cannot sustain over time

Building UI in such a way is hard because there's no much state

data is updated everywhere
designs changes over time
many ways of user input

In the 90s it was actually easier...

Just to refresh the page!

In the 90s it was actually easier...

That's originate the idea of
"reactive rendering"

React Rendering Algorithms

let UI states sourced from a single object
First-class "render" method that transform the state object into a new virtual DOM Tree
...diffs it with the previous rendered virtual DOM
computes the minimal set of DOM mutations required and put them in queue
...batch apply the changes to only the affected nodes

components data flow

Troop widget composition

{{!--camp widget --}}

{{#each camp.sections}}
  <div data-weave="widget/section(section)" data-section={{JSON.stringify(this)}}></div>
{{/each}}

{{!--section widget --}}
                  
{{#each section.tasks}}
  <div data-weave="widget/task(task)" data-task={{JSON.stringify(this)}}></div>
{{/each}}

{{!--task widget --}}

{{#each task.steps}}
  <div data-weave="widget/step(step)" data-step={{JSON.stringify(this)}}></div>
{{/each}}

Troop widget data flow

recursively render the camp data

render each section

render each task

render each step

blocked data flow

when "step" data changes on camp

React component composition

/* camp.jsx */
return camp.sections.map(function(section) {
  return <Section section={section} />;
});

/* section.jsx */
return section.tasks.map(function(task) {
  return <Task task={task} />;
});

/* task.jsx */
return task.steps.map(function(step) {
  return <Step step={step} />;
});

React component data flow

when "step" data changes on camp

update each section

update each task

step is updated!

Re-render the whole app on each update, let the framework to remember which widget is to create or to update

The React Principle

Develop Locally

For the most productive frontiers, local development shall requires no server environment setup at all, yet remains as close as possible to all production environments, aka. load from HTML

Goals

Bootstrap page from a HTML file, not a ASPX file
RequireJS load modules, individually, locally
Import CSS files individually, locally
Load CMS contents locally
Non-local resources including API endpoints, medias and blurbs

Non-local resources

1. local DNS map

campuat.englishtown.com
qa.englishtown.com
www.englishtown.com

campuat.local
qa.local
www.local

#/usr/local/etc/dnsmasq.conf

local=/localhost/
address=/dev/127.0.0.1

example with dnsmasq

Non-local resources

2. virtual host + reverse proxy

static files (.js/.css) -> resolve as local file
.html files -> proxied to local server
API endpoints -> proxied to "the" server
handlers -> proxied to "the" server

Non-local resources

2. virtual host + reverse proxy

# CAMP QA Reverse Proxy
<VirtualHost *:80>
    DocumentRoot /Users/garry/Stash/camp-ui
    ServerName qa.dev
    ProxyPassMatch ^/(.*\.html)$ http://localhost:3000/$1
    ProxyPass /camp/login http://qa.englishtown.com/camp/login
    ProxyPass /camp/enroll http://qa.englishtown.com/camp/enroll
    ProxyPass /camp !
    ProxyPass / http://qa.englishtown.com/
    ProxyPassReverse / http://qa.englishtown.com/
    ProxyPassReverseCookieDomain qa.englishtown.com qa.dev
</VirtualHost>

([^.]*).dev -> $1.englishtown.com

HTML file as handler

Use server variables to assemble the page content, e.g. getTrans::12345, getContent::cms_key
Embed build instructions into the HTML file, e.g. concatenation

HTML file as handler

<!doctype html>
<html lang="en">
<head>
  <title>Englishtown | CAMP | Study </title>

  <!-- build:css css/camp.css -->
  <link rel="stylesheet" href="icons/style.css"/>
  <link rel="stylesheet" href="css/camp.css"/>
  <!-- endbuild -->

  getContent::Camp_common-head

  <!-- build:js camp.js -->
  <script src="bower_components/requirejs/require.js"></script>
  <script src="common-camp.js"></script>
  <!-- endbuild -->

  getContent::Camp_common-tail
</head>
<body class="ets-spinning-global" data-weave="camp-ui/widget/layouts/main">
<div class="spinner"></div>
</body>
</html>

HTML file as handler

Question: how to load such HTML file with server variables resolved?

HTML file as handler

Introduce local handlers

HTML file as handler

Introduce local handlers

Load JS modules

DEV

baseUrl:'bower_components'
all modules are loaded individually

Load JS modules

>UAT

baseUrl:{CCL:configuration.servers.cache}/_shared/camp-ui/{CCL:camp.ui.version}/bower_components (aka. CDN)
mandatory modules are loaded as one single bundled module
lazy modules are loaded individually on demand

Transcompiler & watch

create a "src" directory as source level code
watch over .js/.hbs/.less and compile with JSX/Handlebar/LESS
copy those newer compiled to the "root" directory
requirejs load modules assuming the same package structure

Umbraco from Git

CMS is a good point, but I always miss the developing experience from local files and the safety have everything version controlled in Git

Why Umbraco
breaks the dev flow

Only one single version of the content, when you publish, you push to QA
Permission control differs from Stash
Cannot develop locally
Terrible editing experiences (miss BATCH replace)

But that doesn't mean...

We have escaped from Umbraco

actually we used that exclusively:

Umbraco + Git?

Combine the best from both worlds

track CMS nodes as local files
put them as GIT files for version control
pull from Umbraco when new node has been added
making GIT commits for changes
publish to Umbraco with a GIT push hook

How do you to Umbraco?

A Umbraco CLI

npm install -g umbra

18:50 $ umbra -h   

  Usage: umbra [options] [command]


  Commands:

    push [files...]  publish specific file(s) or the entire "cms" directory to Umbraco
    pull             fetch newly created Umbraco contents back to the "cms" directory

  CLI for publish Umbraco CMS contents from local files

  Options:

    -h, --help     output usage information
    -V, --version  output the version number

09:24 $ ll cms/*landing*.html
-rw-r--r--+ 1 garry  staff   9119 May  7 17:31 cms/Camp_landing-cs.html
-rw-r--r--+ 1 garry  staff   9781 May  7 17:31 cms/Camp_landing-en.html
-rw-r--r--+ 1 garry  staff    773 May  7 17:31 cms/Camp_landing-navbar-cs.html
-rw-r--r--+ 1 garry  staff    767 May  7 17:31 cms/Camp_landing-navbar-en.html
-rw-r--r--+ 1 garry  staff    792 Apr 26 17:21 cms/Camp_landing-navbar-topic-ji-cs.html
-rw-r--r--+ 1 garry  staff    794 Apr 26 17:21 cms/Camp_landing-navbar-topic-ji-en.html
-rw-r--r--+ 1 garry  staff  15103 May  7 17:31 cms/Camp_landing-topic-ji-cs.html
-rw-r--r--+ 1 garry  staff  16307 May  7 17:31 cms/Camp_landing-topic-ji-en.html

09:24 $ umbra push cms/*landing*.html
[ok] camp_landing-en (31744) has been published as "getContent::Camp_landing-navbar-en
<d..."
[ok] camp_landing-navbar-topic-ji-en (33202) has been published as "<nav class="navbar navbar-default nav..."
[ok] camp_landing-topic-ji-cs (31750) has been published as "getContent::Camp_landing-navbar-topic..."
[ok] camp_landing-navbar-cs (33199) has been published as "<nav class="navbar navbar-default nav..."
[ok] camp_landing-cs (31748) has been published as "getContent::Camp_landing-navbar-cs
<d..."
[ok] camp_landing-navbar-topic-ji-cs (33201) has been published as "<nav class="navbar navbar-default nav..."
[ok] camp_landing-navbar-en (33197) has been published as "<nav class="navbar navbar-default nav..."
[ok] camp_landing-topic-ji-en (31751) has been published as "getContent::Camp_landing-navbar-topic..."

Hook into Git

Client hooks

pre-rebase
post-rewrite
post-merge
pre-push

Server hooks

pre-recieve

post-recieve

update

Hook into Git

pre-push (local)

requires developer setup
impossible with changes via pull-request
less fault proof

update (server)

no developer setup

requires server-side (Stash) support

less error prune, more accurate

Hook into Git

git hook? | git diff-tree $ref head | grep "cms" | umbraco push

Umbraco publishing in a pipe

Github hook is better

https://developer.github.com/webhooks/#events

The reworked CMS releasing

create CMS nodes in Umbraco & umbra pull
Develop locally in feature/branch
Review changes locally
commit and pull request to team/branch
git hooks & umbra pushed
Your CMS changes is now published on QA!
Go through the rest of the BIC

Web Recording

Base on Flash

Tech check (Microphone, Chrome)
MP3 Encoder (Open source)
Audio File Upload (Cassandra provide by Adrian Gonzalez)
Trouble Shoot

Upload File to Cassandra

API
http://{hostname}:{port}/opt-media/upload
Authentication by CMus and et_sid
Test Page
http://qa.englishtown.com/opt-media/form
Doc:
https://stash.englishtown.com/...

Pitfalls

A lot of user record noise like

Client Environment

OS
Flash version
Browser - Chrome 31.0.1650.63

Client Environment

OS - XP
Flash version - 11.6
Browser - Chrome 31.0.1650.63

https://bugbase.adobe.com/index.cfm?event=bug&id=3501218

Client Environment

OS - XP
Flash version - 11.6
Browser - Chrome 31.0.1650.63

https://bugbase.adobe.com/index.cfm?event=bug&id=3501218

more "360 safe browser" than we expected

            
 <meta name="renderer" content="webkit">

Cassandra Service

Live Error: Fails to upload recording: 500

Cassandra Service

Live Error: Fails to upload recording: 500

Automation test

ALL（DEV，QA，Live）

Live

How we collect user error

Raygun

Raygun

Default: User Agent, Host, Referer
Version, Tag
Tracking User
Customer Data
Filtering

CAMP Tech Crunch

Goals

Feature Highlights

Facts

Stats: front-end

Stats: front-end

Stats: server-side

Stats: server-side

Agenda

Efecta integration

camp

school

"course structure" revamped

camp

school

school URL schema

camp

school

camp URL schema

camp url segment - base64 encoded UUID

school url segment - uid

completely Routable UI

hash

UI state

CAMP study page

resolve the UI puzzle

challenges

clue

In case you don't know hub events

{lesson: '8861', step:'47198',activity:...}

{step:{...}, lesson: {}}

{id: "lesson!8861" …}

{id: "step!47198" …}

Efecta I/O in troop events

load

(un)load/lesson

(un)load/step

(un)load/activity

step.1 - create a legacy sandbox

step.2 - bridge to sandbox events

step.3 - open a camp "step" in Efecta

step.4 - fulfill data requests from Efecta

step.last - sandbox the CSS

Cassandra storage

Why use Cassandra

Weakness of Cassandra & CQL

Data Model

Try Cassandra

Introduce Solr

Data Model with Solr

Use Solr with CQL

Use Solr with CQL

Key_Space configuration

Tips

References

API docs

we start from, documenting the model

sooner we realize for what we're doing, there's an existing standard...

Swagger!

document a command

document a query

Applyied a few UI magics

You can even try out each API live

kudos to @Teddy

Rethink the UI workflow

#1

#2

#3

If you have the same pain...

Evil

no separation of concerns

The “layers" of UX

Redefine "delivery"

KEYFRAMES

focus

design specs in "web format"

KEYFRAMES

ITERACTIVES

focus

user interaction flow and state transitions

You can even
try out each API live

here is where we usually start from...
a single state UI

what if data#1 is removed,
or even comes after data#2

That's originate the idea of
"reactive rendering"

Why Umbraco
breaks the dev flow