History and future of Tapan

Tapan v1

• Database and Reports layers
• Lot's of SQL queries, queries replica, consistency :(
• Very few typesafety
• Report results are cached, date range is a key cache
• No persistence, no zero downtime restarts
• Exposes CSVs

Tapan v2 (now)

• Database, View, Reports and API
• No SQL (almost), queries replica, consistency :(
• Database layer is mixed with Views :(
• Typesafety, let is crash
• Reports are built based on both Database and Views layers
• Only final Reports are cached, no intermediate caching, but filters and aggregations work (not everywhere)
• No persistence, no zero downtimes / upgrades
• No way to query from outside (ad-hoc, tableau)

Tapan v3

• Separate Views and Database
• Typesafety, skip and report
• Do ETL
• Cache and persist intermediate steps
• Open access to Views and Reports to outside

Tapan v3. Database

• Database, Views, Reports and API
• Database layer is flat and has simple types (CSV)
• No database connection, database is exported
  • export-20160223-0100/platform/tasks.csv
  • export-20160223-0100/analytics/ga.csv
• Export can be consistent
• Expected to be faster
  • needs to be measured

Tapan v3. Views

• Views are built by other Views and Database
• Typesafety through validation
  • don't crash, just log and skip
• Consistency! Because export is consistent
• Cacheable!
• Queriable/Exportable to/from Tableau, R, whatever
• Use typed data frames for aggregations and joins

Tapan v3. Reports

• Built by other Reports and Views
• Cacheable as before
• Fault-tolerant
  • skip and report inconsistent things
  • return last built report if it doesn't work today
• Consistent
  • cross-checks should always match
• Queriable/Exportable to/from Tableau, R, whatever

Tapan v3. API

• Built by other Reports and Views
• Cacheable as before
• Fault-tolerant
• • skip and report inconsistent things
  • return last built report if it doesn't work today
• Queriable/Exportable to/from Tableau, R, whatever

Tapan v3. Deployment

• Zero-downtime upgrade/downgrade/restart
• Build views and reports, and then switch
  • v7.0.0/views/companies
  • v7.0.0/reports/daily_revenue
  • v7.1.0/views/companies
  • v7.1.0/reports/daily_revenue
• Don't upgrade if we failed to build new reports

Tapan v3. Productivity

• Don't recompute things between restarts
• Reports and views are stored locally as CSV
  • grep
  • diff (endpoint-compare)
  • ...
• Computations can be executed remotely
  • Don't spend your CPU and battery power
• Further yolo-mode enchancements