Our journey and lessons learned
John McKim
VP of Product & Technology
A Cloud Guru
@johncmckim
https://acloud.guru
Cloud Training for Organisations and Engineers
Small team and Growing Fast
Oh shit. We need Metrics.
No experience. No problem
DynamoDB, Redshift and Segment
Not fun.
We need help
Challenges
Starting with Firebase
Firebase Data Pipeline
Adopting Fivetran
Fivetran
Data Replication Pipeline
Using DBT for Modelling
Build Models on your Data
{{ config(materialized='table',
sort = 'full_date',
dist = 'full_date') }}
select
created_at::date as full_date,
zendesk_agent_id,
zendesk_group_id,
count(case score when 'offered' then 1 else null end) as surveys_sent,
count(case score when 'offered' then null else 1 end) as responses,
count(case score when 'good' then 1 else null end) as good_ratings,
count(case score when 'bad' then 1 else null end) as bad_ratings
from {{ref('dim_zendesk_satisfaction_rating')}}
group by 1,2,3
Done ?
Maybe not.
Incident Timeline
Incident metrics
Severity: Production Impacted
MTTD: > 5 months
MTTR: > 10 weeks
Build Performance
Ever slowing queries
Sad.
Break through
Orange = Query Planning
No answers. Only suspicions.
New Replication Service & Data Lake
Changing role of Data Eng
https://blog.fishtownanalytics.com/does-my-startup-data-team-need-a-data-engineer-b6f4d68d7da9
Outsourcing has challenges
Questions?
johncmckim.me
twitter.com/@johncmckim
medium.com/@johncmckim