I care about client data!
Mistakes were made.
But hey...
Continuous improvement is possible.
Major scaling of the system is possible.
SENSEI downtime is not a factor**.
**Refers to downtime caused by import. Some other things can still fubar us.
Monitoring
File Metadata
{
"entity": "Type of Data (Measurement)",
"location": "Generated Unique Hash location in S3",
"mimeType": "text/csv",
"originalLocation": "Where file ultimately came from",
"previousLocation": "Where file immediately came from",
"owner": "Customer name",
"state": "State of file (received, transformed, delivered, failed)"
}Improving Metadata
Reception Time
Amazon S3 Versions
Improve Ability to Query Import Data
Fewer Transformation Artifacts (files)
Improving Logging
Reduce Noise
Strategic Logging
Tell a Story
Leave Breadcrumbs
Notifications
- Emails
- Slack
- SMS (Text messages)
- Feedback in our web apps
Learn our Log Tools
Self-Service Imports
Feedback and
Recovery Loops
Apps for configuring, analyzing and maintaining client data
Import Architecture Today
Import System
By Will Vaughn
Import System
An explanation of the current system by which Cascade Energy ingests client data.
