NodeJS → Go
Alan Braithwaite
@Caust1c
github.com/abraithwaite
Segment Inc
Why?
Performance
Security
Reliability
Flexibility
Productivity
Why?
The architecture
Verification of New API
Verification of New API
Release Proceedure
Roll out new API Forwarder pointing to new API
Scale up new API as more traffic is moved over
???
🤑🤑🤑
Incident Free Hermit
Challenges Encountered
Timestamps
Challenges Encountered
var formats = []string{
// 99% case. RFC3339 handles deserializing ISO8601 as well.
time.RFC3339,
// everything else (we've seen so far) that JS accepts
time.RFC1123Z,
time.RFC822Z,
"2006-01-02T15:04:05-0700",
// (seemingly rare) JS case.
"2006-01-02 15:04:05 -0700",
// --- Keep formats with tz offset above this line ---
time.RFC1123,
time.RFC822,
"2006-01-02T15:04:05.00",
"2006-01-02T15:04:05",
"2006-01-02T15:04:05.000",
"Jan _2, 2006 3:04:05 PM",
// counterpart to above
"2006-01-02 15:04:05",
}Timestamps
Challenges Encountered
Timestamps
x = new Date("01 Jan 18 00:01 PST")
Mon Jan 01 2018 00:01:00 GMT-0700 (Pacific Standard Time)const int8_t DateParser::KeywordTable::
array[][DateParser::KeywordTable::kEntrySize] = {
// some omitted for brevity...
{'u', 't', '\0', DateParser::TIME_ZONE_NAME, 0},
{'u', 't', 'c', DateParser::TIME_ZONE_NAME, 0},
{'z', '\0', '\0', DateParser::TIME_ZONE_NAME, 0},
{'g', 'm', 't', DateParser::TIME_ZONE_NAME, 0},
{'c', 'd', 't', DateParser::TIME_ZONE_NAME, -5},
{'c', 's', 't', DateParser::TIME_ZONE_NAME, -6},
{'e', 'd', 't', DateParser::TIME_ZONE_NAME, -4},
{'e', 's', 't', DateParser::TIME_ZONE_NAME, -5},
{'m', 'd', 't', DateParser::TIME_ZONE_NAME, -6},
{'m', 's', 't', DateParser::TIME_ZONE_NAME, -7},
{'p', 'd', 't', DateParser::TIME_ZONE_NAME, -7},
{'p', 's', 't', DateParser::TIME_ZONE_NAME, -8},
// ...
}Other Challenges Encountered
- Javascript is loosely typed
- Falsey
- Null
- Undefined
- Unknown fields must be preserved
-
github.com/tidwall/gjson
-
New Release Proceedure
- Release tiered by customer plan level
- Manual redshift query
- Embedded CSV into API Forwarder to query
- Released over 4 months
SEVs Related to Rewrite
incident-free-hermit (corrupted timestamps for all customers across 8 hours)
incident-needed-emu (imbalanced autoscaling rules between api-forwarder and api)
sev-broad-moose (canary configs in treb caused reset in desired count for service, causing delays)
sev-slow-cockroach (projectId:null handled improperly caused dropped data for 8h on one source)
sev-blop-shibe (marketo integration bad signatures caused <5min dropped marketo data)
sev-lively-mouse (secret in event body for authentication, affected 1 or two customers)
sev-low-hedgehog (performance for overflow. probably would have knocked old API over)
sev-drab-quokka (pentester added library integration, bad behavior from old API)
sev-clumsy-possum (validation errors caused dropping of page/screen calls)
Summary
Rewrites are challenging
Avoid them if possible
Setup a rock solid validation system against real customer data
Go made it easier ;-)
Thank you
Alan Braithwiate
Segment
@Caust1c
https://github.com/abraithwaite
https://segment.com
Rewrite from NodeJS to Go
By Alan Braithwaite
