NodeJS → Go

Alan Braithwaite

@Caust1c

github.com/abraithwaite

Segment Inc

Why?

Performance

Security

Reliability

Flexibility

Productivity

Why?

The architecture

Verification of New API

Release Proceedure

Roll out new API Forwarder pointing to new API

Scale up new API as more traffic is moved over

???

🤑🤑🤑

Incident Free Hermit

Challenges Encountered

Timestamps

Challenges Encountered

var formats = []string{
	// 99% case.  RFC3339 handles deserializing ISO8601 as well.
	time.RFC3339,
	// everything else (we've seen so far) that JS accepts
	time.RFC1123Z,
	time.RFC822Z,
	"2006-01-02T15:04:05-0700",
	// (seemingly rare) JS case.
	"2006-01-02 15:04:05 -0700",

	// --- Keep formats with tz offset above this line ---
	time.RFC1123,
	time.RFC822,
	"2006-01-02T15:04:05.00",
	"2006-01-02T15:04:05",
	"2006-01-02T15:04:05.000",
	"Jan _2, 2006 3:04:05 PM",
	// counterpart to above
	"2006-01-02 15:04:05",
}

Timestamps

Challenges Encountered

Timestamps

x = new Date("01 Jan 18 00:01 PST")
Mon Jan 01 2018 00:01:00 GMT-0700 (Pacific Standard Time)

const int8_t DateParser::KeywordTable::
  array[][DateParser::KeywordTable::kEntrySize] = {
  // some omitted for brevity...
    {'u', 't', '\0', DateParser::TIME_ZONE_NAME, 0},
    {'u', 't', 'c', DateParser::TIME_ZONE_NAME, 0},
    {'z', '\0', '\0', DateParser::TIME_ZONE_NAME, 0},
    {'g', 'm', 't', DateParser::TIME_ZONE_NAME, 0},
    {'c', 'd', 't', DateParser::TIME_ZONE_NAME, -5},
    {'c', 's', 't', DateParser::TIME_ZONE_NAME, -6},
    {'e', 'd', 't', DateParser::TIME_ZONE_NAME, -4},
    {'e', 's', 't', DateParser::TIME_ZONE_NAME, -5},
    {'m', 'd', 't', DateParser::TIME_ZONE_NAME, -6},
    {'m', 's', 't', DateParser::TIME_ZONE_NAME, -7},
    {'p', 'd', 't', DateParser::TIME_ZONE_NAME, -7},
    {'p', 's', 't', DateParser::TIME_ZONE_NAME, -8},
  // ... 
}

Other Challenges Encountered

Javascript is loosely typed
- Falsey
- Null
- Undefined
Unknown fields must be preserved
- ```
github.com/tidwall/gjson
```

New Release Proceedure

Release tiered by customer plan level
Manual redshift query
Embedded CSV into API Forwarder to query
Released over 4 months

SEVs Related to Rewrite

incident-free-hermit (corrupted timestamps for all customers across 8 hours)

incident-needed-emu (imbalanced autoscaling rules between api-forwarder and api)

sev-broad-moose (canary configs in treb caused reset in desired count for service, causing delays)

sev-slow-cockroach (projectId:null handled improperly caused dropped data for 8h on one source)

sev-blop-shibe (marketo integration bad signatures caused <5min dropped marketo data)

sev-lively-mouse (secret in event body for authentication, affected 1 or two customers)

sev-low-hedgehog (performance for overflow. probably would have knocked old API over)

sev-drab-quokka (pentester added library integration, bad behavior from old API)

sev-clumsy-possum (validation errors caused dropping of page/screen calls)

Summary

Rewrites are challenging

Avoid them if possible

Setup a rock solid validation system against real customer data

Go made it easier ;-)

Thank you

Alan Braithwiate
Segment
@Caust1c

https://github.com/abraithwaite

https://segment.com