Serum

a standard for serializable errors

and!  go-serum-analyzer:
a static analysis tool for serum in golang

I'm warpfork.

github: github.com/warpfork

twitter: twitter.com/warpfork

slides: slides.com/warpfork

Serum!

  • `code` -- clarity for machines
  • `message` -- legibility for humans
  • `details` -- everything else!
{"error":{
  "code": "error-codes-matter",
  "message": "messages matter too",
  "details": {"extensible": "yet simple"}
}}

You've got errors.

You need to log them.

 

You need to show them to users.

 

You need to know your programs handle them gracefully.

Serum gives them structure.

JSON.

Incrementally legible.

Mechanically manageable.

Specifies the minimum bits you need; expandable.

love Codes are all you need 🎶🎵🎶

- Errors need a code.

- It should be human-readable.

- It should search-engine'able.

- It should be machine-matchable.

And that's kinda all you need.

Everything else is icing on the cake.

switch err.Code() {
  case "warpforge-error-network":
    return retry()
  default:
    return value
}

// Good documentation should
// describe error codes!
//
// Errors:
//
//    - examples-error-foobar -- comes from DoSomething.
//    - examples-error-discombobulated -- also comes from DoSomething.
func DoMoreThings() error {
	return DoSomething(nil)
}

love Codes are all you need 🎶🎵🎶

- Errors need a code.

 

Treating this like an enum is ridiculously powerful, while still being compellingly simple.

Static analysis tools can help you a lot with just this much structure.  (More on this later.)

switch err.Code() {
  case "warpforge-error-network":
    return retry()
  default:
    return value
}

// Good documentation should
// describe error codes!
//
// Errors:
//
//    - examples-error-foobar -- comes from DoSomething.
//    - examples-error-discombobulated -- also comes from DoSomething.
func DoMoreThings() error {
	return DoSomething(nil)
}

Descriptions are good

- Freetext messages are nice:

- Easy to write

- Easy to read

- ... but not great for machine parsing.  (So use the code for that.)

Serum says: use a string.
(Simple.)

Make it for humans.

return SerumError{
  Code: "myapp-error-foobar",
  Msg:  "error handling workspace"
    +"at %q: %s" % wsPath, cause
}

Nobody can resist details

... so let's just have a map for that.

A key-value map is enough.

It's probably a good idea to repeat the details in the prose, too -- if we want end-users to see those details.

return SerumError{
  Code: "myapp-error-foobar",
  Msg:  "error handling workspace"
    +"at %q: %s" % wsPath, cause,
  Details: {
    "workspacePath": wsPath,
    "otherdetails": whatever,
  },
  Cause: wrapErr(cause),
}

Nobody can resist details

Stack traces are also details.

I know that seems rough,
but stack traces aren't really standardized.  They're freetext.

So treat them like freetext.

What if I want to start using Serum...?

... Great!  Do it!

What if there's no libraries in my language yet..?

... Make one!  :D

   It's just JSON!

What if I want to extend Serum...?

... Great!  Do it!
   (Adding fields to JSON is generally not a compatibility problem...)

What inspired Serum?

  • Error codes in SQL DBs -- they always have a short number, and it's reliably easy to plug into search engines.
     
  • GraphQL APIs -- unions for errors work well.
    (Serum is "code" where in graphQL was "__type__"!)

Positive Experiences:

Negative Experiences:

  • Practical experience of writing an API with too much strong typing of errors...
    It can make APIs become fragile, brittle!

There's a spec!

Find it at

https://github.com/
  serum-errors/serum-spec

go-serum-analyzer

the great thing about error codes is...

  • Enumerable for a human: can reason about it
  • Enumerable for search engine queries
  • Enumerable in that a tool can check for
    exhaustive handling    , and it's about as much complexity as checking off a list

They're enumerable.

and the crap thing about golang is...

it's seriously so bad.

it's the #1 reason I start having trouble wrangling large codebases in golang and producing reliable software.

the error handling.

so let's fix it...

with a static analysis tool!

Golang code using Serum codes...

Looks roughly like this.

It's still very normal golang code.

...Notice the docs!

Golang code using Serum codes...

It's a pretty simple annotation style:

 

Errors:

dash
{code}

dashdash

{describe}

Codes returned by this function

Codes returned by this function

How does this analysis work?

  1. Simple taint model.
  2. Any function that returns `error`: examine it.
  3. Do values returned also have a `func Code() string`?
  4. Look at what strings that `Code()` func can return...
  5. Add that to the set of things the function can be expected to return.


Now diff that with what the docs strings say.

Report any divergence.

Works in a wide
variety of situations...

Multiple return sites separated by any kind of branching work, as you'd expect:

Works in a wide
variety of situations...

Assignments are traced:

Works in a wide
variety of situations...

In finding an error type's code, branching is also understood:

Works in a wide
variety of situations...

Error codes can even be composed:

(as long as it's constant composition)

Works in a wide
variety of situations...

When things get too wild, overrides are an option:

Works in a wide
variety of situations...

Partial overrides?  Yes.

Very Useful when using switch statements to reduce the taint spread!

Works in a wide
variety of situations...

Can you restrict the errors returned by interfaces, and will this result in their implementers being checked?

Yus.

Designed to be easy to adopt

  • Functions that aren't annotated aren't checked.
    • ... unless you use the `-strict` flag; then un-annotated public functions are an error!
  • Scanning quietly continues through unannotated functions!
    • ... You don't have to annotate *every* function!
       
  • Can be applied to one package at a time!
     
  • Will recursively scan other packages...
    • ... but trusts each function that's annotated to be telling the truth, so if something *is* wrong, you hear about it *once*.
      (Report noise levels are manageable!)

Impact

When adopting go-serum-analyzer in even a medium codebase...

We found the analyzer was replacing the equivalent of thousands of manual checks.

 

Development quality was increased.

 

Impact

When adopting go-serum-analyzer in even a medium codebase...

By the numbers, we found:

  • 983 errorcode-sites verified by automation (meaning: error codes per function, in those functions that are explicitly annotated).
  • 59 functions were manually annotated to reach this result.
  • 174 functions in total were covered implicitly, since analysis covers all functions called by an annotated function.
  • (That means annotating 33% of functions was enough to attain the full effect!)

Impact

When adopting go-serum-analyzer in even a medium codebase...

We found we were happy with the result.

Check it out: