Golang At Optiver

Sydney Go Meetup 2014-11-19

Who are we?

Optiver is a propietary global market maker, trading on all major financial markets around the world, putting only its own capital at risk. Making markets in financial products is Optiver's core business.

What technologies do we use?

We're a polyglot company:

 

  • C++
  • C#
  • Python
  • (most recently) Go

 

We're hiring!

http://www.optiver.com/sydney/careersoptiver

What this talk is about

  • How we came to be using Go
  • The awesome parts of our Go experience
  • The not so awesome parts of our Go experience
  • Where we're currently at with using Go.
  • Where we're heading in the future with Go.

How it all started

One of our server-like products is implemented in Python. But it has problems:

 

  • Slow.
  • Client processes experiencing timeouts.
  • Reconnect is especially slow -- positive feedback loop of pain.
  • Server gets swamped.
  • I don't like the Twisted Python framework.

What did we do?

We've been chipping away at this problem for about a year.

 

We tried a few different things before trying out Go.

PyPy

  • Bad interaction with the version of the Python Twisted framework we were using.

C++ Python Protocol Buffers

  • We were originally using native Python Protocol Buffers.
  • The unit tests didn't pass when we evaluated the C++ version!
  • Too risky.

Redis For Caching

  • Didn't align well with our usecase.
  • We needed something more feature-rich.
  • More than just caching key/value pairs!

Enter Go

We implemented a small prototype of our server product using Go. We tried Go because it's advertised to be:

 

  • Fast to execute.
  • Fast to develop with.

 

That was enough to tempt us.

Our Go Server Prototype

  • We had no experience writing Go code.
  • Had a prototype up and running in 2 days.
  • Performance was impressive:
    • ~40 times faster than our Python server.
    • Held up under heavy load testing.

Server Rewrite in Go

  • Two months of effort.
  • Remember, none of us had prior experience with Go!
  • Language features and tools contributed to development speed.

Go Language

  • Minimal and clean syntax.
  • Simple design.
  • Built in concurrency primitives.
  • Very easy to read.
    • People with no Go experience were able to contribute to the code review process.

Go Tooling

  • Gofmt enforces 'one true style'.
    • Writing code is quicker.
    • Reading code is easier.
    • Code review has less 'noise'.
  • Compiler is fast! (compared to C++ compilers)
    • Test Driven Development with short cycles is possible.

Test code not in production

// File: validate_test.go

// Dummy validate validates all inputs as being good,
// since in testing we don't care about validation.
func dummyValidate(x int) bool {
    return true
}
// File: validation.go

var validates func(int) bool
validates = dummyValidate // This won't compile!

if validates(badInput) {
    // stop something bad from happening!
    // ...
}

Deployment to Production

  • Single binary
  • Minimal dependencies, only links with pthreads and libc.
  • Just copy the binary and you're good to go!

3rd party libraries

How can we be confident that they work?

 

Tested?

 

Feature complete?

 

Go is now compared to other technologies such as C++ and Python. Ecosystem isn't as well evolved.

Lib pq

  • 'Just worked'.
  • We still thoroughly tested each part of the library we wanted to use.

Goprotobuf

  • Most of our internal protocols are protobuf based, so this was a massive win.

 

  • But watch out for protocols spanning multiple files,
  • Keep message type enums consistent!

Testing

C++ ⇨ Google Mock and Google Test framework.

 

Initially Go's standard 'testing' library seemed lackluster...

 

Initial thoughts on testing

  • Testify for test suites and assertions.
  • Gomock for mocking.
  • Testify still isn't as powerful as Google Test.
  • Gomock has confusing rules, and you have to read the src to work out how some parts work!

Testing frameworks aren't needed

  • Just as easy to do things with normal Go code.
  • Just using Go code provides the ultimate flexibility.
  • Less dependencies are nice.

Roll your own mocks

type FooInterface interface {
    Bar(int) bool
    Baz(string, int)
}

type MockFoo struct {
    FnBar func(int) bool
    FnBaz func(string, int)
}

func (f *mockFoo) Bar(x int) bool {
    return f.FnBar(x)
}

func (f *mockFoo) Baz(s string, x int) {
    f.FnBaz(s, x)
}

Not all roses...

  • Integration with our existing tools.
  • Interoperability with our existing C++ libs.
  • Centos 5.
  • Dependency management

Existing Tools

  • We have lots of in house app management tools. Solve:
    • What version of an app are we running?
    • What apps communicate with what other apps?
  • We have C++ libs that play nicely with these tools.
    • Have to rewrite these as well!

Configuration

  • All of our apps are configured with XML (and our tools work with XML!).
  • And we want XSD validation.
  • No existing XSD validation libs in Go, but we don't want to write our own.
  • Instead configure via command line argument.
    • But this doesn't scale well.

Existing Libs

  • Many in house libs to provide business specific functionality.
  • We want to leverage these in Go apps.
  • General problem that will exist for all medium to large sized corporate environments.

 

SWIG

  • Connects C/C++ to higher level languages.
  • But our interfaces use complex types:
    • Nested objects,
    • Templates,
    • Standard library containers.
  • SWIG and Go compiler don't integrate nicely.
  • We didn't get it working with our libs.

Transliteration to Go

DB Schema (XML) ⇨ Jinja2/Python script ⇨ C++

Surely we can do the same with Go?

Yes, but it's difficult

Some language features of C++ don't transliterate well:

 

  • Templates
  • Overloaded functions
  • Default parameters
  • Operator overloading
  • Certain types of inheritance.

When we solve this, it's really going to test out Go compiler performance. We expect ~250K Go LOC.

Centos 5

No Go support.

Not a show stopper for us:

​

  • Less precision in timestamps,
  • File descriptors can leak into child processes.

 

We're concerned that there's no guarantee that more things won't break in future Go releases on Centos 5.

Dependency Management

  • Go Get ⇨ Non reproducible builds
  • CI servers have no internet access.

Vendored code is mirrored on internal SVN/Git servers and pulled in as externals.

Project Structure

One GOPATH per project:

.
|-- projectA
|   `-- src
|       `-- optiver.com
|           `-- ...
`-- projectB
    `-- src
        `-- optiver.com
            `-- ...

Future Plans

  • Our first Go app has been in production for some months.
  • Doors are now open to use Go in other places.
    • More server style applications,
    • Backend processes,
    • Tools

Questions?

Made with Slides.com