Self-improving software

https://www.kensho.com/careers

glebbahmutov.com

github.com/bahmutov

KENSHO

Technology that brings transparency to complex systems

Harvard Sq, WTC NYC

https://github.com/kensho

Does your code look like this?

There is a better way

codebase red flags
divide and conquer
semantic versioning
safe dependency updates
safe new version release

Outline

Source out of control - the red flags

Build time > 15 seconds

Multiple tools, environments

Impossible to unit test a feature

You have to use AND to describe the project and its goals

Source control to built and tested > 10 minutes

No code reuse

Giving up

https://github.com/sindresorhus/ama/issues/10

Repo size

single

function

several

files

Google

size

mono

repo

my limit

http://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext

backend

(API + www)

_ is simpler in a small project

Scope
Code and module reuse
Documentation
Testing
API design
Code reviews and standards
Installation / deployment
Onboarding

https://github.com/babel/babel/blob/master/doc/design/monorepo.md

Monorepo Exception*

Babel

PouchDB

...

single product + plugins

https://github.com/lerna/lerna

Q: Why are the giant code bases so difficult to deal with?

Software complexity

= number of interactions

var sum = add(a, b)

4 variables (a, b, add and sum)

4 * (4 - 1) / 2 = 6 interactions

Cognitive research

3 - 7 things at once

"Thinking, Fast and Slow" by Daniel Kahneman

Physical separation

A function isolates internal code

function add(a, b) { ... }
var sum = add(2, 3)
console.log('2 + 3 =', sum)

A module separates code from its dependencies

functions separate variables via scopes
source files separate code
dependencies separate files
teams separate people
slides separate bullet points

App Assembly

Each auto part is

built separately
tested separately
has SKU number
shipped to the auto plant
placed into the car

Assembling apps

same way as cars are built

Heroku's 12 Factor App

https://12factor.net/

Single repo per app

Do not build cars AND boats at the same plant

Share code via dependencies, not code

Do not build cars and tires at the same plant

https://www.linux.com/news/event/Nodejs/2016/state-union-npm

Every library has its own version

Old OEM parts are still made and installed

NPM - the largest library of JavaScript modules

You can build anything by using off the shelf parts ... if you have a huge warehouse

Node module: package.json

{
  "name": "my-module",
  "version": "0.1.0",
  "main": "index.js",
  "dependencies": {
    "foo": "0.1.*",
    "bar": "~1.2.0",
    "baz": "^2.0.1"
  },
  "devDependencies": {
    "grunt-concat": "0.1.1"
  }
}

Worry about the top level only

1.5.0

4.0.1

codebase red flags
divide and conquer
semantic versioning
safe dependency updates
safe new version release

Outline

Semantic versioning

major . minor . patch

break . feature . fix

semver.org

major: I changed how it works 🔥😡
minor: I added a new feature 👏👍
patch: I fixed something ✅❤️

really

Semantic versioning

semver.org

Upstream vs downstream

upstream

downstream

Dependency tree: day 0

0.1.0

2.2.0

0.1.0

Dependency tree: day 1

0.1.0

2.2.0

0.1.0

0.1.1

0.1.2

Dependency tree: day 3 (really)

0.1.0

2.2.0

0.1.0

0.1.1

0.1.2

2.2.1

2.2.2

2.5.0

3.0.0

3.0.1

https://david-dm.org/

Can we update? In theory

0.1.0

2.2.0

0.1.0

0.1.1

0.1.2

2.2.1

2.2.2

2.5.0

3.0.0

3.0.1

patches

0.1.0

2.2.0

0.1.0

0.1.1

0.1.2

2.2.1

2.2.2

2.5.0

3.0.0

3.0.1

patches

new feature

Can we update? In theory

0.1.0

2.2.0

0.1.0

0.1.1

0.1.2

2.2.1

2.2.2

2.5.0

3.0.0

3.0.1

patches

new feature

api change

Can we update? In theory

0.1.0

2.2.0

0.1.0

0.1.1

0.1.2

2.2.1

2.2.2

2.5.0

3.0.0

patches

new feature

api change

Can we update?

3.0.1

No one knows!

Relying on human-supplied semver is like relying on code comments to be 100% accurate

Q: Is this a mess?

A: Yes.

the mess is manageable.
the mess maps nicely to the software development:
- different parts are developed at different speeds.

Versioned dependencies isolate the true mess: constant merging of commits

source: https://static.pexels.com/photos/119661/pexels-photo-119661.jpeg

source: https://static.pexels.com/photos/53930/pexels-photo-53930.jpeg

Before I change my light bulb I must test Niagara Falls station to make sure it still works

Develop + integrate

function add(a, b) { ... }
...
... add(2, 3); 
    ....
add(-1, 100); 
...
...
console.log(add(0, 0));
...

Develop + integrate

function add(a, b) { XXX }
...
... add(2, 3); 
    ....
add(-1, 100); 
...
...
console.log(add(0, 0));
...

change

Develop + integrate

function add(a, b) { XXX }
...
... add(2, 3); 
    ....
add(-1, 100); 
...
...
console.log(add(0, 0));
...

Develop + integrate

function add(a, b) { XXX }
...
... add(2, 3); 
    ....
add(-1, 100); 
...
...
console.log(add(0, 0));
...

Hard to improve "add" AND keep everything working

Alternative: Develop THEN integrate

// package.json "add": "1.0.0"
var add = require('add');
...
... add(2, 3); 
    ....
add(-1, 100); 
...
...
console.log(add(0, 0));

// module ADD@1.0.0
function add(a, b) { ... }

Develop THEN integrate

// package.json "add": "1.0.0"
var add = require('add');
...
... add(2, 3); 
    ....
add(-1, 100); 
...
...
console.log(add(0, 0));

// module ADD@1.1.0
function add(a, b) { XXX }

change

Develop THEN integrate

// package.json "add": "1.1.0"
var add = require('add');
...
... add(2, 3); 
    ....
add(-1, 100); 
...
...
console.log(add(0, 0));

// module ADD@1.1.0
function add(a, b) { XXX }

new

version

Revert, no harm done

// package.json "add": "1.0.0"
var add = require('add');
...
... add(2, 3); 
    ....
add(-1, 100); 
...
...
console.log(add(0, 0));

// module ADD@1.1.0
function add(a, b) { XXX }

revert

version

Q: Can we automate dependency upgrades?

Run unit tests
Install each new dependency
Run unit tests again
1. Keep or revert

(safe upstream)

https://github.com/bahmutov/next-update

A: next-update

watch

next-update

watch

https://github.com/bahmutov/next-update

next-update

If you test your software - you will get 3rd party upgrades with zero effort

https://github.com/bahmutov/next-updater

next-updater

When you have lots of projects to upgrade

GreenKeeper.io - dependency update as a service

Q: Is the update

foo@x.y.z => foo@x.y + 1.z

likely to succeed?

https://github.com/bahmutov/next-update-stats

next-update-stats

next-update-stats

https://github.com/bahmutov/next-update-stats

https://glebbahmutov.com/blog/measuring-semver-adherence/

Q: Are automated upgrades scary?

iOS6 vs iOS7 app updates

Automatic upgrades should not be scary

Your project is well tested
A particular dependency update has a high probability of being successful

codebase red flags
divide and conquer
semantic versioning
safe dependency updates
safe new version release

Outline

https://github.com/semantic-release/semantic-release

Automate NPM publish

$ npm install -g semantic-release-cli
$ semantic-release-cli setup
$ git commit -m "minor(app): add new feature"
$ git push

New minor version is published by your CI if tests pass

https://www.npmjs.com/package/semantic-release-gitlab

Commit message convention

https://github.com/bahmutov/simple-commit-message

$ git commit -m "minor(app): add new feature"
$ git push

$ git commit -m "patch(log): fix log rotation"
$ git commit -m "major(api): change output format"
$ git push

new minor version published

new major version published

Q: Are you going to break everyone?

https://github.com/bahmutov/dont-break

dont-break from module X

Install each dependent project
Replace X@x.y.z with X@current
Run unit tests

https://github.com/bahmutov/dont-crack

If we are breaking dependent projects - maybe we should increment MAJOR

dont-break from module X

plugin for semantic-release

https://glebbahmutov.com/blog/how-i-publish-to-npm/

Automate all the things!

http://www.chicagotribune.com/bluesky/hub/chi-inc-robots-doing-more-office-work-bsi-hub-20150617-story.html

Example: snap-shot

snap-shot

snap-shot-core

schema-shot

snap-shot-jest-test

snap-shot-ava-test

snap-shot-vue-test

...

snap-shot

snap-shot-core

schema-shot

snap-shot-jest-test

snap-shot-ava-test

snap-shot-vue-test

semantic release

with dont-crack

...

Example: snap-shot

snap-shot

snap-shot-core

schema-shot

snap-shot-jest-test

snap-shot-ava-test

snap-shot-vue-test

semantic release

with dont-crack

...

Example: snap-shot

snap-shot

snap-shot-core

schema-shot

snap-shot-jest-test

snap-shot-ava-test

snap-shot-vue-test

semantic release

with dont-crack

semantic release

with dont-crack

...

Example: snap-shot

snap-shot

snap-shot-core

schema-shot

snap-shot-jest-test

snap-shot-ava-test

snap-shot-vue-test

semantic release

with dont-crack

semantic release

with dont-crack

...

Example: snap-shot

Conclusions

Single huge project => lots of small projects
Automate new version release: semantic-release, dont-break
Automate dependencies next-update / greenkeeper.io