Self-improving software

Dr. Gleb Bahmutov PhD

KENSHO

@bahmutov

glebbahmutov.com

#ConFoo

github.com/bahmutov

.CA

KENSHO

Technology that brings transparency to complex systems

Harvard Sq, WTC NYC

https://www.kensho.com/careers

https://github.com/kensho

Does your code look like this?

There is a better way

Source out of control - the red flags

Build time > 15 seconds

Source control to built and tested > 10 minutes

• Large projects are never finished (https://glebbahmutov.com/blog/large-projects-are-never-finished/)

• Monolithic Node does not work (http://www.richardrodger.com/monolithic-nodejs#.WMC6XRLytE4)

• Less is more (http://www.stevemcconnell.com/articles/art06.htm)
• Why Google Stores Billions of Lines of Code in a Single Repository (http://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext)

Large projects are bad: extra reading

 Q: Why are the giant code bases so difficult to deal with?

Software complexity

= number of interactions

Cognitive research

3 - 7 things at once

"Thinking, Fast and Slow" by Daniel Kahneman

A function isolates internal code

function add(a, b) { ... }
var sum = add(2, 3)
console.log('2 + 3 =', sum)

A module separates code from its dependencies

• functions separate variables via scopes

• source files separate code

• dependencies separate files

• teams separate people

• slides separate bullet points

App Assembly

Each auto part is

1. built separately

2. tested separately

3. has SKU number

4. shipped to the auto plant

5. placed into the car

Assembling apps

same way as cars are built

Heroku's 12 Factor App

https://12factor.net/

Single repo per app

Do not build cars AND boats at the same plant

Share code via dependencies, not code

Do not build cars and tires at the same plant

Every library has its own version

Old OEM parts are still made and installed

Node module: package.json

{
  "name": "my-module",
  "version": "0.1.0",
  "main": "index.js",
  "dependencies": {
    "foo": "0.1.*",
    "bar": "~1.2.0",
    "baz": "^2.0.1"
  },
  "devDependencies": {
    "grunt-concat": "0.1.1"
  }
}

Package.json fields

https://nodesource.com/blog/the-basics-of-package-json-in-node-js-and-npm

Init package.json file

https://github.com/bahmutov/generator-node-bahmutov

$ npm init --yes

Worry about the top level only

Semantic versioning

major . minor . patch

semver.org

• major: I changed how it works  🔥😡

• minor: I added a new feature    👏👍

• patch: I fixed something            ✅❤️

really

Semantic versioning

semver.org

Dependency tree: day 0

0.1.0

2.2.0

2.2.0

0.1.0

Dependency tree: day 1

0.1.0

2.2.0

2.2.0

0.1.0

0.1.1

0.1.2

Dependency tree: day 3 (really)

0.1.0

2.2.0

2.2.0

0.1.0

0.1.1

0.1.2

2.2.1

2.2.2

2.5.0

3.0.0

3.0.1

https://david-dm.org/

Can we update? In theory

0.1.0

2.2.0

2.2.0

0.1.0

0.1.1

0.1.2

2.2.1

2.2.2

2.5.0

3.0.0

3.0.1

patches

patches

0.1.0

2.2.0

2.2.0

0.1.0

0.1.1

0.1.2

2.2.1

2.2.2

2.5.0

3.0.0

3.0.1

patches

patches

new feature

Can we update? In theory

0.1.0

2.2.0

2.2.0

0.1.0

0.1.1

0.1.2

2.2.1

2.2.2

2.5.0

3.0.0

3.0.1

patches

patches

new feature

api change

Can we update? In theory

0.1.0

2.2.0

2.2.0

0.1.0

0.1.1

0.1.2

2.2.1

2.2.2

2.5.0

3.0.0

patches

patches

new feature

api change

Can we update? In practice

3.0.1

Versioned dependencies isolate the true mess: constant merging of commits

Before I change my light bulb I must test Niagara Falls station to make sure it still works

Develop + integrate

function add(a, b) { ... }
...
... add(2, 3); 
    ....
add(-1, 100); 
...
...
console.log(add(0, 0));
...

Develop + integrate

function add(a, b) { XXX }
...
... add(2, 3); 
    ....
add(-1, 100); 
...
...
console.log(add(0, 0));
...

change

Develop + integrate

function add(a, b) { XXX }
...
... add(2, 3); 
    ....
add(-1, 100); 
...
...
console.log(add(0, 0));
...

Develop + integrate

function add(a, b) { XXX }
...
... add(2, 3); 
    ....
add(-1, 100); 
...
...
console.log(add(0, 0));
...

Hard to improve "add" AND keep everything working

Alternative: Develop THEN integrate

// module ADD@1.0.0
function add(a, b) { ... }

Develop THEN integrate

// package.json "add": "1.0.0"
var add = require('add');
...
... add(2, 3); 
    ....
add(-1, 100); 
...
...
console.log(add(0, 0));

// module ADD@1.1.0
function add(a, b) { XXX }

change

Develop THEN integrate

// package.json "add": "1.1.0"
var add = require('add');
...
... add(2, 3); 
    ....
add(-1, 100); 
...
...
console.log(add(0, 0));

// module ADD@1.1.0
function add(a, b) { XXX }

new

version

Revert, no harm done

// package.json "add": "1.0.0"
var add = require('add');
...
... add(2, 3); 
    ....
add(-1, 100); 
...
...
console.log(add(0, 0));

// module ADD@1.1.0
function add(a, b) { XXX }

revert

version

Q: Can we automate dependency upgrades?

1. Run unit tests

2. Install each new dependency

3. Run unit tests again

   1. Keep or revert

(safe upstream)

A: next-update

https://github.com/bahmutov/next-update

watch

next-update

watch

https://github.com/bahmutov/next-update

next-update

If you test your software - you will get 3rd party upgrades with zero effort

next-updater

When you have lots of projects to upgrade

https://github.com/bahmutov/next-updater

GreenKeeper.io - dependency update as a service

next-update-stats

https://github.com/bahmutov/next-update-stats

next-update-stats

https://github.com/bahmutov/next-update-stats

https://glebbahmutov.com/blog/measuring-semver-adherence/

Q: Are automated upgrades scary?

iOS6 vs iOS7 app updates

Automatic upgrades should not be scary

• Your project is well tested

• A particular dependency update has a high probability of being successful

NPM     0.x.y

http://npmbynumbers.bocoup.com/ (2015)

We fail to follow semver

Stephen Bonnemann (JSConf Budapest 2015)

https://speakerdeck.com/boennemann/we-fail-to-follow-semver-and-why-it-neednt-matter-1

We fail to follow semver

- and it does matter

rant

We fail to follow semver

- and it does matter

rant

Automate SemVer

https://github.com/semantic-release/semantic-release

$ npm install -g semantic-release-cli
$ semantic-release-cli setup
$ git commit -m "minor(app): add new feature"
$ git push

https://www.npmjs.com/package/semantic-release-gitlab

Commit message convention

https://github.com/bahmutov/simple-commit-message

$ git commit -m "minor(app): add new feature"
$ git push

Q: Are you going to break everyone?

source: http://darkroom.baltimoresun.com/2016/06/10th-annual-tomato-fight-festival-in-colombia/#1

dont-break from module X

1. Install each dependent project

2. Replace X@x.y.z with X@current

3. Run unit tests 

If we are breaking dependent projects - maybe we should increment MAJOR

dont-break from module X

Example: snap-shot

https://github.com/bahmutov/snap-shot

snap-shot

https://glebbahmutov.com/blog/snapshot-testing/

Example: snap-shot

snap-shot

snap-shot-core

schema-shot

snap-shot-jest-test

snap-shot-ava-test

snap-shot-vue-test

semantic release

with dont-crack

https://github.com/bahmutov/snap-shot

https://glebbahmutov.com/blog/snapshot-testing/

...

Example: snap-shot

snap-shot

snap-shot-core

schema-shot

snap-shot-jest-test

snap-shot-ava-test

snap-shot-vue-test

semantic release

with dont-crack

https://github.com/bahmutov/snap-shot

https://glebbahmutov.com/blog/snapshot-testing/

safe updates with Greenkeeper.io

(or next-update)

...

Example: snap-shot

snap-shot

snap-shot-core

schema-shot

snap-shot-jest-test

snap-shot-ava-test

snap-shot-vue-test

semantic release

with dont-crack

https://github.com/bahmutov/snap-shot

https://glebbahmutov.com/blog/snapshot-testing/

semantic release

with dont-crack

safe updates with Greenkeeper.io

(or next-update)

...