Gleb Bahmutov PRO
JavaScript ninja, image processing expert, software quality fanatic
Self-improving software
Dr. Gleb Bahmutov PhD
KENSHO
#ConFoo
.CA
Confoo
KENSHO
Technology that brings transparency to complex systems
Harvard Sq, WTC NYC
Does your code look like this?
There is a better way
-
codebase red flags
-
divide and conquer
-
semantic versioning
-
safe dependency updates
-
safe new version release
Outline
Source out of control - the red flags
Build time > 15 seconds
Multiple tools, environments
Impossible to unit test a feature
You have to use AND to describe the project and its goals
Source control to built and tested > 10 minutes
No code reuse
Giving up
-
Large projects are never finished (https://glebbahmutov.com/blog/large-projects-are-never-finished/)
-
Monolithic Node does not work (http://www.richardrodger.com/monolithic-nodejs#.WMC6XRLytE4)
- Less is more (http://www.stevemcconnell.com/articles/art06.htm)
- Why Google Stores Billions of Lines of Code in a Single Repository (http://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext)
Large projects are bad: extra reading
Things that are simpler in small projects
-
Scope
-
Code and module reuse
-
Documentation
-
Testing
-
API design
-
Code reviews and standards
-
Installation / deployment
-
Onboarding
Q: Why are the giant code bases so difficult to deal with?
Software complexity
= number of interactions
var sum = add(a, b)4 variables (a, b, add and sum)
4 * (4 - 1) / 2 = 6 interactions
Cognitive research
3 - 7 things at once
"Thinking, Fast and Slow" by Daniel Kahneman
Physical separation
A function isolates internal code
function add(a, b) { ... }
var sum = add(2, 3)
console.log('2 + 3 =', sum)
A module separates code from its dependencies
-
functions separate variables via scopes
-
source files separate code
-
dependencies separate files
-
teams separate people
-
slides separate bullet points
App Assembly
Each auto part is
-
built separately
-
tested separately
-
has SKU number
-
shipped to the auto plant
-
placed into the car
Assembling apps
same way as cars are built
Heroku's 12 Factor App
https://12factor.net/
Single repo per app
Do not build cars AND boats at the same plant
Share code via dependencies, not code
Do not build cars and tires at the same plant
Every library has its own version
Old OEM parts are still made and installed
NPM - the largest library of JavaScript modules
You can build anything by using off the shelf parts ... if you have a huge warehouse
Node module: package.json
{
"name": "my-module",
"version": "0.1.0",
"main": "index.js",
"dependencies": {
"foo": "0.1.*",
"bar": "~1.2.0",
"baz": "^2.0.1"
},
"devDependencies": {
"grunt-concat": "0.1.1"
}
}
Package.json fields
Init package.json file
$ npm init --yes
Worry about the top level only
1.5.0
4.0.1
Semantic versioning
major . minor . patch
break . feature . fix
-
major: I changed how it works 🔥😡
-
minor: I added a new feature 👏👍
-
patch: I fixed something ✅❤️
really
Semantic versioning
Upstream vs downstream
upstream
downstream
Dependency tree: day 0
0.1.0
2.2.0
2.2.0
0.1.0
Dependency tree: day 1
0.1.0
2.2.0
2.2.0
0.1.0
0.1.1
0.1.2
Dependency tree: day 3 (really)
0.1.0
2.2.0
2.2.0
0.1.0
0.1.1
0.1.2
2.2.1
2.2.2
2.5.0
3.0.0
3.0.1
Can we update? In theory
0.1.0
2.2.0
2.2.0
0.1.0
0.1.1
0.1.2
2.2.1
2.2.2
2.5.0
3.0.0
3.0.1
patches
patches
0.1.0
2.2.0
2.2.0
0.1.0
0.1.1
0.1.2
2.2.1
2.2.2
2.5.0
3.0.0
3.0.1
patches
patches
new feature
Can we update? In theory
0.1.0
2.2.0
2.2.0
0.1.0
0.1.1
0.1.2
2.2.1
2.2.2
2.5.0
3.0.0
3.0.1
patches
patches
new feature
api change
Can we update? In theory
0.1.0
2.2.0
2.2.0
0.1.0
0.1.1
0.1.2
2.2.1
2.2.2
2.5.0
3.0.0
patches
patches
new feature
api change
Can we update? In practice
No one knows for sure
3.0.1
Relying on human-supplied semver is like relying on code comments to be 100% accurate
Q: Is this a mess?
A: Yes.
- the mess is manageable.
- the mess maps nicely to the software development:
- different parts are developed at different speeds.
Versioned dependencies isolate the true mess: constant merging of commits
source: https://static.pexels.com/photos/119661/pexels-photo-119661.jpeg
source: https://static.pexels.com/photos/53930/pexels-photo-53930.jpeg
Before I change my light bulb I must test Niagara Falls station to make sure it still works
Develop + integrate
function add(a, b) { ... }
...
... add(2, 3);
....
add(-1, 100);
...
...
console.log(add(0, 0));
...Develop + integrate
function add(a, b) { XXX }
...
... add(2, 3);
....
add(-1, 100);
...
...
console.log(add(0, 0));
...change
Develop + integrate
function add(a, b) { XXX }
...
... add(2, 3);
....
add(-1, 100);
...
...
console.log(add(0, 0));
...Develop + integrate
function add(a, b) { XXX }
...
... add(2, 3);
....
add(-1, 100);
...
...
console.log(add(0, 0));
...Hard to improve "add" AND keep everything working
Alternative: Develop THEN integrate
// package.json "add": "1.0.0"
var add = require('add');
...
... add(2, 3);
....
add(-1, 100);
...
...
console.log(add(0, 0));// module ADD@1.0.0
function add(a, b) { ... }
Develop THEN integrate
// package.json "add": "1.0.0"
var add = require('add');
...
... add(2, 3);
....
add(-1, 100);
...
...
console.log(add(0, 0));// module ADD@1.1.0
function add(a, b) { XXX }change
Develop THEN integrate
// package.json "add": "1.1.0"
var add = require('add');
...
... add(2, 3);
....
add(-1, 100);
...
...
console.log(add(0, 0));// module ADD@1.1.0
function add(a, b) { XXX }new
version
Revert, no harm done
// package.json "add": "1.0.0"
var add = require('add');
...
... add(2, 3);
....
add(-1, 100);
...
...
console.log(add(0, 0));// module ADD@1.1.0
function add(a, b) { XXX }revert
version
Q: Can we automate dependency upgrades?
-
Run unit tests
-
Install each new dependency
-
Run unit tests again
-
Keep or revert
-
(safe upstream)
A: next-update
next-update
next-update
If you test your software - you will get 3rd party upgrades with zero effort
next-updater
When you have lots of projects to upgrade
GreenKeeper.io - dependency update as a service
Q: Is the update
foo@x.y.z => foo@x.y + 1.z
likely to succeed?
next-update-stats
next-update-stats
Q: Are automated upgrades scary?
iOS6 vs iOS7 app updates
Automatic upgrades should not be scary
-
Your project is well tested
-
A particular dependency update has a high probability of being successful
-
Minor / patch update
-
Automated release
Automated upstream updates (never fall behind new versions)
Last part: Automate new version release
(safe downstream)
NPM 0.x.y
We fail to follow semver
Stephen Bonnemann (JSConf Budapest 2015)
We fail to follow semver
- and it does matter
Project activity - # of commits
Is your new release likely to NOT work for MY project? - SEMVER
rant
We fail to follow semver
- and it does matter
Backbone@43.0.0 is OK
Backbone 43.0.0 -> 43.0.1 NOT working right away for my web app is NOT OK
rant
see also (angular.js => angular2 => angular 4)
Automate SemVer
$ npm install -g semantic-release-cli
$ semantic-release-cli setup
$ git commit -m "minor(app): add new feature"
$ git pushNew minor version is published by your CI if tests pass
Commit message convention
$ git commit -m "minor(app): add new feature"
$ git push$ git commit -m "patch(log): fix log rotation"
$ git commit -m "major(api): change output format"
$ git pushnew minor version published
new major version published
Q: Are you going to break everyone?
source: http://darkroom.baltimoresun.com/2016/06/10th-annual-tomato-fight-festival-in-colombia/#1
dont-break: let the entire Internet be your test case
dont-break from module X
-
Install each dependent project
-
Replace X@x.y.z with X@current
-
Run unit tests
If we are breaking dependent projects - maybe we should increment MAJOR
dont-break from module X
plugin for semantic-release
Automated upstream updates (never fall behind new versions)
Automated downstream releases (never break your users without warning)
robots from https://robohash.org/
Automate all the things!
http://www.chicagotribune.com/bluesky/hub/chi-inc-robots-doing-more-office-work-bsi-hub-20150617-story.html
Example: snap-shot
snap-shot
snap-shot-core
schema-shot
snap-shot-jest-test
snap-shot-ava-test
snap-shot-vue-test
...
Example: snap-shot
snap-shot
snap-shot-core
schema-shot
snap-shot-jest-test
snap-shot-ava-test
snap-shot-vue-test
semantic release
with dont-crack
...
Example: snap-shot
snap-shot
snap-shot-core
schema-shot
snap-shot-jest-test
snap-shot-ava-test
snap-shot-vue-test
semantic release
with dont-crack
safe updates with Greenkeeper.io
(or next-update)
...
Example: snap-shot
snap-shot
snap-shot-core
schema-shot
snap-shot-jest-test
snap-shot-ava-test
snap-shot-vue-test
semantic release
with dont-crack
semantic release
with dont-crack
safe updates with Greenkeeper.io
(or next-update)
...
Conclusions
-
Single huge project => lots of small projects
-
Automate new version release: semantic-release, dont-break
-
Automate dependencies next-update / greenkeeper.io
Self-improving software
Thank you 👏
#ConFoo
.CA
Confoo
Self-improving software - ConFoo CA 2017
By Gleb Bahmutov
Self-improving software - ConFoo CA 2017
Every JavaScript project requires 3rd party modules. As soon as you depend on specific versions, your module falls behind. I will show how to keep your software up to date without any effort (as long as there are some tests). Everything will be automated and reliable; it is like bringing an army of robots to help you.
