Learn About Node's New Built-in
Test Runner
Gleb Bahmutov
gleb.dev
NERD Summit 2023
Amherst, MA, USA
The climate crisis is:
here
urgent
caused by burning fossil fuels
survival is possible* but we need to act now
Speaker: Gleb Bahmutov PhD
C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing
Gleb Bahmutov
Sr Director of Engineering
Let's Test!!!
(our Node code)
{
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
}
}
jest v29 7 years
mocha v10 11 years
npx available-versions mocha
tape v5 10 years
tap v16 11 years
ava v5 9 years
Node.js 13 years
May 2009
Node.js is a minimal core
Write your own test runner!
JavaScript standard library is tiny
Node.js does not include test runner
Write your own test runner!
gt.test("get N '='", function () {
gt.ok(typeof getLines === "function",
"getLines is a function");
gt.equal(getLines(0), "", "0 character");
gt.equal(getLines(1), "=", "1 character");
});- collect all test hooks and callbacks
- start running hooks and callbacks
- report passes and failures
- expand features:
- add assertions
- better async support
- spying and stubbing
- code coverage
- running tests in parallel
"gt: Gleb Test" framework test sample
gt.async('run a program', function () {
gt.exec('node', ['index.js', 'arg1', 'arg2'], 0,
function (stdout, stderr) {
if (/error/.test(stdout)) {
throw new Error('Errors in output ' + stdout);
}
});
});Async via callbacks was the way 😄
- Assertions:
- Snapshot testing
- Data-driven testing
- X
- Y
- Z
Mocha + Chai + Sinonjs
Ava
Ava (Node)
Cypress (browser)
Node v18 test runner
Ported to v16.x
I am using v19.6.0
Test examples shown in this presentation: https://github.com/bahmutov/node-tests
2022-04-19
2022-04-19
2023-03-08
Testing added 2 things
import test from 'node:test'
New built-in module "node:test"
$ node --test tests/*.mjsNew --test Node CLI flag
The tests
import test from 'node:test'
import assert from 'node:assert/strict'
test('hello', () => {
const message = 'Hello'
assert.equal(message, 'Hello', 'checking the greeting')
})Every test shown in this presentation: https://github.com/bahmutov/node-tests
The tests
import test from 'node:test'
import assert from 'node:assert/strict'
test('hello', () => {
const message = 'Hello'
assert.equal(message, 'Hello', 'checking the greeting')
})
Comparing To Other Test Runners
- installation
- execution
- syntax
- assertions
- hooks
- reporters
- speed
- TypeScript support
"Installation"
$ nvm ls-remote
$ nvm install 19
Downloading and installing node v19.6.0...
Downloading https://nodejs.org/dist/v19.6.0/...
############################################## 100.0%
Computing checksum with shasum -a 256
Checksums matched!
Now using node v19.6.0 (npm v9.4.0)🎁 Just use "nvm" https://github.com/nvm-sh/nvm
# .nvmrc file
19.6.0
$ nvm use
$ nvm use
Found '/node-tests/.nvmrc' with version <19.6.0>
Now using node v19.6.0 (npm v9.4.0)"Installation"
🎁 Just use "nvm" https://github.com/nvm-sh/nvm
| Test runner | NPM packages installed |
|---|---|
| Mocha | 78 |
| Mocha + Chai + Sinon | 445 |
| Ava | 198 |
| Jest | 429 |
Execution
$ node tests/demo.mjs
Tip: you can run tests in any JS file that has import "node:test" command
$ node --watch tests/demo.mjs
$ node --watch --test tests/*.mjsTip: test runner is works with built-in Node --watch mode
Tip: finds the tests by default
$ node --test
# runs tests in "test" subfolderTest files naming convention
Start or end test file names with "test" to automatically recursively find them
tests/
names/
another.test.mjs
my_test.mjs
test-1.mjs
utils.mjs$ node --test tests/names
- tests/names/another.test.mjs
- tests/names/my_test.mjs
- tests/names/test-1.mjsTest file convention
https://nodejs.org/api/test.html#test-runner-execution-model
Parallel
describe('parallel tests', { concurrency: true }, () => {
it('subtest 1', async () => {
console.log('subtest 1 start')
await delay(5000)
console.log('subtest 1 end')
})
it('subtest 2', async () => {
console.log('subtest 2 start')
await delay(5000)
console.log('subtest 2 end')
})
})Parallel
describe('parallel tests', { concurrency: true }, () => {
it('subtest 1', async () => {
console.log('subtest 1 start')
await delay(5000)
console.log('subtest 1 end')
})
it('subtest 2', async () => {
console.log('subtest 2 start')
await delay(5000)
console.log('subtest 2 end')
})
})
The tests
import test from 'node:test'
import assert from 'node:assert/strict'
test('top level test', async (t) => {
await t.test('subtest 1', (t) => {
assert.strictEqual(1, 1)
})
await t.test('subtest 2', (t) => {
assert.strictEqual(2, 2)
})
})
... look weird
Every test shown in this presentation: https://github.com/bahmutov/node-tests
The tests
import test from 'node:test'
import assert from 'node:assert/strict'
test('top level test', async (t) => {
await t.test('subtest 1', (t) => {
assert.strictEqual(1, 1)
})
await t.test('subtest 2', (t) => {
assert.strictEqual(2, 2)
})
})
... look weird
Every test shown in this presentation: https://github.com/bahmutov/node-tests
The tests
import test from 'node:test'
import assert from 'node:assert/strict'
test('top level test', async (t) => {
await test('subtest 1', (t) => {
assert.strictEqual(1, 1)
})
await t.test('subtest 2', (t) => {
assert.strictEqual(2, 2)
})
})
... look weird
Every test shown in this presentation: https://github.com/bahmutov/node-tests
Wrong "test" function
Every test shown in this presentation: https://github.com/bahmutov/node-tests
BDD syntax
import { describe, it } from 'node:test'
import assert from 'node:assert/strict'
describe('top level test', () => {
it('subtest 1', () => {
assert.strictEqual(1, 1)
})
it('subtest 2', () => {
assert.strictEqual(2, 2)
})
})Every test shown in this presentation: https://github.com/bahmutov/node-tests
import { describe, it } from 'node:test'
import assert from 'node:assert/strict'
describe('top level test', () => {
it('subtest 1', () => {
assert.strictEqual(1, 1)
})
it('subtest 2', () => {
assert.strictEqual(2, 2)
})
})1
2
Reporters
$ node --test --test-reporter tap # default
$ node --test --test-reporter spec
$ node --test --test-reporter dot$ node --test --test-reporter dot \
--test-reporter-destination stdout \
--test-reporter spec \
--test-reporter-destination out.txtmultiple reporters
⚠️ Arguments order
🚫 wrong order
✅ test runner arguments come before test filenames
{
"scripts": {
"test": "node --test",
"spec": "node --test --test-reporter spec"
}
}package.json
Via TAP
Test Anything Protocol
Demo "node --test" vs "node --test | npx faucet"
# npm i -D faucet
$ node --test | npx faucet"node --test" plus https://github.com/ljharb/faucet output
Test Statuses
it('works', () => {
assert.equal(1, 1)
})
it('fails', () => {
assert.equal(2, 5)
})
Test Statuses: skip/todo
it.todo('loads data')
// SKIP: <issue link>
it.skip('stopped working', () => {
assert.equal(2, 5)
})
Test Statuses: cancelled
before(() => {
console.log('before hook')
throw new Error('Setup fails')
})
it('works', () => {
assert.equal(1, 1)
})
it('works again', () => {
assert.equal(1, 1)
})
🐞
Speaking of "before" hook
describe('feature', () => {
before(() => {
console.log('before hook')
})
it('works', () => {
assert.equal(1, 1)
})
it('works again', () => {
assert.equal(1, 1)
})
})
Speaking of "before" hook
before(() => {
console.log('before hook')
})
it('works', () => {
assert.equal(1, 1)
})
it('works again', () => {
assert.equal(1, 1)
})
Assertions
via built-in Node assert module
import assert from 'node:assert/strict'
assert.ok(truthy, message)
assert.equal(value, expected, ...)
assert.deepEqual(...)
assert.match(value, regexp)
assert.throws(fn)
assert.rejects(asyncFn)plus ".notX" assertions
quite sparse compared to https://www.chaijs.com/api/bdd/
Missing
expect({ /* object */ }).to.have.property(x)
expect({ /* object */ }).to.have.keys([x, y, z])
expect({ /* large object */ })
.to.deep.include({ ... known properties })// import assert from 'node:assert/strict'
import { assert } from 'chai'Assertions
via built-in Node assert module
import { describe, it } from 'node:test'
import assert from 'node:assert/strict'
describe('Assertions', () => {
it('passes with primitives', () => {
assert.equal('Hello', 'Helloz', 'greeting check')
})
})fail the assertion on purpose
Assertions
via built-in Node assert module
it('fails objects on purpose', () => {
const person = { name: { first: 'Joe' } }
assert.deepEqual(person,
{ name: { first: 'Anna' } }, 'people')
})fail the assertion on purpose
Compare to Ava.js
with its "magic assert"
import test from 'ava'
test('fails objects on purpose', (t) => {
const person = { name: { first: 'Joe' } }
t.deepEqual(person,
{ name: { first: 'Anna' } }, 'people')
})
fail the assertion on purpose
Compare to Ava.js
with its "magic assert"
import test from 'ava'
test('passes with primitives', (t) => {
t.is('Hello', 'Helloz', 'greeting check')
})
fail the assertion on purpose
Compare to Mocha + Chai
import { it } from 'mocha'
import { expect } from 'chai'
it('fails objects on purpose', (t) => {
const person = { name: { first: 'Joe' } }
expect(person).to.deep.equal(
{ name: { first: 'Anna' } })
})fail the assertion on purpose
node:test + Chai
import { it } from 'node:test'
import { expect } from 'chai'
it('fails objects on purpose', () => {
const person = { name: { first: 'Joe' } }
expect(person).to.deep.equal({ name: { first: 'Anna' } })
})
Assertions and error messages are Node Test Runner's weakest link
node:test + spok
const test = require('node:test')
const spok = require('spok').default
test('complex object', (t) => {
const person = { name: { first: 'Joe' } }
spok(t, person, { name: { first: 'Anna' } })
})
spok
test('my object meets the specifications', (t) => {
spok(t, object, {
$topic : 'spok-example'
, one : spok.ge(1)
, two : 2
, three : spok.range(2, 4)
, four : spok.lt(5)
, helloWorld : spok.startsWith('hello')
, anyNum : spok.type('number')
, anotherNum : spok.number
, anArray : spok.array
, anotherArray : hasThreeElements
, anObject : spok.ne(undefined)
})
})
Test filtering
--test-name-pattern
it('works @sanity', () => {
assert.equal(1, 1)
})
it('works @sanity and @feature-a', () => {
assert.equal(2, 2)
})
it('low-priority-test', () => {
assert.equal(3, 3)
})# run tests with "@sanity" in the title
--test --test-name-pattern @sanity
- Usually unclear which tests were skipped
- No "pre-filtering" of test files
Test filtering
--test-name-pattern
it('works @sanity', () => {
assert.equal(1, 1)
})
it('works @sanity and @feature-a', () => {
assert.equal(2, 2)
})
it('low-priority-test', () => {
assert.equal(3, 3)
})# run tests with "@sanity" or "@feature"
--test --test-name-pattern @sanity \
--test-name-pattern @featureRun tests on CI
name: ci
on: push
jobs:
name: test
steps:
- uses: actions/checkout@v3
# https://github.com/actions/setup-node
- uses: actions/setup-node@v3
with:
node-version: 19.6.0
cache: 'npm'
- run: npm ci
- run: npm run spec
🏎️🏎️🏎️🏎️🏎️
Spies and Stubs
1. Spy / stub methods when you have an object reference
const person = {
name () {
return 'Joe'
}
}
stub(person, 'name').return('Anna')Sinon https://sinonjs.org/ 👑👑👑
Node Test Runner
1. Spy / stub methods when you have an object reference
import { it, mock } from 'node:test'
import assert from 'node:assert/strict'
it('returns name', () => {
const person = {
name() {
return 'Joe'
},
}
assert.equal(person.name(), 'Joe')
mock.method(person, 'name', () => 'Anna')
assert.equal(person.name(), 'Anna')
assert.equal(person.name.mock.calls.length, 1)
person.name.mock.restore()
assert.equal(person.name(), 'Joe')
})Node Test Runner
1. Spy / stub methods when you have an object reference
- Great start 👍
- spying and stubbing methods work really well. Reset and restore, information about each call.
- Lack of helpers to deal with async code
- Using plain "assert" with calls is pretty verbose
Spies and Stubs
2. Spy / stub module exports (CommonJS / ESM)
jest.mock('node-fetch')
import fetch, { Response } from 'node-fetch'
import { createUser } from './createUser'
test('createUser', async () => {
fetch.mockReturnValue(Promise.resolve(new Response('4')))
const userId = await createUser()
expect(fetch).toHaveBeenCalledTimes(1)
})export const add = (a, b) => {
console.log('adding %d to %d', a, b)
return a + b
}math.mjs
import { add } from './math.mjs'
export const calculate = (op, a, b) => {
return op === '+' ? add(a, b) : NaN
}calculator.mjs
import { it } from 'node:test'
import assert from 'node:assert/strict'
import { calculate } from './calculator.mjs'
it('adds two numbers', () => {
assert.equal(calculate('+', 2, 3), 5)
})regular test
import { it } from 'node:test'
import assert from 'node:assert/strict'
import esmock from 'esmock'
it('adds two numbers (mocks add)', async () => {
const { calculate } = await esmock('./calculator.mjs', {
'./math.mjs': {
add: () => 20,
},
})
assert.equal(calculate('+', 2, 3), 20)
})mock "add" from "math.mjs"
import { it, mock } from 'node:test'
import assert from 'node:assert/strict'
import esmock from 'esmock'
it('adds two numbers (confirm call)', async () => {
const add = mock.fn(() => 20)
const { calculate } = await esmock('./calculator.mjs', {
'./math.mjs': {
add,
},
})
assert.equal(calculate('+', 2, 3), 20)
assert.deepEqual(add.mock.calls[0].arguments, [2, 3])
})combine with function mocks
TypeScript
$ npm i -D typescript ts-node{
"scripts": {
"ts-test": "node --test --loader ts-node/esm test/**/*.ts"
}
}package.json
import { it } from 'node:test'
import assert from 'node:assert/strict'
type Person = {
name: string
}
it('subtest 1', () => {
console.log('testing the person')
const p: Person = {
name: 'Joe',
}
assert.deepEqual(p, { name: 'Joe' })
})
test/ts-tests.ts
TypeScript .ts file
JavaScript .mjs file
Misc features
it('works for 2 seconds', { timeout: 1000 }, async () => {
await delay(2000)
})Timeouts
Misc features
it('works for 2 seconds', { skip: 'Issue url here' },
async () => {
await delay(2000)
})Named skip
Misc features
it('works for 2 seconds', { only: true }, async () => {
await delay(2000)
})
it('fails', () => {
throw new Error('Nope')
})Only option
Misc features
it('works for 2 seconds', { only: true }, async () => {
await delay(2000)
})
it('fails', () => {
throw new Error('Nope')
})Only option
Misc features
it('succeeds', (done) => {
setTimeout(done, 1000)
})Done callback argument
Misc features
it('succeeds', (done) => {
setTimeout(() => {
done(new Error('A problem'))
}, 1000)
})Done callback argument
import test from 'node:test'
import assert from 'node:assert/strict'
async function fetchTestData() {
return new Promise((resolve) => {
setTimeout(() => {
resolve(['first', 'second', 'third'])
}, 1000)
})
}
test('generated', async (t) => {
const items = await fetchTestData()
for (const item of items) {
await t.test(`test ${item}`, (t) => {
assert.strictEqual(1, 1)
})
}
})Generate tests dynamically
import test from 'node:test'
import assert from 'node:assert/strict'
async function fetchTestData() {
return new Promise((resolve) => {
setTimeout(() => {
resolve(['first', 'second', 'third'])
}, 1000)
})
}
test('generated', async (t) => {
const items = await fetchTestData()
for (const item of items) {
await t.test(`test ${item}`, (t) => {
assert.strictEqual(1, 1)
})
}
})Generate tests dynamically
Code coverage
Built-in v19.7.0
import { it } from 'node:test'
import assert from 'node:assert/strict'
import { calculate } from './calculator.mjs'
it('adds two numbers', () => {
assert.strictEqual(calculate('+', 2, 3), 5, '2+3')
})export const add = (a, b) => {
console.log('adding %d to %d', a, b)
return a + b
}
export const sub = (a, b) => {
console.log('%d - %d', a, b)
return a - b
}math.mjs
--experimental-test-coverage
Other missing features
- Number of expected assertions
t.plan(2)
- Number of expected assertions
- Mocking timers
jest.useFakeTimers()
- Exit on first failure
deno test --fail-fast
- Expected failure
test.failing('found a bug', t => {
// Test will count as passed
t.fail();
});| Feature | Node.js TR | Mocha | Ava | Jest |
|---|---|---|---|---|
| Included with Node | ✅ | 🚫 | 🚫 | 🚫 |
| Watch mode | ✅ | ✅ | ✅ | ✅ |
| Reporters | via TAP | strong | via TAP | lots |
| Assertions | weak | via Chai ✅ | ✅ | ✅ |
| Snapshots | 🚫 | 🚫 | ✅ | ✅ |
| Hooks | ✅ | ✅ | ✅ | ✅ |
| grep support | ✅ | ✅ | ✅ | ✅ |
| spy and stub | ✅ | via Sinon ✅ | via Sinon ✅ | ✅✅ |
| parallel execution | ✅ | ✅ | ✅ | ✅ |
| code coverage | ✅ | via nyc | via c8 | ✅ |
| TS support | via ts-node | via ts-node | via ts-node | via ts-jest |
🎉
👍
😑
🎉
-
Try
node --teston smaller new projects -
Do not port existing tests yet
-
Re-evaluate in 6 months
Node Test Runner can quickly copy from existing test runners
Learn About Node's
New Built-in
Test Runner
Gleb Bahmutov
gleb.dev
👏 Thank You 👏