Learn About Node's New Built-in

Test Runner

Gleb Bahmutov

gleb.dev

Feb 22, 2023 14:15

Westmount 6 ConFoo.CA

https://slides.com/bahmutov/node-test-runner

The climate crisis is:

here

urgent

caused by burning fossil fuels

survival is possible* but we need to act now

@bahmutov

rebellion.global

www.wedonthavetime.org

www.sierraclub.org 350.org

www.greenpeace.org

www.sunrisemovement.org

Speaker: Gleb Bahmutov PhD

@bahmutov

gleb.dev

github.com/bahmutov

glebbahmutov.com/blog

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

https://www.youtube.com/@gleb

Gleb Bahmutov

Sr Director of Engineering

www.mercari.com/careers

https://cypress.tips/courses

Let's Test!!!

(our Node code)

{
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  }
}

gleb.dev https://slides.com/bahmutov/node-test-runner

jest v29 7 years

mocha v10 11 years

npx available-versions mocha

tape v5 10 years

tap v16 11 years

ava v5 9 years

Node.js 13 years

May 2009

gleb.dev https://slides.com/bahmutov/node-test-runner

Node.js is a minimal core

https://github.com/bahmutov/gt

Write your own test runner!

JavaScript standard library is tiny

Node.js does not include test runner

https://github.com/bahmutov/gt

Write your own test runner!

gt.test("get N '='", function () {
  gt.ok(typeof getLines === "function", 
    "getLines is a function");
  gt.equal(getLines(0), "", "0 character");
  gt.equal(getLines(1), "=", "1 character");
});

collect all test hooks and callbacks
start running hooks and callbacks
report passes and failures
expand features:
- add assertions
- better async support
- spying and stubbing
- code coverage
- running tests in parallel

"gt: Gleb Test" framework test sample

gt.async('run a program', function () {
  gt.exec('node', ['index.js', 'arg1', 'arg2'], 0,
    function (stdout, stderr) {
      if (/error/.test(stdout)) {
        throw new Error('Errors in output ' + stdout);
      }
    });
});

Async via callbacks was the way 😄

Assertions:
- https://github.com/bahmutov/lazy-ass
- https://github.com/kensho-technologies/check-more-types
Snapshot testing
- https://github.com/bahmutov/snap-shot-it
Data-driven testing
- https://github.com/bahmutov/snap-shot-it
X
Y
Z

Mocha + Chai + Sinonjs

Ava

Ava (Node)

Cypress (browser)

Node v18 test runner

https://nodejs.org/en/blog/announcements/v18-release-announce/

Ported to v16.x

I am using v19.6.0

Test examples shown in this presentation: https://github.com/bahmutov/node-tests

2022-04-19

https://github.com/nodejs/node/issues/46642

Testing added 2 things

import test from 'node:test'

New built-in module "node:test"

$ node --test tests/*.mjs

New --test Node CLI flag

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('hello', () => {
  const message = 'Hello'
  assert.equal(message, 'Hello', 'checking the greeting')
})

Every test shown in this presentation: https://github.com/bahmutov/node-tests

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('hello', () => {
  const message = 'Hello'
  assert.equal(message, 'Hello', 'checking the greeting')
})

Comparing To Other Test Runners

installation
execution
syntax
assertions
hooks
reporters
speed
TypeScript support

gleb.dev https://slides.com/bahmutov/node-test-runner

"Installation"

$ nvm ls-remote
$ nvm install 19

Downloading and installing node v19.6.0...
Downloading https://nodejs.org/dist/v19.6.0/...
############################################## 100.0%
Computing checksum with shasum -a 256
Checksums matched!
Now using node v19.6.0 (npm v9.4.0)

🎁 Just use "nvm" https://github.com/nvm-sh/nvm

gleb.dev https://slides.com/bahmutov/node-test-runner

# .nvmrc file
19.6.0

$ nvm use
$ nvm use
Found '/node-tests/.nvmrc' with version <19.6.0>
Now using node v19.6.0 (npm v9.4.0)

"Installation"

🎁 Just use "nvm" https://github.com/nvm-sh/nvm

Test runner	NPM packages installed
Mocha	78
Mocha + Chai + Sinon	445
Ava	198
Jest	429

Execution

$ node tests/demo.mjs

Tip: you can run tests in any JS file that has import "node:test" command

$ node --watch tests/demo.mjs
$ node --watch --test tests/*.mjs

Tip: test runner is works with built-in Node --watch mode

Tip: finds the tests by default

$ node --test
# runs tests in "test" subfolder

gleb.dev https://slides.com/bahmutov/node-test-runner

Test files naming convention

Start or end test file names with "test" to automatically recursively find them

tests/
  names/
    another.test.mjs
    my_test.mjs
    test-1.mjs
    utils.mjs

$ node --test tests/names
- tests/names/another.test.mjs
- tests/names/my_test.mjs
- tests/names/test-1.mjs

Test file convention

https://nodejs.org/api/test.html#test-runner-execution-model

Parallel

describe('parallel tests', { concurrency: true }, () => {
  it('subtest 1', async () => {
    console.log('subtest 1 start')
    await delay(5000)
    console.log('subtest 1 end')
  })

  it('subtest 2', async () => {
    console.log('subtest 2 start')
    await delay(5000)
    console.log('subtest 2 end')
  })
})

gleb.dev https://slides.com/bahmutov/node-test-runner

Parallel

describe('parallel tests', { concurrency: true }, () => {
  it('subtest 1', async () => {
    console.log('subtest 1 start')
    await delay(5000)
    console.log('subtest 1 end')
  })

  it('subtest 2', async () => {
    console.log('subtest 2 start')
    await delay(5000)
    console.log('subtest 2 end')
  })
})

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('top level test', async (t) => {
  await t.test('subtest 1', (t) => {
    assert.strictEqual(1, 1)
  })

  await t.test('subtest 2', (t) => {
    assert.strictEqual(2, 2)
  })
})

... look weird

Every test shown in this presentation: https://github.com/bahmutov/node-tests

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('top level test', async (t) => {
  await t.test('subtest 1', (t) => {
    assert.strictEqual(1, 1)
  })

  await t.test('subtest 2', (t) => {
    assert.strictEqual(2, 2)
  })
})

... look weird

Every test shown in this presentation: https://github.com/bahmutov/node-tests

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('top level test', async (t) => {
  await test('subtest 1', (t) => {
    assert.strictEqual(1, 1)
  })

  await t.test('subtest 2', (t) => {
    assert.strictEqual(2, 2)
  })
})

... look weird

Every test shown in this presentation: https://github.com/bahmutov/node-tests

Wrong "test" function

Every test shown in this presentation: https://github.com/bahmutov/node-tests

BDD syntax

import { describe, it } from 'node:test'
import assert from 'node:assert/strict'

describe('top level test', () => {
  it('subtest 1', () => {
    assert.strictEqual(1, 1)
  })

  it('subtest 2', () => {
    assert.strictEqual(2, 2)
  })
})

Every test shown in this presentation: https://github.com/bahmutov/node-tests

import { describe, it } from 'node:test'
import assert from 'node:assert/strict'

describe('top level test', () => {
  it('subtest 1', () => {
    assert.strictEqual(1, 1)
  })

  it('subtest 2', () => {
    assert.strictEqual(2, 2)
  })
})

Reporters

$ node --test --test-reporter tap # default
$ node --test --test-reporter spec
$ node --test --test-reporter dot

$ node --test --test-reporter dot \
  --test-reporter-destination stdout \
  --test-reporter spec \
  --test-reporter-destination out.txt

multiple reporters

gleb.dev https://slides.com/bahmutov/node-test-runner

⚠️ Arguments order

🚫 wrong order

✅ test runner arguments come before test filenames

{
  "scripts": {
    "test": "node --test",
    "spec": "node --test --test-reporter spec"
  }
}

package.json

Via TAP

Test Anything Protocol

https://github.com/sindresorhus/awesome-tap#reporters

Demo "node --test" vs "node --test | npx faucet"

# npm i -D faucet
$ node --test | npx faucet

"node --test" plus https://github.com/ljharb/faucet output

Test Statuses

it('works', () => {
  assert.equal(1, 1)
})

it('fails', () => {
  assert.equal(2, 5)
})

Test Statuses: skip/todo

it.todo('loads data')

// SKIP: <issue link>
it.skip('stopped working', () => {
  assert.equal(2, 5)
})

Test Statuses: cancelled

before(() => {
  console.log('before hook')
  throw new Error('Setup fails')
})

it('works', () => {
  assert.equal(1, 1)
})

it('works again', () => {
  assert.equal(1, 1)
})

🐞

Speaking of "before" hook

describe('feature', () => {
  before(() => {
    console.log('before hook')
  })

  it('works', () => {
    assert.equal(1, 1)
  })

  it('works again', () => {
    assert.equal(1, 1)
  })
})

Speaking of "before" hook

before(() => {
  console.log('before hook')
})

it('works', () => {
  assert.equal(1, 1)
})

it('works again', () => {
  assert.equal(1, 1)
})

Assertions

via built-in Node assert module

https://nodejs.dev/en/api/v19/assert/

import assert from 'node:assert/strict'
assert.ok(truthy, message)
assert.equal(value, expected, ...)
assert.deepEqual(...)
assert.match(value, regexp)
assert.throws(fn)
assert.rejects(asyncFn)

plus ".notX" assertions

quite sparse compared to https://www.chaijs.com/api/bdd/

and https://jestjs.io/docs/expect

Missing

expect({ /* object */ }).to.have.property(x)
expect({ /* object */ }).to.have.keys([x, y, z])
expect({ /* large object */ })
  .to.deep.include({ ... known properties })

Can bring

https://github.com/TritonDataCenter/node-assert-plus

https://github.com/chaijs/chai

// import assert from 'node:assert/strict'
import { assert } from 'chai'

Assertions

via built-in Node assert module

https://nodejs.dev/en/api/v19/assert/

import { describe, it } from 'node:test'
import assert from 'node:assert/strict'

describe('Assertions', () => {
  it('passes with primitives', () => {
    assert.equal('Hello', 'Helloz', 'greeting check')
  })
})

fail the assertion on purpose

Assertions

via built-in Node assert module

https://nodejs.dev/en/api/v19/assert/

it('fails objects on purpose', () => {
  const person = { name: { first: 'Joe' } }
  assert.deepEqual(person, 
    { name: { first: 'Anna' } }, 'people')
})

fail the assertion on purpose

Compare to Ava.js

with its "magic assert"

https://github.com/avajs/ava

import test from 'ava'

test('fails objects on purpose', (t) => {
  const person = { name: { first: 'Joe' } }
  t.deepEqual(person, 
    { name: { first: 'Anna' } }, 'people')
})

fail the assertion on purpose

Compare to Ava.js

with its "magic assert"

https://github.com/avajs/ava

import test from 'ava'

test('passes with primitives', (t) => {
  t.is('Hello', 'Helloz', 'greeting check')
})

fail the assertion on purpose

Compare to Mocha + Chai

import { it } from 'mocha'
import { expect } from 'chai'

it('fails objects on purpose', (t) => {
  const person = { name: { first: 'Joe' } }
  expect(person).to.deep.equal(
    { name: { first: 'Anna' } })
})

fail the assertion on purpose

node:test + Chai

import { it } from 'node:test'
import { expect } from 'chai'

it('fails objects on purpose', () => {
  const person = { name: { first: 'Joe' } }
  expect(person).to.deep.equal({ name: { first: 'Anna' } })
})

Assertions and error messages are Node Test Runner's weakest link

Test filtering

--test-name-pattern

it('works @sanity', () => {
  assert.equal(1, 1)
})

it('works @sanity and @feature-a', () => {
  assert.equal(2, 2)
})

it('low-priority-test', () => {
  assert.equal(3, 3)
})

# run tests with "@sanity" in the title
--test --test-name-pattern @sanity

Usually unclear which tests were skipped
No "pre-filtering" of test files

Test filtering

--test-name-pattern

it('works @sanity', () => {
  assert.equal(1, 1)
})

it('works @sanity and @feature-a', () => {
  assert.equal(2, 2)
})

it('low-priority-test', () => {
  assert.equal(3, 3)
})

# run tests with "@sanity" or "@feature"
--test --test-name-pattern @sanity \
       --test-name-pattern @feature

Run tests on CI

name: ci
on: push
jobs:
  name: test
  steps:
    - uses: actions/checkout@v3
    # https://github.com/actions/setup-node
    - uses: actions/setup-node@v3
      with:
        node-version: 19.6.0
        cache: 'npm'
    - run: npm ci
    - run: npm run spec

https://github.com/bahmutov/node-tests/actions

https://glebbahmutov.com/blog/trying-github-actions/

🏎️🏎️🏎️🏎️🏎️

How to fill the report card

Spies and Stubs

1. Spy / stub methods when you have an object reference

const person = {
  name () {
    return 'Joe'
  }
}
stub(person, 'name').return('Anna')

Sinon https://sinonjs.org/ 👑👑👑

Jest https://jestjs.io/docs/mock-function-api 🥈

Node Test Runner

1. Spy / stub methods when you have an object reference

import { it, mock } from 'node:test'
import assert from 'node:assert/strict'

it('returns name', () => {
  const person = {
    name() {
      return 'Joe'
    },
  }
  assert.equal(person.name(), 'Joe')
  mock.method(person, 'name', () => 'Anna')
  assert.equal(person.name(), 'Anna')
  assert.equal(person.name.mock.calls.length, 1)
  person.name.mock.restore()
  assert.equal(person.name(), 'Joe')
})

Node Test Runner

1. Spy / stub methods when you have an object reference

Great start 👍
- spying and stubbing methods work really well. Reset and restore, information about each call.
Lack of helpers to deal with async code
Using plain "assert" with calls is pretty verbose

Spies and Stubs

2. Spy / stub module exports (CommonJS / ESM)

jest.mock('node-fetch')

import fetch, { Response } from 'node-fetch'
import { createUser } from './createUser'

test('createUser', async () => {
  fetch.mockReturnValue(Promise.resolve(new Response('4')))
  const userId = await createUser()
  expect(fetch).toHaveBeenCalledTimes(1)
})

Jest https://jestjs.io/docs/mock-functions 👑👑👑

export const add = (a, b) => {
  console.log('adding %d to %d', a, b)
  return a + b
}

math.mjs

import { add } from './math.mjs'

export const calculate = (op, a, b) => {
  return op === '+' ? add(a, b) : NaN
}

calculator.mjs

import { it } from 'node:test'
import assert from 'node:assert/strict'
import { calculate } from './calculator.mjs'

it('adds two numbers', () => {
  assert.equal(calculate('+', 2, 3), 5)
})

regular test

import { it } from 'node:test'
import assert from 'node:assert/strict'
import esmock from 'esmock'

it('adds two numbers (mocks add)', async () => {
  const { calculate } = await esmock('./calculator.mjs', {
    './math.mjs': {
      add: () => 20,
    },
  })
  assert.equal(calculate('+', 2, 3), 20)
})

mock "add" from "math.mjs"

https://github.com/iambumblehead/esmock

import { it, mock } from 'node:test'
import assert from 'node:assert/strict'
import esmock from 'esmock'

it('adds two numbers (confirm call)', async () => {
  const add = mock.fn(() => 20)
  const { calculate } = await esmock('./calculator.mjs', {
    './math.mjs': {
      add,
    },
  })
  assert.equal(calculate('+', 2, 3), 20)
  assert.deepEqual(add.mock.calls[0].arguments, [2, 3])
})

combine with function mocks

https://github.com/iambumblehead/esmock

TypeScript

$ npm i -D typescript ts-node

https://github.com/TypeStrong/ts-node

{
  "scripts": {
    "ts-test": "node --test --loader ts-node/esm test/**/*.ts"
  }
}

package.json

gleb.dev https://slides.com/bahmutov/node-test-runner

import { it } from 'node:test'
import assert from 'node:assert/strict'

type Person = {
  name: string
}

it('subtest 1', () => {
  console.log('testing the person')
  const p: Person = {
    name: 'Joe',
  }
  assert.deepEqual(p, { name: 'Joe' })
})

test/ts-tests.ts

TypeScript .ts file

JavaScript .mjs file

Misc features

it('works for 2 seconds', { timeout: 1000 }, async () => {
  await delay(2000)
})

Timeouts

gleb.dev https://slides.com/bahmutov/node-test-runner

Misc features

it('works for 2 seconds', { skip: 'Issue url here' }, 
  async () => {
    await delay(2000)
  })

Named skip

Misc features

it('works for 2 seconds', { only: true }, async () => {
  await delay(2000)
})

it('fails', () => {
  throw new Error('Nope')
})

Only option

Misc features

it('works for 2 seconds', { only: true }, async () => {
  await delay(2000)
})

it('fails', () => {
  throw new Error('Nope')
})

Only option

Misc features

it('succeeds', (done) => {
  setTimeout(done, 1000)
})

Done callback argument

Misc features

it('succeeds', (done) => {
  setTimeout(() => {
    done(new Error('A problem'))
  }, 1000)
})

Done callback argument

import test from 'node:test'
import assert from 'node:assert/strict'

async function fetchTestData() {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve(['first', 'second', 'third'])
    }, 1000)
  })
}

test('generated', async (t) => {
  const items = await fetchTestData()

  for (const item of items) {
    await t.test(`test ${item}`, (t) => {
      assert.strictEqual(1, 1)
    })
  }
})

Generate tests dynamically

import test from 'node:test'
import assert from 'node:assert/strict'

async function fetchTestData() {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve(['first', 'second', 'third'])
    }, 1000)
  })
}

test('generated', async (t) => {
  const items = await fetchTestData()

  for (const item of items) {
    await t.test(`test ${item}`, (t) => {
      assert.strictEqual(1, 1)
    })
  }
})

Generate tests dynamically

Code coverage

Built-in v19.7.0

import { it } from 'node:test'
import assert from 'node:assert/strict'
import { calculate } from './calculator.mjs'

it('adds two numbers', () => {
  assert.strictEqual(calculate('+', 2, 3), 5, '2+3')
})

export const add = (a, b) => {
  console.log('adding %d to %d', a, b)
  return a + b
}

export const sub = (a, b) => {
  console.log('%d - %d', a, b)
  return a - b
}

math.mjs

--experimental-test-coverage

Other missing features

Number of expected assertions

t.plan(2)

Number of expected assertions

Mocking timers

jest.useFakeTimers()

Exit on first failure

deno test --fail-fast

Expected failure

test.failing('found a bug', t => {
  // Test will count as passed 
  t.fail();
});

gleb.dev https://slides.com/bahmutov/node-test-runner

Feature	Node.js TR	Mocha	Ava	Jest
Included with Node	✅	🚫	🚫	🚫
Watch mode	✅	✅	✅	✅
Reporters	via TAP	strong	via TAP	lots
Assertions	weak	via Chai ✅	✅	✅
Snapshots	🚫	🚫	✅	✅
Hooks	✅	✅	✅	✅
grep support	✅	✅	✅	✅
spy and stub	✅	via Sinon ✅	via Sinon ✅	✅✅
parallel execution	✅	✅	✅	✅
code coverage	✅	via nyc	via c8	✅
TS support	via ts-node	via ts-node	via ts-node	via ts-jest

🎉

👍

😑

🎉

gleb.dev https://slides.com/bahmutov/node-test-runner

Try node --test on smaller new projects
Do not port existing tests yet

Re-evaluate in 6 months

Node Test Runner can quickly copy from existing test runners