Learn About Node's New Built-in

Test Runner

Gleb Bahmutov

gleb.dev

Feb 22, 2023 14:15

Westmount 6 ConFoo.CA

The climate crisis is:

here

urgent

caused by burning fossil fuels

survival is possible* but we need to act now

Speaker: Gleb Bahmutov PhD

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

Gleb Bahmutov

Sr Director of Engineering

Let's Test!!!

(our Node code)

{
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  }
}

jest v29 7 years

mocha v10 11 years

npx available-versions mocha

tape v5 10 years

tap v16 11 years

ava v5 9 years

Node.js 13 years

May 2009

Node.js is a minimal core

Write your own test runner!

JavaScript standard library is tiny

Node.js does not include test runner

Write your own test runner!

gt.test("get N '='", function () {
  gt.ok(typeof getLines === "function", 
    "getLines is a function");
  gt.equal(getLines(0), "", "0 character");
  gt.equal(getLines(1), "=", "1 character");
});
  • collect all test hooks and callbacks
  • start running hooks and callbacks
  • report passes and failures
  • expand features:
    • add assertions
    • better async support
    • spying and stubbing
    • code coverage
    • running tests in parallel

"gt: Gleb Test" framework test sample

gt.async('run a program', function () {
  gt.exec('node', ['index.js', 'arg1', 'arg2'], 0,
    function (stdout, stderr) {
      if (/error/.test(stdout)) {
        throw new Error('Errors in output ' + stdout);
      }
    });
});

Async via callbacks was the way 😄

Mocha + Chai + Sinonjs

Ava

Ava (Node)

Cypress (browser)

Node v18 test runner

Ported to v16.x

I am using v19.6.0

Test examples shown in this presentation: https://github.com/bahmutov/node-tests

2022-04-19

Testing added 2 things

import test from 'node:test'

New built-in module "node:test"

$ node --test tests/*.mjs

New --test Node CLI flag

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('hello', () => {
  const message = 'Hello'
  assert.equal(message, 'Hello', 'checking the greeting')
})

Every test shown in this presentation: https://github.com/bahmutov/node-tests

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('hello', () => {
  const message = 'Hello'
  assert.equal(message, 'Hello', 'checking the greeting')
})

Comparing To Other Test Runners

  • installation
  • execution
  • syntax
  • assertions
  • hooks
  • reporters
  • speed
  • TypeScript support

"Installation"

$ nvm ls-remote
$ nvm install 19

Downloading and installing node v19.6.0...
Downloading https://nodejs.org/dist/v19.6.0/...
############################################## 100.0%
Computing checksum with shasum -a 256
Checksums matched!
Now using node v19.6.0 (npm v9.4.0)

🎁 Just use "nvm" https://github.com/nvm-sh/nvm

# .nvmrc file
19.6.0

$ nvm use
$ nvm use
Found '/node-tests/.nvmrc' with version <19.6.0>
Now using node v19.6.0 (npm v9.4.0)

"Installation"

🎁 Just use "nvm" https://github.com/nvm-sh/nvm

Test runner NPM packages installed
Mocha 78
Mocha + Chai + Sinon 445
Ava 198
Jest 429

Execution

$ node tests/demo.mjs

Tip: you can run tests in any JS file that has import "node:test" command

$ node --watch tests/demo.mjs
$ node --watch --test tests/*.mjs

Tip: test runner is works with built-in Node --watch mode

Tip: finds the tests by default

$ node --test
# runs tests in "test" subfolder

Test files naming convention

Start or end test file names with "test" to automatically recursively find them

tests/
  names/
    another.test.mjs
    my_test.mjs
    test-1.mjs
    utils.mjs
$ node --test tests/names
- tests/names/another.test.mjs
- tests/names/my_test.mjs
- tests/names/test-1.mjs

Parallel

describe('parallel tests', { concurrency: true }, () => {
  it('subtest 1', async () => {
    console.log('subtest 1 start')
    await delay(5000)
    console.log('subtest 1 end')
  })

  it('subtest 2', async () => {
    console.log('subtest 2 start')
    await delay(5000)
    console.log('subtest 2 end')
  })
})

Parallel

describe('parallel tests', { concurrency: true }, () => {
  it('subtest 1', async () => {
    console.log('subtest 1 start')
    await delay(5000)
    console.log('subtest 1 end')
  })

  it('subtest 2', async () => {
    console.log('subtest 2 start')
    await delay(5000)
    console.log('subtest 2 end')
  })
})

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('top level test', async (t) => {
  await t.test('subtest 1', (t) => {
    assert.strictEqual(1, 1)
  })

  await t.test('subtest 2', (t) => {
    assert.strictEqual(2, 2)
  })
})

... look weird

Every test shown in this presentation: https://github.com/bahmutov/node-tests

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('top level test', async (t) => {
  await t.test('subtest 1', (t) => {
    assert.strictEqual(1, 1)
  })

  await t.test('subtest 2', (t) => {
    assert.strictEqual(2, 2)
  })
})

... look weird

Every test shown in this presentation: https://github.com/bahmutov/node-tests

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('top level test', async (t) => {
  await test('subtest 1', (t) => {
    assert.strictEqual(1, 1)
  })

  await t.test('subtest 2', (t) => {
    assert.strictEqual(2, 2)
  })
})

... look weird

Every test shown in this presentation: https://github.com/bahmutov/node-tests

Wrong "test" function

Every test shown in this presentation: https://github.com/bahmutov/node-tests

BDD syntax

import { describe, it } from 'node:test'
import assert from 'node:assert/strict'

describe('top level test', () => {
  it('subtest 1', () => {
    assert.strictEqual(1, 1)
  })

  it('subtest 2', () => {
    assert.strictEqual(2, 2)
  })
})

Every test shown in this presentation: https://github.com/bahmutov/node-tests

import { describe, it } from 'node:test'
import assert from 'node:assert/strict'

describe('top level test', () => {
  it('subtest 1', () => {
    assert.strictEqual(1, 1)
  })

  it('subtest 2', () => {
    assert.strictEqual(2, 2)
  })
})

1

2

Reporters

$ node --test --test-reporter tap # default
$ node --test --test-reporter spec
$ node --test --test-reporter dot
$ node --test --test-reporter dot \
  --test-reporter-destination stdout \
  --test-reporter spec \
  --test-reporter-destination out.txt

multiple reporters

⚠️ Arguments order

🚫 wrong order

✅ test runner arguments come before test filenames

{
  "scripts": {
    "test": "node --test",
    "spec": "node --test --test-reporter spec"
  }
}

package.json

Via TAP

Test Anything Protocol

Demo "node --test" vs "node --test | npx faucet"

# npm i -D faucet
$ node --test | npx faucet

"node --test" plus https://github.com/ljharb/faucet output

Test Statuses

it('works', () => {
  assert.equal(1, 1)
})

it('fails', () => {
  assert.equal(2, 5)
})

Test Statuses: skip/todo

it.todo('loads data')

// SKIP: <issue link>
it.skip('stopped working', () => {
  assert.equal(2, 5)
})

Test Statuses: cancelled

before(() => {
  console.log('before hook')
  throw new Error('Setup fails')
})

it('works', () => {
  assert.equal(1, 1)
})

it('works again', () => {
  assert.equal(1, 1)
})

🐞

Speaking of "before" hook

describe('feature', () => {
  before(() => {
    console.log('before hook')
  })

  it('works', () => {
    assert.equal(1, 1)
  })

  it('works again', () => {
    assert.equal(1, 1)
  })
})

Speaking of "before" hook

before(() => {
  console.log('before hook')
})

it('works', () => {
  assert.equal(1, 1)
})

it('works again', () => {
  assert.equal(1, 1)
})

Assertions

via built-in Node assert module

https://nodejs.dev/en/api/v19/assert/

import assert from 'node:assert/strict'
assert.ok(truthy, message)
assert.equal(value, expected, ...)
assert.deepEqual(...)
assert.match(value, regexp)
assert.throws(fn)
assert.rejects(asyncFn)

plus ".notX" assertions

Missing

expect({ /* object */ }).to.have.property(x)
expect({ /* object */ }).to.have.keys([x, y, z])
expect({ /* large object */ })
  .to.deep.include({ ... known properties })
// import assert from 'node:assert/strict'
import { assert } from 'chai'

Assertions

via built-in Node assert module

https://nodejs.dev/en/api/v19/assert/

import { describe, it } from 'node:test'
import assert from 'node:assert/strict'

describe('Assertions', () => {
  it('passes with primitives', () => {
    assert.equal('Hello', 'Helloz', 'greeting check')
  })
})

fail the assertion on purpose

Assertions

via built-in Node assert module

https://nodejs.dev/en/api/v19/assert/

it('fails objects on purpose', () => {
  const person = { name: { first: 'Joe' } }
  assert.deepEqual(person, 
    { name: { first: 'Anna' } }, 'people')
})

fail the assertion on purpose

Compare to Ava.js

with its "magic assert"

https://github.com/avajs/ava

import test from 'ava'

test('fails objects on purpose', (t) => {
  const person = { name: { first: 'Joe' } }
  t.deepEqual(person, 
    { name: { first: 'Anna' } }, 'people')
})

fail the assertion on purpose

Compare to Ava.js

with its "magic assert"

https://github.com/avajs/ava

import test from 'ava'

test('passes with primitives', (t) => {
  t.is('Hello', 'Helloz', 'greeting check')
})

fail the assertion on purpose

Compare to Mocha + Chai

import { it } from 'mocha'
import { expect } from 'chai'

it('fails objects on purpose', (t) => {
  const person = { name: { first: 'Joe' } }
  expect(person).to.deep.equal(
    { name: { first: 'Anna' } })
})

fail the assertion on purpose

node:test + Chai

import { it } from 'node:test'
import { expect } from 'chai'

it('fails objects on purpose', () => {
  const person = { name: { first: 'Joe' } }
  expect(person).to.deep.equal({ name: { first: 'Anna' } })
})

Assertions and error messages are Node Test Runner's weakest link

Test filtering

--test-name-pattern

it('works @sanity', () => {
  assert.equal(1, 1)
})

it('works @sanity and @feature-a', () => {
  assert.equal(2, 2)
})

it('low-priority-test', () => {
  assert.equal(3, 3)
})
# run tests with "@sanity" in the title
--test --test-name-pattern @sanity
  • Usually unclear which tests were skipped
  • No "pre-filtering" of test files

Test filtering

--test-name-pattern

it('works @sanity', () => {
  assert.equal(1, 1)
})

it('works @sanity and @feature-a', () => {
  assert.equal(2, 2)
})

it('low-priority-test', () => {
  assert.equal(3, 3)
})
# run tests with "@sanity" or "@feature"
--test --test-name-pattern @sanity \
       --test-name-pattern @feature

Run tests on CI

name: ci
on: push
jobs:
  name: test
  steps:
    - uses: actions/checkout@v3
    # https://github.com/actions/setup-node
    - uses: actions/setup-node@v3
      with:
        node-version: 19.6.0
        cache: 'npm'
    - run: npm ci
    - run: npm run spec

🏎️🏎️🏎️🏎️🏎️

How to fill the report card

Spies and Stubs

1. Spy / stub methods when you have an object reference

const person = {
  name () {
    return 'Joe'
  }
}
stub(person, 'name').return('Anna')

Sinon https://sinonjs.org/ 👑👑👑

Node Test Runner

1. Spy / stub methods when you have an object reference

import { it, mock } from 'node:test'
import assert from 'node:assert/strict'

it('returns name', () => {
  const person = {
    name() {
      return 'Joe'
    },
  }
  assert.equal(person.name(), 'Joe')
  mock.method(person, 'name', () => 'Anna')
  assert.equal(person.name(), 'Anna')
  assert.equal(person.name.mock.calls.length, 1)
  person.name.mock.restore()
  assert.equal(person.name(), 'Joe')
})

Node Test Runner

1. Spy / stub methods when you have an object reference

  • Great start 👍
    • spying and stubbing methods work really well. Reset and restore, information about each call.
  • Lack of helpers to deal with async code
  • Using plain "assert" with calls is pretty verbose

Spies and Stubs

2. Spy / stub module exports (CommonJS / ESM)

jest.mock('node-fetch')

import fetch, { Response } from 'node-fetch'
import { createUser } from './createUser'

test('createUser', async () => {
  fetch.mockReturnValue(Promise.resolve(new Response('4')))
  const userId = await createUser()
  expect(fetch).toHaveBeenCalledTimes(1)
})
export const add = (a, b) => {
  console.log('adding %d to %d', a, b)
  return a + b
}

math.mjs

import { add } from './math.mjs'

export const calculate = (op, a, b) => {
  return op === '+' ? add(a, b) : NaN
}

calculator.mjs

import { it } from 'node:test'
import assert from 'node:assert/strict'
import { calculate } from './calculator.mjs'

it('adds two numbers', () => {
  assert.equal(calculate('+', 2, 3), 5)
})

regular test

import { it } from 'node:test'
import assert from 'node:assert/strict'
import esmock from 'esmock'

it('adds two numbers (mocks add)', async () => {
  const { calculate } = await esmock('./calculator.mjs', {
    './math.mjs': {
      add: () => 20,
    },
  })
  assert.equal(calculate('+', 2, 3), 20)
})

mock "add" from "math.mjs"

import { it, mock } from 'node:test'
import assert from 'node:assert/strict'
import esmock from 'esmock'

it('adds two numbers (confirm call)', async () => {
  const add = mock.fn(() => 20)
  const { calculate } = await esmock('./calculator.mjs', {
    './math.mjs': {
      add,
    },
  })
  assert.equal(calculate('+', 2, 3), 20)
  assert.deepEqual(add.mock.calls[0].arguments, [2, 3])
})

combine with function mocks

TypeScript

$ npm i -D typescript ts-node
{
  "scripts": {
    "ts-test": "node --test --loader ts-node/esm test/**/*.ts"
  }
}

package.json

import { it } from 'node:test'
import assert from 'node:assert/strict'

type Person = {
  name: string
}

it('subtest 1', () => {
  console.log('testing the person')
  const p: Person = {
    name: 'Joe',
  }
  assert.deepEqual(p, { name: 'Joe' })
})

test/ts-tests.ts

TypeScript .ts file

JavaScript .mjs file

Misc features

it('works for 2 seconds', { timeout: 1000 }, async () => {
  await delay(2000)
})

Timeouts

Misc features

it('works for 2 seconds', { skip: 'Issue url here' }, 
  async () => {
    await delay(2000)
  })

Named skip

Misc features

it('works for 2 seconds', { only: true }, async () => {
  await delay(2000)
})

it('fails', () => {
  throw new Error('Nope')
})

Only option

Misc features

it('works for 2 seconds', { only: true }, async () => {
  await delay(2000)
})

it('fails', () => {
  throw new Error('Nope')
})

Only option

Misc features

it('succeeds', (done) => {
  setTimeout(done, 1000)
})

Done callback argument

Misc features

it('succeeds', (done) => {
  setTimeout(() => {
    done(new Error('A problem'))
  }, 1000)
})

Done callback argument

import test from 'node:test'
import assert from 'node:assert/strict'

async function fetchTestData() {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve(['first', 'second', 'third'])
    }, 1000)
  })
}

test('generated', async (t) => {
  const items = await fetchTestData()

  for (const item of items) {
    await t.test(`test ${item}`, (t) => {
      assert.strictEqual(1, 1)
    })
  }
})

Generate tests dynamically

import test from 'node:test'
import assert from 'node:assert/strict'

async function fetchTestData() {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve(['first', 'second', 'third'])
    }, 1000)
  })
}

test('generated', async (t) => {
  const items = await fetchTestData()

  for (const item of items) {
    await t.test(`test ${item}`, (t) => {
      assert.strictEqual(1, 1)
    })
  }
})

Generate tests dynamically

Code coverage

Built-in v19.7.0

import { it } from 'node:test'
import assert from 'node:assert/strict'
import { calculate } from './calculator.mjs'

it('adds two numbers', () => {
  assert.strictEqual(calculate('+', 2, 3), 5, '2+3')
})
export const add = (a, b) => {
  console.log('adding %d to %d', a, b)
  return a + b
}

export const sub = (a, b) => {
  console.log('%d - %d', a, b)
  return a - b
}

math.mjs

--experimental-test-coverage

Other missing features

  • Number of expected assertions

t.plan(2)

  • Number of expected assertions
  • Mocking timers

jest.useFakeTimers()

  • Exit on first failure

deno test --fail-fast

  • Expected failure
test.failing('found a bug', t => {
  // Test will count as passed 
  t.fail();
});
Feature Node.js TR Mocha Ava Jest
Included with Node 🚫 🚫 🚫
Watch mode
Reporters via TAP strong via TAP lots
Assertions weak via Chai ✅
Snapshots 🚫 🚫
Hooks
grep support
spy and stub via Sinon ✅ via Sinon ✅ ✅✅
parallel execution
code coverage via nyc via c8
TS support via ts-node via ts-node via ts-node via ts-jest

🎉

👍

😑

🎉

  • Try node --test on smaller new projects

  • Do not port existing tests yet

  • Re-evaluate in 6 months

Node Test Runner can quickly copy from existing test runners

Learn About Node's New Built-in

Test Runner

Gleb Bahmutov

gleb.dev

👏 Thank You 👏

Thank you Colin Ihrig @cjihrig for reviewing the slides and working on Node Test Runner

Learn About Node's New Built-in Test Runner

By Gleb Bahmutov

Learn About Node's New Built-in Test Runner

Jest, Mocha, Ava, and other Node test runners long dominated the unit testing. Now Node.js has released its own experimental unit testing framework. Learn how it compares to other test runners, which interesting features it provides, and if you should rewrite your tests to switch to the built-in "test" module. 45 minutes, presented at ConFoo in Montreal, Canada

  • 1,113