The New And Shiny ... Node.js Built-in Test Runner

Gleb Bahmutov

gleb.dev

https://slides.com/bahmutov/shiny-node-test

📝 Blog https://glebbahmutov.com/blog/trying-node-test-runner/

Fight climate crisis together

@bahmutov

rebellion.global 

www.wedonthavetime.org 

www.sierraclub.org 350.org 

www.greenpeace.org 

www.sunrisemovement.org

Speaker: Gleb Bahmutov PhD

@bahmutov

gleb.dev

github.com/bahmutov

glebbahmutov.com/blog

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

https://www.youtube.com/@gleb

Gleb Bahmutov

Sr Director of Engineering

www.mercari.com/careers

https://cypress.tips/courses

Let's Test!!!

(our Node code)

{
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  }
}

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

jest v29 7 years

mocha v10 11 years

npx available-versions mocha

tape v5 10 years

tap v16 11 years

ava v5 9 years

Node.js 13 years

May 2009

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

Node.js is a minimal core

https://github.com/bahmutov/gt

Write your own test runner!

JavaScript standard library is tiny

Node.js does not include test runner

https://github.com/bahmutov/gt

Write your own test runner!

Mocha + Chai + Sinonjs

Ava

Ava (Node)

Cypress (browser)

Node v18 test runner

https://nodejs.org/en/blog/announcements/v18-release-announce/

Ported to v16.x

I am using v19.6.0

Test examples shown in this presentation: https://github.com/bahmutov/node-tests

2022-04-19

2022-04-19

2023-03-08

Testing added 2 things

import test from 'node:test'

New built-in module "node:test"

$ node --test tests/*.mjs

New --test Node CLI flag

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('hello', () => {
  const message = 'Hello'
  assert.equal(message, 'Hello', 'checking the greeting')
})

Every test shown in this presentation: https://github.com/bahmutov/node-tests

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('hello', () => {
  const message = 'Hello'
  assert.equal(message, 'Hello', 'checking the greeting')
})

Comparing To Other Test Runners

  • installation
  • execution
  • syntax
  • assertions
  • hooks
  • reporters
  • speed
  • TypeScript support

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

"Installation"

$ nvm ls-remote
$ nvm install 19

Downloading and installing node v19.6.0...
Downloading https://nodejs.org/dist/v19.6.0/...
############################################## 100.0%
Computing checksum with shasum -a 256
Checksums matched!
Now using node v19.6.0 (npm v9.4.0)

🎁 Just use "nvm" https://github.com/nvm-sh/nvm

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

Test runner NPM packages installed
Mocha 78
Mocha + Chai + Sinon 445
Ava 198
Jest 429

Execution

$ node tests/demo.mjs

Tip: you can run tests in any JS file that has import "node:test" command

$ node --watch tests/demo.mjs
$ node --watch --test tests/*.mjs

Tip: test runner is works with built-in Node --watch mode

Tip: finds the tests by default

$ node --test
# runs tests in "test" subfolder

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

Test files naming convention

Start or end test file names with "test" to automatically recursively find them

tests/
  names/
    another.test.mjs
    my_test.mjs
    test-1.mjs
    utils.mjs
$ node --test tests/names
- tests/names/another.test.mjs
- tests/names/my_test.mjs
- tests/names/test-1.mjs

Test file convention

https://nodejs.org/api/test.html#test-runner-execution-model

Parallel

describe('parallel tests', { concurrency: true }, () => {
  it('subtest 1', async () => {
    console.log('subtest 1 start')
    await delay(5000)
    console.log('subtest 1 end')
  })

  it('subtest 2', async () => {
    console.log('subtest 2 start')
    await delay(5000)
    console.log('subtest 2 end')
  })
})

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

Parallel

describe('parallel tests', { concurrency: true }, () => {
  it('subtest 1', async () => {
    console.log('subtest 1 start')
    await delay(5000)
    console.log('subtest 1 end')
  })

  it('subtest 2', async () => {
    console.log('subtest 2 start')
    await delay(5000)
    console.log('subtest 2 end')
  })
})

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('top level test', async (t) => {
  await t.test('subtest 1', (t) => {
    assert.strictEqual(1, 1)
  })

  await t.test('subtest 2', (t) => {
    assert.strictEqual(2, 2)
  })
})

... look weird

Every test shown in this presentation: https://github.com/bahmutov/node-tests

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('top level test', async (t) => {
  await t.test('subtest 1', (t) => {
    assert.strictEqual(1, 1)
  })

  await t.test('subtest 2', (t) => {
    assert.strictEqual(2, 2)
  })
})

... look weird

Every test shown in this presentation: https://github.com/bahmutov/node-tests

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('top level test', async (t) => {
  await test('subtest 1', (t) => {
    assert.strictEqual(1, 1)
  })

  await t.test('subtest 2', (t) => {
    assert.strictEqual(2, 2)
  })
})

... look weird

Every test shown in this presentation: https://github.com/bahmutov/node-tests

BDD syntax

import { describe, it } from 'node:test'
import assert from 'node:assert/strict'

describe('top level test', () => {
  it('subtest 1', () => {
    assert.strictEqual(1, 1)
  })

  it('subtest 2', () => {
    assert.strictEqual(2, 2)
  })
})

Every test shown in this presentation: https://github.com/bahmutov/node-tests

import { describe, it } from 'node:test'
import assert from 'node:assert/strict'

describe('top level test', () => {
  it('subtest 1', () => {
    assert.strictEqual(1, 1)
  })

  it('subtest 2', () => {
    assert.strictEqual(2, 2)
  })
})

1

2

Reporters

$ node --test --test-reporter tap # default
$ node --test --test-reporter spec
$ node --test --test-reporter dot
$ node --test --test-reporter dot \
  --test-reporter-destination stdout \
  --test-reporter spec \
  --test-reporter-destination out.txt

multiple reporters

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

Via TAP

Test Anything Protocol

https://github.com/sindresorhus/awesome-tap#reporters

Demo "node --test" vs "node --test | npx faucet"

# npm i -D faucet
$ node --test | npx faucet

"node --test" plus https://github.com/ljharb/faucet output

Test Statuses

it('works', () => {
  assert.equal(1, 1)
})

it('fails', () => {
  assert.equal(2, 5)
})

Test Statuses: skip/todo

it.todo('loads data')

// SKIP: <issue link>
it.skip('stopped working', () => {
  assert.equal(2, 5)
})

Test Statuses: cancelled

before(() => {
  console.log('before hook')
  throw new Error('Setup fails')
})

it('works', () => {
  assert.equal(1, 1)
})

it('works again', () => {
  assert.equal(1, 1)
})

🐞

Assertions

via built-in Node assert module

https://nodejs.dev/en/api/v19/assert/

import assert from 'node:assert/strict'
assert.ok(truthy, message)
assert.equal(value, expected, ...)
assert.deepEqual(...)
assert.match(value, regexp)
assert.throws(fn)
assert.rejects(asyncFn)

plus ".notX" assertions

quite sparse compared to https://www.chaijs.com/api/bdd/ 

and https://jestjs.io/docs/expect

Missing

expect({ /* object */ }).to.have.property(x)
expect({ /* object */ }).to.have.keys([x, y, z])
expect({ /* large object */ })
  .to.deep.include({ ... known properties })

Can bring

https://github.com/TritonDataCenter/node-assert-plus

https://github.com/chaijs/chai

// import assert from 'node:assert/strict'
import { assert } from 'chai'

Assertions

via built-in Node assert module

https://nodejs.dev/en/api/v19/assert/

import { describe, it } from 'node:test'
import assert from 'node:assert/strict'

describe('Assertions', () => {
  it('passes with primitives', () => {
    assert.equal('Hello', 'Helloz', 'greeting check')
  })
})

fail the assertion on purpose

Assertions

via built-in Node assert module

https://nodejs.dev/en/api/v19/assert/

it('fails objects on purpose', () => {
  const person = { name: { first: 'Joe' } }
  assert.deepEqual(person, 
    { name: { first: 'Anna' } }, 'people')
})

fail the assertion on purpose

Compare to Ava.js

with its "magic assert"

https://github.com/avajs/ava

import test from 'ava'

test('fails objects on purpose', (t) => {
  const person = { name: { first: 'Joe' } }
  t.deepEqual(person, 
    { name: { first: 'Anna' } }, 'people')
})

fail the assertion on purpose

Compare to Ava.js

with its "magic assert"

https://github.com/avajs/ava

import test from 'ava'

test('passes with primitives', (t) => {
  t.is('Hello', 'Helloz', 'greeting check')
})

fail the assertion on purpose

Compare to Mocha + Chai

import { it } from 'mocha'
import { expect } from 'chai'

it('fails objects on purpose', (t) => {
  const person = { name: { first: 'Joe' } }
  expect(person).to.deep.equal(
    { name: { first: 'Anna' } })
})

fail the assertion on purpose

node:test + Chai

import { it } from 'node:test'
import { expect } from 'chai'

it('fails objects on purpose', () => {
  const person = { name: { first: 'Joe' } }
  expect(person).to.deep.equal({ name: { first: 'Anna' } })
})

Assertions and error messages are Node Test Runner's weakest link

Test filtering

--test-name-pattern

it('works @sanity', () => {
  assert.equal(1, 1)
})

it('works @sanity and @feature-a', () => {
  assert.equal(2, 2)
})

it('low-priority-test', () => {
  assert.equal(3, 3)
})
# run tests with "@sanity" in the title
--test --test-name-pattern @sanity
  • Usually unclear which tests were skipped
  • No "pre-filtering" of test files

Run tests on CI

name: ci
on: push
jobs:
  name: test
  steps:
    - uses: actions/checkout@v3
    # https://github.com/actions/setup-node
    - uses: actions/setup-node@v3
      with:
        node-version: 19.6.0
        cache: 'npm'
    - run: npm ci
    - run: npm run spec

https://github.com/bahmutov/node-tests/actions

https://glebbahmutov.com/blog/trying-github-actions/

🏎️🏎️🏎️🏎️🏎️

Spy and stub

1. Spy / stub methods when you have an object reference

import { it, mock } from 'node:test'
import assert from 'node:assert/strict'

it('returns name', () => {
  const person = {
    name() {
      return 'Joe'
    },
  }
  assert.equal(person.name(), 'Joe')
  mock.method(person, 'name', () => 'Anna')
  assert.equal(person.name(), 'Anna')
  assert.equal(person.name.mock.calls.length, 1)
  person.name.mock.restore()
  assert.equal(person.name(), 'Joe')
})

1. Spy / stub methods when you have an object reference

  • Great start 👍
    • spying and stubbing methods work really well. Reset and restore, information about each call.
  • Lack of helpers to deal with async code
  • Using plain "assert" with calls is pretty verbose

Spy and stub

TypeScript

$ npm i -D typescript ts-node

https://github.com/TypeStrong/ts-node

{
  "scripts": {
    "ts-test": "node --test --loader ts-node/esm test/**/*.ts"
  }
}

package.json

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

import { it } from 'node:test'
import assert from 'node:assert/strict'

type Person = {
  name: string
}

it('subtest 1', () => {
  console.log('testing the person')
  const p: Person = {
    name: 'Joe',
  }
  assert.deepEqual(p, { name: 'Joe' })
})

test/ts-tests.ts

TypeScript .ts file

JavaScript .mjs file

Misc features

it('works for 2 seconds', { timeout: 1000 }, async () => {
  await delay(2000)
})

Timeouts

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

Misc features

it('works for 2 seconds', { skip: 'Issue url here' }, 
  async () => {
    await delay(2000)
  })

Named skip

Misc features

it('works for 2 seconds', { only: true }, async () => {
  await delay(2000)
})

it('fails', () => {
  throw new Error('Nope')
})

Only option

Misc features

it('works for 2 seconds', { only: true }, async () => {
  await delay(2000)
})

it('fails', () => {
  throw new Error('Nope')
})

Only option

Misc features

it('succeeds', (done) => {
  setTimeout(done, 1000)
})

Done callback argument

Misc features

it('succeeds', (done) => {
  setTimeout(() => {
    done(new Error('A problem'))
  }, 1000)
})

Done callback argument

import test from 'node:test'
import assert from 'node:assert/strict'

async function fetchTestData() {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve(['first', 'second', 'third'])
    }, 1000)
  })
}

test('generated', async (t) => {
  const items = await fetchTestData()

  for (const item of items) {
    await t.test(`test ${item}`, (t) => {
      assert.strictEqual(1, 1)
    })
  }
})

Generate tests dynamically

import test from 'node:test'
import assert from 'node:assert/strict'

async function fetchTestData() {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve(['first', 'second', 'third'])
    }, 1000)
  })
}

test('generated', async (t) => {
  const items = await fetchTestData()

  for (const item of items) {
    await t.test(`test ${item}`, (t) => {
      assert.strictEqual(1, 1)
    })
  }
})

Generate tests dynamically

Code coverage

Built-in v19.7.0

import { it } from 'node:test'
import assert from 'node:assert/strict'
import { calculate } from './calculator.mjs'

it('adds two numbers', () => {
  assert.strictEqual(calculate('+', 2, 3), 5, '2+3')
})
export const add = (a, b) => {
  console.log('adding %d to %d', a, b)
  return a + b
}

export const sub = (a, b) => {
  console.log('%d - %d', a, b)
  return a - b
}

math.mjs

--experimental-test-coverage

Only tested half of math.mjs

Other missing features

  • Number of expected assertions

t.plan(2)

  • Number of expected assertions
  • Mocking timers

jest.useFakeTimers()

  • Exit on first failure

deno test --fail-fast

  • Expected failure
test.failing('found a bug', t => {
  // Test will count as passed 
  t.fail();
});

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

Feature Node.js TR Mocha Ava Jest
Included with Node ✅ 🚫 🚫 🚫
Watch mode ✅ ✅ ✅ ✅
Reporters via TAP strong via TAP lots
Assertions weak via Chai ✅ ✅ ✅
Snapshots 🚫 🚫 ✅ ✅
Hooks ✅ ✅ ✅ ✅
grep support ✅ ✅ ✅ ✅
spy and stub ✅ via Sinon ✅ via Sinon ✅ ✅✅
parallel execution ✅ ✅ ✅ ✅
code coverage ✅ via nyc via c8 ✅
TS support via ts-node via ts-node via ts-node via ts-jest

🎉

👍

😑

🎉

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

  • Try node --test on smaller new projects

  • Do not port existing tests yet

  • Re-evaluate in 6 months

Node Test Runner can quickly copy from existing test runners

The New And Shiny ...

Node.js Built-in Test Runner

Gleb Bahmutov

gleb.dev

👏 Thank You 👏

https://slides.com/bahmutov/shiny-node-test

💻 Code https://github.com/bahmutov/node-tests

📝 Blog https://glebbahmutov.com/blog/trying-node-test-runner/

The New And Shiny ... Node.js Built-in Test Runner

By Gleb Bahmutov

The New And Shiny ... Node.js Built-in Test Runner

Jest, Mocha, Ava, and other Node test runners long dominated the unit testing. Now Node.js has released its own experimental unit testing framework. Learn how it compares to other test runners, which interesting features it provides, and if you should rewrite your tests to switch to the built-in "test" module.

  • 1,445

More from Gleb Bahmutov