The New And Shiny ... Node.js Built-in Test Runner

Gleb Bahmutov

gleb.dev

https://slides.com/bahmutov/shiny-node-test

📝 Blog https://glebbahmutov.com/blog/trying-node-test-runner/

Fight climate crisis together

@bahmutov

rebellion.global 

www.wedonthavetime.org 

www.sierraclub.org 350.org 

www.greenpeace.org 

www.sunrisemovement.org

Speaker: Gleb Bahmutov PhD

@bahmutov

gleb.dev

github.com/bahmutov

glebbahmutov.com/blog

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

https://www.youtube.com/@gleb

Gleb Bahmutov

Sr Director of Engineering

www.mercari.com/careers

https://cypress.tips/courses

Let's Test!!!

(our Node code)

{
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  }
}

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

jest v29 7 years

mocha v10 11 years

npx available-versions mocha

tape v5 10 years

tap v16 11 years

ava v5 9 years

Node.js 13 years

May 2009

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

Node.js is a minimal core

https://github.com/bahmutov/gt

Write your own test runner!

JavaScript standard library is tiny

Node.js does not include test runner

https://github.com/bahmutov/gt

Write your own test runner!

Mocha + Chai + Sinonjs

Ava

Ava (Node)

Cypress (browser)

Node v18 test runner

https://nodejs.org/en/blog/announcements/v18-release-announce/

Ported to v16.x

I am using v19.6.0

Test examples shown in this presentation: https://github.com/bahmutov/node-tests

2022-04-19

2022-04-19

2023-03-08

Testing added 2 things

import test from 'node:test'

New built-in module "node:test"

$ node --test tests/*.mjs

New --test Node CLI flag

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('hello', () => {
  const message = 'Hello'
  assert.equal(message, 'Hello', 'checking the greeting')
})

Every test shown in this presentation: https://github.com/bahmutov/node-tests

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('hello', () => {
  const message = 'Hello'
  assert.equal(message, 'Hello', 'checking the greeting')
})

Comparing To Other Test Runners

  • installation
  • execution
  • syntax
  • assertions
  • hooks
  • reporters
  • speed
  • TypeScript support

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

"Installation"

$ nvm ls-remote
$ nvm install 19

Downloading and installing node v19.6.0...
Downloading https://nodejs.org/dist/v19.6.0/...
############################################## 100.0%
Computing checksum with shasum -a 256
Checksums matched!
Now using node v19.6.0 (npm v9.4.0)

🎁 Just use "nvm" https://github.com/nvm-sh/nvm

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

Test runner NPM packages installed
Mocha 78
Mocha + Chai + Sinon 445
Ava 198
Jest 429

Execution

$ node tests/demo.mjs

Tip: you can run tests in any JS file that has import "node:test" command

$ node --watch tests/demo.mjs
$ node --watch --test tests/*.mjs

Tip: test runner is works with built-in Node --watch mode

Tip: finds the tests by default

$ node --test
# runs tests in "test" subfolder

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

Test files naming convention

Start or end test file names with "test" to automatically recursively find them

tests/
  names/
    another.test.mjs
    my_test.mjs
    test-1.mjs
    utils.mjs
$ node --test tests/names
- tests/names/another.test.mjs
- tests/names/my_test.mjs
- tests/names/test-1.mjs

Test file convention

https://nodejs.org/api/test.html#test-runner-execution-model

Parallel

describe('parallel tests', { concurrency: true }, () => {
  it('subtest 1', async () => {
    console.log('subtest 1 start')
    await delay(5000)
    console.log('subtest 1 end')
  })

  it('subtest 2', async () => {
    console.log('subtest 2 start')
    await delay(5000)
    console.log('subtest 2 end')
  })
})

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

Parallel

describe('parallel tests', { concurrency: true }, () => {
  it('subtest 1', async () => {
    console.log('subtest 1 start')
    await delay(5000)
    console.log('subtest 1 end')
  })

  it('subtest 2', async () => {
    console.log('subtest 2 start')
    await delay(5000)
    console.log('subtest 2 end')
  })
})

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('top level test', async (t) => {
  await t.test('subtest 1', (t) => {
    assert.strictEqual(1, 1)
  })

  await t.test('subtest 2', (t) => {
    assert.strictEqual(2, 2)
  })
})

... look weird

Every test shown in this presentation: https://github.com/bahmutov/node-tests

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('top level test', async (t) => {
  await t.test('subtest 1', (t) => {
    assert.strictEqual(1, 1)
  })

  await t.test('subtest 2', (t) => {
    assert.strictEqual(2, 2)
  })
})

... look weird

Every test shown in this presentation: https://github.com/bahmutov/node-tests

The tests

import test from 'node:test'
import assert from 'node:assert/strict'

test('top level test', async (t) => {
  await test('subtest 1', (t) => {
    assert.strictEqual(1, 1)
  })

  await t.test('subtest 2', (t) => {
    assert.strictEqual(2, 2)
  })
})

... look weird

Every test shown in this presentation: https://github.com/bahmutov/node-tests

BDD syntax

import { describe, it } from 'node:test'
import assert from 'node:assert/strict'

describe('top level test', () => {
  it('subtest 1', () => {
    assert.strictEqual(1, 1)
  })

  it('subtest 2', () => {
    assert.strictEqual(2, 2)
  })
})

Every test shown in this presentation: https://github.com/bahmutov/node-tests

import { describe, it } from 'node:test'
import assert from 'node:assert/strict'

describe('top level test', () => {
  it('subtest 1', () => {
    assert.strictEqual(1, 1)
  })

  it('subtest 2', () => {
    assert.strictEqual(2, 2)
  })
})

1

2

Reporters

$ node --test --test-reporter tap # default
$ node --test --test-reporter spec
$ node --test --test-reporter dot
$ node --test --test-reporter dot \
  --test-reporter-destination stdout \
  --test-reporter spec \
  --test-reporter-destination out.txt

multiple reporters

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

Via TAP

Test Anything Protocol

https://github.com/sindresorhus/awesome-tap#reporters

Demo "node --test" vs "node --test | npx faucet"

# npm i -D faucet
$ node --test | npx faucet

"node --test" plus https://github.com/ljharb/faucet output

Test Statuses

it('works', () => {
  assert.equal(1, 1)
})

it('fails', () => {
  assert.equal(2, 5)
})

Test Statuses: skip/todo

it.todo('loads data')

// SKIP: <issue link>
it.skip('stopped working', () => {
  assert.equal(2, 5)
})

Test Statuses: cancelled

before(() => {
  console.log('before hook')
  throw new Error('Setup fails')
})

it('works', () => {
  assert.equal(1, 1)
})

it('works again', () => {
  assert.equal(1, 1)
})

🐞

Assertions

via built-in Node assert module

https://nodejs.dev/en/api/v19/assert/

import assert from 'node:assert/strict'
assert.ok(truthy, message)
assert.equal(value, expected, ...)
assert.deepEqual(...)
assert.match(value, regexp)
assert.throws(fn)
assert.rejects(asyncFn)

plus ".notX" assertions

quite sparse compared to https://www.chaijs.com/api/bdd/ 

and https://jestjs.io/docs/expect

Missing

expect({ /* object */ }).to.have.property(x)
expect({ /* object */ }).to.have.keys([x, y, z])
expect({ /* large object */ })
  .to.deep.include({ ... known properties })

Can bring

https://github.com/TritonDataCenter/node-assert-plus

https://github.com/chaijs/chai

// import assert from 'node:assert/strict'
import { assert } from 'chai'

Assertions

via built-in Node assert module

https://nodejs.dev/en/api/v19/assert/

import { describe, it } from 'node:test'
import assert from 'node:assert/strict'

describe('Assertions', () => {
  it('passes with primitives', () => {
    assert.equal('Hello', 'Helloz', 'greeting check')
  })
})

fail the assertion on purpose

Assertions

via built-in Node assert module

https://nodejs.dev/en/api/v19/assert/

it('fails objects on purpose', () => {
  const person = { name: { first: 'Joe' } }
  assert.deepEqual(person, 
    { name: { first: 'Anna' } }, 'people')
})

fail the assertion on purpose

Compare to Ava.js

with its "magic assert"

https://github.com/avajs/ava

import test from 'ava'

test('fails objects on purpose', (t) => {
  const person = { name: { first: 'Joe' } }
  t.deepEqual(person, 
    { name: { first: 'Anna' } }, 'people')
})

fail the assertion on purpose

Compare to Ava.js

with its "magic assert"

https://github.com/avajs/ava

import test from 'ava'

test('passes with primitives', (t) => {
  t.is('Hello', 'Helloz', 'greeting check')
})

fail the assertion on purpose

Compare to Mocha + Chai

import { it } from 'mocha'
import { expect } from 'chai'

it('fails objects on purpose', (t) => {
  const person = { name: { first: 'Joe' } }
  expect(person).to.deep.equal(
    { name: { first: 'Anna' } })
})

fail the assertion on purpose

node:test + Chai

import { it } from 'node:test'
import { expect } from 'chai'

it('fails objects on purpose', () => {
  const person = { name: { first: 'Joe' } }
  expect(person).to.deep.equal({ name: { first: 'Anna' } })
})

Assertions and error messages are Node Test Runner's weakest link

Test filtering

--test-name-pattern

it('works @sanity', () => {
  assert.equal(1, 1)
})

it('works @sanity and @feature-a', () => {
  assert.equal(2, 2)
})

it('low-priority-test', () => {
  assert.equal(3, 3)
})
# run tests with "@sanity" in the title
--test --test-name-pattern @sanity
  • Usually unclear which tests were skipped
  • No "pre-filtering" of test files

Run tests on CI

name: ci
on: push
jobs:
  name: test
  steps:
    - uses: actions/checkout@v3
    # https://github.com/actions/setup-node
    - uses: actions/setup-node@v3
      with:
        node-version: 19.6.0
        cache: 'npm'
    - run: npm ci
    - run: npm run spec

https://github.com/bahmutov/node-tests/actions

https://glebbahmutov.com/blog/trying-github-actions/

🏎️🏎️🏎️🏎️🏎️

Spy and stub

1. Spy / stub methods when you have an object reference

import { it, mock } from 'node:test'
import assert from 'node:assert/strict'

it('returns name', () => {
  const person = {
    name() {
      return 'Joe'
    },
  }
  assert.equal(person.name(), 'Joe')
  mock.method(person, 'name', () => 'Anna')
  assert.equal(person.name(), 'Anna')
  assert.equal(person.name.mock.calls.length, 1)
  person.name.mock.restore()
  assert.equal(person.name(), 'Joe')
})

1. Spy / stub methods when you have an object reference

  • Great start 👍
    • spying and stubbing methods work really well. Reset and restore, information about each call.
  • Lack of helpers to deal with async code
  • Using plain "assert" with calls is pretty verbose

Spy and stub

TypeScript

$ npm i -D typescript ts-node

https://github.com/TypeStrong/ts-node

{
  "scripts": {
    "ts-test": "node --test --loader ts-node/esm test/**/*.ts"
  }
}

package.json

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

import { it } from 'node:test'
import assert from 'node:assert/strict'

type Person = {
  name: string
}

it('subtest 1', () => {
  console.log('testing the person')
  const p: Person = {
    name: 'Joe',
  }
  assert.deepEqual(p, { name: 'Joe' })
})

test/ts-tests.ts

TypeScript .ts file

JavaScript .mjs file

Misc features

it('works for 2 seconds', { timeout: 1000 }, async () => {
  await delay(2000)
})

Timeouts

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

Misc features

it('works for 2 seconds', { skip: 'Issue url here' }, 
  async () => {
    await delay(2000)
  })

Named skip

Misc features

it('works for 2 seconds', { only: true }, async () => {
  await delay(2000)
})

it('fails', () => {
  throw new Error('Nope')
})

Only option

Misc features

it('works for 2 seconds', { only: true }, async () => {
  await delay(2000)
})

it('fails', () => {
  throw new Error('Nope')
})

Only option

Misc features

it('succeeds', (done) => {
  setTimeout(done, 1000)
})

Done callback argument

Misc features

it('succeeds', (done) => {
  setTimeout(() => {
    done(new Error('A problem'))
  }, 1000)
})

Done callback argument

import test from 'node:test'
import assert from 'node:assert/strict'

async function fetchTestData() {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve(['first', 'second', 'third'])
    }, 1000)
  })
}

test('generated', async (t) => {
  const items = await fetchTestData()

  for (const item of items) {
    await t.test(`test ${item}`, (t) => {
      assert.strictEqual(1, 1)
    })
  }
})

Generate tests dynamically

import test from 'node:test'
import assert from 'node:assert/strict'

async function fetchTestData() {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve(['first', 'second', 'third'])
    }, 1000)
  })
}

test('generated', async (t) => {
  const items = await fetchTestData()

  for (const item of items) {
    await t.test(`test ${item}`, (t) => {
      assert.strictEqual(1, 1)
    })
  }
})

Generate tests dynamically

Code coverage

Built-in v19.7.0

import { it } from 'node:test'
import assert from 'node:assert/strict'
import { calculate } from './calculator.mjs'

it('adds two numbers', () => {
  assert.strictEqual(calculate('+', 2, 3), 5, '2+3')
})
export const add = (a, b) => {
  console.log('adding %d to %d', a, b)
  return a + b
}

export const sub = (a, b) => {
  console.log('%d - %d', a, b)
  return a - b
}

math.mjs

--experimental-test-coverage

Only tested half of math.mjs

Other missing features

  • Number of expected assertions

t.plan(2)

  • Number of expected assertions
  • Mocking timers

jest.useFakeTimers()

  • Exit on first failure

deno test --fail-fast

  • Expected failure
test.failing('found a bug', t => {
  // Test will count as passed 
  t.fail();
});

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

Feature Node.js TR Mocha Ava Jest
Included with Node ✅ 🚫 🚫 🚫
Watch mode ✅ ✅ ✅ ✅
Reporters via TAP strong via TAP lots
Assertions weak via Chai ✅ ✅ ✅
Snapshots 🚫 🚫 ✅ ✅
Hooks ✅ ✅ ✅ ✅
grep support ✅ ✅ ✅ ✅
spy and stub ✅ via Sinon ✅ via Sinon ✅ ✅✅
parallel execution ✅ ✅ ✅ ✅
code coverage ✅ via nyc via c8 ✅
TS support via ts-node via ts-node via ts-node via ts-jest

🎉

👍

😑

🎉

gleb.dev                     https://slides.com/bahmutov/shiny-node-test

  • Try node --test on smaller new projects

  • Do not port existing tests yet

  • Re-evaluate in 6 months

Node Test Runner can quickly copy from existing test runners

The New And Shiny ...

Node.js Built-in Test Runner

Gleb Bahmutov

gleb.dev

👏 Thank You 👏

https://slides.com/bahmutov/shiny-node-test

💻 Code https://github.com/bahmutov/node-tests

📝 Blog https://glebbahmutov.com/blog/trying-node-test-runner/