Node 24 test runner module does not wait for subtests to finish

[mrazauskas]

Version

v24.0.0

Platform

Darwin MacBookPro 24.4.0 Darwin Kernel Version 24.4.0

Subsystem

No response

What steps will reproduce the bug?

Node 24 introduced breaking change in the test runner via #56664

Author of the PR has mentioned the following in #56664 (comment)

The only scenario where this is breaking:

test('foo', async (t) => {
  await t.test('subtest 1');
  // It is no longer guaranteed that the subtest on the previous line has
  // finished by the time this line executes. But the test runner still ensures
  // the order that subtest 1 and subtest 2 are executed in.
  await t.test('subtest 2');
});

In my case a for..of loop creates subtests dynamically and the next test must started only after previous is finished. These tests are triggering event emitter with different data and to collect output the events must appear in predictable sequence.

The example above is using await, but it has no effect. The execution move to the second subtests without waiting for the first one to finish.

By the way, the releases notes claim that this change “makes writing tests more intuitive and reduces common errors related to unhandled promises.” Hm.. but this behaviour looks to me exactly like an unhandled promise and my intuition is rather lost: how to ensure order of execution?

---

Is there a chance to revert the change?

Or perhaps it makes sense adding an option that brings back the old behaviour? I use run() in the project that got broken. So passing one more option would be fine.

How often does it reproduce? Is there a required condition?

Always.

What is the expected behavior? Why is that the expected behavior?

The test runner should only execute one test or subtest at a time within the same file. Or provide a mechanism to ensure such behaviour.

What do you see instead?

All subtest created by the for..of loop run at the same time.

Additional information

Thank you for working on Node.js.

[romainmenke]

This was also a breaking change for us.
The test runner for PostCSS plugins we maintain here is now broken.

I also don't see a good way forwards where we can still cover node 18 -> node 24.

I would also prefer it if this change was reverted as the only benefit it seems to bring is some developer convenience. I don't think it actually added any capabilities, right?

[romainmenke]

A simplified example of one issues we now face:

• first sub test is async
• each test cases has an after callback that should be run after the first sub test

import test from 'node:test';
import assert from 'node:assert';

await test('outer', async (t1) => {
	let a = 1;
	let b = 1;

	await t1.test('inner 1', async () => {
		await new Promise((resolve) => setTimeout(resolve, 50));
		// Run initial assert provided by the test case
		// This passes in Node 22 and earlier but fails in node 24 !!
		assert.equal(a, b);
	});

	// run "after" callback provided by the test case
	await new Promise((resolve) => {
		b = 2;
		resolve();
	});

	await t1.test('inner 2', async () => {
		// Run some extra asserts
		assert.ok(b);
	});
});

I think we can (ab)use the after hooks on context and some bogus sub tests.
This seems to work.

nvm, that didn't work at all :D

[mrazauskas]

I just read through the issue (#56664) closed by the PR which created this breaking change in node:test.

The most of discussion there is about one lint rule of one lint tool. The lint tool is actually looking at types of @types/node. But the change was made in node:test. Hard to explain.

If it ain’t broke, don’t fix it.

[cjihrig]

Anyone is free to open a revert and see how it is received.

[mrazauskas]

Just found the problem. I had to move setup and teardown to t.before() and t.after(). Instead of having line of code before and after the for..of loop. If I remember it right, Mocha had the same requirement.

Here is the code which helped me to understand the problem:

import assert from "node:assert";
import test from "node:test";
import { setTimeout } from "node:timers/promises";

let res = 0;

await test("top level test", async (t) => {
  t.beforeEach(() => {
    console.log("before");

    res = 0;
  });

  t.afterEach(() => {
    console.log("after");

    res = 0;
  });

  console.log("setup");

  for (const x of [2500, 500]) {
    await t.test(`subtest ${x}`, async () => {
      await setTimeout(x);

      res = x;

      console.log(`done ${x}`);

      assert.strictEqual(res, x);
    });
  }

  console.log("teardown");
});

Using Node 24 it logs:

setup
before
teardown   <- problem is here
done 2500
after
before
done 500
after

Using Node 22 it logs:

setup
before
done 2500
after
before
done 500
after
teardown

@cjihrig Really sorry about the noise. Looks like there is only one test executed at the time. My assumptions were wrong and the problem was elsewhere. How do you think, perhaps is it worth adding a note to documentation?

[romainmenke]

@mrazauskas Can you re-open this issue?

The before/after hooks only work as you describe when there is only a single sub test.
If there are multiple it no longer works.

In our case we have:

• test
  • sub test
  • async workload in between sub tests
  • sub test
  • async workload in between sub tests
  • sub test

Such work can no longer be orchestrated in Node 24.

The only way to make things work is to move everything async outside of sub tests.

[pmarchini]

Anyone is free to open a revert and see how it is received.

Considering all the discussions I've seen since the release, I think we should provide a way to optionally roll back to the previous behavior.

I'll take a look over the next few days.

[mrazauskas]

The before/after hooks only work as you describe when there is only a single sub test. If there are multiple it no longer works.

Do you mean having t.after() inside a subtest does not work? Looking at your example:

await test('outer', async (t1) => {
	let a = 1;
	let b = 1;

	await t1.test('inner 1', async (t) => {
		t.after(); // <- here it does not work?

[romainmenke]

By carefully adding bogus sub tests and using before/after it does seem possible to cover most cases.
I can't seem to reproduce the exact failure case I was facing before.

But it is a pretty bad DX.

Having to use before/after and bogus sub tests to orchestrate async workloads is basically callback hell. await was added to get away from callback hell.

[LiviaMedeiros]

Reopening because this is issue is clearly about something that stopped working in stable API without prior deprecation.
Dropping another example of broken code due to this issue: https://github.com/LiviaMedeiros/poteto/blob/f8771297b05872bf887301ec2e93ed5230afb057/test/sequence.test.mjs#L246-L252

// ...
// bunch of sequential reads-writes-appends-stats-etc.

  // fs.fileAppend doesn't work with ReadableStream
  await t.test('further sequence', { skip: isDeno && 'not supported in Deno yet' }, async () => {
    resp = await poteto(_(), { body: bodygen(), method: 'APPEND', duplex: 'half' });
    validate(resp, { status: 201, body: null });

    resp = await poteto(_(), { method: 'APPEND', duplex: 'half' });
    validate(resp, { status: 422, body: null });
// ...
// bunch of sequential reads-writes-appends-stats-etc.
// ...
  });

  await new Promise(_ => setTimeout(_, 9));

  resp = await poteto(_(), { method: 'DELETE' });

This test is strictly sequential as it tests various file operations at once, using a single file.
Adding subtest with skip condition looked like an idiomatic way to make a skip-if-condition block clearly indicated in the console output.
Of course, this can be mitigated (and ideally rewritten into many parallelizable tests anyway), but I'd like to point out that this is extremely unexpected and counter-intuitive when supposedly async function starts ignoring await and allows strictly subsequent tasks to run right away, make its inner tests floating, and make them fail for seemingly no reason.

I'm also trying to not think of how many tests in the wild may continue working after this change like nothing happened, but either become potentially flaky or start hiding issues because subsequent tests or cleanups are "fixing" them.

Additionally, I'm not really seeing any examples of what #56664 actually solved, as in what can be done after that change that was not working before. Making specific linter happy with hanging promises by eliminating ability to explicitly control the flow doesn't sound like a good justification.

Even if there will be consensus that the behaviour after semver-major change is preferable, node:test is a stable API; hence a breaking change like this should go through full deprecation cycle first.