prevent confusing debugging experience when users write garbage to stdout, interfering with the build runner / test runner protocol

[gitusel]

Zig Version

0.11.0-dev.2297+28d6dd75a

Steps to Reproduce and Observed Behavior

1/ zig init-lib
2/ Correct test_step.dependOn(&main_tests.step); to test_step.dependOn(&main_tests.run().step);
3/ src/main.zig
const std = @import("std");
test "t" {
const writer = std.io.getStdOut().writer();
var buffered_writer = std.io.bufferedWriter(writer);
var stdout_writer = buffered_writer.writer();
stdout_writer.print("TEST: t\n", .{}) catch @Panic("PANIC!\n");
buffered_writer.flush() catch @Panic("PANIC!\n");
}

4/ run zig build test. The test stalled and you need to exit using ctrl+c.

Expected Behavior

The test should run and exit.

If using stdErr, e.g. changing const writer = std.io.getStdOut().writer(); to const writer = std.io.getStdErr().writer(); will make it passed.

[jacobly0]

After the build parallel changes, stdout is now used to communicate progress to the build exe so that it can display aggregate progress for all simultaneously running test processes. Do you have a specific use case for writing to stdout instead of stderr from a test?

[gitusel]

If for any reason a function is displaying something then it will be blocked, e.g.:
const std = @import("std");
fn t() u8 {
const writer = std.io.getStdOut().writer();
var buffered_writer = std.io.bufferedWriter(writer);
var stdout_writer = buffered_writer.writer();
stdout_writer.print("TEST: t\n", .{}) catch @Panic("PANIC!\n");
buffered_writer.flush() catch @Panic("PANIC!\n");
return 0;
}

test "t" {
try std.testing.expect(t() == 0);
}

will also never exit.

[jacobly0]

By use case, I mean a specific motivating example, not "any reason". This involves more of an explanation for why the function is required to write to standard out and why it benefits from being tested than necessarily specific code. Based on the example, it vaguely suggests that the function just wants to print some testing status to stdout, which if even this were allowed, wouldn't work very well now that multiple test processes can be running at the same time. The current behavior is to collect all stderr and print it after the test runs, which is useful for descriptions of test failures, but not very useful for live status updates.

I do have an idea for addressing the hanging issue. At least on linux, stdout could be dup'd and closed when used by the test runner so that other code can't accidentally write to it. The descriptor could even be replaced with a handle to /dev/null so that bad code that does use it at least won't error (and to avoid it getting replaced with another open'd descriptior I guess), but I'm not sure which behavior is more desirable.

[gitusel]

Not sure to understand what you are looking for. But for instance, simple logging to stdOut. The code shared is a simplified version to show the issue. The solution to leverage stdOut as an internal channel between a test and build exe does not appear viable.

[jacobly0]

A reduced example is very valuable for demonstrating the issue, and in this case I agree that the example doesn't work as expected and so that has served its purpose. However, by reducing the example, there tends to be a lot of lost context that can be filled in by an explanation of what you were trying to accomplish when you ran into the issue. If the answer is that you don't specifically need stdout to work, but you just wanted to log something and happened to use stdout, expecting it to work, then that is a perfectly valid use case, and you are just missing a description of what you wanted to log (test progress, test result status, failure description, debug prints, actually testing a "log something to stdout" function, etc) and why logging to stderr instead does not solve this use case. Note that std.log uses stderr by default, so there's nothing inherent about the term "logging" that implies a requirement to use stdout in this context.

[gitusel]

(Un)fortunately, in this particular case, the library I translated is using its own logging framework and it relies on stdout. I think that the IPC between the build exe and the test RunStep has to be private.

[gitusel]

It is not confusion but test run never exits.

[karlseguin]

I just ran into this too.

Do you have a specific use case for writing to stdout instead of stderr from a test

It doesn't have to be the test that writes to stdout for it to hang, it can be the code under test. For example, this hangs

fn echo(data: []const u8) !void {
	try std.io.getStdOut().writeAll(data);
}

test "echo" {
	try echo("hello");
}

This is obviously a minimal example to reproduce the issue. But I don't think it's an edge-case that code under test, maybe through a library and maybe directly, might want to use stdout.

[karlseguin]

@andrewrk I guess I can see how being able to use stdout is considered an enhancement. But "zig build test" hanging when a function in the standard library is used should surely be classified as a bug?