Rust test runner: Please support Codewars Output format too #822
Description
Currently, the Rust runner (rightfully) uses the built-in cargo test test runner. It's the defacto framework for Rust tests.
However, that framework also has its limitations, in its rigidity. Sometimes, you just need to control the output format and order, and that's not something that can be done with the Rust runner alone (it runs tests in parallel, and doesn't offer any way of controlling test order or test skipping).
By default, the runner captures any output produced with the println!() and eprintln!() macros (to display this output for failed tests, normally). But content written to a new std::io::stdout() handle is not capured, so a Rust test could still output the Codewars tagged test progress output, simply by not using the [e]println!() macros.
Unfortunately, the current CodeWars Rust runner doesn't support this use-case. While I understand that this repository is no longer reflecting how the CodeWars runners actually work, I'm assuming that it's not too far removed from the rust.js runner still visible here; that runner discards all output other than the standard Rust test success and failure text.
Please support forwarding any tagged output on stdout, so we can produce richer test suites in Rust.
The use case
Why do I want this? Because I want to provide a Rust translation for the BECOME IMMORTAL kata. The test suite for this Kata tells a story. It sets time limits. It executes tests in phases, with colour-coded sections (<:LF:> breaks with added newline and indentation for readability):
<DESCRIBE::>become_immortal
<IT::>example_tests
<PASSED::>Test Passed
<COMPLETEDIN::>[mstime]
<IT::>phase_1
<LOG::><h1>The Elder is interested...</h1>
<:LF:><h2>Young man, you should learn a thing or two...</h2>
<:LF:><p><font color="green">100 test cases
<:LF:>m,n: 2^5 - 2^10
<:LF:>l: 0 - 19
<:LF:>t: 2^5 - 2^15</font></p>
︙<LOG::>The Elder says:
︙ <:LF:><p><font color="green">m=[rnd], n=[rnd], l=[rnd], t=[rnd]</font></p>
⋱ ……… x 100 times
<:LF:><p><font color="green">Completed in [timems] ms</font></p>
<PASSED::>Test Passed
<COMPLETEDIN::>[mstime]
<IT::>phase_2
<LOG::><h1>The Elder is excited!</h1>
<:LF:><h2>You're too young and too simple!</h2>
<:LF:><p><font color="yellow">300 test cases
<:LF:>m,n: 2^8 - 2^20
<:LF:>l: 0 - 9999
<:LF:>t: 2^10 - 2^20</font></p>
︙<LOG::>The Elder says:
︙ <:LF:><p><font color="yellow">m=[rnd], n=[rnd], l=[rnd], t=[rnd]</font></p>
⋱ ……… x 300 times
<:LF:><p><font color="yellow">Completed in [timems] ms</font></p>
<PASSED::>Test Passed
<COMPLETEDIN::>[mstime]
<IT::>phase_3
<LOG::><h1>The Elder is angry!</h1>
<:LF:><h2>And sometimes naive!</h2>
<:LF:><p><font color="red">500 test cases
<:LF:>m,n: 2^32 - 2^64
<:LF:>l: 0 - 999999
<:LF:>t: 2^16 - 2^32</font></p>
︙<LOG::>The Elder says:
︙ <:LF:><p><font color="red">m=[rnd], n=[rnd], l=[rnd], t=[rnd]</font></p>
⋱ ……… x 500 times
<:LF:><p><font color="red">Completed in [timems] ms</font></p>
<PASSED::>Test Passed
<COMPLETEDIN::>[mstime]
<IT::>conclusion
<LOG::>Your final time is... [mstime]ms
<:LF:>You finished all the tests within 1 second. The Elder is very happy! +1s
<PASSED::>Test Passed
<COMPLETEDIN::>[mstime]
<COMPLETEDIN::>[mstime]
And we can't provide this in a Rust translation of the Kata because there are no options to produce the same kind of output.
If the option existed, I'd write one Rust test case, which then uses a few Rust macros to provide the needed test framework support, so we can produce sequential output.
Offer to do it myself
I'd be happy to produce a diff or pull request, if there was a way to access the docker and runner setup somewhere.
It's not much work, provided the macros take care to lock the handle and start output with a newline to avoid interleaving with the test [module]::[testfunc] ... and ok/FAILED output. Of course, the test runner could always be configured to run single-threaded with cargo test -- --test-threads=1...
Sample test case
Here is a sample test snippet that only outputs one of the documentation examples (directly as raw bytes for simplicity):
#[cfg(test)]
mod tests {
use std::io::{stdout,Result,Write};
#[test]
fn test_stdout() -> Result<()> {
stdout().write_all(b"<DESCRIBE::>Foo
<IT::>It should return a string
<PASSED::>Test Passed
<COMPLETEDIN::>23
<IT::>It should return \"foo\"
This is some direct output (i.e. console.log(\"...\"))
<FAILED::>Expected \"foo\" but instead got \"\"
<COMPLETEDIN::>10
<DESCRIBE::>This is a nested describe
<IT::>Should not be null
<PASSED::>Test Passed
<COMPLETEDIN::>20
<COMPLETEDIN::>22
<COMPLETEDIN::>100\n")?;
Ok(())
}
}