Skip to content

Commit 41cf5e9

Browse files
committed
Clarify expected/actual failure colorization
On CI, `test-fixtures-windows` does `GIX_TEST_IGNORE_ARCHIVES=1` test runs on Windows, which have a number of known and expected failures, as described in GitoxideLabs#1358. Since GitoxideLabs#1663, these are listed in `etc/test-fixtures-windows-expected-failures-see-issue-1358.txt` and `test-fixtures-windows` compares those expected failures to the actual failures of a run. If there are any differences, it shows a unified diff with full context and fails the run. The default Git foreground color scheme has been used for the diff, where any tests that are no longer failing are shown as `-` lines in red, and any tests that are newly failing are shown as `+` lines in green. These are the usual default `git diff` colors; we did not pick those colors specifically to be meaningful to this scenario. While the use of `-` and `+` lines and the specific unified diff format chosen are intuitive and informative, this is less so for the color scheme. That is for two reasons: 1. By default and unless another theme is chosen explicitly or selected based on device settings, the GitHub web interface uses a light theme. In this theme, as currently coded/styled, on the pale background in GitHub Actions, foreground normal-styled green text can be hard to distinguish visually from foreground normal-style black text. So it is often not immediately clear which test failures are new. (New test failures are the typical way a `test-fixtures-windows` job fails, when it does.) 2. A test that is no longer failing would usually mean that a test or implementation bug has been fixed, or that compatibility has otherwise been improved. `test-fixtures-windows` fails on these so that we attend to them, usually by dropping newly passing tests from the list of expected failures. Likewise, a test that is failing, especially a newly failing tests, usually if not always means that something is not right, even if it is merely the test itself that needs to be improved. But showing things that have gotten better in red, and things that have gotten worse in green, is unintuitive. This is partly because these colors have the opposite meaning in numerous contexts, including in some parts of the GitHub Actions interface. It is more specifically because they have the opposite meaning in the context of previous step output, where `nextest` shows stderr in red (and specifically shows stderr on tests that have failed), shows `FAIL` in red, and shows `PASS` in green. It may be possible to make red and green show up better. These colors are clear in the `nextest` output. However, the options for `diff` colors, at least when configuring them with `color.diff.*`, are limited. Furthermore, that would only improve (1), not (2). This bolds both `-` and `+` lines, shows `-` lines in magenta, and shows `+` lines in blue. These colors seem to be visible on both light and dark backgrounds, and they seem to be at least as distinguishable from each other as are red and green, in general. This change only applies to the specific diff that shows newly passing and newly failing tests in `test-fixtures-windows`.
1 parent d97f27e commit 41cf5e9

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

.github/workflows/ci.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -305,7 +305,8 @@ jobs:
305305
- name: Compare expected and actual failures
306306
run: |
307307
# Fail on any differences, even unexpectedly passing tests, so they can be investigated.
308-
git --no-pager diff --no-index --exit-code --unified=1000000 --color=always -- `
308+
git --no-pager -c diff.color.old='magenta bold' diff.color.new='blue bold' `
309+
diff --no-index --exit-code --unified=1000000 --color=always -- `
309310
etc/test-fixtures-windows-expected-failures-see-issue-1358.txt actual-failures.txt
310311
311312
test-32bit:

0 commit comments

Comments
 (0)