Release 2.0

There's a few incompatible changes to the CLI interface as it was all
getting quite messy, so release as 2.0.

This also enables TOML 1.1 default (via the github.com/BurntSushi/toml
update).

Fixes #168
Fixes #173

Author: arp242

.github/workflows/test.yml

@@ -5,20 +5,20 @@
     "test": {
       "strategy": {
         "matrix": {
-          "go-version": ["1.19.x", "1.24.x"],
+          "go-version": ["1.19.x", "1.25.x"],
           "os":         ["ubuntu-latest", "macos-latest", "windows-latest"]
         }
       },
       "runs-on": "${{ matrix.os }}",
       "steps": [
         {
           "name": "Install Go",
-          "uses": "actions/setup-go@v5",
+          "uses": "actions/setup-go@v6",
           "with": {"go-version": "${{ matrix.go-version }}"}
         },
         {
           "name": "Checkout code",
-          "uses": "actions/checkout@v4"
+          "uses": "actions/checkout@v6"
         },
         {
           "name": "Test",

CHANGELOG.md

@@ -0,0 +1,138 @@
+v2.0.0 2025-12-18
+-----------------
+This release has a number of incompatible changes. It also sets TOML 1.1 as the
+default version to test.
+
+### Incompatible changes
+- The CLI syntax has been cleaned up to make it easier to use. The main
+  motivation was to enable running all tests (decoder and encoder) with just one
+  `toml-test` invocation, which was hard to do in a compatible way.
+
+  To run the tests, you now need to use the `test` subcommand. And the decoder
+  and encoder binaries are specified via the `-decoder` and `-encoder` flags
+  rather than as positional arguments.
+
+  Before:
+
+      % toml-test my-decoder
+      % toml-test -encoder my-encoder
+
+  Now:
+
+      % toml-test test -decoder=my-decoder -encoder=my-encoder
+  
+- `toml-test -copy` is now `toml-test copy`.
+
+- `toml-test -version` is now `toml-test version`.
+
+- `toml-test -list` has been removed; I don't think anyone was using it(?) and
+  the `copy` command should cover most use cases. It can be added again if
+  someone needs it.
+
+- Rename `-print-skip` flag to `-script`.
+
+- Remove `-testdir` flag to load test files from the filesystem. Tests have been
+  built in the binary since 1.0.0-beta1 (2021), and this flag has not really
+  been useless ever since.
+
+- It now tests TOML 1.1 by default. Use `-toml=1.0` if you want to test TOML
+  1.0.
+
+### Other changes
+- Add various new tests, and rename some existing tests for consistency.
+
+- Add `-json` flag to the `test` command, to print test output as JSON rather
+  than text.
+
+- Allow setting `-toml=latest`.
+
+- Add `-skip-must-err` flag to the `test` command, to treat skipped tests that
+  don't fail as an error.
+
+v1.6.0, 2025-04-15
+------------------
+This is a small maintenance release which fixes a few small bugs in the test
+runner and adds a few small tests. See the git log for details: v1.5.0...v1.6.0
+
+v1.5.0, 2024-05-31
+------------------
+### Changes
+- This release requires Go 1.19 to build.
+
+- Add quite a lot of new test.
+
+- Only "pass" an invalid test if the decoder exits with exactly exit 1, rather
+  than any exit >0. This catches segfaults, panics, and other crashes which
+  shouldn't be considered "passing".
+
+- Tests are now run in parallel, defaulting to the number of available cores.
+  Use the `-parallel` flag to set the number of cores to use.
+
+- Few small improvements to toml-test runner output.
+
+### New features
+- The `-copy` flag copies all tests to the given directory (taking the `-toml`
+  flag in to account). This is much easier than manually copying the files.
+
+- Add `-errors` flag to test expected error messages for invalid tests. See
+  `-help` for details.
+
+- Add `-print-skip`, to print out a small bash/zsh script with `-skip` flags for
+  tests that failed. Useful to get a list of "known failures" for CI
+  integrations and such.
+
+- Add `-timeout` flag to set the maximum execution time per test, to catch
+  infinite loops and/or pathological cases. This defaults to 1s, but can
+  probably be set (much) lower for most implementations.
+
+- Add `-int-as-float` flag, for implementations that treat all numbers as
+  floats.
+
+- Add `-cat` flag to create a large (valid) TOML document, for benchmarks and
+  such.
+
+v1.4.0, 2023-09-29
+------------------
+- Move from github.com/BurntSushi/toml-test to github.com/toml-lang/toml-test
+
+  In most cases things should keep working as GitHub will redirect things, but
+  you'll have to update the path if you install from source with `go install`.
+
+- Both TOML 1.0 and the upcoming TOML 1.1 are now supported.
+
+  If you implemented your own test-runner, then you should only copy/use the
+  files listed in `tests/files-toml-1.0.0` (or `tests/files-toml-1.1.0`). Some
+  things that are invalid in 1.0 are now valid in 1.1.
+
+  Also see "Usage without toml-test binary" in the README.md.
+
+  For the `toml-test` tool the default remains 1.0; add `-toml 1.1.0` to use
+  TOML 1.1.
+
+- Add a few tests, and improve output on test failures a bit.
+
+v1.2.0, 2023-01-15
+------------------
+A few minor fixes and additional tests; see the git log for details:
+v1.2.0...v1.3.0
+
+v1.2.0, 2022-06-02
+------------------
+A few minor fixes and additional tests; see the git log for details:
+v1.1.0...v1.2.0
+
+v1.1.0, 2022-01-12
+------------------
+Adds various tests; see the git log for details: v1.0.0...v1.1.0
+
+v1.0.0, 2021-08-04
+------------------
+Many changes since the last release in 2013: much improved error output, support
+TOML 1.0.0, add several flags to give more control over which tests to run/skip.
+
+Some minor incompatibilities in the test tool:
+
+- You no longer need to add a type hint to arrays.
+- Tests are always referenced as valid/[...] or invalid/[..]
+- The datetime-local, date-local, and time-local types are added. You will need
+  to add support for this in your -encode and -decode test helpers.

README.md

@@ -15,67 +15,59 @@ If you find something in your parser that's not exactly covered by toml-test
 already then it should be added here; just creating an issue is enough: don't
 *need* to create a PR.
 
-Compatible with TOML version [v1.0.0].
+Compatible with TOML versions [v1.0.0] and [v1.1.0].
 
 [TOML]: https://toml.io
 [v1.0.0]: https://toml.io/en/v1.0.0
+[v1.1.0]: https://toml.io/en/v1.1.0
 
 Installation
 ------------
 There are binaries on the [release page]; these are statically compiled and
-should run in most environments. It's recommended you use a binary, or a tagged
+should run in most environments. It's recommended you use a binary or a tagged
 release if you build from source especially in CI environments. This prevents
 your tests from breaking on changes to tests in this tool.
 
 To compile from source you will need Go 1.19 or newer:
 
-    % go install github.com/toml-lang/toml-test/cmd/toml-test@latest
+    % go install github.com/toml-lang/toml-test/v2/cmd/toml-test@latest
 
 This will build a `toml-test` binary in the `~/go/bin` directory. You can change
 that directory by setting `GOBIN`; for example to use the current directory:
 
-    % GOBIN="$(pwd)" go install github.com/toml-lang/toml-test/cmd/toml-test@latest
+    % GOBIN="$(pwd)" go install github.com/toml-lang/toml-test/v2/cmd/toml-test@latest
 
-[release page]: https://github.com/toml-lang/toml-test/releases
-
-Running in CI
--------------
-The [setup-toml-test] action can be used in GitHub. See the README for more
-details.
+See [CHANGELOG.md] for a list of changes.
 
-For other CI systems: the action essentially just downloads a release binary.
-See index.js. Should be easy enough to reproduce elsewhere.
-
-[setup-toml-test]: https://github.com/toml-lang/setup-toml-test
+[release page]: https://github.com/toml-lang/toml-test/releases
+[CHANGELOG.md]: ./CHANGELOG.md
 
 Usage
 -----
-`toml-test` accepts an encoder or decoder as the first positional argument, for
-example:
-
-    % toml-test my-toml-decoder
-    % toml-test my-toml-encoder -encoder
-
-The `-encoder` flag is used to signal that this is an encoder rather than a
-decoder.
-
-For example, to run the tests against the Go TOML library:
+`toml-test test` runs the test suite, and requires an `-decoder` and/or
+`-encoder` flag; for example:
 
     # Install my parser
     % go install github.com/BurntSushi/toml/cmd/toml-test-decoder@master
     % go install github.com/BurntSushi/toml/cmd/toml-test-encoder@master
 
-    % toml-test toml-test-decoder
-    toml-test v2023-10-23 [toml-test-decoder]: using embeded tests: 278 passed
+    # Run tests
+    % toml-test test \
+       -decoder=toml-test-decoder \
+       -encoder=toml-test-encoder
+
+    [..]
 
-    % toml-test -encoder toml-test-encoder
-    toml-test v2023-10-23 [toml-test-encoder]: using embeded tests:  94 passed,  0 failed
+    toml-test v2025-12-16 [toml-test-decoder] [toml-test-encoder]
+      valid tests: 205 passed,  0 failed
+    encoder tests: 205 passed,  0 failed
+    invalid tests: 460 passed, 15 failed
 
 You can use `-run [name]` or `-skip [name]` to run or skip specific tests. Both
 flags can be given more than once and accept glob patterns: `-run
 'valid/string/*'`.
 
-See `toml-test -help` for detailed usage.
+See `toml-test test -help` for detailed usage.
 
 ### Implementing a decoder
 For your decoder to be compatible with `toml-test` it **must** satisfy the
@@ -230,13 +222,13 @@ version of TOML; for example the 1.0.0 tests contain a test that trailing commas
 in tables are invalid, but in 1.1.0 this should be considered valid.
 
 In short: you can't "just" copy all .toml and .json files from the tests/
-directory. The easiest way to copy the correct files is to use `-copy`:
+directory. The easiest way to copy the correct files is to use `copy`:
 
     # Default of TOML 1.0
-    % toml-test -copy ./tests
+    % toml-test copy ./tests
 
     # Use TOML 1.1
-    % toml-test -copy ./tests -toml 1.1.0
+    % toml-test copy -toml 1.1.0 ./tests
 
 Alternatively, the [tests/files-toml-1.0.0] and [tests/files-toml-1.1.0] files
 contain a list of files you should run for that TOML version. You can use them
@@ -257,7 +249,7 @@ mentioned above, tests are split into two groups: invalid and valid tests.
 
 Invalid tests **only check if a decoder rejects invalid TOML data**. Or, in the
 case of testing encoders, invalid tests **only check if an encoder rejects an
-invalid representation of TOML** (e.g., a hetergeneous array). Therefore, all
+invalid representation of TOML** (e.g., a heterogeneous array). Therefore, all
 invalid tests should try to **test one thing and one thing only**. Invalid tests
 should be named after the fault it is trying to expose. Invalid tests for
 decoders are in the `tests/invalid` directory while invalid tests for encoders

cmd/toml-test/copy.go

@@ -8,7 +8,7 @@ import (
 	"sort"
 	"strings"
 
-	tomltest "github.com/toml-lang/toml-test"
+	tomltest "github.com/toml-lang/toml-test/v2"
 	"zgo.at/zli"
 )
 

cmd/toml-test/main.go

@@ -33,44 +33,30 @@ func main() {
 		return
 	}
 	if err != nil {
-		if helpFlag.Set() {
-			if contains(f.Args, "test") {
-				fmt.Print(usageTest)
-			} else {
-				fmt.Print(usage)
-			}
-			return
-		}
 		zli.F(err)
 	}
+	if helpFlag.Set() {
+		f.Args, cmd = []string{cmd}, "help"
+	}
 
 	switch cmd {
 	case "help":
-		if contains(f.Args, "test") {
-			fmt.Print(usageTest)
-		} else {
-			fmt.Print(usage)
+		topic := ""
+		if len(f.Args) > 0 {
+			topic = f.Args[0]
 		}
+		u, ok := helpTopics[topic]
+		if !ok {
+			zli.Fatalf("no help for %q", topic)
+		}
+		fmt.Print(u)
 	case "version":
 		v := f.Bool(false, "v")
 		zli.F(f.Parse())
 		zli.PrintVersion(v.Bool())
 	case "copy", "cp":
 		cmdCopy(f)
 	case "test":
-		if helpFlag.Set() || contains(f.Args, "help") {
-			fmt.Print(usageTest)
-			return
-		}
 		cmdTest(f)
 	}
 }
-
-func contains[S ~[]E, E comparable](s S, v E) bool {
-	for i := range s {
-		if v == s[i] {
-			return true
-		}
-	}
-	return false
-}

cmd/toml-test/test.go

@@ -10,7 +10,7 @@ import (
 	"time"
 
 	"github.com/BurntSushi/toml"
-	tomltest "github.com/toml-lang/toml-test"
+	tomltest "github.com/toml-lang/toml-test/v2"
 	"zgo.at/jfmt"
 	"zgo.at/zli"
 )