CLI toml-test decoder

[moorereason]

Is your feature request related to a problem? Please describe.

I've been doing differential fuzzing against a few TOML decoders to see where they may handle inputs differently. I would like to add tomlplusplus into the mix.

Describe the solution you'd like

Can tomlplusplus provide a CLI toml-test decoder? The decoder takes a given TOML input via STDIN and outputs annotated JSON to STDOUT. Details are in the toml-test README.

Additional context

I've filed several bug reports in other projects and submitted new test cases to toml-test based upon this fuzzing.

Thanks for considering.

[marzer]

There's no toml-test decoder for toml++ currently, though my test suite consumes the test inputs from toml-test and generates C++ unit tests to validate using the test suite, e.g.: conformance_burntsushi_valid.cpp

I update the conformance tests every time I do any work on the library so it's generally up-to-date.

"Why did you bother writing a C++ code generator and not just provide an encoder/decoder?", I hear you ask.

It made sense at the time. Third-party libs in C++ can be very annoying, with or without CMake and friends, so in the early days of toml++ this approach seemed easier. I am going to do another batch of toml++ work at some point in January so if I get time I'll revisit this.

[moorereason]

I'm not advocating that you stop generating unit tests. Every(?) TOML package I've looked at generates local unit tests from the suite. However, I've only found 5 packages that have a toml-test decoder implementation.

I find a toml-test decoder useful for differential fuzzing and submitting bug reports. For example, I'm not a C++ programmer, but I can use a Go fuzzer to find and report bugs in a C++ decoder such as toml11 (e.g. ToruNiina/toml11#180 & ToruNiina/toml11#181). No C++ skills required! 😄

[marzer]

I'm not advocating that you stop generating unit tests.

Oh, certainly. I didn't think you were. Apologies if I gave you that impression. Was just explaining why this repo does the seemingly-harder thing first, instead of the probably-easier-thing, heh.

I do see the appeal. There's even a few tests I have to manually skip the code generation for because various C++ compilers don't like the in-code equivalent (e.g. GCC doesn't like whitespace after a trailing \ in a string literal, and there's a few tests in toml-test that have those). I'll get around to it eventually, hopefully as part of the next release.

[marzer]

@moorereason I've implemented this in the v3 branch - see v3/toml-test for setup info.

Every(?) TOML package I've looked at generates local unit tests from the suite.

Just re-read this. How do you figure that? It certainly isn't true for the C++ TOML ecosystem. In fact toml++ is the only one that chose that route, as far as I can tell - the others (the still-maintained ones, at least) appear to have done the sensible in the first place and built toml-test decoders 😄

In any case, toml++ has now joined the club and provided both an encoder and decoder for toml-test. Let me know how you get on with it.

[moorereason]

Sweet! I'm trying to get through my existing corpus, and I'm getting a UTF-8 error on a case.

Given an input of 0="\u0800", the tt_decoder returns:

terminate called after throwing an instance of 'nlohmann::detail::type_error'
  what():  [json.exception.type_error.316] invalid UTF-8 byte at index 1: 0x80

Not sure what to do with that.

[marzer]

O_o

Are you using any custom tests that aren't currently a part of toml-test? I built it from master today and didn't experience that.

If so, can you share the toml/json files?

[moorereason]

Are you using any custom tests that aren't currently a part of toml-test?

Yes, but I'm doing differential fuzzing against other toml-test decoder implementations via STDIN and comparing the results.

The TOML document in question is 10 characters long:

0="\u0800"

I've tested that input with 6 other decoders (in Go, C99, C++, Fortran, and Dart), and they all return valid and equal results.

[marzer]

Oh, right, fuzzing. Ok well I'll do some digging and see what's going on.