Preservation of comments and whitespace

[ned14]

Firstly, this is one of those very few libraries I've used where I don't think I'd have done much different myself. Well, possibly I'd have made it always non-throwing. Well done on that, and thank you for making it open source.

Another thing that I would have done is optional comment and whitespace preservation, such that if you read in a TOML file, and then write it out, comments and whitespace are preserved. This is useful because I'm using your library a bit like how clang-tidy or doxygen do things, where you can dump out a default TOML config, hand edit it, or supply TOML delta files to merge against the default TOML config, and so on. Therefore, dumping out the post-merge TOML file, with all comments also merged, would be ideal.

For this to work right, you'd need to retain comments and whitespace as an attribute of the first node following the comment. Then, if the node gets replaced by a merge, one could retain the existing comment and whitespace, or replace it with the new comment and whitespace etc.

I'd personally make this opt-in, as plenty of people don't want to waste CPU nor memory on comments and whitespace. My thanks in advance to you for your consideration!

[marzer]

Firstly, this is one of those very few libraries I've used where I don't think I'd have done much different myself.

Damn, that's high praise. Thanks! Glad you like it. It's the first general-purpose C++ library I've released for public consumption so it's been nice to see it well-received.

Another thing that I would have done is optional comment and whitespace preservation, such that if you read in a TOML file, and then write it out, comments and whitespace are preserved. This is useful because I'm using your library a bit like how clang-tidy or doxygen do things, where you can dump out a default TOML config, hand edit it, or supply TOML delta files to merge against the default TOML config, and so on. Therefore, dumping out the post-merge TOML file, with all comments also merged, would be ideal.

Yeah, this is a piece of functionality I've been asked about a couple of times now, though you're the first to actually provide a compelling use-case. It's tentatively on my mental road-map for toml++ though might be a decent while before I can make time to implement it.

[marzer]

Also, reciprocal praise: your work on profiling the STL headers (e.g. here) is something I referred to on a regular basis in optimizing compile times for the library, so thanks for that!

[ned14]

You're very kind. I think if you're a certain kind of C++ dev, stuff like not including headers and, generally speaking, not having the compiler generate opcodes in the first place, leads to a certain shared philosophy. One which I would prefer to see more of on WG21, but that's by the by.

Am I right that your iterators don't implement operator->?

[marzer]

Quite likely. Most library functionality has been implemented in a very YAGNI way, i.e. I've only added something when I've personally needed it myself. I added the iterators to make ranged-for work and haven't really used them directly via begin()/end() at all.

Which of course means it wasn't an active design decision not to implement the operator, it just hasn't been done. I can add that to my to-do list, too, though all PR's welcome! :D

[ned14]

The need to retain comments across modification is becoming more urgent within our use case here, as a wider set of engineers use the tooling. We currently have TOML config files where the original source is commented with documentation describing each option, but once it passes through tomlplusplus all those comments get lost. This is quite unhelpful.

If you even just added the ability for there to be a custom void * per toml table, into which we could store an unordered map of toml identifier to comment, that would be wonderful.

[marzer]

@ned14

If you even just added the ability for there to be a custom void * per toml table, into which we could store an unordered map of toml identifier to comment, that would be wonderful.

Yeah, that'd be doable, though I'd likely make it opt-in to avoid ABI breaks for the masses. What say you to the idea of me exposing it as an "injectable", not unlike ImGui's IM_VEC2_CLASS_EXTRA and friends?

[ned14]

One possible ABI neutral way is an "invisible" node type which doesn't appear in the normal enumerations. It only appears if you explicitly ask for it by name. Said custom node type could carry user defined payload.

I'm not sure if you're already using it, but node_type::none could mean just such an invisible node.

[marzer]

That value is not presently used in that manner, but I'd rather not go down that path.

To be clear, I'm not at all concerned about ABI breaks for those opting in to this feature; it would be "buyer beware" advanced use case. I'm only worried about avoiding them for those not opting in.

[ned14]

I assume node_type::custom is also not your preference?

[marzer]

If I were to implement this fully I'd likely treat the comments as metadata associated with the tables/arrays/values they relate to, similar to how the source information works now, rather than add them as nodes.

But, er, unless I misunderstood what you were asking for as an interim solution, you suggested adding a public void* data member to toml::table, right? In which case I'm saying that I'd be happy to go a little further and allow advanced users to inject whatever they like using a config option. The only caveat for either route would be that I'd gate it behind an opt-in configurable so as to not break ABI for everyone else. It'd very much be a "here be dragons" feature.

[ned14]

I know that toml11 handles comments by attaching them to nodes, but that falls down if there are blocks of comment without any obvious node to attach them to. What most parsers do in this situation is to have comment nodes which act like whitespace i.e. ignored by default.

But separately to all that, what would be ideal for me right now is to be able to store custom data with a secret named key. Then I can do a poor man's regex parse of the incoming TOML, and say for a key named foo I might store an invisible custom node called #comment_foo with the comment text. When reconstituting the TOML for output as text, I can then inject the per-key comments from the original.

I know this probably sounds like I'm just going ahead and implementing comments, but because our TOML config files are super simple, I can hack a nasty solution which will work for our simple TOML. It wouldn't likely scale out well.

Failing that, the custom void * per toml table can also work. Just be aware that we may see mixed toml::table* in our code base because code A might create the custom table pointers, but code B might create default ones. So we'd need a way to disambiguate between them.

I originally thought to simply subclass toml::table with a custom type, then use RTTI to disambiguate, but you made toml::table final :(