Roadmap

[pt300]

Here I'd like present my suggestions for what I believe could be done to improve quality of JSMN. Everyone should feel free to comment on this and give own suggestions. Also, I will need opinion from @zserge on this.

• Split strict and non-strict parsing code
  Idea behind it is that this would make it easier to maintain both versions without having to overcomplicate the code and to eliminate risk of breaking strict version when trying to improve non-strict one

• Make strict mode the default
  Considering that JSMN is mostly aiming at parsing JSON strings and not it's abbreviations the library should do it by default.

• Enable parent links by default
  Parent links feature enables a great speed boost at a quite low memory cost. I believe a lot of people aren't using this library on really memory constrained platforms. But if there's a need for it the user should just disable parent links. It could be probably nice to rename the define to something like JSMN_LOW_MEMORY.

• Strict mode should follow RFC-8259
  There are a few standards with slight differences for JSON. I believe it should be explicitly said that we would follow RFC-8259 as THE standard for our JSON parsing.

• Maybe create a package for Arduino
  Considering the fact JSMN aim at platforms that are similar to what Arduino offers I believe it would be nice to either offer this library as is to Arduino community or separately write a wrapper to make access to this library more C++ like.

• Bit flags for token types Make jsmntype_t values bit flags #108
  Small change that would make it easier to write parsing code in certain places.

Utilities

• Token array traversing
  Using JSMN in a safe manner can be difficult. Because of that I think it would be nice to write an optional header file with utilities to easily and safely traverse JSMN's output. These should work at least for the strict mode.

• Values decoding
  At this point JSMN made the decision of not parsing the values. It could be left this way but then I believe it would be great to optionally offer utilities that could take for example a string from a token and decode it into UTF-8 or a widechar string.

[ahogen]

I've been using JSMN in a somewhat memory constrained application. The 16-bit micro has 64kB internal flash, but due to other required libraries, I didn't have a lot of room to spare. So keeping a low-memory footprint in mind is important to me. JSMN_LOW_MEMORY sounds great.

Utilities

• Values decoding
  ...

Some platforms I've used don't have the non-standard itoa() functions. So when I'm generating JSON replies, I've had to create a local_itoa() sort of function, optimized to only support bases 2 - 16. This is more of a values encoding idea, but maybe something to discuss?

[xdevelnet]

So, first of all I'd like you to thank for such nicely done piece of free software. I really appreciated of it.

And like ahogen, I also use this parser.

Ok, since you're planning to make significant changes to this project, I'd like to post here some thoughts about this:

At this point JSMN made the decision of not parsing the values

For my humble opinion you should at least check if this is even printable character in case of regular strings. If in double quotes pair suddenly happens binary garbage - that should not be parsed anyway, welp.

From other point of view, this project is not JSON validator, but JSON parser. Maybe it's better to throw away any validation since current validation is not even close to what should be and focus on performance and low memory usage. Then, recommend users to use separate modules for JSON validation, like this: http://www.json.org/JSON_checker/JSON_checker.c

[pt300]

Well, for the JSON to be parsable it has to be valid so as long as the parser is never wrong then if we are able to parse the thing then we are basically validating it. There's no way out of it.
In JSMN values aren't really fully parsed and so they are not validated.

How I think it could work is that the utilities for parsing JSON values as seen by JSMN will have to adhere to RFC-8259 anyway and so they will be able to tell you whether there is a problem with value itself. For example if token is of type string but it contains for example a byte of value 0x05, then the value parser should return an error as this is invalid according to the standard.

[pt300]

Also I'm feeling the a lot of pressure here as it seems like the original creator of this project either abandoned it or is too busy/tired to work on it and beside him I am the only person that can do anything with this repository. I have ambition of keeping JSMN as great as it is when it comes to parsing speed and memory usage but make it better at actually parsing JSON. Right now in my opinion there are quite a bit parsing bugs in JSMN which mostly come from the fact of how it's built.

Right now I have an idea of how JSMN could be slightly modified in a sense of idea of how the parsing happens but it would mean somewhat of a large code changes. And so I thought that why not as well just think of what else could be changed and just make JSMN the best thing out there that's easy to use.

[xdevelnet]

In JSMN values aren't really fully parsed and so they are not validated.

Currently, if \n happens in the middle of " " pair, JSMN still will parse it. BTW it could be fixed, so I'd like to ask: are you accepting PR's or you're already started developing an update for this project so there is not reason to even open PR?

Also I'm feeling the a lot of pressure here

Dunno. It's free software, people are free to modify it for their own needs.
For my humble opinion, from human philosophy PoV: If nothing was given then nothing should be expected. This thought partially described here: do-not-theme/do-not-theme.github.io#3 (comment)

I have ambition of keeping JSMN as great as it is when it comes to parsing speed and memory usage but make it better at actually parsing JSON.`

Hmmm... It reminds me an old joke about having a fast, reliable, strong, and cheap vehicle at same time. I have no idea if you can keep the same performance with all required validation.

[pt300]

Haven't really done any work myself yet. So far it's all planning and getting the courage to do any changes to this repository. After all I'm not @zserge and this is basically official repository of JSMN. I was given responsibility I haven't exactly asked for :p

Feel free to create PR. Although I think the way library works should be discussed to make it as close to perfect without weird trade-offs. But you are right, there are limits to what can be achieved here.

[zserge]

@pt300 I really like the suggested list of changes.
Strict mode by default, compatibility with RFC8259, Arduino package and bitflags - all sounds good.

But I want to share my vision of the 'non-strict mode'. Basically, it's not a feature of JSMN, but rather a side-effect of "what would happen if we #undef some of the parser code". This eliminates certain sanity checks, and parses invalid but nicely looking JSON-ish text, with unquoted strings and the absence of the root object. This has never been a priority for JSMN (in my mind), and I don't think it deserves a separate implementation. If, by undef-ing some code form the RFC-compatible strict implementation we can achieve some syntax sugar for JSON - fine, if not - I'd rather forget about non-strict mode at all.

[zserge]

Additionally, I would rather invest into advanced documented supersets of JSON, like JSON5, rather than in the undocumented, barely known non-strict mode.

[pt300]

What about the ability of estimation of token array length? Considering it works only for specific cases I think it's not a really good feature to keep around, especially that it seems to add more complexity to the code than it's worth.

[CaCtus491]

In my case I need to open a JSON object from an embedded file system based storage that may potentially be too large (up to ~64kb) to fit into the available RAM (~4 - 8kB), pull certain configuration values from it, then I'm done. I'm not particularly concerned about speed. Allocating RAM space for the token array is doable.

I currently have JSMN implemented and working well, and have been getting by though minimising the size of the JSON string (remove whitespace, ensure that only required JSON elements are present, etc) and maximising the heap space available to fit larger objects; however we're starting to feel the pinch as other areas of the code grow and start to demand more RAM.

I have been toying with the idea of abstracting the access to the underlying original JSON string so that it can still be backed by a raw array of bytes as it currently is, or alternatively a paged based solution that will only require a portion of the JSON string to be in RAM at once.

I'm unsure if this is even possible with the structure of JSMN, and even if it is, it is probably too much of a change to be considered based the 'light weight and fast' ethos (which I very much appreciate).

If I do get around to implementing it, I will certainly make the changes available.

[chrismerck]

Make strict mode the default / RFC-8259

Non-strict mode is important to the extent that JSON implementations differ in the wild (or, at least, to the extent that the "few standards" differ.)

a paged based solution that will only require a portion of the JSON string to be in RAM at once... if this is even possible with the structure of JSMN

JSMN is simple enough to already be "complete" --- so you should consider forking and modifying as you need, manually merging bugfixes if/when bugs are ever discovered.

---

Something I'd like to see is much clearer documentation of the functions, expectations in error cases, etc. Currently this requires code inspection & experimentation.

[vector76]

Thank you for your work on jsmn. On my private copy I've added some functionality for easier traversal, in particular all tokens have a "first_child" and a "next_sibling". I added a very simple function to populate these values after parsing (speed is not important to me). And then I can iterate over children of a node by:

for (int child = tokbuf[node].first_child; child != -1; child = tokbuf[child].next_sibling) {
   ...
}

I am not asking anything from the jsmn project but if you would like to see the code I would be happy to share (not that it's difficult).