Not returning proper count when used with tokens=NULL and stream data

[givanov256]

Hello, past few days I'm testing JSMN with stream data and NULL token parameter.

Simple example

static const char *JSON_STRING = "{\"user\": \"johndoe\", \"admin\": false}";
uint8_t bigarray[500];

int i=0;
int j=0;
int r;
jsmn_parser p;
jsmntok_t t[10]; /* We expect no more than 10 tokens */

jsmn_init(&p);

r = jsmn_parse(&p, JSON_STRING, strlen(JSON_STRING), NULL, 0);
printf("parse full JSON: %d\n", r);

jsmn_init(&p);
             
while(i<strlen(JSON_STRING))
{
    bigarray[j] = JSON_STRING[i]; //get from fifo
    
    //jsmn_init(&p);
    
    r = jsmn_parse(&p, bigarray, j+1, NULL, 0);
    printf("i=%d, ch=%c, ", j, bigarray[j]);           
    printf("r=%d ", r);           
    fwrite(bigarray, 1, j+1, stdout);
    printf("\n");
   
    
    i++;
    j++;
}

printf("parse stream JSON: %d\n", r);

So, the output is

parse full JSON: 5
i=0, ch={, r=1 {
i=1, ch=", r=-3 {"
i=2, ch=u, r=-3 {"u
i=3, ch=s, r=-3 {"us
i=4, ch=e, r=-3 {"use
i=5, ch=r, r=-3 {"user
i=6, ch=", r=1 {"user"
i=7, ch=:, r=0 {"user":
i=8, ch= , r=0 {"user":
i=9, ch=", r=-3 {"user": "
i=10, ch=j, r=-3 {"user": "j
i=11, ch=o, r=-3 {"user": "jo
i=12, ch=h, r=-3 {"user": "joh
i=13, ch=n, r=-3 {"user": "john
i=14, ch=d, r=-3 {"user": "johnd
i=15, ch=o, r=-3 {"user": "johndo
i=16, ch=e, r=-3 {"user": "johndoe
i=17, ch=", r=1 {"user": "johndoe"
i=18, ch=,, r=0 {"user": "johndoe",
i=19, ch= , r=0 {"user": "johndoe",
i=20, ch=", r=-3 {"user": "johndoe", "
i=21, ch=a, r=-3 {"user": "johndoe", "a
i=22, ch=d, r=-3 {"user": "johndoe", "ad
i=23, ch=m, r=-3 {"user": "johndoe", "adm
i=24, ch=i, r=-3 {"user": "johndoe", "admi
i=25, ch=n, r=-3 {"user": "johndoe", "admin
i=26, ch=", r=1 {"user": "johndoe", "admin"
i=27, ch=:, r=0 {"user": "johndoe", "admin":
i=28, ch= , r=0 {"user": "johndoe", "admin":
i=29, ch=f, r=1 {"user": "johndoe", "admin": f
i=30, ch=a, r=1 {"user": "johndoe", "admin": fa
i=31, ch=l, r=1 {"user": "johndoe", "admin": fal
i=32, ch=s, r=1 {"user": "johndoe", "admin": fals
i=33, ch=e, r=1 {"user": "johndoe", "admin": false
i=34, ch=}, r=0 {"user": "johndoe", "admin": false}
parse stream JSON: 0

Mind that JSMN_STRICT should be defined, because in other case false is decoded like bunch of primitives! But that is not the point.

From the output you can see that with tokens=NULL, the returned count is 1 when new token is found, not the found tokens so far. I've managed to locate that in jsmn.h in jsmn_parse function -> int count = parser->toknext;. So whit NULL, toknext is never updated and the returned count always starts from 0.

Workaround is to use jsmn_init(&p) in every iteration. But it waste time in checking entire buffer .

[pt300]

I will try to look into it, but for now most of the focus is on refactoring. As you can see I already lack the spoons for that.