Need help with parsing JSON with strings in Telegraf inputs.http

[Thidjs]

Relevant conf

telegraf.conf

[[inputs.http]]
urls = [“https://api-url”]
method = “GET”
data_format = “json”
json_query = “stats”
string_fields = [“paid”]

json response

{
    “stats”: {
    “hashes”: “1104589883”,
    “lastShare”: “1536248906”,
    “balance”: “91967”,
    “paid”: “8520900”,
    “minPayoutLevel”: “200000”,
    “hashrate”: “1.91 KH”
}
    “charts”: {
    “payments”: [
                       [
                         1535688385,
                         “184800”
                       ],
                       [
                         1535697940,
                         “123300”
                       ]
                       ]
}

System info

Windows 10 fully updated
Telegraf latest nightly (7 sept)

Steps to reproduce

I want to use the json_query, because I don’t need the “charts” stuff, only the stats-portion.
The issue is that when I leave the json_query out, it only parses the charts-part like charts_payments_0_0 with as field the numeric part like 1535688385.
When I enable the JSON query nothing gets parsed at all.

Who knows what to change? Unfortunately I cannot change the JSON response, so I have to deal with the strings…

Thanx in advance!!

[danielnelson]

Thanks for the help testing, looks like this feature was not working due to some mixup in the variable naming. Once #4656 is merged you should use the option json_string_fields instead of string_fields, your query and other settings are all correct.

[Thidjs]

Thanx for the reply, I make sure I will test it when the changes are merged and report my findings here

[GillesC]

The MQTT consumer has not defined json_string_fields in the latest nightly build for the armhf platform.
I will now build from source and update my comment.
However, I think this should be clearly described in the documentation of the input data format.

Update: string_fields is defined in the nigthly build, but it seems that it does not parse the parameter to a field.

[GillesC]

I finally managed to convert a string-valued json parameter; but not through string_fields or json_string_fields.

1. I added the needed string-valued key to tag_keys

tag_keys = [
    "dev_id",
    "metadata_data_rate"
  ]

2. Than I used the Converter processor to convert the tag to a field

[[processors.converter]]
  order = 1
  [processors.converter.tags]
    string = ["metadata_data_rate"]

In your case you could (probably) also use another target type.

3. Finally, I converted the strings (which could be interpreted as an enum type) through the Enum processor:

[[processors.enum]]
  order = 2
  [[processors.enum.fields]]
    ## Name of the field to map
    source = "metadata_data_rate"
    destination = "spreading_factor"

    default = 0

    ## Table of mappings
    [processors.enum.fields.value_mappings]
      SF12BW125 = 12
      SF11BW125 = 11
      SF10BW125 = 10
      SF9BW125 = 9
      SF8BW125 = 8
      SF7BW125 = 7

[danielnelson]

@GillesC json_string_fields should be working now, can you post an example of the data you are trying to parse and what your settings are. Can you also post the output of telegraf version for the nightly you are using.

BTW, we are switching around the enum processor config before the release #4672

[Thidjs]

@danielnelson Just tested the latest nightly on my config stated earlier. It works like a charm, al strings defined in json_string_fields are being sent to influxdb. Next step for me is to convert some of these strings to float so I can use them in graphs in Grafana :-) Any suggestions?

edit, 5 seconds later...

never mind I should've read the docs better :-)

after inserting this to my config, it works perfectly;

[[processors.converter]]
[processors.converter.fields]
float = ["paid"]

[woozhijun]

If there are many columns, this does not seem to be a good solution.

[danielnelson]

@woozhijun Agreed, we have an issue open to add glob support to the json_string_fields. #4912

[woozhijun]

@danielnelson I tried json_string_fields = ["fields.*"] before, but it did't taken effect

[amir-am]

@danielnelson Has this item been added?
json_string_fields = ["fields.*"]