Add JSON schema based validation to mbed config script #5022

ghost · 2017-09-05T14:10:18Z

Description

The current configuration system allows a user to specify various compile time configuration options for various targets within JSON files in the projects directory mbed_app.json for applications, and mbed_lib.json for libraries. The parameters will be converted to a collection of macros that are defined when compiling the code. The format of the JSON file is defined here.

Currently, the process of the checking the validity of the config file is done through a series of manual checks in tools/config/__init__.py. This PR will replace the manual checks with a solution that uses JSON schema to validate the configuration files.

Status

IN DEVELOPMENT

Todos

Define the JSON schema for app and lib library files
Remove remaining manual checks

Steps to test or reproduce

Test the current validation test cases still all pass:

Linux: $ PYTHONPATH=. python -m pytest tools/test/config/config_test.py

Windows: $ set PYTHONPATH=. && python -m pytest tools/test/config/config_test.py

0xc0170 · 2017-09-05T14:12:00Z

cc @theotherjimmy

theotherjimmy · 2017-09-05T14:16:26Z

@scartmell-arm Wow thanks!

theotherjimmy

Looks great so far! In it's current form, it's a huge improvement over the old validation mechanism. Be aware that #4984 may create a conflict with the test changes.

theotherjimmy · 2017-09-05T14:19:11Z

tools/config/schema_app.json

+        "pattern": "^([A-Za-z0-9_]+|[A-Za-z0-9_]+=[0-9]+|[A-Za-z0-9_]+=\\\".*\\\")$"
+      }
+    },
+    "__config_path": {


Is this key something that we should allow in the configuration file? or is it something that the tools could add after validation?

I wasn't sure when I added it, it was in the allowable keys list, but isn't mentioned anywhere in the specification so it's probably fine to remove.

It's added here

You could do the validation before adding that key.

That seems sensible, I'll do that.

theotherjimmy · 2017-09-05T14:20:06Z

tools/config/schema_lib.json

+  "description": "Configuration file for an mbed library",
+  "type": "object",
+  "$id": "http://example.com/root.json",
+  "definitions": {


These definitions look almost the same. Could they be moved to another file, and imported?

Yeah, that should be possible. I'll have a look at separating out the common parts of them, I don't think there are any differences between app and lib config parameter handling that needs to be handled in the validation.

theotherjimmy · 2017-09-05T14:22:32Z

It looks like travis failed because it does not have the jsonschema package installed. Could you add that to the requirements.txt?

theotherjimmy · 2017-09-05T17:56:57Z

@scartmell-arm Looks like there's something missing from the schema: defining macros as evaluating to other macros. Do we need to add this to the documentation?

ConfigException: 'UVISOR_MODE=UVISOR_DISABLED' does not match '^([A-Za-z0-9_]+|[A-Za-z0-9_]+=[0-9]+|[A-Za-z0-9_]+=\\".*\\")$'

'MEM_FREE=free' does not match '^([A-Za-z0-9_]+|[A-Za-z0-9_]+=[0-9]+|[A-Za-z0-9_]+=\\".*\\")$','MEM_ALLOC=malloc' does not match '^([A-Za-z0-9_]+|[A-Za-z0-9_]+=[0-9]+|[A-Za-z0-9_]+=\\".*\\")$'

ghost · 2017-09-06T09:19:38Z

Looks like there's something missing from the schema: defining macros as evaluating to other macros. Do we need to add this to the documentation?

@theotherjimmy Yeah, that isn't currently included in the documentation. I have a feeling it's going to be a futile endeavour writing a regex pattern to cover the various valid macro forms so I'll also change that to something reasonable.

Also, wrt the documentation, it states:

However, if target_overrides is defined, config must also be defined.

A number of the tests, and potentially some of the actual application configurations don't follow this so I haven't currently added it to the schema. Whilst it might be correct, I can't see any harm in defining target_overrides without config being set.

theotherjimmy · 2017-09-06T14:02:27Z

Yeah, that isn't currently included in the documentation.

I'll make a note to include that capability in the documentation.

I have a feeling it's going to be a futile endeavour writing a regex pattern to cover the various valid macro forms so I'll also change that to something reasonable.

Something reasonable like [A-Z0-9_]+(=.*)?? I think that's the best we have.

A number of the tests, and potentially some of the actual application configurations don't follow this so I haven't currently added it to the schema.

Good. That line is just wrong. Adding to the list of documentation updates.

theotherjimmy · 2017-09-07T16:19:00Z

@scartmell-arm It looks like #4984 changed the way those test worked out from underneath you. Could you rebase to resolve those conflicts?

theotherjimmy

Was this an omission?

theotherjimmy · 2017-09-07T16:21:04Z

tools/config/schema_lib.json

+  "title": "Mbed Library Schema",
+  "description": "Configuration file for an mbed library",
+  "type": "object",
+  "$id": "http://example.com/root.json",


I don't think this is quite right.

theotherjimmy · 2017-09-07T16:21:13Z

tools/config/schema_app.json

+  "title": "Mbed Library Schema",
+  "description": "Configuration file for an mbed library",
+  "type": "object",
+  "$id": "http://example.com/root.json",


This looks off too.

mbed-ci · 2017-10-10T10:52:23Z

Build : FAILURE

Build number : 50
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/5022/

mbed-ci · 2017-10-10T14:04:33Z

Build : FAILURE

Build number : 59
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/5022/

adbridge · 2017-10-16T15:37:54Z

/morph build

adbridge · 2017-10-16T15:38:35Z

@theotherjimmy looks like this needs a re-review

ghost · 2017-10-16T15:46:35Z

It's currently breaking in CI because some examples have mbed-app.json that apparently aren't handled by the schema.

Specifically, mbed-cloud-client-example-sources-internal broke during the CI build because it's configuration contains "mbed-trace.enable": null which is being rejected as invalid, though I'm not entirely sure why.

mbed-ci · 2017-10-16T17:52:32Z

Build : FAILURE

Build number : 184
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/5022/

theotherjimmy · 2017-10-23T15:17:33Z

@studavekar I can't find the failure in the above build. Could you take a look?

ghost · 2017-10-23T15:33:35Z

There's a related one here: Build 729

15:52:08 [K64F_WIFI4 IAR] [ERROR] None is not valid under any of the given schemas,None is not valid under any of the given schemas,None is not valid under any of the given schemas,None is not valid under any of the given schemas,None is not valid under any of the given schemas,None is not valid under any of the given schemas,None is not valid under any of the given schemas,None is not valid under any of the given schemas,None is not valid under any of the given schemas,None is not valid under any of the given schemas,None is not valid under any of the given schemas,None is not valid under any of the given schemas,None is not valid under any of the given schemas,None is not valid under any of the given schemas,None is not valid under any of the given schemas

theotherjimmy · 2017-10-23T15:42:55Z

I think you're right. Should we allow None? I'm thinking not, but it would be fine, as we would treat it the same as False. Up to @sg- really.

adbridge · 2017-10-31T14:48:28Z

@sg- Do you have an opinion on this ?

sg- · 2017-10-31T17:03:45Z

None in the past has led to tracebacks. Is that still the case? If so, no to None, if handled properly then it seems ok.

theotherjimmy · 2017-10-31T18:44:19Z

@sg- I don't think that None will lead to trace backs here, but I can't be sure without inspecting a large amount of code. I'd rather be safe and not allow it.

0xc0170 · 2017-11-22T14:14:47Z

@sg- I don't think that None will lead to trace backs here, but I can't be sure without inspecting a large amount of code. I'd rather be safe and not allow it.

@scartmell-arm Can you update the schema based on the feedback above? As this has been idle for some time, I would recommend rebase as well.

mbed-ci · 2017-11-22T14:51:01Z

Build : SUCCESS

Build number : 575
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/5022/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build

mbed-ci · 2018-02-01T02:23:55Z

Build : SUCCESS

Build number : 1032
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/5022/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build

mbed-ci · 2018-02-01T05:02:23Z

Exporter Build : SUCCESS

Build number : 713
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/5022/

mbed-ci · 2018-02-01T09:59:28Z

Test : SUCCESS

Build number : 841
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/5022/841

0xc0170 · 2018-02-01T10:08:23Z

This is now ready for integration

cmonr · 2018-02-01T16:59:57Z

@scartmell-arm Could you rebase this PR one last time? Looks like a conflict was recently introduced.
I'll keep an eye to push it through CI once the rebase is completed.

- Added app and lib JSON schema definition files which specify the valid keys and values that can be used in mbed library and application configuration files. The primary different between the app and lib schema is that the lib config requires a name key. - Modified the expected error code in some of the test cases. The error message is now issued by the JSON validator. - Added some validation code to the config script which checks the validity of the mbed_app.json file when it is initially loaded. - Added some validation code to config script which checks each of the mbed_lib.json scripts when they are loaded. - Removed manual checks for allowable config keys from within the mbed_app and mbed_lib json files. - Removed the check_dict_types() function which was no longer being called.

- Removed the config_path setting from schema and moved the addition of the config_path value after the validation is done. - Altered the macro validation regex to be more lenient. Now verifies that if '=' is used in the macro definition that something comes after it.

- Moved all common definitions from schema_app and schema_lib into a separate definitions file. - Changed the calling code to resolve multiple schema files correctly.

- Modified the expected error code in some of the test cases. The error message is now issued by the JSON validator. - Stop validation code being called if no application configuration is given to validate. - Change test mock code to check for specific calls, instead of number of calls. - Derive absolute paths for JSON schema files before loading them, which seems to work better with the URI path resolution of the schema parser.

cmonr · 2018-02-01T17:51:53Z

/morph build

mbed-ci · 2018-02-01T18:17:33Z

Build : SUCCESS

Build number : 1043
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/5022/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build

mbed-ci · 2018-02-01T20:13:22Z

Exporter Build : SUCCESS

Build number : 724
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/5022/

mbed-ci · 2018-02-01T21:01:50Z

Test : SUCCESS

Build number : 850
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/5022/850

theotherjimmy

I think the schema can be more restrictive, and therefore more helpful at pointing out errors.

theotherjimmy · 2018-02-01T21:19:50Z

tools/config/definitions.json

+    "type": "object",
+    "patternProperties": {
+      "\\*": {
+        "type": "object",


This should be the same as the pattern property below, "$ref": "#/target_override_entry"

theotherjimmy · 2018-02-01T21:20:37Z

tools/config/definitions.json

+  "target_override_entry": {
+    "type": "object",
+    "patternProperties": {
+      "^\\S+$": {}


This should contain "$ref": "#/config_parameter_short", I think

theotherjimmy

Let's get this in, and I'll submit another PR to make it more strict. @cmonr If you could do the honors

0xc0170 added the needs: review label Sep 5, 2017

theotherjimmy approved these changes Sep 5, 2017

View reviewed changes

theotherjimmy reviewed Sep 7, 2017

View reviewed changes

theotherjimmy added needs: work and removed needs: review labels Sep 7, 2017

theotherjimmy approved these changes Sep 28, 2017

View reviewed changes

adbridge added needs: review and removed needs: work labels Oct 16, 2017

cmonr added the needs: CI label Feb 1, 2018

0xc0170 removed the needs: work label Feb 1, 2018

0xc0170 added ready for merge and removed needs: CI labels Feb 1, 2018

Steven Cartmell added 6 commits February 1, 2018 17:18

Add JSON Schema library to requirements.txt

d208421

Refactor common JSON schema definitions into a seperate file

151f2fa

- Moved all common definitions from schema_app and schema_lib into a separate definitions file. - Changed the calling code to resolve multiple schema files correctly.

Remove placeholder $id from JSON schema and fix description

7010010

cmonr added needs: CI and removed ready for merge labels Feb 1, 2018

theotherjimmy suggested changes Feb 1, 2018

View reviewed changes

theotherjimmy approved these changes Feb 1, 2018

View reviewed changes

cmonr merged commit 860b66a into ARMmbed:master Feb 1, 2018

cmonr removed the needs: CI label Feb 1, 2018

cmonr mentioned this pull request Feb 1, 2018

Add support for the WISE1530 and integration framework for WICED SDK #5758

Merged

adbridge added the release-version: 5.8.0-rc1 label Feb 9, 2018

adbridge mentioned this pull request Feb 23, 2018

Python2+3: mbed compile, mbed test --compile, python unit tests #5848

Merged

1 task

cmonr mentioned this pull request May 11, 2018

Corrected iteritems py2/3 compatability in test_api.py #6886

Merged

ghost deleted the mbed-config-json-schema branch December 5, 2018 16:48

Add JSON schema based validation to mbed config script #5022

Add JSON schema based validation to mbed config script #5022

Uh oh!

Conversation

ghost commented Sep 5, 2017 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Status

Todos

Steps to test or reproduce

Uh oh!

0xc0170 commented Sep 5, 2017

Uh oh!

theotherjimmy commented Sep 5, 2017

Uh oh!

theotherjimmy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

theotherjimmy commented Sep 5, 2017

Uh oh!

theotherjimmy commented Sep 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Sep 6, 2017 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

theotherjimmy commented Sep 6, 2017

Uh oh!

theotherjimmy commented Sep 7, 2017

Uh oh!

theotherjimmy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mbed-ci commented Oct 10, 2017

Uh oh!

mbed-ci commented Oct 10, 2017

Uh oh!

adbridge commented Oct 16, 2017

Uh oh!

adbridge commented Oct 16, 2017

Uh oh!

ghost commented Oct 16, 2017

Uh oh!

mbed-ci commented Oct 16, 2017

Uh oh!

theotherjimmy commented Oct 23, 2017

Uh oh!

ghost commented Oct 23, 2017

Uh oh!

theotherjimmy commented Oct 23, 2017

Uh oh!

adbridge commented Oct 31, 2017

Uh oh!

sg- commented Oct 31, 2017

Uh oh!

theotherjimmy commented Oct 31, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0xc0170 commented Nov 22, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mbed-ci commented Nov 22, 2017

ghost commented Sep 5, 2017 •

edited by ghost

Loading

theotherjimmy commented Sep 5, 2017 •

edited

Loading

ghost commented Sep 6, 2017 •

edited by ghost

Loading

theotherjimmy commented Oct 31, 2017 •

edited

Loading

0xc0170 commented Nov 22, 2017 •

edited

Loading