Standard schema for results, including errors

[trajano]

The schema provides a way of validating the input, but there should be a standard for the validation results as well.

[handrews]

EDIT: This is now tracking overall output formats, success or failure. See also #530 for more specifics about annotation output.

@trajano could you elaborate on this a bit more? Are you talking about input to a specific validator? Or input/results from some other system that you want to use validation with?

[trajano]

I was thinking that we should have something along the lines of a validation result structure akin to https://docs.oracle.com/javaee/7/api/javax/validation/ConstraintViolation.html

For example:

{
   "valid": true | false,
   "violations": [
      {
         "property-path" : "$/path/to[10]/bad/element",
         "invalid-value": "who's bad" | { "some": [ "complex", "object" ] }
         "rule-path": "$/path/in/schema/that/triggered/violation",
     } ,
      {
         "property-path" : "$/path/to[10]/bad/element",
         "invalid-value": "who's bad" | { "some": [ "complex", "object" ] }
         "rule-path": "$/path/in/schema/that/triggered/violation",
          // these may be optionally added but should not be part of the spec
         "message-template": "{0} is who is bad",
         "message": "who's bad is who is bad",
         "position": { "line": 52", "col": 10 }
     } 
    ]
}

[trajano]

That way each application need not develop their own validation error standard messages

[handrews]

@trajano I like the idea, in some form or another.

RFCs generally do not constrain implementations in this way, due to the wide variety of implementation languages, environments, and requirements such as performance vs usability. This idea helps interoperability among users of validators, rather than interoperability in schema processing and outcome. So I don't think this proposal belongs in the spec repo (at least not with its current scope).

I see two possible approaches:

1. Write a schema for the desired output and perhaps we can discuss it with implementation maintainers and publish it as a recommendation on the web site (which is a separate GitHub repo). Depending on how broadly it is adopted maybe there will be a clear need to formalize it further.

2. Approach it like the "application/problem+json" specification. That is an error-reporting format that is defined independent of the system producing the errors. You would obviously want specific schema error-oriented fields, and would not be going for the completely generic approach of that spec, but separating it encourages its use in implementations that can support it, without burdening more constrained implementations with support.

[erayd]

I really like this:

2. Approach it like the "application/problem+json" specification. That is an error-reporting format that is defined independent of the system producing the errors. You would obviously want specific schema error-oriented fields, and would not be going for the completely generic approach of that spec, but separating it encourages its use in implementations that can support it, without burdening more constrained implementations with support.

If it's attractive enough, I would certainly support it in the implementations I'm involved with.

[trajano]

I think application/violation+json as a MIME type sounds better. For those who may not know (like me) the thing before the +json is the MIME subtype.

[handrews]

@trajano @erayd I'm glad to see multiple people taking interest. I do not have time to focus on this right now, but here is what I would recommend:

• Don't worry about making it an actual media type right now. You may want to do that, or you may (for instance) just want to define a specific extension of application/problem+json that can be identified by a schema describing the validation error output. Then you do not need to go through RFC, but can instead just apply two existing technologies and see how that goes to start.
• File an issue to track this over in the json-schema-org.github.io repo. Storing your proposal there as extended documentation / best practices seems like the best starting point to me.
• I'm going to close this as I do not think it belongs in this repo at this time. But that is just me managing the repo scope, not a rejection of the idea at all.

[trajano]

Closing this issue since it is out of scope of this project.

[yurikhan]

I am interested in this. Subscribing for comments, even if only to know which other issue to track.

[handrews]

There have been a few other issues around error reporting and/or formal definitions of validation results, so I have made an "output" label to track these. I will re-open this and label it, at least until we figure out where all of these concerns should really live.

Somewhat related: I'm defining a "recommended" output format for Hyper-Schema, although it is not a strict requirement. It does seem useful, though, which is why I am re-considering whether a recommendation, if not requirement, might be in-scope. The situation in hyper-schema is a little different, though (the mandatory output of the process is much more complex than a boolean result, even ignoring error reporting).

[Anthropic]

@handrews should validation results be a vocabulary? I think it would be awesome to have a consistent format/structure/schema for responses that validation implementations can work toward to ensure compatibility to make transferring from one validator to another more seamless. Just a week ago I posted this issue on djv to support a similar, more detailed, error message format to ajv. I think if ajv and djv the two fastest validators have a similar format then it could act as a starting point for a predictable validation result schema.