Machine-readable config description

[V02460]

Adds a JSON Schema description of the Synapse configuration to the repo, together with a script generating docs/usage/config_documentation.md from it.

This has many advantages, one can:

• Validate Synapse configuration files against the schema.
• Keep configuration and its documentation closer together, avoiding de-synchronization.
• Keep a uniform documentation style.
• Represent enforceable constraints between config values.
• Choose to reduce the amount of hand-rolled config validation code.
• Have IDE support like keyword suggestion and linting when editing a Synapse configuration.

I was surprised how aligned the expressiveness of the schema file is with the current infos from the Synapse documentation. Besides added types and defaults, some whitespace differences, and minor changes to wording and ordering, the config documentation remains mostly unchanged. But still, it is no perfect match, with the most important restrictions being:

1. No native sections for config options (the script patches those in).
2. No comments in examples. Might be handled as part of the description.
3. No native JSON Schema keyword representing the “Added in Synapse <version>” annotations (added to the plain-text description).

I expect there will be some feedback to work into this MR before it can get merged. I’m especially thinking about CI integration, JSON Schema IDs, types and defaults, and discussions about workflow – as well as a good amount of fixes due to the size of the change. Looking forward to your comments :)

Example

Running the script

scripts-dev/gen_config_documentation.py schema/synapse-config.schema.yaml > docs/usage/configuration/config_documentation.md

on a schema containing the following fragment

"soft_file_limit": {
  "type": "integer",
  "description": "Set the soft limit on the number of file descriptors synapse can use. Zero is used to indicate synapse should set the soft limit to the hard limit.",
  "default": 0,
  "examples": [3]
}

yields the corresponding Synapse documentation as Markdown:

### `soft_file_limit`

*(integer)* Set the soft limit on the number of file descriptors synapse can
use. Zero is used to indicate synapse should set the soft limit to the hard
limit. Defaults to `0`.

Example configuration:
```yaml
soft_file_limit: 3
```

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Code style is correct
  (run the linters)

[CLAassistant]

All committers have signed the CLA.

[reivilibre]

I reviewed the diff of the config documentation and fixed up some accidental differences that I saw, as well as some other minor improvements (IMO).

Due to time constraints I have yet to deal with these issues:

• suspicious defaults
  • public_baseurl
  • databases Defaults to {}.
  • session_lifetime
  • refresh_token_lifetime
  • nonrefreshable_access_token_lifetime
  • room_list_publication_rules (should be "*"?)
  • alias_creation_rules (should be "*"?)

Due to a problem with my diff viewer and long lines (???), some descriptions were cut off whilst I was reading them and I have not yet re-reviewed those fragments:

• remote_media_download_burst_count
• remote_media_download_per_second
• media_retention
• For modern Android devices the notification con...
• Ensure the main process and all pusher workers are r...
• and it should be listed...
• This option defaults to off, enable it by provid...

I thought the section on 'resources' in the listeners section looked suspicious but did not thoroughly re-review yet.

---

Questions (aimed at the team):

• should we convert the JSONSchema document itself to YAML?
  That would allow writing comments as well as multiline strings.
  Currently the markdown strings inside the JSON document are a little bit awkward to edit, since they have to be single-line.

Thoughts:

• string|null might look nicer as string or null.
• I'd like to try avoiding throwing too much JSON in the reader's face:
  • 'Defaults to []' might look nicer as 'Defaults to an empty list'
  • 'Defaults to {}' might have a nicer alternative?
  • Maybe we should follow what we used to do and use the word 'none' instead of 'null'...?
  • 'list' instead of 'array'?
  • maybe there's a word for 'object' that would suit better
• In url_preview_url_blacklist and some others, it would probably be nice to preserve the comments in the example. We can probably add some io.element.yaml_example field to the schema with verbatim YAML for such cases, which should be fairly rare.

Things missing from this PR that I think are needed:

• some options are missing descriptions and this shows up as MISSING DESCRIPTION. We should add descriptions for those.
• The config documentation generation should be checked in CI — to make sure the schema is valid and that the doc doesn't fall out of sync by accidental manual edits
• missing extra docs on enable_authenticated_media (due to a concurrent PR)

Things missing from this PR that can probably wait:

• Synapse should enforce the JSONSchema validations on startup (both because this would be useful and because this will help ensure they don't fall out of sync)

---

Risks:

• if the schema is wrong and we start validating it in Synapse, we might break some servers' ability to start up

If we accepted this PR now but for some reason didn't like it later:

• We could regenerate the config documentation, then remove the schema and generator. We'd be back in the same place as now, before accepting the PR.

[anoadragon453]

A few minor things, but we're in the home stretch now!

[anoadragon453]

Happy to say this now LGTM! I double-checked the generated config documentation vs. the hand-written one and all options seem to be present.

Thanks again for your hard work on putting this together, and keeping it updated even through the long review cycles ❤️

[anoadragon453]

Note that merging this PR will require updates for any other currently-open PR that modifies the handcrafted configuration documentation. I'll do my best to spread the word around so contributors are aware.

Before merging, I'll add one more commit to this PR that adds a section to the documentation README explaining how the Configuration Manual is now built.