Per-tenant logging

[MadLittleMods]

Per-tenant logging (exploration)

Tracked internally by https://github.com/element-hq/synapse-small-hosts/issues/48

Dev notes

1. Just set the root log level to logging.DEBUG so that everything is logged
   1. As an optimization, we can instead, take the list of logger levels from the tenant logging config and apply with incremental (we want the superset)
2. Add a handler that puts all of the log entries in a queue (logging.handlers.QueueHandler(queue))
3. For each tenant, spawn a separate listener process which loads the full log config (logging.config.dictConfig(log_config)) and handles logs
   • Also see QueueListener

---

Luckily, the Python docs have a section for what we're trying to do:

For the purposes of illustration, say that you have different web applications, each independent of the other but running in the same Python process and using a library common to them. How can each of these applications have their own log, where all logging messages from the library (and other request processing code) are directed to the appropriate application’s log file, while including in the log additional contextual information such as client IP, HTTP request method and client username?

-- https://docs.python.org/3/howto/logging-cookbook.html#use-of-contextvars

But this doesn't cover using different log levels across applications. Maybe we could log to separate place that has the loggers configured. Perhaps we can add a filter that can take in a log config and filter from there.

Logging to a single file from multiple processes

Although logging is thread-safe, and logging to a single file from multiple threads in a single process is supported, logging to a single file from multiple processes is not supported, because there is no standard way to serialize access to a single file across multiple processes in Python. If you need to log to a single file from multiple processes, one way of doing this is to have all the processes log to a SocketHandler, and have a separate process which implements a socket server which reads from the socket and logs to file. (If you prefer, you can dedicate one thread in one of the existing processes to perform this function.) This section documents this approach in more detail and includes a working socket receiver which can be used as a starting point for you to adapt in your own applications.

You could also write your own handler which uses the Lock class from the multiprocessing module to serialize access to the file from your processes. The stdlib FileHandler and subclasses do not make use of multiprocessing.

Alternatively, you can use a Queue and a QueueHandler to send all logging events to one of the processes in your multi-process application. The following example script demonstrates how you can do this; in the example a separate listener process listens for events sent by other processes and logs them according to its own logging configuration. Although the example only demonstrates one way of doing it (for example, you may want to use a listener thread rather than a separate listener process – the implementation would be analogous) it does allow for completely different logging configurations for the listener and the other processes in your application, and can be used as the basis for code meeting your own specific requirements:

-- https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes

Or maybe we can ...

Using a context manager for selective logging

There are times when it would be useful to temporarily change the logging configuration and revert it back after doing something. For this, a context manager is the most obvious way of saving and restoring the logging context. Here is a simple example of such a context manager, which allows you to optionally change the logging level and add a logging handler purely in the scope of the context manager:

-- https://docs.python.org/3/howto/logging-cookbook.html#using-a-context-manager-for-selective-logging

---

• https://docs.python.org/3/library/logging.html
• https://docs.python.org/3/library/logging.config.html
• https://docs.python.org/3/library/logging.config.html#logging-config-dictschema

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Code style is correct (run the linters)