Test Observer

Test Observer (TO) is a dashboard for viewing the results of tests run on different environments for a particular artefact. A user interested in testing an artefact (a deb, snap, charm or image) under different environments (particular machines or cloud setups) can use TO as means for storing, viewing and comparing results with previous runs or versions of an artefact. The last use case is particularly useful for catching regressions. Additionally, TO provides a mechanism to assign reviewers that can look at results and mark artefacts as approved or failed to gate updates. It is important to note that TO does not run the tests itself, but provides an API with which users can report the results to.

Certification currently deploys an instance of TO that they used for reviewing Stable Release Updates (SRUs). Other teams also use this instance for their tests. You can visit the frontend and view the API docs, although this currently requires Canonical VPN access. There's also a staging deployment of frontend and API that teams can use to test their integration.

Run locally

Docker Compose (Recommended for Development)

The easiest way to run the full Test Observer stack locally is using Docker Compose (installed preferably via the docker.io provided APT source):

# Start the complete stack (backend, frontend, database)
# Migrations and test data seeding happen automatically
docker compose up

# Start without test data seeding
SEED_DATA=false docker compose up

# Run in detached mode
docker compose up -d

# Stop the stack
docker compose down

# Rebuild containers after code changes
docker compose up --build

# Run tests
docker compose exec test-observer-api pytest

This will start:

• Backend API at http://localhost:30000 (with automatic migrations and seeding)
• Frontend at http://localhost:30001
• PostgreSQL database with persistent storage on port 5432
• SimpleSAMLPHP at http://localhost:8080 a local Identity Provider (IdP) for testing

Local authentication

Test Observer (TO) supports authentication through SAML SSO. On staging and production Ubuntu One acts as the Identity Provider (IdP). On development however, SimpleSAMLPHP is deployed alongside TO to act as an IdP. The setup is configures one user account with the username "certbot" and password "password". This configuration resides in backend/saml/simplesamlphp/authsources.php.

Individual Components

For component-specific development, see the backend and frontend documentation.

Production-like Environment

To run the whole system via Terraform, juju and charms simulating production and staging environments, use terraform

Observability

The Test Observer charm provides built-in observability through Prometheus metrics and Grafana dashboards:

• Prometheus Metrics: The charm exposes metrics on port 9000 via the metrics-endpoint relation using the prometheus_scrape interface
• Grafana Dashboard: Pre-configured Grafana dashboards at backend/charm/src/grafana_dashboards/ will be automatically provided via the grafana-dashboard relation

To integrate with monitoring tools:

# Relate to Prometheus for metrics collection
juju relate test-observer-api:metrics-endpoint prometheus

# Relate to Grafana for dashboard provisioning
juju relate test-observer-api:grafana-dashboard grafana

Charm integration tests

A charmed Test Observer deployment can be integration tested using charmcraft test (which itself internally uses a classically confined copy of spread bundled with charmcraft).

The integration tests are runnable either locally or using GitHub. Alternative test execution backends are configurable by adding new adhoc backends next to lxd-vm and localhost in the spread.yaml.

Run integration tests locally

Tests can be run locally with either charmcraft test lxd or charmcraft.spread lxd (the latter gives a bit more verbose log output).

charmcraft.spread -v lxd-vm

Jump into a spread provided shell environment

To debug the charm integration tests, you can jump into a spread provided environment for the test execution either before, instead of the test task, or after a test task has been executed (you could also combine this choosing to run only a specific test scenario).

# open a shell to the execution environment _before_ executing each test task 
charmcraft.spread -v -shell-before lxd-vm

# open a shell to the execution environment _instead of_ executing each test task 
charmcraft.spread -v -shell lxd-vm

# open a shell to the execution environment _after_ of executing each test task 
charmcraft.spread -v -shell-after lxd-vm

Copyright Information

Test Observer uses the REUSE tool to check copyright notices as part of the CI process.

Canonical's guidelines for copyright is that all source code files should have a header/notice for the copyright, and that "in all cases, the year in the copyright notice should be the year the project / file was released.". However, not all files are source code, and so this guideline doesn't always apply. Additionally, some files are either not text files or don't support comments (e.g. JSON), so these are handled in the REUSE.toml file.

We also include auto-generated files in the REUSE.toml, since these would otherwise have the copyright values overwritten. Charm libraries (those fetched by charmcraft fetch-libs) and package locks are included in REUSE.toml for this reason. Furthermore, we use REUSE.toml to tag everything under docs/, because the process of building the docs inserts the copyright notice on each page, but the raw source pages do not include the copyright info.