Skip to content

CLAUDE.md

Latest commit

 

History

History
136 lines (96 loc) · 5.75 KB

CLAUDE.md

File metadata and controls

136 lines (96 loc) · 5.75 KB
Raw

Project Overview

ESSlivedata is a live data reduction visualization framework for the European Spallation Source (ESS). It processes real-time neutron detector data via Kafka streams and provides interactive dashboards for monitoring and data reduction workflows.

Terminology: src/ess/livedata/glossary.md (cross-cutting + backend) and src/ess/livedata/dashboard/glossary.md are the authoritative glossaries; consult them before naming new concepts.

Development Commands

Environment Setup

In the devcontainer, micromamba is auto-activated -- python, pytest, tox etc. just work.

Worktree Setup (when launched with claude -w):

python -m venv .venv && source .venv/bin/activate && pip install -e ".[test]"

Tests

  • Tests in tests/ mirror src/ess/livedata/ structure, files follow *_test.py
  • All unit tests run without Kafka -- use fakes from fakes.py
  • pytest uses --import-mode=importlib
  • Robustness harnesses: tests/helpers/hostile_wire.py is the corpus of malformed/insane wire payloads consumed by the adapter- and service-level robustness tests (add new corruption modes there). Known holes are strict xfails referencing their issue; fixing the issue flips them loudly.
  • pytest -m browser runs Playwright UI tests (fake backend, e.g. the multi-session smoke test); excluded from the default run, CI runs them via tox -e browser.
python -m pytest                      # fast tests only (~25s)
python -m pytest -m "not integration" # include @pytest.mark.slow (~85s)
tox                                   # CI: includes slow tests

Code Quality

Use the linter agent. Tools: ruff (primary, CI-enforced), pylint (optional), mypy (optional). Run tox -e static for all pre-commit checks.

Documentation

tox -e docs       # build HTML docs
tox -e linkcheck  # check links
# Manual: python -m sphinx -v -b html -d .tox/docs_doctrees docs html
# Doctest: python -m sphinx -v -b doctest -d .tox/docs_doctrees docs html

Project Template

Uses Copier with Scipp template. Config in .copier-answers.yml. Update: copier update.

Running Services Locally

Services require Kafka (docker-compose up kafka), which the devcontainer does not have. Use --dev for simplified topic structure. The fakes always publish to the dev topic names, so consuming services need --dev too, or they subscribe to production topics and see nothing.

Services: fake_monitors, fake_detectors, fake_logdata, monitor_data, detector_data, data_reduction, timeseries. Run as: python -m ess.livedata.services.<name> --instrument dummy [--dev], or the whole stack in one terminal with esslivedata-dev <instrument> (see the ess.livedata.scripts.dev docstring).

Dashboard: python -m ess.livedata.dashboard.reduction --instrument dummy

--transport none runs the UI without Kafka (default port 5009; pass --port for a second instance or to dodge a port a prior run still holds). With no backend, started workflows stay PENDING. --transport fake adds an in-process fake backend (no Kafka): started workflows go ACTIVE and plots fill with synthesized data.

To drive or screenshot the UI without Kafka, use scripts/drive_dashboard.py: --launch spawns a fake backend seeded from the dummy fixture, drives it, and tears it down; --map inventories the live tabs and lt-* automation hooks. For the library API, fixture seeding/regeneration, and automation gotchas (stable lt-* hooks, shadow-DOM selectors, modals), see the script's module docstring and "Driving the dashboard with Playwright" in .claude/rules/dashboard-widgets.md.

Tools

src/ess/livedata/nexus_helpers.py -- utilities for extracting Kafka topic and source names from NeXus files.

Architecture Overview

Core Abstractions

Message-driven service architecture:

  • Service: Top-level lifecycle manager running processors in a loop
  • Workflow: Scientific reduction logic processing accumulated data (instances run as Job)
  • MessageSource / MessageSink: Abstractions for consuming/publishing messages (e.g., Kafka)

Message Flow

Kafka Topics -> MessageSource -> OrchestratingProcessor -> Preprocessors -> JobManager -> Workflow -> MessageSink -> Kafka Topics

Key Components

  • core/: service.py (lifecycle), orchestrating_processor.py (job-based batching), preprocessor.py (preprocessor factory/protocols), command_dispatcher.py (dispatches commands to JobManagerAdapter), message.py (Message, StreamId, StreamKind), job_manager.py (scheduling)
  • kafka/: source.py (consumers), sink.py (producers), message_adapter.py (raw -> domain), stream_mapping.py (topic -> stream)
  • config/: YAML + Jinja2 configs. Defaults in config/defaults/, per-instrument in config/instruments/
  • preprocessors/: Preprocessor factories for detector, monitor, and reduction data; accumulators; workflow protocol
  • dashboard/: Panel/HoloViews visualizations, MVC pattern, DataService for data stream subscriptions

Code Style

  • Formatting: ruff (88 char lines)
  • Type hints: required; mypy used but not strictly enforced
  • Docstrings: NumPy format, no type annotations in params (sphinx-autodoc-typehints handles it)
def process(x: int) -> float:
    """Short description.

    Parameters
    ----------
    x:
        Description of x.

    Returns
    -------
    :
        Description of return value.
    """

Single-sentence: """Returns the number of dimensions."""

Instrument Support

Instruments registered in src/ess/livedata/config/instruments/: dummy, dream, bifrost, loki, odin, nmx, tbl, estia, beer. Optional deps installed as extras: pip install esslivedata[dream], etc.