MANGO Ingest

Monitoring and ANalysis for Gravity Operations - Data ingestion and API

Contribution

This repository is publicly readable, but in its current early stage of development is specialised to our internal use-case. As such, we are not accepting external contributions at this time.

Prerequisites

• a running TimescaleDB instance
• conda environment installed and activated
• environment variables set

  export TSDB_HOST='localhost';
  export TSDB_PORT='5432';
  export TSDB_USER='postgres';
  export TSDB_PASSWORD='password';
  export TSDB_DATABASE='masschange';
  
  # optionally, for prod deployment behind a proxy server
  export API_ROOT_PATH='/mango/api/'
  export API_PROXY_HOST= # populate with production API host

Staging Deployment (quick note)

New env vars have been created in the docker-compose to handle user mapping. These are MANGO_UID and MANGO_GID and should match the host-machine user with the necessary file access permissions for the db storage volume path.

In bash, these may be set as expected. On the staging server, we are currently limited to use of tcsh, which will require setenv <key> <value> syntax instead.

Docker Quickstart (in-progress - populate as part of ops server replacement deployment - edunn 20251002)

...

Docker Quickstart (OUTDATED)

1. Clone repository
2. Build image with docker build -t masschange path/to/repository
3. Ensure that input root directory exists and contains data
4. Ensure that output root exists and is globally-writable (this will be removed as a requirement at some point)
5. Run GRACE-FO sample data ingest with docker run -v path/to/input-data-root:/input -v path/to/output-data-root:/data masschange conda run -n masschange python /app/masschange/src/masschange/ingest/datasets/gracefo/ingest.py /input /data
6. Confirm presence of ingested data in output root directory

Dev Quickstart

[!IMPORTANT] If using Rancher Desktop to support dockerized deployment on a development machine, it is critical that the env vars specifying bind-mounted paths used for input and staging be located under allowed paths (ex. /tmp/rancher-desktop, under default settings) See: https://stackoverflow.com/a/78782207

Dependencies

• conda

1. Clone repository
2. Create conda env with conda env create --file ./environment.yml
3. Activate conda env with conda activate masschange
4. Install editable masschange package with pip install -e /app/masschange
5. Run ingestion on GRACE-FO data location with python ./masschange/ingest/datasets/gracefo/ingest.py --dataset GRACEFO_ACC1A --src path/to/input_data_root (add --zipped if data is in tarballs)

To update existing conda environment

1. Edit ./environment.yml
2. Activate conda env with conda activate masschange
3. Update env with conda env update --file ./environment.yml --prune

Tests

Ingest and dataset implementations

Run functional and unit tests from repository root with

PYTHONPATH='./src' && python -m unittest discover

A test database named masschange_functional_tests will be created, written to, and deleted during functional test execution.

API tests

API operational tests are provided by starlette and may be run from the repository root with

pytest path/to/src/masschange/api/tests