Add automated test suite (49 CI + 6 local tests, all passing)

New test files (tests/):
- conftest.py: shared fixtures, 'local'/'demo' markers; local tests
  auto-skip on GitHub CI unless PYFLEXTRKR_TEST_DATA env var is set
- test_steiner_func.py: unit tests for background_intensity,
  make_dilation_step_func, mod_steiner_classification, expand_conv_core
- test_echotop_func.py: unit tests for calc_cloud_boundary and
  echotop_height (1D and 3D height arrays)
- test_vertical_coordinate.py: unit tests for standardize_vertical_coordinate
  covering height/pressure units, scale_factor and units_override options
- test_idcells_synthetic.py: end-to-end integration test using a synthetic
  in-memory NetCDF file; no real data needed
- test_regression_local.py: local regression tests that validate full demo
  outputs (file structure, track counts, coordinate ranges, animations);
  uses recursive glob to handle startdate_enddate subdirectories
- README.md: pytest usage guide for new users

CI configuration:
- pytest.ini: test discovery and marker registration
- requirements-ci.txt: minimal pip-installable deps for CI (numpy, scipy,
  xarray, pandas, netcdf4, healpix, PyYAML, pytz, cftime, pytest);
  replaces requirements.txt + requirements-dev.txt in CI to avoid
  packages that are conda-only (cartopy, xesmf, ffmpeg, etc.)
- .github/workflows/test.yml: use requirements-ci.txt; CI installs only
  what tests need, making builds faster and more reliable

Bug fix:
- echotop_func.py: calc_cloud_boundary crashed on empty idxcld input;
  added early return and filter empty sublists from np.split

Author: feng045

.github/workflows/test.yml

@@ -28,9 +28,8 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install -r requirements.txt
+          pip install -r requirements-ci.txt
           pip install -e .
-          pip install pytest
 
       - name: Run Tests
         run: |

pyflextrkr/echotop_func.py

@@ -21,8 +21,14 @@ def calc_cloud_boundary(height, idxcld, gap, min_thick):
         Cloud-top height for each cloud layer.
     """
 
+    # Handle empty idxcld
+    if len(idxcld) == 0:
+        return np.zeros(0, dtype=np.float32), np.zeros(0, dtype=np.float32)
+
     # Split idxcld into layers
     Layers = np.split(idxcld, np.where(np.diff(idxcld) > gap)[0]+1)
+    # Filter out empty sublists that np.split can produce
+    Layers = [L for L in Layers if len(L) > 0]
     nLayers = len(Layers)
 
     # Create cloud_base, cloud_top arrays

pytest.ini

@@ -0,0 +1,13 @@
+[pytest]
+# Default test discovery path
+testpaths = tests
+
+# Show short summary for all failures/passes
+addopts = -ra -q
+
+# Register custom markers to suppress PytestUnknownMarkWarning
+markers =
+    local: Tests that require locally-mounted data (skipped on GitHub CI).
+           Set env variable PYFLEXTRKR_TEST_DATA to enable.
+    demo:  Full end-to-end demo run (slow, needs data and multiple workers).
+           Set env variable PYFLEXTRKR_TEST_DATA to enable.

requirements-ci.txt

@@ -0,0 +1,26 @@
+# Minimal dependencies for running CI tests (GitHub Actions).
+# This is intentionally a subset of requirements.txt — it contains only
+# what the tested pyflextrkr modules actually import.
+#
+# Full environment setup (local/HPC): use environment.yml with mamba/conda
+# Full dependency list: requirements.txt
+
+# Core scientific stack
+numpy
+scipy>=1.4
+xarray>=0.19
+pandas
+
+# NetCDF I/O (imported by pyflextrkr/netcdf_io.py)
+netcdf4
+
+# HEALPix (imported by pyflextrkr/hp_utilities.py)
+healpix
+
+# Utilities (imported by pyflextrkr/ft_utilities.py)
+PyYAML
+pytz
+cftime
+
+# Testing
+pytest>=7.0

tests/README.md

@@ -0,0 +1,179 @@
+# PyFLEXTRKR Automated Tests
+
+This directory contains automated tests for PyFLEXTRKR.  
+Tests are written with [pytest](https://docs.pytest.org/) and are split into two tiers:
+
+| Tier | Files | Runs on | Needs data? |
+|------|-------|---------|-------------|
+| Unit / synthetic | `test_steiner_func.py`, `test_echotop_func.py`, `test_vertical_coordinate.py`, `test_idcells_synthetic.py`, `test_example.py` | GitHub CI + local | No |
+| Local regression | `test_regression_local.py` | Local / HPC only | Yes (demo outputs) |
+
+---
+
+## Quick start
+
+Make sure pytest is installed (once per environment):
+```bash
+mamba install pytest        # or: pip install pytest
+```
+
+Run all CI-safe tests from the repo root:
+```bash
+cd /global/homes/f/feng045/program/PyFLEXTRKR-dev
+python -m pytest tests/ -v
+```
+
+You should see output like:
+```
+tests/test_echotop_func.py .......
+tests/test_steiner_func.py ..............
+tests/test_vertical_coordinate.py ...................
+...
+49 passed, 6 skipped
+```
+
+The 6 `s` (skipped) tests are the local regression tests — they are automatically
+skipped unless you set the `PYFLEXTRKR_TEST_DATA` environment variable (see below).
+
+---
+
+## Running a specific test file
+
+```bash
+python -m pytest tests/test_steiner_func.py -v
+python -m pytest tests/test_echotop_func.py -v
+python -m pytest tests/test_vertical_coordinate.py -v
+python -m pytest tests/test_idcells_synthetic.py -v
+```
+
+## Running a single test by name
+
+Use the `-k` flag with any part of the test name:
+```bash
+python -m pytest tests/ -v -k "convective"     # all tests with 'convective' in the name
+python -m pytest tests/ -v -k "echotop"
+python -m pytest tests/ -v -k "scale_factor"
+```
+
+## Stopping at the first failure
+
+```bash
+python -m pytest tests/ -v -x
+```
+
+## Show the full error traceback (more detail than default)
+
+```bash
+python -m pytest tests/ -v --tb=long
+```
+
+---
+
+## Understanding the output symbols
+
+| Symbol | Meaning |
+|--------|---------|
+| `.`    | Test passed |
+| `F`    | Test failed |
+| `s`    | Test skipped (e.g., local data not available) |
+| `E`    | Error during test setup (not a test failure itself) |
+
+---
+
+## Test files explained
+
+### `test_steiner_func.py`
+Unit tests for the Steiner convective/stratiform classification functions
+(`background_intensity`, `make_dilation_step_func`, `mod_steiner_classification`,
+`expand_conv_core`). Uses a small synthetic 50×50 km reflectivity field with one
+obvious convective core at the centre — no real radar data needed.
+
+### `test_echotop_func.py`
+Unit tests for echo-top height calculation (`calc_cloud_boundary`, `echotop_height`).
+Tests both 1D and 3D height coordinate inputs.
+
+### `test_vertical_coordinate.py`
+Unit tests for `standardize_vertical_coordinate()` — the function that converts
+vertical coordinates to standard units (metres for height, hPa for pressure).
+Covers: metres, kilometres, Pascals, hPa, `scale_factor` override, `units_override`,
+missing units, and error cases (geopotential height, unknown units).
+
+### `test_idcells_synthetic.py`
+End-to-end integration test. Creates a minimal synthetic NetCDF radar file in a
+temporary directory, then runs the full cell-identification pipeline:
+1. `get_composite_reflectivity_generic` (file reader)
+2. `mod_steiner_classification` (convective cell detection)
+3. `echotop_height` (echo-top calculation)
+4. `expand_conv_core` (cell labelling)
+
+Verifies that the known convective cell in the synthetic data is detected correctly.
+No real data needed.
+
+### `test_regression_local.py`
+Validates the output of full demo runs (e.g., `demo_cell_nexrad.sh`).
+These tests check that track statistics files contain reasonable values
+(correct geographic region, positive durations, finite coordinates) and that
+quicklook plots and animations were produced.
+
+**These tests are skipped automatically unless you set `PYFLEXTRKR_TEST_DATA`:**
+```bash
+# Point to the root directory of your demo output data
+export PYFLEXTRKR_TEST_DATA=/pscratch/sd/f/feng045/demo
+
+# Run the demo first, then validate its outputs:
+bash config/demo_cell_nexrad.sh
+python -m pytest tests/test_regression_local.py -v
+```
+
+---
+
+## Workflow: running tests after making code changes
+
+```bash
+# 1. Make your code changes
+# 2. Run all CI-safe tests — should take < 10 seconds
+python -m pytest tests/ -v
+
+# 3. If you have local demo data, run regression tests too
+export PYFLEXTRKR_TEST_DATA=/pscratch/sd/f/feng045/demo
+python -m pytest tests/ -v
+```
+
+If any test fails after your changes, the output will show exactly which assertion
+failed and what values were produced vs. expected, making it easy to diagnose
+whether the change broke existing behaviour or whether the test needs updating.
+
+---
+
+## Adding new tests
+
+Each test file follows the same pattern:
+
+```python
+import numpy as np
+import pytest
+from pyflextrkr.some_module import some_function
+
+def test_my_function_does_something():
+    # Arrange: set up inputs
+    my_input = np.array([1, 2, 3])
+    # Act: call the function
+    result = some_function(my_input)
+    # Assert: check the result
+    assert result == 6, f"Expected 6, got {result}"
+```
+
+Group related tests in a class for clarity:
+```python
+class TestSomeFunction:
+    def test_normal_case(self): ...
+    def test_edge_case(self): ...
+    def test_raises_on_bad_input(self): ...
+```
+
+To mark a test as local-only (skipped on CI):
+```python
+@pytest.mark.local
+def test_something_needing_data():
+    ...
+```

tests/conftest.py

@@ -0,0 +1,126 @@
+"""
+conftest.py – shared fixtures and custom pytest markers.
+
+Markers
+-------
+local   Tests that require locally-available data (HPC / workstation only).
+        Skip automatically on GitHub CI where no data is mounted.
+demo    Full end-to-end demo run (slow, needs data + multiple CPUs).
+
+Usage
+-----
+Run everything:           pytest tests/
+Skip local/demo tests:    pytest tests/ -m "not local and not demo"
+Run only local tests:     pytest tests/ -m local
+"""
+
+import os
+import numpy as np
+import pytest
+
+
+# ---------------------------------------------------------------------------
+# Register custom markers so pytest does not emit warnings
+# ---------------------------------------------------------------------------
+def pytest_configure(config):
+    config.addinivalue_line(
+        "markers",
+        "local: test requires locally mounted data (skip on GitHub CI)"
+    )
+    config.addinivalue_line(
+        "markers",
+        "demo: full end-to-end demo run (slow, needs data and multiple workers)"
+    )
+
+
+# ---------------------------------------------------------------------------
+# Auto-skip local/demo tests when no data root is found
+# ---------------------------------------------------------------------------
+DATA_ROOT_ENV = "PYFLEXTRKR_TEST_DATA"  # set this env-var on HPC/workstation
+
+
+def pytest_collection_modifyitems(config, items):
+    data_available = os.environ.get(DATA_ROOT_ENV) is not None
+    skip_local = pytest.mark.skip(
+        reason=f"Requires local data: set {DATA_ROOT_ENV} env variable to enable."
+    )
+    for item in items:
+        if "local" in item.keywords and not data_available:
+            item.add_marker(skip_local)
+        if "demo" in item.keywords and not data_available:
+            item.add_marker(skip_local)
+
+
+# ---------------------------------------------------------------------------
+# Shared synthetic-data fixtures (usable by all test files)
+# ---------------------------------------------------------------------------
+
+@pytest.fixture
+def grid_params():
+    """Standard small grid used across tests: 50x50 at 1 km resolution."""
+    return dict(nx=50, ny=50, dx=1000.0, dy=1000.0)
+
+
+@pytest.fixture
+def synthetic_refl_2d(grid_params):
+    """
+    Simple 2D composite reflectivity field with one obvious convective cell.
+
+    Layout (50x50 grid, 1 km spacing):
+    - Background: 20 dBZ everywhere
+    - Stratiform ring (r=5..10 km around centre): 30 dBZ
+    - Convective core (r<5 km around centre):     50 dBZ
+    """
+    nx, ny = grid_params['nx'], grid_params['ny']
+    cx, cy = nx // 2, ny // 2
+
+    yy, xx = np.ogrid[:ny, :nx]
+    r = np.sqrt((xx - cx) ** 2 + (yy - cy) ** 2)
+
+    refl = np.full((ny, nx), 20.0, dtype=np.float32)
+    refl[(r >= 5) & (r < 10)] = 30.0
+    refl[r < 5] = 50.0
+    return refl
+
+
+@pytest.fixture
+def mask_goodvalues(grid_params):
+    """All-valid mask matching the default 50x50 grid."""
+    return np.ones((grid_params['ny'], grid_params['nx']), dtype=int)
+
+
+@pytest.fixture
+def synthetic_dbz3d(grid_params):
+    """
+    Minimal 3D reflectivity DataArray [z, y, x] for echo-top tests.
+
+    10 height levels (0..9 km); convective column above the grid centre
+    has 50 dBZ up to 8 km, 0 dBZ elsewhere.
+    """
+    import xarray as xr
+
+    nx, ny = grid_params['nx'], grid_params['ny']
+    nz = 10
+    height_1d = np.arange(nz, dtype=np.float32) * 1000.0  # 0..9 km in m
+    cx, cy = nx // 2, ny // 2
+
+    data = np.zeros((nz, ny, nx), dtype=np.float32)
+    # Convective column: 50 dBZ below 8 km
+    data[:8, cy - 2:cy + 3, cx - 2:cx + 3] = 50.0
+
+    da = xr.DataArray(
+        data,
+        dims=['z', 'y', 'x'],
+        coords={'z': height_1d},
+    )
+    return da, height_1d
+
+
+@pytest.fixture
+def steiner_types():
+    return {
+        'NO_SURF_ECHO': 1,
+        'WEAK_ECHO': 2,
+        'STRATIFORM': 3,
+        'CONVECTIVE': 4,
+    }