RAIL-BENCH Odometry

This is the official toolbox for the Odometry challenge of RAIL-BENCH, the world's first perception benchmark suite for the railway.

For the other challenges of RAIL-BENCH (Rail, Object, Vegetation, and Tracking), check out the main RAIL-BENCH toolkit and our website.

In this Readme you can find the following information:

1. Available Tools: Overview over the available tools.

2. Further Details: All information that helps you to set up your prediction files and explains further on how to use the dashboard

3. Development Setup: How to set up a python environment and install all required packages.

1 Available Tools:

This toolkit offers you the following tools:

1. Analysis Dashboard: Visualize your predictions, compare them to the ground truth and run the evaluation.

2. Standalone Evaluation: Run the evaluation without the dashboard.

1.1 Analysis Dashboard

An analysis dashboard, implemented using Dash, can be run locally to quickly evaluate the performance of visual odometry tools. It can be used independently of a specific dataset or ground-truth data, but it is optimized for use with the RAIL-BENCH Odometry dataset.

To use the dashboard, the dependencies must first be installed. Follow the installation guide to activate a virtual environment and install all required dependencies. After navigating to the src directory using cd src, the dashboard can be started on a local server with:

python run_dashboard.py

The main features of the dashboard include:

• Support for different file formats
• Overview and metrics plots (including ATE and RTE)
• Various alignment options

Default settings can be configured in the defaults.json file so that commonly used options do not need to be set repeatedly in the dashboard (see the Default Settings section for more details).

1.2 Standalone Evaluation

In addition to the dashboard, predictions can be evaluated directly from the command line without launching the dashboard. After navigating to the src directory using cd src, run:

python run_evaluate.py <gt_path> <pred_path>

Argument	Description
gt_path	Directory containing ground-truth parquet files (e.g. scene_*_train.parquet)
pred_path	Directory containing prediction CSV files (e.g. scene_*_train_est.csv)

The script prints a formatted summary table with per-scene and aggregate RTE / ATE statistics. Scenes for which no matching prediction file is found are skipped with a warning.

The evaluate_odometry function can also be called directly from Python:

from analysis.evaluate import evaluate_odometry

scores = evaluate_odometry("path/to/gt", "path/to/predictions")
# scores is a dict with keys like "scene_31_test_rte_rmse", "rte_mean", ...

1.3 Format Check

If you want to submit predictions to the offical RAIL-BENCH challenge, we advise you to run a format check on the folder with your prediction files.

Your 20 prediction files should be named: scene_31_test_est.csv, scene_32_test_est.csv, ..., scene_50_test_est.csv. Each row in a CSV file contains the 16 entries of a 4 × 4 homogeneous transformation matrix. See 2.1 Pose Details for further details.

After navigating to the src directory using cd src, run:

python check_formatting.py <pred_path>

Argument	Description
pred_path	Directory containing prediction CSV files (scene_*_test_est.csv)

2 Further Details

2.1 Pose Details

The program assumes that poses are provided as a list of homogeneous transformations describing the camera position in world coordinates. The dashboard supports multiple ways of loading poses (see the File Formats section for details).

The camera coordinate system follows these conventions:

• z-axis pointing upward relative to the camera
• x-axis pointing in the viewing direction of the camera
• y-axis pointing to the right of the camera (resulting in a right-handed coordinate system)

2.2 File Formats

The dashboard supports three different file formats:

• .parquet format: Ground-truth trajectories are provided as parquet files. The loader expects these files to contain translation columns, rotation quaternions, and a time step in order to correctly load the data.
• .csv format: In this format, poses are stored by flattening homogeneous transformation matrices. Each row in the CSV file contains the 16 entries of a 4 × 4 homogeneous transformation matrix.
• TUM format: The dashboard supports the widely used TUM trajectory format. More details can be found here.

Additional loaders can be implemented by inheriting from the LoadFuncBase base class and registering the class with the dashboard.

2.3 Alignment Methods

Alignment between the estimated and ground-truth trajectories is necessary because:

1. the estimator cannot estimate the scale (in classical monocular visual odometry), and
2. the estimator does not know the initial orientation of the camera, or more precisely, which coordinate frame is used by the ground truth.

For these reasons, the dashboard provides multiple alignment options that can be selected to analyze the estimator's performance under different alignment assumptions.

Note: Additional alignment methods are implemented (e.g., SE(3) alignment or approximations of Sim(3) alignment using only the first few frames) and are available in the alignment.py file, but they are not exposed in the dashboard by default.

First Frame Alignment

In first frame alignment, all estimated poses are aligned such that the first estimated pose is identical to the first ground-truth pose. This option is selected by default. Drift and scale are not corrected with this option.

First Frame + Scale Alignment

When first frame + scale alignment is selected, the estimated trajectory is first aligned to the first frame, after which the scale is corrected.

The scale correction is applied by multiplying the translational components of the homogeneous transformation matrices by a scaling factor. This factor is computed as the ratio between the total Euclidean distance of the ground-truth trajectory and the total Euclidean distance of the estimated trajectory.

Umeyama Alignment

When this option is selected, the estimated poses $p_e$ are aligned to the ground-truth poses $p_{gt}$ by optimizing a rotation $R$, translation $t$, and scale $s$ such that the difference between the ground-truth poses and the aligned poses $p_a$ is minimized. The aligned poses are computed as

$$ p_a = s R p_e + t $$

2.4 Default Settings

The defaults.json file allows configuration of default options for the dashboard. A restart of the dashboard may be required for changes to take effect.

The following options are available:

Option	Type	Explanation	Default
primary_axis	list[int] (length = 2)	Defines the primary axes of the dataset, i.e., the axes along which the largest motion is expected.	[0,1], corresponding to the ground plane
show_tooltips	bool	Determines whether tooltips are displayed when hovering over certain elements.	false
prefix_to_delta_ts	dict[str, float]	Defines the expected time difference between consecutive frames. This value is used to compute velocity information for trips. The dictionary key corresponds to the filename prefix.	0.1 for 2021 WODATO files, 0.2 for 2022 WODATO files
directory	str	Defined separately for ground truth and estimations. Specifies the default directory used in the loading interface.	Paths to local data directories
suffix	str	Defined separately for ground truth and estimations. Allows filtering files in the directories by a specific suffix.	No filtering for ground truth, "_est" for estimations
loader_id	int	Defined separately for ground truth and estimations. Specifies which loading function is selected by default in the loading interface.	.parquet for ground truth and .csv for estimations

3 Development Setup

3.1 General Notes

This project uses pre-commit hooks to enforce code quality and consistency:

1. Ruff (ruff-check): Lints Python files and automatically fixes issues where possible.
2. MyPy (mypy): Performs static type checking on Python files. Missing type stubs are automatically installed using the --install-types option.
3. Nbstripout (nbstripout): Automatically removes all output cells from Jupyter notebooks when committing. Caution: Any executed output in notebooks will be removed before the commit is created.

3.2 Installation

(For Linux systems)

Create and activate a conda environment (note that the project was developed and tested using Python 3.14):

conda create -n odometry_env python=3.14
conda activate odometry_env

Install dependencies:

pip install -r requirements.txt

Enable pre-commit hooks for linting and type checking:

pre-commit install