comparison groups

[sam0x17]

Overview

• Added opt-in comparison benchmark groups so multiple implementations can be benchmarked together and summarized side-by-side (CLI + HTML).
• Comparison summary now:
  • Ranks benchmarks (1st/2nd/3rd…) by the typical statistic (slope if available, otherwise mean) normalized to 1.00.
  • Colors fastest entries green and others red; highlights the "faster than next" percent in green.
  • Shows change vs previous baseline in the format [score_delta, % change] (score deltas of ~0.000 are neutral/uncolored).
• HTML summary report mirrors the CLI formatting and wording.
• Added a demo comparison bench (benches/benchmarks/comparison_demo.rs) with multiple Fibonacci implementations to exercise the feature.
• Documented the feature in book/src/user_guide/comparing_functions.md.

Rationale

• Users often want to compare several implementations of the same operation in a single run without manual diffing.
• The ranked summary focuses on the headline metric Criterion already uses (typical) and keeps the output concise and readable.
• Coloring and ordinals make it easy to see the winner, relative gaps, and baseline movement at a glance.

Usage

// Enable comparison mode
let mut group = c.comparison_benchmark_group("MyGroup");
// or:
// let mut group = c.benchmark_group("MyGroup");
// group.comparison();

// Add benchmarks as usual
group.bench_function("ImplA", |b| { /* ... */ });
group.bench_function("ImplB", |b| { /* ... */ });
group.finish();

Run your benches (optionally with baselines) as normal; the group summary will include the ranked comparison.

Notable behavior

• Only the typical statistic is shown in the comparison summary (consistent with Criterion’s primary headline metric).
• Percent “faster than next” is always positive and green; change vs baseline is colored for improvement/regression; zero deltas are neutral.
• CLI intro now reads “Higher is better; best performer is 1.00 (typical).”

Testing

• cargo test --workspace
• cargo bench --bench bench_main -- FibonacciComparison --noplot --color always
• cargo clippy --workspace --all-targets -- -D warnings

[sam0x17]

here is what it looks like:

[sam0x17]

updated to handle throughput better and fix a bug: