Add Level column to IMO-AnswerBench

[tlyleung]

Can the Level column (pre-IMO, IMO-easy, IMO-medium, IMO-hard) be added to IMO-AnswerBench, similar to how it is in IMO-ProofBench?

The paper lists the difficulty breakdown for each problem category in Table 2, but it would be useful to have it at a per-problem granularity.

[Pratik00531]

Query for Issue #4: Add Level column to IMO-AnswerBench

---

Hi @tlyleung and maintainers,

Thanks for opening this issue! I've been investigating how to add the Level column to IMO-AnswerBench and wanted to share my findings and request guidance.

What I've Found

Table 2 in the paper provides aggregate difficulty distributions:

Category	Pre-IMO	IMO-Easy	IMO-Medium	IMO-Hard	Total
Algebra	11	46	32	11	100
Combinatorics	4	19	31	46	100
Geometry	13	44	32	11	100
Number Theory	2	20	31	47	100

However, the answerbench.csv doesn't include per-problem difficulty labels, making it impossible to know which specific problem has which difficulty level.

Attempted Approaches & Why They Don't Work

Approach 1: IMO problem position heuristic

• Idea: Use IMO final exam position (P1/P4 → Easy, P2/P5 → Medium, P3/P6 → Hard)
• Problem: Most problems in the dataset are from "IMO Shortlist" or regional competitions (APMO, Vietnam, etc.), NOT final IMO exams
• Result: ❌ Not applicable to this dataset

Approach 2: Shortlist code classification

• Idea: Use Shortlist codes (A1/G1/N1/C1 → Easy, A2/G2/N2/C2 → Medium, A3/G3/N3/C3 → Hard)
• Problem: The Source column only shows "IMO Shortlist 2021" without the specific problem code (e.g., missing A1, A2, A3 suffixes)
• Result: ❌ Insufficient metadata in current dataset

Approach 3: Manual subjective assessment

• Problem: Difficulty labeling requires expert judgment and would likely not match the authors' original classification
• Result: ❌ Would create inconsistent/inaccurate labels

Questions

1. If labels aren't immediately available, what metadata could help?

   • Are there any additional metadata fields from the original curation process that could be shared?

2. Alternative approaches?

   • Are there any heuristics or guidelines the authors used for labeling problems from non-IMO sources (regional olympiads, etc.)?
   • Would a community-driven labeling effort be acceptable, or should we wait for the official labels?

Thanks for considering this request! Happy to help with implementation once guidance is provided.

---

References:

• Paper: Luong et al. (2025) "Towards Robust Mathematical Reasoning" - EMNLP 2025
• Website: https://imobench.github.io