MMLU Redux

[clefourrier]

Getting (on MMLU-Redux-2, vs MMLU-Redux reported in the Qwen3 report)

• Qwen 3-0.6B: 46.11 vs 51.2:
  vllm "model_name=Qwen/Qwen3-0.6B-Base,data_parallel_size=2,max_num_batched_tokens=100000,generation_parameters={temperature:0.6,top_p:0.95,top_k:20,min_p:0,presence_penalty:1,max_new_tokens:38912}" "lighteval|mmlu_redux_2|5|0"
• google/gemma-3-1b-pt: 30.26 vs 25.99
  (same as above with "model_name=google/gemma-3-1b-pt,data_parallel_size=2,max_num_batched_tokens=2001,generation_parameters={temperature:0.6,top_p:0.95,top_k:20,min_p:0,presence_penalty:1,max_new_tokens:5},gpu_memory_utilization=0.8")

It's impossible to know if the implem is exactly the same so we're imo within range (for ex, unsure if the report uses a logprob or generative approach - they use 5 shots which are likely not the same as ours and we know it can lead to up to 3 points diff, etc)

Not adding tests atm as we'll want to do all evals at the same time I think, with a setup running faster than our current one

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.