Skip to content

Conversation

fsschneider
Copy link
Contributor

@fsschneider fsschneider commented Jun 25, 2025

This PR prepares for the upcoming v0.6.0 release of the benchmark. Depends on #874.

Specifically, it changes:

  • Rework Readme to be clearer and easier to understand.
  • Remove deprecated COMPETITON_RULES and CALL_FOR_SUBMISSION docs.
  • Backfill the changelog with the most relevant changes and highlight it in more documents.
  • Increment rules version to 0.6.0 and detail new rule changes:
    • Reduce studies from 5 to 3.
    • Remove held-out workloads.
    • Update API in documentation.
  • Integrate and describe the new version policy (Proposal: Unify Versioning for Codebase, Leaderboard, and Rules #870).
  • Minor formatting improvements.

Todos

There are a few more rule changes to detail. I also want to give the entire DOCUMENTATION.md a thorough review, focusing on readability and removing deprecated statements. The changes include, for example:

  • Remove mentions of model track?
  • Rewrite section on qualification set -> Phrase it as a recommended protocol for reporting partial results if limited resources are available. An AlgoPerf light version?
  • Update note on "how many submissions"
  • Update FAQs (for rolling leaderboard format)
  • Integrated important requirements from the COMPETITON_RULES doc (e.g., Apache 2.0 license, defining software dependencies, CLA/membership/trademark agreement?)
  • Mention that you can change the dropout value in the forward pass now.

I would also like to:

  • Re-organize the algorithms into a single directory. I have a first draft, but I also need to:
    • Fix the tests, that are currently failing due to reference_algorithms being moved.
    • Add a section in README.md about Algorithms (or Sample Submissions) with a short description of what they are.
    • Proper README.md within the algorithms/ directory to enumerate all subfolders and algorithms.
    • What to do with the prize-qualification-baseline? There is no longer such a thing, but we do want to keep "a baseline", right?
    • Clean-up README.md in the individual algorithms/ subfolders.

@fsschneider fsschneider self-assigned this Jun 25, 2025
Copy link

github-actions bot commented Jun 25, 2025

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@fsschneider fsschneider marked this pull request as ready for review August 27, 2025 13:39
@fsschneider fsschneider requested a review from a team as a code owner August 27, 2025 13:39
@priyakasimbeg
Copy link
Contributor

priyakasimbeg commented Aug 28, 2025

EDIT: Disregard below comment, I just noticed they are in the "algorithms" folder

~ Hi Frank, I like that you reorganized the submissions but I was wondering if the algoperf folder is the right place for them. There are a lot of logs in those folders that will inflate the size of the algoperf package ~

priyakasimbeg
priyakasimbeg previously approved these changes Aug 28, 2025
@priyakasimbeg priyakasimbeg merged commit 484e66c into mlcommons:dev Aug 30, 2025
15 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Aug 30, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants