Fix regression in ProcessorMixin._load_tokenizer_from_pretrained for tokenizers at root

[punyamodi]

Summary

In transformers v5.x, a change was introduced to automatically look in subfolder directories named after the sub-processor attribute when loading additional/non-primary tokenizers (e.g. searching for files in bpe_tokenizer/ when the sub-processor name is bpe_tokenizer).

This regression breaks loading for older/existing model repositories where tokenizer files are placed at the root of the repository, but the sub-processor attribute name is configured to something else (for example, the UniversalActionProcessor of physical-intelligence/fast which uses bpe_tokenizer). When attempting to load such processors using AutoProcessor.from_pretrained(), it fails with a ValueError because it cannot locate files in the subfolder.

This PR wraps the subfolder loading in a try-except block. If loading from the subfolder fails, it gracefully logs a deprecation warning and falls back to loading from the root of the repository (or the passed subfolder directory).

Testing

Verified by loading physical-intelligence/fast with AutoProcessor.from_pretrained("physical-intelligence/fast", trust_remote_code=True) and confirming it successfully falls back to root and loads the processor.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a fallback path for loading an “additional tokenizer” when the expected subfolder load fails, while warning that this fallback behavior is deprecated.

Changes:

• Wrap from_pretrained(..., subfolder=tokenizer_subfolder) in a try/except and fall back to subfolder=subfolder on failure
• Emit a warning message indicating the fallback is deprecated

[github-actions]

CI Dashboard: View test results in Grafana

[Rocketknight1]

cc @ArthurZucker @itazap