Fix stop string matching for byte-fragment tokens

[Incheonkirin]

What does this PR do?

Fixes #46519.

This fixes a case where StopStringCriteria can miss stop strings with byte-level tokenizers. The full generated sequence decodes correctly, but some CJK characters are split into byte-fragment tokens, and decoding those tokens one by one can produce replacement characters instead. That means the text may visibly end with the stop string while the stopping criterion still does not match it.

This PR handles this during preprocessing by building the matching table in byte space for ByteLevel and SentencePiece ByteFallback tokenizers. Non-byte tokenizers keep the existing string path.

StopStringCriteria.__call__ is unchanged, so the runtime matching path stays tensor-only.

I ran:

PYTHONPATH=src python -m pytest tests/generation/test_stopping_criteria.py -q -p no:cacheprovider
ruff check src/transformers/generation/stopping_criteria.py tests/generation/test_stopping_criteria.py
ruff format --check src/transformers/generation/stopping_criteria.py tests/generation/test_stopping_criteria.py

Local result: 19 passed.

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

• I confirm that this is not a pure code agent PR.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? See StopStringCriteria misses CJK stop strings on byte-level tokenizers when a character splits into byte-fragment tokens #46519.
• Did you make sure to update the documentation with your changes? No documentation update is needed for this bug fix.
• Did you write any new necessary tests?

Who can review?

cc @Rocketknight1

[Rocketknight1]

Yes, this is nice - keeps the public API but fixes byte-level issues internally without breaking compilation. It should have no effect for non-byte-level tokenizers too. I don't see any major flaws and tests pass, so I'm happy to merge!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

CI Dashboard: View test results in Grafana

[Incheonkirin]

Thanks! Glad this landed — the byte-fragment case was subtle, and it’s good to have it covered now.