Fix byte-level BPE tokenizers reusing SentencePiece classes (Llama 3 …#46686

kaixuanliu wants to merge 1 commit into

huggingface:mainhuggingface/transformers:mainfrom

kaixuanliu:tokenizer-fixkaixuanliu/transformers:tokenizer-fixCopy head branch name to clipboard