Incorporating tokenizer_config.json in DJL HuggingFace Tokenizer

[Soha-Agarwal]

I'm working with the DJL HuggingFace Tokenizer implementation and I've noticed that while it uses tokenizer.json, it doesn't seem to read tokenizer_config.json. I'd like to ensure that all relevant configurations from tokenizer_config.json are applied to the tokenizer.

Current observations:

1. The DJL implementation primarily uses tokenizer.json for initialization.
2. Some configurations are set via a Map<String, String> options or through the Builder pattern.
   Questions:
3. What is the recommended approach to incorporate tokenizer_config.json configurations into the DJL HuggingFace Tokenizer?
4. Are there plans to extend the Java implementation to handle more configurations from tokenizer_config.json directly?
5. How can we ensure that settings like model_max_length, padding_side, truncation_side, etc. from tokenizer_config.json are properly applied?
   Any insights or recommendations on how to best approach this would be greatly appreciated. Thank you!"

[frankfliu]

@Soha-Agarwal
You raised a very good question. I think we should take care of tokenizer_config.json automatically,
The best place is: https://github.com/deepjavalibrary/djl/blob/master/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java#L133

Do you mind contribute to DJL to enhance this?

[Soha-Agarwal]

@frankfliu

When you say we should take care of tokenizer_config.json automatically, I want to clarify:
Looking at the code, it seems we should:

1. Parse tokenizer_config.json
2. Update the tokenizer.json string (by modifying the JsonObject) before sending it to Rust
3. This is because I don't see the Rust implementation handling tokenizer_config.json separately

Is this the correct approach? For example: Adding special tokens from tokenizer_config.json to the added_tokens array in tokenizer.json

Would this be the right way to incorporate tokenizer_config.json configurations?

[frankfliu]

How python code handle this? I didn't get a chance to look into it, but ideally, we should do the same.

[Soha-Agarwal]

I am trying to use HuggingFaceTokenizer for Qwen2.5-1.5B and see that below properties need to be handled in tokenizer_config.json

1. chat_template
2. additional_special_tokens
3. added_tokens_decoder
4. errors
5. tokenizer_class

Will work on another PR to support this.

[Soha-Agarwal]

Hi @frankfliu

I see the pull request is merged but not included in the releases. What timeline is followed for releases?

[frankfliu]

We have some issues with maven publishing due to sonatype migration to maven central. Will release new version after we sort it out.

[Soha-Agarwal]

Hi @frankfliu
I see there is a bug in the code where we prefer tokenizer_config.json properties over options, working on pull request to fix it.

[Soha-Agarwal]

Created a PR to fix the issue - #3785