Search results

Is it support Chinese?

Status: Open.

#53 In sugarme/tokenizer;

· wubin1989 opened on Mar 6, 2025

Support Embedding??

Status: Open.

#51 In sugarme/tokenizer;

· ErfanMomeniii opened on Jan 11, 2025

When Support Unigram?

Status: Open.

#44 In sugarme/tokenizer;

· Doloxetine opened on Aug 22, 2024

regexp dont support ?i

Status: Open.

#43 In sugarme/tokenizer;

· xuxiaoxia96 opened on Aug 6, 2024

Compile to wasm target

Status: Open.

#35 In sugarme/tokenizer;

· tiero opened on Oct 12, 2023

OpenAI CLIP tokenization support?

Status: Open.

#34 In sugarme/tokenizer;

· kristofmaar opened on Sep 12, 2023

int64

enhancementNew feature or requestNew feature or request

Status: Open.

#15 In sugarme/tokenizer;

· sugarme opened on Oct 22, 2020

serialization

enhancementNew feature or requestNew feature or request

Status: Open.

#2 In sugarme/tokenizer;

· sugarme opened on Sep 20, 2020