v0.9.1

What's Changed

• Support o-series models, gpt-5, and add o200k_harmony tokenizer for gpt-oss in #123

Full Changelog: v0.7.0...v0.9.1

v0.7.0

What's Changed

• Remove need for Arc<Mutex<>> on singletons by @benbrandt in #107
• Update to latest tiktoken + new model support by @benbrandt in #106
• Dependency and build/CI updates

New Contributors

• @benbrandt made their first contribution in #107

Full Changelog: v0.6.0...v0.7.0

v0.6.0

Minor version release signifies a breaking change in 0.x

What's Changed

• Add support for chatgpt-4o-latest by @Congyuwang in #85
• Refactor internals to make future updates to tiktoken easier to merge
• Do not expose tiktoken internal modules and functions
• Update dependencies

New Contributors

• @Congyuwang made their first contribution in #85

Full Changelog: v0.5.9...v0.6.0

v0.5.9

What's Changed

• Add gpt-4-0125 and gpt-3.5-turbo-1106 context sizes by @mikaelsouza in #59
• Add gpt-3.5-turbo-0125 by @mikaelsouza in #60
• Update README.md by @imalsogreg in #61
• Add support: o200k_base tokenizer. by @CaffeeLake in #63

New Contributors

• @mikaelsouza made their first contribution in #59
• @imalsogreg made their first contribution in #61
• @CaffeeLake made their first contribution in #63

Full Changelog: v0.5.8...v0.5.9

v0.5.8

What's Changed

• Add tokenizer prefixes for fine-tuned models by @jbgriesner in #53

New Contributors

• @jbgriesner made their first contribution in #53

Full Changelog: v0.5.7...v0.5.8

v0.5.7

What's Changed

• Add gpt-4-1106-preview model by @JosephTLyons in #49

New Contributors

• @JosephTLyons made their first contribution in #49

Full Changelog: v0.5.6...v0.5.7

v0.5.6

What's Changed

• Expose _decode_native methods by @zurawiki in #43

Full Changelog: v0.5.5...v0.5.6

v0.5.5

What's Changed

• return correct context length for text-embedding-ada-002 by @ursachec in #41

New Contributors

• @ursachec made their first contribution in #41

Full Changelog: v0.5.4...v0.5.5

v0.5.4

Restore minimum supported Rust version (MSRV) to 1.61.0.

• No other functional changes

v0.5.3

Update dependencies

Full Changelog: v0.5.1...v0.5.3