Skip to content

Releases: niieani/gpt-tokenizer

1.0.3

18 Apr 08:35
7b9e7fd

Choose a tag to compare

1.0.3 (2023-04-18)

Bug Fixes

1.0.2

18 Apr 08:16

Choose a tag to compare

1.0.2 (2023-04-18)

Bug Fixes

1.0.1

16 Apr 06:44

Choose a tag to compare

1.0.1 (2023-04-16)

Bug Fixes

1.0.0

16 Apr 06:35

Choose a tag to compare

1.0.0 (2023-04-16)

Bug Fixes

  • remove global cache memory leak (40a2985), closes #35

Features

  • add a way to count tokens without encoding the whole text (88c8590)
  • add decodeAsyncGenerator and support any iterable input (41175f4)