RangeError: Invalid array length

[enijar]

Version: 2.6.2

Calling encode on a very large string of text (429,600,824 characters) throws this error:

RangeError: Invalid array length

This is how I'm using it:

import { encode } from "gpt-tokenizer";
console.log(encode("very long string of text").length);

Is there a limit to the string length that encode can handle?

I'm using this to calculate how many tokens my string is, to calculate OpenAI embeddings pricing

[niieani]

Hi @enijar! You seem to be bumping into a limitation of your JavaScript engine.
If you don't need the actual tokens, you can use either the generator version of the encode function to iteratively count the length of encoded values, while discarding the tokens, or, you can use isWithinTokenLimit with limit of Infinity to get the total number of tokens.

E.g.

import { isWithinTokenLimit } from "gpt-tokenizer";
console.log('token count is', isWithinTokenLimit("very long string of text", Infinity));

This should work on infinitely large pieces of text, but beware might take a while on such a large piece of text.

Also, if this is for estimating embedding cost, you should import the tokenizer for the model related to that embedding, e.g.

import { isWithinTokenLimit } from "gpt-tokenizer/model/text-embedding-3-large.js";
console.log('token count is', isWithinTokenLimit("very long string of text", Infinity));

I might add a helper method to just count tokens or even estimate cost.

[github-actions]

🎉 This issue has been resolved in version 2.7.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀