The token count result is inconsistent with the tiktoken result

[caozhiyuan]

[caozhiyuan]

@niieani The token count result is inconsistent with the tiktoken result

[niieani]

thanks for the report, I fixed it (the errors were in the RegExps), new version will be released soon.

[github-actions]

🎉 This issue has been resolved in version 3.3.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀