new models + add AGENTS

[niieani]

Summary

• clarify that the repo uses Yarn Berry with node_modules rather than Plug'n'Play
• note the postinstallDev Beemo scaffold step and how to rerun it manually

fixes #73

Testing

• not run

---

https://chatgpt.com/codex/tasks/task_e_68e82240e9608323af50b8ff81446210

[chatgpt-codex-connector]

💡 Codex Review

gpt-tokenizer/src/bpeRanks/o200k_base.js

Lines 1 to 3 in becff10

	// This file was generated by src/codegen/generateJsBpe.ts.
	// To regenerate, run: yarn codegen:bpe.
	// Source data: data/o200k_base.tiktoken.

Restore full o200k_base BPE rank data

The regenerated src/bpeRanks/o200k_base.js now consists only of three header comment lines and no exported BPE arrays. The previous version was ~2 MB of token data; after this change it is 142 bytes and no longer defines bpe or export default bpe. Any model that depends on the o200k_base encoding (e.g., GPT‑4.1 variants) will throw at import time or produce empty tokenization results. This file appears to have been truncated during the header insertion—please re-run the codegen and commit the full data.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting

• @codex fix this CI failure
• @codex address that feedback

[github-actions]

🎉 This PR is included in version 3.1.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀