fix: load published tokenizer in demo

[niieani]

Summary

• replace the demo layout with a tabbed prompt/chat experience backed by the new TokenInput component and chat playground
• add a theme provider, toggle, and updated surface styling to support light and dark modes across hero, cards, and resources
• refresh shared UI primitives, model insights, and copy to highlight gpt-5 defaults and npm/GitHub calls to action
• depend on the published gpt-tokenizer package for the demo so tokenizers load reliably

Testing

• CI=1 yarn build
• CI=1 yarn test
• cd demo && npm run build

---

https://chatgpt.com/codex/tasks/task_e_68e844c73bec83239c84b52f347b78e3

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting

• @codex fix this CI failure
• @codex address that feedback

[chatgpt-codex-connector]

Link local package without building it

The new Pages workflow still installs the demo with npm install .. --no-save after removing the steps that install dependencies and build the root package. Because the root package.json runs a prepare script that depends on Yarn and compiled artifacts, this command now fails in CI (the repository has no node_modules or compiled esm/cjs output). As a result the demo build cannot complete, so the workflow will not deploy. Restore the root install/build steps or drop the local link and rely on the published gpt-tokenizer package.

Useful? React with 👍 / 👎.

[github-actions]

🎉 This PR is included in version 3.3.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀