[Tracking]: Overhaul CI setup

[ndelangen]

Problem statement

Our CI setup is in need of optimization to save cost and time.

The current setup has many convergent points, which make the runtime long, and cause failures of a part of the system to stop running the remaining tasks for other parts of the system.

We've identified multiple key issues we want to address:

• When one sandbox fails, it blocks all following steps
  • Pre-release sandboxes (especially Next.js) frequently break CI
  • Unrelated failures block work in other areas
• High flakiness across all test types
• Chromatic UI test issues
  • Changes often merged without accepting/denying them
  • Chromatic checks are yellow, not red (non-blocking)
• Package benchmarking often shows incorrect baseline comparisons
• Sandbox generation challenges
  • 24-hour generation cycle creates lag in detecting upstream issues
  • Generation process takes 30-40 minutes, nearly hitting GitHub timeout limits
• CI split between GitHub Actions and CircleCI creates confusion

Here is a list of solutions we want to complete:

• Use CircleCI dynamic config generation to create individual jobs per sandbox
  • Prevents one sandbox failure from blocking others
  • Each sandbox would have its own set of steps (unit tests, stories tests, etc.)
• Implement smart change detection
  • Only run affected sandboxes based on changed files
  • Different rules for core changes vs. framework-specific changes
• Disable NX cache in CI (not bulletproof, causing issues)
• Make Chromatic fail on changes (red instead of yellow)
• Fix package size benchmarking

These tasks will be completed in distinct milestones described below

Milestones

M1: Introduce a dynamic CI config step

Owner: @ndelangen
Complete by: 2025-11-7
PR: #32925

• Add a setup workflow
• Ensure support for existing labels passed to existing workflows

---

M2: Generate the config.generated.yml

Owner: @mention
Complete by: @date
PR: #32944

• Generate the existing workflow config using code
• Delete .circleci/src

---

M3: Generate single workflow

Owner: @mention
Complete by: @date

• Based on label, a singular-workflow config.generated.yml is created
• The created workflow runs selected sandboxes' pipelines in parallel
• Delete code for parallelism

---

M4: Windows & special cases

Owner: @mention
Complete by: @date

• Test at least 1 sandbox on Windows
• Test at least 1 sandbox with yarn pnp
• Test at least 1 sandbox with npm
• Test at least 1 sandbox with pnpm
• Test storybook for React Native (Web?)
• prereleases special handling (nextjs canary failing for example)

---

M5: Re-running compat & SSH

Owner: @mention
Complete by: @date

• Ensure re-running tasks can be done from failed
• Ensure SSH into runner is possible
• Setup rerun failed tests where possible

---

M6: Reporting

Owner: @mention
Complete by: @date

• Ensure size-reporting work
• Ensure bench reporting works
• Ensure size-reporting is concise, and accurate
• Ensure bundle analyzer links are correct
• Code coverage reporting on PR

---

M7: Use CircleCI App integration

Owner: @mention
Complete by: @date

• Enable the official app
• Setup github triggers for label changes
• Setup github triggers for pushes
• Setup circleci daily schedule running all sandboxes, even if nothing changed
• Remove custom github actions triggering CircleCI

---

M8: Simpler monorepo setup

Owner: @mention
Complete by: @date

• Upgrade NX
• Evaluate NX agents
• Merge code and scripts into yarn workspace
• Merge code and scripts into root
• Evaluate using https://rushjs.io/

---

M9: Reduce custom scripting

Owner: @mention
Complete by: @date

• Evaluate tool for running script across packages
• Watch-mode
• Remove custom script such as task and build

---

M10: Stop using yarn

Owner: @mention
Complete by: @date

• Switch from yarn to pnpm in monorepo
• Adjust all scripts to use pnpm npm instead of yarn
• Fix the sb link command

---

M11: Type bundling using rolldown

Owner: @mention
Complete by: @date

• replace rollup-plugin-dts with rolldown Build: Use rolldown instead of rollup for dts generation #32823
• replace esbuild with with tsdown
• add CI check for type bundles Sandboxes: Typecheck check production storybook packages #32447

---

M12: Change detection

Owner: @mention
Complete by: @date

• Determine target branch
• Automatically detect which sandboxes / tasks to run
• Generate config matching change
• Remove labels ci:*

---

M13: Move sandbox generation to scheduled CircleCI job

Owner: @mention
Complete by: @date

• Make separate repos for sandboxes
• Adjust sandbox scripts to clone/download correct repo
• Run schedule task to re-generate all sandboxes, and commit changes to individual repositories

---

M14: Use Changesets for releases

Owner: @mention
Complete by: @date

• Setup changesets Github App
• Setup PR release App
• Write educational material for changesets usage
• Remove custom release workflow/scripts
• Add storybook repo to list https://github.com/changesets/changesets

---