🎞️ ci: Fix Playwright ffmpeg Install Hang and Cache the Download (#15065)

Every Playwright job spent a flat 90s on `npx playwright install ffmpeg`,
and none of them ended up with a usable ffmpeg.

The 2.3MB download finishes in under a second; extraction then hangs until
`timeout -k 10 90` reaps it (exit 124, masked by `continue-on-error`). That
is a Node 24.16.0 readable-stream change (nodejs/node#62557) colliding with
yauzl/fd-slicer never firing `close` after EOF, which hangs extract-zip.
It leaves a truncated `ffmpeg-linux` — 5,055,201 bytes against the zip's
declared 5,101,056, segfaulting on exec — and no INSTALLATION_COMPLETE
marker, so Playwright treated ffmpeg as uninstalled. `video: 'on-first-retry'`
has therefore never worked in CI, and every first retry of a flaky test died
in browserContext.newPage: exactly the failure the step existed to prevent.

Upstream fixed it in Playwright 1.60.0 (microsoft/playwright#40747) and Node
reverted it in 24.18.0 (nodejs/node#63834). Node 24.16.0 is pinned in 17
places including the Dockerfiles, so bump Playwright instead — it is a dev
dependency, and `^1.56.1` already permitted 1.62.1; only the lockfile pinned
it. Staying at or above 1.62.1 also avoids the tsconfig-resolution
regressions in 1.62.0.

Caching alone could not have fixed this: a cold cache still hangs, and what
would have been cached is the corrupt binary. So the ffmpeg download is now
restored from cache keyed on the resolved playwright-core version, the
install is skipped outright on a hit, and the cache is only saved once the
binary is verified to actually execute — a partial extraction can never be
promoted into a cache that every later job restores.

Per job: 90s to ~0s on a hit, ~2s on a miss.

Author: danny-avila

.github/scripts/verify-playwright-ffmpeg.sh

@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+#
+# Verifies that Playwright's ffmpeg download produced a usable binary.
+#
+# `playwright install` is not trustworthy on its own here: under the Node 24.16.0
+# yauzl/extract-zip regression (Playwright < 1.60.0) it would hang mid-extraction
+# and leave a truncated `ffmpeg-linux` behind with no INSTALLATION_COMPLETE marker,
+# so the exit code said nothing about whether ffmpeg actually worked.
+#
+# CI caches the download, so this runs before the cache is saved: checking both the
+# marker and that the binary actually executes is what keeps a partial extraction
+# from being promoted into a cache that every later job would restore. The install
+# directory is read back from Playwright so this stays correct across version bumps
+# and never lets a stale revision vouch for the one actually required.
+
+set -uo pipefail
+
+install_dir=$(npx playwright install --dry-run ffmpeg 2>/dev/null |
+  sed -n 's/^[[:space:]]*Install location:[[:space:]]*//p' | head -1)
+
+if [ -z "${install_dir}" ]; then
+  echo "::warning::Could not determine Playwright's ffmpeg install location; skipping cache save."
+  exit 1
+fi
+
+if [ ! -f "${install_dir}/INSTALLATION_COMPLETE" ]; then
+  echo "::warning::${install_dir} has no INSTALLATION_COMPLETE marker; the download did not finish."
+  exit 1
+fi
+
+binary="${install_dir}/ffmpeg-linux"
+
+if [ ! -x "${binary}" ]; then
+  echo "::warning::${binary} is missing or not executable."
+  exit 1
+fi
+
+if ! "${binary}" -version >/dev/null 2>&1; then
+  echo "::warning::${binary} is present but does not execute; treating it as a partial extraction."
+  exit 1
+fi
+
+echo "Verified Playwright ffmpeg at ${binary}"

.github/workflows/codegraph-e2e-votes.yml

@@ -172,11 +172,37 @@ jobs:
         run: google-chrome --version
 
       # ffmpeg for retry video — see the note in playwright-mock.yml.
-      - name: Install Playwright ffmpeg (best effort)
+      - name: Resolve Playwright version
+        id: playwright-version
+        if: steps.tiers.outputs.count != '0'
+        run: |
+          version=$(node -p "require('./package-lock.json').packages['node_modules/playwright-core'].version")
+          echo "version=${version}" >> "$GITHUB_OUTPUT"
+
+      - name: Restore Playwright ffmpeg cache
+        id: cache-ffmpeg
         if: steps.tiers.outputs.count != '0'
+        uses: actions/cache/restore@v5
+        with:
+          path: ~/.cache/ms-playwright
+          key: playwright-ffmpeg-${{ runner.os }}-${{ steps.playwright-version.outputs.version }}
+
+      - name: Install Playwright ffmpeg (best effort)
+        id: install-ffmpeg
+        if: steps.tiers.outputs.count != '0' && steps.cache-ffmpeg.outputs.cache-hit != 'true'
         timeout-minutes: 3
         continue-on-error: true
-        run: timeout -k 10 90 npx playwright install ffmpeg
+        run: |
+          timeout -k 10 60 npx playwright install ffmpeg
+          .github/scripts/verify-playwright-ffmpeg.sh
+
+      - name: Save Playwright ffmpeg cache
+        if: steps.tiers.outputs.count != '0' && steps.install-ffmpeg.outcome == 'success'
+        continue-on-error: true
+        uses: actions/cache/save@v5
+        with:
+          path: ~/.cache/ms-playwright
+          key: playwright-ffmpeg-${{ runner.os }}-${{ steps.playwright-version.outputs.version }}
 
       # Optional fonts only — see the note in playwright-mock.yml.
       - name: Install optional Playwright font dependencies (best effort)

.github/workflows/playwright-mock.yml

@@ -146,13 +146,48 @@ jobs:
 
       # `video: 'on-first-retry'` needs ffmpeg; without it the first retry dies in
       # browserContext.newPage before the test body runs, so a flaky test loses the
-      # retry that would have recovered it. The CLI can hang after the download
-      # finishes on these runners, so bound it and keep it non-fatal — worst case is
-      # today's behaviour of retrying without video.
+      # retry that would have recovered it.
+      #
+      # This step used to burn its full 90s bound on every job. Playwright's bundled
+      # extractor hangs on Node 24.16.0 (a yauzl/extract-zip regression fixed in
+      # Playwright 1.60.0): the 2.3MB download finished in under a second, then
+      # extraction stalled and the timeout reaped it, leaving a truncated binary and
+      # no INSTALLATION_COMPLETE marker — so ffmpeg was never actually installed and
+      # retries never got video. With Playwright bumped past the fix the install
+      # takes about a second, and a restored cache skips it outright.
+      #
+      # The cache is only saved once the binary is verified to run, so a partial
+      # extraction can never be promoted into a cache every later job restores.
+      # Kept non-fatal: retry video is a debugging aid, not something CI asserts on.
+      - name: Resolve Playwright version
+        id: playwright-version
+        run: |
+          version=$(node -p "require('./package-lock.json').packages['node_modules/playwright-core'].version")
+          echo "version=${version}" >> "$GITHUB_OUTPUT"
+
+      - name: Restore Playwright ffmpeg cache
+        id: cache-ffmpeg
+        uses: actions/cache/restore@v5
+        with:
+          path: ~/.cache/ms-playwright
+          key: playwright-ffmpeg-${{ runner.os }}-${{ steps.playwright-version.outputs.version }}
+
       - name: Install Playwright ffmpeg (best effort)
+        id: install-ffmpeg
+        if: steps.cache-ffmpeg.outputs.cache-hit != 'true'
         timeout-minutes: 3
         continue-on-error: true
-        run: timeout -k 10 90 npx playwright install ffmpeg
+        run: |
+          timeout -k 10 60 npx playwright install ffmpeg
+          .github/scripts/verify-playwright-ffmpeg.sh
+
+      - name: Save Playwright ffmpeg cache
+        if: steps.install-ffmpeg.outcome == 'success'
+        continue-on-error: true
+        uses: actions/cache/save@v5
+        with:
+          path: ~/.cache/ms-playwright
+          key: playwright-ffmpeg-${{ runner.os }}-${{ steps.playwright-version.outputs.version }}
 
       # The runner's Chrome is an apt package, so its real library dependencies are
       # already satisfied; all `install-deps` adds here are optional CJK/Thai/Cyrillic
@@ -246,10 +281,35 @@ jobs:
         run: google-chrome --version
 
       # ffmpeg for retry video — see the note in the e2e_shards job.
+      - name: Resolve Playwright version
+        id: playwright-version
+        run: |
+          version=$(node -p "require('./package-lock.json').packages['node_modules/playwright-core'].version")
+          echo "version=${version}" >> "$GITHUB_OUTPUT"
+
+      - name: Restore Playwright ffmpeg cache
+        id: cache-ffmpeg
+        uses: actions/cache/restore@v5
+        with:
+          path: ~/.cache/ms-playwright
+          key: playwright-ffmpeg-${{ runner.os }}-${{ steps.playwright-version.outputs.version }}
+
       - name: Install Playwright ffmpeg (best effort)
+        id: install-ffmpeg
+        if: steps.cache-ffmpeg.outputs.cache-hit != 'true'
         timeout-minutes: 3
         continue-on-error: true
-        run: timeout -k 10 90 npx playwright install ffmpeg
+        run: |
+          timeout -k 10 60 npx playwright install ffmpeg
+          .github/scripts/verify-playwright-ffmpeg.sh
+
+      - name: Save Playwright ffmpeg cache
+        if: steps.install-ffmpeg.outcome == 'success'
+        continue-on-error: true
+        uses: actions/cache/save@v5
+        with:
+          path: ~/.cache/ms-playwright
+          key: playwright-ffmpeg-${{ runner.os }}-${{ steps.playwright-version.outputs.version }}
 
       # This job deliberately skips the optional font install: its bounded
       # Playwright apt process can outlive the wrapper on a slow mirror and

package-lock.json

@@ -152,7 +152,7 @@
     "@eslint/compat": "^1.2.6",
     "@eslint/eslintrc": "^3.3.4",
     "@eslint/js": "^9.20.0",
-    "@playwright/test": "^1.56.1",
+    "@playwright/test": "^1.62.1",
     "@types/react-virtualized": "^9.22.0",
     "brace-expansion": "^2.1.2",
     "caniuse-lite": "^1.0.30001741",