Improve ROM scan speed with two-phase pipeline and IGDB optimizations

[Praeses0]

Summary

• Split scan into discovery → enrichment phases: ROMs appear in the library within seconds (~14 roms/s), metadata fills in progressively afterward. Previously, each ROM blocked on IGDB API calls before appearing.
• Fix IGDB rate limiter lock-during-sleep bug: The token bucket was sleeping while holding the asyncio lock, serializing all concurrent IGDB requests to ~1 req/s instead of the intended 4 req/s.
• Reduce IGDB API calls per ROM: Merge the two-phase filtered/unfiltered search into a single API call with local game_type preference. Eliminates redundant expanded searches. Worst case drops from 6 to 3 calls per ROM.
• Add per-scan search term dedup cache: Avoids duplicate IGDB API calls when multiple ROMs normalize to the same search term (e.g. regional variants).
• Show scan phase in UI: Frontend now displays "Discovering" vs "Fetching metadata" phase indicator in both the scan page and admin task progress panel.
• Increase default SCAN_WORKERS from 1 to 10: Better I/O overlap between hash computation, DB writes, and metadata API calls.

Benchmark Results

Tested with 1,469 ROMs (1,100 Game Boy + 368 Game Gear + 1 Switch) using IGDB metadata:

Metric	Before	After	Improvement
ROMs visible in library	~29 min	36 seconds	~50x
End-to-end scan time	~29 min	~9.5 min	~3x
IGDB metadata throughput	0.83 roms/s	2.72 roms/s	3.3x

The hard ceiling is IGDB's 4 req/s rate limit — these optimizations maximize throughput within that constraint.

Test plan

• All 714 tests pass (713 existing + 1 new, only pre-existing Docker root-user CHD test fails)
• New regression test for HASHES scan type persisting ROM hash fields
• New unit tests for IGDB search cache semantics (positive hit cached, failures not cached, cache clear)
• Benchmarked on 51-ROM and 1,469-ROM sets
• trunk fmt and trunk check pass
• Verified scan UI shows phase indicator during both discovery and enrichment

AI Disclosure

This PR was authored with the assistance of Claude Code (AI).

[Copilot]

Pull request overview

This PR improves perceived and end-to-end ROM scan performance by splitting scanning into a fast “discovery” phase (DB + hashes) followed by an async “enrichment” phase (metadata + assets), while also optimizing IGDB request throughput and de-duplicating IGDB searches. It also surfaces scan phase in the UI and increases concurrency defaults.

Changes:

• Backend: introduce discovery→enrichment scan pipeline, add scan phase to stats, improve IGDB search efficiency + per-scan dedup cache, and fix IGDB rate limiting concurrency.
• Frontend: display “Discovering” vs “Fetching metadata” phase indicator in scan UI and admin task progress.
• Config/tests: bump default scan worker concurrency and add regression/unit tests for hashes persistence and IGDB search cache behavior.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
frontend/src/views/Scan.vue	Shows scan phase chip in the scan stats footer.
frontend/src/stores/scanning.ts	Extends scan stats type to include scan_phase from backend.
frontend/src/locales/en_US/scan.json	Adds English strings for the new scan phase labels.
frontend/src/locales/en_GB/scan.json	Adds British English strings for the new scan phase labels.
frontend/src/components/Settings/Administration/tasks/ScanTaskProgress.vue	Adds scan phase chip to admin task progress UI with i18n labels.
backend/tests/handler/metadata/test_igdb_handler.py	New unit tests for IGDB per-scan search cache semantics.
backend/tests/endpoints/sockets/test_scan.py	New regression test ensuring HASHES scan persists ROM-level hash fields.
backend/handler/metadata/igdb_handler.py	Adds per-scan search dedup cache and reduces IGDB calls by preferring game types locally.
backend/endpoints/sockets/scan.py	Implements two-phase scanning, adds scan_phase to stats, and clears IGDB search cache per scan.
backend/config/init.py	Increases default SCAN_WORKERS from 1 to 10.
backend/adapters/services/igdb.py	Adds a token-bucket rate limiter and supports offset in IGDB requests.

Comments suppressed due to low confidence (1)

backend/endpoints/sockets/scan.py:299

• In _discover_rom(), calculate_hashes is derived from SKIP_HASH_CALCULATION even for ScanType.HASHES (line 293). If a user sets SKIP_HASH_CALCULATION=true, a HASHES scan will still proceed but fs_rom_handler.get_rom_files(..., calculate_hashes=False) will not compute hashes, so the “recalculate hashes” scan type can’t fulfill its purpose. Consider forcing calculate_hashes=True when scan_type == ScanType.HASHES, or explicitly disabling HASHES scans when hashes are globally skipped.

    if should_update_files:
        calculate_hashes = not cm.get_config().SKIP_HASH_CALCULATION
        if calculate_hashes:
            log.debug(f"Calculating file hashes for {rom.fs_name}...")

        parsed_rom_files = await fs_rom_handler.get_rom_files(
            rom, calculate_hashes=calculate_hashes
        )

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

New i18n keys phase-discovering / phase-enriching were added only to en_US (and en_GB). The repo has frontend/src/locales/check_i18n_locales.py, which enforces that every locale’s scan.json includes the same keys as en_US; missing these keys will fail that check (and will show raw key strings at runtime for non-English locales). Please add these keys to all other locale scan.json files (even if the value is temporarily English) to keep locales in sync.

[Copilot]

The shared module-level _igdb_rate_limiter holds an asyncio.Lock() created on first use. If the process ever runs IGDB requests across different event loops (common in test suites with multiple loops, or in app reload scenarios), reusing an asyncio primitive across loops can raise “attached to a different loop” errors. Consider making the limiter loop-local (e.g., store one per running loop), or avoid asyncio.Lock in a global singleton by using an anyio limiter or a pure-time-based atomic approach. Also prefer asyncio.get_running_loop().time() over get_event_loop() in async code for 3.13+ compatibility.