fix(zebra-state): start without backup instead of aborting on unusable backup dir

[natalieesk]

Motivation

Closes #10544.

NonFinalizedState::with_backup (zebra-state/src/service/non_finalized_state.rs) created the backup directory with create_dir_all(...).expect(...) inside a spawn_blocking. When the path already exists as a regular file, create_dir_all errors and the expect panics; under the workspace's panic = "abort" that aborts the process on every startup — a crash loop until the file is removed by hand.

Solution

Probe the path (create_dir_all then read_dir) out of the blocking task and, on error, log a clear actionable message and start without the non-finalized backup, routing to the existing no-backup path. The backup is a re-downloadable resilience cache, not consensus data, so degrading is preferable to a crash loop. read_dir is also probed because create_dir_all succeeds for a directory that already exists but is unreadable, which would otherwise abort deeper in restore_backup.

A graceful hard-fail instead would have to thread Result through the public zebra_state::init API and all its callers — a much larger change, so this degrades in place. No consensus, RPC, config, or DB-format change.

Tests

with_backup_starts_without_backup_when_path_is_a_file (non_finalized_state/tests/vectors.rs) opens an ephemeral finalized DB, writes a regular file at the backup path, calls with_backup, and asserts it returns (rather than aborting under panic = "abort") and leaves the file untouched. cargo fmt/clippy clean. (An unreadable-directory test isn't included — creating one portably in CI, and its no-op as root, is unreliable; that branch is covered by the same read_dir probe.)

Specifications & References

None.

Follow-up Work

run_backup_task and restore_backup still call read_dir(...).expect(...) in backup.rs, so a backup directory that becomes unreadable while zebrad runs can still abort. Pre-existing and out of scope here; worth a follow-up to convert those to warn-and-skip.

AI Disclosure

• No AI tools were used in this PR
• AI tools were used: Claude (Claude Code) — wrote the fix and the regression test.

PR Checklist

• The PR title follows conventional commits format: type(scope): description
• The PR follows the contribution guidelines.
• This change was discussed in an issue or with the team beforehand.
• The solution is tested.
• The documentation and changelogs are up to date.

[v12-auditor]

Note

Complete: Audit complete. V12 found five issues worth reviewing.

Open the full results here.

Finding	Severity	Details
F-206450	🟠 High	Backup files bypass semantic verification Backup restoration deserializes bytes from the cache directory and immediately constructs SemanticallyVerifiedBlock with SemanticallyVerifiedBlock::from, even though that conversion only derives cached fields and runs no consensus verifier. A mismatched filename hash only produces a warning, so it does not authenticate the file contents. Production restoration then calls validate_and_commit_non_finalized, whose checks are contextual and do not validate the proof-of-work hash, Equihash solution, transaction signatures, or other semantic rules enforced by the normal block verifier. An actor who can write the backup directory can therefore build a contextually consistent sequence whose headers encode valid difficulty targets and times but whose actual proof of work or transactions are invalid. The restored chain can become the advertised best chain and, once the write worker processes another successful non-finalized commit, blocks beyond the reorg limit are written into finalized state without semantic revalidation.
F-206452	🟠 High	Unbounded cache reads exhaust startup memory Backup restoration performs std::fs::read on every hash-named, non-finalized entry before applying any backup-format or block-size validation. This API allocates and reads the complete file into a Vec<u8>, and only afterward does NonFinalizedBlockBackup::from_bytes split the amount prefix and invoke block deserialization. The block deserializer's 2,000,000-byte limit cannot bound this earlier allocation or I/O. An actor with backup-directory write access can place a huge regular or sparse file under a valid 64-hex-character name and trigger the read whenever an eligible persistent node restarts. Because state initialization awaits restoration, the file can exhaust memory or stall every startup before being rejected as malformed.
F-206453	🟡 Medium	Inline filesystem probe can hang startup The patch executes std::fs::create_dir_all and std::fs::read_dir directly while polling the async with_backup future. Both calls can block for an unbounded duration on a hung FUSE mount, hard-mounted network filesystem, or stalled storage device. The error fallback cannot run until the blocking syscall returns, and the surrounding state initialization is awaited before the node finishes startup. The adjacent restoration code and the recurring backup task correctly isolate comparable filesystem work with spawn_blocking, but the new probe is outside that boundary. Thus an optional restart cache can indefinitely occupy the runtime thread driving startup even when restoration itself is disabled.
F-206454	🟠 High	Mutable backup path still aborts node The probe opens and immediately drops the backup directory, so it establishes no property for later path lookups. After a successful probe, startup restoration and the lifetime async backup task reopen the same mutable path through read_backup_dir, which still uses read_dir(...).expect(...). Removing, renaming, unmounting, or making the directory unreadable after the probe therefore panics on the next restore/listing attempt. The async task performs this listing at the beginning of each update cycle, leaving an unbounded exploitation window over the node's lifetime. Because Zebra is built with abort-on-panic, a same-UID process, cache cleaner, volume manager, or operator action that mutates this re-downloadable cache directory terminates the full node.
F-206456	🟠 High	Hash symlink race enables file overwrite The backup writer uses a directory snapshot to decide whether each live block file exists, then later opens backup_dir/<block-hash> with std::fs::write. In the default async path, listing occurs before waiting for a state change and the five-second rate limit, creating a long window in which an attacker who can mutate the backup directory can plant a hash-named symlink absent from the snapshot. When that hash becomes live, the missing-entry test passes and std::fs::write follows the symlink, truncating and replacing the target with serialized backup bytes. A symlink present before listing suppresses the write, but planting it after listing gives a concrete race; synchronous mode has the same race over a shorter list-to-write window. The code uses neither a no-follow/create-new open nor atomic temporary-file replacement to bind the check and write to the intended file.

And six more auto-invalidated findings.

Analyzed one file, diff 05d129b...9cdd2a0.

[conradoplg]

I'd prefer for Zebra to shutdown in that case rather than keep running, which could lead to a misconfiguration being unnoticed. I don't think panicking in that case is a huge deal but if you can find a non-very-intrusive way to gracefully log the error and shutdown zebra, I'd prefer that, otherwise we can just close the PR.

[arya2]

@claude update this PR so that if we fail to create a non-finalized state backup directory at the:

• default path:
  • log a warning
  • try to use a different path
  • log another warning if we can't create the non-finalized state backup directory
• configured path: gracefully log the error and shutdown zebra.