etcdserver: don't read storage version from a closed backend

[jojinkb]

Fixes #22189

What this PR does

Fixes a SIGSEGV when a Maintenance/Status gRPC call races with server shutdown (e.g. the member being removed from the cluster).

The run goroutine's shutdown defer closes s.stopping and then calls Cleanup(), which closes the backend without synchronizing with in-flight readers. A concurrent Status handler that reaches EtcdServer.StorageVersion() in that window calls schema.DetectSchemaVersion() on the closed backend, whose shared read transaction has already been reset to nil by the final CommitAndStop, and backend.(*baseReadTx).UnsafeRange() dereferences the nil *bolt.Tx:

SIGSEGV in bbolt.(*Tx).Bucket
  <- backend.(*baseReadTx).UnsafeRange   (server/storage/backend/read_tx.go:103)
  <- schema.UnsafeReadStorageVersion
  <- schema.DetectSchemaVersion
  <- EtcdServer.StorageVersion           (server/etcdserver/server.go:2193)
  <- serverVersionAdapter.GetStorageVersion
  <- maintenanceServer.Status

How it is fixed

Two small changes in server/etcdserver/server.go:

1. Cleanup() now closes the backend while holding s.bemu (the mutex StorageVersion already takes for reading, because applySnapshot can swap the backend instance).
2. StorageVersion() checks s.stopping after acquiring s.bemu.RLock() and returns nil when the server is stopping.

Since s.stopping is closed strictly before Cleanup() runs, the lock gives the needed happens-before edge: a reader that does not observe stopping as closed cannot have its backend closed while it holds the read lock, and a reader that acquires the lock after the close observes stopping as closed and returns nil. The Status handler already handles a nil storage version by omitting the field, and the internal caller (monitorStorageVersion) exits on s.stopping before Cleanup() runs, so no behavior changes outside the shutdown window.

Relationship to #21966 / #22188

This crash was discovered while verifying the fix for #21966: on current main it is masked in the Status path because the same handler panics earlier in RaftCluster.IsLocalMemberLearner. With #22188 applied, the #21966 reproducer immediately surfaces this second, independent crash. This change is standalone and does not depend on #22188.

Testing

• New regression test TestStorageVersionDuringStop (fails with the SIGSEGV stack above before the fix; the concurrent subtest also deadlocks before the fix because the panic escapes UnsafeRange with txMu held):
  • go test ./etcdserver/ -run TestStorageVersionDuringStop -v -race — PASS
• go test ./etcdserver/ -count=1 — ok
• End-to-end with the Panic in IsLocalMemberLearner when Maintenance/Status is called concurrently with member removal #21966 reproducer against a build including membership: don't panic in IsLocalMemberLearner when local member is absent #22188: before this fix, SIGSEGV on the first run; after, 3/3 clean exits with code 0 (~300k Status calls per run).

---

This change was developed with AI assistance (Claude Code); I have reviewed and tested it.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jojinkb
Once this PR has been reviewed and has the lgtm label, please assign spzala for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kubernetes-prow]

Hi @jojinkb. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.