Maintenance Status RPC can SIGSEGV a member during shutdown: StorageVersion reads from a closed backend

[jojinkb]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

A Maintenance/Status gRPC call processed while the member is shutting down (e.g. after a ConfChangeRemoveNode removing that member) can crash the process with a SIGSEGV instead of letting it exit cleanly.

The shutdown sequence in the run goroutine's defer (server/etcdserver/server.go) closes s.stopping, waits for attached goroutines, and then calls Cleanup(), which closes the backend (s.be.Close()) without any synchronization against in-flight readers. A concurrent Status handler that reaches EtcdServer.StorageVersion() in that window calls schema.DetectSchemaVersion() on the closed backend. The backend's final commit (CommitAndStop) has already reset the shared read transaction to nil, so backend.(*baseReadTx).UnsafeRange() calls (*bolt.Tx).Bucket() on a nil *bolt.Tx and the process dies:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x104c3eb1c]

goroutine 6351 [running]:
go.etcd.io/bbolt.(*Tx).Bucket(...)
	go.etcd.io/bbolt@v1.5.0/tx.go:112
go.etcd.io/etcd/server/v3/storage/backend.(*baseReadTx).UnsafeRange(...)
	server/storage/backend/read_tx.go:103 +0x1ec
go.etcd.io/etcd/server/v3/storage/schema.UnsafeReadStorageVersion(...)
	server/storage/schema/version.go:35 +0x60
go.etcd.io/etcd/server/v3/storage/schema.UnsafeDetectSchemaVersion(...)
	server/storage/schema/schema.go:93 +0x40
go.etcd.io/etcd/server/v3/storage/schema.DetectSchemaVersion(...)
	server/storage/schema/schema.go:88 +0xac
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).StorageVersion(...)
	server/etcdserver/server.go:2193 +0xe4
go.etcd.io/etcd/server/v3/etcdserver.(*serverVersionAdapter).GetStorageVersion(...)
	server/etcdserver/adapters.go:73 +0x20
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.(*maintenanceServer).Status(...)
	server/etcdserver/api/v3rpc/maintenance.go:280 +0x210
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.(*authMaintenanceServer).Status(...)
	server/etcdserver/api/v3rpc/maintenance.go:374 +0x68
go.etcd.io/etcd/api/v3/etcdserverpb._Maintenance_Status_Handler.func1(...)
	api/etcdserverpb/rpc_grpc.pb.go:1243
... (grpc server internals)

Note on discoverability: on current main this crash is masked in the Status path because the same handler panics earlier in RaftCluster.IsLocalMemberLearner (#21966). With that panic fixed (#22188), the reproducer for #21966 immediately surfaces this second, independent crash in the same shutdown window.

What did you expect to happen?

The removed/stopping member exits cleanly with code 0. Status calls racing with shutdown should either succeed or fail with a gRPC error, never crash the server.

How can we reproduce it (as minimally and precisely as possible)?

Use the reproducer program from #21966 unchanged (3-member cluster, 64 goroutines hammering Maintenance/Status on a persistent gRPC connection to m3, then removing m3 via m1), against a build that includes the #22188 fix (otherwise the IsLocalMemberLearner panic fires first and masks this one):

1. Build etcd from main + membership: don't panic in IsLocalMemberLearner when local member is absent #22188 (go build -o bin/etcd ./server).
2. Run the reproducer from Panic in IsLocalMemberLearner when Maintenance/Status is called concurrently with member removal #21966 with that binary on PATH.
3. m3 exits with code 2 and the SIGSEGV stack above in m3.log (reproduced on the first attempt in my runs).

With StorageVersion prevented from reading a closed backend, the same reproducer gives 3/3 clean exits with code 0 (~300k Status calls per run).

Anything else we need to know?

• Root cause is the unsynchronized ordering in the run goroutine's defer: close(s.stopping) → s.wg.Wait() → s.Cleanup() (which calls s.be.Close()), while gRPC handlers may still be executing. StorageVersion() takes s.bemu.RLock(), but Cleanup() does not take bemu before closing the backend, so the lock does not protect readers from the close.
• Internal callers of StorageVersion (e.g. monitorStorageVersion) are safe: they run via GoAttach and are waited on by s.wg.Wait() before Cleanup() runs. Only externally driven paths (the Status RPC via serverVersionAdapter.GetStorageVersion) race.
• The other backend accesses in the Status handler (Backend().Size(), SizeInUse()) read atomics and do not crash.
• A fix is prepared: close the backend under s.bemu in Cleanup() and check s.stopping under the bemu read lock in StorageVersion(), returning nil (the handler already omits the field for a nil version).

Etcd version (please run commands below)

Details

$ etcd --version
etcd Version: 3.8.0-alpha.0
Git SHA: Not provided (use ./build instead of go build)
Go Version: go1.26.5
Go OS/Arch: darwin/arm64

Built from main (14bd0ca) plus #22188. The affected code (Cleanup/StorageVersion ordering) is unchanged on main; the same pattern exists on the release-3.6 branch (StorageVersion reads under bemu.RLock while Cleanup closes the backend without taking bemu), though there the Status path is shadowed by the #21966 panic. release-3.5 is unaffected: it predates the StorageVersion/schema machinery entirely.

Etcd configuration (command line flags or environment variables)

Details

Defaults as set by the #21966 reproducer (3 members, plain HTTP, --log-format console); no non-default tuning is required.