grpcproxy: new watchers may be added to a disconnected watchBroadcast and miss all subsequent events

[maluyazi]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

In server/proxy/grpcproxy, when a watchBroadcast's underlying etcd watch connection is broken (the goroutine exits after for wr := range wch completes), the watchBroadcast remains in watchBroadcasts.bcasts with no indication that it is no longer receiving events.

This causes two problems:

1. add() can place new watchers into a dead broadcast: Since there is no stopped flag, add() happily accepts new watchers into a broadcast whose goroutine has already exited. These watchers will never receive any subsequent events.

2. coalesce() can migrate watchers into a dead broadcast: The coalesce logic only checks nextrev and responses, not whether the target broadcast is still alive. Watchers migrated to a dead broadcast will also stop receiving events.

Additionally, there is no mechanism to reassign existing watchers (orphans) from a dead broadcast to a healthy one.

What did you expect to happen?

1. A disconnected watchBroadcast should be marked as stopped and refuse new watchers.
2. coalesce() should not migrate watchers to a stopped broadcast.
3. When a broadcast disconnects, its existing watchers should be automatically reassigned to a healthy broadcast or a newly created one.

How can we reproduce it (as minimally and precisely as possible)?

1. Set up an etcd gRPC proxy with multiple client watchers coalesced on the same key range.
2. Cause the backend etcd watch connection to break (e.g., network partition, etcd server restart).
3. After the break, create a new watcher on the same key range through the proxy.
4. Observe that the new watcher (and any existing watchers on the dead broadcast) never receives subsequent events.

Anything else we need to know?

No response

Etcd version (please run commands below)

Details

$ etcd --version
v3.5.4

$ etcdctl version
v3.5.4

Etcd configuration (command line flags or environment variables)

Details

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

Details

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

[kunalworldwide]

When a watch client's connection drops, the watchBroadcast remains in the proxy's tracking list but can no longer deliver events, yet new watchers that join while it is in this state are silently attached to the dead broadcast and never receive updates. This creates a silent data loss scenario where client applications believe they are subscribed to changes but are in fact receiving nothing. Adding a stopped flag to the broadcast struct and a check in the watcher-addition path to reassign new watchers to a live broadcast would prevent this class of issue. Is the stopped flag already present in any of the nearby watcher code, or would this require introducing a new field and updating the watcher registration logic?

[maluyazi]

The stopped flag does not currently exist in the proxy watch code. I've submitted a fix #21814 that addresses this issue.

Here's a summary of the changes:

1. Added a stopped bool field to watchBroadcast — it gets set to true when the underlying etcd watch channel closes (i.e., the goroutine exits after for wr := range wch completes).

2. Guard in add() — watchBroadcast.add() now checks if wb.stopped { return false } before attaching a new watcher, so no watcher can be silently added to a dead broadcast.

3. Guard in coalesce() — the coalescing logic now checks !wbswb.stopped before migrating watchers to another broadcast, preventing migration to a dead one.

4. Added a reattach callback — when a broadcast's goroutine exits (connection drop), it invokes reattach(wb) which collects all orphaned watchers from the dead broadcast and reassigns them to an existing live broadcast or creates a new one.

PR: #21814

[serathius]

cc @ahrtr @fuweid @jberkus

Can we announce deprecation of grpcproxy or are we still pretending someone is working on it?

[maluyazi]

Are there any recommended alternatives to grpcproxy? In our use case, we have a large number of watch streams and the watched data volume is significant, so we rely on the proxy to coalesce watches and optimize performance.

[ahrtr]

Can we announce deprecation of grpcproxy

I believe it's useful but we don't have bandwidth to take care of it.

Options:

• Deprecate it and eventually remove it
• Move it into a separate repo, and call for contributors.

Probably we can send a survey? @jberkus

[silentred]

I believe it's useful

Indeed. +1 for a separate repo.