Maintain peers across all data column subnets

[jimmygchen]

Issue Addressed

Closes:

• Maintain peers across all custody column subnets #7865
• Refactor data column RPC peer selection #7855

Changes extracted from earlier PR #7876

This PR fixes two main things with a few other improvements mentioned below:

• Prevent Lighthouse from repeatedly sending DataColumnByRoot requests to an unsynced peer, causing lookup sync to get stuck
• Allows Lighthouse to send discovery requests if there isn't enough synced peers in the required sampling subnets - this fixes the stuck sync scenario where there isn't enough usable peers in sampling subnet but no discovery is attempted.

Proposed Changes

• Make peer discovery queries if custody subnet peer count drops below the minimum threshold
• Update peer pruning logic to prioritise uniform distribution across all data column subnets and avoid pruning sampling peers if the count is below the target threshold (2)
• Check sync status when making discovery requests, to make sure we don't ignore requests if there isn't enough synced peers in the required sampling subnets
• Optimise some of the PeerDB functions checking custody peers
• Only send lookup requests to peers that are synced or advanced

[mergify]

Some required checks have failed. Could you please take a look @jimmygchen? 🙏

[AgeManning]

I think this is an improvement, but we might want to make the data columns first class citizens.

The overall goal of this logic originally was to get Lighthouse to reach a steady-state where it never had to do any discoveries. It would find and maintain a uniform set of subnet peers.

There are two competing factors, discoveries which generate peers and pruning which remove the excess. If the pruning doesn't match what our discovery targets are, we might be in a perpetual state of discovering, then pruning the discovered peers.

Before data columns, we would discover peers if we needed them for attestation subnets, then prune down to maintain a uniform set of attestation subnets, to prevent any future discoveries.

With this change, we now have a new driving requirement, which is has_good_peers_in_custody_subnet(). We will try and discover peers constantly until we meet this requirement.

But the pruning logic is still to maintain uniform attestation subnets. We only now don't prune peers that might help with our custody subnet requirement.

I think now that things have changed, we should prioritize a uniform distribution on the data column custody and as a second priority, manage the attestation subnets. The reason being, is that the attestation subnets don't have a direct maintain_custody_peers() like function causing discoveries, and so I think its therefore less of a priority.

Also, for attestation subnets we really need, we have a min_ttl which prevents them from being pruned when we need them. So we can rely on that to save the crucial ones from being dropped.

So I think the pruning priorities should now be:

1. Maintain uniform distribution of data columns
2. a - Don't remove peers that we need for attestation subnets
3. b - Dont remove peers that we need for sync committees
4. If all of the above are satisfied, remove peers to make attestation subnets uniform.

[jimmygchen]

Thanks @AgeManning , yeah I think your suggestion makes sense, I'll make this change.

[AgeManning]

Left some comments

[AgeManning]

Maybe for a future PR. Rather than calculate this thing for every peer every heartbeat, we just change PeerInfo to store these naturally for each peer.

[jimmygchen]

yeah good idea, with higher peer count it makes even more sense. I'll raise an issue for this.

[ackintosh]

Thanks for this PR, Jimmy! I've left some comments.

[AgeManning]

This looks good to me :)