Fix stuck data column lookups by improving peer selection and retry logic

[jimmygchen]

Issue Addressed

Fixes the issue described in #7980 where Lighthouse repeatedly sends DataColumnsByRoot requests to the same peers that return empty responses, causing sync to get stuck.

The root cause was we don't count empty responses as failures, leading to excessive retries to unresponsive peers.

Proposed Changes

• Track per peer attempts to limit retry attempts per peer (MAX_CUSTODY_PEER_ATTEMPTS = 3)
• Replaced random peer selection with hashing within each lookup to prevent splitting lookup into too many small requests and improve request batching efficiency.
• Added single_block_lookup root span to track all lookups created and added more debug logs:

[jimmygchen]

It seems like we're spliting into way too many requests than needed, and doesn't really help with sync speed

I'm going to try experiment replacing the random component when selecting peer and use something deterministic at each block, so we batch as much as possible, but without ending up selecting the same peer for every block lookup.

[jimmygchen]

Interestingly removing the randomness seems to have caused more issues keeping up with the chain, it seems like the spreading requests out to different peers may be a better approach when peers aren't all serving by root requests well.

[jimmygchen]

^ This is not true, it happened because devnet-3 went into non finality testing mode.

So I think the changes are good, I've compare it with some healthy nodes on devnet-3, and see a similar pattern

and some other unhealthy nodes, that likely ran into the "stuck" scenario and had to fallback to range sync:

[eserilev]

This looks great! I just had one thing I wanted to mention, mostly to make sure I wasn't misunderstanding things, plus a tiny nit that doesn't really matter.