'Please provide at least one reachable node' error misleading in case where nodes have been dropped

[eoghanmurray]

I'm getting the following after many hours of operation in production:

RedisClusterException: Redis Cluster cannot be connected. Please provide at least one reachable node: None

This is thrown at
https://github.com/redis/redis-py/blob/master/redis/asyncio/cluster.py#L1472

Which should include an exception thrown, which is in fact None, so that would indicate that the self.startup_nodes dict is empty.

As the app was started with nodes present, the only bit of code that I can see that could alter the startup_nodes is

https://github.com/redis/redis-py/blob/master/redis/asyncio/cluster.py#L824

where a node is popped after one of (ConnectionError, TimeoutError)

So I'd propose

• throwing a new error at that point when a ConnectionError or TimeoutError is encountered against all nodes, and we've popped the last one

But I'm not sure if this is the right approach; should we be popping the startup_nodes, or just retrying in a different order instead?

I haven't confirmed yet but I believe that the problem stems from the cluster occasionally timing out after a long ZUNIONSTORE, with no ClusterDown error encountered.

[eoghanmurray]

Possibly this has only become a problem recently, possibly related to changes in #3622

[petyaslavova]

Hi @eoghanmurray, thanks for letting us know! We will have a look at the open PR soon.