vine: manager stops scheduling tasks when peer match is not found

[btovar]

When a worker is selected when dispatching a task, the manager checks for possible input file sources with vine_manager_transfer_capacity_available. If this function fails to find peers for all input files, then the task is not dispatched. Since nothing changes in the state of the manager, this dispatch is tried endlessly.

The inner check that fails is vine_file_replica_table_find_worker. Since this check fails, that means that vine_manager_check_inputs_available (with inner vine_file_replica_table_exists_somewhere) should have previously failed. Thus, one hand the manager thinks the file exists somewhere, but then it cannot find it.

Unfortunately this bug is hard to reproduce.

[dthain]

@colinthomas-z80 I think we need your insight here on how to back out of a task dispatch where there is insufficient transfer capacity available.

[colinthomas-z80]

I looks like this may be possible if w->transfer_port_active is set to false? Is there any chance this is the case for some of your workers?

[btovar]

All the workers are the same, from the same factory. Do you mean that the worker failed to set a transfer server port?

[colinthomas-z80]

That is one theory. Without identifying the problem directly I think it might be centered around vine_manager_check_inputs_available not being stringent enough to declare a recovery task? Maybe we should match the satisfying conditions with those of vine_file_replica_table_find_worker

[btovar]

Ok, yes, vine_file_replica_table_exists_somewhere should at least check for w->transfer_port_active for files that do not have a replica at the manager. Do you want to make a pr about that?

[colinthomas-z80]

Sure. I think the issue must be caused by a temp file right?

[btovar]

Yes, these are temp files. The temp files are supposed to be replicated, but I guess they can't if no source is found.

[colinthomas-z80]

Actually it does not look like a worker will keep running if it can't open a transfer port. So the remaining inconsistency is this file state check from the concepts introduced in #3647

[dthain]

Is this one fixed?