Skip to content

Vine: Forsake functions when library exits.#3742

Merged
btovar merged 10 commits intocooperative-computing-lab:masterfrom
dthain:vine-worker-forsake-functions
Mar 26, 2024
Merged

Vine: Forsake functions when library exits.#3742
btovar merged 10 commits intocooperative-computing-lab:masterfrom
dthain:vine-worker-forsake-functions

Conversation

@dthain
Copy link
Copy Markdown
Member

@dthain dthain commented Mar 22, 2024

Proposed changes

Previously, when a library would exit (success or otherwise), two problems resulted:
1 - The function calls sent subsequent to that library would simply sit on the worker doing nothing.
2 - The manager would quickly re-dispatch a library to that same worker, resulting in a tight loop of failures.

This PR fixes these as follows:
1 - The worker now has extended logic for task_can_fit_eventually so that functions will be forsaken if there is no library process at all that could execute them.
2 - Forsaken tasks are now handled correctly by deleting their sandboxes once they are retrieved.
3 - The most recent failure of a library is recorded in the library prototype object and used to throttle re-dispatch of failed library tasks.
4 - Fetching of task stdout is factored out of get_result to make the logic of fetching a result more clearly visible.

Post-change actions

Put an 'x' in the boxes that describe post-change actions that you have done.
The more 'x' ticked, the faster your changes are accepted by maintainers.

  • make test Run local tests prior to pushing.
  • make format Format source code to comply with lint policies. Note that some lint errors can only be resolved manually (e.g., Python)
  • make lint Run lint on source code prior to pushing.
  • Manual Update Did you update the manual to reflect your changes, if appropriate? This action should be done after your changes are approved but not merged.
  • Type Labels Select github labels for the type of this change: bug, enhancement, etc.
  • Product Labels Select github labels for the product affected: TaskVine, Makeflow, etc.
  • PR RTM Mark your PR as ready to merge.

Additional comments

This section is dedicated to changes that are ambitious or complex and require substantial discussions. Feel free to start the ball rolling.

@dthain
Copy link
Copy Markdown
Member Author

dthain commented Mar 25, 2024

Still thinking about this one in two respects:
1 - The FORSAKEN code path in the manager is not overly trod, I want to make sure this behaves as expected.
2 - This will still cause the manger to retry libraries quickly, we need some kind of hysteresis in library attempts.

@dthain dthain force-pushed the vine-worker-forsake-functions branch from 70643c6 to bd7fc68 Compare March 25, 2024 12:47
@dthain dthain changed the title (WIP) Forsake functions without a library. (WIP) Vine: Forsake functions without a library. Mar 25, 2024
@dthain dthain changed the title (WIP) Vine: Forsake functions without a library. Vine: Forsake functions when library exits. Mar 26, 2024
@dthain dthain requested a review from btovar March 26, 2024 12:46
@dthain dthain added bug For modifications that fix a flaw in the code. TaskVine labels Mar 26, 2024
@dthain
Copy link
Copy Markdown
Member Author

dthain commented Mar 26, 2024

Since this is an improvement to a corner case and not a critical bug, I think it can wait until after the pending release...

@btovar btovar merged commit 009693e into cooperative-computing-lab:master Mar 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug For modifications that fix a flaw in the code. TaskVine

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants