Skip to content

vine: consider cached output buffers as replicas at workers#3690

Merged
btovar merged 4 commits intocooperative-computing-lab:masterfrom
btovar:replica_cache_buffer
Mar 6, 2024
Merged

vine: consider cached output buffers as replicas at workers#3690
btovar merged 4 commits intocooperative-computing-lab:masterfrom
btovar:replica_cache_buffer

Conversation

@btovar
Copy link
Copy Markdown
Member

@btovar btovar commented Mar 5, 2024

  • make test Run local tests prior to pushing.
  • make format Format source code to comply with lint policies. Note that some lint errors can only be resolved manually (e.g., Python)
  • make lint Run lint on source code prior to pushing.
  • Manual Update Did you update the manual to reflect your changes, if appropriate? This action should be done after your changes are approved but not merged.
  • Type Labels Select github labels for the type of this change: bug, enhancement, etc.
  • Product Labels Select github labels for the product affected: TaskVine, Makeflow, etc.
  • PR RTM Mark your PR as ready to merge.

Additional comments

This section is dedicated to changes that are ambitious or complex and require substantial discussions. Feel free to start the ball rolling.

@btovar btovar added bug For modifications that fix a flaw in the code. TaskVine labels Mar 5, 2024
@btovar btovar requested a review from dthain March 5, 2024 18:22
@btovar btovar requested a review from dthain March 5, 2024 19:35
@btovar
Copy link
Copy Markdown
Member Author

btovar commented Mar 6, 2024

vine_file_has_changed uses st_size. This is an issue for directories because we would like to have f->size to be the total size of the files inside the directory for scheduling purposes, but we don't want to stat every time all its contents every time to check if it has changed.

One easy fix is to check if the payload is a directory, and if so, drop the check against st_mtime.

@dthain

@btovar btovar merged commit 5021804 into cooperative-computing-lab:master Mar 6, 2024
@btovar btovar deleted the replica_cache_buffer branch March 6, 2024 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug For modifications that fix a flaw in the code. TaskVine

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants