Copy all run and out dirs at once, not in for-loop

[Aariq]

Addresses #3019 with changes to start_model_runs()

Description

• Moves creation of directories on the remote host outside of for loop to do it all at once
• Moves calls to remote.copy.to outside of for loop to copy over all /out and /run directories at once.
• Moves call to remote.copy.from outside of for loop to copy all outputs back to local at once.
• Adds notes for future improvments
• Add informative errors to remote.copy.to() and remote.copy.from()

There aren't any tests for start_model_runs() and I'm not sure how to write them, but I did install this PR and test it with my own setup.

Motivation and Context

Review Time Estimate

• Immediately
• Within one week
• When possible

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My change requires a change to the documentation.
• My name is in the list of CITATION.cff
• I have updated the CHANGELOG.md.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.

[Aariq]

Marked as draft again because I'm getting close (hopefully) to really fixing this.

[Aariq]

Ok, the remaining bug has nothing to do with PEcAn or with R for that matter. I've been able to reproduce it with just rsync. It also seems to be specific to our server as I can't reproduce it from my laptop. Going to open this back up for review. One possible safeguard is to wrap the remote.copy.to commands in PEcAn.utils::retry.func() but I'll wait to get feedback before doing this.

[mdietze]

@robkooper could you look at the changes you requested so we can figure out whether this is ready to pull?

[infotroph]

is removing the shQuote around args intentional?

[Aariq]

Yes, it was intentional, but now that you bring it up I'm not sure it's correct. system2() always shQuotes command, but I guess it doesn't do that for args. I'll add it back and make sure everything still works.

[infotroph]

I appreciate the desire for clarity, but I'd rather send people to the rsync docs for these error messages. We use whatever rsync version is installed on the host system, so hard-coding values here feels like an invitation for them to get out of sync.

I assume rsync doesn't often change the meaning of exit codes between versions, but if they add a new code that's not on this list then this switch will return an empty string instead of the numeric value.

[Aariq]

Yeah, this makes sense. I'm going to keep errors (anything other than 0) as logger.severe() though.

[robkooper]

tested with rabbitmq. please address issues pointed out by @infotroph