Taskification/control-flow inversion for scripts, part 1 #1499
Conversation
jtb20
requested review from
saiislam,
estewart08,
gregrodgers and
ronlieb
as code owners
|
This probably hasn't been tested well enough so far to merge just yet (and apparently needs rebasing already), but I'm interested in feedback on the approach. The build_offload.sh script should do the same thing it does at present, but one thing to note is the build steps take place in a different order -- clean/cmake/build/install for each configuration in turn, rather than cmake/cmake/cmake build/build/build install/install/install. So the build log can't be compared directly. |
|
I just noticed the |
|
What is the motivation/benefit in changing the current approach? |
|
Several things really, e.g.:
...though admittedly, some of these are planned features, rather than things present in the actual patches so far. Think of this more as preparatory work. (The idea is that each script would implement the "introspection" functions -- list_configs and list_tasks in particular. Then some centralised mechanism would enumerate the complete list of tasks, allowing the user to select which are run for some given build or rebuild.) |
This patch splits aomp_utils (shell utility functions) out of aomp_common_vars, in preparation for adding lots more non-configuration-related stuff to the new file. This is mostly straightfoward, but build scripts, etc. need to source the new file also as well as aomp_common_vars.
This is an experimental patch to incrementally refactor the build scripts.
The approach taken is particularly relevant to scripts which build
multiple configurations of a component.
There are several ideas here. Firstly, build scripts can be introspected
so you can see what they are doing:
$ ./build_offload.sh list_configs
asan
perf
perf+asan
debug
debug+asan
$ AOMP_BUILD_SANITIZER=0 ./build_offload.sh list_configs
perf
debug
Build steps are split into different tasks, each of which is wrapped in a
"task_foo" function (which is passed the configuration name):
$ ./build_offload.sh list_tasks perf
clean
cmake
build
install
You can list all tasks list this:
$ ./build_offload.sh list
task_precheck
task_patch
task_clean asan
task_cmake asan
task_build asan
task_install asan
task_clean perf
task_cmake perf
task_build perf
task_install perf
task_clean perf+asan
task_cmake perf+asan
task_build perf+asan
task_install perf+asan
task_clean debug
task_cmake debug
task_build debug
task_install debug
task_postinstall debug
task_clean debug+asan
task_cmake debug+asan
task_build debug+asan
task_install debug+asan
task_unpatch
Or e.g. see what will happen for different env var settings:
$ AOMP_BUILD_DEBUG=0 ./build_offload.sh list
task_precheck
task_patch
task_clean asan
task_cmake asan
task_build asan
task_install asan
task_clean perf
task_cmake perf
task_build perf
task_install perf
task_clean perf+asan
task_cmake perf+asan
task_build perf+asan
task_install perf+asan
task_unpatch
You can still invoke the build scripts the same way as at present:
$ ./build_offload.sh
[...builds everything...]
$ ./build_offload.sh cmake
[...just runs cmake steps...]
$ ./build_offload.sh nocmake
[...just runs build steps...]
$ ./build_offload.sh install
[...just runs install steps...]
Only the build_offload.sh script has been done so far, but this
backward compatibility allows other scripts to be refactored in a similar
way incrementally.
The aim (not yet realised) is to then provide a new or alternate front-end
(to build_aomp.sh) using exactly the same build scripts, allowing
fine-grained control over which components to build/rebuild, and so on.
Internally, the refactoring already done on build_offload.sh demonstrates
very much deduplication, particularly around the setting of cmake
options. Further work will allow much of the "boilerplate" in each of
the build scripts to be replaced by a central implementation.
Environment variables used to control the build are now read via two
functions (get_config_var_string and get_config_var_bool). There are
several reasons for this indirection:
- It allows us to have a single location to check which environment
variables are being used to control the build.
- (For the alternate frontend) the options can be changed via
e.g. command-line options instead of environment variables.
- The _bool function emits the string 'true' or 'false'. This allows
the use of:
if $Result; then ... fi
instead of
if [ "$Result" -eq 1 ]; then ... fi
which is shorter/somewhat easier to read.
"Actions" (the emulated "nocmake", etc. options, and the new "list",
etc. options) are implemented as functions named "do_{foo}", invoked by
the new "command_dispatcher" function.
There are a few other actions, e.g.:
$ ./build_offload.sh show_build_dir perf
/home/julbrown/git/aomp21.0/build/home/julbrown/git/aomp21.0/offload_perf
These will be more useful once more scripts are "taskified".
|
This version adds the (originally intended, but lost during development!) ability to run individual tasks: That should make these changes a bit more immediately useful already. I also realised the quite heavy use of command substitutions isn't really safe unless the shell "errexit" option (set -e) is turned on. I've added this (to the build_offload.sh script only), but there are several places where it might cause inadvertent early exit in non-error cases, so all that needs checking and fixing. |
This is an experimental patch to incrementally refactor the build scripts. The approach taken is particularly relevant to scripts which build multiple configurations of a component.
There are several ideas here. Firstly, build scripts can be introspected so you can see what they are doing:
Build steps are split into different tasks, each of which is wrapped in a "task_{foo}" function (which is passed the configuration name):
You can list all tasks like this:
Or e.g. see what will happen for different environment variable settings:
You can still invoke the build scripts the same way as at present:
Only the build_offload.sh script has been done so far, but this backward compatibility allows other scripts to be refactored in a similar way incrementally.
The aim (not yet realised) is to then provide a new or alternate front-end (to build_aomp.sh) using exactly the same build scripts, allowing fine-grained control over which components to build/rebuild, and so on.
Internally, the refactoring already done on build_offload.sh demonstrates very much deduplication, particularly around the setting of cmake options. Further work will allow much of the "boilerplate" in each of the build scripts to be replaced by a central implementation.
Environment variables used to control the build are now read via two functions (get_config_var_string and get_config_var_bool). There are several reasons for this indirection:
It allows us to have a single location to check which environment variables are being used to control the build.
(For the alternate frontend) the options can be changed via
e.g. command-line options instead of environment variables.
The _bool function emits the string 'true' or 'false'. This allows
the use of:
if $Result; then ... fiinstead of
if [ "$Result" -eq 1 ]; then ... fiwhich is shorter/somewhat easier to read.
"Actions" (the emulated "nocmake", etc. options, and the new "list", etc. options) are implemented as functions named "do_{foo}", invoked by the new "command_dispatcher" function.
There are a few other actions, e.g.:
These will be more useful once more scripts are "taskified".