Improve performance when loading large amount of modules

[xdelaruelle]

Loading a large amount of module (>100) feels slow for users. Especially when comparing to Modules version 3.

This was reported on the following mailing-list discussion: https://lists.hpsf.io/g/modules-interest/message/2

On a test case based on a module tree generated with EasyBuild with R-bundle-Bioconductor/3.19-foss-2023b-R-4.4.1 requiring 136 other modules as dependencies, the module load commands takes 1000ms.

Adding this test case within mb (Modules make bench utility), we get the following results across versions:

$ script/mb load2
              |   v4.1.4 |   v4.3.1 |   v4.5.3 |   v4.7.1 |   v5.1.1 |   v5.3.1 |   v5.4.0 |   v5.5.0 |   v5.6.1 |     main |
--------------+----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
        load2 |      302 |      635 |     1537 |     1600 |      872 |      878 |      842 |      943 |      988 |      988 |

$ script/mb load2_cache
              |   v4.1.4 |   v4.3.1 |   v4.5.3 |   v4.7.1 |   v5.1.1 |   v5.3.1 |   v5.4.0 |   v5.5.0 |   v5.6.1 |     main |
--------------+----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
  load2_cache |        - |        - |        - |        - |        - |     1054 |     1037 |     1129 |     1165 |     1167 |

Time increases across version, due to the addition of new features especially for dependency resolution and their consistency. Here module cache does not help, as test is run on a single user local laptop with SSD storage. Of course module cache should help when facing a more loaded IO storage.

To improve the situation, an objective could be to reduce this elapsed time by half (500ms).