Crash When Attempting to Install Packages on HPC Environment

[novalex98]

Hello,

I am attempting to install packages on the Hyak HPCC system (https://hyak.uw.edu/).

I am using the latest version of Julia (1.9.3)

Some packages installed successfully, but when it got to SimpleUnPack there was a crash, and a dump was produced (first attachment).
JuliaPackageInstallCrashDump.txt

I searched online and found that i the past people have solved this problem by using JuliaUp to install Julia instead of installing it directly https://discourse.julialang.org/t/having-trouble-installing-differentialequations-jl/105217/5

I tried this, and while Julia was installed successfully, this time no packages could be installed and the same dump was produced (second attachment).
JuliaPackageInstallCrashDump_2.txt

Please help me fix this problem. Happy to provide any additional information.

I posted on the llvm project forum and was told the post belonged here llvm/llvm-project#72731

[Keno]

The binary that's crashing here is lld and the error message Resource temporarily unavailable is EAGAIN. However, EAGAIN shouldn't be possible unless somebody is setting up a non-restarting signal handler, which to the best of my knowledge lld does not do. Since this is an HPC environment, my best guess is that there's something in LD_LIBRARY_PATH or LD_PRELOAD that's leaking signal handlers.

[Keno]

Also a symbolized stack trace to see what syscall if failing with EAGAIN would help.

[giordano]

Just because lld is involved, I'll mention that I've seen lld crashing on HPC systems because of parallel precompilation clashing with CPU limits put on login node (for example with cgroups). This is usually resolved by setting the environment variable JULIA_NUM_PRECOMPILE_TASKS to a low value (like 1 or so), or lower than the limit.

However I think the error I've seen was different from the one reported here, so I'm not sure it's the same thing, but I figured I'd mention it for completeness.

[giordano]

Actually, no, the error message was precisely the same: llvm/llvm-project#68192. I guess screenshots and attached files don't make search very easy.

[Keno]

Can we read RLIMIT_NPROC and automatically restrict the number of processes we launch? Alternatively, should we propose a patch to lld to give a better error message at least?