-
Notifications
You must be signed in to change notification settings - Fork 112
Description
This might be a Julia bug. Well, not really a bug, but just something that needs better default behavior.
Right now it seems that Julia workers don't know about the memory allocation of other Julia workers, and so garbage collection does not happen as aggressively as it should.
The solution we can implement in SymbolicRegression.jl is for worker processes internall created by addprocs
to have a much smaller heap-size-limit
(see https://julialang.org/blog/2023/04/julia-1.9-highlights/#memory_usage_hint_for_the_gc_with_--heap-size-hint).
The best resource allocation seems to be letting the head worker have as many threads and a large portion of memory available. But the worker processes should have a single thread and small memory available, so they garbage collect frequently. I think this could be done automatically.