Skip to content

Garbage collection too passive on worker processes #237

@MilesCranmer

Description

@MilesCranmer

This might be a Julia bug. Well, not really a bug, but just something that needs better default behavior.

Right now it seems that Julia workers don't know about the memory allocation of other Julia workers, and so garbage collection does not happen as aggressively as it should.

The solution we can implement in SymbolicRegression.jl is for worker processes internall created by addprocs to have a much smaller heap-size-limit (see https://julialang.org/blog/2023/04/julia-1.9-highlights/#memory_usage_hint_for_the_gc_with_--heap-size-hint).

The best resource allocation seems to be letting the head worker have as many threads and a large portion of memory available. But the worker processes should have a single thread and small memory available, so they garbage collect frequently. I think this could be done automatically.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions