internal/debug: add debug_setMemoryLimit

[georgehao]

when send too many transaction to txpool during my benchmark, the golang gc will take 1/3 CPU profile.

I want to use the ballast mechanism to reduce the gc times to reduce golang GC CPU cost. I try GOGC = 500 and GOMEMLIMIT=10G, but it doesn't work. mainly because this section of code limits the GOGC to never be over 100.

go-ethereum/cmd/utils/flags.go

Lines 1583 to 1597 in 03cc294

	mem, err := gopsutil.VirtualMemory()
	if err == nil {
	if 32<<(^uintptr(0)>>63) == 32 && mem.Total > 2*1024*1024*1024 {
	log.Warn("Lowering memory allowance on 32bit arch", "available", mem.Total/1024/1024, "addressable", 2*1024)
	mem.Total = 2 * 1024 * 1024 * 1024
	}
	allowance := int(mem.Total / 1024 / 1024 / 3)
	if cache := ctx.Int(CacheFlag.Name); cache > allowance {
	log.Warn("Sanitizing cache to Go's GC limits", "provided", cache, "updated", allowance)
	ctx.Set(CacheFlag.Name, strconv.Itoa(allowance))
	}
	}
	// Ensure Go's GC ignores the database cache for trigger percentage
	cache := ctx.Int(CacheFlag.Name)
	gogc := math.Max(20, math.Min(100, 100/(float64(cache)/1024)))

Assuming the memory is 30G, GOGC value is 20 after the calculation. It also means the larger the memory, the smaller the gogc. This will make the Golang GC very frequent, wasting the CPU resources, so suggest removing this part.

If we wonder about the large number of txs that trigger the OOM, the suggestion is to set GOMEMLIMIT depending on the realistic situation.

For my benchmark, I have enough memory, I don't want the CPU waste on GC, I want to use ballast to reduce the GC times, I need a big GOGC value. here is the performance after the adjustment.

The throughput increased 25% from 45Mgas/s to 55Mgas/s

Before update

After update.

And most of the time, just keeping the GOGC = 100 is a good decision

[georgehao]

Some updates:
golang gc statistics by add GODEBUG: "gctrace=1"

GOGC=500: https://gist.github.com/georgehao/ce60f675699bbed77b988446e296c197
GOGC=20: https://gist.github.com/georgehao/8ff506e0bb0d097ce9ec9bfb912f60f9

The average gc interval:

• GOGC=20 --> 1.53s
• GOGC=500 --> 7.51s

This means using large GOGC will enlarge the gc cycle and let the CPU not cost too much on Golang GC. Actually, it doesn't mean the bigger GOGC is better, we should support a way to optimize.

the gc pause time also reduced.

[rjl493456442]

From my understanding, your main requirement is a way to adjust the GOGC value,
which you can actually modify at runtime using:

> debug.setGCPercent(200)  
100  

This allows you to bypass the startup limitations.

---

For several reasons, we prefer not to support a high GOGC value by default.
A large GOGC setting may cause Geth to retain more memory, potentially
leading to out-of-memory issues on machines with limited memory resources.

[rjl493456442]

Regarding the SetMemoryLimit API, we prefer not to use it. This API imposes a hard cap
on Geth’s memory usage. If memory consumption approaches this limit, Go will trigger
garbage collection frequently, potentially causing excessive GC cycles and blocking the
application.

[georgehao]

If memory consumption approaches this limit, Go will trigger
garbage collection frequently, potentially causing excessive GC cycles and blocking the
application.

I don't quite agree with you.

trigger garbage collection frequently, potentially causing excessive GC cycles

you can refer here

GOGC=500: https://gist.github.com/georgehao/ce60f675699bbed77b988446e296c197
GOGC=20: https://gist.github.com/georgehao/8ff506e0bb0d097ce9ec9bfb912f60f9

you will found use GOGC=500, the gc cycle is much less.

Use MemoryLimit is to prolong gc cycle. this is the go ballast mechanism. you can refer this post . golang use GoMemLimit as a graceful way to realize the ballast after Go1.19 (Go added the variable specifically for golang Garbage Collection）

proposal: runtime: add a mechanism for specifying a minimum target heap size

also can refer this project: https://github.com/cortexproject/cortex/blob/fcc41c24d11cffc9454847aa35dfc53f21e393be/cmd/cortex/main.go#L105

locking the application

I think no, with less go gc cycle, the STW is less, it will speed up the application

At least, we shouldn't let geth's default GOGC is very small, this will actually locking the application

Assuming the memory is 30G, GOGC value is 20 after the calculation. It also means the larger the memory, the smaller >the gogc. This will make the Golang GC very frequent, wasting the CPU resources, so suggest removing this part.

[georgehao]

@rjl493456442 hope you can review it again, thx

[georgehao]

For several reasons, we prefer not to support a high GOGC value by default.
A large GOGC setting may cause Geth to retain more memory, potentially
leading to out-of-memory issues on machines with limited memory resources.

I also agree don't use a large GOGC, this should to decide by the realistic env. the default 100 is the recommendation. but current geth's implementation: with large machine memory, the gc will become very small. the min value is 20. I think this is too small, it will trigger gc very frequently and lock the application (you will find the gcMaker will take 1/3 of the pprof
flame)

this PR's change:

• remove with large machine memory, the gc will become very small
• Add SetMemoryLimit API

These two changes make it possible to optimize for their own environment based on their env (If the user has the ability to optimize). And this also will don't affect the current logic.

[georgehao]

@rjl493456442, what's your opinion?

[rjl493456442]

Please restore the logic for sanitizing the GOGC settings, as it remains a meaningful setting for us.

It’s fine to add the SetMemoryLimit API as a debug method, but users should be aware that:

• Geth also allocates memory off-heap, particularly for fastCache and Pebble, which can be non-trivial (a few gigabytes by default).
• Setting the limit too low will cause Geth to become unresponsive.

Now, while the memory limit is clearly a powerful tool, the use of a memory limit does not come without a cost, and certainly doesn't invalidate the utility of GOGC.

Consider what happens when the live heap grows large enough to bring total memory use close to the memory limit. In the steady state visualization above, try turning GOGC off and then slowly lowering the memory limit further and further to see what happens. Notice that the total time the application takes will start to grow in an unbounded manner as the GC is constantly executing to maintain an impossible memory limit.

This situation, where the program fails to make reasonable progress due to constant GC cycles, is called thrashing. It's particularly dangerous because it effectively stalls the program. Even worse, it can happen for exactly the same situation we were trying to avoid with GOGC: a large enough transient heap spike can cause a program to stall indefinitely! Try reducing the memory limit (around 30 MiB or lower) in the transient heap spike visualization and notice how the worst behavior specifically starts with the heap spike.

In many cases, an indefinite stall is worse than an out-of-memory condition, which tends to result in a much faster failure.

---

Normally, Geth is bound with disk io, and most of the CPU resources are idle. After bumping the GOGC from 25 to 100, there is no big performance difference.

[georgehao]

@rjl493456442 fixed

[rjl493456442]

Could you at least add a comment stating that the input limit is specified in bytes? I’m not sure whether we should sanitize excessively low values, as it’s unclear where the threshold should be.

But I can foresee that people might pass the values as the megabytes or so, resulting in an unexpected behavior.

I would also prefer to print out some logs so that users know what's the threshold being set

[georgehao]

Sorry for the late reply. I will submit as soon

[georgehao]

I’m not sure whether we should sanitize excessively low values, as it’s unclear where the threshold should be.

The minimum value is not easy to decide, I added a comment if the setting is too small

[georgehao]

@rjl493456442 is there anything need to update?