perf: Add DrainAll() for batch consumption of perf events

[liuchangyan]

Context

I'm building a continuous profiling function of (ccfos/huatuo) that samples CPU stack traces via perf events. This is a follow-up discussion from #1910 — I want to propose a less intrusive API that addresses the same underlying need.

Problem

My workflow requires a clear batch boundary when consuming perf events:

1. Wait for events (with a controlled timeout)
2. Drain all available records from all rings as a single batch
3. Aggregate the batch (deduplicate stack IDs, accumulate counts)
4. Compare the BPF-side sample counter with records received — if there's a gap, do a short re-poll and drain again

ReadInto in a loop doesn't cleanly express this because:

• No "batch complete" signal. After ReadInto returns, the caller cannot distinguish "more data pending in loaded rings" from "rings exhausted, next call will epoll_wait." The only way to detect the boundary is to set a short deadline and treat ErrDeadlineExceeded as "batch done" — which conflates "no data available right now" with "timeout expired while waiting for new data."

• Unpredictable deadline behavior. SetDeadline only takes effect when epollRings is empty (since ReadInto skips Wait when rings have data). From the caller's perspective, the configured deadline may or may not be used on any given call, making timeout-based flow control unreliable.

• Forced external buffering. To aggregate a batch, I must buffer records outside the library and guess when to stop calling ReadInto. This effectively reimplements the drain logic in user code.

Proposal

Add a single method:

// DrainAll waits for at least one event (respecting SetDeadline), then
// reads all available records from all rings and returns them as a batch.
// When it returns, all rings are empty — giving the caller a clear batch boundary.
func (pr *Reader) DrainAll() ([]*Record, error)

Why this is minimal:

• Single method, no new types or options
• Reuses SetDeadline for timeout control (no new timeout parameter)
• Does not expose internal state (epollRings, poller, etc.)
• Semantically equivalent to "ReadInto in a loop until all rings are empty" — but packaged so the caller gets the batch boundary for free

Why not BPF ring buffer:
We use perf buffers because our deployment includes older kernels where BPF_MAP_TYPE_RINGBUF is not available. Even on newer kernels, perf buffers remain a valid choice for per-CPU isolation without cross-CPU contention.

Intended usage

for {
    reader.SetDeadline(time.Now().Add(500 * time.Millisecond))
    batch, err := reader.DrainAll()
    if errors.Is(err, os.ErrDeadlineExceeded) {
        continue
    }
    if err != nil {
        return err
    }

    process(batch)

    // Short re-poll if BPF counter indicates missed events
    if bpfSampleCount > uint64(len(batch)) {
        reader.SetDeadline(time.Now().Add(100 * time.Millisecond))
        more, _ := reader.DrainAll()
        if len(more) > 0 {
            process(more)
        }
    }
}

I'm happy to submit a PR if there's interest. The implementation is straightforward — ~30 lines composing existing internals (poller.Wait, loadHead, readRecordFromRing).

cc @lmb @florianl

[ti-mo]

@liuchangyan Thanks for the suggestion. I'm not sure if this was called out in any of the previous threads, but your 'batch boundary' concept is something that's inevitably application-specific. In a world with asynchronous, lockless producers and consumers, there is no guarantee that your suggested DrainAll will yield all the samples you'd expect it to. I think your example of calling DrainAll() in a loop further corroborates this. Without explicit coordination between user and kernel space, you cannot correctly draw a batch boundary.

In the example you gave, if I understand correctly, you could substitute DrainAll() with ReadInto() and end up with equivalent behaviour. We shouldn't give callers the false impression they're getting a batch boundary 'for free'; there is simply no such thing in a ring buffer mechanism.

[liuchangyan]

@liuchangyan Thanks for the suggestion. I'm not sure if this was called out in any of the previous threads, but your 'batch boundary' concept is something that's inevitably application-specific. In a world with asynchronous, lockless producers and consumers, there is no guarantee that your suggested DrainAll will yield all the samples you'd expect it to. I think your example of calling DrainAll() in a loop further corroborates this. Without explicit coordination between user and kernel space, you cannot correctly draw a batch boundary.

In the example you gave, if I understand correctly, you could substitute DrainAll() with ReadInto() and end up with equivalent behaviour. We shouldn't give callers the false impression they're getting a batch boundary 'for free'; there is simply no such thing in a ring buffer mechanism.

---

Thanks for the feedback @ti-mo. I agree that in a pure async ring buffer there's no inherent batch boundary — but my use case has explicit kernel/userspace coordination that makes this well-defined.

My design uses A/B buffer swapping + a BPF-side sample counter:

  1. Read transfer_cnt to determine which buffer (A or B) to consume
  2. EpollWait for events to arrive
  3. Increment transfer_cnt — this signals the BPF program to start writing to the other buffer
  4. Drain all records from the current buffer
  5. Compare BPF-side sample_count with records received — if there's a gap (race window where kernel hasn't published yet), do a short re-poll
  6. Reset counter, clean up stack map

So the batch boundary isn't defined by "the ring happens to be empty" — it's defined by the A/B swap moment. After the swap, BPF writes go to the other side, and the current side has a finite, known number of records (equal to sample_count). The re-poll is just handling the brief race window where records are committed but not yet visible to userspace — not evidence that the boundary is unreliable.

What DrainAll (or ReadAllRings in my fork) gives me is not "a guaranteed complete batch" — it's the ability to read all currently-available records from all rings in one call, without having to loop ReadInto and guess whether the next call will do an epoll_wait or return buffered data. The "batch completeness" guarantee comes from my A/B protocol; the API just need to express "drain what's ready now" cleanly.

Happy to discuss further or adjust the proposal if this changes the picture.

[ti-mo]

[...] but my use case has explicit kernel/userspace coordination that makes this well-defined.

That's my whole point, it doesn't make sense to give users the false impression that that mechanism can be generalized to all use cases.

What DrainAll (or ReadAllRings in my fork) gives me is not "a guaranteed complete batch" — it's the ability to read all currently-available records from all rings in one call, without having to loop ReadInto [...]

.. except now you're calling DrainAll in a loop in the exact same fashion, and you still need to make sure to set deadlines, handle timeouts, and keep trying until your expected sample count is met, just like with ReadInto. I just don't see the value-add, sorry.

On a side note, please read our AI policy, your responses feel heavily generated.