Translate Quip's Loadshed-Lock to Go

[bgwines]

Background / Why?

VTTablets under high load need a mechanism to shed excess requests rather than letting queue depth grow unbounded. The Python loadshed-lock system implements this using the CoDel (Controlled Delay) algorithm — the same algorithm used in Linux's fq_codel queueing discipline — to detect persistent queue buildup and drop low-priority requests before they accumulate latency that compounds downstream.

This PR ports that system to Go. The core challenge is that Python's asyncio provides implicit single-threaded synchronization, while Go requires explicit synchronization under true parallelism. The design uses a single shared sync.Mutex across all layers to mirror Python's single-threaded guarantee, with chan error replacing asyncio.Future for per-request signaling.

Architecture

Lock (public API — owns sync.Mutex)
│
├── Acquire(ctx, valveID) → *SafeUnlock
│   ├── uncontended: grant immediately, start max-age timer
│   └── contended: enqueue → select { req.result | ctx.Done() }
│
├── SafeUnlock.Release(errs...)
│   ├── verify nonce (sync.Once for idempotency)
│   ├── run release callbacks (mutex NOT held)
│   └── dequeue current holder, signal next waiter
│
├── SelfContentionAwareCoDelQueue (valve layer)
│   │
│   │  N valves (one per valve ID, typically one per HTTP request
│   │  on the app server). Each valve permits at most 1 request
│   │  into the CoDel queue; the rest wait in a per-ID FIFO slice.
│   │
│   │  Owns drop orchestration: lockedRunScheduledDrop encapsulates
│   │  finding the lowest-priority droppable request and triggering
│   │  valve promotion.
│   │
│   │         valve "A"           valve "B"         valve "C"
│   │        ┌─────────┐        ┌─────────┐       ┌─────────┐
│   │        │ A₂  A₃  │        │ B₂      │       │ (empty) │
│   │        └────┬────┘        └────┬────┘       └────┬────┘
│   │             │ promote          │ promote         │
│   │             ▼                  ▼                 ▼
│   └───► CoDelQueue ◄───────────────────────────────────────
│         (single shared instance)
│
├── CoDelQueue (CoDel algorithm)
│   ├── *list.List: [A₁] [B₁] [C₁]   (O(1) head pop + O(1) removal)
│   ├── healthy ↔ dropping state machine
│   ├── control law: interval / count^exponent
│   └── schedules drops via injected callback — no timer ownership
│
└── Timers (owned by Lock, scheduled via callback injection)
    ├── dropTimer: time.AfterFunc → sq.lockedRunScheduledDrop
    └── maxAgeTimer: time.AfterFunc → force-release stale holder

Request lifecycle events

Event	Mechanism	Result channel touched?	Request leaves CoDel queue?
Grant	lockedMarkNotDroppable + signal(nil)	Yes — nil	No — holder stays until release
Release	lockedDequeue	No (already signaled)	Yes — removed on release
Drop	lockedDropActive → signal(err)	Yes — error	Yes — removed immediately
Cancel	lockedCancel / lockedRemove	Yes — error	Yes — removed immediately

Summary

• request.go — Request with priority (*float64), result channel, signal() for atomic write to inspectable outcome field + blocking channel, container/list back-pointer for O(1) removal, unexported priorityUndroppable sentinel, droppability derived from priority (no separate boolean)
• codelq.go — CoDel queue + DroppedRequestError: healthy/dropping state machine, control law (interval / count^exponent), drop timer scheduling via injected callback, max 100 drops per timer fire, sojourn-time-based state transitions
• selfcontentionaware_codelq.go — Per-valve-ID serialization: at most 1 request per valve ID in CoDel queue at a time, preventing fan-out patterns from inflating queue depth. Owns lockedRunScheduledDrop which encapsulates drop-finding + valve promotion
• lock.go — Lock (public API) + SafeUnlock with nonce verification, Acquire(ctx, valveID) with explicit valve ID parameter, max-age forced release, cancel-vs-grant race handling, release callbacks with panic recovery

Key design decisions

1. Drop timer via callback injection — CoDelQueue calls an injected scheduleDropTimer(delayNs) internally whenever it needs a timer (matching Python's call_later). Lock provides the callback. This eliminates the error-prone out-parameter pattern that caused missed reschedules.
2. Valve ID as explicit parameter — Acquire(ctx, valveID string) takes the valve ID directly rather than via a config callback. Clearer call sites, no ambient state.
3. Droppability derived from priority — No separate droppable boolean. isDroppable() checks priority != priorityUndroppable. lockedMarkNotDroppable sets priority to the sentinel.
4. Cancel-vs-grant race — Go's non-deterministic select when both ctx.Done() and req.result are ready requires explicit detection: the cancel path checks if l.holder == req after acquiring the mutex, and if so calls releaseInternal to prevent lock orphaning.

Differences from the Python implementation

1. Synchronization: Python relies on asyncio's single-threaded event loop (no locking). Go uses a single sync.Mutex on Lock guarding all state
2. Request signaling: Python uses asyncio.Future. Go uses buffered chan error (cap 1) paired with signal() that writes both an inspectable outcome field and the channel
3. Timer ownership: Both Python and Go inject scheduling into the queue. Python uses loop.call_later(), Go uses an injected func(delayNs int64) callback. Lock owns the actual time.AfterFunc and provides idempotency
4. Cancel-vs-grant race: Python has no equivalent — asyncio's cooperative scheduling prevents it. Go's Acquire uses a double-select with holder-check fallback
5. Context cancellation: Python cancels via asyncio.CancelledError. Go takes context.Context in Acquire with explicit lockedCancel/lockedRemove methods
6. Release idempotency: Python uses __del__ fallback. Go wraps Release in sync.Once
7. Max-age timeout: Python cancels the holder's asyncio.Task. Go uses time.AfterFunc with staleness guard via maxAgeHolder pointer comparison
8. activePerValve map: Added for O(1) lookup of which request is in CoDel per valve ID (Python scans callbacks)
9. Release callbacks: Python awaits async coroutines. Go runs func(error) callbacks without mutex held, with recover() for panic safety

Testing

98 tests across 4 test files:

• codelq_test.go (39 tests) — CoDel algorithm unit tests: FIFO ordering, drop priority selection, state machine transitions, control law math, scheduled drop behavior via test recorder
• selfcontentionaware_codelq_test.go (14 tests) — Valve logic: direct entry, valving, promotion on dequeue/drop/cancel, outstanding count lifecycle, FIFO within contention
• lock_test.go (28 tests) — Lock behavior: mutual exclusion, FIFO ordering, context cancellation, cancel-vs-grant race, self-contention serialization, CoDel drops, max-age timeout, SafeUnlock idempotency, release callbacks with panic recovery
• lock_stress_test.go (16 tests) — High-concurrency stress tests with -race: 200 goroutines, rapid acquire/release, cancel-vs-grant races, drop timer + cancel races, max-age under load, goroutine leak detection, starvation prevention, self-contention stress (mutual exclusion, FIFO valve ordering, drop+promotion chains, cancel-in-valve, mixed cancel/drop/grant, sustained high concurrency)
• lock_race_test.go (1 test) — 50k-iteration targeted reproduction of the cancel-vs-grant race

All tests pass with -race -count=5.

---

AI disclosure: Claude Code assisted with development. Every line of code was either written by or carefully reviewed by me :)

[rcohen2000]

this pattern appears to not be idiomatic in go. if the usage of that the lock is taken and released in the same function, maybe we don't need the additional proptection and can get away with a simpler abstraction of an "erroring mutex"

err := emu.Lock()
if err != nil {
	return ...
}
defer emu.Unlock()

[bgwines]

we discussed this verbally today. Assuming we keep the lock(semaphore) holder inside the codelq, the use-case of transactions means we can't do this since it's multiple requests from the client, which implies multipl estackframes.

Fortunately, it's not completely without precedent. Claude claims idiomatic Go for this is store the release handle on the struct that owns the resource lifetime, which is StatefulConnection. It already does exactly this with the DB connection itself:

  // stateful_connection.go
  type StatefulConnection struct {
      dbConn  *connpool.PooledConn   // acquired in Begin, released in Commit/Rollback
      ConnID  tx.ConnID
      // ...
  }

  func (sc *StatefulConnection) Release(reason tx.ReleaseReason) {
      sc.pool.unregister(sc.ConnID, ...)
      sc.dbConn.Recycle()    // ← returns conn to pool
      sc.dbConn = nil
  }

[rcohen2000]

we need to put some thought into whether 1 is the right number here. it made sense when doing single dispatch, but when we do multiple and possibly variable dispatch i don't know offhand what the right answer is.

[bgwines]

for Quip we never overrode the default we set, 100ns -- maybe just don't even expose this?

[bgwines]

Code Review: Interactive Walkthrough Results

Brett and I did a function-by-function review comparing this Go translation against the Python original. 23 items identified — 1 bug, 3 architectural, 5 refactoring, 14 naming/clarity.

Bug Found & Fixed

Cancel-vs-grant race in Acquire's double-select (commit 125651dcea)

Race window: releaser sets l.holder = next and unlocks, but BEFORE calling next.signal(nil), the cancelling goroutine's inner non-blocking select takes default (done channel empty), acquires the mutex, and calls lockedCancel on itself — the current holder. Reproduction rate: 17% (17,202 / 100,000 iterations). Fixed by checking l.holder == req after acquiring mutex in the default branch, draining the pending signal, then calling releaseInternal(). Stress test: 50,000 iterations, 0 failures.

Architecture (3 items)

#	Summary
8	Queue should manage its own drop timer via callback injection (faithful to Python's battle-tested design)
20	Eliminate maxAgeHolder — restructure release so all state manipulation is one atomic mutex hold, run callbacks after granting next holder
21	Make valve ID a mandatory string parameter to Acquire, not a config callable

Refactoring (5 items)

#	Summary
15	Extract lockedClearActiveAndPromote shared helper from promote-on-grant and promote-on-evict
18	Extract release callback execution into runReleaseCBs helper
19	Extract lockedGrantNext helper for duplicated grant logic
13	Inline decrementOutstanding in clearDone loop
23	Collapse NewLock / NewLockWithClock into single constructor

Naming & Clarity (14 items)

#	Summary
1	Use explicit constant for undroppable priority, not nil
2	Remove droppable instance variable — derive from priority
3	Remove enqueuedAt parameter from newRequest
4	Rename done channel to result
5	Remove redundant q.droppableLen > 0 guard
6	Rename clockFunc to now
7	Rename enqueuedAt to enqueuedAtNs
9	Rename lockedCancel to lockedRemove
10	Rename contentionID to valveID throughout
11	Add comment explaining empty valve ID bypass semantics
12	Rename activeRequests to clarify it only tracks valve-ID requests
14	Rename SelfContentionAwareCoDelQueue receiver s → q
17	Fix "run release callbacks without mutex held" comment to explain WHY
22	Rename l.sq to l.q

All items will be addressed in a follow-up commit on this branch.

[salesforce-cla]

Thanks for the contribution! Before we can merge this, we need @mattlord @vitess-bot @nickvanw @maxenglander @mhamza15 @arthurschreiber to sign the Salesforce Inc. Contributor License Agreement.

[bgwines]

valves are FIFO, I'll touch this up in a subsequent PR. I don't think this is O(N) based on the caller pattern, but we can make that a hard guarantee by adjusting the implementation a bit.

[bgwines]

squash-merged in 4191f37