Presently, each subchannel using ALTS credentials pins a 32KB write buffer in memory. When using DirectPath with a load balancing policy that connects to multiple backends (e.g., round-robin, weighted-round-robin), a client may have hundreds of open subchannels, even though only a few have ongoing RPCs at any given time. This leads to significant memory usage that can be reduced.
The writer needs the buffer only when a write is requested by gRPC; it does not need to retain the buffer at other times. The buffered writer (bufWriter) in the gRPC transport uses a sync.Pool of byte slices to allow write buffers to be reused by other subchannels when not in use. A similar mechanism can be adopted in the ALTS conn to reduce memory usage. However, there are a few notable differences compared to bufWriter:
- The ALTS
conn write buffer may grow larger than 32KB.
- The ALTS
conn write buffer may not need the entire 32KB buffer for smaller writes.
gRPC already maintains a default buffer pool that is used by other packages to acquire buffers of any size. The ALTS conn can fetch a buffer from this pool during the Write operation and return it once the data is written to the wire. The global default memory pool initializes (clears) all buffers it returns, which incurs slight overhead. To avoid this, the alts package can maintain its own package-private buffer pool implementation that skips initialization. To share the existing buffer pool implementation, the binaryTieredBufferPool can be moved to an internal/mem package, exposing a new constructor that allows injecting custom sizedBufferPool and simpleBufferPool builders.
For each subchannel, gRPC presently pins the following buffers:
- Read buffer of 32KB in the transport.
- Read buffer of 32KB in the ALTS credentials.
- Write buffer of 32KB in the ALTS credentials.
With this change, one of these buffers can be unpinned, potentially reducing the memory usage of a subchannel by 33%.
Related issue: googleapis/google-cloud-go#13753
Presently, each subchannel using ALTS credentials pins a 32KB write buffer in memory. When using DirectPath with a load balancing policy that connects to multiple backends (e.g., round-robin, weighted-round-robin), a client may have hundreds of open subchannels, even though only a few have ongoing RPCs at any given time. This leads to significant memory usage that can be reduced.
The writer needs the buffer only when a write is requested by gRPC; it does not need to retain the buffer at other times. The buffered writer (
bufWriter) in the gRPC transport uses async.Poolof byte slices to allow write buffers to be reused by other subchannels when not in use. A similar mechanism can be adopted in the ALTSconnto reduce memory usage. However, there are a few notable differences compared tobufWriter:connwrite buffer may grow larger than 32KB.connwrite buffer may not need the entire 32KB buffer for smaller writes.gRPC already maintains a default buffer pool that is used by other packages to acquire buffers of any size. The ALTS
conncan fetch a buffer from this pool during theWriteoperation and return it once the data is written to the wire. The global default memory pool initializes (clears) all buffers it returns, which incurs slight overhead. To avoid this, thealtspackage can maintain its own package-private buffer pool implementation that skips initialization. To share the existing buffer pool implementation, thebinaryTieredBufferPoolcan be moved to aninternal/mempackage, exposing a new constructor that allows injecting customsizedBufferPoolandsimpleBufferPoolbuilders.For each subchannel, gRPC presently pins the following buffers:
With this change, one of these buffers can be unpinned, potentially reducing the memory usage of a subchannel by 33%.
Related issue: googleapis/google-cloud-go#13753