server: Set a pprof label on new stream goroutines by dfinkel · Pull Request #9082 · grpc/grpc-go

dfinkel · 2026-04-21T18:11:18Z

To make stack-traces and some profiles more useful, change sets goroutine labels indicating which gRPC method is being handled.

Goroutine labels are inherited by child goroutines, so this provides useful context in profiles and traces for work that's been farmed out to child goroutines.

These currently show up in three places:

trace labels in pprof CPU profiles
trace labels in runtime/pprof (and http/pprof) debug=0 goroutine profiles (which are pprof format)
debug=1 aggregated text-format goroutine profiles

For Go 1.27, golang/go#76349 adds goroutine labels to tracebacks and by extension debug=2 pprof text-based profiles for go 1.27+ modules.

The naming of the goroutine label currently matches the opentelemetry RPC method tag, and has some similarities to the current proposal for goroutine tag naming for tests in golang/go#75047. I.e. this uses grpc.method.

This change avoids setting anything on the client side due to the lower utility and goroutine lifetime issues.

Include a GRPC_GO_SERVER_GOROUTINE_LABELS environment variable to allow users to easily opt-in of setting these goroutine labels. The value the form grpc.method=true to enable a specific goroutine label, and has special values of all and none which enable and disable all registered goroutine labels respectively.

RELEASE NOTES:

server: Set runtime/pprof goroutine labels for incoming method streams. This may be enabled with the GRPC_GO_SERVER_GOROUTINE_LABELS=grpc.method=true environment variable for just the new goroutine label or GRPC_GO_SERVER_GOROUTINE_LABELS=all to enable all goroutine labels that might be added later. none is available for blanket disabling, as well.

codecov · 2026-04-21T18:15:49Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.08%. Comparing base (1c63fa5) to head (3854612).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9082      +/-   ##
==========================================
- Coverage   83.30%   83.08%   -0.23%     
==========================================
  Files         413      413              
  Lines       33482    33512      +30     
==========================================
- Hits        27892    27842      -50     
- Misses       4190     4243      +53     
- Partials     1400     1427      +27

Files with missing lines	Coverage Δ
internal/envconfig/envconfig.go	`100.00% <100.00%> (ø)`
server.go	`82.83% <100.00%> (-0.15%)`	⬇️

... and 28 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

To make stack-traces and some profiles more useful, change sets goroutine labels indicating which gRPC method is being handled. Goroutine labels are inherited by child goroutines, so this provides useful context in profiles and traces for work that's been farmed out to child goroutines. These currently show up in three places: - trace labels in pprof CPU profiles - trace labels in runtime/pprof (and http/pprof) debug=0 goroutine profiles (which are pprof format) - debug=1 aggregated text-format goroutine profiles For Go 1.27, golang/go#76349 adds goroutine labels to tracebacks and by extension debug=2 pprof text-based profiles for go 1.27+ modules. The naming of the goroutine label currently matches the opencensus RPC method tag, and has some similarities to the current proposal for goroutine tag naming for tests in golang/go#75047. I.e. this uses `grpc.server.method`. This change avoids setting anything on the client side due to the lower utility and goroutine lifetime issues. Include a GRPC_SERVER_METHOD_GOROUTINE_LABELS environment variable to allow users to easily opt-out of setting these goroutine labels. RELEASE NOTES: * server: Set runtime/pprof goroutine labels for incoming method streams. This may be disabled with the GRPC_SERVER_METHOD_GOROUTINE_LABELS=false environment variable. Fixes grpc#9010

arjan-bal · 2026-04-22T08:28:02Z

+	if envconfig.LabelServerStreamGoroutines {
+		// This method always runs in its own goroutine, so we can set a
+		// goroutine label without needing to restore a previous context.
+		ctx = pprof.WithLabels(ctx, pprof.Labels("grpc.server.method", stream.Method()))


Can we use the key "grpc.method" to be consistent with the OpenTelemetry plugins? gRPC is moving away from OpenCensus in favour of OpenTelemetry. See https://grpc.io/docs/guides/opentelemetry-metrics/#server-instruments and https://grpc.io/docs/guides/opentelemetry-metrics/#labelsattributes

arjan-bal · 2026-04-22T08:45:51Z

+	// LabelServerStreamGoroutines controls setting [runtime/pprof.Labels] on the
+	// goroutines spawned to handle incoming requests on the server.
+	// Set "GRPC_SERVER_METHOD_GOROUTINE_LABELS" to "false" to disable.
+	LabelServerStreamGoroutines = boolFromEnv("GRPC_SERVER_METHOD_GOROUTINE_LABELS", true)


Thinking ahead to when we might have more labels, maintaining one environment variable per label seems tedious. Could we instead use a single variable whose value is a comma-separated list of key-value pairs? For example: GRPC_GO_PPROF_LABELS_ENABLED="grpc.method=false".

This would be similar to the parsing of the GODEBUG environment variable

Good idea
... and Done.

Add support for setting a comma-separated list of goroutine labels to enable/disable. Converting this configuration from a bool to a bitfield with a default instead of a solitary boolean made it clear that there needed to be ways to robustly enable disable and toggle the behavior. Tests will be in another commit.

Add a couple test-cases that verify that the contexts include the expected goroutine labels.

Add a test covering the goroutine label configuration parsing.

arjan-bal · 2026-05-08T08:28:39Z

The benchmark's reported QPS typically fluctuates by ±1%, even with identical code. Using the -bufconn flag forces an in-memory transport instead of TCP, which helps reduce this variance. Across several runs, I observed the difference narrow to between -0.7% and +0.3%.

I discussed this with the other maintainers, and we concluded that enabling this by default would cause minor regressions for all users, even those who aren't profiling or using these new labels. While the extra allocations seem unavoidable, developers who need the metadata will likely be willing to pay the performance cost. Therefore, we should make this feature opt-in rather than opt-out.

Also, since users currently have no way to toggle all labels at once, I think we should support special keywords like "all" and "none" as valid environment variable values to simplify this experience.

Make it easy to opt into all goroutine labels now that we're keeping it default disabled.

Add a "none" special value that pairs with "all" so users can opt out of any goroutine labels that happen to have been enabled elsewhere.

…bels

Another commit appears to have removed this import while this change was in-flight. Add it back now that it's needed again.

dfinkel · 2026-05-11T17:59:46Z

Ok, I've updated my branch to default-disabled, updated the PR description/release-notes and merged master because the envconfig import disappeared in a merged PR which caused the tests to fail in CI.

arjan-bal · 2026-05-12T07:55:24Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a mechanism to set runtime/pprof labels on gRPC server goroutines, specifically enabling the grpc.method label via the GRPC_GO_SERVER_GOROUTINE_LABELS environment variable. The implementation includes a bitfield-based configuration and comprehensive tests. Feedback highlights a compatibility issue where the use of strings.SplitSeq and range-over-func syntax breaks support for Go 1.22, suggesting a fallback to strings.Split. Additionally, it was noted that the 'all' and 'none' keywords are currently only recognized as standalone values and should ideally be supported within comma-separated lists for more flexible configuration.

arjan-bal

LGTM! Adding a second reviewer.

dfinkel · 2026-05-12T14:09:01Z

Thanks for the thorough review!

easwars

LGTM, modulo minor nits

easwars · 2026-05-12T18:25:54Z

+	// requests on the server.
+	// Set "GRPC_GO_SERVER_GOROUTINE_LABELS" to "grpc.method=true" to
+	// enable this grpc.method label, or "all" to enable all valid labels.
+	// This variable is a bit-field.


Nit: Does it make sense to get rid of this last sentence? I'm wondering if it makes sense for a user reading this docstring.

I'm not sure.

I think it's rare that a normal user would look at the doc-comments in this package, but I added that info to this doc-comment because some of the other package-level variables in this package have similar comments.

I'm definitely open to deleting this comment or moving some of the syntax/usage info to a doc-comment on goroutineLabelsFromEnv.

@arjan-bal , what do you think?

easwars · 2026-05-12T18:33:38Z

+			ctxLabels := pprofCtxCollectLabels(ctx)
+			if val, ok := ctxLabels["grpc.method"]; !ok {
+				t.Errorf("missing \"grpc.method\" label; found labels: %v", ctxLabels)
+			} else if expVal := "/grpc.testing.TestService/EmptyCall"; val != expVal {


Nit: Here and elsewhere s/expVal/wantVal

See: https://google.github.io/styleguide/go/decisions#got-before-want

easwars · 2026-05-12T18:34:12Z

+	_, err := ss.Client.EmptyCall(ctx, &testpb.Empty{})
+	if err != nil {


Nit: The assignment and the conditional can be on the same line. Here and elsewhere in this test.

Good point.
Done.

dfinkel · 2026-05-28T18:19:55Z

Thanks @easwars and @arjan-bal for the excellent reviews!

Fixes grpc#9010 To make stack-traces and some profiles more useful, change sets goroutine labels indicating which gRPC method is being handled. Goroutine labels are inherited by child goroutines, so this provides useful context in profiles and traces for work that's been farmed out to child goroutines. These currently show up in three places: - trace labels in pprof CPU profiles - trace labels in runtime/pprof (and http/pprof) debug=0 goroutine profiles (which are pprof format) - debug=1 aggregated text-format goroutine profiles For Go 1.27, golang/go#76349 adds goroutine labels to tracebacks and by extension debug=2 pprof text-based profiles for go 1.27+ modules. The naming of the goroutine label currently matches the opentelemetry RPC method tag, and has some similarities to the current proposal for goroutine tag naming for tests in golang/go#75047. I.e. this uses `grpc.method`. This change avoids setting anything on the client side due to the lower utility and goroutine lifetime issues. Include a `GRPC_GO_SERVER_GOROUTINE_LABELS` environment variable to allow users to easily opt-in of setting these goroutine labels. The value the form `grpc.method=true` to enable a specific goroutine label, and has special values of `all` and `none` which enable and disable all registered goroutine labels respectively. RELEASE NOTES: * server: Set runtime/pprof goroutine labels for incoming method streams. This may be enabled with the `GRPC_GO_SERVER_GOROUTINE_LABELS=grpc.method=true` environment variable for just the new goroutine label or `GRPC_GO_SERVER_GOROUTINE_LABELS=all` to enable all goroutine labels that might be added later. `none` is available for blanket disabling, as well.

…(#17) This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [google.golang.org/grpc](https://github.com/grpc/grpc-go) | `v1.81.1` → `v1.82.0` | ![age](https://developer.mend.io/api/mc/badges/age/go/google.golang.org%2fgrpc/v1.82.0?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/go/google.golang.org%2fgrpc/v1.81.1/v1.82.0?slim=true) | --- ### Release Notes <details> <summary>grpc/grpc-go (google.golang.org/grpc)</summary> ### [`v1.82.0`](https://github.com/grpc/grpc-go/releases/tag/v1.82.0): Release 1.82.0 [Compare Source](grpc/grpc-go@v1.81.1...v1.82.0) ### Behavior Changes - server: Remove support for `GRPC_GO_EXPERIMENTAL_DISABLE_STRICT_PATH_CHECKING` environment varibale. Strict incoming RPC path validation (which has been the default since `v1.79.3`) can no longer be disabled. ([#9112](grpc/grpc-go#9112)) - transport: Add environment variable to change the default max header list size from `16MB` to `8KB`. This may be enabled by setting `GRPC_GO_EXPERIMENTAL_ENABLE_8KB_DEFAULT_HEADER_LIST_SIZE=true`. This will be enabled by default in a subsequent release. ([#9019](grpc/grpc-go#9019)) - balancer: Load Balancing policy registry is now case-sensitive. Set `GRPC_GO_EXPERIMENTAL_CASE_SENSITIVE_BALANCER_REGISTRIES=false` (and file an issue) to revert to case-insensitive behavior. ([#9017](grpc/grpc-go#9017)) ### New Features - experimental/stats: Expose a new API, `NewContextWithLabelCallback`, to register a callback that is invoked when telemetry labels are added. ([#8877](grpc/grpc-go#8877)) - Special Thanks: [@seth-epps](https://github.com/seth-epps) - client: Return a portion of the response body in the error message, when the client receives an unexpected non-gRPC HTTP response, to make debugging easier. ([#8929](grpc/grpc-go#8929)) - Special Thanks: [@chengxilo](https://github.com/chengxilo) - server: Add environment variable `GRPC_GO_SERVER_GOROUTINE_LABELS` that controls setting `runtime/pprof.Labels` on goroutines spawned by the server. Set `GRPC_GO_SERVER_GOROUTINE_LABELS=grpc.method=true` to add the `grpc.method` label on goroutines spawned to handle incoming requests. ([#9082](grpc/grpc-go#9082)) - Special Thanks: [@dfinkel](https://github.com/dfinkel) ### Bug Fixes - xds/server: Fix a memory leak of HTTP filter instances occurring when route configurations are updated in-place during a Route Discovery Service (RDS) update. ([#9138](grpc/grpc-go#9138)) - grpc: In the deprecated `gzip` Compressor (used via the deprecated `WithCompressor` dial option), enforce the `MaxRecvMsgSize` limit on the decompressed message buffer, preventing excessive memory allocation from highly compressed payloads. ([#9114](grpc/grpc-go#9114)) - Special Thanks: [@evilgensec](https://github.com/evilgensec) - stats/opentelemetry: Record retry attempts, `grpc.previous-rpc-attempts`, at the call level and not the attempt level. ([#8923](grpc/grpc-go#8923)) - encoding: Ensure `Close()` is always called on readers returned from `Compressor.Decompress` if possible. ([#9135](grpc/grpc-go#9135)) - channelz: Fix the `LastMessageSentTimestamp` and `LastMessageReceivedTimestamp` fields in `SocketMetrics` to ensure they contain correct timestamp values. ([#9109](grpc/grpc-go#9109)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.erwanleboucher.dev/eleboucher/talos-mcp/pulls/17

…(#77) This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [google.golang.org/grpc](https://github.com/grpc/grpc-go) | `v1.81.1` → `v1.82.0` | ![age](https://developer.mend.io/api/mc/badges/age/go/google.golang.org%2fgrpc/v1.82.0?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/go/google.golang.org%2fgrpc/v1.81.1/v1.82.0?slim=true) | --- ### Release Notes <details> <summary>grpc/grpc-go (google.golang.org/grpc)</summary> ### [`v1.82.0`](https://github.com/grpc/grpc-go/releases/tag/v1.82.0): Release 1.82.0 [Compare Source](grpc/grpc-go@v1.81.1...v1.82.0) ### Behavior Changes - server: Remove support for `GRPC_GO_EXPERIMENTAL_DISABLE_STRICT_PATH_CHECKING` environment varibale. Strict incoming RPC path validation (which has been the default since `v1.79.3`) can no longer be disabled. ([#9112](grpc/grpc-go#9112)) - transport: Add environment variable to change the default max header list size from `16MB` to `8KB`. This may be enabled by setting `GRPC_GO_EXPERIMENTAL_ENABLE_8KB_DEFAULT_HEADER_LIST_SIZE=true`. This will be enabled by default in a subsequent release. ([#9019](grpc/grpc-go#9019)) - balancer: Load Balancing policy registry is now case-sensitive. Set `GRPC_GO_EXPERIMENTAL_CASE_SENSITIVE_BALANCER_REGISTRIES=false` (and file an issue) to revert to case-insensitive behavior. ([#9017](grpc/grpc-go#9017)) ### New Features - experimental/stats: Expose a new API, `NewContextWithLabelCallback`, to register a callback that is invoked when telemetry labels are added. ([#8877](grpc/grpc-go#8877)) - Special Thanks: [@seth-epps](https://github.com/seth-epps) - client: Return a portion of the response body in the error message, when the client receives an unexpected non-gRPC HTTP response, to make debugging easier. ([#8929](grpc/grpc-go#8929)) - Special Thanks: [@chengxilo](https://github.com/chengxilo) - server: Add environment variable `GRPC_GO_SERVER_GOROUTINE_LABELS` that controls setting `runtime/pprof.Labels` on goroutines spawned by the server. Set `GRPC_GO_SERVER_GOROUTINE_LABELS=grpc.method=true` to add the `grpc.method` label on goroutines spawned to handle incoming requests. ([#9082](grpc/grpc-go#9082)) - Special Thanks: [@dfinkel](https://github.com/dfinkel) ### Bug Fixes - xds/server: Fix a memory leak of HTTP filter instances occurring when route configurations are updated in-place during a Route Discovery Service (RDS) update. ([#9138](grpc/grpc-go#9138)) - grpc: In the deprecated `gzip` Compressor (used via the deprecated `WithCompressor` dial option), enforce the `MaxRecvMsgSize` limit on the decompressed message buffer, preventing excessive memory allocation from highly compressed payloads. ([#9114](grpc/grpc-go#9114)) - Special Thanks: [@evilgensec](https://github.com/evilgensec) - stats/opentelemetry: Record retry attempts, `grpc.previous-rpc-attempts`, at the call level and not the attempt level. ([#8923](grpc/grpc-go#8923)) - encoding: Ensure `Close()` is always called on readers returned from `Compressor.Decompress` if possible. ([#9135](grpc/grpc-go#9135)) - channelz: Fix the `LastMessageSentTimestamp` and `LastMessageReceivedTimestamp` fields in `SocketMetrics` to ensure they contain correct timestamp values. ([#9109](grpc/grpc-go#9109)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.erwanleboucher.dev/eleboucher/runner-k8s-plugin/pulls/77

This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [google.golang.org/grpc](https://github.com/grpc/grpc-go) | `v1.81.1` → `v1.82.0` | ![age](https://developer.mend.io/api/mc/badges/age/go/google.golang.org%2fgrpc/v1.82.0?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/go/google.golang.org%2fgrpc/v1.81.1/v1.82.0?slim=true) | --- ### Release Notes <details> <summary>grpc/grpc-go (google.golang.org/grpc)</summary> ### [`v1.82.0`](https://github.com/grpc/grpc-go/releases/tag/v1.82.0): Release 1.82.0 [Compare Source](grpc/grpc-go@v1.81.1...v1.82.0) ### Behavior Changes - server: Remove support for `GRPC_GO_EXPERIMENTAL_DISABLE_STRICT_PATH_CHECKING` environment varibale. Strict incoming RPC path validation (which has been the default since `v1.79.3`) can no longer be disabled. ([#9112](grpc/grpc-go#9112)) - transport: Add environment variable to change the default max header list size from `16MB` to `8KB`. This may be enabled by setting `GRPC_GO_EXPERIMENTAL_ENABLE_8KB_DEFAULT_HEADER_LIST_SIZE=true`. This will be enabled by default in a subsequent release. ([#9019](grpc/grpc-go#9019)) - balancer: Load Balancing policy registry is now case-sensitive. Set `GRPC_GO_EXPERIMENTAL_CASE_SENSITIVE_BALANCER_REGISTRIES=false` (and file an issue) to revert to case-insensitive behavior. ([#9017](grpc/grpc-go#9017)) ### New Features - experimental/stats: Expose a new API, `NewContextWithLabelCallback`, to register a callback that is invoked when telemetry labels are added. ([#8877](grpc/grpc-go#8877)) - Special Thanks: [@seth-epps](https://github.com/seth-epps) - client: Return a portion of the response body in the error message, when the client receives an unexpected non-gRPC HTTP response, to make debugging easier. ([#8929](grpc/grpc-go#8929)) - Special Thanks: [@chengxilo](https://github.com/chengxilo) - server: Add environment variable `GRPC_GO_SERVER_GOROUTINE_LABELS` that controls setting `runtime/pprof.Labels` on goroutines spawned by the server. Set `GRPC_GO_SERVER_GOROUTINE_LABELS=grpc.method=true` to add the `grpc.method` label on goroutines spawned to handle incoming requests. ([#9082](grpc/grpc-go#9082)) - Special Thanks: [@dfinkel](https://github.com/dfinkel) ### Bug Fixes - xds/server: Fix a memory leak of HTTP filter instances occurring when route configurations are updated in-place during a Route Discovery Service (RDS) update. ([#9138](grpc/grpc-go#9138)) - grpc: In the deprecated `gzip` Compressor (used via the deprecated `WithCompressor` dial option), enforce the `MaxRecvMsgSize` limit on the decompressed message buffer, preventing excessive memory allocation from highly compressed payloads. ([#9114](grpc/grpc-go#9114)) - Special Thanks: [@evilgensec](https://github.com/evilgensec) - stats/opentelemetry: Record retry attempts, `grpc.previous-rpc-attempts`, at the call level and not the attempt level. ([#8923](grpc/grpc-go#8923)) - encoding: Ensure `Close()` is always called on readers returned from `Compressor.Decompress` if possible. ([#9135](grpc/grpc-go#9135)) - channelz: Fix the `LastMessageSentTimestamp` and `LastMessageReceivedTimestamp` fields in `SocketMetrics` to ensure they contain correct timestamp values. ([#9109](grpc/grpc-go#9109)) </details> --- ### Configuration 📅 **Schedule**: (in timezone America/Los_Angeles) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate).  Co-authored-by: Renovate Bot <renovate-bot@geekfarm.org> Reviewed-on: https://git.geekfarm.org/wu/keyop-messenger/pulls/7

…/forgejo) (#13580) This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [google.golang.org/grpc](https://github.com/grpc/grpc-go) | `v1.79.3` → `v1.82.1` | ![age](https://developer.mend.io/api/mc/badges/age/go/google.golang.org%2fgrpc/v1.82.1?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/go/google.golang.org%2fgrpc/v1.79.3/v1.82.1?slim=true) | --- ### gRPC-Go: xDS RBAC and HTTP/2 Vulnerabilities [GHSA-hrxh-6v49-42gf](GHSA-hrxh-6v49-42gf) <details> <summary>More information</summary> #### Details Multiple security vulnerabilities have been identified and addressed in grpc-go affecting the xDS RBAC authorization engine (internal/xds/rbac) and the HTTP/2 transport server implementation (internal/transport). These vulnerabilities could result in: - Authorization Bypass (Fail-Open) when translating xDS RBAC policies containing `Metadata` or `RequestedServerName` fields. - Denial of Service (High CPU Consumption) due to an HTTP/2 Rapid Reset mitigation bypass during client-initiated stream resets. - Denial of Service (Server Panic) when parsing crafted xDS RBAC policies containing `NOT` rules around unsupported fields. ##### Impact _What kind of vulnerability is it? Who is impacted?_ ##### xDS RBAC Authorization Bypass via `Metadata` & `RequestedServerName` matchers - Affected Component: xDS RBAC - Impact: When building policy matchers for gRPC RBAC from xDS configurations, unsupported `permission` and `principal` rules (specifically `Metadata` and `RequestedServerName`) were silently ignored and treated as no-ops. - If an authorization policy relied purely on these matchers for access control, treating those rules as no-ops effectively removed the restrictions. - If these unsupported rules were nested inside logical `NOT` rules (`Permission_NotRule` / `Principal_NotId`) or multi-condition `OR/AND` rules, silently dropping them changed the boolean logic flow of the authorization engine. As a result, policy evaluation decisions could fail open, allowing unauthorized clients to access protected gRPC services or resources. ##### HTTP/2 Rapid Reset Mitigation Bypass / Denial of Service via Stream Aborts - Affected Component: HTTP/2 transport - Impact: Earlier mitigations in grpc-go for HTTP/2 Rapid Reset only applied threshold checks to items that directly resulted in control frames being written back to the wire, such as `SETTINGS` ACKs or server-initiated `RST_STREAM`s. When a client initiated a rapid flood of stream creation (`HEADERS`) immediately followed by stream termination `RST_STREAM`, items queued up in the control buffer without counting against the transport response frame threshold. An attacker can repeatedly trigger this flood sequence to bypass reader blocking, resulting in high CPU usage, and Denial of Service (DoS). ##### Denial of Service (Panic) in xDS RBAC Engine via Unsupported Fields inside NOT Rules - Affected Component: xDS RBAC - Impact: The xDS RBAC policy translators recursively generate matchers for nested rules. When a `NOT` rule wrapped an unsupported or unhandled field (such as `SourcedMetadata`), the recursive step returned an empty matcher. This could result in a runtime panic when the RBAC engine attempts to authorize an incoming request. An attacker or misconfigured/malicious xDS management server delivering an LDS/RDS update containing a `NOT` rule around an unhandled field causes the gRPC server process to crash immediately (CWE-248 / Denial of Service). ##### Patches _Has the problem been patched? What versions should users upgrade to?_ All three issues have been fixed in `master` and will be released in 1.82.1 shortly. ##### Workarounds _Is there a way for users to fix or remediate the vulnerability without upgrading?_ If upgrading grpc-go immediately is not possible, apply the following workarounds based on your deployment architecture: * For xDS RBAC Vulnerabilities & Panics: Ensure that upstream xDS management servers do not push RBAC policies containing `Metadata`, `RequestedServerName`, or `NOT` rules wrapping unsupported fields (such as `SourcedMetadata`) to grpc-go servers. * For HTTP/2 Rapid Reset DOS: Configure upstream reverse proxies or load balancers (such as Envoy) with strict HTTP/2 `max_concurrent_streams` limits and active rate limiting on `RST_STREAM` frequency per connection. ##### Severity | Vulnerability | Qualitative Severity | Approximate CVSS v3.1 Score | Primary Impact | | :--- | :--- | :--- | :--- | | **xDS RBAC Authorization Bypass** | **High** | `8.2` | Unauthorized Access / Fail-Open | | **HTTP/2 Rapid Reset DOS Bypass** | **High** | `7.5` | High CPU Consumption / Denial of Service | | **xDS RBAC Engine Server Panic** | **Medium** | `5.9` | Process Crash / Denial of Service | #### Severity - CVSS Score: 8.8 / 10 (High) - Vector String: `CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:N/VI:H/VA:H/SC:N/SI:N/SA:N` #### References - [https://github.com/grpc/grpc-go/security/advisories/GHSA-hrxh-6v49-42gf](https://github.com/grpc/grpc-go/security/advisories/GHSA-hrxh-6v49-42gf) - [https://github.com/grpc/grpc-go/pull/9236](https://github.com/grpc/grpc-go/pull/9236) - [https://github.com/grpc/grpc-go/commit/4ea465d4ab98013f72a142fe0fc89c19770b2935](https://github.com/grpc/grpc-go/commit/4ea465d4ab98013f72a142fe0fc89c19770b2935) - [https://github.com/grpc/grpc-go](https://github.com/grpc/grpc-go) - [https://github.com/grpc/grpc-go/releases/tag/v1.82.1](https://github.com/grpc/grpc-go/releases/tag/v1.82.1) This data is provided by [OSV](https://osv.dev/vulnerability/GHSA-hrxh-6v49-42gf) and the [GitHub Advisory Database](https://github.com/github/advisory-database) ([CC-BY 4.0](https://github.com/github/advisory-database/blob/main/LICENSE.md)). </details> --- ### Release Notes <details> <summary>grpc/grpc-go (google.golang.org/grpc)</summary> ### [`v1.82.1`](https://github.com/grpc/grpc-go/releases/tag/v1.82.1): Release 1.82.1 [Compare Source](grpc/grpc-go@v1.82.0...v1.82.1) ### Security - server: Stop reading from the connection when flooded by HTTP/2 frames. The default value for this limit is 100 frames, excluding DATA and HEADERS, and may be changed by setting environment variable `GRPC_GO_EXPERIMENTAL_CONTROL_BUFFER_THROTTLE_LIMIT`. - xds/rbac: Support `Metadata` and `RequestedServerName` permissions matcher fields. If present in a DENY rule, previously these would be ignored and fail-open. - xds/rbac: Fix panic when parsing unsupported fields in `NotRule`/`NotId` permissions. - xds/rbac: Support the deprecated `source_ip` principal identifier by treating it as equivalent to `direct_remote_ip`. ### [`v1.82.0`](https://github.com/grpc/grpc-go/releases/tag/v1.82.0): Release 1.82.0 [Compare Source](grpc/grpc-go@v1.81.1...v1.82.0) ### Behavior Changes - server: Remove support for `GRPC_GO_EXPERIMENTAL_DISABLE_STRICT_PATH_CHECKING` environment varibale. Strict incoming RPC path validation (which has been the default since `v1.79.3`) can no longer be disabled. ([#9112](grpc/grpc-go#9112)) - transport: Add environment variable to change the default max header list size from `16MB` to `8KB`. This may be enabled by setting `GRPC_GO_EXPERIMENTAL_ENABLE_8KB_DEFAULT_HEADER_LIST_SIZE=true`. This will be enabled by default in a subsequent release. ([#9019](grpc/grpc-go#9019)) - balancer: Load Balancing policy registry is now case-sensitive. Set `GRPC_GO_EXPERIMENTAL_CASE_SENSITIVE_BALANCER_REGISTRIES=false` (and file an issue) to revert to case-insensitive behavior. ([#9017](grpc/grpc-go#9017)) ### New Features - experimental/stats: Expose a new API, `NewContextWithLabelCallback`, to register a callback that is invoked when telemetry labels are added. ([#8877](grpc/grpc-go#8877)) - Special Thanks: [@seth-epps](https://github.com/seth-epps) - client: Return a portion of the response body in the error message, when the client receives an unexpected non-gRPC HTTP response, to make debugging easier. ([#8929](grpc/grpc-go#8929)) - Special Thanks: [@chengxilo](https://github.com/chengxilo) - server: Add environment variable `GRPC_GO_SERVER_GOROUTINE_LABELS` that controls setting `runtime/pprof.Labels` on goroutines spawned by the server. Set `GRPC_GO_SERVER_GOROUTINE_LABELS=grpc.method=true` to add the `grpc.method` label on goroutines spawned to handle incoming requests. ([#9082](grpc/grpc-go#9082)) - Special Thanks: [@dfinkel](https://github.com/dfinkel) ### Bug Fixes - xds/server: Fix a memory leak of HTTP filter instances occurring when route configurations are updated in-place during a Route Discovery Service (RDS) update. ([#9138](grpc/grpc-go#9138)) - grpc: In the deprecated `gzip` Compressor (used via the deprecated `WithCompressor` dial option), enforce the `MaxRecvMsgSize` limit on the decompressed message buffer, preventing excessive memory allocation from highly compressed payloads. ([#9114](grpc/grpc-go#9114)) - Special Thanks: [@evilgensec](https://github.com/evilgensec) - stats/opentelemetry: Record retry attempts, `grpc.previous-rpc-attempts`, at the call level and not the attempt level. ([#8923](grpc/grpc-go#8923)) - encoding: Ensure `Close()` is always called on readers returned from `Compressor.Decompress` if possible. ([#9135](grpc/grpc-go#9135)) - channelz: Fix the `LastMessageSentTimestamp` and `LastMessageReceivedTimestamp` fields in `SocketMetrics` to ensure they contain correct timestamp values. ([#9109](grpc/grpc-go#9109)) ### [`v1.81.1`](https://github.com/grpc/grpc-go/releases/tag/v1.81.1): Release 1.81.1 [Compare Source](grpc/grpc-go@v1.81.0...v1.81.1) ### Security - xds/rbac: Fix a potential authorization bypass caused by incorrectly falling through URI/DNS SANs to Subject Distinguished Name (DN) when matching the authenticated principal name. With this fix, only the first non-empty identity source will be used, as per [gRFC A41](https://github.com/grpc/proposal/blob/master/A41-xds-rbac.md). ([#9111](grpc/grpc-go#9111)) - Special Thanks: [@al4an444](https://github.com/al4an444) ### Bug Fixes - otel: Segregate client and server RPC information used for metrics and traces, to avoid one overwriting the other. ([#9081](grpc/grpc-go#9081)) ### [`v1.81.0`](https://github.com/grpc/grpc-go/releases/tag/v1.81.0): Release 1.81.0 [Compare Source](grpc/grpc-go@v1.80.0...v1.81.0) ### Behavior Changes - balancer/rls: Switch gauge metrics to asynchronous emission (once per collection cycle) to reduce telemetry noise and align with other gRPC language implementations. ([#8808](grpc/grpc-go#8808)) ### Dependencies - Minimum supported Go version is now 1.25. ([#8969](grpc/grpc-go#8969)) ### Bug Fixes - xds: Use the leaf cluster's security config for the TLS handshake instead of the aggregate cluster's config. ([#8956](grpc/grpc-go#8956)) - transport: Send a `RST_STREAM` when receiving an `END_STREAM` when the stream is not already half-closed. ([#8832](grpc/grpc-go#8832)) - xds: Fix ADS resource name validation to prevent a panic. ([#8970](grpc/grpc-go#8970)) ### New Features - grpc/stats: Add support for custom labels in per-call metrics ([gRFC A108](https://github.com/grpc/proposal/blob/master/A108-otel-custom-per-call-label.md)). ([#9008](grpc/grpc-go#9008)) - xds: Add support for Server Name Indication (SNI) and SAN validation ([gRFC A101](https://github.com/grpc/proposal/blob/master/A101-SNI-setting-and-SNI-SAN-validation.md)). Disabled by default. To enable, set `GRPC_EXPERIMENTAL_XDS_SNI=true` environment variable. ([#9016](grpc/grpc-go#9016)) - xds: Add support to control which fields get propagated from ORCA backend metric reports to LRS load reports ([gRFC A85](https://github.com/grpc/proposal/blob/master/A85-lrs-custom-metrics-changes.md)). Disabled by default. To enable, set `GRPC_EXPERIMENTAL_XDS_ORCA_LRS_PROPAGATION=true`. ([#9005](grpc/grpc-go#9005)) - xds: Add metrics to track xDS client connectivity and cached resource state ([gRFC A78](https://github.com/grpc/proposal/blob/master/A78-grpc-metrics-wrr-pf-xds.md)). ([#8807](grpc/grpc-go#8807)) - stats/otel: Enhance `grpc.subchannel.disconnections` metric by adding disconnection reason to the `grpc.disconnect_error` label ([gRFC A94](https://github.com/grpc/proposal/blob/master/A94-subchannel-otel-metrics.md)). This provides granular insights into why subchannels are closing. ([#8973](grpc/grpc-go#8973)) - mem: Add `mem.Buffer.Slice()` API to slice the buffer like a slice. ([#8977](grpc/grpc-go#8977)) - Special Thanks: [@ash2k](https://github.com/ash2k) ### Performance Improvements - alts: Pool read buffers to lower memory utilization when sockets are unreadable. ([#8964](grpc/grpc-go#8964)) - transport: Pool HTTP/2 framer read buffers to reduce idle memory consumption. Currently limited to Linux for ALTS and non-encrypted transports (TCP, Unix). To disable, set `GRPC_GO_EXPERIMENTAL_HTTP_FRAMER_READ_BUFFER_POOLING=false` and report any issues. ([#9032](grpc/grpc-go#9032)) ### [`v1.80.0`](https://github.com/grpc/grpc-go/releases/tag/v1.80.0): Release 1.80.0 [Compare Source](grpc/grpc-go@v1.79.3...v1.80.0) ### Behavior Changes - balancer: log a warning if a balancer is registered with uppercase letters, as balancer names should be lowercase. In a future release, balancer names will be treated as case-insensitive; see [#5288](grpc/grpc-go#5288) for details. ([#8837](grpc/grpc-go#8837)) - xds: update resource error handling and re-resolution logic ([#8907](grpc/grpc-go#8907)) - Re-resolve all `LOGICAL_DNS` clusters simultaneously when re-resolution is requested. - Fail all in-flight RPCs immediately upon receipt of listener or route resource errors, instead of allowing them to complete. ### Bug Fixes - xds: support the LB policy configured in `LOGICAL_DNS` cluster resources instead of defaulting to `pick_first`. ([#8733](grpc/grpc-go#8733)) - credentials/tls: perform per-RPC authority validation against the leaf certificate instead of the entire peer certificate chain. ([#8831](grpc/grpc-go#8831)) - xds: enabling A76 ring hash endpoint keys no longer causes EDS resources with invalid proxy metadata to be NACKed when HTTP CONNECT (gRFC A86) is disabled. ([#8875](grpc/grpc-go#8875)) - xds: validate that the sum of endpoint weights in a locality does not exceed the maximum `uint32` value. ([#8899](grpc/grpc-go#8899)) - Special Thanks: [@RAVEYUS](https://github.com/RAVEYUS) - xds: fix incorrect proto field access in the weighted round robin (WRR) configuration where `blackout_period` was used instead of `weight_expiration_period`. ([#8915](grpc/grpc-go#8915)) - Special Thanks: [@gregbarasch](https://github.com/gregbarasch) - xds/rbac: handle addresses with ports in IP matchers. ([#8990](grpc/grpc-go#8990)) ### New Features - ringhash: enable gRFC A76 (endpoint hash keys and request hash headers) by default. ([#8922](grpc/grpc-go#8922)) ### Performance Improvements - credentials/alts: pool write buffers to reduce memory allocations and usage. ([#8919](grpc/grpc-go#8919)) - grpc: enable the use of pooled write buffers for buffering HTTP/2 frame writes by default. This reduces memory usage when connections are idle. Use the [WithSharedWriteBuffer](https://pkg.go.dev/google.golang.org/grpc#WithSharedWriteBuffer) dial option or the [SharedWriteBuffer](https://pkg.go.dev/google.golang.org/grpc#SharedWriteBuffer) server option to disable this feature. ([#8957](grpc/grpc-go#8957)) - xds/priority: stop caching child LB policies removed from the configuration. This will help reduce memory and cpu usage when localities are constantly switching between priorities. ([#8997](grpc/grpc-go#8997)) - mem: add a faster tiered buffer pool; use the experimental [mem.NewBinaryTieredBufferPool](https://pkg.go.dev/google.golang.org/grpc/mem@master#NewBinaryTieredBufferPool) function to create such pools. ([#8775](grpc/grpc-go#8775)) </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - Between 12:00 AM and 03:59 AM (`* 0-3 * * *`) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate).  Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/13580 Reviewed-by: Mathieu Fenniak <mfenniak@noreply.codeberg.org>

dfinkel force-pushed the server_goroutine_labels branch from 5b63d30 to 59a7845 Compare April 21, 2026 18:12

dfinkel force-pushed the server_goroutine_labels branch from 59a7845 to 84dda06 Compare April 21, 2026 18:16

dfinkel marked this pull request as ready for review April 21, 2026 18:56

easwars added Type: Feature New features or improvements in behavior Area: Server Includes Server, Streams and Server Options. labels Apr 21, 2026

easwars requested a review from arjan-bal April 21, 2026 22:51

easwars assigned arjan-bal Apr 21, 2026

arjan-bal added this to the 1.82 Release milestone Apr 22, 2026

arjan-bal reviewed Apr 22, 2026

View reviewed changes

arjan-bal assigned dfinkel and unassigned arjan-bal Apr 22, 2026

dfinkel added 3 commits April 23, 2026 16:17

test: server: verify goroutine labels in ctxs

3b7e587

Add a couple test-cases that verify that the contexts include the expected goroutine labels.

envconfig: add test for goroutineLabelsFromEnv

c3f70d9

Add a test covering the goroutine label configuration parsing.

arjan-bal assigned arjan-bal, ashishksrivastava and atollena and unassigned dfinkel, arjan-bal, ashishksrivastava and atollena Apr 24, 2026

Pranjali-2501 unassigned dfinkel Apr 28, 2026

arjan-bal assigned dfinkel and unassigned arjan-bal May 8, 2026

dfinkel added 5 commits May 11, 2026 13:42

envconfig: add AllGoroutineLabels & "all" value

cbcdf44

Make it easy to opt into all goroutine labels now that we're keeping it default disabled.

envconfig: flip goroutine label default to disabled

e176a2b

envconfig: goroutine labels: add "none" value

05172ce

Add a "none" special value that pairs with "all" so users can opt out of any goroutine labels that happen to have been enabled elsewhere.

Merge remote-tracking branch 'origin/master' into server_goroutine_la…

921d8cb

…bels

server: add the envconfig import back

e238722

Another commit appears to have removed this import while this change was in-flight. Add it back now that it's needed again.

easwars assigned arjan-bal and unassigned dfinkel May 11, 2026

gemini-code-assist Bot reviewed May 12, 2026

View reviewed changes

Comment thread internal/envconfig/envconfig.go

Comment thread internal/envconfig/envconfig.go

arjan-bal approved these changes May 12, 2026

View reviewed changes

arjan-bal requested a review from easwars May 12, 2026 07:59

arjan-bal assigned easwars and unassigned arjan-bal May 12, 2026

easwars approved these changes May 12, 2026

View reviewed changes

easwars assigned dfinkel and unassigned easwars May 12, 2026

test: server: s/expVal/wantVal/ & reduce err scopes

3854612

easwars merged commit b58f32d into grpc:master May 27, 2026
13 of 14 checks passed

		_, err := ss.Client.EmptyCall(ctx, &testpb.Empty{})
		if err != nil {

Uh oh!

Conversation

dfinkel commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dfinkel Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

arjan-bal commented May 8, 2026

Uh oh!

dfinkel commented May 11, 2026

Uh oh!

arjan-bal commented May 12, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

arjan-bal left a comment

Choose a reason for hiding this comment

Uh oh!

dfinkel commented May 12, 2026

Uh oh!

easwars left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dfinkel commented May 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dfinkel commented Apr 21, 2026 •

edited

Loading

codecov Bot commented Apr 21, 2026 •

edited

Loading

dfinkel Apr 23, 2026 •

edited

Loading