Skip to content

Race condition in xds package #8474

New issue
New issue
Open
#8483
Open
Race condition in xds package#8474
#8483
Assignees
eshitachandwani
Labels
P1Type: Bug
@tritone

Description

@tritone
tritone
opened on Jul 25, 2025

What version of gRPC are you using?

v1.74.2

What version of Go are you using (go version)?

v1.24.0

What operating system (Linux, Windows, …) and version?

Ubuntu

What did you do?

I'm a maintainer in google-cloud-go and saw a failure in a GCS integration test run with the -race flag. The stack trace showed only goroutines in the grpc/xds package so it seems to be internal to that. I wasn't able to reproduce the issue myself; it showed up as a one-time failure in a CI run. Issue details are at googleapis/google-cloud-go#12606.

Logs from the race detector:

==================
WARNING: DATA RACE
Read at 0x00c001670ff0 by goroutine 7570:
  google.golang.org/grpc/xds/internal/xdsclient.(*clientImpl).ReportLoad()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/xdsclient/clientimpl_loadreport.go:35 +0x56
  google.golang.org/grpc/xds/internal/balancer/clusterimpl.(*clusterImplBalancer).updateLoadStore()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterimpl/clusterimpl.go:241 +0x67c
  google.golang.org/grpc/xds/internal/balancer/clusterimpl.(*clusterImplBalancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterimpl/clusterimpl.go:283 +0x5c4
  google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/internal/balancer/gracefulswitch/gracefulswitch.go:194 +0x2b4
  google.golang.org/grpc/xds/internal/balancer/outlierdetection.(*synchronizingBalancerWrapper).updateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/outlierdetection/balancer.go:877 +0xf7
  google.golang.org/grpc/xds/internal/balancer/outlierdetection.(*outlierDetectionBalancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/outlierdetection/balancer.go:336 +0x16c4
  google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/internal/balancer/gracefulswitch/gracefulswitch.go:194 +0x2b4
  google.golang.org/grpc/internal/balancergroup.(*subBalancerWrapper).updateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/internal/balancergroup/balancergroup.go:130 +0x2c4
  google.golang.org/grpc/internal/balancergroup.(*BalancerGroup).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/internal/balancergroup/balancergroup.go:458 +0x166
  google.golang.org/grpc/xds/internal/balancer/priority.(*childBalancer).sendUpdate()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer_child.go:113 +0x1a9
  google.golang.org/grpc/xds/internal/balancer/priority.(*childBalancer).start()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer_child.go:106 +0x144
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).switchToChild()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer_priority.go:171 +0x146
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).syncPriority()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer_priority.go:114 +0x816
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).run()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer.go:286 +0x2e5
  google.golang.org/grpc/xds/internal/balancer/priority.bb.Build.gowrap1()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer.go:74 +0x33

Previous write at 0x00c001670ff0 by goroutine 7569:
  google.golang.org/grpc/xds/internal/xdsclient.(*clientImpl).ReportLoad()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/xdsclient/clientimpl_loadreport.go:44 +0x271
  google.golang.org/grpc/xds/internal/balancer/clusterimpl.(*clusterImplBalancer).updateLoadStore()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterimpl/clusterimpl.go:241 +0x67c
  google.golang.org/grpc/xds/internal/balancer/clusterimpl.(*clusterImplBalancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterimpl/clusterimpl.go:283 +0x5c4
  google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/internal/balancer/gracefulswitch/gracefulswitch.go:194 +0x2b4
  google.golang.org/grpc/xds/internal/balancer/outlierdetection.(*synchronizingBalancerWrapper).updateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/outlierdetection/balancer.go:877 +0xf7
  google.golang.org/grpc/xds/internal/balancer/outlierdetection.(*outlierDetectionBalancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/outlierdetection/balancer.go:336 +0x16c4
  google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/internal/balancer/gracefulswitch/gracefulswitch.go:194 +0x2b4
  google.golang.org/grpc/internal/balancergroup.(*subBalancerWrapper).updateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/internal/balancergroup/balancergroup.go:130 +0x2c4
  google.golang.org/grpc/internal/balancergroup.(*BalancerGroup).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/internal/balancergroup/balancergroup.go:458 +0x166
  google.golang.org/grpc/xds/internal/balancer/priority.(*childBalancer).sendUpdate()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer_child.go:113 +0x1a9
  google.golang.org/grpc/xds/internal/balancer/priority.(*childBalancer).start()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer_child.go:106 +0x144
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).switchToChild()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer_priority.go:171 +0x146
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).syncPriority()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer_priority.go:114 +0x816
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).run()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer.go:286 +0x2e5
  google.golang.org/grpc/xds/internal/balancer/priority.bb.Build.gowrap1()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer.go:74 +0x33

Goroutine 7570 (running) created at:
  google.golang.org/grpc/xds/internal/balancer/priority.bb.Build()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer.go:74 +0x9dc
  google.golang.org/grpc/xds/internal/balancer/priority.(*bb).Build()
      :1 +0xa7
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.init.func1()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:52 +0xac
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).updateChildConfig()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:234 +0x179
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).handleResourceUpdate()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:219 +0xcf
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).run()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:341 +0x3af
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.bb.Build.gowrap1()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:94 +0x33

Goroutine 7569 (running) created at:
  google.golang.org/grpc/xds/internal/balancer/priority.bb.Build()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer.go:74 +0x9dc
  google.golang.org/grpc/xds/internal/balancer/priority.(*bb).Build()
      :1 +0xa7
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.init.func1()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:52 +0xac
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).updateChildConfig()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:234 +0x179
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).handleResourceUpdate()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:219 +0xcf
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).run()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:341 +0x3af
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.bb.Build.gowrap1()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:94 +0x33
==================
==================
WARNING: DATA RACE
Write at 0x00c00656fea0 by goroutine 7570:
  sync/atomic.CompareAndSwapInt32()
      /usr/local/go/src/runtime/race_amd64.s:361 +0xb
  sync/atomic.CompareAndSwapInt32()
      :1 +0x18
  sync.(*Mutex).Lock()
      /usr/local/go/src/sync/mutex.go:46 +0x28
  google.golang.org/grpc/xds/internal/clients/lrsclient.(*LRSClient).getOrCreateLRSStream()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/clients/lrsclient/lrsclient.go:100 +0x84
  google.golang.org/grpc/xds/internal/clients/lrsclient.(*LRSClient).ReportLoad()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/clients/lrsclient/lrsclient.go:88 +0x3c4
  google.golang.org/grpc/xds/internal/xdsclient.(*clientImpl).ReportLoad()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/xdsclient/clientimpl_loadreport.go:47 +0x331
  google.golang.org/grpc/xds/internal/balancer/clusterimpl.(*clusterImplBalancer).updateLoadStore()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterimpl/clusterimpl.go:241 +0x67c
  google.golang.org/grpc/xds/internal/balancer/clusterimpl.(*clusterImplBalancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterimpl/clusterimpl.go:283 +0x5c4
  google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/internal/balancer/gracefulswitch/gracefulswitch.go:194 +0x2b4
  google.golang.org/grpc/xds/internal/balancer/outlierdetection.(*synchronizingBalancerWrapper).updateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/outlierdetection/balancer.go:877 +0xf7
  google.golang.org/grpc/xds/internal/balancer/outlierdetection.(*outlierDetectionBalancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/outlierdetection/balancer.go:336 +0x16c4
  google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/internal/balancer/gracefulswitch/gracefulswitch.go:194 +0x2b4
  google.golang.org/grpc/internal/balancergroup.(*subBalancerWrapper).updateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/internal/balancergroup/balancergroup.go:130 +0x2c4
  google.golang.org/grpc/internal/balancergroup.(*BalancerGroup).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/internal/balancergroup/balancergroup.go:458 +0x166
  google.golang.org/grpc/xds/internal/balancer/priority.(*childBalancer).sendUpdate()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer_child.go:113 +0x1a9
  google.golang.org/grpc/xds/internal/balancer/priority.(*childBalancer).start()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer_child.go:106 +0x144
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).switchToChild()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer_priority.go:171 +0x146
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).syncPriority()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer_priority.go:114 +0x816
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).run()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer.go:286 +0x2e5
  google.golang.org/grpc/xds/internal/balancer/priority.bb.Build.gowrap1()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer.go:74 +0x33

Previous write at 0x00c00656fea0 by goroutine 7569:
  google.golang.org/grpc/xds/internal/clients/lrsclient.New()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/clients/lrsclient/lrsclient.go:71 +0x7a
  google.golang.org/grpc/xds/internal/xdsclient.(*clientImpl).ReportLoad()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/xdsclient/clientimpl_loadreport.go:36 +0x137
  google.golang.org/grpc/xds/internal/balancer/clusterimpl.(*clusterImplBalancer).updateLoadStore()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterimpl/clusterimpl.go:241 +0x67c
  google.golang.org/grpc/xds/internal/balancer/clusterimpl.(*clusterImplBalancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterimpl/clusterimpl.go:283 +0x5c4
  google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/internal/balancer/gracefulswitch/gracefulswitch.go:194 +0x2b4
  google.golang.org/grpc/xds/internal/balancer/outlierdetection.(*synchronizingBalancerWrapper).updateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/outlierdetection/balancer.go:877 +0xf7
  google.golang.org/grpc/xds/internal/balancer/outlierdetection.(*outlierDetectionBalancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/outlierdetection/balancer.go:336 +0x16c4
  google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/internal/balancer/gracefulswitch/gracefulswitch.go:194 +0x2b4
  google.golang.org/grpc/internal/balancergroup.(*subBalancerWrapper).updateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/internal/balancergroup/balancergroup.go:130 +0x2c4
  google.golang.org/grpc/internal/balancergroup.(*BalancerGroup).UpdateClientConnState()
      /go/pkg/mod/google.golang.org/[email protected]/internal/balancergroup/balancergroup.go:458 +0x166
  google.golang.org/grpc/xds/internal/balancer/priority.(*childBalancer).sendUpdate()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer_child.go:113 +0x1a9
  google.golang.org/grpc/xds/internal/balancer/priority.(*childBalancer).start()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer_child.go:106 +0x144
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).switchToChild()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer_priority.go:171 +0x146
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).syncPriority()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer_priority.go:114 +0x816
  google.golang.org/grpc/xds/internal/balancer/priority.(*priorityBalancer).run()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer.go:286 +0x2e5
  google.golang.org/grpc/xds/internal/balancer/priority.bb.Build.gowrap1()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer.go:74 +0x33

Goroutine 7570 (running) created at:
  google.golang.org/grpc/xds/internal/balancer/priority.bb.Build()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer.go:74 +0x9dc
  google.golang.org/grpc/xds/internal/balancer/priority.(*bb).Build()
      :1 +0xa7
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.init.func1()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:52 +0xac
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).updateChildConfig()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:234 +0x179
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).handleResourceUpdate()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:219 +0xcf
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).run()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:341 +0x3af
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.bb.Build.gowrap1()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:94 +0x33

Goroutine 7569 (running) created at:
  google.golang.org/grpc/xds/internal/balancer/priority.bb.Build()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/priority/balancer.go:74 +0x9dc
  google.golang.org/grpc/xds/internal/balancer/priority.(*bb).Build()
      :1 +0xa7
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.init.func1()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:52 +0xac
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).updateChildConfig()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:234 +0x179
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).handleResourceUpdate()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:219 +0xcf
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.(*clusterResolverBalancer).run()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:341 +0x3af
  google.golang.org/grpc/xds/internal/balancer/clusterresolver.bb.Build.gowrap1()
      /go/pkg/mod/google.golang.org/[email protected]/xds/internal/balancer/clusterresolver/clusterresolver.go:94 +0x33
==================
    testing.go:1490: race detected during execution of test

Metadata

Metadata

Assignees

Labels

P1Type: Bug

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions