System.Net.Http.Headers.HttpHeaders.ReadStoreValues exception after .NET6 upgrade

[rui-ktei]

Description

Reproduction Steps

Sorry I can't produce detailed steps because it doesn't happen all the time, but when it happens, the stacktrace is like below. This started to happen after we upgraded to .NET6. I'm not really sure this is .NET6 problem or a bug in DynamoDB SDK which was only made occur after .NET6. I did google a bit and I found this which looks very similar:
dotnet/yarp#1337

Logs

at System.Net.Http.Headers.HttpHeaders.ReadStoreValues[T](Span`1 values, Object storeValue, HttpHeaderParser parser, Int32& currentIndex)
   at System.Net.Http.Headers.HttpHeaders.GetStoreValuesAsStringOrStringArray(HeaderDescriptor descriptor, Object sourceValues, String& singleValue, String[]& multiValue)
   at System.Net.Http.Headers.HttpHeaders.GetEnumeratorCore()+MoveNext()
   at Amazon.Runtime.Internal.Transform.HttpClientResponseData.CopyHeaderValues(HttpResponseMessage response)
   at Amazon.Runtime.Internal.Transform.HttpClientResponseData..ctor(HttpResponseMessage response, HttpClient httpClient, Boolean disposeClient)
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.DynamoDBv2.DocumentModel.Search.GetNextSetHelperAsync(CancellationToken cancellationToken)
   at Amazon.DynamoDBv2.DataModel.AsyncSearch`1.GetNextSetAsync(CancellationToken cancellationToken)

Environment

• SDK Version: AWSSDK.DynamoDBv2 - 3.7.0.104
• Package Version: AWSSDK.DynamoDBv2 - 3.7.0.104
• OS Info: I'm not too sure actually, it's the official .net6 runtime Docker image though
• Build Environment AWS CodeBuild
• Targeted .NET Platform: net6.0

Resolution

I've no clue yet.

---

This is a 🐛 bug-report

Thank you very much if you could look into this!

[ashishdhingra]

Most probably related to dotnet/runtime#61798.

[MihaZupan]

I did google a bit and I found this which looks very similar:
dotnet/yarp#1337

The underlying cause is certainly the same, the question is just which component is causing it in your setup.

@rui-ktei Are you using NewRelic for monitoring your application? If so, this is known issue that was likely fixed in the 9.2.0 release of their .NET agent.

FWIW, I looked through some of the code using HttpClient in this repo and did not spot behavior that would cause this.

[rui-ktei]

@MihaZupan thank you for the information. Yes I do use NewRelic to trace my application. I didn’t realise this was caused by that because the stack track is purely from DynamoDB. Let me update NR to latest and try again.

[rui-ktei]

@MihaZupan so I checked my project again: it does have indirect NewRelic.Agent package reference. It's indirect because it references it through another package which references NewRelic. That being said, my project does not use anything in NewRelic.Agent package. There is no using statements for NewRelic at all.

However, I still updated NewRelic.Agent to the latest version anyway but I still get the same errors after that.

If you look at the stacktrace I provided, it actually comes directly from DynamoDB as well. There is no magic behind it, we were simply invoking dynamoDb.LoadAsync<T>.

Looking at the code, I do have some calls to DynamoDB running parallelly, and it looks like this:

var task1 = _dynamoDb.LoadAsync<T1>(...);
var task2 = _dynamoDb.LoadAsync<T2>(...);
await Task.WhenAll(task1, task2);

Could this be the issue? If under the hood, AWS SDK is using the same HttpClient instance from the pool, then this is the only code I think could lead to thread-safety issue

[MihaZupan]

HttpClient is safe to use concurrently. In fact, we encourage it.

The issue here is that the headers from the same response message are being read concurrently.
You will only see the call stack of one thread - not necessarily the one that's misbehaving. The instrumentation will rewrite the original SocketsHttpHandler call site, so you won't spot it from the stack trace.

You don't necessarily need any new relic code in your project. They offer instrumentations that hook into every process on a given host machine.

[rui-ktei]

@MihaZupan I see what you mean now. Thank you for the reply. We do use NewRelic .NET APM. Let me check that. I will get back to you.

[rui-ktei]

Closing this for now. It's highly likely this is NewRelic issue. We're going to update to latest version of NR and check it again. Thank you for all your help guys!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.