Possible Race Condition in v8 Telemetry

[martincostello]

While doing some work on #1144, I had a test run where I got this exception in the same test in two different TFMs in a single test run. Re-running the tests didn't repro it, and I haven't seen it since.

Maybe there's an edge case race-condition somewhere?

 Polly.Core.Tests.Hedging.HedgingResilienceStrategyTests.ExecuteAsync_EnsurePrimaryTaskCancelled_Ok
   Source: HedgingResilienceStrategyTests.cs line 263
   Duration: 7 ms

  Message: 
System.IndexOutOfRangeException : Index was outside the bounds of the array.

  Stack Trace: 
List`1.Add(T item)
CallbackDiagnosticSource.Write(String name, Object value) line 125
ResilienceStrategyTelemetry.Report[TArgs,TResult](String eventName, OutcomeArguments`2 args) line 74
TelemetryUtil.ReportExecutionAttempt[TResult](ResilienceStrategyTelemetry telemetry, ResilienceContext context, Outcome`1 outcome, Int32 attempt, TimeSpan executionTime, Boolean handled) line 37
<UpdateOutcomeAsync>d__54.MoveNext() line 230
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
ConfiguredTaskAwaiter.GetResult()
<ExecutePrimaryActionAsync>d__53`1.MoveNext() line 222
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
ConfiguredTaskAwaiter.GetResult()
<DisposeAsync>d__23.MoveNext() line 96
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
ConfiguredValueTaskAwaiter.GetResult()
<ExecuteCallbackAsync>d__19`1.MoveNext() line 60
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
ValueTask`1.get_Result()
<<ConvertValueTask>g__ConvertValueTaskAsync|4_0>d`1.MoveNext() line 76
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
ValueTask`1.get_Result()
<ExecuteAsync>d__10`1.MoveNext() line 169
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
ValueTask`1.get_Result()
<ExecuteAsync_EnsurePrimaryTaskCancelled_Ok>d__26.MoveNext() line 293
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

[martincostello]

Similar issue here:

[xUnit.net 00:00:01.74]     Polly.Core.Tests.Hedging.HedgingResilienceStrategyTests.ExecuteAsync_EnsureHedgingDelayGeneratorRespected [FAIL]

Error: System.InvalidOperationException : Collection was modified; enumeration operation may not execute.

  Failed Polly.Core.Tests.Hedging.HedgingResilienceStrategyTests.ExecuteAsync_EnsureHedgingDelayGeneratorRespected [3 ms]
  Error Message:
   System.InvalidOperationException : Collection was modified; enumeration operation may not execute.
  Stack Trace:
     at System.Collections.Generic.List`1.Enumerator.MoveNextRare()
   at System.Linq.Enumerable.WhereListIterator`1.MoveNext()
   at Polly.Core.Tests.Hedging.HedgingTimeProvider.Advance(TimeSpan diff) in /_/test/Polly.Core.Tests/Hedging/HedgingTimeProvider.cs:line 17
   at Polly.Core.Tests.Hedging.HedgingResilienceStrategyTests.ExecuteAsync_EnsureHedgingDelayGeneratorRespected() in /_/test/Polly.Core.Tests/Hedging/HedgingResilienceStrategyTests.cs:line 870

[martintmk]

Yeah, it's in hedging where concurrency is involved. The Polly.Core is safe, but we need to put some locks in tests to avoid those race conditions.