Crossgen runs for Linux x64 has failures

[LoopedBard3]

Crossgen runs for Linux x64 are red due to 19 failures, some failures have stack overflow errors bubbled up similar to #2325, and some don't have failures bubbled up. Example stack overflow error below:

[2022/03/21 09:17:32][INFO] Stack overflow.
[2022/03/21 09:17:32][INFO]    at System.Diagnostics.Tracing.EventSource.WriteToAllListeners(System.Diagnostics.Tracing.EventWrittenEventArgs, Int32, EventData*)
[2022/03/21 09:17:32][INFO]    at System.Diagnostics.Tracing.EventSource.WriteEventWithRelatedActivityIdCore(Int32, System.Guid*, Int32, EventData*)
[2022/03/21 09:17:32][INFO]    at System.Buffers.ArrayPoolEventSource.BufferRented(Int32, Int32, Int32, Int32)
[2022/03/21 09:17:32][INFO]    at System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1[[System.Char, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Rent(Int32)
[2022/03/21 09:17:32][INFO]    at System.Runtime.CompilerServices.DefaultInterpolatedStringHandler..ctor(Int32, Int32)
[2022/03/21 09:17:32][INFO]    at System.Diagnostics.Tracing.ActivityTracker+ActivityInfo.Path(ActivityInfo)
[2022/03/21 09:17:32][INFO]    at System.Diagnostics.Tracing.ActivityTracker+ActivityInfo.Path(ActivityInfo)
[2022/03/21 09:17:32][INFO]    at System.Diagnostics.Tracing.ActivityTracker+ActivityInfo.Path(ActivityInfo)
[2022/03/21 09:17:32][INFO]    at System.Diagnostics.Tracing.ActivityTracker+ActivityInfo.Path(ActivityInfo)
[2022/03/21 09:17:32][INFO]    at System.Diagnostics.Tracing.ActivityTracker+ActivityInfo.ToString()
[2022/03/21 09:17:32][INFO]    at System.Diagnostics.Tracing.ActivityTracker+ActivityInfo.LiveActivities(ActivityInfo)
[2022/03/21 09:17:32][INFO]    at System.Diagnostics.Tracing.ActivityTracker+ActivityInfo.LiveActivities(ActivityInfo)
[2022/03/21 09:17:32][INFO]    at System.Diagnostics.Tracing.ActivityTracker.OnStart(System.String, System.String, Int32, System.Guid ByRef, System.Guid ByRef, System.Diagnostics.Tracing.EventActivityOptions, Boolean)
[2022/03/21 09:17:32][INFO]    at System.Diagnostics.Tracing.EventSource.WriteEventWithRelatedActivityIdCore(Int32, System.Guid*, Int32, EventData*)
[2022/03/21 09:17:32][INFO]    at ILCompiler.PerfEventSource.JitMethodStart()
[2022/03/21 09:17:32][INFO]    at ILCompiler.PerfEventSource+StartStopEvents.JitMethodEvents()
[2022/03/21 09:17:32][INFO]    at ILCompiler.ReadyToRunCodegenCompilation.<ComputeDependencyNodeDependencies>b__35_0(ILCompiler.DependencyAnalysisFramework.DependencyNodeCore`1<ILCompiler.DependencyAnalysis.NodeFactory>)
[2022/03/21 09:17:32][INFO]    at System.Threading.Tasks.Parallel+<>c__DisplayClass33_0`2[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<ForEachWorker>b__0(Int32)
[2022/03/21 09:17:32][INFO]    at System.Threading.Tasks.Parallel+<>c__DisplayClass19_0`1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<ForWorker>b__1(System.Threading.Tasks.RangeWorker ByRef, Int32, Boolean ByRef)
[2022/03/21 09:17:32][INFO]    at System.Threading.Tasks.TaskReplicator+Replica`1[[System.Threading.Tasks.RangeWorker, System.Threading.Tasks.Parallel, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ExecuteAction(Boolean ByRef)
[2022/03/21 09:17:32][INFO]    at System.Threading.Tasks.TaskReplicator+Replica.Execute()
[2022/03/21 09:17:32][INFO]    at System.Threading.Tasks.TaskReplicator+Replica+<>c.<.ctor>b__4_0(System.Object)
[2022/03/21 09:17:32][INFO]    at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[2022/03/21 09:17:32][INFO]    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
[2022/03/21 09:17:32][INFO]    at System.Threading.ThreadPoolWorkQueue.Dispatch()
[2022/03/21 09:17:32][INFO]    at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
[2022/03/21 09:17:32][INFO] 
[2022/03/21 09:17:32][INFO] Process 29803 failed to run. Result: ExitedWithError
[2022/03/21 09:17:32][ERROR] Process exited with status 255
[2022/03/21 09:17:32][INFO] Run failure registered

[LoopedBard3]

Maybe or maybe not related, but it appears that near the top during setup, source is not being found as a valid command for activating the virtual environment.

[LoopedBard3]

Still having issues with Crossgen. The current failure is:

[2022/07/08 11:57:11][INFO] Parsing traces/Crossgen2_Throughput_-_Composite_-_framework-r2r_startup.trace.zip
[2022/07/08 11:57:11][INFO] Unhandled exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
[2022/07/08 11:57:11][INFO]  ---> System.InvalidOperationException: Sequence contains no elements
[2022/07/08 11:57:11][INFO]    at System.Linq.ThrowHelper.ThrowNoElementsException()
[2022/07/08 11:57:11][INFO]    at System.Linq.Enumerable.Max(IEnumerable`1 source)
[2022/07/08 11:57:11][INFO]    at Reporting.Reporter.<>c.<WriteResultTable>b__10_4(Counter c) in /__w/1/s/Payload/performance/src/tools/Reporting/Reporting/Reporter.cs:line 129
[2022/07/08 11:57:11][INFO]    at System.Linq.Enumerable.Max[TSource](IEnumerable`1 source, Func`2 selector)
[2022/07/08 11:57:11][INFO]    at Reporting.Reporter.WriteResultTable() in /__w/1/s/Payload/performance/src/tools/Reporting/Reporting/Reporter.cs:line 129
[2022/07/08 11:57:11][INFO]    at ScenarioMeasurement.Startup.CreateTestReport(String scenarioName, IEnumerable`1 counters, String reportJsonPath, Logger logger) in /__w/1/s/Payload/performance/src/tools/ScenarioMeasurement/Startup/Startup.cs:line 525
[2022/07/08 11:57:11][INFO]    at ScenarioMeasurement.Startup.Main(String appExe, MetricType metricType, String scenarioName, String traceName, Boolean processWillExit, Int32 iterations, String iterationSetup, String setupArgs, String iterationCleanup, String cleanupArgs, Int32 timeout, Int32 measurementDelay, String appArgs, String logFileName, String workingDir, Boolean warmup, Boolean guiApp, Boolean skipProfileIteration, String reportJsonPath, String traceDirectory, String environmentVariables, String innerLoopCommand, String innerLoopCommandArgs, Boolean runWithoutExit, Int32 hotReloadIters, Boolean skipMeasurementIteration, Boolean parseOnly, Boolean runWithDotnet) in /__w/1/s/Payload/performance/src/tools/ScenarioMeasurement/Startup/Startup.cs:line 329
[2022/07/08 11:57:11][INFO]    --- End of inner exception stack trace ---
[2022/07/08 11:57:11][INFO]    at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
[2022/07/08 11:57:11][INFO]    at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[2022/07/08 11:57:11][INFO]    at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
[2022/07/08 11:57:11][INFO]    at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
[2022/07/08 11:57:11][INFO]    at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
[2022/07/08 11:57:11][INFO] --- End of stack trace from previous location ---
[2022/07/08 11:57:11][INFO]    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseParseErrorReporting>b__19_0>d.MoveNext()
[2022/07/08 11:57:11][INFO] --- End of stack trace from previous location ---
[2022/07/08 11:57:11][INFO]    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass14_0.<<UseHelp>b__0>d.MoveNext()
[2022/07/08 11:57:11][INFO] --- End of stack trace from previous location ---
[2022/07/08 11:57:11][INFO]    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseVersionOption>b__0>d.MoveNext()
[2022/07/08 11:57:11][INFO] --- End of stack trace from previous location ---
[2022/07/08 11:57:11][INFO]    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass21_0.<<UseTypoCorrections>b__0>d.MoveNext()
[2022/07/08 11:57:11][INFO] --- End of stack trace from previous location ---
[2022/07/08 11:57:11][INFO]    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__20_0>d.MoveNext()
[2022/07/08 11:57:11][INFO] --- End of stack trace from previous location ---
[2022/07/08 11:57:11][INFO]    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseParseDirective>b__18_0>d.MoveNext()
[2022/07/08 11:57:11][INFO] --- End of stack trace from previous location ---
[2022/07/08 11:57:11][INFO]    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseDebugDirective>b__10_0>d.MoveNext()
[2022/07/08 11:57:11][INFO] --- End of stack trace from previous location ---
[2022/07/08 11:57:11][INFO]    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__9_0>d.MoveNext()
[2022/07/08 11:57:11][INFO] --- End of stack trace from previous location ---
[2022/07/08 11:57:11][INFO]    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass5_0.<<ConfigureConsole>b__0>d.MoveNext()
[2022/07/08 11:57:11][INFO] --- End of stack trace from previous location ---
[2022/07/08 11:57:11][INFO]    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass12_0.<<UseExceptionHandler>b__0>d.MoveNext()
[2022/07/08 11:57:11][ERROR] Process exited with status 1