Error reported in diagnostic logs. Please examine the log for more details

[jeffschwMSFT]

Build

https://dev.azure.com/dnceng/internal/_build/results?buildId=2367532

Error:

System.Net.Http.HttpRequestException: No route to host (dev.azure.com:443)
 ---> System.Net.Sockets.SocketException (65): No route to host
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32 lastChangeId, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.TaskServer.ConnectAsync(VssConnection jobConnection) in /Users/runner/work/1/s/src/Microsoft.VisualStudio.Services.Agent/TaskServer.cs:line 42
   at Microsoft.VisualStudio.Services.Agent.Worker.JobRunner.RunAsync(AgentJobRequestMessage message, CancellationToken jobRequestCancellationToken) in /Users/runner/work/1/s/src/Agent.Worker/JobRunner.cs:line 224
   at Microsoft.VisualStudio.Services.Agent.Worker.JobRunner.RunAsync(AgentJobRequestMessage message, CancellationToken jobRequestCancellationToken) in /Users/runner/work/1/s/src/Agent.Worker/JobRunner.cs:line 397
   at Microsoft.VisualStudio.Services.Agent.Worker.Worker.RunAsync(String pipeIn, String pipeOut) in /Users/runner/work/1/s/src/Agent.Worker/Worker.cs:line 97
   at Microsoft.VisualStudio.Services.Agent.Worker.Program.MainAsync(IHostContext context, String[] args) in /Users/runner/work/1/s/src/Agent.Worker/Program.cs:line 52

Error reported in diagnostic logs. Please examine the log for more details

Build leg reported

No response

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "Error reported in diagnostic logs. Please examine the log for more details",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2367532
Error message validated: Error reported in diagnostic logs. Please examine the log for more details
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/1/2024 3:50:06 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2946378	dotnet-maui	Build Device Tests (Mono)
2945744	dotnet-dotnet-monitor	Archive Linux arm64 Release
2945583	dotnet-arcade-validation	Linux Build_Release
1369125	dotnet/runtime	linux-arm64 Debug AllSubsets_CoreCLR		dotnet/runtime#126550
1369068	dotnet/runtime	linux-arm64 Debug AllSubsets_CoreCLR		dotnet/runtime#126561
2945115	dotnet-runtime	browser-wasm linux release Mono_multithread multithread
2943216	dotnet-arcade-validation	Validate_Signing_Linux Real_Signing
2943155	dotnet-arcade-validation	Linux Build_Release
2942590	dotnet-maui	Build Device Tests (Mono)
2942499	dotnet-runtime	browser-wasm linux Release LibraryTests_Smoke_AOT		#59779
2941580	dotnet-dotnet-monitor	Build Linux arm64 Release
2941526	dotnet-runtime	linux-arm64 release Mono_LLVMAOT LLVMAOT
2941503	dotnet-maui	Controls (v18.4) Border,BoxView,Brush,Button
2941150	dotnet-maui	Controls (v18.5)	Log
2940995	dotnet-maui	Controls (vlatest) CarouselView	Log
2940996	dotnet-maui	Build Device Tests (Mono)
2940581	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-cbl-mariner-2.0-crossdeps-graph
2940237	dotnet-maui	Controls (vlatest) Shell	Log
2939837	dotnet-maui	Controls (v18.4) ListView
2939346	dotnet-maui	Controls (v18.5) Navigation	Log
2939142	dotnet-maui	Controls (vlatest)	Log
1356284	dotnet/runtime	ios-arm64 release MonoAOTOffsets
2937367	dotnet-maui	Controls (v18.4) Lifecycle,ManualReview,Maps
2937370	dotnet-maui	Controls (v18.4) WebView
2936433	dotnet-maui	Controls (v18.5) Editor,Effects,Essentials,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log
2936739	dotnet-runtime	android-arm release Mono
2934665	dotnet-maui	Controls (vlatest) RadioButton,RefreshView	Log
2936018	dotnet-runtime	android-x64 release MonoAOTOffsets
2935475	dotnet-maui	Controls (v18.5) Editor,Effects,Essentials,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log
2934997	dotnet-runtime	android-arm64 release Mono
2926962	dotnet-runtime	browser-wasm linux Release wasm_coreclr
2932878	dotnet-maui	Controls (v18.5) Image,ImageButton,IndicatorView,InputTransparent,IsEnabled,IsVisible	Log
2934161	dotnet-runtime	linux-x64 Debug Mono_Interpreter_LibrariesTests		#57892
2933801	dotnet-runtime	linux-x64 Release AllSubsets_Mono_LLVMAOT		#57892
2932447	dotnet-maui	Controls (vlatest) SearchBar,Shape,Slider	Log
2932303	dotnet-maui	Controls (vlatest) ListView	Log
2932894	dotnet-maui	Build Device Tests (Mono)
2931936	dotnet-maui	Controls (v18.5) ListView	Log
2932475	dotnet-maui	Build Device Tests (Mono)
2931549	dotnet-maui	Controls (v18.4) Layout	Log
2932406	dotnet-maui	Controls (v18.5) Label	Log
2931493	dotnet-maui	Controls (vlatest) CarouselView	Log
2932022	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net11.0-android-graph
2931849	dotnet-maui	Controls (v18.5)	Log
2931579	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net11.0-android-graph	Log
2931846	dotnet-maui	Build Device Tests (Mono)
2931698	dotnet-maui	Controls (vlatest) CarouselView	Log
2931543	dotnet-maui	Controls (vlatest) SafeAreaEdges,Shadow	Log
2931333	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net9.0-android-graph
2930751	dotnet-maui	Controls (vlatest) Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log
2930970	dotnet-maui	Controls (v18.5)	Log
2930951	dotnet-runtime	browser-wasm linux release Mono
2930819	dotnet-maui	Controls (vlatest) Page,Performance,Picker,ProgressBar	Log
2930821	dotnet-runtime	browser-wasm linux release Mono_multithread multithread
2930755	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net10.0-android-graph	Log
2930592	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net10.0-android-graph
2930261	dotnet-runtime	browser-wasm linux release Mono_multithread multithread
2930194	dotnet-runtime	linux-arm64 Debug Libraries_CheckedCoreCLR		#57892
2930080	dotnet-runtime	linux-arm64 Debug Libraries_CheckedCoreCLR		#58741
2929887	dotnet-maui	Controls (v18.4) ListView	Log
2929956	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net10.0-android-graph
2929942	dotnet-runtime	linux-arm64 Debug Mono_MiniJIT_LibrariesTests		#59204
2929919	dotnet-runtime	browser-wasm linux Release LibraryTests_EAT		#59204
2929781	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net11.0-android-graph
2929742	dotnet-maui	Controls (v18.4) CollectionView	Log
2929634	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net10.0-android-graph
2929520	dotnet-runtime	linux-arm64 release coreclr
2928850	dotnet-runtime	linux-arm64 Debug Mono_MiniJIT_LibrariesTests	Log	#57892
2928707	dotnet-runtime	browser-wasm linux release bluebird
2928524	dotnet-maui	Controls (v18.4) Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView
2928241	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net10.0-android-graph
2928202	dotnet-runtime	browser-wasm linux release bluebird
2928182	dotnet-runtime	linux-arm64 release AOT
2928183	dotnet-runtime	browser-wasm linux release Mono_multithread multithread
2928168	dotnet-runtime	linux-x64 Debug Mono_MiniJIT_LibrariesTests		#57892
2928143	dotnet-runtime	linux-x64 release coreclr
2928111	dotnet-runtime	linux-arm64 release AOT
2928043	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net10.0-android-graph
2928051	dotnet-runtime	linux-x64 release coreclr
2928094	dotnet-runtime	linux-x64 release AOT
2928045	dotnet-runtime	linux-arm64 release AOT
2924275	dotnet-runtime	linux-x64 release coreclr_r2r_interpreter
2928024	dotnet-runtime	browser-wasm linux Release wasm
2928009	dotnet-runtime	linux-x64 release AOT
2927977	dotnet-runtime	browser-wasm linux Release wasm
2927962	dotnet-runtime	linux-arm64 release AOT
2927847	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net11.0-android-graph
2927944	dotnet-runtime	browser-wasm linux Release wasm
2927900	dotnet-runtime	linux-arm64 release AOT
2927875	dotnet-runtime	linux-arm64 release AOT
2927857	dotnet-runtime	linux-x64 release coreclr
2927854	dotnet-runtime	browser-wasm linux Release wasm_coreclr
2927648	dotnet-maui	Controls (v18.4) Navigation
2927836	dotnet-runtime	linux-x64 release bluebird
2927818	dotnet-runtime	browser-wasm linux Release wasm
2927790	dotnet-runtime	browser-wasm linux Release wasm_coreclr
2927718	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net11.0-android-graph
2927758	dotnet-runtime	linux-arm64 release AOT
2927747	dotnet-runtime	linux-x64 release AOT
2927580	dotnet-maui	Controls (v18.4) TabbedPage,TableView,TimePicker,TitleView,ToolbarItem	Log
Displaying 100 of 207 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
2	10	207

[lewing]

Looks like mostly macs? not sure where Build Source Index runs

[epananth]

Looked at the latest builds all are failing cos out of space on the device

[epananth]

##[error]Error reported in diagnostic logs. Please examine the log for more details. - /Users/runner/runners/3.242.1/_diag/Worker_20240806-221514-utc.log System.IO.IOException: No space left on device : '/Users/runner/runners/3.242.1/_diag/Worker_20240806-221514-utc.log' at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan1 buffer, Int64 fileOffset)
at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.Diagnostics.TextWriterTraceListener.Flush()
at Microsoft.VisualStudio.Services.Agent.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id) in /Users/runner/work/1/s/src/Microsoft.VisualStudio.Services.Agent/HostTraceListener.cs:line 151
at Microsoft.VisualStudio.Services.Agent.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message) in /Users/runner/work/1/s/src/Microsoft.VisualStudio.Services.Agent/HostTraceListener.cs:line 81
at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
at Microsoft.VisualStudio.Services.Agent.Worker.Worker.RunAsync(String pipeIn, String pipeOut) in /Users/runner/work/1/s/src/Agent.Worker/Worker.cs:line 93
at Microsoft.VisualStudio.Services.Agent.Worker.Program.MainAsync(IHostContext context, String[] args) in /Users/runner/work/1/s/src/Agent.Worker/Program.cs:line 52`

[epananth]

I am going to go ahead an add a check for disk utilization, to know more why this is failing.

[epananth]

Is it possible to add this check to your pipeline to get more data (like aspnet-core team did)

https://github.com/dotnet/aspnetcore/blob/42af9fe6ddd7c3f9cde04ac003bf97509881873b/.azure/pipelines/jobs/default-build.yml#L163-L165

[missymessa]

@jeffschwMSFT @lewing is this still an issue?

[lewing]

@jeffschwMSFT @lewing is this still an issue?

yes

[lewing]

I took a look at the recent entries in this issue. All the latest builds hitting this error are from the dotnet-runtime-perf and dotnet-runtime-perf-build pipelines running on NetCore1ESPool-Internal agents.

Checking the build logs (e.g. builds 2926621, 2926620, 2926505, 2926495, 2926451, 2926370), they all show the same root cause: disk space exhaustion on the agents — consistently at 97–100% usage. This triggers a cascade of failures:

• Docker pulls fail with "no space left on device"
• Build processes crash with SIGABRT (exit code 134)
• Timeouts as processes stall on disk I/O
• The generic "Error reported in diagnostic logs" wrapper error

This appears to be the same underlying issue reported by @epananth back in Aug 2024 — the perf build agents need either larger disks or more aggressive cleanup between jobs. The disk utilization check that was added helps surface the problem, but the root cause (insufficient disk capacity for the workload) is still present.

[lewing]

cc @DrewScoggins @mangod9

[DrewScoggins]

Yeah, we are seeing this as well. We have a PR up, dotnet/runtime#125634, to fix this on the build side, and should have another up to fix the perf job side as well.