Investigate what's going on in Symbol Validation

[JohnTortugo]

We got a few failures in symbol validation in the past few days. I'm disabling that step in our release process until I figure out what's happening and have a fix for it.

Some of the failed releases:

https://dnceng.visualstudio.com/internal/_releaseProgress?_a=release-pipeline-progress&releaseId=7929
https://dnceng.visualstudio.com/internal/_releaseProgress?_a=release-pipeline-progress&releaseId=7849
https://dnceng.visualstudio.com/internal/_releaseProgress?_a=release-pipeline-progress&releaseId=7847

@jcagme @mmitche

[JohnTortugo]

The failures observed so far:

2019-05-23T02:06:54.2037133Z Validating Microsoft.DotNet.SwaggerGenerator.MSBuild.1.0.0-beta.19272.12.nupkg 
...
2019-05-23T02:07:03.1245831Z 	 Checking file D:\a\r1\a\PrimaryArtifact\Temp\ad341f27-dbed-42b1-bf7e-00151b55ea5e\tools\net472\SharpYaml.dll ... No symbols found on MSDL or SymWeb!

2019-05-21T23:48:23.6122251Z Validating Microsoft.DotNet.Build.Tasks.Feed.2.2.0-beta.19271.6.nupkg 
...
2019-05-21T23:50:14.5239237Z 	 Checking file D:\a\r1\a\PrimaryArtifact\Temp\0ae89340-6508-4012-916b-e13c78c31b01\tools\net472\Microsoft.DotNet.VersionTools.dll ... No symbols found on MSDL or SymWeb!

These are considered flaky because retry of the same validation pass and moreover there was no change to these packages to justify a sudden validation failure.

[alexperovich]

The first failure is not real, that dll is a 3rd party dependency. There is no way we could have produced symbols for it.

[JohnTortugo]

Yet, there is symbol for that file in the symbol server. I think the reason is because in the absence of a .symbols.nupkg we just copy the .nupkg for the package and assume the regular module contains embedded PDB info. This is explained here: https://github.com/dotnet/arcade/blob/master/src/Microsoft.DotNet.Arcade.Sdk/tools/Publish.proj#L69

BTW, here is how I check for symbol availability (I also check MSDL):

.\SymbolTool\dotnet-symbol.exe --symbols --modules --windows-pdbs --microsoft-symbol-server "C:\temp\SymbolAvailability\Teste\SharpYaml.dll" -o ./TestSymbols

[JohnTortugo]

Some more info:

At 12:53 I executed the validation and got failure for these 4 packages:

Checking file C:\temp\SymbolAvailability\Extract\e845f1b6-4c5b-4e87-be61-da18e8199ba7\tools\net472\Microsoft.DotNet.SwaggerGenerator.CodeGenerator.dll ... No symbols found on MSDL or SymWeb!
Checking file C:\temp\SymbolAvailability\Extract\e845f1b6-4c5b-4e87-be61-da18e8199ba7\tools\net472\Microsoft.DotNet.SwaggerGenerator.MSBuild.dll ... No symbols found on MSDL or SymWeb!
Checking file C:\temp\SymbolAvailability\Extract\e845f1b6-4c5b-4e87-be61-da18e8199ba7\tools\netcoreapp2.1\Microsoft.DotNet.SwaggerGenerator.CodeGenerator.dll ... No symbols found on MSDL or SymWeb!
Checking file C:\temp\SymbolAvailability\Extract\e845f1b6-4c5b-4e87-be61-da18e8199ba7\tools\netcoreapp2.1\Microsoft.DotNet.SwaggerGenerator.MSBuild.dll ... No symbols found on MSDL or SymWeb!

Then I executed the symbol publishing: https://dnceng.visualstudio.com/internal/_releaseProgress?_a=release-pipeline-progress&releaseId=8146

At 12:55 I retried the validation: Still failing
At 13:00 I retried the validation: Still failing
At 13:04 I retried the validation: PASSED

@mikem8361 are you aware of any reason for delays before the symbols be available in the feed?

[mikem8361]

I would guess the 10 minute or so delay it is part of the symbol uploading process. I don’t think there is anything on the client side (SymbolUploader) that could change it. It is certainly way better than the old “CreateRequest.cmd” day or so delay.

[JohnTortugo]

These two files together has 100Kb. The whole package they are in has less than 1MB. The PublishSymbols task from Symbol-Uploader finished in about 10 seconds :-/ .

IIRC these "servers" are backed by AzDO feeds, so I'm considering reach to the AzDO-Artifacts folks to see if they can give more light on this. Do you think that makes sense?

[jcagme]

Does this happen only for these two files?

[JohnTortugo]

Does this happen only for these two files?

Yes, and I think the reason is because those files are what's really changing from build to build; i.e., the other files are kind of cached somehow... I still need to investigate that as well.

[JohnTortugo]

BTW, just got an answer from some folks in an e-mail thread:

This delay is expected. Hits/misses of symbols to those endpoints are cached for 10 minutes. As such, if your pipeline queries for the symbols right after publishing them (and before the services have finished processing them), the cache will record a miss.

[dougbu]

@JohnTortugo where is the cache they mention? If it's similar to the multi-level NuGet cache and is on the client machine somewhere, perhaps we could clear it prior to doing symbol validation…

[JohnTortugo]

@JohnTortugo where is the cache they mention? If it's similar to the multi-level NuGet cache and is on the client machine somewhere, perhaps we could clear it prior to doing symbol validation…

I understood that would be on the server itself. I included you in the thread.