Conan upload to Artifactory is intermittently failing after getting stuck for 10mins

[sharadupadhyaya]

Conan version: 3.2.1
Artifactory version: Enterprise Plus 7.77.11

Scenario:
Jenkins pipeline is building and uploading around 20 builds with 7 variety packages under each of these in parallel from different executor hosts, each package size is under 40MB.

Example build/package structure:

Command used by the pipeline to upload:
conan upload val_val_cortex-m23/0.0+10145 -r fm-conan-prod -c -v trace

This was all working fine two weeks ago. Since couple of weeks one or two uploads are failing intermittently but frequently.

In successful upload cases, the upload takes just under a minute.

In failure cases, the upload command hangs for many minutes with no outcome.
In the pipeline logic the upload command is guarded by 10 mins timer which terminates the upload command after 10mins wait.

Tried various verbose option with conan upload command, but did not get any meaningful logs/traces.

Upload/download configs:

tools.files.download:retry_wait=10
tools.files.download:retry=10
core.net.http:timeout=630
core.net.http:max_retries=10
core.upload:retry=10
core.upload:retry_wait=10

Any advice how to move further to investigate this?
Any command option to get further logging to see what the command is doing and getting stuck at?

[memsharded]

Hi @sharadupadhyaya

Thanks for your feedback.

In some conditions, because the uploads are not fully atomic on the server side, there might be race conditions over the "recipe" part of the upload (that is the export folder containing the conanfile.py and other recipe files).
The best way to avoid it is do a preliminary conan upload of the recipe only (this can be super fast, just at the beginning of the pipeline, a simple conan export + conan upload will do it).

If that is not the case, then maybe it could be some server side issue, I think 7.77 is a relatively older version of Artifactory, so it might be recommended to upgrade. In any case, I'd suggest opening a ticket against Artifactory, because that seems more like a server side issue, I don't know either what could be, as I cannot reproduce it from my end.

As another side note, Conan 3.2.1 version doesn't exist. If it is Conan 2.3, that is also relatively old, like 2 years old, I'd recommend updating to latest Conan 2.23. Thanks

[sharadupadhyaya]

Hi @memsharded ,
Thanks for the response and suggestions.

In case of race condition with respect to recipe you suggested, the failure response from Artifactory should be immediate.
In the case detailed above, the upload command hangs for many minutes with no outcome and then the pipeline needs to forcefully terminating it as guarding it under a 10 min timer.

Sorry the Conan version 3.2.1 was a type, it's 2.3.1.
The version of Artifactory or Conan is not changed recently, it was all working fine two weeks ago with same versions.

[memsharded]

Thanks for the feedback:

Sorry the Conan version 3.2.1 was a type, it's 2.3.1.

Still a bit outdated, we are now in Conan 2.23, that is 20 releases more that got a lot of improvements and fixes, it would be very recommended to upgrade in any case.

The version of Artifactory or Conan is not changed recently, it was all working fine two weeks ago with same versions.

I am afraid that I have no idea what could be happening, without any possible way to reproduce it.
Maybe the first step would be to try to isolate if it is a client issue or a server issue.

For that, the best would be if you could setup a pipeline (or a simple script that runs on a developer computer) with a local ArtifactoryCE and see if it reproduces against it. If it reproduces against it, then it could be some client side issue, please let us know and we will investigate further from that setup.

If it doesn't reproduce, then it seems a server side issue, and the best would be to report it to Artifactory support.

[memsharded]

Hi @sharadupadhyaya

Did you manage to reproduce with a local ArtifactoryCE?