[question] Building and uploading missing packages changes the recipe revision timestamp

[alexsmedin]

What is your question?

Not sure if this is a bug or intentional for some reason.

A conan config change (core.package_id:default_python_mode in global.conf) caused our conan installs to search for different package ids, which were then missing because the recipe revisions that were locked in lockfiles did not have packages for the new config. To address this, we used -b missing in CI and wrote json output to a graph, then ran conan list on the graph and finally an upload on the list, causing the missing packages to be built and uploaded to Artifactory.

So far, everything worked well and projects that locked these revisions could find and install the new packages.
However, when updating lockfiles with -u to get the latest revision, we noticed that the rebuilt revisions are now being picked up as the latest. Checking in Artifactory confirms that the recipe revision .timestamp files have changed for the revisions that we uploaded new packages for.

This is a problem because it means that the revision that is actually supposed to be the latest either needs to be rebuilt and reuploaded or the timestamp manually edited to make it the latest revision again. And since the latest revision is already built with the new config before rebuilding the older revision upload of the latest revision is skipped, making it hard to restore the latest revision (though I guess force uploading may help if that also changes the timestamp). Wouldn't it make more sense if only the .timestamp files in the added packages were new and not the one for the recipe revision, since the recipe hasn't changed?

We are using conan 2.4.1 and Artifactory 7.55.10.

Is there some reason I'm not aware of why the recipe timestamp needs to change or is this a bug? Would it make sense with a feature request to add a flag to the conan upload command to upload packages without altering the recipe timestamp in order to simply add additional packages without changing the order of revs?

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @alexsmedin

Thanks for your question.

I have been trying to reproduce, but so far it seems the behavior is correct, and the ordering of revisions in the server side is stable.

This is what I have tried:

• conan new cmake_exe -d name=myapp -d version=0.1
• conan create . # config Release, revision1
• conan upload "*" -r=artifactory -c # upload revision1 - Release
• Do a change in .cpp file
• conan create . # config Release revision2 because of changes
• ``conan upload "*" -r=artifactory -c" # uploads revision2 - Release
• conan list "*#*" shows correct ordering both in client and server
• Remove the changes in .cpp file to return to previous revision1
• conan create . -s build_type=Debug
• conan upload "*" -r=artifactory # Upload revision1 - Debug
• conan list "*#*" -r=artifactory still shows correct order
• Recover the changes in .cpp file, to force again revision2
• conan create . -s build_type=Debug
• conan upload "*" -r=artifactory # Upload revision2 - Debug
• conan list "*#*" still shows correct order both in client and server

So the revisions order in the server side seems stable as expected, uploading new binaries for older revisions do not alter the order of the revisions

[alexsmedin]

Thank you for the quick reply! If that is the case and the expected behavior is that the revision order persists when adding packages, I'll try to reproduce it manually and share some instructions for doing so if I'm able to. Just wanted to know what was expected before investigating it further!

[alexsmedin]

We figured out what went wrong and the problem was not where we expected it, the upload of additional packages does indeed not change the timestamps.

There was a pipeline that used a lockfile and ran conan upload with --force after building packages. Every time that ran, all the revisions in the lockfile became the latest of its requirements. I guess that is to be expected when reuploading, so the error was on our side and we'll remove that flag. However, there could perhaps also be a warning in the conan upload documentation that the force flag changes the revision order. Closing the issue, thanks again for looking into it.

[memsharded]

I am not so sure either that the --force changes that order too.
We have a test in the test suite, I have done #16621, to test it against Artifactory, in case it was behaving differently. The test is passing correctly.

The test is re-exporting an older revision, so it gets a new timestamp, and uploading it with --force. apparently it doesn't change the revision order either.

[alexsmedin]

Hm, maybe it was the combination of adding new packages with --force? I am able to reproduce it with very similar steps to what you described before.

The first conan create made a myapp/0.1#ebe3dfe396fc585b7e4e3a93994cdcd1, and after editing a cpp file the second run created myapp/0.1#c186e0c3d24ece0f27cf34d323b6e990. Also uploaded them both to artifactory after creating, and running list on the local cache and the remote gives the following output.

> conan list "myapp/*#*"
Found 1 pkg/version recipes matching myapp/* in local cache
Local Cache
  myapp
    myapp/0.1
      revisions
        ebe3dfe396fc585b7e4e3a93994cdcd1 (2024-07-08 11:00:24 UTC)
        c186e0c3d24ece0f27cf34d323b6e990 (2024-07-08 11:01:36 UTC)

> conan list "myapp/*#*" -r artifactory-local
Found 1 pkg/version recipes matching myapp/* in artifactory-local
artifactory-local
  myapp
    myapp/0.1
      revisions
        ebe3dfe396fc585b7e4e3a93994cdcd1 (2024-07-08 11:01:22 UTC)
        c186e0c3d24ece0f27cf34d323b6e990 (2024-07-08 11:02:07 UTC)

After reverting to the original src content and running conan create . -s build_type=Debug, local cache looks like this

> conan list "myapp/*#*"
Found 1 pkg/version recipes matching myapp/* in local cache
Local Cache
  myapp
    myapp/0.1
      revisions
        c186e0c3d24ece0f27cf34d323b6e990 (2024-07-08 11:01:36 UTC)
        ebe3dfe396fc585b7e4e3a93994cdcd1 (2024-07-08 11:04:53 UTC)

I then ran conan upload myapp/0.1#ebe3dfe396fc585b7e4e3a93994cdcd1 -r artifactory-local --force (note the force flag) and listed the revisions in the remote again. As you can see the ebe... revision is now the latest.

> conan list "myapp/*#*" -r artifactory-local
Found 1 pkg/version recipes matching myapp/* in artifactory-local
artifactory-local
  myapp
    myapp/0.1
      revisions
        c186e0c3d24ece0f27cf34d323b6e990 (2024-07-08 11:02:07 UTC)
        ebe3dfe396fc585b7e4e3a93994cdcd1 (2024-07-08 11:07:13 UTC)

[memsharded]

Thanks for the detail.

UPDATE: I have been able to reproduce, I have added a test that fails when used against Artifactory in my PR #16621

We need to carefully check where would be the discrepancy, if it is a failure in the testing server or in Artifactory, it is not fully clear now what was the expected behavior.

The test was failing against my local Artifactory because I was reusing the repo and not cleaning it. After cleaning it, the test, that I think it captures your described scenario, is passing in both cases.

[memsharded]

I have updated my comment above, the test seems to be passing against Artifactory too, I was testing it wrong.

[memsharded]

Maybe the Artifactory version is related? I am using 7.77.5, and you are using Artifactory 7.55.10. For some reason I overlooked the version, I think I read the 55 and confused it for 77.5. That version is like 15 months old, I don't recall right now a explicit bugfix for this, but some other changes were done in between, there might be a chance that this got fixed since then.