[bug] AssertionError: Conan failed to acquire database lock

[Alexolut]

Describe the bug

Windows Server 2022, Conan 2.22.1, Python 3.11

The problem arises after upgrade Conan from 2.21.0 to 2.22.0 (and 2.22.1) on our CI workflow during conan install packages. It's not quite stable fail, sometimes it works as expected. Below you can find logs (some parts are truncated for brevity):

Run py -3 -m pip install conan==2.22.1
Collecting conan==2.22.1
  Downloading conan-2.22.1.tar.gz (542 kB)
     -------------------------------------- 542.4/542.4 kB 3.8 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: requests<3.0.0,>=2.25 in c:\python311\lib\site-packages (from conan==2.22.1) (2.32.4)
Requirement already satisfied: urllib3<3.0,>=1.26.6 in c:\python311\lib\site-packages (from conan==2.22.1) (1.26.20)
Requirement already satisfied: colorama<0.5.0,>=0.4.3 in c:\python311\lib\site-packages (from conan==2.22.1) (0.4.6)
Requirement already satisfied: PyYAML<7.0,>=6.0 in c:\python311\lib\site-packages (from conan==2.22.1) (6.0.2)
Collecting patch-ng<1.19,>=1.18.0 (from conan==2.22.1)
  Downloading patch-ng-1.18.1.tar.gz (17 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: fasteners>=0.15 in c:\python311\lib\site-packages (from conan==2.22.1) (0.19)
Requirement already satisfied: Jinja2<4.0.0,>=3.0 in c:\python311\lib\site-packages (from conan==2.22.1) (3.1.6)
Requirement already satisfied: python-dateutil<3,>=2.8.0 in c:\python311\lib\site-packages (from conan==2.22.1) (2.9.0.post0)
Requirement already satisfied: MarkupSafe>=2.0 in c:\python311\lib\site-packages (from Jinja2<4.0.0,>=3.0->conan==2.22.1) (3.0.2)
Requirement already satisfied: six>=1.5 in c:\python311\lib\site-packages (from python-dateutil<3,>=2.8.0->conan==2.22.1) (1.16.0)
Requirement already satisfied: charset_normalizer<4,>=2 in c:\python311\lib\site-packages (from requests<3.0.0,>=2.25->conan==2.22.1) (3.4.2)
Requirement already satisfied: idna<4,>=2.5 in c:\python311\lib\site-packages (from requests<3.0.0,>=2.25->conan==2.22.1) (3.10)
Requirement already satisfied: certifi>=2017.4.17 in c:\python311\lib\site-packages (from requests<3.0.0,>=2.25->conan==2.22.1) (2025.7.14)
Building wheels for collected packages: conan, patch-ng
  Building wheel for conan (pyproject.toml): started
  Building wheel for conan (pyproject.toml): finished with status 'done'
  Created wheel for conan: filename=conan-2.22.1-py3-none-any.whl size=682077 sha256=4390d9d82d06611c2acd6e87abcd7927a05efd5e4e740782616012df23e02704
  Stored in directory: c:\users\gh-ci\appdata\local\pip\cache\wheels\59\1b\43\6502c708257c50b84469763d7f15a80db57489b28b5c5c3db7
  Building wheel for patch-ng (pyproject.toml): started
  Building wheel for patch-ng (pyproject.toml): finished with status 'done'
  Created wheel for patch-ng: filename=patch_ng-1.18.1-py3-none-any.whl size=17028 sha256=cdc3c7269eeb25b966e061f19daf34445608598f14f8bc54b57b3127a51b1f1a
  Stored in directory: c:\users\gh-ci\appdata\local\pip\cache\wheels\7b\cc\49\8bc9f301f7969862da20988c1bd25c60f20df61718a01b2449
Successfully built conan patch-ng
Installing collected packages: patch-ng, conan
  Attempting uninstall: patch-ng
    Found existing installation: patch-ng 1.17.4
    Uninstalling patch-ng-1.17.4:
      Successfully uninstalled patch-ng-1.17.4
  Attempting uninstall: conan
    Found existing installation: conan 1.61.0
    Uninstalling conan-1.61.0:
      Successfully uninstalled conan-1.61.0
Successfully installed conan-2.22.1 patch-ng-1.18.1

Notice:  A new release of pip is available: 23.2.1 -> 25.3
Notice:  To update, run: python.exe -m pip install --upgrade pip

....

======== Installing packages ========

-------- Downloading 95 packages --------
Downloading binary packages in 16 parallel threads


....



ERROR: Traceback (most recent call last):
  File "C:\Python311\Lib\site-packages\conan\cli\cli.py", line 297, in main
    cli.run(args)
  File "C:\Python311\Lib\site-packages\conan\cli\cli.py", line 193, in run
    command.run(self._conan_api, args[0][1:])
  File "C:\Python311\Lib\site-packages\conan\cli\command.py", line 191, in run
    info = self._method(conan_api, parser, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\conan\cli\commands\install.py", line 75, in install
    conan_api.install.install_binaries(deps_graph=deps_graph, remotes=remotes)
  File "C:\Python311\Lib\site-packages\conan\api\subapi\install.py", line 30, in install_binaries
    installer.install(deps_graph, remotes, install_order=install_order)
  File "C:\Python311\Lib\site-packages\conan\internal\graph\installer.py", line 255, in install
    self._download_bulk(install_order)
  File "C:\Python311\Lib\site-packages\conan\internal\graph\installer.py", line 286, in _download_bulk
    thread_pool.map(self._download_pkg, downloads)
  File "C:\Python311\Lib\multiprocessing\pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\multiprocessing\pool.py", line 774, in get
    raise self._value
  File "C:\Python311\Lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\multiprocessing\pool.py", line 48, in mapstar
    return list(map(*args))
           ^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\conan\internal\graph\installer.py", line 297, in _download_pkg
    self._remote_manager.get_package(node.pref, node.binary_remote)
  File "C:\Python311\Lib\site-packages\conan\internal\rest\remote_manager.py", line 142, in get_package
    pkg_layout = self._cache.create_pkg_layout(pref)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\conan\internal\cache\cache.py", line 161, in create_pkg_layout
    self._db.create_package(package_path, pref, None)
  File "C:\Python311\Lib\site-packages\conan\internal\cache\db\cache_database.py", line 87, in create_package
    self._packages.create(path, ref, build_id=build_id)
  File "C:\Python311\Lib\site-packages\conan\internal\cache\db\packages_table.py", line 80, in create
    with self.db_connection() as conn:
  File "C:\Python311\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\conan\internal\cache\db\table.py", line 26, in db_connection
    assert self._lock.acquire(timeout=20), "Conan failed to acquire database lock"
AssertionError: Conan failed to acquire database lock

ERROR: Conan failed to acquire database lock

It seems highly likely related to parallel download, that is used by default since 2.22 (#19099). I noticed similar fails before on 2.21 when I was experimenting with --core-conf core.download:parallel={os.cpu_count()} for conan install command.

If it's necessary to provide additional details to simplify investigation, please let me know.

I found old similar bug that is marked as fixed: #13924

BTW: Will downloading work in single-thread mode if I add --core-conf core.download:parallel=1? Instead if I'm not able to disable parallel download we will stuck on 2.21 version, due to that error with database lock.

[memsharded]

Hi @Alexolut

Thanks for reporting.

BTW: Will downloading work in single-thread mode if I add --core-conf core.download:parallel=1? Instead if I'm not able to disable parallel download we will stuck on 2.21 version, due to that error with database lock.

Yes, a core.download:parallel=0 disables parallelism completely (the 0 value means "use the default=nothing". It seems that we could also disable the thread pool for 1 value too, but that is another story).
Please use that to deactivate parallelism while we further check this (and allows you to upgrade to Conan 2.22 if there are other features/fixes you'd like to use from 2.22).

[memsharded]

Trying to understand what could be happening, this DB acquisition error only happens in our experience in quite heavily busy systems, with resources used at full/over capacity.
Can you please comment if you think there is some correlation with server load for these failures? Can you run some tests with the server used exclusively for the current build and not others in parallel?
Can you comment on the relative frequency of the error, is it very often in many builds, or relatively infrequent?
Do you have other servers in other platforms, are those failing too? Or is the Windows server machine the only one?

[memsharded]

As some extra information: we didn't just enable the parallel downloads arbitrarily.
We enabled it first in ConanCenter, that has been running a couple of months with it, without any issue like this one, then decided to move it forward as the default in Conan.

It could be related to the specific server configuration like Windows Server 2022, Conan 2.22.1, Python 3.11 and the load/concurrency that the server is managing. ConanCenter handles a lot of activity, but the concurrency isn't necessarily that massive.

[Alexolut]

From my observation this happening in ~30-50% cases. However I didn't do many tests yet. It only happened on Windows machines. Worth noting that on Linux build we use python 3.12 (compared to 3.11 on Windows). It's hard to provide any info regarding CPU/disk/network load at "conan install" phase, but I can say that we don't use same VM for other tasks at this phase, it expected to be exclusively used for "conan install". At the same time I can't guarantee that there's nothing other that can be started by the system in background mode (e.g. antivirus scan or something else).

If you have any ideas what kind of additional logs/actions can help solve this issue please let me know.

Set core.download:parallel=0 works fine at first sight.

[marlamb]

It seems not be limited to Windows. I saw the very same problem on Ubuntu 24.04 with Python 3.12 and conan 2.22.1 in a medium sized project (a little more than 200 packages need to be downloaded). I don't have exact measures how often that happens, but I would guess it is something between 1% and 10%.

[memsharded]

Thanks for the feedback @marlamb
Do you have any insights about that server load when this is happening? Is it like at full capacity, might be struggling with resources?

[marlamb]

Unfortunately I am not able to access the server while the conan install is executed. But the server is apart from default processes from the OS not running any other (potentially resource-hungry) process in parallel.

[memsharded]

You mean that it seems that the server resources might not be depleted or at full capacity?
What is the number of cores/parallelism that is being used in your case? Are you using the Conan default or a different explicit value?

[marlamb]

I am using the Conan defaults. The machine has eight logical (four physical) cores and 32GB RAM. I can't say if it comes to full capacity by the conan install, but apart from that nothing is running that should get it even close to depletion.

[Alexolut]

Interesting observation, I have disabled parallel download via core.download:parallel=0 in our master branch and it works stable on CI with Conan 2.22.1. At the same time I have my branch with parallel download enabled (the diff is only one line mentioned earlier is removed). And this branch works fine as well. I'm start thinking what if the problem relies not on the client side performance, but on the server side where binaries are stored (artifactory instance)?

• Many clients but single thread - OK
• One client with multiple threads - OK too
• Many clients with multiple threads - Lock failed (but due to slow server response?)

[memsharded]

Interesting hypothesis.
I have done a review of the code to see if there was something in the DB access that could be being locked by some slow server reponses. The lock that is failing to acquire is in:

@contextmanager
    def db_connection(self):
        assert self._lock.acquire(timeout=20), "Conan failed to acquire database lock"
        connection = sqlite3.connect(self.filename, isolation_level=None, timeout=20)
        try:
            yield connection
        finally:
            connection.close()
            self._lock.release()

And then it seems all usages of db_connection() context happen synchronously with existing and available data, so I cannot find a place where the server slowness could be causing this issue.

I have started to do some tests for the cache concurrency, I think I will try to do some concurrency stress tests for the DB tables, to see if I can force this error somehow.

[Alexolut]

During another one test run (with parallel download enabled) on our CI I faced with a different error. Again on conan install phase. It may or may not be related to the concurrency, but potentially can point to some issues on Artifactory side, I guess.

lib_name/1.0.1@user/channel: ERROR: Exception while getting package: da39a3ee5e6b4b0d3255bfef95601890afd80709
lib_name/1.0.1@user/channel: ERROR: Exception: <class 'conan.errors.ConanException'> Response ended prematurely. [Remote: my_remote]
...
ERROR: Response ended prematurely. [Remote: my_remote]

[memsharded]

I have been investigating, trying to stress the CacheDatabase thread lock, and reading about possible causes.

One possible reason could be inefficiencies of the Python GIL, that doesn't allow to do progress quick enough.

Unfortunately, stressing the CacheDatabase I have been only to produce timeouts of 1-2 seconds maximum, very far from the current 20 seconds limit.

Could you please tell the size in rows of the sqlite3 DB called cache.sqlite3 inside the Conan home .conan2/p folder, the table called recipes? (when these locking errors are shown).

There is a call there that could be quite inefficient for large tables, and that might be triggered often in case of version ranges usage, maybe we could try to optimize that.