[bug] `conf_info` set by build context package not set by Conan during installation of host packages

[tttapa]

Describe the bug

The values in self.conf_info set in the package_info method of a build-require package do not take effect during the installation of host context dependencies, resulting in (silently) mis-configured packages and/or build failures.

---

I'm using Conan to package a toolchain as described in https://docs.conan.io/2/examples/cross_build/toolchain_packages.html. The toolchain's package_info sets tools.build:compiler_executables and tools.build.cross_building:cross_build:

    def package_info(self):
        ...
        compilers = {
            "c": f"{target}-gcc",
            "cpp": f"{target}-g++",
            "fortran": f"{target}-gfortran",
        }
        self.conf_info.update("tools.build:compiler_executables", compilers)
        self.conf_info.define("tools.build.cross_building:cross_build", True)

https://github.com/tttapa/conan-recipes/blob/bed3326dff0800deb9f04663f45d0247a41cc292/recipes/tttapa-toolchains/binary/conanfile.py#L130

Other packages, such as OpenBLAS rely on these values (using conan.tools.build.cross_building and self.conf.get("tools.build:compiler_executables")).
However, Conan does not always correctly set these values when building host packages.

This results in build failures, for example, because a conanfile for a host package might see cross_building=False, even though it was set to true by the toolchain, or in linker errors in OpenBLAS because libgfortran is not included due to compiler_executables["fortran"] being empty, even though it was set by the toolchain.

How to reproduce it

Add some print statements to the OpenBLAS recipe (or any other recipe):

diff --git a/recipes/openblas/all/conanfile.py b/recipes/openblas/all/conanfile.py
index 6fcc5d830..432b4f006 100644
--- a/recipes/openblas/all/conanfile.py
+++ b/recipes/openblas/all/conanfile.py
@@ -101,6 +101,8 @@ class OpenblasConan(ConanFile):
     @property
     def _fortran_compiler(self):
         comp_exe = self.conf.get("tools.build:compiler_executables")
+        print("Compiler executables:", comp_exe)
+        print("Cross-building:", cross_building(self))
         if comp_exe and "fortran" in comp_exe:
             return comp_exe["fortran"]
         return None

Use the following profile:

[settings]
arch=x86_64
build_type=Debug
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=14
os=Linux

[tool_requires]
tttapa-toolchains/1.0.2

[options]
openblas/*:target=HASWELL

This uses this toolchain https://github.com/tttapa/conan-recipes/blob/main/recipes/tttapa-toolchains/binary/conanfile.py (but any build-require package that sets self.conf_info will probably also demonstrate the issue).

Then build OpenBLAS:

conan create conan-center-index/recipes/openblas/all --version 0.3.27 --build=missing -pr test-profile

You'll see from the print statements that the conf_info is incorrect:

openblas/0.3.27: Calling package()
openblas/0.3.27: Running CMake.install()
openblas/0.3.27: RUN: cmake --install "/home/pieter/.conan2/p/b/openbc9f5b69c5dfef/b/build/Debug" --prefix "/home/pieter/.conan2/p/b/openbc9f5b69c5dfef/p"
-- Install configuration: "Debug"
-- Installing: /home/pieter/.conan2/p/b/openbc9f5b69c5dfef/p/lib/libopenblas.a
[...]

openblas/0.3.27: Package folder /home/pieter/.conan2/p/b/openbc9f5b69c5dfef/p
Compiler executables: {'c': 'x86_64-bionic-linux-gnu-gcc', 'cpp': 'x86_64-bionic-linux-gnu-g++', 'fortran': 'x86_64-bionic-linux-gnu-gfortran'}
Cross-building: True

======== Launching test_package ========

======== Computing dependency graph ========
Graph root
    openblas/0.3.27 (test package): /home/pieter/Downloads/conan-center-index/recipes/openblas/all/test_package/conanfile.py
Requirements
    openblas/0.3.27#98109524ab5c1dee94d06fc816c83442 - Cache
Build requirements
    tttapa-toolchains/1.0.2#73c6482fe9bf9a138c96619f083635bf - Cache

======== Computing necessary packages ========
Requirements
    openblas/0.3.27#98109524ab5c1dee94d06fc816c83442:01ceb14e4b3bb26bc1152b145930df01741d996e#8cca61d8c4b7c53c8dcae31f7ddf1f43 - Cache
Build requirements
    tttapa-toolchains/1.0.2#73c6482fe9bf9a138c96619f083635bf:fbe2cec9c27dea4585e3cc53888517dabd7ab58f#f9cfa7834c59ef749fdf09621418c7a7 - Cache

======== Installing packages ========
tttapa-toolchains/1.0.2: Already installed! (1 of 2)
openblas/0.3.27: Already installed! (2 of 2)
Compiler executables: None
Cross-building: False

Note how during the installation of the dependencies of the test_package, there are no compiler executables set, and cross_building returns false, even though these values were set by the tttapa-toolchains package!

Inspecting the conan-center-index/recipes/openblas/all/test_package/build/gcc-14-x86_64-gnu17-debug/generators/OpenBLAS-debug-x86_64-data.cmake file indeed reveals that Conan incorrectly configured the dependencies: the gfortran library is missing from the openblas_SYSTEM_LIBS_DEBUG variable, which causes linker errors as described above.

mold: error: undefined symbol: _gfortran_compare_string
>>> referenced by ilaenv.f
>>>               /home/runner/work/.conan2/p/b/openbbb6b930d00c7e/p/lib/libopenblas.a(ilaenv.f.o):(ilaenv_)
>>> referenced by ilaenv.f
>>>               /home/runner/work/.conan2/p/b/openbbb6b930d00c7e/p/lib/libopenblas.a(ilaenv.f.o):(ilaenv_)
>>> referenced by ilaenv.f
>>>               /home/runner/work/.conan2/p/b/openbbb6b930d00c7e/p/lib/libopenblas.a(ilaenv.f.o):(ilaenv_)

[tttapa]

A similar problem arises in the validate_build method: the OpenBLAS recipe queries cross_building() there (the target option needs to have a valid value when cross-building), but cross_building() incorrectly returns false, even though it was set to true by the toolchain package.

diff --git a/recipes/openblas/all/conanfile.py b/recipes/openblas/all/conanfile.py
index 6fcc5d830..5a94b7cf8 100644
--- a/recipes/openblas/all/conanfile.py
+++ b/recipes/openblas/all/conanfile.py
@@ -101,6 +101,8 @@ class OpenblasConan(ConanFile):
     @property
     def _fortran_compiler(self):
         comp_exe = self.conf.get("tools.build:compiler_executables")
+        print("Compiler executables:", comp_exe)
+        print("Cross-building:", cross_building(self))
         if comp_exe and "fortran" in comp_exe:
             return comp_exe["fortran"]
         return None
@@ -141,6 +143,7 @@ class OpenblasConan(ConanFile):
                 raise ConanInvalidConfiguration(f'"{self.name}/*:build_relapack=True" option is only supported for GCC and Clang')
 
     def validate_build(self):
+        self._fortran_compiler
         if Version(self.version) < "0.3.22" and cross_building(self, skip_x64_x86=True):
             # OpenBLAS CMake builds did not support some of the cross-compilation targets in 0.3.20/21 and earlier.
             # This was fixed in https://github.com/OpenMathLib/OpenBLAS/pull/3714 and https://github.com/OpenMathLib/OpenBLAS/pull/3958

======== Computing dependency graph ========
Graph root
    cli
Requirements
    openblas/0.3.27#021a483a8e21ddf6afe70a0fb2b24291 - Cache
Build requirements
    tttapa-toolchains/1.0.2#73c6482fe9bf9a138c96619f083635bf - Cache

======== Computing necessary packages ========
Compiler executables: None
Cross-building: False
Connecting to remote 'conancenter' anonymously
Compiler executables: None
Cross-building: False

---

Edit: Even worse, this also means that self.conf contains invalid data inside of the configure() method, resulting in incorrect default options etc., leading to inconsistencies in later steps where self.conf does seem to contain the values set by the toolchain.

[memsharded]

Hi @tttapa

Thanks for your feedback.

I have verified that this test works:

def test_issue():
    # https://github.com/conan-io/conan/issues/18008
    c = TestClient()
    toolchain = textwrap.dedent("""
        from conan import ConanFile
        class Pkg(ConanFile):
            name = "tool"
            version = "0.1"
            def package_info(self):
                self.conf_info.update("tools.build:compiler_executables", {"c": "potato-c"})
        """)
    c.save({"conanfile.py": toolchain})
    c.run("create . --build-require")

    openblas = textwrap.dedent("""
        from conan import ConanFile

        class Pkg(ConanFile):
            name = "openblas"
            version = "0.1"

            def generate(self):
                comp_exe = self.conf.get("tools.build:compiler_executables")
                self.output.info(f"Compiler executables: {comp_exe}")
        """)
    profile = textwrap.dedent("""
        [tool_requires]
        tool/0.1
        """)
    c.save({"conanfile.py": openblas,
            "profile": profile},
           clean_first=True)
    c.run("create . -pr=profile")
    print(c.out)
    assert "openblas/0.1: Compiler executables: {'c': 'potato-c'}" in c.out

But using validate_build() instead of generate() doesn't work.

the reason is that conf_info is populated at generate() time, because that information from conf is intended to be leveraged by generate() only.

Usage in other parts of the recipe, like the usage that the openblas recipe does in package_info(), via the self._fortran_compiler is actually illegal. The package_info() cannot rely on information from the "tool_requires" to define itself, because the tool_requires might not even exist if the package has a precompiled binary and consequently the tool_requires are not necessary and are skipped.
This seems a necessary fix in the recipe.

Something similar happens for validate_build() it runs when computing the package_id, that is when it is decided if a package needs to be built from source or not, so obviously it is also executed even before the package_info() of the tool_requires is executed, because in many cases it will not even be executed as the tool_requires will be "skipped" as unnecessary.

So this seems expected and by design, the only possible places where the conf_info from tool_requires can be correctly defined in a consumer is in the generate(), build() and package() methods, as it is physically impossible otherwise.

This means that some recipes in ConanCenter that use this pattern might need some improvements, and maybe even that an extra recipe check (linter) could make sense to try to avoid these errors, but we need to talk this with the team.

[tttapa]

@memsharded Thanks a lot for the quick response!

Could you describe the proper way to handle the following use cases:

• How to know whether we're cross-compiling during validate_build? It seems to me that determining whether a package can be built depends on whether cross-compilation is enabled for some packages.
  For example:
  • https://github.com/conan-io/conan-center-index/blob/7647699a985ec682c5412b3e9ec3d495d9232b65/recipes/b2/portable/conanfile.py#L72
• Similarly, some key packages require a native version of themselves during cross-compilation, so this requires cross_building to return the correct value in the build_requirements method (it has to be conditional on cross_building to avoid circular dependencies in the case of native builds).
  For example:
  • https://github.com/conan-io/conan-center-index/blob/7647699a985ec682c5412b3e9ec3d495d9232b65/recipes/qt/6.x.x/conanfile.py#L434
  • https://github.com/conan-io/conan-center-index/blob/7647699a985ec682c5412b3e9ec3d495d9232b65/recipes/grpc/all/conanfile.py#L149
  • https://github.com/conan-io/conan-center-index/blob/7647699a985ec682c5412b3e9ec3d495d9232b65/recipes/flex/all/conanfile.py#L44
• Finally, how to deal with system_libs and other properties in package_info depending on values in the configuration, as in OpenBLAS? I guess the only alternative is to somehow save the necessary values during the package() method, but where should one save this information? Manually, using a file in the package folder?

The fact that self.conf is invalid during most methods should probably be documented somewhere (and Conan should preferably even throw a loud error if it is accessed elsewhere), because this is the first time I read about this restriction :)
The documentation of the conf attribute is very brief and doesn't mention this: https://docs.conan.io/2/reference/conanfile/attributes.html#conf
Similar for cross_building: https://docs.conan.io/2/reference/tools/build.html#conan-tools-build-cross-building

[memsharded]

Hi @tttapa

Sorry this was not followed up.

Your last comment seems to contain several different questions/issues, not completely related to the original one in this thread.
The response in the review of conan-io/docs#4037 gave a hint: this is only an issue for the conf_info defined in the tool_requires recipes.

That means that it is safe to define tools.build.cross_building:cross_build in the profile, and it will be defined, and this really makes sense to be defined in a profile and not only in the tool_requires, because it is a critical definition, not depending only of having a tool_requires. So that would probably solve the issues?

[tttapa]

this is only an issue for the conf_info defined in the tool_requires recipes.

Sure, but the fact that Conan silently yields inconsistent values for conf_info in different steps of the build/installation seems like a problem to me. Is it feasible for Conan to detect and report this as an error/warning somehow?

Specifically, I think that the following points together create problems:

• There are no restrictions or guidelines regarding which conf_info values can be changed by tools.
• There are no restrictions or guidelines regarding which conf_info values can be used by recipes, and in which functions they're used.
• Conan does not verify that the conf_info values are consistent between different steps in the build process.

This results in tools and packages that may work in isolation, but certain combinations of tools and other packages may silently generate invalid packages (binaries compiled with the wrong flags, incorrect system_libs, or build failures). IMHO, Conan should fail loudly if it detects such a case.

In general, it seems that if conf_info.xyz is allowed to be modified by a tool's package_info() method, then any use of conf_info.xyz outside of the generate(), build() and package() methods of other packages should result in an error, and vice versa. Is this correct?

---

So that would probably solve the issues?

Setting cross_build in the profile is indeed a workaround for most of the issues I listed.
However, getting back to the original question about OpenBLAS: this doesn't resolve the use of compiler_executables in the OpenBLAS recipe (because only the tool package knows the name and path of the compiler, so I cannot hard-code it in the profile).

It appears like the only solution is to introduce an explicit with_fortran option in the OpenBLAS recipe. However, it's not clear to me how this could be done without breaking backward compatibility:

• Setting with_fortran=false by default breaks things for people relying on the Fortran components.
• Setting with_fortran=true by default causes build failures if no Fortran compiler is available or if libgfortran is missing.
• Checking for a Fortran compiler in config_options() is not possible for the reasons discussed above.

What would you recommend?

[memsharded]

Sure, but the fact that Conan silently yields inconsistent values for conf_info in different steps of the build/installation seems like a problem to me. Is it feasible for Conan to detect and report this as an error/warning somehow?

Well, almost everything is possible in SW engineering. But this is typically a trade-off, and in general Conan doesn't follow a very "defensive" programming approach, it doesn't protect against every potential user error, that adds usually quite some overhead in terms of development, testing and maintenance, so unless those are very common errors, Conan will not protect against them.

This error in particular is almost impossible to detect, because the scope of tool-requires is a bit more limited, but when the consumer reads a certain self.conf.get(), it is not possible to know where that information is supposed to come. Raising an error if the conf.get() is done in certain methods might easily be breaking for users that define that info in the profile files and read in some methods, so this sounds to be risky for the value, even if it doesn't sound great that they use it in other methods, forbidding that will break them and they will complain saying that the previous version worked nicely.

For the possible changes to OpenBlas, it would be necessary to talk with the team there, I'll mark this as look-into to see if I can discuss it with the team next week.

Thanks for the feedback!

[memsharded]

I have been talking with the team regarding this.

It seems this is a recipe limitation that do not properly model the fortran usage.
It would be necessary to introduce to the recipe a with_fortran: [True, False] new option, because the final binary is definitely different if fortran is used or not, and this cannot depend exclusively on configuration items, because everything that affects the binaries should be part of the binary model (mostly settings+options).

I am creating a new ticket in conan-center-index to request this change, and also a ticket in the docs to try to improve it and explain more explicitly the scope of the conf_info from tool-requires, then closing this tickets as resolved (not-a-bug).

Thanks again for your feedback!