CMake: From USCRPL/mbed-cmake to Mbed OS first-party CMake

[ladislas]

Description of defect

Intro

As mentioned in #13974, I'm in the process of moving our current project from USCRPL/mbed-cmake to mbed os first-party CMake.

I thought it might be interesting for others to share our progress, our questions, feedback and issues we might be facing. I think that github is better than the forum for this as it is easier to share code, other repos and reference issues and PR

Our project is quite complex, with a main src directory containing our product firmware (main program). We have different drivers and libraries that are being used by the firmware.

We also have a lot of spikes: standalone projects (with their own main.cpp) to test components, features, libs, technical solutions, etc.

The following repository is a simpler example of our big project, and I'll use it to test all the features.

https://github.com/ladislas/mbed-cmake-template

How it works now

USCRPL/mbed-cmake allows us to have the spikes and the main program live in the same repository and compile together without any issues. We then use openocd to flash the .bin we want to use in our product.

USCRPL/mbed-cmake first creates a mbed-os-static STATIC library (https://github.com/USCRPL/mbed-cmake/blob/c0b0f7d4080bba179b9390e877eb0c1e7467f6d0/cmake/MbedOS-GCCArm.cmake#L35-L37) which is then wrapped into an mbed-os INTERFACE that defined the linker script and link options (https://github.com/USCRPL/mbed-cmake/blob/c0b0f7d4080bba179b9390e877eb0c1e7467f6d0/cmake/MbedOS-GCCArm.cmake#L50-L58). It's this mbed-os INTERFACE that our main program, spikes drivers and libs link with.

From a user perspective, mbed-os-static is compiled once for the main program and for all the spikes, drivers and libs. Adding a new spike with just a main.cpp file, will only increment the whole compilation steps number by one.

First try with Mbed OS first-party CMake

On the other hand, mbed os first-party cmake "recompiles" all the needed sources for each target depending on it. Adding a new spike linking with mbed-os will throw in the compilation a few hundred more "steps". Looking at the output, one can clearly see that mbed os or target files are compiled multiple times for the libs, the spikes and the main program.

Issues

And then it stops as the way it handles the linker script is not made for multiple add_executable targets.

Another issue was that adding another executable and calling mbed_set_mbed_target_linker_script resulted in the following error about .compile_time_defs.txt:

CMake Error: Files to be generated by multiple different commands: "/Users/ladislas/dev/ladislas/mbed-cmake-template/build/compile_time_defs.txt"

The issue comes from here:

mbed-os/tools/cmake/toolchain.cmake

Lines 22 to 23 in 33a7e66

	file(GENERATE OUTPUT "${CMAKE_BINARY_DIR}/compile_time_defs.txt" CONTENT "${_compile_definitions}\n")
	set(${definitions_file} @${CMAKE_BINARY_DIR}/compile_time_defs.txt PARENT_SCOPE)

and is easily fixed by using:

-    file(GENERATE OUTPUT "${CMAKE_BINARY_DIR}/compile_time_defs.txt" CONTENT "${_compile_definitions}\n")
-    set(${definitions_file} @${CMAKE_BINARY_DIR}/compile_time_defs.txt PARENT_SCOPE)
+    file(GENERATE OUTPUT "${CMAKE_BINARY_DIR}/${target}.compile_time_defs.txt" CONTENT "${_compile_definitions}\n")
+    set(${definitions_file} @${CMAKE_BINARY_DIR}/${target}.compile_time_defs.txt PARENT_SCOPE)

Note that the two files generated are identical. If they can never be different, it would be better to juste generate one and use it for all executable.

Questions

Please note I'm neither a CMake expert nor a .ld expert.

From my understanding: for the same mbed target (DISCO_F769NI in our case), the linker script is always the same. If we change target, we must change the linker script. In real life projects this happens seldom as your board/target is usually the same for a long period of time. But even if we need to change it, a new run of mbed-tools configure will provide everything needed.

Currently mbed-os, mbed-core, mbed-rtos are defined as INTERFACE libraries, meaning they are not compiled on their own (as static or object libraries would be), but become part of the target (other library or executable) that links with it.

So calling mbed_set_mbed_target_linker_script for each executable will also call mbed_set_toolchain_options multiple times.

My first question is:

What is the reason for using INTERFACE for the whole mbed-os? Why not use a static library wrapped in an interface?

And secondly:

Is supporting a project structure such as mine with multiple executable targets on the roadmap?

I'd be happy to help in this case.

Thanks for reading till here and also thanks for bringing CMake to mbed-os! :)

(cc @0xc0170 @hugueskamba @rajkan01 @multiplemonomials @ProExpertProg)

Target(s) affected by this defect ?

DISCO_F769NI

Toolchain(s) (name and version) displaying this defect ?

arm-none-eabi-gcc --version
arm-none-eabi-gcc (GNU Arm Embedded Toolchain 9-2020-q2-update) 9.3.1 20200408 (release)

What version of Mbed-os are you using (tag or sha) ?

33a7e66

What version(s) of tools are you using. List all that apply (E.g. mbed-cli)

mbed --version
1.10.4

mbed-tools --version
3.5.1.dev36

How is this defect reproduced ?

to do

[multiplemonomials]

I haven't had a chance to poke around the official CMake scripts that much, but I do agree that building the linker script multiple times seems like the wrong way to do it since it doesn't change. It would be better to just generate it once and then attach it as part of a target's link flags (how mbed-cmake does it).

Also, it definitely doesn't seem right if the entire source code is getting built once for each target. My understanding is that the mbed source code is supposed to get built as an object library, then the objects are supposed to get attached using TARGET_SOURCES to an interface library. But it sounds like maybe the source files themselves got added as the interface sources, rather than the object files.

[ciarmcom]

@ladislas thank you for raising this issue.Please take a look at the following comments:

We cannot automatically identify a release based on the version of Mbed OS that you have provided.
Please provide either a single valid sha of the form #abcde12 or #3b8265d70af32261311a06e423ca33434d8d80de
or a single valid release tag of the form mbed-os-x.y.z .
E.g. 'master' is not a unique single snapshot.
NOTE: If there are fields which are not applicable then please just add 'n/a' or 'None'.This indicates to us that at least all the fields have been considered.
Please update the issue header with the missing information, the issue will not be mirroredto our internal defect tracking system or investigated until this has been fully resolved.

[ciarmcom]

Thank you for raising this detailed GitHub issue. I am now notifying our internal issue triagers.
Internal Jira reference: https://jira.arm.com/browse/IOTOSM-2888

[ladislas]

My understanding is that the mbed source code is supposed to get built as an object library, then the objects are supposed to get attached using TARGET_SOURCES to an interface library.

@multiplemonomials it was since e7c0d93 that changed it from OBJECT to INTERFACE.

Would OBJECT be better than STATIC libraries?

[multiplemonomials]

Yeah that commit is... not right, or at least it's not correct to the design we originally planned (I worked with Martin and Hugues to plan out how the object and interface libraries would be used). The interface libraries do need to be there, but there also need to be object libraries as well. The object libraries would be used to compile the sources, then the compiled objects from those libraries should get added via target_sources to the interface libraries.

[0xc0170]

Thanks for the feedback, we will review and respond to all issues/questions.

[0xc0170]

@ladislas can we convert example blinky to be able to reproduce the issue with spikes? I've checked your template repository, it does not reference mbedtools, so I might need an assistance to reproduce it there (how to).

Is supporting a project structure such as mine with multiple executable targets on the roadmap?

I would say it should work as CMake libraries should be reusable. We are currently looking at testing and it might be the same use case.

1. 2 or more applications
   2 spikes as you call, just 2 different applications using the same mbed-os library ? I've created a branch with 2 main object libraries to test with https://github.com/0xc0170/mbed-os/tree/cmake-fix-objects - can you test ?

2. linker issue
   we will review if we can fix the linker not to be dependent on application target (it's related to the first one so we might want to first fix that and then get to this one).

We initially worked only on object libraries but we faced cmake issues with object libraries (we should be able to find PRs/commits) and converted to interface libraries. We exposed the main libraries mbed-os and mbed-baremetal as interface as well - it has some drawbacks as mentioned here, we should be able to address them. The branch I shared above was a concept we had previously.

From my understanding: for the same mbed target (DISCO_F769NI in our case), the linker script is always the same. If we change target, we must change the linker script. In real life projects this happens seldom as your board/target is usually the same for a long period of time. But even if we need to change it, a new run of mbed-tools configure will provide everything needed.

Consider ld file being a template that is changed based on app needs (sections could be adjusted, an example is bootloader). Therefore we preprocess linker scripts. If you change an app, linker script must be regenerated - reusing the same template. We always wanted to have one .ld template and fill it in.
Also for linker script file, if you can provide us a repo where we can reproduce, we will review in more detail.

[ladislas]

can we convert example blinky to be able to reproduce the issue with spikes? I've checked your template repository, it does not reference mbedtools, so I might need an assistance to reproduce it there (how to).

@0xc0170 yes, I'll work on a PR! I'll also push to my template on a specific branch for testing.

I'll also test your branch and let you know.

And thanks for the explanation on the linker file. 👍

[multiplemonomials]

Okay, I've got time to take a look into what's going on here.
I think I found the source of some of these errors, but before that, I got distracted by a giant, horrible, flashing red flag:

Detecting C Compiler ABI -- Failed

This is the same issue that we talked about at our meeting, and it seems that it still hasn't been fixed. Basically, the compiler flags that CMake is using to do all of its internal tests are extremely messed up, and you guys just papered over the issue with:

set(CMAKE_C_COMPILER_WORKS TRUE)
set(CMAKE_CXX_COMPILER_WORKS TRUE)

This is a really bad idea, because it means that all sorts of things will be subtly screwed up later on, since CMake cannot successfully run compilation or linking tests.
The real way to fix it is to add the needed compiler flags to CMake's global flag variables, like I explained. So, I made a PR to fix this: #13987 . Please take a look when you have a chance!