gnuarmeclipse: cxxsupport failures for few targets on master

[0xc0170]

Description

The failures started on Friday, the first job failing is https://mbed-os.mbedcloudtesting.com/job/mbed-os-ci_exporter-gnuarmeclipse/2550 (internal link).

This are the errors:

15:41:44  [2019-08-12T14:41:44.702Z] + tail -n 100 exporter_build_log_NRF52840_DK_gnuarmeclipse.log
15:41:44  [2019-08-12T14:41:44.702Z]           ^~~~~~~~~~~
15:41:44  [2019-08-12T14:41:44.702Z] /builds/ws/mbed-os-ci_exporter-gnuarmeclipse@3/examples/mbed-os-example-blinky/mbed-os/platform/cxxsupport/mstd_type_traits:1240:21: error: expected initializer before '<' token
15:41:44  [2019-08-12T14:41:44.702Z]   -> std::enable_if_t<std::is_function<F>::value &&
15:41:44  [2019-08-12T14:41:44.702Z]                      ^
15:41:44  [2019-08-12T14:41:44.702Z] mbed-os/rtos/TARGET_CORTEX/subdir.mk:38: recipe for target 'mbed-os/rtos/TARGET_CORTEX/mbed_rtx_idle.o' failed
15:41:44  [2019-08-12T14:41:44.702Z] /builds/ws/mbed-os-ci_exporter-gnuarmeclipse@3/examples/mbed-os-example-blinky/mbed-os/platform/cxxsupport/mstd_type_traits:1250:10: error: 'enable_if_t' in namespace 'std' does not name a template type
15:41:44  [2019-08-12T14:41:44.702Z]   -> std::enable_if_t<std::is_function<F>::value &&
15:41:44  [2019-08-12T14:41:44.702Z]           ^~~~~~~~~~~
15:41:44  [2019-08-12T14:41:44.702Z] /builds/ws/mbed-os-ci_exporter-gnuarmeclipse@3/examples/mbed-os-example-blinky/mbed-os/platform/cxxsupport/mstd_type_traits:1250:21: error: expected initializer before '<' token
15:41:44  [2019-08-12T14:41:44.702Z]   -> std::enable_if_t<std::is_function<F>::value &&
15:41:44  [2019-08-12T14:41:44.702Z]                      ^
15:41:44  [2019-08-12T14:41:44.702Z] /builds/ws/mbed-os-ci_exporter-gnuarmeclipse@3/examples/mbed-os-example-blinky/mbed-os/platform/cxxsupport/mstd_type_traits:1260:10: error: 'enable_if_t' in namespace 'std' does not name a template type
15:41:44  [2019-08-12T14:41:44.702Z]   -> std::enable_if_t<std::is_function<F>::value &&
15:41:44  [2019-08-12T14:41:44.702Z]           ^~~~~~~~~~~
15:41:44  [2019-08-12T14:41:44.702Z] /builds/ws/mbed-os-ci_exporter-gnuarmeclipse@3/examples/mbed-os-example-blinky/mbed-os/platform/cxxsupport/mstd_type_traits:1260:21: error: expected initializer before '<' token
15:41:44  [2019-08-12T14:41:44.702Z]   -> std::enable_if_t<std::is_function<F>::value &&
15:41:44  [2019-08-12T14:41:44.702Z]                      ^
15:41:44  [2019-08-12T14:41:44.702Z] /builds/ws/mbed-os-ci_exporter-gnuarmeclipse@3/examples/mbed-os-example-blinky/mbed-os/platform/cxxsupport/mstd_type_traits:1271:10: error: 'enable_if_t' in namespace 'std' does not name a template type
15:41:44  [2019-08-12T14:41:44.702Z]   -> std::enable_if_t<!std::is_function<F>::value &&
15:41:44  [2019-08-12T14:41:44.702Z]           ^~~~~~~~~~~
15:41:44  [2019-08-12T14:41:44.702Z] /builds/ws/mbed-os-ci_exporter-gnuarmeclipse@3/examples/mbed-os-example-blinky/mbed-os/platform/cxxsupport/mstd_type_traits:1271:21: error: expected initializer before '<' token
15:41:44  [2019-08-12T14:41:44.702Z]   -> std::enable_if_t<!std::is_function<F>::value &&
15:41:44  [2019-08-12T14:41:44.702Z]                      ^
15:41:44  [2019-08-12T14:41:44.702Z] /builds/ws/mbed-os-ci_exporter-gnuarmeclipse@3/examples/mbed-os-example-blinky/mbed-os/platform/cxxsupport/mstd_type_traits:1281:10: error: 'enable_if_t' in namespace 'std' does not name a template type
15:41:44  [2019-08-12T14:41:44.702Z]   -> std::enable_if_t<!std::is_function<F>::value &&
15:41:44  [2019-08-12T14:41:44.702Z]           ^~~~~~~~~~~
15:41:44  [2019-08-12T14:41:44.702Z] /builds/ws/mbed-os-ci_exporter-gnuarmeclipse@3/examples/mbed-os-example-blinky/mbed-os/platform/cxxsupport/mstd_type_traits:1281:21: error: expected initializer before '<' token
15:41:44  [2019-08-12T14:41:44.702Z]   -> std::enable_if_t<!std::is_function<F>::value &&
15:41:44  [2019-08-12T14:41:44.702Z]                      ^
15:41:44  [2019-08-12T14:41:44.702Z] /builds/ws/mbed-os-ci_exporter-gnuarmeclipse@3/examples/mbed-os-example-blinky/mbed-os/platform/cxxsupport/mstd_type_traits:1291:10: error: 'enable_if_t' in namespace 'std' does not name a template type
15:41:44  [2019-08-12T14:41:44.702Z]   -> std::enable_if_t<!std::is_function<F>::value &&
15:41:44  [2019-08-12T14:41:44.702Z]           ^~~~~~~~~~~
15:41:44  [2019-08-12T14:41:44.702Z] /builds/ws/mbed-os-ci_exporter-gnuarmeclipse@3/examples/mbed-os-example-blinky/mbed-os/platform/cxxsupport/mstd_type_traits:1291:21: error: expected initializer before '<' token
15:41:44  [2019-08-12T14:41:44.702Z]   -> std::enable_if_t<!std::is_function<F>::value &&
15:41:44  [2019-08-12T14:41:44.702Z]                      ^
15:41:44  [2019-08-12T14:41:44.702Z] /builds/ws/mbed-os-ci_exporter-gnuarmeclipse@3/examples/mbed-os-example-blinky/mbed-os/platform/cxxsupport/mstd_type_traits:1302:10: error: 'enable_if_t' in namespace 'std' does not name a template type
15:41:44  [2019-08-12T14:41:44.702Z]   -> std::enable_if_t<!std::is_member_pointer<std::decay_t<F>>::value ||
15:41:44  [2019-08-12T14:41:44.702Z]           ^~~~~~~~~~~
15:41:44  [2019-08-12T14:41:44.702Z] /builds/ws/mbed-os-ci_exporter-gnuarmeclipse@3/examples/mbed-os-example-blinky/mbed-os/platform/cxxsupport/mstd_type_traits:1302:21: error: expected initializer before '<' token
15:41:44  [2019-08-12T14:41:44.702Z]   -> std::enable_if_t<!std::is_member_pointer<std::decay_t<F>>::value ||
15:41:44  [2019-08-12T14:41:44.702Z]                      ^
15:41:44  [2019-08-12T14:41:44.702Z] make: *** [mbed-os/rtos/TARGET_CORTEX/mbed_rtx_idle.o] Error 1
15:41:44  [2019-08-12T14:41:44.702Z] make: *** [mbed-os/platform/source/DeepSleepLock.o] Error 1
15:41:44  [2019-08-12T14:41:44.702Z] mbed-os/platform/source/subdir.mk:115: recipe for target 'mbed-os/platform/source/DeepSleepLock.o' failed
15:41:44  [2019-08-12T14:41:44.702Z] mbed-os/platform/source/subdir.mk:115: recipe for target 'mbed-os/platform/source/FileBase.o' failed
15:41:44  [2019-08-12T14:41:44.702Z] make: *** [mbed-os/platform/source/FileBase.o] Error 1

Public link: #11021 (comment), contains build logs. It has been failing often and affecting currently master PRs with rate higher than 50 percent of exporters run.

Strange fact, only two targets failing: NRF52840_DK and NUCLEO_L073RZ.

It started 9th of August, there were few merges that day: https://github.com/ARMmbed/mbed-os/commits/master. The only generic change is #11051 - this changed gnuarmeclipse files so might be the cause.

cc @evedon @ARMmbed/mbed-os-test @kjbracey-arm @ARMmbed/mbed-os-maintainers

Issue request type

[ ] Question
[ ] Enhancement
[X] Bug

[0xc0170]

This commit might be the offender: evedon@050a338. If not, looks like something else had to change on Friday

@evedon Please review

[kjbracey]

There is a change to tools/export/gnuarmeclipse/.cproject.tmpl related to the language standard there that I don't immediately understand.

It's possible that it's failing to write the standard to the XML file, so defaulting to something lower than C++14. The errors generated suggest it's in pre-C++14 mode. (std::enable_if_t appears in C++14).

[kjbracey]

Is this perhaps related to the new "merging two profile files" trick being applied? Is that combination maybe not fully working (for export only?), so info is lost?

[0xc0170]

I add @ARMmbed/mbed-os-tools team as well as this is most likely connected to profiles and the change above

I also don't understand the change in that PR (what is fixing and how it broke). Lets wait for @ARMmbed/mbed-os-core team review and fix.

[evedon]

The change to tools/export/gnuarmeclipse/.cproject.tmpl was made by @madchutney to modify the template to work with partial profile see 3010aaa

[kjbracey]

Don't see the logic in that change - why is compiler.std being singled out? Surely all the other settings next to it using the same basic form would have the same problem if this was a generic "run template on profile fragment" issue as described. Aren't all the nearby settings also missing in the fragment?

Maybe this change is just covering over the fact that compiler.std specifically has gone missing.

I wonder if all the tests should actually be if opts['cpp']['foo'] is defined - maybe they're all broken, and would fail their entry was missing.

[kjbracey]

So my suspicion is that somehow the profile combination process is losing compiler.std, that problem was only revealed by the faulty "if defined" checks in this exporter, and the exporter change is covering it up for this specific entry. The real problem is elsewhere. (As of 5.13.0, not setting the compiler to C++14 is a non-fatal mistake, but current master does need C++14 to be selected).

I'd suggest reverting that change, and digging into whether/why the setting is missing on a complete build.

[madchutney]

The change I authored when helping @evedon was workaround the addition of a partial profile, which was causing CI to fail. A partial profile is apparently something that has been supported but not used internally and as such is not handled by CI.

I'm happy to see the change reverted if necessary and a proper implementation of partial profiles included in Mbed CLI and CI system. This was what I recommended at the time but there were insufficient time and resources to complete the task without accruing more technical debt.

A PR to tackle this technical debt has already been proposed for the next PI.

[kjbracey]

I'm not understanding what the original issue was, and I'm wondering if it was misdiagnosed. What exactly is the CI trying to do with the profile? Is it doing a "for-each" over the "profiles" directory?

Is this enable_if_t failure we see now arising when attempting a compile with "just minimal-printf" profile? If so, we'll need to stop it trying to do that. If we can't fix the tools, do we need to do something to get that profile fragment out of the way? Put it in a subdirectory?

[evedon]

I'm not understanding what the original issue was, and I'm wondering if it was misdiagnosed. What exactly is the CI trying to do with the profile? Is it doing a "for-each" over the "profiles" directory?

Is this enable_if_t failure we see now arising when attempting a compile with "just minimal-printf" profile? If so, we'll need to stop it trying to do that. If we can't fix the tools, do we need to do something to get that profile fragment out of the way? Put it in a subdirectory?

The exporter CI issue arised when attempting a compile with "just minimal-printf" profile. We discussed at the time whether to put the partial profile in a subdirectory or fix CI. If we put the profile in a subdirectory then the command line would become much longer:

mbed compile -m K64F -t GCC_ARM --profile release --profile mbed-os/tools/profiles/extensions/minimal-printf.json

So we decided that it would be better to fix CI, hence the change proposed by @madchutney. I think we need to understand why the change is affecting two targets NRF52840_DK and NUCLEO_L073RZ specifically.
I am happy to revert the change, but this has the impact described above.