Broken Python 2.7.11 support in Mbed OS 5.8.1 for ARMCC

[screamerbg]

Description

• Type: Bug
• Priority: Blocker

---

Bug

Target
NUCLEO_F429ZI

Toolchain:
ARM

Python version:

$ python --version
Python 2.7.11

mbed-cli version:
1.5.0

mbed-os sha:
addec7b Merge pull request #6461 from ARMmbed/release-candidate

DAPLink version:
STLink

Expected behavior
mbed test ... works

Actual behavior
Getting python error:

D:\Work\sdk\ci-test-shield\mbed-os>mbed test -m detect --app-config mbed_app.json -n "tests-*"
[mbed] Detected "NUCLEO_F429ZI" connected to "E:" and using com port "COM6"
Traceback (most recent call last):
  File "D:\Work\sdk\ci-test-shield\mbed-os\tools\test.py", line 31, in <module>
    from tools.test_api import test_path_to_name, find_tests, get_test_config, print_tests, build_tests, test_spec_from_test_builds
  File "D:\Work\sdk\ci-test-shield\mbed-os\tools\test_api.py", line 59, in <module>
    from tools.build_api import build_project, build_mbed_libs, build_lib
  File "D:\Work\sdk\ci-test-shield\mbed-os\tools\build_api.py", line 47, in <module>
    from .toolchains import TOOLCHAIN_CLASSES
  File "D:\Work\sdk\ci-test-shield\mbed-os\tools\toolchains\__init__.py", line 1611, in <module>
    from tools.toolchains.arm import ARM_STD, ARM_MICRO, ARMC6
  File "D:\Work\sdk\ci-test-shield\mbed-os\tools\toolchains\arm.py", line 18, in <module>
    from builtins import str
ImportError: No module named builtins
[mbed] ERROR: "C:\Progra~1\Python\python.exe" returned error code 1.
[mbed] ERROR: Command "C:\Progra~1\Python\python.exe -u D:\Work\sdk\ci-test-shield\mbed-os\tools\test.py -t ARM -m NUCLEO_F429ZI --source D:\Work\sdk\ci-test-shield --build D:\Work\sdk\ci-test-shield\BUILD\tests\NUCLEO_F429ZI\ARM --test-spec D:\Work\sdk\ci-test-shield\BUILD\tests\NUCLEO_F429ZI\ARM\test_spec.json -n tests-* --app-config mbed_app.json" in "D:\Work\sdk\ci-test-shield"
---

Steps to reproduce

$ mbed import ci-test-shield
$ cd ci-test-shield
$ cd mbed-os
$ mbed update mbed-os-5.8.1
$ mbed test -t ARM -m K64F --app-config mbed_app.json -n "tests-*"

If you change to mbed-os-5.7.7 then everything's fine.

Official documentation states Python version 2.7.11

cc @theotherjimmy

[cmonr]

@screamerbg Can you post your output for

> pip list
> pip2 list

What happens if you try the tests inside of a python virtual env?

Have you looked at running pip install -r requirements.txt yet?

[cmonr]

Also, maybe this was just me, but doing a mbed update 5.8.0 (or 5.8.1) in that sequence of steps causes build failures, whereas if I were to only checkout mbed-os and run the same test command, evreything compiles fine.

[screamerbg]

D:\Work\sdk\ci-test-shield\mbed-os>mbed update mbed-os-5.8.1
[mbed] Updating library "mbed-os" to branch/tag "mbed-os-5.8.1"
Enter passphrase for key '/.ssh/id_rsa':
[mbed] Updating reference "mbed-os" -> "https://github.com/ARMmbed/mbed-os/#addec7ba10054be03849eff58a1d17f157391e7d"

D:\Work\sdk\ci-test-shield\mbed-os>pip install -r requirements.txt
Requirement already satisfied (use --upgrade to upgrade): colorama>=0.3.3 in c:\progra~1\python\lib\site-packages (from -r requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): PySerial>=2.7 in c:\progra~1\python\lib\site-packages (from -r requirements.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): PrettyTable>=0.7.2 in c:\progra~1\python\lib\site-packages (from -r requirements.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): Jinja2>=2.7.3 in c:\progra~1\python\lib\site-packages (from -r requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): IntelHex>=1.3 in c:\progra~1\python\lib\site-packages (from -r requirements.txt (line 5))
Requirement already satisfied (use --upgrade to upgrade): junit-xml in c:\progra~1\python\lib\site-packages (from -r requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): pyYAML in c:\progra~1\python\lib\site-packages (from -r requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): requests in c:\progra~1\python\lib\site-packages (from -r requirements.txt (line 8))
Requirement already satisfied (use --upgrade to upgrade): mbed-ls>=0.2.13 in c:\progra~1\python\lib\site-packages (from -r requirements.txt (line 9))
Requirement already satisfied (use --upgrade to upgrade): mbed-host-tests>=1.1.2 in c:\progra~1\python\lib\site-packages (from -r requirements.txt (line 10))
Requirement already satisfied (use --upgrade to upgrade): mbed-greentea>=0.2.24 in c:\progra~1\python\lib\site-packages (from -r requirements.txt (line 11))
Requirement already satisfied (use --upgrade to upgrade): beautifulsoup4>=4 in c:\progra~1\python\lib\site-packages (from -r requirements.txt (line 12))
Requirement already satisfied (use --upgrade to upgrade): fuzzywuzzy>=0.11 in c:\progra~1\python\lib\site-packages (from -r requirements.txt (line 13))
Requirement already satisfied (use --upgrade to upgrade): pyelftools>=0.24 in c:\progra~1\python\lib\site-packages (from -r requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jsonschema>=2.6 in c:\progra~1\python\lib\site-packages (from -r requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): MarkupSafe in c:\progra~1\python\lib\site-packages (from Jinja2>=2.7.3->-r requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): six in c:\progra~1\python\lib\site-packages (from junit-xml->-r requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): fasteners in c:\progra~1\python\lib\site-packages (from mbed-ls>=0.2.13->-r requirements.txt (line 9))
Requirement already satisfied (use --upgrade to upgrade): appdirs>=1.4 in c:\progra~1\python\lib\site-packages (from mbed-ls>=0.2.13->-r requirements.txt (line 9))
Requirement already satisfied (use --upgrade to upgrade): lockfile in c:\progra~1\python\lib\site-packages (from mbed-greentea>=0.2.24->-r requirements.txt (line 11))
Requirement already satisfied (use --upgrade to upgrade): mock in c:\progra~1\python\lib\site-packages (from mbed-greentea>=0.2.24->-r requirements.txt (line 11))
Requirement already satisfied (use --upgrade to upgrade): functools32 in c:\progra~1\python\lib\site-packages (from jsonschema>=2.6->-r requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): monotonic>=0.1 in c:\progra~1\python\lib\site-packages (from fasteners->mbed-ls>=0.2.13->-r requirements.txt (line 9))
Requirement already satisfied (use --upgrade to upgrade): funcsigs>=1 in c:\progra~1\python\lib\site-packages (from mock->mbed-greentea>=0.2.24->-r requirements.txt (line 11))
Requirement already satisfied (use --upgrade to upgrade): pbr>=0.11 in c:\progra~1\python\lib\site-packages (from mock->mbed-greentea>=0.2.24->-r requirements.txt (line 11))
You are using pip version 7.1.2, however version 9.0.3 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

D:\Work\sdk\ci-test-shield\mbed-os>mbed test -t ARM -m K64F --app-config mbed_app.json -n "tests-api-spi*" -c
Traceback (most recent call last):
  File "D:\Work\sdk\ci-test-shield\mbed-os\tools\test.py", line 31, in <module>
    from tools.test_api import test_path_to_name, find_tests, get_test_config, print_tests, build_tests, test_spec_from_test_builds
  File "D:\Work\sdk\ci-test-shield\mbed-os\tools\test_api.py", line 59, in <module>
    from tools.build_api import build_project, build_mbed_libs, build_lib
  File "D:\Work\sdk\ci-test-shield\mbed-os\tools\build_api.py", line 47, in <module>
    from .toolchains import TOOLCHAIN_CLASSES
  File "D:\Work\sdk\ci-test-shield\mbed-os\tools\toolchains\__init__.py", line 1611, in <module>
    from tools.toolchains.arm import ARM_STD, ARM_MICRO, ARMC6
  File "D:\Work\sdk\ci-test-shield\mbed-os\tools\toolchains\arm.py", line 18, in <module>
    from builtins import str
ImportError: No module named builtins
[mbed] ERROR: "c:\progra~1\python\python.exe" returned error code 1.
[mbed] ERROR: Command "c:\progra~1\python\python.exe -u D:\Work\sdk\ci-test-shield\mbed-os\tools\test.py -t ARM -m K64F -c --source D:\Work\sdk\ci-test-shield --build D:\Work\sdk\ci-test-shield\BUILD\tests\K64F\ARM --test-spec D:\Work\sdk\ci-test-shield\BUILD\tests\K64F\ARM\test_spec.json -n tests-api-spi* --app-config mbed_app.json" in "D:\Work\sdk\ci-test-shield"
---

[screamerbg]

@cmonr Did you try to reproduce it with ARMCC and mbed-os-5.8.1? e.g. mbed test -t ARM -m K64F --app-config mbed_app.json -n "tests-api-spi*"

[cmonr]

Interesting. I only tried GCC_ARM. I can give ARM a shot tomorrow.

I did try 5.8.1, which had the same results as 5.8.0.

[0xc0170]

I have installed locally future module that should resolve this. If I remove it, I can see the same error as reported here.

Once uninstalled, I run pip install -r mbed-os\requirements.txt (mbed-os is 5.8.1 tag addec7ba1 - (HEAD, tag: mbed-os-5.8.1, upstream/mbed-os-5.8) Merge pull request #6461 from ARMmbed/release-candidate (16 hours ago)).

Requirement already satisfied: colorama>=0.3.3 in c:\python27\lib\site-packages (from -r mbed-os\requirements.txt (line 1))
Requirement already satisfied: PySerial>=2.7 in c:\python27\lib\site-packages (from -r mbed-os\requirements.txt (line 2))
Requirement already satisfied: PrettyTable>=0.7.2 in c:\python27\lib\site-packages (from -r mbed-os\requirements.txt (line 3))
Requirement already satisfied: Jinja2>=2.7.3 in c:\python27\lib\site-packages (from -r mbed-os\requirements.txt (line 4))
Requirement already satisfied: IntelHex>=1.3 in c:\python27\lib\site-packages (from -r mbed-os\requirements.txt (line 5))
Requirement already satisfied: junit-xml in c:\python27\lib\site-packages (from -r mbed-os\requirements.txt (line 6))
Requirement already satisfied: pyYAML in c:\python27\lib\site-packages (from -r mbed-os\requirements.txt (line 7))
Requirement already satisfied: requests in c:\python27\lib\site-packages (from -r mbed-os\requirements.txt (line 8))
Requirement already satisfied: mbed-ls>=0.2.13 in c:\python27\lib\site-packages (from -r mbed-os\requirements.txt (line 9))
Requirement already satisfied: mbed-host-tests>=1.1.2 in c:\python27\lib\site-packages (from -r mbed-os\requirements.txt (line 10))
Requirement already satisfied: mbed-greentea>=0.2.24 in c:\python27\lib\site-packages (from -r mbed-os\requirements.txt (line 11))
Requirement already satisfied: beautifulsoup4>=4 in c:\python27\lib\site-packages (from -r mbed-os\requirements.txt (line 12))
Requirement already satisfied: fuzzywuzzy>=0.11 in c:\python27\lib\site-packages (from -r mbed-os\requirements.txt (line 13))
Requirement already satisfied: pyelftools>=0.24 in c:\python27\lib\site-packages (from -r mbed-os\requirements.txt (line 14))
Requirement already satisfied: jsonschema>=2.6 in c:\python27\lib\site-packages (from -r mbed-os\requirements.txt (line 15))
Requirement already satisfied: MarkupSafe>=0.23 in c:\python27\lib\site-packages (from Jinja2>=2.7.3->-r mbed-os\requirements.txt (line 4))
Requirement already satisfied: six in c:\python27\lib\site-packages (from junit-xml->-r mbed-os\requirements.txt (line 6))
Requirement already satisfied: urllib3<1.23,>=1.21.1 in c:\python27\lib\site-packages (from requests->-r mbed-os\requirements.txt (line 8))
Requirement already satisfied: idna<2.7,>=2.5 in c:\python27\lib\site-packages (from requests->-r mbed-os\requirements.txt (line 8))
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\python27\lib\site-packages (from requests->-r mbed-os\requirements.txt (line 8))
Requirement already satisfied: certifi>=2017.4.17 in c:\python27\lib\site-packages (from requests->-r mbed-os\requirements.txt (line 8))
Requirement already satisfied: fasteners in c:\python27\lib\site-packages (from mbed-ls>=0.2.13->-r mbed-os\requirements.txt (line 9))
Requirement already satisfied: appdirs>=1.4 in c:\python27\lib\site-packages (from mbed-ls>=0.2.13->-r mbed-os\requirements.txt (line 9))
Requirement already satisfied: pyOCD>=0.8.1a1 in c:\python27\lib\site-packages (from mbed-host-tests>=1.1.2->-r mbed-os\requirements.txt (line 10))
Requirement already satisfied: lockfile in c:\python27\lib\site-packages (from mbed-greentea>=0.2.24->-r mbed-os\requirements.txt (line 11))
Requirement already satisfied: mock in c:\python27\lib\site-packages (from mbed-greentea>=0.2.24->-r mbed-os\requirements.txt (line 11))
Requirement already satisfied: functools32; python_version == "2.7" in c:\python27\lib\site-packages (from jsonschema>=2.6->-r mbed-os\requirements.txt (line 15))
Requirement already satisfied: monotonic>=0.1 in c:\python27\lib\site-packages (from fasteners->mbed-ls>=0.2.13->-r mbed-os\requirements.txt (line 9))
Requirement already satisfied: enum34 in c:\python27\lib\site-packages (from pyOCD>=0.8.1a1->mbed-host-tests>=1.1.2->-r mbed-os\requirements.txt (line 10))
Collecting future (from pyOCD>=0.8.1a1->mbed-host-tests>=1.1.2->-r mbed-os\requirements.txt (line 10))
  Using cached future-0.16.0.tar.gz
Requirement already satisfied: websocket-client in c:\python27\lib\site-packages (from pyOCD>=0.8.1a1->mbed-host-tests>=1.1.2->-r mbed-os\requirements.txt (line 10))
Requirement already satisfied: pywinusb>=0.4.0 in c:\python27\lib\site-packages (from pyOCD>=0.8.1a1->mbed-host-tests>=1.1.2->-r mbed-os\requirements.txt (line 10))
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in c:\python27\lib\site-packages (from mock->mbed-greentea>=0.2.24->-r mbed-os\requirements.txt (line 11))
Requirement already satisfied: pbr>=0.11 in c:\python27\lib\site-packages (from mock->mbed-greentea>=0.2.24->-r mbed-os\requirements.txt (line 11))
Installing collected packages: future
  Running setup.py install for future ... done
Successfully installed future-0.16.0

SHA 14255ca introduced the lines that are reported to be breaking here.

Are we missing future module in requirements? Reference: https://pypi.python.org/pypi/future

[theotherjimmy]

@0xc0170 Good find. Yes, we are missing the future module.