Skip to content

Commit c503ded

Browse files
0xc01700xc0170
authored
Merge pull request #8212 from lorjala/unittests-docs
Unit tests: update README.md
2 parents e7cd6ae + e1f22e4 commit c503ded

File tree

1 file changed

+102
-204
lines changed

1 file changed

+102
-204
lines changed

‎UNITTESTS/README.md

Lines changed: 102 additions & 204 deletions
Original file line numberDiff line numberDiff line change
@@ -1,240 +1,107 @@
1-
# Unit testing Mbed OS
1+
## Unit testing
22

3-
This document describes how to run and write unit tests for Mbed OS.
3+
This document describes how to write and test unit tests for Mbed OS. To prevent and solve problems, please see the [troubleshooting](#troubleshooting) section.
44

5-
## Prerequisites
5+
### Introduction
66

7-
* GNU toolchains installed.
8-
* GCC 6 or later
9-
* MinGW-W64 GCC-6.4.0 or MinGW-W64 GCC-7.3.0 (Windows)
10-
* CMake 3.0+ installed.
11-
* Python 2.7.x or >3.5 and pip 10.0 (or newer) installed.
12-
* gcovr >=4.1
13-
* mbed-cli >= 1.80
7+
Unit tests test code in small sections on a host machine. Unlike other testing tools, unit testing doesn't require embedded hardware, and it doesn't need to build the full operating system. Because of this, unit testing can result in faster tests than other testing tools. Unit testing takes place in a build environment where we test each C or C++ class or module in isolation. This means we build test suites into separate test binaries and stub all access outside to remove dependencies on any specific embedded hardware or software combination. This allows us to complete the testing using native compilers on the build machine.
148

15-
### Installing dependencies on Debian/Ubuntu
9+
### Prerequisites
1610

17-
1. `sudo apt-get -y install build-essential cmake`
18-
2. Install python and pip:
19-
```
20-
sudo apt-get -y install python python-setuptools
21-
sudo easy_install pip
22-
```
23-
3. Install [gcovr](#installing-covr).
24-
4. (Optional) Install [Mbed CLI](https://os.mbed.com/docs/latest/tools/arm-mbed-cli.html).
25-
26-
### Installing dependencies on Mac OS
27-
28-
1. Install [Homebrew](https://brew.sh/).
29-
2. Install gcc compilers and cmake with: `brew install gcc cmake`
30-
3. Install python and pip:
31-
```
32-
brew install python
33-
sudo easy_install pip
34-
```
35-
3. Install [gcovr](#installing-covr).
36-
4. (Optional) Install [Mbed CLI](https://os.mbed.com/docs/latest/tools/arm-mbed-cli.html).
11+
Please install the following dependencies to use Mbed OS unit testing.
3712

38-
### Installing dependencies on Windows
39-
40-
1. Download and install [MinGW-W64](http://mingw-w64.org/).
41-
2. Download CMake binaries from https://cmake.org/download/ and run the installer.
42-
3. Download Python2.7 or Python3 from https://www.python.org/getit/ and run the installer.
43-
4. Add MinGW, CMake and Python into PATH.
44-
5. Install [gcovr](#installing-covr).
45-
6. (Optional) Install [Mbed CLI](https://os.mbed.com/docs/latest/tools/arm-mbed-cli.html).
46-
47-
### Installing covr
48-
49-
Install gcovr code coverage tool globally with `pip install "gcovr>=4.1"` or using virtualenv:
50-
51-
#### virtualenv
52-
53-
1. Install virtualenv if not installed with `pip install virtualenv`
54-
2. Install gcovr with:
55-
56-
**[Debian/Linux/Mac OS]**
57-
```
58-
virtualenv pyenv
59-
. pyenv/bin/activate
60-
pip install 'gcovr>=4.1'
61-
```
62-
**[Windows]**
63-
```
64-
virtualenv pyenv
65-
pyenv\\Scripts\\activate
66-
pip install "gcovr>=4.1"
67-
```
13+
- GNU toolchains.
14+
- GCC 6 or later. We recommend you use MinGW-W64 on Windows, but any Windows port of the above GCC versions works.
15+
- CMake 3.0 or newer.
16+
- Python 2.7.x, 3.5 or newer.
17+
- Pip 10.0 or newer.
18+
- Gcovr 4.1 or newer.
19+
- Mbed CLI 1.8.0 or newer.
6820

69-
## Building and running unit tests
70-
71-
> In case of running into problems see [troubleshooting](#troubleshooting) section.
72-
73-
`UNITTESTS/mbed_unittest.py` contains testing scripts for Mbed OS unit testing. Mbed CLI supports unit testing through `mbed test --unittests` command with the same arguments.
74-
75-
### Testing with Mbed CLI
76-
77-
```
78-
mbed test --unittests
79-
```
80-
81-
A subset of tests can be run by providing `-r` flag for the tool which runs tests matching a regular expression.
82-
83-
e.g. `mbed test --unittests --run -r features-netsocket`
84-
85-
### Build manually without Python tools
86-
87-
1. Create a build directory e.g. `mkdir UNITTESTS/build`.
88-
2. Move to the build directory `cd UNITTESTS/build`.
89-
3. Run CMake with `cmake [RELATIVE PATH TO UNITTESTS DIR] [OPTIONAL ARGUMENTS]` e.g. `cmake ..`:
90-
* Add `-g [generator]` argument if target other than Unix Makefiles e.g. MinGW `-g "MinGW Makefiles"`
91-
4. Run make program (make, gmake, mingw32-make, etc).
92-
93-
##### Run CMake and build Unix Makefiles (GNU make)
94-
95-
```
96-
cmake ..
97-
make
98-
```
99-
100-
##### Run CMake and build MinGW Makefiles (mingw32-make)
101-
102-
```
103-
cmake -G "MinGW Makefiles" ..
104-
mingw32-make
105-
```
21+
Detailed instructions for supported operating systems are below.
10622

107-
#### Custom CMake variables
23+
#### Installing dependencies on Debian or Ubuntu
10824

109-
Usage:
110-
`cmake [RELATIVE PATH TO UNITTESTS DIR] [OPTIONS]`
25+
1. `sudo apt-get -y install build-essential cmake`
26+
1. Install Python and Pip with:
11127

112-
Keyword variables (usage `cmake -D<VARIABLE>(:<TYPE>)=<value>`:
28+
```
29+
sudo apt-get -y install python python-setuptools
30+
sudo easy_install pip
31+
```
11332

114-
| Variable | Type | Accepted values | Description |
115-
| -------- | ---- | --------------- | ----------- |
116-
| COVERAGE | STRING | merged<br>separate | Generate merged or individual reports |
33+
1. Install Gcovr and [Mbed CLI](https://os.mbed.com/docs/latest/tools/arm-mbed-cli.html) with `pip install "gcovr>=4.1" mbed-cli`.
11734

118-
### Run in terminal
35+
#### Installing dependencies on macOS
11936

120-
Unit tests can be run separately from each executable or by using ctest test runner. Run ctest with make program using target test. Options can be passed to ctest using ARGS argument. See [ctest manual](https://cmake.org/cmake/help/v3.0/manual/ctest.1.html) for more information.
37+
1. Install [Homebrew](https://brew.sh/).
38+
1. Install GCC compilers and CMake with: `brew install gcc cmake`.
39+
1. Install Python and Pip:
12140

122-
Run ctest on test suite level:
123-
```
124-
{MAKE_PROGRAM} test -C [RELATIVE PATH TO BUILD DIRECTORY]
125-
```
126-
e.g. `make test -C UNITTESTS/build` or `mingw32-make test -C UNITTESTS/build`
41+
```
42+
brew install python
43+
sudo easy_install pip
44+
```
12745

128-
Run ctest verbose (show each test case):
129-
```
130-
{MAKE_PROGRAM} test -C UNITTESTS/build ARGS="-V"
131-
```
132-
133-
Run ctest dashboard test and create test results:
134-
```
135-
{MAKE_PROGRAM} test --C UNITTESTS/build ARGS="-D ExperimentalTest"
136-
```
46+
1. Install Gcovr and [Mbed CLI](https://os.mbed.com/docs/latest/tools/arm-mbed-cli.html) with `pip install "gcovr>=4.1" mbed-cli`.
13747

138-
### Run with GUI test runner
48+
#### Installing dependencies on Windows
13949

140-
1. Build and/or install *gtest-runner* using the documentation: https://github.com/nholthaus/gtest-runner
50+
1. Download and install [MinGW-W64](http://mingw-w64.org/).
51+
1. Download CMake binaries from https://cmake.org/download/, and run the installer.
52+
1. Download Python 2.7 or Python 3 from https://www.python.org/getit/, and run the installer.
53+
1. Add MinGW, CMake and Python into system PATH.
54+
1. Install Gcovr and [Mbed CLI](https://os.mbed.com/docs/latest/tools/arm-mbed-cli.html) with `pip install "gcovr>=4.1" mbed-cli`.
14155

142-
2. Run the application, add built test executables into the list and run it.
56+
### Test code structure
14357

144-
### Get code coverage
58+
Unit tests are located in the Mbed OS repository under the `UNITTESTS` folder. We recommend unit test files use an identical directory path to the file under test. This makes it easier to find unit tests for a particular class or a module. For example, if the file under test is `some/example/path/ClassName.cpp`, then all the test files are in the `UNITTESTS/some/example/path/ClassName` directory. Each test suite needs to have its own `unittest.cmake` file for test configuration.
14559

146-
Python tools use gcovr to build code coverage reports. Generate html report `UNITTESTS/build/coverage/index.html` with:
147-
```
148-
mbed test --unittests --coverage html
149-
```
60+
#### Test discovery
15061

151-
To get coverage for a single test suite, run gcovr separately for suite coverage data directory. See [gcovr documentation](https://gcovr.com/guide.html#filter-options) for more information.
62+
Registering unit tests for running is automatic, and the test runner handles registration. However, test files are not automatically assigned to be built. We build unit tests by using a separate build system, which searches for unit tests under the `UNITTESTS` directory.
15263

153-
e.g. for features/netsocket/InternetSocket coverage:
64+
For the build system to find and build any test suite automatically, you must include a unit test configuration file named `unittest.cmake` for each unit test suite. This configuration file contains all the source files required for the build.
15465

155-
Debian/Ubuntu/Mac OS:
156-
```
157-
mkdir UNITTESTS/build
158-
cd UNITTESTS/build
159-
cmake -DCMAKE_BUILD_TYPE=Debug -DCOVERAGE:STRING=html ..
160-
make
161-
./features-netsocket-InternetSocket
162-
gcovr -r ../.. --html --html-detail -o ./index.html ./CMakeFiles/features-netsocket-InternetSocket.MbedOS.dir/
163-
```
164-
Windows:
165-
```
166-
mkdir UNITTESTS/build
167-
cd UNITTESTS/build
168-
cmake -DCMAKE_BUILD_TYPE=Debug -DCOVERAGE:STRING=html -g "MinGW Makefiles" ..
169-
mingw32-make
170-
features-netsocket-InternetSocket.exe
171-
gcovr -r ..\.. --html --html-detail -o .\index.html .\CMakeFiles\features-netsocket-InternetSocket.MbedOS.dir\
172-
```
66+
#### Test names
17367

174-
## The structure of unit tests
68+
The build system automatically generates names of test suites. The name is constructed by taking a relative file path from the UNITTESTS directory to the test directory and replacing path separators with dashes. For example, the test suite name for `some/example/path/ClassName.cpp` is `some-example-path-ClassName`. Suite names are used when deciding which test suites to run.
17569

176-
The structure of the unit tests directory looks like this:
177-
```
178-
UNITTESTS
179-
├── mbed_unittest.py Python tool for unit testing
180-
├── unit_test Python tool modules
181-
├── CMakeLists.txt CMake project definition file
182-
├── CMakeSettings.json CMake configurations for Visual Studio 2017
183-
├── README.md
184-
├── googletest-CMakeLists.txt.in CMake project definition file for Google Test
185-
│
186-
├── features
187-
│ └── netsocket Directory tree that mirrors Mbed OS root
188-
│ ├── NetworkInterface Name of the class to be tested
189-
│ │ ├── test_NetworkInterface.cpp
190-
│ │ └── unittest.cmake CMake module for unit test
191-
│ └── Socket
192-
│
193-
├── stubs Shared stubs which can be used for tests.
194-
├── target_h Shared headers which can be used for tests.
195-
└── template Templates for creating new unittests
196-
```
197-
Each unit test has an identical directory tree as seen in the Mbed OS root folder. This is not a mandatory requirement but helps to maintain tests. Each class to be tested have their own `unittest.cmake` which is found by `CMakeLists.txt`.
70+
### Unit testing with Mbed CLI
19871

199-
## Creating a unit test
72+
Mbed CLI supports unit tests through `mbed test --unittests` command. For information on using Mbed CLI, please see the [CLI documentation in handbook](https://os.mbed.com/docs/latest/tools/arm-mbed-cli.html).
20073

201-
Each class to be tested requires two files for unit testing:
202-
1. C++ unit test source file (e.g. `test_NetworkInterface.cpp`)
203-
2. CMake module file for unit test definition (`unittest.cmake`)
74+
### Writing unit tests
20475

205-
A unit test definition file `unittest.cmake` requires variables to be set for a test to be configured. File source paths in `unittest.cmake` files need to be relative to the unit test folder and `CMakeLists.txt`.
76+
Create two files in the test directory for each test suite:
20677

207-
* **unittest-includes** - Include paths for headers needed to build the tests in addition to the base include paths listed in [CMakeLists.txt](CMakeLists.txt). Optional.
208-
* **unittest-sources** - Mbed OS source files and stubs included for the build.
209-
* **unittest-test-sources** - Unit test source files.
78+
- Unit test source file (`test_ClassName.cpp`).
79+
- Unit test configuration file (`unittest.cmake`).
21080

211-
#### Creating unit tests files with Mbed CLI
81+
List all the files required for the build in the `unittest.cmake` file. We recommend you list the file paths relative to the `UNITTESTS` folder. Use the following variables to list the source files and include paths:
21282

213-
```
214-
mbed test --unittests --new <FILEPATH>
215-
```
83+
- **unittest-includes** - List of header include paths. You can use this to extend or overwrite default paths listed in CMakeLists.txt.
84+
- **unittest-sources** - List of files under test.
85+
- **unittest-test-sources** - List of test sources and stubs.
21686

217-
E.g. `mbed test --unittests --new rtos/Semaphore.cpp`
87+
With the following steps, you can write a simple unit test. In this example, `rtos/Semaphore.cpp` is a class under test.
21888

219-
The generator script only creates the files required for a unit test. It does not write unit tests automatically nor does it handle source dependencies.
89+
1. Create a directory for unit test files in `UNITTESTS/rtos/Semaphore`.
90+
1. Create a test configuration file `UNITTESTS/rtos/Semaphore/unittest.cmake` with the following content:
22091

221-
#### Create files manually
92+
```
93+
set(unittest-sources
94+
../rtos/Semaphore.cpp
95+
)
96+
97+
set(unittest-test-sources
98+
stubs/mbed_assert.c
99+
rtos/Semaphore/test_Semaphore.cpp
100+
)
101+
```
222102

223-
For example to create a unit test for `rtos/Semaphore.cpp`:
103+
1. Create a test source file `UNITTESTS/rtos/Semaphore/test_Semaphore.cpp` with the following content:
224104

225-
1. Create a directory for unit test files in `UNITTESTS/rtos/Semaphore`.
226-
2. Create a test definition file `UNITTESTS/rtos/Semaphore/unittest.cmake` with the following content:
227-
```
228-
set(unittest-sources
229-
stubs/mbed_assert.c
230-
../rtos/Semaphore.cpp
231-
)
232-
233-
set(unittest-test-sources
234-
rtos/Semaphore/test_Semaphore.cpp
235-
)
236-
```
237-
3. Create a test source file `UNITTESTS/rtos/Semaphore/test_Semaphore.cpp` with the following content:
238105
```
239106
#include "gtest/gtest.h"
240107
#include "rtos/Semaphore.h"
@@ -285,8 +152,39 @@ TEST_F(TestSemaphore, constructor)
285152
}
286153
```
287154

288-
## Troubleshooting
155+
### Building and running unit tests
156+
157+
Use Mbed CLI to build and run unit tests. For advanced use, you can run CMake and a make program directly.
158+
159+
#### Build tests directly with CMake
160+
161+
1. Create a build directory `mkdir UNITTESTS/build`.
162+
1. Move to the build directory `cd UNITTESTS/build`.
163+
1. Run CMake using a relative path to `UNITTESTS` folder as the argument. So from `UNITTESTS/build` use `cmake ..`:
164+
- Add `-g [generator]` if generating other than Unix Makefiles such in case of MinGW use `-g "MinGW Makefiles"`.
165+
- Add `-DCOVERAGE:True` to add coverage compiler flags.
166+
- See the [CMake manual](https://cmake.org/cmake/help/v3.0/manual/cmake.1.html) for more information.
167+
1. Run a make program (Make, Gmake, Mingw32-make and so on) to build the tests.
168+
169+
#### Run tests directly with CTest
170+
171+
Run a test binary in the build directory to run a unit test suite. To run multiple test suites at once, use CTest test runner. Run CTest with `ctest`. Add `-v` to get results for each test case. See the [CTest manual](https://cmake.org/cmake/help/v3.0/manual/ctest.1.html) for more information.
172+
173+
#### Run tests with GUI test runner
174+
175+
1. Install *gtest-runner* using the [documentation](https://github.com/nholthaus/gtest-runner).
176+
1. Run *gtest-runner*
177+
1. Add test executables into the list.
178+
1. Run them.
179+
180+
### Get code coverage
181+
182+
Use Mbed CLI to generate code coverage reports. For advanced use, you can run Gcovr or any other code coverage tool directly in the build directory.
183+
184+
### Troubleshooting
289185

290-
**Problem:** virus protection identifies files generated by CMake as malicious and quarantines the files on Windows.
186+
**Problem:** Generic problems with CMake or with the build process.
187+
* **Solution**: Delete the build directory. Make sure that CMake, g++, gcc and a make program can be found in the path and are correct versions.
291188

292-
* **Solution**: restore the false positive files from the quarantine.
189+
**Problem:** Virus protection identifies files generated by CMake as malicious and quarantines the files on Windows.
190+
* **Solution**: Restore the false positive files from the quarantine.

0 commit comments

Comments
 (0)