contest: vmksft: support parsing nested tests

Some selftests are executing a lot of different subtests. On the other
hand, the kselftests infrastructure only supports TAP 13 format, which
doesn't support subtests.

If one subtest fails, the whole selftest is marked as failed. It starts
to be really annoying when one subtest starts to be unstable and marked
as "ignored": all the other subtests are then ignored as well. It is
then important to parse subtests to be able to track each subtest
individually, and not loose info about the others when one is unstable.

A workaround to support subtests with TAP 13 is to embed the subtests
result in the comments. This is what is done with MPTCP selftests.
That's not specific to MPTCP, because TC and others written in C and
using kselftest_harness.h are also doing that.

Because it will increase the number of tests, there is a new option,
disabled by default, to enable this feature or not per target:

  [ksft]
  nested_tests = on

When nested TAP results are detected, new tests are being created on
top of the main/parent one. At the end of the comments, the parsing
continue on the main/parent test. Note that the 'output' of these
subtests will not be duplicated in the main test and the subtests ones:
what is specific to the subtests will only be in the 'output' of the
subtests, not the main one.

Signed-off-by: Matthieu Baerts (NGI0) <[email protected]>

Author: matttbe

contest/remote/vmksft.py

@@ -42,6 +42,7 @@
 boot_timeout=45
 [ksft]
 targets=net
+nested_tests=off / on
 
 
 Expected:
@@ -51,14 +52,28 @@
 """
 
 
-def ktap_split(full_run):
+def ktap_split(full_run, parse_nested_tests):
     tests = []
     test = None
     test_id = 0
+    test_main = None
 
     result_re = re.compile(r"(not )?ok (\d+)( -)? ([^#]*[^ ])( # )?([^ ].*)?$")
 
     for line in full_run.split('\n'):
+        if parse_nested_tests:
+            # nested tests support: we parse the comments from 'TAP version'
+            if test_main:
+                if line.startswith("# "):
+                    line = line[2:]
+                else:
+                    # back to the main test
+                    test = test_main
+                    test_main = None
+            elif line.startswith("# TAP version "):
+                test_main = test
+                test = None
+
         if test is None:
             test = {
                 "tid": test_id,
@@ -168,7 +183,9 @@ def test(binfo, rinfo, cbarg):
     full_run = vm.log_out
     vm.dump_log(results_path + '/full', result=retcode, info={"vm_state": vm.fail_state})
 
-    tests = ktap_split(full_run)
+    parse_nested_tests = config.getboolean('ksft', 'nested_tests',
+                                                fallback=False)
+    tests = ktap_split(full_run, parse_nested_tests)
     if tests:
         pfx = ktap_extract_pfx(tests)
         grp_name = namify(pfx)