contest: vmksft-p: support parsing nested tests

Some selftests are executing a lot of different subtests. On the other
hand, the kselftests infrastructure only supports TAP 13 format, which
doesn't support subtests.

If one subtest fails, the whole selftest is marked as failed. It starts
to be really annoying when one subtest starts to be unstable and marked
as "ignored": all the other subtests are then ignored as well. It is
then important to parse subtests to be able to track each subtest
individually, and not loose info about the others when one is unstable.

A workaround to support subtests with TAP 13 is to embed the subtests
result in the comments. This is what is done with MPTCP selftests.
That's not specific to MPTCP, because TC and others written in C and
using kselftest_harness.h are also doing that. The output to parse looks
like that:

  TAP version 13
  1..1
  # timeout set to (...)
  # selftests: (...)
  # <comment>
  # (...)
  # <comment>
  #
  # TAP version 13         |
  # 1..X                   |
  # ok 1 (...)             |
  # nok 2 (...)            | <== That's what we want to parse
  # ok 3 (...) # SKIP      |
  # (...)                  |
  # ok X (...)             |
  ok 1 selftests: {target}: {prog}

The last line is not parsed, this is handled by the existing code, also
checking for VM crash, etc. This means that in case of crash, the
kselftest will be marked as 'fail', but subtests (if any) will be
parsed: the ones previous to the crash are still valid. It might help to
know what was OK, and what was not.

Because it will increase the number of tests, there is a new option,
disabled by default, to enable this feature or not per target, where it
makes sense:

  [ksft]
  nested_tests = on

Signed-off-by: Matthieu Baerts (NGI0) <[email protected]>

Author: matttbe

contest/remote/vmksft-p.py

@@ -47,6 +47,7 @@
 boot_timeout=45
 [ksft]
 targets=net
+nested_tests=off / on
 
 
 Expected:
@@ -73,6 +74,42 @@ def get_prog_list(vm, target):
     return [e.split(":")[1].strip() for e in targets]
 
 
+def _parse_nested_tests(full_run):
+    tests = []
+    nested_tests = False
+
+    result_re = re.compile(r"(not )?ok (\d+)( -)? ([^#]*[^ ])( # )?([^ ].*)?$")
+
+    for line in full_run.split('\n'):
+        # nested subtests support: we parse the comments from 'TAP version'
+        if nested_tests:
+            if line.startswith("# "):
+                line = line[2:]
+            else:
+                nested_tests = False
+        elif line.startswith("# TAP version "):
+            nested_tests = True
+            continue
+
+        if not nested_tests:
+            continue
+
+        if line.startswith("ok "):
+            result = "pass"
+        elif line.startswith("not ok "):
+            result = "fail"
+        else:
+            continue
+
+        v = result_re.match(line).groups()
+        name = v[3]
+        if len(v) > 5 and v[5]:
+            if v[5].lower().startswith('skip') and result == "pass":
+                result = "skip"
+        tests.append((name, result))
+
+    return tests
+
 def _vm_thread(config, results_path, thr_id, in_queue, out_queue):
     target = config.get('ksft', 'target')
     vm = None
@@ -136,6 +173,14 @@ def _vm_thread(config, results_path, thr_id, in_queue, out_queue):
         out_queue.put({'prog': prog, 'test': test_name, 'file_name': file_name,
                        'result': result, 'time': (t2 - t1).seconds})
 
+        if config.getboolean('ksft', 'nested_tests', fallback=False):
+            # this will only parse nested tests inside the TAP comments
+            tests = _parse_nested_tests(vm.log_out)
+
+            for r_name, r_result in tests:
+                out_queue.put({'prog': prog, 'test': namify(r_name),
+                               'file_name': file_name, 'result': r_result})
+
         if vm.fail_state:
             print(f"INFO: thr-{thr_id} VM kernel crashed, destroying it")
             vm.stop()
@@ -214,7 +259,8 @@ def test(binfo, rinfo, cbarg):
     cases = []
     while not out_queue.empty():
         r = out_queue.get()
-        cbarg.prev_runtime[r["prog"]] = r["time"]
+        if 'time' in r:
+            cbarg.prev_runtime[r["prog"]] = r["time"]
         cases.append({'test': r['test'], 'group': grp_name, 'result': r["result"],
                       'link': link + '/' + r['file_name']})
     if not in_queue.empty():