Make it possible to execute luatest suites #310

VitaliyaIoffe · 2021-06-15T11:03:30Z

Provide luatest server, which can find tests by luatest pattern *_test.lua
and execute them luatest -v <name>_test.lua -o tap --shuffle none.

Was changed get_filename_by_test for right working not only with *.test.*,
but also with *_test.*.

Fixes #304

lib/test.py

lib/luatest_server.py

Totktonada · 2021-07-13T19:32:57Z

lib/luatest_server.py

+        command = [os.environ['LUATEST_BIN'], '-v', '-c', self.name,
+                   '-o', 'tap',
+                   '--shuffle', 'none']
+        proc = Popen(command, cwd=server.vardir, stdout=PIPE, stderr=STDOUT)


I'm not very fun about mixing stdout in TAP13 format and stderr in arbitrary format. It is possible that the mix will anyway be parseable as TAP13 (or at least be accepted by given TAP13 parser). Say, if stderr is formatted in YAML, it may be interpreted as a TAP13 YAML block. Depending of certain parser there may be other kinds of output that will be accepted. For example, our TAP13 parser is not very strict (self.strict is always False). AFAIR, I had intention to make the parser strictier, but our tests were not ready for that, so I hid this behaviour under the flag. It would be good to re-check, whether we can enable it now (out of this issue). There were fixes around incorrect test plans.

Back to mixing of stderr into stdout. In the past, AppTest had this behaviour, but I changed it. I think it would be good to don't introduce it again for LuatestTest.

Have to change -c option in luatest
tarantool/luatest#147

But the code in test-run that mixing stdout and stderr still there. It should not.

I'll leave it as is for now.

lib/luatest_server.py

lib/__init__.py

Totktonada · 2021-07-13T20:17:50Z

I glanced over the code (without actual testing — will do on the next review iteration). Aside of comments above I would want to see a luatest test in the test-run repository.

I think this PR should close the issue: further acivities should go under separate tasks.

Provide luatest and its dependency checks as a submodules of test-run. Part of: #304

test/luatest_helpers/server.lua

Totktonada · 2021-10-20T22:26:37Z

I see that some on the helper code were changed in tarantool/tarantool#6181, but not here, so I left my comments in the linked PR. I don't know, which variant of the code is in some transient state and which one is the final. Please, ping me, when there will be one version to review.

I glanced over the code again (the version in this PR) and I don't see problems aside of the unmotivated inconsistency with tarantool's version. So, next time, I hope, I'll just run it in several ways (in-source, out-of-source, pure luatest) to ensure that it works as expected and accept.

test/luatest_helpers.lua

Provide luatest server, which can find tests by luatest pattern *_test.lua and execute them luatest -c -v <name>_test.lua -o tap --shuffle none. Use luatest bin from a submodule. Was changed get_filename_by_test for right working not only with *.test.*, but also with *_test.*. Smoke check lua-test was added with starting server and checks of box_cfg args. Show a test suite type when it is unknown. Fixes #304

Totktonada · 2021-10-22T02:10:03Z

I think this PR should close the issue: further acivities should go under separate tasks.

I changed it in the commit message and in the PR description.

A simple example is in the test-run source tree (`test/test-luatest/smoke_check_test.lua`). tarantool/test-run#304 tarantool/test-run#310

A simple example is in the test-run source tree (`test/test-luatest/smoke_check_test.lua`). tarantool/test-run#304 tarantool/test-run#310 (cherry picked from commit e3ca1ed)

Totktonada · 2021-10-22T02:52:57Z

Updated the test-run submodule in tarantool in 2.10.0-beta1-101-ge3ca1edff, 2.8.2-33-gb65cb91f4, 1.10.11-21-g2ad4e3487.

VitaliyaIoffe requested a review from Totktonada June 15, 2021 11:03

VitaliyaIoffe linked an issue Jun 15, 2021 that may be closed by this pull request

Consider integration with luatest #304

Closed

VitaliyaIoffe requested a review from avtikhon June 15, 2021 11:51

VitaliyaIoffe force-pushed the VitaliyaIoffe/gh-304-include-luatest branch 2 times, most recently from e8bf58b to 3abad5e Compare June 15, 2021 13:54

VitaliyaIoffe force-pushed the VitaliyaIoffe/gh-304-include-luatest branch from 3abad5e to a9c7f53 Compare June 23, 2021 09:43