Run e2e tests with modern probe #967

Molter73 · 2023-03-10T15:24:48Z

What type of PR is this?

/kind feature

Any specific area of the project related to this PR?

/area CI

/area tests

Does this PR require a change in the driver versions?

What this PR does / why we need it:

With the modern BPF probe quickly reaching a production ready state, I think it's time we start running e2e tests with it. This PR achieves this by adding a new set of sinsp-example binaries to be run using this new driver to the tests. A small check was added to prevent the tests from being run on systems that are not supported by the driver.

There are currently 3 tests that are failing:

2 of them are due to the exe_writable flag not being supported yet on the modern probe.
1 test is failing because the test expects proc.exe to be nginx: master proces, but the modern probe sets it to nginx: master process nginx -g daemon off;. This is not a major problem IMO, but some extra investigation is needed to understand why this happens before changing the test to pass.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:
I couldn't get sinsp-example to compile with the system libbpf in CI, so I set -DUSE_BUNDLED_LIBBPF=ON when compiling the tests.

Does this PR introduce a user-facing change?:

NONE

Signed-off-by: Mauro Ezequiel Moltrasio <[email protected]>

FedeDP · 2023-03-10T16:17:08Z

This is truly amazing @Molter73 😍

incertum

@Molter73 🚀 nice!

Does this PR aim to only update the CI or could we also get an update re how to use it on localhost? Currently getting IndexError: list index out of range errors ...

incertum · 2023-03-13T06:19:43Z

test/e2e/tests/test_event_generator/test_run_shell_untrusted.py

    sinsp.generate_id(sinsp_example) for sinsp_example in sinsp_examples
 ]

+# modern probe doesn't support EXE_WRITABLE flag yet


@LucaGuerra is this in your queue for Falco 0.35?

yep, I've in on the roadmap, in Falco 0.35 the modern probe should reach feature parity with other drivers 🎉

incertum · 2023-03-13T06:24:28Z

test/e2e/tests/test_network/test_network.py

 ids = [sinsp.generate_id(sinsp_example) for sinsp_example in sinsp_examples]

+# For some reason, the modern probe gives a longer proc.exe than the legacy
+# drivers, needs further investigation.


1 test is failing because the test expects proc.exe to be nginx: master proces, but the modern probe sets it to nginx: master process nginx -g daemon off;. This is not a major problem IMO, but some extra investigation is needed to understand why this happens before changing the test to pass.

Checked w/ bpf_printk statements while running the e2e tests locally ... not a truncation error, this is all we got in the old eBPF raw_syscalls tracepoints for argv[0] aka exe ... @Andreagit97 would you know some good docs stating why? Clearly the modern_bpf driver has the complete arg.

This demonstrates in an excellent way how much we need these e2e tests 🎉 .

is there a way to reproduce this test locally by running it with just the modern bpf probe?

Andreagit97 · 2023-03-13T08:59:51Z

Thank you @Molter73 ! Sorry for the silly question but Is there a way to run locally a single test, with just the modern bpf?

Molter73 · 2023-03-13T09:13:10Z

Hey everyone!

Does this PR aim to only update the CI or could we also get an update re how to use it on localhost? Currently getting IndexError: list index out of range errors ...

@incertum , this should work both locally and on CI, the problem comes from me doing hardcoded index access like this one:

libs/test/e2e/tests/test_network/test_network.py

Line 26 in 4d15d56

sinsp_examples[2] = pytest.param(sinsp_examples[2], marks=pytest.mark.xfail)

I'll get it fixed in a commit in a minute, sorry about that.

is there a way to reproduce this test locally by running it with just the modern bpf probe?

@Andreagit97 I haven't planned to get the e2e tests to be run with just a single type of driver, since they are pretty fast on their own. What you can do is change this script and point it to the directory/file that has the test you want to run. You can find instructions on how to run the e2e tests in the readme:
https://github.com/falcosecurity/libs/blob/4d15d56756d5bd30c265c17ad371c2a494c92f4f/test/e2e/README.md#running-the-tests

Side note, if you want the tests to stop at the first failure, you can pass -x to pytest in that same script.

Signed-off-by: Mauro Ezequiel Moltrasio <[email protected]>

Andreagit97 · 2023-03-13T11:49:50Z

For some reason, on my host the e2e tests never end, I've to type CTRL+C to stop them :/

----------------------------------------------- generated html file: file:///home/andrea/personal/libs/build/report/report.html ------------------------------------------------
======================================================================== 2 passed, 1 xfailed in 56.36s =========================================================================
^CException ignored in: <module 'threading' from '/usr/lib/python3.10/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1567, in _shutdown
    lock.acquire()
KeyboardInterrupt: 
make[3]: *** [test/e2e/CMakeFiles/e2e-tests.dir/build.make:71: test/e2e/CMakeFiles/e2e-tests] Interrupt
make[2]: *** [CMakeFiles/Makefile2:2055: test/e2e/CMakeFiles/e2e-tests.dir/all] Interrupt
make[1]: *** [CMakeFiles/Makefile2:2062: test/e2e/CMakeFiles/e2e-tests.dir/rule] Interrupt
make: *** [Makefile:832: e2e-tests] Interrupt

I used a python env to run them, and I launch them with the following command:

sudo -E env "PATH=$PATH" make e2e-tests

Molter73 · 2023-03-13T11:54:34Z

For some reason, on my host the e2e tests never end, I've to type CTRL+C to stop them :/

Yeah, there's a weird threading error were this method causes a lock when the last test fails (or xfails) in this case:

libs/test/e2e/tests/commons/sinspqa/sinsp.py

Lines 103 to 110 in acf239f

    
           @staticmethod 
        
           def _readline(stream: IO[AnyStr], queue: Queue): 
        
               with open(os.path.join(LOGS_PATH, 'sinsp.log'), 'a') as f: 
        
                   while SinspProcessStreamer._running: 
        
                       for line in iter(stream.readline, ''): 
        
                           if line != '' and line != 'null': 
        
                               f.write(line) 
        
                               queue.put(line.rstrip())

I believe it's because the inner for loop is waiting to read a line from sinsp-example, which should be stopped by a fixture that is somehow not running.

I'm aware of it and will try to get it sorted, but AFAICT it doesn't break anything major, the tests do end and the HTML report is correctly created.

Andreagit97 · 2023-03-13T12:07:32Z

I'm aware of it and will try to get it sorted, but AFAICT it doesn't break anything major, the tests do end and the HTML report is correctly created.

yep, I can confirm it 👍

Molter73 · 2023-03-13T15:25:56Z

Think I have a fix for the threading thing, looks like some Python non-sense that causes the __del__ destructor for the SinspStreamer to not be called when an assertion exception is thrown. I'll try to add the fix to this same PR or if it gets merged before that open a separate one, don't think this would be a considered a blocker for this PR.

incertum · 2023-03-13T17:15:33Z

@Andreagit97 I haven't planned to get the e2e tests to be run with just a single type of driver, since they are pretty fast on their own. What you can do is change this script and point it to the directory/file that has the test you want to run. You can find instructions on how to run the e2e tests in the readme: https://github.com/falcosecurity/libs/blob/4d15d56756d5bd30c265c17ad371c2a494c92f4f/test/e2e/README.md#running-the-tests

Side note, if you want the tests to stop at the first failure, you can pass -x to pytest in that same script.

@Molter73 above (either explicitly adding to README or supplying elegant options to toggle both the driver used and which tests to run) and fixing the threading thing would make a perfect touch-up PR, but agreed let's first merge this one!

incertum

Tests now run again locally, what's missing is also an updated cmake setup in the README aka add -DBUILD_LIBSCAP_MODERN_BPF=ON

Can be done all together in a possible touch up PR.

/approve

poiana · 2023-03-13T17:18:14Z

LGTM label has been added.

Details

Git tree hash: 166d97c48c15c64381cfceaa0700b57219aaea8c

Andreagit97 · 2023-03-13T17:53:37Z

yep we can merge this, with @FedeDP we found the reason behind this #967 (comment) we will try to come out with a fix in the next few days :)

Andreagit97

/approve

FedeDP

/approve

poiana · 2023-03-13T17:54:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Andreagit97, FedeDP, incertum, Molter73

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [Andreagit97,FedeDP,Molter73,incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LucaGuerra · 2023-05-16T15:44:13Z

/milestone 0.11.0

Molter73 added 2 commits March 10, 2023 14:40

feature(e2e-tests): run e2e-tests with the modern eBPF probe

b971b3a

Signed-off-by: Mauro Ezequiel Moltrasio <[email protected]>

fix(ci): run e2e-tests with the modern_probe in CI

4d15d56

Signed-off-by: Mauro Ezequiel Moltrasio <[email protected]>

poiana added release-note-none dco-signoff: yes kind/feature New feature or request area/CI area/tests labels Mar 10, 2023

poiana requested review from hbrueckner and jasondellaluce March 10, 2023 15:25

poiana added approved size/M labels Mar 10, 2023

Molter73 mentioned this pull request Mar 10, 2023

Extend existing sinsp-example based tests stackrox/collector#1043

Closed

incertum reviewed Mar 13, 2023

View reviewed changes

fix(e2e-tests): prevent out of index access on systems without BTF

be0801f

Signed-off-by: Mauro Ezequiel Moltrasio <[email protected]>

poiana added size/L and removed size/M labels Mar 13, 2023

incertum approved these changes Mar 13, 2023

View reviewed changes

poiana assigned incertum Mar 13, 2023

poiana added the lgtm label Mar 13, 2023

Andreagit97 approved these changes Mar 13, 2023

View reviewed changes

poiana assigned Andreagit97 Mar 13, 2023

FedeDP approved these changes Mar 13, 2023

View reviewed changes

poiana assigned FedeDP Mar 13, 2023

poiana merged commit a055187 into falcosecurity:master Mar 13, 2023

Andreagit97 mentioned this pull request Mar 14, 2023

[WIP] Collect correct arguments when setproctitle is called #976

Closed

FedeDP mentioned this pull request Mar 20, 2023

Investigate setproctitle drivers behavior #988

Closed

poiana added this to the 0.11.0 milestone May 16, 2023

Run e2e tests with modern probe #967

Run e2e tests with modern probe #967

Uh oh!

Conversation

Molter73 commented Mar 10, 2023

Uh oh!

FedeDP commented Mar 10, 2023

Uh oh!

incertum left a comment

Choose a reason for hiding this comment

Uh oh!

incertum Mar 13, 2023

Choose a reason for hiding this comment

Uh oh!

Andreagit97 Mar 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

incertum Mar 13, 2023

Choose a reason for hiding this comment

Uh oh!

Andreagit97 Mar 13, 2023

Choose a reason for hiding this comment

Uh oh!

Andreagit97 commented Mar 13, 2023

Uh oh!

Molter73 commented Mar 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Andreagit97 commented Mar 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Molter73 commented Mar 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Andreagit97 commented Mar 13, 2023

Uh oh!

Molter73 commented Mar 13, 2023

Uh oh!

incertum commented Mar 13, 2023

Uh oh!

incertum left a comment

Choose a reason for hiding this comment

Uh oh!

poiana commented Mar 13, 2023

Uh oh!

Andreagit97 commented Mar 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Andreagit97 left a comment

Choose a reason for hiding this comment

Uh oh!

FedeDP left a comment

Choose a reason for hiding this comment

Uh oh!

poiana commented Mar 13, 2023

Uh oh!

LucaGuerra commented May 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Andreagit97 Mar 13, 2023 •

edited

Loading

Molter73 commented Mar 13, 2023 •

edited

Loading

Andreagit97 commented Mar 13, 2023 •

edited

Loading

Molter73 commented Mar 13, 2023 •

edited

Loading

Andreagit97 commented Mar 13, 2023 •

edited

Loading