Skip to content

C++: Reveal false negative in test case #17275

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 26, 2024

Conversation

asgerf
Copy link
Contributor

@asgerf asgerf commented Aug 21, 2024

Some of the sinks in this test were flagged for the wrong reason.

The flow into these sinks isn't working properly, but this was not revealed by the test since an alternate, spurious path exists for each of them. The spurious path goes through the implicit read at one of the prior sinks and takes a use-use step.

In one case the false negative can be revealed by swapping the order of the sinks, but in the other case I had to remove some of the sinks to avoid the interference.

This came up while trying to fix the issue with use-use flow after an implicit read, and this test started failing because the fix removes the spurious paths that the test was relying on.

@github-actions github-actions bot added the C++ label Aug 21, 2024
asgerf added a commit to asgerf/codeql that referenced this pull request Aug 22, 2024
This reveals that some tests were passing for the wrong reasons.
See github#17275
@asgerf asgerf marked this pull request as ready for review August 22, 2024 09:52
@asgerf asgerf requested a review from a team as a code owner August 22, 2024 09:52
@asgerf asgerf added the no-change-note-required This PR does not need a change note label Aug 22, 2024
asgerf added a commit to asgerf/codeql that referenced this pull request Aug 22, 2024
This reveals that some tests were passing for the wrong reasons.
See github#17275
asgerf added a commit to asgerf/codeql that referenced this pull request Aug 22, 2024
This reveals that some tests were passing for the wrong reasons.
See github#17275
Copy link
Contributor

@geoffw0 geoffw0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look fine.

Is there any particular reason you want to change the tests, rather than just changing the expected results when you fix the issue? (with a note about why they've happened, I suppose)

@geoffw0
Copy link
Contributor

geoffw0 commented Aug 22, 2024

... on a second look, I think some of these tests and the related models are a bit naive about collection content, which we do now have in CPP (mostly). I can perhaps find time to fix things a bit.

@asgerf
Copy link
Contributor Author

asgerf commented Aug 23, 2024

Is there any particular reason you want to change the tests, rather than just changing the expected results when you fix the issue? (with a note about why they've happened, I suppose)

Mainly I wanted to be confident that the fix is 100% correct, and it's just easier to get there when there are no suspicious-looking test changes, like losing a TP. I guess I could have described the issues in the PR, but I worry that the discussion could get a bit muddled in the weeds of a PR that affects multiple languages (Java also had a similar PR).

asgerf added a commit to asgerf/codeql that referenced this pull request Aug 23, 2024
This reveals that some tests were passing for the wrong reasons.
See github#17275
@jketema
Copy link
Contributor

jketema commented Aug 26, 2024

Instead of modifying the existing tests, I think it would be better to add some tests. It's not clear to me that moving things around here means that we are still testing what we were testing before.

@asgerf
Copy link
Contributor Author

asgerf commented Aug 26, 2024

Instead of modifying the existing tests, I think it would be better to add some tests

The problem is that the test is going to fail once I merge #17262, and we obviously can't make it not-fail by adding more tests.

@jketema
Copy link
Contributor

jketema commented Aug 26, 2024

Then the test results should be updated with #17262, and the test shouldn't be modified.

@asgerf asgerf force-pushed the cpp/taint-test-case-false-negative branch from 1053434 to 16c2cf2 Compare August 26, 2024 09:53
@asgerf
Copy link
Contributor Author

asgerf commented Aug 26, 2024

Sounds good to me. I've merged the shared PR. This PR now just updates the inline test annotation, but feel free to close and handle the test however you like.

@jketema
Copy link
Contributor

jketema commented Aug 26, 2024

Thanks.

@asgerf asgerf merged commit 4e3440a into github:main Aug 26, 2024
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C++ no-change-note-required This PR does not need a change note
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants