Skip to content

Cleanup coverage CI job #6606

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jan 22, 2022

Conversation

tautschnig
Copy link
Collaborator

@tautschnig tautschnig commented Jan 21, 2022

This is an attempt to remove any code marked "UNREACHABLE" from the
coverage report. (Such code just artificially reduces our coverage
numbers.)

While at it I also suspected that we might be running lcov twice.

See individual commit messages for numeric data points.

  • Each commit message has a non-empty body, explaining why the change was made.
  • n/a Methods or procedures I have added are documented, following the guidelines provided in CODING_STANDARD.md.
  • n/a The feature or user visible behaviour I have added or modified has been documented in the User Guide in doc/cprover-manual/
  • Regression or unit tests are included, or existing tests cover the modified code (in this case I have detailed which ones those are in the commit message).
  • My commit message includes data points confirming performance improvements (if claimed).
  • My PR is restricted to a single feature or bugfix.
  • n/a White-space or formatting changes outside the feature-related changed lines are in commits of their own.

@tautschnig tautschnig self-assigned this Jan 21, 2022
cmake --target coverage already takes care of running lcov. We ended up
doing exactly the same work twice, just logging to different output
files. This should save about 90 seconds of CI job execution time.
@codecov
Copy link

codecov bot commented Jan 21, 2022

Codecov Report

Merging #6606 (8aa6be8) into develop (5aae9d0) will increase coverage by 0.12%.
The diff coverage is 90.47%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #6606      +/-   ##
===========================================
+ Coverage    76.49%   76.62%   +0.12%     
===========================================
  Files         1578     1578              
  Lines       181464   181180     -284     
===========================================
+ Hits        138816   138833      +17     
+ Misses       42648    42347     -301     
Impacted Files Coverage Δ
src/solvers/lowering/byte_operators.cpp 92.17% <76.92%> (ø)
src/ansi-c/ansi_c_convert_type.cpp 80.00% <100.00%> (+0.81%) ⬆️
src/ansi-c/c_storage_spec.cpp 97.56% <100.00%> (ø)
src/ansi-c/parser.y 79.70% <100.00%> (+0.06%) ⬆️
src/util/c_types.h 100.00% <100.00%> (ø)
src/util/pointer_expr.h 78.77% <100.00%> (ø)
src/util/std_types.cpp 86.81% <100.00%> (+0.94%) ⬆️
src/util/std_types.h 95.77% <100.00%> (ø)
src/util/type.h 100.00% <100.00%> (ø)
... and 147 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cc9d04e...8aa6be8. Read the comment docs.

Code marked "UNREACHABLE" just artificially reduces our coverage
numbers. Tell lcov about this (it would by default exclude lines
containing "LCOV_EXCL_LINE" only). This changes the coverage reported at
present by +0.12 percentage points.
@tautschnig tautschnig force-pushed the cleanup/lcov-unreachable branch from 0935b7a to 8aa6be8 Compare January 21, 2022 18:57
@tautschnig tautschnig changed the title [WIP] Cleanup coverage CI job Cleanup coverage CI job Jan 21, 2022
@tautschnig tautschnig removed their assignment Jan 21, 2022
@tautschnig tautschnig marked this pull request as ready for review January 21, 2022 18:58
@tautschnig tautschnig requested a review from a team as a code owner January 21, 2022 18:58
Copy link
Contributor

@chrisr-diffblue chrisr-diffblue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A clear improvement on what we have, thanks!

@@ -700,7 +700,9 @@ jobs:
- name: Print ccache stats
run: ccache -s
- name: Run CTest and collect coverage statistics
run: cmake --build build --target coverage -- -j2
run: |
echo "lcov_excl_line = UNREACHABLE" > ~/.lcovrc
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I'm not massively familiar with lcov (to my shame) so apologies if these are dumb questions... but:

  • Does this exclude only lines containing UNREACHABLE, or also any other lines following that in the same block/trace?
  • Do we want/need to do something similar for branch coverage with lcov_excl_br_line ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it only affects exactly that one line that has "UNREACHABLE." There are ways to mark entire blocks (details in https://linux.die.net/man/1/geninfo), which we could consider if deemed useful.

I don't think we measure branch coverage at this time, but if needed we could certainly extend the configuration.

@tautschnig tautschnig self-assigned this Jan 22, 2022
@tautschnig tautschnig merged commit 739a158 into diffblue:develop Jan 22, 2022
@tautschnig tautschnig deleted the cleanup/lcov-unreachable branch January 22, 2022 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants