Skip to content

Cleanup coverage CI job #6606

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jan 22, 2022
Merged

Cleanup coverage CI job #6606

merged 2 commits into from
Jan 22, 2022

Conversation

tautschnig
Copy link
Collaborator

@tautschnig tautschnig commented Jan 21, 2022 •
edited
Loading

This is an attempt to remove any code marked "UNREACHABLE" from the
coverage report. (Such code just artificially reduces our coverage
numbers.)

While at it I also suspected that we might be running lcov twice.

See individual commit messages for numeric data points.

  • Each commit message has a non-empty body, explaining why the change was made.
  • n/a Methods or procedures I have added are documented, following the guidelines provided in CODING_STANDARD.md.
  • n/a The feature or user visible behaviour I have added or modified has been documented in the User Guide in doc/cprover-manual/
  • Regression or unit tests are included, or existing tests cover the modified code (in this case I have detailed which ones those are in the commit message).
  • My commit message includes data points confirming performance improvements (if claimed).
  • My PR is restricted to a single feature or bugfix.
  • n/a White-space or formatting changes outside the feature-related changed lines are in commits of their own.

@tautschnig tautschnig self-assigned this Jan 21, 2022
@tautschnig
Coverage-reporting CI job: don't run lcov twice
3f92fcb 
cmake --target coverage already takes care of running lcov. We ended up
doing exactly the same work twice, just logging to different output
files. This should save about 90 seconds of CI job execution time.
@codecov
Copy link

codecov bot commented Jan 21, 2022 •
edited
Loading

Codecov Report

Merging #6606 (8aa6be8) into develop (5aae9d0) will increase coverage by 0.12%.
The diff coverage is 90.47%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #6606      +/-   ##
===========================================
+ Coverage    76.49%   76.62%   +0.12%     
===========================================
  Files         1578     1578              
  Lines       181464   181180     -284     
===========================================
+ Hits        138816   138833      +17     
+ Misses       42648    42347     -301
Impacted Files Coverage Δ
src/solvers/lowering/byte_operators.cpp 92.17% <76.92%> (ø)
src/ansi-c/ansi_c_convert_type.cpp 80.00% <100.00%> (+0.81%) ⬆️
src/ansi-c/c_storage_spec.cpp 97.56% <100.00%> (ø)
src/ansi-c/parser.y 79.70% <100.00%> (+0.06%) ⬆️
src/util/c_types.h 100.00% <100.00%> (ø)
src/util/pointer_expr.h 78.77% <100.00%> (ø)
src/util/std_types.cpp 86.81% <100.00%> (+0.94%) ⬆️
src/util/std_types.h 95.77% <100.00%> (ø)
src/util/type.h 100.00% <100.00%> (ø)
... and 147 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cc9d04e...8aa6be8. Read the comment docs.

@tautschnig
CI/Codecov report: ignore code marked "UNREACHABLE"
8aa6be8 
Code marked "UNREACHABLE" just artificially reduces our coverage
numbers. Tell lcov about this (it would by default exclude lines
containing "LCOV_EXCL_LINE" only). This changes the coverage reported at
present by +0.12 percentage points.
@tautschnig tautschnig force-pushed the cleanup/lcov-unreachable branch from 0935b7a to 8aa6be8 Compare January 21, 2022 18:57
@tautschnig tautschnig changed the title [WIP] Cleanup coverage CI job Cleanup coverage CI job Jan 21, 2022
@tautschnig tautschnig removed their assignment Jan 21, 2022
@tautschnig tautschnig marked this pull request as ready for review January 21, 2022 18:58
@tautschnig tautschnig requested a review from a team as a code owner January 21, 2022 18:58
chrisr-diffblue
chrisr-diffblue approved these changes Jan 22, 2022
View reviewed changes
Copy link
Contributor

@chrisr-diffblue chrisr-diffblue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A clear improvement on what we have, thanks!

.github/workflows/pull-request-checks.yaml
@@ -700,7 +700,9 @@ jobs:
- name: Print ccache stats
run: ccache -s
- name: Run CTest and collect coverage statistics
run: cmake --build build --target coverage -- -j2
run: |
echo "lcov_excl_line = UNREACHABLE" > ~/.lcovrc
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I'm not massively familiar with lcov (to my shame) so apologies if these are dumb questions... but:

  • Does this exclude only lines containing UNREACHABLE, or also any other lines following that in the same block/trace?
  • Do we want/need to do something similar for branch coverage with lcov_excl_br_line ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it only affects exactly that one line that has "UNREACHABLE." There are ways to mark entire blocks (details in https://linux.die.net/man/1/geninfo), which we could consider if deemed useful.

I don't think we measure branch coverage at this time, but if needed we could certainly extend the configuration.

@tautschnig tautschnig self-assigned this Jan 22, 2022
@tautschnig tautschnig merged commit 739a158 into diffblue:develop Jan 22, 2022
@tautschnig tautschnig deleted the cleanup/lcov-unreachable branch January 22, 2022 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chrisr-diffblue chrisr-diffblue chrisr-diffblue approved these changes

Assignees

@tautschnig tautschnig

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@tautschnig @chrisr-diffblue