Add haskell program coverage reports to Hydra tests #2124

rvl · 2020-09-09T06:33:06Z

Issue Number

ADP-99

Overview

This is a bit of an assortment of nix build improvements.

Add a code test coverage report for the Hydra build - implemented by PR First iteration of coverage reports input-output-hk/haskell.nix#762
Add a nix-shell with profiled packages. Profiled packages will be built on Hydra for master branch but not PRs. This means you can download haskell dependencies with profiling enabled, rather than having to build everything yourself.
Update versions of build tools in the nix-shell to latest hackage release - ghcide, hlint and stylish-haskell updated.

Comments

Stack coverage report for comparison

Command

stack build --coverage --fast --test --skip integration --skip jormungandr-integration

Result:

...
Generating unified report            
 26% expressions used (26186/98111)
 44% boolean coverage (136/305)
      42% guards (102/240), 72 always True, 7 always False, 59 unevaluated
      52% 'if' conditions (33/63), 4 always True, 8 always False, 18 unevaluated
      50% qualifiers (1/2), 1 always True
 40% alternatives used (849/2108)
 58% local declarations used (859/1456)
 50% top-level declarations used (1769/3533)
The unified report is available at /home/rodney/iohk/cardano-wallet/.stack-work/install/x86_64-linux/2cecc28bf3aab8c8c3e4a07c1c6c1c846ec8861df3d8a5e9247bce185aeb7542/8.6.5/hpc/combined/all/hpc_index.html
                
An index of the generated HTML coverage reports is available at /home/rodney/iohk/cardano-wallet/.stack-work/install/x86_64-linux/2cecc28bf3aab8c8c3e4a07c1c6c1c846ec8861df3d8a5e9247bce185aeb7542/8.6.5/hpc/index.html
                
--  While building package cardano-wallet-2020.9.30 using:
      /home/rodney/.stack/setup-exe-cache/x86_64-linux/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.5 --builddir=.stack-work/dist/x86_64-linux/Cabal-2.4.0.1 build lib:cardano-wallet exe:cardano-wallet test:unit --ghc-options "-hpcdir .stack-work/dist/x86_64-linux/Cabal-2.4.0.1/hpc -fdiagnostics-color=always"
    Process exited with code: ExitFailure 1
Progress 335/336

sevanspowell

LGTM!

rvl · 2020-10-06T07:25:07Z

bors try

iohk-bors · 2020-10-06T09:06:21Z

try

Build succeeded:

rvl · 2020-10-09T06:10:34Z

bors r+

2124: Add haskell program coverage reports to Hydra tests r=rvl a=rvl ### Issue Number ADP-99 ### Overview This is a bit of an assortment of nix build improvements. 1. Add a code test coverage report for the Hydra build - implemented by PR input-output-hk/haskell.nix#762 2. Add a nix-shell with profiled packages. Profiled packages will be built on Hydra for master branch but not PRs. This means you can download haskell dependencies with profiling enabled, rather than having to build everything yourself. 3. Update versions of build tools in the nix-shell to latest hackage release - ghcide, hlint and stylish-haskell updated. ### Comments - [Hydra jobset](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-2124) - [Coverage report job](https://hydra.iohk.io/job/Cardano/cardano-wallet-pr-2124/musl64.testCoverageReport.x86_64-linux/latest) - [Coverage report from bors try](https://hydra.iohk.io/build/4328848/download/2/hpc_index.html) <details> <summary>Stack coverage report for comparison</summary> #### Command ``` stack build --coverage --fast --test --skip integration --skip jormungandr-integration ``` #### Result: ``` ... Generating unified report 26% expressions used (26186/98111) 44% boolean coverage (136/305) 42% guards (102/240), 72 always True, 7 always False, 59 unevaluated 52% 'if' conditions (33/63), 4 always True, 8 always False, 18 unevaluated 50% qualifiers (1/2), 1 always True 40% alternatives used (849/2108) 58% local declarations used (859/1456) 50% top-level declarations used (1769/3533) The unified report is available at /home/rodney/iohk/cardano-wallet/.stack-work/install/x86_64-linux/2cecc28bf3aab8c8c3e4a07c1c6c1c846ec8861df3d8a5e9247bce185aeb7542/8.6.5/hpc/combined/all/hpc_index.html An index of the generated HTML coverage reports is available at /home/rodney/iohk/cardano-wallet/.stack-work/install/x86_64-linux/2cecc28bf3aab8c8c3e4a07c1c6c1c846ec8861df3d8a5e9247bce185aeb7542/8.6.5/hpc/index.html -- While building package cardano-wallet-2020.9.30 using: /home/rodney/.stack/setup-exe-cache/x86_64-linux/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.5 --builddir=.stack-work/dist/x86_64-linux/Cabal-2.4.0.1 build lib:cardano-wallet exe:cardano-wallet test:unit --ghc-options "-hpcdir .stack-work/dist/x86_64-linux/Cabal-2.4.0.1/hpc -fdiagnostics-color=always" Process exited with code: ExitFailure 1 Progress 335/336 ``` </details> Co-authored-by: Rodney Lorrimar <[email protected]> Co-authored-by: Samuel Evans-Powell <[email protected]>

iohk-bors · 2020-10-09T08:40:09Z

Timed out.

rvl · 2020-10-09T08:50:48Z

Big rebuild...

bors r+

2124: Add haskell program coverage reports to Hydra tests r=rvl a=rvl ### Issue Number ADP-99 ### Overview This is a bit of an assortment of nix build improvements. 1. Add a code test coverage report for the Hydra build - implemented by PR input-output-hk/haskell.nix#762 2. Add a nix-shell with profiled packages. Profiled packages will be built on Hydra for master branch but not PRs. This means you can download haskell dependencies with profiling enabled, rather than having to build everything yourself. 3. Update versions of build tools in the nix-shell to latest hackage release - ghcide, hlint and stylish-haskell updated. ### Comments - [Hydra jobset](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-2124) - [Coverage report job](https://hydra.iohk.io/job/Cardano/cardano-wallet-pr-2124/musl64.testCoverageReport.x86_64-linux/latest) - [Coverage report from bors try](https://hydra.iohk.io/build/4328848/download/2/hpc_index.html) <details> <summary>Stack coverage report for comparison</summary> #### Command ``` stack build --coverage --fast --test --skip integration --skip jormungandr-integration ``` #### Result: ``` ... Generating unified report 26% expressions used (26186/98111) 44% boolean coverage (136/305) 42% guards (102/240), 72 always True, 7 always False, 59 unevaluated 52% 'if' conditions (33/63), 4 always True, 8 always False, 18 unevaluated 50% qualifiers (1/2), 1 always True 40% alternatives used (849/2108) 58% local declarations used (859/1456) 50% top-level declarations used (1769/3533) The unified report is available at /home/rodney/iohk/cardano-wallet/.stack-work/install/x86_64-linux/2cecc28bf3aab8c8c3e4a07c1c6c1c846ec8861df3d8a5e9247bce185aeb7542/8.6.5/hpc/combined/all/hpc_index.html An index of the generated HTML coverage reports is available at /home/rodney/iohk/cardano-wallet/.stack-work/install/x86_64-linux/2cecc28bf3aab8c8c3e4a07c1c6c1c846ec8861df3d8a5e9247bce185aeb7542/8.6.5/hpc/index.html -- While building package cardano-wallet-2020.9.30 using: /home/rodney/.stack/setup-exe-cache/x86_64-linux/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.5 --builddir=.stack-work/dist/x86_64-linux/Cabal-2.4.0.1 build lib:cardano-wallet exe:cardano-wallet test:unit --ghc-options "-hpcdir .stack-work/dist/x86_64-linux/Cabal-2.4.0.1/hpc -fdiagnostics-color=always" Process exited with code: ExitFailure 1 Progress 335/336 ``` </details> 2223: Enable 100 wallet scenario in latency benchmark r=Anviking a=Anviking # Issue Number ADP-469, ADP-473 # Overview - [x] Simply enable `Latencies for 100 fixture wallets` - [x] Delete all wallets in between all latency scenarios - [x] Rename DSL helper `tearDown` to `deleteAllWallets`, and re-use it more # Comments ``` $ stack bench cardano-wallet Benchmark latency: RUNNING... Non-cached run getNetworkInfo - 20.7 ms Latencies for 2 fixture wallets scenario listWallets - 1.4 ms getWallet - 0.5 ms getUTxOsStatistics - 0.4 ms listAddresses - 1.0 ms listTransactions - 2.2 ms postTransactionFee - 86.4 ms listStakePools - 1.2 ms getNetworkInfo - 0.0 ms Latencies for 10 fixture wallets scenario listWallets - 4.8 ms getWallet - 0.4 ms getUTxOsStatistics - 0.3 ms listAddresses - 0.9 ms listTransactions - 2.0 ms postTransactionFee - 93.2 ms listStakePools - 1.3 ms getNetworkInfo - 0.0 ms Latencies for 100 fixture wallets listWallets - 59.9 ms getWallet - 0.5 ms getUTxOsStatistics - 0.4 ms listAddresses - 1.2 ms listTransactions - 2.6 ms postTransactionFee - 139.2 ms listStakePools - 1.1 ms getNetworkInfo - 0.0 ms Latencies for 2 fixture wallets with 10 txs scenario listWallets - 1.1 ms getWallet - 0.5 ms getUTxOsStatistics - 0.4 ms listAddresses - 0.9 ms listTransactions - 3.8 ms postTransactionFee - 104.2 ms listStakePools - 1.3 ms getNetworkInfo - 0.1 ms Latencies for 2 fixture wallets with 20 txs scenario listWallets - 1.1 ms getWallet - 0.5 ms getUTxOsStatistics - 0.4 ms listAddresses - 1.0 ms listTransactions - 5.5 ms postTransactionFee - 97.0 ms listStakePools - 1.2 ms getNetworkInfo - 0.1 ms Latencies for 2 fixture wallets with 100 txs scenario listWallets - 1.1 ms getWallet - 0.5 ms getUTxOsStatistics - 0.4 ms listAddresses - 1.0 ms listTransactions - 14.5 ms postTransactionFee - 95.1 ms listStakePools - 1.7 ms getNetworkInfo - 0.1 ms Latencies for 10 fixture wallets with 10 txs scenario listWallets - 4.4 ms getWallet - 0.5 ms getUTxOsStatistics - 0.3 ms listAddresses - 0.9 ms listTransactions - 4.3 ms postTransactionFee - 106.6 ms listStakePools - 1.3 ms getNetworkInfo - 0.0 ms Latencies for 10 fixture wallets with 20 txs scenario listWallets - 4.2 ms getWallet - 0.4 ms getUTxOsStatistics - 0.3 ms listAddresses - 0.9 ms listTransactions - 4.5 ms postTransactionFee - 106.6 ms listStakePools - 1.3 ms getNetworkInfo - 0.1 ms Latencies for 10 fixture wallets with 100 txs scenario listWallets - 4.5 ms getWallet - 0.5 ms getUTxOsStatistics - 0.4 ms listAddresses - 0.9 ms listTransactions - 15.8 ms postTransactionFee - 96.8 ms listStakePools - 1.6 ms getNetworkInfo - 0.1 ms Latencies for 2 fixture wallets with 100 utxos scenario listWallets - 1.7 ms getWallet - 0.5 ms getUTxOsStatistics - 0.4 ms listAddresses - 1.0 ms listTransactions - 12.2 ms postTransactionFee - 95.9 ms listStakePools - 1.4 ms getNetworkInfo - 0.0 ms Latencies for 2 fixture wallets with 200 utxos scenario listWallets - 1.7 ms getWallet - 0.5 ms getUTxOsStatistics - 0.4 ms listAddresses - 1.0 ms listTransactions - 25.1 ms postTransactionFee - 96.6 ms listStakePools - 1.4 ms getNetworkInfo - 0.1 ms Latencies for 2 fixture wallets with 500 utxos scenario listWallets - 1.6 ms getWallet - 0.5 ms getUTxOsStatistics - 0.4 ms listAddresses - 1.0 ms listTransactions - 69.7 ms postTransactionFee - 87.9 ms listStakePools - 1.5 ms getNetworkInfo - 0.0 ms Latencies for 2 fixture wallets with 1000 utxos scenario ```   Co-authored-by: Rodney Lorrimar <[email protected]> Co-authored-by: Samuel Evans-Powell <[email protected]> Co-authored-by: Johannes Lund <[email protected]>

iohk-bors · 2020-10-09T09:50:08Z

Build failed (retrying...):

buildkite/cardano-wallet

2124: Add haskell program coverage reports to Hydra tests r=rvl a=rvl ### Issue Number ADP-99 ### Overview This is a bit of an assortment of nix build improvements. 1. Add a code test coverage report for the Hydra build - implemented by PR input-output-hk/haskell.nix#762 2. Add a nix-shell with profiled packages. Profiled packages will be built on Hydra for master branch but not PRs. This means you can download haskell dependencies with profiling enabled, rather than having to build everything yourself. 3. Update versions of build tools in the nix-shell to latest hackage release - ghcide, hlint and stylish-haskell updated. ### Comments - [Hydra jobset](https://hydra.iohk.io/jobset/Cardano/cardano-wallet-pr-2124) - [Coverage report job](https://hydra.iohk.io/job/Cardano/cardano-wallet-pr-2124/musl64.testCoverageReport.x86_64-linux/latest) - [Coverage report from bors try](https://hydra.iohk.io/build/4328848/download/2/hpc_index.html) <details> <summary>Stack coverage report for comparison</summary> #### Command ``` stack build --coverage --fast --test --skip integration --skip jormungandr-integration ``` #### Result: ``` ... Generating unified report 26% expressions used (26186/98111) 44% boolean coverage (136/305) 42% guards (102/240), 72 always True, 7 always False, 59 unevaluated 52% 'if' conditions (33/63), 4 always True, 8 always False, 18 unevaluated 50% qualifiers (1/2), 1 always True 40% alternatives used (849/2108) 58% local declarations used (859/1456) 50% top-level declarations used (1769/3533) The unified report is available at /home/rodney/iohk/cardano-wallet/.stack-work/install/x86_64-linux/2cecc28bf3aab8c8c3e4a07c1c6c1c846ec8861df3d8a5e9247bce185aeb7542/8.6.5/hpc/combined/all/hpc_index.html An index of the generated HTML coverage reports is available at /home/rodney/iohk/cardano-wallet/.stack-work/install/x86_64-linux/2cecc28bf3aab8c8c3e4a07c1c6c1c846ec8861df3d8a5e9247bce185aeb7542/8.6.5/hpc/index.html -- While building package cardano-wallet-2020.9.30 using: /home/rodney/.stack/setup-exe-cache/x86_64-linux/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.5 --builddir=.stack-work/dist/x86_64-linux/Cabal-2.4.0.1 build lib:cardano-wallet exe:cardano-wallet test:unit --ghc-options "-hpcdir .stack-work/dist/x86_64-linux/Cabal-2.4.0.1/hpc -fdiagnostics-color=always" Process exited with code: ExitFailure 1 Progress 335/336 ``` </details> Co-authored-by: Rodney Lorrimar <[email protected]> Co-authored-by: Samuel Evans-Powell <[email protected]>

iohk-bors · 2020-10-09T15:17:56Z

Timed out.

Includes Samuel's coverage PR. input-output-hk/haskell.nix#762

KtorZ · 2020-10-16T13:58:25Z

bors merge

iohk-bors · 2020-10-16T16:16:08Z

Build succeeded:

rvl self-assigned this Sep 9, 2020

rvl force-pushed the rvl/test-coverage-reports branch 2 times, most recently from 65ab60a to 40e7a7a Compare October 1, 2020 05:30

rvl added the Improvement Mark a PR as an improvement, for auto-generated CHANGELOG label Oct 1, 2020

rvl requested a review from sevanspowell October 1, 2020 05:34

rvl marked this pull request as ready for review October 1, 2020 05:44

rvl marked this pull request as draft October 1, 2020 07:22

sevanspowell approved these changes Oct 2, 2020

View reviewed changes

rvl force-pushed the rvl/test-coverage-reports branch from fd464f7 to 21659a2 Compare October 6, 2020 03:33

rvl marked this pull request as ready for review October 6, 2020 03:33

iohk-bors bot added a commit that referenced this pull request Oct 6, 2020

Try #2124:

25dbcdc

rvl force-pushed the rvl/test-coverage-reports branch from 2d1ad8c to 1833f9d Compare October 9, 2020 06:10

rvl and others added 8 commits October 16, 2020 15:57

nix: Add shell with profiled haskell packages

ad67e13

release.nix: Fix sourcesOverride

57672a0

Bump iohk-nix to latest master

5e41f7e

Update iohk-nix cardano deployments

97f0976

Bump haskell.nix to latest master

87666d5

Includes Samuel's coverage PR. input-output-hk/haskell.nix#762

Build tests with Haskell Program Coverage enabled

f776685

Add stock coverage reports

51951bd

nix: Update ghcide, hlint, stylish-haskell versions

a853fe8

rvl added 7 commits October 16, 2020 15:57

release.nix: Filter test coverage report to be musl linux only

c70a7db

Fix up Haskell.nix usage and add Hydra job for coverage

30eb486

Remove old script

1aa2585

nix: Remove upstreamed code

f560971

Fix new HLint errors

95c37bc

Use shorter cross-compiler friendly HLint annotations

23e9257

nix-shell: Replace ghcide-0.4.0 with haskell-language-server-0.5.0

e2c02b0

KtorZ force-pushed the rvl/test-coverage-reports branch from 1833f9d to e2c02b0 Compare October 16, 2020 13:58

iohk-bors bot merged commit 05e59a0 into master Oct 16, 2020

iohk-bors bot deleted the rvl/test-coverage-reports branch October 16, 2020 16:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add haskell program coverage reports to Hydra tests #2124

Add haskell program coverage reports to Hydra tests #2124

Uh oh!

rvl commented Sep 9, 2020 •

edited

Loading

Uh oh!

sevanspowell left a comment

Uh oh!

rvl commented Oct 6, 2020

Uh oh!

iohk-bors bot commented Oct 6, 2020

Uh oh!

rvl commented Oct 9, 2020

Uh oh!

iohk-bors bot commented Oct 9, 2020

Uh oh!

rvl commented Oct 9, 2020

Uh oh!

iohk-bors bot commented Oct 9, 2020

Uh oh!

iohk-bors bot commented Oct 9, 2020

Uh oh!

KtorZ commented Oct 16, 2020

Uh oh!

iohk-bors bot commented Oct 16, 2020

Uh oh!

Uh oh!

Add haskell program coverage reports to Hydra tests #2124

Add haskell program coverage reports to Hydra tests #2124

Uh oh!

Conversation

rvl commented Sep 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue Number

Overview

Comments

Command

Result:

Uh oh!

sevanspowell left a comment

Choose a reason for hiding this comment

Uh oh!

rvl commented Oct 6, 2020

Uh oh!

iohk-bors bot commented Oct 6, 2020

try

Uh oh!

rvl commented Oct 9, 2020

Uh oh!

iohk-bors bot commented Oct 9, 2020

Uh oh!

rvl commented Oct 9, 2020

Uh oh!

iohk-bors bot commented Oct 9, 2020

Uh oh!

iohk-bors bot commented Oct 9, 2020

Uh oh!

KtorZ commented Oct 16, 2020

Uh oh!

iohk-bors bot commented Oct 16, 2020

Uh oh!

Uh oh!

rvl commented Sep 9, 2020 •

edited

Loading