Night Statistics Monitoring

[SBOne-Kenobi]

Description

Problem

We want to develop and improve our product and, of course, there are some changes and its combinations which, according to some statistics, can make UTBot worse.

Solution

The main idea is collecting statistics after made changes. But it takes too long to collect statistics on a huge project to do it after each push into master. Thus, we will do it every night when no one makes changes. Also we need to visualize collected statistics to easily analyze them.

Look at docs/NightStatisticsMonitoring.md for more information.

Coverage

I decided to collect coverage information using our ConcreteExecutor and I've provided total instructions count from tested class.

Fixes #700

Type of Change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Manual Scenario

Created cloned repository and run github action. Also run each script with different parameters.

Checklist:

• The change followed the style guidelines of the UTBot project
• Self-review of the code is passed
• The change contains enough commentaries, particularly in hard-to-understand areas
• New documentation is provided or existed one is altered

[sergeypospelov]

PR looks really nice, can't wait to try it out. There are just some minor things. Also here some thoughts about possible enhancements:

• What about adding usedConcreteExecutors field in the future? 😊
• Maybe in the future we could write an interface to extend statistics with a custom statistic produced from UtExecution?
• What about adding a special file from which the StatisticsMonitoringKt::main takes arguments? It could be useful for defining custom monitoring runs without modifying .yml file.

[denis-fokin]

There are some suggestions on the approach to gather and store the statistics (Thanks to @AndreyAkinshin for the inspiration)

Where to store

It is better to store the statistical data in a separate branch or even repository. This way we can update the data without any harm to the main branch.

Metadata

Currently, there is not enough metadata for further data processing. Assuming, we store data in a separate repository or branch we may need the next data:

• commit hash
• build number
• environment

Images

Maybe our way of the results representation is not the best, so we should not stick to the particular images. Images are still useful for brief insight into the specific data, but they should not be stored as bitmap images. It is better to choose some vector format.

The way of representation of overall data with a single picture is a separate challenge. By the way, an independent repository will help us with refining the images.

Data processing

Eventually, it is great to have only data in the repository. There are several solutions on the data postprocessing:

• script in the repository
• service that is able to process the data and build visual representation of it

Granularity

We need to choose a smallest piece of information that we want to represent in our statistics. It seems reasonable to use a test as such an atomic measure. A common scenario that a user can run into is a change of generated tests number. This will lead to the change of the results. In such situations it is good to see the reason right from the statistical data.

Third party solutions

We can take a look at other's solutions:

• GitHub Action that works in similar way: https://github.com/benchmark-action/github-action-benchmark
• JSON file format: https://raw.githubusercontent.com/benchmark-action/github-action-benchmark/gh-pages/dev/bench/data.js
• Static dashboards implemented with ghpages: https://benchmark-action.github.io/github-action-benchmark/dev/bench/

Versioning

It is better to use separate files for each build than rewrite the same JSON file. The name of the file could be a hash of current time, agent, branch and so on.

It is crucial to store a timestamp in the JSON file.

JSON file must have its scheme version (in case we will change it).

We need to add a branch from which the statistics is gathered. This will help us to distinguish different branches (main, RC and so on). We can retrospectively apply exponential smoothing to the particular branch and compare the results with a feature branch before push into the main branch.