Skip to content

System views creation in SchemeShard under FeatureFlag #19377

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

n00bcracker
Copy link
Collaborator

@n00bcracker n00bcracker commented Jun 5, 2025

Changelog entry

  • Created the set of system views in SchemeShard during init stage based on system view descriptions in the code.
  • For this experimental feature a new feature flag EnableRealSystemViewPaths was added
  • To support enabling added feature flag some related tests were made robust to the initial paths creation.

Changelog category

  • Not for changelog (changelog entry is not required)

Description for reviewers

Techical details:

  • For creating sysview paths used a dedicated actor (created during init stage of SchemeShard)
  • Collected .sys dir contents before launch that actor
  • Formed a list of changes and pass it into that actor
  • Fixed all depended tests (disable flag in tests based on counting dir entries and make another version of test for tests which checks system views themself)

@n00bcracker n00bcracker force-pushed the sysview_scheme_shard_sync branch 2 times, most recently from c48496d to 1d8e7a1 Compare June 7, 2025 16:41

This comment was marked as outdated.

This comment was marked as outdated.

@n00bcracker n00bcracker force-pushed the sysview_scheme_shard_sync branch from 1d8e7a1 to 4c43d1f Compare June 9, 2025 13:40

This comment was marked as outdated.

This comment was marked as outdated.

@n00bcracker n00bcracker force-pushed the sysview_scheme_shard_sync branch from 4c43d1f to c49727a Compare June 10, 2025 13:48

This comment was marked as outdated.

This comment was marked as outdated.

@n00bcracker n00bcracker force-pushed the sysview_scheme_shard_sync branch from c49727a to 0bf2b33 Compare June 11, 2025 14:17

This comment was marked as outdated.

This comment was marked as outdated.

@n00bcracker n00bcracker force-pushed the sysview_scheme_shard_sync branch from 0bf2b33 to db7b127 Compare June 11, 2025 16:42

This comment was marked as outdated.

This comment was marked as outdated.

@n00bcracker n00bcracker force-pushed the sysview_scheme_shard_sync branch 3 times, most recently from 219498b to 17582b4 Compare June 14, 2025 18:55

This comment was marked as outdated.

This comment was marked as outdated.

@n00bcracker n00bcracker force-pushed the sysview_scheme_shard_sync branch 2 times, most recently from 357372b to ef10bd7 Compare June 16, 2025 14:56

This comment was marked as outdated.

This comment was marked as outdated.

@n00bcracker n00bcracker force-pushed the sysview_scheme_shard_sync branch from ef10bd7 to 9bab503 Compare June 17, 2025 18:42

This comment was marked as outdated.

Copy link

github-actions bot commented Jun 25, 2025

2025-06-25 20:30:10 UTC Pre-commit check linux-x86_64-release-asan for 322527a has started.
2025-06-25 20:30:21 UTC Artifacts will be uploaded here
2025-06-25 20:34:07 UTC ya make is running...
🔴 2025-06-25 23:44:04 UTC Too many tests failed, NOT going to retry
🟡 2025-06-26 00:15:02 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16408 13853 0 2296 210 49

🟢 2025-06-26 00:17:16 UTC Build successful.
🟡 2025-06-26 00:18:28 UTC ydbd size 3.9 GiB changed* by +344.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 82ee38f merge: 322527a diff diff %
ydbd size 4 192 702 480 Bytes 4 193 055 160 Bytes +344.4 KiB +0.008%
ydbd stripped size 1 453 443 224 Bytes 1 453 526 328 Bytes +81.2 KiB +0.006%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@n00bcracker n00bcracker force-pushed the sysview_scheme_shard_sync branch from e487e8e to 7d4729f Compare June 26, 2025 10:31
Copy link

github-actions bot commented Jun 26, 2025

2025-06-26 10:33:58 UTC Pre-commit check linux-x86_64-release-asan for eb38d1c has started.
2025-06-26 10:34:10 UTC Artifacts will be uploaded here
2025-06-26 10:38:08 UTC ya make is running...
🟡 2025-06-26 13:10:08 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16433 16014 0 122 263 34

🟢 2025-06-26 13:11:38 UTC Build successful.
🟡 2025-06-26 13:12:07 UTC ydbd size 3.9 GiB changed* by +348.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 093001b merge: eb38d1c diff diff %
ydbd size 4 192 778 160 Bytes 4 193 135 368 Bytes +348.8 KiB +0.009%
ydbd stripped size 1 453 500 856 Bytes 1 453 588 664 Bytes +85.8 KiB +0.006%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jun 26, 2025

2025-06-26 10:35:44 UTC Pre-commit check linux-x86_64-relwithdebinfo for eb38d1c has started.
2025-06-26 10:35:49 UTC Artifacts will be uploaded here
2025-06-26 10:39:48 UTC ya make is running...
🟡 2025-06-26 12:29:40 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
38854 36133 0 6 2677 38

2025-06-26 12:33:03 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-06-26 12:46:26 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
930 (only retried tests) 819 0 1 73 37

2025-06-26 12:46:38 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-06-26 12:59:21 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
306 (only retried tests) 272 0 0 3 31

🟢 2025-06-26 12:59:30 UTC Build successful.
🟡 2025-06-26 12:59:53 UTC ydbd size 2.2 GiB changed* by +213.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 093001b merge: eb38d1c diff diff %
ydbd size 2 383 122 576 Bytes 2 383 340 704 Bytes +213.0 KiB +0.009%
ydbd stripped size 498 976 712 Bytes 499 002 696 Bytes +25.4 KiB +0.005%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jun 26, 2025

2025-06-26 19:26:06 UTC Pre-commit check linux-x86_64-release-asan for eb61c09 has started.
2025-06-26 19:26:19 UTC Artifacts will be uploaded here
2025-06-26 19:30:04 UTC ya make is running...
🟡 2025-06-26 22:11:27 UTC ydbd size 3.9 GiB changed* by +343.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 935b735 merge: eb61c09 diff diff %
ydbd size 4 193 756 216 Bytes 4 194 108 416 Bytes +343.9 KiB +0.008%
ydbd stripped size 1 453 748 792 Bytes 1 453 830 456 Bytes +79.8 KiB +0.006%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation
2025-06-26 22:11:29 UTC Check cancelled

Copy link

github-actions bot commented Jun 26, 2025

2025-06-26 19:26:06 UTC Pre-commit check linux-x86_64-relwithdebinfo for eb61c09 has started.
2025-06-26 19:26:20 UTC Artifacts will be uploaded here
2025-06-26 19:30:09 UTC ya make is running...
🟡 2025-06-26 21:21:59 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39173 36412 0 54 2669 38

2025-06-26 21:25:21 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-06-26 21:36:12 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
310 (only retried tests) 234 0 48 2 26

2025-06-26 21:36:23 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-06-26 21:46:33 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
152 (only retried tests) 79 0 47 2 24

🟢 2025-06-26 21:46:41 UTC Build successful.
🟡 2025-06-26 21:47:04 UTC ydbd size 2.2 GiB changed* by +219.7 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 935b735 merge: eb61c09 diff diff %
ydbd size 2 383 680 216 Bytes 2 383 905 160 Bytes +219.7 KiB +0.009%
ydbd stripped size 499 053 288 Bytes 499 087 016 Bytes +32.9 KiB +0.007%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@n00bcracker n00bcracker force-pushed the sysview_scheme_shard_sync branch from 8254eda to ab18bcf Compare June 26, 2025 22:06
Copy link

github-actions bot commented Jun 26, 2025

2025-06-26 22:15:16 UTC Pre-commit check linux-x86_64-release-asan for 2be5dc9 has started.
2025-06-26 22:15:29 UTC Artifacts will be uploaded here
2025-06-26 22:19:27 UTC ya make is running...
🔴 2025-06-27 01:33:09 UTC Too many tests failed, NOT going to retry
🟡 2025-06-27 01:56:03 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16636 14015 0 2306 267 48

🟢 2025-06-27 01:58:22 UTC Build successful.
🟡 2025-06-27 01:59:28 UTC ydbd size 3.9 GiB changed* by +343.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 9a3ba4f merge: 2be5dc9 diff diff %
ydbd size 4 193 756 216 Bytes 4 194 108 416 Bytes +343.9 KiB +0.008%
ydbd stripped size 1 453 748 792 Bytes 1 453 830 456 Bytes +79.8 KiB +0.006%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jun 26, 2025

2025-06-26 22:15:27 UTC Pre-commit check linux-x86_64-relwithdebinfo for 2be5dc9 has started.
2025-06-26 22:15:39 UTC Artifacts will be uploaded here
2025-06-26 22:19:29 UTC ya make is running...
🟡 2025-06-26 23:59:17 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39173 36439 0 25 2669 40

2025-06-27 00:02:41 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-06-27 00:18:01 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
286 (only retried tests) 236 0 18 2 30

2025-06-27 00:18:11 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-06-27 00:33:22 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
192 (only retried tests) 135 0 25 4 28

🟢 2025-06-27 00:33:30 UTC Build successful.
🟡 2025-06-27 00:33:55 UTC ydbd size 2.2 GiB changed* by +219.7 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 9a3ba4f merge: 2be5dc9 diff diff %
ydbd size 2 383 680 216 Bytes 2 383 905 160 Bytes +219.7 KiB +0.009%
ydbd stripped size 499 053 288 Bytes 499 087 016 Bytes +32.9 KiB +0.007%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@n00bcracker n00bcracker force-pushed the sysview_scheme_shard_sync branch from ab18bcf to fa219e9 Compare June 27, 2025 07:35
Copy link

github-actions bot commented Jun 27, 2025

2025-06-27 07:38:39 UTC Pre-commit check linux-x86_64-release-asan for 8521c5a has started.
2025-06-27 07:38:52 UTC Artifacts will be uploaded here
2025-06-27 07:42:44 UTC ya make is running...
🟡 2025-06-27 10:14:36 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16649 16232 0 123 265 29

🟢 2025-06-27 10:16:07 UTC Build successful.
🟡 2025-06-27 10:16:35 UTC ydbd size 3.9 GiB changed* by +342.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: de8d3ca merge: 8521c5a diff diff %
ydbd size 4 193 756 152 Bytes 4 194 106 896 Bytes +342.5 KiB +0.008%
ydbd stripped size 1 453 748 728 Bytes 1 453 829 496 Bytes +78.9 KiB +0.006%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jun 27, 2025

2025-06-27 07:38:49 UTC Pre-commit check linux-x86_64-relwithdebinfo for 8521c5a has started.
2025-06-27 07:39:02 UTC Artifacts will be uploaded here
2025-06-27 07:43:13 UTC ya make is running...
🟡 2025-06-27 09:38:12 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39173 36459 0 3 2670 41

2025-06-27 09:41:43 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-06-27 09:53:07 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
165 (only retried tests) 137 0 0 2 26

🟢 2025-06-27 09:53:15 UTC Build successful.
🟡 2025-06-27 09:53:36 UTC ydbd size 2.2 GiB changed* by +219.7 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: de8d3ca merge: 8521c5a diff diff %
ydbd size 2 383 680 200 Bytes 2 383 905 160 Bytes +219.7 KiB +0.009%
ydbd stripped size 499 053 352 Bytes 499 087 144 Bytes +33.0 KiB +0.007%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@n00bcracker n00bcracker force-pushed the sysview_scheme_shard_sync branch from fa219e9 to 9ef46dd Compare June 27, 2025 15:02

This comment was marked as outdated.

This comment was marked as outdated.

@n00bcracker n00bcracker force-pushed the sysview_scheme_shard_sync branch from 9ef46dd to 07fc392 Compare June 30, 2025 15:36
Copy link

github-actions bot commented Jun 30, 2025

2025-06-30 15:38:31 UTC Pre-commit check linux-x86_64-release-asan for 0ab3e33 has started.
2025-06-30 15:38:44 UTC Artifacts will be uploaded here
2025-06-30 15:42:41 UTC ya make is running...
🟡 2025-06-30 18:06:47 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16631 16224 0 155 223 29

🟢 2025-06-30 18:08:13 UTC Build successful.
🟡 2025-06-30 18:08:42 UTC ydbd size 3.9 GiB changed* by +348.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 8ff355e merge: 0ab3e33 diff diff %
ydbd size 4 195 987 704 Bytes 4 196 344 984 Bytes +348.9 KiB +0.009%
ydbd stripped size 1 454 289 176 Bytes 1 454 375 672 Bytes +84.5 KiB +0.006%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jun 30, 2025

2025-06-30 15:40:17 UTC Pre-commit check linux-x86_64-relwithdebinfo for 0ab3e33 has started.
2025-06-30 15:40:30 UTC Artifacts will be uploaded here
2025-06-30 15:44:17 UTC ya make is running...
🟡 2025-06-30 17:28:40 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39156 36439 0 6 2671 40

2025-06-30 17:32:02 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-06-30 17:42:39 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
272 (only retried tests) 243 0 1 2 26

2025-06-30 17:42:48 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-06-30 17:53:16 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
82 (only retried tests) 54 0 0 2 26

🟢 2025-06-30 17:53:23 UTC Build successful.
🟡 2025-06-30 17:53:46 UTC ydbd size 2.2 GiB changed* by +215.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 8ff355e merge: 0ab3e33 diff diff %
ydbd size 2 384 983 808 Bytes 2 385 204 256 Bytes +215.3 KiB +0.009%
ydbd stripped size 499 215 208 Bytes 499 244 904 Bytes +29.0 KiB +0.006%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link
Collaborator

@nikvas0 nikvas0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kqp looks ok

@n00bcracker n00bcracker merged commit 9c75f0c into ydb-platform:main Jul 1, 2025
17 of 18 checks passed
@n00bcracker n00bcracker deleted the sysview_scheme_shard_sync branch July 1, 2025 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create materialized system view paths in SchemeShard during SchemeShard init stage
4 participants