verify: Added extra statistics for block index; Printable on debug; Interesting results

[bwplotka]

Was playing around a couple of statistics of real block indexes and this is crazy!

For example one customer's block from our production at Red Hat has following stats:

ts=2020-10-30T16:05:15.969081066Z caller=level.go:63 level=debug verifiers=index_known_issues verifier=index_known_issues stats="{TotalSeries:8377876 OutOfOrderSeries:0 OutOfOrderChunks:0 DuplicatedChunks:0 OutsideChunks:0 CompleteOutsideChunks:0 Issue347OutsideChunks:0 OutOfOrderLabels:0 SeriesMinLifeDuration:0s SeriesAvgLifeDuration:13h7m1.399s SeriesMaxLifeDuration:311h50m40.379s SeriesMinLifeDurationWithoutSingleSampleSeries:14.91s SeriesAvgLifeDurationWithoutSingleSampleSeries:13h46m22.193s SeriesMaxLifeDurationWithoutSingleSampleSeries:311h50m40.379s SeriesMinChunks:1 SeriesAvgChunks:8 SeriesMaxChunks:165 TotalChunks:67874256 ChunkMinDuration:0s ChunkAvgDuration:1h37m8.646s ChunkMaxDuration:1h59m56.586s ChunkMinSize:23 ChunkAvgSize:135 ChunkMaxSize:1209 SingleSampleSeries:398902 SingleSampleChunks:1330558}" id=01DN3SK96XDAEKRB1AN30AAW6E

We can actually learn a lot here. This is a non-downsampled, 2-week block that has 8.3 Million of series.

Interesting things we can find from this 🤯 👇

• Despite block being 2 weeks long, avg lifetime of series is just SeriesAvgLifeDuration:13h7m1.399s
• Biggest chunk is just over 1KB
• There are tons of 1 sample chunks and whole series with just 1 sample: SingleSampleSeries:398902 SingleSampleChunks:1330558.. 400k of 8.3M series have one sample, thus most likely are not usable / not graphable. This might mean some sneaky app using Prometheus as event-based system. 🤦

Wonder what your blocs look like, so shipping this code so everyone can check. Even usable for blocks in Prometheus local disk e.g:

#!/usr/bin/env bash

thanos tools bucket verify --log.level debug -i "index_known_issues" --objstore.config "
type: FILESYSTEM
config:
  directory: /<prometheus_local_dir>
"

Is that healthy? @brian-brazil @pracucci @kakkoyun @s-urbaniak @gouthamve @codesome @smarterclayton ;p

Signed-off-by: Bartlomiej Plotka bwplotka@gmail.com

[kakkoyun]

Looks good. I think there are some unintended Config->Context transformations.

[bwplotka]

Fixed 🤗 Thanks @kakkoyun for super quick review

[s-urbaniak]

@bwplotka i think this is wonderful! 🎉 questions: could this data be a) machine-retrievable? and b) available in the UI?

[bwplotka]

could this data be a) machine-retrievable? and b) available in the UI?

a) you mean API? Why not, feel free to add an issue for it 🤗 (This API would need to download block index though)

b) Of what service? There is plan to have BucketViewer to have those features (cc @kunal-kushwaha) and to migrate such view to Prometheus

[pracucci]

This is very interesting. Thanks!