Compactor/bucket utility: overlap error on non-overlapping data blocks

[ivan-kiselev]

Hello! Opening an issue following a slack discussion with @bwplotka: https://improbable-eng.slack.com/archives/CA4UWKEEN/p1552312504323600

Thanos, Prometheus and Golang version used
Thanos bucket util:

$ thanos --version
thanos, version 0.3.2 (branch: HEAD, revision: 4b7320c0e45e3f48a437bd19294f569785bafb02)
  build user:       root@e9a9c28f966a
  build date:       20190304-17:11:48
  go version:       go1.11.5

Thanos compactor: v0.3.2
Prometheus: 2.7.0
What happened
Overlap error on non-overlapping blocks:

What you expected to happen

No overlap error

Full logs to relevant components
level=warn ts=2019-03-11T14:18:37.555442Z caller=overlapped_blocks.go:38 msg="found overlapped blocks" group="0@{some_labels}" overlap="[mint: 1550620800000, maxt: 1550620800000, range: 0s, blocks: 2]: <ulid: 01D44DGZYN912SA0FG0TRTR19Z, mint: 1550620800000, maxt: 1550628000000, range: 2h0m0s>, <ulid: 01D44FSTJC2BR2YXZZK124YZF0, mint: 1550592000000, maxt: 1550620800000, range: 8h0m0s>"

Inspect output related to mentioned in logs blocks:

|            ULID            |        FROM         |        UNTIL        |     RANGE     |  UNTIL-COMP   |  #SERIES  |   #SAMPLES    |  #CHUNKS   | COMP-LEVEL | COMP-FAILED |       LABELS         | RESOLUTION |  SOURCE   |
|----------------------------|---------------------|---------------------|---------------|---------------|-----------|---------------|------------|------------|-------------|----------------------|------------|-----------|
| 01D44DGZYN912SA0FG0TRTR19Z | 20-02-2019 01:00:00 | 20-02-2019 03:00:00 | 2h0m0s        | 38h0m0s       | 244,585   | 158,944,050   | 1,148,746  | 1          | false       | some_matching_labels | 0s         | sidecar   |
| 01D44FSTJC2BR2YXZZK124YZF0 | 19-02-2019 17:00:00 | 20-02-2019 01:00:00 | 8h0m0s        | 32h0m0s       | 1,008,062 | 1,087,766,037 | 9,837,618  | 2          | false       | some_matching_labels | 0s         | compactor |

As you can see - one block starts at 20-02-2019 01:00:00 and another ends at this moment. Doesn't sound like overlap to me, but probably I'm wrong.

Anything else we need to know

I noticed an overlap problem in thanos compactor logs and then dug into thanos bucket outputs. The example that I put here is not what I see in thanos compactor logs, but since compactor halts at the moment it discovered first overlap error - it hasn't probably managed to meet mentioned here problem before halting.

The problem and log I describe here is taken from thanos bucket utility.

[bwplotka]

0 idea what's there, We can try to reproduce it in some unit test - we have those: https://github.com/prometheus/tsdb/blob/master/db_test.go#L1134

I tried to change things to put metas as you have and all works here. I feel like the overlap is on different thing but stats is showing this? There must be bug somewhere ):