No datapoints for queries with a window smaller than 105h with auto-downsampling selected

[antonio]

Thanos, Prometheus and Golang version used

thanos, version 0.3.2 (branch: master, revision: 8e53a42515c591bb5fb58fa38d795f146704034d)
  build user:       root@packages-build-f2e9be33-d451-4201-a013-97a6f0686e26-l995n
  build date:       20190430-09:03:56
  go version:       go1.10.1

This is a custom version, patched with #1094

Prometheus 2.9.2

What happened

We've configured the compactor to keep raw data for 14 days and delete it afterwards, and the query component to auto-downsample data. For queries that span over a week the results look like the ones we expected when compared with another setup in which we have kept raw data. Now, when the query window is smaller than 105h we are not getting any results back, unless we specifically force it in the UI to use 5m downsampled.

What you expected to happen

That the data that does exist would be returned, even though it has a lower resolution.

How to reproduce it (as minimally and precisely as possible):

• Duplicate a bucket.
• Compact the new bucket with a configuration that removes raw data after 14 days and keeps the rest.
• Have another environment use the new bucket.
• Query both environments to see the difference.

Full logs to relevant components

The logs don't show anything odd, at least with the info level. I'm going to restart the store with debug and will add whatever is interesting here, if there's anything.

Anything else we need to know

Nothing I can think of right now.

[antonio]

Having read a few issues in this repo about downsampling, I think this is expected behavior and would like to ask for a confirmation. I'm opening this issue in case someone else is experiencing the same (now, or in the future!).

I wouldn't be surprised if the resolution is that I'm not understanding something correctly and/or doing something wrong though! 😄

[bwplotka]

Thanks for report. I think similar issue was reported somewhere but would be nice to narrow it down.

I think it must be something wrong with logic that chooses which blocks to use (falling back to 5m and then to raw): https://github.com/improbable-eng/thanos/blob/3b6a73bebd200d81a386d3b7cc4a0b2d0ce66abc/pkg/store/bucket.go#L1000

Does it makes sense?
Marking us bug.

[bwplotka]

Also be aware of this: #1104 which will help to debug this I guess

[bwplotka]

Fixes ahead:

• https://github.com/improbable-eng/thanos/pull/1146/files/9c9491843a1a335d18c4ed33fee5291ef5cfcc28..cd3363cfca26a9db80aceeecad57182e60b8557a
• query/api: properly pass downsampling param #1144

[bwplotka]

Should be fixed by #1146