Long Range Queries from Rule component store return weird results

[davtsur]

I think that this relates to a similar issue #396 that was resolved in #718
Rule component uses : pkg/store/tsdb.go , so it still suffers from the same problem.

Thanos version 0.3.2
Prometheus version 2.5.0
Golang version go1.11.5

What happened

Long range query from the rule component store for a time period of a month returns partial results .

How to reproduce it (as minimally and precisely as possible):

When executing a query for a time period of a month from data that exists in the rule component store this will happen always.

This is how it looks when executing the query for a period of 3 weeks:

And this is for 4 weeks:

[bwplotka]

That is most likely same bug but on Ruler indeed 👍 Looking now on potential fix.

[bwplotka]

Wrote:

func TestTSDBStore_Series_SplitSamplesIntoChunksWithMaxSizeOfUint16_e2e(t *testing.T) {
	defer leaktest.CheckTimeout(t, 10*time.Second)()

	db, err := testutil.NewTSDB()
	defer func() { testutil.Ok(t, db.Close()) }()
	testutil.Ok(t, err)

	baseT := timestamp.FromTime(time.Now().AddDate(0, 0, -2)) / 1000 * 1000

	a := db.Appender()

	offset := int64(2*math.MaxUint16 + 5)
	for i := int64(0); i < offset; i++ {
		_, err = a.Add(labels.FromStrings("a", "b"), baseT+i, 1)
		testutil.Ok(t, err)
	}

	testutil.Ok(t, a.Commit())

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	tsdbStore := NewTSDBStore(nil, nil, db, component.Rule, labels.FromStrings("region", "eu-west"))

	srv := newStoreSeriesServer(ctx)
	testutil.Ok(t, tsdbStore.Series(&storepb.SeriesRequest{
		MinTime: baseT,
		MaxTime: baseT + offset,
		Matchers: []storepb.LabelMatcher{
			{Type: storepb.LabelMatcher_EQ, Name: "a", Value: "b"},
			{Type: storepb.LabelMatcher_EQ, Name: "region", Value: "eu-west"},
		},
	}, srv))
	testutil.Equals(t, 1, len(srv.SeriesSet))

	firstSeries := srv.SeriesSet[0]

	testutil.Equals(t, []storepb.Label{
		{Name: "a", Value: "b"},
		{Name: "region", Value: "eu-west"},
	}, firstSeries.Labels)

	testutil.Equals(t, 3, len(firstSeries.Chunks))

	chunk, err := chunkenc.FromData(chunkenc.EncXOR, firstSeries.Chunks[0].Raw.Data)
	testutil.Ok(t, err)
	testutil.Equals(t, math.MaxUint16, chunk.NumSamples())

	chunk, err = chunkenc.FromData(chunkenc.EncXOR, firstSeries.Chunks[1].Raw.Data)
	testutil.Ok(t, err)
	testutil.Equals(t, math.MaxUint16, chunk.NumSamples())

	chunk, err = chunkenc.FromData(chunkenc.EncXOR, firstSeries.Chunks[2].Raw.Data)
	testutil.Ok(t, err)
	testutil.Equals(t, 5, chunk.NumSamples())
}

And it fails, which confirms that it is same bug. Fixing.

[davtsur]

Great thanks! , btw when is the next minor release expected?

[bwplotka]

ASAP we are in full momentum for reviewing last features ): 0.4.0 coming. We might go for rc.0 first maybe.

[bwplotka]

Fix #1041 @davtsur PTAL