Linearizable reads/writes

[srh]

(We didn't have an issue for this.) The feature we want is for read behavior to be linearizable, instead of happily reading outdated data.

The short of it is, we're going to use etcd/raft's ReadIndex functionality. To avoid spamming Raft groups' leaders with MsgReadIndex messages, each follower node will only make one in-flight request at a time. We'll do this from whatever node serves our Raft group.

That's the most basic version -- we could get more clever about how we do ReadIndex in the future to reduce latency. We could allow multiple in-flight requests (and take care not to let them bunch up).
We could send our requests from other machines to avoid extra network hops. Some descriptions of alternatives are here: https://discuss.dgraph.io/t/linearizability/1683/3

We will also need to change the watermark implementation to not use sleep.

[srh]

So, linearizability is implemented (albeit under-tested, and the simplest possible implementation as described). In the branch sam/new1344.

One thing that isn't implemented, though is the question, what should the UI be? How should the user specify a linearizable read vs. a non-linearizable one?

Option 1: an out-of-band flag.
Option 2: the flag gets parsed out of the query/schema read query somehow.
Option 3: it's not a mere flag -- the user could supply a {groupId => readIndex} bundle.

[manishrjain]

All reads should be linearizable for now. If performance becomes an issue, we'll solve for it when it does.

[srh]

I'm getting 14 second reads with a simple curl query while running dgraphloader. Running the Raft ReadIndex operation is quick, but then it takes about 12 seconds between getting a linearizable read index and for waitForAppliedMark to return. It looks like the time duration between a change reaching a majority of nodes and actually being "applied" can be large.

[manishrjain]

It looks like the time duration between a change reaching a majority of nodes and actually being "applied" can be large.

This is because we're using goroutines for each mutation which get scheduled whenever they do. @janardhan1993 is working on a smart scheduler which can prioritize mutations in a roughly FIFO order. That should reduce the applied latency.

But, great to hear read linearizability is working! PR?

[janardhan1993]

To minimise the latency, i guess we need to schedule all the mtuations which came after read request only when the applied watermark catches up.

[srh]

Linearizability won't be fully working until #1345 (for which a basic fix has just been implemented) because it does in fact trigger that scenario sometimes.

[srh]

#1345 was fixed, so this can be closed.