Joining back 3rd manager sometimes causes reelection

[tonistiigi]

In a three daemon cluster a leader is killed, 2 remaining nodes form a new leader. Now the old leader is started again, it shouldn't start a new election but sometimes it does(tested on a slow arm machine).

logs:

daemon1 d20073174 nodeid=2f7116c9f3135f00 https://gist.github.com/tonistiigi/985a6e0b90b5a94b6bb0639c880636cb#file-daemon1-d20073174-log
daemon2 d30297954 nodeid=feb61f32aa6d9a2 https://gist.github.com/tonistiigi/985a6e0b90b5a94b6bb0639c880636cb#file-daemon2-d30297954-log
daemon3 d35310795 nodeid=51dbe61bbca33471 https://gist.github.com/tonistiigi/985a6e0b90b5a94b6bb0639c880636cb#file-daemon3-d35310795-log

timeline:

2016-08-11T00:52:14.175173269Z daemon1(leader) is shut down https://gist.github.com/tonistiigi/985a6e0b90b5a94b6bb0639c880636cb#file-daemon1-d20073174-log-L214
2016-08-11T00:52:17.276697359Z daemon2 starts new election https://gist.github.com/tonistiigi/985a6e0b90b5a94b6bb0639c880636cb#file-daemon2-d30297954-log-L154
2016-08-11T00:52:17.290743959Z daemon2 becomes new leader https://gist.github.com/tonistiigi/985a6e0b90b5a94b6bb0639c880636cb#file-daemon2-d30297954-log-L167
2016-08-11T00:52:21.453692753Z daemon1 is restarted https://gist.github.com/tonistiigi/985a6e0b90b5a94b6bb0639c880636cb#file-daemon1-d20073174-log-L221
2016-08-11T00:52:29.206870451Z deamon1 kicks off new election(why?) https://gist.github.com/tonistiigi/985a6e0b90b5a94b6bb0639c880636cb#file-daemon1-d20073174-log-L370
from the logs it appears that daemon3 rejected to vote for daemon1 and no vote from daemon2
2016-08-11T00:52:31.288503039Z in daemon2 still for 10 sec all sends to daemon1 fail https://gist.github.com/tonistiigi/985a6e0b90b5a94b6bb0639c880636cb#file-daemon2-d30297954-log-L199
2016-08-11T00:52:33.226706096Z in daemon3 logs daemon2 has lost its leader status(nothing in daemon2 logs) https://gist.github.com/tonistiigi/985a6e0b90b5a94b6bb0639c880636cb#file-daemon3-d35310795-log-L176
2016-08-11T00:52:37.603734049Z daemon3 set itself as candidate and becomes leader(voted by daemon1 but not daemon2) https://gist.github.com/tonistiigi/985a6e0b90b5a94b6bb0639c880636cb#file-daemon3-d35310795-log-L193
2016-08-11T00:52:37.680063567Z strange "failed to remove node" in daemon2 https://gist.github.com/tonistiigi/985a6e0b90b5a94b6bb0639c880636cb#file-daemon2-d30297954-log-L216

[LK4D4]

I wonder if we should tune raft options. Now reelection starts after 5 seconds of last node observation. A slow machine, CPU spikes, network problems and sequential nature of our raft package can lead to heartbeat miss.
ping @abronan

[aaronlehmann]

My suspicion is that since the arm machines use a network block device, raft's fsyncs introduce too much latency and lead to timeouts.

Could you please try running the tests on a ramdisk or local storage and see if it makes a difference?

[tonistiigi]

@aaronlehmann I ran it from tmpfs and it failed the test failed the same way. https://www.dropbox.com/s/yfcj8yul12f6bd4/leaderelection-tmpfs.tar.gz?dl=0 has all the logs and swarm state.

[abronan]

@tonistiigi Is it possible to try the test bumping the election-tick and heartbeat-tick to higher values? (like 2 or 3 times the default?)

The default values won't cut it for slow environments or ARM so we should tweak those values as @LK4D4 mentioned.

[tonistiigi]

@abronan Tested with ElectionTick=9 HeartbeatTick=3 but didn't seem to make a difference. Logs: https://www.dropbox.com/s/fisn7d6xpf3hvi4/leadereelction-modraft.tar.gz?dl=0

[tonistiigi]

diff --git a/integration-cli/check_test.go b/integration-cli/check_test.go
index 4dfc2f2..165b5b2 100644
--- a/integration-cli/check_test.go
+++ b/integration-cli/check_test.go
@@ -243,7 +243,10 @@ func (s *DockerSwarmSuite) AddDaemon(c *check.C, joinSwarm, manager bool) *Swarm
                                JoinToken:   token,
                        }), check.IsNil)
                } else {
-                       c.Assert(d.Init(swarm.InitRequest{}), check.IsNil)
+                       req := swarm.InitRequest{}
+                       req.Spec.Raft.ElectionTick = 9
+                       req.Spec.Raft.HeartbeatTick = 3
+                       c.Assert(d.Init(req), check.IsNil)
                }
        }

[aaronlehmann]

Not sure that patch will make a difference, as I believe these settings in the ClusterSpec are ignored (see #1326).

[tonistiigi]

@aaronlehmann @abronan I did 5 runs of this test with ElectionTick=9 HeartbeatTick=3 set in docker source(not through api) and all of them passed.

[LK4D4]

@tonistiigi Cool

[aaronlehmann]

Let's try to understand why it sometimes takes a node more than 3 seconds to send a heartbeat. Is it blocked on I/O? Is it a networking issue?

I think this part of the log suggests something related to networking/GRPC:

time="2016-08-11T00:52:22.279387847Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-11T00:52:22.279672693Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-11T00:52:22.280490509Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-11T00:52:22.280744834Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-11T00:52:25.278396028Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-11T00:52:25.278711474Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-11T00:52:25.279343367Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-11T00:52:28.288733637Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-11T00:52:28.288980962Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-11T00:52:28.289528413Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-11T00:52:31.288264154Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-11T00:52:31.288503039Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-11T00:52:31.289682262Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-11T00:52:34.278710214Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-11T00:52:34.279260705Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-11T00:52:34.279432908Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-11T00:52:37.311223633Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-11T00:52:37.311468318Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 
time="2016-08-11T00:52:37.311629921Z" level=debug msg="feb61f32aa6d9a2 failed to send message to 2f7116c9f3135f00 because it is unreachable [next = 25, match = 0, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]" 

By 52:29, 2f7116c9f3135f00 has already started an election, and is receiving messages from 51dbe61bbca33471, so it surely must be reachable. We are supposed to reset the GRPC connection and replace it with a new one if the RPC times out, but perhaps there is a problem related to this.

[aaronlehmann]

Here's some additional logging to try with the original heartbeat/election tick values:

diff --git a/manager/state/raft/raft.go b/manager/state/raft/raft.go
index 0cfbdfa..37751f2 100644
--- a/manager/state/raft/raft.go
+++ b/manager/state/raft/raft.go
@@ -341,6 +341,8 @@ func (n *Node) Run(ctx context.Context) error {
            n.Tick()

        case rd := <-n.Ready():
+           n.Config.Logger.Debug("processing raft update")
+
            raftConfig := DefaultRaftConfig()
            n.memoryStore.View(func(readTx store.ReadTx) {
                clusters, err := store.FindClusters(readTx, store.ByName(store.DefaultClusterName))
@@ -432,6 +434,7 @@ func (n *Node) Run(ctx context.Context) error {

            // Advance the state machine
            n.Advance()
+           n.Config.Logger.Debug("finished processing raft update")

        case snapshotIndex := <-n.snapshotInProgress:
            if snapshotIndex > n.snapshotIndex {
@@ -751,6 +754,8 @@ func (n *Node) ProcessRaftMessage(ctx context.Context, msg *api.ProcessRaftMessa
        return nil, grpc.Errorf(codes.InvalidArgument, "no message provided")
    }

+   n.Config.Logger.Debugf("received message from %x", msg.Message.From)
+
    // Don't process the message if this comes from
    // a node in the remove set
    if n.cluster.IsIDRemoved(msg.Message.From) {
@@ -777,6 +782,8 @@ func (n *Node) ProcessRaftMessage(ctx context.Context, msg *api.ProcessRaftMessa
        return nil, err
    }

+   n.Config.Logger.Debugf("finished processing message from %x", msg.Message.From)
+
    return &api.ProcessRaftMessageResponse{}, nil
 }

@@ -1083,6 +1090,8 @@ func (n *Node) sendToMember(members map[uint64]*membership.Member, m raftpb.Mess

    _, err := conn.ProcessRaftMessage(ctx, &api.ProcessRaftMessageRequest{Message: &m})
    if err != nil {
+       n.Config.Logger.Debugf("failed to send message to %x: %v", m.To, err)
+
        if grpc.ErrorDesc(err) == ErrMemberRemoved.Error() {
            n.removeRaftFunc()
        }

I have access to an ARM machine, so I can try this myself. If I run TESTFLAGS='-check.f DockerSwarmSuite.*' make test-integration-cli, will it preserve the logs outside the container by default?

[aaronlehmann]

Should be fixed by #1367. Reopen if not.