PlannedReparent improvements

[deepthi]

Overview of the Issue

During a planned reparent, it is possible that PromoteSlaveWhenCaughtUp succeeds without an error but the outer context has been canceled. In that situation, we still call UndoDemoteMaster with a background context.
By design, the old master discovers that it is no longer master and changes its type to REPLICA. However, the call to UndoDemoteMaster is unnecessary insufficient in this situation (as explained further below).
When the outer context is canceled at a point where PromoteSlaveWhenCaughtUp has completed, other replicas are still pointing to the old master, so we can end up with a situation like this:

               New Master <---- Old Master <---- Other replicas

To prevent this, we should consider two additional safeguards:

• add logic in PromoteSlaveWhenCaughtUp to revert any partial changes in case of errors.
• create an UndoPromoteSlave that is called along with UndoDemoteMaster.

Code snippet for reference:
https://github.com/vitessio/vitess/blame/master/go/vt/wrangler/reparent.go#L605

		rp, err = wr.tmc.PromoteSlaveWhenCaughtUp(promoteCtx, masterElectTabletInfo.Tablet, rp)
		if err != nil || (ctx.Err() != nil && ctx.Err() == context.DeadlineExceeded) {
			// If we fail to promote the new master, try to roll back to the
			// original master before aborting.
			// It is possible that we have used up the original context, or that
			// not enough time is left on it before it times out.
			// But at this point we really need to be able to Undo so as not to
			// leave the cluster in a bad state.
			// So we create a fresh context based on context.Background().
			undoCtx, undoCancel := context.WithTimeout(context.Background(), *topo.RemoteOperationTimeout)
			defer undoCancel()
			if err1 := wr.tmc.UndoDemoteMaster(undoCtx, oldMasterTabletInfo.Tablet); err1 != nil {
				log.Warningf("Encountered error %v while trying to undo DemoteMaster", err1)
			}
			return fmt.Errorf("master-elect tablet %v failed to catch up with replication or be upgraded to master: %v", masterElectTabletAliasStr, err)
		}

[deepthi]

cc @enisoc @sougou @rafael @setassociative @zmagg

[deepthi]

After discussion here's the approach we came up with:

• The outer call needs to know whether PromoteSlaveWhenCaughtUp failed because the replica was unable to catch up within the timeout, or due to some other error. In order to facilitate this, we will issue two separate RPCs, one to catchup (WaitForPosition) and then to do the actual promotion (PromoteSlave). Then we can handle any errors from each of those calls with more clarity.

• In PromoteSlave change tablet type to MASTER before setting the underlying mysql to read-write. This will ensure that we never have a situation where PRS fails and we have two writable mysql instances. Vitess/topo level mastership will change correctly even in case of PRS failure because of the shard-watch mechanism.

• Calling PRS on existing master to repair a failed PRS should always work (barring network failures). This will require some changes to the logic in reparent.go that handles this case.

• When PRS is called with a no-master scenario, we try to find the most up-to-date replica and promote that. The way we decide that relies on Executed_Gtid_Set which gives us the replica that has the most up-to-date committed transaction. The most-up-to-date replica is actually the one that has the most advanced Retrieved_Gtid_Set. We need to change our SlaveStatus struct and functions to handle this correctly. This will help avoid errant GTID situations.