add split brain flag

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

Author: timvaillancourt

CLAUDE.md

@@ -238,7 +238,7 @@ return user.NeedsMigration() && migrate(user) || user
 ### EmergencyReparentShard (ERS)
 - ERS must prioritize **certainty** that we picked the most-advanced candidate
 - ERS must error when the most-advanced candidate is not clear, and/or a split-brain is suspected
-- ERS must avoid introducing errant GTIDs on replicas. This includes writes that are considered unacknowledged to the client — at the errant-GTID level, whether or not the transaction was acknowledged to the client is inconsequential, as MySQL cannot rewind GTIDs of any kind
+- ERS must avoid introducing errant GTIDs on replicas. This includes writes that are considered unacknowledged to the client as MySQL cannot rewind GTIDs of any kind
 - Changes should prioritize reducing points of failure - avoid new RPCs or work that may delay or make ERS more brittle
 - ERS must error if a shard contains a mix of GTID-based and non-GTID-based replication. Their position semantics differ (`Combined` = retrieved+executed for GTID vs. executed-only for non-GTID), so a unified split-brain / most-advanced check across both is unsafe
 - For non-GTID flavors, ERS must wait on every candidate and fail on any error. The "filter to leading group + short-circuit on first success" optimization is only safe for GTID-based flavors, where `Combined` is distinct from the executed position

changelog/25.0/25.0.0/summary.md

@@ -100,11 +100,14 @@ See [#19978](https://github.com/vitessio/vitess/issues/19978) for details.
 
 `EmergencyReparentShard` (ERS) on GTID-based shards no longer fails when only some replicas can apply their relay logs. As long as at least one tablet at the leading `Combined` GTID position applies successfully, ERS proceeds; lagging or stuck-SQL-thread replicas are no longer blockers. Pre-existing pre-PR behavior is preserved for non-GTID flavors (FilePos, MariaDB), where ERS still requires every candidate to apply.
 
-When the leading GTID-based candidates have incomparable `Combined` positions (suspected split-brain), ERS now aborts upfront with a clear `FAILED_PRECONDITION` error rather than risk silently picking one side.
+When the leading GTID-based candidates have incomparable `Combined` positions (suspected split-brain), ERS now aborts upfront with a clear `FAILED_PRECONDITION` error naming the diverged tablets, rather than silently picking one side. Pre-PR ERS would pick blindly and let the losing side's unique GTIDs become errant on those tablets — a silent data-integrity incident that surfaced later via lag alerts or downstream consistency checks. See [#20199](https://github.com/vitessio/vitess/issues/20199) for the bug this addresses.
 
-Two new stats are exported for observability:
+A new `--allow-split-brain-promotion` flag is added to `vtctldclient EmergencyReparentShard` (and `--allow_split_brain_promotion` on the legacy `vtctl`). It is **off by default**. Operators who deliberately need to force ERS through a detected split-brain — typically because they already know which side to keep and plan to re-clone the losing side — can set it to convert the abort into a `WARN` log and proceed. The losing side's unique GTIDs will become errant after promotion, so this is an explicit operator override, not a default-on safety knob.
+
+Three new stats are exported for observability:
 
 - `EmergencyReparentFilteredCandidates` — counts replicas excluded from the relay-log wait because their `Combined` position is strictly behind the leading group.
 - `EmergencyReparentRelayLogFailedCandidates` — counts replicas that genuinely failed to apply relay logs (cancellations after a peer succeeded are not counted).
+- `EmergencyReparentSplitBrainOverrides` — counts ERS runs that proceeded despite detected split-brain because `--allow-split-brain-promotion` was set. Stays at zero unless an operator has deliberately invoked the escape hatch.
 
 See [#18707](https://github.com/vitessio/vitess/pull/18707) for details.

go/cmd/vtctldclient/command/reparents.go

@@ -96,6 +96,7 @@ var emergencyReparentShardOptions = struct {
 	IgnoreReplicaAliasStrList []string
 	PreventCrossCellPromotion bool
 	WaitForAllTablets         bool
+	AllowSplitBrainPromotion  bool
 }{}
 
 func commandEmergencyReparentShard(cmd *cobra.Command, args []string) error {
@@ -144,6 +145,7 @@ func commandEmergencyReparentShard(cmd *cobra.Command, args []string) error {
 		WaitReplicasTimeout:       protoutil.DurationToProto(emergencyReparentShardOptions.WaitReplicasTimeout),
 		PreventCrossCellPromotion: emergencyReparentShardOptions.PreventCrossCellPromotion,
 		WaitForAllTablets:         emergencyReparentShardOptions.WaitForAllTablets,
+		AllowSplitBrainPromotion:  emergencyReparentShardOptions.AllowSplitBrainPromotion,
 	})
 	if err != nil {
 		return err
@@ -309,6 +311,7 @@ func init() {
 	EmergencyReparentShard.Flags().StringVar(&emergencyReparentShardOptions.ExpectedPrimaryAliasStr, "expected-primary", "", "Alias of a tablet that must be the current primary in order for the reparent to be processed.")
 	EmergencyReparentShard.Flags().BoolVar(&emergencyReparentShardOptions.PreventCrossCellPromotion, "prevent-cross-cell-promotion", false, "Only promotes a new primary from the same cell as the previous primary.")
 	EmergencyReparentShard.Flags().BoolVar(&emergencyReparentShardOptions.WaitForAllTablets, "wait-for-all-tablets", false, "Should ERS wait for all the tablets to respond. Useful when all the tablets are reachable.")
+	EmergencyReparentShard.Flags().BoolVar(&emergencyReparentShardOptions.AllowSplitBrainPromotion, "allow-split-brain-promotion", false, "Allow ERS to proceed when two leading candidates have incomparable Combined GTID positions (suspected split-brain). Off by default. Operator escape hatch — accepts that the losing side's unique GTIDs will become errant.")
 	EmergencyReparentShard.Flags().StringSliceVarP(&emergencyReparentShardOptions.IgnoreReplicaAliasStrList, "ignore-replicas", "i", nil, "Comma-separated, repeated list of replica tablet aliases to ignore during the emergency reparent.")
 	Root.AddCommand(EmergencyReparentShard)
 

go/vt/proto/vtctldata/vtctldata.pb.go

@@ -1304,6 +1304,7 @@ func (s *VtctldServer) EmergencyReparentShard(ctx context.Context, req *vtctldat
 	span.Annotate("wait_replicas_timeout_sec", waitReplicasTimeout.Seconds())
 	span.Annotate("prevent_cross_cell_promotion", req.PreventCrossCellPromotion)
 	span.Annotate("wait_for_all_tablets", req.WaitForAllTablets)
+	span.Annotate("allow_split_brain_promotion", req.AllowSplitBrainPromotion)
 
 	m := sync.RWMutex{}
 	logstream := []*logutilpb.Event{}
@@ -1314,7 +1315,8 @@ func (s *VtctldServer) EmergencyReparentShard(ctx context.Context, req *vtctldat
 		logstream = append(logstream, e)
 	})
 
-	ev, err := reparentutil.NewEmergencyReparenter(s.ts, s.tmc, logger).ReparentShard(ctx,
+	ev, err := reparentutil.NewEmergencyReparenter(s.ts, s.tmc, logger).ReparentShard(
+		ctx,
 		req.Keyspace,
 		req.Shard,
 		reparentutil.EmergencyReparentOptions{
@@ -1324,6 +1326,7 @@ func (s *VtctldServer) EmergencyReparentShard(ctx context.Context, req *vtctldat
 			WaitAllTablets:            req.WaitForAllTablets,
 			PreventCrossCellPromotion: req.PreventCrossCellPromotion,
 			ExpectedPrimaryAlias:      req.ExpectedPrimary,
+			AllowSplitBrainPromotion:  req.AllowSplitBrainPromotion,
 		},
 	)
 
@@ -3315,7 +3318,8 @@ func (s *VtctldServer) PlannedReparentShard(ctx context.Context, req *vtctldatap
 		logstream = append(logstream, e)
 	})
 
-	ev, err := reparentutil.NewPlannedReparenter(s.ts, s.tmc, logger).ReparentShard(ctx,
+	ev, err := reparentutil.NewPlannedReparenter(s.ts, s.tmc, logger).ReparentShard(
+		ctx,
 		req.Keyspace,
 		req.Shard,
 		reparentutil.PlannedReparentOptions{
@@ -5496,7 +5500,8 @@ func (s *VtctldServer) ValidateVSchema(ctx context.Context, req *vtctldatapb.Val
 			r := &tabletmanagerdatapb.GetSchemaRequest{ExcludeTables: req.ExcludeTables, IncludeViews: req.IncludeViews}
 			primarySchema, err := schematools.GetSchema(ctx, s.ts, s.tmc, si.PrimaryAlias, r)
 			if err != nil {
-				errorMessage := fmt.Sprintf("GetSchema(%s, nil, %v, %v) (%v/%v) failed: %v", si.PrimaryAlias.String(),
+				errorMessage := fmt.Sprintf(
+					"GetSchema(%s, nil, %v, %v) (%v/%v) failed: %v", si.PrimaryAlias.String(),
 					excludeTables, includeViews, keyspace, shard, err,
 				)
 				shardResult.Results = append(shardResult.Results, errorMessage)

go/vt/proto/vtctldata/vtctldata_vtproto.pb.go

@@ -173,6 +173,7 @@ func commandEmergencyReparentShard(ctx context.Context, wr *wrangler.Wrangler, s
 	preventCrossCellPromotion := subFlags.Bool("prevent_cross_cell_promotion", false, "only promotes a new primary from the same cell as the previous primary")
 	ignoreReplicasList := subFlags.String("ignore_replicas", "", "comma-separated list of replica tablet aliases to ignore during emergency reparent")
 	waitForAllTablets := subFlags.Bool("wait_for_all_tablets", false, "should ERS wait for all the tablets to respond. Useful when all the tablets are reachable")
+	allowSplitBrainPromotion := subFlags.Bool("allow_split_brain_promotion", false, "allow ERS to proceed when two leading candidates have incomparable Combined GTID positions (suspected split-brain); off by default — operator escape hatch")
 
 	if err := subFlags.Parse(args); err != nil {
 		return err
@@ -206,6 +207,7 @@ func commandEmergencyReparentShard(ctx context.Context, wr *wrangler.Wrangler, s
 		WaitReplicasTimeout:       *waitReplicasTimeout,
 		IgnoreReplicas:            topoproto.ParseTabletSet(*ignoreReplicasList),
 		PreventCrossCellPromotion: *preventCrossCellPromotion,
+		AllowSplitBrainPromotion:  *allowSplitBrainPromotion,
 	})
 }
 

go/vt/vtctl/grpcvtctldserver/server.go

@@ -63,6 +63,13 @@ type EmergencyReparentOptions struct {
 	WaitReplicasTimeout       time.Duration
 	PreventCrossCellPromotion bool
 	ExpectedPrimaryAlias      *topodatapb.TabletAlias
+	// AllowSplitBrainPromotion lets ERS proceed when two leading candidates have
+	// incomparable Combined GTID positions (suspected split-brain). Off by default —
+	// operator escape hatch. When set, the upfront uniformCombined check and the
+	// secondary AtLeast check in findMostAdvanced log a warning and continue instead
+	// of aborting with FAILED_PRECONDITION. The losing side's unique GTIDs will
+	// become errant on those tablets after promotion.
+	AllowSplitBrainPromotion bool
 
 	// Private options managed internally. We use value passing to avoid leaking
 	// these details back out.
@@ -84,6 +91,10 @@ var (
 		"EmergencyReparentRelayLogFailedCandidates", "Number of candidates that failed to apply relay logs during EmergencyReparentShard",
 		[]string{"Keyspace", "Shard"},
 	)
+	ersSplitBrainOverrides = stats.NewCountersWithMultiLabels(
+		"EmergencyReparentSplitBrainOverrides", "Number of times EmergencyReparentShard proceeded despite detecting incomparable Combined GTID positions, because the operator set AllowSplitBrainPromotion=true. The losing side's unique GTIDs will become errant on those tablets after promotion.",
+		[]string{"Keyspace", "Shard"},
+	)
 )
 
 // NewEmergencyReparenter returns a new EmergencyReparenter object, ready to
@@ -291,7 +302,7 @@ func (erp *EmergencyReparenter) reparentShardLocked(ctx context.Context, ev *eve
 	relayLogWaitCandidates := validCandidates
 	requireAll := true
 	if isGTIDBased {
-		relayLogWaitCandidates, err = erp.filterAndCheckUniform(validCandidates, keyspace, shard, "candidates")
+		relayLogWaitCandidates, err = erp.filterAndCheckUniform(validCandidates, keyspace, shard, "candidates", opts.AllowSplitBrainPromotion)
 		if err != nil {
 			return err
 		}
@@ -331,7 +342,7 @@ func (erp *EmergencyReparenter) reparentShardLocked(ctx context.Context, ev *eve
 		}
 		if !appliedSurvived {
 			erp.logger.Warningf("all originally-applied candidates were removed by errant-GTID detection; running second relay-log-apply wait on surviving candidates before promotion")
-			rewaitCandidates, err := erp.filterAndCheckUniform(validCandidates, keyspace, shard, "surviving unwaited candidates")
+			rewaitCandidates, err := erp.filterAndCheckUniform(validCandidates, keyspace, shard, "surviving unwaited candidates", opts.AllowSplitBrainPromotion)
 			if err != nil {
 				return err
 			}
@@ -485,18 +496,27 @@ type relayLogResult struct {
 // has incomparable Combined positions (suspected split-brain). The descriptor parameter is
 // interpolated into the error message to identify which ERS pipeline stage detected the
 // split-brain (first wait vs. errant-GTID re-wait).
+//
+// If allowSplitBrain is true, an incomparable result logs a warning and returns the filtered
+// set without erroring — the operator-escape-hatch path that accepts errant GTIDs on the
+// losing side in exchange for forcing ERS through.
 func (erp *EmergencyReparenter) filterAndCheckUniform(
 	validCandidates map[string]*RelayLogPositions,
 	keyspace, shard, descriptor string,
+	allowSplitBrain bool,
 ) (map[string]*RelayLogPositions, error) {
 	filtered := filterToMostAdvancedCombined(validCandidates, erp.logger)
 	if !uniformCombined(filtered) {
-		return nil, vterrors.Errorf(
-			vtrpc.Code_FAILED_PRECONDITION,
-			"emergency reparent aborted: %s have incomparable Combined GTID positions (suspected split-brain): %s",
-			descriptor,
-			describeCombinedPositions(filtered),
-		)
+		if !allowSplitBrain {
+			return nil, vterrors.Errorf(
+				vtrpc.Code_FAILED_PRECONDITION,
+				"emergency reparent aborted: %s have incomparable Combined GTID positions (suspected split-brain): %s",
+				descriptor,
+				describeCombinedPositions(filtered),
+			)
+		}
+		erp.logger.Warningf("AllowSplitBrainPromotion=true: %s have incomparable Combined GTID positions (suspected split-brain): %s — proceeding under operator override; losing side's unique GTIDs will become errant", descriptor, describeCombinedPositions(filtered))
+		ersSplitBrainOverrides.Add([]string{keyspace, shard}, 1)
 	}
 	if excluded := int64(len(validCandidates) - len(filtered)); excluded > 0 {
 		ersFilteredCandidates.Add([]string{keyspace, shard}, excluded)
@@ -715,10 +735,14 @@ func (erp *EmergencyReparenter) findMostAdvanced(
 	winningPosition := tabletPositions[0]
 
 	// We have already removed the tablets with errant GTIDs before calling this function. At this point our winning position must be a
-	// superset of all the other valid positions. If that is not the case, then we have a split brain scenario, and we should cancel the ERS
+	// superset of all the other valid positions. If that is not the case, then we have a split brain scenario, and we should cancel the ERS —
+	// unless AllowSplitBrainPromotion is set, in which case the operator has accepted that the losing side's unique GTIDs will become errant.
 	for i, position := range tabletPositions {
 		if !winningPosition.AtLeast(position) {
-			return nil, nil, vterrors.Errorf(vtrpc.Code_FAILED_PRECONDITION, "split brain detected between servers - %v and %v", winningPrimaryTablet.Alias, validTablets[i].Alias)
+			if !opts.AllowSplitBrainPromotion {
+				return nil, nil, vterrors.Errorf(vtrpc.Code_FAILED_PRECONDITION, "split brain detected between servers - %v and %v", winningPrimaryTablet.Alias, validTablets[i].Alias)
+			}
+			erp.logger.Warningf("AllowSplitBrainPromotion=true: split brain detected between %v and %v — proceeding under operator override; losing side's unique GTIDs will become errant", winningPrimaryTablet.Alias, validTablets[i].Alias)
 		}
 	}