Bug Report: `SwitchTraffic` in v15 not backwards compatible with v14 tablets

[arthurschreiber]

Overview of the Issue

We have a source keyspace, running Vitess v14 vttablets, and a target keyspace, running Vitess v15 vttablets. We are moving some tables from the source keyspace into the target keyspace, using the MoveTables workflow.

Setting up the MoveTables workflow between the source and target keyspace worked as expected. Once VReplication finished the copy phase and switched over to the streaming phase, we used SwitchTraffic to first move reads (successfully) to the new cluster, but when we tried to move writes as well, we encountered the following error:

E0113 13:45:52.519880    1353 traffic_switcher.go:586] createReverseVReplication failed: Code: UNKNOWN
rpc error: code = Unknown desc = TabletManager.VReplicationExec on ac4-0171236135 error: Unknown column 'workflow_sub_type' in 'field list' (errno 1054) (sqlstate 42S22) during query: insert into _vt.vreplication (workflow, source, pos, max_tps, max_replication_lag, time_updated, transaction_timestamp, state, db_name, workflow_type, workflow_sub_type) values (...)

workflow_sub_type was added to the _vt.vreplication table in Vitess v15, so the _vt_vreplication table running with v14 vttablets does not have that column yet. The SwitchTraffic command in the v15 vtctl expects that column to exist.

We're now in the process of upgrading the source cluster to v15 and will the try moving traffic again, but I was wondering if Vitess is supposed to have backwards / forwards compatibility for commands that work across keyspaces (that can run with different tablet versions) like this?

Reproduction Steps

n/a

Binary Version

n/a

Operating System and Environment details

n/a

Log Fragments

No response

[rohit-nayak-ps]

This is a bug in the way the workflow_sub_type column was added to _vt.vreplication . We do expect this upgrade path to work. Will work on a fix.

The upgrade to v15 should fix it since the _vt schema "self-heals" when the VReplication controller restarts on the v15 vttablets.

[mattlord]

This is one of many such failures that can happen due to the VReplicationExec API, see:

• Compatibility issues between v11 and v12 for getting workflows #11535
• Tracking Issue: Replace VReplicationExec Command + RPC  #12086