VReplication Workflows and VStream API gets stuck in the copy phase if tablet type is set only as a replica

[rohit-nayak-ps]

Overview of the Issue

VStreams do not stream any data while streaming using the vtgate VStream API If

• you set the tablet type as replica
• enable the lag throttler (https://vitess.io/docs/16.0/reference/features/tablet-throttler)

The rowstreamer starts streaming, but it gets permanently throttled before a single row is streamed and the copy phase times out.

I am also able to reproduce this in the MoveTables step of the local example if tablet_type is set just to replica.

If we modify local_example.sh to enable the throttler, MoveTables never moves out of the copy phase.

./101_initial_cluster.sh

vtctlclient -- UpdateThrottlerConfig --enable --threshold 5.0 commerce

Reproduction Steps

1. Run local_example upto ./201_customer_tablets.sh, so that the commerce keyspace is setup and populated
2. Enable the throttler: vtctlclient -- UpdateThrottlerConfig --enable --threshold 5.0 commerce
3. Modify examples/local/vstream_client.go to stream from commerce and from replica
   reader, err := conn.VStream(ctx, topodatapb.TabletType_REPLICA, vgtid, filter, flags)
4. Stream using the VStream API: go run examples/local/vstream_client.go
5. Nothing gets streamed
6. In a different terminal disable the throttler: vtctlclient -- UpdateThrottlerConfig --disable commerce
7. Data is streamed

Binary Version

`main` at `18ab80e26b71bc25c11ec11919c5b9361576a160`

Operating System and Environment details

-

Log Fragments

I0528 11:08:25.236309  611413 uvstreamer.go:405] Stream() called
I0528 11:08:25.236349  611413 uvstreamer.go:410] TablePKs is not nil: starting vs.copy()
I0528 11:08:25.236352  611413 copy.go:50] catchupAndCopy for customer
I0528 11:08:25.236354  611413 copy.go:58] catchupAndCopy: before copyTable customer
I0528 11:08:25.236358  611413 copy.go:214] Starting copyTable for customer, PK []
I0528 11:08:25.236363  611413 engine.go:279] Streaming rows for query select * from customer, lastpk: []
I0528 11:08:25.238083  611413 rowstreamer.go:291] Streaming query: select customer_id, email from customer order by customer_id
I0528 11:08:25.238300  611413 snapshot_conn.go:121] Locking table customer for copying
I0528 11:08:25.238636  611413 snapshot_conn.go:114] Tables unlocked: customer
I0528 11:12:17.104444  611413 rowstreamer.go:346] Stream ended because of ctx.Done
I0528 11:12:17.104667  611413 uvstreamer.go:412] uvstreamer.Stream() copy returned with err stream ended: context canceled

[shlomi-noach]

fixed by #13195