onlineddl: context-scoped bulk migration operations

[shlomi-noach]

Followup to #20146, similar changes (same changes parser-wise, obviously different implementation onlineddl-executor-wise).

Fixes #20177.
Related: #20145

Extends the ALTER VITESS_MIGRATION ... CONTEXT 'ctx' pattern to all bulk migration operations. Previously only CANCEL CONTEXT 'ctx' existed. Now the full set is supported:

ALTER VITESS_MIGRATION CANCEL CONTEXT 'ctx'
ALTER VITESS_MIGRATION CLEANUP CONTEXT 'ctx'
ALTER VITESS_MIGRATION COMPLETE CONTEXT 'ctx'
ALTER VITESS_MIGRATION POSTPONE COMPLETE CONTEXT 'ctx'
ALTER VITESS_MIGRATION FORCE_CUTOVER CONTEXT 'ctx'
ALTER VITESS_MIGRATION LAUNCH CONTEXT 'ctx'
ALTER VITESS_MIGRATION THROTTLE CONTEXT 'ctx' [EXPIRE 'duration'] [RATIO ratio]
ALTER VITESS_MIGRATION UNTHROTTLE CONTEXT 'ctx'

Each operation affects only migrations belonging to the given migration_context, leaving unrelated contexts untouched.

How it works

The AlterMigration AST node already carries a Context string field (added in the CANCEL CONTEXT PR). Each XxxAllMigrationType executor method now accepts an optional migrationContext string parameter:

• Empty → existing bulk behavior (unchanged)
• Non-empty → filters to migrations matching that context

CLEANUP CONTEXT uses a new SQL constant that adds AND migration_context=? to the existing cleanup query. All other operations iterate readPendingMigrations and skip non-matching contexts, matching the pattern established by CancelPendingMigrations.

THROTTLE/UNTHROTTLE CONTEXT work by iterating pending migrations in the context and throttling each by UUID (the individual-migration throttle path), since the whole-app throttle doesn't support context filtering.

Changes

• go/vt/sqlparser/ — 7 new grammar rules, format branching for all ALL types, 11 parse round-trip tests
• go/vt/vttablet/onlineddl/executor.go — migrationContext param on all 7 "All" methods
• go/vt/vttablet/onlineddl/schema.go — sqlUpdateReadyForCleanupAllByContext constant
• go/vt/vttablet/tabletserver/query_executor.go — pass alterMigration.Context in all ALL-type dispatch cases
• go/test/endtoend/onlineddl/ — 7 new vtgate helpers + e2e scheduler tests for each operation

---

Most of this was written by Claude Code — I provided direction.

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• Apply the release notes (needs details) label if users need to know about this change.
• New features should be documented.
• There should be some code comments as to why things are implemented the way they are.
• There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• Is this flag really necessary?
• Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• vtctl command output order should be stable and awk-able.

[Copilot]

Pull request overview

This PR extends ALTER VITESS_MIGRATION ... CONTEXT 'ctx' support from just CANCEL to the full set of bulk migration operations, so bulk actions can be scoped to a specific migration_context without impacting other contexts.

Changes:

• Extends the SQL grammar/AST formatting to parse + round-trip ... CONTEXT 'ctx' for all bulk operations.
• Threads migrationContext through OnlineDDL executor “All” operations and filters pending migrations accordingly (plus a context-specific cleanup UPDATE).
• Adds end-to-end helpers/tests to validate each context-scoped bulk operation.

Reviewed changes

Copilot reviewed 10 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
go/vt/vttablet/tabletserver/query_executor.go	Passes alterMigration.Context to OnlineDDL executor bulk-operation dispatch.
go/vt/vttablet/onlineddl/schema.go	Adds a context-scoped cleanup UPDATE query constant.
go/vt/vttablet/onlineddl/executor.go	Adds migrationContext param to bulk executor methods and filters to matching contexts.
go/vt/sqlparser/sql.y	Adds CONTEXT token + grammar rules for context-scoped bulk migration operations.
go/vt/sqlparser/parse_test.go	Adds parser round-trip coverage for the new ... CONTEXT statements.
go/vt/sqlparser/keywords.go	Registers context keyword/token.
go/vt/sqlparser/cached_size.go	Updates cached-size accounting for AlterMigration to include Context.
go/vt/sqlparser/ast.go	Adds Context field to the AlterMigration AST node.
go/vt/sqlparser/ast_format.go	Formats bulk operations as ... context 'ctx' when Context is set.
go/vt/sqlparser/ast_format_fast.go	Adds fast formatting for the new Context field (see review comment about escaping).
go/vt/sqlparser/ast_equals.go	Includes Context in AlterMigration equality comparisons.
go/test/endtoend/onlineddl/vtgate_util.go	Adds VTGate helper functions for context-scoped operations (see review comment about query construction).
go/test/endtoend/onlineddl/scheduler/onlineddl_scheduler_test.go	Adds scheduler e2e tests validating context-scoped behavior across operations.

Files not reviewed (3)

• go/vt/sqlparser/ast_equals.go: Language not supported
• go/vt/sqlparser/ast_format_fast.go: Language not supported
• go/vt/sqlparser/cached_size.go: Language not supported

Comments suppressed due to low confidence (7)

go/test/endtoend/onlineddl/vtgate_util.go:253

• This query is built via fmt.Sprintf with migrationContext embedded in quotes; contexts containing quotes/backslashes will break the SQL and can lead to unintended statements. Prefer sqlparser.ParseAndBind with %a + StringBindVariable for migrationContext.

// CheckPostponeCompleteContextMigrations postpones completion of all pending migrations with a given context and expects number of affected rows.
// A negative value for expectCount indicates "don't care, no need to check"
func CheckPostponeCompleteContextMigrations(t *testing.T, vtParams *mysql.ConnParams, migrationContext string, expectCount int) {
	query := fmt.Sprintf("alter vitess_migration postpone complete context '%s'", migrationContext)
	r := VtgateExecQuery(t, vtParams, query, "")

go/test/endtoend/onlineddl/vtgate_util.go:286

• This query is built via fmt.Sprintf with migrationContext embedded in quotes; contexts containing quotes/backslashes will break the SQL and can lead to unintended statements. Prefer sqlparser.ParseAndBind with %a + StringBindVariable for migrationContext.

// CheckCancelContextMigrations cancels all pending migrations with a given context and expect number of affected rows
// A negative value for expectCount indicates "don't care, no need to check"
func CheckCancelContextMigrations(t *testing.T, vtParams *mysql.ConnParams, migrationContext string, expectCount int) {
	cancelQuery := fmt.Sprintf("alter vitess_migration cancel context '%s'", migrationContext)
	r := VtgateExecQuery(t, vtParams, cancelQuery, "")

go/test/endtoend/onlineddl/vtgate_util.go:297

• This query is built via fmt.Sprintf with migrationContext embedded in quotes; contexts containing quotes/backslashes will break the SQL and can lead to unintended statements. Prefer sqlparser.ParseAndBind with %a + StringBindVariable for migrationContext.

// CheckCleanupContextMigrations cleans up terminal migrations with a given context and expects number of affected rows.
// A negative value for expectCount indicates "don't care, no need to check"
func CheckCleanupContextMigrations(t *testing.T, vtParams *mysql.ConnParams, migrationContext string, expectCount int) uint64 {
	query := fmt.Sprintf("alter vitess_migration cleanup context '%s'", migrationContext)
	r := VtgateExecQuery(t, vtParams, query, "")

go/test/endtoend/onlineddl/vtgate_util.go:322

• This query is built via fmt.Sprintf with migrationContext embedded in quotes; contexts containing quotes/backslashes will break the SQL and can lead to unintended statements. Prefer sqlparser.ParseAndBind with %a + StringBindVariable for migrationContext.

// CheckLaunchContextMigrations launches all queued postponed migrations with a given context and expects number of affected rows.
// A negative value for expectCount indicates "don't care, no need to check"
func CheckLaunchContextMigrations(t *testing.T, vtParams *mysql.ConnParams, migrationContext string, expectCount int) {
	query := fmt.Sprintf("alter vitess_migration launch context '%s'", migrationContext)
	r := VtgateExecQuery(t, vtParams, query, "")

go/test/endtoend/onlineddl/vtgate_util.go:343

• This query is built via fmt.Sprintf with migrationContext embedded in quotes; contexts containing quotes/backslashes will break the SQL and can lead to unintended statements. Prefer sqlparser.ParseAndBind with %a + StringBindVariable for migrationContext.

// CheckForceCutOverContextMigrations marks all pending migrations with a given context for forced cut-over and expects number of affected rows.
// A negative value for expectCount indicates "don't care, no need to check"
func CheckForceCutOverContextMigrations(t *testing.T, vtParams *mysql.ConnParams, migrationContext string, expectCount int) {
	query := fmt.Sprintf("alter vitess_migration force_cutover context '%s'", migrationContext)
	r := VtgateExecQuery(t, vtParams, query, "")

go/test/endtoend/onlineddl/vtgate_util.go:493

• This query is built via fmt.Sprintf with migrationContext embedded in quotes; contexts containing quotes/backslashes will break the SQL and can lead to unintended statements. Prefer sqlparser.ParseAndBind with %a + StringBindVariable for migrationContext.

// ThrottleContextMigrations throttles all pending migrations with a given context.
func ThrottleContextMigrations(t *testing.T, vtParams *mysql.ConnParams, migrationContext string) {
	query := fmt.Sprintf("alter vitess_migration throttle context '%s' expire '24h' ratio 1", migrationContext)
	_ = VtgateExecQuery(t, vtParams, query, "")

go/test/endtoend/onlineddl/vtgate_util.go:505

• This query is built via fmt.Sprintf with migrationContext embedded in quotes; contexts containing quotes/backslashes will break the SQL and can lead to unintended statements. Prefer sqlparser.ParseAndBind with %a + StringBindVariable for migrationContext.

// UnthrottleContextMigrations unthrottles all pending migrations with a given context.
func UnthrottleContextMigrations(t *testing.T, vtParams *mysql.ConnParams, migrationContext string) {
	query := fmt.Sprintf("alter vitess_migration unthrottle context '%s'", migrationContext)
	_ = VtgateExecQuery(t, vtParams, query, "")

[Copilot]

Pull request overview

Copilot reviewed 10 out of 14 changed files in this pull request and generated no new comments.

Files not reviewed (3)

• go/vt/sqlparser/ast_equals.go: Language not supported
• go/vt/sqlparser/ast_format_fast.go: Language not supported
• go/vt/sqlparser/cached_size.go: Language not supported

[codecov]

Codecov Report

❌ Patch coverage is 0.84746% with 117 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.05%. Comparing base (70c7a72) to head (120cfc6).
⚠️ Report is 277 commits behind head on main.

Files with missing lines	Patch %	Lines
go/vt/vttablet/onlineddl/executor.go	0.00%	81 Missing ⚠️
go/vt/sqlparser/ast_format.go	3.33%	29 Missing ⚠️
go/vt/vttablet/tabletserver/query_executor.go	0.00%	7 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (70c7a72) and HEAD (120cfc6). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (70c7a72)	HEAD (120cfc6)
	1	0

Additional details and impacted files

@@             Coverage Diff             @@
##             main   #20147       +/-   ##
===========================================
- Coverage   69.67%   62.05%    -7.62%     
===========================================
  Files        1614      135     -1479     
  Lines      216793    30725   -186068     
===========================================
- Hits       151044    19067   -131977     
+ Misses      65749    11658    -54091     

Flag	Coverage Δ
partial	62.05% <0.84%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 14 changed files in this pull request and generated 1 comment.

Files not reviewed (3)

• go/vt/sqlparser/ast_equals.go: Language not supported
• go/vt/sqlparser/ast_format_fast.go: Language not supported
• go/vt/sqlparser/cached_size.go: Language not supported

[shlomi-noach]

OK OK you're right and I give up. Empty context is now rejected by the parser. I like this idea better than my original approach.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 14 changed files in this pull request and generated no new comments.

Files not reviewed (3)

• go/vt/sqlparser/ast_equals.go: Language not supported
• go/vt/sqlparser/ast_format_fast.go: Language not supported
• go/vt/sqlparser/cached_size.go: Language not supported

[Copilot]

Pull request overview

Copilot reviewed 8 out of 10 changed files in this pull request and generated 2 comments.

Files not reviewed (1)

• go/vt/sqlparser/ast_format_fast.go: Language not supported

[Copilot]

Pull request overview

Copilot reviewed 8 out of 10 changed files in this pull request and generated no new comments.

Files not reviewed (1)

• go/vt/sqlparser/ast_format_fast.go: Language not supported

[Copilot]

Pull request overview

Copilot reviewed 8 out of 10 changed files in this pull request and generated no new comments.

Files not reviewed (1)

• go/vt/sqlparser/ast_format_fast.go: Language not supported

[promptless]

Promptless prepared a documentation update related to this change.

Triggered by PR #20147

Added documentation for the new context-scoped bulk migration operations (LAUNCH CONTEXT, COMPLETE CONTEXT, POSTPONE COMPLETE CONTEXT, FORCE_CUTOVER CONTEXT, CLEANUP CONTEXT, THROTTLE CONTEXT, and UNTHROTTLE CONTEXT) to the Online DDL audit and control guide, following the existing patterns for the CANCEL CONTEXT syntax.

Review: Document context-scoped bulk migration operations

[mattlord]

Shouldn't we have CLEANUP CONTEXT use the same exact context matching semantics as the other bulk operations in go/vt/vttablet/onlineddl/schema.go:191-197?

The new cleanup path filters with migration_context=%a in SQL, so it inherits the sidecar table's collation. schema_migrations is CHARSET=utf8mb4 without an explicit binary collation, and the supported MySQL versions today all default to a case-insensitive collation for utf8mb4. The other new context-scoped operations compare pm.migrationContext != migrationContext in Go, which is byte/case-sensitive. Since ValidateMigrationContext allows uppercase, contexts like ctx-a and CTX-a can be treated as distinct by COMPLETE/POSTPONE/CANCEL/LAUNCH/FORCE_CUTOVER/THROTTLE, but CLEANUP CONTEXT 'ctx-a' may mark both contexts for artifact cleanup.

Unless I'm missing something I think that we should either make the SQL cleanup predicate binary/exact, or define/implement context matching as case-insensitive consistently across all of these operations. A regression test with two terminal migrations whose contexts differ only by case would then cover this (maybe it will already pass as I'm missing something).

Otherwise it LGTM!

[mattlord]

Might be worth trimming whitespace from both ends of the strings just in case.

[shlomi-noach]

When looking to apply that, I realize the same could be said about dozens if not more other locations in the code where any migration context or uuid are involved. I'd say skip this.

[shlomi-noach]

Shouldn't we have CLEANUP CONTEXT use the same exact context matching semantics as the other bulk operations in go/vt/vttablet/onlineddl/schema.go:191-197?

That's a good catch. sqlUpdateReadyForCleanupAllByContext is the only query that UPDATEs by migration_context comparison.

In choosing the best approach I saw two alternatives:

1. Introduce a SELECT query that uses same WHERE clause as sqlUpdateReadyForCleanupAllByContext, then compare migration context in golang space, then run one UPDATE query per matched entry; or:
2. Modify the sqlUpdateReadyForCleanupAllByContext query to be case sensitive.

I picked the latter

diff --git a/go/vt/vttablet/onlineddl/schema.go b/go/vt/vttablet/onlineddl/schema.go
index 0e75aef6ba..f6fc0606ef 100644
--- a/go/vt/vttablet/onlineddl/schema.go
+++ b/go/vt/vttablet/onlineddl/schema.go
@@ -194,7 +194,7 @@ const (
                        migration_status IN ('complete', 'cancelled', 'failed')
                        AND cleanup_timestamp IS NULL
                        AND retain_artifacts_seconds > 0
-                       AND migration_context=%a
+                       AND migration_context COLLATE utf8mb4_bin=%a
        `
        sqlUpdateForceCutOver = `UPDATE _vt.schema_migrations
                        SET force_cutover=1

it's a simple enough change and I think worth it, even if this still means some inconsistency with existing logic. It's just so much simpler. Let me know if you disagree.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 10 changed files in this pull request and generated no new comments.

Files not reviewed (1)

• go/vt/sqlparser/ast_format_fast.go: Language not supported

[mattlord]

LGTM!

[mattlord]

This will prevent usage of the index on the migration_context column, and this is an update so it is taking X locks on rows it scans, but that is probably OK. We could alternatively make the column use this collation in the schema.

[shlomi-noach]

Oh I forgot to explain, the index this query uses is:

KEY `cleanup_status_idx` (`cleanup_timestamp`, `migration_status`),

and so migration_context is irrelevant to the index search.

[shlomi-noach]

updated above comment which was poorly worded.