fix: four lexers push onto states that do not exist - #1350
Merged
alecthomas merged 1 commit intoAug 17, 2026
Conversation
alecthomas
approved these changes
Aug 17, 2026
alecthomas
left a comment
Owner
There was a problem hiding this comment.
Just wanted to say thanks for all the PRs, I very much appreciate that they are small targeted fixes, not giant blobs of AI crap.
One suggestion though - add a rule to your AGENTS.md/CLAUDE.md to make the pull request descriptions much more concise while still describing why and what the PR does. The current descriptions are so large that they're almost useless, which is very common for LLMs.
|
|
||
| func (p *pushMutator) MutatorKind() string { return "push" } | ||
|
|
||
| func (p *pushMutator) ValidateMutator(rules CompiledRules) error { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four lexers push onto states that do not exist, so lexing ordinary input for
those languages fails at runtime.
The fix, and why it is one change rather than four
The root cause is that a
pushonto a missing state is only discovered when adocument happens to reach that rule.
regexp.goalready validates emitters atcompile time; this adds the same for mutators, via a
ValidatingMutatorinterface implemented by
pushMutatorandmultiMutator. A bad push then failswhen the lexer is compiled, which the existing
TestCompileAllRegexesruns overevery lexer in the repo.
Turning that on is what surfaces the four broken lexers, and they have to be
fixed in the same commit or the suite goes red. Reverting just the four XML files
with the validation in place gives:
The four:
push state="comment"inside thecommentstate itself, for nestedblock comments. Changed to
<push/>(#push), which is what Pygments does.#WARN/#INFOrule pusheswarn-directive, which was neverdefined. Added, mirroring the sibling
other-directiveandaccept-directivestates.lifetime, never defined. Added, copied fromrust.xml,which Spade derives from.
functionstate. Both are alsounreachable, shadowed by the preceding
[a-zA-Z_]\w*rule, and upstreamPygments has no such rules, so I removed them rather than invent a state.
sourcepawn.expectedis unchanged, which is the evidence they never fired.Verification
TestPushToUndefinedStateIsAnErrorcompiles a lexer that pushes a missing stateand expects an error.
Removing the validation block from
regexp.gofails it:Reverting
yang.xmlback to<push state="comment"/>fails the golden test withthe new compile-time message rather than a runtime one:
Changing the spade
lifetimetoken toTextfails on the token stream:The
psl,spadeandyanggoldens are regenerated withRECORD=trueafterappending a snippet that exercises each fixed path.
go test ./...is ok across every package, andgolangci-lint run ./...exits0.
One judgement call worth flagging: PSL is not a Pygments lexer, so
warn-directivehad no upstream reference. I gave it the same.+$toCommentSinglebody as its sibling directive states, which is what the originalauthor's rule clearly intended, but you may prefer different content.
Disclosure: written with AI assistance (Claude Code). The before and after come from running the real
chromabinary from each tree, and I ran the mutation checks and the coupling check above myself.