Skip to content

fix: four lexers push onto states that do not exist - #1350

Merged
alecthomas merged 1 commit into
alecthomas:masterfrom
VXNCXNX:fix/push-to-undefined-state
Aug 17, 2026
Merged

fix: four lexers push onto states that do not exist#1350
alecthomas merged 1 commit into
alecthomas:masterfrom
VXNCXNX:fix/push-to-undefined-state

Conversation

@VXNCXNX

@VXNCXNX VXNCXNX commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Four lexers push onto states that do not exist, so lexing ordinary input for
those languages fails at runtime.

$ chroma -f noop --lexer yang  file    # /* nested */ comment
before: chroma: error: unknown state comment
after:  /* outer /* inner */ */

$ chroma -f noop --lexer psl   file    # #WARN directive
before: chroma: error: unknown state warn-directive
after:  #WARN something

$ chroma -f noop --lexer spade file    # a lifetime
before: chroma: error: unknown state lifetime
after:  fn f(x: 'a) {}

The fix, and why it is one change rather than four

The root cause is that a push onto a missing state is only discovered when a
document happens to reach that rule. regexp.go already validates emitters at
compile time; this adds the same for mutators, via a ValidatingMutator
interface implemented by pushMutator and multiMutator. A bad push then fails
when the lexer is compiled, which the existing TestCompileAllRegexes runs over
every lexer in the repo.

Turning that on is what surfaces the four broken lexers, and they have to be
fixed in the same commit or the suite goes red. Reverting just the four XML files
with the validation in place gives:

--- FAIL: TestCompileAllRegexes
        PSL: directive: INFO|WARN: invalid push state "warn-directive"
--- FAIL: TestLexers/PSL/testdata/psl.actual
--- FAIL: TestLexers/SourcePawn/testdata/sourcepawn.actual

The four:

  • yang: push state="comment" inside the comment state itself, for nested
    block comments. Changed to <push/> (#push), which is what Pygments does.
  • psl: the #WARN/#INFO rule pushes warn-directive, which was never
    defined. Added, mirroring the sibling other-directive and
    accept-directive states.
  • spade: pushes lifetime, never defined. Added, copied from rust.xml,
    which Spade derives from.
  • sourcepawn: two rules push a nonexistent function state. Both are also
    unreachable, shadowed by the preceding [a-zA-Z_]\w* rule, and upstream
    Pygments has no such rules, so I removed them rather than invent a state.
    sourcepawn.expected is unchanged, which is the evidence they never fired.

Verification

TestPushToUndefinedStateIsAnError compiles a lexer that pushes a missing state
and expects an error.

Removing the validation block from regexp.go fails it:

--- FAIL: TestPushToUndefinedStateIsAnError
    regexp_test.go:230: Expected an error

Reverting yang.xml back to <push state="comment"/> fails the golden test with
the new compile-time message rather than a runtime one:

--- FAIL: TestLexers/YANG/testdata/yang.actual
        YANG: comments: /\*: invalid push state "comment"

Changing the spade lifetime token to Text fails on the token stream:

- {"type":"NameAttribute","value":"'a"}
+ {"type":"NameAttribute","value":"'"}

The psl, spade and yang goldens are regenerated with RECORD=true after
appending a snippet that exercises each fixed path.

go test ./... is ok across every package, and golangci-lint run ./... exits
0.

One judgement call worth flagging: PSL is not a Pygments lexer, so
warn-directive had no upstream reference. I gave it the same .+$ to
CommentSingle body as its sibling directive states, which is what the original
author's rule clearly intended, but you may prefer different content.

Disclosure: written with AI assistance (Claude Code). The before and after come from running the real chroma binary from each tree, and I ran the mutation checks and the coupling check above myself.

@alecthomas alecthomas left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wanted to say thanks for all the PRs, I very much appreciate that they are small targeted fixes, not giant blobs of AI crap.

One suggestion though - add a rule to your AGENTS.md/CLAUDE.md to make the pull request descriptions much more concise while still describing why and what the PR does. The current descriptions are so large that they're almost useless, which is very common for LLMs.

Comment thread mutators.go

func (p *pushMutator) MutatorKind() string { return "push" }

func (p *pushMutator) ValidateMutator(rules CompiledRules) error {

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

@alecthomas
alecthomas merged commit c9b4f2c into alecthomas:master Aug 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants