[Lexer] YAML integer-looking value like 1.0-beta or 2 val misparses

[ZhenShuo2021]

Request type

Lexer highlighting bug

Language

YAML

Sample input

a: 1.0-beta
b: 2 val

https://swapoff.org/chroma/playground/#eyJsYW5ndWFnZSI6IllBTUwiLCJzdHlsZSI6ImdpdGh1YiIsInRleHQiOiJhOiAxLjAtYmV0YVxuYjogMiB2YWwiLCJjbGFzc2VzIjpmYWxzZX0

Syntax references

I'm not a regex expert myself, so I asked Claude to look at the TextMate implementation, and it found a reference version:

https://github.com/textmate/yaml.tmbundle/blob/master/Syntaxes/YAML.tmLanguage

Diff below, combined by Claude from that reference implementation, tested and passing:

diff --git a/lexers/embedded/yaml.xml b/lexers/embedded/yaml.xml
index 0707554..a820471 100644
--- a/lexers/embedded/yaml.xml
+++ b/lexers/embedded/yaml.xml
@@ -68,7 +68,7 @@
       <rule pattern="\d\d\d\d-\d\d-\d\d([T ]\d\d:\d\d:\d\d(\.\d+)?(Z|\s+[-+]\d+)?)?">
         <token type="LiteralDate"/>
       </rule>
-      <rule pattern="\b[+\-]?(0x[\da-f]+|0o[0-7]+|(\d+\.?\d*|\.?\d+)(e[\+\-]?\d+)?|\.inf|\.nan)\b">
+      <rule pattern="[+\-]?(0x[\da-f]+|0o[0-7]+|(\d+\.?\d*|\.?\d+)(e[\+\-]?\d+)?|\.inf|\.nan)(?=\s*$|\s+#|\s+//|\s*:(\s|$)|\s*:[\[\]{},]|\s*[\[\]{},])">
         <token type="LiteralNumber"/>
       </rule>
       <rule pattern="([^\{\}\[\]\?,\:\!\-\*&amp;\@].*)( )+(#.*)">
diff --git a/lexers/testdata/yaml.actual b/lexers/testdata/yaml.actual
index ddb417e..5278c88 100644
--- a/lexers/testdata/yaml.actual
+++ b/lexers/testdata/yaml.actual
@@ -254,3 +254,11 @@ set2:
   item3: null
 
 ...  # document end
+
+# regression test: numeric-looking prefix followed by non-numeric suffix
+layout: 2-col
+version: 3.5-beta
+mixed: 123abc
+port: 8080
+ratio: 3.14
+neg: -17
diff --git a/lexers/testdata/yaml.expected b/lexers/testdata/yaml.expected
index 9cb7372..6756774 100644
--- a/lexers/testdata/yaml.expected
+++ b/lexers/testdata/yaml.expected
@@ -564,8 +564,7 @@
   {"type":"TextWhitespace","value":" "},
   {"type":"CommentPreproc","value":"!!python/complex"},
   {"type":"TextWhitespace","value":" "},
-  {"type":"LiteralNumber","value":"1"},
-  {"type":"Literal","value":"+2j"},
+  {"type":"Literal","value":"1+2j"},
   {"type":"TextWhitespace","value":"\n\n"},
   {"type":"Comment","value":"# We can also use yaml complex keys with language specific tags"},
   {"type":"TextWhitespace","value":"\n"},
@@ -669,5 +668,37 @@
   {"type":"NameNamespace","value":"..."},
   {"type":"TextWhitespace","value":"  "},
   {"type":"Comment","value":"# document end"},
+  {"type":"TextWhitespace","value":"\n\n"},
+  {"type":"Comment","value":"# regression test: numeric-looking prefix followed by non-numeric suffix"},
+  {"type":"TextWhitespace","value":"\n"},
+  {"type":"NameTag","value":"layout"},
+  {"type":"Punctuation","value":":"},
+  {"type":"TextWhitespace","value":" "},
+  {"type":"Literal","value":"2-col"},
+  {"type":"TextWhitespace","value":"\n"},
+  {"type":"NameTag","value":"version"},
+  {"type":"Punctuation","value":":"},
+  {"type":"TextWhitespace","value":" "},
+  {"type":"Literal","value":"3.5-beta"},
+  {"type":"TextWhitespace","value":"\n"},
+  {"type":"NameTag","value":"mixed"},
+  {"type":"Punctuation","value":":"},
+  {"type":"TextWhitespace","value":" "},
+  {"type":"Literal","value":"123abc"},
+  {"type":"TextWhitespace","value":"\n"},
+  {"type":"NameTag","value":"port"},
+  {"type":"Punctuation","value":":"},
+  {"type":"TextWhitespace","value":" "},
+  {"type":"LiteralNumber","value":"8080"},
+  {"type":"TextWhitespace","value":"\n"},
+  {"type":"NameTag","value":"ratio"},
+  {"type":"Punctuation","value":":"},
+  {"type":"TextWhitespace","value":" "},
+  {"type":"LiteralNumber","value":"3.14"},
+  {"type":"TextWhitespace","value":"\n"},
+  {"type":"NameTag","value":"neg"},
+  {"type":"Punctuation","value":":"},
+  {"type":"TextWhitespace","value":" "},
+  {"type":"LiteralNumber","value":"-17"},
   {"type":"TextWhitespace","value":"\n"}
 ]

Claude AI: perf check

Only strings starting with a digit but not fully numeric (e.g. 1.0-beta) get slower; ~0.9µs → ~1.7µs per value. Everything else (non-numeric values, valid numbers) is unaffected or slightly faster.

[farfromexact]

The proposed direction addresses the root cause: the old trailing word boundary can accept the numeric prefix of a larger plain scalar such as 1.0-beta or 2 val. I would keep the fix, but define the boundary in terms of a complete YAML scalar rather than only copying the TextMate expression.

The regression matrix should include:

• plain scalars such as 1.0-beta, 2 val, 123abc, and valid 3.14
• signs, exponent notation, hex, octal, .inf, and .nan
• block comments and values followed by a colon
• flow collections such as [1.0-beta, 2 val]
• tags, anchors, quoted values, and multiline input

The proposed lookahead contains // as a delimiter, but // is not a YAML comment marker, so I would verify that this is intentional for the lexer context rather than inherited from a different grammar. If the regex engine supports the lookahead, the key invariant should be: classify as LiteralNumber only when the numeric pattern consumes the complete plain scalar; otherwise emit one Literal token for the whole value.

Adding the cases to yaml.actual and yaml.expected is the right validation. I would also keep the benchmark because the extra boundary check is cheap, but it is worth checking long non-numeric scalars and flow collections as well as the two examples.