ReviewOS

zig-utils/zig-js

Lexer: `/` after a postfix update, a keyword-named property, or an if/while/for head is misread, rejecting valid code

#944
Closed glennmichael123 opened this 10 days ago ·
10 days ago

Found in the post-#933 survey. Every row below was executed, with node v24.4.1 as the oracle, against a Debug build of main (f2bafeb). node accepts each program; zig-js rejects it before parsing starts.

The lexer tokenizes the whole source first and decides "regex or division" for each / from the previous token. Three of those decisions are wrong, and all three can be decided exactly from tokens alone:

#programnodezig-js
postfix ++/--var i = 1; var z = i++ / 2; z0.5UnterminatedString
var t = 4; var z = t--/2; z (minified)2UnterminatedString
var x = 10, y = 2; var z = x++ /y/ 1; z5UnexpectedToken
var n = 3; var z = n-- / 3; var s = "a/b"; z + s1a/bUnterminatedString
keyword as a property namevar o = { default: 8 }; o.default / 24UnterminatedString
s.delete / s.total, s.in / 3, r.return / 2numbersUnterminatedString
m?.default / 52UnterminatedString
) of an if/while/for headif (1) /"/.test("\"") && (r = 1)1UnexpectedCharacter
if (1) /}/.test("}") && (r = 1)1UnterminatedString
while (i++ < 1) /}/.test("}") && (r = 1)1UnterminatedString
for (var i = 0; i < 1; i++) /"/.test("\"") && (r = 1)1UnexpectedCharacter
function f(a){if(a)/"/.test(a)&&(a=1);return a} (minified)1UnexpectedCharacter

Causes (src/lexer.zig, regexAllowed):

  • after ++/-- a regex is always allowed, but a postfix update ends an expression. Postfix is decidable: an expression ended right before the ++ on the same line (UpdateExpression has [no LineTerminator here]). The prefix form must stay a regex; ++/a/g.lastIndex runs correctly today.
  • after an identifier, the operand-keyword list (return, in, default, delete, …) is consulted even when the name follows ./?., where it is an IdentifierName, not the keyword.
  • after ) a regex is never allowed, but a statement follows the ) of an if/while/for/with/for await head (a do-while's ) gets a semicolon inserted, so a statement follows it too).

One parser bug on the same path: yield /a/.source (in a generator) fails with UnexpectedToken even though its regex has no quotes. The / after yield is lexed as division; parseYield re-reads it as a regex but takes only that primary, so .source is left over. await has the same shortcut.

Out of scope here, filed separately: cases that need the parser's context rather than the previous token. Examples are a regex containing a quote or brace after yield/await, a label's block }, a class body's }, and template substitutions (${ n++ / 2 } is scanned by a separate byte heuristic).

Done: each program above returns node's value, and the prefix ++/re/ and return /re/ controls still lex as regex. Pinned as focused frontend cases. Debug before/after --diag over the affected test262 subtrees shows 0 failures on both sides.

10 days ago

Taking this: the fix is written and its before/after gate (probe corpus, Debug test262 diff, focused suites) is running. It lands together with #944/#945/#946 as separate commits.

10 days ago

Implemented and pushed to main in 70234924.

Evidence:

  • The 16 previously failing probes now return the Node values; prefix ++/a/g.lastIndex and return /a/.source remain unchanged.
  • Added coverage for postfix updates across same-line and line-terminating comments, Unicode line terminators, escaped and optional keyword-named properties, if/while/for/with/for await/do-while heads, nested control heads, and yield/await member tails.
  • Exact-parent Debug test262 comparison: 1,033 affected cases, 0 fail before and 0 fail after.
  • Lexer module: 27/27. Parser module: 224/224. Both focused frontend matrices pass.
  • ReleaseFast allocation counts are unchanged. Hardware instructions are within 0.2% on the ordinary 2,048-identifier row and +1.5% on the deliberately control-heavy 1,024-head stress row.
10 days ago

I checked #944–#948 independently on a Debug build of 538a667a, with node v24.4.1 as the oracle:

  • Survey corpus (345 programs): the old base (f2bafeb) differed from node on 92 programs; main now differs on 13, and none of those involve these issues. Twelve are var arguments declared by a direct eval in parameter expressions, where zig-js follows test262 and JSC and V8 does not implement the rule. The last one is the --eval runner printing a function value as [object Object]. No program that agreed with node before disagrees now.
  • Pinned regex-vs-division set: 19 programs, covering postfix/prefix updates across line breaks and multi-line comments; if/while/for/with/for await/do-while heads; keyword-named and optional properties; a method named if; and a yield operand continuing past its regex. All return node's values.
  • Arrows (#945): a 10-program acceptance set plus the yield-as-identifier value case, and 16 rejections (labels, [Yield], strict parameters). All match.
  • Function constructors (#946): 16 early-closing bodies across the four constructors throw SyntaxError, and the injected code never runs; complete bodies (trailing //, leading -->) still work.
  • #947: labelled and case blocks, a class expression body, and a quoted regex after yield/await, plus the #948 template cases. All match.

I had a parallel fix for #944–#946 gated locally; it is dropped as a duplicate.

Sign in to comment on this issue.