Follow-up to #944. Every row below was executed, with node v24.4.1 as the oracle, on a Debug build of main (f2bafeb) and again with #944's lexer fix applied. The results are the same on both builds:
| program | node | zig-js |
|---|---|---|
var r = 0; l: {} /"/.test("\"") && (r = 1); r (a labelled block) | 1 | UnexpectedCharacter |
var r = 0; switch (0) { case 0: {} /"/.test("\"") && (r = 1); } r (a block in a case) | 1 | UnexpectedCharacter |
var z = class {} / 2; String(z) (a class expression's body) | NaN | UnterminatedString |
function* g(){ yield /"/.source; } g().next().value | " | UnterminatedString |
(async function(){ out = await /"/.source; })() | runs | UnterminatedString |
Why #944 did not fix these: zig-js lexes the whole source before parsing, deciding "regex or division" for each / from the tokens before it. #944 made that decision exact wherever the preceding tokens determine it. These five cases depend on context the tokens do not carry:
- whether a
{after:opens a block (a label orcase) or an object literal value; - whether a class body closes a declaration or an expression;
- whether
yield/awaitare keywords here, which depends on the enclosing function kind and the goal symbol.
The parser does repair a / it meets in operand position, re-reading the source as a regex (since #944, yield /a/.source works). That repair cannot help when lexing the misread bytes fails first: any quote, backtick or brace inside the regex does that. Nor can it reset lexer state, such as the brace stack, that tokens inside the misread span already changed.
Options:
- Teach the lexer the missing context: label and
casecolons, class bodies, and generator/async function bodies, including methods. The pre-tokenized design stays, but part of the parser's state is replicated in the lexer, which is a bug surface of its own. - Tokenize on demand with the parser's goal symbol (InputElementDiv or InputElementRegExp), as the spec and other engines do. The lexer then needs no brace/paren stacks, and re-lexing from a token needs only its offset. The cost is in the lookaheads that scan ahead over tokens (
arrowAhead,classicForHeadAhead, the #933 item 8b paren index): they must lex ahead and discard on a disagreement. That has to be measured with the #926/#928 growth rows so no new quadratic appears. The template-substitution scanner (#948) has the same root cause and would be replaced by the same change.
I'd take option 2 together with #948 and measure first. This is a design choice, so it is recorded here rather than worked around.