ReviewOS

zig-utils/zig-js

Lexer: template substitutions are scanned by byte heuristics, rejecting valid regex and division inside ${}

#948
Closed glennmichael123 opened this 10 days ago ·
10 days ago

Found in the post-#933 survey and re-executed against node v24.4.1, on a Debug build of main (f2bafeb) and with #944's lexer fix applied. The results are the same on both builds. node accepts every program; zig-js rejects each one:

programnodezig-js
var i = 1; var x = `${ i++ / 2 }`; x0.5UnterminatedString
var x = `${ typeof /}/ }`; xobjectUnterminatedString
var x = `${ (() => { return /}/.source })() }`; x}ExpectedToken
var x = `${ void /`/ }`; xundefinedUnterminatedString
var x = `${ [1].map(function(){ return /"/.source })[0] }`; x"UnterminatedString
var x = `${ "source" in /}/ }`; xtrueUnterminatedString
var x = `${ (function(v){ switch (v) { case /}/.source: return 12 } })("}") }`; x12ExpectedToken
function t(s, v){ return v; } var x = t`${ typeof /}/ }`; x (tagged)objectUnterminatedString
s.split("").map(c => { return /"/.test(c) ? """ : c }).join("") inside ${}a"bUnterminatedString

The survey's template probes contain 21 such programs in all. Today 18 of them fail, plus 3 realistic ones (escape/HTML helpers and a counter).

Cause: the end of a ${ … } substitution is found by a byte-level scan (lexTemplate/substEnd in src/lexer.zig) that guesses regex vs division from the last character. It does not know keywords (return, typeof, void, in, case, delete, new, yield, await), postfix ++/--, or whether a } closed a block. Any regex or division whose bytes contain a quote, brace or backtick therefore breaks brace matching. Each nesting level then re-lexes its substitution with a fresh sub-parser.

Cost: as a parse-only probe, a 400 KB array literal nested in D templates takes 0.67 / 0.78 / 1.09 / 1.77 s at D = 32 / 64 / 128 / 256 (one sample each, survey run), against a flat 0.49 / 0.55 / 0.43 / 0.45 s in D parentheses. Each level rescans the payload.

Fix direction: template head/middle/tail tokens emitted into the main token stream, with the substitution lexed by the main lexer rules, as the spec's grammar does. That shares its root with #947 (regex vs division needs the parser's goal symbol), so the two should be designed together. Freeze a D x payload growth row in bench/frontend_parse.zig before changing code.

10 days ago

Completed on main across four small commits:

  • 0837949c: froze nested-template and parenthesis growth rows before changing the lexer.
  • c86ea2e1: replaced byte-level substitution rescanning with template head/middle/tail tokens in the ordinary lexer stream.
  • 38db24c0: checked in the exact-parent performance report and raw samples.
  • 70234924: completed the postfix-update division context, including `${i++ / 2}`.

All nine programs listed in the issue now pass in the focused frontend matrix, including tagged templates and the escape-helper shape. The exact-parent depth-256 diagnostic report records 203.366 ms → 9.774 ms (0.048x), 0.0673x instructions, 0.5246x allocated bytes, and 0.5843x peak RSS across seven order-balanced pairs. Evidence is in docs/.data/exact-parent-template-substitution-nested-256-2026-09-21.md.

Sign in to comment on this issue.