#951 · zig-utils/zig-js

Parent: #899

Problem

A current-main direct differential against macOS JavaScriptCore 625.1.22 still loses the offending token kind and renders the internal parser tag plus origin/position inside .message:

source passed to `eval`	zig-js (`e.message`)	JavaScriptCore
`var q = ;`	`eval: UnexpectedToken at 1:10`	`Unexpected token ';'`
`if (true) {`	`eval: ExpectedToken at 1:12`	`Unexpected end of script`
`var a b`	`eval: UnexpectedToken at 1:7`	`Unexpected identifier 'b'. Expected ';' after variable declaration.`
`1 2`	`eval: UnexpectedToken at 1:3`	`Unexpected number '2'`
`'a' 'b'`	`eval: UnexpectedToken at 1:5`	`Unexpected string literal 'b'`
`2n 3n`	`eval: UnexpectedToken at 1:4`	`Unexpected token '3n'`
`if if`	`eval: ExpectedToken at 1:4`	`Unexpected keyword 'if'. Expected '(' to start an 'if' condition.`

The probe executed both engines directly; the JSC side used the C API so Home's known private-syntax transpilation defect was not in the path.

The parser already has DiagnosticToken, failWithTokenReason, and structured line / column / byteOffset Error properties from earlier #899 slices. A static audit on this revision still finds 159 bare parser/lexer returns of the compact error tags. This child owns the generic token-kind renderer and the first unambiguous call-site conversions, not every context-specific clause in the umbrella.

Requirements

Retain the exact offending source spelling and classify punctuation, identifier, keyword, number, string literal, BigInt, and EOF as JavaScriptCore does.
Render prose through the existing structured diagnostic path; keep position in Error properties rather than .message.
Attach the diagnostic at the grammar point where the error is known. Do not infer it later by rescanning source or mapping a compact error tag globally.
Preserve the compact ParseError categories and all syntax acceptance behavior.
Split context-specific clauses into bounded follow-ups when the parser state needed to render them differs; do not hardcode source strings or special-case the probe.

Acceptance

The seven-case direct differential above is exact against the same raw JavaScriptCore oracle.
Focused parser and eval/Function tests pin token text, token kind, and exact structured location.
Exact-parent test262 comparisons cover every changed grammar subtree with zero regressions.
A post-change audit accounts for the converted bare returns and identifies the next remaining diagnostic clusters on #899.

No public performance claim is required; the generic failure path must not add work to successful parsing.

Landed on main in 54fc295.\n\nThe parser now retains the offending source spelling and classifies punctuation, identifiers, keywords, numbers, strings, BigInt, and EOF through its existing structured diagnostic state. Variable declarations and if conditions carry their JSC context clauses. Function and the generator/async dynamic constructors also distinguish caller-body EOF from their synthetic wrapper suffix without weakening the boundary-injection validation.\n\nEvidence:\n- raw JavaScriptCore 625.1.22: all seven messages exact through eval, and all seven exact through Function\n- parser module: 229/229\n- focused eval/Function diagnostic and dynamic-boundary filters: 4/4 linked tests\n- exact parent e8bb5a6f, test262 expressions: 11,005/11,038 before and after, 33 identical known failures\n- exact parent, test262 statements: 9,304/9,337 before and after, 33 identical known failures\n- both complete failure transcripts byte-identical; zero flips\n\nThe post-change static audit moves every expect call (56 sites) and statement terminator (8 sites) onto failure-only structured rendering and reduces direct bare error returns 159 -> 155. The largest remaining clusters are statement/control rules (16), binding/catch/parameter conflicts (19), object/pattern rules (19), module import rules (13), template escapes (9), and for-head rules (5). Those remain on #899 for bounded follow-ups.