Lexer: enforce Unicode identifier properties

Parent: #496 Related: #493, #538, #539, #460, #461

Problem

The lexer currently treats almost every non-ASCII code point that is not whitespace, a line terminator, ZWNJ/ZWJ-at-start, U+2E2F, or U+180E as an identifier character. That permissive approximation accepts punctuation and symbols that ECMAScript excludes from Unicode ID_Start/ID_Continue. Source text is attacker-controlled, and accepting invalid source changes parse boundaries, early errors, and potentially the meaning of adjacent tokens.

The owned zig-regex dependency already publishes binary-searched Unicode 17.0.0 ID_Start and ID_Continue tables for ECMAScript named captures. The lexer should consume the same exact generated authority rather than maintaining an incomplete approximation or duplicating tables.

Scope

• Commit parser-only 1K/2K/4K raw-Unicode identifier rows before changing validation, with exact AST identities, checksums, warmup, allocation replay, and parser-only timed boundaries.
• Preserve the branch-cheap ASCII $/_/letter/digit path.
• Validate raw and escaped non-ASCII identifier starts/continues against Unicode 17 ID_Start/ID_Continue, plus ECMAScript $, _, U+200C ZWNJ, and U+200D ZWJ additions.
• Reject invalid punctuation, symbols, unassigned/surrogate code points, invalid UTF-8, invalid escapes, and continue-only characters at start without consuming them as identifiers.
• Preserve keyword-after-escape early errors, raw token positions, decoded identifier text, private names, RegExp named captures, source lifetime, and OOM behavior.

No-workaround rules

• No hand-maintained allow/deny exceptions, ASCII-only mode, locale predicates, host-library Unicode behavior, source rewriting, deferred syntax errors, or benchmark recognition.
• Do not duplicate generated Unicode tables into zig-js or change the dependency outside its existing public Unicode property API.
• Do not modify stacks, queues, jobs, Promise/microtask behavior, or Thread/Worker lifecycle.

Acceptance

• Benchmark-first 1K/2K/4K rows freeze exact valid raw-Unicode sources and outputs before the lexer change.
• Raw and escaped start/continue witnesses cover BMP, astral, Other_ID_*, combining marks, digits, connector punctuation, ZWNJ/ZWJ, symbols, punctuation, whitespace, unassigned code points, and malformed UTF-8/escapes.
• Lexer and RegExp named-capture identifier classification agree for every Unicode scalar value, modulo the grammar-specific ASCII/ZWJ additions.
• Exact-parent evidence reports wall, retired instructions, allocations/bytes, RSS, dispersion, and identical checksums; ASCII and escaped-identifier controls do not regress materially.
• Focused lexer/frontend, full units, TSan, identifier/private-name test262, and forced parser/execution paths pass with no regressions.
• Raw evidence and report are checked in before any performance claim.

Benchmark-first commit b7bbea78 is on main with no lexer/runtime change.

Frozen parser-only rows:

• representative_frontend_unicode_identifiers_1024 → checksum 19121
• representative_frontend_unicode_identifiers_2048 → checksum 39345
• representative_frontend_unicode_identifiers_4096 → checksum 79793

Each generated function cycles raw valid Unicode 17 identifier starts from Greek, CJK, astral Deseret, and Other_ID_Start, follows them with a combining ID_Continue mark, and adds a unique ASCII ordinal. Source construction happens once before all warmups and scored parser boundaries. Every scored parse validates the exact decoded prefix and ordinal for every parameter; the checksum is structural, not hash-only.

The existing ASCII and escaped-identifier rows remain available as controls. The next commit will replace only the permissive non-ASCII classifier with the dependency’s public generated Unicode property API while retaining the current ASCII fast path.

Completed on main in three traceable commits:

• b7bbea78 bench(frontend): add Unicode identifier rows (#547)
• 085cb622 fix(lexer): enforce Unicode identifiers (#547)
• 61396be6 docs(frontend): publish Unicode identifier evidence (#547)

The lexer now keeps its local ASCII fast path and classifies every non-ASCII scalar through zig-regex's generated Unicode 17 ID_Start/ID_Continue authority, including the ECMAScript $, _, ZWNJ, and ZWJ additions already owned by that public API. Invalid symbols, punctuation, unassigned code points, surrogate escapes, malformed UTF-8, and continue-only starts are rejected at the lexical boundary for raw and escaped identifiers and private names.

Validation:

• exhaustive lexer/RegExp agreement for every Unicode scalar value
• 18/18 lexer units and 12/12 focused frontend cases
• full units: 1,614 passed, 1 skipped, 0 failed, 0 leaked
• focused lexer and parser TSan: clean
• exact-parent test262 identifiers: 268 → 268, zero failures/flips
• exact-parent filtered private-name invalid cases: 12 → 12, zero failures/flips
• all five checked-in raw artifacts pass performance-attribution validation

The exact-parent diagnostic report publishes the correctness cost rather than hiding it: raw-Unicode instructions rise 7.7–7.8% and stable 4K wall rises 9.3%, with unchanged allocations and near-linear growth. The ASCII and escaped-ASCII controls change instructions by 0.79% and 0.45%; neither establishes a wall regression.

Aggregate evidence: docs/.data/exact-parent-frontend-unicode-identifiers-2026-08-12.md