ReviewOS

zig-utils/zig-js

Frontend: normalize numeric separators in one exact allocation

#544
Closed chris opened this 9 days ago ·
9 days ago

Parent: #496 Related: #461, #480, #493

Problem

The numeric-literal lexer validates separator placement, then stripSeparators grows an arena-backed byte list one byte at a time before parseInt/parseFloat. Wide sources containing many otherwise ordinary separated literals therefore perform allocator growth/remap work proportional to token length even though the normalized output length is known exactly after validation.

This child is benchmark-first. The committed fixture must freeze ordinary decimal-separated 1K/2K/4K rows plus an unseparated 4K semantic control before the lexer changes.

Scope

  • Measure parser-only construction of exact, safely representable decimal literals with multiple separators per token.
  • Normalize validated literal bytes into one exact owned allocation.
  • Preserve the zero-allocation source slice for literals without separators.
  • Preserve decimal/radix/BigInt values, legacy-octal rejection, separator placement errors, source offsets, raw token text, and OOM behavior.

No-workaround rules

  • No pre-normalized benchmark source, source construction in the timed boundary, skipped parsing/value validation, reduced width, hash/cache recognition, or different numeric values in the control.
  • No new arbitrary numeric-length limit and no relaxation of adversarial-complexity or memory bounds.
  • Do not modify stacks, queues, jobs, Promise/microtask behavior, or Thread/Worker lifecycle.

Acceptance

  • Benchmark-first 1K/2K/4K separated rows and an unseparated 4K control commit exact values, checksums, warmup, and parser-only timed boundaries.
  • Exact-parent evidence reports wall, retired instructions, allocation/RSS evidence, dispersion, and identical checksums.
  • Separated normalization uses one exact allocation; unseparated literals allocate no normalization storage.
  • Decimal integer/fraction/exponent, binary/octal/hex, small/large BigInt, invalid placement, offsets, and OOM witnesses pass.
  • Focused frontend/lexer, full units, test262 numeric-literal coverage, and TSan pass with no regressions.
  • Raw evidence/report are checked in before any public performance claim.
9 days ago

Benchmark-first fixture commit: 0a212b51

Implementation commit: 4fdff17b

The lexer now counts validated separators, allocates the exact normalized length once, and fills it in one pass. Literals without separators still borrow their source slice with zero normalization allocation. Added exact-allocation/zero-allocation/OOM witnesses plus decimal integer/fraction/exponent, radix, small/large BigInt, invalid-placement, raw-text, and offset coverage.

Validation:

  • focused separator tests under TSan: 4/4 pass, no leaks
  • lexer tests: 12/12 pass; frontend suite: 12/12 pass
  • full sharded units: 1,608 pass, 1 intentional skip, 0 fail, 0 leak
  • exact-parent test262: numeric literals 157 -> 157 passing; BigInt literals 59 -> 59 passing (including 46 separator cases); 0 flips

The frozen benchmark checksums are 102400000524800 (1K), 204800002098176 (2K), and 409600008390656 (4K separated and unseparated control). Diagnostic A/B indicates lower retired instructions on separated rows with the unseparated control effectively unchanged, but I am not making a public performance claim until exact-parent raw evidence and its report are checked in.

9 days ago

Completed and pushed to main:

  • 0a212b51 bench(frontend): add numeric separator growth rows (#544)
  • 4fdff17b perf(frontend): normalize numeric separators once (#544)
  • ad8403fb bench(frontend): expose exact parser counters (#544)
  • eb657c07 fix(frontend): measure logical parser allocations (#544)
  • c7e4f7f7 docs(frontend): publish numeric separator evidence (#544)

The checked aggregate report links all four raw artifacts and per-row reports.

Seven order-balanced exact-parent pairs show that separated 1K/2K/4K inputs remove exactly one logical allocation/resize request per literal across ten parser jobs (10,240 / 20,480 / 40,960 fewer), reduce cumulative parser allocation bytes by 11.6–14.2%, and reduce stable retired instructions by 5.4–5.6%. The stable 4K wall row is 0.887x parent; smaller wall rows remain explicitly diagnostic because of dispersion. The unseparated 4K control keeps allocation requests and bytes exactly unchanged and records 1.005x wall and 0.9944x peak RSS.

Correctness and safety gates:

  • focused TSan: 4/4 pass, no leaks
  • lexer: 12/12 pass; focused frontend: 12/12 pass
  • full sharded units: 1,608 pass, 1 intentional skip, 0 fail, 0 leak
  • exact-parent test262: numeric literals 157 → 157 passing; BigInt literals 59 → 59 passing, including 46 separator cases; 0 flips
  • exact allocation, zero-allocation plain-literal borrowing, OOM, decimal/fraction/exponent, radix, small/large BigInt, invalid placement, raw-text, and offset witnesses pass
  • all four raw artifacts validate against zig-js-performance-attribution-v1

The docs renderer reached its manifest gate but exposed unrelated repository-wide BunPress output drift; no unrelated manifest change was accepted into this issue.

Sign in to comment on this issue.