ReviewOS

home-lang/home

compression: compile real implementations and migrate them to std.Io

#887
Closed glennmichael123 opened this 9 days ago ·
9 days ago

Problem

packages/compression currently has no build graph that compiles its implementations:

  • zig build test points to nonexistent tests/gzip_test.zig and tests/zstd_test.zig.
  • the default shipped build points to nonexistent examples/gzip_example.zig, examples/zstd_example.zig, and examples/benchmark.zig.
  • the only existing test file, tests/compression_tests.zig, checks format constants and imports none of src/gzip.zig, src/zstd.zig, or the public package root.

Both the shipped and test commands therefore fail at file loading before reaching the package. This masks a substantial Zig 0.18 migration in src/gzip.zig/src/zstd.zig: removed allocator-bound ArrayList operations, removed legacy std.io reader/writer surfaces, and implicit wall-clock access (std.time.timestamp).

This is the expensive std.Io class identified by #884, not a safe blanket rename.

Required work

  • Build and test the actual public compression root.
  • Replace nonexistent build targets with real artifacts or remove unsupported targets deliberately.
  • Inject std.Io wherever compression metadata needs wall-clock time or streams.
  • Port buffers/readers/writers to current APIs without changing wire formats.
  • Add round-trip and malformed-input tests for every advertised implementation.
  • Ensure the same modules/configuration are used by tests and installed artifacts.
  • Verify Debug and ReleaseFast on the pinned compiler and Zig 0.18 master, plus compile-only Linux targets.

No placeholder tests, test weakening, or silent time/entropy fallback.

Related: #884, #66.

9 days ago

Pushed the first Zig 0.18 compression slice directly to main in 3443c2ea5 (fix(compression): restore core codec compatibility (#887)).

This activates and repairs the Brotli/LZ4/Snappy implementations through the real public compression root:

  • migrated allocator-bound ArrayList calls to the explicit-allocator API used by Zig master;
  • replaced removed array repetition expressions and removed std.crypto.random test dependence;
  • made valid empty LZ4 frames retain their required size header;
  • stopped Snappy hash positions from wrapping at 64 KiB;
  • corrected Snappy's maximum encodable offset and copy-tag length encoding;
  • added strict Snappy output-length validation rather than accepting truncated output.

Verification (all under the 3,840 MB guard):

  • Zig 0.18.0-dev.146+35accc06e: 21/21 public-root tests pass in Debug and ReleaseFast.
  • Home pinned Zig 0.17.0-dev.2163+89ff10d56: 21/21 pass in Debug and ReleaseFast.
  • x86_64 Linux ReleaseFast compile-only passes on both compilers.

This does not close the issue. The next slice is the build graph plus gzip/zstd: replace the nonexistent test/example roots, make tests import the same module shipped to users, port the remaining stream APIs to std.Io, and add malformed-input coverage.

9 days ago

Second #887 slice is on main in e0e205d92 (fix(compression): migrate package codecs to std.Io (#887)).

What changed:

  • the package build now compiles the actual public compression module and the existing real test root;
  • removed build references to five nonexistent test/example/benchmark files;
  • added a compile-only check step for cross-target CI;
  • migrated gzip time, readers, writers, buffers, compressor, and decompressor to std.Io;
  • injected std.Io into gzip so mtime comes from the caller's clock;
  • kept gzip CRC32 and uncompressed-size validation explicit (the current std flate reader parses footer metadata but does not compare it);
  • migrated zstd framing/stream facades to std.Io and made truncated blocks, invalid tags/matches, trailing block data, size mismatches, dictionary mismatches, and checksum failures hard errors;
  • replaced constant-only tests with public API round-trip, streaming, checksum, invalid-magic, empty-frame, and truncation tests.

Verification under the 3,840 MB guard:

  • Zig 0.18.0-dev.146+35accc06e: package zig build test passes in Debug and ReleaseFast.
  • Home pinned Zig 0.17.0-dev.2163+89ff10d56: the same passes in Debug and ReleaseFast.
  • zig build check -Dtarget=x86_64-linux-gnu -Doptimize=ReleaseFast passes on both compilers.
  • peak observed tree footprint was 312 MB.

Keeping #887 open for one remaining quality slice: explicit malformed-input tests for the Brotli/LZ4/Snappy public decoders (their round-trip suites are active and green from 3443c2ea5).

9 days ago

Completed the remaining #887 work in two final commits:

  • 995b98946 — runs the shipped module's own tests as well as the external public API tests; rejects malformed Brotli/LZ4/Snappy streams; enforces declared output lengths; supports Snappy's valid four-byte-offset copy tag; fixes valid empty Brotli frames.
  • 4294a44d9 — restores and pins the established zstd block wire bytes with a fixed compatibility fixture (including the legacy empty-frame representation), correcting an encoding change caught during final acceptance review.

Final package matrix, under the 3,840 MB guard:

  • Zig 0.18.0-dev.146+35accc06e: 32/32 tests pass in Debug and ReleaseFast.
  • Home pinned Zig 0.17.0-dev.2163+89ff10d56: 32/32 tests pass in Debug and ReleaseFast.
  • Both compilers compile both test roots for x86_64-linux-gnu ReleaseFast.
  • Highest observed guarded tree footprint: 596 MB.

The 32 tests consist of 24 tests rooted in the same public module Home ships plus 8 external public API tests. Round-trip and malformed-input coverage now executes for gzip, zstd, Brotli, LZ4, and Snappy. The build no longer references nonexistent artifacts, gzip's clock is injected through std.Io, all stream/buffer APIs are current, and the check target is compile-only for cross-target CI.

Sign in to comment on this issue.