ReviewOS

also looking at this

home-lang/home

feat(video): add native AVC/HEVC and synchronized AAC MP4 streaming

#913
Merged chris wants to merge feat/avc-hevc-mp4 into main
14 files +1119 -0
packages/video/NATIVE_MEDIA.mdadded+89-0
Changes to packages/video/NATIVE_MEDIA.md
@@ -0,0 +1,89 @@
1# Native AVC/HEVC and streamed MP4
2
3`src/codecs/video/native_encoder.zig` is the supported standalone Home integration
4for libx264/libx265. Compile `native_encoder.c` against the installed vendor
5headers, then link x264/x265 directly. Only Home-owned fixed scalar structs cross
6the C/Zig boundary; the old handwritten vendor structures in x264.zig/x265.zig
7are not reused. There is no FFmpeg/Libav or encoder subprocess dependency.
8The Zig wrapper works with Zig 0.15.2 and the Home 0.17 development toolchain.
9Vendor libraries keep their GPL/commercial licenses; Home modules remain MIT.
10
11The wrapper accepts checked I420 planes, returns borrowed length-prefixed NAL
12packets, and preserves microsecond PTS. Serialize operations per encoder; packets
13expire on the next call. Encode may return null while AVC VFR buffers one input.
14Always drain `flush()` until null before deinit. Encoding after drain or error is
15rejected. The integration fixes presets to veryfast, one frame thread, no B frames
16or lookahead, closed GOPs, AVC High / HEVC Main, 8-bit 4:2:0. Quality 1..100 maps
17linearly to CRF 51..0 (AVC minimum 1 to retain High-profile compatibility); bitrate >0 selects ABR in bits/sec, not strict CBR. Zero GOP
18means 60. HEVC uses CTU16 across instances to avoid global x265 CTU mismatches.
19
20`rgb_i420.zig` is a bounded, stride-checked conversion from RGB24. It improves
21Home's existing RGB conversion approach with actual 2x2 chroma averaging. Output
22is limited-range BT.601 with sRGB transfer / BT.709 primaries. This is SDR.
23
24`src/containers/mp4_stream.zig` provides a sink-based single-video-track fragmented
25MP4 writer, adapted from Home's existing MP4 box code. It writes complete AVC/HEVC
26sample descriptions, avcC/hvcC from encoder headers, mvex/trex, accurate moof/traf/
27tfhd/trun offsets, 64-bit tfdt, and one mdat per sample. Payload writes go straight
28to the sink; there is no whole-recording buffer or growing sample table.
29
30The writer accepts length-prefixed samples with PTS=DTS and explicit microsecond
31durations. A recorder should retain one encoded packet to compute duration from
32the next timestamp, preserving capture gaps; use nominal duration for the last.
33The writer rejects malformed NAL lengths and non-increasing decode times. Partial
34sink failure poisons the writer. It has a fixed 16KiB header bound, no seek index,
35no audio, and supports the fixed 8-bit profiles emitted by this adapter.
36
37Home's older pure-Zig H.264/HEVC slice encoders and fragmented-muxer implementation
38remain legacy code; their feature lists are not conformance evidence for these
39new modules. The new modules are available through video.native_encoder,
40video.rgb_i420 and video.mp4_stream without pulling in the old codec graph.
41
42From the repository root on Linux:
43
44```sh
45sudo apt install libx264-dev libx265-dev pkg-config
46packages/video/test-native-media.sh
47```
48
49Focused tests validate geometry, original timestamps, all frames through drain,
50invalid lifecycle, averaged chroma, fragment sizes/data offsets/64-bit time and
51sink failure. They pass in Debug/ReleaseSafe with Zig 0.15.2 and Home's installed
520.17 development toolchain. zig-utils/zig-camera integration additionally checks
53MP4 regular/gapped capture timelines and GOPs, and decodes AVC/HEVC using independent
54Apple AVFoundation and Linux OpenH264/libde265. These are synthetic correctness
55tests, not Pi hardware FPS, long-run reliability or compression-quality benchmarks.
56
57References: [x265 API](https://x265.readthedocs.io/en/master/api.html),
58[x264 public header](https://code.videolan.org/videolan/x264/-/blob/master/x264.h),
59[ISO BMFF streaming requirements](https://www.w3.org/TR/mse-byte-stream-format-isobmff/).
60
61## AAC-LC with synchronized fragmented MP4
62
63`mp4_stream.Writer.initAv` extends the same bounded video muxer with an optional
64AAC track. Its `AudioConfig` consumes the encoder's real AudioSpecificConfig,
65rate/channels/bitrate and codec delay. Use the native encoder in
66[Home #918](https://github.com/home-lang/home/pull/918); the muxer itself has no
67codec dependency and remains independently testable.
68
69Audio raw access units use the sample-rate timebase and `mp4a`/`esds`; video
70retains microsecond capture timestamps. Pass a sink with both write and patch
71callbacks. One AAC packet is staged so the final duration can omit padding.
72After draining AAC, `finish(start_us, input_frames)` emits the final packet and
73patches only fixed header fields: 64-bit movie/track/media durations and edit
74lengths. Memory does not grow with recording duration. A seekable sink and clean
75finalization are required; pipe output and crash recovery are not provided.
76
77The audio edit list represents the initial offset and actual encoder delay.
78Global `sgpd` and per-fragment `sbgp` roll associations mark explicit decoder
79pre-roll. Without those associations Apple readers can apply an extra implicit
80priming trim. The final short fragment duration prevents padded tail playback.
81Both are checked by FAAD2/AVFoundation camera integration, including sub-frame
82input, 44.1/48kHz, stereo, video gaps and independently decodable split files.
83Do not infer AAC time from callback scheduling or assume a fixed codec delay.
84The caller must provide continuous PCM to the encoder and preserve sample counts.
85Microsecond movie/edit times can round by less than one PCM sample.
86
87Four focused muxer tests pass on Zig 0.15.2 and the Home development toolchain,
88including sink failure poisoning and fixed edit/duration patch values. No
89FFmpeg/Libav or complete Home media-graph validation is represented here.