also looking at this
feat(video): add native AVC/HEVC and synchronized AAC MP4 streaming
#913
14 files
+1119
-0
| @@ -0,0 +1,89 @@ | ||
| 1 | # Native AVC/HEVC and streamed MP4 | |
| 2 | ||
| 3 | `src/codecs/video/native_encoder.zig` is the supported standalone Home integration | |
| 4 | for libx264/libx265. Compile `native_encoder.c` against the installed vendor | |
| 5 | headers, then link x264/x265 directly. Only Home-owned fixed scalar structs cross | |
| 6 | the C/Zig boundary; the old handwritten vendor structures in x264.zig/x265.zig | |
| 7 | are not reused. There is no FFmpeg/Libav or encoder subprocess dependency. | |
| 8 | The Zig wrapper works with Zig 0.15.2 and the Home 0.17 development toolchain. | |
| 9 | Vendor libraries keep their GPL/commercial licenses; Home modules remain MIT. | |
| 10 | ||
| 11 | The wrapper accepts checked I420 planes, returns borrowed length-prefixed NAL | |
| 12 | packets, and preserves microsecond PTS. Serialize operations per encoder; packets | |
| 13 | expire on the next call. Encode may return null while AVC VFR buffers one input. | |
| 14 | Always drain `flush()` until null before deinit. Encoding after drain or error is | |
| 15 | rejected. The integration fixes presets to veryfast, one frame thread, no B frames | |
| 16 | or lookahead, closed GOPs, AVC High / HEVC Main, 8-bit 4:2:0. Quality 1..100 maps | |
| 17 | linearly to CRF 51..0 (AVC minimum 1 to retain High-profile compatibility); bitrate >0 selects ABR in bits/sec, not strict CBR. Zero GOP | |
| 18 | means 60. HEVC uses CTU16 across instances to avoid global x265 CTU mismatches. | |
| 19 | ||
| 20 | `rgb_i420.zig` is a bounded, stride-checked conversion from RGB24. It improves | |
| 21 | Home's existing RGB conversion approach with actual 2x2 chroma averaging. Output | |
| 22 | is limited-range BT.601 with sRGB transfer / BT.709 primaries. This is SDR. | |
| 23 | ||
| 24 | `src/containers/mp4_stream.zig` provides a sink-based single-video-track fragmented | |
| 25 | MP4 writer, adapted from Home's existing MP4 box code. It writes complete AVC/HEVC | |
| 26 | sample descriptions, avcC/hvcC from encoder headers, mvex/trex, accurate moof/traf/ | |
| 27 | tfhd/trun offsets, 64-bit tfdt, and one mdat per sample. Payload writes go straight | |
| 28 | to the sink; there is no whole-recording buffer or growing sample table. | |
| 29 | ||
| 30 | The writer accepts length-prefixed samples with PTS=DTS and explicit microsecond | |
| 31 | durations. A recorder should retain one encoded packet to compute duration from | |
| 32 | the next timestamp, preserving capture gaps; use nominal duration for the last. | |
| 33 | The writer rejects malformed NAL lengths and non-increasing decode times. Partial | |
| 34 | sink failure poisons the writer. It has a fixed 16KiB header bound, no seek index, | |
| 35 | no audio, and supports the fixed 8-bit profiles emitted by this adapter. | |
| 36 | ||
| 37 | Home's older pure-Zig H.264/HEVC slice encoders and fragmented-muxer implementation | |
| 38 | remain legacy code; their feature lists are not conformance evidence for these | |
| 39 | new modules. The new modules are available through video.native_encoder, | |
| 40 | video.rgb_i420 and video.mp4_stream without pulling in the old codec graph. | |
| 41 | ||
| 42 | From the repository root on Linux: | |
| 43 | ||
| 44 | ```sh | |
| 45 | sudo apt install libx264-dev libx265-dev pkg-config | |
| 46 | packages/video/test-native-media.sh | |
| 47 | ``` | |
| 48 | ||
| 49 | Focused tests validate geometry, original timestamps, all frames through drain, | |
| 50 | invalid lifecycle, averaged chroma, fragment sizes/data offsets/64-bit time and | |
| 51 | sink failure. They pass in Debug/ReleaseSafe with Zig 0.15.2 and Home's installed | |
| 52 | 0.17 development toolchain. zig-utils/zig-camera integration additionally checks | |
| 53 | MP4 regular/gapped capture timelines and GOPs, and decodes AVC/HEVC using independent | |
| 54 | Apple AVFoundation and Linux OpenH264/libde265. These are synthetic correctness | |
| 55 | tests, not Pi hardware FPS, long-run reliability or compression-quality benchmarks. | |
| 56 | ||
| 57 | References: [x265 API](https://x265.readthedocs.io/en/master/api.html), | |
| 58 | [x264 public header](https://code.videolan.org/videolan/x264/-/blob/master/x264.h), | |
| 59 | [ISO BMFF streaming requirements](https://www.w3.org/TR/mse-byte-stream-format-isobmff/). | |
| 60 | ||
| 61 | ## AAC-LC with synchronized fragmented MP4 | |
| 62 | ||
| 63 | `mp4_stream.Writer.initAv` extends the same bounded video muxer with an optional | |
| 64 | AAC track. Its `AudioConfig` consumes the encoder's real AudioSpecificConfig, | |
| 65 | rate/channels/bitrate and codec delay. Use the native encoder in | |
| 66 | [Home #918](https://github.com/home-lang/home/pull/918); the muxer itself has no | |
| 67 | codec dependency and remains independently testable. | |
| 68 | ||
| 69 | Audio raw access units use the sample-rate timebase and `mp4a`/`esds`; video | |
| 70 | retains microsecond capture timestamps. Pass a sink with both write and patch | |
| 71 | callbacks. One AAC packet is staged so the final duration can omit padding. | |
| 72 | After draining AAC, `finish(start_us, input_frames)` emits the final packet and | |
| 73 | patches only fixed header fields: 64-bit movie/track/media durations and edit | |
| 74 | lengths. Memory does not grow with recording duration. A seekable sink and clean | |
| 75 | finalization are required; pipe output and crash recovery are not provided. | |
| 76 | ||
| 77 | The audio edit list represents the initial offset and actual encoder delay. | |
| 78 | Global `sgpd` and per-fragment `sbgp` roll associations mark explicit decoder | |
| 79 | pre-roll. Without those associations Apple readers can apply an extra implicit | |
| 80 | priming trim. The final short fragment duration prevents padded tail playback. | |
| 81 | Both are checked by FAAD2/AVFoundation camera integration, including sub-frame | |
| 82 | input, 44.1/48kHz, stereo, video gaps and independently decodable split files. | |
| 83 | Do not infer AAC time from callback scheduling or assume a fixed codec delay. | |
| 84 | The caller must provide continuous PCM to the encoder and preserve sample counts. | |
| 85 | Microsecond movie/edit times can round by less than one PCM sample. | |
| 86 | ||
| 87 | Four focused muxer tests pass on Zig 0.15.2 and the Home development toolchain, | |
| 88 | including sink failure poisoning and fixed edit/duration patch values. No | |
| 89 | FFmpeg/Libav or complete Home media-graph validation is represented here. | |