ñü.media: images, video and HDR in minimal wire bits

How khensu.ñü sends photos and video at the fewest bits on the wire while keeping resolution, colour and HDR, using the A20 Pro’s Neural Engine and media engines. Status: design. No ñü.media measurements exist yet. Every baseline figure below is an estimate from typical results for Apple’s encoders and the published literature; every ñü.media figure is a target. Section 8 is the benchmark that replaces them. Open questions in OPEN-QUESTIONS.md section H (52 and 55). Summary in PLAN.md, nu-media.

1. hardware it is built for

iPhone 18 Pro and Pro Max (Apple’s specifications page):

The A20 Pro has no “TPU”; Apple’s term is the Neural Engine. The iPhone has no AV1 encoder. Measured on an iPhone Air (section 8.2), Apple’s current hardware HEVC encoder matches real-time AV1 software at 720p and beats it at 4K, at a fraction of the power, so the video base layer is hardware HEVC Main 10 (H.264 only as a fallback for devices without HEVC Main 10). AV1 decodes in hardware on A17 Pro and later. ñü.media works on every iOS 26 device and is tuned for iPhone 18 Pro on iOS 27.

2. the wire budget

Everything is sized against the stealth tick (PROTOCOL.md): with the app open, a burst every 2 s with 3 payload slots of 2,048 bytes.

Bits are the primary unit below. KB means 1,024 bytes; MB means 1,000,000 bytes.

3. images: ImageIO against ñü.media

One full-screen photo on iPhone 18 Pro (3,162,132 px). Bits = bits per pixel (bpp) × pixels. Time is how long it takes to arrive inside the ticks.

What to expect honestly:

• At equal quality, ñü.media targets about half the bits of ImageIO HEIC. That figure is an estimate: the best learned image codecs measure roughly 10–20% fewer bits than VVC intra, VVC intra roughly 25% fewer than HEVC intra (HM reference), and a real-time hardware HEIC encoder needs more bits than HM. Combined, that is about 45–55% below ImageIO HEIC.
• 15–40 KB is a size cap, not “the same picture for free”. It is about what HEIC would need at 30–80 KB: softer than the camera’s original, but clean. Learned codecs degrade by smoothing, not by blocks and ringing, which is why they look better than HEIC at these sizes.
• Fidelity first: no generated texture (no GAN or diffusion decoders), so nothing appears that wasn’t in the photo.
• Live Photos are fully supported (11 Oct 2026). At screen size they travel inside the ticks like any photo: the still first (about 40 KB, shown in about 7 ticks), then up to 3 s of motion coded as a residual from the still, with Opus sound, for about 200 KB in total (design target; about 34 ticks, so it moves at about 70 s). Apple’s HEIC + HEVC pair at the same size is about 1.4 MB. The receiver rebuilds a real Live Photo (PHLivePhoto from a still and a paired video with the same asset identifier), so it plays with a long press and saves to Photos as a Live Photo.
• Originals (full 48 MP, ProRAW, full-resolution Live Photos) don’t go through the ticks. They go through your own node, the charging window or the trickle (PLAN.md, Stealth).

4. video: hardware h.264 and h.265 against ñü.media

4.1 stored video at equal quality

The baselines are VideoToolbox hardware encoders at typical good streaming quality, around VMAF 93, on natural content. ñü.media’s target is the same VMAF with about 40% fewer bits than hardware HEVC; the realistic range from the literature is 30–50%. 60-second clip.

Video doesn’t fit the foreground ticks: one ñü.media HDR minute at 1080p (162,000,000 bits) takes about 110 minutes at 24,576 bit/s. It goes through your own node or the charging window.

4.2 calls at equal bits

Calls run at a fixed bitrate (stealth), so the comparison is what resolution and frame rate each codec delivers for the same bits. ñü.media sends a reduced-resolution base layer and the receiver’s Neural Engine adds super-resolution, artefact removal and frame interpolation (section 5). The table below is the design target; sections 8.1 and 8.2 have the first measurements.

Audio adds 1,920,000 bits per minute (Opus at 32,000 bit/s, constant).

5. how ñü.media works

• Images: a learned transform codec. The analysis and synthesis networks run on the Neural Engine in FP16; the entropy model (hyperprior and context model) runs in integer arithmetic on the CPU, so every device decodes the exact same symbols. Learned codecs that run their entropy model in floating point break when encoder and decoder hardware differ; integer entropy models avoid that. Progressive: the first slot carries a preview, and each later slot refines the whole picture.
• Video, three layers:
  • Base: hardware HEVC Main 10 at constant bitrate with low-latency rate control, so it runs live on every iOS 26 device. It is always decodable on its own. Full resolution by default; reduced resolution only at the lowest call tier, where section 8 shows it helps on older encoders (to be tuned per chip).
  • Enhancement, outside the loop: super-resolution, artefact removal and frame interpolation on the receiver’s Neural Engine. It can’t drift and needn’t be bit-exact. Target budget on iPhone 18 Pro: 8 ms per frame at 60 fps and 300 mW, to be measured.
  • Residual, inside the loop: a learned integer residual layer adds detail on devices that can run it in real time. Stored video only at first.
• Metadata: all EXIF, GPS, device and time data is dropped; only colour and HDR metadata is kept.

6. HDR and colour: wider than the phone

ñü.media carries more colour than the iPhone’s screen shows, so the same file looks right on later displays.

• Display: the receiver tone-maps from PQ or HLG to the screen’s current headroom with Apple’s EDR APIs, so HDR stays HDR at 1,600 nits and degrades cleanly on SDR screens.
• Export: when you save to Photos or share outside khensu.ñü, ñü.media writes HEIC with an ISO 21496-1 gain map through ImageIO (images), or HEVC Main 10 HLG through VideoToolbox (video), so other apps show it correctly.
• Dolby Vision and HDR10+: received camera clips in Dolby Vision profile 8.4 are carried as their HLG base plus our own tone-mapping metadata. We play Dolby Vision and HDR10+ through Apple’s decoders but don’t author either; both need licences from their owners.
• Cost of HDR: about 10–25% more bits than SDR at the same quality (estimate), included in the tables above.

6.1 against what phones store and what streams send

These are properties of the format, not a measured quality score. The comparison is 10-bit 4:2:0 delivery in P3 or BT.2020, which is HEIC, HEVC Main 10, AV1 Main and the streams from the large video services. Area figures are the primary triangle inside the CIE 1931 spectral locus.

The live base stays 10-bit 4:2:0. That is the picture the iPhone’s hardware encoder can emit in real time (HEVC Main 10). The residual layer is what makes a stored file 12-bit and 4:4:4 (nu-media-protocol.md 5.3). Cinema archives already go further than this file: ACES in 16-bit half float holds every visible colour. ñü.media does not claim to beat that. It does hold four times the steps and four times the colour samples of the 10-bit 4:2:0 picture a phone or a stream delivers, in a gamut about 1.7 times P3, at a peak about 6 times the iPhone’s.

7. more than the usual formats

7.1 frame rate: 120 hz, and double the fastest gaming display

• Format: frames carry a 1 MHz timestamp, not a fixed rate, so a stream can run at any rate up to 2,000 fps and change rate mid-stream. That is double the fastest gaming monitors on sale (about 500 to 1,000 Hz in 2026; the exact top figure is to be confirmed before it is quoted on the site).

• Tiers:

  Tier	Rate	Where
  Standard	30 and 60 fps	Every iOS 26 device
  ProMotion	120 fps	iPhone Pro, iPad Pro, MacBook Pro
  Double ProMotion	240 fps	Recording and playback; slow motion from the camera
  Gaming	Up to 2,000 fps	Mac and PC capture to high-refresh displays

• How: above the rate the sender’s encoder can sustain, the sender codes a lower rate (for example 60 fps) and the receiver’s Neural Engine interpolates the missing frames in the enhancement layer (section 5). Interpolated frames cost no bits, so 120 and 240 fps calls use the same bitrate as 60 fps. Stored high-rate video (camera 4K120, 240 fps slow motion, game capture) is coded at its native rate.

• No frame loss: the receiver never drops a coded frame. It shows every coded frame and only adds or skips interpolated ones to match the display. The benchmark counts frames in and out for every run.

• To measure on iPhone Air: hardware encode and Neural Engine super-resolution throughput at each base size (section 8.3), and Apple’s low-latency frame interpolation from 30 or 60 to 120 and 240 fps. A 120 fps test clip is still needed: the NASA 4K clip is 29.97 fps.

7.2 resolution: native from the best cameras, headroom to 42K

• Rule: ñü.media never scales a source down to fit the codec. Video is carried at the camera’s native resolution, frame rate and bit depth; only the bitrate changes. Lowering resolution is the viewer’s choice (a smaller screen or a slower link fetches fewer tiles), not the sender’s.

• Format limits: frame sizes up to 43,008 × 21,504 (42K equirectangular), coded as independent tiles of 512 × 512 to 2,048 × 2,048 so no single decoder has to hold a whole frame. 42K cameras don’t exist yet; the limit is there so the format won’t need to change when they do.

• Cameras to support first:

  Camera	Native video	Live today	Notes
  Insta360 Titan	11K mono, 10K 3D, 30 fps, 10-bit, 8 Micro Four Thirds sensors	4K30 stitched in camera; 8K30 with Insta360’s 8K Live licence and an external stitching server (RTSP, RTMP, RTMPS, HLS, SDI)	insta360.com
  Large-format cinema cameras	12K to 17K (for example Blackmagic URSA Cine 17K 65)	Not as live streams	Model list to be confirmed before it is quoted
  Dome and immersive rigs	Up to about 18K square (Sphere’s Big Sky)	No	Theatre and dome playback

• Viewing 360 and VR: tiles are fetched by viewport (section 7, Places), so a viewer on iPhone or Vision Pro downloads only what they’re looking at in full detail, plus a low-detail full sphere for quick head turns.

• Raw versus native: truly uncompressed 11K30 10-bit is about 25 Gbit/s (10,560 × 5,280 × 30 fps × 15 bits at 4:2:0), which no internet path carries. “Raw” in khensu.ñü means native resolution, never downscaled, at a visually lossless bitrate, with a mathematically lossless mode (section 7, Lossless) for local networks and dedicated fibre.

• Ingest without servers: the encoder runs on the camera operator’s own Mac or stitching machine, which takes the camera’s SDI, RTSP or RTMP feed and sends ñü.media into the network. Nothing is transcoded on khensu.ñü or operator servers. MOQOM Cloud uses the same format and the same ingest tool.

• Theatres: cinemas play features from DCI packages (JPEG 2000, encrypted with per-screen keys) and take live events as alternative content through a decoder on the projector’s SDI or HDMI input. ñü.media targets that alternative-content input first, at the projector’s native 2K or 4K, or up to 16K on LED walls and domes. DCI packaging is a separate licensing and certification job.

• Open questions:

  • how high-bitrate live streams travel: the mixnet’s constant-rate cover traffic makes multi-gigabit streams costly, so large live events may use the two-relay call path instead;
  • decoder memory and power for 11K-plus tiles on iPhone and Vision Pro (to be measured);
  • the exact current top camera and display figures, checked at publication.

7.3 nu-score: the test where VMAF stops

VMAF alone can’t judge nu-media. Its models stop at 3840 × 2160 and assume 8-bit SDR at ordinary frame rates, which is about the ceiling of what Netflix streams (4K HDR). Its single average hides a few bad frames, and it can’t see dropped frames, stalls or banding. bench/nuscore.py reports, for any resolution and frame rate:

First run, on a synthetic 7680 × 4320 clip with x264 at CRF 32: VMAF mean 87, but the worst 1% of frames 62 and the worst tile 78, which a single VMAF number would have hidden.

Test media. bench/ytstream.py saves exactly the stream YouTube delivers, without re-encoding, with a JSON record of the format, codec, resolution, frame rate, HDR transfer, bitrate, licence and SHA-256. The first clip is “Best of 8K Video” (4K Video ULTRA HD on YouTube), format 702: AV1 Main 10-bit, 7680 × 4320, 59.94 fps, PQ (SMPTE ST 2084) in BT.2020, about 51 Mbit/s; 20 seconds from 1:00. The title says 120 fps, but YouTube sends 60. Downloaded streams are for private dev testing; showing them publicly needs the owner’s permission.

7.4 depth for every picture, estimated on the device

Depth is what turns a flat photo into refocus, relighting, a parallax Live Photo, a spatial photo for Vision Pro and an object you can place in a room. Three sources, in order of preference:

Estimation. The estimator is a monocular depth network running in FP16 on the Neural Engine, with two inputs the published models don’t use together:

• the picture itself, for the single-image prior (shape from texture, perspective, known object sizes);
• for a Live Photo, the up to 3 s of motion around the still, from which the parallax between frames gives true relative depth for anything that moved against the background, fused with the prior where nothing moved.

The result is a metric-scaled depth plane where the camera metadata allows (focal length and sensor size, which the camera pipeline keeps and ñü.media’s EXIF stripping removes before sending; estimation therefore happens on the capturing device, and only the plane travels). Where it doesn’t, the plane is relative, and the UI says so.

Where our work is. The reference monocular estimators to measure against are Apple’s Depth Pro (2024), Depth Anything V2, MiDaS v3.1 and Marigold. None of them is designed for a phone’s neural engine, and none fuses Live Photo motion. ñü.media’s contribution, to be built and measured before any claim is made, is an estimator shaped for the Neural Engine’s tiles and FP16 path (batch-1 latency rather than throughput, no dynamic shapes, 8 ms per 12 MP picture as the target), the Live Photo fusion, and a confidence plane the renderer uses to fade effects out where depth is unsure. The algorithm Elliott referred to in the brief (“this algo”, “their algo”) was not attached; this section names the published candidates and will be revised against the intended one.

Published. Like the rest of ñü.media: the format in nu-media-protocol.md, the model weights and the estimator code in khensu-core under its licence, and the benchmark (bench/depth.py, to be written: AbsRel and δ₁ against the NYU Depth V2, KITTI and iBims-1 test sets, plus latency and energy on each chip) beside them. Patent rights and ownership stay with Elliott Wave LLC (PATENTS.md); publication is never held for a filing.

Privacy. Estimated depth is computed on the device and travels only inside the encrypted object. It is never sent to a server, and the estimator never sees a picture the person did not already choose to send or open.

7.5 the owner’s mark: invisible, anonymous, readable only by you

A person who makes or shares a picture wants to know where it went: did a friend forward it, did it end up in a public feed, was it screenshotted and reposted. Everyone else wants that same picture to say nothing about who made it. ñü.media gives both with one mechanism, the owner’s mark, built on the provenance design (nu-provenance.md) and on its rule P1: nothing in a file identifies a device or a person.

• What is embedded. A short code (96 bits), derived on the owner’s device from a secret the owner holds and a per-picture random salt, is embedded invisibly into the picture: in the learned codec’s latent for ñü.media files, and as a robust spread-spectrum watermark in the pixels for anything exported to JPEG, HEIC or HEVC, so it survives re-encoding, scaling, cropping to about a third of the frame and screenshots. The code is indistinguishable from random to anyone without the secret. It carries no name, no key, no device and no time.
• What the owner sees. The owner’s app can test any picture it encounters, in a chat, a feed, a forum or a public webË site, against the owner’s own secret, on the device. A match says: this is mine, from this capture, and it appeared here. The app collects these sightings privately into one view: “this photo was shared on by two people in groups you’re in, and appeared once in a public feed”. Sightings are only ever gathered from places the owner’s own device already visits; nothing crawls on the owner’s behalf.
• What everyone else sees. Nothing. Without the secret, the mark is noise. Two pictures from the same owner can’t be linked by the mark, because each carries a fresh salt. No server, node or operator can test a mark, since the secret never leaves the owner’s devices (it syncs between them inside the owner’s own encrypted conversation with themselves).
• Off this line. The mark covers pictures, video and audio the owner chooses to share. It is never applied to, and the sightings view never includes, the private side of a person’s life in the app: calendar, itinerary, notes, drafts, the memory layer, payments, and media that was received rather than made. Those stay in their own encrypted stores and are never marked, indexed or matched.
• Interactions. When someone reacts to, replies to, forwards or quotes the owner’s media inside khensu.ñü, that is already an interaction the owner’s app knows about through the conversation itself, with no mark needed. The mark adds only the cases the conversation can’t see: a re-upload, a screenshot, a public repost.
• Not evidence. As with perceptual fingerprints (nu-provenance.md 7), a sighting is shown to the owner as “looks like yours”, never asserted to anyone else, never used for moderation, and never connected to the illegal-content hash list.
• To build and measure. Robustness (survival rate against re-encode, scale, crop, screenshot, print-and-scan), false-positive rate at 96 bits over the device’s library, quality cost in nu-score, and the cost in bits inside the learned latent. Prior art to compare against for PATENTS.md: spread-spectrum image watermarking (Cox et al., 1997), Digimarc, Google SynthID, Meta’s Stable Signature and Video Seal, and C2PA’s soft bindings. The candidate novelty is the owner-only, unlinkable, per-picture code combined with sightings gathered solely by the owner’s own device.

7.6 receivers rebuild more than they receive

The enhancement layer (section 5) runs on the receiver and is what a viewer of a ñü.media stream actually sees:

The card on the sites. Where a ñü.media stream is shown on khensu.ñü, the player puts one small card beside it, the same kind the creators page uses, that states what the receiver did: “nu.media rebuilt this at 4K · 25 → 120 fps · BT.709 → BT.2020, 12-bit”. The figures are read from the decoder, never typed in. No before-and-after.

The hero video, honestly. The Earth video behind the home page is today an H.264 1080p25 BT.709 8-bit file, because no browser can decode ñü.media yet. It is not ñü.media and carries no card. The order of work to change that: the WebAssembly decoder for the enhancement layer (SIMD plus WebGPU; the learned transform is out of reach of WebAssembly at 4K, so the browser path is base plus enhancement, not the full codec), then encoding the Earth source as a ñü.media base at a low rate, then the card with measured numbers. Until then the home page says nothing about ñü.media over the video; the claim appears only when the decoder is running.

8. benchmark: replacing the estimates

A Swift test app, nu-media/bench, runs on iPhone 18 Pro and an iOS 26 device from the oldest supported generation and writes the tables above with measured numbers.

• Codecs:
  • ImageIO JPEG and HEIC at quality 0.3–0.95, with ImageIO AVIF and JPEG XL as decode-only references;
  • VideoToolbox hardware H.264 and HEVC Main and Main 10, both variable and constant bitrate, with low-latency rate control for the call tiers;
  • ñü.media images and video.
• Content:
  • images: Kodak (24 images), the CLIC professional validation set, and our own iPhone captures in SDR and HDR;
  • video: the UVG sequences at 1080p and 4K, HDR test sequences, and recorded video-call content.
• Quality metrics:
  • all content: PSNR on Y′CbCr (6:1:1 weighting), MS-SSIM and VMAF;
  • HDR: PU21-encoded PSNR and VMAF, and colour error as ΔE_ITP (ITU-R BT.2124).
• Results: BD-rate against each baseline, bits per image and per minute, encode and decode time per image and per frame, and energy per image and per minute (MetricKit and power logs).
• Pass criteria for shipping:
  • images use at least 35% fewer bits than ImageIO HEIC at equal MS-SSIM;
  • calls score higher VMAF at the displayed resolution than hardware HEVC at the same tier;
  • decode is bit-exact across all supported devices;
  • no frame drops at 60 fps on iPhone 18 Pro.

Measured results go here, with the device, OS build and commit of each run.

8.1 desktop baseline run

bench/codecs.py with ffmpeg, data in bench/results.json. Source: Tears of Steel 1080p (Blender Foundation, CC BY 3.0), 6 s from 28 s, 24 fps, 8-bit 4:2:0. Quality is VMAF (libvmaf, default model) against the 1080p source. H.264 and HEVC use VideoToolbox with software fallback disabled, so these are Apple’s hardware encoders; VideoToolbox undershoots its target by about 30%, so the table uses actual bitrates. ñü.media rows are the base layer only, upscaled with Lanczos: a floor for the enhancement layer to build on.

Bracketed figures are the actual kbit/s where an encoder didn’t land near the column. Equal-bitrate check at 500 kbit/s (each encoder steered to within 3%): H.264 42.6, HEVC 50.9, ñü.media HEVC base 59.2, ñü.media AV1 base 72.0, real-time AV1 at 1080p 72.9.

What this changes and what it doesn’t:

• Superseded by 8.2 for the base layer. The M1 Pro’s hardware encoder is several generations old; on an iPhone Air, hardware HEVC is much stronger, so the base layer stays hardware HEVC.
• Offline AV1 is the bar for the enhancement layer: ñü.media’s target is that quality (about 400 kbit/s for VMAF 80 on this clip) in real time on iPhone. Unmeasured until the enhancement network exists.
• AV2 and learned codecs are not measured here. Published figures: AV2 about 30% fewer bits than AV1 (AOMedia); DCVC-RT 21% fewer than H.266 VTM and real time at 1080p only on an NVIDIA A100 (Jia et al., CVPR 2025). Neither has a phone encoder. The site plots them as estimates scaled from the offline AV1 curve (0.70 and 0.65).
• One clip, one machine, 8-bit SDR. The section 8 pass criteria still need the iPhone app, the full content set, HDR and energy.

8.2 iPhone Air run

bench/iphone.py drives the bench/ios app (NuBench) over devicectl: the Mac copies ProRes 422 clips to the phone, the phone encodes them with VideoToolbox hardware only (real time, no frame reordering, average bitrate capped per second), and the Mac copies the files back and scores them. AV1 is SVT-AV1 at real-time call settings on the Mac (the iPhone has no AV1 encoder). ñü.media rows are the half-size base layer with a Lanczos upscale, no Neural Engine. Data in bench/iphone-reel.json and bench/iphone-aurora.json. Encode fps is the phone’s whole pipeline, including ProRes decode.

720p: a 720 × 1280 portrait reel (supplied by the user; already H.264 at 2.5 Mbit/s, so its compression is baked in), 6 s from 8 s, 30 fps. VMAF default model.

4K: NASA SVS 31375, aurora from the ISS during the November 2025 geomagnetic storm, 3840 × 2160 at 180 Mbit/s, 6 s from 24 s, 30 fps. VMAF 4K model.

What this shows:

• Apple’s current HEVC encoder is the baseline to beat. It matches real-time AV1 software at 720p and beats it by 5 VMAF points at 4K, in hardware. The base layer stays HEVC.
• A half-size base layer alone loses on this hardware (69 vs 73 at 720p, 87 vs 93 at 4K): upscaling can’t restore detail the encoder never sent. Everything ñü.media claims beyond HEVC has to come from the Neural Engine layers, which don’t exist yet. The target stays 40% fewer bits than hardware HEVC at equal VMAF.
• 4K HEVC encodes at about 48 fps on iPhone Air including ProRes decode, so 4K30 is live; the half-size path runs at about 97 fps, leaving headroom for enhancement.
• Limits: two clips, one phone, 8-bit SDR, no energy measurement yet. Publishing the reel on khensu.ñü needs the creator’s permission.

Every codec at 500 kbit/s, 720p, with a first Neural Engine enhancement (bench/grid720.py, data in bench/iphone-720p.json). The source is the Instagram file itself: H.264 High profile, 720 × 1280 at 30 fps, 2.44 Mbit/s video (2.51 Mbit/s with audio). Every encoder is steered to 500 kbit/s ±3%.

• Apple’s low-latency super resolution runs at about 440–500 fps at 360 × 640 → 720 × 1280 on iPhone Air, so a learned enhancement layer has ample real-time budget. It looks sharper but scores below a plain upscale: it is a general-purpose scaler that adds texture the original doesn’t have, which VMAF penalises. ñü.media’s enhancement has to be trained on our own base layer with a fidelity loss, not borrowed.
• Offline encoders show the headroom: x265 slow and AV1 with lookahead reach VMAF 81–86 at the same bits, 9–14 points above the live hardware encoder. That gap is what a learned layer on the receiving phone can try to close without changing what’s sent.
• Bug fixed on the way: the scaler rejects frames that lack the padded layout in sourcePixelBufferAttributes, which AVAssetReader ignores; frames are copied into a pool with those attributes first.