E2EE: H.264 from Intel Quick Sync (Windows Chrome) never decrypts: AES-GCM additionalData includes 3-byte start codes that WebRTC rewrites to 4-byte #2140

Summary

With E2EE enabled, H.264 video published by Chrome on Windows with the Intel Quick Sync hardware encoder is never decrypted by any receiver. Every frame fails with InvalidKey: Decryption failed: OperationError, the receiver decodes 0 frames, and it requests keyframes continuously. The same machines with the NVIDIA hardware encoder, or with OpenH264, work.

The cause: the Intel encoder writes 3-byte start codes (00 00 01) before some NAL units. WebRTC's H.264 depacketizer rebuilds every received frame with 4-byte start codes (00 00 00 01). The E2EE frame cryptor leaves the bytes up to the first slice NAL unit unencrypted, and passes them to AES-GCM as additionalData. The sender authenticates the header with 3-byte start codes, and the receiver verifies it with 4-byte start codes, so the GCM tag check fails on every frame.

Environment

Evidence

An in-browser loopback (below) reads each frame with an encoded transform before encryption would happen, and again on the receiving side after depacketization.

Intel Quick Sync (fails):

The first bytes of the keyframe differ inside the region the cryptor leaves unencrypted:

• sent: … 68 ee 3c b0 00 00 01 6e c0 80 …
• received: … 68 ee 3c b0 00 00 00 01 6e c0 80 …

NVIDIA H.264 Encoder MFT (works), same machine and test:

Other observations:

• No NALU processing failed, falling back to VP8 handling warning appears on the Intel sender. NAL parsing succeeds on both sides, so the parse fallback is not involved.
• Disabling hardware encoding on the same Intel laptop (chrome://flags, Hardware-accelerated video encode set to Disabled) switches Chrome to OpenH264, and the receiver then decrypts normally.
• macOS VideoToolbox also writes 4-byte start codes and works.

Reproduction

Paste this into the DevTools console of any https:// page in Chrome on a Windows machine with Intel graphics. Allow the camera when Chrome asks. It prints the NAL unit layout of two frames as encoded and as received:

(async () => {
  const hex = (b) => [...b].map((x) => x.toString(16).padStart(2, '0')).join(' ');
  const nals = (b) => {
    const out = [];
    for (let i = 0; i + 3 < b.length; i++) {
      if (b[i] === 0 && b[i + 1] === 0 && b[i + 2] === 1) {
        out.push((i > 0 && b[i - 1] === 0 ? '4byte:' : '3byte:') + (b[i + 3] & 31));
        i += 3;
      }
    }
    return out.slice(0, 12).join(' ');
  };
  const sample = (list, f) => {
    if (list.length < 2) {
      const d = new Uint8Array(f.data);
      list.push({ type: f.type, bytes: d.length, nalUnits: nals(d), first48: hex(d.slice(0, 48)) });
    }
  };
  const gathered = (pc) => new Promise((r) => {
    if (pc.iceGatheringState === 'complete') return r();
    pc.addEventListener('icegatheringstatechange', () => pc.iceGatheringState === 'complete' && r());
  });
  const track = (await navigator.mediaDevices.getUserMedia({ video: { width: 1280, height: 720 } })).getVideoTracks()[0];
  const a = new RTCPeerConnection({ encodedInsertableStreams: true });
  const b = new RTCPeerConnection({ encodedInsertableStreams: true });
  const tr = a.addTransceiver(track, { direction: 'sendonly' });
  const h264 = RTCRtpSender.getCapabilities('video').codecs.filter(
    (c) => c.mimeType === 'video/H264' && /packetization-mode=1/.test(c.sdpFmtpLine || ''));
  const high = h264.filter((c) => /profile-level-id=64/.test(c.sdpFmtpLine || ''));
  tr.setCodecPreferences(high.length ? high : h264);
  const log = { sent: [], received: [] };
  const s = tr.sender.createEncodedStreams();
  s.readable.pipeThrough(new TransformStream({ transform(f, c) { sample(log.sent, f); c.enqueue(f); } })).pipeTo(s.writable);
  b.ontrack = (ev) => {
    const r = ev.receiver.createEncodedStreams();
    r.readable.pipeThrough(new TransformStream({ transform(f, c) { sample(log.received, f); c.enqueue(f); } })).pipeTo(r.writable);
  };
  await a.setLocalDescription(await a.createOffer()); await gathered(a);
  await b.setRemoteDescription(a.localDescription);
  await b.setLocalDescription(await b.createAnswer()); await gathered(b);
  await a.setRemoteDescription(b.localDescription);
  await new Promise((r) => setTimeout(r, 5000));
  (await a.getStats()).forEach((x) => {
    if (x.type === 'outbound-rtp' && x.kind === 'video') log.encoder = x.encoderImplementation;
  });
  a.close(); b.close(); track.stop();
  console.log(JSON.stringify(log, null, 1));
})();

In a livekit room, the same machine publishing H.264 with E2EE produces InvalidKey: Decryption failed: OperationError on every receiver, for every frame.

How the high profile was selected matters. On Windows, chrome://gpu lists hardware H.264 encode for baseline, main, and high. With livekit's default negotiation, Chrome usually lands on constrained baseline (42e01f), which it encodes with OpenH264, so the bug stays hidden. The bug appears whenever the Intel hardware encoder produces the frames, for example when the application prefers the high profile with setCodecPreferences before the offer.

Expected

Frames from any spec-compliant H.264 encoder decrypt at the receiver. 3-byte and 4-byte start codes are both valid Annex B.

Possible fixes

1. On the sender, rewrite 3-byte start codes to 4-byte inside the unencrypted header before encrypting, so the header matches what the receiver reconstructs.
2. On both sides, compute additionalData from a normalized form of the header, for example NAL unit payloads without start codes. That is independent of how the depacketizer writes start codes.

Option 2 also covers depacketizer changes in the future. Both options leave the ciphertext unchanged.

Related

• fix(e2ee): continue NALU parsing when leading data is detected #1652 handles leading data before the first start code. That is a different case, and this one does not throw.
• Seed the E2EE sender cryptor with the publish codec #2131 seeds the sender cryptor's codec. It does not affect this case, because the codec is detected correctly here.