[gelasdev]

Motivation

Apps that check AVCaptureSession.connections before a photo capture, or that convert between preview-layer and capture-device coordinates, do not work with idb camera inject yet. The Onfido iOS SDK does both in its document capture step:

• AVCaptureSession.connections returned [] for a session holding the virtual input, so Onfido refused to capture ("Couldn't establish active camera connection on photo capture"). The README lists a session's connections as not covered.
• metadataOutputRectConverted(fromLayerRect:) returned the layer rect un-normalised. Onfido computes its crop rect with it, so the crop fell outside the photo ("Something went wrong whilst cropping the captured image").

The same conversions are what apps use to set rectOfInterest, to crop video-data-output frames to an on-screen guide, and to turn a tap into a focus or exposure point.

Fixes #989.

This change:

• swizzles -[AVCaptureSession connections]: for a session with virtual state it returns the native connections plus the connections of the virtual outputs that have input ports, like inputs and outputs already do;
• swizzles metadataOutputRectOfInterestForRect:, rectForMetadataOutputRectOfInterest:, captureDevicePointOfInterestForPoint: and pointForCaptureDevicePointOfInterest: on AVCaptureVideoPreviewLayer: for a layer bound to a virtual session they use one transform from the normalised native frame to the layer (the frame placed by videoGravity, turned by the preview connection's rotation, mirrored when the connection is mirrored) and its inverse; other layers use the original implementation;
• moves frameSize(_:) out of VirtualCamera so the preview layer uses the same frame shape as its contents;
• adds a roi mode to CameraFixture;
• updates the shim README (covered paths; "Not yet covered" now lists transformedMetadataObjectForMetadataObject: and a preview layer's connection in connections).

Test Plan

iPhone 17 Pro simulator (iOS 26.5), Xcode 26.6, macOS 26.6.2.

1. ./build.sh build Shimulator-iOS.
2. xcrun -sdk iphonesimulator swiftc -parse-as-library -target arm64-apple-ios17.0-simulator CameraFixture/CameraFixture.swift -o CameraFixture.
3. SIMCTL_CHILD_IDB_VIDEO_INJECTION=1 SIMCTL_CHILD_DYLD_INSERT_LIBRARIES=<libShimulator-iOS.dylib> xcrun simctl spawn <UDID> ./CameraFixture roi, then again with SIMCTL_CHILD_CAMFIXTURE_FRONT=1.

The layer is 300x600 pt, the rect is (30, 150, 120, 200) and the point is (75, 450). With resizeAspect at 90 degrees the 1080x1920 frame fills 300x533.3 pt from y = 33.3, so the expected rect is (0.2188, 0.5, 0.375, 0.4). Back camera (excerpt):

CAMFIXTURE roi connections no-output=0
CAMFIXTURE roi connections photo-output=1
CAMFIXTURE roi mirrored=0
CAMFIXTURE roi gravity=AVLayerVideoGravityResizeAspect angle=90 rect=0.2188,0.5000,0.3750,0.4000 point=0.7813,0.7500
CAMFIXTURE roi gravity=AVLayerVideoGravityResizeAspectFill angle=90 rect=0.2500,0.5000,0.3333,0.3556 point=0.7500,0.7222
CAMFIXTURE roi gravity=AVLayerVideoGravityResize angle=180 rect=0.5000,0.4167,0.4000,0.3333 point=0.7500,0.2500
CAMFIXTURE roi connections no-input=0
CAMFIXTURE roi round-trip drift=0.000000

Front camera (preview mirrored), same input:

CAMFIXTURE roi mirrored=1
CAMFIXTURE roi gravity=AVLayerVideoGravityResizeAspect angle=90 rect=0.2188,0.1000,0.3750,0.4000 point=0.7813,0.2500
CAMFIXTURE roi round-trip drift=0.000000

Both runs print all 12 gravity and angle combinations and exit 0. The round trip goes through layerRectConverted(fromMetadataOutputRect:) and layerPointConverted(fromCaptureDevicePoint:).

Onfido iOS SDK 32.6.3, with idb camera inject document.mp4 --loop and the app launched through the companion: before, the shutter fails at once; after the connections change alone, cropping fails. LLDB values before the change:

-[AVCaptureSession connections]                    ()   (from Onfido.CameraLayerView.checkPhotoCaptureConnectionIsActive)
getRelativeImageAreaToCrop (Onfido, expects 0...1)  {244.72, 12.56, 237.66, 376.88}

With both changes Onfido passes its connection check, crops the photo to the document frame, accepts the front and the back, and moves to its selfie step.

Related PRs

None.

[meta-cla]

Hi @gelasdev!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[meta-cla]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!