Introduce the ability to extend camera functionality by bijington · Pull Request #3274 · CommunityToolkit/Maui

bijington · 2026-07-26T18:24:10Z

Current status

This it the remaining TODO list

test out multiple scenarios together
Tested on macOS for both - Shared page crashes
Needs to be built and tested on Android
Needs to be built and tested on Windows

Introduce the ability to extend camera functionality for concepts such as Barcode scanning.

Replaces the original PR attempt at #2841.

Description of Change

Camera Scenarios Usage Guide

The Community Toolkit for MAUI CameraView allows extending its functionality through "Scenarios". There are two primary ways to implement a scenario: PlatformCameraScenario and FrameBasedCameraScenario.

PlatformCameraScenario

PlatformCameraScenario is designed for scenarios where you want to leverage platform-specific high-performance APIs (e.g., Google ML Kit on Android, Apple Vision framework on iOS, or Windows Media Capture APIs).

When to use it

When performance is critical.
When you want to use a feature already provided by the operating system.
When you need fine-grained control over the platform-specific camera pipeline.

Implementation Example (Windows)

In the sample project, PlatformBarcodeScanningScenario on Windows uses the MediaFrameReader to capture frames and decode them using ZXing.Net within the platform-specific code.

// Platforms/Windows/PlatformBarcodeScanningScenario.cs
public class PlatformBarcodeScanningScenario : PlatformCameraScenario
{
    // ... Windows specific implementation using MediaFrameReader
}

FrameBasedCameraScenario

FrameBasedCameraScenario is a specialized version of PlatformCameraScenario that abstracts the frame capture process. It provides a platform-agnostic OnFrameReceived method that gives you access to a CameraFrame containing raw byte data.

When to use it

When you want to share the same processing logic across all platforms.
When you have a cross-platform library (like ZXing.Net) that can process raw image data.
When you don't need the absolute maximum performance of platform-native ML frameworks.

Implementation Example (Shared)

In the sample project, SharedBarcodeScanningScenario demonstrates how to implement barcode scanning in shared code.

// SharedBarcodeScanningScenario.cs
public class SharedBarcodeScanningScenario : FrameBasedCameraScenario
{
    public override void OnFrameReceived(CameraFrame frame)
    {
        // frame.Data contains the raw byte array
        // frame.Width and frame.Height provide dimensions
        
        var luminanceSource = new RGBLuminanceSource(frame.Data, frame.Width, frame.Height);
        var result = barcodeReader.Decode(luminanceSource);

        if (result is not null)
        {
            // Handle result
        }
    }
}

Key Differences

Feature	PlatformCameraScenario	FrameBasedCameraScenario
Logic Location	Platform-specific folders	Shared code
Performance	Highest (Native APIs)	Good (Shared processing)
Complexity	Higher (Needs platform knowledge)	Lower (Abstraction handled for you)
Data Access	Native objects (e.g., `ImageProxy`, `CMSampleBuffer`)	`CameraFrame` (Raw bytes)

Usage in XAML

Both types of scenarios are used in the same way with the CameraView.

<toolkit:CameraView>
    <toolkit:CameraView.Scenarios>
        <!-- Use either a platform-specific scenario -->
        <sample:PlatformBarcodeScanningScenario Command="{Binding OnCodeDetectedCommand}" />
        
        <!-- Or a shared frame-based scenario -->
        <sample:SharedBarcodeScanningScenario Command="{Binding OnCodeDetectedCommand}" />
    </toolkit:CameraView.Scenarios>
</toolkit:CameraView>

Linked Issues

Fixes #

PR Checklist

Has a linked Issue, and the Issue has been approved(bug) or Championed (feature/proposal)
Has tests (if omitted, state reason in description)
Has samples (if omitted, state reason in description)
Rebased on top of main at time of PR
Changes adhere to coding standard
Documentation created or updated: https://github.com/MicrosoftDocs/CommunityToolkit/pulls

Additional information

Example of it working on macOS

scanning.mov

…h as Barcode scanning

CliffAgius · 2026-07-27T12:46:42Z

Are we still supporting Tizen? it's not possible to build a MAUI app on tizen as it's stuck on Net6...

Good point! I needed to patch this for the builds to pass but it won't do anything

Copilot

Pull request overview

This PR adds an extensibility mechanism to CameraView via attachable “scenarios” (e.g., barcode scanning), wires scenario handling into platform camera pipelines (Android/iOS), and provides a sample barcode-scanning page demonstrating the new capability.

Changes:

Introduces ICameraView.Scenarios / CameraView.Scenarios as a collection of CameraScenario instances.
Adds CameraManager scenario management (add/remove/reset) plus platform hooks to integrate platform-specific outputs/use-cases.
Adds a new sample page and platform implementations for barcode scanning (including an Android MLKit package reference).

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
src/CommunityToolkit.Maui.Camera/Views/CameraView.shared.cs	Adds `Scenarios` bindable collection to `CameraView`.
src/CommunityToolkit.Maui.Camera/Interfaces/ICameraView.shared.cs	Exposes `Scenarios` on the camera view contract.
src/CommunityToolkit.Maui.Camera/Handlers/CameraViewHandler.shared.cs	Maps `Scenarios` and reacts to collection changes to update camera pipeline.
src/CommunityToolkit.Maui.Camera/Primitives/CameraScenario.shared.cs	Introduces base scenario lifecycle API (`Initialize`, attach/detach hooks).
src/CommunityToolkit.Maui.Camera/Primitives/PlatformCameraScenario.shared.cs	Adds a platform-specialized scenario base type.
src/CommunityToolkit.Maui.Camera/Primitives/PlatformCameraScenario.android.cs	Declares Android-specific platform scenario surface (`UseCase`).
src/CommunityToolkit.Maui.Camera/Primitives/PlatformCameraScenario.macios.cs	Declares Apple-specific platform scenario surface (`AVCaptureOutput`).
src/CommunityToolkit.Maui.Camera/CameraManager.shared.cs	Adds scenario list + add/remove/reset and preview refresh behavior.
src/CommunityToolkit.Maui.Camera/CameraManager.android.cs	Integrates additional `UseCase`s into rebinding flow.
src/CommunityToolkit.Maui.Camera/CameraManager.macios.cs	Integrates scenario outputs into `AVCaptureSession` startup.
src/CommunityToolkit.Maui.Camera/CameraManager.net.cs	Adds scenario APIs for unsupported platforms (throws NotSupported).
src/CommunityToolkit.Maui.Camera/CameraManager.tizen.cs	Adjusts partial method signatures and adds scenario APIs (throws NotSupported).
samples/CommunityToolkit.Maui.Sample/ViewModels/Views/ViewsGalleryViewModel.cs	Adds a sample gallery entry for barcode scanning.
samples/CommunityToolkit.Maui.Sample/ViewModels/Views/CameraView/BarcodeScanningViewModel.cs	New view model to display detected barcode text and refresh cameras.
samples/CommunityToolkit.Maui.Sample/Pages/Views/CameraView/BarcodeScanningPage.xaml	New page wiring `CameraView.Scenarios` to a barcode-scanning scenario.
samples/CommunityToolkit.Maui.Sample/Pages/Views/CameraView/BarcodeScanningPage.xaml.cs	New page code-behind to refresh camera list on appearing.
samples/CommunityToolkit.Maui.Sample/PlatformBarcodeScanningScenario.cs	Shared scenario surface (bindable `Command`) for barcode detection callbacks.
samples/CommunityToolkit.Maui.Sample/Platforms/PlatformBarcodeScanningScenario.cs	Adds a partial scenario file in Platforms folder.
samples/CommunityToolkit.Maui.Sample/Platforms/Android/PlatformBarcodeScanningScenario.cs	Android-specific scenario implementation using CameraX `ImageAnalysis`.
samples/CommunityToolkit.Maui.Sample/Platforms/Android/BarcodeAnalyzer.cs	Android barcode analyzer implementation using MLKit.
samples/CommunityToolkit.Maui.Sample/Platforms/iOS/PlatformBarcodeScanningScenario.cs	iOS-specific scenario using `AVCaptureMetadataOutput`.
samples/CommunityToolkit.Maui.Sample/Platforms/MacCatalyst/PlatformBarcodeScanningScenario.cs	Mac Catalyst-specific scenario using `AVCaptureMetadataOutput`.
samples/CommunityToolkit.Maui.Sample/MauiProgram.cs	Registers the new barcode scanning page + view model.
samples/CommunityToolkit.Maui.Sample/AppShell.xaml.cs	Adds shell mapping for barcode scanning page.
samples/CommunityToolkit.Maui.Sample/CommunityToolkit.Maui.Sample.csproj	Adds Android-only MLKit barcode scanning package reference.

…n't important

peruchali · 2026-08-03T08:43:08Z

+	/// <param name="data">The raw frame data.</param>
+	/// <param name="width">The width of the frame.</param>
+	/// <param name="height">The height of the frame.</param>
+	public CameraFrame(byte[] data, int width, int height)


@bijington Is there any format that these bytes will conform to? Is it rgba etc? or will it be platform specific?

I was wondering about this over the weekend. I think this will need to be more intelligent, I'll have a play this week

One option can be to add an enum for the Format for the common native formats (YUV*, etc), set it appropriately for different platforms and leave it up to the handler to process as they wish, reduces any perf overhead.

Yes I like this idea, or possibly provide a default converter and let the developer override, then we don't have to worry about support multiple formats?

peruchali · 2026-08-04T08:19:43Z

+				return;
+			}
+
+			var buffer = planes[0].Buffer;


@bijington I think this code will get only the Y plane (luminance) data, which will effectively just be grayscale. I think we should be getting 1 (U) and 2 (V) as well?

peruchali · 2026-08-04T08:58:44Z

+				return;
+			}
+
+			var data = new byte[buffer.Remaining()];


@bijington Not sure if this is out of scope of this scenario, but allocating these byte arrays can end up in the LOH and cause GC slowdown with repeated allocations and releases (if it's anything like decoding video files on disk 😔).
Is it possible to augment FrameBasedCameraScenario to allow a consumer to optionally handle memory management?
For eg. FrameBasedCameraScenario can have a
protected virtual byte[] Allocate(int size) => new byte[size];
which can be plumbed through in FrameAnalyzer.Analyze rather than directly allocating?
That way, a consumer can override FrameBasedCameraScenario to use ArrayPool/provide some preallocated/reusable buffer, etc or any other kind of memory management for these bytes (by coordinating with the use of CameraFrame) to avoid GC slowdown.
Didn't want to re-implement the entire FrameBasedCamera scenario on all platforms just for memory management from PlatformCameraScenario when all the other pieces exist...

For eg. FrameBasedCameraScenario can have a
protected virtual byte[] Allocate(int size) => new byte[size];
which can be plumbed through in FrameAnalyzer.Analyze rather than directly allocating?

I like the idea of giving extension points to our users. But it's better to have a good solution inside the library; most devs will shoot themselves in the foot trying to manage that. So we do provide a better solution, than new byte [size] and allow devs to implement their own if needed.

@pictos What better solution do you suggest? Anything more than new byte[] will place constraints on the usage, lifetime of the byte array - which seems strange out of the box.

…orting / conversion

bijington · 2026-08-04T19:48:35Z

@peruchali it'll be great to hear your thoughts. I have made a couple of changes here based on your feedback:

provide better memory management in the base scenario and allow developers to override the defaults
provide an enum for common camera frame formats and make it possible for developers to provide custom conversions
allow developers to set a preferred format to avoid having to convert post frame capture

I should add most of this is currently untested now but I am keen to iterate over the API and then we can get the underlying implementation working

pictos · 2026-08-05T01:25:50Z

+	/// </summary>
+	/// <param name="size">The size of the byte array to allocate.</param>
+	/// <returns>A new byte array of the specified size.</returns>
+	protected virtual byte[] Allocate(int size) => ArrayPool<byte>.Shared.Rent(size);


We can't do it, users may not know this is an ArrayPool array, and may not return it into the pool. So it can be worse than doing new byte[size];

Also, arrays from ArrayPool have their length as a power of 2 base. So if you ask for 1000, it will give back an array with a length of 1024. And it can be with garbage (when you return or rent an array into the pool it isn't zeroed by default), so users can't foreach over it.

Yes, and also the value of the required length of the byte[] is lost here and downstream (bytes.Length can be > than the required size). Having it out of the box will require adding a new member to CameraFrame containing the length, which makes it more complicated.

pictos · 2026-08-05T01:27:40Z

+	/// Frees the specified byte array.
+	/// </summary>
+	/// <param name="data">The byte array to free.</param>
+	protected virtual void Free(byte[] data) => ArrayPool<byte>.Shared.Return(data);


this is unsafe, if the returned array isn't an array from ArrayPool, this will throw an exception. We can't control if the user will reasign this array or something like. For example:

var data = myFrame.Allocate(1024); data = data.Where( x => x != 0).ToArray(); // this will allocate a new array myFrame.Free(data); //this will throw, because the array doesn't belong into the pool

pictos · 2026-08-05T01:33:18Z

+{
+	ImageAnalysis? imageAnalysis;
+
+	public override Task OnAttached(IExecutorService? cameraExecutor, ResolutionSelector? resolutionSelector)


I believe this should be ValueTask instead of Task

pictos

I've couple of more comments, but I'm not sure about this approach. Maybe I didn't fully understand it yet, but I feel the better would be to expose the CameraManager and let users customize the capabilities and handling there, like we have for MediaManager, this current approach adds so many levels of indirection that can be dangerous.

pictos · 2026-08-05T01:41:36Z

+/// <summary>
+/// Base class for output based processing for camera related activities.
+/// </summary>
+public abstract class CameraScenario : BindableObject, IDisposable


Before moving forward with this, would be good to see if it's possible to work with more than one scenario at once. How the Camera control would behave, and so on

I agree, I've got this on my list to try out

pictos · 2026-08-05T01:47:49Z

+
+		var preferredFormatName = PreferredFormat switch
+		{
+			CameraFrameFormat.Nv12 => "NV12",


would be better to create a Constant file and put those strings there and reuse it?

pictos · 2026-08-05T01:51:00Z

 	static readonly BindablePropertyKey isAvailablePropertyKey =
 		BindableProperty.CreateReadOnly(nameof(IsAvailable), typeof(bool), typeof(CameraView), CameraViewDefaults.IsAvailable);
+
+	internal static readonly BindablePropertyKey ScenariosPropertyKey = BindableProperty.CreateReadOnly(nameof(Scenarios), typeof(IList<CameraScenario>), typeof(CameraView), default(IList<CameraScenario>),


if users can't change the collection type, why not use the concrete type?

bijington · 2026-08-05T04:04:05Z

I've couple of more comments, but I'm not sure about this approach. Maybe I didn't fully understand it yet, but I feel the better would be to expose the CameraManager and let users customize the capabilities and handling there, like we have for MediaManager, this current approach adds so many levels of indirection that can be dangerous.

Thanks @pictos. I'm not stuck on this approach, just spiking it to see if it makes sense, so your feedback here is valuable! I'm also happy to go down the exposing of CameraManager route, as you say I've been exposing more levels here which is introducing much more to maintain now.

My current goal is to find a way to solve the following:

Provide platform specific frame processing
Provide shared frame processing

I think we are closing to having those abilities but then we can evaluate and simplify how we actually expose it

peruchali · 2026-08-05T04:21:33Z

@bijington for:

provide better memory management in the base scenario and allow developers to override the defaults

I'm not sure that the library out of the box can implement any efficient memory management (like ArrayPool) without at the same time introducing constraints on the usage and lifetime of the bytes of CameraFrame (for eg the user should be very careful with passing the bytes around rather than copying). The default should be Allocate(int size) => new byte[size] (we shouldn't need a Free method, that is up to the consumer). This way, out of the box, the user can use as they feel like without worrying about any dangling or corrupted memory. I feel like an example of good memory management using ArrayPool (or even just a single reused byte[] for simplicity) can belong in the samples/guidance by overriding the Scenario and usage of CameraFrame.

To "soften" the blow of the "default" out-of-the-box experience just being the humble new byte[...], I think we should:

Add a property public int SamplingRate to FrameBasedCameraScenario (default - maybe 3?)
Add another override to the FrameBasedCameraScenario:
```
private int _samplingCounter = 0;
protected virtual bool ShouldProcessFrame(...) => (++_samplingCounter % SamplingRate) == 0;
```
and call this from FrameAnalyzer.Analyze. If ShouldProcessFrame is false, just return right away without allocating or returning a CameraFrame, reducing allocations to 1/3rd.
For practical purposes like a bar code scanner, 30 fps is overkill and sampling will ease some of the pressure.
A consumer can override this to provide more custom or variable sampling.

Add a property to FrameBasedCameraScenario to specify the public Size Resolution. It looks like this can be plumbed through in Android:

var resolutionStrategy = new ResolutionStrategy(
     targetSize, 
     ResolutionStrategy.RuleClosestLowerThenHigher
 );

 var customResolutionSelector = new ResolutionSelector.Builder()
     .SetResolutionStrategy(resolutionStrategy)
     .Build();

 imageAnalysis = new ImageAnalysis.Builder()
     .SetResolutionSelector(customResolutionSelector) ....

Setting a cheap default will ensure that the allocations don't end up in the LOH, and so should work decently out of the box.

Other requests:

CameraFrame can have a Timestamp field (image.ImageInfo.Timestamp in Android). This will be useful for any kind of real-time tracking algorithm, etc.

bijington · 2026-08-05T04:24:37Z

@peruchali thanks. Yes I don't think I had fully considered my idea last night 😂. It might be safest to just revert to something very simple and let the user decide

Introduce the ability to extend camera functionality for concepts suc…

559fb00

…h as Barcode scanning

bijington mentioned this pull request Jul 26, 2026

Initial attempt at platform based processing from the CameraView #2841

Closed

6 tasks

CliffAgius reviewed Jul 27, 2026

View reviewed changes

TheCodeTraveler requested a review from Copilot July 28, 2026 17:42

Copilot started reviewing on behalf of TheCodeTraveler July 28, 2026 17:42 View session

Copilot AI reviewed Jul 28, 2026

View reviewed changes

bijington added 4 commits July 29, 2026 20:10

Fix some access modifiers and add missing xml comments

04fa058

More xml docs and sample in Windows using ZXIng for now

3d8a64a

Provide support for a shared code layer for times when performance is…

cddd00d

…n't important

Split out the sample pages

bea6b24

peruchali reviewed Aug 3, 2026

View reviewed changes

peruchali reviewed Aug 4, 2026

View reviewed changes

bijington added 3 commits August 4, 2026 20:15

Provide cleaner memory allocation support and camera frame format rep…

d5ebf8c

…orting / conversion

Attempt to set the preferred format for capture

0a04e50

Add example of how to convert frames

0fdde5d

bijington and others added 3 commits August 4, 2026 20:59

Fix up some bits

c73e3c9

Take specific platform scenarions

5d665b1

Merge branch 'main' into feature/sl-camera-processing

e9de718

pictos reviewed Aug 5, 2026

View reviewed changes

Uh oh!

Conversation

bijington commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Current status

Description of Change

Camera Scenarios Usage Guide

PlatformCameraScenario

When to use it

Implementation Example (Windows)

FrameBasedCameraScenario

When to use it

Implementation Example (Shared)

Key Differences

Usage in XAML

Linked Issues

PR Checklist

Additional information

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

peruchali Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peruchali Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bijington commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peruchali Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pictos Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pictos left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bijington commented Jul 26, 2026 •

edited

Loading

peruchali Aug 3, 2026 •

edited

Loading

peruchali Aug 4, 2026 •

edited

Loading

bijington commented Aug 4, 2026 •

edited

Loading

peruchali Aug 5, 2026 •

edited

Loading

pictos Aug 5, 2026 •

edited

Loading

peruchali commented Aug 5, 2026 •

edited

Loading