Skip to content

feat(ios): 支持画中画播放 - #2646

Open
sibogentleman wants to merge 9 commits into
Predidit:mainfrom
sibogentleman:codex/ios-picture-in-picture
Open

sibogentleman wants to merge 9 commits into
Predidit:mainfrom
sibogentleman:codex/ios-picture-in-picture

Conversation

@sibogentleman

@sibogentleman sibogentleman commented Oct 9, 2026 •

Copy link
Copy Markdown

描述

为 iOS 15 以上的系统添加了小窗播放的功能,在 iOS 27 的 iPhone 16 Pro 上真机测试通过。

关联的 Issues

[feat]: iOS的小窗模式 #2588

PR 类型

  • Bug 修复
  • 添加新功能
  • 重构实现
  • 文档或格式
  • 其它

AI 辅助开发

  • 没有使用 AI 辅助
  • AI 辅助开发了部分功能,但是我已检查并理解 AI 所写的代码
  • 基本上都由 AI 实现,但是我非常了解 AI 的代码与写完的产物

AI 辅助模型: OpenAI GPT-6(Astra)

本次实现及代码整理使用了 AI,由 Codex 复核代码、运行自动检查,提交者已进行真机手动测试。

提交前检查

  • 我已经填写了 PR 模板中的所有内容
  • 我已经阅读了 贡献指引 ,并遵守指引进行开发
  • 这个 PR 的功能已经经过我的测试,并且我完全理解我(或 AI)做出的改动

附注

验证环境:Flutter 3.47.6 / Dart 3.13.5。

  • flutter test --no-pub:261 项通过,覆盖 PiP 通道、状态同步、视频轨道恢复、并发切换及页面销毁 / 会话交接。
  • flutter analyze --no-pub --no-fatal-infos --fatal-warnings:通过,无错误或警告;17 条 info 均位于未修改文件。
  • flutter build ios --release --no-codesign --no-pub:通过。
  • 包含本次审查修复的开发测试包 2.3.8(20310)已安装到 iOS 27.0.1、iPhone 16 Pro;提交者于 2026-10-12 确认已测试。更完整的性能与兼容性回归项目见下述文档。
  • 16 项 Swift 原生 XCTest 已通过编译,覆盖插件原样转发、像素缓冲区引用释放、输出切换及前后台恢复;执行受本机 CoreSimulator 的 runtime path not found 错误阻塞,尚未取得运行结果。
  • 配套 media-kit 的 14 项原生生命周期 / 线程测试通过;macOS 应用构建和设备回归尚未执行。

小窗期间使用同一个 mpv 播放器的软件视频输出,关闭后在前台恢复 GPU 输出;会增加 CPU / 内存带宽开销。Flutter 弹幕及字幕覆盖层不显示在小窗内,暂不自动进入小窗。使用说明和后续性能回归项目见 docs/ios-picture-in-picture.md。

本轮审查修复:

  • 复用 utils 中的 AsyncSingleFlight / AsyncSerialQueue。
  • 删除对 media-kit 内部方法委托的拦截与 Create / SetSize / Dispose 重放,改用插件公开的有类型原生接口。配套接口见 media-kit PR #41,当前两项依赖及 lockfile 均锁定 fork 提交 cb118d7d078dd25fbbad3e67180eb5b6639f6044,无需本地路径覆盖;配套改动合入后可一起切回上游。
  • iOS PiP 激活和进入等待状态通过 MobX 更新控制层,移除相关 setState。

Comment thread lib/pages/player/controller/player_playback_controller.dart
Comment thread lib/pages/player/player_item.dart Outdated
setState(() {});
if (!active && !restored && !backgroundPlayback &&
WidgetsBinding.instance.lifecycleState == AppLifecycleState.paused) {
unawaited(playerController.pause(enableSync: false));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: The deferred background pause does not suspend prefetch

While _iosPip.keepPlaybackInBackground is true, the paused branch of didChangeAppLifecycleState is skipped, and that branch is what calls playback.setPrefetchSuspended(true) ("pause alone keeps prefetching"). This line, and the matching failed-entry path at line 336, are the deferred version of that branch, but they only call pause. So when PiP is closed, or entry fails, while the app is backgrounded with background playback disabled, prefetch keeps running until the next resumed event. Consider calling setPrefetchSuspended(true) here as well, or reusing the paused-branch logic.


Reply with @kilocode-bot fix it to have Kilo Code address this issue.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kilocode-bot fix it

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To use Kilo from GitHub you first need to link your GitHub account to Kilo. Link your Kilo account to continue. After linking, mention me again in this issue or pull request.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kilocode-bot fix it

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To use Kilo from GitHub you first need to link your GitHub account to Kilo. Link your Kilo account to continue. After linking, mention me again in this issue or pull request.

@kilo-code-bot

kilo-code-bot Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: 2 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 1
Issue Details (click to expand)

WARNING

File Line Issue
pubspec.yaml 97 media_kit / media_kit_video (and pubspec.lock) now resolve from the personal fork sibogentleman/media-kit@cb118d7 instead of Predidit/media-kit. These packages supply native libs for all platforms, and nothing shows the fork includes 803c4a2.

SUGGESTION

File Line Issue
ios/Runner/KazumiPipPluginRegistry.swift 200 retryGeneration is captured when the setSoftwareRendering result arrives. A texture registration that lands while the request is still pending is missed, so a .notReady restore may never be retried.
Files Reviewed (14 files)
  • pubspec.yaml - 1 issue
  • pubspec.lock - same concern as pubspec.yaml
  • ios/Runner/KazumiPipPluginRegistry.swift - 1 issue
  • ios/Runner/AppDelegate.swift
  • ios/Runner/IosPictureInPictureController.swift
  • ios/RunnerTests/RunnerTests.swift
  • lib/pages/player/controller/player_panel_controller.dart
  • lib/pages/player/controller/player_panel_controller.g.dart (generated)
  • lib/pages/player/player_item.dart
  • lib/services/player/ios_pip_controller.dart
  • lib/services/player/ios_pip_video_output_transition.dart
  • test/ios_pip_controller_test.dart
  • test/ios_pip_video_output_transition_test.dart
  • docs/ios-picture-in-picture.md

Incremental review (a7a0f33 → b98956d): This commit removes the interception of media-kit's internal VideoOutputManager.* method calls and switches to a typed KazumiPipRendering API published by the plugin. It also replaces the hand-rolled reservation and completer logic with AsyncSingleFlight / AsyncSerialQueue, and moves PiP UI state into MobX observables. I checked several areas and found no problems:

  • The single-flight and serial-queue semantics match the old behaviour: no double resume, no queue deadlock, and errors are isolated per task.
  • All callers match the new resumeVideo(handle, {waitForTexture}) signature.
  • The new observables are read inside the enclosing Observer, and videoPageController.isPip is already a MobX observable, so dropping setState is safe.
  • The new tests match the production branches they exercise.

Previous findings, checked against the current HEAD: the auto-safe decoder issue in player_playback_controller.dart is fixed (line 421 maps it to auto-copy-safe).

Assumptions / gaps: The media-kit fork sources are not available locally, so I could not verify:

  • the MediaKitVideoPlugin / VideoOutputRenderingResult API;
  • which thread setSoftwareRendering calls its completion on;
  • whether the plugin publishes itself under the key MediaKitVideoPlugin.

Per the author's notes, the Swift XCTest cases compile but have still not been run.

Fix these issues in Kilo Cloud

Previous Review Summaries (5 snapshots, latest commit a7a0f33)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit a7a0f33)

Status: No Issues Found | Recommendation: Merge

Files Reviewed (1 file)
  • docs/ios-picture-in-picture.md

Incremental review: Since a57547a, the only change is to docs/ios-picture-in-picture.md. It updates the device-test notes to dev build 2.3.8 (20309) on iOS 27.0.1, which was built after the Swift migration. It also says the 4 new Swift XCTest cases compile but have not run yet, because of a local CoreSimulator runtime path not found error. No code changed and I found no issues in the text.

Previous findings, checked against the current HEAD:

  • The auto-safe decoder comment on player_playback_controller.dart:422 no longer applies. Line 421 now maps 'auto-safe' => 'auto-copy-safe'.
  • The deferred background-pause/prefetch comment on player_item.dart is outdated on GitHub (it no longer maps to a line), so it is not carried forward.

The native XCTest cases still haven't run, per the author's own note, so the Swift PiP adapter has no runtime test results yet.

Previous review (commit a57547a)

Status: No Issues Found | Recommendation: Merge

Files Reviewed (5 files)
  • ios/Runner.xcodeproj/project.pbxproj
  • ios/Runner/KazumiPipPluginRegistry.swift (replaces the deleted KazumiPipPluginRegistry.h / .m)
  • ios/Runner/Runner-Bridging-Header.h
  • ios/RunnerTests/RunnerTests.swift
  • lib/pages/player/controller/player_playback_controller.dart (checked only the earlier finding)

Incremental review: Since cde6091, the Objective-C PiP plugin-registry adapter has been ported to Swift, and the project file and bridging header have been updated to match. I compared the port against the deleted Objective-C version and found the same behavior in each area:

  • Only the MediaKitVideoPlugin registrar is wrapped.
  • Frames are forwarded in textureFrameAvailable, and the retained buffer from copyPixelBuffer is handed to ARC via takeRetainedValue().
  • The Create / SetSize / Dispose arguments are tracked and the generation counter is checked.
  • The output is still recreated in the order suspend → dispose → create → resume.
  • Restoration completions are still delayed until in-flight recreations finish.

The callers in AppDelegate.swift match the new Swift API. The new XCTest cases cover registrar wrapping, buffer ownership, callback order and restoration timing.

The existing auto-safe decoder comment on player_playback_controller.dart:422 is out of date: line 421 now maps 'auto-safe' => 'auto-copy-safe'.

I couldn't compile any of this here. Without Flutter's iOS headers in the sandbox, I couldn't confirm that every forwarded FlutterPluginRegistrar method, such as valuePublished(byPlugin:), is declared on that protocol, so this rests on the author's report of a passing iOS build.

Previous review (commit cde6091)

Status: No Issues Found | Recommendation: Merge

Files Reviewed (1 file)
  • lib/pages/player/player_item.dart

Incremental review: Since ae33a6f, the paused-lifecycle logic has moved into a shared _pauseForBackground() helper. Both the deferred PiP-close pause (line 308) and the failed-entry pause (line 340) now call it, so prefetch is suspended on those paths as well. This fixes the earlier SUGGESTION. The earlier auto-safe decoder WARNING is still fixed in player_playback_controller.dart:421. No new issues found.

Previous review (commit ae33a6f)

Status: 1 Issue Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 0
SUGGESTION 1
Issue Details (click to expand)

SUGGESTION

File Line Issue
lib/pages/player/player_item.dart 304 When PiP closes in the background, the deferred pause does not call setPrefetchSuspended(true), which the normal paused branch does. The failed-entry path at line 336 has the same gap.
Files Reviewed (1 file)
  • lib/pages/player/controller/player_playback_controller.dart - 0 issues

Incremental review: The only change since 1bc6aa0 maps the default decoder auto-safe to auto-copy-safe on PiP-capable devices. This fixes the earlier WARNING. player_item.dart has not changed, and a targeted check shows the SUGGESTION is still present. No test covers the new auto-safe mapping.

Fix these issues in Kilo Cloud

Previous review (commit 1bc6aa0)

Status: 2 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 1
Issue Details (click to expand)

WARNING

File Line Issue
lib/pages/player/controller/player_playback_controller.dart 421 The default decoder auto-safe falls through unchanged and is not mapped to a copy-back decoder. The stated PiP design (copy-back decoding on PiP-capable devices) therefore does not apply to the default configuration.

SUGGESTION

File Line Issue
lib/pages/player/player_item.dart 304 When PiP is closed in the background, the deferred pause skips setPrefetchSuspended(true), unlike the normal paused branch. The same applies to the failed-entry path at line 336.
Files Reviewed (16 files)
  • README.md - 0 issues
  • docs/ios-picture-in-picture.md - 0 issues
  • ios/Podfile - 0 issues
  • ios/Runner.xcodeproj/project.pbxproj - 0 issues
  • ios/Runner/AppDelegate.swift - 0 issues
  • ios/Runner/IosPictureInPictureController.swift - 0 issues
  • ios/Runner/KazumiPipPluginRegistry.h - 0 issues
  • ios/Runner/KazumiPipPluginRegistry.m - 0 issues
  • ios/Runner/Runner-Bridging-Header.h - 0 issues
  • lib/pages/player/controller/player_playback_controller.dart - 1 issue
  • lib/pages/player/player_item.dart - 1 issue
  • lib/pages/player/player_item_panel.dart - 0 issues
  • lib/services/player/ios_pip_controller.dart - 0 issues
  • lib/services/player/ios_pip_video_output_transition.dart - 0 issues
  • test/ios_pip_controller_test.dart - 0 issues
  • test/ios_pip_video_output_transition_test.dart - 0 issues

Note: media-kit's native source is a git dependency and is not in the workspace. I could not confirm the assumption in KazumiPipPluginRegistry.m that textureFrameAvailable: is called on the platform/main thread. onVideoFrame touches UIKit and layer state without dispatching to main, so this depends on that assumption holding.

Fix these issues in Kilo Cloud


Reviewed by claude-opus-5.5 · Input: 266 · Output: 9.1K · Cached: 637.8K

sibogentleman and others added 2 commits October 9, 2026 18:11
Co-authored-by: kilo-code-bot[bot] <240665456+kilo-code-bot[bot]@users.noreply.github.com>
@Predidit

Predidit commented Oct 9, 2026

Copy link
Copy Markdown
Owner

注意到此实现基于objc,为什么没有使用更新的swift并与项目的其他插件保持一致

@sibogentleman

Copy link
Copy Markdown
Author

手写了部分CVPixel使用的实现,objc 仅仅是个人偏好,会改用 swift 实现后再提交

@sibogentleman

Copy link
Copy Markdown
Author

@Predidit 可以看看最新的 swift 实现,已经在 ios 上本地测试过

Comment thread docs/ios-picture-in-picture.md
@sibogentleman

Copy link
Copy Markdown
Author

@Predidit 看看可以合并不

@Predidit

Copy link
Copy Markdown
Owner

我没有可以测试此功能的设备,需要等待其他贡献者审查

@Predidit

Copy link
Copy Markdown
Owner

在其他贡献者审查前,有以下问题可以先处理

  1. 复用 utils 目录下已有的同步原语
  2. KazumiPipPluginRegistry.swift 的实现非常hack ,hook 了 media-kit 的内部方法,这种操作符合预期吗
  3. _handleIOSPIPModeChanged 使用 setState 触发重建,与 android 通过 mobx 触发重建的方案不一致

Comment thread pubspec.yaml
git:
url: https://github.com/Predidit/media-kit.git
ref: 803c4a27912091db6f839d48d1b984a0d9f588c7
url: https://github.com/sibogentleman/media-kit.git

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: media_kit / media_kit_video now resolve from a personal fork

Both packages (lines 97 and 102, and the matching pubspec.lock entries) move from Predidit/media-kit@803c4a2 to sibogentleman/media-kit@cb118d7. Since 4fed48b removed the media_kit_libs_* overrides, these packages also supply the native libraries for every platform, not just iOS. Merging as-is means:

  • a third-party repository controls code shipped on Android, Windows, macOS, Linux and OHOS;
  • any fix that is in 803c4a2 but missing from cb118d7 is silently lost (nothing in this repo shows cb118d7 descends from 803c4a2);
  • future maintainer media-kit bumps cannot be applied until the dependency is switched back.

Consider merging Predidit/media-kit PR #41 first and pinning the resulting Predidit/media-kit commit here. Failing that, confirm that cb118d7 is rebased on 803c4a2 before merging.


Reply with @kilocode-bot fix it to have Kilo Code address this issue.

renderer.setSoftwareRendering(handle: handle, enabled: software) { result in
if !software && result == .notReady {
self.waitingForRegistration.insert(handle)
retryGeneration = self.registrationGeneration

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: The retry generation is captured when the result arrives, not when the request is made

retryGeneration is read inside the setSoftwareRendering completion. Suppose the busy output's texture registers (and textureRegistered() runs) while that request is still pending. retryRegisteredOutputs() skips the handle, because it is in recreatingHandles and not yet in waitingForRegistration. .notReady then records the already-incremented generation, so the generation != registrationGeneration check in finish (line 183) never fires. The output stays on software rendering in the foreground until the next activation or texture registration.

This is reachable if media-kit completes asynchronously, which the comment at line 205 implies ("completes only once output work has drained"). Capturing let requestGeneration = self.registrationGeneration just before line 197, and assigning it here, would cover this case. It would still handle the case where registration happens after the result, which the existing test covers.


Reply with @kilocode-bot fix it to have Kilo Code address this issue.

@sibogentleman

Copy link
Copy Markdown
Author

看了下 media-kit 的封装还是不够理想,画中画本身的实现只是调用 ios 的 pip 能力,但是依然需要获取视频流,依赖的接口修改先提到你的 fork 分支上,可以先审阅看看,也许会有更好的接口封装方案 @Predidit P

@sibogentleman

Copy link
Copy Markdown
Author

@Predidit 改动比较多,但是我觉得这个功能本身还是比较实用的,可以有空的时候给一些意见吗

@Predidit

Copy link
Copy Markdown
Owner

我没有 mac ,这个功能需要其他维护者审查

sibogentleman and others added 3 commits October 14, 2026 09:27
Preserve the Darwin rendering API while incorporating the upstream
Android libmpv 1.3.0 bundle update and Flutter 3.47.7 pin.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants