Repository navigation
RFC: opt-in app access on the phone (Gmail, Uber, Lyft, Google Maps) #70
Description
Sibling of #57 (generative UI track). App cards and brand-colored activity rows are in draft PR #71 (demo video there); this RFC covers the part that PR deliberately leaves out: letting T3 Code use the person's other apps (Gmail, Uber, Lyft, Google Maps, ...) from their phone, opt-in.
Problem
Agents in T3 Code can only reach services that a provider on the host has a tool for. On the phone, the services people actually use day to day (mail, rides, maps) are installed apps with their own sign-in, and the agent can't touch them. Dash and z0bridge already let an agent act on the phone, behind scoped, revocable grants. T3 Code has no equivalent.
Goals
- A person can opt in, per app, to let agents in T3 Code read from or act through that app.
- Reading and acting are separate grants. Every act (book, send, buy) asks first, on the phone, every time, unless the person turns that off for that app.
- Works the same for every provider (Codex, Claude, Cursor, Grok, OpenCode): the phone shows up as one more MCP server.
- Every grant can be seen, paused and revoked in Settings → Connections, and every call is in a per-app log.
- Results render as the app cards from the PR (RideOptions, MailList, Route, PullRequest), so it is obvious which app was used.
Non-goals
- Driving other apps by reading the screen and tapping (AccessibilityService). Fragile, hostile-content risk, and Play restricts it to real accessibility tools.
- Showing third-party logos or pretending to be the app. Cards use the brand name and color only.
- Background access while the phone is locked.
Options compared
| Route | What it gives | Who can use it | Fit |
|---|---|---|---|
| Host MCP connectors (Gmail API, Maps API, Uber/Lyft APIs) | Structured read and act, works for web, desktop and remote | Anyone with an API key or OAuth | Best for desktop and remote. Uber/Lyft public APIs are restricted, so rides need the phone path |
| Android AppFunctions (Android 16+) | Apps declare typed functions; an agent calls them | Callers need EXECUTE_APP_FUNCTIONS, which ordinary apps don't get; few apps ship functions |
Right long-term shape. Not usable by T3 today; design the grant model so it can slot in |
Intents and deep links (geo:, google.navigation:, uber://, lyft://, mailto:) |
Hand off a prepared action to the app; the person confirms there | Any app; needs <queries> entries on Android 11+ |
Best act path on the phone today. Safe by construction: the app shows its own confirm screen |
App widget hosting (AppWidgetHost, user grants bind via ACTION_APPWIDGET_BIND) |
Real, live widgets from the installed app, drawn inside T3 | Any app with a host and a one-time per-widget consent dialog | The only way to show the app's own UI. Display only, sizes vary, and few apps' widgets carry task state. Optional extra, not the base |
| iOS App Intents / Shortcuts | Your own app's actions to Siri; other apps only via a user-made Shortcut (shortcuts://run-shortcut) |
iOS | Read and act through user-built Shortcuts; no direct calls to other apps |
| AccessibilityService | Read and tap anything | Play policy restricts it | Rejected (see non-goals) |
Proposal
- Phone connector. The mobile app exposes a small MCP server over the existing T3 connection, so it works locally, over Tailscale and through T3 Connect, with no new port. Tools are per app and per capability, for example
maps.open_directions,uber.open_ride,gmail.search(read through the host Gmail connector when present). Act tools build an intent or deep link and wait for the person to confirm in the target app. The result says "opened in Uber, not booked" until the app reports otherwise. - Grants. Each app has two toggles, Read and Act, off by default. They are stored on the environment, scoped to the device, and shown in Settings → Connections → This phone. Pause all is one tap. Revoking a grant removes the tools from the next turn. This mirrors z0bridge's scoped tokens (kvnloo/z0bridge#3, [muse] fix(server): bound app-server RPC waits with per-request timeouts #4) and Dash's typed agent→app protocol ([ChatGPT lane] DASH-10: Agent→app UI control: SPEC + typed wire protocol and validator dash#368).
- Approvals. Act calls arrive on the phone as normal T3 approvals with the app card as the preview ("UberX · $31.20 · Ferry Building → SFO"). Declining is the default after a timeout.
- Rendering. Activity rows use the brand mark and sentence from the PR ("Requesting an Uber"). The final answer uses the matching app card.
- Widgets, later and optional. On Android, a person can pin a real widget (for example the Uber ride-status widget) under a card. The bind dialog is the consent. If the widget is removed or the bind is refused, the card remains.
Surfaces and modes
- Mobile: connector, grants UI, approvals. Android first; iOS gets Shortcuts-based actions and the same grants.
- Web and desktop: see the phone's grants and the call log; approvals can be answered on any client, but act approvals default to the phone.
- Remote, relay and tunnel: the connector rides the existing session, so it works the same way in all three. If the phone is offline, its tools are not offered rather than failing mid-turn.
- Contracts: grant records, the connector tool list and the call log are typed in
packages/contracts. - Reverse states: every grant has pause and revoke, and every widget pin has unpin.
Phases
- Grants model and Settings UI with read-only Maps (
open_directions) and an approval round trip. Done when a turn on a remote desktop opens directions on the phone after one approval. - Uber and Lyft deep links plus the RideOptions card. Gmail through the host connector.
- AppFunctions caller, behind a capability check, as soon as a caller role is available to T3.
- Widget pinning (Android).
Open questions
- Should grants live on the environment (shared by every client of that server) or on the phone only?
- Is a per-app act approval enough, or do purchases need a per-amount limit?
- Does the phone connector belong in the stock app or behind a lab flag first?
Prior art
- Dash: agent→app UI control spec and typed wire protocol ([ChatGPT lane] DASH-10: Agent→app UI control: SPEC + typed wire protocol and validator dash#368).
- z0bridge: one pairing and identity layer with scoped tokens (kvnloo/z0bridge#3, [muse] fix(server): bound app-server RPC waits with per-request timeouts #4).
- Android: AppFunctions overview (developer.android.com/ai/appfunctions), widget host guide (developer.android.com/guide/topics/appwidgets/host).
Activity