New recipe: Browser Automation: Invoice Drafts and Web Research

authors.yaml

@@ -52,3 +52,7 @@ colinmurphy326:
   avatar: "https://github.com/colinmurphy326.png?size=200"
   githubUrl: "https://github.com/colinmurphy326"
   linkedinUrl: "https://www.linkedin.com/in/colinmurph"
+
+arunt14:
+  name: "Arun Kumar Thiagarajan"
+  title: "PM"

notebooks/data/browser-automation-invoice-and-research/.env.example

@@ -0,0 +1,15 @@
+AZURE_SUBSCRIPTION_ID=<your-subscription-id>
+AZURE_TENANT_ID=<subscription-tenant-id>
+PROJECT_ENDPOINT=https://<account>.services.ai.azure.com/api/projects/<project>
+MODEL_DEPLOYMENT=<deployment-name>
+BROWSER_CONNECTION_ID=<full-project-connection-resource-id>
+BROWSER_MODE=workspace-mcp
+BROWSER_WORKSPACE_MCP_URL=https://<region>.mcp.playwright.microsoft.com/playwrightworkspaces/<workspace-id>/mcp
+# Required only for BROWSER_MODE=toolbox-mcp; no token or key values.
+BROWSER_TOOLBOX_URL=https://<account>.services.ai.azure.com/api/projects/<project>/toolboxes/<name>/versions/<version>/mcp?api-version=v1
+BROWSER_TOOLBOX_CONNECTION_ID=<temporary-remote-tool-connection-id>
+DEMO_FORM_URL=https://<storage-static-website-host>/invoice-entry.html
+DEMO_EXPECTED_MARKER=<marker-from-local-hosted-marker.json-not-from-a-prompt>
+TEST_DEMO_MARKER=<same-locally-rendered-marker-for-ui-tests-only>
+RESEARCH_WINDOW_START=<YYYY-MM-DD>
+ARTIFACT_DIR=<absolute-local-artifact-directory>

notebooks/data/browser-automation-invoice-and-research/.gitattributes

@@ -0,0 +1 @@
+assets/*.pdf binary

notebooks/data/browser-automation-invoice-and-research/.gitignore

@@ -0,0 +1,5 @@
+.env
+.venv/
+__pycache__/
+.pytest_cache/
+*.pyc

notebooks/data/browser-automation-invoice-and-research/README.md

@@ -0,0 +1,144 @@
+# Browser automation companion
+
+Run the tutorial in `notebooks\browser-automation-invoice-and-research.ipynb`.
+This folder contains its scoped Python package, synthetic fixtures, and setup
+helpers. Do not download the notebook without these files.
+
+The pattern is draft-only and read-only: validate inputs in code, expose minimal
+browser tools, and compare model claims with actual tool evidence. There is no
+production submission, payment, duplicate database, or hard browser network
+allowlist.
+
+## Offline setup
+
+From this directory in PowerShell:
+
+```powershell
+python -m venv .venv
+.\.venv\Scripts\python.exe -m pip install -r requirements.txt
+.\.venv\Scripts\python.exe -m pytest -q
+```
+
+The notebook supports launch from the checkout root, `notebooks`, or this folder.
+Select a kernel with these dependencies installed. Ordinary Run All does not load
+`.env`, create Azure clients, provision resources, or make paid calls, even if
+`LIVE_APPROVED` is inherited. The fixtures are included; do not regenerate or
+overwrite the tutorial notebook.
+The scoped `.gitattributes` keeps PDF fixtures binary so Windows line-ending
+conversion does not rewrite them.
+
+`bat_recipe` contains the contracts, runner, evidence checks, and workflows.
+`scripts` contains local CLI helpers; run their module commands from this folder.
+Fixture and instruction paths are anchored to this directory, not the caller's
+working directory. Only this directory's ignored `.env` is loaded by live commands.
+
+## Browser routes and evidence
+
+The routes are selected explicitly with `BROWSER_MODE`; the runner never changes
+routes after a failure:
+
+| Mode | Foundry integration | Evidence behavior |
+|---|---|---|
+| `native-ga` | GA Browser Automation Tool with a `PlaywrightWorkspace` connection | The current verifier cannot establish snapshot-level URL provenance from this route; results remain unverified. |
+| `workspace-mcp` | Playwright Workspaces remote MCP preview with a `RemoteTool` connection and project managed identity | The verifier requires `browser_snapshot` tool output with structured `current_url` metadata, scoped navigation, and the same closed session. |
+| `toolbox-mcp` | Project toolbox MCP with its own configured endpoint and connection | The current verifier cannot establish snapshot-level URL provenance from this route; results remain unverified. |
+
+The remote MCP service is a preview and is not recommended for production. The
+GA Browser Automation Tool and the separate remote MCP preview are distinct
+integrations and connection categories. The required snapshot URL metadata has
+not been confirmed in a fresh live response; missing metadata fails verification.
+
+## Optional authorized cloud setup
+
+Follow the notebook's full prerequisites before executing setup helpers. The
+route covered by this setup is `BROWSER_MODE=workspace-mcp`, using:
+
+```text
+https://<region>.mcp.playwright.microsoft.com/playwrightworkspaces/<workspace-id>/mcp
+```
+
+Do not use the WebSocket `/browsers` endpoint. The project connection needs
+`RemoteTool`, `ProjectManagedIdentity`, audience
+`https://mcp.playwright.microsoft.com`, and workspace ARM metadata. The project's
+managed identity needs Playwright Workspace Contributor at that workspace's scope.
+The caller separately signs in through Azure CLI to the intended tenant.
+
+After separate authorization, `scripts\create_mcp_connection.ps1` creates a
+uniquely named connection and reads its metadata back. It does not grant roles.
+Render a fresh synthetic form with `python -m scripts.render_demo <new-directory>`
+and host only its HTML using `scripts\host_demo.ps1`. Hosting creates temporary
+Azure resources and uploads public synthetic content; approve it separately.
+Prior trial hosting is deleted and must not be reused.
+
+Copy `.env.example` to `.env` and replace placeholders. Keep the marker from the
+local `hosted-marker.json` out of prompts, instructions, schemas, and URLs. Use
+an artifact directory outside the checkout and a valid research start date.
+There are no credentials in the managed-identity browser configuration.
+Both notebook and CLI reject artifact directories inside the checkout, including
+the recipe's own source directory, before creating files or service clients.
+
+## One recommended live sequence
+
+Use either the notebook's deliberately enabled optional cell or the commands
+below, not both. Authorize pricing and the bounded run before setting the flag:
+
+```powershell
+$env:LIVE_APPROVED = "1"
+.\.venv\Scripts\python.exe -m scripts.diagnose_bat probe
+# Stop on failure. Continue only after actual marker readback passed.
+.\.venv\Scripts\python.exe -m scripts.diagnose_bat validate
+.\.venv\Scripts\python.exe -m scripts.summarize_run '<absolute-artifact-directory>'
+```
+
+The CLI rejects unapproved live execution before agent creation. The validation
+stage also rejects a missing, malformed, or future `RESEARCH_WINDOW_START` before
+creating clients, agents, or invoice requests. The marker-only probe does not
+require a research window; the combined notebook sequence checks it before
+starting either stage.
+The validation gate is tied to the current marker, URL, model, connection, and
+browser mode.
+Validation uses six requests: happy PDF extraction, drafting, three source runs,
+and synthesis. It does not rerun the malicious PDF or repeat the historical
+failed native-tool diagnostics. The ledger permits eight total requests and at
+most two diagnostic probes. A second probe needs a distinct rationale and
+remaining authorization; a failed probe is not automatic retry permission.
+
+Run only one process against an artifact directory. Do not delete the ledger to
+regain budget. Timeouts and request limits are not a hard billing cap or proof
+that remote work stopped. Native/toolbox compatibility helpers remain available
+for existing evidence tests; they are not an alternate recommended live path.
+
+## Review and cleanup
+
+Review `happy-draft.json` together with `happy-draft-verification.json`.
+Successful execution requires a supported `browser_snapshot` response with
+structured observed `current_url` matching the exact form URL, a fresh marker,
+all nine values, zero submission attempts, and same-task session closure.
+Research needs source-scoped observed snapshots, direct quotes/dates or an
+evidence-backed limitation, session closure, and faithful synthesis.
+
+Dated findings include `date_evidence_quote`: a copied publication/update
+statement in the same browser snapshot as the exact browser-observed direct-page
+URL and content quote. The `current_url` must come from structured browser-tool
+metadata, not a navigation argument, agent claim, or text found on the page.
+The verifier compares its date with `published_or_updated`. It accepts labeled
+Published, Updated, or Released statements using ISO `YYYY-MM-DD` or an English
+month name with day and four-digit year, such as `Updated: September 15, 2026`.
+Unrelated event dates or dates from another observation are not proof.
+Unsupported or ambiguous date formats require `date_not_verified`, a null
+`date_evidence_quote`, and an explicit limitation. A model-claimed date without
+matching evidence fails verification; it is not silently corrected. An undated
+finding is logged as `no_verified_findings`, not as a verified in-window change.
+An empty findings result also needs `limitation_evidence_quote` copied from an
+in-scope page observed in the same closed browser session; a limitation alone
+cannot support a no-findings result.
+
+MCP discovery, pending approval, empty output, or final model prose is not proof.
+No findings can be a truthful research outcome, not proof that no change occurred.
+The historical manuscript's cloud results are not a fresh execution of this port.
+
+The runner removes only its owned agent version and closes clients. After
+review and deletion authorization, use a reviewed created-resource inventory
+with `scripts\cleanup_demo.ps1`. Preserve shared resources and existing roles,
+and independently verify deletion and any failed remote-session lifecycle.
+Keep raw service evidence, identifiers, usage records, and artifacts local.

notebooks/data/browser-automation-invoice-and-research/assets/expected_invoice_result.json

@@ -0,0 +1,11 @@
+{
+  "vendor_name": "Contoso Office Supplies",
+  "invoice_number": "INV-10482",
+  "invoice_date": "2026-09-15",
+  "currency": "USD",
+  "subtotal": 1250.00,
+  "tax": 225.00,
+  "total": 1475.00,
+  "purchase_order": "PO-8421",
+  "due_date": "2026-10-15"
+}

notebooks/data/browser-automation-invoice-and-research/bat_recipe/browser_evidence.py

@@ -0,0 +1,162 @@
+from dataclasses import dataclass
+import json
+
+
+NATIVE_CALL_TYPES = {"browser_automation_preview_call", "browser_automation_call"}
+NATIVE_OUTPUT_TYPES = {f"{kind}_output" for kind in NATIVE_CALL_TYPES}
+CALL_ITEM_TYPES = NATIVE_CALL_TYPES | {"mcp_call"}
+TOOL_ITEM_TYPES = CALL_ITEM_TYPES | NATIVE_OUTPUT_TYPES | {
+    "mcp_list_tools", "mcp_approval_request",
+}
+MCP_SERVER_LABEL = "bat_toolbox"
+MCP_TOOL_NAME = "browser_automation"
+WORKSPACE_MCP_LABEL = "bat_workspace"
+WORKSPACE_READ_TOOLS = {
+    "create_browser_session", "browser_navigate", "browser_snapshot",
+    "browser_find", "close_browser_session",
+}
+WORKSPACE_DRAFT_TOOLS = WORKSPACE_READ_TOOLS | {"browser_fill_form"}
+
+
+@dataclass(frozen=True)
+class BrowserObservation:
+    invocation_id: str
+    protocol: str
+    output: str
+    tool_name: str | None = None
+    session_id: str | None = None
+    current_url: str | None = None
+
+
+def _arguments(value) -> dict:
+    if isinstance(value, str):
+        value = json.loads(value)
+    if not isinstance(value, dict):
+        raise RuntimeError("Browser call arguments are not a JSON object")
+    return value
+
+
+def _snapshot_output(value: str) -> tuple[str, str | None]:
+    try:
+        decoded = json.loads(value)
+    except json.JSONDecodeError:
+        return value, None
+    if not isinstance(decoded, dict):
+        return value, None
+
+    structured = decoded.get("structuredContent")
+    if not isinstance(structured, dict):
+        structured = {}
+    current_url = structured.get("current_url")
+    if current_url is None and isinstance(decoded.get("content"), list):
+        current_url = decoded.get("current_url")
+
+    content = []
+    for item in decoded.get("content", []):
+        if isinstance(item, dict) and item.get("type") == "text":
+            text = item.get("text")
+            if isinstance(text, str):
+                content.append(text)
+    for key in ("snapshot", "text", "output"):
+        text = structured.get(key)
+        if isinstance(text, str):
+            content.append(text)
+    if not content:
+        content = [value]
+    return "\n".join(content), current_url if isinstance(current_url, str) else None
+
+
+def reconcile_tool_items(stream_items: list[dict], final_items: list[dict]) -> list[dict]:
+    items = {}
+    for item in stream_items + final_items:
+        if item.get("type") not in {"message", "reasoning"}:
+            key = item.get("id") or (item.get("type"), item.get("call_id"))
+            items[key] = item
+    return list(items.values())
+
+
+def checked_output(value) -> str:
+    if not isinstance(value, str) or not value.strip():
+        raise RuntimeError("BAT returned empty or missing tool output; browser outcome unverified")
+    try:
+        decoded = json.loads(value)
+    except json.JSONDecodeError:
+        decoded = None
+    if isinstance(decoded, dict) and (
+        decoded.get("error") or decoded.get("isError") is True
+        or decoded.get("status") in ("error", "failed")
+    ):
+        raise RuntimeError("BAT returned a tool error, not successful browser evidence")
+    return value
+
+
+def browser_observations(evidence: list[dict], *, mode: str | None = None) -> list[BrowserObservation]:
+    if any(not isinstance(item, dict) or item.get("type") not in TOOL_ITEM_TYPES
+           for item in evidence):
+        raise RuntimeError("Malformed or unsupported BAT evidence item")
+    if any(item["type"] == "mcp_approval_request" for item in evidence):
+        raise RuntimeError("MCP approval is pending; browser execution is not verified")
+    native = [item for item in evidence if item["type"] in NATIVE_CALL_TYPES | NATIVE_OUTPUT_TYPES]
+    mcp = [item for item in evidence if item["type"] == "mcp_call"]
+    if native and mcp:
+        raise RuntimeError("Mixed native and MCP evidence is not a verified single browser mode")
+    observations = []
+    if native:
+        if mode in {"toolbox-mcp", "workspace-mcp"}:
+            raise RuntimeError("Native evidence does not match the configured MCP mode")
+        if any(not isinstance(item.get("call_id"), str) or not item["call_id"] for item in native):
+            raise RuntimeError("BAT call/output is missing its correlation ID")
+        calls = [item for item in native if item["type"] in NATIVE_CALL_TYPES]
+        outputs = [item for item in native if item["type"] in NATIVE_OUTPUT_TYPES]
+        call_ids = {item["call_id"] for item in calls}
+        output_ids = {item["call_id"] for item in outputs}
+        if (len(call_ids) != len(calls) or len(output_ids) != len(outputs)
+                or call_ids != output_ids):
+            raise RuntimeError("BAT call/output correlation is missing, duplicated or mismatched")
+        by_id = {item["call_id"]: item for item in outputs}
+        for call in calls:
+            result = by_id[call["call_id"]]
+            if result["type"] != f"{call['type']}_output":
+                raise RuntimeError("BAT call/output protocol types do not match")
+            if call.get("status") != "completed" or result.get("status") != "completed":
+                raise RuntimeError("BAT call/output did not complete")
+            observations.append(BrowserObservation(
+                call["call_id"], call["type"], checked_output(result.get("output")),
+            ))
+    elif mcp:
+        if mode == "native-ga":
+            raise RuntimeError("MCP evidence does not match the configured native mode")
+        ids = set()
+        for call in mcp:
+            identity = call.get("id")
+            if not isinstance(identity, str) or not identity or identity in ids:
+                raise RuntimeError("MCP invocation ID is missing or duplicated")
+            ids.add(identity)
+            label, name = call.get("server_label"), call.get("name")
+            if label == MCP_SERVER_LABEL:
+                allowed = name == MCP_TOOL_NAME and mode in (None, "toolbox-mcp")
+            elif label == WORKSPACE_MCP_LABEL:
+                allowed = name in WORKSPACE_DRAFT_TOOLS and mode in (None, "workspace-mcp")
+            else:
+                allowed = False
+            if not allowed:
+                raise RuntimeError("MCP call does not target the allowlisted BAT server/tool")
+            if call.get("status") != "completed" or call.get("error"):
+                raise RuntimeError("MCP browser call failed or did not complete")
+            output = checked_output(call.get("output"))
+            tool_name = call["name"]
+            session_id = _arguments(call.get("arguments", "{}")).get("browserSessionId")
+            current_url = None
+            if tool_name == "browser_snapshot":
+                output, current_url = _snapshot_output(output)
+            observations.append(BrowserObservation(
+                identity, "mcp_call", output, tool_name,
+                session_id if isinstance(session_id, str) else None, current_url,
+            ))
+    if not observations:
+        raise RuntimeError("Browser Automation Tool was not invoked")
+    return observations
+
+
+def verify_browser_outputs(evidence: list[dict], *, mode: str | None = None) -> int:
+    return len(browser_observations(evidence, mode=mode))