feat(app): make demo launches asynchronous and feature the 100-table

[ravi-databricks]

Summary

Improve the Databricks App demo experience by replacing the long list of demo
cards with a compact featured-demo picker, making the At-scale Auto Loader
100-table demo the default, and running every demo asynchronously with streamed
logs and clickable Databricks run URLs.

The change also hardens background subprocess execution, makes local App
authentication non-interactive, propagates launcher failures correctly, and
prevents unbounded in-memory job/log growth.

Problem

The existing demo-launch experience has several usability and reliability
issues:

• The Demos page presents a long list of launcher cards that is difficult to
  scan.
• Demo launch requests can remain behind a blocking spinner while the Python
  launcher waits for a Databricks job or pipeline.
• Long-running demos, including the 100-table Auto Loader demo and Interactive
  Notebook demo, may run for 60–90 minutes.
• The progress dialog cannot be dismissed while a demo is running.
• Databricks job and pipeline URLs are printed as plain text instead of
  actionable links.
• Local App launches can fall through to interactive authentication and fail
  with EOFError when no terminal input is available.
• Legacy demo launchers catch exceptions without re-raising them, allowing the
  App to report false success.
• Background subprocesses use a short silence timeout that can terminate a
  healthy remote job waiter.
• Completed job records and verbose logs remain in process memory without
  bounded retention.
• Interval-based browser polling may overlap when a previous HTTP request is
  still active.

Proposed User Experience

Featured demo picker

Replace the six large demo cards with one dropdown:

Featured demo
  At-scale Auto Loader (100 tables)   [default]
  Interactive Notebook
  Cloud Files
  Apply Changes Snapshot
  Silver Fanout
  DAIS Demo

The selected demo displays its title, icon, and short description next to one
Run featured demo action.

Featured onboarding entry

The Onboarding page's Pick a bundled demo dropdown should default to:

★ Featured — At-scale Auto Loader (100 tables)

When selected:

• The main Onboarding action changes to Run Featured Demo.
• The catalog entered on the Onboarding page is reused.
• The 100-table demo launches directly from the same page.
• The user is not redirected to the Demos page.
• Template Preview is disabled because the featured demo is an orchestrated
  launcher rather than one static onboarding template.

Selecting a normal bundled onboarding specification restores the existing
Run Onboarding and Preview behavior.

Asynchronous launch

Every registered demo should follow the same flow:

1. Validate the requested demo and Unity Catalog access.
2. Allocate a background job token.
3. Start the launcher subprocess.
4. Return HTTP 202 immediately.
5. Stream stdout and stderr through the progress API.
6. Detect completion and surface success or failure.

The App page must remain usable while the remote Databricks job continues.

Progress dialog

The progress dialog should:

• Show the selected demo's real title.
• Stream stdout and stderr incrementally.
• Render HTTPS job and pipeline URLs as clickable links.
• Show the number of received log lines.
• Allow Close while the demo continues in the background.
• Stop overlapping requests by scheduling the next poll only after the current
  request completes.
• Surface polling HTTP failures instead of retrying silently forever.

Backend Design

Demo registry

Keep one allow-listed registry mapping UI command names to launcher files,
catalog argument names, and optional fixed arguments.

Registered App demos:

• At-scale Auto Loader
• Interactive Notebook
• Cloud Files
• Apply Changes Snapshot
• Silver Fanout
• DAIS

Unsupported launchers that require Terraform or unavailable external
infrastructure remain excluded from the App.

Shared background runner

Use one subprocess runner for CLI and demo commands.

Required behavior:

• Unbuffered stdout/stderr streaming.
• Separate reader threads for both streams.
• Graceful terminate, followed by kill after a bounded grace period.
• Guaranteed child reaping to avoid zombie processes.
• Partial output preservation when the process fails or times out.
• Optional cleanup of temporary files.
• Configurable idle timeout.

Onboarding and deployment commands retain the shorter default idle timeout.
Demo launchers use a two-hour idle timeout because healthy remote jobs may
produce no output while waiting for completion.

Job registry

The in-process job registry should:

• Use a re-entrant lock for iteration and mutation.
• Return stable snapshots to request handlers.
• Retain at most 5,000 log lines per job.
• Preserve an absolute log offset when old lines are evicted.
• Return next_offset to polling clients.
• Retain at most 50 completed jobs.
• Remove completed jobs after one hour.

Authentication behavior

Deployed Databricks Apps continue to use ambient service-principal
authentication.

Local Flask runs should:

• Forward DATABRICKS_CONFIG_PROFILE to demo launcher commands as
  --profile <name>.
• Avoid setting DATABRICKS_APP_PORT artificially.
• Preserve local workspace notebook paths with their .py suffix.
• Never prompt for workspace URL or token from a background process.

Failure propagation

Legacy demo launchers must re-raise caught exceptions after logging their
tracebacks. A failed launcher must produce a non-zero subprocess exit code so
the App reports failure.

Affected launchers:

• launch_af_cloudfiles_demo.py
• launch_acfs_demo.py
• launch_silver_fanout_demo.py
• launch_dais_demo.py

API Contract

Start a demo

POST /rundemo
Content-Type: application/json

{
  "demo_name": "demo_at_scale_autoloader",
  "uc_name": "main"
}

Accepted response:

{
  "token": "<job-token>"
}

Status: 202 Accepted

Poll logs

GET /api/job/<job-token>/logs?offset=0

Response:

{
  "logs": [
    {
      "stream": "stdout",
      "line": "At-scale demo job: https://..."
    }
  ],
  "next_offset": 1,
  "done": false,
  "returncode": null,
  "error": null
}

When complete, done is true and result contains the parsed command
result.

Invalid offsets return 400. Missing or expired tokens return 404.

Test Plan

Automated

• Run focused App tests:

pytest \
  tests/test_app_subprocess_runner.py \
  tests/test_app_demo_job_logs.py \
  tests/test_app_landing_page.py \
  tests/test_app_onboarding_preview.py

• Run lint and whitespace validation:

flake8 \
  databricks_app/_jobs.py \
  databricks_app/_subprocess_runner.py \
  databricks_app/routes/demo.py \
  tests/test_app_demo_job_logs.py \
  tests/test_app_landing_page.py

git diff --check

Manual

• Run the App locally with an FEVM profile.
• Confirm At-scale Auto Loader is the default featured demo.
• Launch each supported demo and verify immediate progress UI.
• Confirm the job or pipeline URL becomes clickable.
• Close the dialog and verify the remote run continues.
• Verify local launch commands contain --profile fevm.
• Deploy the App to FEVM and repeat the featured launch flow.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.24%. Comparing base (e6e1945) to head (8ea1ed9).
⚠️ Report is 1 commits behind head on v0.1.1.

Additional details and impacted files

@@           Coverage Diff           @@
##           v0.1.1     #492   +/-   ##
=======================================
  Coverage   89.24%   89.24%           
=======================================
  Files          18       18           
  Lines        5030     5030           
  Branches     1029     1029           
=======================================
  Hits         4489     4489           
  Misses        313      313           
  Partials      228      228           

Flag	Coverage Δ
unittests	89.24% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[brij-raghuwanshi-db]

Approved.