RHEL-LAB-002: offline-first KVM Virtual Testbed and Daybreak readiness gates

AGENTS.md

@@ -15,6 +15,14 @@ Develop inspectable, reproducible, **defensive** reliability and security resear
 5. Use an isolated guest network without bridged access by default; never assume virtualization alone prevents egress. Validate containment before every experimental campaign.
 6. On unexpected access, sensitive data exposure, or scope drift: stop execution, preserve minimal evidence securely, and escalate to the owner.
 
+## Local testbed gate
+
+- Never enable GitHub Pages, publish a project URL/dashboard, or add a hosted endpoint for this repository.
+- Keep guest networking disabled by default. No NIC, NAT, bridge, public graphics or host passthrough without separate explicit scope review.
+- Never run scripts/testbed.py prepare or snapshot, virsh define/start, or any other state-changing host step without operator approval. GitHub CI must only test pure/mocked code.
+- Do not represent a successful static XML audit as proof of real network containment or a completed RHEL experiment.
+- Stage Daybreak repository code reviews separately. No scans or automated fixes are authorized merely by creating this PR.
+
 ## Engineering defaults
 
 - Prefer Python standard library; use uv run --no-project for standalone scripts. If dependency management is truly needed, use uv add and uv lock with explicit versions.

README.md

@@ -4,7 +4,7 @@
 
 RHEL-LAB is an Apache-2.0-licensed research workspace for controlled experiments on RHEL installations in isolated virtual machines running on owned or explicitly authorized hardware. The project prioritizes exact provenance, independently repeatable results, and responsible disclosure.
 
-> Status: RHEL-LAB-001 foundation. This repository does not contain a tested RHEL VM image, demonstrated vulnerability, or completed kernel-fuzzing campaign.
+> Status: RHEL-LAB-002 virtual-testbed tooling proposed for review. No RHEL VM has been installed, tested or booted by this repository.
 
 ## Research boundaries
 
@@ -25,6 +25,7 @@ See [AGENTS.md](AGENTS.md), [SECURITY.md](SECURITY.md), and [docs/RESEARCH_PROTO
 | docs/RESEARCH_PROTOCOL.md | Experiment gates, provenance, disclosure |
 | docs/ROADMAP.md | Phased milestones and acceptance criteria |
 | scripts/evidence.py | Dependency-free SHA-256 evidence inventory and verifier |
+| scripts/testbed.py | Operator-gated local KVM profile, auditing and offline disk snapshots |
 | tests/ | Safe standard-library unit tests |
 | .github/workflows/ci.yml | Non-privileged source checks only |
 
@@ -45,6 +46,20 @@ To inventory an evidence directory, write the manifest **outside** the evidence
 
 The example requires the capture directory to exist. The manifest is deterministic (no generated timestamp), records relative paths, byte sizes and SHA-256 digests, rejects symlinks, and will not overwrite an existing manifest. For provenance, separately record target/kernel/package versions, test configuration, seed, exact source SHA, time, tool versions, and whether the test actually executed. A manifest establishes byte identity, not scientific validity.
 
+## Virtual testbed (RHEL-LAB-002)
+
+The optional KVM/QEMU testbed is **offline-first, local-only, and operator-gated**. It uses an unprivileged libvirt user connection, a VM profile with no virtual NIC, serial-console-only installation, and private disk/snapshot state outside the repository. It has no Pages deployment or public endpoint.
+
+- [Virtual Testbed guide](docs/VIRTUAL_TESTBED.md) — plan, manual preparation, definition/audit, offline snapshots, limitations.
+- [Daybreak security gate](docs/DAYBREAK_GATE.md) — source scanning and fixes come **before** any authorized live experimentation.
+- [Lab profile](lab/testbed.json) and [baseline receipt template](templates/baseline_receipt.json).
+
+Once a local, authorized installer ISO is available, inspect the zero-write plan with:
+
+    uv run --no-project python scripts/testbed.py plan --lab-root "$HOME/rhel-lab-state" --iso "$HOME/Downloads/RHEL-INSTALLER.iso"
+
+This command only produces an XML/JSON plan and performs no host mutation. Host-changing commands require separate explicit approval; no actual guest, security scan or isolated network proof has been performed.
+
 ## Research priorities
 
 1. Establish isolated, disposable RHEL guests and capture exact environment baselines.

docs/DAYBREAK_GATE.md

@@ -0,0 +1,25 @@
+# Daybreak Defense Factory — staged security gate
+
+**Status: NOT RUN.** RHEL-LAB-002 code should be reviewed through an ordinary GitHub PR and the non-privileged Python CI tests *before* requesting a Daybreak repository scan. Daybreak has not been authorized to run a scan by this milestone.
+
+## Controlled progression
+
+1. Review all changes, particularly the local command boundaries, operator approval tokens, XML allowlist, disk paths, and snapshot race limitations.
+2. Check CI for the exact PR head SHA. Unit tests prove only the Python utilities' tested behaviors, not RHEL kernel behavior.
+3. Request a **repository source-code security scan** through Daybreak when the owner approves it; use the connected QSOLKCB/RHEL-LAB repository and inspect findings.
+4. Fix validated source-code findings on a new feature branch, reproduce reported problems, add regression tests and repeat CI/reviews. No automatic merging.
+5. Approve and merge only after review gates pass.
+6. Separately prepare/verify a local KVM guest, isolation, permitted licensing, disk capacity and offline access. The owner explicitly authorizes any actual host operation.
+7. Only after those gates approve guest startup, local reliability experiments or other active testing. No Red Hat service/third-party probing.
+
+**Important distinction:** Daybreak scans repository source code. It is not a substitute for inspecting real guest firewall rules, proving isolation, running kernel fuzzing or observing a live RHEL VM. No GitHub Actions job may start a guest or conduct privileged/security testing.
+
+## Evidence to retain
+
+- Exact base/head/merge SHA and PR review records
+- Python CI run ID, completion state, test outcomes and logs
+- Daybreak scan run ID and actual findings once requested/executed
+- Individual fixes, test reproductions, negative results and owner approvals
+- Separately stored private VM evidence, logs, release/kernel package identities, snapshot hashes and disclosure records
+
+Do not claim "Daybreak clean" until the exact scan has completed successfully for the exact commit and its findings have been reviewed. Unfinished scans or zero results from a failed run are inconclusive.

docs/LAB_SETUP.md

@@ -1,6 +1,6 @@
 # Local RHEL lab setup — design checklist
 
-This document specifies containment prerequisites. It is not an installation script and no hardware or guest has been provisioned by this repository.
+This document specifies containment prerequisites. The companion scripts/testbed.py offers opt-in local VM preparation and snapshot commands, but no hardware or guest has been provisioned by repository CI. See VIRTUAL_TESTBED.md and DAYBREAK_GATE.md.
 
 ## Operator prerequisites
 

docs/ROADMAP.md

@@ -6,14 +6,20 @@
 - [x] Document lab containment, experiment protocol and security disclosure.
 - [x] Supply deterministic, standard-library SHA-256 evidence inventory and verification.
 - [x] Supply safe unit tests and CI without guest workloads.
-- [ ] Owner review and merge; CI status to be verified separately.
+- [x] Owner reviewed and merged RHEL-LAB-001; CI verified green on the merge commit.
 
-## RHEL-LAB-002 — authorized guest baseline
+## RHEL-LAB-002 — The Virtual Testbed
 
-- [ ] Select exact RHEL release and validate subscription/installation terms.
-- [ ] Document actual hypervisor and guest configurations.
-- [ ] Record signed-off VM scope, offline networking checks, exact package/kernel baseline.
-- [ ] Create reproducible snapshot and artifact receipts.
+- [x] Provide a strictly offline-only x86_64 KVM profile and serial-console-only libvirt domain XML.
+- [x] Add no-write plan, approved disk preparation, persistent/live XML inspection and guarded offline snapshot commands.
+- [x] Add a baseline receipt template, documented containment checklist, regression tests and Daybreak scan gate.
+- [ ] Owner review, CI validation and security audit of this PR.
+- [ ] Select/download exact RHEL ISO under valid entitlement and verify media integrity.
+- [ ] Confirm actual host KVM/libvirt capability and security settings; record authorized VM scope.
+- [ ] Manually define/install the isolated guest after authorization; verify *actual* network and host isolation.
+- [ ] Capture a real clean baseline snapshot with exact guest/kernel/package evidence.
+
+Source tooling completion is not equivalent to guest installation, execution, isolation proof or a completed Daybreak scan.
 
 ## RHEL-LAB-003 — bounded reliability campaign
 

docs/VIRTUAL_TESTBED.md

@@ -0,0 +1,94 @@
+# RHEL-LAB-002 — The Virtual Testbed
+
+This milestone supplies an offline-first *host-side implementation*, not a preinstalled RHEL guest. No VM, kernel experiment, Red Hat service or Daybreak scan is executed in GitHub Actions. The testbed runs only on an operator-owned Linux host, after explicit approval.
+
+## Design constraints
+
+- KVM/QEMU under the least-privileged user libvirt connection: qemu:///session. No sudo or system-wide daemon required by these scripts.
+- One x86_64 guest, 2 vCPUs, 4096 MiB RAM, standalone 48 GiB QCOW2 virtual disk. Configuration is in lab/testbed.json.
+- **No guest NIC, bridge, NAT, hostdev, host share, USB redirection, graphics listener or public service** in the generated domain XML.
+- Serial PTY console only. Guest ISO must be obtained separately from authorized Red Hat distribution; no proprietary image is committed.
+- Images, private receipts and XML live in an operator-selected private 0700 directory **outside the Git repository**.
+- Every host-changing command is explicit, opt-in, and fail-closed. No automatic boot, package installation, VM definition, snapshot restore, or host network configuration.
+- A definition/active XML audit rejects unexpected disks, device passthrough, guest networking and external graphics. **This is a configuration audit, not proof of effective egress isolation.**
+- No GitHub Pages, public dashboard or network endpoint.
+
+## Prepare the host manually
+
+Use a personally controlled Linux host with KVM support and local libvirt/QEMU tools installed by the operator according to their OS instructions. You need virsh, qemu-img, a working per-user qemu:///session connection, access to /dev/kvm and sufficient disk space. Do not install packages automatically as part of CI or this project.
+
+Check the actual hardware, hypervisor, firmware support, host ownership, VM resource budget, filesystem permissions and guest media authenticity yourself. The scripted plan **does not** check that a physical host is isolated.
+
+Download the correct RHEL installer ISO using your own authorized account, verify its official checksum/signature, and keep it outside Git. No vendor access credential belongs in a script, PR or public log.
+
+## Commands (host only, not GitHub Actions)
+
+From the checked-out repository, set paths before any mutating action:
+
+    LAB="$HOME/rhel-lab-state"
+    ISO="$HOME/Downloads/RHEL-INSTALLER.iso"
+
+Preview the generated domain XML and paths without execution. The ISO must actually exist:
+
+    uv run --no-project python scripts/testbed.py plan --lab-root "$LAB" --iso "$ISO"
+
+After manually reviewing the XML and approving disk preparation, run:
+
+    uv run --no-project python scripts/testbed.py prepare \
+      --lab-root "$LAB" --iso "$ISO" \
+      --approve I_AUTHORIZE_LOCAL_VM_PREPARATION
+
+This creates the private folder, a blank QCOW2 image and domain.xml only. It **does not** define, start, install or connect a guest. The exact approval phrase is deliberate.
+
+Inspect domain.xml and then, **only after explicit owner authorization for this VM**, manually define and boot it with the user libvirt connection:
+
+    virsh -c qemu:///session define "$LAB/domain.xml"
+    virsh -c qemu:///session start rhel-lab-offline-01
+    virsh -c qemu:///session console rhel-lab-offline-01
+
+The console is serial-only; the selected RHEL installer must support or be configured for serial/text installation (for example, console=ttyS0 and inst.text via the ISO's own boot configuration). If your ISO does not support that boot path, stop and make a reviewed change rather than adding a public graphics listener or a network adapter.
+
+Audit both persistent and current domain XML (including possible live changes):
+
+    uv run --no-project python scripts/testbed.py audit-defined --lab-root "$LAB" --iso "$ISO"
+
+Re-run this after installation, host/libvirt configuration changes and before each experiment. The audit does not prove firewall state, process isolation or that the guest cannot access host services via other channels. A no-NIC VM still needs review of storage, console, device and host controls.
+
+## Offline snapshot and receipt
+
+First shut down the guest cleanly and establish that **no other process is writing its disk**. This step is mandatory. A guest-state check alone cannot prevent a process from starting the VM while copying; coordinate exclusive ownership of the host during conversion.
+
+    virsh -c qemu:///session shutdown rhel-lab-offline-01
+    virsh -c qemu:///session domstate rhel-lab-offline-01
+
+Only when it reports shut off and an approved exclusive offline window exists, take a copy of its standalone QCOW2 disk:
+
+    SHA="$(git rev-parse HEAD)"
+    uv run --no-project python scripts/testbed.py snapshot \
+      --lab-root "$LAB" --iso "$ISO" \
+      --label baseline-001 --source-sha "$SHA" \
+      --approve I_AUTHORIZE_OFFLINE_SNAPSHOT
+
+The snapshot command:
+1. Re-audits the persistent and current libvirt definitions.
+2. Checks virsh domstate reports exactly "shut off".
+3. Checks for a standalone QCOW2 image, then invokes qemu-img convert (source to new private image).
+4. Refuses existing destination/receipt names and uses a no-clobber hard-link for final publication.
+5. Records SHA-256, exact source commit, byte count and UTC time in a local receipt.
+
+This is an **offline disk copy**, not a libvirt RAM-state snapshot and not a guaranteed atomic capture against hostile/concurrent writers. Never use qemu-img against a running VM disk.
+
+Verify the resulting evidence directory using scripts/evidence.py, storing its manifest outside the directory being sealed. Preserve guest kernel, release, package versions, original media checksum, VM settings, snapshot identifiers, command line, exit status and negative findings in a separate private baseline receipt.
+
+## What this milestone does not do
+
+The repository does not deploy an RHEL guest automatically, bypass subscription conditions, install a hypervisor, configure the host firewall, prove network egress isolation, take a real snapshot in CI, run kernel tests, or claim vendor acceptance of findings. Owner signoff and independently verified isolation are required before RHEL-LAB-003.
+
+## Test and review gates
+
+    uv run --no-project python -m unittest discover -s tests -v
+    uv run --no-project python -m compileall -q scripts tests
+
+The unit tests simulate unsafe profile changes, network injection, path substitution, root/symlink attacks, missing approvals, running guest snapshot refusal and both libvirt XML views. They do not invoke real QEMU/virsh or use a VM.
+
+Read AGENTS.md, docs/RESEARCH_PROTOCOL.md, and docs/DAYBREAK_GATE.md before approving any live operation.

lab/testbed.json

@@ -0,0 +1,9 @@
+{
+  "schema": "qsol-rhel-testbed/v1",
+  "name": "rhel-lab-offline-01",
+  "architecture": "x86_64",
+  "network": "none",
+  "vcpus": 2,
+  "memory_mib": 4096,
+  "disk_gib": 48
+}