Skip to content

Keep developer evidence capturing with automatic retention #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ElliotDrel merged 1 commit into from
Oct 19, 2026
Merged

Keep developer evidence capturing with automatic retention #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions .agents/skills/read-logs/scripts/logs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,9 +63,9 @@ def make_replay_case(entry, log_dir):
manifest = json.loads(path.read_text(encoding="utf-8"))
except FileNotFoundError as exc:
if developer.get("status") != "pending":
raise ValueError("Developer evidence is not saved yet; retry after capture finishes") from exc
raise ValueError("Developer evidence manifest is missing (possibly pruned); use a saved replay case or complete inline inputs") from exc
manifest_pending = True
evidence_warning = "Developer evidence is pending; replay uses validated complete inline inputs only"
evidence_warning = "Developer evidence is pending or pruned; replay uses validated complete inline inputs only"
if manifest is not None and (not isinstance(manifest, dict)
or manifest.get("schema_version") != 1 or manifest.get("run_id") != detail.get("run_id")):
raise ValueError("Developer evidence schema or run identity does not match")
Expand All @@ -82,7 +82,12 @@ def payload(field, required):
if required:
raise ValueError(f"Incomplete developer evidence: {field}")
return None
raw = _contained_path(path.parent, item["path"]).read_bytes()
try:
raw = _contained_path(path.parent, item["path"]).read_bytes()
except FileNotFoundError as exc:
if required:
raise ValueError(f"Missing replay evidence (possibly partially pruned): {field}") from exc
return None
if len(raw) != item.get("bytes") or hashlib.sha256(raw).hexdigest() != item.get("sha256"):
raise ValueError(f"Developer evidence integrity check failed: {field}")
value = raw.decode("utf-8")
Expand All @@ -94,7 +99,7 @@ def payload(field, required):
if (not isinstance(value, str) or detail.get(field + "_truncated")
or detail.get(count_key, _utf16_chars(value) if isinstance(value, str) else -1) != _utf16_chars(value or "")):
if required:
note = "; developer evidence is pending, retry after capture finishes" if manifest_pending else ""
note = "; developer evidence is pending or pruned; retry pending capture or use a saved replay case" if manifest_pending else ""
raise ValueError(f"Missing or truncated replay evidence: {field}{note}")
return None
return value
Expand Down
4 changes: 2 additions & 2 deletions docs/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Overview

The product is the native Rust tray app and dashboard under `rust/`. `main.rs` bootstraps Velopack; `app.rs` owns tray/hotkey lifetime; `runtime.rs` runs corrections; `dashboard.rs` owns the GUI; `update.rs` owns serialized update work; `store.rs` and `platform.rs` retain compatible settings, DPAPI keys and legacy-data migration. Channel constants are generated from `src/BuildChannel.cs`.
The product is the native Rust tray app and dashboard under `rust/`. `main.rs` bootstraps Velopack; `app.rs` owns tray/hotkey lifetime; `runtime.rs` runs corrections; `dashboard.rs` owns the GUI; `update.rs` owns serialized update work; `store.rs` and `platform.rs` retain compatible settings, DPAPI keys and legacy-data migration. Channel constants are generated from `src/BuildChannel.cs`. Optional evidence uses `evidence.rs` for capture and `evidence_storage.rs` for bounded storage and oldest-first retention; both channels share its existing named mutex.

The sections below document the retained C#/.NET 10 reference under `src/`. It bootstraps via Velopack, registers a global hotkey, and runs a clipboard-capture → API-request → post-process → paste pipeline. AHK is archived at `.archive/ahk-legacy/` and is not part of the active system.

Expand Down Expand Up @@ -229,7 +229,7 @@ Triggered by `v*.*.*` tags. Steps: checkout → dotnet publish Release win-x64
- Use one app-lifetime native WinHTTP session/connection with automatic proxy discovery and HTTP/2 enabled. Keep each request's body, headers and read buffer alive through `HANDLE_CLOSING`; enforce a 30-second deadline per attempt, matching two attempts and a 500 ms retry interval.
- Keep shared stamped log timestamps at event time while writing asynchronously; flush through a bounded barrier on orderly shutdown.
- Save the existing named DPAPI key collection with synchronized temporary-file contents and atomic replacement. Preserve corrupt files rather than overwriting them with defaults.
- Keep bounded screenshot and UIA workers off the correction path; preserve the existing cross-channel conservative evidence quota and explicit failure/busy statuses.
- Keep bounded screenshot and UIA workers off the correction path; preserve the existing cross-channel bounded evidence storage with oldest-first Rust retention and explicit failure/busy statuses.
- Queue and deduplicate restart requests during update checks. Use the existing updater's `start --waitPid` wire behavior, then let normal startup own the unified update check/install flow.
- Keep synthetic C# golden cases under `rust/tests/fixtures/`; do not add real runtime logs, copied selections or credentials to source control.

Expand Down
27 changes: 24 additions & 3 deletions docs/replacements-and-logging.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,8 +288,29 @@ UTF-8 sidecars for `input_text`, `clipboard_html`, `clipboard_rtf`, `clipboard_f
`readback_text/html/rtf/formats`. Entries expose relative `path`, SHA-256 `sha256`, UTF-8 `bytes`,
.NET UTF-16 `chars`, `complete` and `status`. Empty source markup is complete only when the format list confirms that flavor is absent.
An offered-but-empty flavor or an unavailable format list is explicitly incomplete, as are null values. Inline 512K markup caps remain, but the sidecars preserve
larger ordinary selections. A 16MiB per-payload cap and 256MiB total evidence store cap explicitly
return `size_limit` / `storage_limit` instead of deleting existing data. Evidence is local and can
larger ordinary selections. The Rust store retains the 16MiB per-payload cap (`size_limit`) and 256MiB total
cap. The former stop-at-full policy is superseded: when a write needs space,
Rust reconciles the quota counter against actual files, then removes oldest
eligible whole-run folders until the pending write fits near 192MiB. This runs
on the optional evidence worker under the existing shared cross-channel mutex;
it adds no clipboard/UI work or correction-thread waits. Ordinary JSONL logs,
settings, keys, and separately exported replay cases are not rotated.

Only 32-hex run folders containing known regular evidence files are eligible.
Current runs, incomplete runs modified within 24 hours, backups/custom files,
nested directories, and reparse points are preserved. Incomplete abandoned
runs become eligible after 24 hours. If protected content or an I/O error
prevents reclamation, evidence reports `storage_limit` or `failed`; corrections
continue. `developer_evidence_pruned` records removed run/byte counts;
`developer_evidence_storage_failed` records storage errors. A locked run is skipped
so other runs can be reclaimed; `developer_evidence_storage_pressure` records
blocked-run counts or insufficient reclaimable capacity. Storage warnings are
limited to one per run. A partly pruned run can retain its manifest with missing
payloads; replay reports required missing data explicitly and omits missing optional
payloads. Optional storage workers wait up to 15 seconds for their shared mutex;
the correction thread never waits for this work. Historical manifest
references may expire; export a replay case before preserving a specific incident.
The retained C# reference keeps its historical stop-at-full policy. Evidence is local and can
contain selected text and surrounding editor content; API key/settings files are never included.

`contexts` links automatic `before-source` and `after-paste` captures with requested/captured/completed
Expand All @@ -310,7 +331,7 @@ best-effort background STA worker with a 500ms deadline and clipboard sequence c
blocks later readback jobs rather than accumulating workers. Absent formats, failures and races are
explicit; a successful read proves the clipboard state at its timestamp, not the editor rendering.
Evidence admission permits at most two in-flight runs. Shared cross-channel quota reservations
are serialized with a named mutex; a failed write may conservatively overcount space.
are serialized with a named mutex; a failed write may conservatively overcount space until the next pressure-triggered reconciliation.

`--developer-evidence-smoke` runs the rebuilt native executable with a disposable RichTextBox,
real Ctrl+C/Ctrl+V, enabled readback and automatic images, disabled readback, live settings cache
Expand Down
119 changes: 71 additions & 48 deletions rust/src/evidence.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,7 @@ use crate::{
store::{Store, wide},
};
use serde_json::{Value, json};
use sha2::{Digest, Sha256};
use std::{
fs,
path::Path,
ptr::{null, null_mut},
sync::{
Expand Down Expand Up @@ -70,7 +68,14 @@ impl Evidence {
let _ = thread::Builder::new()
.name("evidence-save".into())
.spawn(move || {
let result = save_inner(&store, &detail, &self.source, &paste, vec![before, after]);
let result = save_inner(
&store,
&logger,
&detail,
&self.source,
&paste,
vec![before, after],
);
logger.data(
"developer_evidence_completed",
json!({"run_id":detail["run_id"],"reference":result}),
Expand Down Expand Up @@ -284,7 +289,14 @@ fn screenshot(hwnd: usize) -> Result<Vec<u8>, String> {
Ok(png)
}
}
fn quota_write(root: &Path, id: &str, name: &str, bytes: &[u8]) -> Value {
fn quota_write(
root: &Path,
id: &str,
name: &str,
bytes: &[u8],
logger: &Logger,
warned: &mut bool,
) -> Value {
if bytes.len() > 16 * 1024 * 1024 {
return json!({"status":"size_limit","complete":false});
}
Expand All @@ -293,61 +305,50 @@ fn quota_write(root: &Path, id: &str, name: &str, bytes: &[u8]) -> Value {
if mutex.is_null() {
return json!({"status":"storage_busy"});
}
let status = WaitForSingleObject(mutex, 1000);
let status = WaitForSingleObject(mutex, 15_000);
if status != WAIT_OBJECT_0 && status != WAIT_ABANDONED {
CloseHandle(mutex);
return json!({"status":"storage_busy"});
}
let result = (|| -> Result<Value, String> {
fs::create_dir_all(root).map_err(|e| e.to_string())?;
let counter = root.join(".reserved-bytes");
let used = fs::read_to_string(&counter)
.ok()
.and_then(|s| s.trim().parse::<u64>().ok())
.unwrap_or_else(|| size(root));
if used + bytes.len() as u64 > 256 * 1024 * 1024 {
return Ok(json!({"status":"storage_limit","complete":false}));
}
fs::write(counter, (used + bytes.len() as u64).to_string())
.map_err(|e| e.to_string())?;
let directory = root.join(id);
fs::create_dir_all(&directory).map_err(|e| e.to_string())?;
fs::write(directory.join(name), bytes).map_err(|e| e.to_string())?;
Ok(
json!({"path":name,"sha256":format!("{:x}",Sha256::digest(bytes)),"bytes":bytes.len(),"complete":true,"status":"ok"}),
)
})();
let result = crate::evidence_storage::write_locked(
root,
id,
name,
bytes,
crate::evidence_storage::STORE_LIMIT,
);
ReleaseMutex(mutex);
CloseHandle(mutex);
result.unwrap_or_else(|_| json!({"status":"failed","complete":false}))
}
}
fn size(root: &Path) -> u64 {
let mut total = 0;
let mut count = 0;
let mut pending = vec![root.to_path_buf()];
while let Some(dir) = pending.pop() {
for entry in fs::read_dir(dir).into_iter().flatten().flatten() {
count += 1;
if count > 20000 {
return 256 * 1024 * 1024;
}
if let Ok(meta) = entry.metadata() {
if meta.is_dir() {
pending.push(entry.path());
} else {
total += meta.len();
match result {
Ok((value, rotation)) => {
if rotation.bytes > 0 {
logger.data(
"developer_evidence_pruned",
json!({"runs":rotation.runs,"bytes":rotation.bytes}),
);
}
if (rotation.blocked_runs > 0 || value["status"] == "storage_limit") && !*warned {
logger.data("developer_evidence_storage_pressure", json!({"run_id":id,"status":value["status"],"blocked_runs":rotation.blocked_runs,"stored_bytes":value["stored_bytes"],"required_bytes":value["required_bytes"]}));
*warned = true;
}
value
}
if total > 256 * 1024 * 1024 {
return total;
Err(error) => {
if !*warned {
logger.data(
"developer_evidence_storage_failed",
json!({"run_id":id,"error":error.to_string()}),
);
*warned = true;
}
json!({"status":"failed","complete":false})
}
}
}
total
}
fn save_inner(
store: &Store,
logger: &Logger,
detail: &Value,
source: &(usize, u32, String),
paste: &(usize, u32, String),
Expand All @@ -356,6 +357,7 @@ fn save_inner(
let id = detail["run_id"].as_str().unwrap_or("unknown");
let root = store.log_directory.join("developer-evidence");
let mut payloads = serde_json::Map::new();
let mut storage_warned = false;
let mut detail = detail.clone();
if let Some(readback) = detail["clipboard_readback"].as_object().cloned() {
for (key, value) in readback {
Expand Down Expand Up @@ -383,7 +385,14 @@ fn save_inner(
] {
let result = match detail[name].as_str() {
Some(text) => {
let mut v = quota_write(&root, id, &format!("{name}.txt"), text.as_bytes());
let mut v = quota_write(
&root,
id,
&format!("{name}.txt"),
text.as_bytes(),
logger,
&mut storage_warned,
);
v["chars"] = json!(text.encode_utf16().count());
v
}
Expand All @@ -395,7 +404,14 @@ fn save_inner(
for context in contexts {
let image = match context.image {
Some(rx) => match rx.recv_timeout(Duration::from_millis(1500)) {
Ok(Ok(bytes)) => quota_write(&root, id, &format!("{}.png", context.phase), &bytes),
Ok(Ok(bytes)) => quota_write(
&root,
id,
&format!("{}.png", context.phase),
&bytes,
logger,
&mut storage_warned,
),
Ok(Err(e)) => json!({"status":e}),
Err(_) => json!({"status":"timeout"}),
},
Expand All @@ -409,7 +425,14 @@ fn save_inner(
}
let manifest = json!({"schema_version":1,"run_id":id,"channel":channel::CHANNEL,"app_version":channel::VERSION,"created_at":now(),"source_target":{"hwnd":source.0,"pid":source.1,"process":source.2},"paste_target":{"hwnd":paste.0,"pid":paste.1,"process":paste.2},"payloads":payloads,"contexts":captured,"clipboard_readback_boundary":detail["clipboard_readback"]["boundary"],"capture_contract":"Native screen pixels and bounded UIA; no rendering-completion claim"});
let bytes = serde_json::to_vec(&manifest).unwrap_or_default();
let result = quota_write(&root, id, "manifest.json", &bytes);
let result = quota_write(
&root,
id,
"manifest.json",
&bytes,
logger,
&mut storage_warned,
);
json!({"schema_version":1,"manifest_path":format!("developer-evidence/{id}/manifest.json"),"status":result["status"]})
}
// Verify before injecting input: a reader after SendInput can lock out the target's paste.
Expand Down
Loading
Loading