diff --git a/.agents/skills/deslop/SKILL.md b/.agents/skills/deslop/SKILL.md new file mode 100644 index 000000000..474ede13d --- /dev/null +++ b/.agents/skills/deslop/SKILL.md @@ -0,0 +1,132 @@ +--- +name: deslop +description: Code quality review that reports AI slop, small local design issues, and larger structural refactors with a follow-up prompt. Explicit cleanup requests permit local, behavior-preserving fixes. On TypeScript and JavaScript diffs, also checks low-evidence typing, including casts, widening, and mocks that hide what the author already knew. Use when reviewing a diff, cleaning touched code, removing AI slop, or cleaning a diff before a commit. +--- + +# Cleanup-first code quality review + +Check the diff against `main`. If the user names a different scope, use that +scope instead. A review request reports findings without editing files. An explicit +cleanup request, or a commit workflow that asks for cleanup, permits local, +behavior-preserving edits. In that edit mode, fix the little things first, +then do a brief code-quality pass on the touched area. + +When a commit workflow invokes this skill, the intended staged snapshot is the +scope. Stay inside that boundary so the commit can restage without picking up +unrelated work. + +## Coverage + +- Inventory every changed path in the chosen diff. Inspect every changed hunk; do not rely on a representative sample. +- Account for code, tests, documentation, configuration, migrations, generated files, and lockfiles. Use the right review depth for each file type, but do not silently skip any path. +- Recheck the inventory after cleanup. New or materially changed hunks require another pass. +- Tools such as linters, typecheckers, tests, and searches support the review; they do not replace reading the diff. +- On TypeScript or JavaScript hunks, if the repo already runs anti-slop Oxlint rules, use their output on the touched files as extra evidence; otherwise check the same problems by reading the diff. + +## Direct fixes in edit mode + +- Extra comments that are unnecessary or inconsistent with local style +- Stale, misleading, or obvious comments; preserve comments that explain intent, constraints, ownership, or surprising behavior +- Defensive checks or `try`/`catch` blocks that are abnormal for trusted code paths +- `any`, unsafe casts, or loose types used only to bypass real contracts +- Inline imports in Python that should live with the rest of the imports +- Deeply nested code that should be flattened with early returns or a small helper +- Vague, misleading, or needlessly abbreviated names for variables, functions, types, files, and tests +- Local style inconsistencies with the file and surrounding codebase + +## Quality Signals + +### Good patterns to preserve + +- Clear ownership: one module, component, or service has one main job +- Centralized invariants: shared types, schemas, normalization, and validation live in one place +- Simple control flow: direct branches, early returns, readable state transitions +- Type-safe interfaces: contracts are expressed in types instead of dodge-casts +- Derived state over repaired state: effects bridge external systems instead of bookkeeping local state +- Explicit states and transitions: loading, empty, success, failure, cancellation, retry, and cleanup behavior are coherent where relevant +- Scalable boundaries: split rendering, orchestration, shaping, and side effects when a file is clearly mixing them +- Performance-aware reactive code: avoid duplicate state, repeated heavy work, and unnecessary rerender or effect churn +- Useful comments: explain intent or constraints, not obvious code + +### Bad patterns to report, or fix in edit mode when local and behavior-preserving + +- Patch-on-patch growth: more guards, flags, branches, or effects layered onto already confused logic +- Mixed responsibilities: one file doing orchestration, transformation, rendering, persistence, and error handling at once +- Duplicated business logic: repeated normalization, model mapping, status derivation, or fallback behavior +- Repair logic instead of ownership: `useEffect` or watchers keeping state in sync after the fact +- Missing, duplicated, contradictory, or impossible states; stale state that survives retries, cancellation, navigation, or teardown +- Weak typing: `any`, unsafe casts, loose shapes, or ad hoc object contracts +- Cosmetic abstraction: more wrappers or helpers while the underlying flow gets harder to follow +- Hidden performance costs: effect spam, unnecessary recomputation, repeated expensive work, duplicated subscriptions +- AI-slop artifacts: needless comments, over-defensive code, placeholder prose, awkward helper names, and style drift + +### Design-level patterns to catch + +Use these as review signals, not automatic findings. Tie each claim to the changed code and current goal. + +- Over-engineering the solution. +- Overly defensive programming. +- Hyper-fixating on rare or fictitious edge cases. +- Subsystems with overlapping responsibilities. +- Solving a problem at the wrong architectural layer. +- Duplicating sources of truth, then adding machinery to keep them synchronized. +- Prematurely generalizing a one-off flow. +- Adding timeouts everywhere without a concrete need to bound an operation. +- Adding production code that exists only to satisfy tests. +- Patching a bad premise additively instead of stepping back and deleting or replacing it. + +## Type evidence (TypeScript / JavaScript) + +On changed TypeScript and JavaScript hunks, also treat as slop any code that +discards or fabricates type evidence: the author knew the real type and hid +it from the compiler. Prefer inference, `as const`, `satisfies`, named owner +types, and parsing at the boundary. + +In edit mode, fix directly when the edit is local and behavior-preserving. In review mode, report these findings: + +- Assertion chains that fabricate evidence: `as object as User`, widening a known value to `unknown`, `object`, or an index signature and asserting it back, or annotating a known object as `Record` so its keys disappear. +- `unknown`, `object`, or `{}` standing in for a contract the author could have named, on parameters, returns, aliases, or dictionary values. An explicit error-`cause` convention is fine. +- `Reflect.get` or `Reflect.apply` where ordinary typed access works. + +Route through the structural handoff when the real fix is not local: + +- `vi.mock` / `jest.mock`-style module mocks that stand in for a missing dependency seam. +- Ad hoc `typeof` narrowing of unparsed input that should be parsed once at the boundary. + +A truly necessary assertion stays. If it lacks a `SAFETY:` comment (or the repo's +equivalent) stating the invariant TypeScript cannot express, recommend that +comment in review mode or add it immediately above the assertion in edit mode. A placeholder justification, or a SAFETY comment laundering an +assertion that should not exist, is itself slop. Do not weaken, suppress, or +disable lint rules to pass. + +## Behavior + +1. Direct cleanup pass: report obvious slop in review mode; fix it in edit mode. +2. Small quality pass: identify small, clear, behavior-preserving improvements in the touched area. Apply them only in edit mode. Check names, comments, types, control flow, state transitions, ownership, tests, and local consistency where they apply. On TypeScript and JavaScript hunks, also apply Type evidence. +3. Structural triage: if the issue is larger than a local cleanup, do not do the big rewrite by default. + +Before finishing, re-read changed hunks and confirm that every inventoried path +was handled. + +## Placeholders (minor) + +Real gaps are sometimes written as prose (`// later`, `// for now`, `// temporary`, “should eventually…”, empty stubs) **without** `TODO:`. When the comment clearly means deferred work, recommend one line `TODO: ` so intent is grep-friendly and explicit. Apply that replacement only in edit mode. Skip if the code is done; leave existing `TODO:` or `FIXME:` that already read well. + +## Escalate Instead of Rewrite + +- Report the little things first in review mode; fix them first in edit mode. +- If the structural problem is larger than a modest local refactor, stop short of a broad rewrite. +- Append a compact handoff with: +- `Structural issue: ...` +- `Why it matters: ...` +- `Refactor direction: ...` +- `Follow-up prompt: ...` + +## Guardrails + +- Keep cleanup behavior unchanged. Report clear bugs separately; fixing one requires a separately authorized bug-fix scope, not just cleanup or commit authorization. +- Prefer minimal, focused edits and modest local refactors over broad rewrites. +- In edit mode, use behavior-preserving targeted restructuring when the boundary is obvious and it clearly improves the touched area. In review mode, recommend it without edits. +- Do not claim an exhaustive review when any target path or changed hunk was skipped. State the limitation instead. +- Keep the final summary concise (1-3 sentences) when no escalation is needed. +- Do not install dependencies, rewrite lint config, or expand a commit to vendor tooling. Type-evidence cleanup is in-diff only; plugin setup belongs to a later explicit request. diff --git a/.agents/skills/efficient-implementation-plans/SKILL.md b/.agents/skills/efficient-implementation-plans/SKILL.md new file mode 100644 index 000000000..e4e0a5945 --- /dev/null +++ b/.agents/skills/efficient-implementation-plans/SKILL.md @@ -0,0 +1,63 @@ +--- +name: efficient-implementation-plans +description: Write, review, or revise software implementation plans. Use for planning requests or reviews of planning and testing overhead, not ordinary coding. +--- + +# Efficient implementation plans + +Make the requested outcome reliably achievable and done extremely well with proportionate product, planning, and verification effort. Every additional requirement, abstraction, test, or supporting system needs a concrete benefit for the current goal. + +## Establish the outcome against the code + +Read the current request, agreed constraints, existing plan, and relevant code before proposing replacements. For behavior changes, follow the relevant entry point through the components responsible for the result. Identify what already works and what must change. Investigate a named uncertainty instead of surveying the whole repository. + +In the plan, distinguish verified capabilities from proposed mechanisms. For low-impact, reversible choices, a reasonable assumption or small experiment may be more useful than further investigation. A failed experiment can be useful if its effects are contained and its result informs the next decision. For a consequential uncertain capability, state the assumption and a proportionate check or exploratory implementation that can resolve it before dependent work relies on it. For example, a shared database does not establish that an existing email queue can join its transaction. Verify that capability before promising atomic insertion or budgeting the work as a handler-only change. + +**Define an observable finish first.** State the delivered behavior, scope, relevant dependencies, and sufficient acceptance criteria. Separate binding outcomes and safety boundaries from suggested implementation details. Do not turn "robust" or "production-ready" into an unbounded list of hypothetical requirements. Prefer the simplest design that fully meets the current goal. Discuss the outcome with the user when it is still being shaped or a consequential choice is unsettled, so you are on the same page. Use agreement already established instead of asking for it again. + +Match the requested action: discussion, review, drafting, or editing. Interpret follow-up questions in light of authorization already given. Discussion and review alone do not authorize file changes, implementation, publication, or changes to project rules. Follow existing project contracts; this skill does not authorize silently dropping agreed gates. Propose changes to binding outcomes, scope, or safety requirements for the user's decision. Make routine implementation and verification choices autonomously within the authorized scope. Keep independent work moving while a necessary answer is pending. + +## Make ownership and dependencies concrete + +Name the contracts that matter: inputs and outputs, state that must remain true, and meaningful failure behavior. At an authority boundary, one component owns a decision or state change. State which component owns it and what callers may assume. For example, a client requests approval; the server verifies the current revision and permission before the worker acts. A hidden button alone does not enforce that contract. + +Prefer a small complete behavior through the layers it touches. Each implementation slice needs an outcome, real prerequisites, and sufficient proof. Put the uncertain integration early enough to change the design before surrounding work accumulates. + +System boundaries and work assignments are different. A feature may cross several component owners and still belong in one reviewable PR. Split it when a contract, independent outcome, migration, or permission boundary makes a separate deliverable useful. Scale migration and release safeguards to actual users, retained data, and dependent systems. Where continuity is required, plan usable intermediate states. Otherwise, a direct replacement may be appropriate without compatibility layers or fallback infrastructure. Avoid dividing every feature into disconnected database, API, and UI tasks. + +**Justify expensive tests and new infrastructure. Choose infrastructure for the work it enables.** Reuse existing tools when they fit the goal and simplify the work. Build or replace infrastructure when it better serves the goal. Compare setup and maintenance effort with the benefit to development, verification, or deployment at the project's current stage. A small deployment workflow or repeatable test scene may be the simplest way to get reliable feedback. Make substantial supporting work visible in the plan; do not hide a new testing system under "add tests." + +For multiple PRs, state the base or prerequisite of each and the behavior its own diff adds. For parallel work, the plan names the independent scopes and integration responsibility. Follow the project's existing isolation and dispatch rules; a separate chat is not file isolation. + +## Choose the cheapest sufficient proof + +Development checks, targeted boundary checks, and coherent end-to-end acceptance are available layers, not three mandatory steps for every task. Judge risk by the actual effects of a change; a one-line permission change can affect many actions. Match each consequential acceptance claim to a check that could disprove it. Prefer a focused regression for bug fixes; a costly red baseline for a new feature requires a concrete purpose or explicit contract. Test a strict local invariant directly when it has a precise answer. + +When the claimed outcome is a user journey, specify its normal entry point, destination, and observable result. An internal function call or prepared fixture proves only the boundary exercised. If credentials, hardware, or deployment access are missing, preserve that acceptance gap and continue the work that can be proved locally. + +**Match verification effort to risk.** Ask what plausible incorrect implementation a costly test would catch that cheaper checks would miss. Choose variants for distinct risks and interactions, not an automatic Cartesian product of platforms, providers, and scenarios. Once the relevant checks pass, broaden or repeat them only for new changes, failures, or unresolved concerns. + +**Keep the first working version focused.** For an early prototype, prioritize getting the smallest complete behavior working and evaluating it. Newly introduced hashing, evidence ledgers, or release machinery are a signal to revisit the current goal and implementation stage. In a small prototype they usually mean the steering is off. Leave them out unless a concrete requirement of that stage needs them. Use existing Git and test records for ordinary implementation work. Preserve safeguards for actual effects, and do not invent future risks to justify additional machinery. + +For costly verification, put these conditions in the plan where the implementer needs them: + +- **Rerun according to impact.** A result remains evidence for its tested artifact; assess its applicability to the delivered state using relevant source, dependencies, packaging, build, configuration, and environment changes. A reporter or selector change preserves unrelated evidence only if entry, fixtures, assertions, and success detection remain valid. Renew affected proofs and explain reuse briefly. Investigate uncertain impact and broaden verification where needed. A new revision alone does not require every previous live run. +- **Diagnose before expensive repetition.** Use the failure evidence to choose the next investigation. For example, you can classify failures as product, harness, infrastructure/external service, or access problems at the first failure. Narrow the reproduction when doing so preserves the relevant conditions; keep or broaden the scope when interactions matter. Another expensive run needs to answer a specific question, test a relevant correction, or follow a changed prerequisite. A likely transient failure may justify a bounded retry. Fix the real cause rather than stacking superficial patches, and verify the originally failing behavior -- and oftentimes it is at the smallest reproducible boundary. +- Verify the delivered state collectively. A selector fix cannot make an unreached assertion retrospectively pass. A successful retry establishes that run's outcome; retain original failures and retries when measuring reliability. + +Use unexpected verification effort to reconsider the method, especially when further investigation is unlikely to change the next decision. Improve the approach autonomously within the agreed scope and communicate material deviations. Preserve agreed outcomes and safety requirements. + +## Maintain one current plan + +When plan edits are authorized, update the existing canonical plan in place. Replace superseded decisions and remove obsolete steps when requirements change. Preserve current constraints and the reason behind a non-obvious choice. Carry an old open question forward only if its answer affects the current outcome or next action. Keep execution details in existing task state, PRs, or test reports, with only the links needed to resume or assess the work. Do not prepend updates, preserve old plan versions inside the document, or move all removed prose into a new archive by default. + +For a substantial task, a useful shape is: + +- Goal and scope: the observable finish and the exclusions that prevent likely drift. +- Design: existing components, necessary changes, and consequential contracts or ownership. +- Implementation slices: outcomes in dependency order, with integration and PR boundaries where needed. +- Acceptance and open decisions: sufficient checks, material limitations, and prerequisites still needed. + +Use only the sections the task needs. A small change can be a paragraph. Include verified source locations when they help the implementer start; avoid speculative file inventories and pseudocode that merely restates prose. Keep later stages at outcome level until their dependencies are understood. A plan's length follows its decisions, not a prescribed template or line count. + +**Make reviews subtractive and finishable.** Resolve actionable ambiguities, contradictions, and relevant coverage gaps; remove redundant obligations and obsolete assumptions. A new mandatory requirement needs a current goal or concrete risk. Finish planning when the implementer can identify the next complete outcome, its prerequisites, and sufficient proof. Reopen settled decisions only with new information. Do not launch repeated whole-plan audits or additional reviewers without a specific unresolved question. diff --git a/.agents/skills/pr-writing/SKILL.md b/.agents/skills/pr-writing/SKILL.md new file mode 100644 index 000000000..222db5b2e --- /dev/null +++ b/.agents/skills/pr-writing/SKILL.md @@ -0,0 +1,32 @@ +--- +name: pr-writing +description: Writes or revises pull request titles, descriptions and visual evidence. Use when preparing a PR, improving how it reads, or updating it after its scope changes. Writing the text does not publish the PR. +--- + +# Write a reviewable PR + +Before drafting, read the repository's template and contributor guide, the diff against the PR's real base, and the current PR body. Apply `unslop` and `technical-writing` to everything you write. Keep what others added: human edits, evidence links and sections a bot manages. Fetch the body again right before you publish, so you don't overwrite someone else's change. + +## Explain the change + +Open with the problem and what the code does now. Name the screen, operation or interface it touches, so someone who never saw the task can follow. For a large change, say what used to happen, what that cost, and how the change fixes it. A new feature needs no broken predecessor. Describe what it adds. + +Order independent changes by how much they matter to users or to the rest of the code. Each heading names a change. Filenames and commit order are not an outline. Explain the mechanism when a reviewer needs it to judge correctness or a trade-off. Link the relevant part of the diff, or the file at the reviewed commit when no diff link exists. Cite earlier research only when a choice rests on it, with the source and why it matters. + +Write whole sentences in which someone does something. Leave out clipped status phrases, internal names nobody explained, claims that the work was thorough, and narration about the reviewer or your process. Keep technical detail a reviewer can use, and cut any sentence that changes no decision. Read the short [writing examples](references/writing-examples.md) before drafting. Match how much they explain, not how their sentences are built or how long they are. + +Size the body to the change. A small fix gets a short explanation and the one check that proves it. A broad PR can use one section per behavior, with evidence beside each claim. Put long logs, inventories and test matrices in linked files. Never paste a task transcript, a subagent report or a commit-by-commit diary into the description. + +## Show evidence + +If the change is visible, read [visual evidence](references/visual-evidence.md) and include matched before-and-after captures from the running app. Add a short recording when the change is motion, timing or interaction. Show only the behavior that changed. A new screen gets an after capture alone, labeled as new. If evidence is missing, say so in the body. + +Report the checks you ran and what they returned, with the command or a link to the run. Put each limit next to the claim it weakens: fixture data, simulator-only behavior, a check that failed or was skipped, a service nobody tested. Keep steps to reproduce apart from checks you finished. Green CI says nothing about how the screen looks or whether the product owner will accept it. + +## Use the repository template + +`.github/pull_request_template.md` has sections for What changed, Why, UI changes, Verification and Risk, then a four-item checklist. Keep every section the repository requires. Delete an optional section that would be empty, and don't explain the same thing under two headings. Where a checklist item doesn't apply, replace it with `N/A` and the reason. Tick only work you finished. A large change that belongs together is fine. Don't call it small to tick a box. + +Write the title about the final change, in the repository's title format. In stacked work, name the base branch and any PRs this one needs, and describe only what this PR adds. When the scope changes, update the title, the body and the affected evidence in the same edit. Keep captures that still hold, labeled with the commit they show, so nobody mistakes them for the newer build. + +Publish with a body file or a structured tool argument. Then open the saved PR and check that the text, links, side-by-side images and videos render. A local draft or a successful upload doesn't prove the published body looks right. Opening the PR, pushing and answering reviews are separate steps, and each needs the user's go-ahead. diff --git a/.agents/skills/pr-writing/references/visual-evidence.md b/.agents/skills/pr-writing/references/visual-evidence.md new file mode 100644 index 000000000..e2597ec60 --- /dev/null +++ b/.agents/skills/pr-writing/references/visual-evidence.md @@ -0,0 +1,27 @@ +# Visual evidence + +Decide which behavior changed before you capture anything. Capture the real route or component with the browser, simulator or desktop tools the host supports. Never redraw an interface as a stand-in for a screenshot. A design prototype shows the prototype, not the native app or the deployed product. + +## Capture a fair comparison + +Note the commit for the before and the after. Keep the device or viewport, route, data and state the same in both. For a bug, trigger the same condition on both versions. For a new feature, show the new flow and label it as new. If the old build won't run, label the missing or historical reference. Never fake a baseline. + +Capture the base in a separate worktree or from an existing build, so you don't disturb whoever is working in the main checkout. Use approved test data. Say when fixture data limits a claim. A success screen doesn't prove the backend saved anything. If the capture procedure will be reused, commit it to the repository with the route, fixture and build commands that worked. + +Put Before and After side by side at the same readable size. Label each and caption what changed in behavior. Crop both the same way, and keep enough around the change to judge it. Avoid tall comparison boards with blank gaps, and galleries of screens that didn't change. Keep the original captures when you build a comparison. + +Record motion at normal speed, showing the trigger, the transition and the settled result. Trim idle time, but never cut a failure or change the timing under review. Use H.264 MP4 when the host plays it. Before uploading, watch the encoded file for dropped frames, unreadable text and wrong orientation. + +## Publish and verify + +Attach inline evidence through GitHub's repository attachments, especially in a private repository. Keep sensitive captures off anonymous public image hosts. A private raw-file URL or a local path won't render inline. + +Use the GitHub CLI's attachment support if the installed version has it. Check `gh pr create --help` for `--attach` first. Otherwise use GitHub's attachment UI while signed in, or an uploader the repository already trusts. Don't carry one app's authentication workaround into another project. If a local tool composes the before-and-after image, check where it writes its output before you run it, so it doesn't upload captures somewhere else. + +With CLI attachment support, reference the local files in the Markdown body and pass each file with `--attach` next to `--body-file`. The CLI swaps those references for uploaded URLs. If an upload fails partway, look at the returned PR and its body before retrying. The CLI can exit nonzero after it has already created the PR. Fix that PR instead of opening another. + +Show a pair in a small Markdown table. Give each image alt text that says what it shows. Put a video in its own paragraph so GitHub renders it as a player. Keep the original attachment URLs when you edit an existing body. + +Open the PR's rendered page. Check that both images load at a useful size, each label matches its commit, and the video plays. If you can't publish or view the PR, hand back the draft and the evidence paths, and name the step still left. + +GitHub documents [attachment access and supported formats](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/attaching-files) and [CLI attachment support](https://docs.github.com/en/github-cli/github-cli/attaching-files-with-github-cli). diff --git a/.agents/skills/pr-writing/references/writing-examples.md b/.agents/skills/pr-writing/references/writing-examples.md new file mode 100644 index 000000000..4d0feb443 --- /dev/null +++ b/.agents/skills/pr-writing/references/writing-examples.md @@ -0,0 +1,31 @@ +# Writing examples + +These examples are made up. They show how much to explain, and none of their facts belong in a real PR. + +## Explain the failure and the mechanism + +Weak: "Improved session handling. The stale-response guard is robust and all tests pass." + +Useful: "Switching accounts while the inbox loaded could show the previous account's messages. The response now carries the account that requested it, and the inbox drops it if that account is no longer active. The regression test starts both loads and finishes the older request last." + +The useful version names the trigger, what went wrong and how the fix works. A test count alone wouldn't show that the test covers the race. + +## Keep a small change short + +Title: "fix(export): keep zero values in CSV downloads" + +"CSV downloads left zero-valued cells empty, because the formatter treated every falsy value as missing. It now leaves a cell empty only for null or undefined. `npm run test:csv` passes, including zero, null and undefined cases." + +This change needs no architecture story and no screenshot. Fill in the sections the repository requires, and stop there. + +## Say what a comparison shows + +"A declined payment used to replace the form and throw away the address the customer typed. The error now appears above the submit button, and the address stays filled in for a retry." + +Put the matched images right below that paragraph. A caption such as "Same declined-payment fixture at 390 × 844. Before a12bc34, after d56ef78." tells the reviewer the comparison is fair. If no live charge ran, say so once, next to the verification. The images don't prove that payments work end to end, so don't present them that way. + +## Keep a limit the reviewer needs + +"The migration ran on a copy of the staging schema. Nobody has measured it against production data volume or timed how long it holds locks." + +That hands the reviewer a concrete open risk. "Low risk, thoroughly tested" hides it, and a long history of approaches that failed would bury it. diff --git a/.agents/skills/prompt-writing/SKILL.md b/.agents/skills/prompt-writing/SKILL.md new file mode 100644 index 000000000..5cb82b4bc --- /dev/null +++ b/.agents/skills/prompt-writing/SKILL.md @@ -0,0 +1,47 @@ +--- +name: prompt-writing +description: Writes and reviews instructions for AI agents, such as AGENTS.md lines, SKILL.md files and prompts that hand work to another agent or subagent. Use when writing or editing AGENTS.md, CLAUDE.md or a skill, or when briefing another agent. +--- + +# Prompt writing + +The reader is a capable model that can read the repository. Write only what it can't work out from the code, and say each thing once. + +## Give the outcome, the reason and the finish + +- State the outcome, the scope and what done looks like. An agent with a clear done condition doesn't stop early. +- Give the reason behind each constraint. A model that knows why a rule exists applies it to cases the rule didn't name. +- Point at the exact path, symbol, command or README section. An agent follows "read `server/README.md`, section 'What settles each check'" and skims past "see the server docs". +- Say what to do rather than what to avoid. +- Hand over what the agent can't find in the repository: decisions already made and why, what was tried and failed, who owns which files, and what the user prefers. + +## Keep instructions calm and consistent + +- Write in a normal voice. Current Claude models follow instructions closely, and capitals, "CRITICAL" or "if in doubt, always" make them apply a rule too widely. +- Put each instruction in one place. When a skill, AGENTS.md and the prompt disagree, a model can stop and wait for help. Where instructions can conflict, say which wins. The user's instructions outrank a skill. +- Write the prompt in the form you want back. A prompt in prose tends to get prose back, and one built from headings and bullets tends to get markdown. +- Call each thing by one name throughout. + +## Keep context small + +- Every line of AGENTS.md or CLAUDE.md loads into every session. If only one task in ten needs some material, move it to its own file and leave a pointer. +- Link every reference file directly from the entry file. An agent may read only part of a file it reached through another reference. +- Leave out facts that expire. If a fact must stay, point to the file that holds its current value. + +## Prune by behavior + +- A line does nothing if the agent acts the same without it. Delete the whole sentence, not a few of its words. +- Don't ask an agent to "shorten" or "streamline" a prompt. It will cut for length and drop instructions that mattered. +- Test the prompt by running it. Give a fresh agent a real task with only this prompt, watch where it goes wrong, and fix that failure rather than one you imagine. + +## Write a skill + +- Write the description in the third person. Say what the skill does and when to use it, in the words a user would type. The description is all the agent sees before it picks the skill. +- Keep SKILL.md under 500 lines. Move detail that only one branch of the task needs into a file that SKILL.md links directly. + +## Sources + +- Anthropic, [Prompting best practices](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices): give context and motivation, say what to do instead of what not to do, match prompt style to output, and tone down aggressive language on current models. +- Anthropic, [Skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices): assume the model is capable, write third-person descriptions, keep references one level deep, leave out time-sensitive facts, and test with real tasks. +- OpenAI, [Prompting the latest model](https://developers.openai.com/api/docs/guides/latest-model): conflicting skill guidance can block work early, and the user's instructions should outrank a skill. +- Matt Pocock, [Writing skills for agents](https://aihero.dev/skills-writing-for-agents): pointer wording, done conditions, the no-op test, and testing by running the document. diff --git a/.agents/skills/technical-writing/SKILL.md b/.agents/skills/technical-writing/SKILL.md new file mode 100644 index 000000000..cd08a9e76 --- /dev/null +++ b/.agents/skills/technical-writing/SKILL.md @@ -0,0 +1,137 @@ +--- +name: technical-writing +description: "Checks and writes technical prose against four standards: Diátaxis for document structure, Google developer style for sentences, STE for instructions, and Global English for syntax. Use for /technical-writing or when writing or reviewing docs, RFCs, READMEs, PR descriptions, or commit messages." +--- + +# Technical writing + +Write so a tired engineer understands it on the first read. Four layers get you there, and each answers one question. What kind of document is this? How do sentences address the reader? How much does each sentence carry? Can any sentence be read two ways? Apply all four. + +Three rules sit above the layers: + +- **Cut every word that does no work.** If the sentence survives without a word, the word goes. "In order to" is "to". "It is important to note that" is nothing. +- **Use the short, everyday word.** "Use", not "utilize". "Help", not "facilitate". "Do", not "perform". A long word has to earn its length with precision. +- **When a rule makes a sentence worse, fix the sentence another way or leave it alone.** The rules serve the reader. A sentence that follows every rule and still sounds machine-written has failed. + +The codebase is the word list. Write the real symbol, file, flag, or command name, not a synonym or a description of it. + +Don't invent jargon. Use the words a developer says out loud, such as "move" and "a budget that only decreases" instead of "evacuate" and "ratchet". A named pattern is fine when the doc defines it the first time it appears. To add a word to `unslop`'s abstract-metaphor rule, propose it with its replacement in a pull request. + +## Vary the rhythm + +The layers decide what a document says and how much each sentence carries. A doc can obey all of them and still read as machine-written, with every sentence clipped short, no view anywhere, and nothing specific. + +- Mix sentence lengths on purpose. A short sentence lands a point. A longer one can carry a fact together with its condition or its consequence. +- One thought per sentence does not mean one length per sentence. Split a sentence that carries two thoughts, and keep a long sentence that carries one. +- Take a position where the mode allows it. Explanation weighs trade-offs, so say which one you would pick and why instead of listing pros and cons. Reference stays dry. +- Be specific. "Schema changes can cause issues" tells the reader nothing. "A column rename fails the build" tells them what to watch for. + +## Pick the mode first (Diátaxis) + +One document, one mode. Two questions pick it. Does the content help the reader act or understand? Does it serve someone learning or someone working? + +- Act while learning: **tutorial**. +- Act while working: **how-to**. +- Understand while working: **reference**. +- Understand while learning: **explanation**. + +Use these two questions on a whole document or on one sentence, whenever you are unsure what you are writing. Gut feel often picks the wrong mode. + +**A tutorial teaches by doing.** You are the teacher, and the learner's success is your job. Open with what the learner will build, not what they will "learn". Every step produces a visible result, early and often. Tell them what they should see, such as the expected output, the prompt change, or the log line. Keep explanation to one clause and a link, because a teaching pause breaks the lesson. Stay concrete, and write as "we", in commands: "First, do x. Now, do y." + +**A how-to gets a competent reader to a goal.** It solves a problem a person has, not an operation the machine can perform. Skip teaching and background, and link to them instead. Allow forks and judgment: "If you want x, do y." Name the guide by the task, as in "How to calibrate the radar array", not "Radar array calibration". + +**Reference holds facts for lookup.** It describes and does nothing else, with no instructions and no opinions. State facts, options, limits, and errors plainly, without hedging, and leave nothing out. Mirror the structure of the thing described, so a reader can move between the code and the docs. Put material where readers expect it. Generate reference from code where you can, so it stays true. + +**Explanation builds understanding of why.** It covers one bounded topic and makes sense away from the product. Each title should still read well with "About" in front of it. Anchor on a real why question, and give the context behind it: design decisions, history, constraints, and alternatives. Explanation is the only mode that allows opinion. + +Keep modes apart. A tutorial has no reference tables, reference has no hand-holding, and a how-to has no argument. Split the content into separate documents and link them. + +Source: diataxis.fr, fetched 2026-07-18. + +## Write sentences to the reader (Google developer style) + +- Address the reader as "you", in the present tense. Use "will" only for things that happen later. +- Say who does what: "the compiler checks", not "is checked". Passive is fine only when the actor is unknown or beside the point. +- Write instructions as commands: "Click Submit." State facts plainly. Never write "should be done". +- Put the condition before the instruction: "To delete the document, click Delete." The reader then skips what doesn't apply. +- Put the common case first and exceptions after it. +- Sound like a friend who knows the system. Leave out buzzwords, figures of speech, and "please" in instructions. Never write "simply", "easy", or "quickly" in a procedure. If it were simple, the reader would not be here. +- Don't announce future plans ("we will soon support..."), and don't start consecutive sentences with the same phrase. +- Read an awkward sentence aloud. If it stays awkward, rewrite it. +- Link text says where the link goes, using the page title or a short description. Never write "click here". A sentence of context on the page beats a link off it. +- A heading states the point, not just the topic: "Pick the mode first", not "Modes". Use sentence case. A task heading is a bare verb phrase ("Create an instance"). A concept heading is a noun phrase. Use one h1 per page and skip no levels. +- Use numbered lists for sequences and bullets for everything else. Introduce a list with a complete sentence, and keep its items parallel. +- Put code in code font and UI elements in bold. Use serial commas. Drop "etc.", and say up front when a list is partial. + +Source: developers.google.com/style, fetched 2026-07-18. + +## Make statements load one at a time (STE rules) + +- Give one instruction per sentence, and one thought per sentence everywhere else. +- Split an instruction longer than about 20 words, and any other sentence longer than about 25. +- Put the warning or condition before the step it guards: "If hot oil touches your skin, injuries can occur." +- Keep "the" and "a". "Remove backup file" reads two ways. "Remove the backup file" reads one way. +- Give each word one meaning, and keep it. If "check" means inspect, don't also use it to mean restrain. +- Pick one word per action and stick to it. Write "start" every time, not "start" here and "initiate" there. +- Write procedures as direct commands, never as narration and never in the passive: "Install the component", not "the component must be installed". +- Avoid "-ing" words where you can. They can do several grammatical jobs, so they invite misreadings. + +Source: asd-ste100.org (Issue 9, 2025), fetched 2026-07-18. The numbered rules and the dictionary are in the spec PDF. The principles above are the parts that carry over to software docs. + +## Leave no sentence open to two readings (Global English) + +- Keep words like "only" and "not" next to the word they change. "Only fails on growth" and "fails only on growth" say different things. +- Break up long noun strings. "The proto import budget check script" becomes "the script that checks the proto-import budget". +- Make every "it", "they", and "this" point at one obvious thing. Repeat the noun when in doubt. Never use "this" or "which" to point at a whole clause. +- Don't drop verbs. "Phase 1 moves the converters and Phase 2 the runtime" leaves Phase 2 without one, so give it one. +- Keep the small words that show structure. "Ensure that the switch is off" keeps "that" because it makes the sentence parse one way. Never trade clarity for word count. +- Repeat the article in a series when two things could read as one: "the client and the host", not "the client and host". +- When a sentence can group two ways, show which parts "and" or "or" joins. "Both...and", "either...or", and "if...then" cost nothing and settle it. +- Use periods, not semicolons. Replace an em dash with a new sentence. +- Text in parentheses must be a full grammatical unit, or its own sentence. Never form plurals with "(s)". +- Don't use slashes. Write "a, b, or both" instead of "a/b" or "and/or". +- Call each thing by one name, everywhere. A doc that calls one thing "the gate", "the ratchet", and "the budget check" teaches three things. Rewording a sentence that didn't change costs readers the same way, so leave unchanged sentences alone. +- Skip idioms, slang, Latin abbreviations, and metaphors. A non-native reader, a translator, and an agent all parse plain sentences best. + +Source: Kohl, The Global English Style Guide (SAS Press). Guideline text fetched from the Internet Archive and the SAS sample chapter, 2026-07-18. + +## Apply unslop and the repository rules + +- Apply the `unslop` skill to every doc this skill touches. `unslop` owns the catalog of slop patterns, such as AI vocabulary, filler, hedging, and formatting tells. +- PR descriptions and commit messages are writing too. Every layer except Diátaxis applies to them. +- Product UI strings are not documentation. Follow the product's copy guidelines for those. +- Write real paths and real symbols. Make every count or file-tree claim true at the commit that lands it, and include the command that regenerates it. + +## Worked example + +Before: + +> Configuration of the proto import ratchet budget script parameters is performed via budget.json. Note that it's important to remember that running with --write, which updates the committed budget to reflect the current count, should only be done when lowering it. If exceeded, CI fails. + +After: + +> `budget.mjs` reads the committed budget from `budget.json` and counts the files that import protos. If the count exceeds the budget, CI fails. Run `budget.mjs --write` only to lower the budget. + +Each fix comes from a layer: + +- "Configuration is performed" becomes "`budget.mjs` reads", so someone does something (Google). +- "Ratchet" goes, and the script's real filename names it instead (the jargon rule). +- The five-noun string breaks up into plain clauses (Global English). +- "Note that it's important to remember" is deleted (cut every word that does no work). +- The failure condition moves ahead of the step it explains (STE). +- "Should only be done when lowering" becomes a command with "only" next to what it limits (STE). +- "If exceeded" gets a subject, the count (Global English). + +## Review checklist + +Run these checks on any prose this skill covers. Item 1 applies only to a set of documents. + +1. Is each file one Diátaxis mode, with links where modes meet? +2. Is every instruction a command, with its condition in front? +3. Does any sentence carry two instructions or two thoughts? Split it. +4. Can any word go without losing meaning? Cut it. +5. Is "only" next to the word it changes? Does every "it" point at one thing? Does every clause keep its verb? +6. Does each thing have exactly one name across the docs? +7. Would a developer say these words out loud? Replace invented metaphors and fancy synonyms with the plain word or the real symbol name. +8. Are all symbols, paths, and counts real at this commit, with the commands that regenerate the counts? diff --git a/.agents/skills/unslop/SKILL.md b/.agents/skills/unslop/SKILL.md new file mode 100644 index 000000000..dc2fba687 --- /dev/null +++ b/.agents/skills/unslop/SKILL.md @@ -0,0 +1,105 @@ +--- +name: unslop +description: Removes AI writing tells and adds a human voice. Use when writing or editing any prose, including docs, READMEs, PR descriptions, commit messages, code comments and reports. +--- + +# Unslop + +Edit text to remove AI patterns and add human voice. + +## Judge by density + +Almost nothing on this list is wrong once. One em dash is punctuation. Five in a paragraph is a fingerprint. One or two instances may be coincidental. A passage introducing many of them is among the strongest signals available. + +Where to start looking: + +- Three or more vocabulary-list words inside 100 words. Rewrite the passage, not the words. Swapping synonyms leaves the sentence shape that produced them. +- Two forced triplets in a row, or two "not just X, but Y" constructions in a row. +- Em dashes past about one a paragraph, where the dash is doing structural work a sentence break should do. + +These numbers are judgment, not measurement. No calibration study sits behind them. Adjust once you have read enough of the document to know its own baseline. Em dashes are the deliberate exception. This project sets them at zero, because this tell is cheap to remove and expensive to adjudicate one at a time. + +## Process + +1. Scan for the patterns below. +2. Rewrite. Preserve meaning, match intended tone. +3. Add soul (see next section). +4. Self-audit: "What makes this obviously AI generated?" Fix remaining tells. + +## Adding soul + +Removing patterns is half the job. Sterile, voiceless writing is just as obvious. + +- **Have opinions.** React to facts instead of neutrally listing pros and cons. +- **Vary rhythm.** Short sentences. Then longer ones that take their time. Mix it up. +- **Acknowledge complexity.** "Impressive but also kind of unsettling" beats "impressive." +- **Use "I" when it fits.** First person isn't unprofessional. +- **Let some mess in.** Perfect structure looks machine-made. +- **Be specific.** Not "this is concerning" but "there's something unsettling about agents churning away at 3am." + +## Patterns to detect and fix + +### Content + +1. **Puffery.** "pivotal moment", "testament to", "evolving landscape", "setting the stage for", "indelible mark", "deeply rooted". Cut puffery, state what happened. +2. **Name-dropping.** Listing media outlets without context. Pick one, say what was said. +3. **Superficial -ing phrases.** "highlighting...", "ensuring...", "reflecting...", "showcasing...", "fostering...". Delete or expand with real sources. +4. **Promotional language.** "nestled", "vibrant", "breathtaking", "groundbreaking", "renowned", "stunning", "must-visit". Use neutral descriptions. +5. **Vague attributions.** "Experts believe", "Industry reports suggest", "Some critics argue". Name the source or delete. +6. **Formulaic challenges.** "Despite challenges... continues to thrive." Replace with specific facts. + +### Language + +7. **AI vocabulary.** Additionally, crucial, delve, enduring, enhance, fostering, garner, interplay, intricate, landscape (abstract), pivotal, showcase, tapestry (abstract), testament, underscore, vibrant. Replace with plain words. +8. **Fancy ways to say "is".** "serves as", "stands as", "boasts", "features". Just say "is" or "has". +9. **"Not just X, but Y."** State the point directly instead. +10. **Rule of three.** Forcing ideas into groups of three. Use the natural number. +11. **Synonym cycling.** Protagonist, main character, central figure, hero all in one paragraph. Pick one, repeat it. +12. **False ranges.** "from X to Y" where X and Y aren't on a meaningful scale. List topics directly. + +### Style + +13. **Em dash overuse.** Avoid em dashes entirely. Use periods or commas only (no parentheses, no en dashes, no hyphen-as-dash substitutes). Em dashes are an AI tell, and reaching for parentheses instead just trades one tell for another. If a thought needs separation, end the sentence or use a comma. +14. **Colon overuse.** Colons are fine before a list or example. Not as mid-sentence connectors. "If you're coming from traditional automation: instead of registering event handlers, you describe conditions" adds nothing with the colon. Rewrite to let the point stand on its own without comparison framing. "Describing when the scheduler should fire works best as plain English." Same meaning, no crutch punctuation. +15. **Boldface overuse.** Don't bold every proper noun or acronym. +16. **Inline-header lists.** The tell is a bold label and colon that restates the line: "**Performance:** Performance improved...". Convert those to prose. A bold lead-in that ends in a period, names the item, and is followed by genuinely new detail ("**Schema in TypeScript.** Tables live in one file.") is fine, not a tell. +17. **Title case headings.** Use sentence case. +18. **Decorative emojis.** Remove from headings and bullets. +19. **Curly quotes.** Replace with straight quotes. + +### Communication artifacts + +20. **Chatbot phrases.** "I hope this helps!", "Let me know if...", "Of course!", "Certainly!", "Found the smoking gun!" Remove. +21. **Cutoff disclaimers.** "While specific details are limited..." Find sources or remove. +22. **Sycophantic tone.** "Great question! You're absolutely right!" Respond directly. + +### Filler + +23. **Filler phrases.** "In order to" becomes "To". "Due to the fact that" becomes "Because". "It is important to note that" gets deleted. +24. **Excessive hedging.** "could potentially possibly be argued that it might" becomes "may". +25. **Generic conclusions.** "The future looks bright." State specific plans or facts. + +### Jargon + +26. **Abstract metaphor nouns.** Substrate, wedge, vector, locus, vantage, nexus, primitive (as noun), harness (as metaphor), surface (as in "API surface"), bedrock, scaffolding (as metaphor), modality, paradigm, gold-plating, ratchet (as metaphor), evacuate (for moving code), endgame, north star, flywheel. These read as technical but usually have a plainer concrete word. "Substrate" becomes "base". "Wedge in" becomes "add". "Vector" becomes "way" or "method". "Gold-plating" becomes "more than the job needs". "Ratchet" becomes the mechanism's real name or "a limit that only tightens". "Evacuate" becomes "move out". "Endgame" becomes "the last phase". Pick the concrete word. + +### Plain speech + +27. **Say what it does, not how it feels.** "the database stays close at hand", "SQL you can read", "types that follow your schema" name a feeling. The fix names the mechanism or a number: "`.toSQL()` returns the exact string sent to the database", "a column rename fails the build". Ask what the sentence tells the reader to do or know, then write that. If you can't restate it as a concrete instruction, fact, or number, cut it. One more check: if the sentence could appear unchanged in another project's docs, it says nothing about this one. Cut it. +28. **Shorten or split dense sentences.** If the reader has to backtrack to parse a sentence, break it in two or drop clauses. One idea per sentence. +29. **Active voice.** Prefer it. Catch "is/are/was/were + past participle" and name the actor: "queries are validated" becomes "the compiler validates queries", "the file is parsed by the loader" becomes "the loader parses the file". Passive is fine only when the actor is unknown or genuinely doesn't matter. +30. **Cut adverbs, or use a stronger verb.** "runs quickly" becomes "is fast" or the number. "significantly improves" becomes the measured delta. An adverb propping up a weak verb means the verb is wrong. +31. **Prefer the plain word.** "utilize" becomes "use", "leverage" becomes "use", "facilitate" becomes "help", "numerous" becomes "many", "in the event that" becomes "if". The fancier synonym is rarely clearer. + +## Not evidence + +Do not flag these. Acting on them produces false positives and, worse, a writer who starts avoiding ordinary English. + +- One instance of anything, except a literal machine artifact such as a leftover placeholder or a truncation notice. +- Correct, well-placed punctuation, including a dash a writer clearly chose. +- Any vocabulary-list word in its literal sense. A landscape can be a landscape. +- Clean structure. Organization is not a tell. Evenly weighted structure matching no real shape in the material is. +- Formal register. Some documents are formal. +- Anything written before late 2022. + +Human writing has drifted toward LLM style since 2024, and writers now avoid these patterns on purpose. In controlled studies people identify AI text at close to chance. That is the reason for the density rule. The goal is prose that carries information and sounds like a person wrote it, not prose that beats a detector. Cosmetic cleanup that leaves a vague, unsourced document vague and unsourced has made things worse, because it hid the real problem. diff --git a/.claude/skills/deslop b/.claude/skills/deslop new file mode 120000 index 000000000..efd867d48 --- /dev/null +++ b/.claude/skills/deslop @@ -0,0 +1 @@ +../../.agents/skills/deslop \ No newline at end of file diff --git a/.claude/skills/efficient-implementation-plans b/.claude/skills/efficient-implementation-plans new file mode 120000 index 000000000..fa48659be --- /dev/null +++ b/.claude/skills/efficient-implementation-plans @@ -0,0 +1 @@ +../../.agents/skills/efficient-implementation-plans \ No newline at end of file diff --git a/.claude/skills/pr-writing b/.claude/skills/pr-writing new file mode 120000 index 000000000..8db4f3adb --- /dev/null +++ b/.claude/skills/pr-writing @@ -0,0 +1 @@ +../../.agents/skills/pr-writing \ No newline at end of file diff --git a/.claude/skills/prompt-writing b/.claude/skills/prompt-writing new file mode 120000 index 000000000..05fc91ddb --- /dev/null +++ b/.claude/skills/prompt-writing @@ -0,0 +1 @@ +../../.agents/skills/prompt-writing \ No newline at end of file diff --git a/.claude/skills/technical-writing b/.claude/skills/technical-writing new file mode 120000 index 000000000..bedd96f61 --- /dev/null +++ b/.claude/skills/technical-writing @@ -0,0 +1 @@ +../../.agents/skills/technical-writing \ No newline at end of file diff --git a/.claude/skills/unslop b/.claude/skills/unslop new file mode 120000 index 000000000..158c16b3e --- /dev/null +++ b/.claude/skills/unslop @@ -0,0 +1 @@ +../../.agents/skills/unslop \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index 941b545ad..34e3e520d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,45 +1,59 @@ -# House scanning: agent guide +# Agent guide -Hackathon project (4-person team, started 2026-09-25). Homeowners walk the outside of their house with an iPhone. -The AR session measures the wall, a server works out where a **Base Power battery** can be installed, and the spot -is shown back in AR. +A homeowner scans the wall around their electric meter with an iPhone. The server decides whether a Base Power battery fits and where, and the app shows the spot in AR. [README.md](README.md) maps the repository, and [docs/00-overview.md](docs/00-overview.md) holds the plan, the decisions and the evidence. -## Read before doing anything non-trivial +## Who owns what -1. `docs/00-overview.md`: goal, the core insight, decisions made, **corrections that supersede earlier drafts**, open questions. -2. `docs/01-feature-map.md`: lanes A–D, features with priorities, the `scene.json` data format, the **rules table (numbers still TBD)**. -3. `docs/02-implementation-plan.md`: how each part gets built, code snippets, schedule, risks. -4. `docs/03-stack-research.md`: which library or open-source project for each part, licenses, gotchas, what to skip. -5. `docs/04-prior-art-and-codes.md`: Base's public rules, competitors, NEC / IRC / Austin Energy citations, pitch angle. -6. `docs/eli5.html`: a visual explainer to open in a browser. Step 4 is an interactive demo of how the solver works. +- **Client team.** Sam, working with AI agents, owns the iOS app in `ios/` and the capture packet it sends. Aiden films video and gathers sample datasets. +- **Server team.** Hunter, working with his own agents, owns everything after the packet: the 3D model and the rule checks, in `server/` and `recon/`. -## Hard rules +Most agent work arrives from a manager session that holds the plan's history and the discussions behind it. Report results, blockers and open decisions back to that manager, and let it settle questions about the plan or the product. -- **This repo is PUBLIC.** `private/` holds materials Base gave the team and is gitignored. Never commit, quote or summarize them - in tracked files: prompt text, prompt names, output field names, internal thresholds. Team notes go in `private/internal-notes.md`. -- **The placement decision is plain code, not AI.** Vision models only recognise things (object boxes, panel brand, meter text). - Every clearance number lives in `rules.yaml` with a code citation, never hardcoded. -- **Never run ML on Google Maps/Street View, Mapbox or Esri imagery.** Their terms forbid it. Use StratMap/NAIP. -- **Place AR results relative to the meter's anchor**, not in raw world coordinates. Use `.gravity` world alignment, not `.gravityAndHeading`. -- The camera image is landscape and the intrinsics match it. Rotate one and you must rotate the other. +Edit the other team's files only with that team's OK. Talk to the author before editing files that an open pull request also changes, because two writers on one file cost more time than asking does. -## ⚠️ Open conflict: native Swift vs Expo +## Where to look -The research plan (`docs/00-overview.md`) chose **native Swift + ARKit/RealityKit**. Expo has no first-class ARKit support, -and the capture needs LiDAR mesh export, per-frame position and lens data, and mesh raycasts. -On 2026-09-25 an Expo app was scaffolded at `apps/mobile/` (Expo SDK 57). **The team hasn't decided yet.** -Options: (a) drop Expo for native Swift; (b) Expo shell + a custom native Swift module for the whole AR capture; -(c) Expo for the non-AR screens only. Don't build AR capture in either place until the team picks one. +A component's README wins over the docs. Files marked with a pull request exist only on that branch until it merges. -`apps/mobile/AGENTS.md` has the Expo-specific rules for that folder. +| Task | Read | +| --- | --- | +| The iOS app | `ios/README.md` (PR #10 for guided capture) | +| The capture packet | `packet/README.md` (PR #22) | +| The rules engine, API and scene contract | `server/README.md` (PR #11), especially "What settles each check" | +| Photos to a 3D model | `recon/HANDOFF.md` (PR #20) | +| Accuracy evals and the field test | `experiments/evals/README.md` (PR #12), and the first phone run in `experiments/device-field-test/README.md` (PR #23) | +| Measurement conventions the code relies on | `docs/00-overview.md`, section "Conventions the code relies on" | +| Public rule values, code citations, model and imagery licenses | `docs/04-prior-art-and-codes.md` | +| The live guided-survey design | `docs/05-live-guided-survey-hld.md` | +| Branches, CI checks and TestFlight | `CONTRIBUTING.md` | -The setup pull request adds a native Swift skeleton at `ios/` as the recommended answer; its description gives the reasons. Remove this section once the team agrees. +## Rules -## Working in this repo +- **Keep Base's materials in `private/`, which git ignores.** The repository is public, so tracked files carry none of that material: no quotes, summaries, prompt text, prompt names, output field names or internal thresholds. Team notes go in `private/internal-notes.md`. Real captures, photos of homes and dataset images go in `captures/`, `fixtures/real/` or `data/`, which git also ignores. +- **Make placement decisions in deterministic code.** Models build geometry and recognize things, and plain code passes or fails each check, so every answer traces to a rule and a measurement. Clearance numbers live in rules files with their sources, never in code. +- **Treat unseen as unsure, never as clear.** A gap in the scan could hide a gas meter. When a check depends on an area nobody observed, return UNSURE and name the view that would settle it. +- **Use StratMap or NAIP for aerial imagery.** Google's and Mapbox's terms forbid running ML on their imagery, and Esri allows it only inside ArcGIS for non-commercial use. +- **Place AR results relative to the meter's anchor, with `.gravity` world alignment.** ARKit corrects anchors as tracking improves, so a result tied to the meter moves with it while raw world coordinates drift. `.gravityAndHeading` depends on the compass, which is unreliable next to a house. +- **Rotate the intrinsics whenever you rotate a camera image.** Camera images are landscape sensor images, and the intrinsics match that orientation. A rotated image with unrotated intrinsics produces wrong 3D geometry without any error. +- **Use LiDAR when present, and never require it.** Most homeowners' phones lack LiDAR, so every check must also work from the camera and the phone's poses. -- Layout: `ios/` is the capture app (native Swift, project generated by XcodeGen). `server/` is the Python server (uv). `web/` is a browser toolchain for the reviewer view and zero-install capture experiments. `experiments/` holds one folder per experiment. `sites/landing` is the landing page, a submodule; change it in its own repository. -- `make check` runs every suite. `make ios`, `make server` and `make web` run one each. `CONTRIBUTING.md` lists the matching CI checks. -- Branch from `main`, keep one writer per branch, and open a pull request. People merge; agents never push to `main` or merge. -- Teammates own lanes A–D (`docs/01-feature-map.md`). AI agents own setup, experiments and challenger reviews, and don't edit a lane's files without its owner's OK. -- Don't pick a signing team in Xcode's Signing & Capabilities pane: it writes into `project.pbxproj` and fails the CI drift check. Put `DEVELOPMENT_TEAM` in `ios/Config/Local.xcconfig` instead (see `ios/README.md`). -- After editing `ios/project.yml`, run `make ios-project` (needs XcodeGen 2.46.0) and commit the regenerated project. +## Writing, planning and review + +The repository ships shared skills in `.agents/skills/`, which Codex reads directly. `.claude/skills/` links to the same folders for Claude Code. Agents without skill support can read each `SKILL.md` as a plain guide. + +- `unslop` for any prose, and `technical-writing` for docs and READMEs. +- `pr-writing` for pull request titles and descriptions. +- `prompt-writing` for AGENTS.md, skills and prompts that brief another agent. +- `efficient-implementation-plans` for implementation plans. +- `deslop` for reviewing or cleaning a diff. + +A user's instruction outranks a skill. + +## Working in the repository + +- `make check` runs every suite. `make ios`, `make server` and `make web` run one each. +- Branch from `main`, keep one writer per branch, and open a pull request. People merge, and agents never push to `main` or merge, so a person sees every change before it lands. +- Several agents share one Mac. Exit code 137 means the system killed the process, usually for memory. Find the large allocation before rerunning, because one runaway process can freeze the whole machine. +- Put `DEVELOPMENT_TEAM` in `ios/Config/Local.xcconfig` (copy `Local.xcconfig.example`), not in Xcode's Signing & Capabilities pane. The pane writes into `project.pbxproj`, and CI fails on that drift. +- After editing `ios/project.yml`, run `make ios-project` (it needs XcodeGen 2.46.0) and commit the regenerated project. CI regenerates it and fails on any difference. +- `sites/landing` is a submodule. Change the landing page in its own repository. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d9fd21641..0ce41fefe 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -48,7 +48,7 @@ Keep workflows that run pull-request code away from production credentials and d ### Before the first upload -App Store Connect rejects a build without an app icon. House Scan needs an `AppIcon` set with a 1024×1024 image in an asset catalog inside `ios/HouseScan/`, added by the lane A owner. Measure Lab has one. +App Store Connect rejects a build without an app icon. House Scan needs an `AppIcon` set with a 1024×1024 image in an asset catalog inside `ios/HouseScan/`, added by the client team. Measure Lab has one. Neither `Info.plist` sets `ITSAppUsesNonExemptEncryption`, so each build waits under "Missing Compliance" until someone answers the encryption question on its TestFlight page. Setting the key to `false` removes that step for an app that uses only HTTPS and Apple's system encryption. @@ -67,11 +67,11 @@ Neither `Info.plist` sets `ITSAppUsesNonExemptEncryption`, so each build waits u Set the secrets and variables with the [GitHub CLI](https://cli.github.com), so the key never appears on screen. `gh secret set` without `--body` prompts without echoing. ```sh - base64 -i AuthKey_.p8 | gh secret set ASC_KEY_P8 --env testflight --repo SamGu-NRX/house-scanning - gh secret set ASC_KEY_ID --env testflight --repo SamGu-NRX/house-scanning - gh secret set ASC_ISSUER_ID --env testflight --repo SamGu-NRX/house-scanning - gh variable set APPLE_TEAM_ID --env testflight --repo SamGu-NRX/house-scanning --body - gh variable set BUNDLE_ID_PREFIX --env testflight --repo SamGu-NRX/house-scanning --body + base64 -i AuthKey_.p8 | gh secret set ASC_KEY_P8 --env testflight --repo SamGu-NRX/house-scanning-master + gh secret set ASC_KEY_ID --env testflight --repo SamGu-NRX/house-scanning-master + gh secret set ASC_ISSUER_ID --env testflight --repo SamGu-NRX/house-scanning-master + gh variable set APPLE_TEAM_ID --env testflight --repo SamGu-NRX/house-scanning-master --body + gh variable set BUNDLE_ID_PREFIX --env testflight --repo SamGu-NRX/house-scanning-master --body rm AuthKey_.p8 ``` diff --git a/README.md b/README.md index 7e445e764..67a521112 100644 --- a/README.md +++ b/README.md @@ -2,48 +2,41 @@ We're exploring a simpler way to survey a home for a Base Power battery. -A homeowner walks the outside of their house with a phone app. The app measures the wall around the electric meter, and our code decides whether a Base Power battery can be installed there and, if so, where. The homeowner sees the spot in AR before they put the phone away. +A homeowner walks the outside wall around their electric meter with an iPhone. The app guides the scan until it has seen everything the placement rules need, and sends what it captured. The server builds a 3D model, decides in plain code whether a Base Power battery fits and where, and the app shows the spot in AR. -This is a four-person hackathon project for Base Power. Success means one capture session gives Base enough to decide, with no follow-up photos. A person can still review the result. +This is a four-person hackathon project for Base Power, started 2026-09-25. Success means one capture session gives Base enough to decide, with no follow-up photos. -## Why measure instead of photograph +Two teams share the work. The client team, Sam with AI agents and Aiden on video and sample data, builds the iPhone app and the capture packet it sends. The server team, Hunter with his agents, turns the packet into a 3D model and checks the rules on it. -Base's site survey starts from a few customer photos, and three things go wrong with them: +## Where to start -- Distances are estimated by eye, so borderline clearances go to a reviewer. -- Unordered photos can't show whether two things are on the same wall or how far apart they are. -- Nobody knows what sits just outside the frame, so the homeowner gets asked for more photos. - -An AR session tracks the phone's position as it moves. The app can measure the wall in feet, and it knows which stretches nobody has looked at yet. It asks for those while the homeowner is still standing there. - -## How it works - -1. The homeowner marks the electric meter, walks the wall to its end in each direction, and takes close-ups of the meter and the electrical panel. The app saves frames along with where the camera was for each one. -2. A server lays the walls out as one line, measured in feet from the meter. It slides the battery's footprint along that line and checks each clearance: gas equipment, driveways, doors and windows, room overhead, and the gap to a facing fence. Each check returns pass, fail, or unsure, with the measurement error counted. -3. The app shows the chosen spot, the battery, and the cable route from the meter in AR. - -Plain code makes the placement decision. Vision models only recognize things, such as where a gas meter is or what the meter label says. Several clearance numbers are not final. Base still has to confirm the rules for driveways, pools, the fence gap, and cable length. - -## What we're testing now - -The plan targets iPhones, and a LiDAR iPhone makes some measurements easier. Most homeowners don't have one, so the experiments ask how far an ordinary phone gets: - -- Can AR taps on the ground and the wall measure the wall line, the room overhead, and the gap to a fence without LiDAR? -- Can plain phone video do the job instead? [OOOSplat](https://github.com/ooolabdev/ooosplat) turns a video into a 3D scene. We scale the scene from one known distance and check a second distance against a tape measure. -- How close does each method get? We tape-measure one real house and compare AR, a depth model on plain photos, and a vision model's guesses against the tape. - -A convincing 3D view can still miss an obstacle or get a distance wrong, so every result keeps its original photos beside it. - -## Read next - -| File | What it covers | +| Read | For | | --- | --- | -| [docs/00-overview.md](docs/00-overview.md) | The goal, decisions made so far, and open questions | -| [docs/01-feature-map.md](docs/01-feature-map.md) | The four work lanes, features by priority, the `scene.json` format, and the rules table | -| [docs/02-implementation-plan.md](docs/02-implementation-plan.md) | How each part gets built, the schedule, and risks | -| [docs/03-stack-research.md](docs/03-stack-research.md) | The open-source projects and libraries we build on, and the ones we skip | -| [docs/04-prior-art-and-codes.md](docs/04-prior-art-and-codes.md) | Base's public rules, similar products, and electrical-code citations | -| [docs/05-live-guided-survey-hld.md](docs/05-live-guided-survey-hld.md) | Proposed live guidance and automatic capture architecture, diagrams, evidence flow, and technical resources | -| [docs/eli5.html](docs/eli5.html) | A visual walkthrough with an interactive demo of the placement search | - -Contributors and coding agents start with [AGENTS.md](AGENTS.md). This repository is public, so materials Base gave the team stay in the git-ignored `private/` folder. +| [docs/how-it-works.html](docs/how-it-works.html) | A picture-first walkthrough of the system, about fifteen minutes | +| [docs/00-overview.md](docs/00-overview.md) | The plan, the decisions and their reasons, the evidence so far, and open questions | +| [AGENTS.md](AGENTS.md) | The rules for anyone changing this repository, person or agent | +| [CONTRIBUTING.md](CONTRIBUTING.md) | Branches, CI checks and TestFlight | + +## Repository map + +Paths marked with a pull request exist only on that branch until it merges. + +| Path | What it is | State | +| --- | --- | --- | +| `ios/` | The iPhone app (Swift, ARKit, RealityKit) | On `main`, an AR session that shows tracking. Guided capture is in PR #10, and the live 3D map in PR #21 | +| `packet/` | The capture packet's spec, validator and samples | PR #22 | +| `server/` | The rules engine and placement API (Python, uv) | On `main`, a skeleton. The engine is in PR #11 | +| `recon/` | Turns photos and depth into a 3D model and a coverage map | PR #20, handed to the server team | +| `experiments/` | One folder per experiment | Accuracy evals in PR #12, Measure Lab in PR #7, meter reading in PR #16, panel labels in PR #17, the first device field test in PR #23 | +| `web/` | Browser toolchain for a reviewer view | The review page (PR #15) is parked | +| `docs/` | The overview, the walkthrough, public rules and code citations, and the live-survey design | | +| `.agents/skills/` | Shared agent skills for writing, planning and review, linked from `.claude/skills/` | | +| `sites/landing` | The landing page, a submodule | Change it in its own repository | + +## What's live + +- The demo server at https://house-scanning-server.vercel.app runs the engine from PR #11 with public rules only, and every answer says so. `GET /health` shows which rules are loaded. A second deployment loads Base's rules and requires a key. +- TestFlight builds of the app and Measure Lab start by hand from the Actions tab, as [CONTRIBUTING.md](CONTRIBUTING.md) describes. +- `make check` runs the server, web and iOS suites that CI runs. + +This repository is public. Materials Base gave the team stay in the git-ignored `private/` folder, and photos of real homes never enter git. diff --git a/docs/00-overview.md b/docs/00-overview.md index b0cc8b4fc..20759ef3a 100644 --- a/docs/00-overview.md +++ b/docs/00-overview.md @@ -1,67 +1,66 @@ -# House scanning: overview and decisions - -Hackathon project, 4-person team, started 2026-09-25. The goal: from a homeowner's phone, work out -**where a Base Power battery can be installed** against the outside of the house, and show the spot -back to them in AR. - -> **This repo is public.** Materials Base gave the team live in `private/`, which is gitignored. Never commit, -> quote or summarize them here. Team-only notes are in `private/internal-notes.md`. - -## The core insight (why AR) - -Today homeowners submit photos and reviewers judge them. Three problems come up that ordinary photos can't solve: -1. **Measuring distance.** A single photo has no real scale, so gaps and clearances get estimated by eye, and the borderline cases go to a person. -2. **Fitting photos together.** A handful of unordered photos can't reliably say whether two things are on the same wall, or how far apart they are. -3. **Coverage.** If part of the wall wasn't photographed, nobody can say what's there, so the homeowner gets asked for more photos. - -AR fixes all three. The recognition half (panel brand, meter text, rust, whether a window opens) stays with a -vision model. AR adds nothing there beyond better photos. - -## Decisions made - -- **AR-guided capture, not "upload a video and process it later."** Offline photogrammetry (COLMAP, splats) fails on - blank siding and stucco, has no real scale, and is slow. ARKit already fuses the camera with the motion sensors on the phone. -- **The phone collects data; the server decides.** Swift app → zip upload → Python FastAPI server → result → AR preview. -- **The placement code is plain code, not AI.** Unroll the house walls into one straight line `s` (feet along the wall - from the meter). Most rules become 1-D interval checks plus 2-D polygon distances. Sweep the battery footprint in small steps. - Every check returns PASS / FAIL / UNSURE, with error bars on each measurement. See `02-implementation-plan.md`. -- **The vision model only recognises things.** Gemini boxes → SAM 2 → cast onto the wall using where the phone was. - Recognition prompts run on the close-ups. -- **Native Swift + ARKit/RealityKit on a LiDAR iPhone.** Not Expo (no first-class ARKit), and not Android for v1. -- **Normal dev app for the hackathon; an App Clip is only the pitch's rollout story** (see the gotchas in `03-stack-research.md`). - -## Corrections made after the research (supersede earlier drafts) - -1. Use `worldAlignment = .gravity`, **not** `.gravityAndHeading`. The compass is unreliable near a house - (developers report errors up to ~176°). Take one compass reading away from the meter as a rough hint. Which wall faces the - street comes from map data: Overture building outline + osmnx nearest road. -2. **Place the battery relative to the meter's anchor**, not in raw world coordinates. ARKit corrects anchors as it goes, - and drift is about 2 cm per second. -3. The battery is **39.5″ tall** (Base Core: 39.5 × 30.68 × 22 in), not 36″. Base's older units are 3 × 3 ft, - so keep all dimensions in the rules config. -4. Claude is **not** the box detector; its docs say its coordinates are approximate. Gemini returns boxes natively as - `[ymin, xmin, ymax, xmax]` on 0–1000. -5. **Never run ML on Google, Mapbox or Esri imagery.** Google's ToS §3.2.3(c) bans it outright. Use StratMap/NAIP (public domain). - -## Open questions (team must decide) - -- **Which clearance numbers to use.** The public baseline is Base's help page: within 20 ft of the meter, within 1 ft of the wall, - and 3 ft from gas meters, fences, AC units and other batteries. Other numbers (driveway, pool, passage width, max cable run) are - still TBD. The team's working notes are in `private/internal-notes.md`. -- **Whether the install includes a separate battery disconnect box** beside the meter. Base's public pages mention a wall-mounted - transfer switch and a battery disconnect. -- **Test these in hours 0–4** (the unknowns everything depends on). See `03-stack-research.md` § Test first. -- **Native Swift vs Expo.** On 2026-09-25 an Expo app (SDK 57) was scaffolded at `apps/mobile/`, which conflicts with the - native-Swift decision above. Options: drop Expo; Expo shell + a native Swift module owning all AR capture; or Expo for the - non-AR screens only. Decide before anyone builds AR capture. - -## Doc map - -| File | What | -|---|---| -| `00-overview.md` | this file: goal, insight, decisions | -| `01-feature-map.md` | lanes A–D, feature list with priorities, data format, rules table | -| `02-implementation-plan.md` | how to build each part, code snippets, 24 h schedule, risks | -| `03-stack-research.md` | libraries and open-source projects per part, licenses, gotchas, what to skip | -| `04-prior-art-and-codes.md` | what Base does publicly, competitors, electrical-code citations, pitch angle | -| `eli5.html` | visual explainer for anyone new; step 4 is an interactive "slide the battery" demo | +# House scanning: plan, decisions and evidence + +A homeowner scans the outside wall around their electric meter with an iPhone. The server works out whether a Base Power battery fits there and where, and the app shows the spot in AR. The goal is one capture session that gives Base enough to decide, with no follow-up photos. A person can still review any result. + +Base's survey starts today from customer photos. A photo has no scale, can't show whether two things share a wall, and misses whatever sits just outside the frame. Borderline cases go to a reviewer, and homeowners get asked for more photos. An AR session gives every photo a position and a real scale, and the app knows what it hasn't seen while the homeowner is still standing there. + +Full-house scans, roofs and Android are out of scope. Each component's README is the source of truth for that component and wins where it differs from this file. + +## How the pieces fit + +The work is three problems. The phone guides the homeowner until it has captured everything the rules need. The photos and their metadata become a 3D model. Plain code checks that model against the placement rules, and the answer comes back to the phone. + +**Client team: Sam with AI agents, and Aiden on video and sample datasets.** The app in `ios/` builds a live 3D map on the phone. A haze, the "fog of war", covers what it hasn't seen and lifts as the homeowner scans. The app guides them until everything the rules need is covered, which is a zone around the meter rather than the whole house. Guided capture is in PR #10 and the live map in PR #21. The app's output is the capture packet: photos, camera poses, intrinsics, motion-sensor readings, LiDAR depth and mesh when the phone has LiDAR, and the homeowner's marks. `packet/README.md` (PR #22) specifies it and lists what the app records today. On `main` the app only shows AR tracking. With PR #10 it uploads only its measurements, `scene.json`, and exports the full packet on request. + +**Server team: Hunter with his agents.** The packet goes in, and a 3D model comes out at real scale with a record of which parts were observed. The rules engine in `server/` (PR #11) then checks every candidate spot. The team is comparing reconstruction methods broadly first, then the best in depth. Hunter's research handoff (PR #18) plans the server side: +- Original photos travel to object storage, separate from the compact placement request, because a Vercel function accepts at most 4.5 MB. +- Recognition and reconstruction run asynchronously while the phone keeps guiding, and capture, upload, analysis and assessment finish at different times. +- A new reconstruction can revise earlier geometry. Coverage and measurements that depend on it get recomputed, never carried over. +- OCR confidence doesn't settle an exact identifier, so unreadable or conflicting values stay unresolved. + +The reconstruction worker is in `recon/` (PR #20), the accuracy evals in `experiments/evals/` (PR #12), and Hunter's live-survey design in [05-live-guided-survey-hld.md](05-live-guided-survey-hld.md). + +## Decisions + +- **Guided AR capture, not a video processed later.** Photogrammetry alone has no scale, is slow, and struggles on blank siding and stucco. COLMAP's own tutorial says to avoid texture-less walls. The packet carries ARKit's poses and any LiDAR depth, so the server's model starts at real scale. +- **The server builds a 3D model at real scale.** Learned depth alone is 4 to 12% off in scale and 8 in or worse at edges, so LiDAR and the phone's poses supply the scale. +- **Models build geometry and recognize things. Plain code decides placement.** The engine unrolls the walls into one line, `s`, in feet from the meter, negative to the left. It slides the battery's footprint along that line in 2 in steps and checks each rule at each position. Every answer then traces to a rule and a measurement. +- **Every check answers PASS, FAIL or UNSURE.** A check passes only when its margin beats the measurement's error, and fails only when it misses by more than the error. Everything else is UNSURE. That includes any check on an area nobody observed, and the answer names the view that would settle it. +- **Rule values live in rules files, each with its source.** `server/rules.yaml` (PR #11) holds the public values and marks each value with no public source `placeholder: true`, such as pool 10 ft and driveway 5 ft. Base's values load from a git-ignored private file. `rules.yaml` also holds the battery's size, because models differ. Base Core is 39.5 in tall on a 30.68 × 22 in footprint, and older units are 3 × 3 ft. +- **Native iOS only.** Capture needs mesh export, per-frame pose and lens data, and mesh raycasts, and Expo has no first-class ARKit support. +- **AR results hang off the meter's anchor, with `.gravity` alignment.** ARKit corrects anchors as tracking improves, and drift runs about 2 cm a second, so a result tied to the meter moves with the correction. `.gravityAndHeading` depends on the compass, which developers report off by up to about 176° near a house. + +## Conventions the code relies on + +These are the non-obvious facts behind numbers in the code, and where each number came from. + +- `capturedImage` is landscape, as the sensor reads it, and the intrinsics match it. Any rotation must also rotate the intrinsics and every image-space annotation. Gemini is the planned box detector, because Claude's docs call Claude's coordinates approximate. Gemini returns `[ymin, xmin, ymax, xmax]` on a 0 to 1000 scale, so apply EXIF rotation before inference and before casting a box. +- Depth maps are 256 × 192. Scale the intrinsics down to match. +- ARKit camera space is x right, y up, looking down −z, with image v growing downward. The ray through pixel (u, v) points along ((u − cx)/fx, −(v − cy)/fy, −1) before the camera-to-world rotation. +- Keyframes are kept every 0.5 m or 15° of movement. +- Mesh vertices are local to their `ARMeshAnchor`. Copy each one out of ARKit's buffer, then multiply the copy by the anchor's transform, leaving the buffer untouched. Classified mesh needs `sceneReconstruction = .meshWithClassification` on the world-tracking configuration, on LiDAR phones only. RealityKit also turns classification off unless `automaticallyConfigureSession` is `false`. +- On the mesh, the gap to a facing fence is a ray straight out from the wall at 1.5 ft high. Headroom is a ray straight up from 1 ft out. The mesh has holes, so each measurement is the smallest distance over a small fan of rays. +- LiDAR reads from about 0.5 m to 5 m, so capture stands 1 to 3 m from the wall. A ray that meets nothing within 5 m is unknown, not open. +- The default error bars are day-1 estimates, untested: 0.3 ft for an AR tap, 0.5 ft for the LiDAR mesh and 1.5 ft for a position from photo detection. The server adds 0.16 ft per foot along the wall from the meter, from measured ARKit drift. + +## Evidence so far + +These results come from public datasets with laser-scanned or surveyed ground truth. Only the first phone run below was measured on our phone. The sources are `experiments/evals/README.md` (PR #12), `recon/HANDOFF.md` (PR #20), `experiments/meter-closeup/README.md` (PR #16), `experiments/panel-label/README.md` (PR #17) and `experiments/device-field-test/README.md` (PR #23). + +- **Tracking.** A current iPhone stayed inside the server's error allowance. On the MARViN dataset, an iPhone 14 Pro Max had p90 position error of 8.6, 13.4 and 18.5 in after 10, 20 and 30 ft, against an allowance of 19.2, 38.4 and 57.6 in. That phone has LiDAR, and MARViN's reference may be scaled to its tracking. A 2018 iPhone 6s on ADVIO ran two to three times over, partly from noise in that dataset's reference. +- **Learned depth.** A model's own scale is 4 to 12% off. Rescaled with the phone's poses, walls come within about 5 in at p90 at a realistic 2% pose error, and within 2.8 in with exact poses. Edges stay at 8 in or worse, and clearances are measured from edges. MapAnything's scale shifted with input resolution. +- **Reconstruction worker.** On the ETH3D building, wall p90 is 2.1 in with a laser scan standing in for LiDAR, and 2.8 in from photos only. The two runs measured different walls. +- **Coverage.** The app's first coverage map claimed 1.1 ft of a 19.3 ft wall that no photo saw. Requiring several view angles removed the false claim but also discarded 6 to 8.5 ft of good wall. A depth test on LiDAR phones removed it cleanly. Checked against depth, the reconstruction worker claims at most about 0.46 ft of unseen wall. +- **Meter reading.** Apple's on-device text recognition read the full meter number on 71 of 73 real photos. It picked the right line on only 21 of 75, because nameplates carry several numbers. A barcode held the number on 20 of the 24 photos that had one. A list of three candidates held it on 85% of held-out photos. So the app offers three candidates to tap, barcode match first, and asks for a retake when the photo is blurry or the number too small. +- **Panel labels.** Not tested. Only 7 open photos of US panels exist, and the test needs 40. +- **World models.** Not evaluated yet. +- **First phone run.** The TestFlight build's first run on a real wall (PR #23) set both wall ends at the meter, so the server placed no spot. Two of five measured features came within 4 in of the tape. + +## Open questions + +- **Clearance values.** [04-prior-art-and-codes.md](04-prior-art-and-codes.md) has the public values with their citations. Base's own values go in the private rules file, along with which number on the meter plate Base needs. +- **Disconnect box.** Does the install add a battery disconnect beside the meter? Base's public pages mention a wall-mounted transfer switch and a battery disconnect. +- **The 3D path.** The candidates are traditional reconstruction fused with LiDAR, learned depth rescaled with the phone's poses, and world models, which produce a whole 3D scene from photos or video. The winner needs real scale, edges within the error allowance, and a record of which parts it observed and which it filled in. +- **Accuracy on our phone.** The field test uses a current iPhone without LiDAR, the Measure Lab build (PR #7), a 30 ft tape and a real meter wall. It follows `experiments/evals/field/FIELD_SHEET.md` (PR #12), and a dry run recovered a planted 1.5% scale error. It also checks the default error bars. LiDAR on siding in midday sun is untested too. +- **Pool and driveway.** Their checks need up to 13 ft of photographed ground. The alternative is one yes-or-no question, recorded as the homeowner's answer. +- **Hidden wall.** Something standing in front of the wall can hide it, and nobody owns that check yet. One proposal is for the app to show the homeowner the photo of the chosen spot and ask, and to run the depth test on LiDAR phones. diff --git a/docs/01-feature-map.md b/docs/01-feature-map.md deleted file mode 100644 index 3975bb195..000000000 --- a/docs/01-feature-map.md +++ /dev/null @@ -1,102 +0,0 @@ -# Feature map - -**Priority:** **P0** = needed for the demo · **P1** = strong add · **P2** = stretch -**Lanes (one teammate each):** **A** capture app · **B** object detection and geometry · **C** rules + placement code · **D** output, testing, pitch - -## Hour 1: agree on the data format (everyone) - -Every lane reads or writes this, so lock it first. Then all four people can work in parallel against a hand-built -`fixtures/fake-01/scene.json` until real captures exist. - -```json -{ - "meter": {"pos": [x,y,z], "wall_id": "w1"}, - "walls": [{"id": "w1", "baseline": [[x,z], ...], "height_ft": 9}], - "objects": [{"type": "window|door|garage_door|ac|gas_meter|elec_box|vent|downspout|pool", - "wall_id": "w1", "span_ft": [a, b], "bottom_ft": 3.1, "top_ft": 6.2, - "attrs": {"operable": true, "well": false}, "source": "vlm|tap", "conf": 0.8}], - "ground": [{"type": "drive|concrete|gravel|lawn|mulch|deck", "polygon": [[x,z], ...]}], - "overheads": [{"wall_id": "w1", "span_ft": [a, b], "clearance_ft": 3.0}], - "facing": [{"wall_id": "w1", "span_ft": [a, b], "depth_ft": 4.5}], - "keyframes": [{"id": "k1", "pose": [...16 floats, column-major cam_to_world...], "intrinsics": [fx,fy,cx,cy], - "w": 1920, "h": 1440, "img": "k1.jpg"}], - "stills": {"meter_close": "...", "meter_surround": "...", "panel_wall": "...", "panel_open": "..."}, - "gps": {"lat": 0, "lon": 0}, "heading": {"deg": 0, "accuracy_deg": 0} -} -``` - -- `facing` is the gap from the wall straight out to the nearest fence, hedge or wall. -- `span_ft` is the stretch along the wall an item covers, in feet from the meter (negative = left of the meter). -- Keyframe images are saved **unrotated** (landscape, as the sensor reads them), so `intrinsics` apply to them directly. - -## A: Capture app (Swift, ARKit/RealityKit) - -| # | Feature | Pri | Notes | -|---|---|---|---| -| A1 | Tap to mark the meter → anchor | P0 | The starting point; the result is later placed relative to it | -| A2 | Guided walk left/right, tap wall ends + corners | P0 | Can't finish until both ends are tapped, so nothing goes unphotographed | -| A3 | Wall + ground detection → wall line at ground level | P0 | Real feet | -| A4 | Save frames every ~0.5 m / 15° with position + lens data | P0 | Fork Stray Scanner's recorder | -| A5 | Tap-to-mark fallback: gas meter, door, window, AC, garage, driveway edge | **P0** | The demo never depends on detection. Also gives correct answers to test against | -| A6 | Guided close-ups: meter, panel wall, open panel, disconnect | P0 | Input for the recognition checks | -| A7 | Photo checks: sharpness, meter can fully in frame, text readable on the phone (Vision) | P1 | | -| A8 | One session from the meter outside to the panel inside | P1 | Answers "same wall?" and "directly behind the meter?" exactly | -| A9 | Mesh .ply export (anchor transform applied) + zip + foreground upload | P0 | | - -## B: Object detection and geometry (server) - -| # | Feature | Pri | Notes | -|---|---|---|---| -| B1 | Gemini boxes for window, door, garage door, AC, gas meter, electrical box, vent | P0 | Or tap marks from A5 | -| B2 | Cast each box onto the wall using the phone's position → s-range + height | P0 | Test first: tap a point, cast it back into a saved frame, check the pixel | -| B3 | Window details: ground-level from measured height, opens vs fixed, window well | P1 | | -| B4 | Passage width (wall out to the nearest fence/hedge) from mesh rays | P1 | LiDAR only; otherwise UNSURE | -| B5 | Overhead headroom (porch, eave, stairs) from mesh rays | P1 | LiDAR only | -| B6 | Ground surfaces + whether they connect to the driveway | P2 | Hardest. Demo uses a tapped driveway edge | -| B7 | Pool + pool-equipment detection | P2 | Also from the aerial photos | -| B8 | Street-facing wall from Overture outline + osmnx road (flag for homeowner approval of the look) | P2 | | - -## C: Rules + placement code - -| # | Feature | Pri | Notes | -|---|---|---|---| -| C1 | `rules.yaml`: every number, per-utility overrides (ComEd), a code citation per rule | P0 | No hardcoded distances in the code | -| C2 | Sweep the 31″ × 22″ footprint along the unrolled wall, 2″ steps | P0 | | -| C3 | Clearance checks: gas, driveway, pool, openings, box above, vent, AC, passage width, headroom, ground | P0 | PASS / FAIL / UNSURE with error bars | -| C4 | Cable route along the wall: blocked by door, garage or a stretch with no wall; length ≤ max | P0 | | -| C5 | Result: pass / manual_review / reject + reasons + photos still needed | P0 | | -| C6 | Rank the valid spots: shortest run, prefer side walls | P1 | | -| C7 | Check the disconnect box's wall space + conduit route | P2 | Only if the current product still uses it | - -## D: Output, testing, pitch - -| # | Feature | Pri | Notes | -|---|---|---|---| -| D1 | AR preview: RealityKit box attached to the meter's anchor + cable line | P0 | The demo moment | -| D2 | 2D site plan (drawsvg): blue meter area, purple cable, red battery spot | P0 | | -| D3 | Tape-measure one real house: AR vs depth model vs the vision model's guesses | **P0** | The pitch's headline number | -| D4 | Baseline: Depth Anything 3 metric on plain photos | P1 | | -| D5 | Recognition prompts on the close-ups (prompts stay in `private/`) | P1 | | -| D6 | Meter type from the can's real height in inches | P2 | | -| D7 | Reviewer web view / JSON report | P2 | | - -## Rules table (C1): CONFIRM THESE - -Public sources only. The team's working numbers are in `private/internal-notes.md` (gitignored). - -| Rule | Public source | Value | Use | -|---|---|---|---| -| Distance from meter | Base help page | within 20 ft | 20 ft, TBD | -| Distance from wall | Base help page | within 1 ft | 1 ft | -| Gas meter / regulator | Base help page; Austin Energy §1.9; Texas Gas Service | 3 ft | 3 ft | -| AC units, fences, other batteries | Base help page | 3 ft | 3 ft | -| Doors and windows | IRC R328 (batteries); Austin Energy §1.9.2 (meters, ≥ 1 ft) | 3 ft | 3 ft | -| Working space at meter / panel | NEC 110.26 | 30″ wide × 36″ deep × 6.5 ft headroom | as listed | -| Driveway | — | — | **TBD** | -| Pool | — | — | **TBD** | -| Passage width (wall to fence) | — | — | **TBD** | -| Battery footprint | Base Core spec | 39.5″ × 30.68″ × 22″ (older units 3 × 3 ft) | per model | - -## Not doing - -Android, Expo, rebuilding a 3D model from uploaded video, full-house or roof scans. diff --git a/docs/02-implementation-plan.md b/docs/02-implementation-plan.md deleted file mode 100644 index 956faca5d..000000000 --- a/docs/02-implementation-plan.md +++ /dev/null @@ -1,152 +0,0 @@ -# Implementation plan - -The main trick is to unroll the house's walls into one straight line, measured in feet from the meter. Almost -every rule then becomes a check on a 1-D number line, and the placement code gets simple. - -## Overall setup - -``` -iPhone (Swift, ARKit) ──upload .zip──▶ Python server (FastAPI) ──result──▶ iPhone AR preview - collects measurements + photos 1. read the upload, build the wall line + 2D site plan - 2. find objects, cast them onto the wall - 3. rule checks → pick a spot - 4. recognition prompts on the close-ups -``` - -The phone only collects data; all the logic runs on the server. Placement code can then be tested against saved -captures without anyone walking a house. - -Repo layout: - -``` -/ios Xcode project -/server ingest.py geometry.py detect.py solver.py rules.yaml plan_svg.py prompts/ -/fixtures saved captures (fake-01 hand-built in hour 1; real ones gitignored) -/eval tape-measure ground truth -/docs these docs -/private materials not for publication (gitignored) -``` - -## Lane A: Capture app - -AR setup: - -```swift -let config = ARWorldTrackingConfiguration() -config.planeDetection = [.horizontal, .vertical] -config.worldAlignment = .gravity // NOT .gravityAndHeading (compass unreliable near a house) -if ARWorldTrackingConfiguration.supportsSceneReconstruction(.meshWithClassification) { - config.sceneReconstruction = .meshWithClassification // LiDAR phones only -} -if ARWorldTrackingConfiguration.supportsFrameSemantics(.sceneDepth) { - config.frameSemantics.insert(.sceneDepth) -} -arView.automaticallyConfigureSession = false // otherwise RealityKit turns classification off -arView.session.run(config) -``` - -Capture flow (P0). Use taps rather than automatic wall detection, so the demo can't break: -1. **"Tap the meter."** Raycast from the tap to a vertical surface and add an `ARAnchor`. That's the zero point. -2. **"Walk left until the wall ends, then tap the end."** Then right. Tap each corner. The taps define the wall line. -3. **Save frames while walking** (~every 0.5 m or 15°): `capturedImage` saved unrotated, `camera.transform`, - `camera.intrinsics`, optional `sceneDepth.depthMap` (256×192; the server scales K down to match). - `captureHighResolutionFrame` gives sharper stills and still includes the pose. -4. **Tap-to-mark buttons:** gas meter, door, garage door, window, AC, driveway edge. -5. **Guided close-ups** for the recognition checks. Photo checks run on the phone: sharpness (Laplacian variance), and Apple - Vision `RecognizeTextRequest` + `VNDetectBarcodesRequest` on the meter. -6. **Mesh export:** for each `ARMeshAnchor`, multiply every vertex by `meshAnchor.transform` (vertices are local to the anchor), - **copying them out** of ARKit's `MTLBuffer` rather than writing back into it. Indices are uInt32. Write the PLY by hand - so each face can keep its classification (wall/door/window). -7. Zip `scene.json` + JPEGs + `mesh.ply` and upload with a **foreground** `URLSession`. -8. Take GPS + one compass reading with `CLLocationManager` (trust it only if `headingAccuracy` ≤ 20°). - -## Lane B: Geometry and detection (server) - -**The wall line (`geometry.py`).** Chain the tapped points (left end → corners → meter → corners → right end) into -a line along the ground. Every point on the house gets one coordinate `s`: feet from the meter, negative to the left. -Each segment also stores its outward direction and ground height. - -**Casting a box onto the wall:** - -```python -def pixel_ray(u, v, K, cam_to_world): - # ARKit camera: +x right, +y up, looks down -z; image v grows downward (Rerun calls this RUB) - d = np.array([(u - K[0,2]) / K[0,0], -(v - K[1,2]) / K[1,1], -1.0]) - d = cam_to_world[:3,:3] @ d - return cam_to_world[:3,3], d / np.linalg.norm(d) - -def hit_plane(o, d, p0, n): - denom = d @ n - if abs(denom) < 1e-6: return None - t = ((p0 - o) @ n) / denom - return o + t*d if t > 0 else None -``` - -- **Things on the wall** (windows, doors, boxes): cast rays from the box's left and right edges into the wall plane → the `s` range + height above ground. -- **Things on the ground** (AC unit, gas meter): cast the mask's bottom-center into the ground plane, then snap to the nearest wall point. -- **Merging duplicates:** merge same-type detections whose `s` ranges overlap, and take a majority vote on attributes. -- ⚠️ `capturedImage` is landscape (the way the sensor reads it) and `intrinsics` match it. Rotate one and you must rotate the other. - Gemini returns `[ymin, xmin, ymax, xmax]` on 0–1000, so convert to pixels of the **unrotated** image. -- **Test this in hour 2:** tap a point, cast it back into a saved frame, and check it lands on the right pixel. - -**Mesh measurements (Open3D `RaycastingScene`, float32 rays, a miss = `inf`).** Every 2″ along the wall: -- Passage width: a ray straight out from the wall at 1.5 ft height. -- Headroom: a ray straight up from 1 ft out from the wall. -- The mesh has holes, so cast a small fan of rays and take a robust minimum. A miss beyond ~5 m (the LiDAR limit) means **unknown**, not "open". - -## Lane C: Placement code (`solver.py`) - -Every measurement carries an error estimate: ±0.3 ft for AR taps, ±0.5 ft for the mesh, ±1.5 ft for anything placed from -a model's photo detection. A check answers **PASS** if the margin is larger than the error, **FAIL** if it's clearly -violated, and **UNSURE** otherwise. - -```python -W = rules.battery.width_ft # 31/12 -for s in np.arange(scene.s_min, scene.s_max - W, 1/6): # 2-inch steps - spot = Spot(s, s + W, depth=rules.battery.depth_ft) - checks = [c(scene, spot, rules) for c in CHECKS] # gas, drive, pool, openings, box_above, - route = harness_route(scene, spot, rules) # vent, ac, front, overhead, ground, street_facing -``` - -- **Distance checks** (gas meter, pool, driveway): the spot is a 2D rectangle against the wall; use shapely `distance`. -- **Checks along the wall** (openings, AC, box above, vent): numpy overlap `max(a0,b0) < min(a1,b1)`. The ComEd rule widens each opening by 3 ft on both sides. -- **Cable route:** the path from `s=0` to the spot along the unrolled line. It fails if it crosses a door or garage interval or a stretch with no wall. - Its length is |s| plus a little per corner. Past a configurable "confident reach" it's UNSURE; over the max it's FAIL. -- **Decision:** - - **pass:** some spot passes every check. Pick the shortest run, preferring side walls. - - **manual_review:** some spot has no FAIL but at least one UNSURE. Report those reasons. - - **reject:** both wall ends were reached and every spot FAILs. - - If the wall wasn't walked to its end → manual_review + missing views. -- A brute-force sweep is enough: ~1,200 positions × tens of checks runs in under a second. No OR-Tools or z3. - -## Lane D: Output, prompts, testing - -- **AR preview:** the server returns the spot **relative to the meter's anchor**. The app attaches a `ModelEntity` - (`MeshResource.generateBox`, 31″ × 22″ × 39.5″) to that anchor, plus thin cylinders for the cable. Keep the AR session running - while the server works; relocalizing later with an `ARWorldMap` is P2. -- **2D site plan:** drawsvg; blue meter area, purple cable run, red battery spot and its clearance. -- **Recognition prompts:** load them from `private/` at runtime. `server/prompts/` is gitignored. Never commit prompt text; this repo is public. -- **Testing:** tape-measure 10–15 distances on one real house. Compare AR, Depth Anything 3 on plain photos, and the - vision model's guesses. The error table is the pitch slide. -- **Debugging:** Rerun. Log camera positions (`Pinhole`, `camera_xyz=RUB`), frames, mesh (static), wall line, detections and solver rays. - -## Schedule (~24 h; stretch it for longer) - -| Hours | Everyone | -|---|---| -| 0–2 | Lock `scene.json`. Hand-build `fixtures/fake-01`. Pick the rules numbers | -| 2–10 | Each lane builds against the fake capture: A tap flow + upload, B ray casting + detection, C solver + rules, D AR preview + SVG | -| 10–14 | **First full run on a real wall**, even with tap-only marking | -| 14–20 | Model detection in place of taps where it holds up, mesh measurements, error tuning, prompts on the close-ups | -| 20–24 | Tape-measure test, demo rehearsal on one house, pitch | - -## Risks and fallbacks - -| Risk | Fallback | -|---|---| -| Detection places objects badly | Tap-to-mark is P0 | -| Image orientation / box coordinates wrong | Round-trip test in hour 2 | -| No LiDAR phone | Planes-only mode: passage width and headroom become UNSURE. Borrow a Pro for the demo | -| AR drift | ~2 cm/s; attach results to the meter's anchor; show the tape-measure numbers | -| Driveway rule | Tapped driveway edge; automatic detection is future work | -| Sunlight washes out LiDAR | Scan the real wall at midday in hours 0–4 | diff --git a/docs/03-stack-research.md b/docs/03-stack-research.md deleted file mode 100644 index 7b8451326..000000000 --- a/docs/03-stack-research.md +++ /dev/null @@ -1,240 +0,0 @@ -# Stack research: libraries, open-source projects, how they fit - -This came from five parallel research agents on 2026-09-25 using live web searches. Anything marked -**UNVERIFIED** couldn't be confirmed. Licenses are what each repo or model card said on that date, so re-check them before shipping a product. - -## Data flow, with the tool at each step - -``` -PHONE (Swift) - ARKit: gravity alignment, wall + ground detection, LiDAR mesh, depth ← Apple "Visualizing and interacting with a reconstructed scene" - Frame recorder (position + lens data per frame) ← fork Stray Scanner (MIT) - Which frames to keep ← lift from arkit-3dgs-scanner (Apache-2.0) - Mesh → .ply ← write by hand - Meter close-up checked on the phone ← Apple Vision RecognizeTextRequest + VNDetectBarcodesRequest - GPS + one compass reading away from the meter - │ zip → foreground upload -SERVER (Python, FastAPI) - Read the capture ─────────── Open3D (read .ply, RaycastingScene, segment_plane) + numpy - Map context ──────────────── Census geocoder → Overture building outline (DuckDB) → osmnx nearest road - → street-facing wall; pools/driveways from OSM + samgeo on StratMap/NAIP - Object detection ─────────── Gemini boxes → SAM 2 tightens them → supervision merges them → cast onto the wall - Recognition ──────────────── recognition prompts on Gemini; zxing-cpp double-checks the barcode - Placement code ───────────── shapely + numpy sweep, rules.yaml (each rule cites its code) - Output ───────────────────── JSON + drawsvg 2D site plan - Debugging ────────────────── Rerun (its ARKitScenes example logs exactly our data) - Pitch baseline ───────────── Depth Anything 3 metric (Apache-2.0) on plain photos - │ result, positioned relative to the meter -PHONE: RealityKit box attached to the meter's anchor + cable line -``` - -## Why each piece matters - -| Piece | Rule(s) it serves | Why this tool, and what we're skipping | -|---|---|---| -| ARKit + LiDAR mesh | Passage width, headroom, gaps between openings, cable length | Real measurements (~±6–8 cm on walls, ~5 m range) instead of estimates by eye. Skip photogrammetry. | -| Guided walk to both ends | "please send more photos" follow-ups | Makes "unseen space" impossible | -| One continuous session | same wall? / panel behind meter | Shared coordinates; nothing pieced together from unordered photos | -| Stray Scanner fork | everything downstream | The only maintained MIT recorder of position + lens data + depth. No open-source app also exports the mesh | -| Open3D | passage width, headroom, wall fitting | One library for .ply + planes + batch rays | -| Gemini boxes | windows, doors, garage, AC, gas meter, vents | Best zero-shot box accuracy found; native 0–1000 box output | -| SAM 2 | every rule that uses an object's position | Vision-model boxes are loose; a mask's bottom edge is the right point to cast | -| Apple Vision on the phone + zxing | meter number, CL320 | Instant retake; serial-number OCR is the weak spot (63% vs 97%) | -| Overture + osmnx | which wall faces the street (flag for homeowner approval of the look) | The compass alone can't tell | -| samgeo on StratMap/NAIP | pool, driveway | OSM tags them on < 5% of homes; imagery ML is allowed to run on | -| rules.yaml + citations | all rules | Each "no" cites NEC / IRC / Austin Energy / Texas Gas Service | -| Rerun | debugging | Capture + solver reasoning in one timeline | -| Depth Anything 3 | pitch | The "AR beats photos" comparison | - -## Phone capture (iOS) - -**Use:** -- Apple sample **"Visualizing and interacting with a reconstructed scene"**: RealityKit + `.meshWithClassification`, mesh raycasts, `ARMeshGeometry` extensions. Plane detection flattens the mesh where planes are found, so walls come out clean. -- Apple sample **"Displaying a point cloud using scene depth"**: sceneDepth + confidence + intrinsics → points. -- The battery model is just `MeshResource.generateBox`. Skip Object Capture. - -**Lift code from:** - -| Repo | License | Last commit | Notes | -|---|---|---|---| -| [strayrobots/scanner](https://github.com/strayrobots/scanner) | MIT | 2026-04 | HEVC RGB, 16-bit depth PNG (mm), confidence, `odometry.csv` (position + quaternion + fx fy cx cy), IMU. No mesh. **Fork this** | -| [xiongyiheng/ARKit-Scanner](https://github.com/xiongyiheng/ARKit-Scanner) | Apache-2.0 | 2026-07 | ScanNet++'s capture app; good upload-to-server reference | -| [KuoFengYuan/arkit-3dgs-scanner](https://github.com/KuoFengYuan/arkit-3dgs-scanner) | Apache-2.0 | 2026-09 | Keyframe picker by movement/rotation, COLMAP export | -| [tkuehnl/iPadLIDARScanExport](https://github.com/tkuehnl/iPadLIDARScanExport) | UNVERIFIED | 2020 | ARMeshAnchor → world → OBJ approach; read it, don't copy (it writes back into ARKit's buffer) | -| [jc211/NeRFCapture](https://github.com/jc211/NeRFCapture) | MIT | 2023 | Small, readable, stale | -| [TokyoYoshida/ExampleOfiOSLiDAR](https://github.com/TokyoYoshida/ExampleOfiOSLiDAR) | MIT | 2021 | Snippets | - -**Skip:** -- **ios_logger:** no license, 2020. -- **ARKit-Data-Logger:** poses only. -- **SwiftUI-LiDAR:** no poses. -- **Nerfstudio iOS:** no app of its own; its importers are for closed apps. -- **RoomPlan:** indoor rooms only. - -**Gotchas:** -- `capturedImage` is in the sensor's landscape orientation, and intrinsics are in its pixels. -- Depth is 256×192, so scale K to match. -- LiDAR: stand 1–3 m from the wall (~5 m max, ≥ ~0.5 m min). -- `.gravityAndHeading` needs a location permission, doesn't re-align north after the session starts, and suffers magnetometer interference. -- **App Clips:** - - Size: 15 MB on iOS 16; 100 MB on iOS 17+ if digital-invocation only. - - ARKit isn't on the list of frameworks App Clips can't use. - - **No background URLSession.** - - Location is "When In Use" only, and it resets. - - Whether sceneReconstruction works inside a clip is UNVERIFIED. -- **Android:** only ARKit supports AR Foundation meshing. ARCore Depth is weak on blank walls and has no mesh. Not for v1. - -## Geometry server - -**Use:** - -| Library | Version / license | What for | -|---|---|---| -| [Open3D](https://pypi.org/project/open3d/) | 0.20.0, MIT | `segment_plane`, `detect_planar_patches`, `o3d.t.geometry.RaycastingScene.cast_rays` | -| [shapely](https://pypi.org/project/shapely/) | 2.1.x, BSD-3 | polygon distances; vectorised functions release the GIL | -| numpy | — | 1-D intervals. No interval library needed | -| [Rerun `rerun-sdk`](https://pypi.org/project/rerun-sdk/) | 0.38, MIT/Apache | [ARKitScenes example](https://rerun.io/examples/spatial-computing/arkit_scenes) logs poses + intrinsics + RGB + depth + mesh + 3D boxes | -| [drawsvg](https://pypi.org/project/drawsvg/) | 2.4.x, MIT | site plan; PNG export needs Cairo | - -**Consider:** -- **trimesh 5.1 + embreex 4.4** (MIT/BSD): nicer mesh API, e.g. `section()` for the wall-base line. -- **polyscope:** an interactive viewer. - -**Skip:** - -| Project | Why | -|---|---| -| SpatialLM | CC-BY-NC encoder, indoor, needs CUDA | -| Meta SceneScript | CC BY-NC, Aria input; its FAQ says outdoor results are unpredictable | -| RoomFormer | indoor floorplans | -| pymeshlab | GPL-3 | -| pyransac3d | inactive | -| svgwrite | inactive | -| COLMAP / hloc / nerfstudio / gsplat | COLMAP's own tutorial says to avoid texture-less walls; no scale; slow | - -**Gotchas:** -- The ARKit mesh isn't watertight, so `contains_points` is meaningless. -- Open3D rays must be float32. -- ARKit camera coordinates are RUB (x right, y up, looking down −z), unlike OpenCV (UNVERIFIED from docs; the round-trip test settles it). - -## Object detection and recognition - -**Use:** -1. **Gemini API boxes:** - - Format: `box_2d: [ymin, xmin, ymax, xmax]` on 0–1000; segmentation adds a `mask` polygon. - - Set the thinking level to minimal for segmentation. - - Accuracy: RF100-VL zero-shot mAP is 13.3 for Gemini 2.5 Pro vs 1.5 for GPT-5. Gemini 3 Pro with better prompts reached 26.4 on a subset. - - Docs examples use `gemini-3.8-flash` (model ID UNVERIFIED). -2. **[SAM 2](https://github.com/facebookresearch/sam2)** (Apache-2.0), prompted with each box, to get a tight mask + ground-contact point. [Grounded-SAM-2](https://github.com/IDEA-Research/Grounded-SAM-2) already wires this up. -3. **[Qwen3-VL](https://github.com/QwenLM/Qwen3-VL)** (Apache-2.0; 2B–235B) as a local second opinion. Its coordinate scale is 0–1000 or 0–1 depending on which doc you read, so test it. -4. **[supervision](https://supervision.roboflow.com/)** (MIT) for NMS/NMM, `InferenceSlicer` and drawing. -5. **OCR:** Apple Vision `RecognizeTextRequest` + `VNDetectBarcodesRequest` on the phone. On the server, the vision model reads the text, cross-checked with a `CL\s?(200|320)` regex and [zxing-cpp](https://github.com/zxing-cpp/zxing-cpp) (Apache-2.0). -6. **Driveways:** Gemini segmentation + SAM 2 for the hackathon. Mapillary-Vistas Mask2Former has a driveway class, but its weights are non-commercial. - -**Detector baselines:** - -| Model | License | Notes | -|---|---|---| -| Grounding DINO | Apache-2.0 | | -| OWLv2 | Apache-2.0 | | -| Florence-2 | MIT | 0.77B; also does OCR | -| RF-DETR | Apache-2.0 for Nano to Large | fine-tune target for autodistill | - -My judgement on these: fine on window, door, garage door and fence; weaker on gas meter and AC; poor on window well. - -**Depth from a single photo** (the no-AR baseline): - -| Model | License | Notes | -|---|---|---| -| [Depth Anything 3](https://github.com/ByteDance-Seed/Depth-Anything-3) `DA3METRIC-LARGE` | Apache-2.0 (Giant/Nested are NC) | Needs focal length in pixels: `depth = focal*out/300`. ETH3D δ1 0.917 | -| [MoGe-2](https://github.com/microsoft/MoGe) | code MIT; weights UNVERIFIED | δ1 0.908, ~60 ms, also gives normals | -| Depth Pro | Apple personal-use grant | commercial use unclear | -| UniDepthV2 | CC BY-NC | skip | -| Metric3D v2 | license conflicts between files | skip | - -**Datasets:** -- Roboflow Universe gas meters ([402 images](https://universe.roboflow.com/proba-vwwtl/gas-meter-recognition), [1.4k images](https://universe.roboflow.com/gas-meter-zbuni/gas-meter-g7kh6)). -- Electric meters ([wattwise](https://universe.roboflow.com/wattwise/electric-meter-wzeeg-zk5fd), [1,943 images for reading dials](https://universe.roboflow.com/abhinav-kumar-do8z1/utility-meter-reading-dataset-for-automatic-reading-yolo-z0e1h)). -- Small AC sets (whether they show outdoor condensers is UNVERIFIED). -- Open Images V7 has Window and Door. -- Facade datasets (CMP, ECP, eTRIMS) are European and a poor fit. -- **No public data for window wells or breaker panels.** Label your own. [autodistill](https://github.com/autodistill/autodistill) (Apache-2.0) can auto-label. - -**Skip:** - -| Option | Why | -|---|---| -| GPT for boxes | 1.5 mAP | -| Claude for boxes | its docs call the coordinates approximate and ask for absolute pixels | -| Grounding DINO 1.5 / DINO-X | API only | -| YOLO-World / YOLOE / Ultralytics | GPL / AGPL | -| Molmo 2 | non-commercial training data | -| SAM 3 | custom license, CUDA 12.6+ | - -**Gotchas:** -- Gemini returns [y, x]. Apply EXIF rotation before inference **and** before casting. -- Depth Anything 3 needs the focal length in pixels. -- Zinsco breakers can read "Sylvania" or "Magnetrip", and FPE ones "Stab-Lok". Put those strings in the prompt. - -## Maps and aerial photos - -**Use:** -- **[Overture Maps](https://docs.overturemaps.org/getting-data/duckdb/)** via DuckDB: `read_parquet('s3://overturemaps-us-west-2/release//theme=buildings/type=building/*')` + a bbox. Roads carry `subclass=driveway|parking_aisle|alley`. Buildings and transportation are **ODbL**. -- **[osmnx](https://osmnx.readthedocs.io/)**: `ox.graph_from_point(pt, dist=150, network_type="drive")`, then `ox.distance.nearest_edges(..., return_dist=True)` on a projected graph. -- **NAIP** (public domain) via the Planetary Computer STAC `naip`. Pflugerville TX: 2022 at 0.6 m. Naperville IL: 2023 at 0.3 m. At 0.6 m a 3–5 ft driveway buffer is only ~2 pixels. -- **[TxGIO StratMap / CAPCOG](https://geographic.texas.gov/stratmap/index.html)** orthoimagery: 6″ inside Austin, 12″ elsewhere, public domain. The best free imagery ML can run on in Texas. -- **[segment-geospatial (samgeo)](https://samgeo.gishub.org/examples/text_swimming_pools/)**: text-prompted SAM ("swimming pool", "driveway"), no training. -- **Census Geocoder** (`onelineaddress`, `Public_AR_Current`): free, no key. Nominatim fallback (1 request/s, identifying User-Agent, no bulk use). -- **Google Solar API: a maybe.** Its §20.1 allows use "to determine the feasibility of installing energy systems". The dataLayers DSM and mask are 0.1 m. Whether running inference on it conflicts with §3.2.3(c) is UNVERIFIED, so ask before relying on it. - -**Licensing traps:** - -| Source | Trap | -|---|---| -| **Google Maps ToS §3.2.3(c)(vii)** | Bans using Google Maps content "to train, test, validate or fine-tune" ML models, and bans tracing building outlines from satellite imagery | -| **Mapbox §1.5(ii)** | Bans using it to "train, operate or improve" ML ("operate" includes running inference) | -| **Esri World Imagery** | Restricted (UNVERIFIED) | -| **ODbL share-alike** | Applies if you publish a database derived from it; internal use with attribution is fine | -| **Microsoft Global ML Footprints** | CDLA-Permissive-2.0 (the older US repo is ODbL) | -| **Mapillary** | CC BY-SA | - -**Recipe:** -1. Geocode the address. -2. Get the Overture building within ±60 m (the one containing the point, or the nearest). Project to UTM 14N (Texas) or 16N (Illinois). -3. Get the osmnx drive network, leaving out `service=driveway` edges. -4. For each outline edge, score (cosine between its outward direction and the vector to the nearest road) × 1/distance. The best is the street-facing wall; flag two edges on corner lots. -5. Driveways = OSM/Overture driveway lines buffered 1.5 m, `amenity=parking` polygons and samgeo masks. -6. Pools = `leisure=swimming_pool` plus samgeo. - -OSM coverage (Overpass, 2026-09-25): - -| Area | Houses | Driveways | Pools | -|---|---|---|---| -| Pflugerville | 14,675 | 149 | 102 | -| Round Rock | 9,223 | 476 | 170 | -| Austin | 302k buildings | 6,029 | 3,339 | - -**AR ↔ map registration:** a 2D rigid fit. Snap the AR walls to 90°, sweep yaw (±20° around the compass reading if its accuracy is ≤ 20°, otherwise all 4 quadrants), and fit translation by least squares against the outline edges. `ARGeoTrackingConfiguration` coverage in Austin is UNVERIFIED. - -**Skip:** -- Google Static / Tiles / Street View and Mapbox/Esri imagery for ML. -- Nearmap: paid, no hackathon tier. -- EagleView: a 30-day sandbox exists, so maybe one email, but don't block on it. -- TorchGeo training: no time. -- Old pool CNNs: zero-shot SAM beats them. - -## Test first (hours 0–4) - -| Test | If it fails | -|---|---| -| LiDAR mesh of a real siding wall at midday sun | Planes only; passage width and headroom become UNSURE | -| Tap → cast into a saved frame → correct pixel | Fix image orientation before building detection | -| Gemini boxes on ~30 real photos (gas meter, AC, window well) | Tap-to-mark carries the demo | -| Compass accuracy at the meter | Use the building outline + road alone | - -## Fine for the hackathon, check before it becomes a product - -- **Not for commercial use:** Mapillary-Vistas models, Molmo 2 and Depth Anything V2's larger models. -- **Grey areas:** Depth Pro (Apple's personal-use license) and inference on Google Solar imagery. -- **ODbL share-alike:** if you publish a derived database. -- **GPL/AGPL detectors:** YOLO-World, YOLOE, Ultralytics. diff --git a/docs/04-prior-art-and-codes.md b/docs/04-prior-art-and-codes.md index 5c0066287..a3aad477f 100644 --- a/docs/04-prior-art-and-codes.md +++ b/docs/04-prior-art-and-codes.md @@ -1,78 +1,78 @@ -# Prior art, what Base does publicly, and code citations - -Researched 2026-09-25. UNVERIFIED items are marked. - -## What Base does today (public) - -- **Photo checklist, 9 photos** ([help](https://help.basepowercompany.com/en/articles/10280641)): - - Meter close-up (number legible), meter surroundings from 10+ steps back, area to the left, area to the right, the adjacent wall corner to corner, behind the fence. - - Main breaker box, main disconnect (amperage readable), breaker box surroundings. - - The "Base engineering and installation teams" review them and reach out within 2 days if more is needed. Nothing public says whether AI is involved. -- **Placement rules** ([help](https://help.basepowercompany.com/en/articles/10280705)): - - Footprint 3 × 3 ft, ~36″ tall. Within 20 ft of the meter and within 1 ft of the wall. - - 3 ft from gas meters, fences, AC units and other batteries. Not in front of meters, panels, solar equipment or windows. - - Meter ≤ 6 ft up. Meter and breaker box share a wall; the box can't be in a closet. - - 30 × 36″ working space in front of the meter and panel. - - Transfer switch ~13″ wide with 30″ clearance (spec pages: 3 ft of wall for the transfer switch, then 3 × 3 ft per battery). - - Main breaker 100–200 A (150–200 A in Austin). Dual batteries or solar need a 200 A panel. - - No concrete pads needed (integrated base). -- **Hardware:** - - 25/50 kWh ground units, plus **Base Core**: 39.2 kWh, **39.5 × 30.68 × 22 in**, 11 kW inverter, UL-listed ([pv-magazine](https://pv-magazine-usa.com/2026/08/04/base-power-launches-39-2-kwh-u-s-made-base-core-home-battery-secures-1-billion-in-new-funding/)). - - Wall-mounted transfer switch + battery disconnect. No public mention of a meter collar. -- **Who reviews:** a Deployment Site Designer job post (removed Nov 2025) describes people reviewing customer site photos by hand, using CAD. -- **Operations:** heavy lifting is split from electrical work so crews can do "20 homes in a day". +# Public rules, code citations, prior art and licenses -## How others do it +This was researched on 2026-09-25 from public sources, and Esri's imagery terms were rechecked on 2026-09-26. UNVERIFIED marks a claim nobody could confirm. Licenses change, so recheck one before building a product on it. -- **Tesla Powerwall:** guided self-survey on the phone (~30 min), photo upload, remote design. -- **Qmerit (EV chargers):** photo upload + "Panel Insights" AI (with Schneider Electric) reading breaker spaces and capacity from one panel photo. - The closest precedent for using AI only for recognition ([link](https://qmerit.com/news/qmerit-deploying-ai-for-faster-safer-more-accurate-estimates-for-home-ev-charging-installations/)). -- **Aurora Solar:** Site Surveys API, beta ([docs](https://docs.aurorasolar.com/reference/site-surveys)). -- **Scanifly:** drone photogrammetry. -- **SolarAPP+ (NREL):** instant automated code checks for solar and storage permits. The precedent for rules engines over judgment calls. -- **Heat pumps:** - - Aira, Mitsubishi Ecodan and Alpha Innotec have AR placement, but it's visual only with no clearance checks. - - **Fraunhofer ISE "Heat Pump PlanAR"** (reported 2026-09-23) scans indoor boiler rooms and optimises placement. The nearest analogue, but indoors. +## What Base does today -## Products with APIs +- **Photo checklist.** Base's [help page](https://help.basepowercompany.com/en/articles/10280641) asks for 9 photos. Four cover the meter: a close-up with the number legible, its surroundings from 10 or more steps back, and the areas to its left and right. The others show the adjacent wall corner to corner, the area behind the fence, the main breaker box, the main disconnect with its amperage readable, and the breaker box's surroundings. Base's engineering and installation teams review them and reach out within 2 days if they need more. Nothing public says whether AI is involved. +- **Placement rules.** The [help page](https://help.basepowercompany.com/en/articles/10280705) gives a 3 × 3 ft footprint about 36 in tall. The battery sits within 20 ft of the meter and 1 ft of the wall, and 3 ft from gas meters, fences, AC units and other batteries. It can't stand in front of meters, panels, solar equipment or windows. +- **Meter and panel.** The meter is at most 6 ft up, and the meter and breaker box share a wall. The box can't be in a closet. Both need 30 × 36 in of working space in front. The main breaker is 100 to 200 A, or 150 to 200 A in Austin, and two batteries or solar need a 200 A panel. +- **Transfer switch.** It is about 13 in wide with 30 in of clearance. Base's spec pages ask for 3 ft of wall for the switch, then 3 × 3 ft per battery. +- **Hardware.** Base sells 25 and 50 kWh ground units and Base Core: 39.2 kWh, 39.5 × 30.68 × 22 in, an 11 kW inverter, and a UL listing ([pv magazine](https://pv-magazine-usa.com/2026/08/04/base-power-launches-39-2-kwh-u-s-made-base-core-home-battery-secures-1-billion-in-new-funding/)). The install adds a wall-mounted transfer switch and a battery disconnect. Nothing public mentions a meter collar. The base is integrated, so no concrete pad is needed. +- **Review.** A Deployment Site Designer job post, removed in November 2025, describes people reviewing customer site photos by hand in CAD. Heavy lifting is split from electrical work so crews can do "20 homes in a day". -| Product | What | Notes | -|---|---|---| -| **Hover** | measured exterior 3D model (walls, windows, openings) from phone photos | Real JSON API ([docs](https://developers.hover.to/reference/measurements-and-deliverables)); turnaround likely hours; cross-check later, not the demo | -| EagleView | aerial, plus wall/window/door measurements (2026-03) | Enterprise | -| Polycam | 3D capture | API Enterprise-only | -| Matterport | 3D capture | sandbox is demo models only | -| magicplan | RoomPlan-based | indoor | -| Canvas | 3D capture | says it drifts outside and gives no exterior elevations | +## Public rule values -## Prior art +The demo's `server/rules.yaml` (PR #11) uses these values, except the 1 ft wall distance, because the solver places the battery flush against the wall. Base's own values are private and stay out of tracked files. -- iPhone 12 Pro Max LiDAR on facades: ~±6–8 cm against a total station. -- iPad Pro LiDAR range: ~5 m. -- AR layout research (OCTO+, CHI 2023): places virtual content only, no code compliance. -- Meter OCR (YOLOv5 study): counters read 97%, serial numbers only 63%. +| Rule | Value | Source | +| --- | --- | --- | +| Cable run from the meter | within 20 ft | Base's help page | +| Distance from the wall | within 1 ft | Base's help page | +| Gas meter or regulator | 3 ft | Base's help page, Austin Energy §1.9, Texas Gas Service | +| AC units, fences, other batteries | 3 ft | Base's help page | +| Doors and windows entering the home | 3 ft | IRC R328.4 | +| Working space at the meter and panel | 30 in wide, 36 in deep, 6.5 ft headroom | NEC 110.26 | +| Headroom over the battery | 6.5 ft | NEC 110.26, applied to the battery as a demo choice | +| Driveway, pool | no public value | demo placeholders of 5 ft and 10 ft | -## Why the rules exist (citations for `rules.yaml`) +## Code citations | Code | What it says | -|---|---| -| **NEC 110.26** ([ICC](https://codes.iccsafe.org/s/ISEP2021P1/national-electrical-code-nec-solar-provisions/ISEP2021P1-NEC-Sec110.26)) | Working space 30″ wide (or the equipment width) × 36″ deep × 6.5 ft headroom; no storage in it. Base's page says "30 in high × 36 in wide", probably a garbled version | -| **Austin Energy Design Criteria (Dec 12, 2023)** ([PDF](https://austinenergy.com/-/media/project/websites/austinenergy/contractors/designcriteriamanual.pdf)) | §1.9.2: 30″ wide / 36″ deep / 6′6″ headroom at meters; socket center 30–72″ above ground; ≥ 1 ft from doors and windows. No meter within a 3 ft radius of gas meters, regulators or relief valves. §1.12: generation enclosures keep the 3 ft radius | -| **Texas Gas Service Meter Setting Requirements** ([PDF](https://www.texasgasservice.com/media/tgs/constructionservices/metersettingrequirements_tgs.pdf)) | Electric meters and outlets 3 ft from the regulator relief vent; 3 ft from operable doors and windows | -| **IRC R328 (2021, from NFPA 855)** ([ICC](https://codes.iccsafe.org/s/IRC2021P2/part-iii-building-planning-and-construction/IRC2021P2-Pt03-Ch03-SecR328)) | UL 9540 listed; outdoors or on exterior walls ≥ 3 ft from doors and windows; 3 ft spacing between batteries unless UL 9540A testing allows closer | -| **NEC 706.15** | Battery disconnect within sight, and within 10 ft or lockable | -| **NEC 230.85** | Outdoor emergency disconnect (moved into 230.70 in the 2026 NEC) | -| ComEd-specific opening rules | UNVERIFIED. Cite Austin Energy / Texas Gas Service instead | - -## Pitch angle: what's new - -1. **Nobody does the whole loop outdoors.** Metric AR capture, a code-cited solver and an AR preview. Heat-pump AR tools are visual only, and Fraunhofer's solver is indoors. -2. **Every clearance becomes a traceable check.** "Fails AE §1.9.2: 3 ft from gas" instead of a vision model's judgment. SolarAPP+ for battery siting. -3. **Enforced coverage fixes Base's stated photo failures:** obstructed or too-zoomed views, unreadable meter numbers. -4. **Recognition stays with AI** (Qmerit's model); **measuring stays deterministic.** - -Be honest about the limits: -- LiDAR reaches ~5 m and is accurate to ~±6–8 cm, so add margins. -- LiDAR is only on Pro iPhones. -- Unit sizes differ between Base models, so keep them as configuration. -- Some calls stay human: how it looks, and judgment calls a camera can't settle. +| --- | --- | +| **NEC 110.26** ([ICC](https://codes.iccsafe.org/s/ISEP2021P1/national-electrical-code-nec-solar-provisions/ISEP2021P1-NEC-Sec110.26)) | Working space 30 in wide, or the equipment's width, by 36 in deep, with 6.5 ft of headroom and no storage in it. Base's page says "30 in high × 36 in wide", probably a garbled version. | +| **Austin Energy Design Criteria, Dec 12 2023** ([PDF](https://austinenergy.com/-/media/project/websites/austinenergy/contractors/designcriteriamanual.pdf)) | §1.9.2 asks for 30 in wide, 36 in deep and 6 ft 6 in of headroom at meters, a socket center 30 to 72 in above ground, and at least 1 ft from doors and windows. No meter may sit within a 3 ft radius of gas meters, regulators or relief valves. §1.12 keeps that 3 ft radius for generation enclosures. | +| **Texas Gas Service Meter Setting Requirements** ([PDF](https://www.texasgasservice.com/media/tgs/constructionservices/metersettingrequirements_tgs.pdf)) | Electric meters and outlets stay 3 ft from the regulator relief vent and 3 ft from operable doors and windows. | +| **IRC R328, 2021, from NFPA 855** ([ICC](https://codes.iccsafe.org/s/IRC2021P2/part-iii-building-planning-and-construction/IRC2021P2-Pt03-Ch03-SecR328)) | Batteries are UL 9540 listed and sit outdoors or on exterior walls at least 3 ft from doors and windows. Batteries stay 3 ft apart unless UL 9540A testing allows closer. | +| **NEC 706.15** | The battery disconnect is within sight, and within 10 ft or lockable. | +| **NEC 230.85** | An outdoor emergency disconnect. The 2026 NEC moves it into 230.70. | +| ComEd opening rules | UNVERIFIED. Cite Austin Energy or Texas Gas Service instead. | + +## How others do it + +- **Tesla Powerwall** runs a guided self-survey on the phone of about 30 minutes, then designs remotely from the uploaded photos. +- **Qmerit**, for EV chargers, pairs photo upload with Panel Insights, built with Schneider Electric, which reads breaker spaces and capacity from one panel photo ([announcement](https://qmerit.com/news/qmerit-deploying-ai-for-faster-safer-more-accurate-estimates-for-home-ev-charging-installations/)). It is the closest precedent for using AI only to recognize things. +- **SolarAPP+**, from NREL, runs instant automated code checks for solar and storage permits. It is the precedent for a rules engine in place of judgment calls. +- **Aurora Solar** has a Site Surveys API in beta ([docs](https://docs.aurorasolar.com/reference/site-surveys)). **Scanifly** uses drone photogrammetry. +- **Heat pumps.** Aira, Mitsubishi Ecodan and Alpha Innotec offer AR placement that is visual only, with no clearance checks. Fraunhofer ISE's Heat Pump PlanAR, reported on 2026-09-23, scans indoor boiler rooms and optimizes placement. It is the nearest analogue, but it works indoors. +- **3D capture products.** Hover builds a measured exterior 3D model from phone photos and has a JSON API ([docs](https://developers.hover.to/reference/measurements-and-deliverables)), with a turnaround likely measured in hours, so it suits a later cross-check. EagleView sells wall, window and door measurements to enterprises. Polycam's API and Matterport's sandbox are closed to us, magicplan works indoors, and Canvas says it drifts outside. + +Prior measurements: iPhone 12 Pro Max LiDAR on facades lands within about ±6 to 8 cm of a total station, and iPad Pro LiDAR reaches about 5 m. A YOLOv5 meter-reading study read the counters 97% of the time and serial numbers only 63%. + +Nobody else runs the whole loop outdoors, from metric AR capture through a solver that cites a code for each clearance to an AR preview. The heat-pump AR tools are visual only, and Fraunhofer's solver works indoors. + +## Licenses and terms + +**Aerial imagery.** Use StratMap or NAIP. + +| Source | Terms | +| --- | --- | +| Google Maps, Street View | ToS §3.2.3(c)(vii) bans using the content to "train, test, validate or fine-tune" ML, and bans tracing building outlines from its imagery. The Solar API's §20.1 allows use "to determine the feasibility of installing energy systems". Whether running inference on it conflicts with §3.2.3(c) is UNVERIFIED. | +| Mapbox | §1.5(ii) bans using it to "train, operate or improve" ML, and operating includes inference. | +| Esri World Imagery | Automated extraction must start from an exported tile package, stay inside ArcGIS, and keep derived results non-commercial ([Esri](https://www.esri.com/arcgis-blog/products/arcgis-living-atlas/imagery/learn-to-use-ai-to-extract-information-from-world-imagery)). | +| TxGIO StratMap orthoimagery ([site](https://geographic.texas.gov/stratmap/index.html)) | Public domain, with 6 in pixels inside Austin and 12 in elsewhere. | +| NAIP, via Planetary Computer | Public domain, at 0.6 m in Pflugerville (2022) and 0.3 m in Naperville (2023). At 0.6 m a 3 to 5 ft driveway buffer is about 2 pixels. | +| Overture buildings and transportation, OSM | ODbL, whose share-alike applies only if you publish a derived database. OSM tags driveways and pools on under 5% of homes. On 2026-09-25 Pflugerville had 14,675 houses, 149 driveways and 102 pools. Round Rock had 9,223 houses, 476 driveways and 170 pools. Austin had 302k buildings, 6,029 driveways and 3,339 pools. | + +**Models.** Check a model's weights license separately from its code license. + +| Status | Models | +| --- | --- | +| Noncommercial | MapAnything's default checkpoint (use `facebook/map-anything-apache` instead), Depth Anything 3 Giant and Nested (Metric Large, Base and Small are Apache-2.0), VGGT-Ω, UniDepthV2, Mapillary-Vistas Mask2Former weights, Molmo 2 | +| Restricted or unclear | Depth Pro (Apple personal-use grant), SAM 3 (custom license, gated weights), HY-World 2.0 (custom community license), Metric3D v2 (its files disagree), MoGe-2 weights (UNVERIFIED, though the code is MIT) | +| Copyleft | YOLO-World, YOLOE and Ultralytics (GPL or AGPL), pymeshlab (GPL-3) | +| Permissive | SAM 2, Grounding DINO, OWLv2, Qwen3-VL and zxing-cpp (Apache-2.0), Florence-2, supervision, Open3D and Stray Scanner (MIT), shapely (BSD-3) | + +For recognition boxes, Gemini 2.5 Pro scored 13.3 zero-shot mAP on RF100-VL, against 1.5 for GPT-5. + +**Datasets.** The evals use ADVIO (CC BY-NC 4.0), ETH3D (CC BY-NC-SA 4.0) and MARViN (no license stated) only to measure accuracy, and never redistribute them. Get permission before any commercial use. For training data, Roboflow Universe has gas meter sets of [402](https://universe.roboflow.com/proba-vwwtl/gas-meter-recognition) and [1.4k](https://universe.roboflow.com/gas-meter-zbuni/gas-meter-g7kh6) images and electric meter sets ([wattwise](https://universe.roboflow.com/wattwise/electric-meter-wzeeg-zk5fd), [1,943 images](https://universe.roboflow.com/abhinav-kumar-do8z1/utility-meter-reading-dataset-for-automatic-reading-yolo-z0e1h)). Open Images V7 labels windows and doors. No public data exists for window wells, and open photos of US breaker panels are scarce: PR #17 found 7 against the 40 its test needs. diff --git a/docs/05-live-guided-survey-hld.md b/docs/05-live-guided-survey-hld.md index c7ae5a3b9..b07dc96c8 100644 --- a/docs/05-live-guided-survey-hld.md +++ b/docs/05-live-guided-survey-hld.md @@ -1,8 +1,8 @@ -# Base Power — Live Guided Site Survey +# Base Power: Live Guided Site Survey **High-Level Design · v0.1 · September 25, 2026** -**Status:** Public architecture proposal for review; no implementation or existing lane decisions are changed by this document. +**Status:** Public architecture proposal for review; this document changes no implementation or existing decision. **Audience:** Product, engineering, ML, and Base survey reviewers. @@ -14,7 +14,7 @@ This document describes the intended system, not an implemented or benchmarked p Base customers currently supply site-survey photographs used to assess electrical equipment and possible battery installations. The proposed experience keeps the camera open while the app guides the customer, chooses useful photographs, and develops a spatial understanding of the relevant exterior. -The repository baseline is documented in [the overview](00-overview.md), [feature map](01-feature-map.md), and [implementation plan](02-implementation-plan.md). This HLD records the proposed continuous capture and guidance experience, with public technical resources. Business-rule values and their authority are maintained separately through the approved rules configuration. This public document contains no team-only criteria material. +The repository baseline is documented in [the overview](00-overview.md) and the component READMEs. This HLD records the proposed continuous capture and guidance experience, with public technical resources. Business-rule values and their authority are maintained separately through the approved rules configuration. This public document contains no team-only criteria material. | Status | Decision | |---|---| @@ -22,15 +22,16 @@ The repository baseline is documented in [the overview](00-overview.md), [featur | **Product alignment** | Analyze the live view to understand relevant objects, observed areas, and missing evidence. | | **Product alignment** | Show the customer where to move and where to aim, updating guidance during capture. | | **Product alignment** | Upload selected images during the session and progressively update reconstruction and assessment. | -| **Proposed** | Use native AR for stable spatial guidance, with LiDAR optional and a simpler camera-guidance fallback. | +| **Proposed** | Use native AR for stable spatial guidance, with a simpler camera-guidance fallback. | | **Proposed** | Keep immediate tracking and feedback on the phone; run heavier analysis and reconstruction asynchronously. | | **Experiment proposal** | Compare periodic MapAnything and stateful LingBot-Map reconstruction alongside the native AR geometry baseline. | -| **Open** | Resolve the documented native Swift/Expo integration conflict; select representative iPhones, GPU environment, and acceptable latency/cost. | +| **Decided** | Native Swift capture in `ios/`; LiDAR optional. | +| **Open** | Select representative iPhones, GPU environment, and acceptable latency/cost. | | **Open** | Confirm the versioned rules configuration and aggregation contract before expanding automated decisions. | ### Relationship to the existing repository plan -The current plan targets an iPhone demo with native Swift, ARKit/RealityKit, and LiDAR geometry where supported. It uses native AR measurements and a deterministic server-side placement solver, retaining tap-to-mark capture as a dependable fallback. Its Swift-versus-Expo integration question remains open in [AGENTS.md](../AGENTS.md). The existing implementation plan remains the baseline until the team adopts a change. +The current plan targets an iPhone demo with native Swift, ARKit/RealityKit, and LiDAR geometry where supported. It uses native AR measurements and a deterministic server-side placement solver, retaining tap-to-mark capture as a dependable fallback. Capture is native Swift in `ios/`, and LiDAR is optional. The overview remains the baseline until the team adopts a change. | Existing baseline | Proposal in this HLD | |---|---| @@ -41,7 +42,7 @@ The current plan targets an iPhone demo with native Swift, ARKit/RealityKit, and | Gemini detection, SAM 2 segmentation, and native Apple Vision checks in the research plan | Treat newer OCR/segmentation models as benchmark candidates, not silent replacements. | | iPhone demo | Keep Android and browser AR references as future portability research, outside the initial demo commitment. | -The evidence record below extends the existing `scene.json` concept rather than replacing its contract. Proposed metadata additions require coordination with the lane owners. Preserve its existing units at interfaces with explicit conversions, and keep AR results relative to the meter anchor using gravity world alignment. +The evidence record below extends the existing `scene.json` concept rather than replacing its contract. Proposed metadata additions require coordination with the owning team. Preserve its existing units at interfaces with explicit conversions, and keep AR results relative to the meter anchor using gravity world alignment. ### Objectives @@ -297,7 +298,7 @@ Perception extracts observable facts. Geometry supplies measured or estimated sp | Responsibility | Contract | |---|---| -| Rule configuration | Keep every clearance and equipment dimension in `rules.yaml`, with the required code citation and applicability metadata; never hardcode them in capture logic. | +| Rule configuration | Keep every clearance and equipment dimension in `rules.yaml`, with its source (a code citation, Base's public help page, or a labeled demo placeholder) and applicability metadata; never hardcode them in capture logic. | | Perception | Associate recognized objects and text with original images and an uncertainty estimate. | | Geometry | Keep coordinate frame, units, scale provenance, error bounds, and evidence references with each measurement. | | Solver | Evaluate the configured rules using the repository's wall-line and polygon approach; expose which observations support each check. | @@ -305,7 +306,7 @@ Perception extracts observable facts. Geometry supplies measured or estimated sp | Human judgment | Preserve a review path for uncertain measurements, interpretation, or unresolved configuration. | | Aggregation | Use the agreed API contract to combine findings; a successful individual check does not establish overall completion. | -The public-source starting points and unresolved values are recorded in [the feature-map rules table](01-feature-map.md#rules-table-c1-confirm-these) and [public source research](04-prior-art-and-codes.md). This HLD introduces no new rule values and does not resolve disputed requirements. Team-only material stays outside tracked files. +The public-source starting points and unresolved values are recorded in [public rule values](04-prior-art-and-codes.md#public-rule-values) and the overview's [open questions](00-overview.md#open-questions). This HLD introduces no new rule values and does not resolve disputed requirements. Team-only material stays outside tracked files. ### Decision and recapture flow @@ -344,7 +345,7 @@ When measurement uncertainty crosses a configured threshold, request a useful ad | [Unity AR Foundation](https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@6.1/manual/index.html) | Shared AR interfaces backed by platform provider plug-ins | Alternative if the team prefers Unity and cross-platform development; does not remove native capability differences. | | [Browser video frame callbacks](https://developer.mozilla.org/en-US/docs/Web/API/HTMLVideoElement/requestVideoFrameCallback) and [OpenCV.js camera processing](https://docs.opencv.org/4.11.0/dd/d00/tutorial_js_video_display.html) | Browser frame analysis and image processing | Useful fallback/prototype for camera coaching. A browser camera feed alone does not provide a persistent AR world map. | -**Platform scope:** preserve the iPhone target and resolve the native Swift/Expo boundary before capture implementation. The proposed direction is native AR capture, with camera-and-motion and optional-depth modes evaluated on actual test devices. Android and browser references are future options. Keep the backend independent of the mobile framework. +**Platform scope:** preserve the iPhone target. Capture is native Swift in `ios/`. The proposed direction is native AR capture, with camera-and-motion and optional-depth modes evaluated on actual test devices. Android and browser references are future options. Keep the backend independent of the mobile framework. ### Perception, reconstruction, and review @@ -402,17 +403,17 @@ Authenticate survey access and isolate sessions. Store original photos separatel ## 10. Hackathon implementation plan -### Phase 1 — Prove the closed loop +### Phase 1: Prove the closed loop Use one target device, a meter, and one candidate battery area. Demonstrate automatic capture, one real evidence gap, a targeted corrective instruction, a second capture, and an updated explained finding. Include a visible case where additional evidence changes the initial assessment. Use confirmed public-source rules or clearly labeled synthetic test configuration. Unresolved rule values remain configuration gaps; do not present a synthetic demonstration as installation approval. Provide a basic review screen with original images and the reason behind each result. -### Phase 2 — Add progressive spatial understanding +### Phase 2: Add progressive spatial understanding Integrate periodic reconstruction, coordinate alignment, observed-surface coverage, and a virtual battery footprint. Compare ordinary camera-and-motion capture with optional depth input on the same scene. Preserve image evidence when geometry fails. -### Phase 3 — Evaluate streaming and broader coverage +### Phase 3: Evaluate streaming and broader coverage Test LingBot-Map behind the same reconstruction interface. Expand to adjacent walls, interrupted sessions, panel tasks, and representative phones. Adopt streaming only if its measured benefit exceeds integration, compute, and reliability costs. @@ -442,7 +443,7 @@ Measure incorrect approvals and rejections separately, exact meter-number accura ## 12. Decisions needed before implementation hardens 1. **Policy:** Which publicly justified rules configuration is confirmed, and how does the result contract combine findings? -2. **Device scope:** How will native capture integrate with the app shell, and which iPhones represent actual customers? +2. **Device scope:** Which iPhones represent actual customers? 3. **Measurement standard:** What scale anchor and error tolerance are acceptable near each decision boundary? 4. **Deployment:** What GPU, connectivity, latency, and per-survey cost budget are available? 5. **Review handoff:** What evidence package and result format should Base’s existing review process receive? diff --git a/docs/eli5.html b/docs/eli5.html deleted file mode 100644 index aa86f6ac6..000000000 --- a/docs/eli5.html +++ /dev/null @@ -1,609 +0,0 @@ - - - - - -Where does the battery go? — explained like you're 5 - - - - -
-
-

Where does the battery go? 🔋🏠

-

Some families get a big battery that sits outside, next to their house. It keeps the lights on when the power goes out. - But it can't go just anywhere. Here's how our app finds the perfect spot.

- -
-
- - -
-
-
- THE PROBLEM -

Photos can't measure 📸🤷

-

Right now, families send in photos. But a photo is flat! It can't tell whether the gap between two windows is as big as a bathtub - or as small as a lunchbox. And if someone forgets to photograph part of the wall, nobody knows what's there.

-

For grown-ups: a single photo has no real scale, so distances get estimated by eye. Borderline cases end up with a human reviewer, - and any wall nobody photographed turns into a "please send more photos" follow-up.

-
-
- - - - - - - - - - - - - ?? ft - "is this 3 feet or 8 feet?" 🤔 - -
-
-
- - -
-
- STEP 1 -

The magic measuring phone 📱✨

-

The family holds up their phone and walks along the wall, like playing "follow the wall." The phone counts how far it went, like - counting footsteps. Fancy phones even shoot out invisible laser dots to feel how far things are, the way a bat uses sound! The phone - says "keep going!" until it has seen the wall all the way to the end, both ways.

-
- - - - - - - - - - - - - - - end ✓ - - end ✓ - - - - - - - 👟 - - -
-
Wall seen
- -
-

For grown-ups: ARKit tracks the phone in real feet using the camera plus the motion sensors. LiDAR iPhones add a 3D mesh - (about ±6–8 cm on walls, out to about 5 m). The capture won't finish until both wall ends are tapped, so "space you haven't seen" can't happen.

-
-
-
- - -
-
- STEP 2 -

A robot plays I Spy 🤖🔍

-

The pictures go to a helper computer. A robot looks at them and plays I Spy: "I spy a window! I spy a door! I spy the gas meter!" - Then it sticks a label on each thing, in exactly the right spot along the wall.

-
- - - - - - - - - - - window! - gas meter! - the meter! - door! - AC! - 🔍 - -

For grown-ups: Gemini draws the boxes. It had the best zero-shot box accuracy in the research, and it returns boxes natively. SAM 2 tightens each box, and supervision merges duplicates across frames. Because the phone knows where it was for every photo, each box - gets cast onto the wall and becomes "window, 2–5 ft along the wall, 3–6.5 ft up."

-
-
-
- - -
-
-
- STEP 3 -

Looking down like a bird 🐦🗺️

-

Some things are easier to see from the sky: the driveway where cars go, the swimming pool, and which side of the house faces the street. - So the helper also checks a map, like a bird looking down.

-

For grown-ups: Census geocoder → Overture building outline → osmnx nearest road → street-facing wall. - Pools and driveways come from samgeo on public-domain StratMap/NAIP aerial photos. OSM tags them on under 5% of homes. Don't run ML on Google or Mapbox imagery; - their terms forbid it.

-
-
- - - - - - - - - - pool - - - - house - garage - - - - - - - - - - keep far fromthe pool - cars drivehere! stay away - faces the street 👀 - the meter → - - 🐦 - -
-
-
- - -
-
- STEP 4 · YOUR TURN -

Slide the box and check the rulebook 🧩

-

Drag the battery along the wall. Every time it moves, the helper checks the rulebook. Green means that rule is happy. - Red means "not here!" Then press Let the computer try every spot to watch it test every single position, one tiny step at a time.

-
-
- - - - - - - - - - - - - driveway - - - - - - - - - - - - - - - - - - - - - - windowgasmeter - doorAC - - - - - - - - BASE - - - - - - - -
- - - -
-
- space workers need at the meter - the wire from the meter - a good spot - not allowed - the stripe under the wall = the computer's test results -
-
-
- -
-
    -

    For grown-ups: this is the whole solver. It slides the 31″ battery along the unrolled wall in small steps and runs every - rule as a plain check (shapely + numpy, numbers in rules.yaml). There's no AI in this step, so every "no" comes with a reason - you can cite.

    -
    -
    -
    -
    - - -
    -
    -
    - STEP 5 -

    Show and tell 👀✨

    -

    The phone draws a pretend battery right on the real wall, like a sticker floating in the air, plus a purple line showing where the wire goes. - The family looks through their phone and says, "Oh! That's where it goes!"

    -

    For grown-ups: RealityKit places a box the battery's size (Base Core is 39.5″ × 30.68″ × 22″), attached to the meter's anchor rather than - raw world coordinates, because ARKit corrects anchors as the phone moves. It's also the moment the family gets to say whether they like how it looks.

    -
    -
    - - - - - - - - - - - - - BASE - - - Here it goes! ✨ - - Looks good 👍 - - - -
    -
    -
    - - -
    -
    - WHY IT MATTERS -

    Why this is better 🏆

    -

    Instead of guessing from flat photos, the phone measures, makes sure it saw everything, and a fair rulebook picks the spot.

    -
    -
    📏

    Real feet, not guesses

    -

    A flat photo can't tell 3 feet from 8 feet. The phone measures to within a few inches.

    -
    🔁

    No "take more photos" loop

    -

    The phone won't finish until it has seen the whole wall, so families don't get asked for more pictures later.

    -
    📖

    Every answer has a reason

    -

    "Too close to the gas meter" comes from a real safety rule, not a robot's feeling. That makes it easy to trust and to double-check.

    -
    -
    -
    - - -
    -
    -
    - 🧑‍🔧 Grown-up corner: what each cartoon really is - - - - - - - - - - - - - - - - - - -
    CartoonReal toolsWhy this one
    📱 Magic measuring phoneSwift + ARKit (gravity alignment, wall and ground detection, LiDAR mesh), frame recorder forked from Stray Scanner (MIT), keyframe picker from arkit-3dgs-scannerMeasures in real feet on the phone. Photogrammetry fails on blank siding, and COLMAP's own docs say so.
    🔍 Robot I SpyGemini boxes → SAM 2 → supervision; each box cast onto the wall using where the phone wasBest zero-shot box accuracy (13.3 vs 1.5 mAP for GPT), with native box output.
    🔢 Reading the meterApple Vision text + barcode reading on the phone; zxing-cpp + Gemini on the serverInstant "retake" on the phone. Serial-number OCR is the weak spot (63% read vs 97% for the dial counters).
    🐦 Bird's-eye mapCensus geocoder, Overture building outlines via DuckDB, osmnx, samgeo on StratMap/NAIPFree, public-domain imagery that ML is allowed to run on. Google's and Mapbox's terms ban it.
    🧩 Slide the boxshapely + numpy sweep, Open3D for passage-width and headroom rays, rules.yamlUnder a second to run, easy to explain, and every rule traces to NEC 110.26, IRC R328, Austin Energy or Texas Gas Service.
    ✨ Show and tellRealityKit box attached to the meter's anchor; drawsvg 2D site plan for reviewersThe demo moment, and where the family sees and approves the spot.
    🐞 DebuggingRerun (its ARKitScenes example logs exactly our data)Camera positions, frames, mesh, detections and solver rays in one timeline.
    📊 The pitch baselineDepth Anything 3 metric (Apache-2.0) on plain photosProves AR beats photos, using a tape measure as the answer key.
    -

    The example wall is made up for illustration: window 2–5 ft, gas meter 8–9 ft, meter at 12 ft, door 24–27 ft, AC 30–33 ft, driveway from 37 ft. - The 3 ft gas, AC, door and window clearances and the 20 ft wire limit come from Base's public placement rules and building codes. The driveway distance is made up for the demo.

    -
    -
    -
    - -
    Made for the Base Power hackathon team · Drag the battery in step 4 · Works offline, one file
    - - - - diff --git a/docs/briefings/2026-09-26-morning.html b/docs/how-it-works.html similarity index 60% rename from docs/briefings/2026-09-26-morning.html rename to docs/how-it-works.html index 9e8cb28da..28927613e 100644 --- a/docs/briefings/2026-09-26-morning.html +++ b/docs/how-it-works.html @@ -4,16 +4,16 @@ -House scanning briefing: what we are building, what works, what is blocked +How house scanning works