Skip to content

fix(tests): stop the test suite from overwriting the real ~/.devorch/permissions.yaml - #35

Merged
Amanbig merged 1 commit into
mainfrom
fix/test-isolation
Oct 31, 2026
Merged

Amanbig merged 1 commit into
mainfrom
fix/test-isolation

Conversation

@Amanbig

@Amanbig Amanbig commented Oct 31, 2026

Copy link
Copy Markdown
Owner

Problem (data-loss bug in the tests, from March)

tests/test_integration.py creates Permissions() and calls set_tool_permission("filesystem"|"edit", ALLOW). That method saves, and the target path is a module constant evaluated from Path.home() at import time, so every pytest run on a developer machine overwrote the real ~/.devorch/permissions.yaml with exactly:

tools:
  edit: {level: allow}
  filesystem: {level: allow}

Consequences: any saved rules ("always allow" patterns, deny rules, per-tool levels) were silently replaced, and DevOrch then edited and wrote files without asking in ASK mode. A stray DevOrch session run by the maintainer did exactly that and modified cli/main.py. The suite also wrote ~/.devorch/sessions/registry.json.

Found by running the suite with HOME pointed at an empty directory (only test_integration.py produced permissions.yaml) and bisecting by file.

Fix

New tests/conftest.py, executed before any application module is imported:

  • HOME / USERPROFILE point at a throwaway temp dir (removed at session end), so every Path.home() / ".devorch" constant resolves there
  • PYTHON_KEYRING_BACKEND = null backend: tests never touch the real keychain
  • API-key environment variables are cleared so tests cannot pick up real keys

tests/test_isolation.py fails if the suite ever resolves to the real home, if the application paths are outside the test home, if saving permissions escapes it, or if the real keyring is used.

Verified

  • Full suite (503 tests) run normally, with no HOME override: nothing under the real ~/.devorch was created, changed or removed (file list + mtimes identical before/after)
  • Without the conftest the isolation tests fail

Action for anyone who ran the tests before

Check ~/.devorch/permissions.yaml. If it contains only edit: allow and filesystem: allow, run devorch permissions reset (this also resets your saved rules, which were already overwritten).

tests/test_integration.py calls Permissions().set_tool_permission(...), which
saves to the real ~/.devorch/permissions.yaml, so every pytest run replaced the
developer's saved permissions with 'edit: allow' and 'filesystem: allow'.
Point HOME and USERPROFILE at a throwaway directory before the application
modules are imported, use the null keyring backend, and clear API key
variables. Add tests that fail if the suite ever resolves to the real home.
@Amanbig
Amanbig merged commit 2939f00 into main Oct 31, 2026
9 of 13 checks passed
@Amanbig
Amanbig deleted the fix/test-isolation branch October 31, 2026 10:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant