Skip to content

feat(bench): add native FIDES and bounded task isolation - #50

Open
arsenyinfo wants to merge 8 commits into
mainfrom
codex/agentthreatbench-typed-memory
Open

feat(bench): add native FIDES and bounded task isolation#50
arsenyinfo wants to merge 8 commits into
mainfrom
codex/agentthreatbench-typed-memory

Conversation

@arsenyinfo

@arsenyinfo arsenyinfo commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a distinct FIDES-native arm using automatic hiding and the official quarantine tools, with serialized middleware invocation to prevent thread-local cross-sample races
  • pin the compatible OpenAI 2.x transport and add a timeout-shape regression test so provider failures cannot silently enter the prior retry loop
  • extend guarded memory delegation with optional bounded scalars, fixed-unit measurements, and low-bandwidth compounds
  • discard model response prose and deliver only deterministic field=value serialization of engine-attested known=true fields
  • keep run manifests, audit validation, rendering, smoke inventory, and tests in sync with the fifth arm

Validation

  • ruff format --check src/appa_agentthreatbench tests
  • ruff check src/appa_agentthreatbench tests
  • pytest (51 passed)
  • uv lock --check
  • runtime leakage audit: no upstream Memory Poisoning expected-answer string occurs in tasks.py
  • three final-commit Memory Poisoning seeds:
    • standard: 86.7% ± 5.8 utility, 100% security
    • chaos: 76.7% ± 5.8 utility, 100% security
    • discarded model prose: 56.7% standard / 53.3% chaos utility

No Markdown documentation is included in this PR.

@arsenyinfo arsenyinfo changed the title Improve AgentThreatBench FIDES parity and typed memory utility feat(bench): improve AgentThreatBench FIDES parity Aug 21, 2026
@arsenyinfo arsenyinfo changed the title feat(bench): improve AgentThreatBench FIDES parity feat(bench): add native FIDES evaluation and attested memory delivery Aug 21, 2026
@arsenyinfo arsenyinfo changed the title feat(bench): add native FIDES evaluation and attested memory delivery feat(bench): add native FIDES and bounded task isolation Aug 21, 2026
@arsenyinfo
arsenyinfo force-pushed the codex/agentthreatbench-typed-memory branch 2 times, most recently from b3ebba2 to 11fcdf4 Compare August 22, 2026 11:23
@arsenyinfo
arsenyinfo force-pushed the codex/agentthreatbench-typed-memory branch from 11fcdf4 to d923796 Compare August 22, 2026 11:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant