[earakely-scale]

FD-4352 (M4.4). agent-env run hf://<org>/<repo>[@<revision>][/<path>] runs an AgentEnv bundle straight from a Hugging Face Hub dataset repo, as agent-env run runs a folder, with nothing installed beyond pip install agentenv-framework.

What it does

• Resolve. The revision (branch, tag or commit; the main branch by default; percent-encoded when it holds a /, as the Hub's own hf:// paths take refs%2Fpr%2F3) becomes a commit sha through HfApi.repo_info(repo_type="dataset"). Gated and private repos go through huggingface_hub's own login; a repo the Hub won't serve fails with the Hub's message and a note to hf auth login or set HF_TOKEN.
• Pick the bundle from the commit's file list before downloading anything: /<path> if given; else the only folder under bundles/, or the one the card's agentenv: default names (several with no default stop with the hf:// reference of each); else the repo root when it has a kind folder. bundles/ comes before the root, so a dataset with a root tasks/ of parquet files (the plugin's publish layout, GridSimEnv-AgentEnv) runs the bundle under bundles/.
• Download only that bundle's files and README.md, at the sha, into a plain folder $XDG_CACHE_HOME/agentenv/hf/<org>/<repo>/<sha> (~/.cache when unset or relative), no Hub-cache links. The commit's file list and the bundles downloaded are kept in .cache/agentenv/commit.json there, so a later run of the commit downloads nothing: a reference pinned to the sha makes no network call at all, and a branch or tag costs one repo_info. Commands on the same commit take a lock on it (holding_locks) while they list, download and record.
• Requirements. bundle.toml as today; a bundle without one requires the plugins the card's agentenv: bundles: <name>: plugins lists, refused with the agent-env plugin add '<spec>' line (and, in a terminal, the same offer to install as bundle.toml gets). The card's setup is shown only when setting up could fix the refusal (see below), never run. A card whose agentenv: bundles entry has the wrong shape is refused naming the field.
• Run through the existing run_bundle with id root @hf/<org>/<repo>, plus /<name> for a bundle under bundles/ (a /<path> elsewhere roots at @hf/<org>/<repo>/<path>). The revision is not in the id: a rerun of the commit reuses what the last one wrote, and a new commit writes new versions of the same entities.
• Record. The run first prints <org>/<repo> at <sha12>, bundle <path>; each task run's context carries metadata["bundle_source"] = {hub, repo, revision, sha, path}, stored with the instance, and the summary prints it (From <org>/<repo> at <sha12>, bundle <path>).

New module bundle/hub.py; locate_bundle recognises hf:// before paths and installed names and now returns Located(root, id_root, hub).

Follow-up changes (Edgar, after a VM check of GridSimEnv-AgentEnv)

• Setup note only when setup can fix it. The plan raises NotInStore, a BundleError carrying missing (kind, id), whenever a reference names an agent, env or artifact the store doesn't hold, mixed with other problems or not, and preflight does for a missing default agent. agent-env run attaches the card's setup note only to NotInStore or UnmetRequirements; a refusal with neither (a model-endpoint preflight error, say) and a task that fails at run time no longer get it.

• Requirements before the download. hub._fetch checks before snapshot_download: a bundle with a bundle.toml has that file alone downloaded first and checked by a folder's rules (parse.read_requires, URL refusal included), else the card's plugins are checked. The refusal is the same agent-env plugin add line plus the setup note; OpenEnv's build_app goes through the same path. Since folded into one check on every fetch (below).

• Missing store entities grouped. Each agent, env or artifact the store doesn't hold is one problem at its first use, (and N more uses) in preflight's format; the plan's other problems stay one line each. Folder runs get it too. GridSimEnv-AgentEnv's 2,200 lines become one: tasks/dam-v1-2014-01-03-LZ_SOUTH-2h-c.json: step 'deploy-agent': a2a_agent_id (and 2199 more uses): there is no agent 'gridsim-llm' in the store, then the setup note.

• verifiers main without vf-eval. verifiers main removed vf-eval (refactor(eval): remove vf-eval PrimeIntellect-ai/verifiers#2610) for vf-run, so taskset (verifiers main) skipped and failed the skip policy. tst/verifiers/taskset_test.py now runs vf-eval when it's on PATH (0.4.0, --no-push after the config when its help names it, as before) and vf-run otherwise, with the same config; vf-run exits 1 because the masked row isn't ok, which the test asserts, and both are judged by the same trace assertions. verifiers.yml puts whichever of the two the installed verifiers has on PATH. Reproduced on a fresh VM as the workflow sets up (Python 3.13.18, the nemo-gym extra from uv.lock, verifiers in its own venv): main 0.4.1.dev147 through vf-run and 0.4.0 through vf-eval both pass with no skips, and with neither on PATH the test skips as verifiers_installed.

Maintainability fixes (Edgar, from a review)

• One requirement check. fetch_bundle checks what the bundle requires on every call, a cached commit included (its bundle.toml by a folder's rules, else the card's plugins), and adds the card's setup note itself; HubBundle.check_plugins and its calls in cli/run.py and OpenEnv are gone, and agent-env run has one offer_to_install path, with locate_bundle inside its try.
• HubSource. Where a Hub bundle came from is a frozen HubSource (describe(), to_metadata(), from_metadata()); run_bundle(source=) takes it, and the stored bundle_source keeps exactly its shape.
• Namespace leftovers. store.ids.namespace_prefix() builds @<namespace>/ in local_namespace, validate_local_id and RoutingDocumentStore.check_id; a bare declared id's message says what this bundle may declare ('@local/', and for a Hub bundle its own root); Ledger.for_plan casts the namespace of a write's id, always local. config/runtime.py keeps its local store.ids import and "local" defaults: importing agent_env.store.ids at its top is circular (agent_env.store imports agent_env.config), checked on the VM.
• One name per Hub bundle. _bundle_name keys both the card's table and the id root: the folder under bundles/, the path for a folder elsewhere (so /a/x and /b/x no longer share a table), the repo's name for the root.
• _fetch split into _commit (the commit's record and card), a pure _select (pick, id root, needs, source) and _download (the requirement check, then the download), with the per-commit lock and record as before.
• Docs once. The pick order, cache path and card shape live in _pick's docstring and the README (now with the card's YAML); the run help keeps the reference format and points to the README; the hub module docstring is three lines.
• cache_root() moved to config/paths.py beside state_root(), same XDG rule ($XDG_CACHE_HOME/agentenv, else ~/.cache/agentenv).
• Conventions. Field and inline comments removed; long docstrings cut to three lines.
• Tested on a fresh VM: ruff check clean; full unit suite 12656 passed, 7 skipped; run_bundle_test.py plus tst/integration/store 22 passed; tst/integration/openenv 7 passed. New tests: a cached commit refused by the CLI and OpenEnv with no Hub call, the pure _select, /a/x and /b/x kept apart, cache_root, and a Hub bundle's declarable-prefix message.

Decisions (Edgar, Nov 2)

• D1: @hf is a second local namespace, routed to the local stores like @local, with its own documents file (hf.db beside local.db) and inside the write fence; services still refuse every @ id.
• D2: the command lives in core's agent-env run; the plugin's hf run becomes a thin call into it later.
• D3: /<path>, then bundles/ (only one, or the card's default), then the root.
• D4: no prompt and no --yes; --dry-run downloads, checks and shows what would run, and runs nothing.

@hf as a local namespace

• store/ids.py: is_local_id() and validate_local_id() know both namespaces; new local_namespace(). An @hf id encodes to its own local/hf-…-<hash> segment; @local encodings are unchanged.
• store/routing.py: RoutingDocumentStore takes a store per local namespace; a run's records follow its namespace (local_run_namespace()), unplaced reads consult the run's own namespace first, and the fence and outward-reference refusal cover both. The bundle ledger keeps each row in its entity's namespace store, so an installed bundle's @local entity pulled into an @hf run keeps one row.
• Sites moved to is_local_id(): plan.py (alias lookup, bundle-ref check, routing guard), importer.py _output_names, resolve.py output ids, parse.py declared ids, routing.py, document_store.reject_reserved_id; the existing is_local_id() callers (object/image routing in config/runtime.py, snapshot and collect steps, verifiers, deploy_agent, the GitHub-build refusal) now cover @hf too.
• Kept @local-only, since each builds or matches an installed bundle's or a folder's id, and those are always @local: plugins/_bundles.py InstalledBundle.id_root; parse.py _id_root (a folder's id root) and segment_problem (uses @local/ only as a vehicle to validate one name); plan.py _pulled and _bundle_name; importer.py _installed_for; register.py RegistrationLedger._bundle (installed:<dist>/<name>).
• Narrow rule for declared ids (parse.foreign_hub_id): an @hf id is accepted only under the bundle's own id root, which only hf:// produces. A folder or installed bundle can't declare one (nor name a task output by one, checked in resolve.py), and a Hub bundle can't declare another repo's. @local declarations are unchanged, Hub bundles included.
• HF org and repo names (letters, digits, -, _, ., at most 96 each) all pass validate_local_id and the byte limit; the reference parser refuses anything else, and a /<path> that can't be an id root is refused with the reason.

Also: on a case-insensitive filesystem, agent-env run desktop run from ~ took ~/Desktop as the folder and failed with "not a bundle". Bare text now counts as a folder only when each name in it is spelled as written in its parent's listing (NFC-compared); path-like text stays a path.

Packaging and docs

• huggingface_hub>=1.0 is a direct dependency (already in every install through litellm → tokenizers, locked at 1.33.0). Its HTTP errors are caught by its own HfHubHTTPError, since 2.x bases them on httpx2, not httpx (the CodeBuild run resolved 2.x and caught that). uv.lock gains two lines (relocked with uv 0.10.0; uv==0.13.1 lock --check passes). THIRD_PARTY_NOTICES.md already lists huggingface_hub 1.33.0.
• README: a short "Run a bundle from the Hub" section; AGENTS.md's bundle/ and store/ rows mention hf:// and @hf.
• Importing huggingface_hub adds about 70 ms to agent-env --help (1.39 s → 1.46 s on the test VM).
• agent_env.openenv.build_app takes its bundle through locate_bundle, so it now also accepts an hf:// reference, and checks the card's plugins as the CLI does.

Tests

On an agent-environments-admin EC2 VM (c6i.2xlarge, Ubuntu 24.04, Python 3.12, Docker 29.9 and registry:2 on :5000):

• ruff check: clean.

• pytest tst/unit packages/agentenv-protocol/tests -n 8: 12641 passed, 7 skipped; the 2 failures (test_chromium_app, run_docker_container_test) needed unzip, failed the same on main, and pass once it's installed.

• New tst/unit/bundle/hub_test.py (fake HfApi): references with and without @revision and /<path> and bad ones; picking by path, the only bundle under bundles/ beside a root tasks/ of data files, the card's default, several with no default, the root, no bundle; only the bundle and card downloaded; a commit reused with no call (pinned) or only repo_info (branch); the Hub's refusal with the login note; through the CLI, the source line first, the @hf/me/ds/hello id root, bundle_source in each instance, the rerun reusing v1, ledger rows in hf.db and nothing in local.db; a dry run that downloads and runs nothing; card plugins refused with the plugin add line and ignored beside a bundle.toml; the card's setup shown on failure.

• Extended: routing_test (@hf documents and run records in hf.db, the fence, outward references, derived ids, objects routed locally, hf.db can't be the configured store), local_id_acceptance_test (services and the @local store refuse @hf), ids_test, parse_test and resolve_test (@hf ids only under a Hub bundle's own root), installed_test (Desktop/ doesn't hide an installed desktop with a case-insensitive lookup simulated; an exactly spelled folder still wins), plan_test, explorer_app_test.

• Integration (tst/integration/cli/run_bundle_test.py, real Hub): agent-env run hf://earakely-scale/agentenv-hf-smoke@5fbd709207a73e2aeb73a77e272a61003d1be768/bundles/hello passes, a rerun reuses every write, only bundles/hello/ and the card are downloaded, the instance records the source; without the path the same commit stops listing bundles/hello and bundles/v3. With the rest of run_bundle_test.py and tst/integration/store: 22 passed.

• By hand on the VM: hf://earakely-scale/GridSimEnv-AgentEnv picks bundles/gridsim-v1 past the root's tasks/, runs/ and other result folders (5637 files listed, the bundle alone downloaded); hf://earakely-scale/CuaDriverBench-AgentEnv, which has no bundles/ yet, takes its root and stops with "no tasks or evals to run"; a repo that doesn't exist fails with the Hub's 401 message and the login note.

• hub_test.py and the real-Hub integration test pass on huggingface_hub 1.0.0, 1.33.0 and 2.4.0.

• Follow-up changes, on a fresh VM (c6i.2xlarge): full unit suite 12651 passed, 7 skipped; run_bundle_test.py plus tst/integration/store (real Hub included) 22 passed. New and extended tests: plan_test (three tasks sharing one missing agent give one grouped line and NotInStore; a refusal without a store gap isn't one), hub_test (the card's plugins and a bundle.toml, by requirement and by URL, refused with no snapshot_download; the setup note on a mixed refusal and not on one without a gap; OpenEnv refused before the download). By hand: hf://earakely-scale/GridSimEnv-AgentEnv at 744e139e without the plugin stops having downloaded only the card (544K, no bundle), and with it a dry run prints 5 lines.

Follow-ups

• agentenv-hf plugin: hf run becomes a thin call into core (no prompt, @hf ids) in a release requiring this version.
• Website docs: FD-4716.
• The integration fixture is the existing public earakely-scale/agentenv-hf-smoke, pinned by sha; a dedicated fixture repo could replace it.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no new blocking issues were found.

Summary

Adds agent-env run hf://… to fetch and run bundles from Hugging Face Hub datasets.

• Bundles can run directly from Hugging Face dataset references.
• Hub bundle records use their own local store namespace.
• Run checks group missing store items and show setup guidance only when it can help.
• Bundle lookup matches folder names only when their spelling matches.
• The verifiers check uses whichever evaluation command is installed.

No new actionable issues were found in the changes since the previous review. The earlier threads were also rechecked.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A["hf:// reference"] --> B["Resolve commit and read file list"]
  B --> C["Pick bundle and read card"]
  C --> D["Check plugin requirements"]
  D --> E["Download selected bundle or reuse cache"]
  E --> F["Plan and run bundle"]
  F --> G["Store entities and runs under @hf"]
  F --> H["Record bundle_source"]

Loading

_{Reviews (7) · Last reviewed commit: "docs(bundle): the Hub bundle rules once,..." · Reviewed by Greptile}