Add 'rootstock usage push': send aggregated rollups to the dashboard

README.md

@@ -89,7 +89,7 @@ Rootstock ships an experimental LAMMPS `fix` that spawns a worker subprocess, gi
 
 ## Usage Statistics
 
-On shared installs where the maintainer has provisioned it, each worker session drops one small anonymous JSON record into the install's `{cache_root}/usage/` directory: which environment/checkpoint/device ran, when, for how long, how many force calls it served, and which entry point started it. Nothing phones home — records stay on the cluster's shared filesystem until the install's maintainer aggregates them (`rootstock usage report`). No job ids, no simulation data, and no raw usernames are recorded: the only per-person field is a salted hash used to count distinct users. Runs with your own weights record only the `<family>:custom` checkpoint id — never the weights path.
+On shared installs where the maintainer has provisioned it, each worker session drops one small anonymous JSON record into the install's `{cache_root}/usage/` directory: which environment/checkpoint/device ran, when, for how long, how many force calls it served, and which entry point started it. Nothing phones home — records stay on the cluster's shared filesystem until the install's maintainer aggregates them (`rootstock usage report`) and, if the cluster is registered with the dashboard, pushes the aggregated monthly counts (`rootstock usage push`). No job ids, no simulation data, and no raw usernames are recorded: the only per-person field is a salted hash used to count distinct users, and checkpoints you register yourself with `rootstock add-local` appear as `(local)`, never by name. Only derived counts ever leave the cluster — the hashes stay in the spool.
 
 If any of this rubs you the wrong way, opt out for all of your sessions with:
 

docs/cluster-setup.md

@@ -137,7 +137,7 @@ After `rootstock init` runs, verify ACLs landed correctly with `getfacl` on a fr
 
 ### The usage-record spool
 
-`setup-perms` also creates `{cache_root}/usage/` with mode `1777` (sticky, `/tmp`-style). This is the spool for [anonymous per-session usage records](https://github.com/Garden-AI/rootstock#usage-statistics): any user's session must be able to drop a record file, and the sticky bit keeps users from deleting each other's. Records are created `O_EXCL` with unique names, so the directory behaves like `/tmp`, not like a writable code path — but some centers' security scans flag any world-writable directory in project space. If that's a problem on your cluster (or you just don't want usage collection), pass `--no-usage-spool`: a missing spool means collection is off for the install, and `check-perms` treats its absence as a deliberate choice, not an issue. Aggregate with `rootstock usage report`, and fold raw records into monthly rollups with `rootstock usage compact` (safe to run from the same cron as the nightly smoke-test).
+`setup-perms` also creates `{cache_root}/usage/` with mode `1777` (sticky, `/tmp`-style). This is the spool for [anonymous per-session usage records](https://github.com/Garden-AI/rootstock#usage-statistics): any user's session must be able to drop a record file, and the sticky bit keeps users from deleting each other's. Records are created `O_EXCL` with unique names, so the directory behaves like `/tmp`, not like a writable code path — but some centers' security scans flag any world-writable directory in project space. If that's a problem on your cluster (or you just don't want usage collection), pass `--no-usage-spool`: a missing spool means collection is off for the install, and `check-perms` treats its absence as a deliberate choice, not an issue. Aggregate with `rootstock usage report`, fold raw records into monthly rollups with `rootstock usage compact`, and send the aggregated counts to the dashboard with `rootstock usage push` (uses the same API credentials as `manifest push`; all three are safe to run from the same cron as the nightly smoke-test).
 
 On clusters where your write access to the install is granted temporarily and later revoked (e.g. Delta), don't put the real spool inside the install — you'd eventually lose the ability to fix its permissions or prune it. Pass `--usage-dir <path>` pointing somewhere you permanently control (a directory under your home works): setup-perms creates the `1777` directory there and symlinks `{cache_root}/usage` to it, so sessions and the collector follow the link without any configuration. Make sure the target's ancestors grant world traverse (`chmod o+x ~` if the spool lives under your home). If the spool ever does become unwritable, user calculations are unaffected — sessions silently skip recording — and `check-perms` will tell you.
 

rootstock/cli.py

@@ -48,9 +48,10 @@
 
     rootstock usage report [--root <path>] [--json]
     rootstock usage compact [--root <path>]
-        Aggregate (read-only) or compact the anonymous usage-record spool at
-        {cache_root}/usage/. The spool is provisioned by setup-perms; without
-        it, usage collection is off.
+    rootstock usage push [--root <path>] [--dry-run]
+        Aggregate (read-only), compact, or push to the dashboard backend the
+        anonymous usage-record spool at {cache_root}/usage/. The spool is
+        provisioned by setup-perms; without it, usage collection is off.
 """
 
 import argparse
@@ -75,6 +76,7 @@
     cmd_smoke_test,
     cmd_status,
     cmd_usage_compact,
+    cmd_usage_push,
     cmd_usage_report,
 )
 from .commands.common import ROOTSTOCK_ROOT_ENV
@@ -564,7 +566,8 @@ def main():
             "Maintainer-side view of the anonymous usage records that "
             "calculator sessions spool to {cache_root}/usage/. 'report' "
             "aggregates read-only; 'compact' folds raw records into "
-            "per-month rollup files."
+            "per-month rollup files; 'push' sends the aggregated rollup "
+            "rows to the dashboard backend."
         ),
     )
     usage_subparsers = usage_parser.add_subparsers(
@@ -603,6 +606,26 @@ def main():
     )
     usage_compact_parser.set_defaults(func=cmd_usage_compact)
 
+    usage_push_parser = usage_subparsers.add_parser(
+        "push",
+        help="Push aggregated rollup rows to the dashboard backend",
+    )
+    usage_push_parser.add_argument(
+        "--root",
+        default=os.environ.get(ROOTSTOCK_ROOT_ENV),
+        help=f"Root directory (default: ${ROOTSTOCK_ROOT_ENV})",
+    )
+    usage_push_parser.add_argument(
+        "--cache-root",
+        help="Cache root override (default: the install's own declaration)",
+    )
+    usage_push_parser.add_argument(
+        "--dry-run",
+        action="store_true",
+        help="Print the endpoint URL and payload instead of pushing",
+    )
+    usage_push_parser.set_defaults(func=cmd_usage_push)
+
     # parse_known_args instead of parse_args so `rootstock benchmark ...` can
     # forward arbitrary flags to the benchmark's own parser. Every other
     # command keeps strict parsing via the explicit error below.

rootstock/client.py

@@ -1,7 +1,7 @@
 """
 HTTP client for Rootstock backend API.
 
-Handles pushing manifests to the central registry.
+Handles pushing manifests and usage rollups to the central registry.
 """
 
 from __future__ import annotations
@@ -43,10 +43,39 @@ def push_manifest(self, manifest: Manifest) -> tuple[bool, str]:
         if not self.config.api_url:
             return False, "API URL not configured"
 
-        data = json.dumps(manifest.to_dict()).encode("utf-8")
+        return self._post(self.config.api_url, manifest.to_dict(), "Manifest pushed successfully")
+
+    def push_usage(self, cluster: str, rows: list[dict]) -> tuple[bool, str]:
+        """
+        Push aggregated usage rollup rows to the backend usage endpoint.
+
+        Args:
+            cluster: Cluster name the rollups are filed under
+            rows: Rollup rows as aggregated by ``summarize_spool`` — counts
+                only, never the user hashes
+
+        Returns:
+            (success, message) tuple
+        """
+        if not self.config.api_key:
+            return False, "API key not configured"
+        if not self.config.api_secret:
+            return False, "API secret not configured"
+        url = self.config.resolve_usage_api_url()
+        if not url:
+            return False, "Usage API URL not configured (set usage_api_url)"
+
+        return self._post(
+            url,
+            {"cluster": cluster, "rows": rows},
+            f"Pushed {len(rows)} rollup row(s) to {url}",
+        )
+
+    def _post(self, url: str, payload: dict, success_message: str) -> tuple[bool, str]:
+        data = json.dumps(payload).encode("utf-8")
 
         request = Request(
-            self.config.api_url,
+            url,
             data=data,
             headers={
                 "Content-Type": "application/json",
@@ -59,7 +88,7 @@ def push_manifest(self, manifest: Manifest) -> tuple[bool, str]:
         try:
             with urlopen(request, timeout=30) as response:
                 if response.status in (200, 201, 204):
-                    return True, "Manifest pushed successfully"
+                    return True, success_message
                 return False, f"Unexpected status: {response.status}"
         except HTTPError as e:
             return False, f"HTTP error {e.code}: {e.reason}"

rootstock/commands/__init__.py

@@ -12,7 +12,7 @@
 from .setup_perms import cmd_setup_perms
 from .smoke_test import cmd_smoke_test
 from .status import cmd_list, cmd_status
-from .usage import cmd_usage_compact, cmd_usage_report
+from .usage import cmd_usage_compact, cmd_usage_push, cmd_usage_report
 
 __all__ = [
     "cmd_add",
@@ -31,5 +31,6 @@
     "cmd_smoke_test",
     "cmd_status",
     "cmd_usage_compact",
+    "cmd_usage_push",
     "cmd_usage_report",
 ]

rootstock/commands/usage.py

@@ -1,9 +1,10 @@
-"""``rootstock usage`` — report on and compact the usage-record spool.
+"""``rootstock usage`` — report on, compact, and push the usage-record spool.
 
 The spool ({cache_root}/usage/) fills with one small JSON record per
 calculator session (see rootstock/usage.py). ``report`` aggregates it
 read-only; ``compact`` folds raw records into per-month rollup files so the
-spool doesn't accumulate thousands of tiny files. Both are login-node,
+spool doesn't accumulate thousands of tiny files; ``push`` sends the
+aggregated rollup rows to the dashboard backend. All are login-node,
 maintainer-side operations — the write side never needs them.
 """
 
@@ -12,6 +13,9 @@
 import json
 import sys
 
+from ..client import RootstockClient
+from ..config import load_config
+from ..manifest import load_manifest
 from ..usage import SpoolSummary, compact_spool, summarize_spool, usage_dir
 from .common import get_root_or_exit, resolve_cache_root
 
@@ -85,6 +89,75 @@ def cmd_usage_report(args) -> int:
     return 0
 
 
+def cmd_usage_push(args) -> int:
+    """Push the aggregated rollup rows to the dashboard backend.
+
+    Pushes the same aggregation ``report`` shows — rollup files plus any
+    not-yet-compacted raw records — filed under the manifest's cluster name.
+    The backend stores rollups per month and replaces only the months
+    present in the push, so pushing repeatedly (e.g. from the smoke-test
+    cron) is idempotent and never erases previously pushed history. Only
+    derived counts are sent; the salted user hashes stay in the spool.
+    """
+    root = get_root_or_exit(args)
+    cache_root = resolve_cache_root(root, args.cache_root)
+    config = load_config()
+
+    valid, error = config.validate()
+    if not valid:
+        print(f"Error: {error}", file=sys.stderr)
+        print(
+            "Configure API credentials in ~/.config/rootstock/config.toml",
+            file=sys.stderr,
+        )
+        return 1
+    url = config.resolve_usage_api_url()
+    if not url:
+        print(
+            "Error: no usage endpoint — api_url doesn't follow the standard "
+            "rootstock-admin naming, so set usage_api_url in "
+            "~/.config/rootstock/config.toml explicitly.",
+            file=sys.stderr,
+        )
+        return 1
+
+    manifest = load_manifest(root)
+    if manifest is None or not manifest.cluster:
+        print(
+            f"No manifest at {root}/manifest.json — the push is filed under "
+            "the manifest's cluster name; run 'rootstock manifest init "
+            "--cluster <name>' first.",
+            file=sys.stderr,
+        )
+        return 1
+
+    summary = summarize_spool(cache_root)
+    if summary is None:
+        print(
+            f"No usage spool at {usage_dir(cache_root)} — usage collection is "
+            "off for this install (rootstock setup-perms provisions it).",
+            file=sys.stderr,
+        )
+        return 1
+    if not summary.rows:
+        print("No usage recorded yet — nothing to push.")
+        return 0
+
+    if args.dry_run:
+        payload = {"cluster": manifest.cluster, "rows": summary.rows}
+        print(f"Would POST to {url}:")
+        print(json.dumps(payload, indent=2))
+        return 0
+
+    client = RootstockClient(config)
+    success, message = client.push_usage(manifest.cluster, summary.rows)
+    if not success:
+        print(f"Error: {message}", file=sys.stderr)
+        return 1
+    print(message)
+    return 0
+
+
 def cmd_usage_compact(args) -> int:
     """Fold raw records into per-month rollup files."""
     root = get_root_or_exit(args)

rootstock/config.py

@@ -27,6 +27,7 @@
 ROOTSTOCK_API_KEY_ENV = "ROOTSTOCK_API_KEY"
 ROOTSTOCK_API_SECRET_ENV = "ROOTSTOCK_API_SECRET"
 ROOTSTOCK_API_URL_ENV = "ROOTSTOCK_API_URL"
+ROOTSTOCK_USAGE_API_URL_ENV = "ROOTSTOCK_USAGE_API_URL"
 ROOTSTOCK_ROOT_ENV = "ROOTSTOCK_ROOT"
 
 # Default config location
@@ -42,6 +43,7 @@ class UserConfig:
     api_key: str | None = None
     api_secret: str | None = None
     api_url: str | None = None
+    usage_api_url: str | None = None
     name: str | None = None
     email: str | None = None
     is_maintainer: bool = False
@@ -50,6 +52,22 @@ def is_push_enabled(self) -> bool:
         """Check if api_key, api_secret, and api_url are configured for pushing."""
         return bool(self.api_key and self.api_secret and self.api_url)
 
+    def resolve_usage_api_url(self) -> str | None:
+        """The endpoint usage rollups are pushed to.
+
+        Deployments of rootstock-admin expose the manifest and usage ingest
+        functions side by side (``...-manifest.modal.run`` /
+        ``...-usage.modal.run``), so when ``usage_api_url`` isn't configured
+        it is derived from ``api_url`` by swapping the function name. An
+        ``api_url`` that doesn't follow that naming can't be derived from —
+        set ``usage_api_url`` explicitly.
+        """
+        if self.usage_api_url:
+            return self.usage_api_url
+        if self.api_url and "-manifest" in self.api_url:
+            return self.api_url.replace("-manifest", "-usage", 1)
+        return None
+
     def validate(self) -> tuple[bool, str]:
         """
         Validate configuration for push operations.
@@ -91,6 +109,7 @@ def load_config(config_path: Path | None = None) -> UserConfig:
         config.api_key = data.get("api_key")
         config.api_secret = data.get("api_secret")
         config.api_url = data.get("api_url")
+        config.usage_api_url = data.get("usage_api_url")
         config.is_maintainer = data.get("is_maintainer", False)
         maintainer = data.get("maintainer", {})
         config.name = maintainer.get("name")
@@ -103,6 +122,8 @@ def load_config(config_path: Path | None = None) -> UserConfig:
         config.api_secret = os.environ.get(ROOTSTOCK_API_SECRET_ENV)
     if not config.api_url:
         config.api_url = os.environ.get(ROOTSTOCK_API_URL_ENV)
+    if not config.usage_api_url:
+        config.usage_api_url = os.environ.get(ROOTSTOCK_USAGE_API_URL_ENV)
 
     return config
 
@@ -147,6 +168,8 @@ def save_config(config: UserConfig, config_path: Path | None = None) -> None:
         data["api_secret"] = config.api_secret
     if config.api_url:
         data["api_url"] = config.api_url
+    if config.usage_api_url:
+        data["usage_api_url"] = config.usage_api_url
 
     maintainer = {}
     if config.name: