fix double billing when timeout triggered, and record memory with no …

EVALUATIONS.md

@@ -387,7 +387,7 @@ except Exception as exc:
     # the legacy views below work on every engine
 ```
 
-The legacy views are not affected - they call the long-standing `/custom-agent-evaluations/evaluation-settings` and `/agent-monitoring/online-evaluators` routes - so they remain the portable choice for code that must run against engines you do not control. This does not affect the `scorer_id` kwarg on `.run()`, which is client-side naming over a field the wire has always had.
+The legacy views are not affected - they call the long-standing `/custom-agent-evaluations/evaluation-settings` and `/monitor/online-evaluators` routes (only the calibration/tuning calls ride `/agent-monitoring`) - so they remain the portable choice for code that must run against engines you do not control. This does not affect the `scorer_id` kwarg on `.run()`, which is client-side naming over a field the wire has always had.
 
 #### Legacy views
 
@@ -896,7 +896,7 @@ This run + gate flow is the **self-host CI path**. The separate CI-runs API in [
 
 ### AI analysis report
 
-`.analyze()` is the last step in the chain. It runs the same durable, multi-stage pipeline as the dashboard's "Analyze" button: each response is scored by 1-3 LLM judges, then reduced through question- and cluster-level summaries into one final qualitative report, returned as the `Report` object. Because of this, `.analyze()` polls until the job finishes rather than returning instantly, and can take noticeably longer than a single LLM call for larger runs (progress is shown in the terminal while it waits).
+`.analyze()` is the last step in the chain. It triggers the same analysis as the dashboard's "Analyze" button: each response is scored by 1-3 LLM judges and reduced into one final qualitative report, returned as the `Report` object. On self-host this is a single synchronous pass - the request returns when the whole analysis is done, so the poll loop sees a terminal status on its first check and the per-level progress percentages never populate. Either way, `.analyze()` can take noticeably longer than a single LLM call for larger runs (a spinner is shown in the terminal while it waits).
 
 ```python
 report = client.evaluations.run(...).execute(my_agent).finalize().analyze(

TRACING.md

@@ -140,7 +140,7 @@ with tracer.trace("orchestrator") as root:
     root.output = reply
 ```
 
-The active-span stack is **thread-local**. Work submitted to a `ThreadPoolExecutor` (or any other thread) doesn't see a span opened on the calling thread - wrap the worker body in `tracer.use_span(span)` to attach it:
+The active-span stack is **context-local** (a `ContextVar`): bare threads start with an empty stack, while asyncio tasks inherit a copy of their creator's. Work submitted to a `ThreadPoolExecutor` (or any other thread) doesn't see a span opened on the calling thread - wrap the worker body in `tracer.use_span(span)` to attach it:
 
 ```python
 with tracer.trace("orchestrator") as span:
@@ -380,7 +380,7 @@ with tracer.trace("support-agent") as span:
     span.output = answer
 ```
 
-`tracer.record_memory(name, operation=..., query=..., output=..., duration_ms=...)` is the after-the-fact form. `operation` is free text - conventionally `"read"` or `"write"` - carried in the span's metadata, while the kind itself stays one value so dashboards and scorers can select all memory activity at once. With no active span, both forms queue the record and merge it into the next trace this tracer sends (the patched-client flow where the memory op runs just before a standalone completions call) instead of silently dropping it.
+`tracer.record_memory(name, operation=..., query=..., output=..., duration_ms=...)` is the after-the-fact form. `operation` is free text - conventionally `"read"` or `"write"` - carried in the span's metadata, while the kind itself stays one value so dashboards and scorers can select all memory activity at once. With no active span the record is dropped (with a debug log) - the pending queue rides the next trace's retrieval steps, and memory content must never feed the RAG judges' retrieval context - so wrap the call in `tracer.trace()`.
 
 ---
 

agentx/evaluations/client.py

@@ -207,6 +207,12 @@ def _request(
                 return resp.json()
             except Exception:
                 return resp.text
+        # A timeout keeps its type: runner._flush_batch catches requests.Timeout specifically
+        # (the engine may still be scoring the batch - a blind retry double-bills every judge
+        # call), and wrapping it in AgentXEvaluationsError here made that guard unreachable.
+        # Applies to retry=False calls too, where the single attempt lands straight here.
+        if isinstance(last_exc, requests.Timeout):
+            raise last_exc
         raise AgentXEvaluationsError(f"Request failed after retries: {last_exc}")
 
     # ------------------------------------------------------------------
@@ -227,7 +233,9 @@ def list_models(self, provider: Optional[str] = None) -> List[ModelInfo]:
     # ------------------------------------------------------------------
 
     def create_dataset(self, payload: dict) -> Dataset:
-        data = self._request("POST", "/datasets", json=self._with_workspace(payload))
+        # Server-side write: a timeout after the dataset row was created would be
+        # retried into a duplicate dataset, so no transport retry.
+        data = self._request("POST", "/datasets", json=self._with_workspace(payload), retry=False)
         return Dataset(**data)
 
     def delete_dataset(self, dataset_id: str) -> None:
@@ -255,8 +263,9 @@ def get_dataset(self, dataset_id: str) -> Dataset:
     # ------------------------------------------------------------------
 
     def create_evaluation_settings(self, payload: dict) -> EvaluationSettings:
+        # Server-side write - no transport retry (see init_run's comment).
         data = self._request(
-            "POST", "/evaluation-settings", json=self._with_workspace(payload)
+            "POST", "/evaluation-settings", json=self._with_workspace(payload), retry=False
         )
         return EvaluationSettings(**data)
 
@@ -281,12 +290,14 @@ def get_evaluation_settings(self, evaluation_settings_id: str) -> EvaluationSett
 
     # ------------------------------------------------------------------
     # Prompt registry endpoints - see agentx.evaluations.prompts.PromptClient for the concept
-    # (the external-agent analog to native autotune). Deliberately read-mostly: no publish here,
-    # a new version only ever comes from the dashboard's human-approved propose/publish flow.
+    # (the external-agent analog to native autotune). propose_prompt never publishes;
+    # publish_prompt_version below IS the explicit approval step - call it only after a human
+    # reviewed the proposal.
     # ------------------------------------------------------------------
 
     def create_prompt(self, payload: dict) -> Prompt:
-        data = self._request("POST", "/prompts", json=self._with_workspace(payload))
+        # Server-side write - no transport retry (see init_run's comment).
+        data = self._request("POST", "/prompts", json=self._with_workspace(payload), retry=False)
         return Prompt(**data)
 
     def list_prompts(self) -> List[Prompt]:
@@ -322,7 +333,9 @@ def init_run(
         alias and keeps working. ``split`` records the named case subset this run covers.
         ``additional_scorer_ids`` (self-host): extra judge scorers that each pass their own
         verdict on every result from the same single agent execution - verdicts land in each
-        result row's ``judgeScorerResults`` and the run's ``scorerBreakdown``."""
+        result row's ``judgeScorerResults`` and the run's ``scorerBreakdown``. When
+        ``scorer_group_id`` is set, the engine nulls ``additionalScorerIds`` on the run too -
+        the group is the whole grading story, not a layer on top of extra scorers."""
         from agentx.version import VERSION
 
         grader_id = _resolve_scorer_id(scorer_id, evaluation_settings_id)
@@ -343,7 +356,8 @@ def init_run(
         if additional_scorer_ids:
             payload["additionalScorerIds"] = additional_scorer_ids
         # Scorer group grading (self-host): the group's weighted 0-10 aggregate fills the rating
-        # column and member verdicts land per row. Mutually exclusive with scorer_id (group wins).
+        # column and member verdicts land per row. Mutually exclusive with scorer_id (group
+        # wins), and the engine also nulls additionalScorerIds when a group grades the run.
         if scorer_group_id:
             payload["scorerGroupId"] = scorer_group_id
         if split:
@@ -408,7 +422,10 @@ def gate_run(
         # scorer's own per-result verdicts. Unknown names are a hard 400 from the engine.
         if scorer:
             params["scorer"] = scorer
-        return self._request("GET", f"/runs/{run_id}/gate", params=params)
+        # record=True is a server-side write despite the GET verb (it persists a gate-history
+        # row): a timeout after the row was stored would be retried into a duplicate verdict,
+        # so no transport retry - same precedent as init_run.
+        return self._request("GET", f"/runs/{run_id}/gate", params=params, retry=not record)
 
     def analyze_run(
         self,

agentx/evaluations/datasets.py

@@ -81,7 +81,8 @@ def __init__(
             self._payload["codeScorers"] = [
                 {
                     "id": scorer.get("id") or _uuid.uuid4().hex[:12],
-                    "name": scorer["name"],
+                    # Name may be omitted - the engine defaults it, so don't KeyError here.
+                    "name": scorer.get("name"),
                     "code": scorer["code"],
                     "enabled": scorer.get("enabled", True),
                 }

agentx/evaluations/evaluation_settings.py

@@ -64,7 +64,8 @@ def __init__(
         if rouge_score:
             self._payload["rougeScore"] = {"enabled": True}
         # Sovereignty & Portability - the models to compare when this config runs
-        # (use client.evaluations.list_models() to discover valid ids).
+        # (use client.evaluations.list_models() to discover valid ids). Self-host: accepted
+        # on the wire but not acted on by the engine (same caveat as DatasetBuilder's).
         if sovereignty_models:
             self._payload["sovereigntyIndex"] = {
                 "enabled": True,

agentx/evaluations/models.py

@@ -248,6 +248,10 @@ class LiveStatistics(BaseModel):
     min_rating: Optional[float] = Field(default=None, alias="minRating")
     max_rating: Optional[float] = Field(default=None, alias="maxRating")
     rated_count: int = Field(default=0, alias="ratedCount")
+    # Rows the judge could not score / rows submitted with an error - the difference between
+    # "everything rated 8" and "half the run never got a verdict".
+    skipped_count: int = Field(default=0, alias="skippedCount")
+    failed_count: int = Field(default=0, alias="failedCount")
 
     class Config:
         populate_by_name = True

agentx/evaluations/prompts.py

@@ -17,9 +17,9 @@ class PromptClient:
     existing version-comparison view (``client.evaluations`` run comparisons on a dataset) can
     tell you which published version actually scored higher.
 
-    Deliberately read-mostly from here: there is no ``publish`` on this client. A prompt only
-    gets a new version through the dashboard's human-approved propose/publish flow, so a
-    rewritten prompt never reaches your running agent without someone explicitly approving it.
+    ``propose()`` never publishes anything; ``publish_version()`` IS the explicit approval
+    step - call it only after a human reviewed the proposal, since a published version is
+    what your running agent pulls as its live prompt.
 
     Example::
 

agentx/evaluations/runner.py

@@ -342,6 +342,10 @@ def _fetch_submitted_keys(self) -> Set[str]:
     # ------------------------------------------------------------------
 
     def finalize(self) -> "EvaluationRunContext":
+        """Mark the run completed server-side. A failed finalize is raised, not swallowed
+        (same fail-loud posture as _flush_batch): it leaves the run in_progress - a state a
+        CI pipeline MUST treat as a failure, since gates and baselines only consider
+        completed runs."""
         _say()
         with Spinner("Finalizing - submitting results"):
             try:
@@ -353,6 +357,7 @@ def finalize(self) -> "EvaluationRunContext":
             except Exception as exc:
                 _say(f"  {red('✗')}  Finalize failed: {dim(str(exc))}")
                 logger.error("Finalize failed: %s", exc)
+                raise
         return self
 
     def gate(

agentx/integrations/crewai.py

@@ -212,6 +212,9 @@ def _build_steps_from_timings(
                 "end_time": end,
                 "input": description,
                 "output": output_text,
+                # A CrewAI task is an agent turn, not a model call - without this,
+                # _merge_child_run's default stamped every task span "llm".
+                "kind": "agent",
             })
             if description is not None or task_output_text is not None:
                 tool_calls.append({"name": name, "input": description, "output": task_output_text})

agentx/integrations/google_adk.py

@@ -231,6 +231,7 @@ async def after_model_callback(
                 model=call_start.get("model") if call_start else None,
                 input_tokens=call_input_tokens,
                 output_tokens=call_output_tokens,
+                span_kind="llm",
             )
 
     async def on_model_error_callback(
@@ -261,6 +262,7 @@ async def on_model_error_callback(
                 output=f"ERROR: {error}",
                 model=call_start.get("model") if call_start else None,
                 error=str(error),
+                span_kind="llm",
             )
 
     # ------------------------------------------------------------------
@@ -290,7 +292,8 @@ async def after_tool_callback(
         tool_input = _safe_serialize(tool_args)
         tool_output = str(result) if result is not None else None
         state["root_span"].child_span(
-            tool_name, start_time=start_t, end_time=end_t, input=tool_input, output=tool_output
+            tool_name, start_time=start_t, end_time=end_t, input=tool_input, output=tool_output,
+            span_kind="tool",
         )
 
     async def on_tool_error_callback(
@@ -311,5 +314,6 @@ async def on_tool_error_callback(
         tool_input = _safe_serialize(tool_args)
         tool_output = f"ERROR: {error}"
         state["root_span"].child_span(
-            tool_name, start_time=start_t, end_time=end_t, input=tool_input, output=tool_output, error=str(error)
+            tool_name, start_time=start_t, end_time=end_t, input=tool_input, output=tool_output, error=str(error),
+            span_kind="tool",
         )

agentx/integrations/langchain.py

@@ -277,6 +277,10 @@ class AgentXCallbackHandler(BaseCallbackHandler):
     become child spans, and each LLM call / tool call / retrieval becomes a span parented under
     the node that ran it - so the engine's Execution Timeline shows the actual graph trajectory
     (which nodes ran, in what order, and what each did), not a flat step list.
+
+    Retriever runs are stamped ``retrieval`` - LangChain cannot distinguish memory-backed
+    retrievers, so a Mem0/Zep-style store exposed as a retriever classifies as retrieval too.
+    Use ``tracer.trace_memory`` for lookups that should classify as memory.
     """
 
     def __init__(

agentx/integrations/openai_agents.py

@@ -251,6 +251,7 @@ def on_span_end(self, span: Any) -> None:
                     model=call_model,
                     input_tokens=call_input_tokens,
                     output_tokens=call_output_tokens,
+                    span_kind="llm",
                 )
 
         elif span_type == "response":
@@ -300,6 +301,7 @@ def on_span_end(self, span: Any) -> None:
                     model=call_model,
                     input_tokens=call_input_tokens,
                     output_tokens=call_output_tokens,
+                    span_kind="llm",
                 )
 
         elif span_type == "function":
@@ -313,6 +315,7 @@ def on_span_end(self, span: Any) -> None:
                 duration_ms=latency if t0 is None or t1 is None else None,
                 input=span_data.input,
                 output=tool_output,
+                span_kind="tool",
             )
 
     def force_flush(self) -> None:

agentx/monitor/client.py

@@ -235,7 +235,9 @@ def _request(
     # ------------------------------------------------------------------
 
     def create_pattern(self, payload: dict) -> MonitorPattern:
-        data = self._request("POST", "/patterns", json=self._with_workspace(payload))
+        # Server-side write: a timeout after the pattern row was created would be
+        # retried into a duplicate pattern, so no transport retry.
+        data = self._request("POST", "/patterns", json=self._with_workspace(payload), retry=False)
         return MonitorPattern(**data["pattern"])
 
     def list_patterns(self) -> List[MonitorPattern]:
@@ -253,7 +255,10 @@ def get_pattern(self, pattern_id: str) -> MonitorPattern:
     # ------------------------------------------------------------------
 
     def create_online_evaluator(self, payload: dict) -> MonitorOnlineEvaluator:
-        data = self._request("POST", "/online-evaluators", json=self._with_workspace(payload))
+        # Server-side write - no transport retry (see create_pattern's comment).
+        data = self._request(
+            "POST", "/online-evaluators", json=self._with_workspace(payload), retry=False
+        )
         return MonitorOnlineEvaluator(**data["evaluator"])
 
     @property

agentx/monitor/models.py

@@ -12,6 +12,11 @@ class MonitorPattern(BaseModel):
     A "failure" pattern (the default) raises a signal to triage; a "proper" pattern logs a
     healthy tally instead. Only one of ``include_terms``/``regex``/``semantic_prompt`` is
     meaningful at a time, selected by ``detector_kind``.
+
+    On self-host the engine stores a pattern as a list of ``conditions`` (each with its own
+    detector kind and match settings) - the flat ``include_terms``/``regex``/
+    ``semantic_prompt`` fields are display-only projections derived from the first condition;
+    ``conditions`` is the truth.
     """
 
     id: str = Field(alias="_id")
@@ -26,6 +31,8 @@ class MonitorPattern(BaseModel):
     exclude_terms: List[str] = Field(default_factory=list, alias="excludeTerms")
     regex: Optional[str] = None
     semantic_prompt: Optional[str] = Field(default=None, alias="semanticPrompt")
+    # The engine's stored detection rules (self-host) - see the class docstring.
+    conditions: List[Dict[str, Any]] = Field(default_factory=list)
     severity: str = "medium"
     polarity: str = "failure"
     enabled: bool = True

agentx/monitor/profile.py

@@ -46,7 +46,7 @@ def update(
         """Update (and enable, if not already) this agent's Monitor profile. Only fields passed
         here are changed; everything else on the existing profile is left as is.
 
-        Self-host only: ``coverage_mode``/``sample_rate``/``retention_days``,
+        Self-host only: ``coverage_mode``/``sample_rate``/``retention_days``, ``dataset_id``,
         and ``threshold_overrides["latencyMs"]`` are project-level defaults now (see
         ``MonitorProfile``'s docstring) - set them via the dashboard's Platform Settings screen
         instead, passing them here is accepted but has no effect. ``enabled``/