diff --git a/docs/framework/v2-extraction-roadmap.md b/docs/framework/v2-extraction-roadmap.md new file mode 100644 index 00000000..4df2977a --- /dev/null +++ b/docs/framework/v2-extraction-roadmap.md @@ -0,0 +1,118 @@ +# v2 extraction roadmap — what the platform taught us belongs in the framework + +The ActiveAgents platform (activeagents/activeagents) has been the field lab +for the gems: every gap in `activeagent` or `solid_agent` shows up there as +app-level code. This document is the audit of that code, drawing the line for +the three-layer architecture: + +- **activeagent** — execution: agents, providers, tools, telemetry +- **solid_agent** — persistence: contexts, generations, tool streams, memory, + pricing +- **the platform** — accounts, billing, quotas, hosted UI, multi-tenancy + +The solid_agent side of this audit already landed (enriched tool persistence, +`ModelPricing`, memory/tool-cache contracts). What follows is the +framework-shaped remainder, proposed for v2 — each item exists today as +platform code that any serious consumer of the gem would have to rebuild. + +## Absorb from the platform + +### 1. Per-model capability gating — ✅ shipped +`ActiveAgent::ModelCapabilities` strips parameters the target model rejects +(temperature/top_p on thinking-first Claude, OpenAI o-series/GPT-5) from +prepared prompt parameters before they reach the provider. Extensible via +`ModelCapabilities.register(pattern, unsupported:)`; disable with +`ModelCapabilities.enabled = false`. Remaining for v2: max_tokens vs +max_completion_tokens switching and reasoning-effort awareness. + +### 2. Provider model catalogs +The platform's `/api/provider_models` queries Ollama's live model list, the +Anthropic Models API, and OpenRouter's catalog, with curated fallbacks. +v2: `Provider#models` on the provider contract (vendor SDKs all expose a +listing endpoint), so model pickers and validation stop being app problems. + +### 3. Tool-loop safety — ✅ shipped (turns) +`max_tool_turns` (default 25, per agent/prompt override) now bounds the +tool-calling recursion; hitting the cap emits +`tool_turns_exceeded.active_agent` and returns the messages gathered so far. +Remaining for v2: a token/cost budget alongside the turn cap. + +### 4. Agent-to-agent delegation +`tools_function` only routes back to `self`. The platform built `call_agent` +(sub-agent invocation with a `Thread.current` depth cap) as an app tool. +v2: a first-class delegation primitive — invoke another agent class/instance +as a tool, with depth limits and shared trace/context correlation. + +### 5. Provider error taxonomy + fallback — ✅ taxonomy shipped +`ActiveAgent::Providers::Errors` (RateLimited, ContextLengthExceeded, +AuthenticationFailed, ContentFiltered, ServiceUnavailable, InvalidRequest) +now normalizes vendor exceptions in `with_exception_handling` — classified +by SDK class name, HTTP status, and message heuristics, original preserved +as `#cause` — so `rescue_from` policy is portable across providers. +Remaining for v2: the `generate_with ... fallback: [:anthropic, :ollama]` +chain the taxonomy makes expressible. + +### 6. A real MCP story +Today MCP is pass-through only: `mcps:` options are normalized into each +vendor's *remote* MCP format (the LLM vendor's servers do the connecting). +There is no MCP client (stdio/HTTP, `tools/list` discovery → routable +actions) and no server facade. The platform built an MCP server over its +agents (`run_` tools, `agent://` resources, Bearer auth) as a +controller. v2: both halves — a client that turns any MCP server's tools +into agent actions, and a mountable engine that presents agents as an MCP +server. + +### 7. Tool DSL / schema derivation +`lib/active_agent.rb`'s docstring advertises a `tool def get_weather(...)` +macro that does not exist; tools are hand-written JSON Schema hashes. +solid_agent's `HasTools` (DSL + JSON view templates) already fills this — +v2 should either absorb it or bless it as the canonical declaration path, +not leave two half-standards. + +### 8. Server-side tool implementations +The platform's `AgentToolbox` (safe `fetch_url` with SSRF guard + redirect +caps, `web_search`, a no-eval `Calculator`, allowlisted `browse_page`) is +generic execution code with zero app coupling. It belongs beside the +framework's tool routing, not in a dashboard app — persistence/caching of +results stays solid_agent's. + +## Fix in place (bugs and dead seams found during the audit) + +- ✅ **Fixed**: `telemetry/instrumentation.rb` — provider/model/message-count + attributes now record (was: nonexistent `generation_provider`, private + helpers hidden from `respond_to?`, nonexistent `Base#messages`); the dead + `around_generate` registration is removed. +- ✅ **Fixed**: `redact_attributes` is now consumed — span and span-event + attribute values matching the configured patterns become `[REDACTED]` in + `build_trace_payload`, covering both transmission and local storage. + `capture_bodies` remains reserved (telemetry spans capture no bodies yet; + the raw response on the ActiveSupport::Notifications payload is + in-process only). +- `Observers`/`Interceptors` call `Prompt.register_observer` on an + `ActiveAgent::Prompt` class that doesn't exist; nothing in the generation + path notifies them. Either implement the ActionMailer-style seam + (persistence layers want it) or delete it. +- The dashboard engine ships orphaned platform-shaped models + (`Dashboard::Agent`, `AgentRun`, `SandboxSession`, jobs, migrations) with + no controllers or routes. Decide: wire them (framework-level run + orchestration — run records, status, cancellation would pair with the + tool-loop limits above) or drop them from the gem. + +## Stays in the platform + +Accounts/users, billing and plan quotas, encrypted API/provider key storage, +trace retention by plan, the hosted React dashboard, sandboxes/session +recordings, and the account-scoped `TelemetryTrace` subclass. These touch +tenancy and money; the gems should expose seams (auth hooks, quota +callbacks), never implementations. + +## solid_agent follow-ups (tracked there, listed for completeness) + +- ✅ **Shipped**: `agent_runs` + persisted run progress events — the install + generator now ships an `AgentRun` model with lifecycle, `append_event`, + trace correlation, and `SolidAgent::RunFingerprint` instruction cohorts. +- Evaluation datasets — `docs/agent-md-spec.md` already specifies + `*.test.yml` cases; the platform's rule-criteria scorer and LLM-judge + plumbing are the reference implementation. +- Fold the platform's drifted model copies back onto the generator + templates once the app's Gemfile.lock reaches solid_agent 0.2. diff --git a/lib/active_agent.rb b/lib/active_agent.rb index 7aeb9a31..6c2dcc74 100644 --- a/lib/active_agent.rb +++ b/lib/active_agent.rb @@ -105,6 +105,7 @@ module ActiveAgent autoload :Preview, "active_agent/concerns/preview" autoload :Previews, "active_agent/concerns/preview" autoload :GenerationJob + autoload :ModelCapabilities autoload :Observers, "active_agent/concerns/observers" autoload :Provider, "active_agent/concerns/provider" autoload :Rescue, "active_agent/concerns/rescue" diff --git a/lib/active_agent/base.rb b/lib/active_agent/base.rb index a665b7f6..54a383bc 100644 --- a/lib/active_agent/base.rb +++ b/lib/active_agent/base.rb @@ -304,6 +304,10 @@ def prepare_prompt_parameters # Render out proc/lamda attributes before rendering templates parameters.deep_transform_values! { _1.respond_to?(:call) ? _1.call : _1 } + # Strip parameters the target model rejects (e.g. temperature/top_p + # on thinking-first models) before they reach the provider. + ModelCapabilities.sanitize!(parameters) + # Apply Callbacks parameters.merge!( trace_id: prompt_options[:trace_id] || SecureRandom.uuid, diff --git a/lib/active_agent/model_capabilities.rb b/lib/active_agent/model_capabilities.rb new file mode 100644 index 00000000..9e9c5f84 --- /dev/null +++ b/lib/active_agent/model_capabilities.rb @@ -0,0 +1,89 @@ +# frozen_string_literal: true + +module ActiveAgent + # Per-model capability quirks, applied before a request reaches the + # provider. Vendors ship models that reject otherwise-standard sampling + # parameters (thinking-first models steered by prompting/effort instead) + # with an API 400 — this registry strips those parameters up front so an + # agent configured with a shared temperature keeps working across model + # switches. + # + # The built-in rules cover the known families; apps can extend the + # registry for new or self-hosted models: + # + # @example Register a custom rule + # ActiveAgent::ModelCapabilities.register(/\Amy-reasoning-model/, unsupported: [:temperature, :top_p]) + # + # @example Disable sanitization entirely + # ActiveAgent::ModelCapabilities.enabled = false + module ModelCapabilities + SAMPLING_PARAMS = [ :temperature, :top_p ].freeze + + # Model families that reject sampling parameters with a 400: + # - Anthropic thinking-first models (Opus 4.7+, Opus 5, Sonnet 5, + # Fable 5 / Mythos 5) + # - OpenAI reasoning models (o-series, GPT-5 family) + BUILTIN_RULES = [ + { pattern: /\Aclaude-(opus-5|opus-4-[78]|sonnet-5|fable-5|mythos-5)/, unsupported: SAMPLING_PARAMS }, + { pattern: /\A(o1|o3|o4)(-|$)/, unsupported: SAMPLING_PARAMS }, + { pattern: /\Agpt-5/, unsupported: SAMPLING_PARAMS } + ].freeze + + class << self + # Master switch; on by default. Set false to send parameters through + # untouched (the vendor then enforces its own rules). + attr_writer :enabled + + def enabled + return @enabled unless @enabled.nil? + + true + end + + # Registers an app-defined capability rule ahead of the built-ins. + # + # @param pattern [Regexp] matched against the model name + # @param unsupported [Array] parameter keys the model rejects + def register(pattern, unsupported:) + custom_rules << { pattern: pattern, unsupported: unsupported.map(&:to_sym) } + end + + def custom_rules + @custom_rules ||= [] + end + + def reset! + @custom_rules = [] + @enabled = nil + end + + # @return [Array] parameter keys the model rejects + def unsupported_params(model) + return [] if model.nil? + + (custom_rules + BUILTIN_RULES).each do |rule| + return rule[:unsupported] if model.to_s.match?(rule[:pattern]) + end + [] + end + + def sampling_supported?(model) + (unsupported_params(model) & SAMPLING_PARAMS).empty? + end + + # Strips parameters the model rejects, in place. Returns the removed + # keys (empty when nothing applied). + # + # @param parameters [Hash] prepared prompt parameters (must carry :model) + # @return [Array] removed parameter keys + def sanitize!(parameters) + return [] unless enabled + return [] unless parameters.is_a?(Hash) + + removed = unsupported_params(parameters[:model]).select { |key| parameters.key?(key) } + removed.each { |key| parameters.delete(key) } + removed + end + end + end +end diff --git a/lib/active_agent/providers/_base_provider.rb b/lib/active_agent/providers/_base_provider.rb index f2323fbd..165fa8f7 100644 --- a/lib/active_agent/providers/_base_provider.rb +++ b/lib/active_agent/providers/_base_provider.rb @@ -55,7 +55,13 @@ class ProvidersError < StandardError; end :request, :message_stack, # Runtime :stream_broadcaster, :streaming, # Callback (Streams) :tools_function, # Callback (Tools) - :usage_stack # Usage Tracking + :usage_stack, # Usage Tracking + :max_tool_turns, :tool_turns # Tool-loop safety + + # Upper bound on tool-calling round-trips within one generation. A + # model that keeps emitting tool calls otherwise recurses until the + # provider stops it — override per agent/prompt with max_tool_turns:. + DEFAULT_MAX_TOOL_TURNS = 25 # @return [String] e.g., "Anthropic", "OpenAI" def self.service_name @@ -106,6 +112,8 @@ def initialize(kwargs = {}) self.stream_broadcaster = kwargs.delete(:stream_broadcaster) self.streaming = false self.tools_function = kwargs.delete(:tools_function) + self.max_tool_turns = kwargs.delete(:max_tool_turns) || DEFAULT_MAX_TOOL_TURNS + self.tool_turns = 0 self.options = options_klass.new(kwargs.extract!(*options_klass.keys)) self.context = kwargs self.message_stack = [] @@ -344,7 +352,7 @@ def process_prompt_finished(api_response = nil) message_stack.push(*api_messages) end - if (tool_calls = process_prompt_finished_extract_function_calls)&.any? + if (tool_calls = process_prompt_finished_extract_function_calls)&.any? && tool_turn_allowed? process_function_calls(tool_calls) resolve_prompt else @@ -373,6 +381,19 @@ def process_prompt_finished(api_response = nil) end end + # Counts a tool round-trip against the per-generation cap. When the + # cap is hit the loop finishes cleanly with the messages gathered so + # far (a partial result) instead of recursing indefinitely. + # + # @return [Boolean] whether another tool round-trip may run + def tool_turn_allowed? + self.tool_turns += 1 + return true if max_tool_turns.nil? || tool_turns <= max_tool_turns + + instrument("tool_turns_exceeded.active_agent", limit: max_tool_turns) + false + end + # @abstract # @param api_response [Object] # @return [Array, nil] diff --git a/lib/active_agent/providers/concerns/exception_handler.rb b/lib/active_agent/providers/concerns/exception_handler.rb index 9abb1d08..4431e4b3 100644 --- a/lib/active_agent/providers/concerns/exception_handler.rb +++ b/lib/active_agent/providers/concerns/exception_handler.rb @@ -1,5 +1,7 @@ # frozen_string_literal: true +require_relative "../errors" + module ActiveAgent module Providers # Provides exception handling for provider operations. @@ -54,7 +56,16 @@ def configure_exception_handler(exception_handler: nil) def with_exception_handling(&block) yield rescue => exception - rescue_with_handler(exception) || raise + # Vendor API failures are normalized into the framework taxonomy + # (Errors::RateLimited, Errors::ContextLengthExceeded, ...) so + # rescue_from policy is portable across providers; the original + # exception is preserved as #cause. Anything unrecognizable — + # including ordinary Ruby errors — passes through untouched. + exception = Errors::Taxonomy.normalize( + exception, + provider_tag: (tag_name if respond_to?(:tag_name)) + ) + rescue_with_handler(exception) || raise(exception) nil # Discard handler return value to prevent polluting raw_response end diff --git a/lib/active_agent/providers/errors.rb b/lib/active_agent/providers/errors.rb new file mode 100644 index 00000000..bcb7d725 --- /dev/null +++ b/lib/active_agent/providers/errors.rb @@ -0,0 +1,140 @@ +# frozen_string_literal: true + +module ActiveAgent + module Providers + # Typed provider failures, normalized across vendor SDKs. + # + # Every vendor raises its own exception classes for the same underlying + # conditions (rate limits, context overflows, content filters, outages), + # which makes retry/backoff/fallback policy impossible to express + # portably. The taxonomy classifies vendor errors into a small set of + # framework types — the original exception is preserved as +#cause+, so + # nothing is lost. + # + # @example Portable retry policy + # rescue_from ActiveAgent::Providers::Errors::RateLimited do |error| + # retry_job wait: 30.seconds + # end + # + # @example Fallback on outage + # rescue_from ActiveAgent::Providers::Errors::ServiceUnavailable do |error| + # FallbackAgent.with(params).ask.generate_later + # end + module Errors + # Base class for normalized provider failures. + class ProviderError < StandardError + # @return [Integer, nil] HTTP status from the vendor error, when known + attr_reader :status + + # @return [String, nil] provider tag (e.g. "Anthropic", "OpenAI::Chat") + attr_reader :provider_tag + + def initialize(message = nil, status: nil, provider_tag: nil) + super(message) + @status = status + @provider_tag = provider_tag + end + end + + # 429s / vendor rate & quota limits. Retryable with backoff. + class RateLimited < ProviderError; end + + # The prompt exceeded the model's context window. Not retryable + # without shrinking the input. + class ContextLengthExceeded < ProviderError; end + + # Invalid, expired, or unauthorized credentials (401/403). + class AuthenticationFailed < ProviderError; end + + # The vendor's safety layer refused the request or response. + class ContentFiltered < ProviderError; end + + # Vendor-side failure or overload (5xx, timeouts, connection drops). + # Retryable; a natural trigger for provider fallback. + class ServiceUnavailable < ProviderError; end + + # Malformed or unsupported request the vendor rejected (400/422) + # that doesn't classify more specifically. + class InvalidRequest < ProviderError; end + + # Classifies vendor SDK exceptions into the taxonomy. Unrecognizable + # exceptions (including ordinary Ruby errors) pass through untouched — + # only errors that look like vendor API failures are normalized. + module Taxonomy + # Vendor SDK class names (demodulized) → taxonomy class. Covers the + # official anthropic/openai gems and SDKs following their naming. + NAME_MAP = { + "RateLimitError" => RateLimited, + "AuthenticationError" => AuthenticationFailed, + "PermissionDeniedError" => AuthenticationFailed, + "ContentFilterError" => ContentFiltered, + "InternalServerError" => ServiceUnavailable, + "APIConnectionError" => ServiceUnavailable, + "APIConnectionTimeoutError" => ServiceUnavailable, + "APITimeoutError" => ServiceUnavailable, + "OverloadedError" => ServiceUnavailable, + "ServiceUnavailableError" => ServiceUnavailable, + "BadRequestError" => InvalidRequest, + "UnprocessableEntityError" => InvalidRequest + }.freeze + + STATUS_MAP = { + 400 => InvalidRequest, + 401 => AuthenticationFailed, + 403 => AuthenticationFailed, + 408 => ServiceUnavailable, + 422 => InvalidRequest, + 429 => RateLimited, + 529 => ServiceUnavailable # Anthropic "overloaded" + }.freeze + + CONTEXT_LENGTH_PATTERN = /context length|context_length|maximum context|context window|too many tokens|prompt is too long|input (?:is )?too long/i + CONTENT_FILTER_PATTERN = /content (?:filter|policy|management)|filtered due to|blocked by|safety (?:system|filter)/i + + class << self + # @param exception [Exception] + # @param provider_tag [String, nil] + # @return [Exception] a taxonomy error, or the original exception + # when it doesn't classify + def normalize(exception, provider_tag: nil) + return exception if exception.is_a?(ProviderError) + + klass = classify(exception) + return exception unless klass + + klass.new(exception.message, status: status_of(exception), provider_tag: provider_tag) + end + + # @return [Class, nil] + def classify(exception) + name = exception.class.name.to_s.demodulize + status = status_of(exception) + api_error = NAME_MAP.key?(name) || !status.nil? + return nil unless api_error + + message = exception.message.to_s + return ContextLengthExceeded if CONTEXT_LENGTH_PATTERN.match?(message) + return ContentFiltered if CONTENT_FILTER_PATTERN.match?(message) + + NAME_MAP[name] || STATUS_MAP[status] || (status && status >= 500 ? ServiceUnavailable : nil) + end + + # @return [Integer, nil] + def status_of(exception) + [ :status, :status_code, :http_status, :code ].each do |reader| + next unless exception.respond_to?(reader) + + value = begin + exception.public_send(reader) + rescue StandardError + nil + end + return value if value.is_a?(Integer) + end + nil + end + end + end + end + end +end diff --git a/lib/active_agent/telemetry/configuration.rb b/lib/active_agent/telemetry/configuration.rb index 6ba61474..fc4967c8 100644 --- a/lib/active_agent/telemetry/configuration.rb +++ b/lib/active_agent/telemetry/configuration.rb @@ -50,8 +50,10 @@ class Configuration # @note Reserved: not yet consumed by the tracer/instrumentation. attr_accessor :capture_bodies - # @return [Array] Attributes to redact from traces - # @note Reserved: not yet consumed by the tracer/instrumentation. + # @return [Array] Attribute-key patterns to redact from + # traces (case-insensitive substring match against span and + # span-event attribute keys; matching values become "[REDACTED]" + # before the payload leaves the process) attr_accessor :redact_attributes # @return [String] Service name for trace attribution diff --git a/lib/active_agent/telemetry/instrumentation.rb b/lib/active_agent/telemetry/instrumentation.rb index c1643102..68900c8b 100644 --- a/lib/active_agent/telemetry/instrumentation.rb +++ b/lib/active_agent/telemetry/instrumentation.rb @@ -23,11 +23,6 @@ module Telemetry module Instrumentation extend ActiveSupport::Concern - included do - # Hook into generation lifecycle - around_generate :trace_generation if respond_to?(:around_generate) - end - class_methods do # Installs instrumentation on the agent class. # @@ -60,18 +55,20 @@ def process_prompt Telemetry.trace("#{self.class.name}.#{action_name}", span_type: :root, **{ trace_id: trace_id }.compact) do |span| span.set_attribute("agent.class", self.class.name) span.set_attribute("agent.action", action_name.to_s) - span.set_attribute("agent.provider", provider_name) if respond_to?(:provider_name) - span.set_attribute("agent.model", model_name) if respond_to?(:model_name) + span.set_attribute("agent.provider", provider_name) + span.set_attribute("agent.model", model_name) # Add prompt span prompt_span = span.add_span("agent.prompt", span_type: :prompt) - prompt_span.set_attribute("messages.count", messages.size) if respond_to?(:messages) + if (message_stack = prompt_options[:messages]).respond_to?(:size) + prompt_span.set_attribute("messages.count", message_stack.size) + end prompt_span.finish # Execute generation with LLM span llm_span = span.add_span("llm.generate", span_type: :llm) - llm_span.set_attribute("llm.provider", provider_name) if respond_to?(:provider_name) - llm_span.set_attribute("llm.model", model_name) if respond_to?(:model_name) + llm_span.set_attribute("llm.provider", provider_name) + llm_span.set_attribute("llm.model", model_name) begin result = super @@ -127,7 +124,7 @@ def process_embed Telemetry.trace("#{self.class.name}.embed", span_type: :embedding) do |span| span.set_attribute("agent.class", self.class.name) span.set_attribute("agent.action", "embed") - span.set_attribute("agent.provider", provider_name) if respond_to?(:provider_name) + span.set_attribute("agent.provider", provider_name) begin result = super @@ -150,7 +147,8 @@ def process_embed private def provider_name - self.class.generation_provider&.to_s || "unknown" + klass = prompt_provider_klass + klass.respond_to?(:tag_name) ? klass.tag_name : "unknown" rescue StandardError "unknown" end diff --git a/lib/active_agent/telemetry/tracer.rb b/lib/active_agent/telemetry/tracer.rb index 0edfbad1..3796e868 100644 --- a/lib/active_agent/telemetry/tracer.rb +++ b/lib/active_agent/telemetry/tracer.rb @@ -141,10 +141,26 @@ def build_trace_payload(root_span) environment: configuration.environment, timestamp: Time.current.iso8601(6), resource_attributes: configuration.resource_attributes, - spans: flatten_spans(root_span) + spans: redact_spans(flatten_spans(root_span)) } end + # Redacts span (and span-event) attributes whose keys match any + # configured redact_attributes entry, before the payload leaves the + # process. Matching is case-insensitive substring — deliberately + # over-broad: better to redact a harmless "max_tokens" than to ship + # an "api_key". + # + # @param spans [Array] flattened span data + # @return [Array] + def redact_spans(spans) + patterns = Array(configuration.redact_attributes).map(&:to_s).reject(&:empty?) + return spans if patterns.empty? + + matcher = Regexp.union(patterns.map { |pattern| Regexp.new(Regexp.escape(pattern), Regexp::IGNORECASE) }) + spans.map { |span| redact_span(span, matcher) } + end + # Flattens span hierarchy into array. # # @param span [Span] Root span @@ -157,6 +173,30 @@ def flatten_spans(span) result end + # @param span [Hash] + # @param matcher [Regexp] + # @return [Hash] span with matching attribute values replaced + def redact_span(span, matcher) + span = span.dup + span[:attributes] = redact_hash(span[:attributes], matcher) if span[:attributes].is_a?(Hash) + if span[:events].is_a?(Array) + span[:events] = span[:events].map do |event| + next event unless event.is_a?(Hash) && event[:attributes].is_a?(Hash) + + event.merge(attributes: redact_hash(event[:attributes], matcher)) + end + end + span + end + + REDACTED = "[REDACTED]" + + def redact_hash(attributes, matcher) + attributes.to_h do |key, value| + [ key, key.to_s.match?(matcher) ? REDACTED : value ] + end + end + # Returns whether this trace should be sampled. # # @return [Boolean] diff --git a/test/dashboard/telemetry_correlation_test.rb b/test/dashboard/telemetry_correlation_test.rb index 3d3623f8..1d29304d 100644 --- a/test/dashboard/telemetry_correlation_test.rb +++ b/test/dashboard/telemetry_correlation_test.rb @@ -94,6 +94,37 @@ def ping swap_global_tracer(original_tracer) end + test "instrumented generations record provider and model attributes" do + original_tracer = swap_global_tracer(ActiveAgent::Telemetry::Tracer.new(@configuration)) + + agent_class = Class.new(ApplicationAgent) do + def self.name = "AttributeProbeAgent" + generate_with :mock, model: "mock-model" + + def ping + prompt(message: "hello") + end + end + agent_class.include(ActiveAgent::Telemetry::Instrumentation) + agent_class.instrument_telemetry! + + agent_class.with({}).ping.generate_now + ActiveAgent::Telemetry.flush + + trace = ActiveAgent::TelemetryTrace.order(:created_at).last + root = trace.spans.find { |span| span["type"] == "root" } + llm = trace.spans.find { |span| span["type"] == "llm" } + prompt_span = trace.spans.find { |span| span["type"] == "prompt" } + + assert_equal "Mock", root.dig("attributes", "agent.provider") + assert_equal "mock-model", root.dig("attributes", "agent.model") + assert_equal "Mock", llm.dig("attributes", "llm.provider") + assert_equal "mock-model", llm.dig("attributes", "llm.model") + assert_operator prompt_span.dig("attributes", "messages.count").to_i, :>=, 1 + ensure + swap_global_tracer(original_tracer) + end + private def stored_payload diff --git a/test/dashboard/telemetry_redaction_test.rb b/test/dashboard/telemetry_redaction_test.rb new file mode 100644 index 00000000..120bf6b3 --- /dev/null +++ b/test/dashboard/telemetry_redaction_test.rb @@ -0,0 +1,71 @@ +# frozen_string_literal: true + +require "test_helper" +require_relative "telemetry_trace_test" + +# redact_attributes: configured key patterns are scrubbed from span and +# span-event attributes before the payload leaves the process (transmission +# and local storage share build_trace_payload). +class TelemetryRedactionTest < ActiveSupport::TestCase + TelemetryTraceTest.ensure_table! + + def setup + ActiveAgent::TelemetryTrace.delete_all + + @configuration = ActiveAgent::Telemetry::Configuration.new + @configuration.enabled = true + @configuration.local_storage = true + @configuration.service_name = "dummy" + end + + test "redacts matching span and event attributes before storage" do + tracer = ActiveAgent::Telemetry::Tracer.new(@configuration) + + tracer.trace("SupportAgent.respond") do |span| + span.set_attribute("llm.api_key", "sk-live-123") + span.set_attribute("http.authorization_token", "Bearer abc") + span.set_attribute("llm.model", "claude-sonnet-5") + span.add_event("tool.call", { "password" => "hunter2", "tool.name" => "fetch_url" }) + end + tracer.flush + + trace = ActiveAgent::TelemetryTrace.first + root = trace.spans.first + + assert_equal "[REDACTED]", root.dig("attributes", "llm.api_key") + assert_equal "[REDACTED]", root.dig("attributes", "http.authorization_token") + assert_equal "claude-sonnet-5", root.dig("attributes", "llm.model") + + event = root["events"].first + assert_equal "[REDACTED]", event.dig("attributes", "password") + assert_equal "fetch_url", event.dig("attributes", "tool.name") + end + + test "custom redact_attributes replace the defaults" do + @configuration.redact_attributes = %w[ssn] + tracer = ActiveAgent::Telemetry::Tracer.new(@configuration) + + tracer.trace("SupportAgent.respond") do |span| + span.set_attribute("user.ssn", "000-00-0000") + span.set_attribute("llm.api_key", "left-alone-by-custom-config") + end + tracer.flush + + root = ActiveAgent::TelemetryTrace.first.spans.first + assert_equal "[REDACTED]", root.dig("attributes", "user.ssn") + assert_equal "left-alone-by-custom-config", root.dig("attributes", "llm.api_key") + end + + test "empty redact_attributes disables scrubbing" do + @configuration.redact_attributes = [] + tracer = ActiveAgent::Telemetry::Tracer.new(@configuration) + + tracer.trace("SupportAgent.respond") do |span| + span.set_attribute("llm.api_key", "sk-live-123") + end + tracer.flush + + root = ActiveAgent::TelemetryTrace.first.spans.first + assert_equal "sk-live-123", root.dig("attributes", "llm.api_key") + end +end diff --git a/test/integration/anthropic/common_format/preview_test.rb b/test/integration/anthropic/common_format/preview_test.rb index fa6303ad..1c893b65 100644 --- a/test/integration/anthropic/common_format/preview_test.rb +++ b/test/integration/anthropic/common_format/preview_test.rb @@ -81,8 +81,11 @@ def comprehensive_test assert_includes preview, "Get detailed weather forecast" assert_includes preview, "Find popular attractions" - # Check parameters in YAML section - assert_includes preview, "temperature: 0.7" + # Check parameters in YAML section. The configured temperature is + # stripped by ModelCapabilities — claude-sonnet-5 is a + # thinking-first model that rejects sampling params — so the + # preview reflects the request that will actually be sent. + refute_includes preview, "temperature" assert_includes preview, "max_tokens: 2000" end end diff --git a/test/integration/open_ai/chat/common_format/preview_test.rb b/test/integration/open_ai/chat/common_format/preview_test.rb index 1ab4e91e..e9b0283d 100644 --- a/test/integration/open_ai/chat/common_format/preview_test.rb +++ b/test/integration/open_ai/chat/common_format/preview_test.rb @@ -82,8 +82,11 @@ def comprehensive_test assert_includes preview, "Get detailed weather forecast" assert_includes preview, "Find popular attractions" - # Check parameters in YAML section - assert_includes preview, "temperature: 0.7" + # Check parameters in YAML section. The configured temperature + # is stripped by ModelCapabilities — gpt-5 is a reasoning model + # that rejects sampling params — so the preview reflects the + # request that will actually be sent. + refute_includes preview, "temperature" assert_includes preview, "max_tokens: 2000" end end diff --git a/test/model_capabilities_test.rb b/test/model_capabilities_test.rb new file mode 100644 index 00000000..30c3c453 --- /dev/null +++ b/test/model_capabilities_test.rb @@ -0,0 +1,94 @@ +# frozen_string_literal: true + +require "test_helper" + +class ModelCapabilitiesTest < ActiveSupport::TestCase + teardown { ActiveAgent::ModelCapabilities.reset! } + + test "thinking-first Claude models reject sampling params" do + %w[claude-sonnet-5 claude-opus-5 claude-fable-5 claude-mythos-5 claude-opus-4-7 claude-opus-4-8].each do |model| + assert_not ActiveAgent::ModelCapabilities.sampling_supported?(model), "expected #{model} to reject sampling" + end + end + + test "OpenAI reasoning models reject sampling params" do + %w[o1 o1-mini o3-mini o4-mini gpt-5.1 gpt-5].each do |model| + assert_not ActiveAgent::ModelCapabilities.sampling_supported?(model), "expected #{model} to reject sampling" + end + end + + test "conventional models keep sampling params" do + %w[claude-haiku-4-5 claude-sonnet-4-5 gpt-4o-mini gpt-4.1 qwen3:8b llama3.1:8b].each do |model| + assert ActiveAgent::ModelCapabilities.sampling_supported?(model), "expected #{model} to support sampling" + end + end + + test "sanitize! strips only the rejected params and reports them" do + parameters = { model: "claude-sonnet-5", temperature: 0.7, top_p: 0.9, max_tokens: 512 } + + removed = ActiveAgent::ModelCapabilities.sanitize!(parameters) + + assert_equal [ :temperature, :top_p ], removed.sort_by(&:to_s) + assert_equal({ model: "claude-sonnet-5", max_tokens: 512 }, parameters) + end + + test "sanitize! leaves conventional models untouched" do + parameters = { model: "gpt-4o-mini", temperature: 0.7 } + + assert_empty ActiveAgent::ModelCapabilities.sanitize!(parameters) + assert_equal 0.7, parameters[:temperature] + end + + test "custom rules are consulted ahead of the built-ins" do + ActiveAgent::ModelCapabilities.register(/\Ahouse-model/, unsupported: [ :temperature ]) + + assert_equal [ :temperature ], ActiveAgent::ModelCapabilities.unsupported_params("house-model-2") + parameters = { model: "house-model-2", temperature: 1.0, top_p: 0.5 } + ActiveAgent::ModelCapabilities.sanitize!(parameters) + assert_nil parameters[:temperature] + assert_equal 0.5, parameters[:top_p] + end + + test "disabling the switch passes parameters through" do + ActiveAgent::ModelCapabilities.enabled = false + parameters = { model: "claude-sonnet-5", temperature: 0.7 } + + assert_empty ActiveAgent::ModelCapabilities.sanitize!(parameters) + assert_equal 0.7, parameters[:temperature] + end + + test "prepared prompt parameters are sanitized for the configured model" do + agent_class = Class.new(ApplicationAgent) do + def self.name = "SanitizeProbeAgent" + generate_with :mock, model: "claude-sonnet-5", temperature: 0.7, top_p: 0.9, max_tokens: 256 + + def ping + prompt(message: "hello") + end + end + + agent = agent_class.new + agent.params = {} + agent.process(:ping) + parameters = agent.send(:prepare_prompt_parameters) + + assert_nil parameters[:temperature] + assert_nil parameters[:top_p] + assert_equal 256, parameters[:max_tokens] + assert_equal "claude-sonnet-5", parameters[:model] + end + + test "generation still succeeds end-to-end with stripped params" do + agent_class = Class.new(ApplicationAgent) do + def self.name = "SanitizeRunProbeAgent" + generate_with :mock, model: "claude-fable-5", temperature: 0.2 + + def ping + prompt(message: "hello") + end + end + + response = agent_class.with({}).ping.generate_now + assert response.message.content.present? + end +end diff --git a/test/providers/base_provider_tool_turns_test.rb b/test/providers/base_provider_tool_turns_test.rb new file mode 100644 index 00000000..286c0976 --- /dev/null +++ b/test/providers/base_provider_tool_turns_test.rb @@ -0,0 +1,76 @@ +# frozen_string_literal: true + +require "test_helper" +require_relative "../../lib/active_agent/providers/mock_provider" + +# Tool-loop safety: process_prompt_finished re-enters resolve_prompt while +# the model keeps emitting tool calls. The max_tool_turns cap bounds that +# recursion and finishes cleanly with the messages gathered so far. +class BaseProviderToolTurnsTest < ActiveSupport::TestCase + # A mock provider whose "model" emits a tool call on every response — + # unbounded, this would recurse forever. + class LoopingMockProvider < ActiveAgent::Providers::MockProvider + # Type resolution (service_name/namespace) derives from the class + # name; keep the Mock identity for this test-local subclass. + def self.name = "ActiveAgent::Providers::MockProvider" + + attr_reader :tool_rounds + + def process_prompt_finished_extract_function_calls + [ { name: "spin", input: {}, id: "call_#{object_id}_#{@tool_rounds}" } ] + end + + def process_function_calls(_calls) + @tool_rounds = (@tool_rounds || 0) + 1 + message_stack.push({ role: "user", content: "tool result #{@tool_rounds}" }) + end + end + + test "max_tool_turns bounds the tool-calling recursion" do + provider = LoopingMockProvider.new( + messages: [ { role: "user", content: "go" } ], + max_tool_turns: 3 + ) + + response = provider.prompt + + assert_equal 3, provider.tool_rounds + assert response.present?, "capped loop should still return a response" + assert response.messages.any? + end + + test "hitting the cap emits a tool_turns_exceeded notification" do + events = [] + subscription = ActiveSupport::Notifications.subscribe("tool_turns_exceeded.active_agent") do |*, payload| + events << payload + end + + LoopingMockProvider.new( + messages: [ { role: "user", content: "go" } ], + max_tool_turns: 2 + ).prompt + + assert_equal 1, events.length + assert_equal 2, events.first[:limit] + ensure + ActiveSupport::Notifications.unsubscribe(subscription) + end + + test "the default cap applies when none is configured" do + provider = LoopingMockProvider.new(messages: [ { role: "user", content: "go" } ]) + + provider.prompt + + assert_equal ActiveAgent::Providers::BaseProvider::DEFAULT_MAX_TOOL_TURNS, provider.tool_rounds + end + + test "generations without tool calls are unaffected" do + provider = ActiveAgent::Providers::MockProvider.new( + messages: [ { role: "user", content: "hello there" } ] + ) + + response = provider.prompt + + assert response.message.content.present? + end +end diff --git a/test/providers/errors_taxonomy_test.rb b/test/providers/errors_taxonomy_test.rb new file mode 100644 index 00000000..d607dada --- /dev/null +++ b/test/providers/errors_taxonomy_test.rb @@ -0,0 +1,83 @@ +# frozen_string_literal: true + +require "test_helper" +require_relative "../../lib/active_agent/providers/mock_provider" + +class ErrorsTaxonomyTest < ActiveSupport::TestCase + Errors = ActiveAgent::Providers::Errors + + # Vendor-SDK-shaped exceptions (the official gems expose #status). + class FakeStatusError < StandardError + def initialize(message, status) + super(message) + @status = status + end + attr_reader :status + end + + class RateLimitError < StandardError; end + + test "classifies by vendor class name" do + error = Errors::Taxonomy.normalize(RateLimitError.new("slow down")) + assert_instance_of Errors::RateLimited, error + assert_equal "slow down", error.message + end + + test "classifies by HTTP status" do + assert_instance_of Errors::RateLimited, Errors::Taxonomy.normalize(FakeStatusError.new("429", 429)) + assert_instance_of Errors::AuthenticationFailed, Errors::Taxonomy.normalize(FakeStatusError.new("bad key", 401)) + assert_instance_of Errors::InvalidRequest, Errors::Taxonomy.normalize(FakeStatusError.new("bad params", 400)) + assert_instance_of Errors::ServiceUnavailable, Errors::Taxonomy.normalize(FakeStatusError.new("boom", 503)) + assert_instance_of Errors::ServiceUnavailable, Errors::Taxonomy.normalize(FakeStatusError.new("overloaded", 529)) + end + + test "context overflow and content filter win over the generic 400" do + context_error = Errors::Taxonomy.normalize(FakeStatusError.new("prompt is too long: 250000 tokens > maximum context", 400)) + assert_instance_of Errors::ContextLengthExceeded, context_error + + filter_error = Errors::Taxonomy.normalize(FakeStatusError.new("Response blocked by content filter", 400)) + assert_instance_of Errors::ContentFiltered, filter_error + end + + test "captures status and provider tag" do + error = Errors::Taxonomy.normalize(FakeStatusError.new("429", 429), provider_tag: "Anthropic") + assert_equal 429, error.status + assert_equal "Anthropic", error.provider_tag + end + + test "ordinary Ruby errors pass through untouched" do + original = NoMethodError.new("undefined method") + assert_same original, Errors::Taxonomy.normalize(original) + + plain = StandardError.new("something odd") + assert_same plain, Errors::Taxonomy.normalize(plain) + end + + test "already-normalized errors pass through" do + original = Errors::RateLimited.new("again") + assert_same original, Errors::Taxonomy.normalize(original) + end + + test "provider raises the typed error with the original as cause" do + provider = ActiveAgent::Providers::MockProvider.new(service: "Mock") + + raised = assert_raises(Errors::RateLimited) do + provider.send(:with_exception_handling) { raise FakeStatusError.new("too fast", 429) } + end + + assert_instance_of FakeStatusError, raised.cause + assert_equal "Mock", raised.provider_tag + end + + test "exception_handler receives the typed error" do + seen = nil + provider = ActiveAgent::Providers::MockProvider.new( + service: "Mock", + exception_handler: ->(exception) { seen = exception } + ) + + provider.send(:with_exception_handling) { raise FakeStatusError.new("too fast", 429) } + + assert_instance_of Errors::RateLimited, seen + end +end