diff --git a/A365_DOCUMENTATION.md b/A365_DOCUMENTATION.md index c0c0155f..dfdd21f9 100644 --- a/A365_DOCUMENTATION.md +++ b/A365_DOCUMENTATION.md @@ -180,6 +180,68 @@ Provided agent ID, name, blueprint ID, platform ID, and version are recorded und `version`). Other target identity fields are not emitted, and the source agent and tenant attributes remain unchanged. +### Tool skill and command refinements + +Supply explicit `ToolCallDetails.refinementDetails` from the tool API or runtime state, +including on failed attempts. The exported `ExecuteToolRefinementDetails` union accepts +`SkillRefinementDetails` (`kind: "load_skill"` or `"read_skill_resource"`) and +`ExecuteCommandRefinementDetails` (`kind: "execute_command"`). Skill refinements do not +accept process fields. An invalid `kind` throws before creating a span. + +All refinements accept `skillName`, `description`, `sourceUri`, and `resourceName`, +emitted as `gen_ai.skill.name`, `gen_ai.skill.description`, `gen_ai.skill.source.uri`, +and `gen_ai.skill.resource.name`. Command refinements additionally accept +`executableName`, `executablePath`, and numeric `exitCode`, emitted as +`process.executable.name`, `process.executable.path`, and `process.exit.code`. +Only supply executable name/path when the tool directly executes a single process, +not for an opaque shell expression or pipeline. String fields that are absent, empty, +or whitespace-only are omitted; nonblank values are recorded unchanged. + +```typescript +import { ExecuteToolScope } from "@microsoft/opentelemetry"; +import type { ToolCallDetails } from "@microsoft/opentelemetry"; + +const details: ToolCallDetails = { + toolName: "shell", + refinementDetails: { + kind: "execute_command", + skillName: "code-review", + sourceUri: "file:///opt/skills/code-review", + resourceName: "scripts/check.py", + executableName: "python", + executablePath: "/usr/bin/python", + }, +}; +const scope = ExecuteToolScope.start( + { conversationId: "conv-123" }, + details, + { agentId: "agent-1", tenantId: "tenant-1" }, +); +// Record the actual exit code reported by the runtime, including zero for success. +scope.recordProcessExitCode(0); +scope.dispose(); +``` + +For `load_skill`, the span name is `execute_tool {toolName} {skillName}`, falling back +to `execute_tool {toolName}` without a nonblank skill name. For resource readers and +commands, a nonblank resource name takes precedence: +`execute_tool {toolName} {skillName} {resourceName}` (omit the skill name if unavailable). +Without a resource name, commands append the executable name if available; resource +readers retain the ordinary tool span name, even if a skill name is present. + +The operation remains `execute_tool`; no extra spans are created. No refinement is +inferred from a tool name or opaque arguments. Existing tool, endpoint, transfer, +source-agent, and tenant attributes are unchanged. `recordProcessExitCode()` records +the reported numeric value, including zero, and can replace an initially supplied +exit code. It does not infer failure or clear an existing error; use `recordError()` +when the tool reports an error. + +Skill descriptions and source URIs may contain sensitive information. Supply the +skill's source URI, not a temporary materialization path, and remove credentials, +access tokens, and other sensitive components before recording it. Refinement metadata +is explicitly supplied and is not automatically scrubbed or gated by `enableSensitiveData`. +Skill identity/version and skill metrics proposals are not part of this API. + ### Request and response attributes `InvokeAgentScope`, `InferenceScope`, and `ExecuteToolScope` accept `request.sessionId`. diff --git a/CHANGELOG.md b/CHANGELOG.md index 99fb1f55..4d7c338c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ - **Breaking change on Unix (Linux/macOS):** Agent365 offline buffering/replay now requires an explicitly configured, existing, absolute `durableDelivery.storageDirectory`. No automatic Unix temporary-directory fallback or root creation. Missing/invalid roots are logged and disable only disk buffering/replay; online export continues. Customers provision and maintain ownership, permissions (normally a private service-account-owned `0700` root), ACLs, and path-replacement protection; the SDK does not inspect or repair Unix access controls. Windows behavior, explicit-root cache identities, record compatibility, and retention limits are unchanged. Old temporary backlogs are not automatically migrated; see [storage provisioning and migration](A365_DOCUMENTATION.md#storage-provisioning-and-migration). ### Features Added +- Add explicit skill-loading, skill-resource-reading, and command refinements to `ExecuteToolScope` through typed `refinementDetails`. Emit skill/process attributes and semantic-convention span names, and support recording process exit codes after execution without changing error status. Ordinary tool calls remain unchanged. - Add optional full tool endpoint URLs (`endpoint.url` -> `url.full`) and typed execute-tool transfer metadata for agent, human, and workflow targets. Export transfer enums and contracts, validate incompatible targets before creating spans, and preserve source agent identity. - Add privacy-safe Model Observation attributes to `InferenceScope`, including the canonical served response model and cache read/write input-token counts. Attributes can be supplied when the scope starts or recorded after the response is available; inference operations accept enum values or canonical strings and use canonical OpenTelemetry names for both operation attributes and span names. Cache-token semantic-convention constants are exported. [#261](https://github.com/microsoft/opentelemetry-distro-javascript/pull/261) diff --git a/src/a365/constants.ts b/src/a365/constants.ts index fd7d2c06..8a574916 100644 --- a/src/a365/constants.ts +++ b/src/a365/constants.ts @@ -199,6 +199,21 @@ export class OpenTelemetryConstants { /** Attribute key for the tool type (`gen_ai.tool.type`). */ public static readonly GEN_AI_TOOL_TYPE_KEY = "gen_ai.tool.type"; + /** Skill name supplied by the tool runtime. */ + public static readonly GEN_AI_SKILL_NAME_KEY = "gen_ai.skill.name"; + /** Skill description supplied by the tool runtime. */ + public static readonly GEN_AI_SKILL_DESCRIPTION_KEY = "gen_ai.skill.description"; + /** Caller-scrubbed source URI of the skill. */ + public static readonly GEN_AI_SKILL_SOURCE_URI_KEY = "gen_ai.skill.source.uri"; + /** Skill-relative resource or script name. */ + public static readonly GEN_AI_SKILL_RESOURCE_NAME_KEY = "gen_ai.skill.resource.name"; + /** Name of a directly executed process. */ + public static readonly PROCESS_EXECUTABLE_NAME_KEY = "process.executable.name"; + /** Path of a directly executed process. */ + public static readonly PROCESS_EXECUTABLE_PATH_KEY = "process.executable.path"; + /** Reported exit code of a command or process. */ + public static readonly PROCESS_EXIT_CODE_KEY = "process.exit.code"; + // ── Tenant ─────────────────────────────────────────────────────── /** Attribute key for the tenant identifier (`microsoft.tenant.id`). */ diff --git a/src/a365/contracts.ts b/src/a365/contracts.ts index f4e73743..edfc20fe 100644 --- a/src/a365/contracts.ts +++ b/src/a365/contracts.ts @@ -480,6 +480,46 @@ export interface InvokeAgentScopeDetails { responseParameters?: GenAiResponseParameters; } +/** Explicit metadata for skill loading or resource reading. Process fields are command-only. */ +export interface SkillRefinementDetails { + /** The skill operation performed by the tool. */ + kind: "load_skill" | "read_skill_resource"; + /** Skill name, when available. */ + skillName?: string; + /** Skill description. May contain sensitive information. */ + description?: string; + /** Skill source URI, not a temporary path. Callers must remove credentials and tokens. */ + sourceUri?: string; + /** Skill-relative resource name, such as `references/policy.md`. */ + resourceName?: string; + executableName?: never; + executablePath?: never; + exitCode?: never; +} + +/** Explicit command metadata, optionally associated with a skill's bundled script. */ +export interface ExecuteCommandRefinementDetails { + /** Identifies a command-executing tool. */ + kind: "execute_command"; + /** Executable name, only when directly executing a single process. */ + executableName?: string; + /** Executable path, only when directly executing a single process. */ + executablePath?: string; + /** Reported exit code for a single command or process, including zero. */ + exitCode?: number; + /** Skill name, when executing a skill's script. */ + skillName?: string; + /** Skill description. May contain sensitive information. */ + description?: string; + /** Skill source URI, not a temporary path. Callers must remove credentials and tokens. */ + sourceUri?: string; + /** Skill-relative script name, such as `scripts/check.py`. */ + resourceName?: string; +} + +/** Explicit skill or command metadata; never inferred from tool names or arguments. */ +export type ExecuteToolRefinementDetails = SkillRefinementDetails | ExecuteCommandRefinementDetails; + /** Details of a tool call made by an agent. */ export interface ToolCallDetails { /** Name of the tool being called (required). */ @@ -496,6 +536,8 @@ export interface ToolCallDetails { endpoint?: ServiceEndpoint; /** Optional explicit transfer performed by this tool call. */ transferDetails?: TransferDetails; + /** Optional explicit skill or command refinement, including on failed attempts. */ + refinementDetails?: ExecuteToolRefinementDetails; } /** Details for an inference call. */ diff --git a/src/a365/index.ts b/src/a365/index.ts index 5a1c32f4..6957e844 100644 --- a/src/a365/index.ts +++ b/src/a365/index.ts @@ -83,6 +83,9 @@ export type { GenAiResponseParameters, InvokeAgentScopeDetails, ToolCallDetails, + ExecuteToolRefinementDetails, + SkillRefinementDetails, + ExecuteCommandRefinementDetails, TransferDetails, InferenceDetails, InferenceResponse, diff --git a/src/a365/scopes/ExecuteToolScope.ts b/src/a365/scopes/ExecuteToolScope.ts index 648eab82..7ca374b7 100644 --- a/src/a365/scopes/ExecuteToolScope.ts +++ b/src/a365/scopes/ExecuteToolScope.ts @@ -63,6 +63,31 @@ export class ExecuteToolScope extends OpenTelemetryScope { } } + const refinement = details.refinementDetails; + if ( + refinement && + refinement.kind !== "load_skill" && + refinement.kind !== "read_skill_resource" && + refinement.kind !== "execute_command" + ) { + throw new RangeError("ExecuteToolScope: invalid refinement kind"); + } + + let spanName = `${OpenTelemetryConstants.EXECUTE_TOOL_OPERATION_NAME} ${details.toolName}`; + if (refinement?.kind === "load_skill") { + if (refinement.skillName?.trim()) { + spanName += ` ${refinement.skillName}`; + } + } else if (refinement) { + if (refinement.resourceName?.trim()) { + spanName += refinement.skillName?.trim() + ? ` ${refinement.skillName} ${refinement.resourceName}` + : ` ${refinement.resourceName}`; + } else if (refinement.kind === "execute_command" && refinement.executableName?.trim()) { + spanName += ` ${refinement.executableName}`; + } + } + const resolvedSpanDetails: SpanDetails = { spanKind: SpanKind.INTERNAL, ...spanDetails, @@ -70,7 +95,7 @@ export class ExecuteToolScope extends OpenTelemetryScope { super( OpenTelemetryConstants.EXECUTE_TOOL_OPERATION_NAME, - `${OpenTelemetryConstants.EXECUTE_TOOL_OPERATION_NAME} ${details.toolName}`, + spanName, agentDetails, resolvedSpanDetails, userDetails, @@ -90,6 +115,27 @@ export class ExecuteToolScope extends OpenTelemetryScope { this.setTagMaybe(OpenTelemetryConstants.GEN_AI_TOOL_CALL_ID_KEY, toolCallId); this.setTagMaybe(OpenTelemetryConstants.GEN_AI_TOOL_DESCRIPTION_KEY, description); + if (refinement) { + const textAttributes: [string, string | undefined][] = [ + [OpenTelemetryConstants.GEN_AI_SKILL_NAME_KEY, refinement.skillName], + [OpenTelemetryConstants.GEN_AI_SKILL_DESCRIPTION_KEY, refinement.description], + [OpenTelemetryConstants.GEN_AI_SKILL_SOURCE_URI_KEY, refinement.sourceUri], + [OpenTelemetryConstants.GEN_AI_SKILL_RESOURCE_NAME_KEY, refinement.resourceName], + ]; + if (refinement.kind === "execute_command") { + textAttributes.push( + [OpenTelemetryConstants.PROCESS_EXECUTABLE_NAME_KEY, refinement.executableName], + [OpenTelemetryConstants.PROCESS_EXECUTABLE_PATH_KEY, refinement.executablePath], + ); + this.setTagMaybe(OpenTelemetryConstants.PROCESS_EXIT_CODE_KEY, refinement.exitCode); + } + for (const [key, value] of textAttributes) { + if (value?.trim()) { + this.setTagMaybe(key, value); + } + } + } + if (transfer) { this.setTagMaybe(OpenTelemetryConstants.TRANSFER_MODE_KEY, transfer.mode); this.setTagMaybe(OpenTelemetryConstants.TRANSFER_TARGET_TYPE_KEY, targetType); @@ -119,6 +165,11 @@ export class ExecuteToolScope extends OpenTelemetryScope { } } + /** Records a reported command or process exit code without changing the span's error status. */ + public recordProcessExitCode(exitCode: number): void { + this.setTagMaybe(OpenTelemetryConstants.PROCESS_EXIT_CODE_KEY, exitCode); + } + /** * Records response information for telemetry tracking. * Objects are serialized to JSON automatically. diff --git a/src/index.ts b/src/index.ts index 4f11faf3..49ab5212 100644 --- a/src/index.ts +++ b/src/index.ts @@ -82,6 +82,9 @@ export type { GenAiResponseParameters, InvokeAgentScopeDetails, ToolCallDetails, + ExecuteToolRefinementDetails, + SkillRefinementDetails, + ExecuteCommandRefinementDetails, TransferDetails, InferenceDetails, InferenceResponse, diff --git a/test/internal/unit/a365/scopes.test.ts b/test/internal/unit/a365/scopes.test.ts index a68f0d0c..dc69cef0 100644 --- a/test/internal/unit/a365/scopes.test.ts +++ b/test/internal/unit/a365/scopes.test.ts @@ -2,7 +2,7 @@ // Licensed under the MIT License. import { describe, it, expect, beforeAll, afterAll, afterEach, vi, beforeEach } from "vitest"; -import { trace, SpanKind, context as otelContext } from "@opentelemetry/api"; +import { trace, SpanKind, SpanStatusCode, context as otelContext } from "@opentelemetry/api"; import { BasicTracerProvider, InMemorySpanExporter, @@ -31,6 +31,9 @@ import type { GenAiResponseParameters, InvokeAgentScopeDetails, ToolCallDetails, + ExecuteToolRefinementDetails, + SkillRefinementDetails, + ExecuteCommandRefinementDetails, InferenceDetails, UserDetails, OutputResponse, @@ -39,6 +42,33 @@ import type { import { InferenceOperationType, MessageRole } from "../../../../src/a365/index.js"; import { safeSerializeToJson } from "../../../../src/a365/message-utils.js"; import { partitionByIdentity } from "../../../../src/a365/exporter/utils.js"; +import type { + ExecuteToolRefinementDetails as RootExecuteToolRefinementDetails, + SkillRefinementDetails as RootSkillRefinementDetails, + ExecuteCommandRefinementDetails as RootExecuteCommandRefinementDetails, +} from "../../../../src/index.js"; + +type Expect = T; +type _RefinementRootExportMatches = Expect< + RootExecuteToolRefinementDetails extends ExecuteToolRefinementDetails ? true : false +>; +type _SkillRootExportMatches = Expect< + RootSkillRefinementDetails extends SkillRefinementDetails ? true : false +>; +type _CommandRootExportMatches = Expect< + RootExecuteCommandRefinementDetails extends ExecuteCommandRefinementDetails ? true : false +>; +type _SkillCannotAcceptProcessFields = Expect< + { kind: "load_skill"; executableName: string } extends SkillRefinementDetails ? false : true +>; +type _ResourceCannotAcceptProcessFields = Expect< + { kind: "read_skill_resource"; exitCode: number } extends ExecuteToolRefinementDetails + ? false + : true +>; +type _InvalidRefinementKindIsRejected = Expect< + { kind: "unknown" } extends ExecuteToolRefinementDetails ? false : true +>; let sharedExporter: InMemorySpanExporter; let contextManager: AsyncLocalStorageContextManager; @@ -491,6 +521,261 @@ describe("Scopes", () => { }); describe("ExecuteToolScope", () => { + it.each<{ refinementDetails: ExecuteToolRefinementDetails; expectedName: string }>([ + { + refinementDetails: { kind: "load_skill", skillName: "code-review" }, + expectedName: "execute_tool custom_tool code-review", + }, + { + refinementDetails: { + kind: "load_skill", + skillName: "code-review", + resourceName: "SKILL.md", + }, + expectedName: "execute_tool custom_tool code-review", + }, + { + refinementDetails: { kind: "load_skill", skillName: " " }, + expectedName: "execute_tool custom_tool", + }, + { + refinementDetails: { + kind: "read_skill_resource", + skillName: "code-review", + resourceName: "references/policy.md", + }, + expectedName: "execute_tool custom_tool code-review references/policy.md", + }, + { + refinementDetails: { + kind: "read_skill_resource", + resourceName: "references/policy.md", + }, + expectedName: "execute_tool custom_tool references/policy.md", + }, + { + refinementDetails: { kind: "read_skill_resource", skillName: "code-review" }, + expectedName: "execute_tool custom_tool", + }, + { + refinementDetails: { + kind: "execute_command", + skillName: "code-review", + resourceName: "scripts/check.py", + executableName: "python", + }, + expectedName: "execute_tool custom_tool code-review scripts/check.py", + }, + { + refinementDetails: { + kind: "execute_command", + resourceName: "scripts/check.py", + executableName: "python", + }, + expectedName: "execute_tool custom_tool scripts/check.py", + }, + { + refinementDetails: { + kind: "execute_command", + skillName: "code-review", + resourceName: " \t", + executableName: "python", + }, + expectedName: "execute_tool custom_tool python", + }, + { + refinementDetails: { kind: "execute_command", skillName: "code-review" }, + expectedName: "execute_tool custom_tool", + }, + { + refinementDetails: { kind: "execute_command" }, + expectedName: "execute_tool custom_tool", + }, + ])("uses refinement span-name rules: $expectedName", ({ refinementDetails, expectedName }) => { + const scope = ExecuteToolScope.start( + testRequest, + { toolName: "custom_tool", refinementDetails }, + testAgentDetails, + ); + scope.dispose(); + + const span = sharedExporter.getFinishedSpans().at(-1)!; + expect(span.name).toBe(expectedName); + expect(span.kind).toBe(SpanKind.INTERNAL); + expect(span.attributes["gen_ai.operation.name"]).toBe("execute_tool"); + expect(span.attributes["gen_ai.tool.name"]).toBe("custom_tool"); + }); + + it.each(["load_skill", "read_skill_resource"] as const)( + "records explicit %s metadata without process attributes", + (kind) => { + const refinementDetails: SkillRefinementDetails = { + kind, + skillName: "code-review", + description: "Review changes", + sourceUri: "file:///opt/skills/code-review", + resourceName: "references/policy.md", + }; + Object.assign(refinementDetails, { + executableName: "must-not-be-emitted", + executablePath: "/must-not-be-emitted", + exitCode: 2, + }); + const scope = ExecuteToolScope.start( + testRequest, + { toolName: "custom_tool", refinementDetails }, + testAgentDetails, + ); + scope.dispose(); + + const attributes = sharedExporter.getFinishedSpans().at(-1)!.attributes; + expect(attributes).toMatchObject({ + "gen_ai.skill.name": "code-review", + "gen_ai.skill.description": "Review changes", + "gen_ai.skill.source.uri": "file:///opt/skills/code-review", + "gen_ai.skill.resource.name": "references/policy.md", + }); + expect(Object.keys(attributes).some((key) => key.startsWith("process."))).toBe(false); + }, + ); + + it.each([0, 2])("records command metadata and numeric exit code %i on failure", (exitCode) => { + const scope = ExecuteToolScope.start( + testRequest, + { + toolName: "run", + refinementDetails: { + kind: "execute_command", + skillName: "code-review", + description: "Review changes", + sourceUri: "file:///opt/skills/code-review", + resourceName: "scripts/check.py", + executableName: "python", + executablePath: "/usr/bin/python", + exitCode, + }, + transferDetails: { mode: TransferMode.RETURN_TO_CALLER }, + }, + testAgentDetails, + ); + scope.recordError(new Error("Command failed")); + scope.dispose(); + + const span = sharedExporter.getFinishedSpans().at(-1)!; + expect(span.attributes).toMatchObject({ + "gen_ai.skill.name": "code-review", + "gen_ai.skill.description": "Review changes", + "gen_ai.skill.source.uri": "file:///opt/skills/code-review", + "gen_ai.skill.resource.name": "scripts/check.py", + "process.executable.name": "python", + "process.executable.path": "/usr/bin/python", + "process.exit.code": exitCode, + "microsoft.a365.transfer.mode": "return_to_caller", + "gen_ai.agent.id": testAgentDetails.agentId, + }); + expect(span.status.code).toBe(SpanStatusCode.ERROR); + }); + + it("omits blank refinement attributes without trimming supplied values", () => { + const scope = ExecuteToolScope.start( + testRequest, + { + toolName: "shell", + refinementDetails: { + kind: "execute_command", + skillName: "", + description: " \t", + sourceUri: "\n", + resourceName: " ", + executableName: " python ", + executablePath: "", + }, + }, + testAgentDetails, + ); + scope.dispose(); + + const span = sharedExporter.getFinishedSpans().at(-1)!; + expect(span.name).toBe("execute_tool shell python "); + expect(span.attributes["process.executable.name"]).toBe(" python "); + expect(span.attributes["process.executable.path"]).toBeUndefined(); + expect(span.attributes["process.exit.code"]).toBeUndefined(); + expect(Object.keys(span.attributes).some((key) => key.startsWith("gen_ai.skill."))).toBe( + false, + ); + }); + + it("does not infer refinements from tool names or arguments", () => { + const scope = ExecuteToolScope.start( + testRequest, + { toolName: "load_skill", arguments: { name: "code-review", executableName: "python" } }, + testAgentDetails, + ); + scope.dispose(); + + const span = sharedExporter.getFinishedSpans().at(-1)!; + expect(span.name).toBe("execute_tool load_skill"); + expect( + Object.keys(span.attributes).some( + (key) => key.startsWith("gen_ai.skill.") || key.startsWith("process."), + ), + ).toBe(false); + }); + + it.each([0, 2])("records late exit code %i without inferring an error", (exitCode) => { + const scope = ExecuteToolScope.start( + testRequest, + { + toolName: "shell", + refinementDetails: { kind: "execute_command", executableName: "python", exitCode: 1 }, + }, + testAgentDetails, + ); + scope.recordProcessExitCode(exitCode); + scope.dispose(); + + const span = sharedExporter.getFinishedSpans().at(-1)!; + expect(span.attributes["process.exit.code"]).toBe(exitCode); + expect(span.status.code).toBe(SpanStatusCode.UNSET); + }); + + it("preserves an explicit error when recording a successful process exit", () => { + const scope = ExecuteToolScope.start( + testRequest, + { toolName: "shell", refinementDetails: { kind: "execute_command" } }, + testAgentDetails, + ); + scope.recordError(new Error("Tool failed after the command completed")); + scope.recordProcessExitCode(0); + scope.dispose(); + + const span = sharedExporter.getFinishedSpans().at(-1)!; + expect(span.attributes["process.exit.code"]).toBe(0); + expect(span.status.code).toBe(SpanStatusCode.ERROR); + }); + + it.each(["unknown", "", undefined])( + "rejects an invalid refinement kind before starting a span: %s", + (kind) => { + const startSpan = vi.spyOn( + trace.getTracer(OpenTelemetryConstants.SOURCE_NAME), + "startSpan", + ); + try { + expect(() => + ExecuteToolScope.start( + testRequest, + { toolName: "custom_tool", refinementDetails: { kind } } as ToolCallDetails, + testAgentDetails, + ), + ).toThrow(/refinement/i); + expect(startSpan).not.toHaveBeenCalled(); + } finally { + startSpan.mockRestore(); + } + }, + ); + it("records the full tool URL without changing host and port attributes", () => { const scope = ExecuteToolScope.start( testRequest,