Skip to content

docs(TSP-1427): add eval metrics to analytics breakdown tables - #800

Open
claude[bot] wants to merge 1 commit into
mainfrom
docs/TSP-1427
Open

claude[bot] wants to merge 1 commit into
mainfrom
docs/TSP-1427

Conversation

@claude

@claude claude Bot commented Sep 1, 2026

Copy link
Copy Markdown

Summary

Updates enterprise/analytics.mdx to document the new evaluation metrics features added in PR #17207:

  • Workforce Breakdown: Added Eval score, Alarm status, and Live evals columns; updated Actions & Credits description to note it includes live eval monitoring spend
  • Agent Breakdown: Same three new columns plus the Actions & Credits live eval spend note
  • Cost breakdown modal: New accordion documenting how to click Credits/Actions values to open the cost breakdown modal, which separates task execution spend from live eval monitoring spend (and is distinct from the eval run breakdown on individual eval results)
  • Empty states: Documented 'Set up evals →' CTA and 'Insufficient data' states within the Eval score column descriptions
  • Feature flag callout: Added <Info> callout to Workforce and Agent Breakdown sections noting that eval columns are gated behind account-level Evals access, using the same rollout language as the Evals page
  • Cross-references: Linked to /build/agents/build-your-agent/evals#monitor when referencing Monitor dashboards and eval alarms

Linear issue: https://linear.app/relevance/issue/TSP-1427/

Documents the new Eval score, Alarm status, and Live evals columns in
the Workforce and Agent Breakdown sections, adds the cost breakdown
modal description, documents empty states for evals, and adds the
Evals feature flag callout to relevant sections.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@claude claude Bot added the docs-drafter Documentation drafted by Claude label Sep 1, 2026
@mintlify

mintlify Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
relevanceai 🟢 Ready View Preview Sep 1, 2026, 4:37 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@linear

linear Bot commented Sep 1, 2026

Copy link
Copy Markdown

TSP-1427

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

🎯 Vibe check

Reviewed: 1 file (1 with issues, 0 clean)

Scores

Dimension Score What's holding it back
🔴 Consistency 4/10 Pervasive heading/title case errors across all four heading levels and every Card, Accordion, and Step component. Multiple "agent"/"agents"/"workforce"/"workforces" lowercase misses for product terms. Bold labels inside callouts in three places.
🟡 Technical clarity 8/10 Content is accurate, internal anchor links verified (#monitor, #cost-and-billing, #monitoring-concurrency-usage all resolve). UI element names are specific and actionable throughout.
🟡 Non-technical clarity 7/10 Good plain-English explanations with concrete examples. The bold-label callout fragments (**Pro tip:**, **Filter and Focus**:) read as mini-headers trapped in the wrong container, which is mildly disorienting.
🟡 Structure 6/10 Two accordions ("Task and Action Trends", "Credits") use bullet lists inside them, violating the CLAUDE.md rule on accordion content. Bold labels inside callouts break the same rules. Page-level flow is otherwise logical.

Overall vibe: The content here is genuinely useful — good use of tables for metric definitions, the cost breakdown modal explanation is clear, and the concurrency note correctly cross-references the plans page. The main problem is mechanical: nearly every title-like element is in title case instead of sentence case, and there are several callout/accordion component violations that need fixing before this is merge-ready.

🔧 Issues (29)

Heading sentence case — only proper nouns and product names get capitals:

  • enterprise/analytics.mdx:19## Getting Started## Getting started
  • enterprise/analytics.mdx:37## Key Metrics at a Glance## Key metrics at a glance
  • enterprise/analytics.mdx:63## Dashboard Sections## Dashboard sections
  • enterprise/analytics.mdx:161## Why Use Analytics?## Why use Analytics?

Step titles:

  • enterprise/analytics.mdx:25<Step title="Set Your Filters">"Set your filters"
  • enterprise/analytics.mdx:28<Step title="Explore Your Data">"Explore your data"

Card titles:

  • enterprise/analytics.mdx:45<Card title="Tasks to Review">"Tasks to review"
  • enterprise/analytics.mdx:48<Card title="Actions Used">"Actions used"
  • enterprise/analytics.mdx:51<Card title="Credits Used">"Credits used"
  • enterprise/analytics.mdx:164<Card title="Cost Control">"Cost control"
  • enterprise/analytics.mdx:168<Card title="Performance Optimization">"Performance optimization"
  • enterprise/analytics.mdx:172<Card title="Error Monitoring">"Error monitoring"
  • enterprise/analytics.mdx:176<Card title="Capacity Planning">"Capacity planning"
  • enterprise/analytics.mdx:180<Card title="ROI Visibility">"ROI visibility"

Accordion titles:

  • enterprise/analytics.mdx:65<Accordion title="Task and Action Trends">"Task and action trends"
  • enterprise/analytics.mdx:87<Accordion title="Workforce Breakdown">"Workforce breakdown"
  • enterprise/analytics.mdx:108<Accordion title="Agent Breakdown">"Agent breakdown"
  • enterprise/analytics.mdx:135<Accordion title="Action Breakdown">"Action breakdown"

Product term capitalization — Agent, Workforce are Relevance AI proper nouns:

  • enterprise/analytics.mdx:9 — "how your AI workforce is performing" → "Workforce"
  • enterprise/analytics.mdx:58 — "view individual agent performance" (inside the Note) → "Agent"
  • enterprise/analytics.mdx:87 (accordion body) — "each of your AI workforces" → "Workforces"
  • enterprise/analytics.mdx:113 — "individual agent performance" → "Agent"
  • enterprise/analytics.mdx:153 — "not data specific to a single agent" → "Agent"
  • enterprise/analytics.mdx:169 — "Sort agents by Credits/Task" → "Agents"
  • enterprise/analytics.mdx:173 — "which agents or actions are failing" → "Agents"
  • enterprise/analytics.mdx:185 — "agents with abnormally high" → "Agents"

Bold labels inside callouts — CLAUDE.md: callouts must be a single short paragraph with no bold labels:

  • enterprise/analytics.mdx:33<Tip>**Pro tip:** — remove the bold label; the <Tip> component already signals the register. Rewrite as plain prose: "Scroll to the bottom of any breakdown table to see summary stats…"
  • enterprise/analytics.mdx:58<Note>**Filter and Focus**: — remove the bold label; rewrite as a plain sentence.
  • enterprise/analytics.mdx:89<Info>**Evals feature flag**: inside the Workforce Breakdown accordion (same pattern repeated at line 110 in the Agent Breakdown accordion) — remove bold labels; the <Info> container is sufficient framing.
🧩 Component suggestions (3)
  • enterprise/analytics.mdx:65–73 — "Task and Action Trends" accordion uses **Quickly identify:** + bullet list + **How to use it:**. CLAUDE.md says accordion content should use flowing sentences, not bullet lists. Rewrite as prose: "Hover over any bar to see exact numbers for that day. The tallest bars show peak demand; unexpected spikes or drops may signal issues. Use the date range picker to zoom into specific weeks, and compare task volume against the Credits chart to understand cost patterns."

  • enterprise/analytics.mdx:77–84 — "Credits" accordion has the same structure (bold subheads + bullets). Same fix: convert to flowing sentences that cover expensive days, cost anomalies, and how to compare with the Tasks chart.

  • enterprise/analytics.mdx:41–53 — "Key Metrics at a Glance" uses a four-card <CardGroup> without href links. Per CLAUDE.md, non-navigable descriptors work better as a table or prose. These four metrics (Tasks, Tasks to Review, Actions Used, Credits Used) each have a consistent two-field structure (name + what it shows), making a two-column markdown table a natural fit and easier to scan than cards.

⚠️ Contradictions (0)

No contradictions found. The concurrency chart description at line 153 ("organization-wide concurrency data — not data specific to a single agent") matches the plans page wording exactly. The Evals rollout language in the Info banners (lines 89–90, 110–111) is consistent with the rollout note on evals.mdx:8. All internal anchors resolve.

🔋 Credit usage
Item Count
Files reviewed 1
Context pages read 3
Total lines processed ~1,296

Files read: enterprise/analytics.mdx (187 lines), build/agents/build-your-agent/evals.mdx (447 lines), admin/subscriptions/plans.mdx (203 lines), enterprise/quick-start-guide.mdx (459 lines)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-drafter Documentation drafted by Claude

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant