Skip to content

tags.md

Latest commit

 

History

History
274 lines (208 loc) · 7.44 KB

tags.md

File metadata and controls

274 lines (208 loc) · 7.44 KB
Raw

Robot Framework Test Tags Reference

This document defines the standard tags used across the Chronicle test suite.

Simplified Tag Set

Chronicle uses 11 business tags and three execution-requirement tags. The 14 permitted tags are defined in robot-tags.json, which is the canonical machine-readable allowlist.

Tag Format

IMPORTANT: Tags must be tab-separated, not space-separated.

# Correct - tabs between tags
[Tags]    permissions	conversation

# Incorrect - spaces between tags
[Tags]    permissions conversation

Approved Tags

Core Component Tags

permissions - Authentication, authorization, access control, user management

  • User login/logout
  • Admin operations
  • Role-based access control
  • Data isolation between users
  • Security tests

conversation - Conversation management and transcription

  • Conversation CRUD operations
  • Transcript processing and versioning
  • Speaker diarization
  • Conversation metadata

memory - Memory extraction, storage, and retrieval

  • Memory creation from conversations
  • Semantic search
  • Memory versioning
  • Memory reprocessing

chat - Chat service and sessions

  • Chat session management
  • Message handling
  • Chat history
  • Chat statistics

queue - Job queue management and monitoring

  • RQ worker status
  • Job tracking
  • Queue health
  • Background task processing

health - System health and readiness checks

  • Health endpoints
  • Service status
  • Readiness probes
  • System metrics

infra - Infrastructure and system-level operations

  • Service configuration
  • System administration
  • Client management
  • General system operations

Audio Processing Tags

audio-upload - Audio file upload and batch processing

  • File upload endpoints
  • Batch audio processing
  • Audio file CRUD operations

audio-batch - Batch audio processing operations

  • Multiple file processing
  • Batch job management

audio-streaming - Real-time audio streaming

  • WebSocket audio streaming
  • Real-time transcription
  • Live audio processing

Integration Tags

e2e - End-to-end integration tests

  • Multi-component workflows
  • Full pipeline testing
  • Cross-service integration

Special Tags

There is no requires-api-keys tag. Tests are never selected by whether a credential is present. Which backing services are real is a property of the service profile a run uses (tests/profiles.yml), not of individual tests, and the suite is identical in every profile. If a test seems to need a real provider, the stub is not faithful enough -- record a cassette (make record-cassettes) rather than tagging the test out of CI.

slow - Tests requiring long timeouts (>30s) or infrastructure operations

  • Backend restart tests (service stop/start cycles)
  • Connection resilience tests
  • Heavy integration tests with multiple service restarts
  • Excluded from default make test runs for faster feedback
  • Run explicitly with make test-slow or make test-all-with-slow-and-sdk

sdk - Tests for unreleased SDK functionality

  • SDK integration tests
  • SDK authentication tests
  • SDK API endpoint tests
  • Excluded from default make test runs until SDK is released
  • Run explicitly with make test-sdk when developing SDK features

requires-gpu - Tests requiring GPU hardware and CUDA

  • Actual ASR model loading and inference
  • GPU-accelerated transcription quality validation
  • Tests using real NeMo/Parakeet models
  • Excluded from standard CI runs (no GPU available)
  • Run explicitly with make test-asr-gpu on GPU-enabled systems

Tag Usage Guidelines

Single Tag per Test (Preferred)

Most tests should have one primary tag indicating the main component being tested:

# Authentication test
[Tags]    permissions

# Memory search test
[Tags]    memory

# WebSocket streaming test
[Tags]    audio-streaming

Multiple Tags (When Necessary)

Use 2-3 tags only when testing interactions between components:

# Conversation creates memories
[Tags]    conversation	memory

# Admin managing queues
[Tags]    permissions	queue

# E2E audio upload to memory
[Tags]    e2e	audio-upload	memory

Tag Selection Decision Tree

  1. Is it about users/auth/security? → permissions
  2. Is it about audio upload/files? → audio-upload
  3. Is it about WebSocket/streaming? → audio-streaming
  4. Is it about conversations? → conversation
  5. Is it about memories? → memory
  6. Is it about chat? → chat
  7. Is it about queues/jobs? → queue
  8. Is it about health checks? → health
  9. Is it end-to-end? → e2e
  10. Is it infrastructure/config? → infra
  11. Does it take >30s or restart services? → Add slow tag
  12. Is it for unreleased SDK features? → Add sdk tag
  13. Does it require GPU hardware? → Add requires-gpu tag

Examples

# Good - Clear single tag
[Tags]    permissions
[Tags]    conversation
[Tags]    memory

# Good - Component interaction
[Tags]    conversation	memory
[Tags]    permissions	queue

# Good - E2E with components
[Tags]    e2e	audio-streaming	conversation

# Bad - Too many tags
[Tags]    permissions	conversation	memory	queue	health

# Bad - Use infra instead
[Tags]    configuration
[Tags]    system-admin

# Bad - Non-existent tags
[Tags]    negative
[Tags]    positive
[Tags]    security  # Use 'permissions' instead

Prohibited Tags

DO NOT create or use any tags other than the 14 approved tags above.

Commonly misused tags that should NOT be used:

  • ❌ positive, negative - Test outcome is in the results, not tags
  • ❌ security, auth, admin, user - Use permissions instead
  • ❌ websocket, streaming - Use audio-streaming instead
  • ❌ upload, crud - Use audio-upload instead
  • ❌ integration - Use e2e instead
  • ❌ system, config, service - Use infra instead
  • ❌ rq, jobs, worker - Use queue instead
  • ❌ Any other tags not in the approved list above

Running Tests by Tag

# Run all permission tests
robot --include permissions tests/

# Run conversation and memory tests
robot --include conversationORmemory tests/

# Run only E2E tests
robot --include e2e tests/

# Run everything except E2E
robot --exclude e2e tests/

# Run audio-related tests
robot --include audio-upload --include audio-streaming tests/

Updating Tags

Before Adding a New Tag

STOP! Ask yourself:

  1. Can I use one of the 11 business tags?
  2. Does the test genuinely need one of the three execution-requirement tags?
  3. Is a new tag really necessary for test selection?
  4. Have I checked with the team?

New tags require team approval and must be added to this document first.

Changing Existing Tags

When updating tags across test files:

  1. Update all affected test files
  2. Update robot-tags.json
  3. Update this document (tags.md)
  4. Update TESTING_GUIDELINES.md if rules changed
  5. Document the change in the commit message

Tag Statistics

Current distribution (approximate):

  • permissions: 38 tests
  • infra: 18 tests
  • chat: 14 tests
  • queue: 18 tests
  • e2e: 14 tests
  • conversation: 12 tests
  • memory: 11 tests
  • health: 9 tests
  • audio-streaming: 4 tests
  • audio-upload: 3 tests
  • slow: 2 tests (backend restart tests)
  • sdk: 2 tests (SDK integration tests)
  • requires-gpu: 5 tests (ASR GPU integration tests)
  • audio-batch: 0 tests (reserved for future use)

Last Updated: 2026-01-29 Total Approved Tags: 14 Enforcement: Mandatory - no exceptions