Flakiness.io Vitest Reporter

A custom Vitest reporter that generates Flakiness Reports from your Vitest test runs. The reporter automatically converts Vitest test results into the standardized Flakiness JSON format, capturing test outcomes, system utilization, and environment information.

Table of Contents

• Requirements
• Installation
• Quick Start
• Uploading Reports
• Viewing Reports
• Features
  • Test Location Tracking
  • Attachments
  • Handling Test Duplicates
  • Environment Detection
  • CI Integration
• Configuration Options
  • title?: string
  • flakinessProject?: string
  • endpoint?: string
  • token?: string
  • outputFolder?: string
  • duplicates?: 'fail' | 'rename'
  • disableUpload?: boolean
• Environment Variables
• Example Configuration

Requirements

• Node.js 20.17.0 or higher (22.9.0 or higher on Node 22)
• Vitest 4.0 or higher
• Node.js project with a git repository (for commit information)
• Valid Flakiness.io access token (for uploads)

Installation

npm install @flakiness/vitest

Quick Start

Add the reporter to your vitest.config.ts:

import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    includeTaskLocation: true,
    reporters: [
      'default',
      ['@flakiness/vitest', { flakinessProject: 'my-org/my-project' }]
    ],
  },
});

Note

The flakiness reporter should be added alongside your other reporters. Include 'default' (or another built-in reporter) to retain the standard terminal output, since Vitest only uses the reporters listed in the reporters array.

Tip

Setting includeTaskLocation: true is recommended to enable test locations in the report.

Run your tests. The report will be automatically generated in the ./flakiness-report folder:

npx vitest run

View the interactive report:

npx flakiness show ./flakiness-report

Uploading Reports

Reports are automatically uploaded to Flakiness.io after test completion. Authentication is resolved in the following order, and the first available method wins:

• Access token: Provide a token via the token option or the FLAKINESS_ACCESS_TOKEN environment variable.

• GitHub OIDC: When running in GitHub Actions with no access token, the reporter can authenticate using GitHub's OIDC token. This requires three conditions:

  1. The flakinessProject option must be set to your Flakiness.io project identifier (org/project).
  2. The Flakiness.io project must be bound to the GitHub repository that runs the GitHub Actions workflow.
  3. The workflow must grant the id-token: write permission.

• GitLab OIDC: When running in GitLab CI/CD with no access token, the reporter can authenticate using a GitLab ID token. GitLab mints ID tokens when the job starts, so the job must declare one named FLAKINESS_ID_TOKEN whose audience is your project identifier:

  test:
    id_tokens:
      FLAKINESS_ID_TOKEN:
        aud: my-org/my-project   # must match the `flakinessProject` option
    script:
      - npx vitest run

  The flakinessProject option must be set, and the Flakiness.io project must be bound to the GitLab project running the pipeline.

If no method is available the upload is skipped, and if an upload fails the report is still available locally in the output folder. Either way the test run is unaffected.

Viewing Reports

After test execution, you can view the report using:

npx flakiness show ./flakiness-report

Features

See features.md for this reporter's status against the Flakiness Report spec.

Test Location Tracking

When includeTaskLocation: true is set in your Vitest config, the reporter records the exact file, line, and column for each test. This enables precise navigation from the Flakiness.io dashboard back to your source code.

Attachments

Browser Mode records screenshots as Vitest artifacts. The reporter writes them into the local Flakiness report and uploads them along with it. Identical content is stored only once, no matter how many tests attach it.

Two kinds of screenshot are captured:

• Failure screenshots. Vitest screenshots every failing test when browser.screenshotFailures is enabled — which is the default outside of the Vitest UI. It is reported as failure-screenshot.png, and needs no changes to your tests.
• Visual regressions. Each failed toMatchScreenshot() comparison contributes screenshot-1-expected.png, screenshot-1-actual.png and screenshot-1-diff.png, numbered per comparison within the test. Vitest does not take a failure screenshot for these, since the comparison images already show the page.

Attachments of custom artifacts recorded through Vitest's recordArtifact() are named after the artifact type (my-plugin:artifact-1). Attachments of test annotations (context.annotate()) are not collected.

Handling Test Duplicates

Vitest allows creating tests with identical names:

// Trivial duplicates
it('should work', () => { /* ... */ });
it('should work', () => { /* ... */ });

// More common way to end up with test duplicates:
it.for([
  { input: 1, expected: 2 },
  { input: 1, expected: 3 },
])('should handle $input', ({ input, expected }) => { /* ... */ });

Flakiness.io, however, does not allow test duplicates.

Flakiness.io considers two tests to be duplicates when they:

1. Share the same parent suite hierarchy
2. Have the same test name
3. And run in the same Vitest project

Flakiness.io relies on full test names to construct test history, so each test must have a unique full name. When Vitest reporter detects tests with identical full names, it issues warnings and handles them according to the duplicates option.

Environment Detection

For each Vitest project, the reporter creates a unique environment. If a project has a name, it will be used as the environment name in Flakiness.io. Tests run in different projects get separate history timelines.

Environment variables prefixed with FK_ENV_ are automatically included in the environment metadata. The prefix is stripped and the key is converted to lowercase.

Example:

export FK_ENV_DEPLOYMENT=staging
export FK_ENV_REGION=us-east-1

This will result in the environment containing:

{
  "metadata": {
    "deployment": "staging",
    "region": "us-east-1"
  }
}

Flakiness.io will create a dedicated history for tests executed in each unique environment. This means tests run with FK_ENV_DEPLOYMENT=staging will have a separate timeline from tests run with FK_ENV_DEPLOYMENT=production, allowing you to track flakiness patterns specific to each deployment environment.

CI Integration

The reporter automatically detects CI environments and includes:

• CI run URLs (GitHub Actions, Azure DevOps, Jenkins, GitLab CI)
• Git commit information
• System environment data

Configuration Options

The reporter accepts the following options:

title?: string

Optional human-readable report title. Typically used to name a CI run, matrix shard, or other execution group. Defaults to the FLAKINESS_TITLE environment variable, or empty otherwise.

reporters: [
  ['@flakiness/vitest', { title: 'Shard 1/4 — Linux Chrome' }]
]

flakinessProject?: string

The Flakiness.io project identifier in org/project format. Required for CI OIDC authentication. When set, and when the Flakiness.io project is bound to the repository running the pipeline, the reporter authenticates uploads via a GitHub Actions or GitLab CI/CD OIDC token with no access token required. See Uploading Reports for the per-provider requirements.

reporters: [
  ['@flakiness/vitest', { flakinessProject: 'my-org/my-project' }]
]

endpoint?: string

Custom Flakiness.io endpoint URL for uploading reports. Defaults to the FLAKINESS_ENDPOINT environment variable, or https://flakiness.io if not set.

Use this option to point to a custom or self-hosted Flakiness.io instance.

reporters: [
  ['@flakiness/vitest', { endpoint: 'https://custom.flakiness.io' }]
]

token?: string

Access token for authenticating with Flakiness.io when uploading reports. Defaults to the FLAKINESS_ACCESS_TOKEN environment variable.

If no token is provided, the reporter falls back to CI OIDC on GitHub Actions and GitLab CI/CD. If that is unavailable too, the report is still generated locally but won't be uploaded automatically.

reporters: [
  ['@flakiness/vitest', { token: 'your-access-token' }]
]

outputFolder?: string

Directory path where the Flakiness report will be written. Defaults to flakiness-report in the current working directory, or the FLAKINESS_OUTPUT_DIR environment variable if set.

reporters: [
  ['@flakiness/vitest', { outputFolder: './test-results/flakiness' }]
]

duplicates?: 'fail' | 'rename'

Controls how the reporter handles tests with duplicate full names. Defaults to 'fail'.

Warning

The 'rename' mode is not recommended for regular use. There is no guarantee that test histories will remain stable for duplicate tests, since the renaming is based on internal Vitest identifiers that may change between runs. This mode exists to help evaluate the reporter against large Vitest projects that have not yet resolved their duplicate test names.

• 'fail' (default): Duplicate tests are marked as failed with a descriptive error message and a dupe annotation. The first duplicate gets a single failed attempt explaining the problem; the remaining duplicates are stripped of their attempts (effectively hidden from the report). This is the recommended mode — it surfaces the problem so you can fix it by renaming your tests.
• 'rename': Duplicate tests are automatically renamed by appending a suffix (e.g., – dupe #2, – dupe #3) to their titles. Each renamed test also receives a dupe annotation on all its attempts. The first test with a given name keeps its original title; only the subsequent duplicates are renamed.

reporters: [
  ['@flakiness/vitest', { duplicates: 'rename' }]
]

disableUpload?: boolean

When set to true, prevents uploading the report to Flakiness.io. The report is still generated locally. Can also be controlled via the FLAKINESS_DISABLE_UPLOAD environment variable.

reporters: [
  ['@flakiness/vitest', { disableUpload: true }]
]

Environment Variables

The reporter respects the following environment variables:

• FLAKINESS_TITLE: Human-readable report title (equivalent to title option)
• FLAKINESS_ACCESS_TOKEN: Access token for Flakiness.io uploads (equivalent to token option)
• FLAKINESS_ENDPOINT: Custom Flakiness.io endpoint URL (equivalent to endpoint option)
• FLAKINESS_OUTPUT_DIR: Output directory for reports (equivalent to outputFolder option)
• FLAKINESS_DISABLE_UPLOAD: When set, disables report uploads (equivalent to disableUpload option)
• FLAKINESS_ID_TOKEN: GitLab CI/CD ID token used for OIDC authentication. Set by GitLab for jobs that declare it under id_tokens: (see Uploading Reports)

Example Configuration

Here's a complete example with all options:

import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    includeTaskLocation: true,
    reporters: [
      'default',
      ['@flakiness/vitest', {
        title: 'My Test Run',
        flakinessProject: 'my-org/my-project',
        endpoint: process.env.FLAKINESS_ENDPOINT,
        token: process.env.FLAKINESS_ACCESS_TOKEN,
        outputFolder: './flakiness-report',
        duplicates: 'fail',
        disableUpload: false,
      }]
    ],
  },
});