Add support for custom checks for deterministic scoring

[joshblack]

We currently offer deterministic checks through scenario.test.ts but this is fairly limited for things that folks might want to do. As a result, we want to add support for custom checks that can be run for a scenario. This could include eslint checks, custom scoring for files, running checks in playwright, etc. In the future, the test API could migrate to these checks, as well, instead of living as its own file.

API

import {defineConfig} from '@primer/agent-eval/scenario';

export default defineConfig({
  description: '...',
  prompt: '...',
  checks: [
    {
      name: 'eslint',
      description: 'Scores files against eslint rules seeing how many files have violations',
      // Files are ignored when the scenario is copied over and then later brought over during the scoring phase
      // Files should also support glob patterns
      files: ['eslint.config.standbox.ts'],
      async run({ sandbox }) {
        // Set up eslint and config
        // Run eslint, either parse output into results or write to json file
        // total would be files.
        
        // Results are used to determine total number of cases and success rate (as a percentage)
        return {
          results: [
            {
              status: 'success',
            },
            // ...
          ],
        }
      },
    },
    {
      name: 'tests',
      description: 'Runs vitest node or browser-based tests',
      files: ['vitest.config.ts'],
      async run ({ sandbox }) {
        // Set up vitest
        // Run
        // Parse test results
        // Return details
      },
    },
    {
      name: 'diff',
      description: 'Compares the page against a target screenshot',
      files: ['screenshots'],
      async run({ sandbox }) {
        // Setup playwright
        // Copy over screenshots for comparison
        // Compare

        return {
          type: 'value',
          value: 0.01,
          min: 0,
          max: 1,
        }
      },
    },
  ],
});

It may be helpful for us to offer tooling to help make some of this easier, in particular for screenshots where it would be helpful to be able to generate an ideal easily that could then be compared against.