Skip to content

fix(ci): retry a quarantined test that fails in a run of several files #624

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
klNuno merged 1 commit into from
Oct 19, 2026
Merged

fix(ci): retry a quarantined test that fails in a run of several files #624

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions scripts/ci/flaky.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,19 @@ test('failed tests are named only when every counted failure has a name', () =>
expect(failedTests('(fail) a\n 1 fail\n(fail) b\n 1 fail\n')).toEqual(['a', 'b']);
});

test('a run of several files names each failure once, not again in its closing recap', () => {
// bun test over several files lists every skip and failure again before the counts.
const several = [
'(fail) relays reads [4246.43ms]', '(pass) other [1.00ms]', '',
'1 tests skipped:', '(skip) killed outright', '', '',
'1 tests failed:', '(fail) relays reads [4246.43ms]', '',
' 111 pass', ' 1 skip', ' 1 fail', 'Ran 113 tests across 39 files. [447.52s]', ''
].join('\r\n');
expect(failedTests(several)).toEqual(['relays reads']);
const two = '(fail) a [1ms]\n(fail) b [1ms]\n\n2 tests failed:\n(fail) a [1ms]\n(fail) b [1ms]\n\n 2 fail\n';
expect(failedTests(two)).toEqual(['a', 'b']);
});

test('a retry needs every failure quarantined in a file that ran', () => {
const list = [entry('tests/e2e/a.test.ts', 'one'), entry('tests/e2e/b.test.ts', 'two')];
const absolute = join(ROOT, 'tests', 'e2e', 'a.test.ts');
Expand Down
3 changes: 2 additions & 1 deletion scripts/ci/flaky.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ export const QUARANTINE: Quarantined[] = JSON.parse(readFileSync(resolve(import.

/** Names of the failed tests in bun test output, or undefined unless every counted failure has one. */
export function failedTests(output: string): string[] | undefined {
const clean = output.replace(/\x1b\[[0-9;]*m/g, '');
// Over several files, bun lists every failure again under "N tests failed:" before the counts.
const clean = output.replace(/\x1b\[[0-9;]*m/g, '').replace(/^\d+ tests? failed:\r?\n(?:\(fail\) .*\r?\n)*/gm, '');
const names = [...clean.matchAll(/^\(fail\) (.+?)(?: \[[\d.]+m?s\])?\r?$/gm)].map(match => match[1]!);
const count = [...clean.matchAll(/^\s*(\d+) fail\r?$/gm)].reduce((sum, match) => sum + Number(match[1]), 0);
return names.length > 0 && names.length === count ? names : undefined;
Expand Down
Loading