From c28a2e34b61c3559ba5aafea0ab201378bfad513 Mon Sep 17 00:00:00 2001 From: Pushpak-Jaiswal Date: Wed, 14 Oct 2026 15:01:25 +0530 Subject: [PATCH 1/2] feat(ci): list failed tests with trace links in the GitHub Actions job summary (#137) --- CHANGELOG.md | 2 + docs-site/docs/changelog.md | 2 + docs-site/docs/ci/github-actions.md | 13 + docs-site/docs/guides/trace-viewer.md | 62 ++++ docs-site/sidebars.js | 1 + .../ci/CiEnvironmentDetector.java | 7 + .../seleniumboot/ci/GitHubActionsSummary.java | 210 ++++++++++++ .../junit5/SeleniumBootLauncherListener.java | 5 +- .../listeners/SuiteExecutionListener.java | 6 +- .../unit/CiEnvironmentDetectorTest.java | 10 + .../unit/GitHubActionsSummaryTest.java | 301 ++++++++++++++++++ 11 files changed, 617 insertions(+), 2 deletions(-) create mode 100644 docs-site/docs/guides/trace-viewer.md create mode 100644 src/main/java/com/seleniumboot/ci/GitHubActionsSummary.java create mode 100644 src/test/java/com/seleniumboot/unit/GitHubActionsSummaryTest.java diff --git a/CHANGELOG.md b/CHANGELOG.md index 915392e..046324a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Unreleased +- **Failed tests with trace links in GitHub Actions job summary** — when running inside GitHub Actions and `tracing.enabled: true` is on, failed tests are automatically summarized in a Markdown table (test name, error message, and path to the HTML trace file) appended to `$GITHUB_STEP_SUMMARY`. Triage failures and locate trace paths directly from the Actions job summary without downloading artifact bundles; no-op outside Actions or when all tests pass. (#137) + - **Bounded failure detail record** — every failed test now exports one optional-field `failureDetail` object with the failing locator, top application frame, last successful step, page URL/title, heal events, console errors, retry/history status, and representative same-cause context. The HTML failure panel and opt-in AI prompt reuse this record; raw errors and stack traces remain available. Text/list/URL limits and the exact AI data disclosure are documented. (#142) - **`Locator`: `waitFor(State)`, `allTextContents()`, and element `screenshot()`** — `waitFor(State)` (and `waitFor()`) waits for an element to reach a target lifecycle state (`VISIBLE`, `HIDDEN`, `ATTACHED`, `DETACHED`), retrying up to `timeouts.explicit`. `allTextContents()` returns the text content of all matched elements in a single pass without waiting if empty, re-resolving if an element goes stale during read. `screenshot()` returns the element's PNG bytes, and `screenshot(Path)` writes them to a file path; both terminal actions re-resolve on every poll to survive stale elements. (#135) diff --git a/docs-site/docs/changelog.md b/docs-site/docs/changelog.md index 6f108e9..9ef5138 100644 --- a/docs-site/docs/changelog.md +++ b/docs-site/docs/changelog.md @@ -11,6 +11,8 @@ All notable changes to Selenium Boot are documented here. ## [Unreleased] +- **Failed tests with trace links in GitHub Actions job summary** — when running inside GitHub Actions and `tracing.enabled: true` is on, failed tests are automatically summarized in a Markdown table (test name, error message, and path to the HTML trace file) appended to `$GITHUB_STEP_SUMMARY`. Triage failures and locate trace paths directly from the Actions job summary without downloading artifact bundles; no-op outside Actions or when all tests pass. (#137) + - **Bounded failure detail record** — every failed test now exports one optional-field `failureDetail` object with the failing locator, top application frame, last successful step, page URL/title, heal events, console errors, retry/history status, and representative same-cause context. The HTML failure panel and opt-in AI prompt reuse this record; raw errors and stack traces remain available. Text/list/URL limits and the exact AI data disclosure are documented. (#142) - **`Locator`: `waitFor(State)`, `allTextContents()`, and element `screenshot()`** — `waitFor(State)` (and `waitFor()`) waits for an element to reach a target lifecycle state (`VISIBLE`, `HIDDEN`, `ATTACHED`, `DETACHED`), retrying up to `timeouts.explicit`. `allTextContents()` returns the text content of all matched elements in a single pass without waiting if empty, re-resolving if an element goes stale during read. `screenshot()` returns the element's PNG bytes, and `screenshot(Path)` writes them to a file path; both terminal actions re-resolve on every poll to survive stale elements. (#135) diff --git a/docs-site/docs/ci/github-actions.md b/docs-site/docs/ci/github-actions.md index d1fe786..020dc5f 100644 --- a/docs-site/docs/ci/github-actions.md +++ b/docs-site/docs/ci/github-actions.md @@ -59,11 +59,24 @@ When it detects a CI environment (GitHub Actions, Jenkins, CircleCI and others), - `execution.threadCount` is derived from the available CPU cores, capped at `execution.maxActiveSessions` — only when you left it at the default of `1` and `parallel` is not `none`. - Inside Docker or Kubernetes, Chrome and Firefox get the container flags they need (for Chrome: `--no-sandbox`, `--disable-dev-shm-usage`, `--disable-gpu`, `--window-size=1920,1080`). - JUnit XML is written to `target/surefire-reports/TEST-SeleniumBoot.xml` on every run. +- When running in GitHub Actions and `tracing.enabled: true` is on, failed tests are summarized in a Markdown table (test name, error, and path to `target/traces/...`) appended to `$GITHUB_STEP_SUMMARY` so failures are visible without opening artifacts. Each override is printed to the console as `[Selenium Boot] CI override: …`. --- +## Job summary with trace links + +When `tracing.enabled: true` is enabled in `selenium-boot.yml` and tests fail, Selenium Boot automatically appends a Markdown table to `$GITHUB_STEP_SUMMARY`: + +| Test | Error | Trace | +| --- | --- | --- | +| `com.example.LoginTest#testInvalidLogin` | Expected [Dashboard] but found [401 Unauthorized] | target/traces/LoginTest/testInvalidLogin-trace.html | + +This lists every failing test, its error message, and the path to its trace file directly in the GitHub Actions run summary page. If no tests fail or outside GitHub Actions, nothing is written. + +--- + ## Publish JUnit XML test results ```yaml diff --git a/docs-site/docs/guides/trace-viewer.md b/docs-site/docs/guides/trace-viewer.md new file mode 100644 index 0000000..fcf5207 --- /dev/null +++ b/docs-site/docs/guides/trace-viewer.md @@ -0,0 +1,62 @@ +--- +description: "Self-contained HTML trace viewer per test with clickable step timeline, per-step screenshots, failure snapshot, error details, and GitHub Actions summary integration." +id: trace-viewer +title: Trace Viewer +sidebar_label: Trace Viewer +sidebar_position: 9 +--- + +# Trace Viewer + +Selenium Boot generates a standalone, self-contained HTML trace viewer per test with an interactive step timeline, per-step screenshots, final-state failure snapshot, and full error details — with zero external CDN dependencies. + +--- + +## Enabling Trace Recording + +Trace recording is opt-in and configured in `selenium-boot.yml`: + +```yaml title="selenium-boot.yml" +tracing: + enabled: true + captureOnPass: false # set true to generate traces for passing tests too +``` + +When enabled, a self-contained HTML file is generated for each failed test at: + +``` +target/traces/{ClassName}/{testMethod}-trace.html +``` + +(Or `build/traces/...` in Gradle projects, honoring `-Dseleniumboot.reports.dir`). + +--- + +## What the Trace Viewer captures + +Each trace file is an independent, single-file HTML document featuring: + +1. **Metadata header** — test method name, class, browser name, duration in milliseconds, status badge, and timestamp. +2. **Interactive step timeline** — every named step recorded via `StepLogger.step(...)` appears in the timeline with its offset. Steps that captured screenshots display a camera icon (📷) and open the interactive screenshot panel on click. +3. **Final-state screenshot** — full-browser screenshot captured automatically at the exact moment of test failure. +4. **Failure details** — highlighted error message and full scrollable stack trace. +5. **Zero CDN dependencies** — CSS styles and interactive JavaScript are completely inlined so traces can be viewed offline, archived, or shared without broken assets. + +--- + +## Viewing traces + +### From the HTML report +In the main Selenium Boot HTML report (`target/selenium-boot-report.html`), every failed test in the Failures tab features a **View Trace** link in its failure detail panel that opens the corresponding trace file. + +### Directly from the file system +Open `target/traces/{ClassName}/{testMethod}-trace.html` in any modern web browser. + +### In GitHub Actions +When running in GitHub Actions and `tracing.enabled: true` is on, failed tests are automatically summarized in a Markdown table appended to `$GITHUB_STEP_SUMMARY`: + +| Test | Error | Trace | +| --- | --- | --- | +| `com.example.CartTest.testCheckout` | `Timeout waiting for element` | `target/traces/CartTest/testCheckout-trace.html` | + +This lets teams triage failures and locate the corresponding HTML trace file directly on the Actions job summary page without downloading and extracting artifact archives. diff --git a/docs-site/sidebars.js b/docs-site/sidebars.js index d50328e..bc790c3 100644 --- a/docs-site/sidebars.js +++ b/docs-site/sidebars.js @@ -34,6 +34,7 @@ const sidebars = { 'guides/retry', 'guides/screenshots', 'guides/step-logging', + 'guides/trace-viewer', 'guides/browser-lifecycle', 'guides/parallel', 'guides/smart-locator', diff --git a/src/main/java/com/seleniumboot/ci/CiEnvironmentDetector.java b/src/main/java/com/seleniumboot/ci/CiEnvironmentDetector.java index 9060348..b67c5ed 100644 --- a/src/main/java/com/seleniumboot/ci/CiEnvironmentDetector.java +++ b/src/main/java/com/seleniumboot/ci/CiEnvironmentDetector.java @@ -31,6 +31,13 @@ public static boolean isCI() { || isEnvSet("BITBUCKET_BUILD_NUMBER"); } + /** + * Returns true if running inside GitHub Actions. + */ + public static boolean isGitHubActions() { + return isEnvSet("GITHUB_ACTIONS"); + } + /** * Returns a human-readable name for the detected CI provider. */ diff --git a/src/main/java/com/seleniumboot/ci/GitHubActionsSummary.java b/src/main/java/com/seleniumboot/ci/GitHubActionsSummary.java new file mode 100644 index 0000000..3934cb7 --- /dev/null +++ b/src/main/java/com/seleniumboot/ci/GitHubActionsSummary.java @@ -0,0 +1,210 @@ +package com.seleniumboot.ci; + +import com.seleniumboot.config.SeleniumBootConfig; +import com.seleniumboot.internal.SeleniumBootContext; +import com.seleniumboot.metrics.TestTiming; +import com.seleniumboot.reporting.ReportPaths; + +import java.io.File; +import java.io.FileWriter; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.Collection; +import java.util.List; + +/** + * Appends a Markdown summary table of failed tests with their error message and trace links + * to GitHub Actions' {@code $GITHUB_STEP_SUMMARY}. + * + *

Activated when running in GitHub Actions, {@code tracing.enabled: true} is configured, + * and at least one test failed. Outside GitHub Actions or when no tests failed, this is a no-op. + */ +public final class GitHubActionsSummary { + + private GitHubActionsSummary() {} + + /** + * Builds the Markdown table string for any failed tests in {@code timings}. + * Returns an empty string if {@code timings} is null or contains no failed tests. + */ + public static String buildMarkdown(Collection timings) { + if (timings == null || timings.isEmpty()) { + return ""; + } + + List failed = timings.stream() + .filter(t -> "FAILED".equalsIgnoreCase(t.getStatus())) + .toList(); + + if (failed.isEmpty()) { + return ""; + } + + String baseDir = ReportPaths.baseDir(); + + StringBuilder sb = new StringBuilder(); + sb.append("### Failed Tests\n\n"); + sb.append("| Test | Error | Trace |\n"); + sb.append("| --- | --- | --- |\n"); + + for (TestTiming t : failed) { + String testId = escapeTableCell(t.getTestId() != null ? t.getTestId() : "Unknown"); + String error = escapeTableCell(t.getErrorMessage() != null ? t.getErrorMessage() : "—"); + String tracePath = resolveTracePath(t, baseDir); + + sb.append("| ").append(testId) + .append(" | ").append(error) + .append(" | ").append(tracePath) + .append(" |\n"); + } + + return sb.toString(); + } + + /** + * Appends failed tests with trace links to {@code summaryFile} when tracing is enabled + * and at least one test failed. + * + * @return {@code true} if summary content was appended; {@code false} otherwise. + */ + public static boolean appendSummary(SeleniumBootConfig config, + Collection timings, + File summaryFile) { + if (config == null) { + return false; + } + SeleniumBootConfig.Tracing tracing = config.getTracing(); + if (tracing == null || !tracing.isEnabled()) { + return false; + } + if (timings == null || timings.isEmpty() || summaryFile == null) { + return false; + } + + boolean hasFailures = timings.stream() + .anyMatch(t -> "FAILED".equalsIgnoreCase(t.getStatus())); + if (!hasFailures) { + return false; + } + + String markdown = buildMarkdown(timings); + if (markdown.isEmpty()) { + return false; + } + + try { + File parent = summaryFile.getParentFile(); + if (parent != null && !parent.exists()) { + parent.mkdirs(); + } + + boolean prependNewline = summaryFile.exists() && summaryFile.length() > 0; + String content = (prependNewline ? "\n" : "") + markdown; + + try (FileWriter writer = new FileWriter(summaryFile, StandardCharsets.UTF_8, true)) { + writer.write(content); + } + + System.out.println("[Selenium Boot] Appended failed tests to GitHub Actions summary → " + + summaryFile.getPath()); + return true; + } catch (IOException e) { + System.err.println("[Selenium Boot] Failed to write to $GITHUB_STEP_SUMMARY: " + + e.getMessage()); + return false; + } + } + + /** + * Exports failed tests to {@code $GITHUB_STEP_SUMMARY} if running in GitHub Actions. + * + * @return {@code true} if written; {@code false} if skipped or failed. + */ + public static boolean export(SeleniumBootConfig config, Collection timings) { + String testOverride = System.getProperty("seleniumboot.github.step.summary"); + boolean isActions = CiEnvironmentDetector.isGitHubActions() + || (testOverride != null && !testOverride.isBlank()); + if (!isActions) { + return false; + } + + String summaryPath = (testOverride != null && !testOverride.isBlank()) + ? testOverride.trim() + : System.getenv("GITHUB_STEP_SUMMARY"); + if (summaryPath == null || summaryPath.isBlank()) { + return false; + } + + return appendSummary(config, timings, new File(summaryPath)); + } + + /** + * Overload using the globally configured {@link SeleniumBootConfig}. + */ + public static boolean export(Collection timings) { + SeleniumBootConfig config = null; + try { + config = SeleniumBootContext.getConfig(); + } catch (Exception ignored) {} + return export(config, timings); + } + + static String resolveTracePath(TestTiming timing, String baseDir) { + String base = (baseDir != null && !baseDir.isBlank()) + ? baseDir.replace('\\', '/').trim() + : "target"; + if (base.endsWith("/")) { + base = base.substring(0, base.length() - 1); + } + + String path = timing.getTracePath(); + if (path != null && !path.isBlank()) { + String normalized = path.replace('\\', '/').trim(); + if (normalized.startsWith("/")) { + normalized = normalized.substring(1); + } + if (normalized.startsWith(base + "/")) { + return normalized; + } + return base + "/" + normalized; + } + + String className = timing.getTestClassName() != null && !timing.getTestClassName().isBlank() + ? timing.getTestClassName() + : "UnknownClass"; + String testId = timing.getTestId(); + String methodName = extractMethodName(testId); + return base + "/traces/" + className + "/" + sanitize(methodName) + "-trace.html"; + } + + static String extractMethodName(String testId) { + if (testId == null || testId.isBlank()) { + return "unknown"; + } + int hashIdx = testId.indexOf('#'); + if (hashIdx >= 0 && hashIdx < testId.length() - 1) { + return testId.substring(hashIdx + 1); + } + int dotIdx = testId.lastIndexOf('.'); + if (dotIdx >= 0 && dotIdx < testId.length() - 1) { + return testId.substring(dotIdx + 1); + } + return testId; + } + + static String escapeTableCell(String value) { + if (value == null || value.isBlank()) { + return "—"; + } + return value.replace("\r\n", " ") + .replace('\n', ' ') + .replace('\r', ' ') + .replace("|", "\\|") + .trim(); + } + + private static String sanitize(String s) { + return s == null ? "unknown" : s.replaceAll("[^a-zA-Z0-9_\\-]", "_"); + } +} + diff --git a/src/main/java/com/seleniumboot/junit5/SeleniumBootLauncherListener.java b/src/main/java/com/seleniumboot/junit5/SeleniumBootLauncherListener.java index f146bc7..ad129fd 100644 --- a/src/main/java/com/seleniumboot/junit5/SeleniumBootLauncherListener.java +++ b/src/main/java/com/seleniumboot/junit5/SeleniumBootLauncherListener.java @@ -2,6 +2,7 @@ import com.seleniumboot.api.SeleniumBootApi; import com.seleniumboot.ci.BuildThresholdEnforcer; +import com.seleniumboot.ci.GitHubActionsSummary; import com.seleniumboot.config.SeleniumBootConfig; import com.seleniumboot.flakiness.FlakinessAnalyzer; import com.seleniumboot.hooks.HookRegistry; @@ -37,12 +38,14 @@ public void testPlanExecutionFinished(TestPlan testPlan) { JUnitXmlReporter.export(ExecutionMetrics.getTimings(), System.currentTimeMillis()); + SeleniumBootConfig config = SeleniumBootContext.getConfig(); + GitHubActionsSummary.export(config, ExecutionMetrics.getTimings()); + ReportAdapterRegistry.generateAll(); PreConditionRunner.clearAll(); HookRegistry.onSuiteEnd(); TestManagementReporter.getInstance().onSuiteEnd(); - SeleniumBootConfig config = SeleniumBootContext.getConfig(); if (config != null) { BuildThresholdEnforcer.enforce(config, ExecutionMetrics.getTimings()); } diff --git a/src/main/java/com/seleniumboot/listeners/SuiteExecutionListener.java b/src/main/java/com/seleniumboot/listeners/SuiteExecutionListener.java index ac7fd92..074b226 100644 --- a/src/main/java/com/seleniumboot/listeners/SuiteExecutionListener.java +++ b/src/main/java/com/seleniumboot/listeners/SuiteExecutionListener.java @@ -1,6 +1,7 @@ package com.seleniumboot.listeners; import com.seleniumboot.ci.BuildThresholdEnforcer; +import com.seleniumboot.ci.GitHubActionsSummary; import com.seleniumboot.flakiness.FlakinessAnalyzer; import com.seleniumboot.healing.HealLog; import com.seleniumboot.config.SeleniumBootConfig; @@ -99,6 +100,10 @@ public void onFinish(ISuite suite) { suite.getAllInvokedMethods().size() > 0 ? System.currentTimeMillis() : 0L); + // GitHub Actions job summary (appends failed tests with trace links) + SeleniumBootConfig config = SeleniumBootContext.getConfig(); + GitHubActionsSummary.export(config, ExecutionMetrics.getTimings()); + ReportAdapterRegistry.generateAll(); PreConditionRunner.clearAll(); DriverManager.quitAllSuiteDrivers(); // per-suite lifecycle — quits all kept-alive drivers @@ -108,7 +113,6 @@ public void onFinish(ISuite suite) { PluginRegistry.unloadAll(); // Build quality gates — must run last so all metrics are recorded - SeleniumBootConfig config = SeleniumBootContext.getConfig(); BuildThresholdEnforcer.enforce(config, ExecutionMetrics.getTimings()); } } diff --git a/src/test/java/com/seleniumboot/unit/CiEnvironmentDetectorTest.java b/src/test/java/com/seleniumboot/unit/CiEnvironmentDetectorTest.java index 5fa9183..e815aa8 100644 --- a/src/test/java/com/seleniumboot/unit/CiEnvironmentDetectorTest.java +++ b/src/test/java/com/seleniumboot/unit/CiEnvironmentDetectorTest.java @@ -81,4 +81,14 @@ public void isCI_returnsBooleanWithoutThrowing() { boolean result = CiEnvironmentDetector.isCI(); assertTrue(result || !result); } + + // ---------------------------------------------------------- + // isGitHubActions() — returns boolean without throwing + // ---------------------------------------------------------- + + @Test + public void isGitHubActions_returnsBooleanWithoutThrowing() { + boolean result = CiEnvironmentDetector.isGitHubActions(); + assertTrue(result || !result); + } } diff --git a/src/test/java/com/seleniumboot/unit/GitHubActionsSummaryTest.java b/src/test/java/com/seleniumboot/unit/GitHubActionsSummaryTest.java new file mode 100644 index 0000000..47b998b --- /dev/null +++ b/src/test/java/com/seleniumboot/unit/GitHubActionsSummaryTest.java @@ -0,0 +1,301 @@ +package com.seleniumboot.unit; + +import com.seleniumboot.ci.GitHubActionsSummary; +import com.seleniumboot.config.SeleniumBootConfig; +import com.seleniumboot.metrics.TestTiming; +import org.testng.annotations.AfterMethod; +import org.testng.annotations.BeforeMethod; +import org.testng.annotations.Test; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.Collections; +import java.util.List; + +import static org.testng.Assert.*; + +/** + * Unit tests for {@link GitHubActionsSummary}. + */ +public class GitHubActionsSummaryTest { + + private File tempSummaryFile; + + @BeforeMethod + public void setUp() throws IOException { + tempSummaryFile = File.createTempFile("github-step-summary-test", ".md"); + tempSummaryFile.deleteOnExit(); + } + + @AfterMethod + public void tearDown() { + if (tempSummaryFile != null && tempSummaryFile.exists()) { + tempSummaryFile.delete(); + } + } + + private TestTiming timing(String testId, String className, String status, String error, String tracePath) { + TestTiming t = new TestTiming(testId, "main"); + t.setTestClassName(className); + t.setStatus(status); + t.setErrorMessage(error); + if (tracePath != null) { + t.setTracePath(tracePath); + } + return t; + } + + private SeleniumBootConfig configWithTracing(boolean enabled) { + SeleniumBootConfig config = new SeleniumBootConfig(); + SeleniumBootConfig.Tracing tracing = new SeleniumBootConfig.Tracing(); + tracing.setEnabled(enabled); + config.setTracing(tracing); + return config; + } + + // ========================================================== + // Markdown Generation + // ========================================================== + + @Test + public void buildMarkdown_nullOrEmpty_returnsEmptyString() { + assertEquals("", GitHubActionsSummary.buildMarkdown(null)); + assertEquals("", GitHubActionsSummary.buildMarkdown(Collections.emptyList())); + } + + @Test + public void buildMarkdown_onlyPassingOrSkipped_returnsEmptyString() { + List timings = List.of( + timing("t1", "LoginTest", "PASSED", null, null), + timing("t2", "CheckoutTest", "SKIPPED", null, null) + ); + assertEquals("", GitHubActionsSummary.buildMarkdown(timings)); + } + + @Test + public void buildMarkdown_singleFailedTest_generatesTable() { + TestTiming failed = timing( + "com.example.LoginTest#testInvalidCredentials", + "LoginTest", + "FAILED", + "Expected [Login] but found [404 Not Found]", + "traces/LoginTest/testInvalidCredentials-trace.html" + ); + + String markdown = GitHubActionsSummary.buildMarkdown(List.of(failed)); + + assertTrue(markdown.contains("### Failed Tests"), "Must contain Failed Tests heading"); + assertTrue(markdown.contains("| Test | Error | Trace |"), "Must contain table header row"); + assertTrue(markdown.contains("| --- | --- | --- |"), "Must contain markdown separator row"); + assertTrue(markdown.contains("com.example.LoginTest#testInvalidCredentials")); + assertTrue(markdown.contains("Expected [Login] but found [404 Not Found]")); + assertTrue(markdown.contains("target/traces/LoginTest/testInvalidCredentials-trace.html") + || markdown.contains("build/traces/LoginTest/testInvalidCredentials-trace.html"), + "Trace path must include base dir and relative trace file path"); + } + + @Test + public void buildMarkdown_multipleFailedTests_generatesMultipleRows() { + List timings = List.of( + timing("t1", "TestA", "PASSED", null, null), + timing("t2", "TestB", "FAILED", "Error B", "traces/TestB/t2-trace.html"), + timing("t3", "TestC", "FAILED", "Error C", "traces/TestC/t3-trace.html") + ); + + String markdown = GitHubActionsSummary.buildMarkdown(timings); + + assertTrue(markdown.contains("t2")); + assertTrue(markdown.contains("Error B")); + assertTrue(markdown.contains("t3")); + assertTrue(markdown.contains("Error C")); + assertFalse(markdown.contains("t1")); // passed test not included + } + + @Test + public void buildMarkdown_escapesPipeAndNewlinesInError() { + TestTiming failed = timing( + "com.example.PipeTest#testPipe", + "PipeTest", + "FAILED", + "Value a | b failed\r\nnext line", + "traces/PipeTest/testPipe-trace.html" + ); + + String markdown = GitHubActionsSummary.buildMarkdown(List.of(failed)); + + // The error line in markdown table row must escape '|' as '\|' + assertTrue(markdown.contains("Value a \\| b failed next line")); + // Check that table row doesn't break into multiple unescaped lines + String[] lines = markdown.split("\n"); + long rowLines = java.util.Arrays.stream(lines).filter(l -> l.startsWith("| com.example.PipeTest")).count(); + assertEquals(1, rowLines, "Table row must be a single line"); + } + + @Test + public void buildMarkdown_nullErrorMessage_usesFallbackDash() { + TestTiming failed = timing( + "com.example.NullErr#testNull", + "NullErr", + "FAILED", + null, + "traces/NullErr/testNull-trace.html" + ); + + String markdown = GitHubActionsSummary.buildMarkdown(List.of(failed)); + + assertTrue(markdown.contains("| — |")); + } + + // ========================================================== + // Trace Path Resolution + // ========================================================== + + @Test + public void resolveTracePath_handlesRelativeAndPrefixedPaths() { + TestTiming withRelative = timing("com.example.ClassA.methodA", "ClassA", "FAILED", "err", + "traces/ClassA/methodA-trace.html"); + String resolvedRel = GitHubActionsSummary.buildMarkdown(List.of(withRelative)); + assertTrue(resolvedRel.contains("target/traces/ClassA/methodA-trace.html") + || resolvedRel.contains("build/traces/ClassA/methodA-trace.html")); + + TestTiming alreadyPrefixed = timing("com.example.ClassB.methodB", "ClassB", "FAILED", "err", + "target/traces/ClassB/methodB-trace.html"); + String resolvedPrefixed = GitHubActionsSummary.buildMarkdown(List.of(alreadyPrefixed)); + assertTrue(resolvedPrefixed.contains("target/traces/ClassB/methodB-trace.html")); + assertFalse(resolvedPrefixed.contains("target/target/")); + } + + @Test + public void resolveTracePath_nullTracePath_generatesDefaultTracePath() { + TestTiming withoutTrace = timing("com.example.FallbackTest#testFallback", "FallbackTest", "FAILED", "err", null); + String markdown = GitHubActionsSummary.buildMarkdown(List.of(withoutTrace)); + assertTrue(markdown.contains("traces/FallbackTest/testFallback-trace.html")); + } + + // ========================================================== + // appendSummary + // ========================================================== + + @Test + public void appendSummary_writesMarkdownToFile() throws IOException { + SeleniumBootConfig config = configWithTracing(true); + List timings = List.of( + timing("t1", "ClassOne", "FAILED", "Something exploded", "traces/ClassOne/t1-trace.html") + ); + + boolean result = GitHubActionsSummary.appendSummary(config, timings, tempSummaryFile); + + assertTrue(result, "appendSummary should return true on success"); + String content = Files.readString(tempSummaryFile.toPath()); + assertTrue(content.contains("### Failed Tests")); + assertTrue(content.contains("Something exploded")); + } + + @Test + public void appendSummary_appendsToExistingFileContent() throws IOException { + Files.writeString(tempSummaryFile.toPath(), "## Existing Job Summary Content\n"); + + SeleniumBootConfig config = configWithTracing(true); + List timings = List.of( + timing("t1", "ClassOne", "FAILED", "Fail 1", "traces/ClassOne/t1-trace.html") + ); + + boolean result = GitHubActionsSummary.appendSummary(config, timings, tempSummaryFile); + + assertTrue(result); + String content = Files.readString(tempSummaryFile.toPath()); + assertTrue(content.startsWith("## Existing Job Summary Content\n")); + assertTrue(content.contains("### Failed Tests")); + } + + @Test + public void appendSummary_tracingDisabled_noOp() { + SeleniumBootConfig config = configWithTracing(false); + List timings = List.of( + timing("t1", "ClassOne", "FAILED", "Fail 1", "traces/ClassOne/t1-trace.html") + ); + + boolean result = GitHubActionsSummary.appendSummary(config, timings, tempSummaryFile); + + assertFalse(result, "appendSummary must no-op when tracing.enabled is false"); + assertEquals(0, tempSummaryFile.length(), "File must not be written to"); + } + + @Test + public void appendSummary_noFailedTests_noOp() { + SeleniumBootConfig config = configWithTracing(true); + List timings = List.of( + timing("t1", "ClassOne", "PASSED", null, null) + ); + + boolean result = GitHubActionsSummary.appendSummary(config, timings, tempSummaryFile); + + assertFalse(result, "appendSummary must no-op when nothing failed"); + assertEquals(0, tempSummaryFile.length(), "File must not be written to"); + } + + @Test + public void appendSummary_createsParentDirectoryIfMissing() { + File nested = new File(tempSummaryFile.getParentFile(), "nested-dir/summary.md"); + nested.deleteOnExit(); + if (nested.exists()) nested.delete(); + if (nested.getParentFile().exists()) nested.getParentFile().delete(); + + SeleniumBootConfig config = configWithTracing(true); + List timings = List.of( + timing("t1", "ClassOne", "FAILED", "Fail 1", "traces/ClassOne/t1-trace.html") + ); + + boolean result = GitHubActionsSummary.appendSummary(config, timings, nested); + + assertTrue(result); + assertTrue(nested.exists()); + nested.delete(); + nested.getParentFile().delete(); + } + + // ========================================================== + // export (Environment detection) + // ========================================================== + + @Test + public void export_outsideGitHubActions_noOp() { + // Unless GITHUB_ACTIONS=true is present in the OS environment, export must return false + if (System.getenv("GITHUB_ACTIONS") == null && System.getProperty("seleniumboot.github.step.summary") == null) { + SeleniumBootConfig config = configWithTracing(true); + List timings = List.of( + timing("t1", "ClassOne", "FAILED", "Fail 1", "traces/ClassOne/t1-trace.html") + ); + boolean result = GitHubActionsSummary.export(config, timings); + assertFalse(result, "export must no-op outside GitHub Actions"); + } + } + + @Test + public void export_withSystemPropertyOverride_writesToFile() throws IOException { + String propKey = "seleniumboot.github.step.summary"; + String previous = System.getProperty(propKey); + try { + System.setProperty(propKey, tempSummaryFile.getAbsolutePath()); + SeleniumBootConfig config = configWithTracing(true); + List timings = List.of( + timing("t1", "ClassOne", "FAILED", "Fail in Actions", "traces/ClassOne/t1-trace.html") + ); + + boolean result = GitHubActionsSummary.export(config, timings); + + assertTrue(result, "export should succeed when summary path property is provided"); + String content = Files.readString(tempSummaryFile.toPath()); + assertTrue(content.contains("### Failed Tests")); + assertTrue(content.contains("Fail in Actions")); + } finally { + if (previous != null) { + System.setProperty(propKey, previous); + } else { + System.clearProperty(propKey); + } + } + } +} + From ab10b61ad5448ecb9f8af7358e4e966121667b8a Mon Sep 17 00:00:00 2001 From: Pushpak-Jaiswal Date: Wed, 14 Oct 2026 15:03:45 +0530 Subject: [PATCH 2/2] feat: ci list failed tests --- docs-site/docs/guides/trace-viewer.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs-site/docs/guides/trace-viewer.md b/docs-site/docs/guides/trace-viewer.md index fcf5207..cdd76e8 100644 --- a/docs-site/docs/guides/trace-viewer.md +++ b/docs-site/docs/guides/trace-viewer.md @@ -60,3 +60,4 @@ When running in GitHub Actions and `tracing.enabled: true` is on, failed tests a | `com.example.CartTest.testCheckout` | `Timeout waiting for element` | `target/traces/CartTest/testCheckout-trace.html` | This lets teams triage failures and locate the corresponding HTML trace file directly on the Actions job summary page without downloading and extracting artifact archives. +