Skip to content

test: add golden outputs for docs.bzl scenarios - #804

Merged
AlexanderLanin merged 6 commits into
eclipse-score:mainfrom
etas-contrib:refactor/docs-bzl-golden-tests
Sep 8, 2026
Merged

test: add golden outputs for docs.bzl scenarios#804
AlexanderLanin merged 6 commits into
eclipse-score:mainfrom
etas-contrib:refactor/docs-bzl-golden-tests

Conversation

@AlexanderLanin

Copy link
Copy Markdown
Member

Why

The public docs.bzl scenario tests kept many expected outputs encoded as Python assertions. That made the scenarios incomplete and difficult to understand or update.

What

  • Add direct _expected/ golden files and directories for the positive docs.bzl scenarios.
  • Centralize the mapping from short expected-output names to Bazel targets and output roots.
  • Compare complete JSON documents in readable canonical form and preserve byte-exact HTML fixtures.
  • Add a scoped updater for refreshing existing golden files.
  • Keep focused semantic tests for runtime behavior and failure cases.
  • Split CI execution into cacheable and slow test groups, with the cacheable group first.

@MaximilianSoerenPollak MaximilianSoerenPollak left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have to understand it a bit more, some questions etc.
But overall it makes sense to me in the most parts.

Once they are answered and I understand it a bit better I think we could look towards merging it.

Comment thread pyproject.toml

markers = [
"bazel_cached: successful build-only test expected to reuse Bazel actions",
"bazel_slow: runtime execution or an uncached expected Bazel failure",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not quiet understand this marker?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the idea was to run fast tests first and then the slow tests. however "fast" is not quite true. They are only fast because mostly they are cached. So I ended up with "cached" and "slow". Those are horrible categories, but so far I dont have a better approach.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But they are only cached on second run right?

@AlexanderLanin

Copy link
Copy Markdown
Member Author

That last one is addressed in useblocks/sphinx-needs#1914

@AlexanderLanin

Copy link
Copy Markdown
Member Author

And that one in useblocks/sphinx-mounts#47

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Documentation preview for this pull request is available at:
pr-804: https://eclipse-score.github.io/docs-as-code/pr-804/


# TEMP UNTIL UPSTREAM FIX - BEGIN
# Bug ref: https://github.com/useblocks/sphinx-mounts/issues/47
def configure_mounted_source_controls(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be in this PR?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need that to generate identical html every time, as currently /home/<username>/.cache/... is added to the urls

Comment on lines -479 to -495
<dialog id="pst-secondary-sidebar-modal"></dialog>
<div id="pst-secondary-sidebar" class="bd-sidebar-secondary bd-toc"><div class="sidebar-secondary-items sidebar-secondary__inner">


<div class="sidebar-secondary-item">


<div class="tocsection editthispage">
<a href="https://github.com/eclipse-score/docs-as-code/edit/main/src/tests/docs_bzl/scenarios/data_files_runfiles/docs/legacy_data_test/indexl/_bazel_alex/ec58ecca617edd0b2864120a40405cd7/execroot/_main/bazel-out/k8-fastbuild/bin/src/tests/docs_bzl/scenarios/data_files_runfiles/legacy_generated/index.rst">
<i class="fa-solid fa-pencil"></i>



Edit on GitHub


</a>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deleted on purpose? Why is the edit button missing in the new golden file?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stuff that is mounted via data is not editable for now. this is not a regression as the link was broken up to now. better no link than broken link.

@MaximilianSoerenPollak MaximilianSoerenPollak left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Talked about in many calls, seems good to me.

Size is this big cause of big golden files and scaffolding needed for testing.

@AlexanderLanin
AlexanderLanin merged commit 405b3a9 into eclipse-score:main Sep 8, 2026
18 of 20 checks passed
@AlexanderLanin
AlexanderLanin deleted the refactor/docs-bzl-golden-tests branch September 8, 2026 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

2 participants