Make the source links really resolve to source files instead of the rendered Doxygen HTML

cmake/registry.cmake

@@ -316,6 +316,21 @@ function(add_docs_from_registry)
         endforeach()
       endif()
 
+      # testmodule.coverage_source_docs: the testcoverage directive reads the
+      # source listings of these Doxygen documents (deploy/xml/<id>/), so the
+      # same stage-2 edge, once per document.
+      string(JSON _zdocs_coverage_n LENGTH "${_zdocs_entry}" "testmodule_coverage_source_docs")
+      if(_zdocs_coverage_n GREATER 0)
+        math(EXPR _zdocs_coverage_last "${_zdocs_coverage_n} - 1")
+        foreach(_zdocs_coverage_i RANGE ${_zdocs_coverage_last})
+          string(JSON _zdocs_coverage_dox GET "${_zdocs_entry}"
+            "testmodule_coverage_source_docs" ${_zdocs_coverage_i})
+          foreach(_zdocs_builder ${_zdocs_builders})
+            add_dependencies(${_zdocs_id}-${_zdocs_builder} ${_zdocs_coverage_dox})
+          endforeach()
+        endforeach()
+      endif()
+
       string(JSON _zdocs_testmodule_spec GET "${_zdocs_entry}" "testmodule_spec")
       if(NOT _zdocs_testmodule_spec STREQUAL "")
         add_dependencies(${_zdocs_id}-index ${_zdocs_testmodule_spec}-index)

doc/manual/howto/render-test-specifications.rst

@@ -62,6 +62,10 @@ literally named ``ZTEST`` — expanding it also injects ``\ingroup suite`` from
 the macro's own argument, which is what attaches a test case to its suite
 without a hand-written (and driftable) ``@ingroup``.
 
+Optional: ``SOURCE_BROWSER = YES`` makes Doxygen write a source listing of
+each file. The Source line of each test case then links to the test's line in
+that listing, not to the test's entry on the group page.
+
 4. Declare the needs vocabulary
 ------------------------------------
 

doc/manual/reference/directives-and-roles.rst

@@ -105,6 +105,13 @@ of ``testcase.yaml``, ``tests.yaml`` and ``sample.yaml`` that exists. Every
 ``ZTEST``/``ZTEST_SUITE``/... in the named group and its inner suite/procedure
 groups becomes one need each — nothing is written by hand per test case.
 
+The Source line of a test case or a procedure (``main.c (line 290)``) links
+to the Doxygen source listing of its file, at that line, if the test
+specification's Doxyfile sets ``SOURCE_BROWSER = YES``. zdocs reads the
+setting from the ``Doxyfile.xml`` that Doxygen writes beside the XML. Without
+the listings, the Source line links to the member's documentation in the
+group page.
+
 A test case need's ``suite`` field is its inner suite group's name, so name
 that group after the ``ZTEST_SUITE``: ``testreport`` finds the test case for a
 twister result by the pair (suite, function). Two test modules that declare the
@@ -382,9 +389,37 @@ requirement. Its body has:
 * The verifying tests, and the fields that the consumer does not declare
   (``verdict``, ``evidence``, ``coverage_run``).
 
-The table cells use the roles ``zd-adq-ok``, ``zd-adq-miss``, ``zd-adq-none``
-and ``zd-adq-loc``. Each gives the text a class of the same name. The zdocs
-need card (:ref:`need-layouts`) colors them.
+The table cells use the roles ``zd-adq-ok``, ``zd-adq-miss`` and
+``zd-adq-none``. Each gives the text a class of the same name. The form and
+place of a body is a paragraph of class ``zd-adq-loc``. The zdocs need card
+(:ref:`need-layouts`) colors them.
+
+The place of a body (``kernel/thread.c:344-358``) links to the first line of
+the body in a Doxygen source listing. The registry key
+``testmodule.coverage_source_docs`` names the Doxygen documents to look in, in
+order; the first that has a listing of the file wins:
+
+.. code-block:: yaml
+
+   test-report:
+     kind: sphinx
+     testmodule:
+       spec: test-specification
+       coverage_source_docs: [dox-detailed-design, dox-api]
+
+The registry gives this list to the build as ``testcoverage_source_docs``, and
+each builder of the document waits for these Doxygen documents. The links
+read the XML of each document (``deploy/xml/<id>/``), so the top-level
+``doxygen_xml: true`` of the registry is necessary. Without it, there is no
+XML and no link. See :doc:`registry-schema`.
+
+The body path is relative to ``testmodule_root``; the Doxygen path can have
+more or fewer directories in front (``zephyr/kernel/thread.c``), if only one
+file of the document matches. A Doxygen document lists every code file with
+``SOURCE_BROWSER = YES``, and only the headers with ``VERBATIM_HEADERS = YES``.
+The default is ``[]``: no links. A ``conf.py`` can also set
+``testcoverage_source_docs``, but then the build does not wait for the
+Doxygen documents.
 
 The id of a need is ``ADQ-<run>/<requirement>``. ``testcoverage_id_prefix``
 sets the prefix. The type, the link and the fields are roles, as for the other
@@ -444,8 +479,11 @@ Each need is titled with the symbol name and has the id
 ``<TYPE>-<symbol>``, where ``<TYPE>`` is the need type's name in upper case
 (``IMPL-k_queue_init``). Its body gives the kind, the brief, a link to the
 symbol's Doxygen page and the file it is declared in; no custom field is
-needed for them. The need type and link are engine roles, named by the
-consumer like the test directives' (ADR-0009):
+needed for them. "Declared in" links to the declaration's line in Doxygen's
+source listing, if the run wrote one: with ``SOURCE_BROWSER = YES`` for every
+file, with ``VERBATIM_HEADERS = YES`` (the Doxygen default) for the headers.
+The need type and link are engine roles, named by the consumer like the test
+directives' (ADR-0009):
 
 .. code-block:: python
 

doc/manual/reference/registry-schema.rst

@@ -324,6 +324,16 @@ Each entry in ``documents:``, keyed by its id:
       It shows as a literal in the "See also" line, without a link. The
       line has the targets of all ``@see`` lines of the test, in order.
 
+   ``coverage_source_docs``
+      A list of ``kind: doxygen`` document ids. The ``testcoverage``
+      directive links the place of each body to the source listing of the
+      first of these documents that has the file (see
+      :doc:`directives-and-roles`). Every stage-2 builder of this document
+      waits for each of them, as for ``doxygen_source``. The links read the
+      XML (``deploy/xml/<id>/``), so the top-level ``doxygen_xml: true`` is
+      necessary. A value that is not a list, or an id that is not an
+      existing ``kind: doxygen`` document, is a configure-time error.
+
    ``spec``
       Id of the sphinx document whose exported needs a ``testreport`` document
       correlates against. Must exist, and must carry ``needs: {source: json}``

scripts/docrefs.py

@@ -144,7 +144,7 @@ def _registry(registry):
 #: typo'd ``doxygen_src:``) must be a configure-time error, not silently
 #: ignored: silently ignoring it would degrade to "no XML" with no
 #: diagnostic at all.
-_TESTMODULE_KEYS = ("doxygen_source", "api_reference", "spec")
+_TESTMODULE_KEYS = ("doxygen_source", "api_reference", "spec", "coverage_source_docs")
 
 
 #: Allowed keys inside a document's ``symbol_needs:`` block. ``doxygen_source``
@@ -358,6 +358,33 @@ def _validate(data):
                         f"'kind: doxygen' document"
                     )
 
+            # The Doxygen documents whose source listings the testcoverage
+            # directive links the bodies to, in order. A list, so each id is
+            # checked as doxygen_source is; the build waits for each of them.
+            coverage_docs = testmodule.get("coverage_source_docs")
+            if coverage_docs is not None:
+                if not isinstance(coverage_docs, list):
+                    raise ValueError(
+                        f"docrefs: document '{doc_id}' has "
+                        f"testmodule.coverage_source_docs: '{coverage_docs}' "
+                        f"— use a list of 'kind: doxygen' document ids"
+                    )
+                for ref_id in coverage_docs:
+                    ref_meta = data.get("documents", {}).get(ref_id)
+                    if ref_meta is None:
+                        raise ValueError(
+                            f"docrefs: document '{doc_id}' has "
+                            f"testmodule.coverage_source_docs: '{ref_id}', "
+                            f"which does not exist in the registry"
+                        )
+                    ref_kind = ref_meta.get("kind", "sphinx")
+                    if ref_kind != "doxygen":
+                        raise ValueError(
+                            f"docrefs: document '{doc_id}' has "
+                            f"testmodule.coverage_source_docs: '{ref_id}', "
+                            f"which is kind '{ref_kind}', not 'doxygen'"
+                        )
+
             spec_id = testmodule.get("spec")
             if spec_id is not None:
                 spec_meta = data.get("documents", {}).get(spec_id)
@@ -643,6 +670,7 @@ def __init__(
         deploy_dirs=None,
         testmodule=None,
         symbol_needs=None,
+        doxygen_docs=None,
     ):
         self.reference_groups = reference_groups
         self.intersphinx_mapping = intersphinx_mapping
@@ -666,6 +694,11 @@ def __init__(
         #: loaded). A dict with ``xml_dir`` (the Doxygen document's XML) and
         #: ``doxygen_url`` (its HTML, relative to this document's root).
         self.symbol_needs = symbol_needs
+        #: Every ``kind: doxygen`` document: ``{id: {"xml_dir", "html_url"}}``,
+        #: its XML and its HTML relative to this document's root. For an
+        #: extension that a ``conf.py`` points at a Doxygen document by id
+        #: (``testcoverage_source_docs``).
+        self.doxygen_docs = doxygen_docs or {}
 
 
 def _resolve_external_url(url, external_base_url):
@@ -963,6 +996,7 @@ def _nav_href(doc_id, meta):
             "api_url": rel_urls.get(api_reference, "") if api_reference else "",
             "needs_json": needs_json,
             "tag_urls": tag_urls(documents, deploy, rel_urls),
+            "coverage_source_docs": list(testmodule_block.get("coverage_source_docs") or []),
         }
 
     symbol_needs = None
@@ -974,6 +1008,12 @@ def _nav_href(doc_id, meta):
             "doxygen_url": rel_urls.get(source, ""),
         }
 
+    doxygen_docs = {
+        doc_id: {"xml_dir": str(deploy / "xml" / doc_id), "html_url": rel_urls[doc_id]}
+        for doc_id, meta in documents.items()
+        if meta.get("kind") == "doxygen" and doc_id in rel_urls
+    }
+
     return Refs(
         reference_groups,
         intersphinx_mapping,
@@ -985,6 +1025,7 @@ def _nav_href(doc_id, meta):
         deploy_dirs=deploy_dirs,
         testmodule=testmodule,
         symbol_needs=symbol_needs,
+        doxygen_docs=doxygen_docs,
     )
 
 
@@ -1271,6 +1312,10 @@ def manifest(registry=None):
         never its own spec), so this cannot cycle; it is deliberately NOT
         generalised to every needs-importer/publisher pair, which CAN cycle
         and is its own, deferred step.
+      * ``testmodule_coverage_source_docs`` — this document's own
+        ``testmodule.coverage_source_docs`` (``[]`` without it). Every
+        stage-2 builder of the document waits for each of these Doxygen
+        documents, as for ``testmodule_doxygen_source``.
       * ``symbol_needs_doxygen_source`` — this document's own
         ``symbol_needs:`` block's ``doxygen_source`` (``None`` without the
         block). Wired like ``testmodule_doxygen_source``: every stage-2
@@ -1294,6 +1339,9 @@ def manifest(registry=None):
                 "remote_tagfile": meta.get("remote-tagfile"),
                 "testmodule_doxygen_source": testmodule.get("doxygen_source"),
                 "testmodule_spec": testmodule.get("spec"),
+                "testmodule_coverage_source_docs": list(
+                    testmodule.get("coverage_source_docs") or []
+                ),
                 "symbol_needs_doxygen_source": (meta.get("symbol_needs") or {}).get(
                     "doxygen_source"
                 ),

sphinx/_extensions/_tests/test_adequacy.py

@@ -37,13 +37,17 @@
 
 import json
 import subprocess
+import sys
 from pathlib import Path
 
 import adequacy as A
 import pytest
 import test_coverage as tc
 from rst_builders import _DEFAULT_NEED_NAMES
 
+sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "scripts"))
+import docrefs  # noqa: E402
+
 OBJ_C = """\
 #include <zephyr/kernel.h>
 
@@ -494,14 +498,34 @@ def test_adequacy_rst_gives_one_row_per_body(tree):
     # The function whose body was read, its form and place, the own lines of
     # the lines any test ran, and the other test cases.
     assert "      * - ``z_impl_k_obj_init``" in rst
-    assert f":zd-adq-loc:`implementation, kernel/obj.c:{IMPL[0]}-{IMPL[1]}`" in rst
+    assert (
+        "          .. rst-class:: zd-adq-loc\n\n"
+        f"          implementation, kernel/obj.c:{IMPL[0]}-{IMPL[1]}\n"
+    ) in rst
     assert ":zd-adq-none:`no test ran it`" in rst
     assert "      * - ``z_vrfy_k_obj_init``" in rst
     assert ":zd-adq-ok:`3 of 3 lines`" in rst
     # The own tests run every body they should: no list of other tests.
     assert "Other tests that run" not in rst
 
 
+def test_adequacy_rst_links_the_place_of_a_body_to_its_listing(tree):
+    run, results, _, _ = _assess(tree)
+    calls = []
+
+    def body_url(path, line):
+        calls.append((path, line))
+        return f"dd/{path}#l{line}" if path == "kernel/obj.c" else ""
+
+    rst = "\n".join(tc.build_adequacy_rst("R-VRFY", results["R-VRFY"], run, body_url=body_url))
+    a, b = IMPL
+    assert f"implementation, `kernel/obj.c:{a}-{b} <dd/kernel/obj.c#l{a}>`__" in rst
+    assert ("kernel/obj.c", a) in calls
+    rst = "\n".join(tc.build_adequacy_rst("R-INLINE", results["R-INLINE"], run, body_url=body_url))
+    a, b = INLINE
+    assert f"static inline, include/zephyr/sys/k_inline.h:{a}-{b}\n" in rst  # no listing
+
+
 def test_adequacy_rst_names_the_other_tests_of_a_missed_body(tree):
     run, results, _, _ = _assess(tree)
     rst = "\n".join(tc.build_adequacy_rst("R-BROKEN", results["R-BROKEN"], run))
@@ -634,6 +658,106 @@ def test_directive_reads_the_consumers_file_set(tree, make_app, tmp_path):
     assert "include/zephyr/sys/**/*.h" in html and "kernel/*.c" not in html
 
 
+def _doxygen_run(path, files):
+    """A Doxygen XML dir with SOURCE_BROWSER = YES and ``files`` ({refid: path})."""
+    path.mkdir()
+    (path / "Doxyfile.xml").write_text(
+        "<doxyfile><option id='SOURCE_BROWSER'><value>YES</value></option></doxyfile>"
+    )
+    (path / "index.xml").write_text("<doxygenindex>" + "".join(
+        f"<compound refid='{r}' kind='file'><name>{Path(f).name}</name></compound>"
+        for r, f in files.items()
+    ) + "</doxygenindex>")
+    for r, f in files.items():
+        (path / f"{r}.xml").write_text(
+            f"<doxygen><compounddef id='{r}' kind='file'><location file='{f}'/>"
+            "</compounddef></doxygen>"
+        )
+    return path
+
+
+def test_directive_links_each_body_to_its_source_listing(tree, make_app, tmp_path):
+    root, _, run_dir, needs_json = tree
+    xml = _doxygen_run(tmp_path / "dd-xml", {"zephyr_2kernel_2plain_8c": "zephyr/kernel/plain.c"})
+    src = tmp_path / "doc"
+    src.mkdir()
+    (src / "conf.py").write_text(_CONF.format(
+        ext=str(Path(A.__file__).parent), needs=str(needs_json), run=str(run_dir),
+        root=str(root),
+    ) + (
+        f"zdocs_doxygen_docs = {{'dd': {{'xml_dir': {str(xml)!r}, 'html_url': '../dd'}}}}\n"
+        "testcoverage_source_docs = ['nope', 'dd']\n"
+    ))
+    (src / "index.rst").write_text("Adequacy\n########\n\n.. testcoverage::\n")
+    app = make_app("html", srcdir=src)
+    app.build()
+    html = (Path(app.outdir) / "index.html").read_text()
+    # The body path is from the tree root, Doxygen's from one level up.
+    assert f'href="../dd/zephyr_2kernel_2plain_8c_source.html#l{PLAIN[0]:05d}"' in html
+    log = app._warning.getvalue()
+    assert "testcoverage_source_docs names 'nope'" in log
+    inputs = set(app.env.zdocs_report_inputs["index"])
+    for name in ("Doxyfile.xml", "index.xml", "zephyr_2kernel_2plain_8c.xml"):
+        assert str(xml / name) in inputs
+
+
+# ---------------------------------------------------------------------------
+# Registry: testmodule.coverage_source_docs
+# ---------------------------------------------------------------------------
+
+REGISTRY = """\
+groups:
+  - id: reference
+    title: Reference
+documents:
+  report:
+    kind: sphinx
+    group: reference
+    builders: [html]
+    testmodule:
+      coverage_source_docs: [dox-dd, dox-api]
+  dox-dd:
+    kind: doxygen
+    group: reference
+  dox-api:
+    kind: doxygen
+    group: reference
+"""
+
+
+def _registry(tmp_path, text=REGISTRY):
+    path = tmp_path / "documents.yaml"
+    path.write_text(text)
+    return path
+
+
+def test_registry_gives_the_coverage_source_docs(tmp_path, monkeypatch):
+    monkeypatch.setenv("OUTPUT_DIR", str(tmp_path / "deploy" / "html" / "report"))
+    refs = docrefs.load("report", _registry(tmp_path))
+    assert refs.testmodule["coverage_source_docs"] == ["dox-dd", "dox-api"]
+    assert set(refs.doxygen_docs) == {"dox-dd", "dox-api"}
+
+
+def test_manifest_names_the_coverage_source_docs(tmp_path):
+    manifest = docrefs.manifest(_registry(tmp_path))
+    entries = {e["id"]: e["testmodule_coverage_source_docs"] for e in manifest}
+    assert entries == {"report": ["dox-dd", "dox-api"], "dox-dd": [], "dox-api": []}
+
+
+@pytest.mark.parametrize(
+    ("after", "message"),
+    [
+        ("coverage_source_docs: dox-dd", "use a list"),
+        ("coverage_source_docs: [nowhere]", "does not exist"),
+        ("coverage_source_docs: [report]", "not 'doxygen'"),
+    ],
+)
+def test_invalid_coverage_source_docs_fail_validation(tmp_path, after, message):
+    text = REGISTRY.replace("coverage_source_docs: [dox-dd, dox-api]", after)
+    with pytest.raises(ValueError, match=message):
+        docrefs.manifest(_registry(tmp_path, text))
+
+
 def test_directive_without_a_run_says_so(make_app, tmp_path):
     src = tmp_path / "doc"
     src.mkdir()