Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 4 additions & 9 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,16 +59,11 @@ jobs:
- name: Check the browser compatibility
run: uv run marimo check app.py --select MW

# `marimo check` does not find a name that two cells both define.
# `app.run()` does, because it builds the cell graph.
# `marimo check` does not find a name that two cells both define, and a plain
# `app.run()` takes only the path without a result. The test checks every cell parameter
# and runs the result view with a fitted demo result.
- name: Run the notebook cells
run: |
uv run python -c "
import importlib.util, sys
spec = importlib.util.spec_from_file_location('notebook_app', 'app.py')
module = importlib.util.module_from_spec(spec); sys.modules['notebook_app'] = module
spec.loader.exec_module(module); module.app.run(); print('cells ok')
"
run: uv run pytest tests/test_notebook.py --no-cov

dependency-review:
name: Dependency review
Expand Down
25 changes: 14 additions & 11 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,8 @@ These rules apply only to Python that runs in a browser. A server target has no

## 8. Project notes — Simple Topic Modeling

These rules come from measured behaviour of `marimo` 0.24.2. Do not change them without a new test.
These rules come from measured behaviour of `marimo` 0.25.1 on Pyodide 314. Do not change them without a new test.
`pyproject.toml` pins `marimo` exactly, because its version fixes the Pyodide version of the export.

### Package layout

Expand Down Expand Up @@ -161,18 +162,20 @@ the page. The first cell therefore installs the wheel again, under a `sys.platfo
guard, using `mo.notebook_location()` to build an absolute URL. Keep both paths. The PEP 723 entry
also stops marimo from building its own wheel.

### Check the notebook with `app.run()`
### Check the notebook with `tests/test_notebook.py`

`marimo check` does not catch a name that two cells both define. `app.run()` does. Run this after you
edit `app.py`:
`marimo check` does not catch a name that two cells both define. A plain `app.run()` runs only the
path without a result. `tests/test_notebook.py` closes both gaps:

* It checks that some cell returns each cell parameter.
* It runs `app.run()` without a result.
* It runs `app.run(defs={"display_result": ...})` with a fitted newspaper demo and a fitted book
demo. The result view then runs every tab.

The gate runs this file. The CI `Notebook` job runs it too. Run it after you edit `app.py`:

```bash
uv run python -c "
import importlib.util, sys
spec = importlib.util.spec_from_file_location('notebook_app', 'app.py')
module = importlib.util.module_from_spec(spec); sys.modules['notebook_app'] = module
spec.loader.exec_module(module); module.app.run(); print('cells ok')
"
uv run pytest tests/test_notebook.py --no-cov
```

Prefix every cell-local variable with `_`. marimo requires a unique name across cells.
Expand Down Expand Up @@ -266,7 +269,7 @@ The remote is `github.com/maehr/simple-topic-modeling`. The published app is
Three workflows run:

* `ci.yml` runs the section 4 gate in the `Gate` job. The `Notebook` job runs `marimo check` and
the `app.run()` check. The `dependency-review` job runs on a pull request only.
`tests/test_notebook.py`. The `dependency-review` job runs on a pull request only.
* `pages.yml` builds the wheel, exports the app, and publishes `dist/`. It fails when the wheel
file name does not match the project version.
* `codeql.yml` scans the Python code.
Expand Down
10 changes: 3 additions & 7 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,15 +47,11 @@ uv run marimo edit app.py
```

Check the cell graph after you edit `app.py`. `marimo check` does not find a name that two cells
both define.
both define. The test checks every cell parameter. It also runs the result view with a fitted
demo result.

```bash
uv run python -c "
import importlib.util, sys
spec = importlib.util.spec_from_file_location('notebook_app', 'app.py')
module = importlib.util.module_from_spec(spec); sys.modules['notebook_app'] = module
spec.loader.exec_module(module); module.app.run(); print('cells ok')
"
uv run pytest tests/test_notebook.py --no-cov
```

The export runs Pyodide in the browser:
Expand Down
8 changes: 2 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,14 +270,10 @@ uv run pytest --doctest-modules --cov --cov-fail-under=100
```

Check the notebook after you edit it. `marimo check` does not find a name that two cells both define.
The test checks every cell parameter. It also runs the result view with a fitted demo result.

```bash
uv run python -c "
import importlib.util, sys
spec = importlib.util.spec_from_file_location('notebook_app', 'app.py')
module = importlib.util.module_from_spec(spec); sys.modules['notebook_app'] = module
spec.loader.exec_module(module); module.app.run(); print('cells ok')
"
uv run pytest tests/test_notebook.py --no-cov
```

Build the export and open it in a browser before you call the work done. The notebook and the export
Expand Down
2 changes: 1 addition & 1 deletion SPECS.md
Original file line number Diff line number Diff line change
Expand Up @@ -474,7 +474,7 @@ Every chart needs a title, hover tooltip, readable labels, and an empty state.

**Topic card:** name, top five terms, prevalence, click to select.

**Topic map:** bubble position = 2-D centroid, size = prevalence, click = select topic.
**Topic map:** bubble position = 2-D centroid, size = prevalence, label = topic number, click = select topic.

**Similarity heatmap:** cosine similarity between normalized topic-term vectors.

Expand Down
4 changes: 2 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,12 @@ Repository = "https://github.com/maehr/simple-topic-modeling"

[dependency-groups]
dev = [
"marimo>=0.24.2",
"marimo==0.25.1",
"prek>=0.5.3",
"pytest>=9.1.1",
"pytest-cov>=7.1.0",
"ruff>=0.16.8",
"ty==0.0.82",
"ty==0.0.84",
]

[build-system]
Expand Down
44 changes: 33 additions & 11 deletions simple_topic_modeling/plots.py
Original file line number Diff line number Diff line change
Expand Up @@ -237,35 +237,57 @@ def _pick_topic() -> alt.Parameter:
return alt.selection_point(fields=["topic_id"], on="click", clear=False)


def topic_map(result: TopicModelResult, selected: int | None = None) -> alt.Chart:
def topic_map(result: TopicModelResult, selected: int | None = None) -> alt.LayerChart:
"""Place each topic on the 2-D map. Bubble size is the prevalence.

A click selects a topic. The selected bubble keeps full opacity and gains an outline.
Each bubble shows its topic number, so a reader can find a topic without its colour. A click
selects a topic. The selected bubble keeps full opacity and gains an outline. The selection
dims the bubbles only, so every number keeps its contrast. The data sits on the layer chart,
so the notebook can still read the click through the chart value.

>>> from simple_topic_modeling.result import _example_result
>>> topic_map(_example_result()).to_dict()["mark"]["type"]
'circle'
>>> spec = topic_map(_example_result()).to_dict()
>>> [layer["mark"]["type"] for layer in spec["layer"]]
['circle', 'text']
"""
frame = topic_map_frame(result).assign(
selected=lambda data: (data["topic_id"] == selected).astype(int)
number=lambda data: data["topic_id"] + 1,
selected=lambda data: (data["topic_id"] == selected).astype(int),
)
return (
alt.Chart(frame, title="Topic map")
bubbles = (
alt.Chart()
.mark_circle(stroke="#000")
.encode(
**_emphasis(selected),
x=alt.X("x:Q", axis=_blank_axis()),
y=alt.Y("y:Q", axis=_blank_axis()),
size=alt.Size("prevalence:Q", title="Prevalence", scale=alt.Scale(range=[100, 2000])),
color=alt.Color("topic:N", legend=None),
)
.add_params(_pick_topic())
)
numbers = (
alt.Chart()
.mark_text(fontWeight="bold", fontSize=12, color="#000")
.encode(text=alt.Text("number:Q"))
)
return (
alt.layer(bubbles, numbers, data=frame, title="Topic map")
.encode(
x=alt.X("x:Q", axis=_blank_axis()),
y=alt.Y("y:Q", axis=_blank_axis()),
tooltip=[
alt.Tooltip("number:Q", title="Number"),
alt.Tooltip("topic:N", title="Topic"),
alt.Tooltip("top_terms:N", title="Top terms"),
alt.Tooltip("prevalence:Q", title="Prevalence", format=".1%"),
],
)
.add_params(_pick_topic())
.properties(height=380)
.properties(
height=380,
description=(
"Topic map. Each circle is one topic and shows its number. The circle size shows"
" the prevalence. " + AXIS_NOTE
),
)
)


Expand Down
90 changes: 90 additions & 0 deletions tests/test_notebook.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
"""Run the notebook outside the browser.

`marimo check` does not find a name that two cells both define, and a plain `app.run()` takes only
the path without a result. These tests close both gaps: every cell parameter must come from a
cell, and the result view must run with a fitted result.
"""

import ast
import importlib.util
import sys
from pathlib import Path
from typing import TypeGuard

import pytest

from simple_topic_modeling.config import AppConfig, ModelConfig
from simple_topic_modeling.io import build_corpus, demo_table, demo_text, split_long_document
from simple_topic_modeling.modeling import fit_topic_model

APP_PATH = Path(__file__).resolve().parent.parent / "app.py"


def _is_cell(node: ast.AST) -> TypeGuard[ast.FunctionDef | ast.AsyncFunctionDef]:
if not isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
return False
for decorator in node.decorator_list:
target = decorator.func if isinstance(decorator, ast.Call) else decorator
if isinstance(target, ast.Attribute) and target.attr == "cell":
return True
return False


def _returned_names(cell: ast.FunctionDef | ast.AsyncFunctionDef) -> set[str]:
names: set[str] = set()
for statement in cell.body:
if isinstance(statement, ast.Return) and statement.value is not None:
values = statement.value
items = values.elts if isinstance(values, ast.Tuple) else [values]
names.update(item.id for item in items if isinstance(item, ast.Name))
return names


@pytest.fixture(scope="module")
def notebook():
spec = importlib.util.spec_from_file_location("notebook_app", APP_PATH)
assert spec is not None
assert spec.loader is not None
module = importlib.util.module_from_spec(spec)
sys.modules["notebook_app"] = module
spec.loader.exec_module(module)
return module


def test_every_cell_parameter_is_returned_by_a_cell():
tree = ast.parse(APP_PATH.read_text(encoding="utf-8"))
cells = [node for node in ast.walk(tree) if _is_cell(node)]
returned = set().union(*(_returned_names(cell) for cell in cells))
missing = {
f"{cell.lineno}:{argument.arg}"
for cell in cells
for argument in cell.args.args
if argument.arg not in returned
}
assert not missing, f"Cell parameters that no cell returns: {sorted(missing)}"


def test_the_notebook_runs_without_a_result(notebook):
_, definitions = notebook.app.run()
assert definitions["display_result"] is None


def test_the_result_view_runs_with_the_demo_corpus(notebook):
table = demo_table()
metadata = table[["category", "date"]].rename(columns={"category": "group"})
corpus, _ = build_corpus(
table["text"].tolist(), table["document_id"].astype(str).tolist(), metadata
)
result = fit_topic_model(corpus, AppConfig())
_, definitions = notebook.app.run(defs={"display_result": result})
assert definitions["map_chart"] is not None
assert definitions["selected_index"] == 0


def test_the_result_view_runs_with_the_long_text_demo(notebook):
segments, identifiers, metadata = split_long_document(demo_text(), "demo.txt")
corpus, _ = build_corpus(segments, identifiers, metadata)
config = AppConfig(model=ModelConfig(n_topics=4), analyse_as="long_document")
result = fit_topic_model(corpus, config)
_, definitions = notebook.app.run(defs={"display_result": result})
assert definitions["map_chart"] is not None
29 changes: 23 additions & 6 deletions tests/test_plots.py
Original file line number Diff line number Diff line change
Expand Up @@ -112,10 +112,27 @@ def _spec(chart):
return chart.to_dict()


def _marks(spec):
"""Return the encoding of the marks that carry the selection: the bubbles on a layer chart."""
return spec["layer"][0]["encoding"] if "layer" in spec else spec["encoding"]


def test_topic_map_marks_size_by_prevalence(result):
bubbles = _spec(topic_map(result))["layer"][0]
assert bubbles["mark"]["type"] == "circle"
assert bubbles["encoding"]["size"]["field"] == "prevalence"


def test_topic_map_labels_each_bubble_with_its_number(result):
spec = _spec(topic_map(result))
assert spec["mark"]["type"] == "circle"
assert spec["encoding"]["size"]["field"] == "prevalence"
numbers = spec["layer"][1]
assert numbers["mark"]["type"] == "text"
assert numbers["encoding"]["text"]["field"] == "number"
rows = spec["datasets"][spec["data"]["name"]]
assert [row["number"] for row in rows] == list(range(1, result.n_topics + 1))
assert "description" in spec
assert "fillOpacity" not in numbers["encoding"]
assert "fillOpacity" not in _spec(topic_map(result, 1))["layer"][1]["encoding"]


def test_projection_axes_are_de_emphasized(result):
Expand Down Expand Up @@ -239,8 +256,8 @@ def test_selected_topic_is_marked_in_every_data_view(result):
spec = _spec(chart)
rows = spec["datasets"][spec["data"]["name"]]
assert [row["selected"] for row in rows] == [i == 1 for i in range(result.n_topics)]
assert "condition" in spec["encoding"]["fillOpacity"]
assert "condition" in spec["encoding"]["strokeWidth"]
assert "condition" in _marks(spec)["fillOpacity"]
assert "condition" in _marks(spec)["strokeWidth"]
heat = _spec(similarity_heatmap(result, 0))
rows = heat["datasets"][heat["data"]["name"]]
assert [row["selected"] for row in rows] == [
Expand All @@ -252,8 +269,8 @@ def test_selected_topic_is_marked_in_every_data_view(result):
def test_no_selection_dims_nothing(result):
for chart in (topic_map(result), prevalence_bars(result), similarity_heatmap(result)):
spec = _spec(chart)
assert spec["encoding"]["fillOpacity"]["value"] == 0.85
assert spec["encoding"]["strokeWidth"]["value"] == 0
assert _marks(spec)["fillOpacity"]["value"] == 0.85
assert _marks(spec)["strokeWidth"]["value"] == 0


def test_topic_card_html_shows_a_custom_name_verbatim():
Expand Down
Loading
Loading