Add notebook smoke tests and rebuilt lectures

.github/workflows/notebook-smoke.yml

@@ -0,0 +1,77 @@
+name: Notebook smoke
+
+on:
+  pull_request:
+    paths:
+      - "_notebooks/**/*.ipynb"
+      - "scripts/smoke_notebooks.py"
+      - "requirements-smoke.txt"
+      - ".github/workflows/notebook-smoke.yml"
+  schedule:
+    # Weekly full suite (Sunday 04:00 UTC)
+    - cron: "0 4 * * 0"
+  workflow_dispatch:
+    inputs:
+      only:
+        description: "Optional notebook prefixes (space-separated), e.g. 01 12. Empty = all except skipped."
+        required: false
+        default: ""
+
+concurrency:
+  group: notebook-smoke-${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  smoke:
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+          cache-dependency-path: requirements-smoke.txt
+
+      - name: Install smoke dependencies
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install -r requirements-smoke.txt
+
+      - name: Cache pods / notebook data downloads
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.pods_data
+            _notebooks/data
+          key: pods-data-${{ runner.os }}-${{ hashFiles('requirements-smoke.txt') }}
+          restore-keys: |
+            pods-data-${{ runner.os }}-
+
+      - name: Run notebook smoke tests
+        env:
+          MPLBACKEND: Agg
+          JUPYTER_PLATFORM_DIRS: "1"
+        run: |
+          set -euo pipefail
+          ONLY="${{ github.event.inputs.only }}"
+          if [ -n "${ONLY}" ]; then
+            # shellcheck disable=SC2086
+            python scripts/smoke_notebooks.py --only ${ONLY}
+          else
+            python scripts/smoke_notebooks.py
+          fi
+
+      - name: Upload smoke results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: notebook-smoke-results
+          path: |
+            _notebooks/test-results/summary.json
+            _notebooks/test-results/*.ok
+            _notebooks/test-results/*.fail
+          if-no-files-found: ignore

.gitignore

@@ -0,0 +1,54 @@
+# Jupyter and notebook checkpoints
+.ipynb_checkpoints/
+
+# OS and editor files
+.DS_Store
+*.swp
+*.swo
+*~
+\#*\#
+.\#*
+
+# Local workspace files
+*.code-workspace
+
+# Python cache
+__pycache__/
+*.pyc
+*.pyo
+*.pyd
+
+# Backups
+*.bak
+
+# Build artefacts
+*.o
+*.so
+*.out
+*.log
+
+# LaMD intermediate build artefacts (maketalk) under _lamd/
+# Keep sources (*.md), _lamd.yml, compile.sh, lectures.csv tracked.
+_lamd/*.gpp.markdown
+_lamd/*.ipynb
+_lamd/*.ipynb.markdown
+_lamd/*.html
+_lamd/*.html.markdown
+_lamd/*.notes.*
+_lamd/*.posts.*
+_lamd/*.slides.*
+_lamd/makefile
+_lamd/*.log
+_lamd/talk-people.gpp
+_lamd/*-e
+_lamd/*.tmp.markdown
+
+# Compiled Jekyll site
+_site/
+
+# Ruby bundler
+.bundle/
+
+# Local Python env and notebook smoke artefacts
+.venv/
+_notebooks/test-results/

README.md

@@ -22,6 +22,21 @@ The lecture notes will all be given in the form of Jupyter Notebooks and are ava
 * [Week 12](./week12.ipynb) Special Topic: Gaussian Processes
 
 
+## Notebook smoke tests
+
+Execute numbered lecture notebooks under `_notebooks/` with CI-safe patches (Agg backend, mocked stdin, capped GPy `optimize`, no widget hangs). Lecture 05 is skipped by default because it needs a live MariaDB and credential widgets.
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -r requirements-smoke.txt
+python scripts/smoke_notebooks.py
+# subset: python scripts/smoke_notebooks.py --only 01 12
+# include lecture 05: python scripts/smoke_notebooks.py --only 05
+```
+
+GitHub Actions runs the same harness on pull requests that touch notebooks or the smoke files, weekly on a schedule, and via **workflow_dispatch** (optional `--only` input). Results land in `_notebooks/test-results/` (gitignored) and as a workflow artifact.
+
 ## ToDo
 
 There are some refinements to the notebook slides required. Particularly for Week 8 (unsupervised learning) 9 and 10. 

_lamd/_lamd.yml

@@ -18,6 +18,7 @@ notesdir: ../_notes/
 notebooksdir: ../_notebooks/
 practicalsdir: ../_practicals/
 snippetsdir: $HOME/lawrennd/snippets
+macrosdir: $HOME/lawrennd/lamd/lamd/macros
 writediagramsdir: .
 scriptsdir: https://inverseprobability.com/talks/scripts/
 diagramsdir: ../slides/diagrams/

_lamd/basis-functions.md

@@ -11,6 +11,10 @@ abstract: |
 \include{_mlai/includes/mlai-notebook-setup.md}
 \include{_ml/includes/non-linear-regression-intro.md}
 
+\define{designVector}{\basisVector}
+\define{designVariable}{Phi}
+\define{designMatrix}{\basisMatrix}
+
 \include{_ml/includes/basis-functions.md}
 \include{_ml/includes/basis-function-models.md}
 

_lamd/matrix-factorization.md

@@ -46,7 +46,7 @@ are going to take a slightly different tack.}
 
 \include{_software/includes/pods-software.md}
 
-\include{_ml/includes/movie-body-count-data.md}
+\include{_data-science/includes/movie-body-count-pandas.md}
 
 \writeassignment{Data ethics. If you find data available on the
 internet, can you simply use it without consequence? If you are given data by a