diff --git a/CHANGELOG.md b/CHANGELOG.md index a005175..6a88eec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [unreleased] + +### Added + +- build:: + - Support setuptools.packages.find.where directive in setup converter. + - Support setuptools.data-files directives in setup converter. + ## [1.8.0] - 2026-05-22 ### Added diff --git a/src/build/README.md b/src/build/README.md index ab21336..0743694 100644 --- a/src/build/README.md +++ b/src/build/README.md @@ -2,6 +2,128 @@ Utilities to help backport builds of Python projects. +## PEP 517 setup converter + +Some platforms ship `pip` and `setuptools` without full [PEP 517](https://peps.python.org/pep-0517/) +support — they cannot build a package by invoking the `[build-system]` backend declared in +`pyproject.toml` (for example Python 3.6 on Rocky Linux 8, Ubuntu Jammy, or openSUSE Leap +15). The setup converter is a small `setup.py` shim that reads `pyproject.toml` from the +current working directory and calls `setuptools.setup()` with mapped parameters. + +The script is shipped as package data in `rfl.build.scripts` and is meant to be copied +next to `pyproject.toml` as `setup.py` before running legacy `pip install`. + +### Usage + +Copy the converter into a package tree with `pyproject.toml`: + +```bash +rfl-install-setup-generator +``` + +This writes `./setup.py` in the current working directory. + +When installing RFL from source, `install.sh` can copy the converter into every package +under `src/` automatically: + +```bash +PEP517_SETUP_WRAPPER=1 bash install.sh +``` + +### Supported `pyproject.toml` fields + +The converter maps a subset of modern metadata to `setuptools.setup()` keyword +arguments: + +| Source | `setup()` argument | +|--------|-------------------| +| `[project]` `name`, `version` | `name`, `version` | +| `[project]` `authors[0]` | `author`, `author_email` | +| `[project]` `scripts` | `entry_points["console_scripts"]` | +| `[project]` `dependencies` | `install_requires` | +| `[project]` `optional-dependencies` | `extras_require` | +| `[project]` `license.text` | `license` | +| `[project]` `license.file` | `license_files` | +| `[project.urls]` `Homepage` | `url` | +| `[tool.setuptools]` `packages` (explicit list) | `packages` | +| `[tool.setuptools.packages.find]` `include` | `packages` (namespace detection; filtered discovery only when `where != "."`) | +| `[tool.setuptools.packages.find]` `where` | `packages` via `find_packages(where=…)` when not `"."`; sets `package_dir={"": where}` | +| `[tool.setuptools.packages.find]` `exclude` | passed to `find_packages(exclude=…)` when `where != "."` | +| `[tool.setuptools]` `package-data` | `package_data`, `include_package_data=True` | +| `[tool.setuptools]` `data-files` | `data_files` (with glob expansion) | + +`platforms` is always set to `["GNU/Linux"]`. + +The `[tool.setuptools]` section is optional. If `[project]` is missing, the script +prints a message and exits with code 0. + +### Package discovery + +`[tool.setuptools.packages.find]` supports `where`, `include`, and `exclude`: + +- `where` (default `["."]`): directory searched for packages. Use `where = ["src"]` + for src-layout projects without an explicit `include`. +- `include` (default `["*"]`): on flat layout (`where` is `"."`), used only for + namespace detection (see below), not passed to `find_packages()`. On src layout, + glob patterns are passed to `find_packages(include=…)`. +- `exclude` (default `[]`): passed to `find_packages(exclude=…)` on src layout only. +- When `where` is not `"."`, the converter also sets `package_dir = {"": where}` so + setuptools 39 can locate packages under `src/` (PEP 517 backends infer this + automatically; the shim must set it explicitly). +- Only the first `where` entry is used when multiple directories are listed. + +When `include` patterns contain a dot (for example `rfl.build*`), entries without a +dot are ignored. For each dotted pattern, the converter inspects the top-level +directory under `where` (the part before the first dot). If that directory has no +`__init__.py`, it is treated as a native namespace package. In that case, wildcard +characters are stripped from include patterns and the resulting names are passed +explicitly to `setup()` — `find_packages()` is not used. + +If every inspected top-level directory contains an `__init__.py`, `find_packages()` +is used. On flat layout it is called without filters (legacy behaviour). On src +layout it is called with the configured `where`, `include`, and `exclude` values. + +This workaround exists because `find_namespace_packages()` requires setuptools ≥ +40.1.0 and is unavailable on Rocky Linux 8 (setuptools 39.2.0). + +### Data files + +`[tool.setuptools.data-files]` maps install destinations to source path lists. +Glob patterns (`*`, `?`, `[…]`) are expanded relative to the project root before +calling `setup()`. Setuptools 39 does not expand globs in `data_files` natively; +the converter mirrors modern PEP 517 behaviour. + +Example: + +```toml +[tool.setuptools.data-files] +"myapp/conf" = ["conf/app.yml", "conf/app.ini.example"] +"myapp/templates" = ["web/templates/*"] +``` + +### Limits + +The converter is intentionally minimal. It does **not** map many common +`pyproject.toml` fields, including: + +- `description`, `readme`, `requires-python`, `keywords`, `classifiers` +- license metadata as a plain SPDX string (only `license.text` and `license.file` + tables are handled explicitly) +- project URLs other than `Homepage` +- entry points other than `[project.scripts]` console scripts +- dynamic metadata, `pyproject.toml` `[build-system]` options, and most + `[tool.setuptools]` directives beyond packages, package data, and data files +- `[tool.setuptools]` `package-dir` as an explicit pyproject key (only inferred from + `where`) + +Only the first `[project]` author entry is used. + +The script must be executed from the directory that contains `pyproject.toml`. It +requires `tomllib` (Python 3.11+) or the `tomli` package on older Python versions. + +On modern toolchains with full PEP 517 support, prefer building directly from +`pyproject.toml` and do not copy this shim. + ## `rfl.build.testing` subpackage The `rfl.build.testing` subpackage provides reusable helpers for writing unit tests in diff --git a/src/build/rfl/build/scripts/setup b/src/build/rfl/build/scripts/setup index 1f510ef..57181bc 100644 --- a/src/build/rfl/build/scripts/setup +++ b/src/build/rfl/build/scripts/setup @@ -11,8 +11,9 @@ projects that provide pyproject.toml with old versions of pip/python (eg. python without PEP 518 support. """ -import sys +import glob import os +import sys from setuptools import setup, find_packages @@ -23,6 +24,34 @@ except ImportError: header = "rfl-build" +_GLOB_CHARACTERS = {"*", "?", "[", "]", "{", "}"} + + +def _expand_glob_patterns(patterns, root_dir="."): + """Expand glob patterns to explicit file paths relative to root_dir. + + Each entry in patterns is either a literal path or a glob. Literal paths are + returned unchanged apart from normalisation to a forward-slash path relative to + root_dir. Glob patterns are expanded with glob.iglob; matching paths are sorted + and returned with the same normalisation. + + Returns a list of source file paths suitable for setuptools setup(data_files=…). + """ + expanded = [] + for value in patterns: + if any(char in value for char in _GLOB_CHARACTERS): + glob_path = os.path.abspath(os.path.join(root_dir, value)) + expanded.extend( + sorted( + os.path.relpath(path, root_dir).replace(os.sep, "/") + for path in glob.iglob(glob_path) + ) + ) + else: + expanded.append(os.path.relpath(value, root_dir).replace(os.sep, "/")) + return expanded + + with open("pyproject.toml", "rb") as fh: pyproject = tomllib.load(fh) @@ -55,26 +84,46 @@ if "scripts" in pyproject["project"]: # this case, the subpackages are explicity declared (without wildcard) in setup(). If no # namespace is detected, the packages are automatically detected with find_packages(). -if ( - "tool" in pyproject - and "setuptools" in pyproject["tool"] -): +if "tool" in pyproject and "setuptools" in pyproject["tool"]: if "packages" in pyproject["tool"]["setuptools"]: - if "find" in pyproject["tool"]["setuptools"]["packages"]: + find_config = pyproject["tool"]["setuptools"]["packages"]["find"] + # Same defaults as setuptools when keys are omitted from pyproject.toml. + where = find_config.get("where", ["."]) + # where is a list in pyproject.toml; find_packages() accepts one directory. + where_dir = where[0] if isinstance(where, list) else where + include = find_config.get("include", ["*"]) + exclude = find_config.get("exclude", []) + autofind = True packages = [] - for include in pyproject["tool"]["setuptools"]["packages"]["find"]["include"]: - if "." not in include: + for pattern in include: + if "." not in pattern: continue - topfolder = include.split(".", 1)[0] + # Namespace check must run under where_dir (eg. src/rfl, not rfl). + topfolder = os.path.join(where_dir, pattern.split(".", 1)[0]) if "__init__.py" not in os.listdir(topfolder): autofind = False - packages.append(include.replace("*", "")) + packages.append(pattern.replace("*", "")) if autofind: - kwargs["packages"] = find_packages() + if where_dir != ".": + # Src layout: honour where/include/exclude like a PEP 517 backend. + kwargs["packages"] = find_packages( + where=where_dir, + include=tuple(include), + exclude=tuple(exclude), + ) + # Setuptools 39 needs this; it looks at the project root otherwise. + kwargs["package_dir"] = {"": where_dir} + else: + # Flat layout: include is for namespace detection only; do not + # filter find_packages(). + kwargs["packages"] = find_packages() else: kwargs["packages"] = packages + if where_dir != ".": + # Explicit package names still live under where_dir on src layout. + kwargs["package_dir"] = {"": where_dir} else: # Explicit packages listing without find kwargs["packages"] = pyproject["tool"]["setuptools"]["packages"] @@ -89,6 +138,16 @@ if ( kwargs["include_package_data"] = True print(f"{header}: package data {kwargs['package_data']}") + if "data-files" in pyproject["tool"]["setuptools"]: + # Modern setuptools expands globs in [tool.setuptools.data-files] over PEP + # 517, but setuptools 39 setup(data_files=…) expects an explicit file list. + # Expand patterns here so legacy pip install matches PEP 517 behaviour. + kwargs["data_files"] = [ + (dest, _expand_glob_patterns(patterns)) + for dest, patterns in pyproject["tool"]["setuptools"]["data-files"].items() + ] + print(f"{header}: data files {kwargs['data_files']}") + if "dependencies" in pyproject["project"]: kwargs["install_requires"] = pyproject["project"]["dependencies"] diff --git a/src/build/rfl/tests/test_setup.py b/src/build/rfl/tests/test_setup.py index 8afa86e..8d1f1d9 100644 --- a/src/build/rfl/tests/test_setup.py +++ b/src/build/rfl/tests/test_setup.py @@ -152,6 +152,127 @@ def layout(root): expected_packages, ) + @patch("sys.exit") + @patch("setuptools.setup") + def test_autofind_packages_with_top_level_wildcard(self, mock_setup, mock_exit): + extra = """ + [tool.setuptools.packages.find] + include = ["mypkg*"] + """ + + def layout(root): + mypkg = root / "mypkg" + mypkg.mkdir() + mypkg.joinpath("__init__.py").write_text("", encoding="utf-8") + + with _project_dir(_minimal_pyproject(extra), layout=layout): + _run_setup_script() + expected_packages = find_packages() + + mock_setup.assert_called_once() + self.assertEqual( + mock_setup.call_args[1]["packages"], + expected_packages, + ) + + @patch("sys.exit") + @patch("setuptools.setup") + def test_autofind_flat_layout_ignores_include_filter(self, mock_setup, mock_exit): + extra = """ + [tool.setuptools.packages.find] + include = ["mypkg.*"] + """ + + def layout(root): + mypkg = root / "mypkg" + mypkg.mkdir() + mypkg.joinpath("__init__.py").write_text("", encoding="utf-8") + other = root / "other" + other.mkdir() + other.joinpath("__init__.py").write_text("", encoding="utf-8") + + with _project_dir(_minimal_pyproject(extra), layout=layout): + _run_setup_script() + expected_packages = find_packages() + + mock_setup.assert_called_once() + self.assertEqual( + mock_setup.call_args[1]["packages"], + expected_packages, + ) + self.assertEqual(sorted(expected_packages), ["mypkg", "other"]) + + @patch("sys.exit") + @patch("setuptools.setup") + def test_autofind_packages_src_layout_where_only(self, mock_setup, mock_exit): + extra = """ + [tool.setuptools.packages.find] + where = ["src"] + """ + + def layout(root): + mypkg = root / "src" / "mypkg" + mypkg.mkdir(parents=True) + mypkg.joinpath("__init__.py").write_text("", encoding="utf-8") + + with _project_dir(_minimal_pyproject(extra), layout=layout): + _run_setup_script() + expected_packages = find_packages(where="src") + + mock_setup.assert_called_once() + kwargs = mock_setup.call_args[1] + self.assertEqual(kwargs["packages"], expected_packages) + self.assertEqual(kwargs["package_dir"], {"": "src"}) + + @patch("sys.exit") + @patch("setuptools.setup") + def test_namespace_packages_with_where(self, mock_setup, mock_exit): + extra = """ + [tool.setuptools.packages.find] + where = ["src"] + include = ["myapp.core*"] + """ + + def layout(root): + myapp_core = root / "src" / "myapp" / "core" + myapp_core.mkdir(parents=True) + myapp_core.joinpath("__init__.py").write_text("", encoding="utf-8") + + with _project_dir(_minimal_pyproject(extra), layout=layout): + _run_setup_script() + + mock_setup.assert_called_once() + kwargs = mock_setup.call_args[1] + self.assertEqual(kwargs["packages"], ["myapp.core"]) + self.assertEqual(kwargs["package_dir"], {"": "src"}) + + @patch("sys.exit") + @patch("setuptools.setup") + def test_autofind_packages_src_layout_with_include(self, mock_setup, mock_exit): + extra = """ + [tool.setuptools.packages.find] + where = ["src"] + include = ["mypkg.*"] + """ + + def layout(root): + mypkg = root / "src" / "mypkg" + mypkg.mkdir(parents=True) + mypkg.joinpath("__init__.py").write_text("", encoding="utf-8") + + with _project_dir(_minimal_pyproject(extra), layout=layout): + _run_setup_script() + expected_packages = find_packages( + where="src", + include=("mypkg.*",), + exclude=(), + ) + + mock_setup.assert_called_once() + kwargs = mock_setup.call_args[1] + self.assertEqual(kwargs["packages"], expected_packages) + self.assertEqual(kwargs["package_dir"], {"": "src"}) + @patch("sys.exit") @patch("setuptools.setup") def test_explicit_packages_without_find(self, mock_setup, mock_exit): @@ -183,6 +304,123 @@ def test_package_data(self, mock_setup, mock_exit): self.assertEqual(kwargs["package_data"], {"pkg.data": ["*.json"]}) self.assertTrue(kwargs["include_package_data"]) + @patch("sys.exit") + @patch("setuptools.setup") + def test_data_files_literals(self, mock_setup, mock_exit): + extra = """ + [tool.setuptools.data-files] + "slurm-quota/conf" = [ + "conf/serve.yml", + "conf/serve.ini.example", + ] + """ + + def layout(root): + conf = root / "conf" + conf.mkdir() + conf.joinpath("serve.yml").write_text("yml", encoding="utf-8") + conf.joinpath("serve.ini.example").write_text("ini", encoding="utf-8") + + with _project_dir(_minimal_pyproject(extra), layout=layout): + _run_setup_script() + + mock_setup.assert_called_once() + self.assertEqual( + mock_setup.call_args[1]["data_files"], + [ + ( + "slurm-quota/conf", + ["conf/serve.yml", "conf/serve.ini.example"], + ), + ], + ) + + @patch("sys.exit") + @patch("setuptools.setup") + def test_data_files_glob_expansion(self, mock_setup, mock_exit): + extra = """ + [tool.setuptools.data-files] + "slurm-quota/web/templates" = ["web/templates/*"] + "slurm-quota/web/static" = ["web/static/*"] + """ + + def layout(root): + templates = root / "web" / "templates" + templates.mkdir(parents=True) + templates.joinpath("a.html").write_text("html", encoding="utf-8") + static = root / "web" / "static" + static.mkdir(parents=True) + static.joinpath("b.css").write_text("css", encoding="utf-8") + + with _project_dir(_minimal_pyproject(extra), layout=layout): + _run_setup_script() + + mock_setup.assert_called_once() + self.assertEqual( + mock_setup.call_args[1]["data_files"], + [ + ("slurm-quota/web/templates", ["web/templates/a.html"]), + ("slurm-quota/web/static", ["web/static/b.css"]), + ], + ) + + @patch("sys.exit") + @patch("setuptools.setup") + def test_data_files_mixed_literals_and_globs(self, mock_setup, mock_exit): + extra = """ + [tool.setuptools.data-files] + "slurm-quota/conf" = [ + "conf/serve.yml", + "conf/serve.ini.example", + "conf/slurm-quota-web.default", + ] + "slurm-quota/web/wsgi" = ["web/wsgi/slurm-quota-web.wsgi"] + "slurm-quota/web/templates" = ["web/templates/*"] + "slurm-quota/web/static" = ["web/static/*"] + """ + + def layout(root): + conf = root / "conf" + conf.mkdir() + conf.joinpath("serve.yml").write_text("yml", encoding="utf-8") + conf.joinpath("serve.ini.example").write_text("ini", encoding="utf-8") + conf.joinpath("slurm-quota-web.default").write_text( + "default", encoding="utf-8" + ) + wsgi = root / "web" / "wsgi" + wsgi.mkdir(parents=True) + wsgi.joinpath("slurm-quota-web.wsgi").write_text("wsgi", encoding="utf-8") + templates = root / "web" / "templates" + templates.mkdir(parents=True) + templates.joinpath("index.html").write_text("html", encoding="utf-8") + static = root / "web" / "static" + static.mkdir(parents=True) + static.joinpath("app.css").write_text("css", encoding="utf-8") + + with _project_dir(_minimal_pyproject(extra), layout=layout): + _run_setup_script() + + mock_setup.assert_called_once() + self.assertEqual( + mock_setup.call_args[1]["data_files"], + [ + ( + "slurm-quota/conf", + [ + "conf/serve.yml", + "conf/serve.ini.example", + "conf/slurm-quota-web.default", + ], + ), + ( + "slurm-quota/web/wsgi", + ["web/wsgi/slurm-quota-web.wsgi"], + ), + ("slurm-quota/web/templates", ["web/templates/index.html"]), + ("slurm-quota/web/static", ["web/static/app.css"]), + ], + ) + @patch("sys.exit") @patch("setuptools.setup") def test_install_requires(self, mock_setup, mock_exit):