diff --git a/.agents/skills/edg-changes-example/SKILL.md b/.agents/skills/edg-changes-example/SKILL.md index 6494d1e2e6..f882316738 100644 --- a/.agents/skills/edg-changes-example/SKILL.md +++ b/.agents/skills/edg-changes-example/SKILL.md @@ -1,14 +1,16 @@ --- name: edg-changes-example description: >- - Encodes and decodes non-UTF-8 example bytes in Changes changelogs with - edg-changes-example («HHHH...» markers) and writes Changes files as UTF-8. - Use when editing any Changes file (src/Changes, util/Changes, …), adding + Writes Changes entries as pending_changes/*.txt files (collected + nightly by edg-collect-changes, checked with edg-collect-changes --check), + and encodes/decodes non-UTF-8 example bytes with edg-changes-example + («HHHH...» markers). Use when adding a Changes entry, editing any Changes + file (src/Changes, util/Changes, …) or pending_changes/ entry, adding examples with Latin-1/EUC-JP/Shift-JIS/other non-UTF-8 bytes, or verifying changelog encoding. --- -# `edg-changes-example` and Changes files +# Changes entries, `edg-collect-changes`, and `edg-changes-example` ## Policy (all `Changes` files) @@ -74,7 +76,50 @@ bytes. No other escapes are interpreted. ## Workflow for a new changelog entry -1. Draft the entry as UTF-8 prose. +New entries are **not** written into a `Changes` file. Each `Changes` file +has a `pending_changes/` directory beside it (`src/pending_changes/`, +`util/pending_changes/`, …); add one entry file there per change, named +`*.txt` (suggested: `gh--.txt`). A nightly job +(`.github/workflows/collect-changes.yml`, midnight US Eastern) runs +`edg-collect-changes`, which moves every entry into its `Changes` file under +the newest `Version x.y` line, adds the date, orders that night's entries by +summary line, and deletes the entry files. + +Entry file format: + +```text +//allowed_words: reflectable memoized +[GH #335] +Type transformations of qualified function types + +Previously, the __add_lvalue_reference, __add_rvalue_reference, ... +For example, with --c++11: + + using F = void() const; + using F = __add_pointer(F); // Previously a spurious error, now okay. +``` + +- Optional leading `//key: value` directives (EDG C++ line directive + syntax). `//allowed_words:` lists space-separated words the spell check + should accept (identifiers, jargon); repeat the line as needed. Words the + target `Changes` file already uses (`constexpr`, `lvalue`, …) are accepted + without it. Directives are dropped when the entry is collected. +- The tag block: one or more comma-separated `GH #N` tags in brackets + (`[GH #12,GH #34]`; only `GH #N`, never `[]`). It may wrap onto several + lines; the tool re-wraps it the way existing entries are wrapped. +- **No date.** The tool adds it (`m/d/yy`, padded to 8 columns) in front of + the tags. +- The summary line, one blank line, then the description, in the same style + as existing entries (unindented prose wrapped at 79 columns, two spaces + after periods, examples indented 2 spaces). +- UTF-8, no tabs, at most **79** columns per line. Trailing whitespace is + tolerated (it's stripped on collection). + +Steps: + +1. Draft the entry as UTF-8 prose in + `/pending_changes/.txt` (the `pending_changes/` beside + the `Changes` file the entry belongs in, usually `src/`). 2. If an example needs non-UTF-8 bytes, author the example in a small raw file (or produce the markers by hand only when the bytes are already known), then run `edg-changes-example encode` and paste the UTF-8 result @@ -83,20 +128,35 @@ bytes. No other escapes are interpreted. `Note: The above example must be decoded using the edg-changes-example tool.` -4. Insert the entry into the appropriate `Changes` file with a **UTF-8** - write (keep the rest of the file valid UTF-8). -5. Verify the file still decodes as UTF-8 (raises on failure): +4. Check the entry (format, spelling, typos, line length). This is the + check `edg-check-file-changes` does for `pending_changes/*.txt` files, + which pull request CI runs and which makes the pre-push hook refuse the + push (no override) if anything is wrong: ```bash - python3 -c "open('path/to/Changes', encoding='utf-8').read()" + edg-exec -- edg-collect-changes --check src/Changes src/pending_changes ``` - Use the real path (for example `src/Changes`). A clean exit means the - whole file is valid UTF-8; a `UnicodeDecodeError` means the write - introduced non-UTF-8 bytes and must be fixed before continuing. + (Name the `Changes` file and `pending_changes/` directory the entry is + in. `edg-exec` runs it in the dev-env container, which has `aspell`.) For + misspellings that are really identifiers or terms, add them to an + `//allowed_words:` line at the very top of the entry, as the failure + message shows; fix real misspellings instead. +5. Optionally preview the result without changing anything: + `edg-collect-changes --dry-run src/Changes src/pending_changes`. + +Edit a `Changes` file directly only to correct an entry that has already +been collected; then write it as **UTF-8** and verify the file still decodes +(raises on failure): + +```bash +python3 -c "open('path/to/Changes', encoding='utf-8').read()" +``` ## Do not +- Add new entries directly to a `Changes` file, or put a date in a + `pending_changes/` entry. - Commit raw Latin-1 / Shift-JIS / etc. bytes inside a `Changes` file. - Invent alternate note wording; match the existing sentence above. - Run `encode` on the entire historical `src/Changes` and rewrite it unless diff --git a/.agents/skills/edg-cpfe-frontend/SKILL.md b/.agents/skills/edg-cpfe-frontend/SKILL.md index efd7da0233..0c16152b9d 100644 --- a/.agents/skills/edg-cpfe-frontend/SKILL.md +++ b/.agents/skills/edg-cpfe-frontend/SKILL.md @@ -45,15 +45,19 @@ Language: `--c++11` … `--c++26`. Emulation: `--gnu=XXYYZZ`, ## Editing `Changes` files -Write every `Changes` file as **UTF-8** (`src/Changes`, `util/Changes`, and -any other `Changes` in the tree). For non-UTF-8 example bytes, `«HHHH…»` -markers, the required note line, encode/decode commands, and verification, -follow skill **edg-changes-example** — do not invent a parallel procedure here. +Add new entries as `pending_changes/*.txt` files beside the `Changes` +file (usually `src/pending_changes/`), not in the `Changes` file itself; a +nightly job collects them with `edg-collect-changes`. Write everything as +**UTF-8**. For the entry format, `//allowed_words:`, `edg-collect-changes +--check`, non-UTF-8 example bytes (`«HHHH…»` markers and the note line), and +verification, follow skill **edg-changes-example** — do not invent a parallel +procedure here. ## Related skills - **edg-cpfe-style** — coding style; [code_style.rst](../../../doc/source/code_style.rst). -- **edg-changes-example** — UTF-8 Changes writes; `edg-changes-example` CLI. +- **edg-changes-example** — Changes entries (`pending_changes/`), UTF-8; + `edg-collect-changes` and `edg-changes-example` CLIs. - **edg-cpfe-build-test** — rebuild, `edg-docker-test`, recording. - **edg-cpfe-bisect** — `edg-docker-test-bisect` between good/bad commits. - Do not invent a separate defect-tracker workflow; use the project's current diff --git a/.git-hooks/pre-push.py b/.git-hooks/pre-push.py index aba11c4a77..27f447753a 100755 --- a/.git-hooks/pre-push.py +++ b/.git-hooks/pre-push.py @@ -42,6 +42,7 @@ import sys import tempfile +import edgchanges import edgutil from pathlib import Path @@ -170,6 +171,11 @@ def confirm_okay(self): print('Too many attempts.') sys.exit(1) + def reject(self): + '''Record a problem that can't be accepted, without asking.''' + print(' Not allowed: CI would reject this; fix it and push again.') + self.errors = 1 + def check_commits(self): '''Walk commits until one already on a remote branch is found. @@ -287,6 +293,23 @@ def check_changed_files(self, first_new_commit): for file_path_str, file_results in results.items(): print(f" Checking file: {file_path_str}") + # A pending Changes entry with any of these problems fails the + # changes-fragments CI check, so there's no accepting them here. + is_changes_entry = edgchanges.is_fragment_path(file_path_str) + confirm_okay = ( + self.reject if is_changes_entry else self.confirm_okay + ) + + format_errors = [ + edgchanges.FormatError(*error) + for error in file_results['format_errors'] + ] + if len(format_errors) != 0: + print(' Badly formatted Changes entry:') + for error in format_errors: + print(f" - {edgchanges.describe_error(file_path_str, error)}") + confirm_okay() + misspellings = file_results['misspellings'] if misspellings is None: print(' Spell check failed:') @@ -297,14 +320,20 @@ def check_changed_files(self, first_new_commit): print(' New misspelled words:') for misspelling in misspellings: print(f" - {misspelling}") - self.confirm_okay() + if is_changes_entry: + hint_lines = edgchanges.allowed_words_hint( + file_path_str, + misspellings + ) + print_indented('\n'.join(hint_lines), ' ') + confirm_okay() typos = file_results['typos'] if len(typos) != 0: print(' New typos:') for typo in typos: print(f" - {typo}") - self.confirm_okay() + confirm_okay() coding_errors = file_results['coding_errors'] if len(coding_errors) != 0: @@ -327,14 +356,9 @@ def check_changed_files(self, first_new_commit): overlong_lines = file_results['overlong_lines'] if len(overlong_lines) != 0: - for line_no, (line_start, lines) in overlong_lines.items(): - print(f" Line {line_no} too long:") - line_end = line_start + (len(lines) - 1) - print(f" {line_start} -> {line_end} ".center(79, '=')) - for line in lines: - print(line, end = '') - print('=' * 79) - self.confirm_okay() + for line in edgutil.format_overlong_lines(overlong_lines, ' '): + print(line) + confirm_okay() def run(self) -> bool: '''Execute the full pre-push check sequence.''' diff --git a/.github/workflows/build-and-test-ci.yml b/.github/workflows/build-and-test-ci.yml index 3d5dbdb9f9..f7b65a2405 100644 --- a/.github/workflows/build-and-test-ci.yml +++ b/.github/workflows/build-and-test-ci.yml @@ -30,6 +30,10 @@ on: push: branches: - main + # The nightly collect-changes job's commits only move Changes entries. + paths-ignore: + - '**/Changes' + - '**/pending_changes/**' pull_request: jobs: diff --git a/.github/workflows/changes-fragments.yml b/.github/workflows/changes-fragments.yml new file mode 100644 index 0000000000..a1fef0c2ed --- /dev/null +++ b/.github/workflows/changes-fragments.yml @@ -0,0 +1,69 @@ +# Part of the EDG Compiler Project, under the Apache License v2.0 with LLVM +# Exceptions. +# See https://edgcpp.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# Check the pending Changes entries (*/pending_changes/*.txt) with +# `edg-collect-changes --check` (format, spelling, typos, line length), so a +# bad entry is caught before the nightly collect-changes job meets it, and run +# the unit tests of the tools involved. + +name: Pending Changes Entries + +on: + push: + branches: + - main + paths: + - '**/pending_changes/**' + - .github/workflows/changes-fragments.yml + - dev_tools/bin/edg-check-file-changes + - dev_tools/bin/edg-collect-changes + - dev_tools/pylibs/edgchanges/** + - dev_tools/pylibs/edgutil/** + - dev_tools/tests/** + pull_request: + paths: + - '**/pending_changes/**' + - .github/workflows/changes-fragments.yml + - dev_tools/bin/edg-check-file-changes + - dev_tools/bin/edg-collect-changes + - dev_tools/pylibs/edgchanges/** + - dev_tools/pylibs/edgutil/** + - dev_tools/tests/** + +permissions: + contents: read + +jobs: + check: + name: check-pending-changes + runs-on: ubuntu-latest + timeout-minutes: 10 + env: + PYTHONPATH: ${{ github.workspace }}/dev_tools/pylibs + # Each holds a Changes file and its pending_changes/ directory (keep in + # step with the list in collect-changes.yml). + CHANGES_DIRS: src util lib_src include_c++ nt_util + steps: + - name: Check out repository + uses: actions/checkout@v7 + + # The same dictionary as the dev-env image + # (dev_tools/docker/dev-env/setup-scripts/install-software.sh). + - name: Install aspell + run: | + sudo apt-get update + sudo apt-get install -y aspell aspell-en + + - name: Run unit tests + run: python3 -m unittest discover -s dev_tools/tests -v + + - name: Check pending Changes entries + run: | + status=0 + for dir in ${CHANGES_DIRS}; do + dev_tools/bin/edg-collect-changes --check \ + "${dir}/Changes" "${dir}/pending_changes" || status=1 + done + exit "${status}" diff --git a/.github/workflows/collect-changes.yml b/.github/workflows/collect-changes.yml new file mode 100644 index 0000000000..46eca99b78 --- /dev/null +++ b/.github/workflows/collect-changes.yml @@ -0,0 +1,135 @@ +# Part of the EDG Compiler Project, under the Apache License v2.0 with LLVM +# Exceptions. +# See https://edgcpp.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# Nightly (midnight US Eastern) collection of the pending Changes entries: +# edg-collect-changes moves every /pending_changes/*.txt entry into +# /Changes, dated with the day that just ended, and this job commits and +# pushes the result to main. The directories are listed in CHANGES_DIRS +# below (keep it in step with the one in changes-fragments.yml). +# +# Malformed entries are left in place and fail the run (after the good ones +# are pushed), so they get noticed. +# +# One-time setup (repository/organization admin): +# +# 1. Create a GitHub App for the organization with only the "Contents: +# Read and write" repository permission, and install it on this +# repository only. +# 2. Add the repository secrets CHANGELOG_APP_CLIENT_ID (the App's client +# ID) and CHANGELOG_APP_PRIVATE_KEY (a private key generated for it). +# 3. Add the App to the bypass list of the ruleset protecting main, as +# "Always allow". (Classic branch protection doesn't document Apps as +# bypass actors; move main to a ruleset.) +# +# If an App isn't an option, a deploy key with write access can be the +# ruleset bypass actor instead: drop the token step and check out with +# `ssh-key: ${{ secrets.CHANGELOG_DEPLOY_KEY }}`. + +name: Collect Changes Entries + +on: + schedule: + - cron: '0 0 * * *' + timezone: 'America/New_York' + workflow_dispatch: + inputs: + date: + description: 'Date for the entries (YYYY-MM-DD; default: yesterday, US Eastern)' + required: false + default: '' + +permissions: + contents: read + +concurrency: + group: collect-changes + cancel-in-progress: false + +jobs: + collect: + name: collect-changes + runs-on: ubuntu-latest + timeout-minutes: 10 + env: + PYTHONPATH: ${{ github.workspace }}/dev_tools/pylibs + # Each holds a Changes file and its pending_changes/ directory. + CHANGES_DIRS: src util lib_src include_c++ nt_util + steps: + - name: Create App token + id: app + uses: actions/create-github-app-token@v3 + with: + client-id: ${{ secrets.CHANGELOG_APP_CLIENT_ID }} + private-key: ${{ secrets.CHANGELOG_APP_PRIVATE_KEY }} + permission-contents: write + + - name: Check out main + uses: actions/checkout@v7 + with: + ref: main + token: ${{ steps.app.outputs.token }} + + - name: Determine the entry date + id: date + env: + INPUT_DATE: ${{ inputs.date }} + run: | + # The day that just ended; a late start doesn't change it. + date="${INPUT_DATE:-$(TZ=America/New_York date -d yesterday +%F)}" + echo "date=${date}" >> "${GITHUB_OUTPUT}" + + - name: Collect pending Changes entries + id: collect + env: + ENTRY_DATE: ${{ steps.date.outputs.date }} + run: | + # A released newest version gets a new section for the version + # src/Changes has open. + status=0 + for dir in ${CHANGES_DIRS}; do + dev_tools/bin/edg-collect-changes \ + --date "${ENTRY_DATE}" --version-from src/Changes \ + "${dir}/Changes" "${dir}/pending_changes" || status=1 + done + echo "status=${status}" >> "${GITHUB_OUTPUT}" + + - name: Commit and push + env: + ENTRY_DATE: ${{ steps.date.outputs.date }} + APP_SLUG: ${{ steps.app.outputs.app-slug }} + GH_TOKEN: ${{ steps.app.outputs.token }} + run: | + # Stage the updated Changes files and the deleted entries. + for dir in ${CHANGES_DIRS}; do + git add -- "${dir}/Changes" "${dir}/pending_changes" + done + if git diff --cached --quiet; then + echo 'No pending Changes entries.' + exit 0 + fi + + bot_user_id="$(gh api "/users/${APP_SLUG}%5Bbot%5D" --jq .id)" + git config user.name "${APP_SLUG}[bot]" + git config user.email \ + "${bot_user_id}+${APP_SLUG}[bot]@users.noreply.github.com" + git commit -m "Collect pending Changes entries for ${ENTRY_DATE} []" + + # Retry if main moved on while this ran; a real conflict (e.g., a + # hand edit of a Changes file) stops the rebase and fails the job. + for attempt in 1 2 3; do + if git push origin HEAD:main; then + exit 0 + fi + git pull --rebase origin main + done + echo 'Could not push to main.' >&2 + exit 1 + + - name: Report malformed entries + if: steps.collect.outputs.status != '0' + run: | + echo 'edg-collect-changes left malformed entries in place; see the' \ + '"Collect pending Changes entries" step.' >&2 + exit 1 diff --git a/.github/workflows/strict-build-check.yml b/.github/workflows/strict-build-check.yml index 04817a2dba..019622ce4f 100644 --- a/.github/workflows/strict-build-check.yml +++ b/.github/workflows/strict-build-check.yml @@ -12,6 +12,10 @@ on: push: branches: - main + # The nightly collect-changes job's commits only move Changes entries. + paths-ignore: + - '**/Changes' + - '**/pending_changes/**' pull_request: jobs: diff --git a/.github/workflows/typecheck-python.yml b/.github/workflows/typecheck-python.yml index 263e645543..5c5820462b 100644 --- a/.github/workflows/typecheck-python.yml +++ b/.github/workflows/typecheck-python.yml @@ -14,6 +14,10 @@ on: push: branches: - main + # The nightly collect-changes job's commits only move Changes entries. + paths-ignore: + - '**/Changes' + - '**/pending_changes/**' pull_request: jobs: diff --git a/AGENTS.md b/AGENTS.md index 170bb7a721..c21ffc886b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -92,9 +92,21 @@ Hard always-on reminders (details in the RST / skill): ## Editing `Changes` files -Any `Changes` changelog in the tree (not only `src/Changes`) must be written -and kept as **valid UTF-8**. Prefer UTF-8 writes when adding or editing -entries. +New changelog entries go in `pending_changes/*.txt` files beside the +`Changes` file (e.g. `src/pending_changes/gh-123-short-slug.txt`), not +in the `Changes` file itself: tag block (`[GH #N]`, no date), summary line, +blank line, description; optional leading `//allowed_words: …` lines accept +words the spell check rejects (words already in the target `Changes` file +pass without them). A nightly job runs `edg-collect-changes` to +date them and move them into `Changes`. Check entries with +`edg-exec -- edg-collect-changes --check src/Changes src/pending_changes` +(for the directory in question). Edit a `Changes` file directly +only to correct an already-collected entry. Details: skill +**edg-changes-example**. + +Any `Changes` changelog in the tree (not only `src/Changes`) and every +pending entry must be written and kept as **valid UTF-8**. Prefer UTF-8 +writes when adding or editing entries. Examples that need non-UTF-8 bytes must use `edg-changes-example` (`«HHHH…»` markers) and the standard note line — see skill **edg-changes-example**. After @@ -136,6 +148,6 @@ If an agent does not auto-load skills, open the matching | `edg-cpfe-frontend` | [`.agents/skills/edg-cpfe-frontend/SKILL.md`](.agents/skills/edg-cpfe-frontend/SKILL.md) | Editing `src/`, IL/templates/overload, `Changes` | | `edg-cpfe-style` | [`.agents/skills/edg-cpfe-style/SKILL.md`](.agents/skills/edg-cpfe-style/SKILL.md) | Front-end coding style (`doc/source/code_style.rst`) | | `edg-python-style` | [`.agents/skills/edg-python-style/SKILL.md`](.agents/skills/edg-python-style/SKILL.md) | Python under `dev_tools/` (no formal RST; match in-tree) | -| `edg-changes-example` | [`.agents/skills/edg-changes-example/SKILL.md`](.agents/skills/edg-changes-example/SKILL.md) | Encoding non-UTF-8 examples in any `Changes` file | +| `edg-changes-example` | [`.agents/skills/edg-changes-example/SKILL.md`](.agents/skills/edg-changes-example/SKILL.md) | Adding a Changes entry (`pending_changes/`); encoding non-UTF-8 examples in any `Changes` file | | `acknowledg-client` | [`.agents/skills/acknowledg-client/SKILL.md`](.agents/skills/acknowledg-client/SKILL.md) | Batch/Direct client, reconnect, provider CLIs | | `acknowledg-bench` | [`.agents/skills/acknowledg-bench/SKILL.md`](.agents/skills/acknowledg-bench/SKILL.md) | `.cgout` / Ir / baselines / bench review | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 91d0753e5f..53ddcf7e6b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -104,8 +104,9 @@ implementation characeristics). - [ ] You've put effort into maintaining code quality: - [ ] Your code is formatted in "EDG Style". - [ ] Your code has good comments. - - [ ] Your code has a Changes entry (not needed for `dev_tools` or build - system changes). + - [ ] Your code has a [Changes entry](#changes-entries) in + `pending_changes/` (not needed for `dev_tools` or build system + changes). - [ ] You used `edg-docker-test`: - [ ] You have added tests (or there are existing tests) that FAIL before your change but PASS after your change. @@ -114,6 +115,31 @@ implementation characeristics). - [ ] You've updated the default recordings after reviewing differences with `edg-docker-test -W`. +### Changes Entries + +The project includes a number of `Changes` files (effectively changelog/NEWS +files). Due to the number of concurrent changes, it has become impractical to +directly maintain the `Changes` files (particularly `src/Changes`). + +Instead, there is a nightly job that updates the `Changes` files for all the +work that happened over the last 24-hours. To take advantage, add a file +named `.txt` (e.g., `gh-335-qualified-function-types.txt`) +to the `pending_changes/` directory beside the `Changes` file your work belongs +with: + +```text +[GH #335] +Type transformations of qualified function types + +Previously, the __add_lvalue_reference, __add_rvalue_reference, __add_pointer, +and __decay intrinsics produced incorrect types for function types with cv- or +ref-qualifiers. They now leave such types unchanged. For example, with +--c++11: + + using F = void() const; + using F = __add_pointer(F); // Previously a spurious error, now okay. +``` + ### Early Design Feedback Please use the open issue for discussion about the proposed design if diff --git a/dev_tools/bin/edg-check-file-changes b/dev_tools/bin/edg-check-file-changes index 131d5fa5f4..bd022df514 100755 --- a/dev_tools/bin/edg-check-file-changes +++ b/dev_tools/bin/edg-check-file-changes @@ -7,6 +7,7 @@ import argparse import contextlib +import functools import re import json import shutil @@ -19,6 +20,7 @@ from concurrent.futures import ThreadPoolExecutor from pathlib import Path from typing import Any, List, Optional, Set, Tuple +import edgchanges import edgutil TOOL_PATH_ASPELL = shutil.which('aspell') @@ -80,6 +82,9 @@ def find_the_the(path): with open(path, 'r', errors = 'replace') as handle: lines = handle.read().splitlines() + return _find_the_the_in_lines(lines) + +def _find_the_the_in_lines(lines: List[str]) -> List[str]: matches = [] for regex, excludes in _FIND_THE_THE_CHECKS_COMPILED: # grep -w: match must be bounded by non-word characters (or line edges). @@ -139,6 +144,40 @@ def check_spelling(old_file: Path, new_file: Path) -> Optional[List[str]]: return sorted(new_mispelled_words - old_mispelled_words) +@functools.lru_cache(maxsize = None) +def _changes_file_misspellings(changes_path: Path) -> Set[str]: + '''Return the words aspell rejects in changes_path (empty if there's no + such file).''' + if not changes_path.is_file(): + return set() + return _run_spell_command(changes_path) + +def _check_fragment_spelling(repo_path: str, + new_file: Path) -> Optional[List[str]]: + '''Report misspellings in a Changes entry. + + The directives and the tag block aren't checked, and words the entry's + //allowed_words: directives list aren't reported. Neither are words + already in the Changes file the entry will be collected into (so a term + like "constexpr" needn't be allowed in every entry). + ''' + if not _is_aspell_working(): + return None + + allowed_words, prose_lines = edgchanges.read_prose(new_file) + with tempfile.TemporaryDirectory() as tmp_dir_str: + text_path = Path(tmp_dir_str) / 'prose.txt' + with open(text_path, 'w', encoding = 'utf-8') as handle: + handle.write('\n'.join(prose_lines) + '\n') + mispelled_words = _run_spell_command(text_path) + + # Accept the words the entry allows and those its Changes file already + # uses. + target_path = edgchanges.target_changes_path(Path.cwd() / repo_path) + accepted_words = allowed_words | _changes_file_misspellings(target_path) + + return sorted(mispelled_words - accepted_words) + def _file_check_config(file_path: Path): '''Return spelling/coding/line-length check settings for a path.''' suffix = file_path.suffix @@ -149,7 +188,7 @@ def _file_check_config(file_path: Path): if suffix in CODE_FILE_EXTENSIONS: check_for_coding_errors = True check_for_overlong_lines = True - elif suffix == 'txt': + elif suffix == '.txt': pass else: check_for_overlong_lines = True @@ -184,14 +223,18 @@ def _check_coding_errors(new_file: Path) -> Any: return coding_errors -def _check_overlong_lines(new_file: Path) -> Any: +def _check_overlong_lines(new_file: Path, + ignore_trailing_whitespace: bool = False) -> Any: '''Report lines longer than 79 columns after tab expansion.''' overlong_lines = {} with open(new_file, 'r', errors = 'replace') as file_handle: lines = file_handle.readlines() for line_no, line in enumerate(lines, start = 1): - if len(line.rstrip('\n').expandtabs()) <= 79: + checked_line = ( + line.rstrip() if ignore_trailing_whitespace else line.rstrip('\n') + ) + if len(checked_line.expandtabs()) <= 79: continue overlong_lines[line_no] = [ @@ -201,11 +244,41 @@ def _check_overlong_lines(new_file: Path) -> Any: return overlong_lines -def check_file(old_file: Path, new_file: Path) -> Any: +def check_changes_fragment(repo_path: str, new_file: Path) -> Any: + '''Check new_file, the contents of the pending Changes entry repo_path. + + Besides the usual checks, its format is checked (see edgchanges). Its + spelling is checked against the Changes file it will be collected into, not + against an old version of the entry (see _check_fragment_spelling), and + trailing whitespace (stripped when the entry is collected) doesn't count + toward the line length. + ''' + _, format_errors = edgchanges.parse_fragment( + new_file, + check_line_length = False + ) + _, prose_lines = edgchanges.read_prose(new_file) + return { + 'format_errors': format_errors, + 'misspellings': _check_fragment_spelling(repo_path, new_file), + 'typos': _find_the_the_in_lines(prose_lines), + 'coding_errors': {}, + 'overlong_lines': _check_overlong_lines( + new_file, + ignore_trailing_whitespace = True + ) + } + +def check_file(repo_path: str, old_file: Path, new_file: Path) -> Any: + '''Check new_file, the new contents of repo_path, against old_file.''' + if edgchanges.is_fragment_path(repo_path): + return check_changes_fragment(repo_path, new_file) + check_for_coding_errors, check_for_overlong_lines = ( _file_check_config(new_file) ) return { + 'format_errors': [], 'misspellings': check_spelling(old_file, new_file), 'typos': find_the_the(new_file), 'coding_errors': ( @@ -237,11 +310,15 @@ def parse_args() -> argparse.Namespace: def main() -> None: args = parse_args() - TOOL_PATH_GET_COMMENTS.exit_if_not_found() - with open(args.input_spec, 'r', encoding = 'utf-8') as file_handle: input_dict = json.load(file_handle) + # Only code files need get-comments, so checking just Changes entries (as + # CI does) works without the built tools. + if any(Path(new_file).suffix in CODE_FILE_EXTENSIONS + for _, new_file in input_dict.values()): + TOOL_PATH_GET_COMMENTS.exit_if_not_found() + output_dict = dict.fromkeys(input_dict) with ThreadPoolExecutor() as executor: @@ -251,6 +328,7 @@ def main() -> None: for key, (old_file, new_file) in input_dict.items(): output_dict[key] = executor.submit( check_file, + key, cwd.joinpath(old_file), cwd.joinpath(new_file) ) diff --git a/dev_tools/bin/edg-collect-changes b/dev_tools/bin/edg-collect-changes new file mode 100755 index 0000000000..c6a8d107f6 --- /dev/null +++ b/dev_tools/bin/edg-collect-changes @@ -0,0 +1,401 @@ +#!/usr/bin/env python3 + +# Part of the EDG Compiler Project, under the Apache License v2.0 with LLVM +# Exceptions. +# See https://edgcpp.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +'''Collect pending Changes entries into a Changes file. + +Each Changes file in the tree has a pending_changes/ directory beside it. +Rather than editing a Changes file directly (which makes every pull request +conflict with every other), a change adds one *.txt entry there: + + //allowed_words: reflectable memoized + [GH #335] + Type transformations of qualified function types + + Previously, ... + +The leading //key: value directives are optional (//allowed_words: lists +words the spell check should accept). Then comes the tag block, one or more +comma-separated "GH #N" tags in brackets (it may wrap onto several lines), the +summary line, a blank line, and the description (see edgchanges). + +Given a Changes file and its pending_changes/ directory, this tool moves +every entry into the Changes file, just below the newest "Version x.y" line, +with the date added in front of the tags. The entries added in one run are +ordered by their summary lines. If the newest version has already been +released (its header has a date), a new section is opened for the next +version (--version or --version-from). A nightly job runs it for each Changes +file (see .github/workflows/collect-changes.yml). + +With --check, the entries are only checked, by running edg-check-file-changes +on them (format, spelling, typos, line length), the same check the pre-push +hook does. +''' + +import argparse +import datetime +import json +import re +import subprocess +import sys +import tempfile + +import edgchanges +import edgutil + +from pathlib import Path +from typing import Any, List, Optional, Tuple + +PENDING_CHANGES_KEEP_FILE_NAME = '.gitignore' + +SEPARATOR_LINE = '-' * edgchanges.MAX_LINE_LENGTH +NO_CHANGES_LINE = 'No changes.' + +# An open version is "Version 7.1"; a released one is +# "Version 7.0, September 28, 2026". +VERSION_HEADER_REGEX = re.compile( + r'^Version ([0-9]+\.[0-9]+)(, [A-Z][a-z]+ [0-9]{1,2}, [0-9]{4})?$' +) +SEPARATOR_REGEX = re.compile(r'^-+$') + +def list_pending_dir(pending_dir: Path) -> Tuple[List[Path], List[Path]]: + '''Return the entries in pending_dir, and the other files there (apart + from the .gitignore that keeps the directory in git).''' + fragment_paths = [] + other_paths = [] + for path in sorted(pending_dir.iterdir()): + if edgchanges.is_fragment_path(str(path)): + fragment_paths.append(path) + elif path.name != PENDING_CHANGES_KEEP_FILE_NAME: + other_paths.append(path) + return fragment_paths, other_paths + +def _describe_misnamed(path: Path) -> str: + return ( + f"{path}: not a pending Changes entry (entries are named " + f"*{edgchanges.FRAGMENT_SUFFIX})" + ) + +def _find_newest_version_header(lines: List[str]) -> Optional[int]: + '''Return the index of the newest Version header line. + + Body text can also start with "Version", so a header must be the first line + of the file or directly follow a separator line. + ''' + for line_index, line in enumerate(lines): + follows_separator = ( + line_index == 0 or + SEPARATOR_REGEX.match(lines[line_index - 1]) is not None + ) + if follows_separator and VERSION_HEADER_REGEX.match(line) is not None: + return line_index + return None + +def read_open_version(changes_path: Path) -> Optional[str]: + '''Return the newest version of changes_path if it's still open (i.e., its + header has no release date).''' + with open(changes_path, 'r', encoding = 'utf-8') as file_handle: + lines = file_handle.read().split('\n') + + header_index = _find_newest_version_header(lines) + if header_index is None: + return None + header_match = VERSION_HEADER_REGEX.match(lines[header_index]) + assert header_match is not None + if header_match.group(2) is not None: + return None + return header_match.group(1) + +def insert_entries(changes_text: str, + entries: List[edgchanges.ChangesEntry], date_text: str, + next_version: Optional[str]) -> Tuple[str, List[str]]: + '''Insert entries under the newest version of changes_text. + + The entries are added just below the Version line, above the entries + already there. If the newest version is released, a new section for + next_version is opened above it instead. + + Returns the new text and the lines of the section as inserted. Raises + ValueError if there's nowhere to put the entries. + ''' + lines = changes_text.split('\n') + header_index = _find_newest_version_header(lines) + if header_index is None: + raise ValueError('no "Version x.y" line found') + header_match = VERSION_HEADER_REGEX.match(lines[header_index]) + assert header_match is not None + + entry_lines: List[str] = [] + for entry in sorted(entries, key = edgchanges.ChangesEntry.sort_key): + entry_lines += entry.format(date_text) + ['', ''] + + if header_match.group(2) is None: + rest_index = header_index + 1 + if rest_index < len(lines) and lines[rest_index] == '': + rest_index += 1 + if rest_index < len(lines) and lines[rest_index] == NO_CHANGES_LINE: + rest_index += 1 + while rest_index < len(lines) and lines[rest_index] == '': + rest_index += 1 + inserted_lines = [lines[header_index], ''] + entry_lines + new_lines = lines[:header_index] + inserted_lines + lines[rest_index:] + else: + if next_version is None: + raise ValueError( + f"the newest version ({header_match.group(1)}) is released; use " + '--version or --version-from to open a new one' + ) + inserted_lines = ( + [f"Version {next_version}", ''] + entry_lines + [SEPARATOR_LINE] + ) + new_lines = ( + lines[:header_index] + inserted_lines + lines[header_index:] + ) + + return '\n'.join(new_lines), inserted_lines + +def _run_file_checks(fragment_paths: List[Path]) -> Any: + '''Run edg-check-file-changes on fragment_paths and return its results. + + The checker wants an old version of each file too; an entry's is ignored, + so an empty file stands in. + ''' + check_tool_path = Path(__file__).resolve().parent / 'edg-check-file-changes' + with tempfile.TemporaryDirectory() as tmp_dir_str: + tmp_dir = Path(tmp_dir_str) + empty_file = tmp_dir / 'empty.txt' + empty_file.touch() + input_spec = { + str(path): [str(empty_file), str(path.resolve())] + for path in fragment_paths + } + input_file = tmp_dir / 'input.json' + output_file = tmp_dir / 'output.json' + with open(input_file, 'w', encoding = 'utf-8') as file_handle: + json.dump(input_spec, file_handle) + + # The checker finds each entry's Changes file (for the spell check) + # from its key, relative to the working directory, as here. + completed_process = subprocess.run( + [sys.executable, str(check_tool_path), str(input_file), str(output_file)] + ) + if completed_process.returncode != 0: + raise RuntimeError( + f"edg-check-file-changes failed (exit {completed_process.returncode})" + ) + with open(output_file, 'r', encoding = 'utf-8') as file_handle: + return json.load(file_handle) + +def _report_file_check_results(display_path: str, results: Any) -> bool: + '''Print the problems edg-check-file-changes found; return True if there + were any.''' + report_lines = [] + for line_no, message in results['format_errors']: + report_lines.append( + f" {message}" if line_no is None else f" line {line_no}: {message}" + ) + + misspellings = results['misspellings'] + if misspellings is None: + report_lines.append( + ' spell check failed: aspell not found or missing dictionary' + ) + elif len(misspellings) != 0: + report_lines.append(' misspelled words:') + report_lines += [f" - {word}" for word in misspellings] + report_lines += [ + f" {line}" + for line in edgchanges.allowed_words_hint(display_path, misspellings) + ] + + if len(results['typos']) != 0: + report_lines.append(' typos:') + report_lines += [f" - {typo}" for typo in results['typos']] + + report_lines += edgutil.format_overlong_lines( + results['overlong_lines'], + ' ' + ) + + if len(report_lines) != 0: + print(f"{display_path}:") + for line in report_lines: + print(line) + return len(report_lines) != 0 + +def check(pending_dir: Path) -> bool: + '''Check every entry in pending_dir with edg-check-file-changes; return + True if all are good.''' + fragment_paths, other_paths = list_pending_dir(pending_dir) + for path in other_paths: + print(_describe_misnamed(path)) + if len(fragment_paths) == 0: + return len(other_paths) == 0 + + all_results = _run_file_checks(fragment_paths) + found_problems = len(other_paths) != 0 + for path in fragment_paths: + # edg-check-file-changes leaves out a file it failed on (with a message). + if str(path) not in all_results: + print(f"{path}: could not be checked") + found_problems = True + elif _report_file_check_results(str(path), all_results[str(path)]): + found_problems = True + + return not found_problems + +def collect(changes_path: Path, pending_dir: Path, date: datetime.date, + next_version: Optional[str], *, dry_run: bool) -> bool: + '''Move the entries in pending_dir into changes_path. + + Malformed entries are reported and left in place; the others are still + collected. Returns True if every entry was collected. Raises ValueError + if changes_path has nowhere to put them. + ''' + fragment_paths, other_paths = list_pending_dir(pending_dir) + for path in other_paths: + edgutil.eprint(f"warning: {_describe_misnamed(path)}; ignored") + + entries = [] + errors = [] + for path in fragment_paths: + entry, format_errors = edgchanges.parse_fragment(path) + errors += [ + edgchanges.describe_error(str(path), error) for error in format_errors + ] + if entry is not None: + entries.append(entry) + for error in errors: + edgutil.eprint(error) + if len(errors) != 0: + edgutil.eprint('Malformed entries were left in place.') + if len(entries) == 0: + return len(errors) == 0 + + with open(changes_path, 'r', encoding = 'utf-8') as file_handle: + changes_text = file_handle.read() + try: + new_text, inserted_lines = insert_entries( + changes_text, + entries, + edgchanges.format_date(date), + next_version + ) + except ValueError as ex: + raise ValueError(f"{changes_path}: {ex}") + + if dry_run: + print(f"=== {changes_path}") + print('\n'.join(inserted_lines)) + else: + # Only drop the entries once they're safely in the Changes file. + with open(changes_path, 'w', encoding = 'utf-8', + newline = '\n') as file_handle: + file_handle.write(new_text) + for entry in entries: + entry.path.unlink() + print( + f"{changes_path}: collected {len(entries)} " + f"{'entry' if len(entries) == 1 else 'entries'}" + ) + + return len(errors) == 0 + +def _parse_date(date_text: str) -> datetime.date: + try: + return datetime.datetime.strptime(date_text, '%Y-%m-%d').date() + except ValueError: + raise argparse.ArgumentTypeError(f"not a YYYY-MM-DD date: {date_text}") + +def _parse_version(version_text: str) -> str: + if re.match(r'^[0-9]+\.[0-9]+$', version_text) is None: + raise argparse.ArgumentTypeError(f"not an x.y version: {version_text}") + return version_text + +def _parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description = ( + 'Move the pending entries in a pending_changes/ directory into a ' + 'Changes file.' + ) + ) + parser.add_argument( + 'changes_file', + type = Path, + help = 'the Changes file (e.g. src/Changes)' + ) + parser.add_argument( + 'pending_dir', + type = Path, + help = 'its pending entries directory (e.g. src/pending_changes)' + ) + parser.add_argument( + '--date', + type = _parse_date, + default = datetime.date.today(), + help = 'the date (YYYY-MM-DD) to give the entries (default: today)' + ) + version_group = parser.add_mutually_exclusive_group() + version_group.add_argument( + '--version', + type = _parse_version, + help = ( + 'the version (x.y) of the section to open if the newest version of ' + 'the Changes file is released' + ) + ) + version_group.add_argument( + '--version-from', + type = Path, + metavar = 'CHANGES_FILE', + help = ( + 'like --version, but take the version from the newest (open) version ' + 'of another Changes file' + ) + ) + mode_group = parser.add_mutually_exclusive_group() + mode_group.add_argument( + '--check', + action = 'store_true', + help = 'only check the pending entries (format, spelling, line length)' + ) + mode_group.add_argument( + '--dry-run', + action = 'store_true', + help = 'print the section that would be inserted; change nothing' + ) + return parser.parse_args() + +def main() -> int: + args = _parse_args() + + if args.check: + return 0 if check(args.pending_dir) else 1 + + next_version = args.version + if args.version_from is not None: + next_version = read_open_version(args.version_from) + if next_version is None: + edgutil.eprint( + f"error: {args.version_from}: the newest version is released" + ) + return 1 + + try: + succeeded = collect( + args.changes_file, + args.pending_dir, + args.date, + next_version, + dry_run = args.dry_run + ) + except ValueError as ex: + edgutil.eprint(f"error: {ex}") + return 1 + return 0 if succeeded else 1 + +if __name__ == '__main__': + sys.exit(main()) diff --git a/dev_tools/pylibs/edgchanges/__init__.py b/dev_tools/pylibs/edgchanges/__init__.py new file mode 100644 index 0000000000..154dd39aec --- /dev/null +++ b/dev_tools/pylibs/edgchanges/__init__.py @@ -0,0 +1,259 @@ +# Part of the EDG Compiler Project, under the Apache License v2.0 with LLVM +# Exceptions. +# See https://edgcpp.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +'''Pending Changes entries: the *.txt files in each pending_changes/ +directory, which edg-collect-changes moves into the Changes file beside it +(the nightly collect-changes workflow lists the directories). + +An entry looks like: + + //allowed_words: reflectable memoized + [GH #335] + Type transformations of qualified function types + + Previously, ... + +The leading //key: value directives (EDG C++ line directive syntax) are +optional; //allowed_words: lists words the spell check should accept (beyond +those the target Changes file already uses, which it accepts anyway). Then +comes the tag block, one or more comma-separated "GH #N" tags in brackets (it +may wrap onto several lines), the summary line, a blank line, and the +description. Lines are at most MAX_LINE_LENGTH columns, not counting trailing +whitespace (which is stripped when the entry is collected). + +The format is checked by edg-check-file-changes (which the pre-push hook and +edg-collect-changes --check run) and relied on by edg-collect-changes. +''' + +import datetime +import re + +import edgutil + +from pathlib import Path +from typing import Iterable, List, NamedTuple, Optional, Set, Tuple + +FRAGMENT_SUFFIX = '.txt' +ALLOWED_WORDS_DIRECTIVE = 'allowed_words' +PENDING_CHANGES_DIR_NAME = 'pending_changes' +CHANGES_FILE_NAME = 'Changes' + +MAX_LINE_LENGTH = 79 +# The date ("10/21/26") is left-justified in this many columns before the +# tags. +DATE_WIDTH = 8 + +TAG_REGEX = re.compile(r'^GH #[0-9]+$') + +# A problem with an entry's format; line_no is None for the whole file. +FormatError = NamedTuple( + 'FormatError', + [('line_no', Optional[int]), ('message', str)] +) + +def is_fragment_path(path_str: str) -> bool: + '''Return True if path_str names a pending Changes entry: a *.txt file in a + pending_changes/ directory.''' + path = Path(path_str) + return (path.suffix == FRAGMENT_SUFFIX and + path.parent.name == PENDING_CHANGES_DIR_NAME) + +def target_changes_path(fragment_path: Path) -> Path: + '''Return the Changes file the entry at fragment_path will be collected + into (the one beside its pending_changes/ directory).''' + return fragment_path.parent.parent / CHANGES_FILE_NAME + +def describe_error(display_path: str, error: FormatError) -> str: + '''Return error as a "path:line: message" string.''' + if error.line_no is None: + return f"{display_path}: {error.message}" + return f"{display_path}:{error.line_no}: {error.message}" + +def allowed_words_hint(display_path: str, words: Iterable[str]) -> List[str]: + '''Return the lines telling the user how to accept words an entry's spell + check rejected.''' + return [ + 'If these words are correct, accept them by adding this line at the very', + f"top of {display_path}:", + f" //{ALLOWED_WORDS_DIRECTIVE}: {' '.join(words)}" + ] + +def format_date(date: datetime.date) -> str: + '''Format date as m/d/yy, the format used throughout the Changes files.''' + return f"{date.month}/{date.day}/{date.year % 100:02d}" + +def format_tag_lines(date_text: str, tags: List[str]) -> List[str]: + '''Return the first line(s) of an entry: the date padded to DATE_WIDTH + columns, a space, and the bracketed tags. + + Tags are packed onto each line as long as they fit in MAX_LINE_LENGTH + columns; continuation lines line the tags up under the first one. + ''' + first_line_prefix = f"{date_text:<{DATE_WIDTH}} [" + continuation_prefix = ' ' * len(first_line_prefix) + + lines = [] + current_line = first_line_prefix + line_has_tag = False + for tag_index, tag in enumerate(tags): + is_last_tag = tag_index == len(tags) - 1 + piece = tag + (']' if is_last_tag else ',') + if line_has_tag and len(current_line) + len(piece) > MAX_LINE_LENGTH: + lines.append(current_line) + current_line = continuation_prefix + current_line += piece + line_has_tag = True + lines.append(current_line) + + return lines + +class ChangesEntry: + '''A parsed pending Changes entry.''' + + def __init__(self, path: Path, tags: List[str], summary: str, + body_lines: List[str]) -> None: + self.path = path + self.tags = tags + self.summary = summary + self.body_lines = body_lines + + def sort_key(self) -> Tuple[str, str, str]: + return (self.summary.casefold(), self.summary, self.path.name) + + def format(self, date_text: str) -> List[str]: + '''Return the lines of this entry as it appears in a Changes file.''' + return ( + format_tag_lines(date_text, self.tags) + + [self.summary, ''] + + self.body_lines + ) + +def _find_tag_block_end(lines: List[str]) -> Optional[int]: + '''Return the index of the line closing the tag block that starts lines.''' + return next( + (index for index, line in enumerate(lines) if ']' in line), + None + ) + +def parse_fragment(path: Path, *, check_line_length: bool = True + ) -> Tuple[Optional[ChangesEntry], List[FormatError]]: + '''Parse the pending Changes entry at path. + + Returns the entry (None if it's malformed) and the format errors found. + Line lengths are checked only if check_line_length (edg-check-file-changes + has its own check); trailing whitespace never counts. + ''' + # The lines keep their newlines; everything below strips them anyway. + try: + with open(path, 'r', encoding = 'utf-8') as handle: + lines = handle.readlines() + except UnicodeDecodeError as ex: + return None, [FormatError(None, f"not valid UTF-8 ({ex})")] + + errors = [] + for line_no, line in enumerate(lines, start = 1): + if '\t' in line: + errors.append(FormatError(line_no, 'tab character; use spaces')) + if check_line_length and len(line.rstrip()) > MAX_LINE_LENGTH: + errors.append( + FormatError(line_no, f"line longer than {MAX_LINE_LENGTH} columns") + ) + + _, remaining_lines = edgutil.split_leading_cpp_line_directives(lines) + first_line_no = len(lines) - len(remaining_lines) + 1 + remaining_lines = [line.rstrip() for line in remaining_lines] + while remaining_lines and remaining_lines[-1] == '': + remaining_lines.pop() + + if not remaining_lines or not remaining_lines[0].startswith('['): + errors.append( + FormatError(first_line_no, "expected the tag block, e.g. '[GH #123]'") + ) + return None, errors + + close_index = _find_tag_block_end(remaining_lines) + if close_index is None: + errors.append(FormatError(first_line_no, "tag block has no closing ']'")) + return None, errors + if not remaining_lines[close_index].endswith(']'): + errors.append( + FormatError( + first_line_no + close_index, + "text after the tag block's ']'" + ) + ) + return None, errors + + tag_text = ' '.join( + line.strip() for line in remaining_lines[:close_index + 1] + )[1:-1] + tags = [tag.strip() for tag in tag_text.split(',')] + if tags == ['']: + errors.append( + FormatError(first_line_no, "empty tag block; use e.g. '[GH #123]'") + ) + else: + seen_tags = set() + for tag in tags: + if TAG_REGEX.match(tag) is None: + errors.append( + FormatError(first_line_no, f"invalid tag '{tag}'; use 'GH #N'") + ) + elif tag in seen_tags: + errors.append(FormatError(first_line_no, f"tag '{tag}' is repeated")) + seen_tags.add(tag) + + summary_index = close_index + 1 + summary_line_no = first_line_no + summary_index + if (summary_index >= len(remaining_lines) or + remaining_lines[summary_index] == ''): + errors.append( + FormatError(summary_line_no, 'expected the summary line after the tags') + ) + elif remaining_lines[summary_index][0].isspace(): + errors.append( + FormatError(summary_line_no, 'summary line must not be indented') + ) + elif (summary_index + 2 >= len(remaining_lines) or + remaining_lines[summary_index + 1] != '' or + remaining_lines[summary_index + 2] == ''): + errors.append( + FormatError( + summary_line_no + 1, + 'expected one blank line and then the description after the summary' + ) + ) + + if len(errors) != 0: + return None, errors + + return ChangesEntry( + path, + tags, + remaining_lines[summary_index], + remaining_lines[summary_index + 2:] + ), [] + +def read_prose(path: Path) -> Tuple[Set[str], List[str]]: + '''Return the words an entry's //allowed_words: directives accept, and the + lines of the entry to spell check (everything after the directives and the + tag block).''' + with open(path, 'r', encoding = 'utf-8', errors = 'replace') as handle: + lines = [line.rstrip('\n') for line in handle] + directive_map, remaining_lines = ( + edgutil.split_leading_cpp_line_directives(lines) + ) + + allowed_words: Set[str] = set() + for value in directive_map.get(ALLOWED_WORDS_DIRECTIVE, []): + allowed_words.update(value.split()) + + if len(remaining_lines) != 0 and remaining_lines[0].startswith('['): + close_index = _find_tag_block_end(remaining_lines) + if close_index is None: + close_index = len(remaining_lines) - 1 + remaining_lines = remaining_lines[close_index + 1:] + + return allowed_words, remaining_lines diff --git a/dev_tools/pylibs/edgchanges/py.typed b/dev_tools/pylibs/edgchanges/py.typed new file mode 100644 index 0000000000..e69de29bb2 diff --git a/dev_tools/pylibs/edgutil/__init__.py b/dev_tools/pylibs/edgutil/__init__.py index e198570ca7..a06c6a39a6 100644 --- a/dev_tools/pylibs/edgutil/__init__.py +++ b/dev_tools/pylibs/edgutil/__init__.py @@ -28,6 +28,7 @@ List, Optional, Set, + Tuple, cast ) @@ -189,6 +190,25 @@ def indent_text_block(text_block: str, n_times: int = 1) -> str: indent = ' ' * n_times return re.sub(r'^(.)', f"{indent}\\1", text_block, flags = re.MULTILINE) +def format_overlong_lines(overlong_lines: Dict[str, Any], + heading_indent: str = '') -> List[str]: + '''Format the overlong_lines result of edg-check-file-changes ({line number: + [first context line number, context lines]}) for display. + + Each overlong line gets a heading (indented by heading_indent) and its + context between two 79-column rules. The context isn't indented, so the + columns line up with the rules. + ''' + report_lines = [] + for line_no, (line_start, lines) in sorted(overlong_lines.items(), + key = lambda item: int(item[0])): + line_end = line_start + (len(lines) - 1) + report_lines.append(f"{heading_indent}Line {line_no} too long:") + report_lines.append(f" {line_start} -> {line_end} ".center(79, '=')) + report_lines += [line.rstrip('\n') for line in lines] + report_lines.append('=' * 79) + return report_lines + def is_env_set(env_var: str) -> bool: '''Return True if the given environment variable is set to a non-zero value.''' @@ -269,6 +289,24 @@ def collect_cpp_line_directives(file_path: Path) -> Dict[str, List[str]]: return directive_map +def split_leading_cpp_line_directives( + lines: List[str]) -> Tuple[Dict[str, List[str]], List[str]]: + '''Split the EDG C++ directive block off the start of lines. + + Unlike collect_cpp_line_directives, only the unbroken block of directive + lines starting at the very first line is collected, so directive-looking + lines later on (e.g., in an example) are left alone. + + Returns the directive map and the lines that follow the block. + ''' + directive_map: Dict[str, List[str]] = {} + line_index = 0 + while (line_index < len(lines) and + _collect_directive_line(directive_map, lines[line_index])): + line_index += 1 + + return directive_map, lines[line_index:] + def replace_cpp_line_directives(file_path: Path, directive_lines: Iterable[str]) -> None: '''Replace the leading EDG C++ directive block in file_path. diff --git a/dev_tools/tests/test_edg_collect_changes.py b/dev_tools/tests/test_edg_collect_changes.py new file mode 100644 index 0000000000..c22b0988f1 --- /dev/null +++ b/dev_tools/tests/test_edg_collect_changes.py @@ -0,0 +1,630 @@ +# Part of the EDG Compiler Project, under the Apache License v2.0 with LLVM +# Exceptions. +# See https://edgcpp.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +'''Tests for edg-collect-changes, edgchanges (the pending Changes entry +format), the entry checks in edg-check-file-changes, and the edgutil directive +helper they use. + +Run with: python3 -m unittest discover -s dev_tools/tests +(with dev_tools/pylibs on PYTHONPATH). +''' + +import contextlib +import datetime +import importlib.machinery +import importlib.util +import io +import json +import os +import shutil +import subprocess +import sys +import tempfile +import unittest + +import edgchanges +import edgutil + +from pathlib import Path +from typing import Any, Dict + +DEV_TOOLS_DIR = Path(__file__).resolve().parents[1] +REPO_DIR = DEV_TOOLS_DIR.parent +COLLECT_TOOL_PATH = DEV_TOOLS_DIR / 'bin' / 'edg-collect-changes' +CHECK_TOOL_PATH = DEV_TOOLS_DIR / 'bin' / 'edg-check-file-changes' + +def _load_collect_tool() -> Any: + loader = importlib.machinery.SourceFileLoader( + 'edg_collect_changes', + str(COLLECT_TOOL_PATH) + ) + spec = importlib.util.spec_from_loader(loader.name, loader) + assert spec is not None + module = importlib.util.module_from_spec(spec) + # Keep dev_tools/bin free of __pycache__. + dont_write_bytecode = sys.dont_write_bytecode + sys.dont_write_bytecode = True + try: + loader.exec_module(module) + finally: + sys.dont_write_bytecode = dont_write_bytecode + return module + +collect_tool = _load_collect_tool() + +OPEN_CHANGES = '''Version 7.1 + +10/20/26 [GH #38] +Existing entry + +Existing body. + + +------------------------------------------------------------------------------- +Version 7.0, September 28, 2026 + +9/1/26 [EDGcpfe/1] +Old entry + +Version 6.9 of the front end is mentioned in this body. +''' + +RELEASED_CHANGES = '''Version 7.0, September 28, 2026 + +No changes. + + +------------------------------------------------------------------------------- +Version 6.10, June 1, 2026 + +No changes. +''' + +def _fragment(tags: str, summary: str, body: str = 'Body.') -> str: + return f"{tags}\n{summary}\n\n{body}\n" + +class FakeTree: + '''A scratch tree of Changes files and pending_changes/ directories.''' + + def __init__(self, root: Path) -> None: + self.root = root + + def add_changes(self, dir_name: str, text: str) -> Path: + dir_path = self.root / dir_name + (dir_path / 'pending_changes').mkdir(parents = True) + (dir_path / 'pending_changes' / '.gitignore').write_text('# Keep.\n') + changes_path = dir_path / 'Changes' + changes_path.write_text(text, encoding = 'utf-8') + return changes_path + + def add_fragment(self, dir_name: str, name: str, text: str) -> Path: + path = self.root / dir_name / 'pending_changes' / name + path.write_text(text, encoding = 'utf-8') + return path + + def collect(self, dir_name: str = 'src', + date: datetime.date = datetime.date(2026, 10, 21), + version: Any = None, dry_run: bool = False) -> bool: + with contextlib.redirect_stdout(io.StringIO()), \ + contextlib.redirect_stderr(io.StringIO()): + return collect_tool.collect( + self.root / dir_name / 'Changes', + self.root / dir_name / 'pending_changes', + date, + version, + dry_run = dry_run + ) + +class TreeTestCase(unittest.TestCase): + + def setUp(self) -> None: + tmp_dir = tempfile.TemporaryDirectory() + self.addCleanup(tmp_dir.cleanup) + self.tree = FakeTree(Path(tmp_dir.name)) + +class FormatTests(unittest.TestCase): + + def test_date_padding(self) -> None: + for date, expected in [ + (datetime.date(2026, 1, 2), '1/2/26 [GH #1]'), + (datetime.date(2026, 1, 21), '1/21/26 [GH #1]'), + (datetime.date(2026, 10, 21), '10/21/26 [GH #1]') + ]: + date_text = edgchanges.format_date(date) + self.assertEqual( + edgchanges.format_tag_lines(date_text, ['GH #1']), + [expected] + ) + + def test_wrap_matches_existing_changes(self) -> None: + # src/Changes wraps long tag lists this way (see the 3/5/26 entry for + # "Pragma processing corrections"). + tags = [ + 'EDGcpfe/23609', 'EDGcpfe/23845', 'EDGcpfe/23927', 'EDGcpfe/24247', + 'EDGcpfe/27220', 'EDGcpfe/28636' + ] + self.assertEqual( + edgchanges.format_tag_lines('3/5/26', tags), + [ + '3/5/26 [EDGcpfe/23609,EDGcpfe/23845,EDGcpfe/23927,EDGcpfe/24247,', + ' EDGcpfe/27220,EDGcpfe/28636]' + ] + ) + + def test_wrap_exactly_at_limit(self) -> None: + # 10 columns of prefix + 69 columns of tags fill the line exactly. + tags = ['GH #1234567', 'GH #1234567', 'GH #1234567', 'GH #1234567', + 'GH #1234567', 'GH #1234'] + lines = edgchanges.format_tag_lines('10/21/26', tags) + self.assertEqual(len(lines), 1) + self.assertEqual(len(lines[0]), 79) + + lines = edgchanges.format_tag_lines('10/21/26', tags + ['GH #1']) + self.assertEqual( + lines, + [ + '10/21/26 [GH #1234567,GH #1234567,GH #1234567,GH #1234567,' + 'GH #1234567,GH #1234,', + ' GH #1]' + ] + ) + +class ParseTests(TreeTestCase): + + def _parse(self, text: str, check_line_length: bool = True) -> Any: + self.tree.root.joinpath('x.txt').write_text(text, encoding = 'utf-8') + return edgchanges.parse_fragment( + self.tree.root / 'x.txt', + check_line_length = check_line_length + ) + + def test_good_fragment(self) -> None: + entry, errors = self._parse( + '//allowed_words: frob \n' + '[GH #2,\n' + ' GH #1]\n' + 'Summary \n' + '\n' + 'Body line. \n' + '\n' + ' //not: a directive\n' + '\n' + '\n' + ) + self.assertEqual(errors, []) + self.assertEqual(entry.tags, ['GH #2', 'GH #1']) + self.assertEqual(entry.summary, 'Summary') + self.assertEqual( + entry.body_lines, + ['Body line.', '', ' //not: a directive'] + ) + + def test_crlf(self) -> None: + entry, errors = self._parse('[GH #1]\r\nSummary\r\n\r\nBody.\r\n') + self.assertEqual(errors, []) + self.assertEqual(entry.body_lines, ['Body.']) + + def _assert_error(self, text: str, expected: str) -> None: + entry, errors = self._parse(text) + self.assertIsNone(entry) + self.assertTrue( + any(expected in error.message for error in errors), + f"{expected!r} not in {errors!r}" + ) + + def test_errors(self) -> None: + self._assert_error('Summary\n\nBody.\n', 'expected the tag block') + self._assert_error(_fragment('[]', 'Summary'), 'empty tag block') + self._assert_error( + _fragment('[EDGcpfe/1]', 'Summary'), + "invalid tag 'EDGcpfe/1'" + ) + self._assert_error( + _fragment('[GH #1,GH #1]', 'Summary'), + "tag 'GH #1' is repeated" + ) + self._assert_error(_fragment('[GH #1', 'Summary'), "no closing ']'") + self._assert_error( + _fragment('[GH #1] Summary', 'More'), + "text after the tag block's ']'" + ) + self._assert_error('[GH #1]\n\nBody.\n', 'expected the summary line') + self._assert_error( + _fragment('[GH #1]', ' Summary'), + 'summary line must not be indented' + ) + self._assert_error('[GH #1]\nSummary\nBody.\n', 'expected one blank line') + self._assert_error('[GH #1]\nSummary\n', 'expected one blank line') + self._assert_error( + '[GH #1]\nSummary\n\n\nBody.\n', + 'expected one blank line' + ) + self._assert_error(_fragment('[GH #1]', 'Sum\tmary'), 'tab character') + self._assert_error( + _fragment('[GH #1]', 'Summary', 'x' * 80), + 'line longer than 79 columns' + ) + + def test_line_length(self) -> None: + entry, errors = self._parse( + _fragment('[GH #1]', 'Summary', 'x' * 79 + ' ') + ) + self.assertEqual(errors, []) + + entry, errors = self._parse( + _fragment('[GH #1]', 'Summary', 'x' * 80), + check_line_length = False + ) + self.assertEqual(errors, []) + + def test_not_utf8(self) -> None: + self.tree.root.joinpath('x.txt').write_bytes( + b'[GH #1]\nSummary\n\n\xe9\n' + ) + entry, errors = edgchanges.parse_fragment(self.tree.root / 'x.txt') + self.assertIsNone(entry) + self.assertIsNone(errors[0].line_no) + self.assertIn('not valid UTF-8', errors[0].message) + +class CollectTests(TreeTestCase): + + def test_open_section(self) -> None: + changes_path = self.tree.add_changes('src', OPEN_CHANGES) + zeta = self.tree.add_fragment( + 'src', 'gh-2.txt', _fragment('[GH #2]', 'zeta', 'Zeta body. ') + ) + alpha = self.tree.add_fragment( + 'src', 'gh-1.txt', + '//allowed_words: frob\n' + _fragment('[GH #1]', 'Alpha') + ) + + self.assertTrue(self.tree.collect()) + + self.assertEqual( + changes_path.read_text(encoding = 'utf-8'), + 'Version 7.1\n' + '\n' + '10/21/26 [GH #1]\n' + 'Alpha\n' + '\n' + 'Body.\n' + '\n' + '\n' + '10/21/26 [GH #2]\n' + 'zeta\n' + '\n' + 'Zeta body.\n' + '\n' + '\n' + + OPEN_CHANGES[len('Version 7.1\n\n'):] + ) + self.assertFalse(zeta.exists()) + self.assertFalse(alpha.exists()) + + def test_matches_real_changes_layout(self) -> None: + # Collecting the newest src/Changes entry, written as a pending entry, + # into the file without it reproduces the file. + real_text = (REPO_DIR / 'src' / 'Changes').read_text(encoding = 'utf-8') + lines = real_text.split('\n') + self.assertRegex(lines[0], r'^Version [0-9]+\.[0-9]+$') + self.assertEqual(lines[1], '') + next_entry_index = next( + index for index in range(3, len(lines)) + if lines[index - 2:index] == ['', ''] and lines[index] != '' + ) + entry_lines = lines[2:next_entry_index - 2] + date_text, tags_text = entry_lines[0].split(' ', 1) + if (any(line.startswith(' ') for line in entry_lines[1:2]) or + '[GH #' not in tags_text): + self.skipTest('newest src/Changes entry is not a single-line GH entry') + + changes_path = self.tree.add_changes( + 'src', + '\n'.join(lines[:2] + lines[next_entry_index:]) + ) + self.tree.add_fragment( + 'src', 'newest.txt', + '\n'.join([tags_text.strip()] + entry_lines[1:]) + '\n' + ) + month, day, year = (int(part) for part in date_text.split('/')) + self.assertTrue( + self.tree.collect(date = datetime.date(2000 + year, month, day)) + ) + self.assertEqual(changes_path.read_text(encoding = 'utf-8'), real_text) + + def test_released_section(self) -> None: + self.tree.add_changes('src', OPEN_CHANGES) + changes_path = self.tree.add_changes('lib_src', RELEASED_CHANGES) + self.tree.add_fragment( + 'lib_src', 'gh-5.txt', _fragment('[GH #5]', 'Library') + ) + + self.assertTrue(self.tree.collect('lib_src', version = '7.1')) + + self.assertEqual( + changes_path.read_text(encoding = 'utf-8'), + 'Version 7.1\n' + '\n' + '10/21/26 [GH #5]\n' + 'Library\n' + '\n' + 'Body.\n' + '\n' + '\n' + f"{'-' * 79}\n" + + RELEASED_CHANGES + ) + + def test_released_section_needs_version(self) -> None: + self.tree.add_changes('lib_src', RELEASED_CHANGES) + self.tree.add_fragment( + 'lib_src', 'gh-5.txt', _fragment('[GH #5]', 'Library') + ) + with self.assertRaisesRegex(ValueError, 'use --version'): + self.tree.collect('lib_src') + + self.assertTrue(self.tree.collect('lib_src', version = '8.0')) + changes_path = self.tree.root / 'lib_src' / 'Changes' + self.assertTrue( + changes_path.read_text(encoding = 'utf-8').startswith('Version 8.0\n') + ) + + def test_read_open_version(self) -> None: + # What --version-from reads. + self.assertEqual( + collect_tool.read_open_version( + self.tree.add_changes('src', OPEN_CHANGES) + ), + '7.1' + ) + self.assertIsNone( + collect_tool.read_open_version( + self.tree.add_changes('lib_src', RELEASED_CHANGES) + ) + ) + + def test_no_changes_replaced(self) -> None: + changes_path = self.tree.add_changes( + 'src', + 'Version 7.1\n\nNo changes.\n\n\n' + RELEASED_CHANGES.split('\n', 5)[5] + ) + self.tree.add_fragment('src', 'a.txt', _fragment('[GH #1]', 'A')) + + self.assertTrue(self.tree.collect()) + + self.assertEqual( + changes_path.read_text(encoding = 'utf-8'), + 'Version 7.1\n\n10/21/26 [GH #1]\nA\n\nBody.\n\n\n' + + RELEASED_CHANGES.split('\n', 5)[5] + ) + + def test_header_needs_separator(self) -> None: + # "Version 6.9 ..." in a body isn't a header, and neither is a version + # line that doesn't follow a separator. + lines = OPEN_CHANGES.split('\n') + self.assertEqual( + collect_tool._find_newest_version_header(lines[2:] + lines[:2]), + lines[2:].index('Version 7.0, September 28, 2026') + ) + self.assertIsNone( + collect_tool._find_newest_version_header(['', 'Version 7.1']) + ) + + def test_sort_tie_break(self) -> None: + changes_path = self.tree.add_changes('src', OPEN_CHANGES) + self.tree.add_fragment('src', 'b.txt', _fragment('[GH #2]', 'Same')) + self.tree.add_fragment('src', 'a.txt', _fragment('[GH #1]', 'Same')) + self.tree.add_fragment('src', 'c.txt', _fragment('[GH #3]', 'same')) + + self.assertTrue(self.tree.collect()) + + tag_lines = [ + line for line in changes_path.read_text(encoding = 'utf-8').split('\n') + if line.startswith('10/21/26') + ] + self.assertEqual( + tag_lines, + ['10/21/26 [GH #1]', '10/21/26 [GH #2]', '10/21/26 [GH #3]'] + ) + + def test_bad_fragment_left_in_place(self) -> None: + changes_path = self.tree.add_changes('src', OPEN_CHANGES) + good = self.tree.add_fragment( + 'src', 'good.txt', _fragment('[GH #1]', 'Good') + ) + bad = self.tree.add_fragment( + 'src', 'bad.txt', _fragment('[EDGcpfe/1]', 'Bad') + ) + + self.assertFalse(self.tree.collect()) + + self.assertFalse(good.exists()) + self.assertTrue(bad.exists()) + text = changes_path.read_text(encoding = 'utf-8') + self.assertIn('Good', text) + self.assertNotIn('Bad', text) + + def test_nothing_pending(self) -> None: + changes_path = self.tree.add_changes('src', OPEN_CHANGES) + self.tree.add_fragment('src', 'notes.md', 'Not an entry.\n') + before = changes_path.stat().st_mtime_ns + + self.assertTrue(self.tree.collect()) + + self.assertEqual(changes_path.stat().st_mtime_ns, before) + self.assertEqual(changes_path.read_text(encoding = 'utf-8'), OPEN_CHANGES) + + def test_dry_run(self) -> None: + changes_path = self.tree.add_changes('src', OPEN_CHANGES) + fragment = self.tree.add_fragment( + 'src', 'a.txt', _fragment('[GH #1]', 'A') + ) + + self.assertTrue(self.tree.collect(dry_run = True)) + + self.assertTrue(fragment.exists()) + self.assertEqual(changes_path.read_text(encoding = 'utf-8'), OPEN_CHANGES) + + def test_check_rejects_misnamed_file(self) -> None: + self.tree.add_changes('src', OPEN_CHANGES) + self.tree.add_fragment('src', 'notes.md', _fragment('[GH #1]', 'A')) + with contextlib.redirect_stdout(io.StringIO()) as output: + self.assertFalse( + collect_tool.check(self.tree.root / 'src' / 'pending_changes') + ) + self.assertIn('notes.md: not a pending Changes entry', output.getvalue()) + +class FragmentPathTests(unittest.TestCase): + + def test_is_fragment_path(self) -> None: + self.assertTrue(edgchanges.is_fragment_path('src/pending_changes/a.txt')) + self.assertTrue( + edgchanges.is_fragment_path('include_c++/pending_changes/gh-1-x.txt') + ) + self.assertFalse(edgchanges.is_fragment_path('src/pending_changes/a.md')) + self.assertFalse( + edgchanges.is_fragment_path('src/pending_changes/.gitignore') + ) + self.assertFalse(edgchanges.is_fragment_path('src/error_msg.txt')) + self.assertFalse( + edgchanges.is_fragment_path('src/pending_changes/sub/a.txt') + ) + +class DirectiveTests(unittest.TestCase): + + def test_leading_block_only(self) -> None: + directive_map, remaining_lines = ( + edgutil.split_leading_cpp_line_directives([ + '//allowed_words: a b', + '//allowed_words: c', + '[GH #1]', + '//allowed_words: d' + ]) + ) + self.assertEqual(directive_map, {'allowed_words': ['a b', 'c']}) + self.assertEqual(remaining_lines, ['[GH #1]', '//allowed_words: d']) + + def test_no_block(self) -> None: + directive_map, remaining_lines = ( + edgutil.split_leading_cpp_line_directives(['[GH #1]', '//x: y']) + ) + self.assertEqual(directive_map, {}) + self.assertEqual(remaining_lines, ['[GH #1]', '//x: y']) + +def _aspell_works() -> bool: + aspell_path = shutil.which('aspell') + if aspell_path is None: + return False + return subprocess.run( + [aspell_path, 'list'], + input = b'cat', + stdout = subprocess.DEVNULL, + stderr = subprocess.DEVNULL + ).returncode == 0 + +class CheckFileChangesTests(unittest.TestCase): + '''Changes entry rules in edg-check-file-changes.''' + + def setUp(self) -> None: + tmp_dir = tempfile.TemporaryDirectory() + self.addCleanup(tmp_dir.cleanup) + self.tmp_dir = Path(tmp_dir.name) + + def _check(self, repo_path: str, text: str) -> Dict[str, Any]: + # Like the pre-push hook's, the temporary files aren't in a + # pending_changes/ directory; only the key is. + old_file = self.tmp_dir / 'oldXYZ.txt' + old_file.write_text('') + new_file = self.tmp_dir / 'newXYZ.txt' + new_file.write_text(text, encoding = 'utf-8') + input_file = self.tmp_dir / 'input.json' + output_file = self.tmp_dir / 'output.json' + input_file.write_text( + json.dumps({repo_path: [str(old_file), str(new_file)]}) + ) + # The checker looks for the entry's Changes file relative to its working + # directory. + subprocess.run( + [sys.executable, str(CHECK_TOOL_PATH), str(input_file), + str(output_file)], + check = True, + cwd = str(self.tmp_dir), + env = dict(os.environ, PYTHONPATH = str(DEV_TOOLS_DIR / 'pylibs')) + ) + return json.loads(output_file.read_text())[repo_path] + + def test_format_errors(self) -> None: + results = self._check( + 'src/pending_changes/a.txt', + '//allowed_words: x\n[EDGcpfe/1]\nSummary\n\nBody.\n' + ) + self.assertEqual( + results['format_errors'], + [[2, "invalid tag 'EDGcpfe/1'; use 'GH #N'"]] + ) + + results = self._check( + 'src/pending_changes/a.txt', + _fragment('[GH #1]', 'Summary') + ) + self.assertEqual(results['format_errors'], []) + + # Other files have no format to check. + results = self._check('notes.txt', 'Anything.\n') + self.assertEqual(results['format_errors'], []) + + def test_trailing_whitespace_not_overlong(self) -> None: + text = _fragment('[GH #1]', 'Summary', 'x' * 79 + ' ') + results = self._check('src/pending_changes/a.txt', text) + self.assertEqual(results['overlong_lines'], {}) + + results = self._check('src/pending_changes/a.txt', + _fragment('[GH #1]', 'Summary', 'x' * 80)) + self.assertEqual(list(results['overlong_lines']), ['4']) + + @unittest.skipUnless(_aspell_works(), 'aspell is not available') + def test_allowed_words(self) -> None: + text = _fragment('[GH #1,\n GH #2]', 'Summary', 'A frobnicator lvalue.') + results = self._check('src/pending_changes/a.txt', text) + # The tag block isn't spell checked. + self.assertEqual(results['misspellings'], ['frobnicator', 'lvalue']) + + results = self._check( + 'src/pending_changes/a.txt', + '//allowed_words: frobnicator\n//allowed_words: lvalue\n' + text + ) + self.assertEqual(results['misspellings'], []) + + @unittest.skipUnless(_aspell_works(), 'aspell is not available') + def test_words_in_target_changes_file(self) -> None: + # Words the target Changes file already uses aren't reported; other + # Changes files don't count. + (self.tmp_dir / 'src').mkdir() + (self.tmp_dir / 'src' / 'Changes').write_text( + 'Version 7.1\n\nAn lvalue.\n' + ) + (self.tmp_dir / 'util').mkdir() + (self.tmp_dir / 'util' / 'Changes').write_text( + 'Version 7.1\n\nA frobnicator.\n' + ) + text = _fragment('[GH #1]', 'Summary', 'A frobnicator lvalue.') + + results = self._check('src/pending_changes/a.txt', text) + self.assertEqual(results['misspellings'], ['frobnicator']) + + results = self._check('util/pending_changes/a.txt', text) + self.assertEqual(results['misspellings'], ['lvalue']) + + @unittest.skipUnless(_aspell_works(), 'aspell is not available') + def test_plain_text_file_unchanged(self) -> None: + # Outside pending_changes/, directives get no special treatment. + results = self._check( + 'notes.txt', + '//allowed_words: frobnicator\nA frobnicator.\n' + ) + self.assertIn('frobnicator', results['misspellings']) + +if __name__ == '__main__': + unittest.main() diff --git a/dev_tools/typecheck-python.sh b/dev_tools/typecheck-python.sh index 6ef53e6ff4..60c7817375 100755 --- a/dev_tools/typecheck-python.sh +++ b/dev_tools/typecheck-python.sh @@ -57,6 +57,7 @@ pushd bin/ > /dev/null check_types_strict edg-bench-run-delta check_types_strict edg-bench-run-save check_types edg-cmakedef + check_types_strict edg-collect-changes check_types_strict edg-docker-bench check_types_strict edg-docker-test check_types edg-examine-test @@ -95,6 +96,9 @@ pushd pylibs/ > /dev/null pushd edgbench/ > /dev/null check_types_strict __init__.py popd > /dev/null + pushd edgchanges/ > /dev/null + check_types_strict __init__.py + popd > /dev/null pushd edgcppgen/ > /dev/null check_types __init__.py popd > /dev/null diff --git a/dev_tools/win-bin/edg-collect-changes.bat b/dev_tools/win-bin/edg-collect-changes.bat new file mode 100644 index 0000000000..5457bf2752 --- /dev/null +++ b/dev_tools/win-bin/edg-collect-changes.bat @@ -0,0 +1,2 @@ +@echo off +python.exe "%~dp0..\bin\edg-collect-changes" %* \ No newline at end of file diff --git a/include_c++/pending_changes/.gitignore b/include_c++/pending_changes/.gitignore new file mode 100644 index 0000000000..1aac391c1b --- /dev/null +++ b/include_c++/pending_changes/.gitignore @@ -0,0 +1,3 @@ +# This file only keeps pending_changes/ in git; it ignores nothing. +# Add pending Changes entries here as *.txt files (see CONTRIBUTING.md). +# A nightly job moves them into ../Changes with edg-collect-changes. diff --git a/lib_src/pending_changes/.gitignore b/lib_src/pending_changes/.gitignore new file mode 100644 index 0000000000..1aac391c1b --- /dev/null +++ b/lib_src/pending_changes/.gitignore @@ -0,0 +1,3 @@ +# This file only keeps pending_changes/ in git; it ignores nothing. +# Add pending Changes entries here as *.txt files (see CONTRIBUTING.md). +# A nightly job moves them into ../Changes with edg-collect-changes. diff --git a/nt_util/pending_changes/.gitignore b/nt_util/pending_changes/.gitignore new file mode 100644 index 0000000000..1aac391c1b --- /dev/null +++ b/nt_util/pending_changes/.gitignore @@ -0,0 +1,3 @@ +# This file only keeps pending_changes/ in git; it ignores nothing. +# Add pending Changes entries here as *.txt files (see CONTRIBUTING.md). +# A nightly job moves them into ../Changes with edg-collect-changes. diff --git a/src/pending_changes/.gitignore b/src/pending_changes/.gitignore new file mode 100644 index 0000000000..1aac391c1b --- /dev/null +++ b/src/pending_changes/.gitignore @@ -0,0 +1,3 @@ +# This file only keeps pending_changes/ in git; it ignores nothing. +# Add pending Changes entries here as *.txt files (see CONTRIBUTING.md). +# A nightly job moves them into ../Changes with edg-collect-changes. diff --git a/util/pending_changes/.gitignore b/util/pending_changes/.gitignore new file mode 100644 index 0000000000..1aac391c1b --- /dev/null +++ b/util/pending_changes/.gitignore @@ -0,0 +1,3 @@ +# This file only keeps pending_changes/ in git; it ignores nothing. +# Add pending Changes entries here as *.txt files (see CONTRIBUTING.md). +# A nightly job moves them into ../Changes with edg-collect-changes.