Repository navigation
Merge upstream master 12 - #22
Open
Mopsgeschwindigkeit wants to merge 54 commits into
Open
Mopsgeschwindigkeit wants to merge 54 commits into
Mopsgeschwindigkeit wants to merge 54 commits into
Conversation
Resolve conflict in faust/transport/drivers/__init__.py by keeping the FactoryMapping-based transport registry (with the confluent driver registered), consistent with every other faust registry (stores, web cache backends, fixups) and preserving lazy driver loading so the optional confluent_kafka dependency is only imported when the confluent transport is actually selected.
The driver depended on Consumer.io_event_enable(), an eventfd-based notification API that only exists in the unmerged 2022 fork stephan-hof/confluent-kafka-python@features/io_event_enable and is absent from every released confluent-kafka. This made faust[ckafka] install an abandoned personal fork and the driver unusable on the official package. Remove the eventfd notification machinery from AsyncConsumer (eventfd, io_event_enable, add_reader/__eventfd_ready, the waiters set and the weakref finalizer) and reduce poll() to a thin wrapper over the blocking confluent_kafka.Consumer.poll(). This mechanism was already unused by the real fetch path -- ConfluentConsumerThread.getmany() drives the blocking Consumer.consume() inside the dedicated consumer thread via call_thread() -- so no functional behavior changes. Point requirements/extras/ckafka.txt at the official confluent-kafka>=2.0.0 release, and drop the now-unused os, struct and weakref imports plus a dead cast in Consumer.on_stop().
The previous tests/unit/transport/drivers/test_confluent.py (~1375 lines) was copied wholesale from the aiokafka test suite and asserted against methods the confluent driver never had (_fetch_records, _max_poll_records, transactional producer methods that raise NotImplementedError, ...), so 60 of 85 tests failed and several blocked in librdkafka. Replace it with a focused suite that exercises the driver's actual surface -- server_list, Consumer, AsyncConsumer, ConfluentConsumerThread, Producer, ProducerThread, ProducerProduceFuture and Transport -- with the underlying confluent_kafka Consumer/Producer mocked so no broker is needed. 56 tests, all passing, runs in <1s. Includes a regression guard that AsyncConsumer construction never calls the removed io_event_enable.
Consumer.create_topic() computed retention as int(want_seconds(retention) * 1000.0) unconditionally, so the default retention=None raised TypeError (None * 1000.0) whenever a topic was declared without an explicit retention. Guard the conversion the same way the aiokafka driver does (pass None through), and add a regression test.
confluent-kafka is an optional extra (faust[ckafka]) and is not part of
requirements/test.txt, so it is absent in the CI test matrix. The test
module imports the driver (which imports confluent_kafka) at import time,
which turned every matrix job into a collection error. Guard the module
with pytest.importorskip('confluent_kafka') so it skips cleanly when the
optional dependency is unavailable, and runs in full where it is.
confluent.py is only importable when the ckafka extra (faust[ckafka]) is installed, which the CI test environment does not do -- so its tests are skipped there and the file shows as 0% covered, dragging project/patch coverage down. Omit it from coverage like the memory driver already is; the driver's own tests still run wherever confluent-kafka is available.
black and isort were unpinned in requirements/test.txt, so CI always installed the newest release. When isort 8.0.1 and black 26.5.1 shipped, their changed formatting output started failing 'Check linting' on otherwise-unchanged trees -- lint was passing or failing depending only on when the job happened to run. Pin both to the current versions (the ones the tree is already formatted for) so linting is reproducible. Bump them deliberately, together with a tree-wide reformat, in future.
black 26.5.1 and isort 8.0.1 require Python >= 3.10, so pinning them unconditionally broke 'pip install -r requirements/test.txt' on the Python 3.9 test-matrix jobs. black/isort are lint-only and only run in the lint job (PYTHON_LATEST), so pin them on 3.10+ and fall back to an unpinned install on 3.9, where they are installed but never invoked.
…n-lint-versions
The confluent readd had switched examples/word_count.py to 'confluent://localhost:9092'. Confluent is an optional, experimental transport (installed via faust[ckafka]); mainstream examples should use the default aiokafka transport. Revert the example broker to 'kafka://localhost:9092'.
…ust-prs-sync-fixes-63nsnu
…min request compatibility Ensure worker consumers use the Faust assignor whenever changelog tables are present, even with zero standby replicas, so table recovery mapping remains correct. Also update metadata/create-topics request construction for newer aiokafka APIs.
…reaming#684) faust-streaming#649 bumped upload/download-artifact from v3 to v4 but kept the v3 pattern where every build_wheels matrix leg (and the sdist) uploaded under a shared artifact name and the publish job downloaded a single name. v3 merged same-named uploads; v4 does not -- each upload is its own immutable artifact and uploading the same name again fails. So the three matrix legs collide on 'wheel-artifacts', and the wheel download used pattern '*artifact*' with no path (landing outside dist/, which is what gh-action-pypi-publish publishes from). The last release (v0.11.3) predates faust-streaming#649 and used v3, so this is latent -- the next release would fail to publish or ship incomplete artifacts. Give each uploader a unique name (cibw-wheels-<os>, cibw-sdist), collect them all in the publish job with pattern: cibw-* + merge-multiple into dist/, add skip-existing so re-runs can backfill, and if-no-files-found: error so an empty build fails loudly instead of silently publishing nothing.
Convert three coverage gaps into genuine tests instead of leaving them excluded: - utils/functional.py: add a test for translate() (only consecutive_numbers was covered -> 75%); now 100%. Remove its .coveragerc omit. - utils/platforms.py: test max_open_files() across all branches (normal hard limit, RLIM_INFINITY on non-Darwin, RLIM_INFINITY on Darwin via sysctl, and missing 'resource' module); now 100%. Remove its omit. - utils/agent_stopper.py: test that agent_stopper() crashes the app with a RuntimeError; 0% -> 100% (was measured but untested). platforms tests use stacked @patch decorators rather than a parenthesized multi-context 'with' so the file stays valid on Python 3.9.
The new MetadataRequest([]) / CreateTopicsRequest(...) forms require the aiokafka 0.13.0 protocol refactor; on aiokafka < 0.13 MetadataRequest is a versioned list (not callable) and CreateTopicsRequest must be indexed by version. Faust keeps its aiokafka>=0.10 floor and already branches on _AIOKAFKA_HAS_API_VERSION, so use the version-appropriate form at both call sites instead of the new one unconditionally: - _get_controller_node: MetadataRequest_v1([]) on <0.13, MetadataRequest([]) on >=0.13 - _really_create_topic: CreateTopicsRequest[1](...) on <0.13, CreateTopicsRequest(...) on >=0.13 This also fixes master's latent breakage where CreateTopicsRequest[version] already fails on aiokafka 0.13+. Also drop a duplicate _AIOKAFKA_HAS_API_VERSION definition introduced by the rebase.
…okafka-changelog-assignor
Adds 3.14 to the test matrix, producing required 'Python 3.14/Cython: true' and 'Python 3.14/Cython: false' jobs alongside the existing 3.13 legs. All matrix legs feed the aggregate '✅ Ensure the required checks passing' job, so they become required wherever branch protection requires that check. Python 3.9 was already removed from the matrix and from python_requires (>=3.10); the lingering 'Python 3.9/Cython: *' required status checks exist only in branch protection and must be deleted there. Note: the 3.14 legs need a mode-streaming release carrying the event-loop fix (faust-streaming/mode#72) to pass, since Python 3.14 removed implicit event-loop creation that mode relied on at import time. Assisted-by: Claude Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgnKFtXZbCjXNoVNWa5JLd
…9, require 3.14) (faust-streaming#690) * Modernize release wheel-build runners The build_wheels matrix pinned ubuntu-20.04, whose runner image GitHub retired in April 2025 -- that job can no longer schedule. macos-13 is also on the deprecation path, and pypa/cibuildwheel@v2.10.1 (Sept 2022) predates Python 3.12/3.13 wheel support and newer runner images. This only runs on release, so the breakage is latent. - os matrix: ubuntu-20.04 -> ubuntu-latest; macos-13 -> macos-15-intel (still x86_64), keeping macos-14 for arm64. - pypa/cibuildwheel: v2.10.1 -> v2.21.3, so release wheels cover the same Python range (3.10-3.13) the test matrix already exercises. cibuildwheel honors python_requires (>=3.10), so older interpreters stay skipped. The other workflows (codeql-analysis, gh-pages) already use ubuntu-latest and current action versions; no changes needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL * Build Windows wheels on release Add windows-2022 to the release wheel matrix. pyproject.toml already declares Windows support and carries a [tool.cibuildwheel] config (build = cp3*, archs = auto64, before-build installs Cython), and setup.py already handles the win32 case (links ws2_32, skips the gcc-only warning flags), so the existing build config applies unchanged -- windows-2022 produces AMD64 CPython wheels for 3.10-3.13. The per-runner artifact name (cibw-wheels-${{ matrix.os }}) stays unique, so the publish job's `cibw-*` download+merge picks up the Windows wheels alongside the Linux/macOS ones. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL * Drop Python 3.9 from CI, add required 3.14 test coverage - test-pytest matrix: add 3.14 (with and without Cython). 3.13 was already present; both now run under the required `check` gate. Python 3.14 is GA (Oct 2025), so it needs no prerelease handling. - test-pypy: bump pypy3.9 -> pypy3.11 so the project no longer targets 3.9 while keeping a PyPy smoke lane (still non-required / continue-on-error). - requirements/test.txt: drop the 3.9 formatter fallbacks; the pins now install unconditionally since the minimum supported Python is 3.10. - pyproject.toml: add the 3.13 and 3.14 classifiers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…g#710) * docs: correct stale dependency and version references The user-facing docs still carried robinhood-era and pre-0.12 details: - `redis` extra listed the long-deprecated `aredis`; it is backed by `redis-py` now (since faust-streaming#635). - The "What do I need?" sidebar claimed Python 3.8 while the prose right below it already says 3.10 (the real minimum) - align it to 3.10. - Dropped two stale "Faust version 1.0" self-references. - Document the `faust[ckafka]` bundle (confluent-kafka / `confluent://` transport), which had no mention in the install docs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL * docs: refresh copyright year and document missing extras - Bump the docs copyright to 2026 and note the faust-streaming project's continued maintenance (docs/conf.py and copyright.rst). - Document the install bundles that existed but were undocumented: `faust[rocksdict]` (Stores), and `faust[prometheus]` / `faust[sentry]` (Sensors). Left `aerospike` out on purpose: it ships a requirements file but is not in setup.py's BUNDLES, so `faust[aerospike]` is not an installable extra. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…st-streaming#708) * docs: prepare v0.12.0 release notes and changelog Resume the Keep a Changelog format (dormant since v0.8.10) with a v0.12.0 section, and add standalone GitHub release notes covering the changes since v0.11.3 plus the pending fix stack. Highlights: two offset data-loss fixes (faust-streaming#606/faust-streaming#707, faust-streaming#316/faust-streaming#692), the re-added confluent-kafka driver, Python 3.14 support (3.8/3.9 dropped), OpenTracing/OpenTelemetry made optional, and a live-broker CI harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL * docs: drop closed codecov.yml PR (faust-streaming#683) from v0.12.0 notes PR faust-streaming#683 (codecov.yml with a 1% coverage threshold) was closed without merging, so remove it from the changelog and release notes to keep the v0.12.0 change list accurate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL * docs: derive Sphinx version from the package instead of a stale constant `docs/conf.py` hardcoded `version_dev='1.1'` / `version_stable='1.0'` - robinhood-era values that never matched faust-streaming's 0.x line, so the published GitHub Pages docs advertised the wrong version. Derive the documented major.minor from `faust.__version__` (which setuptools_scm resolves from the git tag), so the docs always report the real version and this can't silently drift between releases. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL * docs: scope v0.12.0 notes to merged work; note dependency updates Per review, the v0.12.0 changelog/release notes should describe only what is already on master, not work still in open PRs. - Remove the not-yet-merged items: the offset-commit data-loss fixes (faust-streaming#606/faust-streaming#707, faust-streaming#316/faust-streaming#692), the optional OpenTracing/OpenTelemetry extras (faust-streaming#685/faust-streaming#686, faust-streaming#688/faust-streaming#681), web_application_options (faust-streaming#704), and the reported-issue fix stack (faust-streaming#693-faust-streaming#703, faust-streaming#705). These will be added back as they merge. - Add a Dependencies section noting the current runtime/client libraries: mode-streaming >= 0.4.0, aiokafka >= 0.10.0 (compatible with recent 0.13/0.14 releases), the new confluent-kafka >= 2.0.0 for faust[ckafka], and the faust-cchardet fork replacing unmaintained cchardet. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL * docs: set v0.12.0 changelog date to 2026-07-19 Replace the UNRELEASED placeholder with the release date and drop the now-satisfied "set the date at tag time" note. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL * Delete RELEASE_NOTES_v0.12.0.md --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
v0.12.1 is a packaging-only patch: v0.12.0's wheels published but its sdist was rejected by PyPI under PEP 625. Add the changelog entry and standalone release notes for the fix (faust-streaming#712). Versioning is setuptools_scm-driven (the git tag is the version) and the docs version is derived from faust.__version__, so no version files need bumping; the GitHub Pages changelog picks this up via docs/changelog.rst's mdinclude of CHANGELOG.md, and the docs version banner updates from the tag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL
…streaming#712) PyPI rejected the v0.12.0 sdist upload: 400 Filename 'faust-streaming-0.12.0.tar.gz' is invalid, should be 'faust_streaming-0.12.0.tar.gz'. The release job built the sdist with the deprecated `setup.py sdist` under an old setuptools floor, which emits the legacy hyphenated distribution filename. PyPI now enforces PEP 625 and requires the normalized name (underscores). Build the sdist with `python -m build --sdist` instead, and raise the build-system setuptools floor to >=69 (which normalizes sdist filenames), so the artifact is named `faust_streaming-<version>.tar.gz`. The wheels were already fine (cibuildwheel emits the normalized name). Verified locally: `python -m build --sdist` now produces `faust_streaming-<version>.tar.gz`. Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…lease-v0.12.1-prep
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
bring master branch up-to-date with until the v0.12.1 of the upstream/master branch.
Everything afterwards is heavily AI generated and has to be reviewed separately.