diff --git a/AGENTS.md b/AGENTS.md index a59671e..98acbc4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,7 +14,8 @@ Guidance for AI coding agents working in this repository. - `tests/zero_bounce_test_case.py`: scripted unit tests. They patch `zerobouncesdk.zerobouncesdk.requests`. - `tests/test_offline_edges.py`, `tests/test_warmup_offline.py`, `tests/test_security_offline.py` and `tests/test_streaming_and_regions_offline.py`: more offline suites on the same principle (error bodies and every optional field; the Warmup API; where keys travel and that they never reach an exception; streamed downloads and the regional-host warning). - `tests/test_offline_api.py`: offline suite. It patches `requests.sessions.Session.send`, so the real `requests` client builds the URL and body and the fake answers without the network. -- `tests/zero_bounce_integration_test.py`: live API tests. `pytest` does not collect this file. Run it with `python -m unittest tests.zero_bounce_integration_test` when `ZEROBOUNCE_API_KEY` is set. +- `tests/live_api_test.py`: opt-in live suite that calls every method of both clients (`ZEROBOUNCE_API_KEY`, `ZEROBOUNCE_WARMUP_API_KEY`). Free calls by default; `ZEROBOUNCE_LIVE_SPEND=1` adds credit-spending calls and bulk round trips, `ZEROBOUNCE_LIVE_CHANGES=1` reversible account changes. `pytest` and CI never run it: `python -m unittest tests.live_api_test -v`. +- `tests/zero_bounce_integration_test.py`: the original live tests. `pytest` does not collect this file. Run it with `python -m unittest tests.zero_bounce_integration_test` when `ZEROBOUNCE_API_KEY` is set. - `pyproject.toml`: the only version field, `version = "x.y.z"`. `setup.py` calls `setup()` and reads that file. ## Commands @@ -50,7 +51,7 @@ CI (`.github/workflows/sdk_ci.yml`) runs `ruff check`, `compileall`, then `pytho - **Do not commit `.env`.** It is gitignored and may hold a real key. Tests must not call the live API. - Match the surrounding style. Do not reformat unrelated code. -- **The live API wins over the docs.** Where they disagree, follow what the API accepts and say so in the docstring. Known cases: `validate_batch` `timeout` must be a JSON string; `send_evaluated_file` takes the evaluator's `file_id`, not an upload; S3 / SFTP jobs need `return_url` (and `validateSFTPfile` needs `file_name`); Warmup `check-domain-dmarc` needs an email address, not a domain. Check a new endpoint against the live API (`tests/zero_bounce_integration_test.py` style, sandbox addresses where possible) before release. +- **The live API wins over the docs.** Where they disagree, follow what the API accepts and say so in the docstring. Known cases: `validate_batch` `timeout` must be a JSON string; `send_evaluated_file` takes the evaluator's `file_id`, not an upload; S3 / SFTP jobs need `return_url` (and `validateSFTPfile` needs `file_name`); Warmup `check-domain-dmarc` needs an email address, not a domain. Add every new method to `tests/live_api_test.py` (free calls by default, behind `ZEROBOUNCE_LIVE_SPEND` / `ZEROBOUNCE_LIVE_CHANGES` when it spends credits or changes the account, never anything the API cannot undo) and run it before a release. ## Changelog @@ -65,7 +66,13 @@ CI (`.github/workflows/sdk_ci.yml`) runs `ruff check`, `compileall`, then `pytho Publishing and pushing tags need a maintainer's explicit approval. Do not bump the version, tag, or publish unless they asked. +**Every release needs a manual live API test run first.** The offline suites cannot catch what only the live API rejects (a `timeout` type, a required field the docs call optional). The run needs real keys, which only a maintainer has, so it is a manual step: + +- Tell the maintainer as early as possible, at the start of any release work and as soon as a change adds or alters an API call, that a live run is required, which keys it needs (`ZEROBOUNCE_API_KEY`, plus `ZEROBOUNCE_WARMUP_API_KEY` when Warmup changed), and which flags (`ZEROBOUNCE_LIVE_SPEND=1` when bulk or credit-spending methods changed, `ZEROBOUNCE_LIVE_CHANGES=1` when filters or Warmup sending accounts changed). Do not leave it for the moment before tagging. +- Do not tag or publish until the run has passed with every flag the change calls for. Record the result (date, flags, pass or failures) in the release pull request or the release notes. +- Never write a key into a file, a commit, a log, or a message. Pass keys only as environment variables for the run. + 1. Move `[Unreleased]` entries under `## [x.y.z] - YYYY-MM-DD`, leave an empty `[Unreleased]` section, and update the compare links at the bottom of `CHANGELOG.md`. 2. Set the same version in `pyproject.toml` (`version = "x.y.z"`). The publish workflow fails if it differs from the tag. That is the only version field. There is no runtime `__version__` and no user-agent constant. `setup.py` does not contain a version. -3. `ruff check src tests`, `python -m compileall -q src tests`, and `python -m pytest`. +3. `ruff check src tests`, `python -m compileall -q src tests`, and `python -m pytest`; then the manual live run (see above): `python -m unittest tests.live_api_test -v` with the keys and flags the release needs. A failure blocks the release. 4. Tag `vx.y.z` on that commit and run the Publish workflow (`.github/workflows/publish.yml`) with that tag. It publishes `zerobouncesdk` to PyPI through PyPI Trusted Publishing: the PyPI project trusts this repository, `publish.yml`, environment `release`. There is no PyPI token, and none should be added. Keep the workflow file name and environment name, or update the trusted publisher on PyPI first. diff --git a/README.md b/README.md index 704b67a..5db899c 100644 --- a/README.md +++ b/README.md @@ -630,14 +630,28 @@ python -m pytest `python -m pytest` runs every offline suite: the scripted unit tests and the `tests/test_*.py` suites, which answer through a fake transport and never call the API. CI runs the same commands, with coverage. ### Live API tests -`tests/zero_bounce_integration_test.py` calls the real API and is not collected by `pytest`. Run it with a valid key: +Two suites call the real API. Neither is collected by `pytest` or run in CI; without a key their tests are skipped. + +`tests/live_api_test.py` calls every SDK method, both clients, to check the API accepts each payload and the responses parse: ```bash -export ZEROBOUNCE_API_KEY= -python -m unittest tests.zero_bounce_integration_test -v +export ZEROBOUNCE_API_KEY= # validation API +export ZEROBOUNCE_WARMUP_API_KEY= # Warmup API (optional) +python -m unittest tests.live_api_test -v ``` -Without `ZEROBOUNCE_API_KEY` its tests are skipped. It covers the original single-request methods; the newer APIs are tested offline only. +By default it makes only free calls (sandbox addresses, reads, and requests expected to fail, such as unknown file IDs or S3 / SFTP without an integration) and fails if the credit balance changed. Opt in to more: + +- `ZEROBOUNCE_LIVE_SPEND=1`: spends credits on scoring, activity, the email finder, and full bulk round trips (upload, wait, download, delete) for validation, scoring, email finder, domain search and the list evaluator. Every job is deleted afterwards. +- `ZEROBOUNCE_LIVE_CHANGES=1`: reversible account changes, undone in cleanup even when a test fails: a test allowlist filter and a self-managed Warmup sending account. + +It never creates an Inbox Placement test, sets its webhook, or starts ZB-managed Warmup onboarding: the API cannot undo those. + +`tests/zero_bounce_integration_test.py` is the original, smaller live suite for the single-request methods: + +```bash +python -m unittest tests.zero_bounce_integration_test -v +``` ### Run tests with Docker ```bash diff --git a/tests/live_api_test.py b/tests/live_api_test.py new file mode 100644 index 0000000..8aca72d --- /dev/null +++ b/tests/live_api_test.py @@ -0,0 +1,381 @@ +"""Opt-in live API suite: calls every SDK method against the real API to check payloads and response shapes. + +Not collected by ``pytest`` and not run by CI. Run it explicitly: + + export ZEROBOUNCE_API_KEY=... # validation API tests (skipped without it) + export ZEROBOUNCE_WARMUP_API_KEY=... # Warmup API tests (skipped without it) + python -m unittest tests.live_api_test -v + +By default only free calls run: sandbox addresses (no credits), reads, and requests that are expected to +fail (unknown file IDs, S3 / SFTP without an integration) to check the API accepts the payload. The free +tests check that the credit balance did not change. Two flags opt in to more: + + ZEROBOUNCE_LIVE_SPEND=1 spends credits: scoring, activity, email finder, and full bulk round trips + (upload, wait, download, delete) for validation, scoring, email finder, + domain search and the list evaluator + ZEROBOUNCE_LIVE_CHANGES=1 reversible account changes, each undone in cleanup even when a test fails: + a test allowlist filter, and a self-managed Warmup sending account + +Never called, because they cannot be undone through the API: creating an Inbox Placement test (it cannot be +deleted while running), setting the Inbox Placement webhook (it cannot be cleared), and ZB-managed Warmup +onboarding (it starts mailbox provisioning). +""" + +import io +import os +import tempfile +import time +import unittest +import uuid +import warnings +from datetime import date, timedelta + +from zerobouncesdk import ( + ZBApiException, + ZBApiUrl, + ZBClientException, + ZBGetFileOptions, + ZBValidateBatchElement, + ZBValidateStatus, + ZeroBounce, + ZeroBounceWarmup, +) + +API_KEY = os.environ.get("ZEROBOUNCE_API_KEY", "").strip() +WARMUP_KEY = os.environ.get("ZEROBOUNCE_WARMUP_API_KEY", "").strip() +SPEND = os.environ.get("ZEROBOUNCE_LIVE_SPEND") == "1" +CHANGES = os.environ.get("ZEROBOUNCE_LIVE_CHANGES") == "1" + +# A well-formed ID that matches no file: a malformed one can be blocked by Cloudflare instead of answered. +UNKNOWN_ID = "0d1b7a4e-1111-2222-3333-444455556666" +CALLBACK = "https://example.com/zerobounce-sdk-live-test" +JOB_TIMEOUT_S = 300 + + +def _wait(check, timeout=JOB_TIMEOUT_S, every=10): + """Call ``check`` until it returns a truthy value; return it, or None after ``timeout`` seconds.""" + deadline = time.time() + timeout + while time.time() < deadline: + value = check() + if value: + return value + time.sleep(every) + return None + + +@unittest.skipUnless(API_KEY, "ZEROBOUNCE_API_KEY not set") +class FreeLiveTest(unittest.TestCase): + """Free calls only; the credit balance must not change.""" + + @classmethod + def setUpClass(cls): + cls.zb = ZeroBounce(API_KEY) + cls.credits_before = int(cls.zb.get_credits().credits) + + @classmethod + def tearDownClass(cls): + after = int(cls.zb.get_credits().credits) + if after != cls.credits_before: + raise AssertionError(f"free live tests spent credits: {cls.credits_before} -> {after}") + + def assertPayloadAccepted(self, call, *rejections): + """The call may fail, but not because the API rejected the payload's fields.""" + try: + call() + except ZBApiException as error: + message = str(error).lower() + for text in ("field is required", "missing param", "not permitted", "invalid request format", *rejections): + self.assertNotIn(text, message, f"payload rejected: {error}") + + # -- account ------------------------------------------------------------------------------ + + def test_credits_and_usage_get_and_post(self): + self.assertGreaterEqual(int(self.zb.get_credits(use_post=True).credits), 0) + end = date.today() + for use_post in (False, True): + usage = self.zb.get_api_usage(end - timedelta(days=30), end, use_post=use_post) + self.assertIsInstance(usage.total, int) + self.assertEqual(usage.end_date, end) + + def test_regions(self): + for region in (ZBApiUrl.API_USA_URL, ZBApiUrl.API_EU_URL): + self.assertGreaterEqual(int(ZeroBounce(API_KEY, base_url=region).get_credits().credits), 0) + + def test_regional_base_url_warns_about_the_bulk_host(self): + client = ZeroBounce(API_KEY, base_url=ZBApiUrl.API_EU_URL) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + with self.assertRaises(ZBApiException): + client.file_status(UNKNOWN_ID) + self.assertEqual(sum("regional host" in str(w.message) for w in caught), 1) + + # -- validation (sandbox addresses are free) ---------------------------------------------- + + def test_validate_every_option_get_and_post(self): + for use_post in (False, True): + result = self.zb.validate("valid@example.com", ip_address="203.0.113.7", timeout=30, activity_data=True, + verify_plus=False, domain_info=True, use_post=use_post) + self.assertEqual(result.status, ZBValidateStatus.valid) + self.assertIsInstance(result.mx_found, bool) + self.assertIsNotNone(result.domain_website_exists) + self.assertIsNotNone(result.processed_at) + + def test_validate_greylist_options_are_accepted(self): + for kwargs in ({"greylist_processing": True}, {"greylist_processing": True, "callback_url": CALLBACK}): + result = self.zb.validate("unknown@example.com", **kwargs) + self.assertTrue(result.status is not None or result.message, kwargs) + + def test_validate_batch_with_every_option(self): + # timeout goes out as a string: the API rejects the batch when it is a JSON number. + result = self.zb.validate_batch( + [ZBValidateBatchElement("valid@example.com"), ZBValidateBatchElement("invalid@example.com", "203.0.113.7")], + timeout=30, activity_data=True, verify_plus=False, + ) + self.assertEqual([e.status for e in result.email_batch], [ZBValidateStatus.valid, ZBValidateStatus.invalid]) + + def test_email_finder_and_domain_without_a_result_are_free(self): + # Charged only when a format is found; example.com has none. + for use_post in (False, True): + found = self.zb.find_email_format(domain="example.com", use_post=use_post) + self.assertEqual(found.domain, "example.com") + self.zb.find_domain(domain="example.com", use_post=use_post) + self.zb.guess_format("example.com", first_name="John") + + def test_filters_list_get_and_post(self): + self.assertIsInstance(self.zb.list_filters(), list) + self.assertIsInstance(self.zb.list_filters(use_post=True), list) + + # -- inbox placement tester (read-only) --------------------------------------------------- + + def test_inbox_placement_reads(self): + providers = self.zb.get_inbox_placement_providers() + self.assertTrue(providers["emailProviders"]["items"]) + self.assertTrue(self.zb.get_inbox_placement_test_types()["testTypes"]) + listing = self.zb.list_inbox_placement_tests(status="finished", created_after="2026-01-01", page=1, page_size=5) + self.assertIn("results", listing) + self.assertIn("webhookUrl", self.zb.get_inbox_placement_webhook()) + + # -- payloads checked through expected failures ------------------------------------------- + + def test_unknown_files_fail_with_a_short_api_error(self): + out = os.path.join(tempfile.mkdtemp(), "never.csv") + for call in ( + lambda: self.zb.file_status(UNKNOWN_ID), + lambda: self.zb.scoring_file_status(UNKNOWN_ID), + lambda: self.zb.email_finder_file_status(UNKNOWN_ID), + lambda: self.zb.domain_search_file_status(UNKNOWN_ID), + lambda: self.zb.get_file(UNKNOWN_ID, out, ZBGetFileOptions(download_type="combined", activity_data=True)), + lambda: self.zb.get_file(UNKNOWN_ID, out, stream=True), + lambda: self.zb.scoring_get_file(UNKNOWN_ID, out), + lambda: self.zb.email_finder_get_file(UNKNOWN_ID, out), + lambda: self.zb.domain_search_get_file(UNKNOWN_ID, out, stream=True), + lambda: self.zb.delete_file(UNKNOWN_ID), + lambda: self.zb.scoring_delete_file(UNKNOWN_ID), + lambda: self.zb.email_finder_delete_file(UNKNOWN_ID), + lambda: self.zb.domain_search_delete_file(UNKNOWN_ID), + lambda: self.zb.get_list_evaluator_result(UNKNOWN_ID), + lambda: self.zb.send_evaluated_file(UNKNOWN_ID), + ): + with self.assertRaises(ZBApiException) as caught: + call() + self.assertLess(len(str(caught.exception)), 300) + self.assertNotIn("