CertSweep

A passive domain discovery and reconnaissance tool that finds real websites by TLD or geographic region. It uses public Certificate Transparency Logs, the Internet Archive, and urlscan.io — no intrusive scanning, no target interaction during discovery.

---

What it does

Given a TLD like .co or a region like latam, CertSweep:

1. Discovers domain names from three public sources in parallel
2. Verifies each domain with a DNS lookup and an HTTP/HTTPS probe
3. Reports which sites are live — including IP address, TLS status, HTTP status code, page title, and server header
4. Caches everything locally in SQLite so data accumulates over time and the tool keeps working even when remote sources are down

---

How it works internally

Phase 1 — Discovery

Three sources run in parallel via ThreadPoolExecutor:

Source	Method	What it finds
crt.sh	GET /?q=%.{tld}&output=json	All domains with a public SSL certificate for that TLD
Wayback Machine CDX API	GET /cdx/search/cdx?url=*.{tld}	Domains crawled by the Internet Archive (including HTTP-only and historical sites)
urlscan.io	GET /api/v1/search/?q=domain:*.{tld}	Additional coverage from browser-based scans (optional API key)

If the global crt.sh query doesn't reach the domain limit, a prefix fallback kicks in: it queries a%.{tld}, b%.{tld}, ..., z%.{tld} in parallel batches of 5 to extract more results.

crt.sh has a known instability problem — its PostgreSQL backend goes down under load. CertSweep handles this with patient retry logic: up to 4 attempts, 60s timeout per attempt, 30s wait between retries. While waiting, it shows the user the retry status in real time.

Phase 2 — Verification

Each discovered domain goes through:

1. DNS resolution — resolves the A record to get the IP. Domains that fail DNS are marked and skipped for HTTP.
2. HTTP probe — tries HTTPS first, falls back to HTTP. Follows redirects, reads up to 50KB of the response body, extracts the page <title> tag.
3. Parallelized — all domains are verified concurrently with a configurable thread pool (default: 15 workers).

Phase 3 — Cache

Results are saved to certsweep/cache.db (SQLite) automatically after every successful scan:

• Domain discovery is saved after Phase 1
• Verification results (IP, status, title, server) are saved after Phase 2
• INSERT OR IGNORE prevents duplicates — if a domain is already in the database, it is skipped without overwriting
• Cache is strictly isolated by TLD — querying .co only reads and writes .co entries, never .com or anything else
• If crt.sh is unavailable and no results come from any source, the tool falls back to the local cache for that exact TLD and shows previously discovered domains

Phase 1: crt.sh + Wayback + urlscan ──► merge ──► save to cache
                                                        │
                              if all sources fail ◄─────┘
                                        │
                              load from cache (same TLD only)

---

Requirements

• Python 3.10 or higher
• Internet connection (for discovery phases)
• No API keys required (urlscan.io works in free tier with reduced results)

Python dependencies

requests>=2.33.0
dnspython>=2.8.0
urllib3>=2.6.0
beautifulsoup4>=4.14.0
rich>=15.0.0

Install with:

pip install -r requirements.txt

---

Installation

git clone https://github.com/YoshiBigSmoke/certsweep.git
cd certsweep
pip install -r requirements.txt
python3 certsweep.py --help

Works on any system with Python 3.10+. Tested on Arch Linux with Python 3.13.

---

Usage

# Single TLD
python3 certsweep.py -t .co
python3 certsweep.py -t .jp

# Multiple TLDs in one run
python3 certsweep.py -t .ke .ng .za

# Full geographic region
python3 certsweep.py -r latam
python3 certsweep.py -r africa
python3 certsweep.py -r europe

# Generic and modern gTLDs
python3 certsweep.py -r generic       # .com .net .org .io .co .app .dev .ai
python3 certsweep.py -r new-gtld      # .xyz .online .site .tech .store ...

# Tune the scan
python3 certsweep.py -t .au -l 50     # 50 domains per TLD (default: 30)
python3 certsweep.py -t .au -w 20     # 20 parallel threads (default: 15)
python3 certsweep.py -t .au -o out.txt  # save live domains to a text file

# Cache
python3 certsweep.py --cache-info     # show what's stored locally
python3 certsweep.py -t .co --no-cache  # skip cache fallback (still saves new data)

Flags

Flag	Description	Default
-t	One or more TLDs to scan	—
-r	Predefined region	—
-l	Max domains to discover per TLD	30
-w	Parallel verification threads	15
-o	Save live results to a text file	—
--cache-info	Show local cache summary	—
--no-cache	Don't use cache as fallback	—
--list-regions	List all available regions	—

---

Available regions

Region	TLDs
africa	.za .ng .ke .gh .tz .ug .et .sn .cm .ci .ma .dz .tn .eg
asia	.jp .kr .th .vn .id .ph .my .sg .bd .lk .np .mm
latam	.mx .br .ar .co .cl .pe .ve .ec .bo .py .uy .gt .hn .sv .ni .cr .pa .do .cu
europe	.de .fr .es .it .pt .nl .be .pl .ro .cz .sk .hu .hr .rs .bg .ua .gr
middleeast	.sa .ae .ir .iq .sy .jo .lb .il .tr .pk
oceania	.au .nz .fj .pg .sb .vu .ws .to
generic	.com .net .org .io .co .app .dev .ai
new-gtld	.xyz .online .site .tech .store .shop .cloud .digital .world .space .life .live .media .news
usa	.us .gov .edu .mil
canada	.ca
uk	.uk .co.uk .org.uk .me.uk

---

Cache database

The cache is a SQLite file at certsweep/cache.db, created automatically on first run. Schema:

CREATE TABLE domains (
    tld         TEXT     NOT NULL,
    domain      TEXT     NOT NULL,
    ts          INTEGER  NOT NULL,   -- first discovery timestamp
    ip          TEXT,                -- resolved IP address
    alive       INTEGER,             -- 1 = responded, 0 = did not respond
    https       INTEGER,             -- 1 = HTTPS, 0 = HTTP only
    status      INTEGER,             -- HTTP status code
    title       TEXT,                -- page <title>
    server      TEXT,                -- Server response header
    verified_at INTEGER,             -- last verification timestamp
    PRIMARY KEY (tld, domain)
)

Query examples:

# All live .co domains with their IPs and titles
sqlite3 certsweep/cache.db \
  "SELECT domain, ip, status, title FROM domains WHERE tld='co' AND alive=1;"

# Domain count per TLD
sqlite3 certsweep/cache.db \
  "SELECT tld, COUNT(*) FROM domains GROUP BY tld ORDER BY COUNT(*) DESC;"

---

Data sources

All sources are public and free:

Source	API	Notes
crt.sh	https://crt.sh/?q=%.{tld}&output=json	Certificate Transparency aggregator by Sectigo
Wayback Machine	CDX API	Internet Archive crawl index
urlscan.io	REST API	Free tier works, set URLSCAN_API_KEY in the script for better results

---

Tested on

• OS: Arch Linux (kernel 6.x)
• Python: 3.13
• TLDs tested: .co, .mx, .jp, .au, .ke, .de, .io, .com

---

Disclaimer

CertSweep only queries public APIs and open datasets. It does not send requests to target domains during the discovery phase. HTTP verification in Phase 2 is a standard web request identical to a browser visit. This tool is intended for educational and research purposes.