Free, open-source pentest CLI for AI chatbots. Six attack classes. 37 probes. Single Python file you can read in 10 minutes.
pip install cveasy-chatbot-pentest
cveasy-chatbot-pentest --url https://your-bot/api/chat
Or just download and run:
curl -O https://raw.githubusercontent.com/CVEasy/cveasy-chatbot-pentest/main/cveasy_chatbot_pentest.py
python3 cveasy_chatbot_pentest.py --url https://your-bot/api/chat
Six attack classes that work against deployed AI chatbots regardless of the underlying model (GPT, Claude, Gemini, Llama, anything else):
| Class | Probes | What it tests |
|---|---|---|
| System prompt extraction | 8 | Whether the bot can be coerced into revealing its internal instructions. Leaked system prompts expose proprietary prompt engineering and the exact list of guardrails attackers then engineer around. |
| Direct prompt injection | 7 | Whether the bot will emit attacker-chosen output. Successful injection means the bot can be turned into an attack proxy. XSS payloads. Phishing in the bot's verified voice. |
| Encoding bypass | 6 | Whether the bot decodes and executes instructions hidden in base64, ROT13, hex, Unicode confusables, leetspeak, or reversed text. Most input-side keyword filters miss these. |
| Authority claiming | 5 | Whether the bot complies with claims of administrative, developer, or legal authority asserted in user input. Production bots that enforce guardrails via prompt text often comply. |
| Multi-language bypass | 5 | Whether instructions in Welsh, Basque, Esperanto, Hawaiian, or Tagalog slip past an English-trained safety classifier. The model usually still understands the instruction even when the filter does not. |
| Jailbreak via role-play | 6 | Whether role-play frames (DAN, developer mode, time travel, the grandmother attack, two-character role-play) defeat the bot's safety policy. |
Output is a styled HTML report with severity ratings (low/medium/high/critical) and an overall posture score (0 to 100). Also supports --format json for CI/CD pipelines and --format markdown for Slack/email.
# Basic scan
cveasy-chatbot-pentest --url https://your-bot/api/chat
# With auth
cveasy-chatbot-pentest --url https://your-bot/api/chat --auth-header "Bearer sk-..."
# Anthropic-style endpoint
cveasy-chatbot-pentest --url https://api.anthropic.com/v1/messages --endpoint-style anthropic --auth-header "x-api-key: ..."
# Custom request shape
cveasy-chatbot-pentest --url https://your-bot/chat --endpoint-style custom \
--body-template '{"input": "{{PROMPT}}", "session": "test"}'
# Quick mode (3 probes per class instead of all)
cveasy-chatbot-pentest --url https://your-bot/api/chat --mode quick
# Safe mode (skips jailbreak class for very conservative scans)
cveasy-chatbot-pentest --url https://your-bot/api/chat --mode safe
# JSON output for CI/CD
cveasy-chatbot-pentest --url https://your-bot/api/chat --format json --output report.json --ci
# Exit code 2 if any attack class triggers; 0 if all cleanFull options: cveasy-chatbot-pentest --help
Every Series B SaaS company has shipped an AI chatbot in the last 18 months. Burp Suite does not test prompt injection. Nessus does not test jailbreak. The OWASP Top 10 has an LLM list now but no productized scanner against it.
Garak (NVIDIA's research toolkit) covers most of this and more, but it has a learning curve and a heavy dependency tree. This tool is the pragmatic version. Single file. Read in 10 minutes. Run in 60 seconds. Free.
The paid edition, CVEasy AI, covers the other 9 attack classes (indirect prompt injection via RAG, tool/function abuse, cross-tenant leakage, cost exhaustion, hallucination weaponization, multi-turn coercion, side-channel timing, model fingerprinting, citation poisoning) plus the full 12-layer TRIS scoring framework, continuous monitoring, and audit-binder-grade reports.
GitHub Actions example:
name: AI Chatbot Pentest
on:
schedule:
- cron: '0 6 * * 1' # Mondays at 6am
workflow_dispatch:
jobs:
pentest:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: pip install cveasy-chatbot-pentest
- name: Pentest the chatbot
env:
BOT_AUTH: ${{ secrets.CHATBOT_AUTH_HEADER }}
run: |
cveasy-chatbot-pentest \
--url https://your-bot.staging.example.com/api/chat \
--auth-header "$BOT_AUTH" \
--format json \
--output pentest-report.json \
--ci
- uses: actions/upload-artifact@v4
if: always()
with:
name: pentest-report
path: pentest-report.jsonMore examples in examples/.
Default rate limit: 1 request/second. Default max tokens per probe: 250. Hard cap of 60 probes total per run. Aborts on 3 consecutive errors.
This is intentional. You should not run a chatbot pentest at full throttle against a production bot. Even with permission, you would rack up the target's LLM API bill and trigger rate-limit / WAF defenses.
Only run this tool against chatbots you own or have explicit written authorization to test. See SECURITY.md for the full responsible-use policy.
- Not a model evaluation framework. This tests deployments, not models. The same underlying model (Claude, GPT-4o, Gemini) can score very differently depending on how the deploying team configured the system prompt, guardrails, and output filters.
- Not a replacement for Burp Suite. Use both. Burp covers HTTP-layer pentest. This covers prompt-layer pentest. They are complements.
- Not a defensive tool. This is offense. For defense, look at Lakera Guard, Prompt Security, or built-in safety stacks from the model vendor.
- Not exhaustive. Six attack classes is a subset. The paid edition has fifteen. Even fifteen is not the whole landscape. New attack classes are being researched constantly; this is a snapshot of 2026 production-relevant patterns.
See CONTRIBUTING.md.
MIT. See LICENSE. Fork, modify, redistribute. Attribution appreciated.
Topher Boker (MIL4N). 10 patents filed on TRIS 12-layer scoring, BAS methodology, and AI chatbot pentest. Building CVEasy AI as part of BlueTeamAutomation.
MIL4N