drudl

A command-line tool for downloading Drupal site content with CAS authentication support. Enumerates content via /admin/content and converts pages to Markdown.

Features

• CAS Authentication: Automatically detects CAS authentication and opens a browser for interactive login
• Content Enumeration: Paginates through /admin/content to find all site content
• Markdown Conversion: Converts HTML pages to clean Markdown format
• Section Trimming: Configurable removal of navigation, menus, and other boilerplate sections
• Progress Indicator: Visual progress bar during download
• Safety First: Filters out dangerous action links (edit, delete, replicate, etc.)

Installation

# Clone the repository
git clone https://github.com/YOUR_USERNAME/drudl.git
cd drudl

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

Requirements

• Python 3.8+
• Chrome browser (for Selenium-based authentication)

Usage

Basic Usage

python drudl https://your-drupal-site.com -o output_directory

With Section Trimming

Remove specific sections from the downloaded Markdown:

# Using individual flags
python drudl https://example.com \
  --trim-section "## Footer" \
  --trim-section "## Main Menu" \
  --trim-section "## Sidebar"

# Using a file
python drudl https://example.com --trim-file trim_sections.txt

Trim File Format

Create a text file with one section name per line (matches any heading level):

# Comments start with hash
Footer
Main Menu
Navigation
Sidebar

Command-Line Options

Option	Description
url	Base URL of the Drupal site (required)
-o, --output	Output directory (default: downloaded_site)
--trim-section	Markdown header to remove (can be used multiple times)
--trim-file	File containing headers to trim, one per line

Environment Variables

Variable	Description
EXTRA_HTTP_HEADERS	JSON object of header name/value pairs sent with every request. Useful for Cloudflare or WAF bypass tokens.
DRUDL_NOVNC_PORT	Host port for the Docker noVNC desktop (default: 6080)
DRUDL_BYPASS_HEADER	Older single-header form, Header-Name: value. Still supported.

# CLI
export EXTRA_HTTP_HEADERS='{"x-wdsoit-bot-bypass": "true"}'
python drudl https://example.com -o output

# Docker
docker run -e 'EXTRA_HTTP_HEADERS={"x-wdsoit-bot-bypass": "true"}' -p 6080:6080 ...

For docker-compose, add the variable under environment in your docker-compose.yml.

Invalid JSON is ignored with a warning rather than failing the run, and only header names are printed since values may be secrets. Both forms may be set at once; they merge.

Shared conventions with bsp

EXTRA_HTTP_HEADERS uses the same name and JSON shape as bsp, so one setting configures the WAF bypass for both tools on the same site.

Port variables stay tool-prefixed on purpose — DRUDL_NOVNC_PORT here, BSP_NOVNC_PORT there — so the two can run side by side without their noVNC desktops colliding.

How It Works

1. Connection Test: Verifies the site is accessible
2. Authentication: If CAS authentication is detected, opens Chrome for interactive login
3. Cookie Transfer: Transfers authenticated session cookies from browser to requests
4. Enumeration: Paginates through /admin/content to collect all content URLs
5. Download: Fetches each page, converts to Markdown, and saves to output directory

Permissions Required

The authenticated user must have permission to access /admin/content. This typically requires:

• "Administer content" permission, or
• "Access content overview" permission

Security Notes

• The script only makes GET requests (never POST/DELETE)
• Dangerous action URLs are filtered out (edit, delete, replicate, unpublish, etc.)
• Uses a standard browser User-Agent to avoid bot detection
• Session cookies are stored in memory only (not persisted to disk)

Running Tests

# Local
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python -m pytest test_drudl.py -v

# Containerized
docker-compose -f docker-compose.test.yml run --rm test

conftest.py loads the extensionless drudl script as an importable module — the CLI is invoked as python drudl, so it carries no .py extension and a plain from drudl import ... cannot find it.

Docker

The Docker version includes a noVNC desktop for interactive CAS authentication.

Using docker-compose

1. Copy the example configuration:

   cp docker-compose.example.yml docker-compose.yml

2. Edit docker-compose.yml to set your target URL in the command line

3. Create output directory and run:

   mkdir -p output
   docker-compose up --build

4. Open http://localhost:6080 in your browser to view the Chromium window (or whichever port DRUDL_NOVNC_PORT is set to)

5. Complete CAS authentication when prompted

The download will resume automatically if interrupted. State is saved in output/.drudl_state.json.

Manual Docker commands

# Build the image
docker build -t drudl .

# Run with noVNC (access browser at http://localhost:6080)
mkdir -p output
docker run -p 6080:6080 -v $(pwd)/output:/app/output drudl \
  https://your-drupal-site.com -o /app/output

# With trim file
docker run -p 6080:6080 -v $(pwd)/output:/app/output \
  -v $(pwd)/trim_sections.txt:/app/trim_sections.txt \
  drudl https://your-drupal-site.com -o /app/output --trim-file /app/trim_sections.txt