An end-to-end data pipeline for analysing ~22,500 German IT job postings scraped from LinkedIn and Indeed. Raw postings flow through six components — scraping, ingestion, extraction, cleaning, analysis, and an interactive web dashboard — producing structured, analysis-ready data with 29 enriched columns per job. Extraction supports two modes: a default LLM mode (DeepSeek V3) and a fully deterministic --regex-only mode that costs nothing and runs in seconds.
poetry run python orchestrate.py
│
┌─────────────────────────┼─────────────────────────┐
▼ ▼ ▼
--dry-run (default) --regex-only
show plan LLM mode Regex mode
no changes $6–10, ~30 min $0, instant, deterministic
│ │
┌──────────────────────────────┴─────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────┐
│ Step 0 scrape opt-in (--scrape) │
│ Indeed (jobspy) + LinkedIn (guest API) │
│ → data/raw/scrape_YYYY-MM-DD/ │
├──────────────────────────────────────────────────────────────┤
│ Step 1 ingest Load 3 source CSVs + scrape_*/ │
│ Schema unification, date parsing │
│ → data/ingestion/combined_jobs.csv │
│ (22,526 rows · 7 cols) │
├──────────────────────────────────────────────────────────────┤
│ Step 2 prepare Input validation, location parse, │
│ title normalization, non-DE filter │
├──────────────────────────────────────────────────────────────┤
│ Step 3 deduplicate Filter flagged rows, URL exact, │
│ composite key, SHA-256 grouping, │
│ MinHash LSH near-dup grouping │
├──────────────────────────────────────────────────────────────┤
│ Step 4 regex_extract Tier 1: 8 deterministic fields │
│ (salary, contract, modality, │
│ experience, seniority, languages, │
│ education) │
│ --regex-only: also Tier 2 (skills, │
│ benefits, soft skills, job_family) │
├──────────────────────────────────────────────────────────────┤
│ Step 5 extract LLM mode only (skipped --regex-only)│
│ (LLM) DeepSeek V3, async + semaphore, │
│ ~10k unique descriptions │
├──────────────────────────────────────────────────────────────┤
│ Step 6 validate LLM mode only (skipped --regex-only)│
│ Hallucination detect, cross-field │
│ checks, salary sanity, C++ fix │
├──────────────────────────────────────────────────────────────┤
│ Step 7 clean_enrich Merge Tier 1 + Tier 2, NA fix, │
│ city normalize, categorical remap, │
│ benefit/soft-skill enrichment │
├──────────────────────────────────────────────────────────────┤
│ Step 8 export Drop internals, reorder columns, │
│ assert 9 invariants │
│ → data/cleaning/cleaned_jobs[_regex].csv │
├──────────────────────────────────────────────────────────────┤
│ Step 9 export_light Drop description/flags/title │
│ → data/cleaning/*_light.csv │
└──────────────────────────────────────────────────────────────┘
│
▼
11 Jupyter notebooks · 35 charts
Interactive web dashboard (React 19)
- Python 3.11+ installed
- Poetry for dependency management
- DeepSeek API key for LLM extraction (Step 5 only)
- python-jobspy (optional) — required for Indeed scraping; if not installed, Indeed is automatically skipped and LinkedIn continues
# Clone the repository
git clone <repo-url>
cd extractors
# Install dependencies via Poetry (from repo root)
poetry install# Linux/Mac
export DEEPSEEK_API_KEY=sk-...
# Windows PowerShell
$env:DEEPSEEK_API_KEY = "sk-..."Never hardcode the API key. Use environment variables or a
.envfile.
# Default pipeline — uses existing raw data (scraping skipped)
poetry run python orchestrate.py
# Full pipeline including scraping (fetch fresh data first)
poetry run python orchestrate.py --scrape
# Quick scrape test (4 keywords, 2 cities, 7 days)
poetry run python orchestrate.py --scrape-test --only scrape
# Or use Make
make pipelinecd notebooks/
poetry run jupyter notebook
# Open any notebook (01–11) and run all cells| Command | Description |
|---|---|
poetry run python orchestrate.py |
Run all pending steps (auto-resume, scraping skipped) |
poetry run python orchestrate.py --scrape |
Include scraping step (fetch fresh data before pipeline) |
poetry run python orchestrate.py --full |
Alias for --scrape — run everything including scraping |
poetry run python orchestrate.py --scrape-test |
Quick scrape test (4 keywords, 2 cities, 7 days) |
poetry run python orchestrate.py --only scrape |
Run only the scraping step |
poetry run python orchestrate.py --only ingest |
Run only the ingest step |
poetry run python orchestrate.py --only extract |
Run only the extract step (LLM) |
poetry run python orchestrate.py --from validate |
Resume from validate onwards |
poetry run python orchestrate.py --dry-run |
Show what would run (no changes) |
poetry run python orchestrate.py --only export_light |
Run only the light export step |
poetry run python orchestrate.py --list |
List all steps with descriptions |
poetry run python orchestrate.py --reset |
Clear progress and start fresh |
poetry run python orchestrate.py --regex-only |
Extract ALL fields via regex ($0, instant, deterministic) |
poetry run python orchestrate.py --no-llm |
Alias for --regex-only (backward compat) |
Incremental processing: After scraping new data (
data/raw/scrape_*/), just re-runpoetry run python orchestrate.py— no--resetneeded. The pipeline auto-detects new scrape CSVs, invalidates steps from ingest onward, and re-processes only new rows. Expensive steps (location parsing, MinHash dedup) use cached outputs from previous runs, so only the ~50 new rows go through the full pipeline. Previously LLM-extracted rows are skipped via checkpoint DB ($0 cost). Existing cleaned output is backed up (timestamped.bak) before overwriting. Use--resetto force a full re-run (clears caches).
| Target | Description |
|---|---|
make pipeline |
Run all 9 core steps (scrape is opt-in, see make targets in Makefile) |
make regex-only |
Run with regex-only extraction ($0 API cost) |
make ingest / prepare / deduplicate / regex-extract / extract / validate / clean-enrich / export |
Run a single step |
make resume-from-<step> |
Resume from a given step onward |
make dry-run / make list-steps |
Plan inspection |
make analyze |
Launch the Jupyter notebooks |
make test / make lint / make type-check |
Dev checks |
# Run all tests (from repo root)
poetry run pytest tests/ -v
# Run tests for a specific domain
poetry run pytest tests/scraping/ -v
poetry run pytest tests/extraction/ -v
poetry run pytest tests/cleaning/ -v
poetry run pytest tests/ingestion/ -v
poetry run pytest tests/analysis/ -v
poetry run pytest tests/integration/ -v
# Lint
poetry run ruff check . --fix
# Type check
poetry run mypy src/extraction/ src/shared/ --ignore-missing-imports
# Check pipeline progress
poetry run python orchestrate.py --dry-runextractors/ ← git root
│
├── orchestrate.py ← MAIN ENTRY POINT (9-step pipeline)
├── pyproject.toml ← Single Poetry config + all dependencies
├── Makefile ← Make targets for all operations
│
├── src/ ← ALL source packages
│ ├── pipeline_state.py ← Typed state container
│ ├── scraping/ ← Opt-in job board scrapers (Indeed + LinkedIn)
│ │ ├── pipeline.py, indeed_scraper.py, linkedin_scraper.py
│ │ ├── normalizer.py, checkpoint.py
│ │ └── config/ ← settings.yaml (keywords, cities, board config)
│ ├── ingestion/ ← Schema normalization + date parsing
│ │ ├── pipeline.py, loader.py, date_parser.py
│ │ └── config/ ← settings.yaml (source definitions)
│ ├── extraction/ ← LLM extraction pipeline
│ │ ├── preprocessing/ ← Input validation, text/title/location normalization
│ │ ├── dedup/ ← URL dedup + description near-duplicate grouping
│ │ ├── llm/ ← DeepSeek client, async processor, parser
│ │ ├── validators/ ← Skill hallucination, cross-field, salary checks
│ │ ├── reporting/ ← Cost, quality, evaluation reports
│ │ ├── checkpoint.py, exporter.py, post_extraction.py
│ │ └── config/ ← YAML prompts, JSON schemas, settings
│ ├── cleaning/ ← Post-extraction data quality + enrichment
│ │ ├── pipeline.py, skill_normalizer.py, benefit_categorizer.py, ...
│ │ └── config/ ← Category mappings (benefits, skills, cities)
│ ├── analysis/ ← Chart helpers + notebook utilities
│ │ └── utils.py, style.py, charts.py, filters.py, compute.py
│ ├── steps/ ← Thin orchestrator wrappers (10 step modules)
│ │ └── scrape.py, ingest.py, prepare.py, ..., export.py, export_light.py
│ └── shared/ ← Cross-component utilities
│ └── io.py, config.py, schemas.py, constants.py, logging.py, ...
│
├── tests/ ← ALL tests, structured by domain
│ ├── scraping/ ← 53 tests
│ ├── extraction/ ← 760+ tests
│ ├── cleaning/ ← 75+ tests
│ ├── ingestion/ ← 6 tests
│ ├── analysis/ ← Analysis utility tests
│ └── integration/ ← End-to-end + step smoke tests
│
├── data/ ← ALL pipeline I/O, scoped by domain
│ ├── raw/ ← Source CSVs (read-only, or produced by --scrape)
│ ├── scraping/ ← Scraper checkpoints (resume support)
│ ├── ingestion/ ← combined_jobs.csv
│ ├── extraction/ ← batches/, deduped/, extracted/, reports/
│ ├── cleaning/ ← cleaned_jobs[_regex].csv + light variants
│ └── analysis/ ← figures/
│
├── notebooks/ ← 11 Jupyter analysis notebooks (01–11)
│
├── website/ ← INTERACTIVE WEB DASHBOARD (React 19 + TS)
│ ├── src/pages/ ← 14 page components
│ ├── src/components/ ← Layout, charts, shared UI
│ ├── src/hooks/ ← Data fetching, filters, theme
│ ├── src/lib/ ← Aggregation engine, explore, colors
│ ├── src/types/ ← TypeScript interfaces + enums
│ └── public/data/ ← Pre-aggregated JSON + jobs-slim.json
│
├── scripts/
│ └── generate_dashboard_data.py ← Pipeline → dashboard JSON exporter
│
└── backlog/ ← Planning docs + archive
Each component has its own documentation file (e.g. EXTRACTION.md, CLEANING.md).
scraping/config/settings.yaml 37 keywords, 34 cities, board config
│
│ [Step 0: Scrape — opt-in, requires --scrape flag]
│ Indeed (via python-jobspy) + LinkedIn (guest API)
│ Writes to data/raw/scrape_YYYY-MM-DD/
▼
data/raw/*.csv 3+ CSVs, 22,500+ raw postings
│
│ [Step 1: Ingest]
│ Schema normalization, date parsing
▼
data/ingestion/combined_jobs.csv 22,526 rows · 7 columns
│
│ [Step 2: Prepare]
│ Input validation, location parsing, title normalization
│ Non-German rows filtered out
▼
~19,000 rows · 7+ columns
│
│ [Step 3: Deduplicate]
│ Filter privacy-wall + invalid-URL rows
│ URL exact, composite key, description grouping, MinHash LSH
▼
~18,500 rows (description groups formed)
│
│ [Step 4: Regex Extract]
│ Tier 1: 8 regex fields (salary, contract, modality, etc.)
│ --regex-only: ALSO Tier 2 (skills, benefits, soft skills, job family)
▼
+ 8 regex_* columns
+ extraction_results_regex.json (if --regex-only)
│
├──── (default — LLM mode) ─────────────────────────────────────┐
│ │
│ [Step 5: Extract — LLM] │
│ DeepSeek V3 API (async, semaphore-limited) │
│ ~10,000 unique descriptions (grouped to save ~20% cost) │
│ Extracts: skills, benefits, tasks, job_family, summary │
▼ │
+ 7 LLM-extracted fields │
│ │
│ [Step 6: Validate] │
│ Skill hallucination detection + removal │
│ Cross-field consistency, salary sanity │
▼ │
Corrections applied │
│ │
├──── (--regex-only) Steps 5-6 SKIPPED ─────────────────────────┘
│ Regex results flow directly to Step 7
│
│ [Step 7: Clean + Enrich]
│ Merge Tier 1 + Tier 2 results (LLM or regex)
│ City normalization, benefit/soft skill categorization
│ Description quality flagging
▼
~18,500 rows · 29 columns
│
│ [Step 8: Export]
│ 9 invariant checks, column ordering
▼
data/cleaning/cleaned_jobs.csv LLM mode: ~18,500 rows · 29 columns
data/cleaning/cleaned_jobs_regex.csv Regex mode: same structure, _regex suffix
│
│ [Step 9: Export Light]
│ Drop description, validation_flags, description_quality, title
▼
data/cleaning/cleaned_jobs_light.csv LLM mode (lightweight)
data/cleaning/cleaned_jobs_regex_light.csv Regex mode (lightweight)
│
│ [Analysis Notebooks + Web Dashboard]
▼
data/analysis/figures/*.png 35 charts across 11 notebooks
| Technology | Version | Purpose |
|---|---|---|
| Python | 3.11+ | Runtime |
| Poetry | latest | Dependency management, virtual environment |
| pandas | ^3.0 | DataFrame processing, CSV handling |
| DeepSeek V3 | API | LLM for semantic field extraction (OpenAI-compatible) |
| OpenAI SDK | ^2.21 | Client library for DeepSeek API (compatible protocol) |
| SQLite | built-in | Checkpoint database for crash-safe pipeline resume |
| Technology | Purpose |
|---|---|
| PyYAML | Configuration files (prompts, schemas, mappings) |
| jsonschema | LLM response validation against JSON Schema |
| datasketch | MinHash LSH for near-duplicate description detection |
| pandera | DataFrame schema validation at step boundaries |
| python-jobspy | Indeed scraper backend (optional — Indeed skipped if not installed) |
| Technology | Purpose |
|---|---|
| Jupyter | Interactive analysis notebooks |
| matplotlib | Chart rendering (35 charts across 11 notebooks) |
| seaborn | Statistical visualization theming |
| Technology | Purpose |
|---|---|
| pytest | Test framework (per-component test suites under tests/) |
| ruff | Fast Python linter (line-length 100, E/F/W/I rules) |
| mypy | Static type checking (strict for key modules) |
| tqdm | Progress bars for LLM extraction |
- Scraping is skipped by default — the pipeline starts from existing raw CSVs
- Use
--scrapeto fetch fresh data from Indeed + LinkedIn before running the pipeline - Output goes to a dated folder (
data/raw/scrape_YYYY-MM-DD/) — original raw data is never overwritten - Indeed requires
python-jobspy(optional dep); if missing, Indeed is automatically skipped and LinkedIn continues - Why: Separates data collection from data processing. Most runs reuse existing data; scraping is expensive and slow
- Default (LLM): Tier 1 via regex (8 fields) + Tier 2 via DeepSeek V3 API (7 fields). Best quality, $6–10/run, ~30 min.
--regex-only(alias--no-llm): ALL fields via dictionary-based regex. $0, instant, deterministic. Lower recall (~250-skill dictionary), higher precision (no hallucinations). Skips Steps 5–6 and writes to_regex-suffixed files so it never overwrites LLM output.- Why: Avoids wasting LLM calls on fields regex handles reliably; regex-only mode enables CI/testing and environments without API access.
- Identical/near-identical descriptions are grouped (SHA-256 hash + MinHash LSH at 95% Jaccard similarity)
- Only one representative per group is sent to the LLM; results are fanned out to all members
- Saves ~20% of LLM API calls (~4,192 duplicate descriptions across 22,500 postings)
- The extraction prompt explicitly forbids skill inference: "ONLY extract a skill if its name appears literally in the description"
- A post-extraction skill verifier cross-checks 100% of extracted skills against the description text
- Skills not found in the description are removed (not just flagged)
- In-memory: Python
None(never the string"NA") - On disk (CSV):
"NA"string - Boundary:
read_csv_safe()converts"NA"→Noneon read;write_csv_safe()convertsNone→"NA"on write - Why: Prevents pandas from silently converting legitimate data to NaN
- SQLite checkpoint tracks per-row stage progress (atomic commits)
- JSON progress file tracks per-step completion with timestamps
- Pipeline resumes automatically from last completed step/row on restart
- Salary:
50.000= 50,000 EUR (German thousands separator) - Titles: Gender suffix patterns stripped —
(m/w/d),(m,w,d),(gn),(all genders),mwd,:in,*in,/-in - Locations: German city/state parsing, state abbreviations (BY, BW, NRW)
- Languages: Bilingual prompt handles German + English job descriptions
- Benefits: German-specific keywords (Urlaub, Gleitzeit, Bahncard, etc.)
The final data/cleaning/cleaned_jobs.csv contains:
| Column | Type | Description |
|---|---|---|
row_id |
string | Unique identifier (SHA-256 of job_url) |
job_url |
string | URL to original posting |
date_posted |
string | ISO date (YYYY-MM-DD) |
company_name |
string | Hiring company |
city |
string | Parsed German city |
state |
string | German federal state |
title |
string | Original job title |
title_cleaned |
string | Normalized title (gender suffixes removed, EN translation) |
job_family |
enum | One of 40 canonical IT role categories (incl. Other) |
job_summary |
string | LLM-generated role summary |
seniority_from_title |
string | Junior / Mid / Senior / Lead |
contract_type |
string | Full-time / Part-time / Contract / Freelance |
work_modality |
string | Remote / Hybrid / On-site |
salary_min |
float | Annual salary floor (EUR) |
salary_max |
float | Annual salary ceiling (EUR) |
experience_years |
int | Required years of experience |
education_level |
string | Highest degree required |
technical_skills |
JSON[] | Required technologies. LLM returns {name, source} evidence objects (max 35); after cleaning, the CSV stores a flat list of canonical skill names |
soft_skills |
JSON[] | Interpersonal skills (flat strings, max 20) |
nice_to_have_skills |
JSON[] | Optional technologies (same shape as technical_skills) |
benefits |
JSON[] | Tangible perks offered (LLM returns {name, source}; CSV stores flat names) |
tasks |
JSON[] | Main responsibilities — short English phrases, max 10 (max_tasks in settings.yaml) |
languages |
JSON[] | Language requirements with CEFR levels |
benefit_categories |
JSON[] | Categorized benefits (11 categories) |
soft_skill_categories |
JSON[] | Categorized soft skills (8 categories) |
description_quality |
string | "clean" or "concatenated" |
site |
string | "indeed" or "linkedin" |
validation_flags |
JSON[] | Quality flags from validation |
description |
string | Full job description text |
| # | Notebook | Focus | Charts |
|---|---|---|---|
| 01 | Market Overview | Dataset shape, job families, top companies, posting trends | 4 |
| 02 | Skills Demand | Top skills, skill×job family heatmap, co-occurrence, req vs nice-to-have | 4 |
| 03 | Salary Analysis | Distribution, by job family, by seniority, by city, experience correlation | 5 |
| 04 | Location Insights | Jobs by state/city, city×role heatmap, remote by state | 4 |
| 05 | Remote Work | Modality split, remote by role, by seniority, language correlation | 4 |
| 06 | Seniority & Experience | Seniority distribution, experience by role, junior cities, salary premium | 4 |
| 07 | Benefits Landscape | Top categories, benefits×role heatmap, vacation prevalence, size correlation | 4 |
| 08 | Language Requirements | German % by role, CEFR levels, English-only roles, language×modality | 4 |
| 09 | Job Seeker Guide | 4 personas (Junior/Senior/Remote/Career-Changer), skills roadmap, cities, salary | 2 + tables |
| 10 | Role Deep Dives | 6 target roles: skills, salary, progression, benefits | charts + tables |
| 11 | Education & Soft Skills | Education by family/seniority/salary, soft skill heatmaps | charts + tables |
Total: 35+ charts across 11 notebooks.
Live: it-in-de.web.app
In addition to the Jupyter notebooks, the project includes a full-featured web dashboard built with React 19 and TypeScript. It provides an interactive, filterable interface for exploring the same dataset in a browser.
Pipeline Output Web Dashboard
─────────────── ─────────────
cleaned_jobs.csv ──> generate_dashboard_data.py ──> public/data/*.json
(~19K rows) (pre-aggregate) (static + slim)
│
▼
React SPA (14 pages)
├── 10 analytics pages
├── 2 deep dive pages
└── 2 advanced exploration
Zero backend — Python generates static JSON files; React fetches and computes everything client-side. Deployed on Cloudflare Workers.
| Route | Page | Description |
|---|---|---|
/ |
Overview | Market snapshot — jobs, companies, remote %, salary coverage, trends |
/skills |
Skills | Top 25 skills, required vs optional, salary premium, skill×family heatmap |
/salary |
Salary | P25/median/P75 ranges by role, seniority, and city |
/location |
Location | Interactive Germany choropleth map, top cities, modality by state |
/remote |
Remote Work | Remote/hybrid/on-site breakdown by role and seniority |
/seniority |
Seniority | Seniority distribution with by-family heatmap |
/benefits |
Benefits | 11 benefit categories, by-family heatmap |
/languages |
Languages | German/English % by role, CEFR level distribution |
/education |
Education | Education levels by family/seniority, soft skill heatmaps |
/companies |
Companies | Top 20 hiring companies, job family diversity |
/roles |
Role Deep Dives | 6 target roles — skills, salary progression, seniority, top cities |
/personas |
Personas | 4 job seeker profiles with tailored recommendations |
/explore |
Advanced Info | 7 interactive tabs (salary estimator, skill matcher, role comparison, etc.) |
/egg |
Easter Egg | Advanced Info without paywall gate |
The Advanced Information page provides real-time interactive analysis computed client-side on the full ~19K job dataset:
| Tab | What It Does |
|---|---|
| By Role | Select a role → full market profile (skills, salary, cities, companies, seniority) |
| By City | Select a city → local IT market breakdown |
| By Skills | Select your skills → personalized role recommendations with match % and skills to learn |
| Salary Estimator | Filter by role + city + seniority → salary percentiles from real data |
| No-German Finder | Jobs without German requirement — ideal for international professionals |
| Skill Combos | Most common skill pairs with salary correlation |
| Role vs Role | Side-by-side comparison of any two job families |
All analytics pages support 6 filter dimensions (role, city, state, seniority, modality, contract type) encoded in the URL:
- No filters active → Displays pre-aggregated static JSON (instant page loads)
- Filters active → Loads full dataset (
jobs-slim.json, ~12MB) and computes aggregations in real-time on the client
| Technology | Purpose |
|---|---|
| React 19 + TypeScript 5.9 | UI framework |
| TanStack Router + Query | Client-side routing + data caching |
| Tailwind CSS 4.2 + shadcn/ui | Styling + component library |
| Recharts 3.8 | Bar, pie, and line charts |
| react-simple-maps | Interactive Germany choropleth |
| Vite 6.4 | Build tooling |
| Cloudflare Workers | Deployment |
cd website/
# Install dependencies
npm install
# Development server
npm run dev
# Production build
npm run build
# Generate dashboard data (from repo root)
python scripts/generate_dashboard_data.pyFull documentation: website/DASHBOARD.md
| Component | Documentation | Description |
|---|---|---|
| Scraping | src/scraping/SCRAPING.md | Opt-in Indeed + LinkedIn scrapers, checkpoint resume, dated output |
| Ingestion | src/ingestion/INGESTION.md | Schema normalization, date parsing, configuration |
| Extraction | src/extraction/EXTRACTION.md | Full pipeline (9 steps), three-mode extraction (LLM / regex-only / legacy) |
| Cleaning | src/cleaning/CLEANING.md | 12 cleaning + 5 enrichment steps, 9 invariant checks |
| Analysis | src/analysis/ANALYSIS.md | 11 notebooks, chart types, visualization gallery |
| Shared | src/shared/SHARED.md | IO utilities, constants, schemas, logging |
| Dashboard | website/DASHBOARD.md | React web app — 14 pages, 7 advanced tabs, interactive filters |
| Raw Data | data/raw/RAW_DATA.md | Raw data sources, CSV schemas, column mapping |
This project was developed as part of TechLabs Data Science coursework.