Skip to content

Proposal: Firecrawl as a fourth job source #28

Closed

Description

@rahulrathore95

Following up on #22 — splitting out the smaller changes into their own PRs (#25, #26, #27), and opening this issue first for Firecrawl specifically, as you asked, since it's a bigger change and adds a paid dependency.

The idea: JSearch and Adzuna each index one source. Firecrawl instead runs a real web search, so Dossier doesn't need a bespoke scraper per job board (which breaks the moment a site's markup changes, and risks its terms of use). Concretely:

  • A fourth JobSourceAdapter, discovery-only: it returns postings from a search, and reuses the existing guarded fetchPosting() for each posting's full text, exactly the way Adzuna's adapter already does.
  • Two Firecrawl /v2/search calls per page — one query biased toward a fixed list of job-board domains (LinkedIn, Indeed, Naukri, Glassdoor, Greenhouse, Lever, Ashby, Workday, SmartRecruiters, …), one with no site restriction at all — merged and de-duplicated by URL. The open query catches a posting on a board outside that list, or a company's own careers page; the biased one keeps results relevant when the open query is noisy.
  • Needs a Firecrawl API key (DOSSIER_FIRECRAWL_KEY, or saved in Settings → Job search, same pattern as the other sources' keys).
  • One Prisma migration, for the new PostingSource enum value.
  • Participates in the source-fallback chain from Fall back across job sources when one can't answer #27 like any other source, once that's in.

Trade-offs worth flagging up front:

  • It's the first job source that costs money per search rather than offering a free tier sized for personal use (Firecrawl has a free trial, then pay-per-search).
  • It's the first source where Dossier doesn't control the result shape at all — whatever Firecrawl's search index returns, including sites Dossier wouldn't otherwise touch — so the usual "Dossier only talks to sources you choose" framing needs a sentence acknowledging that a web search is broader by nature.

A working reference implementation already exists on my fork (rahulrathore95/dossier, src/lib/job-search/firecrawl.ts, with tests) if you want to look at the actual code before we agree on the approach. Happy to adjust the design based on your take, then open the PR once we're aligned.

Activity

  1. rahulrathore95 commented on Oct 28, 2026

    @rahulrathore95
    Author

    Closing — I'm going to keep working on my own fork rather than contributing this upstream. Thanks for engaging with it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions