Skip to content

Fix JSearch's timeout; add blocked companies, a Firecrawl source, source fallback, and resume hyperlinks - #22

Closed
rahulrathore95 wants to merge 2 commits into
AienTech:mainfrom
rahulrathore95:main
Closed

rahulrathore95 wants to merge 2 commits into
AienTech:mainfrom
rahulrathore95:main

Conversation

@rahulrathore95

@rahulrathore95 rahulrathore95 commented Oct 28, 2026 •

Copy link
Copy Markdown

Summary

Changes made while using Dossier for my own job search, across two commits. Happy to split these into separate PRs if that's easier to review.

Fix: JSearch's 10 s timeout was shorter than its real-world latency. A live request against OpenWeb Ninja's JSearch API took 10.3 s to answer, so every search lost the race against src/lib/job-search/http.ts's hard-coded 10 s limit and came back "JSearch isn't answering right now" even with a valid key. Raised to 25 s. (Deliberately kept as a plain constant, not read from the environment — an earlier version of this fix read it from env at module scope, which forced full environment validation on every module that imports job-search code, breaking ~15 unrelated test suites and any code path that registers job types without a job source configured. Caught and reverted during testing.)

Feature: blocked companies (Settings → Job search). A list of companies that Find jobs, the job agent and the Claude connector's search_jobs/save_job_scores/save_application/draft_application never show, score or draft an application for again. Matched against a posting's company, title and description, so a staffing listing that only names the employer in the body text is still caught. Filtered centrally in askSource() and the job agent's search step, with a defense-in-depth check in the connector tools for postings reached by posting_id/posting_url/paste_id instead of a search.

Feature: Firecrawl, a fourth job source. Runs two web searches per page — one biased toward LinkedIn, Indeed, Naukri, Glassdoor, Monster, Greenhouse, Lever, Ashby, Workday and SmartRecruiters, one with no site restriction at all for whatever that fixed list misses (a smaller board, a company's own careers page) — merged into one result, closer to what a person googling "data engineer jobs in Bengaluru" would see than Dossier maintaining a scraper per site. Discovery-only: posting details are fetched with the existing guarded fetchPosting(), exactly like Adzuna's adapter. Needs a Firecrawl key (DOSSIER_FIRECRAWL_KEY or Settings → Job search) and one migration for the new PostingSource enum value.

Feature: falls back across job sources. searchWithFallback() tries the source chosen in Settings first, then every other source with a key saved (each source keeps its own key independently of which is "chosen"), in case the first is out of quota, down, or doesn't cover the query's country. A source with no key configured is skipped silently rather than counted as a failed attempt, so with only one source set up — the common case — the error message is unchanged from before this existed. A kept search's "Show more" always stays pinned to the exact source it started with, since its pagination cursor only means something there. Shared by askSource (Find jobs, search_jobs) and the job agent, which additionally excludes the Bundesagentur für Arbeit from its own fallback chain (it never searches that source unattended, #223).

Feature: hyperlinks on the resume. Email, website and LinkedIn in the header were plain text; they're clickable links now (mailto:, tel:, https://), styled the same muted, underline-free way a publication's citation link already was. ResumeContent.contacts changed shape (string[] → {text, href}[]) for this; the cover letter's own sender block, which is free text the user can edit, is untouched.

Test plan

  • yarn typecheck, yarn lint, yarn test (full suite) all pass — run repeatedly for stability across both commits, 1717 tests, 0 failures
  • The new Prisma migration (20261028120000_firecrawl_source) applies cleanly against a fresh and an existing database
  • A production image built from this Dockerfile starts, passes its health check, and keeps existing data and sessions intact across the swap — verified against both a fresh database and a real one already in use
  • Verified in a real browser against a live instance: chose Firecrawl as the source, saved and tested a key, added and removed blocked companies, confirmed persistence across a reload
  • Resume snapshots regenerated and reviewed by hand (only the new <a> wrapping changed, nothing else moved)
  • New unit/integration tests: the blocklist matcher, the Firecrawl adapter (including its two-search merge), the fallback chain (single-source behavior unchanged, falls back on quota, combined message when everything fails, "Show more" stays pinned), and the linked contact fields — plus updates to every existing test the new fields/sources touch

🤖 Generated with Claude Code

Three changes on top of upstream, made for my own job search:

- Fix: JSearch's hard-coded 10 s request timeout was shorter than its
  real-world latency (10.3 s observed against OpenWeb Ninja), so every
  search failed with "isn't answering right now" even with a working
  key. Raised to 25 s in src/lib/job-search/http.ts.

- Blocked companies (Settings -> Job search): Find jobs, the job agent
  and the Claude connector's search_jobs/save_job_scores/save_application
  /draft_application never show, score or draft a posting that names a
  blocked company, matched against its company, title and description so
  a staffing listing for a role placed there is caught too. Filtered
  centrally in askSource() and the job agent's search step, with a
  defense-in-depth check in the connector tools for postings reached by
  posting_id/posting_url/paste_id instead of a search.

- New job source, Firecrawl: one web search biased toward LinkedIn,
  Indeed, Naukri, Glassdoor, Monster, Greenhouse, Lever, Ashby, Workday
  and SmartRecruiters at once, instead of one API per board. Discovery
  only; posting details are fetched with the existing guarded fetcher,
  same as Adzuna. Needs a Firecrawl key (DOSSIER_FIRECRAWL_KEY or
  Settings -> Job search) and a Prisma migration for the new
  PostingSource enum value.

Verified: yarn typecheck, yarn lint and yarn test (full suite, twice)
all pass; the new Prisma migration applies cleanly; a production Docker
image built from this Dockerfile starts, passes its health check and
serves Settings -> Job search with Firecrawl and Blocked companies
working end to end in a browser.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@AienTech

AienTech commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner

Hi, thanks for the contribution. I'll take a look at it and will let you know. so far, having separate PRs for this is better. since JSearch's timeout is already being released today.

new comment added

Three more changes, on request:

- Falls back across job sources: searchWithFallback() in find.ts tries
  the chosen source first, then every other source with a key saved
  (JSearch, Adzuna, Firecrawl each keep their own, independently of
  which one is chosen), in case the first is out of quota, down, or
  doesn't cover the query's country. A source with no key configured is
  skipped silently rather than counted as a failed attempt, so with
  only one source set up (the common case) the error message is
  unchanged from before. A kept search's "Show more" stays pinned to
  its own source, since its cursor only means something there. Shared
  by askSource (Find jobs, search_jobs) and the job agent, which
  additionally excludes the Bundesagentur für Arbeit from the chain
  (it never searches that source on its own, #223).

- Firecrawl now runs two searches per page instead of one: the existing
  job-board-biased query, and the same query with no site restriction
  at all, merged and de-duplicated. Catches a posting on a board outside
  the fixed domain list, or a company's own careers page.

- Resume contact details link out: email (mailto:), phone (tel:),
  website and LinkedIn (https://) were plain text in the header: ran
  that profile is in Email, website and LinkedIn were plain text in the
  resume's contact line; they're clickable links now, styled the same
  muted, underline-free way as a publication's citation link already
  was. ResumeContent.contacts changed shape (string[] -> {text, href}[])
  for this; the cover letter's own sender block, which is free text the
  user can edit, is untouched.

Also noted in the README: one-page resumes were already a setting
(Settings -> Documents -> Resume length), just not the default -
nothing to build there.

Verified: yarn typecheck, yarn lint and yarn test (full suite) all
pass, 1717 tests, 0 failures; resume snapshots regenerated and reviewed
(only the new <a> wrapping changed, nothing else moved); new tests for
the fallback chain (single-source behavior unchanged, falls back on
quota, combined message when everything fails, Show more stays pinned)
and the linked contact fields.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@rahulrathore95 rahulrathore95 changed the title Fix JSearch's too-short timeout; add blocked companies and a Firecrawl job source Fix JSearch's timeout; add blocked companies, a Firecrawl source, source fallback, and resume hyperlinks Oct 28, 2026
@AienTech

Copy link
Copy Markdown
Owner

Hi @rahulrathore95, thank you for your contributions. I know Dossier is still missing features and has bugs, and I appreciate every contribution, including ones made with AI. But I can't accept a single PR this size that I can't review properly or test with confidence.

Please split it into smaller PRs, one change each, small enough for a person to read and test:

  • blocked companies
  • Firecrawl as a job source
  • falling back across job sources
  • hyperlinks on the resume

The JSearch timeout fix is covered by today's release, so you can leave that one out. For bigger features like Firecrawl, please open an issue first so we can agree on the approach before you write the code.

I'm closing this PR for now. Thanks again for taking the time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants