Repository navigation
Fix JSearch's timeout; add blocked companies, a Firecrawl source, source fallback, and resume hyperlinks - #22
rahulrathore95 wants to merge 2 commits into
Conversation
Three changes on top of upstream, made for my own job search: - Fix: JSearch's hard-coded 10 s request timeout was shorter than its real-world latency (10.3 s observed against OpenWeb Ninja), so every search failed with "isn't answering right now" even with a working key. Raised to 25 s in src/lib/job-search/http.ts. - Blocked companies (Settings -> Job search): Find jobs, the job agent and the Claude connector's search_jobs/save_job_scores/save_application /draft_application never show, score or draft a posting that names a blocked company, matched against its company, title and description so a staffing listing for a role placed there is caught too. Filtered centrally in askSource() and the job agent's search step, with a defense-in-depth check in the connector tools for postings reached by posting_id/posting_url/paste_id instead of a search. - New job source, Firecrawl: one web search biased toward LinkedIn, Indeed, Naukri, Glassdoor, Monster, Greenhouse, Lever, Ashby, Workday and SmartRecruiters at once, instead of one API per board. Discovery only; posting details are fetched with the existing guarded fetcher, same as Adzuna. Needs a Firecrawl key (DOSSIER_FIRECRAWL_KEY or Settings -> Job search) and a Prisma migration for the new PostingSource enum value. Verified: yarn typecheck, yarn lint and yarn test (full suite, twice) all pass; the new Prisma migration applies cleanly; a production Docker image built from this Dockerfile starts, passes its health check and serves Settings -> Job search with Firecrawl and Blocked companies working end to end in a browser. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
new comment added |
Three more changes, on request:
- Falls back across job sources: searchWithFallback() in find.ts tries
the chosen source first, then every other source with a key saved
(JSearch, Adzuna, Firecrawl each keep their own, independently of
which one is chosen), in case the first is out of quota, down, or
doesn't cover the query's country. A source with no key configured is
skipped silently rather than counted as a failed attempt, so with
only one source set up (the common case) the error message is
unchanged from before. A kept search's "Show more" stays pinned to
its own source, since its cursor only means something there. Shared
by askSource (Find jobs, search_jobs) and the job agent, which
additionally excludes the Bundesagentur für Arbeit from the chain
(it never searches that source on its own, #223).
- Firecrawl now runs two searches per page instead of one: the existing
job-board-biased query, and the same query with no site restriction
at all, merged and de-duplicated. Catches a posting on a board outside
the fixed domain list, or a company's own careers page.
- Resume contact details link out: email (mailto:), phone (tel:),
website and LinkedIn (https://) were plain text in the header: ran
that profile is in Email, website and LinkedIn were plain text in the
resume's contact line; they're clickable links now, styled the same
muted, underline-free way as a publication's citation link already
was. ResumeContent.contacts changed shape (string[] -> {text, href}[])
for this; the cover letter's own sender block, which is free text the
user can edit, is untouched.
Also noted in the README: one-page resumes were already a setting
(Settings -> Documents -> Resume length), just not the default -
nothing to build there.
Verified: yarn typecheck, yarn lint and yarn test (full suite) all
pass, 1717 tests, 0 failures; resume snapshots regenerated and reviewed
(only the new <a> wrapping changed, nothing else moved); new tests for
the fallback chain (single-source behavior unchanged, falls back on
quota, combined message when everything fails, Show more stays pinned)
and the linked contact fields.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Hi @rahulrathore95, thank you for your contributions. I know Dossier is still missing features and has bugs, and I appreciate every contribution, including ones made with AI. But I can't accept a single PR this size that I can't review properly or test with confidence. Please split it into smaller PRs, one change each, small enough for a person to read and test:
The JSearch timeout fix is covered by today's release, so you can leave that one out. For bigger features like Firecrawl, please open an issue first so we can agree on the approach before you write the code. I'm closing this PR for now. Thanks again for taking the time. |
Summary
Changes made while using Dossier for my own job search, across two commits. Happy to split these into separate PRs if that's easier to review.
Fix: JSearch's 10 s timeout was shorter than its real-world latency. A live request against OpenWeb Ninja's JSearch API took 10.3 s to answer, so every search lost the race against
src/lib/job-search/http.ts's hard-coded 10 s limit and came back "JSearch isn't answering right now" even with a valid key. Raised to 25 s. (Deliberately kept as a plain constant, not read from the environment — an earlier version of this fix read it fromenvat module scope, which forced full environment validation on every module that imports job-search code, breaking ~15 unrelated test suites and any code path that registers job types without a job source configured. Caught and reverted during testing.)Feature: blocked companies (Settings → Job search). A list of companies that Find jobs, the job agent and the Claude connector's
search_jobs/save_job_scores/save_application/draft_applicationnever show, score or draft an application for again. Matched against a posting's company, title and description, so a staffing listing that only names the employer in the body text is still caught. Filtered centrally inaskSource()and the job agent's search step, with a defense-in-depth check in the connector tools for postings reached byposting_id/posting_url/paste_idinstead of a search.Feature: Firecrawl, a fourth job source. Runs two web searches per page — one biased toward LinkedIn, Indeed, Naukri, Glassdoor, Monster, Greenhouse, Lever, Ashby, Workday and SmartRecruiters, one with no site restriction at all for whatever that fixed list misses (a smaller board, a company's own careers page) — merged into one result, closer to what a person googling "data engineer jobs in Bengaluru" would see than Dossier maintaining a scraper per site. Discovery-only: posting details are fetched with the existing guarded
fetchPosting(), exactly like Adzuna's adapter. Needs a Firecrawl key (DOSSIER_FIRECRAWL_KEYor Settings → Job search) and one migration for the newPostingSourceenum value.Feature: falls back across job sources.
searchWithFallback()tries the source chosen in Settings first, then every other source with a key saved (each source keeps its own key independently of which is "chosen"), in case the first is out of quota, down, or doesn't cover the query's country. A source with no key configured is skipped silently rather than counted as a failed attempt, so with only one source set up — the common case — the error message is unchanged from before this existed. A kept search's "Show more" always stays pinned to the exact source it started with, since its pagination cursor only means something there. Shared byaskSource(Find jobs,search_jobs) and the job agent, which additionally excludes the Bundesagentur für Arbeit from its own fallback chain (it never searches that source unattended, #223).Feature: hyperlinks on the resume. Email, website and LinkedIn in the header were plain text; they're clickable links now (
mailto:,tel:,https://), styled the same muted, underline-free way a publication's citation link already was.ResumeContent.contactschanged shape (string[]→{text, href}[]) for this; the cover letter's own sender block, which is free text the user can edit, is untouched.Test plan
yarn typecheck,yarn lint,yarn test(full suite) all pass — run repeatedly for stability across both commits, 1717 tests, 0 failures20261028120000_firecrawl_source) applies cleanly against a fresh and an existing databaseDockerfilestarts, passes its health check, and keeps existing data and sessions intact across the swap — verified against both a fresh database and a real one already in use<a>wrapping changed, nothing else moved)🤖 Generated with Claude Code