1.5.17: Fail the canary probe on an empty track instead of transcribing it

[samson-art]

Closes #59

What and why

The canary is the periodic test request of the HTTP server. It finds out whether captions still come back from the platform. The probe in src/canary.ts calls validateAndDownloadSubtitles for one video with type: "official", lang: "en" and skipCache. The review of #50 found two faults in this path. Both are older than #50.

1. With speech-to-text on (WHISPER_MODE is local or api), an empty track on the canary video started the speech-to-text fallback. When the fallback answered, the call resolved with source: 'whisper'. The canary counted that answer as a working caption path. It set transcriptor_canary_ok to 1 while captions failed, and it ended an open failure streak.
2. In handleExplicitRequestFlow (src/validation.ts), a speech-to-text job that ran longer than WHISPER_TIMEOUT wrote its answer to the cache later. That late write ignored skipCache. So a late answer to the probe went into the cache as the official English track of the canary video. Calls that asked for that track by name then got speech-to-text text instead of captions until the entry expired (CACHE_TTL_SUBTITLES_SECONDS, 7 days by default).

Two tests showed both faults before the fix (see Verified).

The fix:

• validateAndDownloadSubtitles gets a second option, skipWhisper. handleExplicitRequestFlow starts speech-to-text only when skipWhisper is not set and WHISPER_MODE is not off. The same value goes to whisperTried, so the "no subtitles" text does not say that speech-to-text ran. The canary passes { skipCache: true, skipWhisper: true }. An empty track is now a failed probe: the gauge goes to 0 and the streak goes on.
• The late write returns early when skipCache is set. This is one guard in the shared function. Real calls do not set skipCache, so they keep the late write.

Only the canary passes these options. The 5 other callers of validateAndDownloadSubtitles (2 in src/index.ts, 3 in src/mcp-core.ts) pass none, and both options are off by default. The tool contract, the caller texts of real calls, metric names and labels, and the per-call log line do not change. No caption request is added. With speech-to-text on, a failed probe no longer downloads audio or runs a transcription.

Commits: 2dcf21d (the fix and the tests), e19a890 (review 1, text only).

Decisions

The maintainer took these decisions on 2026-09-26, in the brief:

• The probe skips the speech-to-text fallback entirely. An empty or missing track on the canary video is a failed probe: the gauge goes to 0 and the streak goes on. This answers the open question of the issue: skip the fallback (cheaper), and do not run it and ignore its answer.
• Fix the root cause of fault 2 in handleExplicitRequestFlow: the late speech-to-text write after WHISPER_TIMEOUT obeys skipCache.
• The canary makes no extra caption requests (ADR 003). ADR 003 changes where it describes the probe and speech-to-text.
• AC 1 changes from "a probe that returns source: 'whisper' counts as a failure" to "a probe whose track is empty never starts speech-to-text, counts as a failure, sets the gauge to 0 and keeps the streak going".
• The branch is fix/1.5.17-canary-no-whisper. The entries go under ## [Unreleased].

Acceptance criteria

• AC 1: a probe whose track is empty never starts speech-to-text, counts as a failure, sets the gauge to 0 and keeps the streak going. Tests:
  • src/validation.test.ts: does not start speech-to-text for a probe whose track is empty. It sets WHISPER_MODE to local, gives an empty track and a speech-to-text job that answers, and passes the options of the canary. It expects NotFoundError, no call to startOrReuseWhisperJob, and an error text that does not name speech-to-text.
  • src/canary.test.ts: counts an empty track as a failure when speech-to-text is on. Two ticks give transcriptor_canary_ok 0 and one canary: transcript path failing with reason not_found.
  • src/canary.test.ts: probes CANARY_URL with one explicit language and skips the cache and speech-to-text.
• AC 2: a skipCache call whose speech-to-text job finishes after WHISPER_TIMEOUT writes nothing to the cache. Test in src/validation.test.ts: does not write a late speech-to-text answer to the cache when skipCache is set. The existing test should call cache.set when Whisper finishes after WHISPER_TIMEOUT (explicit lang) still passes, so real calls keep the late write.
• The canary makes no extra caption requests, and ADR 003 describes the new probe (brief). No request is added. ADR 003 is updated (see Plan).

Plan

Files that change

• src/validation.ts: the options of validateAndDownloadSubtitles get skipWhisper?: boolean. A doc comment says that the options apply only to a request that names lang. handleExplicitRequestFlow takes { skipCache, skipWhisper } instead of skipCache alone. The value transcribe (skipWhisper not set and WHISPER_MODE not off) decides the fallback and whisperTried. The late write returns early when skipCache is set.
• src/canary.ts: the probe passes skipWhisper: true. The header comment says that an empty track is a failed probe.
• src/canary.test.ts: one test renamed and changed to expect skipWhisper, and one new test.
• src/validation.test.ts: two new tests.
• docs/adr/003-caption-request-budget.md: the Decision says that the probe never falls back to speech-to-text, that an empty track is a failed probe, and that the probe does not read a cached transcript and does not store one. The stale sentence "When Whisper answers the probe, the run includes the whole transcription" is removed. A new Do-not rule says not to let speech-to-text answer the probe. The Date and Sources lines name The canary counts a speech-to-text answer as a working caption path, and its late result fills the cache #59.
• CHANGELOG.md: two entries under [Unreleased], Fixed.

No env var changes, so .env.example and the README do not change.

Order of work

1. Wrote the three new tests and changed the canary test for the probe options. Saved the red run (red-59.txt): 2 tests failed in each suite.
2. Added skipWhisper, the transcribe value and the skipCache guard. Passed skipWhisper from the canary. Updated ADR 003 and the CHANGELOG. Commit 2dcf21d.
3. Review 1 found two minor points and no unmet criteria. First, ADR 003 said that the probe does not read or write the cache. That is false for a failed probe. The "no subtitles" answer reads the track list, and on a cache miss one metadata run fills the avail, info and chapters entries. Second, the CHANGELOG said that real calls for the canary video got speech-to-text text. Only calls that named the official English track read it, because auto-discovery skips a speech-to-text answer stored under a track name (test does not take a speech-to-text answer stored under a track name for that track). Fixed in e19a890: text only, no code change.

Risks

• With speech-to-text on, an empty probe track no longer counts in subtitles_extraction_failures_total{reason="no_subtitles"}. That counter counts a failure only when speech-to-text ran (1.5.13). The CHANGELOG says so. Names and labels do not change.
• A failed probe still reads the track list in throwNoSubtitlesError. On a cache miss, this is one metadata run. The probe made this run before with speech-to-text off. With speech-to-text on, the failed probe now makes this run instead of an audio download.
• The fix does not remove a speech-to-text answer that a late write stored before the deploy. That entry stays until its TTL ends. See After merge.

Verified

• Red before the fix (an excerpt of red-59.txt, on 152a609 with the new tests):

  $ npx jest src/canary.test.ts -w 2 -t "empty track|skips the cache and speech-to-text"
    ✕ probes CANARY_URL with one explicit language and skips the cache and speech-to-text
    ✕ counts an empty track as a failure when speech-to-text is on
      Expected pattern: /^transcriptor_canary_ok\{[^}]*\} 0$/m
      Received string:  "# HELP http_requests_total Total HTTP requests
  Tests:       2 failed, 14 skipped, 16 total
  
  $ npx jest src/validation.test.ts -w 2 -t "late speech-to-text answer|probe whose track is empty"
    ● ... › does not write a late speech-to-text answer to the cache when skipCache is set
      expect(jest.fn()).not.toHaveBeenCalled()
      Received number of calls: 1
      1: "sub:https://www.youtube.com/watch?v=dQw4w9WgXcQ:auto:en:srt", "{...\"source\":\"whisper\"}", 604800
    ● ... › does not start speech-to-text for a probe whose track is empty
      Expected constructor: NotFoundError
      Received constructor: Object
  Tests:       2 failed, 118 skipped, 120 total
  

• Green after the fix, on 2dcf21d:

  • npx tsc --noEmit -p .: no errors.
  • npx jest src/canary.test.ts -w 2: 16 of 16.
  • npx jest src/validation.test.ts -w 2: 120 of 120.
  • npx eslint on the 4 changed .ts files: no problems.

• On e19a890: npx jest src/canary.test.ts src/validation.test.ts -w 2 with a name filter for the acceptance-criteria tests: 7 passed.

• make check-no-smoke (format-check, lint, typecheck, full Jest, build) passed in the pre-commit hook on 2dcf21d and on e19a890. On 2dcf21d it ran twice: on the first commit, and on an amend that restored the ADR 003 italics after a prettier run changed them. The test count of the gate was not kept.

• Mutation drill on 2dcf21d: 4 of 4 mutations fail a named test. The file was restored after each mutation.

  Mutation	Tests that failed
  src/canary.ts: remove skipWhisper: true from the probe call.	probes CANARY_URL with one explicit language and skips the cache and speech-to-text and counts an empty track as a failure when speech-to-text is on (2 of 16 in canary.test.ts).
  src/validation.ts: const transcribe = whisperConfig.mode !== 'off' (ignore skipWhisper).	does not start speech-to-text for a probe whose track is empty (1 of 120 in validation.test.ts).
  src/validation.ts: whisperTried: whisperConfig.mode !== 'off' instead of transcribe.	does not start speech-to-text for a probe whose track is empty, because the error text says that speech-to-text was tried (1 of 120).
  src/validation.ts: the late write guard goes back to if (!text?.trim()) (no skipCache).	does not write a late speech-to-text answer to the cache when skipCache is set (1 of 120).

• After review 1, mutations 1, 2 and 4 ran again on e19a890 and failed the same tests (3 of 3). The review fix changes only text, so it has no mutation of its own.

Not verified

• make check, smoke, e2e and load tests: not run. They call real YouTube from this IP.
• A real speech-to-text run and a real Redis: not used. The tests mock downloadSubtitles, startOrReuseWhisperJob, getWhisperConfig and the cache set.
• A real canary probe on an empty track: not done. The canary video has an official English track, so an empty track needs a platform failure to happen.
• The full Jest suite by hand: not run. It ran only inside the pre-commit gate, because one heavy Jest run at a time keeps the machine in memory.

Not in scope

• The canary video and its interval (the issue says so).
• The metadata run in throwNoSubtitlesError on a failed probe, and the avail, info and chapters entries that it fills. It was there before with speech-to-text off, and this PR does not add it. To remove it, pass the track list to throwNoSubtitlesError, or skip the list when skipCache is set. The plan puts it out of scope, and the review did not ask for it.
• skipCache and skipWhisper for a request without lang. handleAutoDiscoverFlow ignores them, as it did before. The canary always names lang, and the doc comment on the options says so.
• A cleanup of a speech-to-text answer that a late write stored before this deploy. See After merge, step 4.

After merge

After the deploy, on the MCP HTTP server. Only the HTTP server runs the canary.

1. Make sure that transcriptor_canary_ok is 1 and that transcriptor_canary_last_success_timestamp_seconds moves forward. The canary video has an official English track, so on a normal day the probe gets a track and nothing changes.
2. Find out whether the deployment sets WHISPER_MODE to local or api. If it does not, neither fault happens there, and only step 1 applies.
3. If speech-to-text is on: when the canary logs canary: transcript fetch failed or canary: transcript path failing with reason not_found, whisper_requests_total does not grow at the time of that probe. The gauge stays at 0 until a track comes back.
4. If speech-to-text is on and CACHE_MODE=redis: read the entry sub:<canary URL>:official:en:<format>. The canary URL is CANARY_URL, or the default in src/canary.ts. The format is YT_DLP_SUB_FORMAT, or srt by default. If its JSON has "source":"whisper", a late write from an old probe can have stored it. Delete that one key, or wait until its TTL ends.

🤖 Generated with Claude Code