DictQuery returns 404 for every word: EDRDG JMdictDB is now behind a site-wide password gate #54

Re-diagnosed 2026-09-14. The original root cause (name="e" → name="g" markup change) is superseded — see Superseded diagnosis at the bottom for the original text. The endpoint is still broken, but for a different and more fundamental reason, and the fix proposed in the first version of this issue would not have worked.

Summary

POST /api/DictQuery/ returns 404 No results found for every word. The cause is not a markup change: the entire https://www.edrdg.org/jmwsgi/ tree (JMdictDB) is now behind a site-wide password gate, so the scraper never reaches a results page at all.

The gate returns HTTP 200 with an access-request page, so httpx raises nothing, BeautifulSoup finds 0 rows, and api/dict_query.py:125-130 emits its own 404. The endpoint reports "no results" when the real condition is "access denied".

Evidence

Probed 2026-09-14. Every /jmwsgi/ path returns the same JMdictDB - Site Access page:

The gate page itself states the terms:

To use the JMdictDB Dictionary Database you need to have a password. Once the password has been entered, it will be set in your browser as a cookie, and will not need to be entered again until it changes.
Currently the password consists of four digits representing the current month.

So it is a cookie set by POST /jmwsgi/access.py, keyed on a value that rotates monthly.

Why the previously proposed fix does not apply

The original fix was to swap the e= selector for g= in three places in api/dict_query.py. That code is still unchanged (api/dict_query.py:66 is still row.find("input", {"name": "e"})), but changing it would fix nothing — both the e= and g= entry URLs now hit the same access wall, as the table above shows. The parser never gets far enough for the selector to matter.

More importantly, the gate is a deliberate anti-automation measure on a rotating credential. Programmatically defeating it is not an option worth pursuing: it would mean re-implementing a monthly-rotating login against a server that has explicitly asked us not to, and it would break again on the operator's next change. JMdictDB has to be abandoned as a runtime dependency.

Secondary defect (still valid, now the more damaging one)

The hardening note from the original report is now the main observable problem: a 200 with 0 parsed rows is indistinguishable from a genuine empty result, so a total upstream loss surfaces as a routine 404. This is why the outage was mis-diagnosed as a markup change in the first place.

Whatever the data-source decision, the parser should distinguish no results from parse/access failure and log the latter as an error.

Note also that no test covers DictQuery (tests/ has no dict_query module), so nothing in CI would have caught this.

Options

A. Bake JMdict locally (recommended)

Drop the scrape and ship the dictionary, matching what MarkAccent already does — the README describes the accent pipeline as "fully local and offline… no network calls", and this endpoint is the last thing contradicting that.

Measured 2026-09-14:

Against the image's existing ~1.3 GB UniDic footprint this is negligible (<1%). <ent_seq> maps directly onto the existing WordResult.id, and <k_ele> / <r_ele> / <sense> map onto kanji / furigana / definitions, so the response schema is preservable.

Licence: JMdict is CC BY-SA 4.0 (EDRDG licence statement, §"Creative Commons Attribution-ShareAlike Licence (V4.0)"). Commercial use is permitted with attribution + ShareAlike — unlike tdmelodic, which was rejected in #62 as non-commercial. A NOTICE/attribution entry would be required (the repo currently has no LICENSE file at all).

B. Re-point at WWWJDIC

https://www.edrdg.org/cgi-bin/wwwjdic/wwwjdic is a different, ungated EDRDG endpoint — and api/sentence_query.py:55 already uses it, so it is a known-working path in this codebase. Verified returning clean EDICT-format data:

$ curl 'https://www.edrdg.org/cgi-bin/wwwjdic/wwwjdic?1ZUJ先生'
先生 [せんせい] /(n) (1) teacher/instructor/master/(n,n-suf) (2) (hon) sensei/...
先生方 [せんせいがた] /(n) (hon) (See 方・1) doctors/teachers/lawyers/

Easier to parse than the HTML it replaces. But it loses the id field — I scanned display codes 0–B and none emit JMdict sequence numbers (EntL…), so WordResult.id would have to be dropped or faked. It also keeps DictQuery network-dependent and scrape-shaped, i.e. the same fragility class that produced this issue and #57/#58.

C. Retire the endpoint

#62 records that DictQuery / SentenceQuery / UsageQuery "see little use" and explicitly parks their bugs as out of scope. If that still holds, removing the endpoint is cheaper than either fix above, and is the honest answer to shipping something that has been returning 404 for every input.

Decision needed

This is blocked on the #62 repo-split question rather than on implementation difficulty. A → B → C in descending order of effort; A is the only one consistent with the project's stated offline-first direction, and its cost is far lower than it looks (10 MB against a 1.3 GB image).

Reproduce

curl -s -X POST localhost:8000/api/DictQuery/ -H 'content-type: application/json' -d '{"word":"先生"}'
# -> 404 "No results found"

# the actual upstream condition:
curl -s 'https://www.edrdg.org/jmwsgi/srchres.py?s1=1&y1=1&t1=%E5%85%88%E7%94%9F&src=1&search=Search&svc=jmdict' | grep -o '<title>.*</title>'
# -> <title>JMdictDB - Site Access</title>

Superseded diagnosis (original report, 2026-05-27)