Skip to content

viz: --dictionary content_type date format is discarded; date parsing follows the global QSV_PREFER_DMY instead #4303

Description

@jqnatividad

Summary

A Data Dictionary declares the date format per column, e.g. content_type: "date:%m/%d/%Y". viz parses that token only to route the column (route_from_content_type), then throws the format away. Every date is instead parsed according to the global QSV_PREFER_DMY env var.

When the two disagree, qsv silently misreads every date in the column. There is no warning, and both readings look plausible in the output.

Repro

dmybug.csv — deliberately ambiguous dates (01/03 is 3 Jan day-first, 1 Mar month-first):

Created Date,Amount
01/03/2026,10
02/05/2026,20
03/07/2026,30
04/09/2026,40
05/11/2026,50

dmybug.schema.json — the dictionary declares month-first, so the correct reading is unambiguous:

{
  "properties": {
    "Created Date": {
      "type": ["string"], "title": "Created Date",
      "x-qsv": { "qsv_type": "Date", "content_type": "date:%m/%d/%Y",
                 "role": "timestamp", "concept": "time.event_timestamp" }
    }
  }
}
qsv viz smart dmybug.csv --dictionary dmybug.schema.json --dict-info
QSV_PREFER_DMY=1 qsv viz smart dmybug.csv --dictionary dmybug.schema.json --dict-info

Result

Dates parsed (both the chart axis values and the data-viewer sort keys):

source dictionary says %m/%d/%Y no QSV_PREFER_DMY QSV_PREFER_DMY=1
01/03/2026 2026-01-03 2026-01-03 ✅ 2026-03-01
02/05/2026 2026-02-05 2026-02-05 ✅ 2026-05-02
03/07/2026 2026-03-07 2026-03-07 ✅ 2026-07-03
04/09/2026 2026-04-09 2026-04-09 ✅ 2026-09-04
05/11/2026 2026-05-11 2026-05-11 ✅ 2026-11-05

The QSV_PREFER_DMY=1 column is wrong in every row, against an explicit per-column declaration. It happens to be correct today only because the default (month-first) coincides with what this dictionary declares — a user whose other datasets are day-first, and who therefore exports QSV_PREFER_DMY=1, gets silently wrong output for this one.

examples/viz/nyc311_dict.schema.json is a real instance: it declares date:%m/%d/%Y on Created Date, Closed Date and Resolution Action Updated Date, and datetime:%m/%d/%Y %I:%M:%S %p on Due Date.

Cause

route_from_content_type (src/cmd/viz.rs) splits the format suffix off and drops it — it only needs the base token to pick a route:

let base = content_type
    .split_once(':')
    .map_or(content_type, |(b, _)| b)
    .trim();

Nothing else ever reads that suffix. Every date-parsing site instead resolves the preference globally:

let prefer_dmy = util::get_envvar_flag("QSV_PREFER_DMY");

Scope

Wider than one chart — every temporal path in viz takes prefer_dmy: bool sourced from the env var, and none of them consult the dictionary:

  • parse_record_date and its callers — trend/overview, animated geo, slider frames, bivariate temporal panels
  • collect_datatable_rows — the data viewer's date sort keys (issue viz: add datatables viewer to smart dashboards #4283)
  • qsv stats itself ORs in QSV_PREFER_DMY (src/cmd/stats.rs), which is what types the column Date/DateTime in the first place

Stats and viz therefore agree with each other and disagree with the dictionary together, which is why nothing looks inconsistent on the page.

Suggested direction

Treat a dictionary-declared format as authoritative for that column, falling back to the global preference only where the dictionary is silent:

  • Parse the :<strftime> suffix off content_type into the DictRow alongside the existing fields, rather than discarding it.
  • Derive a per-column DMY/MDY preference from it (%d before %m → day-first, %m before %d → month-first) and thread that through the date-parsing sites in place of the single global flag. DictData is already in scope at the relevant call sites.
  • Where the dictionary is absent or the format unparseable, keep today's QSV_PREFER_DMY behaviour exactly.

Worth considering whether a declared format that contradicts QSV_PREFER_DMY should also warn, since either choice silently changes every date in the column.

Noticed while working on #4302; that PR does not touch this.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions