Repository navigation
fix: read a date-only timestamp as midnight US/Eastern - #114
Conversation
The API writes a date as %Y-%m-%d, meaning that calendar day in US/Eastern: on the default format for option expirations, and under dateformat=timestamp for candles of a day or longer and for earnings, news and market status dates. ToDateTime parsed every string with DateTimeStyles.AssumeUniversal, so a date became UTC midnight and, once converted to America/New_York, 20:00 or 19:00 of the day before. A string of exactly the yyyy-MM-dd shape is now midnight US/Eastern of that day, with the EST or EDT offset in force then. Every other string keeps the offset-aware parse, so datetimes with an offset and Unix timestamps read as before. A length test keeps datetime values off the exact date parse, which would otherwise cost about 200 ns per value when it fails. The explicit non-string branch surfaces a path the old combined condition hid from coverlet, which shares one hit counter per jump target; that path now has its own test. The live expirations test now checks that every expiration is a midnight, the DateFormat docs describe what the API sends, and BUG_FINDING 4.1 compares the default, unix and timestamp formats instead of running spreadsheet through a typed method, which rejects it.
|
/run-all-os |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #114 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 69 69
Lines 2187 2190 +3
Branches 272 274 +2
=========================================
+ Hits 2187 2190 +3
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
|
❌ On-demand OS matrix |
✅ Ready — Code reviewVersion impact: minor (1.0.0 -> 1.1.0). The internal parser now treats date-only API timestamps as midnight US/Eastern while preserving offset-bearing and Unix timestamp behavior, with endpoint-level, integration, documentation, and changelog updates. The same date-normalization defect can exist in the Python and PHP sibling SDKs; Java has a different spreadsheet-date defect, while Go and JavaScript do not perform this typed decoding. Rating patch 6/6 (exemplary) · proof 6/6 (comprehensive) · overall 6/6 Reviewed |
|
✅ On-demand OS matrix |
Midnight alone also accepts a value decoded as UTC midnight. Each live expiration must carry the offset US/Eastern has on its date: -04:00 or -05:00, both of which the AAPL list spans.
|
/run-all-os |
|
✅ On-demand OS matrix |
marketdata-code-review
Bot
added
review: ready
and removed
review: needs changes
labels
Closes #111
Summary
A date the API sends without a time (
2026-09-21) now reads as midnight America/New_York of that day. Onmainit read as UTC midnight, so it came back as 20:00 (EDT) or 19:00 (EST) of the day before. That hit:GetExpirationsAsyncon the default date format. With nodateformat, the API sendsexpirationsas dates ("2026-09-23"), so every expiration came back as the evening before.DateFormat.Timestamp: theTimeof stock and fund candles of a day or longer, theDate,ReportDateandUpdatedof an earning, thePublicationDateandUpdatedof the news, and the market statusDate.Datetimes with an offset (
2026-09-21 14:46:05 -04:00) and Unix timestamps read as before.Version impact
Version impact: minor (1.0.0 -> 1.1.0). The diff touches no public symbol: the change is inside the internal
JsonResponseParser, plus the XML doc wording ofDateFormat.UnixandDateFormat.Timestamp. Under the SDK review rules a non-breaking change takes the next minor. The affected values change: those are the values that were wrong.The change
The API writes a date as
%Y-%m-%d, meaning that calendar day in US/Eastern, and a datetime as%Y-%m-%d %H:%M:%S %:z(marketDataApi/common/util/date_helper.py::DateHelper.format_date).JsonResponseParser.ToDateTimeparsed every string withDateTimeStyles.AssumeUniversal.yyyy-MM-ddshape is now midnight US/Eastern of that day, with the EST or EDT offset in force then. US/Eastern changes its clocks at 02:00, so that midnight always exists and is never ambiguous.DateOnly.TryParseExactadded about 200 ns to every datetime on .NET 10 (best-of-5 Stopwatch microbenchmark, not BenchmarkDotNet). With it, the datetime path measures as before and a date costs about 50–70 ns more; the decode allocates nothing new.DateFormat.Unixno longer claims to be the default for every field, andDateFormat.Timestampdescribes the two shapes the API sends instead of "ISO-8601"..github/BUG_FINDING.md4.1 now compares the default, unix and timestamp formats on daily candles and expirations. It ranDateFormat.Spreadsheetthrough a typed method, which rejects it, and only counted rows.Left as it was, on purpose: a datetime without an offset (
2026-09-21 14:46:05) still reads as UTC, and a test pins it. The API never sends that shape. The issue's definition of done says "a timestamp string without an offset"; this PR reads it as the date shape the API sends. Reading every offset-less string as Eastern wall-clock time was tried and rejected: throughDateTime.TryParsewithAdjustToUniversal,0001-01-01 00:00:00 +01:00goes from null to a value a day off (the UTC adjustment wraps negative ticks) and9999-12-31 23:00:00starts throwing.Tests
JsonResponseParserInternalTests: the shapes of the definition of done (a date, a datetime with an offset in EDT and in EST), an EST date, both DST switch days (2026-03-08→-05:00,2026-11-01→-04:00) and an offset-less datetime (UTC). The assertions compare wall-clock time and offset as text, becauseAssert.EqualonDateTimeOffsetcompares instants only. Alsotrue,{}and the date-lengthnot-a-date, each null.JsonParsingTests: one test per endpoint the issue lists, through the public client underDateFormat.Timestamp: daily stock and fund candlest; earningsdate,reportDateandupdated; newspublicationDateand the root-levelupdated; optionsexpirationsbeside a datetimeupdated; markets statusdate. PlusGetExpirationsAsyncon the default format with the default response shape: dates forexpirations, a Unixupdated, and nodateformatin the request.OptionsIntegrationTests.Expirations_ReturnExpectedShapenow checks that every live expiration is midnight with the US/Eastern offset of its date, so a value decoded as UTC midnight fails too. The AAPL list spans both offsets (-04:00and-05:00, up to2029-01-19). Run live againstmain, the midnight check fails; on this branch all 25 pass.main, 11 of the new cases fail with the reported symptom (2026-09-20T20:00:00-04:00for2026-09-21). The datetime cases and the null cases pass there too, as expected.String && TryParsecondition hid: coverlet shares one hit counter per jump target, sonot-a-timestampalso counted for a non-string element. That path now has its own test.Run on Windows (host zone UTC−3), .NET SDK 10.0.302: 323 passed on
net8.0andnet10.0, 100% line and branch coverage on both, 0 build warnings,dotnet format --verify-no-changesclean.Proof
Measured live on 2026-09-22 through the public client in demo mode (no token),
AAPL. "Matches unix" compares every row, instant and offset, against the same call underDateFormat.Unix.main3438f7f)2026-09-232026-09-22 20:00 -04:002026-09-23 00:00 -04:002026-09-232026-09-22 20:00 -04:002026-09-23 00:00 -04:00t2026-09-152026-09-14 20:00 -04:002026-09-15 00:00 -04:00date2026-09-302026-09-29 20:00 -04:002026-09-30 00:00 -04:00reportDate2026-10-292026-10-28 20:00 -04:002026-10-29 00:00 -04:00updated2026-09-172026-09-16 20:00 -04:002026-09-17 00:00 -04:00publicationDate2026-09-222026-09-21 20:00 -04:002026-09-22 00:00 -04:00updated2026-09-222026-09-21 20:00 -04:002026-09-22 00:00 -04:00t(control)2026-09-22 09:30:00 -04:002026-09-22 09:30 -04:002026-09-22 09:30 -04:00Weekly, monthly and yearly candles are sent as dates too under
timestamp(2026-09-14,2026-08-03,2025-01-01) and go through the same path. On the default format, every other field checked (candles, earnings, news, the expirationsupdated) is a Unix number.Not verified live:
markets/statusreturns 401 without a token andfunds/candlesreturned no rows in demo mode, so both rely on the unit tests. The live integration suite did not run here (no token on this machine); this PR'sIntegration Tests (live)job runs the new expirations check.Cross-platform: the on-demand OS matrix passed on Windows and macOS on this head (
2ded4b8), run 35861445686. On051306c, the first run's Windows leg passed every step, both test runs under the coverage gate included, but was cancelled while closing: Codecov's comment started a new run of the same workflow, and itsconcurrencygroup cancels the run in progress, which the Report job counted as a failure. Its rerun passed. The ubuntu leg covers the IANA time-zone data.Sibling SDKs
Date and time normalisation is shared logic. The API sends option expirations as dates even when no
dateformatis requested, so that case is the one to check in each SDK.dateformat(DateFormatis CSV-only), andExpirationsbuilds each date withCarbon::parseand no zone, so2026-09-23is midnight in the server'sdate.timezone: 20:00 of the day before in New York on a UTC server. Checked in the code at ec5181b, not run on a PHP host.timestampshapes right; its defect is the spreadsheet serial (dateformat=spreadsheet timestamps are four or five hours early: the serial is read as UTC, the API writes US/Eastern wall-clock time sdk-java#77), per the survey in fix(dataframe): read timestamp and spreadsheet dates as the API writes them sdk-py#144.