fix(models): build INTERNAL results that carry a column the model does not declare

CHANGELOG.md

@@ -21,9 +21,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Money in the `OutputFormat.INTERNAL` models is `decimal.Decimal` instead of `float`, built from the digits in the response body, which those resources now decode with `parse_float=Decimal` on the INTERNAL path: `ask`, `bid`, `mid`, `last`, `change` of `StockQuote`, `mid` and `change` of `StockPrice`, `o`, `h`, `l`, `c` of `StockCandle` and `FundsCandle`, the three EPS lists of `StockEarnings`, `strike`, `bid`, `mid`, `ask`, `last`, `intrinsicValue`, `extrinsicValue`, `underlyingPrice` of `OptionsChain` and `OptionsQuotes`, and the same fields of every human-readable twin. Greeks, IV and percentages stay `float`, sizes and counts `int`; DataFrames (pandas and polars), JSON and CSV output keep the plain float parse. Migration: arithmetic mixing `Decimal` and `float` raises `TypeError`, and a comparison with a float literal compares against its binary value (`Decimal("65.1") == 65.1` is `False`), so use `Decimal` literals or `int`; `float(value)` gives a float back, and `OutputFormat.DATAFRAME` is the float path for analysis (a pandas DataFrame built from models gets `object` columns, and `json.dumps(..., default=str)` writes the money as strings). A model built by hand converts its money arguments: a float or a numpy scalar through its shortest repr (`65.1` is `Decimal("65.1")`), a NaN number, which is how pandas writes a missing value, to `None`; a value that is not a number, a string that reads as NaN, or an infinity raises and names the field, while an array (anything that is not a string and has `__array__` and `__len__`: a numpy array, a pandas or polars Series, a pandas Index) is left as it came, and a `dict`, a `set` or a generator raises. On the API path the same refusal is a `ParseError` naming the field and the body, not a bare `TypeError` (#50)
 - Dates on `OutputFormat.INTERNAL` are read with the API's own rule and always come back as US/Eastern datetimes. A number from 10000 up to 200000 is a spreadsheet serial of US/Eastern wall-clock time, from 200000 Unix seconds, from `1e10` milliseconds and from `1e13` nanoseconds. A number under 10000 is a relative range for the API, not a date, and raises `ParseError`. The SDK used a boundary of its own at 60000, so from April 2064 a serial would have read as 1970. A `timestamp` date (`2026-09-21`) and a spreadsheet serial are now midnight and wall-clock time in US/Eastern, where they came back as naive datetimes, and a serial is rounded to the second. The intraday `from_date` and `to_date` of `stocks.candles()` are read only when they are ISO dates or numbers the API reads as dates (a spreadsheet serial or a Unix time), which the SDK splits into one request per year: a relative range or a keyword goes to the API as it is, in one request, where `"60"` was sent as 1900-02-28 and `"yesterday"` was refused. `UserRateLimits.reset_time` takes a datetime or Unix seconds, and refuses anything else (#116). Migration: compare INTERNAL dates against aware datetimes (`pytz.timezone("US/Eastern").localize(...)`), not naive ones
 - A response body carrying a `NaN`, `Infinity` or `-Infinity` literal raises `ParseError` on every output format that decodes it, where `INTERNAL`, `JSON` and `DATAFRAME` used to return it as `nan` or `inf`. Those literals are not JSON, and the renderer behind the API's data endpoints refuses them, answering `500`, so one in their bodies was written by something in between (`/status/` and `/headers/` are served elsewhere). `CSV` output writes the API's body as it came, since it reads no number, except for `client.utilities`, which builds its CSV from the decoded body and refuses it too. A number past what a float holds is not refused: `JSON` and `DATAFRAME` read `1e400` as `inf` while `INTERNAL` keeps it exact, and `INTERNAL` refuses one past what a `Decimal` holds. That renderer does not write such a number, and refusing it on the float path would slow every decode. Migration: code that read `nan` or `inf` from such a body catches `ParseError` (or `BaseMarketdataException`) instead; no setting brings the literals back (#50)
-- With `OutputFormat.INTERNAL`, a body its model refuses raises `ParseError` on every resource, `markets.status()`, `options.expirations()`, `options.lookup()`, `stocks.news()` and `client.utilities` included: a date it cannot parse, which used to escape as a bare `ValueError`, and a key the model does not take or a column it needs that the body lacks, which used to escape as a bare `TypeError`. For `stocks.candles()` and `options.quotes()` the error names the chunk or the symbol that sent the value, with that answer's own reason. A body that is not a JSON object (a list, a string, a number or `null`) now raises `ParseError`, where it raised a bare `TypeError`, on `stocks.earnings()`, `options.lookup()`, and on `options.chain()` and `options.expirations()` with the API's names; elsewhere it raises what it raised before, a bare built-in except on `stocks.candles()` and `options.quotes()`. An empty object still fails with a bare `ValueError`, before any model reads it, on `stocks.quotes()`, `stocks.prices()`, `stocks.news()`, `funds.candles()`, `markets.status()` and `client.utilities.status()` (#110 has the table). Migration: an `except ValueError` or `except TypeError` around those calls no longer catches these errors; catch `ParseError` (or `BaseMarketdataException`) instead (#50)
+- With `OutputFormat.INTERNAL`, a body its model refuses raises `ParseError` on every resource, `markets.status()`, `options.expirations()`, `options.lookup()`, `stocks.news()` and `client.utilities` included: a date it cannot parse, which used to escape as a bare `ValueError`, and a column it needs that the body lacks, which used to escape as a bare `TypeError`. For `stocks.candles()` and `options.quotes()` the error names the chunk or the symbol that sent the value, with that answer's own reason. A body that is not a JSON object (a list, a string, a number or `null`) now raises `ParseError`, where it raised a bare `TypeError` or `AttributeError`, on `stocks.earnings()`, `options.lookup()`, `options.chain()` and `options.expirations()`, with the API's names and the human-readable ones. Their `JSON` and `DATAFRAME` output raise the same `ParseError`, where they returned the body and a frame built from it (an empty one for `null`), and their CSV output raises `ParseError` too, since it now refuses a header row with none of the resource's columns and, under `add_headers=False`, an answer whose `Content-Type` names JSON or HTML, a lone string, number or boolean included, or whose body reads as either, where it wrote the body to the file. The CSV output of `stocks.candles()` and `options.quotes()` under `add_headers=False` refuses such an answer the same way. On those four resources an object with none of their columns, like a proxy's JSON error page, raises `ParseError` on `JSON` and `DATAFRAME` too, where they returned it, as it does on `INTERNAL` and CSV. Elsewhere `INTERNAL` raises what it raised before, a bare built-in except on `stocks.candles()` and `options.quotes()`. An empty object still fails with a bare `ValueError`, before any model reads it, on `stocks.quotes()`, `stocks.prices()`, `stocks.news()`, `funds.candles()`, `markets.status()` and `client.utilities.status()` (#110 has the table). Migration: an `except ValueError` or `except TypeError` around those calls no longer catches these errors; catch `ParseError` (or `BaseMarketdataException`) instead (#50)
 - `options.chain(delta=...)` checks a number before it sends it, the way it checks `strike`, and keeps it as the text it sends. A `bool`, a NaN, an infinity, a number a float cannot hold and a delta outside -1 to 1 now raise `ValidationError` before any request, where they used to be sent: `delta=nan` came back as the whole chain, since the API reads it as no filter, `delta=True` was sent as `1.0`, and `delta=30` was sent as `30.0`. `OptionsChainInput.delta` holds that text (`"0.5"`) instead of a `float` (#101). Migration: pass a finite number from -1 to 1, the way the public docs write a delta (`0.3`), a `DeltaFilter`, or `delta=None` for no filter; code that reads `OptionsChainInput(...).delta` gets the text sent to the API, and `float()` turns a plain number back.
 - Every date field of the `OutputFormat.INTERNAL` models admits `None` in its annotation: `datetime.datetime | None` for one date, `list[datetime.datetime | None]` for a column of them, and `datetime.date | None` for `MarketStatus.date`, as `OptionsExpirations.updated` already did. A `null` date in an answer reads as `None` in its own row (see Fixed), and the annotations now say so. Migration: a type checker now asks for a `None` check before a `datetime` method is called on a date field (`if quote.updated is not None: quote.updated.date()`). At run time nothing that used to return changes: an answer with a `null` date used to raise `ParseError`, or on `OptionsQuotes` shift the column (#106)
+- A column the API adds before the SDK declares it no longer fails the call. On `OutputFormat.INTERNAL` the model is built without it, where the call raised a bare `TypeError`, and `marketdata.get_extra(model)` returns it as the API sent it: the whole column on a single-object model, the row's own value on each row of a list result. The column is named once per answer at DEBUG on `marketdata.logger`. `StockEarnings.from_dict()`, `StockPrice.from_dict()`, `StockQuote.from_dict()` and their human-readable twins keep such a key for `get_extra()` the same way, where they raised `TypeError`. A column the model needs and the body lacks still raises `ParseError`, and so does an object with none of the model's columns. `stocks.candles()` and `options.quotes()` carry such a column through their merge on every format, INTERNAL through `get_extra()`, as long as every chunk or symbol sends it: like a declared column, an answer without it, or with a list of another length, fails the call with `ParseError` on every format. The other resources that answer in columns, `funds.candles()`, `markets.status()`, `options.chain()`, `options.expirations()`, `options.lookup()`, `stocks.earnings()`, `stocks.news()`, `stocks.prices()`, `stocks.quotes()` and `client.utilities.status()`, hold each answer to the same rule: a list, declared or not, longer or shorter than the others raises `ParseError` on `INTERNAL`, `JSON` and `DATAFRAME`, and on the CSV of `client.utilities.status()`, which is written from the decoded answer, where `INTERNAL` built the rows up to the shortest declared column, `DATAFRAME` padded the short columns with nulls and `JSON` returned the body; a single value instead of a list is still kept as sent. The CSV output of `stocks.candles()` and `options.quotes()` keeps an added column, where it raised `ParseError`, because `merge_csv_responses()` now refuses only a header with none of the resource's columns. Migration: code that caught the `TypeError` or `ParseError` an added column raised now gets the result; read the column with `get_extra()` until the model declares it, and catch `ParseError` (or `BaseMarketdataException`) for an answer whose columns differ in length (#111)
 
 ### Changed
 
@@ -40,6 +41,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- `marketdata.get_extra(model)` reads the columns an `OutputFormat.INTERNAL` answer carried that its model does not declare, keyed as the API sent them (#111)
 - `StrikeFilter` builds the `strike` filter of `options.chain` from its shapes instead of from a hand-written string: `exact`, `any_of`, `between`, `at_least`, `at_most`, `above`, `below`, and `expression` for anything the others do not name, the way sdk-go and sdk-csharp give them. `DeltaFilter` does the same for `delta`, which the API reads with the same grammar, with `nearest` in place of `exact`: the API answers a single delta with the contract nearest to it. Every `DeltaFilter` constructor but `expression` raises `ValueError` for a delta outside -1 to 1. Both parameters also take a plain `int`, `float` or `Decimal` now, which `strike` refused before while the public docs typed it `float`, so a strike read from an INTERNAL chain can be passed straight back. A negative **number** is refused rather than sent: the API reads a number by its absolute value, so `strike=-250` used to answer with the strikes at `250` and no error. A string is still passed through as written, since it is the expression form and the API is what reads it. A negative delta is kept, since there the absolute value is the documented filter and both sides come back either way. A number travels as plain digits, never in exponent form: the API reads a query of `delta=0.00005`, and answers `delta=5e-05` with a `400`, since it splits that text at its minus. A number a float cannot hold is refused, since the API reads each one with `float()` and would take it for an infinity or for 0 (#101)
 - The API's IP headers, for accounts that restrict access by IP (#44). A 403 that blocks an address raises `ForbiddenError` with `authorized_ip`, the address the account is authorized for, taken from `X-API-Authorized-IP` and repeated in the message, so a caller who prints the error knows what to allow. `marketdata.get_meta(result).detected_ip` is the address the call came from, taken from `X-API-Detected-IP` on every answer the API serves. Both read `None` when the API sent no header: an account allowed to call from several addresses, a staff token, the Sheets add-on, or an address it could not resolve. The legacy `X-API-BLOCKED-IP` is not read: the API is removing it (MarketData-App/api#202)
 - `client.utilities` resource with `status()`, `headers()` and `user()` for the `/status/`, `/headers/` and `/user/` endpoints, in every output format (#63)

README.md

@@ -353,6 +353,28 @@ Mixing `Decimal` and `float` in arithmetic raises `TypeError`, and comparing the
 
 The other formats keep the plain parse. A DataFrame never holds a `Decimal`: it keeps the plain parse, so every column has the dtype it always had on both pandas and polars (`float64` for prices with a fraction). A DataFrame is for vectorized analysis, pandas has no decimal dtype, and the same call returning a different dtype depending on which library is installed would be a trap. `OutputFormat.JSON` returns the decoded JSON with standard `float` numbers, and `OutputFormat.CSV` writes the API's text as it came (`client.utilities` builds its CSV from the decoded body). A `NaN` or an `Infinity` in a JSON body is not JSON and fails the call on every format that decodes the body; a number past what a float holds (`1e400`) is `inf` on `JSON` and `DATAFRAME` and exact on `INTERNAL`.
 
+### Columns a model does not declare
+
+The API adds columns without an SDK release. A model is built from the columns it declares, and `marketdata.get_extra()` returns the others, named as the API sent them, until a release declares them:
+
+```python
+import marketdata
+from marketdata import MarketDataClient, OutputFormat
+
+client = MarketDataClient()
+chain = client.options.chain("AAPL", output_format=OutputFormat.INTERNAL)
+marketdata.get_extra(chain)          # {} while the model declares every column
+
+quotes = client.stocks.quotes("AAPL", output_format=OutputFormat.INTERNAL)
+marketdata.get_extra(quotes[0])      # the row's value of each column StockQuote lacks
+```
+
+- **Which results:** every `OutputFormat.INTERNAL` result built from an answer's columns, the merged answers of `stocks.candles()` and `options.quotes()` included, and a model built with `from_dict()` on `StockEarnings`, `StockPrice`, `StockQuote` and their human-readable twins. A single-object result (`options.chain()`, `options.expirations()`, `options.lookup()`, `options.quotes()`, `stocks.earnings()`) holds each column whole; a list result (`stocks.quotes()`, `stocks.prices()`, `stocks.candles()`, `stocks.news()`, `funds.candles()`, `markets.status()`, `utilities.status()`) gives each row its own value. `utilities.user()` and `utilities.headers()` keep nothing there: the first reads only its own keys, the second keeps every key as a header.
+- **The values:** decoded as the rest of the answer, so a number with a fraction is a `Decimal` on the resources whose money is. Each answer that brings such a column names it once at DEBUG on `marketdata.logger`.
+- **How long they last:** as long as the object, which holds them itself, like the response metadata: they survive `copy.copy()`, `copy.deepcopy()` and `pickle`. They are not fields, so `==`, the repr and `dataclasses.fields()` ignore them, and a model built by hand, or from an answer without such a column, returns `{}`.
+- **The other formats:** `JSON` and `DATAFRAME` carry the column by the API's name and `CSV` writes it in the file. On `stocks.candles()` and `options.quotes()` every chunk or symbol must carry it, as it must carry a declared column: an answer without it fails the call with `ParseError`, on every format.
+- **One value per row:** like a declared column, an added one that is a list must be as long as the others. A list longer or shorter than the rest fails the call with `ParseError` on `INTERNAL`, `JSON` and `DATAFRAME`, whatever the resource, and on the CSV of `utilities.status()`, which is written from the decoded answer. A single value instead of a list is kept as sent, on every row of a list result; the merges of `stocks.candles()` and `options.quotes()` carry only lists.
+
 ## Universal Parameters
 
 All resource methods support universal parameters that can be used to customize the API request and response:

docs/funds.md

@@ -13,6 +13,10 @@ funds = client.funds
 
 All methods in the funds resource include API status checking and automatic retry logic. See the [README](../README.md) for general information about error handling, retry mechanisms, and output formats.
 
+## Columns a model does not declare
+
+With `OutputFormat.INTERNAL`, a column the API sends before the SDK declares it does not fail the call: the result is built from the columns its model declares, and `marketdata.get_extra(candle)` returns the others, named as the API sent them, with each row's own value. `JSON` and `DATAFRAME` carry it as a column, and `CSV` writes it in the file. See [Columns a model does not declare](../README.md#columns-a-model-does-not-declare) in the README for how long the values last and how they are decoded.
+
 ## Methods
 
 ### `candles()`

docs/markets.md

@@ -13,6 +13,10 @@ markets = client.markets
 
 All methods in the markets resource include API status checking and automatic retry logic. See the [README](../README.md) for general information about error handling, retry mechanisms, and output formats.
 
+## Columns a model does not declare
+
+With `OutputFormat.INTERNAL`, a column the API sends before the SDK declares it does not fail the call: the result is built from the columns its model declares, and `marketdata.get_extra(row)` returns the others, named as the API sent them, with each row's own value. `JSON` and `DATAFRAME` carry it as a column, and `CSV` writes it in the file. See [Columns a model does not declare](../README.md#columns-a-model-does-not-declare) in the README for how long the values last and how they are decoded.
+
 ## Methods
 
 ### `status()`

docs/options.md

@@ -13,6 +13,10 @@ options = client.options
 
 All methods in the options resource include API status checking and automatic retry logic. See the [README](../README.md) for general information about error handling, retry mechanisms, and output formats.
 
+## Columns a model does not declare
+
+With `OutputFormat.INTERNAL`, a column the API sends before the SDK declares it does not fail the call: the result is built from the columns its model declares, and `marketdata.get_extra(result)` returns the others, named as the API sent them, each column whole, since `chain()`, `expirations()`, `lookup()` and `quotes()` each return one object. On `quotes()` every symbol must send such a column, as it must send a declared one, or the call raises `ParseError`. `JSON` and `DATAFRAME` carry it as a column, and `CSV` writes it in the file. See [Columns a model does not declare](../README.md#columns-a-model-does-not-declare) in the README for how long the values last and how they are decoded.
+
 ## Methods
 
 ### `expirations()`