diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a08b89..dfc9da6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,9 +21,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Money in the `OutputFormat.INTERNAL` models is `decimal.Decimal` instead of `float`, built from the digits in the response body, which those resources now decode with `parse_float=Decimal` on the INTERNAL path: `ask`, `bid`, `mid`, `last`, `change` of `StockQuote`, `mid` and `change` of `StockPrice`, `o`, `h`, `l`, `c` of `StockCandle` and `FundsCandle`, the three EPS lists of `StockEarnings`, `strike`, `bid`, `mid`, `ask`, `last`, `intrinsicValue`, `extrinsicValue`, `underlyingPrice` of `OptionsChain` and `OptionsQuotes`, and the same fields of every human-readable twin. Greeks, IV and percentages stay `float`, sizes and counts `int`; DataFrames (pandas and polars), JSON and CSV output keep the plain float parse. Migration: arithmetic mixing `Decimal` and `float` raises `TypeError`, and a comparison with a float literal compares against its binary value (`Decimal("65.1") == 65.1` is `False`), so use `Decimal` literals or `int`; `float(value)` gives a float back, and `OutputFormat.DATAFRAME` is the float path for analysis (a pandas DataFrame built from models gets `object` columns, and `json.dumps(..., default=str)` writes the money as strings). A model built by hand converts its money arguments: a float or a numpy scalar through its shortest repr (`65.1` is `Decimal("65.1")`), a NaN number, which is how pandas writes a missing value, to `None`; a value that is not a number, a string that reads as NaN, or an infinity raises and names the field, while an array (anything that is not a string and has `__array__` and `__len__`: a numpy array, a pandas or polars Series, a pandas Index) is left as it came, and a `dict`, a `set` or a generator raises. On the API path the same refusal is a `ParseError` naming the field and the body, not a bare `TypeError` (#50) - Dates on `OutputFormat.INTERNAL` are read with the API's own rule and always come back as US/Eastern datetimes. A number from 10000 up to 200000 is a spreadsheet serial of US/Eastern wall-clock time, from 200000 Unix seconds, from `1e10` milliseconds and from `1e13` nanoseconds. A number under 10000 is a relative range for the API, not a date, and raises `ParseError`. The SDK used a boundary of its own at 60000, so from April 2064 a serial would have read as 1970. A `timestamp` date (`2026-09-21`) and a spreadsheet serial are now midnight and wall-clock time in US/Eastern, where they came back as naive datetimes, and a serial is rounded to the second. The intraday `from_date` and `to_date` of `stocks.candles()` are read only when they are ISO dates or numbers the API reads as dates (a spreadsheet serial or a Unix time), which the SDK splits into one request per year: a relative range or a keyword goes to the API as it is, in one request, where `"60"` was sent as 1900-02-28 and `"yesterday"` was refused. `UserRateLimits.reset_time` takes a datetime or Unix seconds, and refuses anything else (#116). Migration: compare INTERNAL dates against aware datetimes (`pytz.timezone("US/Eastern").localize(...)`), not naive ones - A response body carrying a `NaN`, `Infinity` or `-Infinity` literal raises `ParseError` on every output format that decodes it, where `INTERNAL`, `JSON` and `DATAFRAME` used to return it as `nan` or `inf`. Those literals are not JSON, and the renderer behind the API's data endpoints refuses them, answering `500`, so one in their bodies was written by something in between (`/status/` and `/headers/` are served elsewhere). `CSV` output writes the API's body as it came, since it reads no number, except for `client.utilities`, which builds its CSV from the decoded body and refuses it too. A number past what a float holds is not refused: `JSON` and `DATAFRAME` read `1e400` as `inf` while `INTERNAL` keeps it exact, and `INTERNAL` refuses one past what a `Decimal` holds. That renderer does not write such a number, and refusing it on the float path would slow every decode. Migration: code that read `nan` or `inf` from such a body catches `ParseError` (or `BaseMarketdataException`) instead; no setting brings the literals back (#50) -- With `OutputFormat.INTERNAL`, a body its model refuses raises `ParseError` on every resource, `markets.status()`, `options.expirations()`, `options.lookup()`, `stocks.news()` and `client.utilities` included: a date it cannot parse, which used to escape as a bare `ValueError`, and a key the model does not take or a column it needs that the body lacks, which used to escape as a bare `TypeError`. For `stocks.candles()` and `options.quotes()` the error names the chunk or the symbol that sent the value, with that answer's own reason. A body that is not a JSON object (a list, a string, a number or `null`) now raises `ParseError`, where it raised a bare `TypeError`, on `stocks.earnings()`, `options.lookup()`, and on `options.chain()` and `options.expirations()` with the API's names; elsewhere it raises what it raised before, a bare built-in except on `stocks.candles()` and `options.quotes()`. An empty object still fails with a bare `ValueError`, before any model reads it, on `stocks.quotes()`, `stocks.prices()`, `stocks.news()`, `funds.candles()`, `markets.status()` and `client.utilities.status()` (#110 has the table). Migration: an `except ValueError` or `except TypeError` around those calls no longer catches these errors; catch `ParseError` (or `BaseMarketdataException`) instead (#50) +- With `OutputFormat.INTERNAL`, a body its model refuses raises `ParseError` on every resource, `markets.status()`, `options.expirations()`, `options.lookup()`, `stocks.news()` and `client.utilities` included: a date it cannot parse, which used to escape as a bare `ValueError`, and a column it needs that the body lacks, which used to escape as a bare `TypeError`. For `stocks.candles()` and `options.quotes()` the error names the chunk or the symbol that sent the value, with that answer's own reason. A body that is not a JSON object (a list, a string, a number or `null`) now raises `ParseError`, where it raised a bare `TypeError` or `AttributeError`, on `stocks.earnings()`, `options.lookup()`, `options.chain()` and `options.expirations()`, with the API's names and the human-readable ones. Their `JSON` and `DATAFRAME` output raise the same `ParseError`, where they returned the body and a frame built from it (an empty one for `null`), and their CSV output raises `ParseError` too, since it now refuses a header row with none of the resource's columns and, under `add_headers=False`, an answer whose `Content-Type` names JSON or HTML, a lone string, number or boolean included, or whose body reads as either, where it wrote the body to the file. The CSV output of `stocks.candles()` and `options.quotes()` under `add_headers=False` refuses such an answer the same way. On those four resources an object with none of their columns, like a proxy's JSON error page, raises `ParseError` on `JSON` and `DATAFRAME` too, where they returned it, as it does on `INTERNAL` and CSV. Elsewhere `INTERNAL` raises what it raised before, a bare built-in except on `stocks.candles()` and `options.quotes()`. An empty object still fails with a bare `ValueError`, before any model reads it, on `stocks.quotes()`, `stocks.prices()`, `stocks.news()`, `funds.candles()`, `markets.status()` and `client.utilities.status()` (#110 has the table). Migration: an `except ValueError` or `except TypeError` around those calls no longer catches these errors; catch `ParseError` (or `BaseMarketdataException`) instead (#50) - `options.chain(delta=...)` checks a number before it sends it, the way it checks `strike`, and keeps it as the text it sends. A `bool`, a NaN, an infinity, a number a float cannot hold and a delta outside -1 to 1 now raise `ValidationError` before any request, where they used to be sent: `delta=nan` came back as the whole chain, since the API reads it as no filter, `delta=True` was sent as `1.0`, and `delta=30` was sent as `30.0`. `OptionsChainInput.delta` holds that text (`"0.5"`) instead of a `float` (#101). Migration: pass a finite number from -1 to 1, the way the public docs write a delta (`0.3`), a `DeltaFilter`, or `delta=None` for no filter; code that reads `OptionsChainInput(...).delta` gets the text sent to the API, and `float()` turns a plain number back. - Every date field of the `OutputFormat.INTERNAL` models admits `None` in its annotation: `datetime.datetime | None` for one date, `list[datetime.datetime | None]` for a column of them, and `datetime.date | None` for `MarketStatus.date`, as `OptionsExpirations.updated` already did. A `null` date in an answer reads as `None` in its own row (see Fixed), and the annotations now say so. Migration: a type checker now asks for a `None` check before a `datetime` method is called on a date field (`if quote.updated is not None: quote.updated.date()`). At run time nothing that used to return changes: an answer with a `null` date used to raise `ParseError`, or on `OptionsQuotes` shift the column (#106) +- A column the API adds before the SDK declares it no longer fails the call. On `OutputFormat.INTERNAL` the model is built without it, where the call raised a bare `TypeError`, and `marketdata.get_extra(model)` returns it as the API sent it: the whole column on a single-object model, the row's own value on each row of a list result. The column is named once per answer at DEBUG on `marketdata.logger`. `StockEarnings.from_dict()`, `StockPrice.from_dict()`, `StockQuote.from_dict()` and their human-readable twins keep such a key for `get_extra()` the same way, where they raised `TypeError`. A column the model needs and the body lacks still raises `ParseError`, and so does an object with none of the model's columns. `stocks.candles()` and `options.quotes()` carry such a column through their merge on every format, INTERNAL through `get_extra()`, as long as every chunk or symbol sends it: like a declared column, an answer without it, or with a list of another length, fails the call with `ParseError` on every format. The other resources that answer in columns, `funds.candles()`, `markets.status()`, `options.chain()`, `options.expirations()`, `options.lookup()`, `stocks.earnings()`, `stocks.news()`, `stocks.prices()`, `stocks.quotes()` and `client.utilities.status()`, hold each answer to the same rule: a list, declared or not, longer or shorter than the others raises `ParseError` on `INTERNAL`, `JSON` and `DATAFRAME`, and on the CSV of `client.utilities.status()`, which is written from the decoded answer, where `INTERNAL` built the rows up to the shortest declared column, `DATAFRAME` padded the short columns with nulls and `JSON` returned the body; a single value instead of a list is still kept as sent. The CSV output of `stocks.candles()` and `options.quotes()` keeps an added column, where it raised `ParseError`, because `merge_csv_responses()` now refuses only a header with none of the resource's columns. Migration: code that caught the `TypeError` or `ParseError` an added column raised now gets the result; read the column with `get_extra()` until the model declares it, and catch `ParseError` (or `BaseMarketdataException`) for an answer whose columns differ in length (#111) ### Changed @@ -40,6 +41,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- `marketdata.get_extra(model)` reads the columns an `OutputFormat.INTERNAL` answer carried that its model does not declare, keyed as the API sent them (#111) - `StrikeFilter` builds the `strike` filter of `options.chain` from its shapes instead of from a hand-written string: `exact`, `any_of`, `between`, `at_least`, `at_most`, `above`, `below`, and `expression` for anything the others do not name, the way sdk-go and sdk-csharp give them. `DeltaFilter` does the same for `delta`, which the API reads with the same grammar, with `nearest` in place of `exact`: the API answers a single delta with the contract nearest to it. Every `DeltaFilter` constructor but `expression` raises `ValueError` for a delta outside -1 to 1. Both parameters also take a plain `int`, `float` or `Decimal` now, which `strike` refused before while the public docs typed it `float`, so a strike read from an INTERNAL chain can be passed straight back. A negative **number** is refused rather than sent: the API reads a number by its absolute value, so `strike=-250` used to answer with the strikes at `250` and no error. A string is still passed through as written, since it is the expression form and the API is what reads it. A negative delta is kept, since there the absolute value is the documented filter and both sides come back either way. A number travels as plain digits, never in exponent form: the API reads a query of `delta=0.00005`, and answers `delta=5e-05` with a `400`, since it splits that text at its minus. A number a float cannot hold is refused, since the API reads each one with `float()` and would take it for an infinity or for 0 (#101) - The API's IP headers, for accounts that restrict access by IP (#44). A 403 that blocks an address raises `ForbiddenError` with `authorized_ip`, the address the account is authorized for, taken from `X-API-Authorized-IP` and repeated in the message, so a caller who prints the error knows what to allow. `marketdata.get_meta(result).detected_ip` is the address the call came from, taken from `X-API-Detected-IP` on every answer the API serves. Both read `None` when the API sent no header: an account allowed to call from several addresses, a staff token, the Sheets add-on, or an address it could not resolve. The legacy `X-API-BLOCKED-IP` is not read: the API is removing it (MarketData-App/api#202) - `client.utilities` resource with `status()`, `headers()` and `user()` for the `/status/`, `/headers/` and `/user/` endpoints, in every output format (#63) diff --git a/README.md b/README.md index 9af93ee..990b302 100644 --- a/README.md +++ b/README.md @@ -353,6 +353,28 @@ Mixing `Decimal` and `float` in arithmetic raises `TypeError`, and comparing the The other formats keep the plain parse. A DataFrame never holds a `Decimal`: it keeps the plain parse, so every column has the dtype it always had on both pandas and polars (`float64` for prices with a fraction). A DataFrame is for vectorized analysis, pandas has no decimal dtype, and the same call returning a different dtype depending on which library is installed would be a trap. `OutputFormat.JSON` returns the decoded JSON with standard `float` numbers, and `OutputFormat.CSV` writes the API's text as it came (`client.utilities` builds its CSV from the decoded body). A `NaN` or an `Infinity` in a JSON body is not JSON and fails the call on every format that decodes the body; a number past what a float holds (`1e400`) is `inf` on `JSON` and `DATAFRAME` and exact on `INTERNAL`. +### Columns a model does not declare + +The API adds columns without an SDK release. A model is built from the columns it declares, and `marketdata.get_extra()` returns the others, named as the API sent them, until a release declares them: + +```python +import marketdata +from marketdata import MarketDataClient, OutputFormat + +client = MarketDataClient() +chain = client.options.chain("AAPL", output_format=OutputFormat.INTERNAL) +marketdata.get_extra(chain) # {} while the model declares every column + +quotes = client.stocks.quotes("AAPL", output_format=OutputFormat.INTERNAL) +marketdata.get_extra(quotes[0]) # the row's value of each column StockQuote lacks +``` + +- **Which results:** every `OutputFormat.INTERNAL` result built from an answer's columns, the merged answers of `stocks.candles()` and `options.quotes()` included, and a model built with `from_dict()` on `StockEarnings`, `StockPrice`, `StockQuote` and their human-readable twins. A single-object result (`options.chain()`, `options.expirations()`, `options.lookup()`, `options.quotes()`, `stocks.earnings()`) holds each column whole; a list result (`stocks.quotes()`, `stocks.prices()`, `stocks.candles()`, `stocks.news()`, `funds.candles()`, `markets.status()`, `utilities.status()`) gives each row its own value. `utilities.user()` and `utilities.headers()` keep nothing there: the first reads only its own keys, the second keeps every key as a header. +- **The values:** decoded as the rest of the answer, so a number with a fraction is a `Decimal` on the resources whose money is. Each answer that brings such a column names it once at DEBUG on `marketdata.logger`. +- **How long they last:** as long as the object, which holds them itself, like the response metadata: they survive `copy.copy()`, `copy.deepcopy()` and `pickle`. They are not fields, so `==`, the repr and `dataclasses.fields()` ignore them, and a model built by hand, or from an answer without such a column, returns `{}`. +- **The other formats:** `JSON` and `DATAFRAME` carry the column by the API's name and `CSV` writes it in the file. On `stocks.candles()` and `options.quotes()` every chunk or symbol must carry it, as it must carry a declared column: an answer without it fails the call with `ParseError`, on every format. +- **One value per row:** like a declared column, an added one that is a list must be as long as the others. A list longer or shorter than the rest fails the call with `ParseError` on `INTERNAL`, `JSON` and `DATAFRAME`, whatever the resource, and on the CSV of `utilities.status()`, which is written from the decoded answer. A single value instead of a list is kept as sent, on every row of a list result; the merges of `stocks.candles()` and `options.quotes()` carry only lists. + ## Universal Parameters All resource methods support universal parameters that can be used to customize the API request and response: diff --git a/docs/funds.md b/docs/funds.md index 10e57d4..6e1cde9 100644 --- a/docs/funds.md +++ b/docs/funds.md @@ -13,6 +13,10 @@ funds = client.funds All methods in the funds resource include API status checking and automatic retry logic. See the [README](../README.md) for general information about error handling, retry mechanisms, and output formats. +## Columns a model does not declare + +With `OutputFormat.INTERNAL`, a column the API sends before the SDK declares it does not fail the call: the result is built from the columns its model declares, and `marketdata.get_extra(candle)` returns the others, named as the API sent them, with each row's own value. `JSON` and `DATAFRAME` carry it as a column, and `CSV` writes it in the file. See [Columns a model does not declare](../README.md#columns-a-model-does-not-declare) in the README for how long the values last and how they are decoded. + ## Methods ### `candles()` diff --git a/docs/markets.md b/docs/markets.md index e99ceb6..787089b 100644 --- a/docs/markets.md +++ b/docs/markets.md @@ -13,6 +13,10 @@ markets = client.markets All methods in the markets resource include API status checking and automatic retry logic. See the [README](../README.md) for general information about error handling, retry mechanisms, and output formats. +## Columns a model does not declare + +With `OutputFormat.INTERNAL`, a column the API sends before the SDK declares it does not fail the call: the result is built from the columns its model declares, and `marketdata.get_extra(row)` returns the others, named as the API sent them, with each row's own value. `JSON` and `DATAFRAME` carry it as a column, and `CSV` writes it in the file. See [Columns a model does not declare](../README.md#columns-a-model-does-not-declare) in the README for how long the values last and how they are decoded. + ## Methods ### `status()` diff --git a/docs/options.md b/docs/options.md index c16f1eb..648cd37 100644 --- a/docs/options.md +++ b/docs/options.md @@ -13,6 +13,10 @@ options = client.options All methods in the options resource include API status checking and automatic retry logic. See the [README](../README.md) for general information about error handling, retry mechanisms, and output formats. +## Columns a model does not declare + +With `OutputFormat.INTERNAL`, a column the API sends before the SDK declares it does not fail the call: the result is built from the columns its model declares, and `marketdata.get_extra(result)` returns the others, named as the API sent them, each column whole, since `chain()`, `expirations()`, `lookup()` and `quotes()` each return one object. On `quotes()` every symbol must send such a column, as it must send a declared one, or the call raises `ParseError`. `JSON` and `DATAFRAME` carry it as a column, and `CSV` writes it in the file. See [Columns a model does not declare](../README.md#columns-a-model-does-not-declare) in the README for how long the values last and how they are decoded. + ## Methods ### `expirations()` diff --git a/docs/stocks.md b/docs/stocks.md index 69a1f38..d9ac434 100644 --- a/docs/stocks.md +++ b/docs/stocks.md @@ -13,6 +13,10 @@ stocks = client.stocks All methods in the stocks resource include API status checking and automatic retry logic. See the [README](../README.md) for general information about error handling, retry mechanisms, and output formats. +## Columns a model does not declare + +With `OutputFormat.INTERNAL`, a column the API sends before the SDK declares it does not fail the call: the result is built from the columns its model declares, and `marketdata.get_extra()` returns the others, named as the API sent them: on each row of `prices()`, `quotes()`, `candles()` and `news()` the row's own value, and on `earnings()`, which returns one object, the whole column. On `candles()` every chunk must send such a column, as it must send a declared one, or the call raises `ParseError`. The `from_dict()` of `StockEarnings`, `StockPrice`, `StockQuote` and their human-readable twins keep such a key the same way. `JSON` and `DATAFRAME` carry it as a column, and `CSV` writes it in the file. See [Columns a model does not declare](../README.md#columns-a-model-does-not-declare) in the README for how long the values last and how they are decoded. + ## Methods ### `prices()` diff --git a/docs/utilities.md b/docs/utilities.md index df61a36..4df109c 100644 --- a/docs/utilities.md +++ b/docs/utilities.md @@ -15,6 +15,10 @@ All three methods include automatic retry logic. They never consume API credits, These endpoints live at the API root (`/status/`, `/headers/`, `/user/`), only speak JSON and accept no query parameters. The SDK therefore sends them bare and produces every output format client-side from the decoded JSON. Of the universal parameters, only `output_format` and `filename` apply. +## Columns a model does not declare + +With `OutputFormat.INTERNAL`, a column the API sends before the SDK declares it does not fail the call: the result is built from the columns its model declares, and `marketdata.get_extra(row)` returns the others on each row of `status()`, named as the API sent them, with the row's own value. `user()` reads only its own keys and `headers()` keeps every key as a header, so neither keeps anything there. `JSON` and `DATAFRAME` carry it as a column, and `CSV` writes it in the file. See [Columns a model does not declare](../README.md#columns-a-model-does-not-declare) in the README for how long the values last and how they are decoded. + ## Methods ### `status()` diff --git a/src/marketdata/__init__.py b/src/marketdata/__init__.py index a8dcdf3..acf8140 100644 --- a/src/marketdata/__init__.py +++ b/src/marketdata/__init__.py @@ -20,6 +20,7 @@ from marketdata.input_types.base import DateFormat, Mode, OutputFormat from marketdata.input_types.filters import DeltaFilter, StrikeFilter from marketdata.meta import ResponseMeta, get_meta +from marketdata.output_types.columns import get_extra from marketdata.types import UserRateLimits __all__ = [ @@ -32,6 +33,7 @@ "ResponseMeta", "UserRateLimits", "get_meta", + "get_extra", "BaseMarketdataException", "MarketdataHttpError", "BadRequestError", diff --git a/src/marketdata/output_types/columns.py b/src/marketdata/output_types/columns.py index 9e2b770..15e12bf 100644 --- a/src/marketdata/output_types/columns.py +++ b/src/marketdata/output_types/columns.py @@ -1,8 +1,77 @@ +import logging from dataclasses import fields, is_dataclass from typing import Any from marketdata.utils import column_key +# By name, not `get_logger()`: importing the package must not attach a handler. +_logger = logging.getLogger("marketdata.logger") + +_EXTRA_ATTRIBUTE = "_marketdata_extra" + + +def get_extra(model: Any) -> dict[str, Any]: + """Read the columns of an API answer that the model built from it does not + declare. + + Args: + model: A model a resource call returned on ``OutputFormat.INTERNAL``, + one row of a list result, or a model built with ``from_dict``. + + Returns: + Each such column, named as the API sent it, with its value as the + model's answer was decoded: a number with a fraction is a ``Decimal`` + on the resources that decode money exactly. A single-object model + holds the whole column, a row its own value. Empty when there is none. + """ + namespace = getattr(model, "__dict__", None) + if not isinstance(namespace, dict): + return {} + return dict(namespace.get(_EXTRA_ATTRIBUTE, {})) + + +def _with_extra(model: Any, extra: dict[str, Any]) -> Any: + """Keep the undeclared columns of an answer on the model built from it. + + Args: + model: The model built from the answer. + extra: The columns the model does not declare, keyed as the API sent + them. + + Returns: + ``model``, which ``get_extra`` reads ``extra`` from. + """ + if extra: + model.__dict__[_EXTRA_ATTRIBUTE] = extra + return model + + +def _with_row_extra(models: list[Any], extra: dict[str, Any]) -> list[Any]: + """Keep the undeclared columns of a column-oriented answer on its rows. + + Args: + models: The models built from the answer's rows, in row order. + extra: The columns the model does not declare, keyed as the API sent + them. + + Returns: + ``models``. A column that is a list as long as the rows gives each row + its own value; any other value goes to every row as the API sent it. + """ + for index, model in enumerate(models): + _with_extra( + model, + { + key: ( + value[index] + if isinstance(value, list) and len(value) == len(models) + else value + ) + for key, value in extra.items() + }, + ) + return models + def _column_names(output_model: type) -> dict[str, str]: """Map each field of an output model to the name of its column in the API's @@ -52,19 +121,36 @@ def _column_types(output_model: type) -> dict[str, Any]: } -def _to_fields(output_model: type, data: dict) -> dict: - """Rename the keys of an API answer from column names to field names. +def _split_fields(output_model: type, data: Any) -> tuple[Any, dict[str, Any]]: + """Split an API answer into the fields of the output model it is built into + and the columns the model does not declare. Args: - output_model: The output model the answer is built into. + output_model: The output model dataclass the answer is built into. data: The answer, keyed by column name. Returns: - The same values keyed by field name. A key that names no column is - kept as it is. + The values of the keys that name a column or a field of the model, + keyed by field name, and every other key but the status flag ``s`` + with its value, as the API sent them. Those are logged at DEBUG, once + per answer. An answer that is not an object, or that carries none of + the model's columns, comes back as it is with no other key, for the + model to refuse. """ - by_column = {column: name for name, column in _column_names(output_model).items()} - return {by_column.get(key, key): value for key, value in data.items()} + columns = _column_names(output_model) + if not isinstance(data, dict) or not any(key in data for key in columns.values()): + return data, {} + by_key = {field.name: field.name for field in fields(output_model)} + by_key.update({column: name for name, column in columns.items()}) + extra = { + key: value for key, value in data.items() if key not in by_key and key != "s" + } + if extra: + _logger.debug( + f"The API sent the columns {list(extra)!r}, which" + f" {output_model.__name__} does not declare; get_extra() holds them" + ) + return {by_key[key]: value for key, value in data.items() if key in by_key}, extra def _model_columns(output_model: type, requested: list[str] | None = None) -> list[str]: diff --git a/src/marketdata/output_types/stocks_earnings.py b/src/marketdata/output_types/stocks_earnings.py index c9ffc4f..91a8ea5 100644 --- a/src/marketdata/output_types/stocks_earnings.py +++ b/src/marketdata/output_types/stocks_earnings.py @@ -3,7 +3,7 @@ from decimal import Decimal from typing import ClassVar -from marketdata.output_types.columns import _to_fields +from marketdata.output_types.columns import _split_fields, _with_extra from marketdata.output_types.money import coerce_numbers from marketdata.utils import format_timestamp @@ -69,7 +69,21 @@ def __str__(self) -> str: @classmethod def from_dict(cls, data: dict) -> "StockEarnings": - return cls(**data) + """Build the model from an API answer keyed by column name. + + Args: + data: The answer, keyed as the API names its columns. + + Returns: + The earnings. A column the model does not declare is kept for ``get_extra``. + + Raises: + TypeError: If a field is missing or a number field holds something + that is not a number. + ValueError: If a value cannot be read as its field's type. + """ + fields, extra = _split_fields(cls, data) + return _with_extra(cls(**fields), extra) @dataclass @@ -130,5 +144,20 @@ def __str__(self) -> str: @classmethod def from_dict(cls, data: dict) -> "StockEarningsHumanReadable": - """Build the model from an API answer keyed by column name.""" - return cls(**_to_fields(cls, data)) + """Build the model from an API answer keyed by column name. + + Args: + data: The earnings, keyed as the API names its + human-readable columns. + + Returns: + The earnings. A column the model does not declare is + kept for ``get_extra``. + + Raises: + TypeError: If a field is missing or a number field holds something + that is not a number. + ValueError: If a value cannot be read as its field's type. + """ + fields, extra = _split_fields(cls, data) + return _with_extra(cls(**fields), extra) diff --git a/src/marketdata/output_types/stocks_prices.py b/src/marketdata/output_types/stocks_prices.py index a9789e9..65fb0e1 100644 --- a/src/marketdata/output_types/stocks_prices.py +++ b/src/marketdata/output_types/stocks_prices.py @@ -3,7 +3,7 @@ from decimal import Decimal from typing import ClassVar -from marketdata.output_types.columns import _to_fields +from marketdata.output_types.columns import _split_fields, _with_extra from marketdata.output_types.money import coerce_numbers from marketdata.utils import format_timestamp @@ -36,7 +36,21 @@ def __str__(self) -> str: @classmethod def from_dict(cls, data: dict) -> "StockPrice": - return cls(**data) + """Build the model from an API answer keyed by column name. + + Args: + data: One price, keyed as the API names its columns. + + Returns: + The price. A column the model does not declare is kept for ``get_extra``. + + Raises: + TypeError: If a field is missing or a number field holds something + that is not a number. + ValueError: If a value cannot be read as its field's type. + """ + fields, extra = _split_fields(cls, data) + return _with_extra(cls(**fields), extra) @dataclass @@ -72,5 +86,20 @@ def __str__(self) -> str: @classmethod def from_dict(cls, data: dict) -> "StockPricesHumanReadable": - """Build the model from an API answer keyed by column name.""" - return cls(**_to_fields(cls, data)) + """Build the model from an API answer keyed by column name. + + Args: + data: One price, keyed as the API names its + human-readable columns. + + Returns: + The price. A column the model does not declare is + kept for ``get_extra``. + + Raises: + TypeError: If a field is missing or a number field holds something + that is not a number. + ValueError: If a value cannot be read as its field's type. + """ + fields, extra = _split_fields(cls, data) + return _with_extra(cls(**fields), extra) diff --git a/src/marketdata/output_types/stocks_quotes.py b/src/marketdata/output_types/stocks_quotes.py index 32481f3..1ab6e46 100644 --- a/src/marketdata/output_types/stocks_quotes.py +++ b/src/marketdata/output_types/stocks_quotes.py @@ -3,7 +3,7 @@ from decimal import Decimal from typing import ClassVar -from marketdata.output_types.columns import _to_fields +from marketdata.output_types.columns import _split_fields, _with_extra from marketdata.output_types.money import coerce_numbers from marketdata.utils import format_timestamp @@ -65,14 +65,15 @@ def from_dict(cls, data: dict) -> "StockQuote": data: One quote, keyed as the API names its columns. Returns: - The quote. + The quote. A column the model does not declare is kept for ``get_extra``. Raises: - TypeError: If a key names no field, a field is missing or a number - field holds something that is not a number. + TypeError: If a field is missing or a number field holds something + that is not a number. ValueError: If a value cannot be read as its field's type. """ - return cls(**_to_fields(cls, data)) + fields, extra = _split_fields(cls, data) + return _with_extra(cls(**fields), extra) @dataclass @@ -129,5 +130,20 @@ def __str__(self) -> str: @classmethod def from_dict(cls, data: dict) -> "StockQuotesHumanReadable": - """Build the model from an API answer keyed by column name.""" - return cls(**_to_fields(cls, data)) + """Build the model from an API answer keyed by column name. + + Args: + data: One quote, keyed as the API names its + human-readable columns. + + Returns: + The quote. A column the model does not declare is + kept for ``get_extra``. + + Raises: + TypeError: If a field is missing or a number field holds something + that is not a number. + ValueError: If a value cannot be read as its field's type. + """ + fields, extra = _split_fields(cls, data) + return _with_extra(cls(**fields), extra) diff --git a/src/marketdata/resources/base.py b/src/marketdata/resources/base.py index 3108b9b..cda3206 100644 --- a/src/marketdata/resources/base.py +++ b/src/marketdata/resources/base.py @@ -1,3 +1,4 @@ +import csv from collections.abc import Callable, Collection, Iterator from contextlib import contextmanager from typing import TYPE_CHECKING, Any @@ -16,6 +17,11 @@ from marketdata.output_types.columns import _column_names, _model_columns from marketdata.settings import settings from marketdata.utils import ( + _check_column_lengths, + _csv_records, + _reads_as_json_or_html, + _unreadable_csv, + column_key, csv_header, parse_error, parse_json, @@ -52,6 +58,86 @@ def model_columns(output_model: type, requested: list[str] | None = None) -> lis return _model_columns(output_model, requested) +def _parse_json_object( + response: Response, output_model: type, *, exact: bool = False +) -> dict[str, Any]: + """Decode a response body that must be a JSON object carrying at least one + of the output model's columns, with every column one length. + + Args: + response: The answer to decode. + output_model: The output model of the resource. + exact: Decode each number with a fraction as a ``Decimal``, as + ``parse_json`` does. + + Returns: + The decoded object. + + Raises: + ParseError: If the body is not valid JSON, holds a number that is not + finite, is not a JSON object, carries none of the model's columns, + or holds lists of different lengths under two of its keys but the + status flag ``s``. + """ + data = parse_json(response, exact=exact) + if not isinstance(data, dict): + raise parse_error(response, "not a JSON object") + if not any(column in data for column in model_columns(output_model)): + raise parse_error(response, "none of this resource's fields") + _check_column_lengths(response, data) + return data + + +def _parse_json_columns(response: Response, *, exact: bool = False) -> Any: + """Decode a column-oriented answer whose columns must all be one length. + + Args: + response: The answer to decode. + exact: Decode each number with a fraction as a ``Decimal``, as + ``parse_json`` does. + + Returns: + The decoded body. + + Raises: + ParseError: If the body is not valid JSON, holds a number that is not + finite, or holds lists of different lengths under two of its keys + but the status flag ``s``. + """ + data = parse_json(response, exact=exact) + _check_column_lengths(response, data) + return data + + +def _check_csv_header( + response: Response, output_model: type, *, with_header: bool +) -> None: + """Refuse a CSV answer whose header names none of the resource's columns. + + Args: + response: The CSV answer. + output_model: The output model of the resource. + with_header: Whether the answer starts with a header row. + + Raises: + ParseError: If the answer starts with a header row that names none of + the model's columns, or with one the ``csv`` module cannot read, + or, without a header row, if it is labeled or shaped as JSON or + HTML. + """ + if not with_header: + if _reads_as_json_or_html(response): + raise parse_error(response, "JSON or HTML, not CSV") + return + try: + incoming = next(_csv_records(response.text), None) + except csv.Error as exc: + raise _unreadable_csv(response, exc) from exc + known = {column_key(name) for name in model_columns(output_model)} + if incoming is not None and not any(column_key(name) in known for name in incoming): + raise parse_error(response, f"unknown columns {incoming!r}") + + @contextmanager def model_errors(response: Response) -> Iterator[None]: """Turn a model refusing a value into a ``ParseError``. diff --git a/src/marketdata/resources/funds/candles.py b/src/marketdata/resources/funds/candles.py index c3f1230..39ae274 100644 --- a/src/marketdata/resources/funds/candles.py +++ b/src/marketdata/resources/funds/candles.py @@ -5,14 +5,19 @@ from marketdata.input_types.base import OutputFormat, UserUniversalAPIParams from marketdata.input_types.funds import FundsCandlesInput from marketdata.output_handlers import get_dataframe_output_handler +from marketdata.output_types.columns import _split_fields, _with_row_extra from marketdata.output_types.funds_candles import FundsCandle, FundsCandlesHumanReadable from marketdata.params import universal_params -from marketdata.resources.base import BaseResource, model_errors, no_data_result +from marketdata.resources.base import ( + BaseResource, + _parse_json_columns, + model_errors, + no_data_result, +) from marketdata.utils import ( encode_path_segment, get_data_records, is_no_data, - parse_json, ) @@ -61,21 +66,22 @@ def candles( ) if user_universal_params.output_format == OutputFormat.DATAFRAME: - data = parse_json(response) + data = _parse_json_columns(response) handler = get_dataframe_output_handler() return handler(data, output_model, user_universal_params).get_result( index_columns=["t", "Date"] ) elif user_universal_params.output_format == OutputFormat.INTERNAL: - data = parse_json(response, exact=True) - data = get_data_records(data, exclude_keys=["s"]) + data = _parse_json_columns(response, exact=True) + fields, extra = _split_fields(output_model, data) + data = get_data_records(fields, exclude_keys=["s"]) with model_errors(response): - return [output_model(**row) for row in data] + return _with_row_extra([output_model(**row) for row in data], extra) elif user_universal_params.output_format == OutputFormat.JSON: - return parse_json(response) + return _parse_json_columns(response) elif user_universal_params.output_format == OutputFormat.CSV: return user_universal_params.write_file(response.text) diff --git a/src/marketdata/resources/markets/status.py b/src/marketdata/resources/markets/status.py index 55b0f40..a39180a 100644 --- a/src/marketdata/resources/markets/status.py +++ b/src/marketdata/resources/markets/status.py @@ -5,13 +5,19 @@ from marketdata.input_types.base import OutputFormat, UserUniversalAPIParams from marketdata.input_types.markets import MarketStatusInput from marketdata.output_handlers import get_dataframe_output_handler +from marketdata.output_types.columns import _split_fields, _with_row_extra from marketdata.output_types.markets_status import ( MarketStatus, MarketStatusHumanReadable, ) from marketdata.params import universal_params -from marketdata.resources.base import BaseResource, model_errors, no_data_result -from marketdata.utils import get_data_records, is_no_data, parse_json +from marketdata.resources.base import ( + BaseResource, + _parse_json_columns, + model_errors, + no_data_result, +) +from marketdata.utils import get_data_records, is_no_data @api_error_handler(service="/v1/markets/status/") @@ -58,20 +64,21 @@ def status( ) if user_universal_params.output_format == OutputFormat.DATAFRAME: - data = parse_json(response) + data = _parse_json_columns(response) handler = get_dataframe_output_handler() return handler(data, output_model, user_universal_params).get_result( index_columns=["Date", "date"] ) elif user_universal_params.output_format == OutputFormat.INTERNAL: - data = parse_json(response) - data = get_data_records(data, exclude_keys=["s"]) + data = _parse_json_columns(response) + fields, extra = _split_fields(output_model, data) + data = get_data_records(fields, exclude_keys=["s"]) with model_errors(response): - return [output_model(**row) for row in data] + return _with_row_extra([output_model(**row) for row in data], extra) elif user_universal_params.output_format == OutputFormat.JSON: - return parse_json(response) + return _parse_json_columns(response) elif user_universal_params.output_format == OutputFormat.CSV: return user_universal_params.write_file(response.text) diff --git a/src/marketdata/resources/options/chain.py b/src/marketdata/resources/options/chain.py index 35602ab..719ec6d 100644 --- a/src/marketdata/resources/options/chain.py +++ b/src/marketdata/resources/options/chain.py @@ -5,14 +5,20 @@ from marketdata.input_types.base import OutputFormat, UserUniversalAPIParams from marketdata.input_types.options import OptionsChainInput from marketdata.output_handlers import get_dataframe_output_handler -from marketdata.output_types.columns import _to_fields +from marketdata.output_types.columns import _split_fields, _with_extra from marketdata.output_types.options_chain import ( OptionsChain, OptionsChainHumanReadable, ) from marketdata.params import universal_params -from marketdata.resources.base import BaseResource, model_errors, no_data_result -from marketdata.utils import encode_path_segment, is_no_data, parse_json +from marketdata.resources.base import ( + BaseResource, + _check_csv_header, + _parse_json_object, + model_errors, + no_data_result, +) +from marketdata.utils import encode_path_segment, is_no_data @api_error_handler(service="/v1/options/chain/") @@ -61,23 +67,27 @@ def chain( ) if user_universal_params.output_format == OutputFormat.DATAFRAME: - data = parse_json(response) + data = _parse_json_object(response, output_model) handler = get_dataframe_output_handler() return handler(data, output_model, user_universal_params).get_result( index_columns=["optionSymbol", "Symbol"] ) elif user_universal_params.output_format == OutputFormat.INTERNAL: - data = parse_json(response, exact=True) - if user_universal_params.use_human_readable: - data = _to_fields(output_model, data) + data = _parse_json_object(response, output_model, exact=True) + fields, extra = _split_fields(output_model, data) with model_errors(response): - return output_model(**data) + return _with_extra(output_model(**fields), extra) elif user_universal_params.output_format == OutputFormat.JSON: - return parse_json(response) + return _parse_json_object(response, output_model) elif user_universal_params.output_format == OutputFormat.CSV: + _check_csv_header( + response, + output_model, + with_header=user_universal_params.add_headers is not False, + ) return user_universal_params.write_file(response.text) # This line should never be reached due to the universal_params decorator validating the output format diff --git a/src/marketdata/resources/options/expirations.py b/src/marketdata/resources/options/expirations.py index 824d03a..5f91daf 100644 --- a/src/marketdata/resources/options/expirations.py +++ b/src/marketdata/resources/options/expirations.py @@ -5,14 +5,20 @@ from marketdata.input_types.base import OutputFormat, UserUniversalAPIParams from marketdata.input_types.options import OptionsExpirationsInput from marketdata.output_handlers import get_dataframe_output_handler -from marketdata.output_types.columns import _to_fields +from marketdata.output_types.columns import _split_fields, _with_extra from marketdata.output_types.options_expirations import ( OptionsExpirations, OptionsExpirationsHumanReadable, ) from marketdata.params import universal_params -from marketdata.resources.base import BaseResource, model_errors, no_data_result -from marketdata.utils import encode_path_segment, is_no_data, parse_json +from marketdata.resources.base import ( + BaseResource, + _check_csv_header, + _parse_json_object, + model_errors, + no_data_result, +) +from marketdata.utils import encode_path_segment, is_no_data @api_error_handler(service="/v1/options/expirations/") @@ -73,23 +79,27 @@ def expirations( ) if user_universal_params.output_format == OutputFormat.DATAFRAME: - data = parse_json(response) + data = _parse_json_object(response, output_model) handler = get_dataframe_output_handler() return handler(data, output_model, user_universal_params).get_result( index_columns=index_columns ) elif user_universal_params.output_format == OutputFormat.INTERNAL: - data = parse_json(response) - if user_universal_params.use_human_readable: - data = _to_fields(output_model, data) + data = _parse_json_object(response, output_model) + fields, extra = _split_fields(output_model, data) with model_errors(response): - return output_model(**data) + return _with_extra(output_model(**fields), extra) elif user_universal_params.output_format == OutputFormat.JSON: - return parse_json(response) + return _parse_json_object(response, output_model) elif user_universal_params.output_format == OutputFormat.CSV: + _check_csv_header( + response, + output_model, + with_header=user_universal_params.add_headers is not False, + ) return user_universal_params.write_file(response.text) # This line should never be reached due to the universal_params decorator validating the output format diff --git a/src/marketdata/resources/options/lookup.py b/src/marketdata/resources/options/lookup.py index 66e9d89..6edf477 100644 --- a/src/marketdata/resources/options/lookup.py +++ b/src/marketdata/resources/options/lookup.py @@ -5,13 +5,20 @@ from marketdata.input_types.base import OutputFormat, UserUniversalAPIParams from marketdata.input_types.options import OptionsLookupInput from marketdata.output_handlers import get_dataframe_output_handler +from marketdata.output_types.columns import _split_fields, _with_extra from marketdata.output_types.options_lookup import ( OptionsLookup, OptionsLookupHumanReadable, ) from marketdata.params import universal_params -from marketdata.resources.base import BaseResource, model_errors, no_data_result -from marketdata.utils import encode_path, is_no_data, parse_json +from marketdata.resources.base import ( + BaseResource, + _check_csv_header, + _parse_json_object, + model_errors, + no_data_result, +) +from marketdata.utils import encode_path, is_no_data @api_error_handler(service="/v1/options/lookup/") @@ -64,21 +71,27 @@ def lookup( ) if user_universal_params.output_format == OutputFormat.DATAFRAME: - data = parse_json(response) + data = _parse_json_object(response, output_model) handler = get_dataframe_output_handler() return handler(data, output_model, user_universal_params).get_result( index_columns=["optionSymbol", "Symbol"] ) elif user_universal_params.output_format == OutputFormat.INTERNAL: - data = parse_json(response) + data = _parse_json_object(response, output_model) + fields, extra = _split_fields(output_model, data) with model_errors(response): - return output_model(**data) + return _with_extra(output_model(**fields), extra) elif user_universal_params.output_format == OutputFormat.JSON: - return parse_json(response) + return _parse_json_object(response, output_model) elif user_universal_params.output_format == OutputFormat.CSV: + _check_csv_header( + response, + output_model, + with_header=user_universal_params.add_headers is not False, + ) return user_universal_params.write_file(response.text) # This line should never be reached due to the universal_params decorator validating the output format diff --git a/src/marketdata/resources/options/quotes.py b/src/marketdata/resources/options/quotes.py index 46a6218..a62f8a2 100644 --- a/src/marketdata/resources/options/quotes.py +++ b/src/marketdata/resources/options/quotes.py @@ -11,7 +11,7 @@ from marketdata.input_types.options import OptionsQuotesInput from marketdata.internal_settings import MAX_CONCURRENT_REQUESTS, VALID_STATUS_CODES from marketdata.output_handlers import get_dataframe_output_handler -from marketdata.output_types.columns import _to_fields +from marketdata.output_types.columns import _split_fields, _with_extra from marketdata.output_types.options_quotes import ( OptionsQuotes, OptionsQuotesHumanReadable, @@ -139,10 +139,11 @@ def _model_alone(response: Response) -> object: [response], alone, output_model.answer_keys() ) joined = output_model.join_dicts(alone, keys) - return output_model(**_to_fields(output_model, joined)) + return output_model(**_split_fields(output_model, joined)[0]) + fields, extra = _split_fields(output_model, data) with merged_model_errors(usable, _model_alone): - return output_model(**_to_fields(output_model, data)) + return _with_extra(output_model(**fields), extra) if user_universal_params.output_format == OutputFormat.JSON: return data diff --git a/src/marketdata/resources/stocks/candles.py b/src/marketdata/resources/stocks/candles.py index 5025e66..ed846b8 100644 --- a/src/marketdata/resources/stocks/candles.py +++ b/src/marketdata/resources/stocks/candles.py @@ -12,6 +12,7 @@ from marketdata.input_types.stocks import StocksCandlesInput from marketdata.internal_settings import MAX_CONCURRENT_REQUESTS from marketdata.output_handlers import get_dataframe_output_handler +from marketdata.output_types.columns import _split_fields, _with_row_extra from marketdata.output_types.stocks_candles import ( StockCandle, StockCandlesHumanReadable, @@ -161,16 +162,20 @@ def _get_responses_data( ) elif user_universal_params.output_format == OutputFormat.INTERNAL: - data = _get_responses_data(responses, exact=True) - data = get_data_records(data, exclude_keys=["s"]) + fields, extra = _split_fields( + output_model, _get_responses_data(responses, exact=True) + ) + data = get_data_records(fields, exclude_keys=["s"]) def _rows_alone(response: httpx.Response) -> list: - alone = _get_responses_data([response], exact=True) + alone = _split_fields( + output_model, _get_responses_data([response], exact=True) + )[0] rows = get_data_records(alone, exclude_keys=["s"]) return [output_model(**row) for row in rows] with merged_model_errors(responses, _rows_alone): - return [output_model(**row) for row in data] + return _with_row_extra([output_model(**row) for row in data], extra) elif user_universal_params.output_format == OutputFormat.JSON: data = _get_responses_data(responses) diff --git a/src/marketdata/resources/stocks/earnings.py b/src/marketdata/resources/stocks/earnings.py index ef3f089..bb2ed27 100644 --- a/src/marketdata/resources/stocks/earnings.py +++ b/src/marketdata/resources/stocks/earnings.py @@ -10,8 +10,14 @@ StockEarningsHumanReadable, ) from marketdata.params import universal_params -from marketdata.resources.base import BaseResource, model_errors, no_data_result -from marketdata.utils import encode_path_segment, is_no_data, parse_json +from marketdata.resources.base import ( + BaseResource, + _check_csv_header, + _parse_json_object, + model_errors, + no_data_result, +) +from marketdata.utils import encode_path_segment, is_no_data @api_error_handler(service="/v1/stocks/earnings/") @@ -60,21 +66,26 @@ def earnings( ) if user_universal_params.output_format == OutputFormat.DATAFRAME: - data = parse_json(response) + data = _parse_json_object(response, output_model) handler = get_dataframe_output_handler() return handler(data, output_model, user_universal_params).get_result( index_columns=["symbol", "Symbol"] ) elif user_universal_params.output_format == OutputFormat.INTERNAL: - data = parse_json(response, exact=True) + data = _parse_json_object(response, output_model, exact=True) with model_errors(response): return output_model.from_dict(data) elif user_universal_params.output_format == OutputFormat.JSON: - return parse_json(response) + return _parse_json_object(response, output_model) elif user_universal_params.output_format == OutputFormat.CSV: + _check_csv_header( + response, + output_model, + with_header=user_universal_params.add_headers is not False, + ) return user_universal_params.write_file(response.text) # This line should never be reached due to the universal_params decorator validating the output format diff --git a/src/marketdata/resources/stocks/news.py b/src/marketdata/resources/stocks/news.py index 547ed88..04300e4 100644 --- a/src/marketdata/resources/stocks/news.py +++ b/src/marketdata/resources/stocks/news.py @@ -5,14 +5,19 @@ from marketdata.input_types.base import OutputFormat, UserUniversalAPIParams from marketdata.input_types.stocks import StocksNewsInput from marketdata.output_handlers import get_dataframe_output_handler +from marketdata.output_types.columns import _split_fields, _with_row_extra from marketdata.output_types.stocks_news import StockNews, StockNewsHumanReadable from marketdata.params import universal_params -from marketdata.resources.base import BaseResource, model_errors, no_data_result +from marketdata.resources.base import ( + BaseResource, + _parse_json_columns, + model_errors, + no_data_result, +) from marketdata.utils import ( encode_path_segment, get_data_records, is_no_data, - parse_json, ) @@ -61,19 +66,20 @@ def news( ) if user_universal_params.output_format == OutputFormat.DATAFRAME: - data = parse_json(response) + data = _parse_json_columns(response) handler = get_dataframe_output_handler() return handler(data, output_model, user_universal_params).get_result( index_columns=["symbol", "Symbol"] ) elif user_universal_params.output_format == OutputFormat.INTERNAL: - data = get_data_records(parse_json(response), exclude_keys=["s"]) + fields, extra = _split_fields(output_model, _parse_json_columns(response)) + data = get_data_records(fields, exclude_keys=["s"]) with model_errors(response): - return [output_model(**row) for row in data] + return _with_row_extra([output_model(**row) for row in data], extra) elif user_universal_params.output_format == OutputFormat.JSON: - return parse_json(response) + return _parse_json_columns(response) elif user_universal_params.output_format == OutputFormat.CSV: return user_universal_params.write_file(response.text) diff --git a/src/marketdata/resources/stocks/prices.py b/src/marketdata/resources/stocks/prices.py index 2674dbc..6187514 100644 --- a/src/marketdata/resources/stocks/prices.py +++ b/src/marketdata/resources/stocks/prices.py @@ -8,10 +8,11 @@ ) from marketdata.input_types.stocks import StocksPricesInput from marketdata.output_handlers import get_dataframe_output_handler +from marketdata.output_types.columns import _split_fields, _with_row_extra from marketdata.output_types.stocks_prices import StockPrice, StockPricesHumanReadable from marketdata.params import universal_params -from marketdata.resources.base import model_errors, no_data_result -from marketdata.utils import get_data_records, is_no_data, parse_json +from marketdata.resources.base import _parse_json_columns, model_errors, no_data_result +from marketdata.utils import get_data_records, is_no_data @api_error_handler(service="/v1/stocks/prices/") @@ -60,19 +61,22 @@ def prices( ) if user_universal_params.output_format == OutputFormat.DATAFRAME: - data = parse_json(response) + data = _parse_json_columns(response) handler = get_dataframe_output_handler() return handler(data, output_model, user_universal_params).get_result( index_columns=["symbol", "Symbol"] ) elif user_universal_params.output_format == OutputFormat.INTERNAL: - data = get_data_records(parse_json(response, exact=True)) + fields, extra = _split_fields( + output_model, _parse_json_columns(response, exact=True) + ) + data = get_data_records(fields) with model_errors(response): - return [output_model.from_dict(row) for row in data] + return _with_row_extra([output_model(**row) for row in data], extra) elif user_universal_params.output_format == OutputFormat.JSON: - return parse_json(response) + return _parse_json_columns(response) elif user_universal_params.output_format == OutputFormat.CSV: return user_universal_params.write_file(response.text) diff --git a/src/marketdata/resources/stocks/quotes.py b/src/marketdata/resources/stocks/quotes.py index 63d6645..f231b3d 100644 --- a/src/marketdata/resources/stocks/quotes.py +++ b/src/marketdata/resources/stocks/quotes.py @@ -5,10 +5,16 @@ from marketdata.input_types.base import OutputFormat, UserUniversalAPIParams from marketdata.input_types.stocks import StocksQuotesInput from marketdata.output_handlers import get_dataframe_output_handler +from marketdata.output_types.columns import _split_fields, _with_row_extra from marketdata.output_types.stocks_quotes import StockQuote, StockQuotesHumanReadable from marketdata.params import universal_params -from marketdata.resources.base import BaseResource, model_errors, no_data_result -from marketdata.utils import get_data_records, is_no_data, parse_json +from marketdata.resources.base import ( + BaseResource, + _parse_json_columns, + model_errors, + no_data_result, +) +from marketdata.utils import get_data_records, is_no_data @api_error_handler(service="/v1/stocks/quotes/") @@ -60,19 +66,22 @@ def quotes( ) if user_universal_params.output_format == OutputFormat.DATAFRAME: - data = parse_json(response) + data = _parse_json_columns(response) handler = get_dataframe_output_handler() return handler(data, output_model, user_universal_params).get_result( index_columns=["symbol", "Symbol"] ) elif user_universal_params.output_format == OutputFormat.INTERNAL: - data = get_data_records(parse_json(response, exact=True), exclude_keys=["s"]) + fields, extra = _split_fields( + output_model, _parse_json_columns(response, exact=True) + ) + data = get_data_records(fields, exclude_keys=["s"]) with model_errors(response): - return [output_model.from_dict(row) for row in data] + return _with_row_extra([output_model(**row) for row in data], extra) elif user_universal_params.output_format == OutputFormat.JSON: - return parse_json(response) + return _parse_json_columns(response) elif user_universal_params.output_format == OutputFormat.CSV: return user_universal_params.write_file(response.text) diff --git a/src/marketdata/resources/utilities/base.py b/src/marketdata/resources/utilities/base.py index 8237e6e..e9ccb2e 100644 --- a/src/marketdata/resources/utilities/base.py +++ b/src/marketdata/resources/utilities/base.py @@ -14,8 +14,15 @@ from marketdata.input_types.base import OutputFormat, UserUniversalAPIParams from marketdata.output_handlers import get_dataframe_output_handler +from marketdata.output_types.columns import _split_fields, _with_row_extra from marketdata.resources.base import BaseResource, model_errors, no_data_result -from marketdata.utils import dict_to_csv, get_data_records, is_no_data, parse_json +from marketdata.utils import ( + _check_column_lengths, + dict_to_csv, + get_data_records, + is_no_data, + parse_json, +) def resolve_output_params( @@ -44,10 +51,26 @@ def render( as_records: bool, index_columns: list[str] | None = None, ): - """Turn the response into the requested output format. + """Turn a utilities answer into the requested output format. - ``as_records`` distinguishes column-oriented payloads (one row per entry, - like ``/status/``) from flat objects (``/headers/``, ``/user/``). + Args: + user_universal_params: The call's universal parameters. + response: The API's answer. + output_model: The output model of the resource. + as_records: Whether the answer is column-oriented, one row per entry + (``/status/``), rather than a flat object (``/headers/``, + ``/user/``). + index_columns: The index columns of the DataFrame. + + Returns: + The rows or the model on INTERNAL, each row keeping the columns its + model does not declare for ``get_extra``; the decoded answer on JSON; + a DataFrame; or the path of the CSV file, written from the decoded + answer. The empty result for a ``no_data`` answer. + + Raises: + ParseError: If the body is not valid JSON, a column-oriented answer + holds lists of different lengths, or the model refuses it. """ output_format = user_universal_params.output_format @@ -60,6 +83,8 @@ def render( response=response, ) data = parse_json(response) + if as_records: + _check_column_lengths(response, data) if output_format == OutputFormat.DATAFRAME: handler = get_dataframe_output_handler() @@ -69,9 +94,10 @@ def render( elif output_format == OutputFormat.INTERNAL: if as_records: - rows = get_data_records(data, exclude_keys=["s"]) + fields, extra = _split_fields(output_model, data) + rows = get_data_records(fields, exclude_keys=["s"]) with model_errors(response): - return [output_model(**row) for row in rows] + return _with_row_extra([output_model(**row) for row in rows], extra) with model_errors(response): return output_model.from_dict(data) diff --git a/src/marketdata/utils.py b/src/marketdata/utils.py index 151441f..7db6772 100644 --- a/src/marketdata/utils.py +++ b/src/marketdata/utils.py @@ -359,12 +359,14 @@ def merge_csv_responses( Args: responses: The answers, in the order their rows go in the file. - known_columns: The resource's column names. A header name must match - one of them through ``column_key``, which also compares the - headers of two answers, so a different spelling of the same - column still merges. + known_columns: The resource's column names. A header must carry at + least one of them, matched through ``column_key``, which also + compares the headers of two answers, so a different spelling of + the same column still merges. A column of the header that matches + none of them is kept. with_header: Whether the bodies start with a header row. Without one, - only the row width is checked, against the first row seen. + only the row width is checked, against the first row seen, and an + answer labeled or shaped as JSON or HTML is refused. Returns: The first answer's header, if any, then every row of every answer, @@ -373,10 +375,11 @@ def merge_csv_responses( Raises: ParseError: Naming the offending response, for a body the ``csv`` - module cannot read, a body with no header row, a header with a - column the resource does not have or with other columns, or in - another order, than the first answer's, or a row not as wide as - the header. + module cannot read, a body with no header row, a header with none + of the resource's columns or with other columns, or in another + order, than the first answer's, an answer without a header row + labeled or shaped as JSON or HTML, or a row not as wide as the + header. """ known = {column_key(name) for name in known_columns} header: list[str] | None = None @@ -386,15 +389,16 @@ def merge_csv_responses( writer = csv.writer(output) for response in responses: + if not with_header and _reads_as_json_or_html(response): + raise parse_error(response, "JSON or HTML, not CSV") rows = _csv_records(response.text) try: if with_header: incoming = next(rows, None) if incoming is None: raise parse_error(response, "no header row") - unknown = [name for name in incoming if column_key(name) not in known] - if unknown: - raise parse_error(response, f"unknown columns {unknown!r}") + if not any(column_key(name) in known for name in incoming): + raise parse_error(response, f"unknown columns {incoming!r}") incoming_key = [column_key(name) for name in incoming] if header is None: header, header_key, width = incoming, incoming_key, len(incoming) @@ -438,26 +442,54 @@ def _unreadable_csv(response: Response, error: csv.Error) -> ParseError: return parse_error(response, f"unreadable CSV: {error}") +def _reads_as_json_or_html(response: Response) -> bool: + """Tell whether a CSV answer is a JSON value or an HTML page instead. + + Args: + response: The answer. + + Returns: + Whether its ``Content-Type`` names JSON or HTML or, under any other + type or none, its body, past a BOM and white space, starts with + ``[``, ``{`` or ``<``, or is ``null``. + """ + # A one-cell CSV can be valid JSON (1790740800), so only the label tells a + # JSON scalar apart. + media_type = response.headers.get("content-type", "").partition(";")[0] + media_type = media_type.strip().lower() + if media_type.endswith(("/json", "+json")) or media_type in ( + "text/html", + "application/xhtml+xml", + ): + return True + body = response.text.lstrip(BOM).strip() + return body[:1] in ("[", "{", "<") or body == "null" + + def json_answer_columns( responses: list[Response], answers: list[Any], keys: list[str] ) -> list[str]: - """The columns a fan-out merges from its decoded JSON answers (#90). - - They are the ``keys`` (the model's columns) that any answer carries, keys - the model does not know being left out. Since every answer must carry - them all, a successful merge has the first answer's columns in the first - answer's order: under ``columns=`` the API sends the requested columns - only, in the order they were requested, which is also the order of the - empty result and of every single-request resource. Every answer must be a - JSON object - carrying all of them as lists of one length, since the merge concatenates - column by column and a missing or short column would shift every later - row into the wrong symbol or chunk. Anything else raises ``ParseError`` - naming the offending response: a body that is not an object (``null``, a - list), answers with none of the keys (a proxy's JSON error page), an - answer missing a column another one carries, whichever it is, or a column - that is not a list of the same length as the others. A merge with no - columns would read as "no data" (#82). + """List the columns a fan-out merges from its decoded JSON answers. + + Every answer must carry every column as a list, all of one length, since + the merge concatenates column by column and a missing or short column + would shift every later row into the wrong symbol or chunk. + + Args: + responses: The answers, in merge order. + answers: Their decoded bodies. + keys: The model's columns. + + Returns: + The ``keys`` any answer carries, and every other key but the status + flag ``s`` that holds a list, in the order the answers first carry + them: under ``columns=``, the order the columns were requested in. + + Raises: + ParseError: Naming the offending response, for a body that is not a + JSON object, answers with none of ``keys`` (a proxy's JSON error + page), an answer missing a column another one carries, or a + column that is not a list as long as the others. """ for response, answer in zip(responses, answers, strict=True): if not isinstance(answer, dict): @@ -465,10 +497,12 @@ def json_answer_columns( known = set(keys) columns: list[str] = [] for answer in answers: - for key in answer: - if key in known and key not in columns: + for key, value in answer.items(): + if key not in columns and ( + key in known or (key != "s" and isinstance(value, list)) + ): columns.append(key) - if not columns: + if not any(key in known for key in columns): raise parse_error(responses[0], "none of this resource's fields") for response, answer in zip(responses, answers): missing = [key for key in columns if key not in answer] @@ -477,12 +511,37 @@ def json_answer_columns( not_lists = [key for key in columns if not isinstance(answer[key], list)] if not_lists: raise parse_error(response, f"columns {not_lists!r} are not lists") - lengths = {key: len(answer[key]) for key in columns} - if len(set(lengths.values())) > 1: - raise parse_error(response, f"columns of different lengths {lengths!r}") + _check_column_lengths(response, answer, columns) return columns +def _check_column_lengths( + response: Response, answer: Any, columns: list[str] | None = None +) -> None: + """Refuse a column-oriented answer whose columns differ in length. + + Args: + response: The answer, named in the error. + answer: Its decoded body. A body that is not an object passes. + columns: The keys to compare, in the order the error lists them; by + default every key but the status flag ``s`` that holds a list. + + Raises: + ParseError: If two of those keys hold lists of different lengths. + """ + if not isinstance(answer, dict): + return + if columns is None: + columns = [ + key + for key, value in answer.items() + if key != "s" and isinstance(value, list) + ] + lengths = {key: len(answer[key]) for key in columns} + if len(set(lengths.values())) > 1: + raise parse_error(response, f"columns of different lengths {lengths!r}") + + _ONE_DAY = datetime.timedelta(days=1) diff --git a/src/tests/test_money_decimal.py b/src/tests/test_money_decimal.py index 4b42501..0f88ef6 100644 --- a/src/tests/test_money_decimal.py +++ b/src/tests/test_money_decimal.py @@ -7,6 +7,7 @@ """ import array +import copy import csv import dataclasses import datetime @@ -15,6 +16,7 @@ import json import math import pathlib +import pickle import pkgutil import typing from decimal import Decimal, InvalidOperation, localcontext @@ -347,12 +349,24 @@ def _load_fixture(name: str) -> dict: return json.loads(path.read_text(encoding="utf-8")) -def _respond(respx_mock, case: Case, body: bytes | None = None) -> None: +def _respond( + respx_mock, + case: Case, + body: bytes | None = None, + content_type: str | None = "application/json", +) -> None: + """Answer every request to a case's URL with one body. + + Args: + respx_mock: The router. + case: The resource. + body: The body; the case's fixture as JSON when omitted. + content_type: The ``Content-Type`` header, or ``None`` for none. + """ if body is None: body = json.dumps(_load_fixture(case.fixture)).encode() - respx_mock.get(case.url).respond( - content=body, headers={"content-type": "application/json"} - ) + headers = {"content-type": content_type} if content_type else {} + respx_mock.get(case.url).respond(content=body, headers=headers) def _holds_decimal(value) -> bool: @@ -684,26 +698,16 @@ def test_a_value_the_model_cannot_hold_is_a_parse_error_not_a_builtin( @pytest.mark.parametrize( - "name, change, refusal", - [ - ("options.lookup", "extra", "got an unexpected keyword argument 'extra'"), - ("markets.status", "extra", "got an unexpected keyword argument 'extra'"), - ("utilities.user", "missing", "missing 1 required positional argument"), - ], + "name", ["options.expirations", "markets.status", "utilities.user"] ) -def test_a_body_whose_keys_the_model_does_not_take_is_a_parse_error( - respx_mock, client, name, change, refusal +def test_a_body_missing_a_column_the_model_needs_is_a_parse_error( + respx_mock, client, name ): - """A key the model does not have, or one it lacks, used to escape as a - bare `TypeError` from the model's constructor (#50 review). What the SDK - should do with a column the API adds is #111; this pins only that the - refusal is an SDK exception.""" + """A column the model needs and the body lacks is refused with an SDK + exception, not a bare `TypeError` from the model's constructor.""" case = OTHER_CASES[name] data = _load_fixture(case.fixture) - if change == "extra": - data["extra"] = data[next(key for key in data if key != "s")] - else: - del data[next(key for key in data if key != "s")] + del data[next(key for key in data if key != "s")] _respond(respx_mock, case, json.dumps(data).encode()) with pytest.raises(ParseError) as failure: @@ -711,7 +715,490 @@ def test_a_body_whose_keys_the_model_does_not_take_is_a_parse_error( message = failure.value.message assert message.startswith("Response body is not a valid answer of this resource (") - assert refusal in message + assert "missing 1 required positional argument" in message + + +# Every INTERNAL resource whose answer is built into a model through its +# columns: `utilities.headers` keeps every key as a header, and +# `utilities.user` reads only the keys it knows. +MODEL_CASES = { + name: case + for name, case in {**MONEY_CASES, **OTHER_CASES}.items() + if name not in ("utilities.headers", "utilities.user") +} + + +def _undeclared_columns_logged(caplog) -> list[str]: + return [ + record.getMessage() + for record in caplog.records + if "does not declare" in record.getMessage() + ] + + +def _add_new_column(data: dict) -> list[str] | str: + """Add a column no model declares to an API answer. + + Args: + data: The answer, keyed by column name; changed in place. + + Returns: + The column's value: one text per row when the answer's first column + is a list, one text otherwise. + """ + first = data[next(key for key in data if key != "s")] + data["brandNew"] = ( + [f"new {index}" for index in range(len(first))] + if isinstance(first, list) + else "new" + ) + return data["brandNew"] + + +@pytest.mark.parametrize("name", MODEL_CASES) +def test_a_column_no_model_declares_is_kept_for_get_extra( + respx_mock, client, caplog, name +): + """A column the API adds before the model declares it does not break the + call: the fields are the ones the answer builds without it, ``get_extra`` + holds the column, a row its own value, and the column is named once at + DEBUG however many rows the answer has, the fan-outs' merged answer + included.""" + case = MODEL_CASES[name] + data = _load_fixture(case.fixture) + _respond(respx_mock, case, json.dumps(data).encode()) + with caplog.at_level("DEBUG", logger="marketdata.logger"): + expected = case.call(client, output_format=OutputFormat.INTERNAL) + assert _undeclared_columns_logged(caplog) == [] + + column = _add_new_column(data) + _respond(respx_mock, case, json.dumps(data).encode()) + caplog.clear() + with caplog.at_level("DEBUG", logger="marketdata.logger"): + result = case.call(client, output_format=OutputFormat.INTERNAL) + + assert result == expected + models = result if isinstance(result, list) else [result] + extras = [marketdata.get_extra(model) for model in models] + if isinstance(result, list) and isinstance(column, list): + assert extras == [{"brandNew": value} for value in column] + else: + assert extras == [{"brandNew": column}] * len(models) + assert _undeclared_columns_logged(caplog) == [ + f"The API sent the columns ['brandNew'], which {type(models[0]).__name__}" + " does not declare; get_extra() holds them" + ] + + +@pytest.mark.parametrize("name", MODEL_CASES) +def test_a_column_no_model_declares_reaches_every_format( + respx_mock, client, tmp_path, name +): + """An answer with a column the model does not declare carries it on every + format, the fan-outs' merge included: INTERNAL through ``get_extra``, JSON + and DataFrame as a column, and CSV in the file, which is the API's text + or, on the utilities, the file written from the decoded answer.""" + case = MODEL_CASES[name] + data = _load_fixture(case.fixture) + _add_new_column(data) + _respond(respx_mock, case, json.dumps(data).encode()) + + result = case.call(client, output_format=OutputFormat.INTERNAL) + as_json = case.call(client, output_format=OutputFormat.JSON) + frame = case.call(client, output_format=OutputFormat.DATAFRAME) + if not name.startswith("utilities."): + first = next(key for key in data if key != "s") + _respond(respx_mock, case, f"{first},brandNew\r\nx,new 0\r\n".encode()) + path = case.call( + client, output_format=OutputFormat.CSV, filename=tmp_path / "answer.csv" + ) + + models = result if isinstance(result, list) else [result] + assert all("brandNew" in marketdata.get_extra(model) for model in models) + assert as_json["brandNew"] == data["brandNew"] + assert "brandNew" in frame.columns + header = pathlib.Path(path).read_text(encoding="utf-8").splitlines()[0] + assert "brandNew" in header.split(",") + + +def test_an_undeclared_value_that_is_not_a_list_goes_to_every_row(respx_mock, client): + """A column the model does not declare that holds one value instead of a + list gives that value to every row, on every format that decodes it.""" + case = MODEL_CASES["funds.candles"] + data = _load_fixture(case.fixture) + data["brandNew"] = "new" + _respond(respx_mock, case, json.dumps(data).encode()) + + rows = case.call(client, output_format=OutputFormat.INTERNAL) + as_json = case.call(client, output_format=OutputFormat.JSON) + frame = case.call(client, output_format=OutputFormat.DATAFRAME) + + assert [marketdata.get_extra(row) for row in rows] == [{"brandNew": "new"}] * 7 + assert as_json["brandNew"] == "new" + assert list(frame["brandNew"]) == ["new"] * 7 + + +def _list_columns(data: dict) -> list[str]: + """Name the keys of a decoded answer that hold a list, but ``s``. + + Args: + data: The decoded answer. + + Returns: + The keys, in the answer's order. + """ + return [ + key for key, value in data.items() if key != "s" and isinstance(value, list) + ] + + +def _parse_errors_on_decoded_formats( + client, case: Case, path: pathlib.Path +) -> set[str]: + """Call a resource on every format that decodes its answer. + + Args: + client: The client. + case: The resource. + path: The file ``utilities.status`` writes its CSV to, from the decoded + answer. + + Returns: + The messages of the ``ParseError`` each call raised. + """ + calls = [ + {"output_format": OutputFormat.INTERNAL}, + {"output_format": OutputFormat.JSON}, + {"output_format": OutputFormat.DATAFRAME}, + ] + if case is MODEL_CASES["utilities.status"]: + calls.append({"output_format": OutputFormat.CSV, "filename": path}) + messages = set() + for kwargs in calls: + with pytest.raises(ParseError) as failure: + case.call(client, **kwargs) + messages.add(failure.value.message) + return messages + + +UNEVEN_CASES = [name for name in MODEL_CASES if name != "options.lookup"] +SHORT_CASES = [name for name in UNEVEN_CASES if "expirations" not in name] + + +@pytest.mark.parametrize("name", UNEVEN_CASES) +def test_an_undeclared_list_of_another_length_is_a_parse_error_on_every_format( + respx_mock, client, tmp_path, name +): + """A column the model does not declare whose list has one value more than + the rows raises one ``ParseError`` on INTERNAL, JSON and DataFrame, and on + the CSV ``utilities.status`` writes from the decoded answer.""" + case = MODEL_CASES[name] + data = _load_fixture(case.fixture) + rows = len(data[_list_columns(data)[0]]) + data["brandNew"] = list(range(rows + 1)) + _respond(respx_mock, case, json.dumps(data).encode()) + + messages = _parse_errors_on_decoded_formats(client, case, tmp_path / "a.csv") + + assert len(messages) == 1 + message = messages.pop() + assert "(columns of different lengths {" in message + assert f"'brandNew': {rows + 1}}})" in message + + +@pytest.mark.parametrize("name", SHORT_CASES) +def test_a_declared_column_of_another_length_is_a_parse_error_on_every_format( + respx_mock, client, tmp_path, name +): + """A column the model declares that is one value short of the others + raises one ``ParseError`` on INTERNAL, JSON and DataFrame, and on the CSV + ``utilities.status`` writes from the decoded answer.""" + case = MODEL_CASES[name] + data = _load_fixture(case.fixture) + first = _list_columns(data)[0] + rows = len(data[first]) + data[first] = data[first][:-1] + _respond(respx_mock, case, json.dumps(data).encode()) + + messages = _parse_errors_on_decoded_formats(client, case, tmp_path / "a.csv") + + assert len(messages) == 1 + message = messages.pop() + assert "(columns of different lengths {" in message + assert f"{first!r}: {rows - 1}" in message + + +def test_get_extra_is_empty_for_an_object_with_no_dict(): + """``get_extra`` finds nothing on an object that has no ``__dict__`` to + hold an undeclared column.""" + assert marketdata.get_extra(None) == {} + assert marketdata.get_extra(object()) == {} + + +@pytest.mark.parametrize("name", MODEL_CASES) +def test_an_object_with_none_of_the_model_columns_is_a_parse_error( + respx_mock, client, name +): + """An object that carries none of the model's columns, like a proxy's JSON + error page, is not an answer of this resource, so no row is built from + what is left of it.""" + case = MODEL_CASES[name] + _respond(respx_mock, case, json.dumps({"error": "upstream timeout"}).encode()) + + with pytest.raises(ParseError) as failure: + case.call(client, output_format=OutputFormat.INTERNAL) + + assert failure.value.message.startswith( + "Response body is not a valid answer of this resource (" + ) + + +SINGLE_OBJECT_MODEL_CASES = [ + "options.chain", + "options.chain human", + "options.expirations", + "options.expirations human", + "options.lookup", + "stocks.earnings", + "stocks.earnings human", +] + + +@pytest.mark.parametrize("name", SINGLE_OBJECT_MODEL_CASES) +def test_a_foreign_object_is_a_parse_error_on_every_format( + respx_mock, client, tmp_path, name +): + """A single-object resource refuses a JSON object that carries none of its + columns, like a proxy's JSON error page, on every format: the same + ``ParseError`` on the three that decode it, and on CSV the one a header + with none of the resource's columns gets or, without a header row, the one + a body that reads as JSON gets.""" + case = MODEL_CASES[name] + text = json.dumps({"error": "upstream timeout"}) + _respond(respx_mock, case, text.encode()) + + messages = set() + for output_format in ( + OutputFormat.INTERNAL, + OutputFormat.JSON, + OutputFormat.DATAFRAME, + ): + with pytest.raises(ParseError) as failure: + case.call(client, output_format=output_format) + messages.add(failure.value.message) + with pytest.raises(ParseError) as refused: + case.call( + client, output_format=OutputFormat.CSV, filename=tmp_path / "answer.csv" + ) + with pytest.raises(ParseError) as headerless: + case.call( + client, + output_format=OutputFormat.CSV, + add_headers=False, + filename=tmp_path / "answer.csv", + ) + + assert messages == { + "Response body is not a valid answer of this resource (none of this" + f" resource's fields): {text!r}" + } + assert refused.value.message == ( + "Response body is not a valid answer of this resource (unknown columns" + f" [{text!r}]): {text!r}" + ) + assert "(JSON or HTML, not CSV)" in headerless.value.message + assert not (tmp_path / "answer.csv").exists() + + +@pytest.mark.parametrize("body", [b"[1]", b"null", b'"x"', b"5", b"true"]) +@pytest.mark.parametrize("name", SINGLE_OBJECT_MODEL_CASES) +def test_a_body_that_is_not_an_object_is_a_parse_error_on_every_format( + respx_mock, client, tmp_path, name, body +): + """A single-object resource refuses a JSON answer that is not an object on + every format, under the API's names and the human-readable ones: the same + ``ParseError`` on the three that decode it, and on CSV the one a header + with none of the resource's columns gets or, without a header row, the one + a JSON answer gets.""" + case = MODEL_CASES[name] + _respond(respx_mock, case, body) + + messages = set() + for output_format in ( + OutputFormat.INTERNAL, + OutputFormat.JSON, + OutputFormat.DATAFRAME, + ): + with pytest.raises(ParseError) as failure: + case.call(client, output_format=output_format) + messages.add(failure.value.message) + with pytest.raises(ParseError) as refused: + case.call( + client, output_format=OutputFormat.CSV, filename=tmp_path / "answer.csv" + ) + with pytest.raises(ParseError) as headerless: + case.call( + client, + output_format=OutputFormat.CSV, + add_headers=False, + filename=tmp_path / "answer.csv", + ) + + text = body.decode() + header = next(csv.reader([text])) + assert messages == { + "Response body is not a valid answer of this resource (not a JSON object):" + f" {text!r}" + } + assert refused.value.message == ( + "Response body is not a valid answer of this resource (unknown columns" + f" {header!r}): {text!r}" + ) + assert headerless.value.message == ( + "Response body is not a valid answer of this resource (JSON or HTML, not" + f" CSV): {text!r}" + ) + assert not (tmp_path / "answer.csv").exists() + + +SINGLE_OBJECT_CASES = [ + "options.chain", + "options.expirations", + "options.lookup", + "stocks.earnings", +] +FAN_OUT_CASES = ["stocks.candles", "options.quotes"] + + +@pytest.mark.parametrize( + "body", [b"AAPL,200\r\n", b"1790740800\r\n"], ids=["a row", "one cell"] +) +@pytest.mark.parametrize("name", SINGLE_OBJECT_CASES + FAN_OUT_CASES) +def test_a_csv_without_a_header_row_is_written_as_it_came( + respx_mock, client, tmp_path, name, body +): + """Under ``add_headers=False`` an answer labeled CSV has no header row to + check, so the file is the API's text as it came, a lone number that also + reads as JSON included.""" + case = MODEL_CASES[name] + _respond(respx_mock, case, body, "text/csv; charset=utf-8") + + path = case.call( + client, + output_format=OutputFormat.CSV, + add_headers=False, + filename=tmp_path / "answer.csv", + ) + + assert pathlib.Path(path).read_bytes() == body + + +@pytest.mark.parametrize( + ("body", "content_type"), + [ + (b'{"error": "upstream timeout"}', None), + (b"502 Bad Gateway", "text/plain; charset=utf-8"), + (b"502 Bad Gateway", "text/html; charset=utf-8"), + (b'"x"', "application/problem+json"), + ], + ids=[ + "a JSON object, unlabeled", + "an HTML page labeled text", + "text labeled HTML", + "a string labeled problem+json", + ], +) +def test_a_headerless_csv_that_reads_as_json_or_html_is_a_parse_error( + respx_mock, client, tmp_path, body, content_type +): + """Under ``add_headers=False`` an answer labeled JSON or HTML, or whose + body reads as either under another label or none, is not a CSV answer, so + it is refused and no file is written.""" + case = MODEL_CASES["options.chain"] + _respond(respx_mock, case, body, content_type) + + with pytest.raises(ParseError) as refused: + case.call( + client, + output_format=OutputFormat.CSV, + add_headers=False, + filename=tmp_path / "answer.csv", + ) + + assert "(JSON or HTML, not CSV)" in refused.value.message + assert not (tmp_path / "answer.csv").exists() + + +@pytest.mark.parametrize( + ("body", "content_type"), + [ + (b"[1]", None), + (b'"x"', "application/json"), + (b"5", "application/json"), + (b"true", "application/json"), + ], + ids=["a list, unlabeled", "a string", "a number", "a boolean"], +) +@pytest.mark.parametrize("name", FAN_OUT_CASES) +def test_a_fan_out_headerless_csv_that_reads_as_json_is_a_parse_error( + respx_mock, client, tmp_path, name, body, content_type +): + """The fan-outs' merge refuses a headerless answer labeled JSON, or whose + body reads as JSON under no label, as their JSON output refuses a body + that is not an object.""" + case = MODEL_CASES[name] + _respond(respx_mock, case, body, content_type) + + with pytest.raises(ParseError) as refused: + case.call( + client, + output_format=OutputFormat.CSV, + add_headers=False, + filename=tmp_path / "answer.csv", + ) + with pytest.raises(ParseError) as decoded: + case.call(client, output_format=OutputFormat.JSON) + + assert "(JSON or HTML, not CSV)" in refused.value.message + assert "(not a JSON object)" in decoded.value.message + assert not (tmp_path / "answer.csv").exists() + + +def test_get_extra_survives_copy_and_pickle(respx_mock, client): + """The columns a model does not declare live on the model itself, so a + copy, a deep copy and a pickled model keep them.""" + case = MODEL_CASES["stocks.quotes"] + data = _load_fixture(case.fixture) + _add_new_column(data) + _respond(respx_mock, case, json.dumps(data).encode()) + + quote = case.call(client, output_format=OutputFormat.INTERNAL)[0] + + for twin in ( + copy.copy(quote), + copy.deepcopy(quote), + pickle.loads(pickle.dumps(quote)), + ): + assert marketdata.get_extra(twin) == {"brandNew": "new 0"} + + +def test_a_csv_header_the_csv_module_cannot_read_is_a_parse_error( + respx_mock, client, tmp_path +): + """A header past the ``csv`` module's field size limit cannot be read, so + the answer is refused and no file is written.""" + case = MODEL_CASES["options.chain"] + _respond(respx_mock, case, b"x" * (csv.field_size_limit() + 1)) + + with pytest.raises(ParseError) as refused: + case.call( + client, output_format=OutputFormat.CSV, filename=tmp_path / "answer.csv" + ) + + assert refused.value.message.startswith( + "Response body is not a valid answer of this resource (unreadable CSV: " + ) + assert not (tmp_path / "answer.csv").exists() @pytest.mark.parametrize("name", OTHER_DATE_KEYS.keys()) diff --git a/src/tests/test_options_chain.py b/src/tests/test_options_chain.py index 9b82572..3b2fe37 100644 --- a/src/tests/test_options_chain.py +++ b/src/tests/test_options_chain.py @@ -324,14 +324,19 @@ def test_get_options_chain_status_offline(load_json, respx_mock, client): def test_get_options_chain_response_200_csv(respx_mock, client): + """A CSV answer whose header names the resource's columns is written to + the file as it came.""" respx_mock.get("https://api.marketdata.app/v1/options/chain/AAPL/").respond( - text="AS RECEIVED FROM API", + text="optionSymbol,underlying,strike\nAAPL260116C00200000,AAPL,200\n", status_code=200, ) output = client.options.chain( "AAPL", output_format=OutputFormat.CSV, filename="test.csv" ) - assert pathlib.Path(output).read_text() == "AS RECEIVED FROM API" + assert ( + pathlib.Path(output).read_text() + == "optionSymbol,underlying,strike\nAAPL260116C00200000,AAPL,200\n" + ) def test_options_chain_input_date_range_aliases_on_wire(load_json, respx_mock, client): diff --git a/src/tests/test_options_expirations.py b/src/tests/test_options_expirations.py index 0ee0e8c..d9c37a3 100644 --- a/src/tests/test_options_expirations.py +++ b/src/tests/test_options_expirations.py @@ -332,11 +332,16 @@ def test_get_options_expirations_internal_ignores_a_column_filter(respx_mock, cl def test_get_options_expirations_response_200_csv(respx_mock, client): + """A CSV answer whose header names the resource's columns is written to + the file as it came.""" respx_mock.get("https://api.marketdata.app/v1/options/expirations/AAPL/").respond( - text="AS RECEIVED FROM API", + text="expirations,updated\n2026-01-16,1769000000\n", status_code=200, ) output = client.options.expirations( symbol="AAPL", output_format=OutputFormat.CSV, filename="test.csv" ) - assert pathlib.Path(output).read_text() == "AS RECEIVED FROM API" + assert ( + pathlib.Path(output).read_text() + == "expirations,updated\n2026-01-16,1769000000\n" + ) diff --git a/src/tests/test_options_lookup.py b/src/tests/test_options_lookup.py index c1e2fd5..8332525 100644 --- a/src/tests/test_options_lookup.py +++ b/src/tests/test_options_lookup.py @@ -167,10 +167,12 @@ def test_get_options_lookup_status_offline(respx_mock, client): def test_get_options_lookup_response_200_csv(respx_mock, client): + """A CSV answer whose header names the resource's columns is written to + the file as it came.""" respx_mock.get( "https://api.marketdata.app/v1/options/lookup/AAPL 28-00-2023 200.0 call/" ).respond( - text="AS RECEIVED FROM API", + text="optionSymbol\nAAPL230728C00200000\n", status_code=200, ) output = client.options.lookup( @@ -178,4 +180,4 @@ def test_get_options_lookup_response_200_csv(respx_mock, client): output_format=OutputFormat.CSV, filename="test.csv", ) - assert pathlib.Path(output).read_text() == "AS RECEIVED FROM API" + assert pathlib.Path(output).read_text() == "optionSymbol\nAAPL230728C00200000\n" diff --git a/src/tests/test_options_quotes.py b/src/tests/test_options_quotes.py index 9704e1a..1a7aa2c 100644 --- a/src/tests/test_options_quotes.py +++ b/src/tests/test_options_quotes.py @@ -879,6 +879,59 @@ def test_options_quotes_csv_merges_every_symbol_under_the_api_header( ) +def test_options_quotes_csv_keeps_a_column_the_api_adds(respx_mock, client, tmp_path): + """A column the model does not declare is written with the others for + every symbol, as the CSV of a single-request resource carries it.""" + respx_mock.get(CALL_URL).respond(text=f"{CSV_HEADER},brandNew\n{CALL_ROW},x\n") + respx_mock.get(PUT_URL).respond(text=f"{CSV_HEADER},brandNew\n{PUT_ROW},y\n") + + output = client.options.quotes( + symbols=["AAPL271217C00255000", "AAPL271217P00255000"], + output_format=OutputFormat.CSV, + filename=tmp_path / "test.csv", + ) + + assert pathlib.Path(output).read_bytes() == ( + f"{CSV_HEADER},brandNew\r\n{CALL_ROW},x\r\n{PUT_ROW},y\r\n".encode() + ) + + +def test_options_quotes_an_added_column_one_symbol_lacks_is_a_parse_error( + load_json, respx_mock, client, tmp_path +): + """A column the model does not declare must come in every symbol's + answer, as a declared one must: a symbol without it fails the call on + every format, naming that symbol's answer.""" + call = load_json("options_quotes_response_200") + call["brandNew"] = ["x"] * len(call["optionSymbol"]) + respx_mock.get(CALL_URL).respond(json=call, status_code=200) + respx_mock.get(PUT_URL).respond( + json=load_json("options_quotes_response_200"), status_code=200 + ) + + for output_format in ( + OutputFormat.INTERNAL, + OutputFormat.JSON, + OutputFormat.DATAFRAME, + ): + with pytest.raises(ParseError) as failure: + client.options.quotes( + symbols=THREE_SYMBOLS[:2], output_format=output_format + ) + assert failure.value.request_url.startswith(PUT_URL) + assert "missing columns ['brandNew']" in failure.value.message + respx_mock.get(CALL_URL).respond(text=f"{CSV_HEADER},brandNew\n{CALL_ROW},x\n") + respx_mock.get(PUT_URL).respond(text=f"{CSV_HEADER}\n{PUT_ROW}\n") + with pytest.raises(ParseError) as failure: + client.options.quotes( + symbols=THREE_SYMBOLS[:2], + output_format=OutputFormat.CSV, + filename=tmp_path / "test.csv", + ) + assert failure.value.request_url.startswith(PUT_URL) + assert "differs from" in failure.value.message + + def test_options_quotes_csv_keeps_the_requested_columns(respx_mock, client, tmp_path): """Issue #86: under `columns=` the API answers with the requested columns only; the merge used to drop every row because the header did not match diff --git a/src/tests/test_stocks_candles.py b/src/tests/test_stocks_candles.py index af4ff9a..a13a6a3 100644 --- a/src/tests/test_stocks_candles.py +++ b/src/tests/test_stocks_candles.py @@ -516,6 +516,27 @@ def test_stocks_candles_csv_merges_every_chunk_under_the_api_header( assert pathlib.Path(output).read_bytes() == b"t,c\r\n1,2\r\n3,4\r\n" +def test_stocks_candles_csv_keeps_a_column_the_api_adds(respx_mock, client, tmp_path): + """A column the model does not declare is written with the others from + every chunk, as the CSV of a single-request resource carries it.""" + respx_mock.get(HOURLY_URL).mock( + side_effect=by_chunk( + dict(text="t,c,brandNew\r\n1,2,x\r\n"), + dict(text="t,c,brandNew\r\n3,4,y\r\n"), + ) + ) + + output = client.stocks.candles( + symbol="AAPL", + resolution="H", + output_format=OutputFormat.CSV, + filename=tmp_path / "test.csv", + **TWO_CHUNKS, + ) + + assert pathlib.Path(output).read_bytes() == b"t,c,brandNew\r\n1,2,x\r\n3,4,y\r\n" + + def test_stocks_candles_csv_undecodable_chunk_body_is_a_parse_error( respx_mock, client, tmp_path ): diff --git a/src/tests/test_stocks_earnings.py b/src/tests/test_stocks_earnings.py index 7540d16..d0e147d 100644 --- a/src/tests/test_stocks_earnings.py +++ b/src/tests/test_stocks_earnings.py @@ -256,14 +256,19 @@ def test_get_stocks_earnings_status_offline(respx_mock, client): def test_get_stocks_earnings_response_200_csv(respx_mock, client): + """A CSV answer whose header names the resource's columns is written to + the file as it came.""" respx_mock.get("https://api.marketdata.app/v1/stocks/earnings/AAPL/").respond( - text="AS RECEIVED FROM API", + text="symbol,fiscalYear,fiscalQuarter\nAAPL,2026,1\n", status_code=200, ) output = client.stocks.earnings( symbol="AAPL", output_format=OutputFormat.CSV, filename="test.csv" ) - assert pathlib.Path(output).read_text() == "AS RECEIVED FROM API" + assert ( + pathlib.Path(output).read_text() + == "symbol,fiscalYear,fiscalQuarter\nAAPL,2026,1\n" + ) def test_earnings_internal_reads_timestamp_dates_as_us_eastern_midnight( diff --git a/src/tests/test_stocks_prices.py b/src/tests/test_stocks_prices.py index 787c09b..b92ba64 100644 --- a/src/tests/test_stocks_prices.py +++ b/src/tests/test_stocks_prices.py @@ -7,6 +7,7 @@ import pytest import pytz +import marketdata from marketdata.exceptions import ParseError, ServerError from marketdata.input_types.base import OutputFormat from marketdata.output_types.stocks_prices import StockPrice, StockPricesHumanReadable @@ -49,6 +50,57 @@ def test_stock_prices_human_readable_str(): assert isinstance(str(instance), str) +def test_stock_price_from_dict_keeps_an_undeclared_column_for_get_extra(): + """StockPrice.from_dict builds the price from one row keyed by the API's + names and keeps a column the model does not declare for get_extra.""" + instance = StockPrice.from_dict( + { + "s": "ok", + "symbol": "AAPL", + "mid": 280.02, + "change": -0.68, + "changepct": -0.0024, + "updated": 1765478200, + "brandNew": 1, + } + ) + + assert instance == StockPrice( + s="ok", + symbol="AAPL", + mid=280.02, + change=-0.68, + changepct=-0.0024, + updated=1765478200, + ) + assert marketdata.get_extra(instance) == {"brandNew": 1} + + +def test_stock_prices_human_readable_from_dict_keeps_an_undeclared_column(): + """StockPricesHumanReadable.from_dict builds the price from one row keyed by + the human-readable names and keeps a column the model does not declare for + get_extra.""" + instance = StockPricesHumanReadable.from_dict( + { + "Symbol": "AAPL", + "Mid": 280.02, + "Change $": -0.68, + "Change %": -0.0024, + "Date": 1765478200, + "brandNew": 1, + } + ) + + assert instance == StockPricesHumanReadable( + Symbol="AAPL", + Mid=280.02, + Change_Price=-0.68, + Change_Percent=-0.0024, + Date=1765478200, + ) + assert marketdata.get_extra(instance) == {"brandNew": 1} + + def test_get_stocks_prices_response_200_internal(load_json, respx_mock, client): mock_data = load_json("stocks_prices_response_200") diff --git a/src/tests/test_stocks_quotes.py b/src/tests/test_stocks_quotes.py index b5960c3..cee88d9 100644 --- a/src/tests/test_stocks_quotes.py +++ b/src/tests/test_stocks_quotes.py @@ -7,6 +7,7 @@ import pytest import pytz +import marketdata from marketdata.exceptions import ServerError from marketdata.input_types.base import DateFormat, OutputFormat from marketdata.output_types.stocks_quotes import StockQuote, StockQuotesHumanReadable @@ -181,6 +182,53 @@ def test_stock_quotes_human_readable_from_dict(): ) +def test_stock_quote_from_dict_keeps_an_undeclared_column_for_get_extra(): + """StockQuote.from_dict builds the quote from one row keyed by the API's + names and keeps a column the model does not declare for get_extra.""" + row = { + "symbol": "AAPL", + "ask": 278.02, + "askSize": 100, + "bid": 277.97, + "bidSize": 100, + "mid": 277.995, + "last": 278.0188, + "change": -0.0112, + "changepct": 0.0, + "volume": 4964676, + "updated": 1765478200, + } + + instance = StockQuote.from_dict({**row, "brandNew": 1}) + + assert instance == StockQuote.from_dict(row) + assert marketdata.get_extra(instance) == {"brandNew": 1} + + +def test_stock_quotes_human_readable_from_dict_keeps_an_undeclared_column(): + """StockQuotesHumanReadable.from_dict builds the quote from one row keyed by + the human-readable names and keeps a column the model does not declare for + get_extra.""" + row = { + "Symbol": "AAPL", + "Ask": 278.02, + "Ask Size": 100, + "Bid": 277.97, + "Bid Size": 100, + "Mid": 277.995, + "Last": 278.0188, + "Change $": 0.51, + "Change %": 0.0018, + "Volume": 4964676, + "Date": 1765478200, + } + + instance = StockQuotesHumanReadable.from_dict({**row, "brandNew": 1}) + + assert instance == StockQuotesHumanReadable.from_dict(row) + assert marketdata.get_extra(instance) == {"brandNew": 1} + + def test_get_stocks_quotes_response_200_internal(load_json, respx_mock, client): mock_data = load_json("stocks_quotes_response_200") updated = datetime.datetime.fromtimestamp( diff --git a/src/tests/test_utils.py b/src/tests/test_utils.py index 85c9c1e..32da326 100644 --- a/src/tests/test_utils.py +++ b/src/tests/test_utils.py @@ -163,6 +163,17 @@ def test_merge_csv_responses_accepts_human_readable_names_for_the_model_fields() assert result == "Expiration Date,Strike\r\n1,2\r\n" +def test_merge_csv_responses_keeps_a_column_the_resource_does_not_have(): + responses = [ + _csv_response("t,c,brandNew\n1,2,x\n"), + _csv_response("t,c,brandNew\n3,4,y\n"), + ] + + result = merge_csv_responses(responses, COLUMNS) + + assert result == "t,c,brandNew\r\n1,2,x\r\n3,4,y\r\n" + + @pytest.mark.parametrize( ("bodies", "reason"), [ @@ -321,14 +332,16 @@ def test_merge_csv_responses_reports_an_unreadable_body_before_a_misaligned_row( def test_json_answer_columns_follow_the_order_the_answers_send(): """Under `columns=` the API sends the requested keys only, in request - order (checked live: `columns=v,c` answers `{"v": [...], "c": [...]}`). - The merge keeps that order, which is the order of the empty result and of - every single-request resource, not the model's; the status flag and keys - the model does not know are not columns.""" + order, and the merge keeps that order, not the model's. A key the model + does not declare is a column too when it holds a list; the status flag and + a key that holds one value are not.""" responses = [_csv_response(""), _csv_response("")] - answers = [{"c": [1], "t": [2], "x": [0]}, {"c": [4], "t": [3], "s": "ok"}] + answers = [ + {"c": [1], "t": [2], "x": [0], "note": "one value"}, + {"c": [4], "t": [3], "x": [5], "s": "ok"}, + ] - assert json_answer_columns(responses, answers, COLUMNS) == ["c", "t"] + assert json_answer_columns(responses, answers, COLUMNS) == ["c", "t", "x"] @pytest.mark.parametrize( @@ -347,6 +360,7 @@ def test_json_answer_columns_follow_the_order_the_answers_send(): ([{"t": [1], "c": [2]}, {"t": [3], "c": []}], "different lengths", 1), ([{"t": [1], "c": [2]}, {"t": [3], "c": "4.5"}], "are not lists", 1), ([{"t": [1], "c": [2]}, {"t": [3], "c": None}], "are not lists", 1), + ([{"t": [1], "x": [0]}, {"t": [3]}], "missing columns ['x']", 1), ], ids=[ "null", @@ -358,6 +372,7 @@ def test_json_answer_columns_follow_the_order_the_answers_send(): "an-empty-column", "a-column-that-is-a-string", "a-null-column", + "an-undeclared-column-one-answer-lacks", ], ) def test_json_answer_columns_name_the_answer_that_breaks_the_merge(