A pandas DataFrame with a null date can raise FloatingPointError at random #155

What happens

A pandas DataFrame whose date column holds a null sometimes raises FloatingPointError: overflow encountered in multiply instead of returning NaT in that row. It happens under the default date format and under DateFormat.SPREADSHEET. The same body reads fine on JSON, INTERNAL, CSV and polars, so the output format decides whether the call raises. The SDK does not catch it, and the bare FloatingPointError reaches the caller.

It shows up in CI as a flaky test, test_money_decimal.py::test_a_null_date_keeps_its_row_on_the_other_decoded_formats[stocks.news-pandas]. It failed on main in runs 36037576926 and 36139219264 and on #154 in run 36150284155, and passes on other runs:

pandas/_libs/tslibs/conversion.pyx:149: in pandas._libs.tslibs.conversion.cast_from_unit_vectorized
.venv/lib/python3.10/site-packages/numpy/_core/fromnumeric.py:3710: in round
E           FloatingPointError: overflow encountered in multiply

Output format: OutputFormat.DATAFRAME, pandas
SDK module: output_handlers/pandas.py, _convert_timestamp_columns and _from_serial

Why

This is pandas-dev/pandas#58419. In pandas 2.3.3, pd.to_datetime(..., unit="s") on a float column goes through cast_from_unit_vectorized, which:

1. allocates its buffer of fractional parts with np.empty and never writes the slots of the null values;
2. rounds the whole buffer with np.round(frac, 9) under np.errstate(over="raise"), outside the try that turns an overflow into OutOfBoundsDatetime.

When the memory left in a null slot holds a float above about 1.8e299, the rounding overflows. Whether it does depends on what that memory held before, so the failure is random. The CI test fails on stocks.news because its body sets updated to null, which fills all 1840 rows of that column with nulls. All three failures go through pandas/core/tools/datetimes.py:1072, the path that converts that whole column.

pandas 3.0.0 fixed it in pandas-dev/pandas#61022 (np.zeros). 2.3.3, the last 2.x release, still has it. pandas 3 needs Python 3.11, so on Python 3.10 the SDK always runs an affected pandas. On 3.11 and later, so does every environment that already has pandas 2.3.x, because it satisfies pandas>=2.3.3 and pip keeps it.

Reproduction

This makes it deterministic by giving pandas dirty memory on purpose:

from unittest import mock

import numpy as np
import pandas as pd

real_empty = np.empty


def dirty_empty(shape, dtype=float, *args, **kwargs):
    array = real_empty(shape, dtype, *args, **kwargs)
    if array.dtype == np.float64:
        array.fill(1e300)
    return array


with mock.patch.object(np, "empty", dirty_empty):
    pd.to_datetime(pd.Series([np.nan, 1.7e9]), unit="s", utc=True)

On pandas 2.3.3 this raises FloatingPointError: overflow encountered in multiply. On pandas 3.0.6 it returns NaT and the date. Under the same patch the CI test above fails every time with the same traceback, and so does _from_serial with a null serial.

Definition of done

• A null in a date column reads as NaT on a pandas DataFrame whatever the memory pandas works in holds, under the default date format and under DateFormat.SPREADSHEET.
• Every other value, the dtype, and the exceptions the conversion raises stay what they are today, on pandas 2.3.3 and on pandas 3.
• A test that drives the public client fails on main every time, not at random.