Task Summary
The JSON Lines and Arrow sources implement StandaloneCodeGenerator. Arrow is read into pandas' nullable dtypes, since the file states of every value whether it is there. JSONL states nothing, so each column is typed the way the engine types it, and a timestamp is read the way the engine reads one.
Matching Arrow also changes the engine. A TIMESTAMP reaches a Python worker as a zoneless microsecond Arrow timestamp instead of a millisecond one labelled UTC. That is what the Python side already declares, and what a script holds.
Step 12b in #8325. It needs #8341 for the shared source code and #8597 for the timestamp helpers.
Task Summary
The JSON Lines and Arrow sources implement
StandaloneCodeGenerator. Arrow is read into pandas' nullable dtypes, since the file states of every value whether it is there. JSONL states nothing, so each column is typed the way the engine types it, and a timestamp is read the way the engine reads one.Matching Arrow also changes the engine. A TIMESTAMP reaches a Python worker as a zoneless microsecond Arrow timestamp instead of a millisecond one labelled UTC. That is what the Python side already declares, and what a script holds.
Step 12b in #8325. It needs #8341 for the shared source code and #8597 for the timestamp helpers.