feat(operator): read a timestamp cast over the years the engine reaches

[kz930]

What changes were proposed in this PR?

Type Casting to a timestamp read text with pd.to_datetime(col, errors="coerce") in the exported script. That is wrong in two ways, and both empty a cell the run itself filled.

pandas parses into nanoseconds, which reach only 1677-09-21 to 2262-04-11. The engine holds a java.sql.Timestamp, where 2500-01-01 00:00:00 is an ordinary moment: it reads it, and the exported script answered with an empty cell. The same for 1500-06-15 08:30:00 and 9999-12-31 23:59:59.

pandas also infers one format for the whole column and coerces every row that does not match it. The engine hands DateParserUtils one field at a time, so a row states its own format. A column holding 2024-03-05 14:09:07 and March 5, 2024 kept the first and emptied the second.

The text branch now reads the column cell by cell and holds the result at microsecond resolution, which covers the years the engine covers. It still coerces: the engine accepts a set of formats no single pandas call states, so text neither side can read leaves an empty cell rather than ending an exported run halfway. That part is unchanged, and a test now says so.

Review and the verification's edge values found four more places where the cast parted from the engine:

• A reading that states an offset, such as Z or +05:30, stopped the cast. The engine moves it to the machine's zone and keeps that wall clock, and the script now does the same.
• A column that already held timestamps went through the text parse and lost its nanoseconds. parseField returns a java.sql.Timestamp untouched, so the script now hands such a column back at the resolution it arrived in.
• Text with digits past the millisecond kept them. DateParserUtils reads into a java.util.Date, which counts milliseconds, so the script now cuts there too.
• Casting to text wrote a value the way Python's str() does. Java writes a double past 1e7 or under 1e-3 in E notation, a boolean in lower case, and a timestamp with its nanoseconds and at least .0. A shared _texera_java_text now writes each one as Java's toString does.

The number branch is left alone. A LONG is read as epoch milliseconds and has the same window, but it also disagrees about the calendar before 1582-10-15, where new Timestamp(long) is Julian and pandas is proleptic Gregorian. Widening the window there would answer with a moment eight days off instead of an empty cell, so what that branch should do is a separate decision. The issue records it.

Any related issues, documentation, discussions?

Closes #8595. #8480 closed the same nanosecond gap for Extract Date/Time Fields, which parses strictly and could take the one-line form this operator cannot.

#8341, #8704, #8705 and #8709 call the timestamp and text helpers this adds, so this PR merges before them.

How was this PR tested?

Four tests in TypeCastingOpDescSpec run the generated Python and compare it cell by cell against AttributeTypeUtils.parseField. They are tagged to run in the integration job, which has pandas.

• The text-to-timestamp test reads moments in three formats, the rows either side of the nanosecond edge, 2500-01-01, 1500-06-15, 9999-12-31, readings with Z and +05:30, and readings past the millisecond. It fails on the old generator.
• One test gives a column that already holds timestamps and checks its nanoseconds are kept.
• The cast-to-string test covers a double at 1e7 and 1e-4.
• One test checks that unreadable text still leaves an empty cell, where the engine refuses it.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• No candidates found from git blame history.

[github-actions]

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 7 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main 5264df2 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	374	0.228	25,856/38,936/38,936 us	🔴 +14.3% / 🔴 +128.4%
🔴	bs=100 sw=10 sl=64	780	0.476	124,965/186,256/186,256 us	🔴 +19.8% / 🔴 +61.8%
⚪	bs=1000 sw=10 sl=64	906	0.553	1,100,566/1,191,617/1,191,617 us	⚪ within ±5% / 🔴 +7.6%

Baseline details

Latest main 5264df2 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	374 tuples/sec	405 tuples/sec	721.84 tuples/sec	-7.7%	-48.2%
bs=10 sw=10 sl=64	MB/s	0.228 MB/s	0.247 MB/s	0.441 MB/s	-7.7%	-48.2%
bs=10 sw=10 sl=64	p50	25,856 us	23,839 us	13,416 us	+8.5%	+92.7%
bs=10 sw=10 sl=64	p95	38,936 us	34,072 us	17,046 us	+14.3%	+128.4%
bs=10 sw=10 sl=64	p99	38,936 us	34,072 us	20,349 us	+14.3%	+91.3%
bs=100 sw=10 sl=64	throughput	780 tuples/sec	797 tuples/sec	914.39 tuples/sec	-2.1%	-14.7%
bs=100 sw=10 sl=64	MB/s	0.476 MB/s	0.486 MB/s	0.558 MB/s	-2.1%	-14.7%
bs=100 sw=10 sl=64	p50	124,965 us	124,577 us	108,507 us	+0.3%	+15.2%
bs=100 sw=10 sl=64	p95	186,256 us	155,430 us	115,131 us	+19.8%	+61.8%
bs=100 sw=10 sl=64	p99	186,256 us	155,430 us	127,181 us	+19.8%	+46.4%
bs=1000 sw=10 sl=64	throughput	906 tuples/sec	917 tuples/sec	937.6 tuples/sec	-1.2%	-3.4%
bs=1000 sw=10 sl=64	MB/s	0.553 MB/s	0.56 MB/s	0.572 MB/s	-1.3%	-3.4%
bs=1000 sw=10 sl=64	p50	1,100,566 us	1,089,207 us	1,065,461 us	+1.0%	+3.3%
bs=1000 sw=10 sl=64	p95	1,191,617 us	1,147,507 us	1,107,736 us	+3.8%	+7.6%
bs=1000 sw=10 sl=64	p99	1,191,617 us	1,147,507 us	1,133,556 us	+3.8%	+5.1%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,535.32,200,128000,374,0.228,25855.61,38936.13,38936.13
1,100,10,64,20,2565.65,2000,1280000,780,0.476,124964.59,186255.81,186255.81
2,1000,10,64,20,22070.32,20000,12800000,906,0.553,1100565.86,1191616.72,1191616.72

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.79%. Comparing base (2ab8ee0) to head (189bf79).
⚠️ Report is 49 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##               main    #8597    +/-   ##
==========================================
  Coverage     92.78%   92.79%            
- Complexity     4898     4923    +25     
==========================================
  Files          1236     1238     +2     
  Lines         52121    52305   +184     
  Branches       6405     6434    +29     
==========================================
+ Hits          48359    48534   +175     
+ Misses         2186     2178     -8     
- Partials       1576     1593    +17     

Flag	Coverage Δ		*Carryforward flag
access-control-service	77.38% <ø> (+5.59%)	⬆️
agent-service	99.32% <ø> (ø)		Carriedforward from ca67ce1
amber	88.59% <ø> (-0.02%)	⬇️	Carriedforward from ca67ce1
computing-unit-managing-service	60.41% <ø> (+5.21%)	⬆️
config-service	87.37% <ø> (ø)
file-service	81.53% <ø> (ø)		Carriedforward from ca67ce1
frontend	96.58% <ø> (ø)		Carriedforward from ca67ce1
notebook-migration-service	83.73% <ø> (ø)
pyamber	98.48% <ø> (ø)		Carriedforward from ca67ce1
workflow-compiling-service	74.09% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[carloea2]

The new timestamp helper crashes on 2024-03-05T14:09:07Z. I ran the generated helper with pandas 2.2.3. Parsing keeps the timezone, then astype rejects the conversion to a timestamp without a timezone. The previous code accepts this value. Please handle timestamps with a timezone and add a test with Z and an explicit offset.

[kz930]

Fixed in c07493387. The helper now does what the engine does with an offset — DateParserUtils reads it and java.sql.Timestamp keeps no zone, so the moment is held as the wall clock of the machine's own zone (...T14:09:07Z is 06:09:07 eight hours behind UTC) — and both spellings joined the cases the test compares against parseField cell by cell, so the expectation comes from the engine rather than being written down.

[carloea2]

The timezone crash is fixed. I ran the current helper with Z, an explicit offset, year 2500, and null. Those checks passed.

[carloea2]

Reproduced locally with the generated Python code. One correctness issue below. I did not run a frontend workflow or the full Scala suite.

[carloea2]

Retested ca67ce1. Timezone inputs, extended years and nanosecond precision pass. The previous findings are fixed. Looks good.

[kz930]

@carloea2 Since your approval, text read into a timestamp is cut to the millisecond as DateParserUtils does, and a cast to text writes a value as Java's toString does.

[carloea2]

Reproduced with the generated helper and JDK 17: 1e23 becomes 1.0E23 here, but Double.toString returns 9.999999999999999E22. String comparisons and joins after a cast can then differ. Please match Java's digit selection and add a regression test.

[kz930]

Confirmed on JDK 17. Matching it exactly means reproducing the digit selection in OpenJDK's FloatingDecimal, which is GPL code, so is it OK to write a Python version of it here, or should the helper keep the shortest form that JDK 19+ gives?