You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
A workflow cannot be taken out of Texera as a script that runs on its own #8325
A workflow can be built and read in the editor, but there is no form of it that runs anywhere else. A user who wants to keep a pipeline after the fact, hand it to a colleague who does not run Texera, or step through it in a notebook has nothing to take away.
What is missing is an export: given a plan, produce a single Python file that reads the same sources, applies the same operators in the same order, and prints its results. The operators already describe their work as Python for the engine, so the pieces exist; what is absent is a form of that description that stands on its own, without the runtime around it, and something to stitch the pieces into one script.
Proposed Solution or Design
An operator says how it reads outside the engine by implementing a StandaloneCodeGenerator trait, returning a block of pandas that names its inputs and outputs by position: in1df, in2df, out1df. A translator walks the plan in topological order, gives every output port a variable, substitutes those placeholders for the variables its upstreams were given, and prints the leaves. A variadic port, of which Union is the one example, needs the whole list of upstreams rather than a fixed count, since any count the operator states would be wrong for some workflow.
The compiling service exposes the result as an endpoint, so the editor can offer the script for a plan the user has open.
An operator that has no generator yet leaves a commented placeholder in the script rather than a line that looks like it works, so the export is useful before every operator implements it.
Each is sized to be read in one sitting. Eight entries were split for growing past that: the export itself into the trait and the operators that use it, the text operators from the relational ones, the coordinate-system charts from the domain ones, the table comparison from the chart comparison, the editor's export button from the property panel, deriving a configuration into the entry that produces one and the entry that sweeps it into variants, the relational operators into joins and sets, Aggregate with the sort family, and the samplers with control flow, and the sources into the CSV family, JSON Lines with Arrow, and the line sources. The export comes first, then the machinery that runs an operator both ways, then the operator families that implement the trait, then the two changes that read every operator at once — those come after the families so their assertions hold as written — and the editor last. This issue stays open until the guide lands.
Related, outside the set
Three new operators read the trait but make no existing operator exportable, so they are not among the 27: #8480 (the whole-number parts of a timestamp), #8481 (bins over a numeric column) and #8512 (a Parquet source). They wait for the set to merge. Two fixes were found along the way. #8514 bounds a file scan's window at zero (#8513) and stands on its own. #8597 reads text into a timestamp and writes a value as text the way the engine does (#8595). Entries 11b, 12a, 12b and 12c call the helpers it adds, so it merges before them. #8766 gives a port that carried no rows its declared columns (#8735). Entry 4 hands that schema to the Python operator it runs, and entry 23 then stops withholding the operators that failed on an empty table, so it merges before both.
@carloea2@mengw15 — this is now open as a set of twenty changes rather than one. The list above gives the order; each is sized to be read in one sitting, most under a thousand lines.
The place to start is #8327, the export itself: an operator says how it reads outside the engine, a translator stitches the blocks into a script, and an endpoint serves it. Nothing else can land before it. After it come the pieces that run an operator both ways and compare the answers, then the operator families that implement the trait, then the two changes that read every operator at once — those sit after the families so their assertions hold as written — and the editor last.
They are not independent: each is cut from main and does not compile until the ones before it land, so the diffs read best in order.
One thing worth saying about the risk: a partial merge leaves nothing broken.
The export is usable as soon as #8327 lands. Each operator family after it is independent of the others, so every one that merges adds the operators it carries, and the ones left out simply stay as they are today. Nothing has to be taken whole, and nothing half-landed leaves the tree in a worse state than it is now.
The verification is what needs the whole set: it reads every operator at once, which is why the two changes that do that sit near the end.
Feature Summary
A workflow can be built and read in the editor, but there is no form of it that runs anywhere else. A user who wants to keep a pipeline after the fact, hand it to a colleague who does not run Texera, or step through it in a notebook has nothing to take away.
What is missing is an export: given a plan, produce a single Python file that reads the same sources, applies the same operators in the same order, and prints its results. The operators already describe their work as Python for the engine, so the pieces exist; what is absent is a form of that description that stands on its own, without the runtime around it, and something to stitch the pieces into one script.
Proposed Solution or Design
An operator says how it reads outside the engine by implementing a
StandaloneCodeGeneratortrait, returning a block of pandas that names its inputs and outputs by position:in1df,in2df,out1df. A translator walks the plan in topological order, gives every output port a variable, substitutes those placeholders for the variables its upstreams were given, and prints the leaves. A variadic port, of which Union is the one example, needs the whole list of upstreams rather than a fixed count, since any count the operator states would be wrong for some workflow.The compiling service exposes the result as an endpoint, so the editor can offer the script for a plan the user has open.
An operator that has no generator yet leaves a commented placeholder in the script rather than a line that looks like it works, so the export is useful before every operator implements it.
The set, in order
Each is sized to be read in one sitting. Eight entries were split for growing past that: the export itself into the trait and the operators that use it, the text operators from the relational ones, the coordinate-system charts from the domain ones, the table comparison from the chart comparison, the editor's export button from the property panel, deriving a configuration into the entry that produces one and the entry that sweeps it into variants, the relational operators into joins and sets, Aggregate with the sort family, and the samplers with control flow, and the sources into the CSV family, JSON Lines with Arrow, and the line sources. The export comes first, then the machinery that runs an operator both ways, then the operator families that implement the trait, then the two changes that read every operator at once — those come after the families so their assertions hold as written — and the editor last. This issue stays open until the guide lands.
Related, outside the set
Three new operators read the trait but make no existing operator exportable, so they are not among the 27: #8480 (the whole-number parts of a timestamp), #8481 (bins over a numeric column) and #8512 (a Parquet source). They wait for the set to merge. Two fixes were found along the way. #8514 bounds a file scan's window at zero (#8513) and stands on its own. #8597 reads text into a timestamp and writes a value as text the way the engine does (#8595). Entries 11b, 12a, 12b and 12c call the helpers it adds, so it merges before them. #8766 gives a port that carried no rows its declared columns (#8735). Entry 4 hands that schema to the Python operator it runs, and entry 23 then stops withholding the operators that failed on an empty table, so it merges before both.
Affected Area
Workflow Engine (Amber), Workflow UI