Running an operator inside a container

[chenlica]

In one of the recent use cases of the system, a collaborator (@hhvu0102) expressed the need of running an operator (of workflow) inside a container, using a docker image possibly hosted on Docker Hub. I would like start a discussion about this feature.

@hhvu0102: If possible, please provide more details of the use case. Thanks.

[sshiv012]

Adding details on the use case and what we built for it.

The use case: isolation Ha's pipeline is three Python steps that each pin their own dependencies. Running them in one computing unit means one Python environment, and a single environment can hold only one version of a shared library. The steps may pin different numpy versions (1.26.4 / 2.1.0 / 2.0.2), one of them needs pysam, another scikit-image, and in practice any two bio tools disagree on something. The failure mode is not performance; it is "this workflow cannot be installed." Nextflow solves this by giving each process its own container image, which is why the pipeline existed as a Nextflow workflow first. The ask was: let a Texera operator do the same.

How it looks from the UI. On top of Chris's (@kunwp1) rented-compute prototype we made the image a per-operator property. In the operator's property panel there is an Offload toggle; turning it on reveals Instance type (a catalog entry such as local-2g, which sets the container's memory cap) and Image (any Docker tag). Leave Image blank and the operator runs on the platform's default worker image; set it to myorg/qc-step:1.0 and that image is used for that operator only. When the workflow runs, the master issues docker run <image> per offloaded operator, which pulls the tag from Docker Hub (or any registry the daemon is logged into) if it is not already local, exactly like Nextflow's container = 'myorg/qc-step:1.0' directive. The one constraint today: the image must be built FROM texera-computing-unit-worker, because the container has to start a Texera worker and join the cluster. The step's own libraries go on top of that base, so each operator gets its own environment.

flowchart LR
  subgraph UI["Workflow editor"]
    P["Operator property panel<br/>Offload: on<br/>Instance type: local-2g<br/>Image: myorg/qc-step:1.0"]
  end
  subgraph CU["Computing-unit master (JVM, Pekko cluster seed :2552)"]
    O["RuntimeOffloadOrchestrator<br/>rent one container per offloaded operator<br/>pin the operator's workers to it"]
  end
  subgraph D["Docker daemon"]
    C1["container: qc<br/>image qc:1.0 (numpy 1.26)<br/>Texera worker :2560"]
    C2["container: plot_qc<br/>image plot-qc:1.0 (numpy 2.1)<br/>Texera worker :2561"]
    C3["container: interactive<br/>image interactive:1.0 (numpy 2.0)<br/>Texera worker :2562"]
  end
  P -->|run| O
  O -->|docker run --memory=2g image| C1 & C2 & C3
  C1 & C2 & C3 <-->|"Pekko Artery TCP (Kryo tuples + control)"| CU

Loading

Same design as Nextflow, different transport. In both systems the unit of isolation is "one container per step, each with its own image." The difference is what crosses the container boundary. Nextflow stages input files into the task's work directory and runs a shell command; the container is a process sandbox and data moves as files. In Texera the container runs a full worker that joins the master's Pekko cluster over Artery TCP, and the operator's input and output tuples are streamed to it the same way they are streamed between in-process workers. Nothing in the operator changes; placement is the only difference, and the dataflow engine keeps working across the boundary (backpressure, pause/resume, stats).

[sshiv012]

Python virtual environments were also considered, but Ha might be using R workflows - hence it is not a complete solution.