Repository navigation
Add Python Virtual Environment Support #4296
Description
Feature Summary
Currently, workflows are executed using the system Python environment. This limits users who need custom packages to run their workflows. In addition, some workflows may contain multiple UDF operators that require different dependencies.
The Python Virtual Environment feature addresses this limitation by allowing users to create multiple environments with custom installed packages. Users can then select the desired environment for each UDF operator through the operator’s property panel, enabling different operators within the same workflow to run in different environments.
Proposed Solution or Design
In the first iteration of our design, environments will be stored on the local volume of each computing unit. The Python UDF operator will include a dropdown menu that allows users to select from the available environments. During execution, when a UDF operator is encountered, the system will spawn a Python process using the Python executable from the selected virtual environment.
Impact / Priority
(P1)High – significantly improves user experience
Affected Area
Workflow Engine (Amber)
Activity
Yicong-Huang commentedon Mar 16, 2026
that's a well needed feature! It will be good to include some diagram here.
chenlica commentedon Mar 17, 2026
@SarahAsad23 Please write down the sub-tasks as a list for easy understanding.
SarahAsad23 commentedon Mar 18, 2026
After the first iteration—where users can create multiple environments and select one per UDF—we received the following feedback:
Yicong-Huang commentedon Mar 18, 2026
@SarahAsad23 please go ahead create an issue for each step. I can help you link them under this issue as sub-issues.
meanwhile, @Ma77Ball is working on an automation to help contributors create sub-issues by themselves.
chenlica commentedon Mar 18, 2026
@SarahAsad23 Please do as suggested.
SarahAsad23 commentedon Mar 18, 2026
@chenlica @Yicong-Huang I have created the sub-issues. Please help me link them.
Yicong-Huang commentedon Mar 18, 2026
done
1 remaining item
feat: Add Python Virtual Environment Support: Creating a Virtual Envi…
feat: Add Python Virtual Environment Support: Creating a Virtual Envi…
feat: Add Python Virtual Environment Support: Installing User Defined…
feat: Add Python Virtual Environment Support: Uninstalling User Defin…
feat: Add Python Virtual Environment Support: Execute Python UDFs Ins…
feat: Add Python Virtual Environment Support: Add k8s Gateway Configu…
feat: Add Python Virtual Environment Support: Creating a Virtual Envi…
feat: Add Python Virtual Environment Support: Installing User Defined…
feat: Add Python Virtual Environment Support: Uninstalling User Defin…
feat: Add Python Virtual Environment Support: Execute Python UDFs Ins…
feat: Add Python Virtual Environment Support: Add k8s Gateway Configu…