[DSIP-79][Task] Add datavines task for data quality - #16863
Open
zixi0825 wants to merge 75 commits into
Open
Conversation
zixi0825
requested review from
EricGao888,
Gallardot,
SbloodyS,
caishunfeng and
songjianet
as code owners
December 1, 2024 02:49
… dev-datavines # Conflicts: # dolphinscheduler-ui/src/views/projects/workflow/components/dag/dag.module.scss
zixi0825
force-pushed
the
dev-datavines
branch
from
January 28, 2025 01:36
9267125 to
8ad0a2a
Compare
zixi0825
force-pushed
the
dev-datavines
branch
from
January 29, 2025 02:37
27c3e67 to
5d4691c
Compare
SbloodyS
requested changes
Apr 15, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new Datavines task type to DolphinScheduler to support triggering Datavines data quality jobs from workflows (UI configuration + worker task plugin + docs/E2E coverage).
Changes:
- UI: register
DATAVINEStask type, add form fields (address/jobId/token/failureBlock), and add DAG icon + i18n strings. - Backend: add
dolphinscheduler-task-datavinesplugin module implementing task channel, parameters, and HTTP calls to Datavines OpenAPI. - Docs/E2E: add Datavines task docs (EN/ZH) and an E2E workflow test with a mock server.
Reviewed changes
Copilot reviewed 33 out of 38 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| dolphinscheduler-ui/src/views/projects/workflow/components/dag/dag.module.scss | Adds Datavines task icon + hover icon styling in the DAG. |
| dolphinscheduler-ui/src/views/projects/task/constants/task-type.ts | Adds DATAVINES to UI task type union and task type metadata map. |
| dolphinscheduler-ui/src/views/projects/task/components/node/types.ts | Extends task parameter typing to include Datavines fields. |
| dolphinscheduler-ui/src/views/projects/task/components/node/tasks/use-datavines.ts | New UI task node model + form composition for Datavines. |
| dolphinscheduler-ui/src/views/projects/task/components/node/tasks/index.ts | Registers DATAVINES to the task form factory map. |
| dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts | Serializes Datavines form fields into taskParams. |
| dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-datavines.ts | New Datavines-specific form fields (address/jobId/token/failureBlock + custom params). |
| dolphinscheduler-ui/src/views/projects/task/components/node/fields/index.ts | Exports the new Datavines field builder. |
| dolphinscheduler-ui/src/store/project/types.ts | Adds DATAVINES to the store-side task type union. |
| dolphinscheduler-ui/src/store/project/task-type.ts | Adds DATAVINES to store task type metadata map. |
| dolphinscheduler-ui/src/locales/zh_CN/project.ts | Adds Chinese i18n strings for Datavines form fields. |
| dolphinscheduler-ui/src/locales/en_US/project.ts | Adds English i18n strings for Datavines form fields. |
| dolphinscheduler-task-plugin/pom.xml | Adds dolphinscheduler-task-datavines module to task plugins build. |
| dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/pom.xml | New Datavines task plugin module (deps + shade config). |
| dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/main/java/.../DatavinesTask.java | Implements the Datavines remote task execution/tracking/cancel logic. |
| dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/main/java/.../DatavinesParameters.java | Defines Datavines task parameters and validation. |
| dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/main/java/.../DatavinesTaskChannel.java | Implements parameter parsing + task creation for the plugin. |
| dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/main/java/.../DatavinesTaskChannelFactory.java | Registers the Datavines task channel via SPI. |
| dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/main/java/.../DatavinesTaskConstants.java | Adds Datavines OpenAPI endpoints/status/constants. |
| dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/main/java/.../DatavinesTaskException.java | Adds a Datavines-specific runtime exception. |
| dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/main/java/.../utils/RequestUtils.java | Adds HTTP helper utilities for calling Datavines OpenAPI. |
| dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/test/java/.../DatavinesTaskTest.java | Adds unit tests for task behavior (with static mocking). |
| dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/test/java/.../DatavinesTaskChannelTest.java | Adds unit tests for channel parsing and task creation. |
| dolphinscheduler-task-plugin/dolphinscheduler-task-datavines/src/test/java/.../DatavinesTaskChannelFactoryTest.java | Adds unit tests for SPI factory name and instantiation. |
| dolphinscheduler-task-plugin/dolphinscheduler-task-all/pom.xml | Includes the Datavines task artifact in the “task-all” bundle. |
| dolphinscheduler-e2e/.../docker/datavines-task/mockserver-config.json | MockServer expectations for Datavines OpenAPI endpoints used in E2E. |
| dolphinscheduler-e2e/.../docker/datavines-task/docker-compose.yaml | E2E compose to run standalone DS + mock Datavines server. |
| dolphinscheduler-e2e/.../pages/project/workflow/task/DatavinesTaskForm.java | Adds Selenium page object for filling Datavines task form. |
| dolphinscheduler-e2e/.../pages/project/workflow/WorkflowForm.java | Adds DATAVINES task type wiring to workflow form helper. |
| dolphinscheduler-e2e/.../cases/tasks/DatavinesTaskE2ETest.java | Adds E2E test that creates/runs a workflow with Datavines task. |
| dolphinscheduler-api/src/main/resources/task-type-config.yaml | Adds DATAVINES under a new dataQuality group (and updates ML list). |
| docs/docs/zh/guide/task/datavines.md | Adds Chinese documentation page for Datavines task. |
| docs/docs/en/guide/task/datavines.md | Adds English documentation page for Datavines task. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Purpose of the pull request
See #16113 for detail.
close #16113
Brief change log
Verify this pull request
This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(or)
Pull Request Notice
Pull Request Notice
If your pull request contain incompatible change, you should also add it to
docs/docs/en/guide/upgrede/incompatible.md