Mixing of PrepID Tasks

[dabercro]

There seem to be workflows that mix tasks together. For example, in this screen shot, both PrepIDs are including tasks that look like they should belong to the other PrepID, based on the numbers "00413" and "00561". This results in duplicate tasks.

[vargasa]

I checked and It doesn't seem to be a problem on the front-end, rather, this is coming directly from the database:

This is coming from here. As of now the same workflow and task names may belong to different preps, that's why is being displayed twice

[phylsix]

This is what I see from wmstats API
https://cmsweb.cern.ch/wmstatsserver/data/request/pdmvserv_task_PPD-PhaseIIMTDTDRAutumn18GS-00015__v1_T_190126_074935_1059

Some tasks' prepID seems to be end with 00009, but this workflow's prepID seems to be consistent to me, ending with 00015

[dabercro]

Okay. This matches what Andres sees, so it's likely that the tasks don't have to match the workflow name.

Also, from the wmstats API, it looks like different tasks are under different PrepIDs. @sharad1126 @vlimant Does that make sense that different tasks under the same workflow have different PrepIDs? We will have to change the presentation hierarchy if that's the case.

[vlimant]

yes, multiple prepids can be present by workflow, the two first column of your screenshot should be "inverted"

[dabercro]

How do we group ACDCs together then?

[dabercro]

@vargasa It looks like each workflow is still only associated with one PrepID. We can keep ACDCs together this way. You can usually tell this based on the name, or you can add the same check that the old console does through request manager: https://github.com/dabercro/WorkflowWebTools/blob/master/workflowwebtools/workflowinfo.py#L454
https://github.com/dabercro/WorkflowWebTools/blob/master/workflowwebtools/workflowinfo.py#L256

[vargasa]

Hi @dabercro. It is not the case, for instance the workflow pdmvserv_task_PPD-PhaseIIMTDTDRAutumn18GS-00015__v1_T_190126_074935_1059 is associated with two prep ids PPD-PhaseIIMTDTDRAutumn18DR-00009 and with PPD-PhaseIIMTDTDRAutumn18GS-00015 :

                        {
                            "prep": {
                                "name": "PPD-PhaseIIMTDTDRAutumn18DR-00009",
                                "campaign": "PhaseIIMTDTDRAutumn18GS",
                                "priority": 900000
                            },
                            "statuses": [
                                {
                                    "site": "T1_IT_CNAF",
                                    "dataset": "/DYToLL_M-50_14TeV_pythia8/PhaseIIMTDTDRAutumn18GS-pilot_103X_upgrade2023_realistic_v2_ext4-v1/GEN-SIM",
                                    "success_count": 2,
                                    "failed_count": 0
                                },
                                {
                                    "site": "T2_US_Purdue",
                                    "dataset": "/DYToLL_M-50_14TeV_pythia8/PhaseIIMTDTDRAutumn18GS-pilot_103X_upgrade2023_realistic_v2_ext4-v1/GEN-SIM",
                                    "success_count": 1,
                                    "failed_count": 0
                                }
                            ],
                            "task_action": null,
                            "name": "/pdmvserv_task_PPD-PhaseIIMTDTDRAutumn18GS-00015__v1_T_190126_074935_1059/PPD-PhaseIIMTDTDRAutumn18GS-00015_0/PPD-PhaseIIMTDTDRAutumn18GS-00015_0MergeRAWSIMoutput",
                            "short_name": "PPD-PhaseIIMTDTDRAutumn18GS-00015_0MergeRAWSIMoutput",
                            "workflow": "pdmvserv_task_PPD-PhaseIIMTDTDRAutumn18GS-00015__v1_T_190126_074935_1059",
                            "parent_workflow": "",
                            "job_type": "Merge",
                            "failures_count": 0
                        }
                    ],
                    "name": "pdmvserv_task_PPD-PhaseIIMTDTDRAutumn18GS-00015__v1_T_190126_074935_1059",
                    "parent_workflow": ""
                }
            ],
            "name": "PPD-PhaseIIMTDTDRAutumn18DR-00009",
            "campaign": "PhaseIIMTDTDRAutumn18GS",
            "priority": 900000,
            "updated": "2019-01-29T00:31:35.921000"
        },
        {
            "workflows": [
                {
                    "tasks": [
                        {
                            "prep": {
                                "name": "PPD-PhaseIIMTDTDRAutumn18GS-00015",
                                "campaign": "PhaseIIMTDTDRAutumn18GS",
                                "priority": 900000
                            },
                            "statuses": [
                                {
                                    "site": "T1_IT_CNAF",
                                    "dataset": "/DYToLL_M-50_14TeV_pythia8/PhaseIIMTDTDRAutumn18MiniAOD-PU200_pilot_103X_upgrade2023_realistic_v2_ext4-v1/MINIAODSIM",
                                    "success_count": 1,
                                    "failed_count": 0
                                },
                                {
                                    "site": "T2_US_Purdue",
                                    "dataset": "/DYToLL_M-50_14TeV_pythia8/PhaseIIMTDTDRAutumn18MiniAOD-PU200_pilot_103X_upgrade2023_realistic_v2_ext4-v1/MINIAODSIM",
                                    "success_count": 1,
                                    "failed_count": 0
                                }
                            ],
                            "task_action": null,
                            "name": "/pdmvserv_task_PPD-PhaseIIMTDTDRAutumn18GS-00015__v1_T_190126_074935_1059/PPD-PhaseIIMTDTDRAutumn18GS-00015_0/PPD-PhaseIIMTDTDRAutumn18GS-00015_0MergeRAWSIMoutput/PPD-PhaseIIMTDTDRAutumn18DR-00009_0/PPD-PhaseIIMTDTDRAutumn18DR-00009_1/PPD-PhaseIIMTDTDRAutumn18DR-00009_1MergeFEVTDEBUGHLToutput/PPD-PhaseIIMTDTDRAutumn18MiniAOD-00002_0/PPD-PhaseIIMTDTDRAutumn18MiniAOD-00002_0MergeMINIAODSIMoutput",
                            "short_name": "PPD-PhaseIIMTDTDRAutumn18MiniAOD-00002_0MergeMINIAODSIMoutput",
                            "workflow": "pdmvserv_task_PPD-PhaseIIMTDTDRAutumn18GS-00015__v1_T_190126_074935_1059",
                            "parent_workflow": "",
                            "job_type": "Merge",
                            "failures_count": 0
                        },

[dabercro]

Yeah, the tasks still might be associated with multiple PrepIDs, but the workflow itself is only associated with one. In this case, it would be PPD-PhaseIIMTDTDRAutumn18GS-00015. See the top of https://cmsweb.cern.ch/reqmgr2/data/request?name=pdmvserv_task_PPD-PhaseIIMTDTDRAutumn18GS-00015__v1_T_190126_074935_1059. The first occurrence of the PrepID corresponds to the workflow. The other instances of the PrepID keys are for the tasks.

We can tell which PrepID to pick from the workflow name too.

[dabercro]

@hbakhshi So doing this with the docker-compose up images, we can quickly reproduce this problem by using "Filter by": pdmvserv_task_PPD-PhaseIIMTDTDRAutumn18GS-0001. Here, we can see the tasks pdmvserv_task_PPD-PhaseIIMTDTDRAutumn18GS-00010__v1_T_190130_112047_7218 and pdmvserv_task_PPD-PhaseIIMTDTDRAutumn18GS-00011__v1_T_190130_112044_5709 show up three times.

[vargasa]

It starts rendering the Prep:

wtc-console/src/static/containers/Tasks/index.js

Line 618 in de1d11f

<PrepWorkflowsTreeTable

Then it renders each workflow for the prep:

wtc-console/src/static/components/PrepWorkflowsTreeTable.js

Line 229 in de1d11f

renderWorkflow = (workflow, level, fill) => {

And finally the list of tasks:

wtc-console/src/static/components/PrepWorkflowsTreeTable.js

Line 209 in de1d11f

if ( task.failures_count > 0. || !!showAllTasks) {

@dabercro I remember you mentioned some tasks that were being checked simultaneously as a way to identify them and prevent rerendering. Do you have a filter showing this on your notes?

[dabercro]

Sorry for the late reply. Yeah, I see this happen for the duplicates when I use the filter pdmvserv_task_PPD-PhaseIIMTDTDRAutumn18GS-0001 mentioned above. Hamed found out that this only happens in Safari though, not Firefox or Chrome.