Repository navigation
[ASIM] Add Proofpoint POD MailLog ingestion-time parser - #15463
adbhogeGit wants to merge 7 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
| TargetASIMTable: 'ASimEmailEventLogs' | ||
| ParserFilterQuery: | | ||
| source | ||
| | where isnotempty(tostring(sm.qid)) and (bag_has_key(sm, "from") or array_length(sm["to"]) > 0) |
There was a problem hiding this comment.
The fixture of ProofpointPODMailLog_CL should contain rows that return false with this condition.
Here is the ITP validation workflow result: https://github.com/Azure/Azure-Sentinel/actions/runs/40631618722?pr=15463
There was a problem hiding this comment.
Added a rejected fixture row with a nonempty sm.qid, no sm.from, and sm.to: [], plus a positive case with an empty sender. The canary on 4802006582 confirms filtering passes: 25 input rows, 24 selected, 1 rejected.
I also replaced unsupported bag_has_key with isnotnull(sm["from"]) in both queries and made delivery classification boolean to avoid a DCR numeric type mismatch. DCR compilation/provisioning succeeded and all 25 fixture rows were accepted for upload, but target visibility reached the 15-minute deadline before data validation/parity. That deadline result did not retain observed target counts, so it does not establish whether there were no rows or an incomplete expected set.
The failed-jobs rerun (attempt 2) stopped before runtime validation because asim-ingestion-template-asimemaileventproofpointpodmaillog-40786158180-2 was not found; the static job's artifact exists only with the attempt-1 suffix. Keeping this thread open until the runtime gate passes.
There was a problem hiding this comment.
Retried the entire workflow (attempt 3), not just failed jobs, so the matching template artifact was regenerated and the attempt-2 artifact error is no longer the blocker: https://github.com/Azure/Azure-Sentinel/actions/runs/40786158180/attempts/3 . The filter again passed with 25 raw rows, 24 selected, and 1 rejected. The DCR compiled and accepted all 25 input rows, but target-visibility polling again exhausted the validation deadline. The artifact does not retain observed target counts, so it cannot establish whether no rows arrived or only some are missing. Runtime data/parity validation was not reached. Read-only diagnosis from this session is blocked because its Azure CLI account cannot access the shared canary subscription. Keeping this thread open pending canary diagnostics; no further blind retries or validation bypasses.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Required items, please complete
Change(s):
Reason for Change(s):
Version Updated:
Testing Completed:
python -m unittest discover -s ".script\tests\asimParsersTest\ingestTimeParser\tests" -p "test_asim_*.py").python .script\tests\asimParsersTest\ingestTimeParser\asim_ingest_time_validation.py validate Parsers\ASimEmailEvent\IngestTimeParsers\ASimEmailEventProofpointPODMailLog.yaml --repo-root . --query-time-path Parsers\ASimEmailEvent\Parsers\ASimEmailEventProofpointPODMailLog.yaml).python .script\tests\asimParsersTest\ingestTimeParser\asim_fixture_preparation.py prepare "Sample Data\ASIM\Proofpoint_POD MailLog_EmailEvent_ProofpointPODMailLog_CL_IngestedLogs.csv" --expected-source-raw-table ProofpointPODMailLog_CL --target-id ASimEmailEventProofpointPODMailLog --target-group emailevent --run-id local-maillog-draft --schema-path "Sample Data\ASIM\ProofpointPODMailLog_CL_Schema.csv").pythonalias was unavailable; JSON outputs were saved outside the repository.Checked that the validations are passing and have addressed any issues that are present: