Skip to content

[ASIM] Add Proofpoint POD MailLog ingestion-time parser - #15463

Open
adbhogeGit wants to merge 7 commits into
masterfrom
adbhoge-microsoft-proofpoint-maillog-draft-pr
Open

adbhogeGit wants to merge 7 commits into
masterfrom
adbhoge-microsoft-proofpoint-maillog-draft-pr

Conversation

@adbhogeGit

@adbhogeGit adbhogeGit commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Required items, please complete

Change(s):

  • Add the Proofpoint POD MailLog ingestion-time EmailEvent parser targeting ASimEmailEventLogs.
  • Add its source-table-specific CSV fixture and filename-matched changelog, preserving existing query-time history.
  • Keep envelope recipients in AdditionalFields, retain authored parse_ipv4/parse_ipv6 calls, and remove obsolete conversion-blocker prose now that that concern is resolved.
  • This independent Draft PR contains only the MailLog parser, fixture, and changelog; no Message parser, converter KQL, or conversion reports.

Reason for Change(s):

  • Support ingestion-time normalization of Proofpoint POD MailLog SMTP receipt and delivery records using the existing query-time parser's mappings.

Version Updated:

  • N/A for detection templates. This new ingestion-time parser starts at version 0.1.2, matching its source query-time parser.

Testing Completed:

  • Passed the documented ingestion-time unit suite: 147 tests (python -m unittest discover -s ".script\tests\asimParsersTest\ingestTimeParser\tests" -p "test_asim_*.py").
  • Passed repository static ingestion-time parser validation and source-table-specific fixture discovery; the explicitly paired query-time parser is eligible for parity (python .script\tests\asimParsersTest\ingestTimeParser\asim_ingest_time_validation.py validate Parsers\ASimEmailEvent\IngestTimeParsers\ASimEmailEventProofpointPODMailLog.yaml --repo-root . --query-time-path Parsers\ASimEmailEvent\Parsers\ASimEmailEventProofpointPODMailLog.yaml).
  • Passed local typed fixture preparation (python .script\tests\asimParsersTest\ingestTimeParser\asim_fixture_preparation.py prepare "Sample Data\ASIM\Proofpoint_POD MailLog_EmailEvent_ProofpointPODMailLog_CL_IngestedLogs.csv" --expected-source-raw-table ProofpointPODMailLog_CL --target-id ASimEmailEventProofpointPODMailLog --target-group emailevent --run-id local-maillog-draft --schema-path "Sample Data\ASIM\ProofpointPODMailLog_CL_Schema.csv").
  • Commands above used the existing session-local Python runtime because the system python alias was unavailable; JSON outputs were saved outside the repository.
  • No live KQL sample execution, destination acceptance, connector route validation, workspace queries, deployment, or ingestion was performed locally. Static validation and fixture preparation do not establish runtime correctness.
  • The user explicitly authorized the repository's automatic ASIM canary workflow on Draft PR open/synchronize. Live filtering, ASIM data validation, and eligible query-time parity remain gated on that CI run, not the user's workspace.

Checked that the validations are passing and have addressed any issues that are present:

  • Local checks listed above pass. Live validation is pending the authorized automatic CI canary; this PR remains Draft.

Aditi Bhoge and others added 3 commits October 27, 2026 19:23
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@adbhogeGit
adbhogeGit marked this pull request as ready for review October 28, 2026 19:43
@adbhogeGit
adbhogeGit requested review from a team as code owners October 28, 2026 19:43
TargetASIMTable: 'ASimEmailEventLogs'
ParserFilterQuery: |
source
| where isnotempty(tostring(sm.qid)) and (bag_has_key(sm, "from") or array_length(sm["to"]) > 0)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fixture of ProofpointPODMailLog_CL should contain rows that return false with this condition.
Here is the ITP validation workflow result: https://github.com/Azure/Azure-Sentinel/actions/runs/40631618722?pr=15463

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a rejected fixture row with a nonempty sm.qid, no sm.from, and sm.to: [], plus a positive case with an empty sender. The canary on 4802006582 confirms filtering passes: 25 input rows, 24 selected, 1 rejected.

I also replaced unsupported bag_has_key with isnotnull(sm["from"]) in both queries and made delivery classification boolean to avoid a DCR numeric type mismatch. DCR compilation/provisioning succeeded and all 25 fixture rows were accepted for upload, but target visibility reached the 15-minute deadline before data validation/parity. That deadline result did not retain observed target counts, so it does not establish whether there were no rows or an incomplete expected set.

The failed-jobs rerun (attempt 2) stopped before runtime validation because asim-ingestion-template-asimemaileventproofpointpodmaillog-40786158180-2 was not found; the static job's artifact exists only with the attempt-1 suffix. Keeping this thread open until the runtime gate passes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retried the entire workflow (attempt 3), not just failed jobs, so the matching template artifact was regenerated and the attempt-2 artifact error is no longer the blocker: https://github.com/Azure/Azure-Sentinel/actions/runs/40786158180/attempts/3 . The filter again passed with 25 raw rows, 24 selected, and 1 rejected. The DCR compiled and accepted all 25 input rows, but target-visibility polling again exhausted the validation deadline. The artifact does not retain observed target counts, so it cannot establish whether no rows arrived or only some are missing. Runtime data/parity validation was not reached. Read-only diagnosis from this session is blocked because its Azure CLI account cannot access the shared canary subscription. Keeping this thread open pending canary diagnostics; no further blind retries or validation bypasses.

Aditi Bhoge and others added 4 commits October 28, 2026 16:39
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants