[nithil (nithillsethupathi)]

Required items, please complete

Change(s):

• Submit the parser content initially proposed in Add ASIM EmailEvent parsers for Proofpoint TAP permitted clicks #15480 using a same-repository branch.
• Add parameterless and filtering ASIM EmailEvent parsers for ProofPointTAPClicksPermittedV2_CL.
• Propose EmailInteraction as an additional EmailEvent event type for interactions with email content, such as a permitted URL click. Preserve the existing EmailDelivery and EmailInspection enum values.
• Normalize permitted URL Defense clicks as EmailInteraction, retain clicksPermitted as the original event type, and do not infer email delivery or HTTP success.
• Read the optional source id defensively using tostring(column_ifexists("id", "")), matching the merged POD MailLog approach in ASIM: Add Proofpoint POD MailLog EmailEvent parsers #15410. Preserve real IDs when available and leave EventOriginalUid empty when the column is absent; do not fabricate replacement identifiers.
• Register both parsers with the EmailEvent unifying parsers, including parameter forwarding, optional packing, and watchlist disable controls.
• Add paired changelogs, a fully synthetic ten-record sample covering the connector's 17 columns, and generated ARM templates.
• Align the KQL CI table fixture with all 17 solution-defined connector columns and the required Type metadata column; remove the stale threatsInfoMap, messageParts, and subject declarations absent from this click-table schema.

Reason for Change(s):

• Add query-time normalization for Proofpoint TAP permitted-click telemetry; existing Proofpoint POD parsers cover different sources.
• Distinguish interactions with email content from message delivery and inspection. The canonical enum change is proposed in this PR; published schema documentation and shared master-backed validators have not yet adopted the new value.
• Preserve one normalized event per source row and retain message, campaign, sender-IP, and user-agent context through optional packing.
• A missing optional identifier must not prevent otherwise valid events from normalizing. This parser-side change does not modify the shared ingestion helper or restore its ID-column coverage.
• Ingestion-time conversion is outside this PR and can follow the merged query-time parser.

Version Updated:

• Source parsers: 0.1.2 (initial version 0.1.0; EmailInteraction added in 0.1.1).
• EmailEvent unifying parsers: 1.0.2 to 1.0.3.
• Target normalized schema: EmailEvent 1.0.0.

Testing Completed:

• Partial; validation limitations remain disclosed below. No customer exports, live identities, or lab resource identifiers are included.
• Current optional-ID revision: both query bodies differ from 0.1.1 only by replacing the unconditional ID rename with the defensive optional-ID mapping. Filtering and all other mappings are unchanged.
• Four read-only functional queries passed against query-scoped synthetic tables: both parsers preserve a real ID, retain an empty ID, and execute successfully when the source table has no id column. Row counts and EmailInteraction classification were preserved. No tables, DCRs, tester functions, or Azure resources were changed.
• The earlier canonical CSV change remains limited to the EmailEvent EventType allow-list; both old values and the original CSV line endings are preserved.
• Both updated queries preserved all 727 lab rows with EventType=EmailInteraction and EventOriginalType=clicksPermitted. These were read-only queries; no tester functions or Azure resources were changed.
• | getschema | invoke ASimSchemaTester('EmailEvent') - no Error-level findings for either updated parser; the same 17 unavailable recommended-field warnings remain.
• | limit 1000 | invoke ASimDataTester('EmailEvent') - not passed for this revision: both existing helpers reject EmailInteraction in all 727 inspected rows because they load ASimTester.csv from master, where the proposed enum extension is not yet merged. The local branch's canonical allow-list includes the new value. The helpers and their schema URLs were not modified to bypass this pre-merge dependency.
• git -c core.whitespace=blank-at-eol,blank-at-eof,space-before-tab,cr-at-eol diff --check origin/master...HEAD - passed while preserving the canonical CSV's existing CRLF format.
• Existing scoped .NET KQL tests - passed, 4/4: both new source parsers and both EmailEvent unifiers. The command below uses PRNUM=15483 for PR-file discovery; <local-evidence-directory> replaces only the session-local TRX output path.

$env:PRNUM = '15483'
dotnet test '.script\tests\KqlvalidationsTests\Kqlvalidations.Tests.csproj' --no-restore --filter 'FullyQualifiedName~Validate_ParsersFunctions_HaveValidKql' --logger 'console;verbosity=normal' --logger 'trx;LogFileName=optional-id-kql.trx' --results-directory '<local-evidence-directory>'

• The previous KQL job failed because the fixture omitted nine real connector fields and Type. The fixture now matches the connector's column names and scalar types; no query mappings, validator logic, or skip lists were changed.
• Earlier GitHub KQL validation - passed, 14/14, with zero failures or skips on enum-extension commit 6f95f74b3ff306af84b0b2164a047324707891f9. The hosted checks for the later optional-ID revision are rerunning and are not recorded as passed here.
• python -, with the stdin invocation below of the existing VerifyASimParserTemplate.extract_and_check_properties - passed, 15 checks per source parser against their unifying parsers. No validator implementation or exclusion was changed.

import importlib.util
import re
from pathlib import Path
spec = importlib.util.spec_from_file_location('asim_template_checker', Path(r'.script\tests\asimParsersTest\VerifyASimParserTemplate.py'))
checker = importlib.util.module_from_spec(spec)
spec.loader.exec_module(checker)
root = Path(r'Parsers\ASimEmailEvent\Parsers')
failed = False
for kind, source, union in [('ASim', 'ASimEmailEventProofpointTAPClicksPermitted.yaml', 'ASimEmailEvent.yaml'), ('vim', 'vimEmailEventProofpointTAPClicksPermitted.yaml', 'imEmailEvent.yaml')]:
    results = checker.extract_and_check_properties(checker.read_local_yaml(root / source), checker.read_local_yaml(root / union), kind, str(Path(r'Sample Data\ASIM')))
    failures = [result for result in results if re.sub(r'\x1b\[[0-9;]*m', '', result[-1]) != 'Pass']
    print(f'{source}: {len(results) - len(failures)}/{len(results)} existing template checks passed')
    if failures:
        failed = True
        checker.print_results_table(failures)
raise SystemExit(1 if failed else 0)

• Earlier validation of the unchanged parser package, carried forward from Add ASIM EmailEvent parsers for Proofpoint TAP permitted clicks #15480:
  • git diff --cached --check - passed for the originally prepared package.
  • pwsh -NoLogo -NoProfile -File .\.script\kqlFuncYaml2Arm.ps1 - passed with the declared generator dependency available; generated only the EmailEvent package and retained every previously tracked ARM file. ARM templates were regenerated again when incorporating Microsoft Defender for Office 365 support.
  • Both parser queries executed against 727 lab records and preserved all 727 rows with EmailInspection and EventResult=NA.
  • Both packaged queries executed against the ten fully synthetic sample records. Recipient, IP, URL, threat-status normalization, and pack forwarding behaved as expected.
  • Inline schema-validation command: | getschema | invoke ASimSchemaTester('EmailEvent') - no Error-level findings for either parser; 17 unavailable recommended-field warnings remain.
  • Inline data-validation command: | limit 1000 | invoke ASimDataTester('EmailEvent') - no Error-level findings on the original sparse lab sample, which omitted recipient/IP/URL values and produced an empty-Dst warning. With populated synthetic URLs, the unchanged tester reports an EmailUrls array-element-type error described below.
  • The prescribed asimFilterTest.ps1 runner was attempted and exited with Schema: EmailEvent - Not supported by the validations script. Individual filter assertions were not executed; further filter testing was explicitly waived for this submission. This is not recorded as a passing filter suite.
• The full repository-wide .NET suite was not run locally; local KQL execution was scoped to the four PR parser cases above.

Checked that the validations are passing and have addressed any issues that are present:

• Need Help / not all checks pass. Known schema/test-definition issues are disclosed rather than suppressed:
  • The shared ASIM data tester loads its enum definition from master; it will continue rejecting the proposed EmailInteraction value until that schema extension is adopted. This PR includes the canonical enum change, not changes to validator code or its source URL.
  • The unchanged sample-ingestion helper includes id in its reserved-column list, which can omit that legitimate Proofpoint field from an inferred custom-table schema. The parser now tolerates this optional-column absence, but hosted ingestion still does not exercise real ID preservation; that behavior was verified separately with the read-only functional fixtures above.
  • The canonical ASimTester.csv defines EmailUrls as an array without an ArrayValuesType. The current ASimDataTester consequently rejects the nine valid string arrays in the populated sample because it expects an empty element type.
  • The supplied local filter validator has no EmailEvent schema entry, as noted above.
• Validator-support changes are limited to the ProofPointTAPClicksPermittedV2_CL table-schema JSON fixture and the explicitly proposed EmailEvent EventType enum extension in ASimTester.csv. KQL validator logic, skip lists, other canonical schema rows, filter-validator implementation, ingestion-helper logic, and workflow configuration are unchanged. The remaining issues above still require follow-up before claiming full CI readiness.