Re-enable Native Auth SSPR tests with Mail.tm, Fixes AB#3729032 by djanardhan98 · Pull Request #2553 · AzureAD/microsoft-authentication-library-for-android

Disha Janardhan (djanardhan98) · 2026-08-12T23:15:07Z

Summary

Re-enable the Native Auth SSPR end-to-end tests using authenticated Mail.tm inboxes.
Update Native Auth end-to-end tests to create registered inboxes when OTP retrieval is required and record checkpoints before requesting or resending an OTP.
Run the complete Native Auth end-to-end test package in PR validation with the secured Mail.tm password.
Update the Common submodule for the companion Mail.tm integration.

Motivation

Native Auth end-to-end tests need both dynamically created signup inboxes and an authenticated persistent inbox for SSPR. The Mail.tm integration in Common supports both workflows and allows the previously disabled SSPR coverage to run in CI.

Validation

All MSAL tests passed after applying these changes against Common commit ddb306a3884f9621e53b6eb3256fc14ea1497c7c.

Pipeline configuration

PR validation passes the secured EMAIL_PROVIDER_PASSWORD variable to the Native Auth end-to-end test task.
The dedicated task runs the complete Native Auth end-to-end test package, including SSPR.

Dependency and merge sequencing

Merge the Common PR first, then update this PR's Common submodule reference to the merged commit before merging.
Fixes AB#3729032

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add a dedicated PR-branch test invocation with the Native Auth and Mail.tm inputs required by SSPRTest. Use a deterministic too-short password for the invalid-format case and update Common for Mail.tm HTTP/1.1 compatibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Update Common to the latest mail-tm-native-auth-e2e commit and broaden the dedicated PR test filter to the complete Native Auth E2E package, including SSPR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-08-14T05:00:10Z

✅ Work item link check complete. Description contains link AB#3720943 to an Azure Boards work item.

Co-authored-by: djanardhan98 <307255345+djanardhan98@users.noreply.github.com>

Point the Common submodule at the latest commit from AzureAD/microsoft-authentication-library-common-for-android#3219. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 83947e30-5bc9-4ec7-a8ed-c03d4415da92

Copilot

Pull request overview

Migrates Native Auth E2E tests to Mail.tm and re-enables SSPR coverage. The current Common submodule API is incompatible with the shared test helper, causing compilation failure.

Changes:

Adds registered inboxes and OTP checkpoints.
Runs Native Auth E2E tests in PR validation.
Updates Common and AB ID extraction.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`.github/workflows/validate-pr-ab-id.yml`	Extracts explicit AB IDs.
`azure-pipelines/pull-request-validation/pr-msal.yml`	Adds Native Auth E2E validation.
`common`	Updates the Common submodule.
`NativeAuthPublicClientApplicationAbstractTest.kt`	Centralizes email service and retries.
`SSPRTest.kt`	Re-enables Mail.tm-backed SSPR tests.
`SignInEmailOTPTest.kt`	Adds OTP checkpoints.
`SignInJITTest.kt`	Creates authenticated inboxes.
`SignInMFATest.kt`	Adds MFA OTP checkpoints.
`SignUpEmailPasswordTest.kt`	Migrates signup inbox handling.
`SignUpEmailPasswordAttributesTest.kt`	Migrates attribute-flow inboxes.
`SignUpEmailOTPTest.kt`	Migrates OTP signup inboxes.
`SignUpEmailOTPAttributesTest.kt`	Migrates attributed OTP signup inboxes.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 83947e30-5bc9-4ec7-a8ed-c03d4415da92

Restore validate-pr-ab-id.yml to the dev branch version. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f268d2a-b8c4-4a22-826e-de2d9b9f59a6

…3219) ## Summary - Replace the Native Auth 1secmail inbox integration with authenticated Mail.tm mailboxes. - Pass the Mail.tm password through the existing Gradle `BuildConfig`/`BuildValues` pattern without committing credentials. - Add checkpoint-aware OTP polling, resend handling, MIME-aware OTP extraction, and bounded retries. - Force HTTP/1.1 for compatibility with the existing OkHttp client and Mail.tm. - Remove obsolete 1secmail response models and add regression coverage for the Mail.tm transport and inbox behavior. ## Motivation Native Auth end-to-end tests need both dynamically created signup inboxes and an authenticated persistent inbox for SSPR. Mail.tm supports both workflows, while the previous unauthenticated 1secmail integration cannot support the persistent SSPR mailbox used by the re-enabled tests. ## Validation - `.\gradlew.bat :testutils:clean :testutils:assembleDebug :testutils:testDebugUnitTest --no-daemon`: passed locally. - All MSAL tests passed after applying the corresponding MSAL changes against Common commit `ddb306a3884f9621e53b6eb3256fc14ea1497c7c`. - The corresponding MSAL changes will be submitted in a separate PR. ## Pipeline configuration - The MSAL consumer-validation task passes the secured Azure DevOps variable `EMAIL_PROVIDER_PASSWORD` from the pipeline's linked variable groups to Common as `-PemailProviderPassword=$(EMAIL_PROVIDER_PASSWORD)`; Common exposes it through generated `BuildConfig` and `BuildValues` without committing the secret. ## Related work - Fixes [AB#3720943](https://identitydivision.visualstudio.com/fac9d424-53d2-45c0-91b5-ef6ba7a6bf26/_workitems/edit/3720943) - AzureAD/microsoft-authentication-library-for-android#2553 ## Merge sequencing This PR is the expand step of a cross-repository migration: it temporarily retains `generateRandomEmailAddressLocally()` so both `dev` branches remain buildable. After AzureAD/microsoft-authentication-library-for-android#2553 migrates callers to `createRandomEmailAddress()` or `generateRandomUnregisteredEmailAddress()` and merges, the compatibility method will be removed in a Common follow-up. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Copilot-Session: 83947e30-5bc9-4ec7-a8ed-c03d4415da92

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Resolve the Common submodule pointer in favor of the current Common dev commit containing the merged Mail.tm integration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Shahzaib (shahzaibj) · 2026-08-24T18:00:02Z

+      - task: Gradle@2
+        displayName: Run Native Auth E2E Tests
+        inputs:
+          tasks: msal:testLocalDebugUnitTest -Plabtest -ProbolectricSdkVersion=${{variables.robolectricSdkVersion}} -PnativeAuthConfigString=$(NATIVE_AUTH_CONFIG_STRING) --tests com.microsoft.identity.client.e2e.tests.network.nativeauth.*


Are we gating every PR on these now? They hit mail.tm + a live tenant, and these are the same tests that were iGnOrE (@ignore)'d for OTP flakiness. I feel like unrelated PRs will start failing validation whenever mail.tm has a hiccup or the tenant throttles OTPs. Could these run on a nightly/scheduled pipeline instead of the per-PR gate?

Fair point — a mail.tm hiccup shouldn't fail a PR that never touched Native Auth.

I went with an escape hatch rather than nightly: a skip-native-auth-e2e-tests label drops the E2E task (same thing iOS does in pr-validation.yml), plus a 30 min timeout. The label check fail-safes to skipping, so if the labels API is down it can't block anything either.

Keeping them on the gate is what makes a regression attributable to the PR that caused it. The label gives us the unblock without giving up that attribution.

Let me know if you think that's a reasonable solution

Responds to review comments on PR #2553, aligning with the equivalent iOS/ObjC implementation (microsoft-authentication-library-for-objc). - pr-msal.yml: add a "skip-native-auth-e2e-tests" label opt-out so a PR is not blocked by live mail.tm / tenant availability. The check fail-safes to skipping when the labels API is unreachable or the build is not a PR build. Also bound the E2E task with timeoutInMinutes: 30. - Replace the brittle AADSTS701014 substring match with a typed classifier (NativeAuthEmailOTPErrorClassifier), mirroring iOS MSALNativeAuthEmailOTPErrorClassifier. AssertUtils now throws NativeAuthAssertionError carrying the typed Error, so the OTP throttle is detected from Error.errorCodes first, with the description substring retained only for failures that never carry typed codes. Adds 12 unit tests for the classifier. - Reduce throttle retry cost: 5 retries with uncapped exponential backoff (~155s) becomes 3 retries with a 20s cap (~35s worst case). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8ace3950-0a5f-47cf-bc7e-33687e701f5f

Shahzaib (shahzaibj) · 2026-08-25T18:35:53Z

+          echo "Fetching PR labels from ${LABELS_URL}"
+          # Capture the body and the HTTP status separately, so that a failed API call is
+          # distinguishable from a successful call that simply returned no matching label.
+          HTTP_STATUS=$(curl -sS -o "$LABELS_FILE" -w "%{http_code}" "$LABELS_URL" || echo "000")


This label check hits api.github.com unauthenticated. Unauth github API is 60/hr per IP, and on the shared windows-2022 pool that egress IP is shared across a ton of jobs, so on a busy day this'll come back 403 rate-limited. The fail-safe right below then treats any non-200 as skip=true, so the native auth e2e tests we're re-enabling would just silently stop running with a green pipeline and nobody notices. kinda defeats the point of making them a PR gate.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 556e2a2c-f4a4-4c93-8aa6-e1e414b4dd8e

Disha Janardhan (djanardhan98) · 2026-08-26T19:47:01Z

Copilot resolve the merge conflicts in this pull request, point the submodule to latest common dev branch

Co-authored-by: djanardhan98 <307255345+djanardhan98@users.noreply.github.com>

Copilot · 2026-08-26T19:52:26Z

Copilot resolve the merge conflicts in this pull request, point the submodule to latest common dev branch

Resolved the merge conflict and updated common to current Common dev commit a69c0c7ae14474193a8ce4060bcafc140d3ec5ab in merge commit e05aa50.

Shahzaib (shahzaibj) · 2026-08-26T20:18:45Z

+        if (errorCodes?.contains(THROTTLE_ERROR_CODE) == true) {
+            return true
+        }
+        return errorDescription?.contains(THROTTLE_ERROR_DESCRIPTION_MARKER) ?: false


I don't think the typed errorCodes path here fires for SSPR or sign-up at all. ResetPasswordChallengeApiResponse and SignUpChallengeApiResponse in common don't parse error_codes, and NativeAuthMsalController builds the APIError for those two challenge failures without errorCodes (reset password around line 1310, signup around 2025), so Error.errorCodes comes back null on both. Only signin/JIT challenge responses carry error_codes. 701014 also isn't in any of common's known-code lists.

So for the flows this PR re-enables (SSPR is right in the title) isThrottleError only ever reaches the errorDescription.contains("AADSTS701014") fallback. That's the same substring match we had before the classifier. The typed-first shape kinda implies the structured code is the primary signal, but for these two flows it's always null.

Are we ok leaning on the AADSTS701014 substring for SSPR/signup, given the typed path can't fire there?

Shahzaib (shahzaibj) · 2026-08-26T20:18:45Z

+        val actual = assertThrows(IllegalStateException::class.java) {
+            retryOperation(maxRetries = 1) {
+                attempts++
+                throw expected


IllegalStateException isn't an AssertionError, so it never reaches the catch (e: AssertionError) in retryOperation. it just falls straight through here, so this is only really checking that a stray RuntimeException isn't swallowed, not the throttle gating.

the branch I'd want pinned is a non-701014 AssertionError getting rethrown with no retry (the isThrottleError false -> throw e path). can we add that case? attempts should still be 1 and the original error should come back out.

Re-enable Native Auth SSPR tests with Mail.tm

a4b718a

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions Bot added the msal label Aug 12, 2026

Disha Janardhan (djanardhan98) and others added 4 commits August 12, 2026 16:20

Update Common after dev rebase

4ad73aa

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Document Native Auth E2E CI design

e9d6059

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Correct Native Auth E2E task design

dd6b5ce

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions Bot added the docs label Aug 13, 2026

Disha Janardhan (djanardhan98) mentioned this pull request Aug 13, 2026

Migrate Native Auth test email provider to Mail.tm, Fixes AB#3720943 AzureAD/microsoft-authentication-library-common-for-android#3215

Closed

Disha Janardhan (djanardhan98) and others added 2 commits August 13, 2026 21:24

Delete docs/superpowers/specs/2026-08-12-native-auth-e2e-ci-design.md

a6aba48

Run all Native Auth E2E tests in PR validation

a90fc17

Update Common to the latest mail-tm-native-auth-e2e commit and broaden the dedicated PR test filter to the complete Native Auth E2E package, including SSPR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions Bot changed the title ~~Re-enable Native Auth SSPR tests with Mail.tm~~ Re-enable Native Auth SSPR tests with Mail.tm, Fixes AB#3720943 Aug 14, 2026

Update pr-msal.yml

f56e844

Disha Janardhan (djanardhan98) mentioned this pull request Aug 14, 2026

Migrate Native Auth test email provider to Mail.tm, Fixes AB#3720943 AzureAD/microsoft-authentication-library-common-for-android#3219

Merged

Azure Active Directory (AzureAD) deleted a comment from github-actions Bot Aug 14, 2026

Disha Janardhan (djanardhan98) changed the title ~~Re-enable Native Auth SSPR tests with Mail.tm, Fixes AB#3720943~~ Re-enable Native Auth SSPR tests with Mail.tm, Fixes AB#3729032 Aug 14, 2026

Copilot started work on behalf of Disha Janardhan (djanardhan98) August 14, 2026 15:58 View session

Copilot AI and others added 3 commits August 14, 2026 15:59

Fix AB ID extraction in PR workflow

e94d321

Co-authored-by: djanardhan98 <307255345+djanardhan98@users.noreply.github.com>

Restore retries for SSPR E2E tests

3c4a8e3

Co-authored-by: djanardhan98 <307255345+djanardhan98@users.noreply.github.com>

Pass email password through pipeline env

e1124b5

Co-authored-by: djanardhan98 <307255345+djanardhan98@users.noreply.github.com>

Copilot finished work on behalf of Disha Janardhan (djanardhan98) August 14, 2026 16:05

Copilot started work on behalf of Disha Janardhan (djanardhan98) August 14, 2026 16:06 View session

Copilot finished work on behalf of Disha Janardhan (djanardhan98) August 14, 2026 16:07

Copilot started work on behalf of Disha Janardhan (djanardhan98) August 14, 2026 16:31 View session

Restore email password Gradle property

97dcc4b

Co-authored-by: djanardhan98 <307255345+djanardhan98@users.noreply.github.com>

Copilot finished work on behalf of Disha Janardhan (djanardhan98) August 14, 2026 16:33

Copilot started work on behalf of Disha Janardhan (djanardhan98) August 14, 2026 17:00 View session

Skip rate-limited SSPR OTP E2E scenarios

f259295

Co-authored-by: djanardhan98 <307255345+djanardhan98@users.noreply.github.com>

Copilot finished work on behalf of Disha Janardhan (djanardhan98) August 14, 2026 17:04

Copilot AI and others added 2 commits August 14, 2026 17:07

Retry rate-limited Native Auth E2E tests

97f1df6

Co-authored-by: djanardhan98 <307255345+djanardhan98@users.noreply.github.com>

Preserve Native Auth retry failure cause

41b12b8

Co-authored-by: djanardhan98 <307255345+djanardhan98@users.noreply.github.com>

Copilot finished work on behalf of Disha Janardhan (djanardhan98) August 14, 2026 17:10

Disha Janardhan (djanardhan98) and others added 2 commits August 14, 2026 13:51

Update Common Mail.tm integration

9686b50

Point the Common submodule at the latest commit from AzureAD/microsoft-authentication-library-common-for-android#3219. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Update Common runtime email credential integration

e356e84

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 83947e30-5bc9-4ec7-a8ed-c03d4415da92

Disha Janardhan (djanardhan98) requested a balanced review from Copilot August 17, 2026 20:03

Copilot started reviewing on behalf of Disha Janardhan (djanardhan98) August 17, 2026 20:04 View session

Copilot AI reviewed Aug 17, 2026

View reviewed changes

Comment thread ...dentity/client/e2e/tests/network/nativeauth/NativeAuthPublicClientApplicationAbstractTest.kt Outdated

Fix Native Auth email service initialization

a599345

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 83947e30-5bc9-4ec7-a8ed-c03d4415da92

Disha Janardhan (djanardhan98) marked this pull request as ready for review August 18, 2026 05:44

Disha Janardhan (djanardhan98) requested review from a team as code owners August 18, 2026 05:44

Revert PR AB ID workflow change

4bd47e5

Restore validate-pr-ab-id.yml to the dev branch version. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f268d2a-b8c4-4a22-826e-de2d9b9f59a6

Silviu Petrescu (spetrescu84) approved these changes Aug 21, 2026

View reviewed changes

Disha Janardhan (djanardhan98) and others added 2 commits August 24, 2026 09:40

Update Common to latest dev

62183f7

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge dev into Native Auth SSPR changes

6bbd3b1

Resolve the Common submodule pointer in favor of the current Common dev commit containing the merged Mail.tm integration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Disha Janardhan (djanardhan98) requested review from Sergei Demchenko (antrix1989) and fadidurah August 24, 2026 16:58

Shahzaib (shahzaibj) reviewed Aug 25, 2026

View reviewed changes

Shahzaib (shahzaibj) reviewed Aug 26, 2026

View reviewed changes

Stop retrying non-throttle Native Auth failures

03d18c5

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 556e2a2c-f4a4-4c93-8aa6-e1e414b4dd8e

Copilot started work on behalf of Disha Janardhan (djanardhan98) August 26, 2026 19:47 View session

Merge dev and update Common

e05aa50

Co-authored-by: djanardhan98 <307255345+djanardhan98@users.noreply.github.com>

Copilot finished work on behalf of Disha Janardhan (djanardhan98) August 26, 2026 19:52

Shahzaib (shahzaibj) reviewed Aug 27, 2026

View reviewed changes

Disha Janardhan (djanardhan98) marked this pull request as draft August 27, 2026 17:47

Navigation Menu

Uh oh!

Re-enable Native Auth SSPR tests with Mail.tm, Fixes AB#3729032 - #2553

Re-enable Native Auth SSPR tests with Mail.tm, Fixes AB#3729032#2553
Disha Janardhan (djanardhan98) wants to merge 24 commits into
devfrom
djanardhan/mail-tm-native-auth-e2e

Disha Janardhan (djanardhan98) commented Aug 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 14, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Shahzaib (shahzaibj) Aug 24, 2026

Uh oh!

Disha Janardhan (djanardhan98) Aug 25, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Shahzaib (shahzaibj) Aug 25, 2026

Uh oh!

Uh oh!

Disha Janardhan (djanardhan98) commented Aug 26, 2026

Uh oh!

Copilot AI commented Aug 26, 2026

Uh oh!

Shahzaib (shahzaibj) Aug 26, 2026

Uh oh!

Shahzaib (shahzaibj) Aug 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

Disha Janardhan (djanardhan98) commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Validation

Pipeline configuration

Dependency and merge sequencing

Uh oh!

github-actions Bot commented Aug 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Shahzaib (shahzaibj) Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

Disha Janardhan (djanardhan98) Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Shahzaib (shahzaibj) Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Disha Janardhan (djanardhan98) commented Aug 26, 2026

Uh oh!

Copilot AI commented Aug 26, 2026

Uh oh!

Shahzaib (shahzaibj) Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

Shahzaib (shahzaibj) Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Disha Janardhan (djanardhan98) commented Aug 12, 2026 •

edited

Loading