[ANR] FCMBroadcastReceiver goAsync starved by 2-thread IO pool during init HTTP (android_params.js / sdk/features) #2722

Summary

FCMBroadcastReceiver ANRs on Android when a push arrives while OneSignal’s 2-thread IO pool is blocked on init HTTP to api.onesignal.com (android_params.js and sdk/features). goAsync() does not extend the ~10s broadcast timeout, so pendingResult.finish() never runs in time.

This is not a main-thread Application.onCreate stall. initWithContext returns in ~100ms on the main thread; the ANR is the broadcast watchdog.

Related: #2709 (open). Play Vitals data there showed 5.9.5 prewarm did not reduce FCMBroadcastReceiver ANR share. That issue attributes the problem to dispatcher construction race. Our traces show the dispatchers are already warm and both IO workers are stuck in TCP connect — pool saturation + timeout mismatch (and on many devices, IPv6 with no Happy Eyeballs).

Also related (different root cause, same symptom string): #1542, #1436, #2324, #2619.

Environment

Repro

1. Force-stop the app (or wait until the process is dead).
2. Send an FCM / OneSignal push so Play services delivers com.google.android.c2dm.intent.RECEIVE (or open the app and send a push in the first ~60s while init HTTP is still in flight).
3. Within ~10s: ANR Broadcast of Intent { act=com.google.android.c2dm.intent.RECEIVE cmp=.../com.onesignal.notifications.receivers.FCMBroadcastReceiver }.

Warm process after init HTTP has finished: later pushes usually do not ANR.

Timeline (5.9.8, one occurrence)

Times local GMT+8, pid 20919:

Same pattern on 5.9.9 with the app already in the foreground: init finished in 132ms; both GETs still connecting; Timeout of broadcast ... started 10001ms ago → ANR.

What the ANR dump shows

Main thread: typically idle (MessageQueue.nativePollOnce). Sometimes unrelated Compose draw. Not blocked in OneSignal.

OneSignal-IO-1 **and **OneSignal-IO-2 (both Native / poll):

at libcore.io.Linux.poll
at libcore.io.IoBridge.isConnected
at libcore.io.IoBridge.connectErrno
at java.net.PlainSocketImpl.socketConnect
at java.net.Socket.connect
at com.android.okhttp.internal.io.RealConnection.connectSocket
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getResponseCode
at com.onesignal.core.internal.http.impl.HttpClient$makeRequestIODispatcher$job$1.invokeSuspend(HttpClient.kt:206)
at com.onesignal.common.threading.ThreadUtilsKt$launchOnIO$1.invokeSuspend
at com.onesignal.common.threading.OneSignalDispatchers$launchOnIO$1.invokeSuspend(OneSignalDispatchers.kt:202)

SDK code (5.9.9) that matches this

1. FCMBroadcastReceiver.onReceive: OneSignalDispatchers.prewarm() then goAsync() then suspendifyOnIO { initWithContext; processBundle; pendingResult.finish() }.
2. OneSignalDispatchers: ThreadPoolExecutor(core=2, max=3, LinkedBlockingQueue(200)). JDK semantics: queue fills before max, so effective concurrency is 2.
3. ConfigModel.httpGetTimeout default 60000ms; HttpClient applies it to connect and read timeout. One hung GET holds an IO thread for up to 60s.
4. Cold start fires two GETs at once (android_params.js + sdk/features) → both IO threads busy.
5. Broadcast budget is ~10s. goAsync() does not extend it (Chronic background ANRs (FCMBroadcastReceiver, NotificationDismissReceiver, BootUpReceiver, SyncJobService) unchanged after 5.9.5 prewarm (PR #2664) — version-code data #2709). 60s vs 10s.

sequenceDiagram
  participant Play as PlayServices
  participant Rx as FCMBroadcastReceiver
  participant IO as OneSignal_IO_pool_2
  participant API as api.onesignal.com

  Note over IO: GET android_params.js
  Note over IO: GET sdk/features
  IO->>API: TCP connect (can hang 60s)
  Play->>Rx: c2dm RECEIVE
  Rx->>Rx: goAsync
  Rx->>IO: suspendifyOnIO processBundle
  Note over IO: queued, both workers in connect
  Note over Rx: 10s, finish never called, ANR

IPv6 / Happy Eyeballs (why connect hangs)

Same device, same URL, same session:

• Device has a global IPv6 address.
• api.onesignal.com has AAAA (Cloudflare). IPv6 path to that anycast is broken on this carrier.
• Java HttpURLConnection / InetAddress has no Happy Eyeballs (RFC 8305); it serializes addresses and sits on IPv6 until connectTimeout (60s).
• System.setProperty("java.net.preferIPv4Stack", "true") in Application.attachBaseContext did not change SDK connect behavior on this device.

This also fits #2709’s device mix (emerging-market OEMs where IPv6 is on but Cloudflare IPv6 routing is poor).

What we already ruled out (app side)

• Main-thread OneSignal.User / feature-gate runBlocking (separate ANR class; we moved those reads off main — main is idle in these dumps).
• Native POST_NOTIFICATIONS vs OneSignal.Notifications.requestPermission.
• R8 / missing -keep (staging/debug, minify off; stacks show full OneSignal names).
• UncaughtExceptionHandler (this is a watchdog timeout, not a crash).
• Downgrade-only experiments: 5.9.8 reproduced; 5.9.9 reproduced. [Bug]: One Signal causing too many ANRS #2619 also reported the same broadcast ANR string on 5.7.7 (different inner stack).
• App FirebaseMessagingService: Play still delivers c2dm to OneSignal’s receiver at priority 999 first.

Requested fix

Any of these would close the 10s window:

1. Do not share the 2-thread IO pool with goAsync work. Use a dedicated thread/pool (or Dispatchers.IO unbounded) so processBundle + finish() cannot wait behind 60s connects.
2. **Enqueue WorkManager then **finish() immediately — you already have NotificationWorkManager; today enqueue still sits behind the hung GETs.
3. Happy Eyeballs or IPv4-first for HttpURLConnection (or switch the SDK HTTP stack to OkHttp, which does HE).
4. Much shorter connect timeout for init GETs (seconds, not 60s), without blocking the FCM receiver on those calls.

prewarm() only helps pool construction, not workers stuck in connect().

Happy to attach a truncated dropbox excerpt (IO threads only) if useful.