device-mapper-multipath: resume DM map after failed remove

[suresh-thelkar]

Summary

_dm_flush_map() in multipath-tools 0.9.6 could leave a device-mapper map
suspended indefinitely. After a failed DM_DEVICE_REMOVE, it treated a
dm_is_mpath() query error (-1) as "removed externally" and returned success
without resuming; its normal-path resume was also a single, unverified
DM_DEVICE_RESUME. A leaked suspend hangs any flush / sync() on that device in
uninterruptible D state and spreads node-wide until reboot.

This PR adds a downstream patch that verifies the suspend, treats only
dm_is_mpath() == 0 as an external removal, and adds a verified
dm_resume_and_verify() that retries and confirms the resume — reporting failure
instead of leaking a suspended map.

Root cause

During map teardown, dm_flush_map suspends the DM map, then DM_DEVICE_REMOVE
fails with -EBUSY because the device is still open (in production, udev ran
blkid on the map's suspend/change event, raising open_count from 0 to 1 in the
window between multipath's pre-check and the remove ioctl). The failed-remove path
then mis-classified the dm_is_mpath() error and never issued a successful
DM_DEVICE_RESUME, leaving the map SUSPENDED. Any subsequent page-cache flush /
sync() on the device blocks forever in D state.

The fix

Patch0: 0001-resume-dm-map-after-failed-remove.patch (against 0.9.6):

• Verify the suspend state rather than assuming it.
• Treat only dm_is_mpath() == 0 as an external removal; a -1 query error is
  not a removal and must not short-circuit the resume.
• Add dm_resume_and_verify() — retries and confirms DM_DEVICE_RESUME, and
  reports failure instead of leaving the map suspended.

Testing

Reproducers (run as root in a disposable VM; scsi_debug / fake devices only).
Full sources are inline below. Build the injector first:
gcc -shared -fPIC -o dmfault.so dmfault.c -ldl.

Notes: the exact behavior of each reproducer — what the shell
scripts (repro-icm-843545099-inject.sh, repro-icm-843545099.sh) and the
dmfault.c injector do, step by step — is documented in detail in the header
comments of their respective source files (inline below).

• Deterministic before/after (recommended): repro-icm-843545099-inject.sh multipath-inject
  — uses dmfault.so (LD_PRELOAD, built from dmfault.c) to force the exact
  failed-remove + dm_is_mpath()→-1 condition.
  • unpatched → [ FAIL ] map left Suspended
  • patched → [ PASS ] map Active (resumed + verified)
• Kernel-mechanism proof (safe): repro-icm-843545099.sh mechanism — shows a
  suspended map wedges a flush in D state and a resume drains it.

Expected result with this build: multipath-inject reports PASS (map Active).

Reproducible sources are attached.

1. dmfault.c
2. repro-icm-843545099-inject.sh

Risk / impact

• Scope: teardown/flush path of device-mapper-multipath only; no on-disk
  format or ABI change.
• Behavioral change: on a failed remove the map is now resumed and verified
  instead of being left suspended — strictly safer (removes the hang).