Skip to content

Make device shutdown fail-safe - #281

Open
TobiasRoeddiger wants to merge 3 commits into
mainfrom
codex/shutdown-gpio-failure
Open

TobiasRoeddiger wants to merge 3 commits into
mainfrom
codex/shutdown-gpio-failure

Conversation

@TobiasRoeddiger

@TobiasRoeddiger TobiasRoeddiger commented Sep 16, 2026

Copy link
Copy Markdown
Member

Problem

A button or automatic shutdown must either enter charging-only mode while USB is present or reach nRF System OFF on battery. On main, several shutdown operations could fail, block forever, assert, or choose the wrong USB branch. Because the status LED was turned off early, any of those failures could leave the application and peripheral rails powered while the device appeared off.

This PR makes that shutdown sequence fail-safe. Normal shutdown still tries to stop sensors, finish the SD log, stop the DAC, and turn off the LED cleanly. A retained shutdown marker and a 30-second guard prevent those optional cleanup steps from keeping the device powered indefinitely. If cleanup stalls or asserts, the rebooted firmware detects the marker, skips all optional teardown and I2C access, drives the switched rails low directly, and enters System OFF. With USB connected, the same marker selects charging-only operation until the user explicitly starts the application or unplugs USB.

Failure handling

Area Previous failure and battery effect Fail-safe change and rationale Where
Overall shutdown progress Sensor, SD, I2C, LED, or device-PM code can wait forever. The task watchdog is not a reliable escape because other work can continue feeding it. Set a retained “stay off” marker before cleanup and start a 30-second shutdown guard. A timeout reboots into a minimal shutdown attempt that skips every optional blocker. PowerManager::power_down() in src/Battery/PowerManager.cpp; marker in src/Battery/BootState.h
Recovery after a stalled shutdown A normal reboot previously set power_on = true, so rebooting to escape a failed shutdown would start the application again. Read the retained marker before battery I2C access. On battery, go directly to minimal shutdown; on USB, remain in charging-only mode. Clear the marker only after an explicit start or immediately before successful System OFF. PowerManager::begin() and PowerManager::power_down()
Fatal assertions The OpenEarable debug fatal handler disabled interrupts and looped forever. An assertion during teardown therefore guaranteed continued drain. Store the same shutdown marker and cold-reboot. The next boot follows the minimal shutdown path instead of returning to the application or entering the debug loop. error_handler() in src/utils/error_handler.c
Bluetooth disconnect Disconnecting peers and stopping advertising issue synchronous HCI commands. An unresponsive network core can time out and assert during shutdown. Remove Bluetooth HCI commands from power_down(). System OFF or the charging reset already terminates the radio and connections. PowerManager::power_down()
Sensor manager Sensor stops use shared I2C mutexes, audio teardown can assert, and work-queue drain waits forever. Keep the normal clean stop, but bound the whole operation with the shutdown guard. The recovery attempt skips the sensor manager completely. stop_sensor_manager() call in PowerManager::power_down()
Full SD card FatFs can return a zero-byte write when the volume is full. The old flush loop consumed zero bytes and retried the same buffer forever. Treat zero progress as -ENOSPC, release the ring-buffer claim, and return. Still close the file so already-written data and metadata can be synced before shutdown continues. SDLogger::flush() and SDLogger::end() in src/SD_Card/SDLogger/SDLogger.cpp
DAC and task watchdog Their teardown results were ignored, and runtime-PM code can block. Log returned errors and continue. The shutdown guard covers a non-returning call; the recovery attempt skips both operations. PowerManager::power_down()
Log flushing LOG_PANIC() synchronously drains the logging backend and can wait indefinitely for UART output. Logs are not required for System OFF. Remove LOG_PANIC() from the shutdown path. PowerManager::power_down()
Console suspend The UARTE suspend path contains an unbounded hardware wait and is unnecessary because System OFF stops the peripheral. Remove explicit console suspension from the critical shutdown path. PowerManager::power_down()
Status LED Firmware deliberately turned the LED off before sensor, SD, wake-source, watchdog, and DAC cleanup. A later stall therefore looked like a successful shutdown. Move explicit LED shutdown after every optional cleanup step. If that final I2C operation stalls, the guard reboots and the recovery path skips LED I2C and removes its supply rails directly. An LED that is already dark is not forced on. PowerManager::power_down()
USB detection gpio_pin_get_dt() returns a negative error code on failure. Converting that value directly to bool reported USB as present and selected the charging-reset path indefinitely. Treat only a positive /PG reading as connected; log a read error and fail toward battery/System OFF. BQ25120a::power_connected() in src/Battery/BQ25120a.cpp
Wake GPIO setup A charger or gauge GPIO error returned from power_down() before rails were disabled and before System OFF. Log the error and continue shutting down. Charger /PG and /INT remain the intended button/USB wake sources. PowerManager::power_down()
Fuel-gauge wake BQ27220 GPOUT was enabled as a level wake source even though gauge events cannot perform useful work while the application is off. It could wake the nRF into an apparently-off application state. Disable the nRF interrupt for GPOUT before System OFF. Gauge interrupts remain active while the application runs; button and USB wake remain available through the charger. BQ27220::disable_wakeup_int() and PowerManager::power_down()
Switched rails The board PM callback ignored GPIO errors and always returned success. PowerManager then overwrote every result. A failed output change could leave SD, 3.3 V, or 1.8 V powered. Retry a failed GPIO write by reconfiguring the pin to the requested output state, return the real result, and check each rail result. After a timeout/fatal reboot, bypass device-PM locks and drive all three GPIOs low directly. set_load_switch(), generic_pm_control(), and openearable_power_rails_off() in boards/teco/openearable_v2/board_init.c; PowerManager::power_down()
Charging reset state A USB shutdown must reboot to show charging, but the software-reset reason previously forced the application on. The charging UI state structure was also partly uninitialized. Preserve the off request in GPREGRET across the reset, keep power_on false, and zero-initialize the status state. A button press explicitly clears the request and starts the application. PowerManager::begin() and PowerManager::power_down()
Code after System OFF The old sleep-and-reboot fallback appeared to protect sys_poweroff(), but sys_poweroff() is declared non-returning, so that code was unreachable. End with sys_poweroff() and CODE_UNREACHABLE; recovery is handled before that point by the guard and retained marker. PowerManager::power_down()

Why the fuel-gauge interrupt is disabled only while off

While the application runs, BQ27220 events are useful: the callback publishes charge state and starts orderly low-voltage shutdown. They are not useful after System OFF because no application remains to consume them and the nRF cannot charge the cell. With the retained gauge defaults, GPOUT can pulse for state-of-charge changes and BatteryStatus()[SYSDWN]; the firmware programs the SYSDWN set threshold to 3.25 V and clear threshold to 3.35 V. Disabling only the nRF GPIO interrupt before System OFF prevents those events from waking an intentionally off device. See the BQ27220 Technical Reference Manual, Table 4-6.

If charger wake setup itself fails, the firmware still enters System OFF instead of leaving the application running. A long button press remains the hardware reset recovery path on v2.7 through the BQ25120A-to-nRF RESETN connection.

Validation and scope

  • The complete signed FOTA/sysbuild for openearable_v2/nrf5340/cpuapp succeeds, including MCUboot, application core, network core, merged images, and dfu_application.zip.
  • A temporary external harness exercised the production SD flush/close code for a full card, partial write followed by full card, negative write error, successful partial writes, empty buffer, and close error. All six scenarios terminate correctly; the harness is not committed.
  • The PR changes eight implementation/header files. It adds no test files, configuration, documentation, or README changes.

@github-actions

Copy link
Copy Markdown

Build output available:
openearable_v2_firmware.elf.zip
openearable_v2_fota.zip

@TobiasRoeddiger
TobiasRoeddiger force-pushed the codex/shutdown-gpio-failure branch from 3347ab5 to c4f0a61 Compare September 17, 2026 03:52
@TobiasRoeddiger TobiasRoeddiger changed the title Complete shutdown when wake-source setup fails Complete shutdown and block fuel-gauge wakeups Sep 17, 2026
@github-actions

Copy link
Copy Markdown

Build output available:
openearable_v2_firmware.elf.zip
openearable_v2_fota.zip

@github-actions

Copy link
Copy Markdown

Build output available:
openearable_v2_firmware.elf.zip
openearable_v2_fota.zip

@TobiasRoeddiger TobiasRoeddiger changed the title Complete shutdown and block fuel-gauge wakeups Ensure battery shutdown reaches System OFF Sep 17, 2026
@github-actions

Copy link
Copy Markdown

Build output available:
openearable_v2_firmware.elf.zip
openearable_v2_fota.zip

@TobiasRoeddiger TobiasRoeddiger changed the title Ensure battery shutdown reaches System OFF Make device shutdown fail-safe Sep 17, 2026
@github-actions

Copy link
Copy Markdown

Build output available:
openearable_v2_firmware.elf.zip
openearable_v2_fota.zip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant