-
Notifications
You must be signed in to change notification settings - Fork 13
Postmortem 2026‐01‐07 ‐ Premature startup sequence completion PROVES ‐ Alcyone
The PROVES - Alcyone satellite has a premature antenna deployment in the clean room before integration into the launch pod.
At 10:48am the remove before flight (RBF) pin was removed, a pre-loaded startup sequence began running with a 45 minute wait before deploying the burn wire. At 10:56am the team synced the time from the stock RTC time of January 1, 2000 to January 7, 2026.
At 10:56am the startup sequence completed prematurely.
The antenna burn wire burned which released the antenna and payload door.
The team heard the antenna deploy which made an audible “twang” and saw the antennas deploy and the payload door open.
At 10:56am Ines Khouider engaged the satellite inhibit switch on the foot of the spacecraft to disconnect the battery and verified there was no physical damage to the satellite. Then Michael Pham reinserted the remove before flight (RBF) to persist the inhibited state.
At 10:57am Michael Pham began reviewing the logs to determine the cause.
At 10:59am Michael Pham determined the cause of the fault and called Nate Gay to explain the fault.
At 11:03am Ines Khouider exited the clean room and began attempting to reproduce the issue with Nate Gay and Saidi Adams with spare flight control boards.
At 11:11am Nate Gay confirmed with Michael Pham that the issue had been recreated and confirmed the root cause.
At 11:03am Evan Jellison entered the clean room to retie the burn wire.
At 11:12am Evan Jellison completed retying the antenna burn wire.
- 10:44 – Satellite unpacked.
- 10:48 – Remove Before Flight pin removed.
- 10:49 – Startup sequence removed.
- 10:52 – All sequences uploaded.
- 10:53 – Antenna deploy file removed.
- 10:54 – 11300 sequence number confirmed.
- 10:56 – Team syncs RTC from stock time (January 1, 2000) to January 7, 2026 (UTC).
- 10:56 – Startup sequence completes prematurely; antenna burn wire fires, deploying the antenna and opening the payload door (audible “twang,” visual confirmation of deployment).
- 10:56 – Ines Khouider engages the satellite inhibit switch to disconnect the battery and verifies no physical damage.
- 10:56 – Michael Pham reinserts the RBF pin to persist the inhibited state.
- 10:57 – Michael Pham begins log review to determine the cause of the fault.
- 10:59 – Michael Pham identifies the cause and calls Nate Gay to explain the fault.
- 11:03 – Ines Khouider exits the clean room and begins reproducing the issue with Nate Gay and Saidi Adams using spare flight control boards.
- 11:03 – Evan Jellison enters the clean room to begin retie of the antenna burn wire.
- 11:11 – Nate Gay confirms with Michael Pham that the issue was successfully reproduced and the root cause confirmed.
- 11:12 – Evan Jellison completes retying the antenna burn wire.
Operator initiated RTC time change while sequence is running
- Any events that change the time can cause unknown behavior
- We did not consider the impact of changing system time
- Immediately after the incident, the team amended the procedure to cancel all sequences prior to setting the time.
- Saidi Adams created a ticket to have the RTC component cancel all running sequences when a time set command has been initiated. Open-Source-Space-Foundation/proves-core-reference#282
- Nate Gay created a ticket to discuss having managers that monitor sequences may want to monitor for unexpected changes in time and cancel running sequences if one is detected. Open-Source-Space-Foundation/proves-core-reference#283
FPrime GDS Logs Postmortem 2026-01-07 - Premature startup sequence completion PROVES - Alcyone.txt