Fix OTA retry accounting and radio delivery races

Keep catalog, manifest and leaf requests pending when the local transmit
queue rejects them instead of spending their network retry allowance.

Keep the packet selected before CAD so work becoming due during sensing
cannot bypass its own channel or pacing checks. Retire cancelled radio
retries before waiting for BUSY hardware or their retry deadline.

Wait for active transmission completion before applying firmware. Stop
admitting new transmissions after the apply drain deadline so busy queues
cannot postpone reboot indefinitely, and keep those deadlines armed across
millis rollover. Retain the existing persistence-flush backoff.

Add failing-before/fixed-after regression coverage for all four issues,
including the production OTA apply and TX guards in CI.

Validation: 1,496 native tests, 42 Python integration tests, and five
firmware builds passed across ESP32-S3, nRF52, and STM32 targets.
This commit is contained in:
mikecarper committed 2026-09-15 06:45:28 -07:00
1 parent b2f2b5bf02
commit 5d69aa461f
11 files changed
+312 -24

No files matched your search

+12 -5
View File
@@ -413,16 +413,16 @@ void __attribute__((noinline)) Mesh::serviceLoopMaintenance() {
ota::OtaContext& oc = ota::ota_ctx();
if (oc.apply_pending) {
if (oc.apply_at == 0) {
oc.apply_at = futureMillis(1500);
oc.apply_hard = futureMillis(15000);
} else if (millisHasNowPassed(oc.apply_at) &&
oc.apply_at = ota::armedDeadline(_ms->getMillis(), 1500);
oc.apply_hard = ota::armedDeadline(_ms->getMillis(), 15000);
} else if (!hasOutbound() && millisHasNowPassed(oc.apply_at) &&
(_mgr->getOutboundTotal() == 0 || millisHasNowPassed(oc.apply_hard))) {
if (!prepareForOtaReboot()) {
// A role-specific persistence flush failed or is in bounded
// backoff. Keep the verified apply armed without hammering storage
// on every pass through the event loop.
oc.apply_at = futureMillis(1000);
oc.apply_hard = futureMillis(15000);
oc.apply_at = ota::armedDeadline(_ms->getMillis(), 1000);
oc.apply_hard = ota::armedDeadline(_ms->getMillis(), 15000);
} else if (oc.bootloader_apply_pending) {
ota::ota_reboot_to_bootloader_update();
} else {
@@ -521,6 +521,13 @@ void __attribute__((noinline)) Mesh::serviceLoopMaintenance() {
}
bool Mesh::allowPacketTransmit(const Packet* packet) const {
#if defined(ENABLE_OTA) && !defined(OTA_SEEDER_ONLY)
// Once the apply drain limit expires, let the active TX finish but decline
// new work. Otherwise a continuous queue can either prevent reboot forever
// or make the hard deadline cut off a packet that is still on air.
const ota::OtaContext* oc = ota::ota_context_if_active();
if (oc && oc->apply_pending && oc->apply_at != 0 && millisHasNowPassed(oc->apply_hard)) return false;
#endif
// This is an egress guard, separate from the receive-side TempRadio check below. A relay can queue an OTA
// packet just before its temporary window closes; never let that delayed packet leak onto the normal channel.
if (packet != NULL && packet->getPayloadType() == PAYLOAD_TYPE_OTA