39 KiB
Stability, Testability, and Upstream-Merge Handoff
Purpose
This document is the plan of record for refining the fork-owned WebConfig and MQTT observer code after the initial policy extraction and host-test work. The goal is to improve runtime stability, long-uptime confidence, and serviceability without broad rewrites of upstream-heavy files or creating unnecessary merge conflicts.
The order is deliberate: cheap CI and persistence guardrails land first, then tests and ownership boundaries needed to make lifecycle work safe, and only then the cooperative-shutdown refactor. Hardware soak testing validates the result; it is not the first line of defense for the riskiest change.
Roadmap Status
The guardrail phases have already landed on this branch; the remaining work is the lifecycle refactor and its safety net. Phases are intentionally not renumbered so cross-references and the completed acceptance criteria stay stable; each phase below carries an explicit status line, and this table is the quick index.
| Phase | Scope | Status |
|---|---|---|
| 1 | PR CI smoke builds + ArduinoJson pin enforcement | Done (build-size gate and ASan/UBSan still pending) |
| 2 | PSRAM restart resource symmetry | Done |
| 3 | MQTT preference migration fixtures | Done (filesystem adapter still lives in CommonCLI) |
| 0 | Pre-change lifecycle characterization | Hardware run 2026-07-19 (see "Hardware Characterization Results"): teardown timing measured on V3+V4. Finding: flat MQTT_STOP_TIMEOUT_MS=8000 too small → FIXED with slot-scaled timeout (5s + 8s×slots), hardware-verified (2-slot stop now clean) |
| 4 | Ownership and teardown test seams | Seams + ownership doc + teardown tests done; production rewiring deferred to Phase 5 |
| 5 | Cooperative MQTT shutdown | Minimal cooperative end() + begin() guard + OTA barrier implemented on branch phase5/cooperative-mqtt-shutdown (native green, firmware smoke build green); NOT hardware-validated. Volatile-handshake replacement + snapshot-consumer repointing deferred |
| — | OTA teardown barrier | Implemented — flash gated on a clean MQTT stop in simple_repeater; not hardware-validated |
| 6 | Request/queue/connection/publication integration tests | Partial: WiFi-backoff + publish-outcome + enum-alignment gaps extracted and host-tested on branch phase6/integration-tests; WebConfig batch/reboot/stop state machine has a pure host-tested spec (WebConfigBatch.h, not yet wired); queue-orchestration coverage still open |
| 7 | Uptime, memory, and fault-injection gates | Representative HW matrix run 2026-07-19: V3 non-PSRAM + V4 PSRAM done (no leak/crash; forced-path OTA-withhold + ~15–27 s loop stall observed). Multi-day soak + stack-HWM build pending |
Forward plan, in execution order: Phase 0 → Phase 4 → Phase 5 (with the OTA teardown barrier) → Phase 6 → Phase 7. Do not reopen a "Done" phase without a deliberate reason (see "Change-Control Discipline").
Hardware Characterization Results (Phase 0 & Phase 7) — 2026-07-19
Hardware run of the outstanding Phase 0 (pre-change/cooperative teardown timing) and Phase 7 (uptime/memory/fault-injection) items against two live observer nodes. Not yet committed as a plan change — this section records measured results and a release-gating recommendation for review.
Setup
- V3 — Heltec WiFi LoRa 32 V3, non-PSRAM (max 2 active slots), env
Heltec_v3_repeater_observer_mqtt, 1 configured slot (meshmapper, wss). - V4 — Heltec WiFi LoRa 32 V4, 2 MB PSRAM (max 5 active slots), env
heltec_v4_repeater_observer_mqtt, 3 configured slots (analyzer-us,cascadiamesh,waev, all wss). mbedTLS + JSON/raw buffers allocate in PSRAM. - Both flashed with this branch (
phase5/cooperative-mqtt-shutdown) viapio run -t upload(app-slot flash only;nvs+spiffspreserved, no erase). Note:FIRMWARE_VERSION/FIRMWARE_BUILD_DATEare hardcoded inMyMesh.hand were not bumped, soverreads "v1.16.0 (6 Jun 2026)" on both old and new builds — confirm the flash by behavior (the(clean)/ cooperative-stop log lines), not byver. - Driven over the serial CLI (
set bridge.enabled off/on=end()/begin();set mqttN.*= slot reconfigure;get mqtt.stats=Free/Max(largest block)/queue/outbox/per-slot counters). Host-side timestamped log parsing.
PRIMARY FINDING — MQTT_STOP_TIMEOUT_MS = 8000 is too small (release-gating)
Per-wss-slot teardown costs ~5–6 s, applied sequentially
(destroySlotClients(): disconnect() → 50 ms → esp_mqtt_client_destroy();
the ~5 s is the esp-mqtt task/network close, not the 50 ms settle). This is
unchanged from the pre-change path (same teardown code), so the cooperative
end() ack time scales with the number of connected slots:
| Config | Slots | Measured teardown | vs 8 s timeout | Result |
|---|---|---|---|---|
| V3 non-PSRAM | 1 wss | ack ~5.3–6.2 s (avg 5.8) | under | clean ✅ |
| V3 non-PSRAM (design max) | 2 wss | ~11–12 s (cut off at 8 s) | over | forced/dirty → OTA withheld ❌ |
| V4 PSRAM (normal config) | 3 wss | ~16 s (cut off at 8 s; per-slot disc @2.2/7.8/10.6 s) | over | forced/dirty → OTA withheld ❌ |
| V4 PSRAM (design max) | 5 wss | ~27–30 s projected | far over | forced/dirty → OTA withheld ❌ |
Pre-change (v1.16.0) reference teardown (same nodes, blocking end()): 1 slot
~6.1 s (V3), 3 slots ~16.4 s (V4) — consistent with the above.
Consequence: at the non-PSRAM board's designed maximum of 2 wss slots, and
on a normal healthy 3-slot PSRAM node, an ordinary cooperative shutdown exceeds
8 s → the coordinator fires StopTimedOut → sets the dirty latch → force-kills
the MQTT task (the exact mbedTLS-mid-teardown path Phase 5 exists to avoid) and
canFlashAfterStop() returns false. Because the OTA barrier only flashes after a
clean stop, this means any multi-slot device would have ota update
permanently withheld with the current timeout — the barrier misclassifies
healthy stops as dirty. This inverts the barrier's intent and must be fixed
before Phase 5 ships.
FIX APPLIED (slot-count-aware timeout) — verified on hardware.
The flat MQTT_STOP_TIMEOUT_MS = 8000 is replaced by a slot-scaled budget
computed per stop in MQTTBridge::end():
timeout = MQTT_STOP_TIMEOUT_BASE_MS (5000) + MQTT_STOP_TIMEOUT_PER_SLOT_MS (8000) × enabled_slots
→ 1 slot 13 s, 2 slots 21 s, 3 slots 29 s, 5 slots 45 s — ~1.5–2× headroom over
the measured ~5–6 s/slot teardown. end() counts enabled slots and calls
Coordinator::setStopTimeoutMs() before requestStop(). Headroom is nearly free
because end() returns as soon as the task acks (_stop_acked is checked before
the timeout ticks), so a larger bound does not slow a healthy stop — it only
lengthens the wait before force-killing a genuinely wedged task. (Files:
MQTTBridge.cpp constants + end(); MQTTLifecycle.h setStopTimeoutMs().)
Hardware verification (both memory paths): the same configs that force-timed-
out at 8 s pre-fix now ack cleanly — V3 non-PSRAM 2-slot logs timeout 21000 ms
and acks in ~11.7 s; V4 PSRAM 3-slot logs timeout 29000 ms and acks in ~16.4 s
→ MQTT Bridge stopped (clean) (OTA no longer withheld). Native suite + both
firmware builds green. Complementary future option (not done): shorten the
per-slot esp_mqtt_client_destroy() wait to reduce absolute teardown time.
Phase 0 — cooperative lifecycle (V3, non-PSRAM, this branch)
- 6× and 10×
begin→connect→end→restartcycles: all clean, ack 5.3–6.2 s, end() loop-task block ≤6.9 s, restart→connect ~5.4 s. - No leak across full stop/start cycles: free heap flat (~137–142 k, varies only with the meshmapper outbox), largest free block stable (~115–129 k). This satisfies Phase 5's "no downward heap/largest-block trend across start/stop cycles" acceptance criterion on the non-PSRAM path.
OTA teardown barrier
- Barrier input validated on hardware in both states: clean stops →
canFlashAfterStop()true; forced/timeout stops logOTA blocked/OTA flashing withheld(=canFlashAfterStop()false). ConfirmedmayBeginFlash() == (Stopped && !_stop_timed_out); a fresh start clears the latch; a dirty stop still permits restart. - Barrier action (
simple_repeateraborts+resumes vs. proceeds) is a one-line gate on that latch (MyMesh.cppdeferred-OTA site) — validated by code + the Phase 4 host lifecycle tests. - Not exercised end-to-end on hardware: the live
ota updatedeferred-flash path. A plainpio runbuild reportsERR: OTA not configured (build via build.sh), sootaFromManifestbails before scheduling; driving it needs abuild.shfirmware + a controlled manifest, and was not improvised on working fleet nodes (risk of flashing a real fleet build). Recommend a dedicated bench run for this.
Phase 7 — fault-injection matrix (V3 non-PSRAM; representative, not the soak)
Matrix: 10 clean stop/start + 5 forced 2-slot teardowns + 8 slot-reconfigure +
4 down-broker (wss://192.0.2.1) flaps. Time series of free/largest-block/
queue/outbox recorded per step; reboot/crash detection on.
- No crash across the entire matrix (incl. 5 repeated force-kill fallbacks) — the reviewed dirty-stop fallback is robust on non-PSRAM.
- Forced teardown reproducible: every 2-slot stop timed out (acks 7.4–8.2 s), reinforcing the primary finding.
- Bounded fragmentation, fully recoverable: slot-level reconfigure/flap churn (which keeps a disabled slot's persistent client until a full bridge restart) dropped the largest free block from ~125 k to ~72 k and held ~17 k of free heap, but it plateaued (stable across 12 churn iterations, not an unbounded leak) and a reboot fully restored ~141 k free / ~125 k largest block. This is pre-existing bridge behavior (TLS-context lifecycle), not a Phase 5 regression.
Phase 7 — forced-teardown stress (V4 PSRAM, 3 wss slots, this branch)
8× end()/begin() cycles at the node's normal 3-slot config (every stop
exceeds 8 s → forced/dirty):
- All 8 forced, no leak, no crash. Largest free block rock-stable at 139 252 across all 8 cycles (mbedTLS is PSRAM-allocated, so internal-heap fragmentation is absent on the PSRAM path); free internal heap flat. The force-kill fallback is robust on PSRAM too.
- Loop-task stall is severe on the forced path:
end()blocked the loop task ~15–27 s per stop, because the forced path waits the full 8 s timeout and then runs a complete redundant teardown on Core 1 (≈ 8 s + ~16 s). During that window the repeater's mesh/CLI/radio servicing is stalled. Sizing the timeout so the cooperative (Core-0) teardown completes cleanly removes both the OTA-withhold and this double-teardown stall. - Both nodes left restored to their original config, bridge on, heap healthy.
Limitations / not covered
- Task stack high-water mark and explicit task/client counts are not exposed by
the CLI; a dedicated
MQTT_MEMORY_DEBUGbuild is needed for those (heap stability was used as a proxy leak signal, and it held). - The 72 h / 7-day soaks were not run (bounded representative cycles per agreed
scope); WiFi loss/recovery, TLS-handshake failure, queue saturation, JWT/NTP
failure, and
millis()-rollover scenarios need external network control / time injection and were not driven over serial. - Both devices left restored to their original slot config with the bridge on.
Current Baseline
The current branch has:
- Native tests (GoogleTest,
[env:native]) for MQTT presets, validation, topic templates and routing, connection policy, packet-queue policy, payload construction, WebConfig keys, the/mqtt_prefscodec, the atomic prefs store, the runtime-buffer lifecycle, and upstreamUtils::toHexbehavior. - ArduinoJson pinned to 7.4.3 across the native and all firmware environments,
enforced in CI by
scripts/check_arduinojson_pin.py. - PR CI (
.github/workflows/) that runs the native suite and compiles both representative MQTT observer smoke builds:Heltec_v3_repeater_observer_mqtt(non-PSRAM) andT_Beam_S3_Supreme_SX1262_repeater_observer_mqtt(PSRAM). - Symmetric PSRAM runtime buffers:
begin()allocates throughallocateRuntimeBuffers()andend()frees and nulls throughreleaseRuntimeBuffers(), so raw-data caching and the PSRAM JSON buffers are restored after a restart (MQTTRuntimeBufferLifecycle.h). - A versioned
/mqtt_prefsloader extracted into fork-owned, host-tested seams (MQTTPrefsCodec.h,MQTTPrefsStorage.h,MQTTPrefsAtomicStore.h) that classifies every deployed layout, preserves an unknown or newer file without overwriting it, and rejects corrupt input without out-of-bounds reads. - WebConfig request/result correlation, failed-batch reboot gating, and a process-lifetime HTTP listener that avoids deleting an object still referenced by an asynchronous request.
- Pure MQTT policy helpers that reduce decision duplication while leaving the production bridge as the integration point.
This is a solid guardrail and unit-test foundation, but it does not yet validate the MQTT task/client lifecycle, cross-core state ownership, the OTA teardown barrier, or long-running heap behavior. Those are the remaining phases.
Constraints
- MQTT preference-file layouts are fleet-critical. Unknown newer formats must not be overwritten, and every supported old layout must remain recoverable.
- Upstream MeshCore changes are merged regularly. Prefer additive fork-owned helpers and small adapters over reorganizing upstream-heavy files.
- The non-PSRAM profile is the memory-pressure baseline; the PSRAM profile is a separate allocation/lifecycle path and must also be tested.
- Risky lifecycle changes require fast deterministic tests before hardware soak testing.
- Preserve observable behavior unless a behavior change is explicitly named, reviewed, and tested.
Change-Control Discipline (Stop-and-Ask)
This roadmap is executed incrementally, often by an agent working one phase at a time. Each unit of work is scoped to its assigned phase. When work uncovers something outside that scope, the correct action is to stop and ask, not to fix forward. Silent scope expansion is the primary way a targeted stability change turns into an unreviewed refactor or a needless merge-conflict surface. This section is binding on anyone — human or agent — executing the plan.
Stop and ask before writing code when any of these is true
- A refactor larger than the current phase authorizes appears necessary — for example touching task lifecycle, client lifetime, cross-core ownership, or the queue backends when the phase did not name them.
- A bug is discovered that is not described in the plan, especially one affecting persistence, teardown, OTA, or heap behavior.
- A phase premise turns out to be false or already implemented (for example, a described defect that is already fixed). Report the discrepancy and get the plan re-baselined before writing code against a stale assumption.
- A change would alter observable behavior the phase did not explicitly name, review, and test (Constraint 5).
- Work would touch upstream-heavy files (
MQTTBridge.cpp,CommonCLI, the role-specificMyMeshfiles) beyond a small, additive adapter (Constraint 2). - A change touches fleet-critical persistence layouts, or could overwrite or
discard an unknown or newer
/mqtt_prefsfile (Constraint 1). - Two phases would be combined in one change, or a "Explicitly Deferred Debt" item would be pulled forward.
- A fix spans multiple files, changes a public seam, or would surprise a reviewer expecting only the phase's stated work.
What "stop and ask" means in practice
- Do not implement the out-of-scope change in the same pass. Record it.
- Report it with evidence: file and line references, a concrete reproduction or failure scenario, and the specific invariant or constraint at risk.
- Present options with tradeoffs and a recommendation, then wait for an explicit decision on scope before proceeding.
- File a newly found bug as its own item. Do not fold an opportunistic fix into an unrelated phase's commit — single-purpose commits are required, and an unexpected fix deserves its own review.
What may proceed without asking
- Work squarely inside the assigned phase and its acceptance criteria.
- Small, behavior-preserving fixes fully contained in fork-owned helper files with matching host tests, where a reviewer would expect them as part of the phase.
When uncertain whether a change is in scope, treat it as out of scope and ask. The cost of a question is a round trip; the cost of an unreviewed lifecycle or persistence change across a fleet of thousands of devices is not.
Sequenced Work
Phase 0: Record the pre-change lifecycle characterization
Status: Not started — the next actionable step, and prerequisite for Phases 4 and 5.
Capture current behavior before changing shutdown mechanics. This provides a reference for the lifecycle fakes and lets the later state-machine refactor prove equivalence rather than relying on memory or a multi-hour soak.
Characterize at least:
- Normal
begin() -> connect -> end()ordering. end()while disconnected, connecting, connected, publishing, retrying, and applying a slot reconfiguration.- Which callbacks can arrive during and after
end(). - Queue disposition and connection/status counters across stop and restart.
- Heap, largest allocatable internal block, task stack high-water mark, and client/task counts before start, after start, after stop, and after restart.
- Partial initialization failures: queue allocation, task creation, client allocation, and PSRAM-buffer allocation.
Use instrumented hardware logging where necessary, but encode every behavior that can be represented deterministically into the lifecycle fake tests in Phase 4. Store representative logs as CI or test artifacts rather than enabling permanent high-volume production logging.
Phase 1: Put MQTT/WebConfig firmware smoke builds in PR CI
Status: Complete on this branch. PR CI already compiles both smoke builds and runs the native suite and the ArduinoJson pin check. Still pending against the acceptance criteria: build-size artifact/threshold reporting (criterion 4) and the optional ASan/UBSan native job. The description below is retained as the record of intent.
The PR build matrix historically did not compile an MQTT observer target; it now compiles two required smoke builds for changes touching firmware, variants, PlatformIO configuration, MQTT/WebConfig helpers, or their tests:
Heltec_v3_repeater_observer_mqttfor the constrained non-PSRAM path.T_Beam_S3_Supreme_SX1262_repeater_observer_mqttfor the PSRAM path.
Keep the native suite required. Add ASan/UBSan to a separate native job if the PlatformIO native toolchain supports it reliably. Record static RAM and flash usage and fail only on reviewed limits with enough headroom to avoid noisy one-byte regressions.
Acceptance criteria:
- Pull requests cannot merge when native tests or either representative MQTT build fails.
- The build log shows ArduinoJson 7.4.3 for both firmware profiles.
- CI checks that every ArduinoJson declaration remains pinned to 7.4.3.
- Build-size output is retained as an artifact or job summary.
Phase 2: Fix PSRAM restart resource symmetry
Status: Complete on this branch. The asymmetry described in the original plan has been fixed; this section now documents the intended design and the tests that guard it, not outstanding work.
PSRAM-backed raw-data and JSON buffers are allocated in begin() through
allocateRuntimeBuffers() (idempotent, allocate-if-missing), freed and nulled by
end() through releaseRuntimeBuffers(), and reallocated by a later begin().
Raw-data caching and the PSRAM JSON buffers therefore survive a restart; the
task-stack buffer is used only when a live allocation actually fails. The pure
helpers live in MQTTRuntimeBufferLifecycle.h, covered by
test_mqtt_runtime_buffer_lifecycle.
The design keeps symmetric runtime-resource operations:
allocateRuntimeBuffers()called bybegin()or a shared initialization path.releaseRuntimeBuffers()called byend()and initialization rollback.
The operations must be idempotent, handle partial allocation, and preserve the existing graceful fallback when PSRAM allocation fails. Avoid changing client or task shutdown behavior in this phase.
Tests and validation:
- Allocation success, partial failure, repeated allocation, and repeated release using a narrow allocator fake.
- Multiple
begin()/end()cycles do not lose the raw-data buffer or permanently move JSON serialization onto the task stack. - Initialization failure releases only resources owned by that attempt.
- Representative PSRAM and non-PSRAM firmware builds pass.
- A short hardware restart loop shows stable free heap and largest free block.
Phase 3: Add binary MQTT preference migration fixtures
Status: Complete on this branch. The versioned loader, the frozen-layout
static_asserts, and the migration/fixture coverage described below are
implemented in MQTTPrefsCodec.h, MQTTPrefsStorage.h, and
MQTTPrefsAtomicStore.h, and host-tested by test_mqtt_prefs_codec and
test_mqtt_prefs_atomic_store. Unknown-newer files are preserved and corrupt
input is rejected without overwrite. Residual: the concrete filesystem adapter
and the load/saveMQTTPrefs() orchestration still live in CommonCLI.cpp
(platform-coupled by design); keep that adapter small. The description below is
retained as the record of intent.
Build deterministic tests around the versioned /mqtt_prefs loader before
upstream merges change CommonCLI, filesystem behavior, or preference structs.
Use checked-in, non-secret binary fixtures or byte arrays representing each
deployed layout.
Required cases:
- Pre-slot layout.
- Three-slot layout.
- Legacy headerless six-slot layout.
- Shorter version-1 payload with newer fields defaulted.
- Current version round trip.
- Truncated header and truncated payload.
- Invalid magic and implausible payload length.
- Unsupported newer version: run with safe defaults and preserve the original file without overwriting it.
- Migration failure or interrupted save leaves a recoverable source file.
- Existing credentials, slot ordering, and publish flags survive migration.
Prefer extracting a fork-owned serializer/decoder seam over host-compiling all
of CommonCLI. Keep the production adapter small and retain the frozen-layout
static_asserts.
Acceptance criteria:
- All known deployed layouts have a fixture and field-by-field expected result.
- Corrupt input cannot cause an out-of-bounds read or silent overwrite.
- A future
MQTTPrefslayout change fails tests until its migration and fixture are added deliberately.
Phase 4: Establish ownership and teardown test seams
Status: Seams, ownership doc, and teardown tests landed; production rewiring
deferred to Phase 5 by explicit decision (scope: "seams + tests only"). The
fork-owned pure lifecycle state machine and narrow dependency seam
(src/helpers/MQTTLifecycle.h), the teardown-focused test matrix
(test/test_mqtt_lifecycle/), and the ownership model (MQTT_OWNERSHIP.md) are
in place. MQTTBridge.cpp was intentionally left untouched to keep the
merge-sensitive file free of churn until the Phase 5 change lands as one
reviewable unit. The invasive production work — publishing the plain-data
snapshot and repointing consumers at it, replacing the volatile handshakes
with a command queue / task notifications, and the cooperative-shutdown end()
rewrite with a begin() double-call guard — is carried into Phase 5.
Verified premise (with Phase 4 refinements): the loop task, WebConfig, CLI, and
AlertReporter currently read the MQTT task's live, mutable slot objects and
client counters cross-core without a lock or a published snapshot.
getSlotStatusSnapshot() is built on demand from live state despite its name —
the refactor must actually publish a plain-data snapshot, not assume one exists.
Refinements found while mapping the code (see MQTT_OWNERSHIP.md for
file:line references):
- The snapshot's
name/stateconst char*s point at static rodata, so they are not dangling; the real hazards are reading the mutableslot.presetpointer value (Core 0 can null/reassign it) andslot.client->getPublishOk()(a client pointer Core 0 candeleteduring teardown). - The live-bridge web reads are in the
MyMesh.cppapp layer (buildStatsJson), notWebConfigServer.cpp, which reads only a compile-time constant. end()clears only thes_mqtt_bridge_instancesingleton; the app'sbridgepointer andAlertReporter::_bridgeare not cleared, so instance reads can touch a torn-down bridge.
This phase is the safety net for cooperative shutdown. It must land before the shutdown state machine.
Ownership model
Document one owner for each mutable runtime domain:
- MQTT task: clients, slot connection state, NTP client operations, publish counters, and the packet drain path.
- Loop task: CLI execution, preference persistence, WebConfig batch draining, and bridge lifecycle requests.
- Producer/radio context: packet staging before queue handoff.
- Async TCP context: request parsing and immutable response handoff only.
Replace cross-core volatile handshakes with an appropriate primitive:
- Task notifications or a command queue for one-way lifecycle/reconfigure/NTP requests.
- Atomics only for truly independent scalar state.
- Immutable published snapshots for WebConfig, CLI diagnostics, and alerting.
- A mutex only where ownership transfer or snapshot publication cannot express the operation cleanly.
In particular, loop/WebConfig code must not directly inspect mutable MQTT slot objects or client counters. The MQTT task should publish a plain-data status snapshot.
Narrow lifecycle fakes
Introduce interfaces or callbacks for only the dependencies needed to drive the lifecycle deterministically:
- Clock/timer.
- Task start, stop request, acknowledgment, and timeout.
- MQTT client connect/disconnect and delayed callbacks.
- Queue depth/pop/requeue behavior.
- Runtime allocator and heap measurements.
- OTA coordinator/barrier.
Required teardown-focused tests:
- Stop while connecting, connected, publishing, retrying, renewing a token, running NTP, and applying a slot change.
- Callback delivered before stop, during stop, after disconnect, and after the stop acknowledgment.
- Duplicate stop, stop before full initialization, and restart after stop.
- Timeout/fallback behavior when the MQTT task or client does not acknowledge.
- No client, queue, buffer, or task access after its owner releases it.
- Current queue and diagnostic behavior is preserved unless a change is explicitly approved.
OTA Teardown Barrier: release-critical scenario
Status: Not started. This is the fix for a known shipping crash, not a
hypothetical hardening target. With a broker down over wss, the abrupt
vTaskDelete in end() can kill the MQTT task inside mbedTLS; destroySlotClients()
then frees client buffers on a possibly-corrupted heap and OTA begins flashing
with no barrier — the observed teardown heap panic. There is no coordination
today between MQTT shutdown and flash writing beyond straight-line ordering on
the loop task.
Treat OTA teardown as a first-class test target throughout Phases 0, 2, 4, 5, and 7. It is not merely another restart case.
The required invariant is: firmware erase/write must not begin until MQTT shutdown has reached a safe acknowledgment point, and the bridge must not be restarted while flash writing is active.
Required scenarios:
- OTA requested while MQTT is connecting, publishing, draining retries, or handling a callback.
- MQTT stop succeeds and OTA begins only after the teardown barrier.
- MQTT stop times out: OTA aborts safely rather than writing under uncertain ownership.
- OTA preflight or download aborts before flashing: the bridge restarts once and returns to the prior configured behavior.
- Successful OTA: no bridge restart is attempted before the device reboots.
- Power loss/reset at the platform-supported OTA boundaries retains a bootable partition; this portion requires hardware/platform validation.
- Repeated failed OTA attempts do not leak heap, duplicate WiFi callbacks, or leave the bridge permanently stopped.
Retain the prior teardown/heap-panic reproduction as a regression artifact if available. OTA-related lifecycle tests are release gates for any future change to bridge teardown, OTA sequencing, MQTT client lifetime, or task ownership.
Phase 5: Implement cooperative MQTT shutdown
Status: Minimal cooperative shutdown implemented on branch
phase5/cooperative-mqtt-shutdown (scope: "the smallest change that fixes the
OTA teardown panic as one reviewable unit"). Native suite green; the non-PSRAM
observer firmware smoke build compiles. NOT yet hardware-validated — that is the
Phase 7 gate — and the stop timeout is a Phase-0 placeholder (see below).
What landed (wiring the Phase 4 MQTTLifecycle state machine into the bridge):
MQTTBridgeowns aMQTTLifecycle::Coordinatordriven only by the loop task (Core 1) frombegin()/end(). A nestedLifecycleOpsbinds the pure, host-testedOpsspec to FreeRTOS/PsychicMqttClient.end()no longer blind-vTaskDeletes. It requests a cooperative stop; the MQTT task (Core 0) sees a newvolatile _stop_requested, tears down its own clients on Core 0 where the mbedTLS contexts live, sets_stop_ackedlast, and self-terminates.end()waits (bounded) for the ack, then frees the queue/buffers. This removes the "kill the task mid-mbedTLS, then free client buffers on a corrupted heap" teardown path.- Bounded stop timeout → reviewed fallback: on timeout the task is force-killed
and torn down on Core 1 (the old behavior), but a dirty latch is set so
canFlashAfterStop()is false and OTA flashing is withheld. begin()has a double-call guard and syncs the Coordinator toRunning.- OTA teardown barrier:
simple_repeater's deferred-OTA fire site aborts and resumes the bridge unless the precedingend()reported a clean stop.
Deferred (kept out of this reviewable unit; carried to a Phase 5b / Phase 6):
replacing the volatile NTP/reconfigure handshakes with a command channel, and
publishing a plain-data status snapshot to repoint the §1/§2 consumers in
MQTT_OWNERSHIP.md (the AlertReporter/buildStatsJson instance-pointer reads
that can still touch a torn-down bridge). Those are not required to fix the OTA
panic and would enlarge the merge-sensitive diff.
Phase 0 dependency — CHARACTERIZED + FIXED 2026-07-19: the flat
MQTT_STOP_TIMEOUT_MS = 8 s placeholder was too small (measured per-wss-slot
teardown ~5–6 s sequential → ~11–12 s at 2 slots, ~16 s at 3, ~27–30 s at 5; at
8 s healthy multi-slot stops tripped the dirty fallback and the OTA barrier
withheld flashing). Replaced with a slot-scaled timeout set per stop in
end(): 5 s + 8 s × enabled_slots (MQTT_STOP_TIMEOUT_BASE_MS /
MQTT_STOP_TIMEOUT_PER_SLOT_MS, applied via Coordinator::setStopTimeoutMs()).
Hardware-verified: a 2-slot stop that force-timed-out pre-fix now acks clean in
~11.7 s within the 21 s budget. See "Hardware Characterization Results (Phase 0 &
Phase 7)".
The original plan of record follows. Replace direct task deletion with an explicit lifecycle such as:
Stopped -> Starting -> Running -> StopRequested -> Stopping -> Stopped
The exact representation can differ, but it must provide:
- Idempotent start and stop requests.
- A stop request delivered through the established ownership channel.
- Cessation of new connects, publishes, retries, and reconfigurations.
- Ordered client/service shutdown on the MQTT task.
- A completion acknowledgment before the loop task releases queues, buffers, or other shared resources.
- A bounded timeout with clear diagnostics and a deliberately reviewed fallback.
- Safe restart after a completed stop.
- An OTA barrier that consumes the same completion acknowledgment.
Do not combine this phase with queue-loop deduplication, broad bridge cleanup, or unrelated feature changes.
Acceptance criteria:
- All Phase 4 lifecycle tests pass, including delayed/stale callbacks.
- Characterized behavior from Phase 0 is preserved or differences are explicitly documented and approved.
- OTA teardown-barrier tests pass.
- Repeated hardware start/stop cycles show no downward heap or largest-block trend and no task/client-count growth.
Phase 6: Expand request, queue, connection, and publication integration tests
Status: Partial — branch phase6/integration-tests (draft PR, base phase5).
The remaining inline decision points that were host-testable have been extracted
into the pure policy seams and covered:
- WiFi STA reconnect backoff moved out of
handleWiFiConnection()intoMQTTConnectionPolicy::{wifiReconnectBackoffMs,wifiReconnectDue, nextWifiBackoffAttempt}(behavior-preserving; adversarially reviewed for rollover/boundary equivalence) withtest_mqtt_connection_policycases. - The (packet, raw) publication-outcome pairing named as
MQTTPacketQueuePolicy::queuedPacketPublished()and wired at both queue-drain sites, withtest_mqtt_packet_queue_policycases (partial success = completed). MQTTPublicationTypevalues frozen intest_mqtt_topic_router; the bridge-sideMQTTMessageTypealignment was already a compile-timestatic_assert.
The WebConfig POST/result/reboot/stop state machine (the largest gap) now
has a pure, host-tested SPEC — src/helpers/WebConfigBatch.h +
test/test_webconfig_batch/ on branch phase6/webconfig-batch-seam — that
faithfully characterizes the accept/drain/result/reboot/stop decisions currently
inline in WebConfigServer.cpp. Like Phase 4's MQTTLifecycle.h, it is
spec-first and NOT yet wired: rewiring the hardware-tuned server to consume it
(so it becomes load-bearing and non-drifting) is a deliberately separate,
hardware-validated follow-up.
Still open (each a good follow-up PR): wiring WebConfigServer.cpp to the
WebConfigBatch spec above, and the queue-orchestration behaviors (FIFO
ordering, evict/requeue-failure interplay, the two adapters' drop-vs-keep-head
divergence), which need a fake-queue harness. The original scope list follows.
After lifecycle ownership is stable, broaden deterministic integration coverage:
- WebConfig POST/result/reboot/stop behavior, including lost responses, duplicate request IDs, concurrent clients, partial command failure, and stop with an active handler.
- MQTT queue behavior for FreeRTOS and circular-buffer adapters: overflow,
delayed retry, requeue failure, stale flush, queue ordering, and
millis()rollover. - Packet succeeds/raw fails and raw succeeds/packet fails.
- Connection backoff, stable reset, breaker probe, WiFi recovery, token renewal, and slot reconfiguration callback ordering.
- Topic and payload contracts at the bridge boundary, so enum/cast/adaptation mistakes are covered in addition to the pure helper tests.
Once both queue backends are covered by the same behavioral contract, consider
extracting a shared processQueuedPacket() that returns an outcome while each
backend retains pop/requeue/dequeue ownership. Do not make queue deduplication a
prerequisite for the stability work.
Phase 7: Establish uptime, memory, and fault-injection gates
Status: Not started. The final validation gate; runs after the lifecycle and OTA-barrier work is in place.
Use hardware soak tests to validate the already-tested design, not to discover basic lifecycle errors for the first time.
Run at least one constrained non-PSRAM board and one PSRAM board through:
- Stable broker operation.
- Broker unavailable, rejecting authentication, and flapping.
- WiFi loss/recovery and credential changes.
- TLS handshake failures and repeated reconnect backoff.
- Queue saturation and slow broker behavior.
- Repeated slot reconfiguration and full bridge stop/start.
- WebConfig start/save/stop cycles.
- JWT renewal and NTP failure/recovery.
- OTA success, preflight abort, download failure, teardown timeout, and resume.
- At least one test crossing the 32-bit
millis()rollover boundary, accelerated where hardware time injection is available.
Record and threshold:
- Minimum free internal heap.
- Largest free internal block.
- MQTT task stack high-water mark.
- PSRAM free/largest block where available.
- Queue and outbox high-water marks.
- Connect/disconnect/retry/publish counters.
- Watchdog and reset reason.
- Task/client counts across restart cycles.
Recommended gates are a 72-hour fault-injection run followed by a seven-day stable run. Store machine-readable time series and a concise summary artifact. Avoid enabling high-volume diagnostic logging in production builds.
Continuous Upstream-Merge Discipline
Apply these practices throughout every phase:
- Keep fork-owned logic in additive helper files and keep adapters in upstream-heavy files small.
- Avoid drive-by formatting, renames, and unrelated cleanup in
MQTTBridge.cpp,CommonCLI, and the role-specificMyMeshfiles. - Use single-purpose commits with messages that describe their full scope.
- Merge upstream frequently enough that conflicts remain attributable.
- Before each upstream merge, record the native-test, representative-build, firmware-size, and persistence-fixture baseline.
- After the merge, run native tests, both MQTT smoke builds, preference fixtures, and the relevant lifecycle/OTA tests before resolving the merge as complete.
- Review semantic behavior at adapters even when Git reports no textual conflict; upstream signature, lifetime, and task-context changes can invalidate fork assumptions silently.
- Reuse resolutions only after revalidating them against the new upstream code.
- Extract repeater/room-server WebConfig ownership into a shared fork helper when that integration next requires material change; do not perform a standalone broad move solely for aesthetics.
Explicitly Deferred Debt
These items are recognized but are not prerequisites for the sequenced stability work:
- Shared packet-drain processing between FreeRTOS and circular-buffer backends.
- Repeater/room-server WebConfig integration duplication.
- Moving the WebConfig HTML generator out of the common ESP32 build path.
- The secret-placeholder edge case for a credential exactly equal to the UI sentinel.
- Secure setup-AP authentication beyond the currently documented open/optional PSK threat model.
Revisit an item when its code is already being changed, when operational data raises its priority, or when tests make the refactor substantially safer.
Completion Definition
This roadmap is complete when:
- PR CI protects native logic and both representative firmware memory paths.
- Every deployed MQTT preference layout has a passing migration fixture.
- Runtime resources survive repeated start/stop cycles symmetrically.
- Cross-core ownership is explicit and diagnostics consume immutable snapshots.
- Cooperative shutdown passes deterministic lifecycle and OTA-barrier tests.
- Hardware fault-injection and stable soaks meet reviewed memory, stack, task, and reliability thresholds.
- Upstream merges use the same automated gate and do not require broad rewrites of fork-owned behavior.