3-slot PSRAM stop (V4) that force-timed-out at 8s pre-fix now logs
timeout 29000 ms and acks clean in ~16.4s -- confirms the slot-scaled
timeout on both memory paths (V3 non-PSRAM 2-slot + V4 PSRAM 3-slot).
Phase 0 hardware characterization showed real mbedTLS/wss client teardown
takes ~5-6s per connected slot, sequentially, so the flat 8s
MQTT_STOP_TIMEOUT_MS force-timed-out healthy multi-slot stops (2-slot
non-PSRAM ~11-12s, 3-slot PSRAM ~16s), which sets the dirty latch and
makes the OTA barrier withhold flashing -- multi-slot nodes could never
ota update.
Replace the flat constant with a slot-scaled budget computed per stop in
end(): 5s base + 8s per enabled slot (~1.5-2x headroom over measured),
applied via new MQTTLifecycle::Coordinator::setStopTimeoutMs() before
requestStop(). Headroom is nearly free: end() returns as soon as the task
acks (checked before the timeout ticks), so a larger bound only lengthens
the wait before force-killing a genuinely wedged task.
Hardware-verified on V3: a 2-slot stop that force-timed-out at 8s pre-fix
now logs "timeout 21000 ms" and acks clean in ~11.7s. Native suite +
both observer firmware builds green.
Hardware run of the outstanding Phase 0 (teardown timing) and Phase 7
(fault-injection) items on V3 (non-PSRAM, 1 wss slot) and V4 (PSRAM,
3 wss slots) observer nodes, both flashed with this branch.
Primary finding (release-gating): MQTT_STOP_TIMEOUT_MS=8000 is too
small. Per-wss-slot teardown is ~5-6s sequential, so healthy multi-slot
stops (2 slots at the non-PSRAM max, 3+ on PSRAM) exceed 8s, trip the
dirty/timeout fallback, and the OTA barrier withholds flashing -- so
multi-slot nodes could never ota update, and the forced path also stalls
the loop task ~15-27s. Recommend raising the timeout (slot-count-aware
preferred, e.g. 4s + 6s*active_slots) before Phase 5 ships.
Phase 7: no heap leak or crash across the representative fault-injection
matrix on either board (incl. repeated forced teardowns); non-PSRAM
reconfigure-churn fragmentation is bounded and fully reboot-recoverable;
PSRAM largest-block rock-stable. OTA barrier latch validated on hardware
in both clean and dirty states.
Improvements:
- Add condition to only turn off display if on (avoids wedging the shutdown on some displays)
- Reset LoRa radio before calling radio_driver.powerOff() to ensure it is in a known good state (and will accept the command)
- Ensure SPI is started so LoRa radio commands can be sent (some shutdowns can happen before this)
- Add gate for P_LORA_NSS pin defined before writing to the pin
The WebConfig POST/result/reboot/stop batch state machine was the largest
remaining Phase 6 coverage gap (all inline in WebConfigServer.cpp, coupled to
AsyncWebServer/ArduinoJson and untestable on host). Extract its decision + timing
CORE into a pure, dependency-free spec mirroring MQTTLifecycle.h:
- src/helpers/WebConfigBatch.h: classifyPost (replay/busy/accept/no-changes with
the DONE-vs-PENDING reqid asymmetry), drain pacing (signed 25 ms gate, sticky
all_ok, 30 s reboot fallback), result classification + arm-once 3 s reboot,
signed-wrap-safe reboot-due / isRebootPending, and stop gating (finalize when
refs==0, warn-once, never force teardown). Constants verbatim from the source.
- test/test_webconfig_batch/: full host coverage incl. exact boundaries and
millis() rollover.
Spec-first, exactly like Phase 4's MQTTLifecycle.h: this is NOT yet wired into
WebConfigServer.cpp. That server is hardware-tuned (debugged against real iOS
captive-portal + HTTP-caching + route-ordering behavior), so making the spec
load-bearing is a deliberately separate, hardware-validated follow-up.
Faithfulness independently reviewed against WebConfigServer.cpp; native suite
green (14 dirs). No production behavior change.
Close the host-testable gaps named in Phase 6 of STABILITY_TESTABILITY_HANDOFF.md
by moving the last inline decision logic into the pure, host-tested policy seams:
- WiFi STA reconnect backoff: extract the inline ladder + wrap-safe timing from
handleWiFiConnection() into MQTTConnectionPolicy::{wifiReconnectBackoffMs,
wifiReconnectDue,nextWifiBackoffAttempt}. Behavior-preserving (elapsedMs is the
wrap-safe form of the old ULONG_MAX branch); ladder/clamp/attempt-cap unchanged.
- Publication outcome pairing: name the (packet, raw) -> delivered contract as
MQTTPacketQueuePolicy::queuedPacketPublished() and wire both queue-drain sites;
partial success = completed, not retried.
- Freeze MQTTPublicationType enum values in a test (the bridge-side MQTTMessageType
alignment is already enforced by a compile-time static_assert).
Adds host tests for all three (exact boundaries + millis() rollover). Native suite
green (13 dirs); non-PSRAM observer firmware smoke build compiles.
WebConfig batch/reboot/stop state-machine extraction and queue-orchestration
coverage remain open (tracked in the Phase 6 status).
Wire the Phase 4 MQTTLifecycle state machine into MQTTBridge to replace the
blind vTaskDelete teardown that could kill the MQTT task mid-mbedTLS and then
free client buffers on a corrupted heap (the observed OTA teardown panic).
- MQTTBridge owns a MQTTLifecycle::Coordinator driven only by the loop task
(Core 1) from begin()/end(); a nested LifecycleOps binds the host-tested Ops
spec to FreeRTOS/PsychicMqttClient.
- end() requests a cooperative stop; the MQTT task (Core 0) tears down its own
clients where the mbedTLS contexts live, acks via _stop_acked, and
self-terminates. end() waits (bounded) for the ack, then frees queue/buffers.
- Bounded stop timeout -> reviewed fallback (force kill + Core-1 teardown) sets
a dirty latch that withholds OTA flashing.
- begin() gains an idempotent double-call guard and syncs the Coordinator to
Running.
- OTA teardown barrier: simple_repeater's deferred flash aborts/resumes unless
end() reported a clean stop (canFlashAfterStop()).
Scope: minimal cooperative-shutdown unit. The volatile NTP/reconfigure handshake
replacement and the plain-data snapshot / consumer repointing (MQTT_OWNERSHIP.md
sections 1-3) are deferred. MQTT_STOP_TIMEOUT_MS is a Phase-0 placeholder pending
on-hardware characterization.
Native suite green (incl. test_mqtt_lifecycle); both observer firmware smoke
builds compile. Not yet hardware-validated (Phase 7 gate).
Fork-owned, host-tested MQTT bridge lifecycle state machine and the narrow
dependency seam to drive it deterministically, plus the cross-core ownership
model. This is the Phase 4 "ownership and teardown test seams" safety net that
must land before the Phase 5 cooperative-shutdown refactor.
- src/helpers/MQTTLifecycle.h: pure state machine
(Stopped->Starting->Running->StopRequested->Stopping->Stopped) + injected
Ops seam (clock / task / resource owner / OTA barrier) + Coordinator with a
bounded stop timeout. No Arduino/FreeRTOS/WiFi deps.
- test/test_mqtt_lifecycle/: 18 GoogleTest cases covering the phase's teardown
matrix (stop during every activity, callback timing, duplicate/early stop,
restart, timeout fallback, no-access-after-release) and the OTA-barrier
scenarios.
- MQTT_OWNERSHIP.md: one owner per mutable runtime domain, current cross-core
hazards with file:line references, target primitives, and Phase-0-pending
(hardware-characterization) items.
- STABILITY_TESTABILITY_HANDOFF.md: Phase 4 status and verified premise
refinements.
Scope: MQTTBridge.cpp is intentionally untouched. The production rewiring
(plain-data snapshot publication, volatile-handshake replacement, cooperative
shutdown) is deferred to Phase 5.
The handoff plan had drifted from the branch. Phases 1-3 are already
implemented, so this updates the plan of record to match reality and adds
a change-control section governing how agents extend the work.
Re-baseline:
- Add a Roadmap Status table and an explicit forward-plan execution order
(Phase 0 -> 4 -> 5 + OTA barrier -> 6 -> 7).
- Rewrite Current Baseline to include the landed CI, PSRAM buffer symmetry,
and versioned /mqtt_prefs migration work.
- Mark Phases 1-3 Complete with their residuals (build-size gate + ASan
pending; filesystem prefs adapter still in CommonCLI).
- Replace Phase 2's now-false premise (buffers are allocated in begin() /
released in end() / reallocated on restart, per MQTTRuntimeBufferLifecycle).
- Annotate Phases 0/4/5/6/7 with verified status, including the on-demand
getSlotStatusSnapshot() naming and the begin() double-call leak.
- Reframe the OTA teardown barrier as the fix for a known shipping heap panic.
Add "Change-Control Discipline (Stop-and-Ask)": agents must stop and ask
before implementing out-of-plan refactors or fixing newly discovered bugs,
report with evidence, and keep fixes single-purpose.
Introduced a new MQTT preset for the meshtexas connection, allowing
users to connect to mqtt.meshtexas.org with JWT authentication over WSS.
This addition increases the number of built-in presets to 27, enhancing
the flexibility of the MQTT implementation.
Rework the /mqtt_prefs load/save path so preference migrations are
crash-safe and, for the first time, unit-testable on the host.
Most of this is extraction. The multi-format migration that previously
lived inline in CommonCLI.cpp (and could only run on-device) is moved
into three dependency-free headers so it can be exercised without
Arduino, a filesystem, or the radio stack:
- MQTTPrefsStorage.h frozen layout structs for every /mqtt_prefs
format ever shipped, with static_asserts that
fail the build if any on-flash offset changes.
- MQTTPrefsCodec.h pure format classification, field-copy
migration, and plausibility validation.
- MQTTPrefsAtomicStore.h transactional writer plus the power-cut
upgrade gate, both host-testable.
New behavior, beyond the refactor:
- Atomic writes: /mqtt_prefs is written to /mqtt_prefs.tmp, verified,
then published with an atomic rename; the writer never removes the
existing file. A failed or interrupted save leaves the current
config intact.
- Power-cut ordering: LegacyUpgradeGate guarantees /mqtt_prefs is
durably committed before the legacy /com_prefs (or /node_prefs)
carrying the observer tail is compacted or removed, so an
interrupted two-file upgrade retries on the next boot without
losing settings.
- Corrupt, unsupported-version, and newer-than-known files are
preserved and the device boots on in-RAM defaults, rather than
overwriting a file this firmware cannot fully decode.
- Headerless legacy formats are validated for plausibility before
they are trusted and rewritten (raw prefs carry no checksum).
The full historical format matrix is migrated forward to the versioned
v1 layout: pre-slot (including pre-wifi-power), 3-slot (base and
token/topic tails), and headerless 6-slot (base, audience, rx, ntp).
Scope note: only /mqtt_prefs and the one-time /node_prefs -> /com_prefs
name migration use the atomic path. Ordinary /com_prefs saves remain a
direct rewrite, unchanged by this commit.
Tests: adds two host GoogleTest suites (pio test -e native).
- test_mqtt_prefs_codec: format classification, migration fixtures,
v1 header integrity, downgrade preservation.
- test_mqtt_prefs_atomic_store: transactional writes, short-write
detection, begin/finish/rename failure cleanup, original-file
preservation.
Replace direct JSON construction in MQTTMessageBuilder with calls to
MQTTPayloadBuilder for building status, packet, and raw messages.
This change improves code maintainability and reduces duplication by
centralizing message formatting logic. Additionally, update platformio.ini
to include ArduinoJson dependency for JSON handling.
Improve the mqttSubstituteTopic function to ensure that it reports
overflow when the output buffer is full, preventing silent truncation
of topics. Update related tests to verify the new behavior for
literal overflow and exact fit scenarios, enhancing robustness and
correctness of topic handling.
Add validation for request IDs in the web configuration server to ensure
they conform to the expected format. Enhance error responses for invalid
or unknown request IDs, improving the robustness of request handling.
Update the web UI to reflect these changes, ensuring that clients can
properly handle errors related to request ID mismatches.
setTxTimeoutMs() and setTxBufferSize() are HWCDC-only APIs; calling
them on HardwareSerial (UART-bridge boards like T-Beam) fails to compile.
Co-Authored-By: Adam Gessaman <adam@gessaman.com>
The new test_kiss_modem suite needs KissModem headers and sources that
only native_kiss_modem provides. Scope native to test_utils and run both
envs in the unit test workflow.
Switch ESP32-S3 KISS modem environments to HWCDC and move outbound KISS writes to a non-blocking queued frame path so loop() and TX completion keep progressing when the host reads slowly. Add native backpressure regression tests and correct the native_kiss_modem test filter so this suite runs directly with pio test.
Include detailed instructions for local testing of observer and WiFi
functionality without hardware. Document the use of a mock backend and
Wokwi ESP32-S3 simulation for easier development and testing.
Enhance the MQTT implementation documentation to improve developer
experience and facilitate testing workflows.
Enhance the WebConfigServer to track in-flight requests, ensuring that
the server is only freed once all requests have completed or a hard
timeout has been reached. This prevents crashes due to live connections
during server deletion. Update route handlers to log requests and
manage their lifecycle more effectively, improving stability and
performance of the web configuration portal.
Introduce a web configuration portal for easier node management and
provisioning without serial CLI. Enhance MQTT functionality with
improved IATA code validation, dynamic slot management, and
background NTP synchronization. Update web UI elements for better
user experience and security notes regarding open AP usage.
publishStatusToSlot hardcoded the retain flag to true, ignoring the preset's
allow_retain field that the packets and neighbors paths already respect. The
waev preset (allow_retain=false, MeshCore topic style) has therefore been
publishing retained status to a broker that rejects the retain flag; the same
would now apply to meshrank, which just gained status publishing. Use
slot.preset->allow_retain, defaulting to true for custom slots to preserve
their existing behavior.
MeshRank slots now publish every message type (status/packets/raw/neighbors)
under meshrank/uplink/{token}/{device}/, matching the standard MeshCore topic
layout, instead of packets only. The four duplicated topic-suffix ternaries are
folded into messageTypeSuffix().
Fix the discover.neighbors -> discover.scopes race: a scopes request issued
while a zero-hop discovery is still collecting responses (from discover.neighbors
or the periodic timer) now queues behind that 60-second window and runs against
the refreshed table, reporting the wait in its reply. A queued one-shot request
is no longer cancelled by `mqtt.neighbors off`; only the periodic timer's own
refresh is. Precondition checks are shared via neighborDiscoverReady() so queuing
cannot report OK for a request that will fail once the window closes.
Add the periodic-neighbors schedule to `get mqtt.status` as a trailing
nbr:<next>/<last> field (time to next publish + last result). The mesh owns the
timer and reports a summary to the bridge across the core boundary via atomics.
Also clamp the running offset in formatMqttStatusReply: snprintf returns the
untruncated length, so a full 6-slot line could push the offset past the buffer
and underflow the remaining size on the next append. Reachable today with real
preset-name combinations at the 160-byte reply size.
agessaman
entr0p1
taco
Liam Cottle
Jarosław Domański
mikecarper
ViezeVingertjesandAdam Gessaman
Scott Powell
Valentin
Ben AllfreeandCursor