Commit Graph
192 Commits
Author SHA1 Message Date
mikecarper ef80f29337 Expose broker publish acknowledgments for MQTT validation 2026-10-10 15:01:14 -07:00
mikecarper 7acc637468 Preserve MQTT TLS connection time and internal DMA headroom 2026-10-10 14:43:57 -07:00
mikecarper 894d033503 Harden recovery authentication and signed OTA admission 2026-10-10 01:25:08 -07:00
mikecarper 8c4e09275e fix(esp32): defer station maintenance while an AP owns WiFi
Starting or restarting MQTT during browser OTA could remove the active AP by forcing STA mode. Defer shared MQTT startup and recovery, and Companion WiFi startup, pending credential reloads and saved-SSID retries, until the live SDK AP releases the driver. Preserve the setup owner's deliberate credential handoff and cooperative stop acknowledgement.

Validated 91 source host tests, including 20 new AP ownership cases and nine reverted controls under ASan/UBSan. Hardware and firmware build qualification remain separate; the Arduino mode snapshot is not atomic.
2026-10-07 02:44:18 -07:00
mikecarper 1e421ff4fc Fix MQTT startup after ESP32 WebConfig provisioning 2026-10-06 19:28:10 -07:00
mikecarper 1d0ae06800 Keep the console responsive during MQTT shutdown 2026-10-05 19:05:58 -07:00
mikecarper 3add4695cc Harden USB logging and persist watchdog events in management reports 2026-10-03 15:29:19 -07:00
mikecarper 43f474a806 Integrate upstream fixes and paired nRF52 sensor OTA release profiles 2026-10-02 17:52:12 -07:00
mikecarper 95aca52b4e Add per-profile TX routing and bridge filtering
Persist Companion TX preferences per contact and channel. Default repeater,
room, and sensor replies to both active TX profiles, with a reply-only force
option for an RX-only secondary profile. Track both reply copies before a
temporary-radio handoff and account for OTA copies under queue backpressure.

Add bridge/crossover filter modes and the third built-in wardriving filter,
plus compact filter CLI syntax, documentation, and CI coverage.

Fix quoted target names being interpreted as slot/key selectors, partial
recovery of invalid saved reply settings, filter suspension during tempradio2,
and unnecessary packet allocation while OTA traffic is throttled.

Validation: 1,477 native tests, 52 Python integration tests, 29 filter UI tests,
and five firmware builds covering ESP32, nRF52, and STM32 passed.
2026-09-15 01:58:43 -07:00
mikecarper 0e0aa8e014 Add WiFi, ESP-NOW and combined wireless CLI controls 2026-09-15 00:01:45 -07:00
mikecarper 1629725d7d Add safe ESP32 transports and Indicator UI
Reserve board-specific GPIOs, make ESP-NOW channel/format handling configurable, and coordinate WiFi, BLE, NTP, MQTT, and host actions safely.

Add the Indicator message page, exclusive transport selector, dynamic 480/320 rendering, larger transport typography, pairing cleanup, and corresponding tests and documentation.
2026-08-31 14:29:42 -07:00
mikecarper 9fd580c89e Harden marathon-tested OTA and hardware paths
Add durable TempRadio, ACL, clock, bridge, GPS/I2C, USB logging, and Indicator recovery safeguards together with expanded automation, build/profile contracts, release metadata, and the hardware validation ledger.
2026-08-30 19:26:15 -07:00
mikecarper 6d6c89e411 fix: harden ESP-NOW and full companion networking
Add selectable shared ESP-NOW channels, raw/wrapped bridge compatibility, and bounded fragmentation for maximum-size packets.\n\nGive ESP32 Full profiles WiFi coexistence, stabilize WebConfig and USB terminal lifecycles, and keep disabled bridges stopped during observer updates.\n\nUpdate generated UI, firmware selection, documentation, and regression coverage.
2026-08-28 16:36:34 -07:00
mikecarper d6519baa0f feat: consolidate full companion runtime profiles 2026-08-28 08:47:26 -07:00
mikecarper bd3a94bddb fix: harden firmware builds and runtime reliability 2026-08-26 15:42:12 -07:00
mikecarper 5362d36ac7 Unify full firmware profiles and add dual USB logging 2026-08-23 19:22:45 -07:00
mikecarper 1e4fee0750 Merge remote-tracking branch 'review/observer-firmware' into keymindCascade
# Conflicts:
#	MQTT_IMPLEMENTATION.md
#	examples/simple_repeater/MyMesh.cpp
#	examples/simple_room_server/MyMesh.cpp
#	examples/simple_sensor/SensorMesh.cpp
#	src/helpers/CommonCLI.h
#	src/helpers/MQTTPresets.h
#	src/helpers/bridges/MQTTBridge.cpp
#	src/helpers/bridges/MQTTBridge.h
#	src/helpers/radiolib/CustomLLCC68Wrapper.h
#	src/helpers/ui/SH1106Display.cpp
2026-08-19 13:45:47 -07:00
mikecarper e742333a0e Update MeshCore features and OTA layout handling 2026-08-18 23:17:42 -07:00
mikecarper 7605b12cc7 Merge remote-tracking branch 'review/observer-firmware' into keymindCascade
# Conflicts:
#	.github/workflows/build-observer-firmwares.yml
#	.gitignore
#	MQTT_IMPLEMENTATION.md
#	MQTT_INTERNALS.md
#	README.md
#	STABILITY_TESTABILITY_HANDOFF.md
#	docs/cli_commands.md
#	examples/companion_radio/MyMesh.cpp
#	examples/simple_repeater/MyMesh.cpp
#	examples/simple_repeater/UITask.cpp
#	examples/simple_room_server/MyMesh.cpp
#	examples/simple_room_server/UITask.cpp
#	examples/simple_sensor/SensorMesh.cpp
#	platformio.ini
#	scripts/generate_webconfig_html.py
#	scripts/webconfig_mock_server.py
#	src/helpers/CommonCLI.cpp
#	src/helpers/CommonCLI.h
#	src/helpers/JWTHelper.cpp
#	src/helpers/MQTTDefaults.h
#	src/helpers/MQTTMessageBuilder.cpp
#	src/helpers/MQTTPayloadBuilder.h
#	src/helpers/MQTTPresets.h
#	src/helpers/bridges/MQTTBridge.h
#	src/helpers/esp32/WebConfigServer.cpp
#	src/helpers/esp32/WebConfigServer.h
#	src/helpers/ui/SH1106Display.cpp
#	variants/lilygo_tbeam_SX1262/platformio.ini
#	variants/lilygo_tbeam_SX1276/platformio.ini
#	variants/lilygo_tlora_v2_1/platformio.ini
#	variants/thinknode_m7/platformio.ini
#	webui/index.html
2026-08-14 08:53:02 -07:00
mikecarper 968ca3363b Add runtime FEM control and fix full Companion boot 2026-08-10 17:10:18 -07:00
agessaman dbdf2d732f fix(mqtt): scan every slot for a stale JWT after a clock correction
The post-NTP-correction refresh looped to _max_active_slots, which is a count of
activation positions and never an index bound. The indices holding those
positions are not contiguous: a slot passed over by isSlotReady() -- or, since
the demand-driven work, by a failed setup -- leaves a higher index activated. On
a two-position board that meant slots 1 and 2 could be live while only indices 0
and 1 were scanned, so slot 3 kept a JWT issued against the pre-correction clock
until its own expiry or a reconnect regenerated it.

Now bounded by RUNTIME_MQTT_SLOTS. The existing guard already skips disabled,
non-JWT, and clientless slots, so widening the range cannot touch a slot that
was never set up.

Pre-existing (the loop predates the demand-driven work) and kept as its own
commit so it can be picked separately. Audited the other _max_active_slots uses:
all are count comparisons or log arguments, so this was the only misuse.
2026-08-04 14:04:39 -07:00
agessaman 53bae5684e fix(mqtt): measure the setup-retry interval from the failure
last_reconnect_attempt starts at zero and teardownSlot() re-zeroes it, so the
retry gate in maintainSlotConnections() reduced to "uptime >=
SLOT_SETUP_RETRY_INTERVAL". Past 60 s of uptime a failed setup was therefore
retried on the next maintenance pass rather than 60 s later -- in the same task
iteration for a live reconfigure, since reconfigure processing runs before
maintenance in the loop.

setupSlot() now stamps last_reconnect_attempt on each failure that represents a
real attempt (client allocation, and both JWT token paths), so all three callers
get the interval measured from the failure. The bounds check and the !enabled
early return are not attempts and stay unstamped. The reconnect ladder reads
this field only for slots with initial_connect_done set, which a failed setup
never sets, so reconnect timing is unaffected.

The retry path's own pre-call stamp is kept as a backstop for any future
false-returning path that does not stamp itself.
2026-08-04 14:04:08 -07:00
agessaman 6be84eb4eb fix(mqtt): let a retried setup consume the reconnect allowance
reconnect_attempted_this_cycle is computed once before the maintenance loop and
only maintainSlotConnection() was setting it. The deferred-setup retry armed the
15 s cross-slot guard via _last_slot_reconnect_ms but left the local flag false,
so a disconnected slot later in the same pass still saw "no reconnect yet" and
started a second TLS handshake. Two concurrent ~40 KB sessions is exactly the
contention the guard prevents, and it landed in the one situation where internal
heap is already known to be short -- a failed allocation is why the retry runs.

Set on success only: setupSlot() returns true only after client->connect(), so
success means a handshake was launched. A failed retry launches nothing and
continues to spend only setup_retry_this_cycle, which rate-limits the allocation
attempts without consuming the handshake allowance.
2026-08-04 14:03:21 -07:00
agessaman e242cc6fd3 fix(mqtt): address review of the demand-driven slot work
[P1] A failed setup no longer strands the slot. setupSlot() returns bool and the
startup loops count only successful activations, so a slot that fails on a client
allocation neither consumes an active-slot position (starving a later healthy
broker on capped hardware) nor sits dead forever: maintainSlotConnections()
previously skipped clientless slots and the reconnect ladder is gated on
initial_connect_done, so nothing retried it. It now retries an enabled but
unactivated slot on a 60 s timer, one per cycle, gated on the same
_slots_setup_done ordering so the NTP-deferred setup sequence is preserved.

[P2] JWT setup no longer proceeds without a usable token. Both the preset and
custom-audience paths returned after ignoring createSlotAuthToken()'s result,
then called connect() and latched initial_connect_done -- so the token-allocation
failure introduced by the previous commit produced an unauthenticated attempt
exactly when memory was exhausted. They now return false and let the retry path
handle it.

[P2] ensureSlotAuthToken() no longer clears an existing token. It cleared
unconditionally, so every renewal wiped the current token before JWTHelper ran;
a renewal that then failed left an empty password where the inline buffer used to
preserve working credentials (JWTHelper writes only on success). Only freshly
allocated buffers are initialised now.

[P2] Raw publications reuse the shared document. buildRawJSON() reached
MQTTPayloadBuilder::buildRawMessage(), which constructed its own default
JsonDocument and therefore malloc'd and freed an internal-heap variant pool per
message -- on the highest-rate topic. The document is threaded through both
builders and the bridge passes _json_scratch_doc.

[P3] The writeTo() guard validates the source fields, not just the destination.
A corrupt payload_len of MAX_PACKET_PAYLOAD + 1 still leaves getRawLength()
inside MAX_TRANS_UNIT, so writeTo() read past packet->payload. Sizing and
validation moved to a pure MQTTWireScratch header with host tests covering the
accept/reject edges, matching the MQTTPacketFilter/MQTTConnectionPolicy pattern.
Two findings fell out: MAX_PATH_SIZE one-byte hops is not encodable (the hop
count is 6 bits, so 64 & 63 == 0; 32 two-byte hops is the widest real path), and
a zero-payload packet serializes but does not survive readFrom() -- pinned as a
test because it constrains any future wire-only queue.

[P3] Corrected the pool-size comment: these targets are 32-bit, so
ARDUINOJSON_SLOT_ID_SIZE is 2 and a pool block is 128 slots / 1024 bytes, not
4096. The 4096 figure came from a pre-existing comment near
NEIGHBORS_DOC_POOL_BUDGET, which is left alone -- its byte measurements are
empirical and still stand, only the block-size attribution is wrong.

Activation is now centralized in activatedSlotCount()/canActivateSlot(), used by
both startup loops, the retry path, and applySlotPreset(). That closes the
pre-existing divergence where a live preset change called setupSlot() without
consulting _max_active_slots, letting a non-PSRAM board reach three concurrent
TLS sessions against a cap of two. BEHAVIOUR CHANGE: a reconfigure that would
exceed the cap now logs and leaves the slot inactive instead of connecting.
Reconfiguring an already-active slot still works, because teardownSlot() releases
its position first.

272/272 native tests pass (5 new); both observer envs and an nRF52 repeater build
clean. Flash 1593249 B non-PSRAM, 1555625 B PSRAM.
2026-08-04 10:13:04 -07:00
agessaman b93b6b1560 perf(mqtt): keep slot JWT tokens in PSRAM where available
ensureSlotAuthToken()/releaseSlotAuthToken() now use the bridge's existing
psram_malloc/psram_free rather than malloc/free, so a JWT slot's 768-byte token
comes out of PSRAM instead of internal DRAM. On PSRAM boards this is the larger
half of the auth_token work: the previous commit reclaims 768 B per slot that
never creates a token, while this reclaims 768 B per slot that does -- up to
3840 B of internal DRAM on a fully configured five-slot board, which is where
the mbedTLS working set is competing for space.

No effect on non-PSRAM boards: psram_malloc falls back to internal DRAM, so the
buffer stays exactly where the previous commit left it. Same for a
BOARD_HAS_PSRAM board whose PSRAM fails to initialise.

Safe to move because every access is a CPU copy on the bridge task, never DMA,
an ISR, or a cache-disabled window: JWTHelper memcpy's the token into this
buffer, and esp-mqtt copies it out of _mqtt_cfg into its own internal-DRAM
storage when connect() applies the config. This is unlike the PSRAM-backed MQTT
task stack that was tried and reverted for resetting Heltec V4 boards, where
the fault was PSRAM execution context rather than a plain buffer read.

Split from the previous commit so it can be reverted alone if hardware soak
shows any PSRAM-related instability on the JWT path.

flash  non-PSRAM 1592801 -> 1592865 B (+64)
           PSRAM 1555213 -> 1555237 B (+24)
267/267 native tests pass; both observer envs and an nRF52 repeater build clean.
2026-08-04 09:47:06 -07:00
agessaman 9141ad854f perf(mqtt): allocate slot JWT token buffers on demand
MQTTSlot carried an inline char auth_token[768] -- 768 of its 1192 bytes --
for every runtime slot, whether the slot was JWT, username/password, disabled,
or above the active-slot cap. Now a char* allocated by ensureSlotAuthToken()
from createSlotAuthToken(), the sole writer, which runs only once a slot is
confirmed to have a JWT audience.

Because _slots[] is a fixed member array, this shrinks the bridge object
unconditionally at construction and re-adds only what is used:

  MQTTSlot     1192 ->  428 B  (-764)
  MQTTBridge  non-PSRAM 12032 -> 9740 B  (-2292)
                  PSRAM 10080 -> 5496 B  (-4584)

The object is allocated as one contiguous block at boot, so a smaller boot-time
chunk is easier to satisfy and leaves a larger contiguous remainder; JWT slots
then take 768 B each as separate blocks. Net steady-state saving is 768 B per
slot that never creates a token -- 768 B on a fully configured board, up to
3840 B on a PSRAM board with one JWT slot. 22 of the 29 presets are
MQTT_AUTH_JWT, so this is mostly reclaiming unconfigured slots rather than
non-JWT ones.

Lifetime rules, which are the whole risk here:
- setCredentials() stores this pointer in _mqtt_cfg rather than copying, and
  esp-mqtt re-reads it whenever a later connect() re-applies a dirtied config.
  So the buffer is freed only alongside the client, in destroySlotClients(),
  after the delete. MQTTSlot::broker_uri already carries an "avoids dangling
  pointer" comment from this same hazard with setServer().
- teardownSlot() still clears the token to an empty string but keeps the
  buffer: the client survives teardown and does not clear cfg->password (only
  setupSlot()'s reconfigure branch does). Freeing there would dangle.
- Never freed per reconnect, preserving the churn-avoidance the inline buffer
  was there for.

Allocation goes through MQTTRuntimeBufferLifecycle (already host-tested) using
plain malloc, so the buffer stays in internal DRAM exactly where it was when
inline. Failure propagates through createSlotAuthToken()'s existing bool.

The two call sites that create a token and then test it (preset JWT in
setupSlot, and the custom-slot audience path) now null-check first; the five
other readers are all inside if (createSlotAuthToken(...)) success branches and
need no change. destroySlotClients() releases the token unconditionally rather
than after its client null-check, so a token can never outlive its slot.

flash  non-PSRAM 1592697 -> 1592801 B (+104)
           PSRAM 1555109 -> 1555213 B (+104)
267/267 native tests pass; both observer envs build clean.
2026-08-04 09:45:30 -07:00
agessaman c833da10e9 perf(mqtt): allocate slot clients on demand
initSlotClients() created a PsychicMqttClient for every one of
RUNTIME_MQTT_SLOTS at begin(), without consulting slot.enabled or the active
cap -- even though presets are applied and _max_active_slots is computed
earlier in the same function. RUNTIME_MQTT_SLOTS is deliberately cap+1
(6/5 with PSRAM, 3/2 without), so at least one client was always unusable,
and a lightly-configured board wasted several.

Replaced with ensureSlotClient(index), called from setupSlot() -- i.e. only
for a slot that is enabled, inside the cap, and ready to connect. A client
that never reaches setupSlot() is completely inert: the reconnect ladder is
gated on initial_connect_done, which only setupSlot() sets. Retained for the
bridge lifetime as before, so reconfigure/reconnect still reuse one mbedTLS
context; that context is built by connect(), not by the constructor, so
deferring costs nothing but the 1284-byte object.

Internal DRAM saved, by configured slot count:
  non-PSRAM (3 runtime / 2 cap)   1 configured  2568 B
                                 2-3           1284 B
  PSRAM     (6 runtime / 5 cap)   1 configured  6420 B
                                 2             5136 B
                                 5-6           1284 B
A BOARD_HAS_PSRAM board whose PSRAM fails to init gets cap 2 against 6
runtime slots, so it saves at least 5136 B -- on the board that just lost its
PSRAM.

Two gates used "client != nullptr" as a proxy for "configured". Under eager
allocation that conjunct was always true and therefore harmless; with lazy
allocation it would have made shouldQueuePacketType() drop every packet
received before the post-NTP-sync slot setup, which is precisely the window
the queue exists to cover. Both now key on slot.enabled, matching the
documented intent above eligiblePacketSlots() that a configured-but-
disconnected broker is still a target.

formatSlotDiagReply() gains the !isSlotReady() -> "wait" branch that
get mqtt.status and getSlotStatusSnapshot() already have; that state
previously fell through to "disc" and read as a network fault when it is
really a missing token/IATA/credential. "no client" now means only what its
name says: the slot is ready but the client could not be allocated. That
state is newly reachable, so the allocation uses new (std::nothrow) -- this
framework enables C++ exceptions, and a throwing new on exhaustion would
panic the node instead of degrading one slot.

flash  non-PSRAM 1592573 -> 1592697 B (+124)
           PSRAM 1555001 -> 1555109 B (+108)
267/267 native tests pass; both observer envs build clean.
2026-08-04 09:16:47 -07:00
agessaman c28548aecf perf(mqtt): tranche 1 memory reductions in the bridge
Five independent, behavior-preserving reductions. Measured on
Heltec_v3_repeater_observer_mqtt (non-PSRAM) and
ThinkNode_M7_repeater_observer_mqtt (PSRAM), 267/267 native tests green.

Drop the unused static-task bookkeeping. StaticTask_t _mqtt_task_tcb (344 B)
and StackType_t* _mqtt_task_stack were never used: there is no
xTaskCreateStatic call, the pointer was assigned nullptr immediately before
xTaskCreatePinnedToCore, and the two psram_free() calls on it were dead.

Size the wire-format scratch buffers from the protocol maximum. raw_hex[1024]
at three sites becomes 2*MAX_TRANS_UNIT+1, and raw_buf/reconstructed[512]
become MAX_TRANS_UNIT. Both writeTo() sites now check getRawLength() first:
writeTo() does not bounds-check and returns uint8_t, so the old 512-byte
buffers were the only thing absorbing a malformed payload_len, and the
post-hoc "raw_len > sizeof(buf)" test ran after the overrun.

Pass the already-known serialized length into publishToSlot() instead of
re-running strlen() per destination slot (up to 2 KB per packet per slot, and
NEIGHBORS_JSON_BUFFER_SIZE per neighbor snapshot). Same for the direct
publish in publishStatusToSlot().

Share one JSON buffer and one document across packet, raw, and status. All
publish paths serialize on the bridge task, so the separate status buffer and
document were never concurrent. Status keeps STATUS_JSON_BUFFER_SIZE as its
serialization ceiling, so which oversized status documents get dropped is
unchanged.

Route the document's pools through a PSRAM-preferring allocator. Under
ArduinoJson 7 StaticJsonDocument<N> is a deprecated empty subclass of
JsonDocument whose template argument only feeds capacity(); the object is
64 B and each pool block (4096 B here) came from plain malloc(), i.e. the
internal DRAM the mbedTLS working set needs. Mirrors NeighborsDocAllocator.
The old comment claiming an inline pool has been corrected.

Measured:
  sizeof(MQTTBridge)  non-PSRAM 13208 -> 12032 B  (-1176, internal heap)
                          PSRAM 10492 -> 10080 B  (-412, plus one fewer
                                                   768 B PSRAM allocation)
  stack, non-PSRAM    buildPacketJSON[FromRaw] 1264 -> 736 B
                      buildRawJSON             1136 -> 608 B
                      publishPacket             688 -> ~432 B
                      packetToHex               560 -> ~304 B
                      deepest publish chain    ~2.6 -> ~1.8 KB of 8 KB
  flash               1592513 -> 1592573 B (+60)
  static RAM          74656 B unchanged -- MQTTBridge is heap-allocated, so
                      these savings are internal heap, not the linker figure

Deferred from the review: the QueuedPacket wire-only redesign (reward is
1.56 KB on non-PSRAM only, and Packet::readFrom() rejects payload_len == 0),
demand-driven slot clients/JWT tokens, and pool retention across publishes --
JsonDocument::to<T>() always calls clear(), which destroys pools, so
"retain pools by clearing the root object" needs a string-pool lifetime
analysis first.
2026-08-03 22:15:12 -07:00
agessaman a3a0a94dc4 feat(mqtt-neighbors): support non-PSRAM observer boards
Neighbors publication was gated on BOARD_HAS_PSRAM. Removing that gate alone
was not enough: the feature built but was inert without PSRAM, because three
allocation sites asked for MALLOC_CAP_SPIRAM (which returns null with no
PSRAM), the bridge's persistent buffer was allocated inside a BOARD_HAS_PSRAM
block, and neighborDiscoverReady() rejected every pass at runtime on
psramFound(). The entry table also did not fit: finishNeighborDiscover put
pubkey_hex[50][65] plus entries[50] on the stack, a 4752-byte frame against
the mesh loop task's 8 KB.

- Gate on MAX_NEIGHBOURS plus PSRAM or an explicit per-variant
  MQTT_NEIGHBORS_WITHOUT_PSRAM opt-in.
- Move the entry table and its hex strings into one heap block sized to the
  pass; the frame drops from 4752 to 304 bytes.
- Prefer PSRAM and fall back to internal DRAM in the mesh-side allocations
  and the ArduinoJson pool; hoist the bridge's persistent buffer out of the
  BOARD_HAS_PSRAM block (psram_malloc already falls back).
- Keep the runtime psramFound() check only where the buffers are sized for
  PSRAM, so a board whose PSRAM failed to init still refuses.
- Size for internal DRAM without PSRAM: 4 KB text buffer and 20 entries per
  publish, keeping the pool to a single block and the peak near 13 KB rather
  than ~35 KB. Oversized tables truncate and report total_neighbors as before.

Enabled on the ESP32-S3 observer envs (Heltec V3/WSL3, RAK3112, Heltec
Tracker v1.1/v2). Left off for the classic ESP32 T-LoRa V2.1-1.6, which is
already limited to one active TLS slot.

Costs ~7.4 KB static DRAM on repeaters and ~9.6 KB on room servers. The
prefs layout is unchanged, so this is neutral for existing devices.
2026-08-03 10:29:54 -07:00
agessaman 616fe96d1e feat(mqtt): publish all message types except raw to MeshRank
MeshRank slots previously took packets only. The broker tolerates status and
neighbors (and its maintainer intends to look at using the neighbors data), so
those now publish under meshrank/uplink/{token}/{device}/{type}, using the same
type suffixes as the MeshCore layout. Raw stays excluded: it is the
highest-volume topic and the broker does not consume it, so `set mqtt.raw on`
has no effect on a MeshRank slot.

All MeshRank gating funnels through the topic router, so relaxing that single
guard was sufficient - publishStatusToSlot, publishStatus, publishRaw,
publishNeighbors, and eligiblePacketSlots already skip slots that cannot form a
topic. A per-slot token is still required. Because eligiblePacketSlots resolves
topic support before serialising, raw JSON is never built for a MeshRank slot
rather than built and discarded.

observer-firmware sends raw to MeshRank; this supersedes that decision, so raw
must stay excluded when the branches merge. MeshRankTakesEveryTypeExceptRaw
fails if a merge re-enables it.
2026-07-29 15:51:28 -07:00
agessaman 52bd27190f feat(mqtt): enhance packet filter functionality with named types
Add support for named packet types in per-slot filters, allowing users
to specify packet types using descriptive names alongside numeric values.
This improves usability and clarity in configuring MQTT slot filters.
Updates include modifications to the parsing logic, WebConfig interface,
and related documentation to reflect the new naming conventions.
2026-07-27 22:29:11 -07:00
agessaman e00b29b4b8 feat(mqtt): add per-slot packet filters for MQTT slots
Introduce per-slot packet filters to allow users to specify which
packet types are uploaded for each MQTT slot. This feature enhances
the flexibility of the MQTT bridge by enabling users to configure
allowlists for packet types, improving the efficiency of data
transmissions. The implementation includes updates to the WebConfig
interface, internal handling of packet filters, and necessary
modifications to the MQTT preferences structure.
2026-07-27 18:34:04 -07:00
agessaman 3c5b25c2cf feat(mqtt): enhance neighbor discovery JSON handling
Introduce new methods to manage neighbor discovery JSON budget and
entries in MyMesh. This includes tracking the number of queried and
published neighbors, measuring JSON sizes, and handling truncation
when the buffer limit is reached. These improvements optimize the
neighbor discovery process and ensure efficient JSON message
construction for MQTT communications.
2026-07-26 19:10:51 -07:00
mikecarper ae2af7ef4d Harden observer preferences and portable controls 2026-07-25 07:59:44 -07:00
mikecarper c1caa5ad81 Merge observer firmware and webconfig updates
# Conflicts:
#	MQTT_INTERNALS.md
#	build.sh
#	docs/cli_commands.md
#	examples/simple_repeater/MyMesh.cpp
#	examples/simple_repeater/MyMesh.h
#	examples/simple_repeater/UITask.cpp
#	examples/simple_room_server/MyMesh.cpp
#	examples/simple_room_server/MyMesh.h
#	examples/simple_room_server/UITask.cpp
#	examples/simple_sensor/SensorMesh.cpp
#	platformio.ini
#	scripts/generate_webconfig_html.py
#	src/MeshCore.h
#	src/helpers/CommonCLI.cpp
#	src/helpers/CommonCLI.h
#	src/helpers/CommonCLI_Observer.cpp
#	src/helpers/ESP32Board.h
#	src/helpers/MQTTMessageBuilder.cpp
#	src/helpers/MQTTPresets.h
#	src/helpers/NRF52Board.cpp
#	src/helpers/NRF52Board.h
#	src/helpers/bridges/MQTTBridge.cpp
#	src/helpers/bridges/MQTTBridge.h
#	src/helpers/esp32/WebConfigServer.cpp
#	src/helpers/esp32/WebConfigServer.h
#	src/helpers/radiolib/RadioLibWrappers.cpp
#	src/helpers/radiolib/RadioLibWrappers.h
#	variants/lilygo_tbeam_SX1262/platformio.ini
#	variants/lilygo_tbeam_SX1276/platformio.ini
#	variants/lilygo_tlora_v2_1/platformio.ini
#	webui/index.html
2026-07-24 02:16:47 -07:00
mikecarper b98a7a7b54 Keep ESP32 OTA images within legacy slot 2026-07-22 15:04:40 -07:00
mikecarper 371d86c4b1 Normalize tracked text to ASCII 2026-07-22 00:43:38 -07:00
agessaman 7daccd9efd feat(mqtt): include outbox drop statistics in MQTT stats reply
Enhance the MQTT stats reply to include outbox drop statistics,
providing users with better insight into message handling and
memory pressure. This change improves the clarity of the status
message and aids in diagnosing potential issues with message
delivery.
2026-07-21 08:42:25 -07:00
agessaman 2d55b95ee5 refactor(mqtt): improve command handling and memory pressure management
Enhance the command handling in CommonCLI_Observer to clarify the
compilation behavior under different configurations. Additionally,
optimize memory pressure checks in MQTTBridge to reduce unnecessary
heap walks, improving performance during sustained memory pressure.
This change ensures that publishes are skipped only when necessary,
enhancing overall stability.
2026-07-20 15:31:54 -07:00
agessaman 1b84311183 fix(mqtt): remove unused time conversion calls in syncTimeWithNTP
Eliminate unnecessary calls to gmtime and localtime in the
syncTimeWithNTP function. This cleanup improves code clarity
and reduces potential confusion regarding time handling.
2026-07-20 15:20:58 -07:00
agessaman c2b157dda0 feat(mqtt): add pending status publish handling for MQTT slots
Introduce a mechanism to manage on-connect status publishing for MQTT
slots. This change allows the MQTT task to handle status updates
safely, ensuring that shared resources are accessed in a controlled
manner. The new flag `_status_publish_pending` is set in the onConnect
callback and processed in the main MQTT task loop, preventing race
conditions and improving overall stability.
2026-07-20 10:34:19 -07:00
agessaman fed561113a refactor(mqtt): optimize MQTT status and diagnostic reply formatting
Replace multiple snprintf calls with a single replyAppendf function to
improve buffer management and prevent overflow. This change enhances
the readability and maintainability of the code while ensuring that
the reply buffer is handled safely across various MQTT status and
diagnostic replies.
2026-07-20 10:00:48 -07:00
agessaman ebca9bdd3e feat(mqtt): add mesh-chaun14 and wcmesh presets
Support "{pubkey}" username sentinel so brokers can auth with the
device public key without enlarging the prefs username field.
2026-07-19 23:45:28 -07:00
agessaman de320bc4df feat(mqtt): add neighbors publish path to MQTTBridge
Port the periodic-neighbors publication from mqtt-bridge-implementation-flex,
adapted to this branch's structure:

- Add MQTT_PUBLICATION_NEIGHBORS ("neighbors") to the pure MQTTTopicRouter
  instead of flex's messageTypeSuffix() helper (this branch already routes
  every publication type through mqttBuildPublicationTopic()). Neighbors is a
  MeshCore/custom publication type, so it resolves to
  meshcore/{iata}/{device}/neighbors and honors custom templates.
- Deliberately do NOT port flex's "all message types to MeshRank" change:
  this branch documents and host-tests a packets-only MeshRank contract
  (MQTTPresets.h, MQTT_IMPLEMENTATION.md, MeshRankContractIsPacketsOnly). So
  neighbors follows status/raw and is rejected on MeshRank slots.
- WITH_MQTT_NEIGHBORS guard (PSRAM + MAX_NEIGHBOURS) gates all new surface.
- MSG_NEIGHBORS message type + enum-drift static_assert.
- Persistent ~10KB PSRAM neighbors buffer allocated/freed via the existing
  MQTTRuntimeBufferLifecycle path (allocate/release), not the ctor as flex did.
- Core1->Core0 handoff: requestPublishNeighbors() (mesh) fills the buffer with
  a release store; the MQTT task consumes it with an acquire load, publishes
  via publishNeighbors() (QoS1, retain = preset->allow_retain, custom=false),
  and clears the pending flag. A second snapshot is dropped while one is
  in flight.
- setNeighborsSchedule()/NeighborsPhase let the mesh report the timer summary;
  formatMqttStatusReply() gains a "nbr: <when>/<last>" field via formatDuration.

Also fix the on-connect status publish (publishStatusToSlot) to honor
preset->allow_retain instead of hardcoding retain=true, matching the periodic
publishStatus() path. Brokers with allow_retain=false (e.g. the waev MeshCore
preset) reject retained publishes, so the on-connect status was being dropped
there. This is flex followup 028a5dca, reconciled to this branch's custom-slot
default of non-retained.

Extends the host topic-router test to cover the neighbors type across all
routes and freezes the new enum value. Bridge itself is on-target only.
2026-07-19 22:39:41 -07:00
agessaman 0e7a07afff Merge remote-tracking branch 'fork/phase6/webconfig-batch-seam' into HEAD
# Conflicts:
#	STABILITY_TESTABILITY_HANDOFF.md
2026-07-19 11:10:17 -07:00
agessaman 4a6fe4e2be fix(mqtt): use ASCII hyphens in serial log strings
UTF-8 em-dashes (U+2014) inside MQTT_DEBUG_PRINTLN / CLI reply strings
render as mojibake ("aEUR"-style, e.g. cooperative stop garbles) on
consoles that don't decode UTF-8. Replace the em-dashes in printed
strings with ASCII '-'. Comments are intentionally left unchanged (they
never reach the console; rewriting them would be needless churn in a
merge-sensitive file).

Verified on V3: the stop message now emits pure ASCII -- zero non-ASCII
bytes in the boot+stop console capture.
2026-07-19 08:02:29 -07:00
agessaman 7767760fae fix(mqtt): slot-count-aware cooperative-stop timeout
Phase 0 hardware characterization showed real mbedTLS/wss client teardown
takes ~5-6s per connected slot, sequentially, so the flat 8s
MQTT_STOP_TIMEOUT_MS force-timed-out healthy multi-slot stops (2-slot
non-PSRAM ~11-12s, 3-slot PSRAM ~16s), which sets the dirty latch and
makes the OTA barrier withhold flashing -- multi-slot nodes could never
ota update.

Replace the flat constant with a slot-scaled budget computed per stop in
end(): 5s base + 8s per enabled slot (~1.5-2x headroom over measured),
applied via new MQTTLifecycle::Coordinator::setStopTimeoutMs() before
requestStop(). Headroom is nearly free: end() returns as soon as the task
acks (checked before the timeout ticks), so a larger bound only lengthens
the wait before force-killing a genuinely wedged task.

Hardware-verified on V3: a 2-slot stop that force-timed-out at 8s pre-fix
now logs "timeout 21000 ms" and acks clean in ~11.7s. Native suite +
both observer firmware builds green.
2026-07-19 07:34:27 -07:00
agessaman b0cf29fb33 test(mqtt): extract + test remaining inline MQTT decision points (Phase 6)
Close the host-testable gaps named in Phase 6 of STABILITY_TESTABILITY_HANDOFF.md
by moving the last inline decision logic into the pure, host-tested policy seams:

- WiFi STA reconnect backoff: extract the inline ladder + wrap-safe timing from
  handleWiFiConnection() into MQTTConnectionPolicy::{wifiReconnectBackoffMs,
  wifiReconnectDue,nextWifiBackoffAttempt}. Behavior-preserving (elapsedMs is the
  wrap-safe form of the old ULONG_MAX branch); ladder/clamp/attempt-cap unchanged.
- Publication outcome pairing: name the (packet, raw) -> delivered contract as
  MQTTPacketQueuePolicy::queuedPacketPublished() and wire both queue-drain sites;
  partial success = completed, not retried.
- Freeze MQTTPublicationType enum values in a test (the bridge-side MQTTMessageType
  alignment is already enforced by a compile-time static_assert).

Adds host tests for all three (exact boundaries + millis() rollover). Native suite
green (13 dirs); non-PSRAM observer firmware smoke build compiles.

WebConfig batch/reboot/stop state-machine extraction and queue-orchestration
coverage remain open (tracked in the Phase 6 status).
2026-07-19 00:50:15 -07:00
agessaman 2e1a1410ce feat(mqtt): cooperative MQTT shutdown + OTA teardown barrier (Phase 5)
Wire the Phase 4 MQTTLifecycle state machine into MQTTBridge to replace the
blind vTaskDelete teardown that could kill the MQTT task mid-mbedTLS and then
free client buffers on a corrupted heap (the observed OTA teardown panic).

- MQTTBridge owns a MQTTLifecycle::Coordinator driven only by the loop task
  (Core 1) from begin()/end(); a nested LifecycleOps binds the host-tested Ops
  spec to FreeRTOS/PsychicMqttClient.
- end() requests a cooperative stop; the MQTT task (Core 0) tears down its own
  clients where the mbedTLS contexts live, acks via _stop_acked, and
  self-terminates. end() waits (bounded) for the ack, then frees queue/buffers.
- Bounded stop timeout -> reviewed fallback (force kill + Core-1 teardown) sets
  a dirty latch that withholds OTA flashing.
- begin() gains an idempotent double-call guard and syncs the Coordinator to
  Running.
- OTA teardown barrier: simple_repeater's deferred flash aborts/resumes unless
  end() reported a clean stop (canFlashAfterStop()).

Scope: minimal cooperative-shutdown unit. The volatile NTP/reconfigure handshake
replacement and the plain-data snapshot / consumer repointing (MQTT_OWNERSHIP.md
sections 1-3) are deferred. MQTT_STOP_TIMEOUT_MS is a Phase-0 placeholder pending
on-hardware characterization.

Native suite green (incl. test_mqtt_lifecycle); both observer firmware smoke
builds compile. Not yet hardware-validated (Phase 7 gate).
2026-07-19 00:25:37 -07:00
agessaman b5deaf93fc fix(mqtt): restore PSRAM buffers after restart 2026-07-18 16:41:27 -07:00