From 6b3c543cbae04d37a14a3a8a273186ff471fb10f Mon Sep 17 00:00:00 2001 From: agessaman Date: Sun, 7 Jun 2026 11:24:39 -0700 Subject: [PATCH 01/87] Add MQTT neighbors functionality for PSRAM builds - Introduced commands for periodic neighbors publishing: `get mqtt.neighbors`, `set mqtt.neighbors on|off`, `get mqtt.neighbors.interval`, and `set mqtt.neighbors.interval `. - Updated MQTT topics to include a new neighbors topic for cached zero-hop repeater neighbors. - Added JSON message format for neighbors, including fields for SNR, last-heard age, and status. - Enhanced CLI documentation to reflect new commands and settings. - Implemented neighbor discovery logic in MyMesh class, including handling responses and publishing neighbor data. - Adjusted MQTTBridge to manage neighbors JSON buffer and publish logic. This update enhances the MQTT implementation by allowing devices to share neighbor information, improving network awareness and connectivity. --- MQTT_IMPLEMENTATION.md | 34 +++- docs/cli_commands.md | 26 +++ examples/simple_repeater/MyMesh.cpp | 250 ++++++++++++++++++++++++++++ examples/simple_repeater/MyMesh.h | 30 ++++ src/helpers/CommonCLI.h | 6 + src/helpers/CommonCLI_Observer.cpp | 31 ++++ src/helpers/MQTTDefaults.h | 2 + src/helpers/MQTTMessageBuilder.cpp | 35 ++++ src/helpers/MQTTMessageBuilder.h | 23 +++ src/helpers/bridges/MQTTBridge.cpp | 74 +++++++- src/helpers/bridges/MQTTBridge.h | 19 ++- 11 files changed, 525 insertions(+), 5 deletions(-) diff --git a/MQTT_IMPLEMENTATION.md b/MQTT_IMPLEMENTATION.md index 301b281a..cb1d0153 100644 --- a/MQTT_IMPLEMENTATION.md +++ b/MQTT_IMPLEMENTATION.md @@ -396,6 +396,8 @@ These settings apply across all MQTT slots: - `get mqtt.rx` - Get RX packet uplinking setting (on/off) - `get mqtt.tx` - Get TX packet uplinking setting (on/off/advert) - `get mqtt.interval` - Get status publish interval +- `get mqtt.neighbors` - Get periodic neighbors publish (PSRAM builds: on/off) +- `get mqtt.neighbors.interval` - Get neighbors publish interval in hours - `get mqtt.ntp` - Get effective NTP server hostname - `get mqtt.ntp.diag` - Probe every configured NTP server for connectivity (does not change the clock; serial console shows each server's reported time, LoRa shows a compact ` ok|fail` list) - `get mqtt.owner` - Get owner public key (serial console only) @@ -413,6 +415,8 @@ These settings apply across all MQTT slots: - `advert` - Uplink only this node's own advert packets (self-originated) - `off` - Disable TX packet uplinking - `set mqtt.interval ` - Set status publish interval (1-60 minutes) +- `set mqtt.neighbors on|off` - Enable/disable periodic neighbors/scopes publish (PSRAM builds only) +- `set mqtt.neighbors.interval ` - Set neighbors publish interval (12-8760 hours, default 24) - `set mqtt.ntp ` - Set custom NTP server (validated with immediate sync); `none` reverts to default - `set mqtt.owner <64-hex-char-public-key>` - Set owner public key - `set mqtt.email ` - Set owner email address @@ -507,7 +511,7 @@ The CLI commands are organized into two levels: ## MQTT Topics -The bridge publishes to three main topics with the following structure: +The bridge publishes to four main topics with the following structure: ### Status Topic: `meshcore/{IATA}/{DEVICE_PUBLIC_KEY}/status` Device connection status and metadata (retained messages). @@ -518,6 +522,9 @@ Full packet data with RF characteristics and metadata. ### Raw Topic: `meshcore/{IATA}/{DEVICE_PUBLIC_KEY}/raw` Minimal raw packet data for map integration. +### Neighbors Topic: `meshcore/{IATA}/{DEVICE_PUBLIC_KEY}/neighbors` +Cached zero-hop repeater neighbors with SNR, last-heard age, and flood-allowed scopes (retained, QoS 1). Published on `discover.scopes` or periodically when `mqtt.neighbors` is enabled (PSRAM observer builds only). + **Note**: `{DEVICE_PUBLIC_KEY}` is the device's public key in hexadecimal format (64 characters). ## JSON Message Formats @@ -596,6 +603,31 @@ Minimal raw packet data for map integration. } ``` +### Neighbors Message (PSRAM observer builds) +```json +{ + "timestamp": "2026-06-07T12:00:00.000000+00:00", + "origin": "MQTT Observer", + "origin_id": "DEVICE_PUBLIC_KEY", + "self": { + "scopes": "Europe,UK,France" + }, + "neighbors": [ + { + "pubkey": "NEIGHBOR_PUBLIC_KEY", + "snr": 8.5, + "heard_secs_ago": 120, + "scopes": "*,Europe", + "status": "responded" + } + ] +} +``` + +**Notes:** +- `status` per neighbor: `responded`, `timeout`, or `send_failed` (scope query only; `self` has no status field). +- Trigger with `discover.scopes` or enable periodic publish via `set mqtt.neighbors on`. + ## Key Features ### Slot-Based Preset System diff --git a/docs/cli_commands.md b/docs/cli_commands.md index b0badbcd..def7da17 100644 --- a/docs/cli_commands.md +++ b/docs/cli_commands.md @@ -113,6 +113,17 @@ This document provides an overview of CLI commands that can be sent to MeshCore --- +### Discover neighbor scopes and publish to MQTT (PSRAM observer builds) + +**Usage:** +- `discover.scopes` + +Queries cached zero-hop repeater neighbors for flood-allowed region scopes (`ANON_REQ_TYPE_REGIONS`), then publishes a retained JSON snapshot to `meshcore/{IATA}/{device}/neighbors`. Includes this node's own scopes in the `self` object. + +**Note:** Requires `WITH_MQTT_BRIDGE`, `MAX_NEIGHBOURS`, and PSRAM (`BOARD_HAS_PSRAM`). Returns `Err - not supported (requires PSRAM)` on other builds. + +--- + ## Statistics ### Clear Stats @@ -1103,6 +1114,21 @@ Requires WiFi connected and the MQTT bridge running. --- +#### Periodic neighbors/scopes MQTT publish (PSRAM observer builds) + +**Usage:** +- `get mqtt.neighbors` +- `set mqtt.neighbors on|off` +- `get mqtt.neighbors.interval` +- `set mqtt.neighbors.interval ` + +**Parameters:** +- `hours`: Interval between automatic `discover.scopes` runs (minimum **12**, default **24**) + +**Note:** Requires PSRAM. Publishes to `meshcore/{IATA}/{device}/neighbors` (retained, QoS 1). Use `discover.scopes` for a one-shot publish. + +--- + #### View or change the speed of the bridge (RS-232 only) **Usage:** - `get bridge.baud` diff --git a/examples/simple_repeater/MyMesh.cpp b/examples/simple_repeater/MyMesh.cpp index 88a44fb5..72810b79 100644 --- a/examples/simple_repeater/MyMesh.cpp +++ b/examples/simple_repeater/MyMesh.cpp @@ -666,6 +666,16 @@ void MyMesh::onAnonDataRecv(mesh::Packet *packet, const uint8_t *secret, const m int MyMesh::searchPeersByHash(const uint8_t *hash) { int n = 0; +#if defined(WITH_MQTT_NEIGHBORS) + if (neighbor_discover_active) { + for (int i = 0; i < neighbor_discover_count && n < MAX_CLIENTS; i++) { + auto& nb = neighbours[neighbor_discover[i].neighbour_idx]; + if (nb.heard_timestamp > 0 && nb.id.isHashMatch(hash)) { + matching_peer_indexes[n++] = NEIGHBOR_DISCOVER_PEER_BASE + i; + } + } + } +#endif for (int i = 0; i < acl.getNumClients(); i++) { if (acl.getClientByIdx(i)->id.isHashMatch(hash)) { matching_peer_indexes[n++] = i; // store the INDEXES of matching contacts (for subsequent 'peer' methods) @@ -676,6 +686,15 @@ int MyMesh::searchPeersByHash(const uint8_t *hash) { void MyMesh::getPeerSharedSecret(uint8_t *dest_secret, int peer_idx) { int i = matching_peer_indexes[peer_idx]; +#if defined(WITH_MQTT_NEIGHBORS) + if (neighbor_discover_active && i >= NEIGHBOR_DISCOVER_PEER_BASE) { + int oi = i - NEIGHBOR_DISCOVER_PEER_BASE; + if (oi >= 0 && oi < neighbor_discover_count) { + self_id.calcSharedSecret(dest_secret, neighbours[neighbor_discover[oi].neighbour_idx].id); + return; + } + } +#endif if (i >= 0 && i < acl.getNumClients()) { // lookup pre-calculated shared_secret memcpy(dest_secret, acl.getClientByIdx(i)->shared_secret, PUB_KEY_SIZE); @@ -707,6 +726,15 @@ void MyMesh::onAdvertRecv(mesh::Packet *packet, const mesh::Identity &id, uint32 void MyMesh::onPeerDataRecv(mesh::Packet *packet, uint8_t type, int sender_idx, const uint8_t *secret, uint8_t *data, size_t len) { int i = matching_peer_indexes[sender_idx]; +#if defined(WITH_MQTT_NEIGHBORS) + if (neighbor_discover_active && i >= NEIGHBOR_DISCOVER_PEER_BASE) { + int oi = i - NEIGHBOR_DISCOVER_PEER_BASE; + if (type == PAYLOAD_TYPE_RESPONSE && oi >= 0 && oi < neighbor_discover_count) { + handleNeighborDiscoverResponse(oi, data, len); + } + return; + } +#endif if (i < 0 || i >= acl.getNumClients()) { // get from our known_clients table (sender SHOULD already be known in this context) MESH_DEBUG_PRINTLN("onPeerDataRecv: invalid peer idx: %d", i); return; @@ -986,6 +1014,14 @@ MyMesh::MyMesh(mesh::MainBoard &board, mesh::Radio &radio, mesh::MillisecondCloc pending_discover_tag = 0; pending_discover_until = 0; +#if defined(WITH_MQTT_NEIGHBORS) + neighbor_discover_count = 0; + neighbor_discover_active = false; + neighbor_discover_until = 0; + next_neighbors_publish = 0; + self_scopes_buf[0] = 0; +#endif + memset(default_scope.key, 0, sizeof(default_scope.key)); } @@ -1390,6 +1426,19 @@ void MyMesh::handleCommand(uint32_t sender_timestamp, char *command, char *reply sendNodeDiscoverReq(); strcpy(reply, "OK - Discover sent"); } +#if defined(WITH_MQTT_NEIGHBORS) + } else if (memcmp(command, "discover.scopes", 15) == 0) { + const char* sub = command + 15; + while (*sub == ' ') sub++; + if (*sub != 0) { + strcpy(reply, "Err - discover.scopes has no options"); + } else if (!startNeighborDiscover(reply)) { + // reply already set + } +#elif defined(WITH_MQTT_BRIDGE) + } else if (memcmp(command, "discover.scopes", 15) == 0) { + strcpy(reply, "Err - not supported (requires PSRAM)"); +#endif } else{ _cli.handleCommand(sender_timestamp, command, reply); // common CLI commands } @@ -1462,6 +1511,20 @@ void MyMesh::loop() { _alerter.onLoop(now); #endif +#if defined(WITH_MQTT_NEIGHBORS) + if (neighbor_discover_active) { + loopNeighborDiscover(); + } else if (_cli.getObserverPrefs()->mqtt_neighbors_enabled && bridge && bridge->isRunning()) { + if (next_neighbors_publish == 0 || + (next_neighbors_publish != 0 && millisHasNowPassed(next_neighbors_publish))) { + char tmp_reply[80]; + if (startNeighborDiscover(tmp_reply)) { + MESH_DEBUG_PRINTLN("%s", tmp_reply); + } + } + } +#endif + #ifdef WITH_SNMP // Push radio stats to SNMP agent every 2 seconds if (_snmp_agent.isRunning()) { @@ -1482,6 +1545,193 @@ void MyMesh::loop() { #endif } +#if defined(WITH_MQTT_NEIGHBORS) + +#include "helpers/MQTTMessageBuilder.h" +#if defined(ESP_PLATFORM) +#include +#endif + +void MyMesh::getLocalScopes(char* buf, size_t len) { + if (!buf || len == 0) return; + buf[0] = 0; + region_map.exportNamesTo(buf, (int)len, REGION_DENY_FLOOD); +} + +bool MyMesh::sendAnonRegionsReq(const mesh::Identity& target, uint32_t& tag) { + uint8_t secret[PUB_KEY_SIZE]; + self_id.calcSharedSecret(secret, target); + + tag = getRTCClock()->getCurrentTimeUnique(); + uint8_t inner[6]; + memcpy(inner, &tag, 4); + inner[4] = 0x01; // ANON_REQ_TYPE_REGIONS + inner[5] = 0x00; // zero-hop reply path + + mesh::Packet* pkt = createAnonDatagram(PAYLOAD_TYPE_ANON_REQ, self_id, target, secret, inner, sizeof(inner)); + if (!pkt) return false; + sendDirect(pkt, NULL, 0, 0); + return true; +} + +void MyMesh::handleNeighborDiscoverResponse(int overlay_idx, const uint8_t* data, size_t len) { + if (overlay_idx < 0 || overlay_idx >= neighbor_discover_count) return; + NeighborDiscoverEntry& entry = neighbor_discover[overlay_idx]; + if (entry.status != ND_PENDING || len < 8) return; + + uint32_t tag; + memcpy(&tag, data, 4); + if (tag != entry.tag) return; + + size_t scope_len = len - 8; + if (scope_len >= sizeof(entry.scopes)) { + scope_len = sizeof(entry.scopes) - 1; + } + memcpy(entry.scopes, &data[8], scope_len); + entry.scopes[scope_len] = 0; + entry.status = ND_RESPONDED; +} + +void MyMesh::finishNeighborDiscover() { + getLocalScopes(self_scopes_buf, sizeof(self_scopes_buf)); + + char self_pubkey_hex[65]; + mesh::Utils::toHex(self_pubkey_hex, self_id.pub_key, PUB_KEY_SIZE); + + char origin[32]; + MQTTBridge::getEffectiveMqttOrigin(&_prefs, _cli.getObserverPrefs(), origin, sizeof(origin)); + + char timestamp[40]; + MQTTMessageBuilder::formatIsoTimestampForMqtt(getRTCClock()->getCurrentTime(), 0, nullptr, timestamp, sizeof(timestamp)); + + char pubkey_hex[MAX_NEIGHBOURS][65]; + MQTTMessageBuilder::NeighborsMessageEntry entries[MAX_NEIGHBOURS]; + uint32_t now_secs = getRTCClock()->getCurrentTime(); + + for (int i = 0; i < neighbor_discover_count; i++) { + auto& nb = neighbours[neighbor_discover[i].neighbour_idx]; + mesh::Utils::toHex(pubkey_hex[i], nb.id.pub_key, PUB_KEY_SIZE); + entries[i].pubkey_hex = pubkey_hex[i]; + entries[i].snr = nb.snr / 4.0f; + entries[i].heard_secs_ago = (nb.heard_timestamp > 0 && now_secs >= nb.heard_timestamp) + ? (now_secs - nb.heard_timestamp) : 0; + entries[i].scopes = neighbor_discover[i].scopes; + switch (neighbor_discover[i].status) { + case ND_RESPONDED: entries[i].status = "responded"; break; + case ND_SEND_FAILED: entries[i].status = "send_failed"; break; + default: entries[i].status = "timeout"; break; + } + } + +#if defined(ESP_PLATFORM) + char* json_buf = (char*)heap_caps_malloc(MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE, MALLOC_CAP_SPIRAM); +#else + char* json_buf = (char*)malloc(MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE); +#endif + if (!json_buf) { + neighbor_discover_active = false; + neighbor_discover_count = 0; + if (_cli.getObserverPrefs()->mqtt_neighbors_enabled) { + next_neighbors_publish = futureMillis(_cli.getObserverPrefs()->mqtt_neighbors_interval); + } + return; + } + + DynamicJsonDocument doc(MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE); + int json_len = MQTTMessageBuilder::buildNeighborsMessage( + doc, origin, self_pubkey_hex, timestamp, self_scopes_buf, + entries, neighbor_discover_count, + json_buf, MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE); + + if (json_len > 0 && bridge) { + bridge->requestPublishNeighbors(json_buf, (size_t)json_len); + } + +#if defined(ESP_PLATFORM) + heap_caps_free(json_buf); +#else + free(json_buf); +#endif + + neighbor_discover_active = false; + neighbor_discover_count = 0; + if (_cli.getObserverPrefs()->mqtt_neighbors_enabled) { + next_neighbors_publish = futureMillis(_cli.getObserverPrefs()->mqtt_neighbors_interval); + } +} + +void MyMesh::loopNeighborDiscover() { + if (!neighbor_discover_active) return; + + bool all_done = true; + for (int i = 0; i < neighbor_discover_count; i++) { + if (neighbor_discover[i].status == ND_PENDING) { + all_done = false; + break; + } + } + + if (!all_done && !millisHasNowPassed(neighbor_discover_until)) { + return; + } + + if (!all_done) { + for (int i = 0; i < neighbor_discover_count; i++) { + if (neighbor_discover[i].status == ND_PENDING) { + neighbor_discover[i].status = ND_TIMEOUT; + } + } + } + finishNeighborDiscover(); +} + +bool MyMesh::startNeighborDiscover(char* reply) { + if (neighbor_discover_active) { + strcpy(reply, "Err - neighbor discover already active"); + return false; + } +#if defined(ESP_PLATFORM) + if (!psramFound()) { + strcpy(reply, "Err - PSRAM not available"); + return false; + } +#endif + if (!bridge || !bridge->isRunning()) { + strcpy(reply, "Err - MQTT bridge not running"); + return false; + } + + getLocalScopes(self_scopes_buf, sizeof(self_scopes_buf)); + neighbor_discover_count = 0; + for (int i = 0; i < MAX_NEIGHBOURS; i++) { + if (neighbours[i].heard_timestamp > 0) { + neighbor_discover[neighbor_discover_count].neighbour_idx = (uint8_t)i; + neighbor_discover[neighbor_discover_count].scopes[0] = 0; + neighbor_discover[neighbor_discover_count].status = ND_PENDING; + uint32_t tag; + if (sendAnonRegionsReq(neighbours[i].id, tag)) { + neighbor_discover[neighbor_discover_count].tag = tag; + } else { + neighbor_discover[neighbor_discover_count].status = ND_SEND_FAILED; + } + neighbor_discover_count++; + } + } + + neighbor_discover_active = true; + neighbor_discover_until = futureMillis(NEIGHBOR_DISCOVER_TIMEOUT_MS); + + if (neighbor_discover_count == 0) { + finishNeighborDiscover(); + strcpy(reply, "OK - neighbor discover started (0 neighbors, self only)"); + } else { + sprintf(reply, "OK - neighbor discover started (%u neighbors)", (unsigned)neighbor_discover_count); + } + return true; +} + +#endif // WITH_MQTT_NEIGHBORS + // To check if there is pending work bool MyMesh::hasPendingWork() const { #if defined(WITH_BRIDGE) diff --git a/examples/simple_repeater/MyMesh.h b/examples/simple_repeater/MyMesh.h index d2a45b81..c30c3120 100644 --- a/examples/simple_repeater/MyMesh.h +++ b/examples/simple_repeater/MyMesh.h @@ -136,6 +136,36 @@ class MyMesh : public mesh::Mesh, public CommonCLICallbacks { AlertReporter _alerter; #endif +#if defined(WITH_MQTT_NEIGHBORS) + enum NeighborDiscoverStatus : uint8_t { + ND_PENDING = 1, + ND_RESPONDED = 2, + ND_TIMEOUT = 3, + ND_SEND_FAILED = 4, + }; + struct NeighborDiscoverEntry { + uint8_t neighbour_idx; + uint32_t tag; + char scopes[96]; + uint8_t status; + }; + NeighborDiscoverEntry neighbor_discover[MAX_NEIGHBOURS]; + uint8_t neighbor_discover_count; + bool neighbor_discover_active; + unsigned long neighbor_discover_until; + unsigned long next_neighbors_publish; + char self_scopes_buf[96]; + + bool sendAnonRegionsReq(const mesh::Identity& target, uint32_t& tag); + bool startNeighborDiscover(char* reply); + void loopNeighborDiscover(); + void finishNeighborDiscover(); + void handleNeighborDiscoverResponse(int overlay_idx, const uint8_t* data, size_t len); + void getLocalScopes(char* buf, size_t len); + static const int NEIGHBOR_DISCOVER_PEER_BASE = 1000; + static const unsigned long NEIGHBOR_DISCOVER_TIMEOUT_MS = 30000; +#endif + void putNeighbour(const mesh::Identity& id, uint32_t timestamp, float snr); uint8_t handleLoginReq(const mesh::Identity& sender, const uint8_t* secret, uint32_t sender_timestamp, const uint8_t* data, bool is_flood); uint8_t handleAnonRegionsReq(const mesh::Identity& sender, uint32_t sender_timestamp, const uint8_t* data); diff --git a/src/helpers/CommonCLI.h b/src/helpers/CommonCLI.h index ef4a797c..d470e7b1 100644 --- a/src/helpers/CommonCLI.h +++ b/src/helpers/CommonCLI.h @@ -156,6 +156,12 @@ struct MQTTPrefs { uint16_t alert_min_interval_min; // min minutes between same-fault alerts, default 60 char alert_hashtag[24]; // readback for `get alert.hashtag` char alert_region[31]; // optional region override; empty = default_scope + + // Neighbors publishing (PSRAM boards only). Appended at the end of the observer + // tail so a shorter /mqtt_prefs payload from earlier firmware still loads with + // these defaulting off (see applyMQTTDefaults); keeps the format at VERSION 1. + uint8_t mqtt_neighbors_enabled; // Periodic neighbors/scopes MQTT publish (PSRAM boards only) + uint32_t mqtt_neighbors_interval; // Neighbors publish interval (ms), default 24h, min 12h }; // /mqtt_prefs is written with an 8-byte header so the format is self-describing. diff --git a/src/helpers/CommonCLI_Observer.cpp b/src/helpers/CommonCLI_Observer.cpp index 22de73cf..1166553e 100644 --- a/src/helpers/CommonCLI_Observer.cpp +++ b/src/helpers/CommonCLI_Observer.cpp @@ -246,6 +246,26 @@ bool CommonCLI::handleObserverSetCmd(uint32_t sender_timestamp, const char* conf } else { strcpy(reply, "Error: interval must be between 1-60 minutes"); } +#if defined(BOARD_HAS_PSRAM) && defined(MAX_NEIGHBOURS) && MAX_NEIGHBOURS > 0 + } else if (memcmp(config, "mqtt.neighbors ", 14) == 0) { + _mqtt_prefs.mqtt_neighbors_enabled = memcmp(&config[14], "on", 2) == 0; + savePrefs(); + strcpy(reply, "OK"); + } else if (memcmp(config, "mqtt.neighbors.interval ", 24) == 0) { + uint32_t hours = _atoi(&config[24]); + if (hours >= 12 && hours <= 8760) { + _mqtt_prefs.mqtt_neighbors_interval = hours * 3600000UL; + savePrefs(); + sprintf(reply, "OK - neighbors interval set to %u hours (%lu ms)", hours, + (unsigned long)_mqtt_prefs.mqtt_neighbors_interval); + } else { + strcpy(reply, "Error: neighbors interval must be between 12-8760 hours"); + } +#elif defined(WITH_MQTT_BRIDGE) + } else if (memcmp(config, "mqtt.neighbors ", 14) == 0 || + memcmp(config, "mqtt.neighbors.interval ", 24) == 0) { + strcpy(reply, "Err - not supported (requires PSRAM)"); +#endif } else if (memcmp(config, "mqtt.ntp ", 9) == 0) { const char* host = &config[9]; while (*host == ' ') host++; @@ -709,6 +729,17 @@ bool CommonCLI::handleObserverGetCmd(uint32_t sender_timestamp, const char* conf } else if (memcmp(config, "mqtt.interval", 13) == 0) { uint32_t minutes = (_mqtt_prefs.mqtt_status_interval + 29999) / 60000; sprintf(reply, "> %u minutes (%lu ms)", minutes, (unsigned long)_mqtt_prefs.mqtt_status_interval); +#if defined(BOARD_HAS_PSRAM) && defined(MAX_NEIGHBOURS) && MAX_NEIGHBOURS > 0 + } else if (memcmp(config, "mqtt.neighbors.interval", 23) == 0) { + uint32_t hours = (_mqtt_prefs.mqtt_neighbors_interval + 3599999) / 3600000; + sprintf(reply, "> %u hours (%lu ms)", hours, (unsigned long)_mqtt_prefs.mqtt_neighbors_interval); + } else if (memcmp(config, "mqtt.neighbors", 14) == 0) { + sprintf(reply, "> %s", _mqtt_prefs.mqtt_neighbors_enabled ? "on" : "off"); +#elif defined(WITH_MQTT_BRIDGE) + } else if (memcmp(config, "mqtt.neighbors", 14) == 0 || + memcmp(config, "mqtt.neighbors.interval", 23) == 0) { + strcpy(reply, "Err - not supported (requires PSRAM)"); +#endif } else if (memcmp(config, "mqtt.ntp.diag", 13) == 0 && (config[13] == '\0' || config[13] == ' ')) { #ifdef ESP_PLATFORM // Connectivity probe across all configured NTP servers; never updates the clock. diff --git a/src/helpers/MQTTDefaults.h b/src/helpers/MQTTDefaults.h index f3c117eb..2617ef7c 100644 --- a/src/helpers/MQTTDefaults.h +++ b/src/helpers/MQTTDefaults.h @@ -68,6 +68,8 @@ static inline void applyMQTTDefaults(MQTTPrefs* prefs) { prefs->mqtt_tx_enabled = 2; prefs->mqtt_rx_enabled = 1; prefs->mqtt_status_interval = 300000; + prefs->mqtt_neighbors_enabled = 0; + prefs->mqtt_neighbors_interval = 24UL * 3600UL * 1000UL; prefs->wifi_power_save = 1; mqttDefaultSlotPreset(prefs->mqtt_slot_preset[0], sizeof(prefs->mqtt_slot_preset[0]), diff --git a/src/helpers/MQTTMessageBuilder.cpp b/src/helpers/MQTTMessageBuilder.cpp index a29b3d96..75be1613 100644 --- a/src/helpers/MQTTMessageBuilder.cpp +++ b/src/helpers/MQTTMessageBuilder.cpp @@ -227,6 +227,41 @@ int MQTTMessageBuilder::buildRawMessage( return (len > 0 && len < buffer_size) ? len : 0; } +int MQTTMessageBuilder::buildNeighborsMessage( + JsonDocument& doc, + const char* origin, + const char* origin_id, + const char* timestamp, + const char* self_scopes, + const NeighborsMessageEntry* neighbors, + int neighbor_count, + char* buffer, + size_t buffer_size +) { + doc.clear(); + JsonObject root = doc.to(); + + root["timestamp"] = timestamp; + root["origin"] = origin; + root["origin_id"] = origin_id; + + JsonObject self = root.createNestedObject("self"); + self["scopes"] = self_scopes ? self_scopes : ""; + + JsonArray arr = root.createNestedArray("neighbors"); + for (int i = 0; i < neighbor_count; i++) { + JsonObject nb = arr.createNestedObject(); + nb["pubkey"] = neighbors[i].pubkey_hex; + nb["snr"] = neighbors[i].snr; + nb["heard_secs_ago"] = neighbors[i].heard_secs_ago; + nb["scopes"] = neighbors[i].scopes ? neighbors[i].scopes : ""; + nb["status"] = neighbors[i].status; + } + + size_t len = serializeJson(root, buffer, buffer_size); + return (len > 0 && len < buffer_size) ? len : 0; +} + int MQTTMessageBuilder::buildPacketJSON( JsonDocument& doc, mesh::Packet* packet, diff --git a/src/helpers/MQTTMessageBuilder.h b/src/helpers/MQTTMessageBuilder.h index 8e3a6d45..c230741f 100644 --- a/src/helpers/MQTTMessageBuilder.h +++ b/src/helpers/MQTTMessageBuilder.h @@ -153,6 +153,29 @@ public: size_t buffer_size ); + struct NeighborsMessageEntry { + const char* pubkey_hex; + float snr; + uint32_t heard_secs_ago; + const char* scopes; + const char* status; + }; + + /** + * Build neighbors table JSON for meshcore/{iata}/{device}/neighbors topic. + */ + static int buildNeighborsMessage( + JsonDocument& doc, + const char* origin, + const char* origin_id, + const char* timestamp, + const char* self_scopes, + const NeighborsMessageEntry* neighbors, + int neighbor_count, + char* buffer, + size_t buffer_size + ); + /** * Convert packet to JSON message * diff --git a/src/helpers/bridges/MQTTBridge.cpp b/src/helpers/bridges/MQTTBridge.cpp index eeae09d2..fe09024b 100644 --- a/src/helpers/bridges/MQTTBridge.cpp +++ b/src/helpers/bridges/MQTTBridge.cpp @@ -506,6 +506,11 @@ MQTTBridge::MQTTBridge(NodePrefs *prefs, MQTTPrefs *obs, mesh::PacketManager *mg _last_raw_data = (uint8_t*)psram_malloc(LAST_RAW_DATA_SIZE); _publish_json_buffer = (char*)psram_malloc(PUBLISH_JSON_BUFFER_SIZE); _status_json_buffer = (char*)psram_malloc(STATUS_JSON_BUFFER_SIZE); + #if defined(WITH_MQTT_NEIGHBORS) + _neighbors_json_buffer = (char*)psram_malloc(NEIGHBORS_JSON_BUFFER_SIZE); + _neighbors_publish_pending = false; + _neighbors_publish_len = 0; + #endif #else memset(_last_raw_data, 0, sizeof(_last_raw_data)); #endif @@ -804,6 +809,11 @@ void MQTTBridge::end() { psram_free(_last_raw_data); _last_raw_data = nullptr; psram_free(_publish_json_buffer); _publish_json_buffer = nullptr; psram_free(_status_json_buffer); _status_json_buffer = nullptr; + #if defined(WITH_MQTT_NEIGHBORS) + psram_free(_neighbors_json_buffer); _neighbors_json_buffer = nullptr; + _neighbors_publish_pending = false; + _neighbors_publish_len = 0; + #endif #endif // JSON documents are now StaticJsonDocument inline members — no heap allocation to free. @@ -1023,6 +1033,17 @@ void MQTTBridge::mqttTaskLoop() { // Process packet queue processPacketQueue(); +#if defined(WITH_MQTT_NEIGHBORS) + if (_neighbors_publish_pending) { + _neighbors_publish_pending = false; + if (publishNeighbors()) { + MQTT_DEBUG_PRINTLN("Neighbors published"); + } else { + MQTT_DEBUG_PRINTLN("Neighbors publish failed"); + } + } +#endif + #ifdef WITH_SNMP // SNMP agent loop — process incoming UDP requests if (_snmp_agent) { @@ -1820,7 +1841,10 @@ bool MQTTBridge::publishToAllSlots(const char* topic, const char* payload, bool // Presets use hardcoded topic logic; custom slots support user-defined templates. // --------------------------------------------------------------------------- bool MQTTBridge::substituteTopicTemplate(const char* tmpl, MQTTMessageType type, int slot_index, char* buf, size_t buf_size) { - const char* type_str = (type == MSG_STATUS) ? "status" : (type == MSG_PACKETS) ? "packets" : "raw"; + const char* type_str = (type == MSG_STATUS) ? "status" + : (type == MSG_PACKETS) ? "packets" + : (type == MSG_RAW) ? "raw" + : "neighbors"; const char* token = _obs->mqtt_slot_token[slot_index]; size_t out = 0; @@ -1878,7 +1902,10 @@ bool MQTTBridge::buildTopicForSlot(int index, MQTTMessageType type, char* topic_ } // MQTT_TOPIC_MESHCORE (default for all other presets) if (!isIATAValid()) return false; - const char* type_str = (type == MSG_STATUS) ? "status" : (type == MSG_PACKETS) ? "packets" : "raw"; + const char* type_str = (type == MSG_STATUS) ? "status" + : (type == MSG_PACKETS) ? "packets" + : (type == MSG_RAW) ? "raw" + : "neighbors"; snprintf(topic_buf, buf_size, "meshcore/%s/%s/%s", _iata, _device_id, type_str); return true; } @@ -1889,7 +1916,10 @@ bool MQTTBridge::buildTopicForSlot(int index, MQTTMessageType type, char* topic_ } // Default: meshcore format if (!isIATAValid()) return false; - const char* type_str = (type == MSG_STATUS) ? "status" : (type == MSG_PACKETS) ? "packets" : "raw"; + const char* type_str = (type == MSG_STATUS) ? "status" + : (type == MSG_PACKETS) ? "packets" + : (type == MSG_RAW) ? "raw" + : "neighbors"; snprintf(topic_buf, buf_size, "meshcore/%s/%s/%s", _iata, _device_id, type_str); return true; } @@ -2716,6 +2746,44 @@ bool MQTTBridge::publishStatus() { return false; } +#if defined(WITH_MQTT_NEIGHBORS) +void MQTTBridge::requestPublishNeighbors(const char* json, size_t len) { + if (!_neighbors_json_buffer || !json || len == 0) return; + if (len >= NEIGHBORS_JSON_BUFFER_SIZE) { + len = NEIGHBORS_JSON_BUFFER_SIZE - 1; + } + memcpy(_neighbors_json_buffer, json, len); + _neighbors_json_buffer[len] = '\0'; + _neighbors_publish_len = len; + _neighbors_publish_pending = true; +} + +bool MQTTBridge::publishNeighbors() { + if (!_neighbors_json_buffer || _neighbors_publish_len == 0) { + return false; + } + if (!_cached_has_connected_slots) { + return false; + } + + refreshOriginFromPrefs(); + + bool published = false; + char topic[128]; + for (int i = 0; i < RUNTIME_MQTT_SLOTS; i++) { + if (_slots[i].enabled && _slots[i].client && _slots[i].connected) { + if (buildTopicForSlot(i, MSG_NEIGHBORS, topic, sizeof(topic))) { + bool use_retain = _slots[i].preset ? _slots[i].preset->allow_retain : false; + if (publishToSlot(i, topic, _neighbors_json_buffer, use_retain, 1)) { + published = true; + } + } + } + } + return published; +} +#endif + bool MQTTBridge::publishPacket(mesh::Packet* packet, bool is_tx, const uint8_t* raw_data, int raw_len, float snr, float rssi) { diff --git a/src/helpers/bridges/MQTTBridge.h b/src/helpers/bridges/MQTTBridge.h index ee5b8b98..70de88ad 100644 --- a/src/helpers/bridges/MQTTBridge.h +++ b/src/helpers/bridges/MQTTBridge.h @@ -35,6 +35,10 @@ class MeshSNMPAgent; // Forward declaration #ifdef WITH_MQTT_BRIDGE +#if defined(BOARD_HAS_PSRAM) && defined(MAX_NEIGHBOURS) && MAX_NEIGHBOURS > 0 +#define WITH_MQTT_NEIGHBORS 1 +#endif + /** * @brief Bridge implementation using MQTT protocol for packet transport * @@ -257,6 +261,11 @@ private: // class object so these allocations don't interleave with large TLS buffers at startup. static const size_t PUBLISH_JSON_BUFFER_SIZE = 2048; static const size_t STATUS_JSON_BUFFER_SIZE = 768; + #if defined(WITH_MQTT_NEIGHBORS) + char* _neighbors_json_buffer; + size_t _neighbors_publish_len; + bool _neighbors_publish_pending; + #endif #if defined(BOARD_HAS_PSRAM) char* _publish_json_buffer; char* _status_json_buffer; @@ -318,7 +327,7 @@ private: mesh::MillisecondClock* _ms; // For uptime // Topic building - enum MQTTMessageType { MSG_STATUS, MSG_PACKETS, MSG_RAW }; + enum MQTTMessageType { MSG_STATUS, MSG_PACKETS, MSG_RAW, MSG_NEIGHBORS }; bool buildTopicForSlot(int index, MQTTMessageType type, char* topic_buf, size_t buf_size); bool substituteTopicTemplate(const char* tmpl, MQTTMessageType type, int slot_index, char* buf, size_t buf_size); @@ -349,6 +358,9 @@ private: void processPacketQueue(); bool publishStatus(); // Returns true if status was successfully published + #if defined(WITH_MQTT_NEIGHBORS) + bool publishNeighbors(); + #endif bool handleWiFiConnection(unsigned long now); // FreeRTOS task function (runs on Core 0) @@ -477,6 +489,11 @@ public: static void formatSlotDiagReply(char* buf, size_t bufsize, int slot_index); static uint8_t getLastWifiDisconnectReason(); static unsigned long getLastWifiDisconnectTime(); + + #if defined(WITH_MQTT_NEIGHBORS) + void requestPublishNeighbors(const char* json, size_t len); + static const size_t NEIGHBORS_JSON_BUFFER_SIZE = 10240; + #endif static const char* wifiReasonStr(uint8_t reason); static const char* tlsErrorStr(int32_t err); From 1454e4cac12cf35860cab6f08c9d01585bebf10a Mon Sep 17 00:00:00 2001 From: agessaman Date: Thu, 16 Jul 2026 17:28:33 -0700 Subject: [PATCH 02/87] fix(mqtt): update neighbors interval range in documentation and code Adjusted the valid range for the `mqtt.neighbors.interval` setting from 12-8760 hours to 12-336 hours to reflect new constraints. Updated related documentation and CLI commands to ensure consistency across the codebase. This change improves clarity and prevents potential misconfigurations in neighbor publishing intervals. --- .gitignore | 3 +- MQTT_IMPLEMENTATION.md | 2 +- docs/cli_commands.md | 2 +- examples/simple_repeater/MyMesh.cpp | 54 ++++++++++++++++++++++++++--- examples/simple_repeater/MyMesh.h | 2 +- src/helpers/CommonCLI.cpp | 10 ++++++ src/helpers/CommonCLI.h | 12 ++++++- src/helpers/CommonCLI_Observer.cpp | 11 +++--- src/helpers/MQTTDefaults.h | 2 +- src/helpers/MQTTMessageBuilder.cpp | 14 +++++++- src/helpers/bridges/MQTTBridge.cpp | 7 ++-- src/helpers/bridges/MQTTBridge.h | 5 ++- 12 files changed, 103 insertions(+), 21 deletions(-) diff --git a/.gitignore b/.gitignore index 29fa3674..bee82048 100644 --- a/.gitignore +++ b/.gitignore @@ -22,4 +22,5 @@ ssl_certs/cacert.pem platformio.local.ini .cursor/* .claude/* -.cursorrules \ No newline at end of file +.cursorrules +src/helpers/esp32/WebConfigHtml.h diff --git a/MQTT_IMPLEMENTATION.md b/MQTT_IMPLEMENTATION.md index cb1d0153..f52fed70 100644 --- a/MQTT_IMPLEMENTATION.md +++ b/MQTT_IMPLEMENTATION.md @@ -416,7 +416,7 @@ These settings apply across all MQTT slots: - `off` - Disable TX packet uplinking - `set mqtt.interval ` - Set status publish interval (1-60 minutes) - `set mqtt.neighbors on|off` - Enable/disable periodic neighbors/scopes publish (PSRAM builds only) -- `set mqtt.neighbors.interval ` - Set neighbors publish interval (12-8760 hours, default 24) +- `set mqtt.neighbors.interval ` - Set neighbors publish interval (12-336 hours, default 24) - `set mqtt.ntp ` - Set custom NTP server (validated with immediate sync); `none` reverts to default - `set mqtt.owner <64-hex-char-public-key>` - Set owner public key - `set mqtt.email ` - Set owner email address diff --git a/docs/cli_commands.md b/docs/cli_commands.md index def7da17..c740ce1f 100644 --- a/docs/cli_commands.md +++ b/docs/cli_commands.md @@ -1123,7 +1123,7 @@ Requires WiFi connected and the MQTT bridge running. - `set mqtt.neighbors.interval ` **Parameters:** -- `hours`: Interval between automatic `discover.scopes` runs (minimum **12**, default **24**) +- `hours`: Interval between automatic `discover.scopes` runs (**12-336**, default **24**) **Note:** Requires PSRAM. Publishes to `meshcore/{IATA}/{device}/neighbors` (retained, QoS 1). Use `discover.scopes` for a one-shot publish. diff --git a/examples/simple_repeater/MyMesh.cpp b/examples/simple_repeater/MyMesh.cpp index 72810b79..1ca5adba 100644 --- a/examples/simple_repeater/MyMesh.cpp +++ b/examples/simple_repeater/MyMesh.cpp @@ -670,13 +670,17 @@ int MyMesh::searchPeersByHash(const uint8_t *hash) { if (neighbor_discover_active) { for (int i = 0; i < neighbor_discover_count && n < MAX_CLIENTS; i++) { auto& nb = neighbours[neighbor_discover[i].neighbour_idx]; + // ACL clients already have a matching peer entry and shared secret. Adding + // a second overlay entry would decrypt first and intercept their normal + // CLI/request traffic for the duration of discovery. + if (acl.getClient(nb.id.pub_key, PUB_KEY_SIZE) != nullptr) continue; if (nb.heard_timestamp > 0 && nb.id.isHashMatch(hash)) { matching_peer_indexes[n++] = NEIGHBOR_DISCOVER_PEER_BASE + i; } } } #endif - for (int i = 0; i < acl.getNumClients(); i++) { + for (int i = 0; i < acl.getNumClients() && n < MAX_CLIENTS; i++) { if (acl.getClientByIdx(i)->id.isHashMatch(hash)) { matching_peer_indexes[n++] = i; // store the INDEXES of matching contacts (for subsequent 'peer' methods) } @@ -741,6 +745,20 @@ void MyMesh::onPeerDataRecv(mesh::Packet *packet, uint8_t type, int sender_idx, } ClientInfo* client = acl.getClientByIdx(i); +#if defined(WITH_MQTT_NEIGHBORS) + // Neighbors that are already ACL clients are intentionally not added to the + // temporary peer overlay above. Consume only a matching discovery response + // here, then leave every other payload on the normal ACL path. + if (neighbor_discover_active && type == PAYLOAD_TYPE_RESPONSE) { + for (int oi = 0; oi < neighbor_discover_count; oi++) { + auto& nb = neighbours[neighbor_discover[oi].neighbour_idx]; + if (client->id.matches(nb.id) && handleNeighborDiscoverResponse(oi, data, len)) { + return; + } + } + } +#endif + if (type == PAYLOAD_TYPE_REQ) { // request (from a Known admin client!) uint32_t timestamp; memcpy(×tamp, data, 4); @@ -1574,14 +1592,14 @@ bool MyMesh::sendAnonRegionsReq(const mesh::Identity& target, uint32_t& tag) { return true; } -void MyMesh::handleNeighborDiscoverResponse(int overlay_idx, const uint8_t* data, size_t len) { - if (overlay_idx < 0 || overlay_idx >= neighbor_discover_count) return; +bool MyMesh::handleNeighborDiscoverResponse(int overlay_idx, const uint8_t* data, size_t len) { + if (overlay_idx < 0 || overlay_idx >= neighbor_discover_count) return false; NeighborDiscoverEntry& entry = neighbor_discover[overlay_idx]; - if (entry.status != ND_PENDING || len < 8) return; + if (entry.status != ND_PENDING || len < 8) return false; uint32_t tag; memcpy(&tag, data, 4); - if (tag != entry.tag) return; + if (tag != entry.tag) return false; size_t scope_len = len - 8; if (scope_len >= sizeof(entry.scopes)) { @@ -1590,6 +1608,19 @@ void MyMesh::handleNeighborDiscoverResponse(int overlay_idx, const uint8_t* data memcpy(entry.scopes, &data[8], scope_len); entry.scopes[scope_len] = 0; entry.status = ND_RESPONDED; + return true; +} + +static bool neighborPublishEntryComesBefore( + const MQTTMessageBuilder::NeighborsMessageEntry& lhs, + const MQTTMessageBuilder::NeighborsMessageEntry& rhs) { + if (lhs.heard_secs_ago != rhs.heard_secs_ago) { + return lhs.heard_secs_ago < rhs.heard_secs_ago; // newer first + } + if (lhs.snr != rhs.snr) { + return lhs.snr > rhs.snr; // stronger first when equally recent + } + return strcmp(lhs.pubkey_hex, rhs.pubkey_hex) < 0; } void MyMesh::finishNeighborDiscover() { @@ -1623,6 +1654,19 @@ void MyMesh::finishNeighborDiscover() { } } + // The JSON builder drops entries from the tail when the fixed MQTT buffer is + // full, so order the useful/recent neighbors first. Insertion sort is small, + // allocation-free, and sufficient for MAX_NEIGHBOURS (currently at most 50). + for (int i = 1; i < neighbor_discover_count; i++) { + MQTTMessageBuilder::NeighborsMessageEntry entry = entries[i]; + int j = i; + while (j > 0 && neighborPublishEntryComesBefore(entry, entries[j - 1])) { + entries[j] = entries[j - 1]; + j--; + } + entries[j] = entry; + } + #if defined(ESP_PLATFORM) char* json_buf = (char*)heap_caps_malloc(MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE, MALLOC_CAP_SPIRAM); #else diff --git a/examples/simple_repeater/MyMesh.h b/examples/simple_repeater/MyMesh.h index c30c3120..a2cfc162 100644 --- a/examples/simple_repeater/MyMesh.h +++ b/examples/simple_repeater/MyMesh.h @@ -160,7 +160,7 @@ class MyMesh : public mesh::Mesh, public CommonCLICallbacks { bool startNeighborDiscover(char* reply); void loopNeighborDiscover(); void finishNeighborDiscover(); - void handleNeighborDiscoverResponse(int overlay_idx, const uint8_t* data, size_t len); + bool handleNeighborDiscoverResponse(int overlay_idx, const uint8_t* data, size_t len); void getLocalScopes(char* buf, size_t len); static const int NEIGHBOR_DISCOVER_PEER_BASE = 1000; static const unsigned long NEIGHBOR_DISCOVER_TIMEOUT_MS = 30000; diff --git a/src/helpers/CommonCLI.cpp b/src/helpers/CommonCLI.cpp index b9ad2eea..b941d179 100644 --- a/src/helpers/CommonCLI.cpp +++ b/src/helpers/CommonCLI.cpp @@ -633,6 +633,16 @@ void CommonCLI::loadMQTTPrefs(FILESYSTEM* fs) { memcpy(_mqtt_prefs.alert_region, _legacy_tail.alert_region, sizeof(_mqtt_prefs.alert_region)); MESH_DEBUG_PRINTLN("MQTT: Migrated observer settings from legacy /com_prefs trailing block"); } + + // Keep persisted values inside the signed-delta millis() scheduling window. + // This also repairs any manually-written or experimental value from firmware + // that briefly accepted intervals longer than the supported two-week cap. + if (_mqtt_prefs.mqtt_neighbors_interval < MQTT_NEIGHBORS_MIN_INTERVAL_MS || + _mqtt_prefs.mqtt_neighbors_interval > MQTT_NEIGHBORS_MAX_INTERVAL_MS) { + _mqtt_prefs.mqtt_neighbors_interval = MQTT_NEIGHBORS_DEFAULT_INTERVAL_MS; + MESH_DEBUG_PRINTLN("MQTT: invalid neighbors interval reset to %u hours", + (unsigned)MQTT_NEIGHBORS_DEFAULT_INTERVAL_HOURS); + } _legacy_tail.valid = false; } diff --git a/src/helpers/CommonCLI.h b/src/helpers/CommonCLI.h index d470e7b1..255ea0bc 100644 --- a/src/helpers/CommonCLI.h +++ b/src/helpers/CommonCLI.h @@ -161,9 +161,19 @@ struct MQTTPrefs { // tail so a shorter /mqtt_prefs payload from earlier firmware still loads with // these defaulting off (see applyMQTTDefaults); keeps the format at VERSION 1. uint8_t mqtt_neighbors_enabled; // Periodic neighbors/scopes MQTT publish (PSRAM boards only) - uint32_t mqtt_neighbors_interval; // Neighbors publish interval (ms), default 24h, min 12h + uint32_t mqtt_neighbors_interval; // Neighbors publish interval (ms), default 24h, range 12-336h }; +// Neighbor discovery is scheduled with the wrap-safe millis() helpers, whose +// signed-delta comparison requires intervals below INT32_MAX milliseconds. +// Two weeks stays comfortably inside that range. +static const uint32_t MQTT_NEIGHBORS_MIN_INTERVAL_HOURS = 12; +static const uint32_t MQTT_NEIGHBORS_MAX_INTERVAL_HOURS = 336; +static const uint32_t MQTT_NEIGHBORS_DEFAULT_INTERVAL_HOURS = 24; +static const uint32_t MQTT_NEIGHBORS_MIN_INTERVAL_MS = MQTT_NEIGHBORS_MIN_INTERVAL_HOURS * 3600000UL; +static const uint32_t MQTT_NEIGHBORS_MAX_INTERVAL_MS = MQTT_NEIGHBORS_MAX_INTERVAL_HOURS * 3600000UL; +static const uint32_t MQTT_NEIGHBORS_DEFAULT_INTERVAL_MS = MQTT_NEIGHBORS_DEFAULT_INTERVAL_HOURS * 3600000UL; + // /mqtt_prefs is written with an 8-byte header so the format is self-describing. // Files with no header are legacy (versionless) and detected by size in loadMQTTPrefs. // The magic leads with a non-ASCII byte so it can never collide with the first diff --git a/src/helpers/CommonCLI_Observer.cpp b/src/helpers/CommonCLI_Observer.cpp index 1166553e..3f597107 100644 --- a/src/helpers/CommonCLI_Observer.cpp +++ b/src/helpers/CommonCLI_Observer.cpp @@ -247,22 +247,23 @@ bool CommonCLI::handleObserverSetCmd(uint32_t sender_timestamp, const char* conf strcpy(reply, "Error: interval must be between 1-60 minutes"); } #if defined(BOARD_HAS_PSRAM) && defined(MAX_NEIGHBOURS) && MAX_NEIGHBOURS > 0 - } else if (memcmp(config, "mqtt.neighbors ", 14) == 0) { - _mqtt_prefs.mqtt_neighbors_enabled = memcmp(&config[14], "on", 2) == 0; + } else if (memcmp(config, "mqtt.neighbors ", 15) == 0) { + _mqtt_prefs.mqtt_neighbors_enabled = memcmp(&config[15], "on", 2) == 0; savePrefs(); strcpy(reply, "OK"); } else if (memcmp(config, "mqtt.neighbors.interval ", 24) == 0) { uint32_t hours = _atoi(&config[24]); - if (hours >= 12 && hours <= 8760) { + if (hours >= MQTT_NEIGHBORS_MIN_INTERVAL_HOURS && + hours <= MQTT_NEIGHBORS_MAX_INTERVAL_HOURS) { _mqtt_prefs.mqtt_neighbors_interval = hours * 3600000UL; savePrefs(); sprintf(reply, "OK - neighbors interval set to %u hours (%lu ms)", hours, (unsigned long)_mqtt_prefs.mqtt_neighbors_interval); } else { - strcpy(reply, "Error: neighbors interval must be between 12-8760 hours"); + strcpy(reply, "Error: neighbors interval must be between 12-336 hours"); } #elif defined(WITH_MQTT_BRIDGE) - } else if (memcmp(config, "mqtt.neighbors ", 14) == 0 || + } else if (memcmp(config, "mqtt.neighbors ", 15) == 0 || memcmp(config, "mqtt.neighbors.interval ", 24) == 0) { strcpy(reply, "Err - not supported (requires PSRAM)"); #endif diff --git a/src/helpers/MQTTDefaults.h b/src/helpers/MQTTDefaults.h index 2617ef7c..72c58b1a 100644 --- a/src/helpers/MQTTDefaults.h +++ b/src/helpers/MQTTDefaults.h @@ -69,7 +69,7 @@ static inline void applyMQTTDefaults(MQTTPrefs* prefs) { prefs->mqtt_rx_enabled = 1; prefs->mqtt_status_interval = 300000; prefs->mqtt_neighbors_enabled = 0; - prefs->mqtt_neighbors_interval = 24UL * 3600UL * 1000UL; + prefs->mqtt_neighbors_interval = MQTT_NEIGHBORS_DEFAULT_INTERVAL_MS; prefs->wifi_power_save = 1; mqttDefaultSlotPreset(prefs->mqtt_slot_preset[0], sizeof(prefs->mqtt_slot_preset[0]), diff --git a/src/helpers/MQTTMessageBuilder.cpp b/src/helpers/MQTTMessageBuilder.cpp index 75be1613..a49f7f06 100644 --- a/src/helpers/MQTTMessageBuilder.cpp +++ b/src/helpers/MQTTMessageBuilder.cpp @@ -238,6 +238,8 @@ int MQTTMessageBuilder::buildNeighborsMessage( char* buffer, size_t buffer_size ) { + if (!buffer || buffer_size == 0) return 0; + doc.clear(); JsonObject root = doc.to(); @@ -249,6 +251,8 @@ int MQTTMessageBuilder::buildNeighborsMessage( self["scopes"] = self_scopes ? self_scopes : ""; JsonArray arr = root.createNestedArray("neighbors"); + if (measureJson(root) >= buffer_size) return 0; + for (int i = 0; i < neighbor_count; i++) { JsonObject nb = arr.createNestedObject(); nb["pubkey"] = neighbors[i].pubkey_hex; @@ -256,6 +260,14 @@ int MQTTMessageBuilder::buildNeighborsMessage( nb["heard_secs_ago"] = neighbors[i].heard_secs_ago; nb["scopes"] = neighbors[i].scopes ? neighbors[i].scopes : ""; nb["status"] = neighbors[i].status; + + // Callers order neighbors from most to least useful (newest, then + // strongest). Stop as soon as the next entry would fill the fixed publish + // buffer, keeping document growth bounded and dropping the remaining tail. + if (measureJson(root) >= buffer_size) { + arr.remove(arr.size() - 1); + break; + } } size_t len = serializeJson(root, buffer, buffer_size); @@ -470,4 +482,4 @@ void MQTTMessageBuilder::packetToHex(mesh::Packet* packet, char* hex, size_t hex // Convert serialized packet to hex bytesToHex(raw_buf, raw_len, hex, hex_size); -} \ No newline at end of file +} diff --git a/src/helpers/bridges/MQTTBridge.cpp b/src/helpers/bridges/MQTTBridge.cpp index fe09024b..aac22561 100644 --- a/src/helpers/bridges/MQTTBridge.cpp +++ b/src/helpers/bridges/MQTTBridge.cpp @@ -1034,13 +1034,13 @@ void MQTTBridge::mqttTaskLoop() { processPacketQueue(); #if defined(WITH_MQTT_NEIGHBORS) - if (_neighbors_publish_pending) { - _neighbors_publish_pending = false; + if (_neighbors_publish_pending.load(std::memory_order_acquire)) { if (publishNeighbors()) { MQTT_DEBUG_PRINTLN("Neighbors published"); } else { MQTT_DEBUG_PRINTLN("Neighbors publish failed"); } + _neighbors_publish_pending.store(false, std::memory_order_release); } #endif @@ -2749,13 +2749,14 @@ bool MQTTBridge::publishStatus() { #if defined(WITH_MQTT_NEIGHBORS) void MQTTBridge::requestPublishNeighbors(const char* json, size_t len) { if (!_neighbors_json_buffer || !json || len == 0) return; + if (_neighbors_publish_pending.load(std::memory_order_acquire)) return; if (len >= NEIGHBORS_JSON_BUFFER_SIZE) { len = NEIGHBORS_JSON_BUFFER_SIZE - 1; } memcpy(_neighbors_json_buffer, json, len); _neighbors_json_buffer[len] = '\0'; _neighbors_publish_len = len; - _neighbors_publish_pending = true; + _neighbors_publish_pending.store(true, std::memory_order_release); } bool MQTTBridge::publishNeighbors() { diff --git a/src/helpers/bridges/MQTTBridge.h b/src/helpers/bridges/MQTTBridge.h index 70de88ad..0623a17e 100644 --- a/src/helpers/bridges/MQTTBridge.h +++ b/src/helpers/bridges/MQTTBridge.h @@ -7,6 +7,7 @@ #include #include #include +#include #include "helpers/JWTHelper.h" #include "helpers/MQTTPresets.h" @@ -264,7 +265,9 @@ private: #if defined(WITH_MQTT_NEIGHBORS) char* _neighbors_json_buffer; size_t _neighbors_publish_len; - bool _neighbors_publish_pending; + // Release/acquire handoff from the Arduino loop (Core 1) to the MQTT task + // (Core 0). A second snapshot is dropped while the current one is publishing. + std::atomic _neighbors_publish_pending; #endif #if defined(BOARD_HAS_PSRAM) char* _publish_json_buffer; From df13d181b24263b4a0e392b54bf2966563a12919 Mon Sep 17 00:00:00 2001 From: agessaman Date: Thu, 16 Jul 2026 18:48:22 -0700 Subject: [PATCH 03/87] feat(mqtt): enhance periodic neighbors publishing with refresh logic Updated the MQTT neighbors functionality to include a two-stage refresh process for periodic publishing. The first stage performs a 60-second zero-hop neighbor refresh, followed by querying the refreshed table for scopes before publishing. This change improves the accuracy of neighbor data and ensures timely updates in the network. Documentation has been updated to reflect these changes. --- MQTT_IMPLEMENTATION.md | 2 ++ docs/cli_commands.md | 2 ++ examples/simple_repeater/MyMesh.cpp | 34 +++++++++++++++++++++++++---- examples/simple_repeater/MyMesh.h | 1 + 4 files changed, 35 insertions(+), 4 deletions(-) diff --git a/MQTT_IMPLEMENTATION.md b/MQTT_IMPLEMENTATION.md index dcc17502..188fc4f4 100644 --- a/MQTT_IMPLEMENTATION.md +++ b/MQTT_IMPLEMENTATION.md @@ -511,6 +511,8 @@ Minimal raw packet data for map integration. ### Neighbors Topic: `meshcore/{IATA}/{DEVICE_PUBLIC_KEY}/neighbors` Cached zero-hop repeater neighbors with SNR, last-heard age, and flood-allowed scopes (retained, QoS 1). Published on `discover.scopes` or periodically when `mqtt.neighbors` is enabled (PSRAM observer builds only). +Periodic publishing first runs a 60-second zero-hop neighbor refresh equivalent to `discover.neighbors`, then queries the refreshed table for scopes and publishes when the scope-query phase completes. Manual `discover.scopes` remains a one-shot query of the current cache. + **Note**: `{DEVICE_PUBLIC_KEY}` is the device's public key in hexadecimal format (64 characters). ## JSON Message Formats diff --git a/docs/cli_commands.md b/docs/cli_commands.md index c740ce1f..d8618e67 100644 --- a/docs/cli_commands.md +++ b/docs/cli_commands.md @@ -1127,6 +1127,8 @@ Requires WiFi connected and the MQTT bridge running. **Note:** Requires PSRAM. Publishes to `meshcore/{IATA}/{device}/neighbors` (retained, QoS 1). Use `discover.scopes` for a one-shot publish. +Automatic publishes run in two stages: first the repeater performs the same 60-second zero-hop refresh as `discover.neighbors`, then it queries the refreshed neighbors for scopes and publishes after those queries complete. + --- #### View or change the speed of the bridge (RS-232 only) diff --git a/examples/simple_repeater/MyMesh.cpp b/examples/simple_repeater/MyMesh.cpp index 1ca5adba..c6010871 100644 --- a/examples/simple_repeater/MyMesh.cpp +++ b/examples/simple_repeater/MyMesh.cpp @@ -1035,6 +1035,7 @@ MyMesh::MyMesh(mesh::MainBoard &board, mesh::Radio &radio, mesh::MillisecondCloc #if defined(WITH_MQTT_NEIGHBORS) neighbor_discover_count = 0; neighbor_discover_active = false; + neighbor_table_refresh_active = false; neighbor_discover_until = 0; next_neighbors_publish = 0; self_scopes_buf[0] = 0; @@ -1530,16 +1531,41 @@ void MyMesh::loop() { #endif #if defined(WITH_MQTT_NEIGHBORS) + bool periodic_neighbors_enabled = _cli.getObserverPrefs()->mqtt_neighbors_enabled; if (neighbor_discover_active) { loopNeighborDiscover(); - } else if (_cli.getObserverPrefs()->mqtt_neighbors_enabled && bridge && bridge->isRunning()) { - if (next_neighbors_publish == 0 || - (next_neighbors_publish != 0 && millisHasNowPassed(next_neighbors_publish))) { + } else if (neighbor_table_refresh_active) { + if (!periodic_neighbors_enabled) { + // Turning periodic publishing off cancels a refresh that has not yet + // reached the scope-query/publish phase. Re-enabling starts fresh. + neighbor_table_refresh_active = false; + pending_discover_tag = 0; + next_neighbors_publish = 0; + } else if (pending_discover_tag == 0 || millisHasNowPassed(pending_discover_until)) { + // The zero-hop discover.neighbors response window is complete. Freeze the + // refreshed table into the scope-query overlay, then publish when those + // per-neighbor requests finish. + pending_discover_tag = 0; + neighbor_table_refresh_active = false; char tmp_reply[80]; if (startNeighborDiscover(tmp_reply)) { - MESH_DEBUG_PRINTLN("%s", tmp_reply); + MESH_DEBUG_PRINTLN("MQTT periodic %s", tmp_reply); + } else { + // Avoid retrying a failed refresh on every loop iteration. + next_neighbors_publish = futureMillis(_cli.getObserverPrefs()->mqtt_neighbors_interval); + MESH_DEBUG_PRINTLN("MQTT periodic neighbor scope discovery failed: %s", tmp_reply); } } + } else if (periodic_neighbors_enabled && bridge && bridge->isRunning()) { + if (next_neighbors_publish == 0 || + (next_neighbors_publish != 0 && millisHasNowPassed(next_neighbors_publish))) { + // Refresh the zero-hop neighbor cache first (the same operation as the + // discover.neighbors CLI command). Scope queries begin after its 60-second + // collection window instead of publishing a stale cached table. + sendNodeDiscoverReq(); + neighbor_table_refresh_active = true; + MESH_DEBUG_PRINTLN("MQTT periodic neighbor table refresh started"); + } } #endif diff --git a/examples/simple_repeater/MyMesh.h b/examples/simple_repeater/MyMesh.h index a2cfc162..bb2c6698 100644 --- a/examples/simple_repeater/MyMesh.h +++ b/examples/simple_repeater/MyMesh.h @@ -152,6 +152,7 @@ class MyMesh : public mesh::Mesh, public CommonCLICallbacks { NeighborDiscoverEntry neighbor_discover[MAX_NEIGHBOURS]; uint8_t neighbor_discover_count; bool neighbor_discover_active; + bool neighbor_table_refresh_active; unsigned long neighbor_discover_until; unsigned long next_neighbors_publish; char self_scopes_buf[96]; From b3bd5ecd3c6f18a81855460604e1c795c7a501da Mon Sep 17 00:00:00 2001 From: agessaman Date: Fri, 17 Jul 2026 07:59:42 -0700 Subject: [PATCH 04/87] feat(mqtt): send all message types to MeshRank and harden neighbors flow MeshRank slots now publish every message type (status/packets/raw/neighbors) under meshrank/uplink/{token}/{device}/, matching the standard MeshCore topic layout, instead of packets only. The four duplicated topic-suffix ternaries are folded into messageTypeSuffix(). Fix the discover.neighbors -> discover.scopes race: a scopes request issued while a zero-hop discovery is still collecting responses (from discover.neighbors or the periodic timer) now queues behind that 60-second window and runs against the refreshed table, reporting the wait in its reply. A queued one-shot request is no longer cancelled by `mqtt.neighbors off`; only the periodic timer's own refresh is. Precondition checks are shared via neighborDiscoverReady() so queuing cannot report OK for a request that will fail once the window closes. Add the periodic-neighbors schedule to `get mqtt.status` as a trailing nbr:/ field (time to next publish + last result). The mesh owns the timer and reports a summary to the bridge across the core boundary via atomics. Also clamp the running offset in formatMqttStatusReply: snprintf returns the untruncated length, so a full 6-slot line could push the offset past the buffer and underflow the remaining size on the next append. Reachable today with real preset-name combinations at the 160-byte reply size. --- MQTT_IMPLEMENTATION.md | 14 ++-- docs/cli_commands.md | 11 ++- examples/simple_repeater/MyMesh.cpp | 81 ++++++++++++++++++---- examples/simple_repeater/MyMesh.h | 2 + src/helpers/MQTTPresets.h | 2 +- src/helpers/bridges/MQTTBridge.cpp | 101 ++++++++++++++++++++++------ src/helpers/bridges/MQTTBridge.h | 17 +++++ 7 files changed, 185 insertions(+), 43 deletions(-) diff --git a/MQTT_IMPLEMENTATION.md b/MQTT_IMPLEMENTATION.md index 188fc4f4..60ec72f7 100644 --- a/MQTT_IMPLEMENTATION.md +++ b/MQTT_IMPLEMENTATION.md @@ -296,7 +296,7 @@ set mqtt3.preset meshrank set mqtt3.token FE1B34242C5938C39225310081FD6718 ``` -The token is generated on the MeshRank website and is tied to your account. MeshRank only receives packet data (no status or raw messages). +The token is generated on the MeshRank website and is tied to your account. MeshRank receives the same message types as any other preset (`status`, `packets`, `raw`, `neighbors`), each under `meshrank/uplink/{token}/{device_id}/`, subject to the usual `mqtt.status`/`mqtt.raw`/`mqtt.neighbors` toggles. #### Example: Configure MeshMapper on Slot 3 ```bash @@ -376,7 +376,7 @@ These settings apply across all MQTT slots: - `get mqtt.iata` - Get IATA code - `get mqtt.presets` - List available MQTT presets (paginated, comma-separated) - `get mqtt.presets ` - Continue list from index shown in `... next:` -- `get mqtt.status` - Get MQTT status summary (connection info per slot) +- `get mqtt.status` - Get MQTT status summary (connection info per slot, plus the periodic neighbors schedule when `mqtt.neighbors` is on) - `get mqtt.packets` - Get packet message setting (on/off) - `get mqtt.raw` - Get raw message setting (on/off) - `get mqtt.rx` - Get RX packet uplinking setting (on/off) @@ -509,11 +509,15 @@ Full packet data with RF characteristics and metadata. Minimal raw packet data for map integration. ### Neighbors Topic: `meshcore/{IATA}/{DEVICE_PUBLIC_KEY}/neighbors` -Cached zero-hop repeater neighbors with SNR, last-heard age, and flood-allowed scopes (retained, QoS 1). Published on `discover.scopes` or periodically when `mqtt.neighbors` is enabled (PSRAM observer builds only). +Cached zero-hop repeater neighbors with SNR, last-heard age, and flood-allowed scopes. Published on `discover.scopes` or periodically when `mqtt.neighbors` is enabled (PSRAM observer builds only). Goes to every configured slot's `neighbors` topic at QoS 1, retained only where the preset allows it. -Periodic publishing first runs a 60-second zero-hop neighbor refresh equivalent to `discover.neighbors`, then queries the refreshed table for scopes and publishes when the scope-query phase completes. Manual `discover.scopes` remains a one-shot query of the current cache. +Periodic publishing first runs a 60-second zero-hop neighbor refresh equivalent to `discover.neighbors`, then queries the refreshed table for scopes and publishes when the scope-query phase completes. -**Note**: `{DEVICE_PUBLIC_KEY}` is the device's public key in hexadecimal format (64 characters). +Manual `discover.scopes` normally queries the current cache in one shot. If a `discover.neighbors` refresh is already collecting responses — whether started from the CLI or by the periodic timer — the scope queries are queued behind its 60-second window instead, so they run against the refreshed table. The reply reports the wait, e.g. `OK - scopes queued (47s discovery remaining)`. A queued one-shot request survives `set mqtt.neighbors off`; only the periodic timer's own refresh is cancelled by it. + +While `mqtt.neighbors` is on, `get mqtt.status` appends `nbr: /` — time to the next automatic publish (`3h12m`, `12m`, `45s`, or `active`/`due`) and the last publish result (`ok`, `failed`, or `none`). + +**Note**: `{DEVICE_PUBLIC_KEY}` is the device's public key in hexadecimal format (64 characters). MeshRank slots use `meshrank/uplink/{token}/{DEVICE_PUBLIC_KEY}/neighbors` instead. ## JSON Message Formats diff --git a/docs/cli_commands.md b/docs/cli_commands.md index d8618e67..55f8cd64 100644 --- a/docs/cli_commands.md +++ b/docs/cli_commands.md @@ -118,7 +118,9 @@ This document provides an overview of CLI commands that can be sent to MeshCore **Usage:** - `discover.scopes` -Queries cached zero-hop repeater neighbors for flood-allowed region scopes (`ANON_REQ_TYPE_REGIONS`), then publishes a retained JSON snapshot to `meshcore/{IATA}/{device}/neighbors`. Includes this node's own scopes in the `self` object. +Queries cached zero-hop repeater neighbors for flood-allowed region scopes (`ANON_REQ_TYPE_REGIONS`), then publishes a JSON snapshot to each configured MQTT slot's `neighbors` topic. Includes this node's own scopes in the `self` object. + +If a zero-hop neighbor discovery is already collecting responses — from `discover.neighbors` or from the periodic `mqtt.neighbors` refresh — the scope queries are queued until its 60-second response window ends, so they run against the refreshed table rather than the stale cache. The reply reports the wait, e.g. `OK - scopes queued (47s discovery remaining)`. A queued request is not cancelled by `set mqtt.neighbors off`. **Note:** Requires `WITH_MQTT_BRIDGE`, `MAX_NEIGHBOURS`, and PSRAM (`BOARD_HAS_PSRAM`). Returns `Err - not supported (requires PSRAM)` on other builds. @@ -1125,10 +1127,15 @@ Requires WiFi connected and the MQTT bridge running. **Parameters:** - `hours`: Interval between automatic `discover.scopes` runs (**12-336**, default **24**) -**Note:** Requires PSRAM. Publishes to `meshcore/{IATA}/{device}/neighbors` (retained, QoS 1). Use `discover.scopes` for a one-shot publish. +**Note:** Requires PSRAM. Publishes to each configured MQTT slot's `neighbors` topic at QoS 1 (retained only when that preset allows it). Use `discover.scopes` for a one-shot publish. Automatic publishes run in two stages: first the repeater performs the same 60-second zero-hop refresh as `discover.neighbors`, then it queries the refreshed neighbors for scopes and publishes after those queries complete. +While enabled, `get mqtt.status` reports the schedule as a trailing `nbr: /` field, e.g. `nbr: 3h12m/ok`: + +- `` — time until the next automatic publish (`3h12m`, `12m`, `45s`), or `active` while a refresh or scope query is in flight, or `due` when a publish is waiting on the bridge or WiFi to come up. +- `` — result of the most recent publish attempt: `ok`, `failed`, or `none` if none has been attempted since boot. + --- #### View or change the speed of the bridge (RS-232 only) diff --git a/examples/simple_repeater/MyMesh.cpp b/examples/simple_repeater/MyMesh.cpp index c6010871..098d60b1 100644 --- a/examples/simple_repeater/MyMesh.cpp +++ b/examples/simple_repeater/MyMesh.cpp @@ -1036,6 +1036,7 @@ MyMesh::MyMesh(mesh::MainBoard &board, mesh::Radio &radio, mesh::MillisecondCloc neighbor_discover_count = 0; neighbor_discover_active = false; neighbor_table_refresh_active = false; + neighbor_table_refresh_periodic = false; neighbor_discover_until = 0; next_neighbors_publish = 0; self_scopes_buf[0] = 0; @@ -1451,6 +1452,25 @@ void MyMesh::handleCommand(uint32_t sender_timestamp, char *command, char *reply while (*sub == ' ') sub++; if (*sub != 0) { strcpy(reply, "Err - discover.scopes has no options"); + } else if (pending_discover_tag != 0 && + !millisHasNowPassed(pending_discover_until) && + !neighbor_discover_active) { + // Reuse an in-flight zero-hop discovery rather than querying the old + // neighbor table. An explicit command takes ownership of a periodic + // refresh so `mqtt.neighbors off` cannot cancel this one-shot request. + // Validate up front so queuing cannot report OK for a request that is + // certain to fail once the discovery window closes. + if (!neighborDiscoverReady(reply)) { + // reply already set + } else { + neighbor_table_refresh_active = true; + neighbor_table_refresh_periodic = false; + long remaining_ms = (long)(pending_discover_until - futureMillis(0)); + unsigned remaining_secs = remaining_ms > 0 + ? (unsigned)(((unsigned long)remaining_ms + 999UL) / 1000UL) : 0; + sprintf(reply, "OK - scopes queued (%us discovery remaining)", remaining_secs); + MESH_DEBUG_PRINTLN("Neighbor scopes queued behind active discovery (%us remaining)", remaining_secs); + } } else if (!startNeighborDiscover(reply)) { // reply already set } @@ -1535,25 +1555,33 @@ void MyMesh::loop() { if (neighbor_discover_active) { loopNeighborDiscover(); } else if (neighbor_table_refresh_active) { - if (!periodic_neighbors_enabled) { + if (neighbor_table_refresh_periodic && !periodic_neighbors_enabled) { // Turning periodic publishing off cancels a refresh that has not yet // reached the scope-query/publish phase. Re-enabling starts fresh. + // pending_discover_tag is left alone: this refresh may have joined a + // discover.neighbors the user started, and clearing the tag would drop + // its remaining responses. An unused tag expires on its own. neighbor_table_refresh_active = false; - pending_discover_tag = 0; + neighbor_table_refresh_periodic = false; next_neighbors_publish = 0; } else if (pending_discover_tag == 0 || millisHasNowPassed(pending_discover_until)) { // The zero-hop discover.neighbors response window is complete. Freeze the // refreshed table into the scope-query overlay, then publish when those // per-neighbor requests finish. + bool was_periodic = neighbor_table_refresh_periodic; pending_discover_tag = 0; neighbor_table_refresh_active = false; + neighbor_table_refresh_periodic = false; char tmp_reply[80]; + const char* origin_str = was_periodic ? "periodic" : "manual"; if (startNeighborDiscover(tmp_reply)) { - MESH_DEBUG_PRINTLN("MQTT periodic %s", tmp_reply); + MESH_DEBUG_PRINTLN("MQTT %s %s", origin_str, tmp_reply); } else { - // Avoid retrying a failed refresh on every loop iteration. - next_neighbors_publish = futureMillis(_cli.getObserverPrefs()->mqtt_neighbors_interval); - MESH_DEBUG_PRINTLN("MQTT periodic neighbor scope discovery failed: %s", tmp_reply); + if (periodic_neighbors_enabled) { + // Avoid retrying a failed refresh on every loop iteration. + next_neighbors_publish = futureMillis(_cli.getObserverPrefs()->mqtt_neighbors_interval); + } + MESH_DEBUG_PRINTLN("MQTT %s neighbor scope discovery failed: %s", origin_str, tmp_reply); } } } else if (periodic_neighbors_enabled && bridge && bridge->isRunning()) { @@ -1562,9 +1590,29 @@ void MyMesh::loop() { // Refresh the zero-hop neighbor cache first (the same operation as the // discover.neighbors CLI command). Scope queries begin after its 60-second // collection window instead of publishing a stale cached table. - sendNodeDiscoverReq(); + if (pending_discover_tag == 0 || millisHasNowPassed(pending_discover_until)) { + pending_discover_tag = 0; + sendNodeDiscoverReq(); + MESH_DEBUG_PRINTLN("MQTT periodic neighbor table refresh started"); + } else { + MESH_DEBUG_PRINTLN("MQTT periodic refresh joined active neighbor discovery"); + } neighbor_table_refresh_active = true; - MESH_DEBUG_PRINTLN("MQTT periodic neighbor table refresh started"); + neighbor_table_refresh_periodic = true; + } + } + + // Report the periodic-neighbors schedule for `get mqtt.status`. The timer lives + // here, so the wrap-safe millis math stays on this side of the handoff. + if (bridge) { + if (neighbor_discover_active || neighbor_table_refresh_active) { + bridge->setNeighborsSchedule(MQTTBridge::NBR_ACTIVE, 0); + } else if (next_neighbors_publish == 0 || millisHasNowPassed(next_neighbors_publish)) { + bridge->setNeighborsSchedule(MQTTBridge::NBR_DUE, 0); + } else { + long remaining_ms = (long)(next_neighbors_publish - futureMillis(0)); + uint32_t remaining_secs = remaining_ms > 0 ? (uint32_t)(remaining_ms / 1000) : 0; + bridge->setNeighborsSchedule(MQTTBridge::NBR_SCHEDULED, remaining_secs); } } #endif @@ -1755,11 +1803,7 @@ void MyMesh::loopNeighborDiscover() { finishNeighborDiscover(); } -bool MyMesh::startNeighborDiscover(char* reply) { - if (neighbor_discover_active) { - strcpy(reply, "Err - neighbor discover already active"); - return false; - } +bool MyMesh::neighborDiscoverReady(char* reply) { #if defined(ESP_PLATFORM) if (!psramFound()) { strcpy(reply, "Err - PSRAM not available"); @@ -1770,6 +1814,17 @@ bool MyMesh::startNeighborDiscover(char* reply) { strcpy(reply, "Err - MQTT bridge not running"); return false; } + return true; +} + +bool MyMesh::startNeighborDiscover(char* reply) { + if (neighbor_discover_active) { + strcpy(reply, "Err - neighbor discover already active"); + return false; + } + if (!neighborDiscoverReady(reply)) { + return false; // reply already set + } getLocalScopes(self_scopes_buf, sizeof(self_scopes_buf)); neighbor_discover_count = 0; diff --git a/examples/simple_repeater/MyMesh.h b/examples/simple_repeater/MyMesh.h index bb2c6698..b223a62f 100644 --- a/examples/simple_repeater/MyMesh.h +++ b/examples/simple_repeater/MyMesh.h @@ -153,11 +153,13 @@ class MyMesh : public mesh::Mesh, public CommonCLICallbacks { uint8_t neighbor_discover_count; bool neighbor_discover_active; bool neighbor_table_refresh_active; + bool neighbor_table_refresh_periodic; unsigned long neighbor_discover_until; unsigned long next_neighbors_publish; char self_scopes_buf[96]; bool sendAnonRegionsReq(const mesh::Identity& target, uint32_t& tag); + bool neighborDiscoverReady(char* reply); bool startNeighborDiscover(char* reply); void loopNeighborDiscover(); void finishNeighborDiscover(); diff --git a/src/helpers/MQTTPresets.h b/src/helpers/MQTTPresets.h index 8ab063e0..38289d72 100644 --- a/src/helpers/MQTTPresets.h +++ b/src/helpers/MQTTPresets.h @@ -23,7 +23,7 @@ enum MQTTAuthType : uint8_t { enum MQTTTopicStyle : uint8_t { MQTT_TOPIC_MESHCORE, // meshcore/{iata}/{device_id}/{status|packets|raw} - MQTT_TOPIC_MESHRANK, // meshrank/uplink/{token}/{device_id}/packets (packets only) + MQTT_TOPIC_MESHRANK, // meshrank/uplink/{token}/{device_id}/{status|packets|raw|neighbors} }; struct MQTTPresetDef { diff --git a/src/helpers/bridges/MQTTBridge.cpp b/src/helpers/bridges/MQTTBridge.cpp index 72a5bf26..93959837 100644 --- a/src/helpers/bridges/MQTTBridge.cpp +++ b/src/helpers/bridges/MQTTBridge.cpp @@ -201,6 +201,19 @@ unsigned long MQTTBridge::getWifiConnectedAtMillis() { return s_wifi_connected_at; } +#if defined(WITH_MQTT_NEIGHBORS) +// Compact "3h12m" / "12m" / "45s" for CLI replies that share a 160-byte buffer. +static void formatDuration(char* buf, size_t bufsize, uint32_t secs) { + if (secs >= 3600) { + snprintf(buf, bufsize, "%uh%um", (unsigned)(secs / 3600), (unsigned)((secs % 3600) / 60)); + } else if (secs >= 60) { + snprintf(buf, bufsize, "%um", (unsigned)(secs / 60)); + } else { + snprintf(buf, bufsize, "%us", (unsigned)secs); + } +} +#endif + void MQTTBridge::formatMqttStatusReply(char* buf, size_t bufsize, const MQTTPrefs* obs) { if (buf == nullptr || bufsize == 0) return; const char* msgs = (obs && obs->mqtt_status_enabled) ? "on" : "off"; @@ -246,8 +259,40 @@ void MQTTBridge::formatMqttStatusReply(char* buf, size_t bufsize, const MQTTPref state = "disc"; } pos += snprintf(buf + pos, bufsize - pos, ", %d: %s (%s)", i + 1, name, state); + // snprintf returns the untruncated length, so clamp before reusing pos as an + // offset: an overflowing pos would walk past buf and underflow bufsize - pos. + if (pos >= (int)bufsize) { pos = (int)bufsize - 1; break; } } - snprintf(buf + pos, bufsize - pos, ", q:%d", q); + if (pos < (int)bufsize - 1) { + pos += snprintf(buf + pos, bufsize - pos, ", q:%d", q); + if (pos >= (int)bufsize) pos = (int)bufsize - 1; + } + +#if defined(WITH_MQTT_NEIGHBORS) + // Periodic neighbors: time to next publish + how the last one went. + if (obs && obs->mqtt_neighbors_enabled && pos < (int)bufsize - 1) { + char when[16]; + switch (b->_neighbors_phase.load(std::memory_order_relaxed)) { + case NBR_ACTIVE: + strcpy(when, "active"); + break; + case NBR_DUE: + strcpy(when, "due"); + break; + default: + formatDuration(when, sizeof(when), + b->_neighbors_secs_until_next.load(std::memory_order_relaxed)); + break; + } + const char* last; + switch (b->_neighbors_last_result.load(std::memory_order_relaxed)) { + case NBR_RESULT_OK: last = "ok"; break; + case NBR_RESULT_FAIL: last = "failed"; break; + default: last = "none"; break; + } + snprintf(buf + pos, bufsize - pos, ", nbr: %s/%s", when, last); + } +#endif } // On-demand publish-health + heap snapshot for the `get mqtt.stats` CLI command. @@ -533,6 +578,9 @@ MQTTBridge::MQTTBridge(NodePrefs *prefs, MQTTPrefs *obs, mesh::PacketManager *mg _neighbors_json_buffer = (char*)psram_malloc(NEIGHBORS_JSON_BUFFER_SIZE); _neighbors_publish_pending = false; _neighbors_publish_len = 0; + _neighbors_last_result = NBR_RESULT_NONE; + _neighbors_phase = NBR_SCHEDULED; + _neighbors_secs_until_next = 0; #endif #else memset(_last_raw_data, 0, sizeof(_last_raw_data)); @@ -1059,7 +1107,10 @@ void MQTTBridge::mqttTaskLoop() { #if defined(WITH_MQTT_NEIGHBORS) if (_neighbors_publish_pending.load(std::memory_order_acquire)) { - if (publishNeighbors()) { + bool ok = publishNeighbors(); + _neighbors_last_result.store(ok ? NBR_RESULT_OK : NBR_RESULT_FAIL, + std::memory_order_relaxed); + if (ok) { MQTT_DEBUG_PRINTLN("Neighbors published"); } else { MQTT_DEBUG_PRINTLN("Neighbors publish failed"); @@ -1864,11 +1915,17 @@ bool MQTTBridge::publishToAllSlots(const char* topic, const char* payload, bool // Topic building - resolves the correct topic for a given slot and message type. // Presets use hardcoded topic logic; custom slots support user-defined templates. // --------------------------------------------------------------------------- +const char* MQTTBridge::messageTypeSuffix(MQTTMessageType type) { + switch (type) { + case MSG_STATUS: return "status"; + case MSG_PACKETS: return "packets"; + case MSG_RAW: return "raw"; + default: return "neighbors"; + } +} + bool MQTTBridge::substituteTopicTemplate(const char* tmpl, MQTTMessageType type, int slot_index, char* buf, size_t buf_size) { - const char* type_str = (type == MSG_STATUS) ? "status" - : (type == MSG_PACKETS) ? "packets" - : (type == MSG_RAW) ? "raw" - : "neighbors"; + const char* type_str = messageTypeSuffix(type); const char* token = _obs->mqtt_slot_token[slot_index]; size_t out = 0; @@ -1917,20 +1974,18 @@ bool MQTTBridge::buildTopicForSlot(int index, MQTTMessageType type, char* topic_ // Preset slots: use hardcoded topic logic if (slot.preset) { if (slot.preset->topic_style == MQTT_TOPIC_MESHRANK) { - // MeshRank: packets only, uses per-slot token in topic path - if (type != MSG_PACKETS) return false; + // MeshRank uses a per-slot token in its topic path. Publish every message + // type using the same suffixes as the standard MeshCore topic layout. const char* token = _obs->mqtt_slot_token[index]; if (!token || token[0] == '\0') return false; - snprintf(topic_buf, buf_size, "meshrank/uplink/%s/%s/packets", token, _device_id); + snprintf(topic_buf, buf_size, "meshrank/uplink/%s/%s/%s", token, _device_id, + messageTypeSuffix(type)); return true; } // MQTT_TOPIC_MESHCORE (default for all other presets) if (!isIATAValid()) return false; - const char* type_str = (type == MSG_STATUS) ? "status" - : (type == MSG_PACKETS) ? "packets" - : (type == MSG_RAW) ? "raw" - : "neighbors"; - snprintf(topic_buf, buf_size, "meshcore/%s/%s/%s", _iata, _device_id, type_str); + snprintf(topic_buf, buf_size, "meshcore/%s/%s/%s", _iata, _device_id, + messageTypeSuffix(type)); return true; } @@ -1940,11 +1995,8 @@ bool MQTTBridge::buildTopicForSlot(int index, MQTTMessageType type, char* topic_ } // Default: meshcore format if (!isIATAValid()) return false; - const char* type_str = (type == MSG_STATUS) ? "status" - : (type == MSG_PACKETS) ? "packets" - : (type == MSG_RAW) ? "raw" - : "neighbors"; - snprintf(topic_buf, buf_size, "meshcore/%s/%s/%s", _iata, _device_id, type_str); + snprintf(topic_buf, buf_size, "meshcore/%s/%s/%s", _iata, _device_id, + messageTypeSuffix(type)); return true; } @@ -1958,7 +2010,7 @@ void MQTTBridge::publishStatusToSlot(int index) { // Build per-slot topic (handles IATA check for meshcore, token check for meshrank) char status_topic[128]; if (!buildTopicForSlot(index, MSG_STATUS, status_topic, sizeof(status_topic))) { - return; // Slot doesn't support status (e.g., meshrank) or missing required config + return; // Slot is missing required topic configuration } // Reuse pre-allocated buffer to avoid heap alloc/free churn under memory pressure. @@ -2755,8 +2807,8 @@ bool MQTTBridge::publishStatus() { } } } - // If no connected slot accepts status topics (e.g. meshrank is packets-only), - // treat as success to avoid infinite retry loops + // If no connected slot accepts status topics, treat as success to avoid + // infinite retry loops. if (published || !any_slot_wants_status) { if (published) MQTT_DEBUG_PRINTLN("Status published"); return true; @@ -2767,6 +2819,11 @@ bool MQTTBridge::publishStatus() { } #if defined(WITH_MQTT_NEIGHBORS) +void MQTTBridge::setNeighborsSchedule(NeighborsPhase phase, uint32_t secs_until_next) { + _neighbors_phase.store((uint8_t)phase, std::memory_order_relaxed); + _neighbors_secs_until_next.store(secs_until_next, std::memory_order_relaxed); +} + void MQTTBridge::requestPublishNeighbors(const char* json, size_t len) { if (!_neighbors_json_buffer || !json || len == 0) return; if (_neighbors_publish_pending.load(std::memory_order_acquire)) return; diff --git a/src/helpers/bridges/MQTTBridge.h b/src/helpers/bridges/MQTTBridge.h index 23dd452a..5e7e2d01 100644 --- a/src/helpers/bridges/MQTTBridge.h +++ b/src/helpers/bridges/MQTTBridge.h @@ -268,6 +268,12 @@ private: // Release/acquire handoff from the Arduino loop (Core 1) to the MQTT task // (Core 0). A second snapshot is dropped while the current one is publishing. std::atomic _neighbors_publish_pending; + // Written by the MQTT task (Core 0), read by the CLI (Core 1). + enum NeighborsResult : uint8_t { NBR_RESULT_NONE, NBR_RESULT_OK, NBR_RESULT_FAIL }; + std::atomic _neighbors_last_result; + // Written by the mesh loop (Core 1), read by the CLI (Core 1). + std::atomic _neighbors_phase; + std::atomic _neighbors_secs_until_next; #endif #if defined(BOARD_HAS_PSRAM) char* _publish_json_buffer; @@ -331,6 +337,7 @@ private: // Topic building enum MQTTMessageType { MSG_STATUS, MSG_PACKETS, MSG_RAW, MSG_NEIGHBORS }; + static const char* messageTypeSuffix(MQTTMessageType type); bool buildTopicForSlot(int index, MQTTMessageType type, char* topic_buf, size_t buf_size); bool substituteTopicTemplate(const char* tmpl, MQTTMessageType type, int slot_index, char* buf, size_t buf_size); @@ -495,6 +502,16 @@ public: #if defined(WITH_MQTT_NEIGHBORS) void requestPublishNeighbors(const char* json, size_t len); static const size_t NEIGHBORS_JSON_BUFFER_SIZE = 10240; + + // Periodic-neighbors schedule, reported by the mesh loop for `get mqtt.status`. + // The mesh owns the timer; the bridge only caches the summary, which keeps the + // wrap-safe millis math on the one side that already has those helpers. + enum NeighborsPhase : uint8_t { + NBR_SCHEDULED, // waiting for the next publish; secs_until_next is valid + NBR_ACTIVE, // zero-hop refresh or scope queries in flight + NBR_DUE, // publish is due, waiting on the bridge/WiFi to come up + }; + void setNeighborsSchedule(NeighborsPhase phase, uint32_t secs_until_next); #endif static const char* wifiReasonStr(uint8_t reason); static const char* tlsErrorStr(int32_t err); From 028a5dcaddd69b4be80cc595d33d450be1c46e8b Mon Sep 17 00:00:00 2001 From: agessaman Date: Fri, 17 Jul 2026 08:01:09 -0700 Subject: [PATCH 05/87] fix(mqtt): honor preset retain policy for status publishes publishStatusToSlot hardcoded the retain flag to true, ignoring the preset's allow_retain field that the packets and neighbors paths already respect. The waev preset (allow_retain=false, MeshCore topic style) has therefore been publishing retained status to a broker that rejects the retain flag; the same would now apply to meshrank, which just gained status publishing. Use slot.preset->allow_retain, defaulting to true for custom slots to preserve their existing behavior. --- MQTT_IMPLEMENTATION.md | 4 ++-- src/helpers/bridges/MQTTBridge.cpp | 6 +++++- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/MQTT_IMPLEMENTATION.md b/MQTT_IMPLEMENTATION.md index 60ec72f7..f6f2a6b0 100644 --- a/MQTT_IMPLEMENTATION.md +++ b/MQTT_IMPLEMENTATION.md @@ -296,7 +296,7 @@ set mqtt3.preset meshrank set mqtt3.token FE1B34242C5938C39225310081FD6718 ``` -The token is generated on the MeshRank website and is tied to your account. MeshRank receives the same message types as any other preset (`status`, `packets`, `raw`, `neighbors`), each under `meshrank/uplink/{token}/{device_id}/`, subject to the usual `mqtt.status`/`mqtt.raw`/`mqtt.neighbors` toggles. +The token is generated on the MeshRank website and is tied to your account. MeshRank receives the same message types as any other preset (`status`, `packets`, `raw`, `neighbors`), each under `meshrank/uplink/{token}/{device_id}/`, subject to the usual `mqtt.status`/`mqtt.raw`/`mqtt.neighbors` toggles. Its broker does not accept the retain flag, so those publishes go out unretained. #### Example: Configure MeshMapper on Slot 3 ```bash @@ -500,7 +500,7 @@ The CLI commands are organized into two levels: The bridge publishes to four main topics with the following structure: ### Status Topic: `meshcore/{IATA}/{DEVICE_PUBLIC_KEY}/status` -Device connection status and metadata (retained messages). +Device connection status and metadata, QoS 1. Retained, except on presets whose broker rejects the retain flag (`meshrank`, `waev`). ### Packets Topic: `meshcore/{IATA}/{DEVICE_PUBLIC_KEY}/packets` Full packet data with RF characteristics and metadata. diff --git a/src/helpers/bridges/MQTTBridge.cpp b/src/helpers/bridges/MQTTBridge.cpp index 93959837..9a9d4628 100644 --- a/src/helpers/bridges/MQTTBridge.cpp +++ b/src/helpers/bridges/MQTTBridge.cpp @@ -2080,7 +2080,11 @@ void MQTTBridge::publishStatusToSlot(int index) { ); if (len > 0) { - int result = slot.client->publish(status_topic, 1, true, json_buffer, strlen(json_buffer)); + // Honor the preset's retain policy, as the packets/neighbors paths do — + // brokers that set allow_retain=false (meshrank, waev) reject retained + // publishes. Custom slots keep the long-standing retained status behavior. + bool use_retain = slot.preset ? slot.preset->allow_retain : true; + int result = slot.client->publish(status_topic, 1, use_retain, json_buffer, strlen(json_buffer)); if (result <= 0) { MQTT_DEBUG_PRINTLN("MQTT%d status publish failed", index + 1); } From 90467ddd13941a41cbd26935d60f8d9d896539a9 Mon Sep 17 00:00:00 2001 From: agessaman Date: Sat, 18 Jul 2026 18:57:01 -0700 Subject: [PATCH 06/87] feat(mqtt): add meshtexas preset for MQTT connections Introduced a new MQTT preset for the meshtexas connection, allowing users to connect to mqtt.meshtexas.org with JWT authentication over WSS. This addition increases the number of built-in presets to 27, enhancing the flexibility of the MQTT implementation. --- .build-wt-028a5dca | 1 + MQTT_IMPLEMENTATION.md | 1 + src/helpers/MQTTPresets.h | 3 ++- 3 files changed, 4 insertions(+), 1 deletion(-) create mode 160000 .build-wt-028a5dca diff --git a/.build-wt-028a5dca b/.build-wt-028a5dca new file mode 160000 index 00000000..028a5dca --- /dev/null +++ b/.build-wt-028a5dca @@ -0,0 +1 @@ +Subproject commit 028a5dcaddd69b4be80cc595d33d450be1c46e8b diff --git a/MQTT_IMPLEMENTATION.md b/MQTT_IMPLEMENTATION.md index f6f2a6b0..40e56fd3 100644 --- a/MQTT_IMPLEMENTATION.md +++ b/MQTT_IMPLEMENTATION.md @@ -120,6 +120,7 @@ The MQTT bridge uses a slot-based architecture with up to 6 concurrent connectio | `flmesh` | mcmqtt.jntconnections.com:443 | JWT (Ed25519) | WSS | | `inwmesh` | scope.inwmesh.org:8883 | Username/password (per slot via `mqttN.username` / `mqttN.password`) | MQTT over TLS | | `rflab` | mqtt.rflab.io:443 | JWT (Ed25519) | WSS | +| `meshtexas` | mqtt.meshtexas.org:443 | JWT (Ed25519) | WSS | | `custom` | User-configured | Username/Password | MQTT or WSS | | `none` | (disabled) | — | — | diff --git a/src/helpers/MQTTPresets.h b/src/helpers/MQTTPresets.h index 38289d72..b3fcf5a8 100644 --- a/src/helpers/MQTTPresets.h +++ b/src/helpers/MQTTPresets.h @@ -105,7 +105,7 @@ static const char ISRG_ROOT_X1[] PROGMEM = "-----END CERTIFICATE-----\n"; // Number of built-in presets -static const int MQTT_PRESET_COUNT = 26; +static const int MQTT_PRESET_COUNT = 27; // Built-in preset definitions (stored in flash) static const MQTTPresetDef MQTT_PRESETS[MQTT_PRESET_COUNT] = { @@ -141,6 +141,7 @@ static const MQTTPresetDef MQTT_PRESETS[MQTT_PRESET_COUNT] = { { "ipnt.uk", "wss://mqtt.ipnt.uk:443", "mqtt.ipnt.uk", ISRG_ROOT_X1, MQTT_AUTH_JWT, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, { "flmesh", "wss://mcmqtt.jntconnections.com:443", "mcmqtt.jntconnections.com", GTS_ROOT_R4, MQTT_AUTH_JWT, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, { "corecomms", "wss://mqtt.corecomms.net:443/mqtt", "mqtt.corecomms.net", GTS_ROOT_R4, MQTT_AUTH_JWT, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, + { "meshtexas", "wss://mqtt.meshtexas.org:443/mqtt", "mqtt.meshtexas.org", GTS_ROOT_R4, MQTT_AUTH_JWT, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, }; // Find a preset by name, returns nullptr if not found From e06bd756ee78ac74cee3a6527b1a42877f42eadc Mon Sep 17 00:00:00 2001 From: agessaman Date: Sun, 19 Jul 2026 21:25:07 -0700 Subject: [PATCH 07/87] ci: source slim manifests from build output; accept observer-firmware branch name MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Flip gen-slim-manifests.py to its new --bin-dir mode (flasher repo PR #1): the slim OTA manifests are now generated from out/ — the assets actually uploaded to the release — instead of the config.json entries, which the /releases feed migration will convert to github release defs. STATIC_PATH joins the workflow env; it must stay consistent with config.json's staticPath. Also add observer-firmware alongside mqtt-bridge-implementation-flex in both push triggers, ahead of renaming the branch: GitHub does not rewrite branch filters on rename, and a rename with only the old name in the filter silently stops production publishing. The old name is dropped once a publish is verified on the new one. Workflow-only commit (.github/** is in paths-ignore): pushing this triggers no build. --- .github/workflows/build-observer-firmwares.yml | 18 ++++++++++++++---- .github/workflows/sync-flasher-content.yml | 4 ++++ 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/.github/workflows/build-observer-firmwares.yml b/.github/workflows/build-observer-firmwares.yml index 3c13bf0c..6e7beee5 100644 --- a/.github/workflows/build-observer-firmwares.yml +++ b/.github/workflows/build-observer-firmwares.yml @@ -7,7 +7,11 @@ on: workflow_dispatch: push: branches: + # Both names during the flex -> observer-firmware rename: GitHub does not + # rewrite branch filters on rename, so the old name is kept until a + # publish is verified on the new one (then it gets dropped). - mqtt-bridge-implementation-flex + - observer-firmware # Only rebuild firmware when something that affects the binaries changes. # Docs, the changelog, the changelog generator, CI files, and repo-meta # files do not change firmware output — docs/changelog are handled by @@ -38,6 +42,10 @@ env: FIRMWARE_VERSION: v1.16.0 # Rolling release tag that hosts the latest observer binaries. RELEASE_TAG: observer-mqtt-latest + # Download host serving RELEASE_TAG's assets (cloudflare-worker in the flasher + # repo). Baked into the slim OTA manifests' file URLs; must stay consistent + # with config.json's staticPath. + STATIC_PATH: https://observer-fw.gessaman.com jobs: @@ -233,11 +241,13 @@ jobs: BUILD_NUMBER: ${{ needs.enumerate.outputs.build_number }} run: | # Derive the slim per-variant manifests (flasher/v/.json) that the - # firmware fetches for `ota check`/`ota update`, from the just-updated - # config.json (single source of truth), stamping this build's number. - # Then persist the counter so the next run increments from here. + # firmware fetches for `ota check`/`ota update`, from the build output + # in out/ — the assets actually uploaded to the release — stamping this + # build's number. Then persist the counter so the next run increments + # from here. python3 flasher/scripts/gen-slim-manifests.py \ - --config flasher/config.json \ + --bin-dir out \ + --static-path "$STATIC_PATH" \ --out-dir flasher/v \ --base-version "$FIRMWARE_VERSION" \ --build "$BUILD_NUMBER" \ diff --git a/.github/workflows/sync-flasher-content.yml b/.github/workflows/sync-flasher-content.yml index e2bd5033..cd3c8428 100644 --- a/.github/workflows/sync-flasher-content.yml +++ b/.github/workflows/sync-flasher-content.yml @@ -14,7 +14,11 @@ on: workflow_dispatch: push: branches: + # Both names during the flex -> observer-firmware rename: GitHub does not + # rewrite branch filters on rename, so the old name is kept until a + # publish is verified on the new one (then it gets dropped). - mqtt-bridge-implementation-flex + - observer-firmware paths: - '**.md' - 'docs/**' From 80b4e4051c590a87b064584c7645bc75617715f1 Mon Sep 17 00:00:00 2001 From: agessaman Date: Sun, 19 Jul 2026 21:29:51 -0700 Subject: [PATCH 08/87] ci: drop retired mqtt-bridge-implementation-flex trigger name The rename to observer-firmware is verified: the rename's own push event ran both workflows on the new name (branch creation bypasses path filters), sync-flasher-content pushed to the flasher repo, and the production publish completed on observer-firmware. The transition dual trigger is no longer needed. Also update the MQTT_INTERNALS.md prose reference to the old branch name. Workflow+docs-only commit: pushing this triggers no build. --- .github/workflows/build-observer-firmwares.yml | 4 ---- .github/workflows/sync-flasher-content.yml | 4 ---- MQTT_INTERNALS.md | 3 ++- 3 files changed, 2 insertions(+), 9 deletions(-) diff --git a/.github/workflows/build-observer-firmwares.yml b/.github/workflows/build-observer-firmwares.yml index 6e7beee5..3390c23e 100644 --- a/.github/workflows/build-observer-firmwares.yml +++ b/.github/workflows/build-observer-firmwares.yml @@ -7,10 +7,6 @@ on: workflow_dispatch: push: branches: - # Both names during the flex -> observer-firmware rename: GitHub does not - # rewrite branch filters on rename, so the old name is kept until a - # publish is verified on the new one (then it gets dropped). - - mqtt-bridge-implementation-flex - observer-firmware # Only rebuild firmware when something that affects the binaries changes. # Docs, the changelog, the changelog generator, CI files, and repo-meta diff --git a/.github/workflows/sync-flasher-content.yml b/.github/workflows/sync-flasher-content.yml index cd3c8428..c534c2f4 100644 --- a/.github/workflows/sync-flasher-content.yml +++ b/.github/workflows/sync-flasher-content.yml @@ -14,10 +14,6 @@ on: workflow_dispatch: push: branches: - # Both names during the flex -> observer-firmware rename: GitHub does not - # rewrite branch filters on rename, so the old name is kept until a - # publish is verified on the new one (then it gets dropped). - - mqtt-bridge-implementation-flex - observer-firmware paths: - '**.md' diff --git a/MQTT_INTERNALS.md b/MQTT_INTERNALS.md index fbe30d70..426d4b9c 100644 --- a/MQTT_INTERNALS.md +++ b/MQTT_INTERNALS.md @@ -81,7 +81,8 @@ default; a newer, longer one is truncated harmlessly. - **`/mqtt_prefs`** — if the file has the version header it is read directly. Otherwise it is a legacy headerless file and its layout is detected by size: pre-slot (`OldMQTTPrefs`), 3-slot (`ThreeSlotMQTTPrefs`), or the 6-slot layout shipped on - `mqtt-bridge-implementation-flex` (`Legacy6SlotMQTTPrefs`). Each is field-copied into + `observer-firmware` back when it was named `mqtt-bridge-implementation-flex` + (`Legacy6SlotMQTTPrefs`). Each is field-copied into the current compact `MQTTPrefs` and re-saved with the version header — which also drops the vestigial `_legacy_*` fields the flex layout carried mid-struct. This is a one-time rewrite; every deployed device performs it on its first boot of versioned From 26db31f62567314f383e4c7bed38c9c2c5b3063d Mon Sep 17 00:00:00 2001 From: agessaman Date: Sun, 19 Jul 2026 21:46:16 -0700 Subject: [PATCH 09/87] ci: stop rewriting flasher config.json; sync release notes to the release body MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The flasher's versions become feed-driven (/releases on the firmware proxy Worker): config.json's observer entries turn into github release defs with no embedded filenames, so update-firmware.py would find zero matches and exit nonzero — remove the call before the config conversion lands. Changelog notes now ride the release body via a non-fatal 'gh release edit --notes-file firmware-notes.html' in the publish step. MUST land before the flasher config.json conversion, or the next build goes red at Update Flasher Config. Workflow-only commit: pushing this triggers no build. --- .../workflows/build-observer-firmwares.yml | 22 ++++++++++++------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/.github/workflows/build-observer-firmwares.yml b/.github/workflows/build-observer-firmwares.yml index 3390c23e..a33ac359 100644 --- a/.github/workflows/build-observer-firmwares.yml +++ b/.github/workflows/build-observer-firmwares.yml @@ -189,6 +189,13 @@ jobs: # not published as release assets (and they'd break the hash-based prune). gh release upload "$RELEASE_TAG" $(find out -maxdepth 1 -type f ! -name '*.partsig') --clobber + # Keep the release body in sync with the repo's notes source: the + # /releases feed serves it as the flasher dropdown's changelog + # (config.json no longer carries per-version notes). Non-fatal — + # stale notes beat a red build whose binaries are already live. + gh release edit "$RELEASE_TAG" --notes-file firmware-notes.html \ + || echo "WARNING: release notes sync failed" >&2 + # Prune old assets, but RETAIN the most recent KEEP_BUILDS build hashes # (not just the current one). The flasher SPA reads config.json once at # page load and the embedded git short-hash is what links it to assets; @@ -226,11 +233,10 @@ jobs: token: ${{ secrets.FLASHER_DISPATCH_TOKEN }} path: flasher - - name: Update Flasher Config - run: | - python3 flasher/scripts/update-firmware.py \ - "${{ steps.sha.outputs.short }}" \ - "$GITHUB_WORKSPACE/firmware-notes.html" + # update-firmware.py is no longer called here: config.json's observer + # entries are github release defs (no embedded filenames to rewrite), the + # flasher's versions come from the Worker's /releases feed, and the + # changelog notes ride the release body (see the publish step). - name: Generate Slim Per-Variant Manifests + Persist Build Counter env: @@ -289,9 +295,9 @@ jobs: - name: Commit & Push Flasher Config working-directory: flasher run: | - # Stage everything: update-firmware.py edits config.json in place, but the - # slim manifests (v/*.json) and the build counter can be NEW files, which - # `commit -am` would miss — so add -A and check the staged diff. + # Stage everything: the slim manifests (v/*.json) and the build counter + # can be NEW files, which `commit -am` would miss — so add -A and check + # the staged diff. (config.json is no longer rewritten per build.) git add -A if git diff --cached --quiet; then echo "No flasher changes to commit." From 5cb3fc18670faa310038a06cfdf98cf9e9e287c4 Mon Sep 17 00:00:00 2001 From: agessaman Date: Sun, 19 Jul 2026 23:45:28 -0700 Subject: [PATCH 10/87] feat(mqtt): add mesh-chaun14 and wcmesh presets Support "{pubkey}" username sentinel so brokers can auth with the device public key without enlarging the prefs username field. --- src/helpers/CommonCLI_Observer.cpp | 6 +++++ src/helpers/MQTTPresets.h | 34 +++++++++++++++++++++++++---- src/helpers/bridges/MQTTBridge.cpp | 35 +++++++++++++++++++----------- 3 files changed, 58 insertions(+), 17 deletions(-) diff --git a/src/helpers/CommonCLI_Observer.cpp b/src/helpers/CommonCLI_Observer.cpp index 002d6a15..37982da4 100644 --- a/src/helpers/CommonCLI_Observer.cpp +++ b/src/helpers/CommonCLI_Observer.cpp @@ -390,6 +390,12 @@ bool CommonCLI::handleObserverSetCmd(uint32_t sender_timestamp, const char* conf } else if (p && p->topic_style == MQTT_TOPIC_MESHCORE && (strlen(_mqtt_prefs.mqtt_iata) == 0 || strcmp(_mqtt_prefs.mqtt_iata, "XXX") == 0)) { sprintf(reply, "OK - slot %d preset: %s (run 'set mqtt.iata ' to publish)", slot + 1, preset_name); + } else if (p && mqttPresetNeedsSlotPassword(p) && + _mqtt_prefs.mqtt_slot_password[slot][0] == '\0' && + !mqttPresetNeedsSlotUsername(p)) { + sprintf(reply, + "OK - slot %d preset: %s (run 'set mqtt%d.password ' to connect)", + slot + 1, preset_name, slot + 1); } else if (p && mqttPresetNeedsSlotCredentials(p) && (_mqtt_prefs.mqtt_slot_username[slot][0] == '\0' || _mqtt_prefs.mqtt_slot_password[slot][0] == '\0')) { diff --git a/src/helpers/MQTTPresets.h b/src/helpers/MQTTPresets.h index b3fcf5a8..bf263799 100644 --- a/src/helpers/MQTTPresets.h +++ b/src/helpers/MQTTPresets.h @@ -40,10 +40,32 @@ struct MQTTPresetDef { const char* userpass_password; // MQTT_AUTH_USERPASS: embedded password, or nullptr to use mqttN.password }; -// True when preset uses MQTT_AUTH_USERPASS but credentials come from slot prefs (mqttN.username/password). -static inline bool mqttPresetNeedsSlotCredentials(const MQTTPresetDef* preset) { +// Sentinel: resolve MQTT username from device public-key hex at connect time. +// Braces match topic placeholders ({device}/{iata}); never send this string to the broker. +static const char MQTT_USERPASS_USERNAME_PUBKEY[] = "{pubkey}"; + +static inline bool mqttPresetUsesDevicePubkeyUsername(const MQTTPresetDef* preset) { return preset && preset->auth_type == MQTT_AUTH_USERPASS && - (!preset->userpass_username || !preset->userpass_password); + preset->userpass_username && + strcmp(preset->userpass_username, MQTT_USERPASS_USERNAME_PUBKEY) == 0; +} + +// True when USERPASS username must come from mqttN.username (null embedded username). +// "{pubkey}" is an embedded sentinel, so it does not need a slot username. +static inline bool mqttPresetNeedsSlotUsername(const MQTTPresetDef* preset) { + return preset && preset->auth_type == MQTT_AUTH_USERPASS && + !preset->userpass_username; +} + +// True when USERPASS password must come from mqttN.password (null embedded password). +static inline bool mqttPresetNeedsSlotPassword(const MQTTPresetDef* preset) { + return preset && preset->auth_type == MQTT_AUTH_USERPASS && + !preset->userpass_password; +} + +// True when preset uses MQTT_AUTH_USERPASS but at least one credential comes from slot prefs. +static inline bool mqttPresetNeedsSlotCredentials(const MQTTPresetDef* preset) { + return mqttPresetNeedsSlotUsername(preset) || mqttPresetNeedsSlotPassword(preset); } // Google Trust Services - GTS Root R4 (used by LetsMesh Analyzer) @@ -105,7 +127,7 @@ static const char ISRG_ROOT_X1[] PROGMEM = "-----END CERTIFICATE-----\n"; // Number of built-in presets -static const int MQTT_PRESET_COUNT = 27; +static const int MQTT_PRESET_COUNT = 29; // Built-in preset definitions (stored in flash) static const MQTTPresetDef MQTT_PRESETS[MQTT_PRESET_COUNT] = { @@ -142,6 +164,10 @@ static const MQTTPresetDef MQTT_PRESETS[MQTT_PRESET_COUNT] = { { "flmesh", "wss://mcmqtt.jntconnections.com:443", "mcmqtt.jntconnections.com", GTS_ROOT_R4, MQTT_AUTH_JWT, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, { "corecomms", "wss://mqtt.corecomms.net:443/mqtt", "mqtt.corecomms.net", GTS_ROOT_R4, MQTT_AUTH_JWT, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, { "meshtexas", "wss://mqtt.meshtexas.org:443/mqtt", "mqtt.meshtexas.org", GTS_ROOT_R4, MQTT_AUTH_JWT, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, + // Username is device pubkey hex at connect; password from mqttN.password. No TLS. + { "mesh-chaun14", "mqtt://mqtt.mesh.chaun14.fr:1884", nullptr, nullptr, MQTT_AUTH_USERPASS, MQTT_TOPIC_MESHCORE, 0, true, 60, MQTT_USERPASS_USERNAME_PUBKEY, nullptr }, + // LetsMesh-compatible JWT; TLS is Let's Encrypt (ISRG Root X1), not GTS. + { "wcmesh", "wss://mqtt.wcmesh.com:443", "mqtt.wcmesh.com", ISRG_ROOT_X1, MQTT_AUTH_JWT, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, }; // Find a preset by name, returns nullptr if not found diff --git a/src/helpers/bridges/MQTTBridge.cpp b/src/helpers/bridges/MQTTBridge.cpp index 9a9d4628..2066f9c6 100644 --- a/src/helpers/bridges/MQTTBridge.cpp +++ b/src/helpers/bridges/MQTTBridge.cpp @@ -1395,10 +1395,19 @@ void MQTTBridge::setupSlot(int index) { slot.client->setCredentials(_jwt_username, slot.auth_token); } } else if (slot.preset->auth_type == MQTT_AUTH_USERPASS) { - if (slot.preset->userpass_username && slot.preset->userpass_password) { - slot.client->setCredentials(slot.preset->userpass_username, slot.preset->userpass_password); - } else if (strlen(slot.username) > 0) { - slot.client->setCredentials(slot.username, slot.password); + const char* user = nullptr; + const char* pass = slot.preset->userpass_password + ? slot.preset->userpass_password + : slot.password; + if (mqttPresetUsesDevicePubkeyUsername(slot.preset)) { + user = _device_id; // never send "{pubkey}" literally + } else if (slot.preset->userpass_username) { + user = slot.preset->userpass_username; + } else if (slot.username[0] != '\0') { + user = slot.username; + } + if (user && user[0] != '\0' && pass && pass[0] != '\0') { + slot.client->setCredentials(user, pass); } } } else { @@ -2310,15 +2319,15 @@ bool MQTTBridge::isSlotReady(int index, char* reason_buf, size_t reason_size) co return false; } } - if (mqttPresetNeedsSlotCredentials(slot.preset)) { - if (_obs->mqtt_slot_username[index][0] == '\0') { - if (reason_buf) snprintf(reason_buf, reason_size, "set mqtt%d.username ", index + 1); - return false; - } - if (_obs->mqtt_slot_password[index][0] == '\0') { - if (reason_buf) snprintf(reason_buf, reason_size, "set mqtt%d.password ", index + 1); - return false; - } + if (mqttPresetNeedsSlotUsername(slot.preset) && + _obs->mqtt_slot_username[index][0] == '\0') { + if (reason_buf) snprintf(reason_buf, reason_size, "set mqtt%d.username ", index + 1); + return false; + } + if (mqttPresetNeedsSlotPassword(slot.preset) && + _obs->mqtt_slot_password[index][0] == '\0') { + if (reason_buf) snprintf(reason_buf, reason_size, "set mqtt%d.password ", index + 1); + return false; } } else { // Custom slot without a topic template uses meshcore format, needs IATA From 59ca834fa0a3c901c886792c20fa17d674288042 Mon Sep 17 00:00:00 2001 From: agessaman Date: Mon, 20 Jul 2026 08:10:57 -0700 Subject: [PATCH 11/87] feat(mqtt): implement neighbor discovery and management features Added functionality for neighbor discovery, including the ability to store and manage neighbor information. This includes methods for sending discovery requests, handling responses, and updating the neighbor table. The changes enhance the mesh networking capabilities by allowing devices to discover and interact with nearby peers more effectively. Updated relevant structures and methods in MyMesh.cpp and MyMesh.h to support these features. --- examples/simple_repeater/MyMesh.cpp | 2 +- examples/simple_room_server/MyMesh.cpp | 550 +++++++++++++++++- examples/simple_room_server/MyMesh.h | 53 +- variants/heltec_t190/platformio.ini | 1 + variants/heltec_tracker_v2/platformio.ini | 2 + variants/heltec_v3/platformio.ini | 2 + variants/heltec_v4/platformio.ini | 2 + variants/lilygo_t3s3/platformio.ini | 1 + variants/lilygo_tbeam_1w/platformio.ini | 1 + variants/lilygo_tbeam_SX1262/platformio.ini | 1 + variants/lilygo_tbeam_SX1276/platformio.ini | 1 + .../platformio.ini | 1 + variants/lilygo_tlora_v2_1/platformio.ini | 1 + variants/rak3112/platformio.ini | 1 + variants/station_g2/platformio.ini | 1 + variants/xiao_s3_wio/platformio.ini | 1 + 16 files changed, 616 insertions(+), 5 deletions(-) diff --git a/examples/simple_repeater/MyMesh.cpp b/examples/simple_repeater/MyMesh.cpp index 098d60b1..8c4b67d6 100644 --- a/examples/simple_repeater/MyMesh.cpp +++ b/examples/simple_repeater/MyMesh.cpp @@ -1282,7 +1282,7 @@ void MyMesh::formatNeighborsReply(char *reply) { // add next neighbour uint32_t secs_ago = getRTCClock()->getCurrentTime() - neighbour->heard_timestamp; - sprintf(dp, "%s:%d:%d", hex, secs_ago, neighbour->snr); + sprintf(dp, "%s:%u:%d", hex, secs_ago, neighbour->snr); while (*dp) dp++; // find end of string } diff --git a/examples/simple_room_server/MyMesh.cpp b/examples/simple_room_server/MyMesh.cpp index 81389ab1..535f0822 100644 --- a/examples/simple_room_server/MyMesh.cpp +++ b/examples/simple_room_server/MyMesh.cpp @@ -1,4 +1,5 @@ #include "MyMesh.h" +#include #include #define REPLY_DELAY_MILLIS 1500 @@ -418,7 +419,21 @@ void MyMesh::onAnonDataRecv(mesh::Packet *packet, const uint8_t *secret, const m int MyMesh::searchPeersByHash(const uint8_t *hash) { int n = 0; - for (int i = 0; i < acl.getNumClients(); i++) { +#if defined(WITH_MQTT_NEIGHBORS) + if (neighbor_discover_active) { + for (int i = 0; i < neighbor_discover_count && n < MAX_CLIENTS; i++) { + auto& nb = neighbours[neighbor_discover[i].neighbour_idx]; + // ACL clients already have a matching peer entry and shared secret. Adding + // a second overlay entry would decrypt first and intercept their normal + // CLI/request traffic for the duration of discovery. + if (acl.getClient(nb.id.pub_key, PUB_KEY_SIZE) != nullptr) continue; + if (nb.heard_timestamp > 0 && nb.id.isHashMatch(hash)) { + matching_peer_indexes[n++] = NEIGHBOR_DISCOVER_PEER_BASE + i; + } + } + } +#endif + for (int i = 0; i < acl.getNumClients() && n < MAX_CLIENTS; i++) { if (acl.getClientByIdx(i)->id.isHashMatch(hash)) { matching_peer_indexes[n++] = i; // store the INDEXES of matching contacts (for subsequent 'peer' methods) } @@ -428,6 +443,15 @@ int MyMesh::searchPeersByHash(const uint8_t *hash) { void MyMesh::getPeerSharedSecret(uint8_t *dest_secret, int peer_idx) { int i = matching_peer_indexes[peer_idx]; +#if defined(WITH_MQTT_NEIGHBORS) + if (neighbor_discover_active && i >= NEIGHBOR_DISCOVER_PEER_BASE) { + int oi = i - NEIGHBOR_DISCOVER_PEER_BASE; + if (oi >= 0 && oi < neighbor_discover_count) { + self_id.calcSharedSecret(dest_secret, neighbours[neighbor_discover[oi].neighbour_idx].id); + return; + } + } +#endif if (i >= 0 && i < acl.getNumClients()) { // lookup pre-calculated shared_secret memcpy(dest_secret, acl.getClientByIdx(i)->shared_secret, PUB_KEY_SIZE); @@ -439,11 +463,35 @@ void MyMesh::getPeerSharedSecret(uint8_t *dest_secret, int peer_idx) { void MyMesh::onPeerDataRecv(mesh::Packet *packet, uint8_t type, int sender_idx, const uint8_t *secret, uint8_t *data, size_t len) { int i = matching_peer_indexes[sender_idx]; +#if defined(WITH_MQTT_NEIGHBORS) + if (neighbor_discover_active && i >= NEIGHBOR_DISCOVER_PEER_BASE) { + int oi = i - NEIGHBOR_DISCOVER_PEER_BASE; + if (type == PAYLOAD_TYPE_RESPONSE && oi >= 0 && oi < neighbor_discover_count) { + handleNeighborDiscoverResponse(oi, data, len); + } + return; + } +#endif if (i < 0 || i >= acl.getNumClients()) { // get from our known_clients table (sender SHOULD already be known in this context) MESH_DEBUG_PRINTLN("onPeerDataRecv: invalid peer idx: %d", i); return; } auto client = acl.getClientByIdx(i); + +#if defined(WITH_MQTT_NEIGHBORS) + // Neighbors that are already ACL clients are intentionally not added to the + // temporary peer overlay above. Consume only a matching discovery response + // here, then leave every other payload on the normal ACL path. + if (neighbor_discover_active && type == PAYLOAD_TYPE_RESPONSE) { + for (int oi = 0; oi < neighbor_discover_count; oi++) { + auto& nb = neighbours[neighbor_discover[oi].neighbour_idx]; + if (client->id.matches(nb.id) && handleNeighborDiscoverResponse(oi, data, len)) { + return; + } + } + } +#endif + if (type == PAYLOAD_TYPE_TXT_MSG && len > 5) { // a CLI command or new Post uint32_t sender_timestamp; memcpy(&sender_timestamp, data, 4); // timestamp (by sender's RTC clock - which could be wrong) @@ -633,6 +681,107 @@ void MyMesh::onAckRecv(mesh::Packet *packet, uint32_t ack_crc) { } } +#if defined(WITH_MQTT_NEIGHBORS) + +#define CTL_TYPE_NODE_DISCOVER_REQ 0x80 +#define CTL_TYPE_NODE_DISCOVER_RESP 0x90 + +void MyMesh::putNeighbour(const mesh::Identity &id, uint32_t timestamp, float snr) { + // find existing neighbour, else use least recently updated + uint32_t oldest_timestamp = 0xFFFFFFFF; + NeighbourInfo *neighbour = &neighbours[0]; + for (int i = 0; i < MAX_NEIGHBOURS; i++) { + // if neighbour already known, we should update it + if (id.matches(neighbours[i].id)) { + neighbour = &neighbours[i]; + break; + } + + // otherwise we should update the least recently updated neighbour + if (neighbours[i].heard_timestamp < oldest_timestamp) { + neighbour = &neighbours[i]; + oldest_timestamp = neighbour->heard_timestamp; + } + } + + // update neighbour info + neighbour->id = id; + neighbour->advert_timestamp = timestamp; + neighbour->heard_timestamp = getRTCClock()->getCurrentTime(); + neighbour->snr = (int8_t)(snr * 4); +} + +static bool isShare(const mesh::Packet *packet) { + if (packet->hasTransportCodes()) { + return packet->transport_codes[0] == 0 && packet->transport_codes[1] == 0; // codes { 0, 0 } means 'send to nowhere' + } + return false; +} + +void MyMesh::onAdvertRecv(mesh::Packet *packet, const mesh::Identity &id, uint32_t timestamp, + const uint8_t *app_data, size_t app_data_len) { + mesh::Mesh::onAdvertRecv(packet, id, timestamp, app_data, app_data_len); // chain to super impl + + // if this a zero hop advert (and not via 'Share'), add it to neighbours + if (packet->getPathHashCount() == 0 && !isShare(packet)) { + AdvertDataParser parser(app_data, app_data_len); + if (parser.isValid() && parser.getType() == ADV_TYPE_REPEATER) { // just keep neigbouring Repeaters + putNeighbour(id, timestamp, packet->getSNR()); + } + } +} + +void MyMesh::onControlDataRecv(mesh::Packet* packet) { + uint8_t type = packet->payload[0] & 0xF0; // just test upper 4 bits + // NOTE: a room server is ADV_TYPE_ROOM, so it does NOT answer node-discover + // requests (which filter for repeaters). It only records repeater responses + // to its own discovery, to build the neighbour table. + if (type == CTL_TYPE_NODE_DISCOVER_RESP && packet->payload_len >= 6) { + uint8_t node_type = packet->payload[0] & 0x0F; + if (node_type != ADV_TYPE_REPEATER) { + return; + } + if (packet->payload_len < 6 + PUB_KEY_SIZE) { + MESH_DEBUG_PRINTLN("onControlDataRecv: DISCOVER_RESP pubkey too short: %d", (uint32_t)packet->payload_len); + return; + } + + if (pending_discover_tag == 0 || millisHasNowPassed(pending_discover_until)) { + pending_discover_tag = 0; + return; + } + uint32_t tag; + memcpy(&tag, &packet->payload[2], 4); + if (tag != pending_discover_tag) { + return; + } + + mesh::Identity id(&packet->payload[6]); + if (id.matches(self_id)) { + return; + } + putNeighbour(id, getRTCClock()->getCurrentTime(), packet->getSNR()); + } +} + +void MyMesh::sendNodeDiscoverReq() { + uint8_t data[10]; + data[0] = CTL_TYPE_NODE_DISCOVER_REQ; // prefix_only=0 + data[1] = (1 << ADV_TYPE_REPEATER); + getRNG()->random(&data[2], 4); // tag + memcpy(&pending_discover_tag, &data[2], 4); + pending_discover_until = futureMillis(60000); + uint32_t since = 0; + memcpy(&data[6], &since, 4); + + auto pkt = createControlData(data, sizeof(data)); + if (pkt) { + sendZeroHop(pkt); + } +} + +#endif // WITH_MQTT_NEIGHBORS + MyMesh::MyMesh(mesh::MainBoard &board, mesh::Radio &radio, mesh::MillisecondClock &ms, mesh::RNG &rng, mesh::RTCClock &rtc, mesh::MeshTables &tables) : mesh::Mesh(radio, ms, rng, rtc, *createObserverPacketManager(32), tables), @@ -702,6 +851,18 @@ MyMesh::MyMesh(mesh::MainBoard &board, mesh::Radio &radio, mesh::MillisecondCloc memset(posts, 0, sizeof(posts)); _num_posted = _num_post_pushes = 0; +#if defined(WITH_MQTT_NEIGHBORS) + pending_discover_tag = 0; + pending_discover_until = 0; + neighbor_discover_count = 0; + neighbor_discover_active = false; + neighbor_table_refresh_active = false; + neighbor_table_refresh_periodic = false; + neighbor_discover_until = 0; + next_neighbors_publish = 0; + self_scopes_buf[0] = 0; +#endif + memset(default_scope.key, 0, sizeof(default_scope.key)); } @@ -936,6 +1097,65 @@ void MyMesh::clearStats() { ((SimpleMeshTables *)getTables())->resetStats(); } +void MyMesh::formatNeighborsReply(char *reply) { +#if defined(WITH_MQTT_NEIGHBORS) + char *dp = reply; + + // create copy of neighbours list, skipping empty entries so we can sort it separately from main list + int16_t neighbours_count = 0; + NeighbourInfo* sorted_neighbours[MAX_NEIGHBOURS]; + for (int i = 0; i < MAX_NEIGHBOURS; i++) { + auto neighbour = &neighbours[i]; + if (neighbour->heard_timestamp > 0) { + sorted_neighbours[neighbours_count] = neighbour; + neighbours_count++; + } + } + + // sort neighbours newest to oldest + std::sort(sorted_neighbours, sorted_neighbours + neighbours_count, [](const NeighbourInfo* a, const NeighbourInfo* b) { + return a->heard_timestamp > b->heard_timestamp; // desc + }); + + for (int i = 0; i < neighbours_count && dp - reply < 134; i++) { + NeighbourInfo *neighbour = sorted_neighbours[i]; + + // add new line if not first item + if (i > 0) *dp++ = '\n'; + + char hex[10]; + // get 4 bytes of neighbour id as hex + mesh::Utils::toHex(hex, neighbour->id.pub_key, 4); + + // add next neighbour + uint32_t secs_ago = getRTCClock()->getCurrentTime() - neighbour->heard_timestamp; + sprintf(dp, "%s:%u:%d", hex, secs_ago, neighbour->snr); + while (*dp) + dp++; // find end of string + } + if (dp == reply) { // no neighbours, need empty response + strcpy(dp, "-none-"); + dp += 6; + } + *dp = 0; // null terminator +#else + strcpy(reply, "not supported"); +#endif +} + +void MyMesh::removeNeighbor(const uint8_t *pubkey, int key_len) { +#if defined(WITH_MQTT_NEIGHBORS) + for (int i = 0; i < MAX_NEIGHBOURS; i++) { + NeighbourInfo *neighbour = &neighbours[i]; + if (memcmp(neighbour->id.pub_key, pubkey, key_len) == 0) { + neighbours[i] = NeighbourInfo(); // clear neighbour entry + } + } +#else + (void)pubkey; (void)key_len; +#endif +} + void MyMesh::formatStatsReply(char *reply) { StatsFormatHelper::formatCoreStats(reply, board, *_ms, _err_flags, _mgr); } @@ -1027,6 +1247,47 @@ void MyMesh::handleCommand(uint32_t sender_timestamp, char *command, char *reply Serial.printf("\n"); } reply[0] = 0; +#if defined(WITH_MQTT_NEIGHBORS) + } else if (memcmp(command, "discover.neighbors", 18) == 0) { + const char* sub = command + 18; + while (*sub == ' ') sub++; + if (*sub != 0) { + strcpy(reply, "Err - discover.neighbors has no options"); + } else { + sendNodeDiscoverReq(); + strcpy(reply, "OK - Discover sent"); + } + } else if (memcmp(command, "discover.scopes", 15) == 0) { + const char* sub = command + 15; + while (*sub == ' ') sub++; + if (*sub != 0) { + strcpy(reply, "Err - discover.scopes has no options"); + } else if (pending_discover_tag != 0 && + !millisHasNowPassed(pending_discover_until) && + !neighbor_discover_active) { + // Reuse an in-flight zero-hop discovery rather than querying the old + // neighbor table. An explicit command takes ownership of a periodic + // refresh so `mqtt.neighbors off` cannot cancel this one-shot request. + // Validate up front so queuing cannot report OK for a request that is + // certain to fail once the discovery window closes. + if (!neighborDiscoverReady(reply)) { + // reply already set + } else { + neighbor_table_refresh_active = true; + neighbor_table_refresh_periodic = false; + long remaining_ms = (long)(pending_discover_until - futureMillis(0)); + unsigned remaining_secs = remaining_ms > 0 + ? (unsigned)(((unsigned long)remaining_ms + 999UL) / 1000UL) : 0; + sprintf(reply, "OK - scopes queued (%us discovery remaining)", remaining_secs); + MESH_DEBUG_PRINTLN("Neighbor scopes queued behind active discovery (%us remaining)", remaining_secs); + } + } else if (!startNeighborDiscover(reply)) { + // reply already set + } +#elif defined(WITH_MQTT_BRIDGE) + } else if (memcmp(command, "discover.scopes", 15) == 0) { + strcpy(reply, "Err - not supported (requires PSRAM)"); +#endif } else{ _cli.handleCommand(sender_timestamp, command, reply); // common CLI commands } @@ -1129,4 +1390,291 @@ void MyMesh::loop() { #ifdef WITH_MQTT_BRIDGE _alerter.onLoop(now); #endif + +#if defined(WITH_MQTT_NEIGHBORS) + bool periodic_neighbors_enabled = _cli.getObserverPrefs()->mqtt_neighbors_enabled; + if (neighbor_discover_active) { + loopNeighborDiscover(); + } else if (neighbor_table_refresh_active) { + if (neighbor_table_refresh_periodic && !periodic_neighbors_enabled) { + // Turning periodic publishing off cancels a refresh that has not yet + // reached the scope-query/publish phase. Re-enabling starts fresh. + // pending_discover_tag is left alone: this refresh may have joined a + // discover.neighbors the user started, and clearing the tag would drop + // its remaining responses. An unused tag expires on its own. + neighbor_table_refresh_active = false; + neighbor_table_refresh_periodic = false; + next_neighbors_publish = 0; + } else if (pending_discover_tag == 0 || millisHasNowPassed(pending_discover_until)) { + // The zero-hop discover.neighbors response window is complete. Freeze the + // refreshed table into the scope-query overlay, then publish when those + // per-neighbor requests finish. + bool was_periodic = neighbor_table_refresh_periodic; + pending_discover_tag = 0; + neighbor_table_refresh_active = false; + neighbor_table_refresh_periodic = false; + char tmp_reply[80]; + const char* origin_str = was_periodic ? "periodic" : "manual"; + if (startNeighborDiscover(tmp_reply)) { + MESH_DEBUG_PRINTLN("MQTT %s %s", origin_str, tmp_reply); + } else { + if (periodic_neighbors_enabled) { + // Avoid retrying a failed refresh on every loop iteration. + next_neighbors_publish = futureMillis(_cli.getObserverPrefs()->mqtt_neighbors_interval); + } + MESH_DEBUG_PRINTLN("MQTT %s neighbor scope discovery failed: %s", origin_str, tmp_reply); + } + } + } else if (periodic_neighbors_enabled && bridge && bridge->isRunning()) { + if (next_neighbors_publish == 0 || + (next_neighbors_publish != 0 && millisHasNowPassed(next_neighbors_publish))) { + // Refresh the zero-hop neighbor cache first (the same operation as the + // discover.neighbors CLI command). Scope queries begin after its 60-second + // collection window instead of publishing a stale cached table. + if (pending_discover_tag == 0 || millisHasNowPassed(pending_discover_until)) { + pending_discover_tag = 0; + sendNodeDiscoverReq(); + MESH_DEBUG_PRINTLN("MQTT periodic neighbor table refresh started"); + } else { + MESH_DEBUG_PRINTLN("MQTT periodic refresh joined active neighbor discovery"); + } + neighbor_table_refresh_active = true; + neighbor_table_refresh_periodic = true; + } + } + + // Report the periodic-neighbors schedule for `get mqtt.status`. The timer lives + // here, so the wrap-safe millis math stays on this side of the handoff. + if (bridge) { + if (neighbor_discover_active || neighbor_table_refresh_active) { + bridge->setNeighborsSchedule(MQTTBridge::NBR_ACTIVE, 0); + } else if (next_neighbors_publish == 0 || millisHasNowPassed(next_neighbors_publish)) { + bridge->setNeighborsSchedule(MQTTBridge::NBR_DUE, 0); + } else { + long remaining_ms = (long)(next_neighbors_publish - futureMillis(0)); + uint32_t remaining_secs = remaining_ms > 0 ? (uint32_t)(remaining_ms / 1000) : 0; + bridge->setNeighborsSchedule(MQTTBridge::NBR_SCHEDULED, remaining_secs); + } + } +#endif } + +#if defined(WITH_MQTT_NEIGHBORS) + +#include "helpers/MQTTMessageBuilder.h" +#if defined(ESP_PLATFORM) +#include +#endif + +void MyMesh::getLocalScopes(char* buf, size_t len) { + if (!buf || len == 0) return; + buf[0] = 0; + region_map.exportNamesTo(buf, (int)len, REGION_DENY_FLOOD); +} + +bool MyMesh::sendAnonRegionsReq(const mesh::Identity& target, uint32_t& tag) { + uint8_t secret[PUB_KEY_SIZE]; + self_id.calcSharedSecret(secret, target); + + tag = getRTCClock()->getCurrentTimeUnique(); + uint8_t inner[6]; + memcpy(inner, &tag, 4); + inner[4] = 0x01; // ANON_REQ_TYPE_REGIONS + inner[5] = 0x00; // zero-hop reply path + + mesh::Packet* pkt = createAnonDatagram(PAYLOAD_TYPE_ANON_REQ, self_id, target, secret, inner, sizeof(inner)); + if (!pkt) return false; + sendDirect(pkt, NULL, 0, 0); + return true; +} + +bool MyMesh::handleNeighborDiscoverResponse(int overlay_idx, const uint8_t* data, size_t len) { + if (overlay_idx < 0 || overlay_idx >= neighbor_discover_count) return false; + NeighborDiscoverEntry& entry = neighbor_discover[overlay_idx]; + if (entry.status != ND_PENDING || len < 8) return false; + + uint32_t tag; + memcpy(&tag, data, 4); + if (tag != entry.tag) return false; + + size_t scope_len = len - 8; + if (scope_len >= sizeof(entry.scopes)) { + scope_len = sizeof(entry.scopes) - 1; + } + memcpy(entry.scopes, &data[8], scope_len); + entry.scopes[scope_len] = 0; + entry.status = ND_RESPONDED; + return true; +} + +static bool neighborPublishEntryComesBefore( + const MQTTMessageBuilder::NeighborsMessageEntry& lhs, + const MQTTMessageBuilder::NeighborsMessageEntry& rhs) { + if (lhs.heard_secs_ago != rhs.heard_secs_ago) { + return lhs.heard_secs_ago < rhs.heard_secs_ago; // newer first + } + if (lhs.snr != rhs.snr) { + return lhs.snr > rhs.snr; // stronger first when equally recent + } + return strcmp(lhs.pubkey_hex, rhs.pubkey_hex) < 0; +} + +void MyMesh::finishNeighborDiscover() { + getLocalScopes(self_scopes_buf, sizeof(self_scopes_buf)); + + char self_pubkey_hex[65]; + mesh::Utils::toHex(self_pubkey_hex, self_id.pub_key, PUB_KEY_SIZE); + + char origin[32]; + MQTTBridge::getEffectiveMqttOrigin(&_prefs, _cli.getObserverPrefs(), origin, sizeof(origin)); + + char timestamp[40]; + MQTTMessageBuilder::formatIsoTimestampForMqtt(getRTCClock()->getCurrentTime(), 0, nullptr, timestamp, sizeof(timestamp)); + + char pubkey_hex[MAX_NEIGHBOURS][65]; + MQTTMessageBuilder::NeighborsMessageEntry entries[MAX_NEIGHBOURS]; + uint32_t now_secs = getRTCClock()->getCurrentTime(); + + for (int i = 0; i < neighbor_discover_count; i++) { + auto& nb = neighbours[neighbor_discover[i].neighbour_idx]; + mesh::Utils::toHex(pubkey_hex[i], nb.id.pub_key, PUB_KEY_SIZE); + entries[i].pubkey_hex = pubkey_hex[i]; + entries[i].snr = nb.snr / 4.0f; + entries[i].heard_secs_ago = (nb.heard_timestamp > 0 && now_secs >= nb.heard_timestamp) + ? (now_secs - nb.heard_timestamp) : 0; + entries[i].scopes = neighbor_discover[i].scopes; + switch (neighbor_discover[i].status) { + case ND_RESPONDED: entries[i].status = "responded"; break; + case ND_SEND_FAILED: entries[i].status = "send_failed"; break; + default: entries[i].status = "timeout"; break; + } + } + + // The JSON builder drops entries from the tail when the fixed MQTT buffer is + // full, so order the useful/recent neighbors first. Insertion sort is small, + // allocation-free, and sufficient for MAX_NEIGHBOURS (currently at most 50). + for (int i = 1; i < neighbor_discover_count; i++) { + MQTTMessageBuilder::NeighborsMessageEntry entry = entries[i]; + int j = i; + while (j > 0 && neighborPublishEntryComesBefore(entry, entries[j - 1])) { + entries[j] = entries[j - 1]; + j--; + } + entries[j] = entry; + } + +#if defined(ESP_PLATFORM) + char* json_buf = (char*)heap_caps_malloc(MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE, MALLOC_CAP_SPIRAM); +#else + char* json_buf = (char*)malloc(MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE); +#endif + if (!json_buf) { + neighbor_discover_active = false; + neighbor_discover_count = 0; + if (_cli.getObserverPrefs()->mqtt_neighbors_enabled) { + next_neighbors_publish = futureMillis(_cli.getObserverPrefs()->mqtt_neighbors_interval); + } + return; + } + + DynamicJsonDocument doc(MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE); + int json_len = MQTTMessageBuilder::buildNeighborsMessage( + doc, origin, self_pubkey_hex, timestamp, self_scopes_buf, + entries, neighbor_discover_count, + json_buf, MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE); + + if (json_len > 0 && bridge) { + bridge->requestPublishNeighbors(json_buf, (size_t)json_len); + } + +#if defined(ESP_PLATFORM) + heap_caps_free(json_buf); +#else + free(json_buf); +#endif + + neighbor_discover_active = false; + neighbor_discover_count = 0; + if (_cli.getObserverPrefs()->mqtt_neighbors_enabled) { + next_neighbors_publish = futureMillis(_cli.getObserverPrefs()->mqtt_neighbors_interval); + } +} + +void MyMesh::loopNeighborDiscover() { + if (!neighbor_discover_active) return; + + bool all_done = true; + for (int i = 0; i < neighbor_discover_count; i++) { + if (neighbor_discover[i].status == ND_PENDING) { + all_done = false; + break; + } + } + + if (!all_done && !millisHasNowPassed(neighbor_discover_until)) { + return; + } + + if (!all_done) { + for (int i = 0; i < neighbor_discover_count; i++) { + if (neighbor_discover[i].status == ND_PENDING) { + neighbor_discover[i].status = ND_TIMEOUT; + } + } + } + finishNeighborDiscover(); +} + +bool MyMesh::neighborDiscoverReady(char* reply) { +#if defined(ESP_PLATFORM) + if (!psramFound()) { + strcpy(reply, "Err - PSRAM not available"); + return false; + } +#endif + if (!bridge || !bridge->isRunning()) { + strcpy(reply, "Err - MQTT bridge not running"); + return false; + } + return true; +} + +bool MyMesh::startNeighborDiscover(char* reply) { + if (neighbor_discover_active) { + strcpy(reply, "Err - neighbor discover already active"); + return false; + } + if (!neighborDiscoverReady(reply)) { + return false; // reply already set + } + + getLocalScopes(self_scopes_buf, sizeof(self_scopes_buf)); + neighbor_discover_count = 0; + for (int i = 0; i < MAX_NEIGHBOURS; i++) { + if (neighbours[i].heard_timestamp > 0) { + neighbor_discover[neighbor_discover_count].neighbour_idx = (uint8_t)i; + neighbor_discover[neighbor_discover_count].scopes[0] = 0; + neighbor_discover[neighbor_discover_count].status = ND_PENDING; + uint32_t tag; + if (sendAnonRegionsReq(neighbours[i].id, tag)) { + neighbor_discover[neighbor_discover_count].tag = tag; + } else { + neighbor_discover[neighbor_discover_count].status = ND_SEND_FAILED; + } + neighbor_discover_count++; + } + } + + neighbor_discover_active = true; + neighbor_discover_until = futureMillis(NEIGHBOR_DISCOVER_TIMEOUT_MS); + + if (neighbor_discover_count == 0) { + finishNeighborDiscover(); + strcpy(reply, "OK - neighbor discover started (0 neighbors, self only)"); + } else { + sprintf(reply, "OK - neighbor discover started (%u neighbors)", (unsigned)neighbor_discover_count); + } + return true; +} + +#endif // WITH_MQTT_NEIGHBORS diff --git a/examples/simple_room_server/MyMesh.h b/examples/simple_room_server/MyMesh.h index cd556be3..d052d58a 100644 --- a/examples/simple_room_server/MyMesh.h +++ b/examples/simple_room_server/MyMesh.h @@ -94,6 +94,13 @@ struct PostInfo { char text[MAX_POST_TEXT_LEN+1]; }; +struct NeighbourInfo { + mesh::Identity id; + uint32_t advert_timestamp; + uint32_t heard_timestamp; + int8_t snr; // multiplied by 4, user should divide to get float value +}; + class MyMesh : public mesh::Mesh, public CommonCLICallbacks { FILESYSTEM* _fs; uint32_t last_millis; @@ -123,6 +130,43 @@ class MyMesh : public mesh::Mesh, public CommonCLICallbacks { uint8_t pending_sf; uint8_t pending_cr; int matching_peer_indexes[MAX_CLIENTS]; +#if defined(WITH_MQTT_NEIGHBORS) + NeighbourInfo neighbours[MAX_NEIGHBOURS]; + uint32_t pending_discover_tag; + unsigned long pending_discover_until; + enum NeighborDiscoverStatus : uint8_t { + ND_PENDING = 1, + ND_RESPONDED = 2, + ND_TIMEOUT = 3, + ND_SEND_FAILED = 4, + }; + struct NeighborDiscoverEntry { + uint8_t neighbour_idx; + uint32_t tag; + char scopes[96]; + uint8_t status; + }; + NeighborDiscoverEntry neighbor_discover[MAX_NEIGHBOURS]; + uint8_t neighbor_discover_count; + bool neighbor_discover_active; + bool neighbor_table_refresh_active; + bool neighbor_table_refresh_periodic; + unsigned long neighbor_discover_until; + unsigned long next_neighbors_publish; + char self_scopes_buf[96]; + + void putNeighbour(const mesh::Identity& id, uint32_t timestamp, float snr); + void sendNodeDiscoverReq(); + bool sendAnonRegionsReq(const mesh::Identity& target, uint32_t& tag); + bool neighborDiscoverReady(char* reply); + bool startNeighborDiscover(char* reply); + void loopNeighborDiscover(); + void finishNeighborDiscover(); + bool handleNeighborDiscoverResponse(int overlay_idx, const uint8_t* data, size_t len); + void getLocalScopes(char* buf, size_t len); + static const int NEIGHBOR_DISCOVER_PEER_BASE = 1000; + static const unsigned long NEIGHBOR_DISCOVER_TIMEOUT_MS = 30000; +#endif #ifdef WITH_MQTT_BRIDGE MQTTBridge* bridge; #endif @@ -176,6 +220,10 @@ protected: void onPeerDataRecv(mesh::Packet* packet, uint8_t type, int sender_idx, const uint8_t* secret, uint8_t* data, size_t len) override; bool onPeerPathRecv(mesh::Packet* packet, int sender_idx, const uint8_t* secret, uint8_t* path, uint8_t path_len, uint8_t extra_type, uint8_t* extra, uint8_t extra_len) override; void onAckRecv(mesh::Packet* packet, uint32_t ack_crc) override; +#if defined(WITH_MQTT_NEIGHBORS) + void onAdvertRecv(mesh::Packet* packet, const mesh::Identity& id, uint32_t timestamp, const uint8_t* app_data, size_t app_data_len) override; + void onControlDataRecv(mesh::Packet* packet) override; +#endif #if ENV_INCLUDE_GPS == 1 void applyGpsPrefs() { @@ -226,9 +274,8 @@ public: void dumpLogFile() override; void setTxPower(int8_t power_dbm) override; - void formatNeighborsReply(char *reply) override { - strcpy(reply, "not supported"); - } + void formatNeighborsReply(char *reply) override; + void removeNeighbor(const uint8_t* pubkey, int key_len) override; void formatStatsReply(char *reply) override; void formatRadioStatsReply(char *reply) override; void formatPacketStatsReply(char *reply) override; diff --git a/variants/heltec_t190/platformio.ini b/variants/heltec_t190/platformio.ini index 5a9e8db2..147e3df4 100644 --- a/variants/heltec_t190/platformio.ini +++ b/variants/heltec_t190/platformio.ini @@ -205,6 +205,7 @@ build_flags = -D ADMIN_PASSWORD='"password"' -D ROOM_PASSWORD='"hello"' -D WITH_MQTT_BRIDGE=1 + -D MAX_NEIGHBOURS=50 -D OTA_MANIFEST_BASE='"https://observer.gessaman.com/v"' -D MAX_MQTT_BROKERS=3 -D MQTT_MAX_PACKET_SIZE=1024 diff --git a/variants/heltec_tracker_v2/platformio.ini b/variants/heltec_tracker_v2/platformio.ini index 512a4543..5e1d2d45 100644 --- a/variants/heltec_tracker_v2/platformio.ini +++ b/variants/heltec_tracker_v2/platformio.ini @@ -257,6 +257,7 @@ build_flags = -D ADMIN_PASSWORD='"password"' -D ROOM_PASSWORD='"hello"' -D WITH_MQTT_BRIDGE=1 + -D MAX_NEIGHBOURS=50 -D MAX_MQTT_BROKERS=3 -D MQTT_MAX_PACKET_SIZE=1024 -D MQTT_DEBUG=1 @@ -293,6 +294,7 @@ build_flags = -D ADMIN_PASSWORD='"password"' -D ROOM_PASSWORD='"hello"' -D WITH_MQTT_BRIDGE=1 + -D MAX_NEIGHBOURS=50 -D MAX_MQTT_BROKERS=3 -D MQTT_MAX_PACKET_SIZE=1024 -D MQTT_DEBUG=1 diff --git a/variants/heltec_v3/platformio.ini b/variants/heltec_v3/platformio.ini index 2d13237b..1bb69a79 100644 --- a/variants/heltec_v3/platformio.ini +++ b/variants/heltec_v3/platformio.ini @@ -194,6 +194,7 @@ build_flags = -D ADMIN_PASSWORD='"password"' -D ROOM_PASSWORD='"hello"' -D WITH_MQTT_BRIDGE=1 + -D MAX_NEIGHBOURS=50 -D OTA_MANIFEST_BASE='"https://observer.gessaman.com/v"' -D MAX_MQTT_BROKERS=3 -D MQTT_MAX_PACKET_SIZE=1024 @@ -533,6 +534,7 @@ build_flags = -D ADMIN_PASSWORD='"password"' -D ROOM_PASSWORD='"hello"' -D WITH_MQTT_BRIDGE=1 + -D MAX_NEIGHBOURS=50 -D OTA_MANIFEST_BASE='"https://observer.gessaman.com/v"' -D MAX_MQTT_BROKERS=3 -D MQTT_MAX_PACKET_SIZE=1024 diff --git a/variants/heltec_v4/platformio.ini b/variants/heltec_v4/platformio.ini index f3622bc5..e066c2b6 100644 --- a/variants/heltec_v4/platformio.ini +++ b/variants/heltec_v4/platformio.ini @@ -317,6 +317,7 @@ build_flags = -D ADMIN_PASSWORD='"password"' -D ROOM_PASSWORD='"hello"' -D WITH_MQTT_BRIDGE=1 + -D MAX_NEIGHBOURS=50 -D OTA_MANIFEST_BASE='"https://observer.gessaman.com/v"' -D MAX_MQTT_BROKERS=3 -D MQTT_MAX_PACKET_SIZE=1024 @@ -357,6 +358,7 @@ build_flags = -D ADMIN_PASSWORD='"password"' -D ROOM_PASSWORD='"hello"' -D WITH_MQTT_BRIDGE=1 + -D MAX_NEIGHBOURS=50 -D OTA_MANIFEST_BASE='"https://observer.gessaman.com/v"' -D MAX_MQTT_BROKERS=3 -D MQTT_MAX_PACKET_SIZE=1024 diff --git a/variants/lilygo_t3s3/platformio.ini b/variants/lilygo_t3s3/platformio.ini index 4a259ee7..9d5b160b 100644 --- a/variants/lilygo_t3s3/platformio.ini +++ b/variants/lilygo_t3s3/platformio.ini @@ -162,6 +162,7 @@ build_flags = -D ADMIN_PASSWORD='"password"' -D ROOM_PASSWORD='"hello"' -D WITH_MQTT_BRIDGE=1 + -D MAX_NEIGHBOURS=50 -D OTA_MANIFEST_BASE='"https://observer.gessaman.com/v"' -D MAX_MQTT_BROKERS=3 -D MQTT_MAX_PACKET_SIZE=1024 diff --git a/variants/lilygo_tbeam_1w/platformio.ini b/variants/lilygo_tbeam_1w/platformio.ini index 827cac38..4a40c0f7 100644 --- a/variants/lilygo_tbeam_1w/platformio.ini +++ b/variants/lilygo_tbeam_1w/platformio.ini @@ -256,6 +256,7 @@ build_flags = -D ADMIN_PASSWORD='"password"' -D ROOM_PASSWORD='"hello"' -D WITH_MQTT_BRIDGE=1 + -D MAX_NEIGHBOURS=50 -D OTA_MANIFEST_BASE='"https://observer.gessaman.com/v"' -D MAX_MQTT_BROKERS=3 -D MQTT_MAX_PACKET_SIZE=1024 diff --git a/variants/lilygo_tbeam_SX1262/platformio.ini b/variants/lilygo_tbeam_SX1262/platformio.ini index 2e0d8334..f4734f73 100644 --- a/variants/lilygo_tbeam_SX1262/platformio.ini +++ b/variants/lilygo_tbeam_SX1262/platformio.ini @@ -204,6 +204,7 @@ build_flags = -D ADMIN_PASSWORD='"password"' -D ROOM_PASSWORD='"hello"' -D WITH_MQTT_BRIDGE=1 + -D MAX_NEIGHBOURS=50 -D OTA_MANIFEST_BASE='"https://observer.gessaman.com/v"' -D MAX_MQTT_BROKERS=3 -D MQTT_MAX_PACKET_SIZE=1024 diff --git a/variants/lilygo_tbeam_SX1276/platformio.ini b/variants/lilygo_tbeam_SX1276/platformio.ini index 41f56115..db287a52 100644 --- a/variants/lilygo_tbeam_SX1276/platformio.ini +++ b/variants/lilygo_tbeam_SX1276/platformio.ini @@ -203,6 +203,7 @@ build_flags = -D ADMIN_PASSWORD='"password"' -D ROOM_PASSWORD='"hello"' -D WITH_MQTT_BRIDGE=1 + -D MAX_NEIGHBOURS=50 -D OTA_MANIFEST_BASE='"https://observer.gessaman.com/v"' -D MAX_MQTT_BROKERS=3 -D MQTT_MAX_PACKET_SIZE=1024 diff --git a/variants/lilygo_tbeam_supreme_SX1262/platformio.ini b/variants/lilygo_tbeam_supreme_SX1262/platformio.ini index 7c588e7c..38dd076c 100644 --- a/variants/lilygo_tbeam_supreme_SX1262/platformio.ini +++ b/variants/lilygo_tbeam_supreme_SX1262/platformio.ini @@ -163,6 +163,7 @@ build_flags = -D ADMIN_PASSWORD='"password"' -D ROOM_PASSWORD='"hello"' -D WITH_MQTT_BRIDGE=1 + -D MAX_NEIGHBOURS=50 -D OTA_MANIFEST_BASE='"https://observer.gessaman.com/v"' -D MAX_MQTT_BROKERS=3 -D MQTT_MAX_PACKET_SIZE=1024 diff --git a/variants/lilygo_tlora_v2_1/platformio.ini b/variants/lilygo_tlora_v2_1/platformio.ini index 37a909f8..450a978e 100644 --- a/variants/lilygo_tlora_v2_1/platformio.ini +++ b/variants/lilygo_tlora_v2_1/platformio.ini @@ -203,6 +203,7 @@ build_flags = -D ADMIN_PASSWORD='"password"' -D ROOM_PASSWORD='"hello"' -D WITH_MQTT_BRIDGE=1 + -D MAX_NEIGHBOURS=50 -D OTA_MANIFEST_BASE='"https://observer.gessaman.com/v"' -D MAX_MQTT_BROKERS=3 -D MQTT_MAX_PACKET_SIZE=1024 diff --git a/variants/rak3112/platformio.ini b/variants/rak3112/platformio.ini index 130b8344..775c50b5 100644 --- a/variants/rak3112/platformio.ini +++ b/variants/rak3112/platformio.ini @@ -177,6 +177,7 @@ build_flags = -D ADMIN_PASSWORD='"password"' -D ROOM_PASSWORD='"hello"' -D WITH_MQTT_BRIDGE=1 + -D MAX_NEIGHBOURS=50 -D OTA_MANIFEST_BASE='"https://observer.gessaman.com/v"' -D MAX_MQTT_BROKERS=3 -D MQTT_MAX_PACKET_SIZE=1024 diff --git a/variants/station_g2/platformio.ini b/variants/station_g2/platformio.ini index 338f4c31..d41c062f 100644 --- a/variants/station_g2/platformio.ini +++ b/variants/station_g2/platformio.ini @@ -187,6 +187,7 @@ build_flags = -D ADMIN_PASSWORD='"password"' -D ROOM_PASSWORD='"hello"' -D WITH_MQTT_BRIDGE=1 + -D MAX_NEIGHBOURS=50 -D OTA_MANIFEST_BASE='"https://observer.gessaman.com/v"' -D MAX_MQTT_BROKERS=3 -D MQTT_MAX_PACKET_SIZE=1024 diff --git a/variants/xiao_s3_wio/platformio.ini b/variants/xiao_s3_wio/platformio.ini index 8e4314cf..bdb96f53 100644 --- a/variants/xiao_s3_wio/platformio.ini +++ b/variants/xiao_s3_wio/platformio.ini @@ -147,6 +147,7 @@ build_flags = -D ADMIN_PASSWORD='"password"' -D ROOM_PASSWORD='"hello"' -D WITH_MQTT_BRIDGE=1 + -D MAX_NEIGHBOURS=50 -D OTA_MANIFEST_BASE='"https://observer.gessaman.com/v"' -D MAX_MQTT_BROKERS=3 -D MQTT_MAX_PACKET_SIZE=1024 From ed674fdf9db30c1b61a1097f900ea62f2ed5ae42 Mon Sep 17 00:00:00 2001 From: agessaman Date: Mon, 20 Jul 2026 09:00:56 -0700 Subject: [PATCH 12/87] feat(docs): update CLI commands for OTA firmware update options Enhance the documentation for the `start ota` command to clarify its behavior when connected to a Wi-Fi network versus when using the `start ota ap` option. This provides users with better guidance on how to initiate OTA updates under different network conditions. (cherry picked from commit 2019a8c0caef92f7b337094011ec8cd003cf6bfa) --- docs/cli_commands.md | 3 ++- src/MeshCore.h | 2 +- src/helpers/CommonCLI.cpp | 10 ++++++++-- src/helpers/ESP32Board.cpp | 9 ++++++--- src/helpers/ESP32Board.h | 2 +- src/helpers/NRF52Board.cpp | 2 +- src/helpers/NRF52Board.h | 2 +- src/helpers/stm32/STM32Board.h | 2 +- 8 files changed, 21 insertions(+), 11 deletions(-) diff --git a/docs/cli_commands.md b/docs/cli_commands.md index 55f8cd64..22014a6f 100644 --- a/docs/cli_commands.md +++ b/docs/cli_commands.md @@ -71,7 +71,8 @@ This document provides an overview of CLI commands that can be sent to MeshCore ### Start an Over-The-Air (OTA) firmware update **Usage:** -- `start ota` +- `start ota` — serves the ElegantOTA web upload page on the station IP if joined to a Wi-Fi network, otherwise raises the `MeshCore-OTA` Wi-Fi hotspot. +- `start ota ap` — always raises the `MeshCore-OTA` Wi-Fi hotspot, even when joined to a network. Use this when the network applies client isolation and the station IP isn't reachable. --- diff --git a/src/MeshCore.h b/src/MeshCore.h index f0c8f387..05f7d021 100644 --- a/src/MeshCore.h +++ b/src/MeshCore.h @@ -63,7 +63,7 @@ public: virtual void setGpio(uint32_t values) {} virtual uint8_t getStartupReason() const = 0; virtual bool getBootloaderVersion(char* version, size_t max_len) { return false; } - virtual bool startOTAUpdate(const char* id, char reply[]) { return false; } // not supported + virtual bool startOTAUpdate(const char* id, char reply[], bool force_ap = false) { return false; } // not supported // Pull-based OTA: fetch the firmware build for this variant from a baked-in manifest and flash it. // current_ver is the running firmware version string (used to skip if already up to date); when // dry_run is true the build is only reported, not flashed. Observer (ESP32+WiFi) builds only. diff --git a/src/helpers/CommonCLI.cpp b/src/helpers/CommonCLI.cpp index b941d179..03d548c7 100644 --- a/src/helpers/CommonCLI.cpp +++ b/src/helpers/CommonCLI.cpp @@ -742,8 +742,14 @@ void CommonCLI::handleCommand(uint32_t sender_timestamp, char* command, char* re (int)heap_caps_get_free_size(MALLOC_CAP_SPIRAM), (int)heap_caps_get_total_size(MALLOC_CAP_SPIRAM)); } else if (memcmp(command, "start ota", 9) == 0) { - // Manual OTA: bring up the ElegantOTA SoftAP for a hand-uploaded binary. - if (!_board->startOTAUpdate(_prefs->node_name, reply)) { + // Manual OTA: bring up the ElegantOTA web UI for a hand-uploaded binary. + // Plain "start ota" serves on the station IP when joined to WiFi, else + // raises the MeshCore-OTA SoftAP. "start ota ap" forces the SoftAP even + // when connected, so the UI is reachable when the network applies client + // isolation and the station IP can't be reached. (&& short-circuits keep + // the [10]/[11] reads in-bounds when command == "start ota".) + bool force_ap = (command[9] == ' ' && command[10] == 'a' && command[11] == 'p'); + if (!_board->startOTAUpdate(_prefs->node_name, reply, force_ap)) { strcpy(reply, "Error"); } } else if (memcmp(command, "clock", 5) == 0) { diff --git a/src/helpers/ESP32Board.cpp b/src/helpers/ESP32Board.cpp index 5a457638..9893c2c7 100644 --- a/src/helpers/ESP32Board.cpp +++ b/src/helpers/ESP32Board.cpp @@ -10,14 +10,17 @@ #include -bool ESP32Board::startOTAUpdate(const char* id, char reply[]) { +bool ESP32Board::startOTAUpdate(const char* id, char reply[], bool force_ap) { inhibit_sleep = true; // prevent sleep during OTA // If the device is already on a WiFi network (e.g. an observer joined in STA // mode), serve ElegantOTA on the station IP so it's reachable from the LAN // without joining a separate AP. Otherwise raise the MeshCore-OTA SoftAP. + // force_ap ("start ota ap") always raises the SoftAP, so the OTA UI stays + // reachable even when the joined network applies client isolation and the + // station IP can't be reached. IPAddress ip; - if (WiFi.status() == WL_CONNECTED) { + if (!force_ap && WiFi.status() == WL_CONNECTED) { ip = WiFi.localIP(); } else { WiFi.softAP("MeshCore-OTA", NULL); @@ -49,7 +52,7 @@ bool ESP32Board::startOTAUpdate(const char* id, char reply[]) { } #else -bool ESP32Board::startOTAUpdate(const char* id, char reply[]) { +bool ESP32Board::startOTAUpdate(const char* id, char reply[], bool force_ap) { return false; // not supported } #endif diff --git a/src/helpers/ESP32Board.h b/src/helpers/ESP32Board.h index eae9ab78..eb6c3a9d 100644 --- a/src/helpers/ESP32Board.h +++ b/src/helpers/ESP32Board.h @@ -150,7 +150,7 @@ public: esp_restart(); } - bool startOTAUpdate(const char* id, char reply[]) override; + bool startOTAUpdate(const char* id, char reply[], bool force_ap = false) override; bool otaFromManifest(const char* current_ver, bool dry_run, char reply[]) override; // Heavy body (TLS + JSON / HTTPUpdate). Runs in a dedicated large-stack task // spawned by otaFromManifest() — public only so that task entry point can call diff --git a/src/helpers/NRF52Board.cpp b/src/helpers/NRF52Board.cpp index 17265f04..84bdcb3d 100644 --- a/src/helpers/NRF52Board.cpp +++ b/src/helpers/NRF52Board.cpp @@ -316,7 +316,7 @@ bool NRF52Board::getBootloaderVersion(char* out, size_t max_len) { return false; } -bool NRF52Board::startOTAUpdate(const char *id, char reply[]) { +bool NRF52Board::startOTAUpdate(const char *id, char reply[], bool force_ap) { // Config the peripheral connection with maximum bandwidth // more SRAM required by SoftDevice // Note: All config***() function must be called before begin() diff --git a/src/helpers/NRF52Board.h b/src/helpers/NRF52Board.h index 17065cf4..c7d1e99e 100644 --- a/src/helpers/NRF52Board.h +++ b/src/helpers/NRF52Board.h @@ -51,7 +51,7 @@ public: virtual float getMCUTemperature() override; virtual void reboot() override { NVIC_SystemReset(); } virtual bool getBootloaderVersion(char* version, size_t max_len) override; - virtual bool startOTAUpdate(const char *id, char reply[]) override; + virtual bool startOTAUpdate(const char *id, char reply[], bool force_ap = false) override; virtual void sleep(uint32_t secs) override; bool isExternalPowered() override; diff --git a/src/helpers/stm32/STM32Board.h b/src/helpers/stm32/STM32Board.h index 06bc768f..cc33dfe5 100644 --- a/src/helpers/stm32/STM32Board.h +++ b/src/helpers/stm32/STM32Board.h @@ -41,5 +41,5 @@ public: } #endif - bool startOTAUpdate(const char* id, char reply[]) override { return false; }; + bool startOTAUpdate(const char* id, char reply[], bool force_ap = false) override { return false; }; }; \ No newline at end of file From 20b5b2e9c9bcd64c2f347d39e5a73e9293dea4ad Mon Sep 17 00:00:00 2001 From: agessaman Date: Mon, 20 Jul 2026 16:11:11 -0700 Subject: [PATCH 13/87] feat(mqtt): add max active slots logic for MQTT connections Implement logic to determine the maximum number of active MQTT slots based on available memory. This change allows the system to warn users when attempting to connect more slots than supported by the hardware, improving user experience and preventing connection issues. The logic is centralized in the new getMaxActiveSlots() method, ensuring consistency across the codebase. --- src/helpers/CommonCLI_Observer.cpp | 28 ++++++++++++++++++++++++++++ src/helpers/bridges/MQTTBridge.cpp | 24 +++++++++++++++--------- src/helpers/bridges/MQTTBridge.h | 4 ++++ 3 files changed, 47 insertions(+), 9 deletions(-) diff --git a/src/helpers/CommonCLI_Observer.cpp b/src/helpers/CommonCLI_Observer.cpp index 37982da4..12c403dc 100644 --- a/src/helpers/CommonCLI_Observer.cpp +++ b/src/helpers/CommonCLI_Observer.cpp @@ -405,6 +405,34 @@ bool CommonCLI::handleObserverSetCmd(uint32_t sender_timestamp, const char* conf } else { sprintf(reply, "OK - slot %d preset: %s", slot + 1, preset_name); } + // Warn when this slot won't actually connect on this hardware. The set + // is never blocked — prefs persist so the config carries over if the + // device is moved to a board with more slots — but flag it, or the + // operator waits for a connection that never comes. Mirrors the bridge + // setup loop: slots past the runtime array (RUNTIME_MQTT_SLOTS) are + // never iterated; within it, only the first getMaxActiveSlots() + // *enabled* slots connect (each WSS/TLS link costs ~40 KB heap). + if (strcmp(preset_name, MQTT_PRESET_NONE) != 0) { + size_t used = strlen(reply); + if (slot >= RUNTIME_MQTT_SLOTS) { + if (used < 158) { + snprintf(reply + used, 160 - used, " (slot inactive on this hardware)"); + } + } else { + const int max_active = MQTTBridge::getMaxActiveSlots(); + int rank = 0; // this slot's position among enabled slots, by index + for (int s = 0; s <= slot; s++) { + if (_mqtt_prefs.mqtt_slot_preset[s][0] != '\0' && + strcmp(_mqtt_prefs.mqtt_slot_preset[s], MQTT_PRESET_NONE) != 0) { + rank++; + } + } + if (rank > max_active && used < 158) { + snprintf(reply + used, 160 - used, + " (won't connect: %d-slot limit on this hardware)", max_active); + } + } + } } } else { strcpy(reply, "Error: unknown preset. Use 'get mqtt.presets'"); diff --git a/src/helpers/bridges/MQTTBridge.cpp b/src/helpers/bridges/MQTTBridge.cpp index 2066f9c6..39cd494d 100644 --- a/src/helpers/bridges/MQTTBridge.cpp +++ b/src/helpers/bridges/MQTTBridge.cpp @@ -337,6 +337,18 @@ void MQTTBridge::formatMqttStatsReply(char* buf, size_t bufsize) { uint8_t MQTTBridge::getLastWifiDisconnectReason() { return s_wifi_disconnect_reason; } unsigned long MQTTBridge::getLastWifiDisconnectTime() { return s_wifi_disconnect_time; } +// Each WSS/TLS connection needs ~40KB for mbedTLS buffers. Without PSRAM even 3 +// concurrent connections would exhaust internal heap, so cap at 2; with PSRAM +// cap at 5 (6 configurable but 5 active max). Static so the CLI can report, at +// config time, that a slot beyond this cap won't connect. +int MQTTBridge::getMaxActiveSlots() { +#if defined(ESP_PLATFORM) && defined(BOARD_HAS_PSRAM) + return psramFound() ? 5 : 2; +#else + return 2; +#endif +} + unsigned long MQTTBridge::getSlotCurrentOutageStartMs(int slot_index) const { if (slot_index < 0 || slot_index >= RUNTIME_MQTT_SLOTS) return 0; return _slots[slot_index].current_outage_started_ms; @@ -627,15 +639,9 @@ void MQTTBridge::begin() { MQTT_DEBUG_PRINTLN("PSRAM: not configured for this board (no BOARD_HAS_PSRAM)"); #endif - // Limit active slots based on available memory. - // Each WSS/TLS connection needs ~40KB for mbedTLS buffers. - // Without PSRAM, even 3 concurrent connections would exhaust internal heap. - // With PSRAM, cap at 5 for safety (6 configurable but 5 active max). - #if defined(ESP_PLATFORM) && defined(BOARD_HAS_PSRAM) - _max_active_slots = psramFound() ? 5 : 2; - #else - _max_active_slots = 2; - #endif + // Limit active slots based on available memory (single source of truth in + // getMaxActiveSlots(), which the CLI also uses to warn about inactive slots). + _max_active_slots = getMaxActiveSlots(); MQTT_DEBUG_PRINTLN("Max active slots: %d", _max_active_slots); // Check if WiFi credentials are configured first diff --git a/src/helpers/bridges/MQTTBridge.h b/src/helpers/bridges/MQTTBridge.h index 5e7e2d01..17346c43 100644 --- a/src/helpers/bridges/MQTTBridge.h +++ b/src/helpers/bridges/MQTTBridge.h @@ -498,6 +498,10 @@ public: static void formatSlotDiagReply(char* buf, size_t bufsize, int slot_index); static uint8_t getLastWifiDisconnectReason(); static unsigned long getLastWifiDisconnectTime(); + /** Max slots that can be connected at once on this hardware (each WSS/TLS link + * needs ~40 KB internal heap): 5 with PSRAM, 2 without. Note this is below + * RUNTIME_MQTT_SLOTS, so more slots can be configured than will connect. */ + static int getMaxActiveSlots(); #if defined(WITH_MQTT_NEIGHBORS) void requestPublishNeighbors(const char* json, size_t len); From 333e7869cf418be7e920abff75dec6280b9b1b21 Mon Sep 17 00:00:00 2001 From: agessaman Date: Tue, 21 Jul 2026 08:54:02 -0700 Subject: [PATCH 14/87] ci(mqtt): add preset name-parity check across channels Keep observer-firmware and observer-firmware-dev from drifting on built-in MQTT preset names while allowing config details to differ. --- .../workflows/check-mqtt-preset-parity.yml | 90 ++++++ scripts/check_mqtt_preset_parity.py | 282 ++++++++++++++++++ 2 files changed, 372 insertions(+) create mode 100644 .github/workflows/check-mqtt-preset-parity.yml create mode 100755 scripts/check_mqtt_preset_parity.py diff --git a/.github/workflows/check-mqtt-preset-parity.yml b/.github/workflows/check-mqtt-preset-parity.yml new file mode 100644 index 00000000..e1039a40 --- /dev/null +++ b/.github/workflows/check-mqtt-preset-parity.yml @@ -0,0 +1,90 @@ +name: Check MQTT Preset Name Parity + +# Ensures observer-firmware and observer-firmware-dev share the same built-in +# MQTT preset *names* (config details may differ). See scripts/check_mqtt_preset_parity.py. + +permissions: + contents: read + +on: + workflow_dispatch: + pull_request: + branches: + - observer-firmware + - observer-firmware-dev + paths: + - 'src/helpers/MQTTPresets.h' + - 'scripts/check_mqtt_preset_parity.py' + - '.github/workflows/check-mqtt-preset-parity.yml' + push: + branches: + - observer-firmware + - observer-firmware-dev + paths: + - 'src/helpers/MQTTPresets.h' + - 'scripts/check_mqtt_preset_parity.py' + - '.github/workflows/check-mqtt-preset-parity.yml' + +jobs: + parity: + runs-on: ubuntu-latest + steps: + - name: Clone Repo + uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Fetch channel branches + run: | + git fetch --no-tags origin observer-firmware observer-firmware-dev + + - name: Materialize presets from both channels + env: + EVENT_NAME: ${{ github.event_name }} + PR_BASE: ${{ github.event.pull_request.base.ref }} + PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }} + run: | + set -euo pipefail + PRESET_PATH=src/helpers/MQTTPresets.h + mkdir -p /tmp/preset-parity + + if [ "$EVENT_NAME" = "pull_request" ]; then + # Ensure the PR head commit is reachable (merge checkout may not keep it). + git fetch --no-tags origin "$PR_HEAD_SHA" + # Proposed state of the PR's base channel vs current tip of the sibling. + git show "${PR_HEAD_SHA}:${PRESET_PATH}" > /tmp/preset-parity/pr-head.h + case "$PR_BASE" in + observer-firmware-dev) + cp /tmp/preset-parity/pr-head.h /tmp/preset-parity/dev.h + git show "origin/observer-firmware:${PRESET_PATH}" > /tmp/preset-parity/prod.h + ;; + observer-firmware) + git show "origin/observer-firmware-dev:${PRESET_PATH}" > /tmp/preset-parity/dev.h + cp /tmp/preset-parity/pr-head.h /tmp/preset-parity/prod.h + ;; + *) + echo "::error::unexpected PR base branch: $PR_BASE" >&2 + exit 1 + ;; + esac + else + # push / workflow_dispatch: compare current channel tips. + git show "origin/observer-firmware:${PRESET_PATH}" > /tmp/preset-parity/prod.h + git show "origin/observer-firmware-dev:${PRESET_PATH}" > /tmp/preset-parity/dev.h + fi + + echo "=== production (observer-firmware) head preview ===" + head -n 5 /tmp/preset-parity/prod.h + echo "=== dev (observer-firmware-dev) head preview ===" + head -n 5 /tmp/preset-parity/dev.h + + - name: Self-test checker + run: python3 scripts/check_mqtt_preset_parity.py --self-test + + - name: Compare preset names + run: | + python3 scripts/check_mqtt_preset_parity.py \ + /tmp/preset-parity/prod.h \ + /tmp/preset-parity/dev.h \ + --label-a observer-firmware \ + --label-b observer-firmware-dev diff --git a/scripts/check_mqtt_preset_parity.py b/scripts/check_mqtt_preset_parity.py new file mode 100755 index 00000000..d213a258 --- /dev/null +++ b/scripts/check_mqtt_preset_parity.py @@ -0,0 +1,282 @@ +#!/usr/bin/env python3 +"""Compare MQTT built-in preset *names* between two MQTTPresets.h files. + +Only the first string field of each ``MQTT_PRESETS`` entry is compared (set +equality, case-sensitive). URL, auth, CA, keepalive, and credentials are +ignored so channel branches may diverge on config details without failing CI. + +Usage:: + + python3 scripts/check_mqtt_preset_parity.py FILE_A FILE_B \\ + --label-a observer-firmware --label-b observer-firmware-dev + + python3 scripts/check_mqtt_preset_parity.py --self-test +""" + +from __future__ import annotations + +import argparse +import re +import sys +from pathlib import Path +from tempfile import TemporaryDirectory + + +COUNT_PATTERN = re.compile( + r"static\s+const\s+int\s+MQTT_PRESET_COUNT\s*=\s*(\d+)\s*;" +) +# Start of the preset table; allow optional whitespace/newlines. +TABLE_START_PATTERN = re.compile( + r"static\s+const\s+MQTTPresetDef\s+MQTT_PRESETS\s*\[\s*[^\]]*\]\s*=\s*\{", + re.MULTILINE, +) +# First quoted string after an opening brace at entry start (after optional WS). +ENTRY_NAME_PATTERN = re.compile(r'\{\s*"([^"]+)"\s*,') + + +def _strip_line_comment(line: str) -> str: + """Remove ``//`` comments; do not treat ``://`` in URLs as comments.""" + + in_string = False + i = 0 + while i < len(line): + ch = line[i] + if ch == '"' and (i == 0 or line[i - 1] != "\\"): + in_string = not in_string + elif not in_string and ch == "/" and i + 1 < len(line) and line[i + 1] == "/": + # Prefer not to cut URL schemes: require that the previous char is not ':'. + if i == 0 or line[i - 1] != ":": + return line[:i] + i += 1 + return line + + +def extract_preset_block(text: str) -> str: + """Return the interior of the ``MQTT_PRESETS[...] = { ... };`` initializer.""" + + match = TABLE_START_PATTERN.search(text) + if not match: + raise ValueError("MQTT_PRESETS table not found") + + depth = 1 + i = match.end() + in_string = False + while i < len(text): + ch = text[i] + if ch == '"' and (i == 0 or text[i - 1] != "\\"): + in_string = not in_string + elif not in_string: + if ch == "{": + depth += 1 + elif ch == "}": + depth -= 1 + if depth == 0: + return text[match.end() : i] + i += 1 + raise ValueError("MQTT_PRESETS table is not closed") + + +def parse_presets(text: str, *, source: str = "") -> tuple[list[str], int | None]: + """Return (names in file order, MQTT_PRESET_COUNT or None if absent).""" + + count_match = COUNT_PATTERN.search(text) + declared_count = int(count_match.group(1)) if count_match else None + + block = extract_preset_block(text) + names: list[str] = [] + for raw_line in block.splitlines(): + line = _strip_line_comment(raw_line).strip() + if not line: + continue + entry = ENTRY_NAME_PATTERN.match(line) + if entry: + names.append(entry.group(1)) + + if not names: + raise ValueError(f"{source}: no preset entries found in MQTT_PRESETS") + + seen: set[str] = set() + dupes: list[str] = [] + for name in names: + if name in seen: + dupes.append(name) + seen.add(name) + if dupes: + raise ValueError(f"{source}: duplicate preset name(s): {', '.join(sorted(set(dupes)))}") + + if declared_count is not None and declared_count != len(names): + raise ValueError( + f"{source}: MQTT_PRESET_COUNT is {declared_count} but found {len(names)} preset entries" + ) + + return names, declared_count + + +def format_missing(label: str, names: set[str]) -> str: + if not names: + return f"Missing from {label}: (none)" + return f"Missing from {label}: {', '.join(sorted(names))}" + + +def compare( + names_a: set[str], + names_b: set[str], + *, + label_a: str, + label_b: str, +) -> list[str]: + """Return human-readable error lines if sets differ; empty list if equal.""" + + only_a = names_a - names_b + only_b = names_b - names_a + if not only_a and not only_b: + return [] + return [ + format_missing(label_b, only_a), # in A but not B → missing from B + format_missing(label_a, only_b), # in B but not A → missing from A + ] + + +def load_names(path: Path) -> set[str]: + text = path.read_text(encoding="utf-8") + names, _ = parse_presets(text, source=str(path)) + return set(names) + + +def self_test() -> int: + """Exercise equal-with-different-URLs, missing-name, and duplicate cases.""" + + header = """ +static const int MQTT_PRESET_COUNT = {count}; +static const MQTTPresetDef MQTT_PRESETS[MQTT_PRESET_COUNT] = {{ +{body} +}}; +""" + + with TemporaryDirectory() as directory: + root = Path(directory) + equal_a = root / "a.h" + equal_b = root / "b.h" + missing = root / "missing.h" + dupes = root / "dupes.h" + + equal_a.write_text( + header.format( + count=2, + body=' { "alpha", "wss://a.example/mqtt", nullptr },\n' + ' { "beta", "wss://b.example/mqtt", nullptr },', + ), + encoding="utf-8", + ) + # Same names, different URLs/config — must pass. + equal_b.write_text( + header.format( + count=2, + body=' { "beta", "mqtt://other:1883", nullptr },\n' + ' { "alpha", "wss://changed.example/mqtt", nullptr },', + ), + encoding="utf-8", + ) + if compare(load_names(equal_a), load_names(equal_b), label_a="a", label_b="b"): + print("self-test failed: equal name sets with different URLs were rejected.", file=sys.stderr) + return 1 + + missing.write_text( + header.format( + count=1, + body=' { "alpha", "wss://a.example/mqtt", nullptr },', + ), + encoding="utf-8", + ) + errs = compare(load_names(equal_a), load_names(missing), label_a="a", label_b="missing") + if len(errs) != 2 or "beta" not in errs[0] or "(none)" not in errs[1]: + print(f"self-test failed: missing-name case unexpected: {errs!r}", file=sys.stderr) + return 1 + + try: + parse_presets( + header.format( + count=2, + body=' { "alpha", "wss://a.example/mqtt", nullptr },\n' + ' { "alpha", "wss://b.example/mqtt", nullptr },', + ), + source="dupes", + ) + print("self-test failed: duplicate names were accepted.", file=sys.stderr) + return 1 + except ValueError as error: + if "duplicate" not in str(error): + print(f"self-test failed: unexpected duplicate error: {error}", file=sys.stderr) + return 1 + + try: + parse_presets( + header.format( + count=99, + body=' { "alpha", "wss://a.example/mqtt", nullptr },', + ), + source="count-mismatch", + ) + print("self-test failed: count mismatch was accepted.", file=sys.stderr) + return 1 + except ValueError as error: + if "MQTT_PRESET_COUNT" not in str(error): + print(f"self-test failed: unexpected count error: {error}", file=sys.stderr) + return 1 + + # URL with :// must not be treated as a line comment. + url_comment = root / "url.h" + url_comment.write_text( + header.format( + count=1, + body=' { "alpha", "wss://mqtt.example:443/mqtt", nullptr }, // note', + ), + encoding="utf-8", + ) + if load_names(url_comment) != {"alpha"}: + print("self-test failed: URL scheme was treated as a comment.", file=sys.stderr) + return 1 + + # Silence unused path in fixture layout. + dupes.write_text("// unused\n", encoding="utf-8") + + print("MQTT preset parity checker self-test passed.") + return 0 + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("file_a", type=Path, nargs="?", help="First MQTTPresets.h (e.g. production)") + parser.add_argument("file_b", type=Path, nargs="?", help="Second MQTTPresets.h (e.g. dev)") + parser.add_argument("--label-a", default="file_a", help="Label for file_a in reports") + parser.add_argument("--label-b", default="file_b", help="Label for file_b in reports") + parser.add_argument("--self-test", action="store_true") + args = parser.parse_args(argv) + + if args.self_test: + return self_test() + + if args.file_a is None or args.file_b is None: + parser.error("FILE_A and FILE_B are required unless --self-test is set") + + try: + names_a = load_names(args.file_a) + names_b = load_names(args.file_b) + except (OSError, ValueError) as error: + print(f"MQTT preset parity check could not run: {error}", file=sys.stderr) + return 2 + + errors = compare(names_a, names_b, label_a=args.label_a, label_b=args.label_b) + if errors: + print("MQTT preset name parity check failed:", *errors, sep="\n ", file=sys.stderr) + return 1 + + print( + f"MQTT preset name parity check passed: {len(names_a)} preset name(s) " + f"match between {args.label_a} and {args.label_b}." + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From 76f44d7d80465bc868df556cff0a0633a0bba872 Mon Sep 17 00:00:00 2001 From: agessaman Date: Tue, 21 Jul 2026 10:27:35 -0700 Subject: [PATCH 15/87] feat(mqtt): implement deferred OTA update scheduling in MyMesh Add functionality to schedule deferred OTA updates in the MyMesh class. This includes a new member variable to track the update time and a method to initiate the update process, allowing for a smoother OTA experience by managing timing and state transitions effectively. --- examples/simple_room_server/MyMesh.cpp | 17 +++++++++++++++++ examples/simple_room_server/MyMesh.h | 10 ++++++++++ 2 files changed, 27 insertions(+) diff --git a/examples/simple_room_server/MyMesh.cpp b/examples/simple_room_server/MyMesh.cpp index 535f0822..0527f971 100644 --- a/examples/simple_room_server/MyMesh.cpp +++ b/examples/simple_room_server/MyMesh.cpp @@ -1374,6 +1374,23 @@ void MyMesh::loop() { MESH_DEBUG_PRINTLN("Radio params restored"); } +#if defined(WITH_MQTT_BRIDGE) && defined(OTA_MANIFEST_BASE) + if (_ota_update_at && millisHasNowPassed(_ota_update_at)) { // deferred `ota update` + _ota_update_at = 0; // clear timer + // The "Beginning update..." reply has now gone out. Free the bridge for heap + // headroom, then flash: otaFromManifest reboots into the new image on success + // (so this never returns); on any abort (already up to date, partition change, + // download error) it returns and we resume the bridge. + Serial.println("OTA: starting update"); + setBridgeState(false); + char ota_reply[160]; + if (!_cli.getBoard()->otaFromManifest(getFirmwareVer(), false, ota_reply)) { + Serial.print("OTA: aborted, resuming bridge - "); Serial.println(ota_reply); + setBridgeState(true); + } + } +#endif + // is pending dirty contacts write needed? if (dirty_contacts_expiry && millisHasNowPassed(dirty_contacts_expiry)) { acl.save(_fs, MyMesh::saveFilter); diff --git a/examples/simple_room_server/MyMesh.h b/examples/simple_room_server/MyMesh.h index d052d58a..8ec944b5 100644 --- a/examples/simple_room_server/MyMesh.h +++ b/examples/simple_room_server/MyMesh.h @@ -125,6 +125,7 @@ class MyMesh : public mesh::Mesh, public CommonCLICallbacks { RegionEntry* recv_pkt_region; TransportKey default_scope; unsigned long set_radio_at, revert_radio_at; + unsigned long _ota_update_at = 0; // deferred `ota update` fire time (0 = none scheduled) float pending_freq; float pending_bw; uint8_t pending_sf; @@ -352,6 +353,15 @@ public: #endif } + // Schedule the pull-OTA flash to run from loop() in ~2.5 s, leaving time for the + // "Beginning update..." CLI reply (CLI_REPLY_DELAY_MILLIS = 600 ms) to transmit + // before the flash blocks the loop and reboots. + bool beginDeferredOtaUpdate() override { + _ota_update_at = millis() + 2500; + if (_ota_update_at == 0) _ota_update_at = 1; // 0 means "none" + return true; + } + int getQueueSize() override { return bridge ? bridge->getQueueSize() : 0; } From d14ec41301697029c0ef6e6f22eec39ac568747b Mon Sep 17 00:00:00 2001 From: agessaman Date: Tue, 21 Jul 2026 11:03:04 -0700 Subject: [PATCH 16/87] feat(mqtt): add SNMP support for radio diagnostics and stats Implement SNMP functionality in MyMesh to provide radio diagnostics and statistics updates. This includes a new method for formatting radio diagnostic replies and periodic updates to the SNMP agent with radio stats, enhancing monitoring capabilities for users with SNMP enabled. (cherry picked from commit fbe608d6c000ad4bceb3ab4e0fd3abdfdd57b160) --- examples/simple_room_server/MyMesh.cpp | 23 +++++++++++++++++++++++ examples/simple_room_server/MyMesh.h | 15 +++++++++++++++ variants/heltec_v3/platformio.ini | 6 ++++++ variants/heltec_v4/platformio.ini | 6 ++++++ variants/rak3112/platformio.ini | 3 +++ variants/station_g2/platformio.ini | 3 +++ 6 files changed, 56 insertions(+) diff --git a/examples/simple_room_server/MyMesh.cpp b/examples/simple_room_server/MyMesh.cpp index 0527f971..ada72858 100644 --- a/examples/simple_room_server/MyMesh.cpp +++ b/examples/simple_room_server/MyMesh.cpp @@ -1164,6 +1164,10 @@ void MyMesh::formatRadioStatsReply(char *reply) { StatsFormatHelper::formatRadioStats(reply, _radio, radio_driver, getTotalAirTime(), getReceiveAirTime()); } +void MyMesh::formatRadioDiagReply(char *reply) { + StatsFormatHelper::formatRadioDiag(reply, _radio, radio_driver, *_ms, _err_flags, hasOutbound()); +} + void MyMesh::formatPacketStatsReply(char *reply) { StatsFormatHelper::formatPacketStats(reply, radio_driver, getNumSentFlood(), getNumSentDirect(), getNumRecvFlood(), getNumRecvDirect()); @@ -1474,6 +1478,25 @@ void MyMesh::loop() { } } #endif + +#ifdef WITH_SNMP + // Push radio stats to SNMP agent every 2 seconds + if (_snmp_agent.isRunning()) { + static unsigned long last_snmp_stats = 0; + if (now - last_snmp_stats >= 2000) { + last_snmp_stats = now; + _snmp_agent.updateRadioStats( + radio_driver.getPacketsRecv(), radio_driver.getPacketsSent(), + radio_driver.getPacketsRecvErrors(), + (int16_t)_radio->getNoiseFloor(), + (int16_t)radio_driver.getLastRSSI(), + (int16_t)(radio_driver.getLastSNR() * 4), + getNumSentFlood(), getNumSentDirect(), + getNumRecvFlood(), getNumRecvDirect(), + getTotalAirTime() / 1000, uptime_millis / 1000); + } + } +#endif } #if defined(WITH_MQTT_NEIGHBORS) diff --git a/examples/simple_room_server/MyMesh.h b/examples/simple_room_server/MyMesh.h index 8ec944b5..57833d8c 100644 --- a/examples/simple_room_server/MyMesh.h +++ b/examples/simple_room_server/MyMesh.h @@ -30,6 +30,10 @@ #define WITH_BRIDGE #endif +#ifdef WITH_SNMP +#include "helpers/SNMPAgent.h" +#endif + /* ------------------------------ Config -------------------------------- */ #ifndef FIRMWARE_BUILD_DATE @@ -171,6 +175,9 @@ class MyMesh : public mesh::Mesh, public CommonCLICallbacks { #ifdef WITH_MQTT_BRIDGE MQTTBridge* bridge; #endif +#ifdef WITH_SNMP + MeshSNMPAgent _snmp_agent; +#endif #ifdef WITH_MQTT_BRIDGE AlertReporter _alerter; #endif @@ -279,6 +286,7 @@ public: void removeNeighbor(const uint8_t* pubkey, int key_len) override; void formatStatsReply(char *reply) override; void formatRadioStatsReply(char *reply) override; + void formatRadioDiagReply(char *reply) override; void formatPacketStatsReply(char *reply) override; void startRegionsLoad() override; bool saveRegions() override; @@ -313,6 +321,13 @@ public: bridge->setBuildDate(getBuildDate()); #ifdef WITH_MQTT_BRIDGE bridge->setStatsSources(this, _radio, _cli.getBoard(), _ms); +#ifdef WITH_SNMP + if (_cli.getObserverPrefs()->snmp_enabled) { + _snmp_agent.setNodeName(_prefs.node_name); + _snmp_agent.setFirmwareVersion(getFirmwareVer()); + bridge->setSNMPAgent(&_snmp_agent); + } +#endif #endif bridge->begin(); #ifdef WITH_MQTT_BRIDGE diff --git a/variants/heltec_v3/platformio.ini b/variants/heltec_v3/platformio.ini index 1bb69a79..1715e361 100644 --- a/variants/heltec_v3/platformio.ini +++ b/variants/heltec_v3/platformio.ini @@ -204,11 +204,13 @@ build_flags = -D CONFIG_MBEDTLS_CERTIFICATE_BUNDLE=y -D ESP32_CPU_FREQ=160 -D MQTT_WIFI_TX_POWER=WIFI_POWER_11dBm + -D WITH_SNMP=1 build_src_filter = ${Heltec_lora32_v3.build_src_filter} + + + + + + +<../examples/simple_room_server> lib_deps = ${Heltec_lora32_v3.lib_deps} @@ -218,6 +220,7 @@ lib_deps = arduino-libraries/NTPClient JChristensen/Timezone paulstoffregen/Time@1.6.1 + 0neblock/SNMP_Agent [env:Heltec_v3_terminal_chat] extends = Heltec_lora32_v3 @@ -545,10 +548,12 @@ build_flags = -D CONFIG_MBEDTLS_CERTIFICATE_BUNDLE=y -D ESP32_CPU_FREQ=160 -D MQTT_WIFI_TX_POWER=WIFI_POWER_11dBm + -D WITH_SNMP=1 build_src_filter = ${Heltec_lora32_v3.build_src_filter} + + + + + +<../examples/simple_room_server> lib_deps = ${Heltec_lora32_v3.lib_deps} @@ -558,6 +563,7 @@ lib_deps = arduino-libraries/NTPClient JChristensen/Timezone paulstoffregen/Time@1.6.1 + 0neblock/SNMP_Agent [env:Heltec_v3_kiss_modem] extends = Heltec_lora32_v3 diff --git a/variants/heltec_v4/platformio.ini b/variants/heltec_v4/platformio.ini index e066c2b6..de5dc2e0 100644 --- a/variants/heltec_v4/platformio.ini +++ b/variants/heltec_v4/platformio.ini @@ -327,11 +327,13 @@ build_flags = -D CONFIG_MBEDTLS_CERTIFICATE_BUNDLE=y -D ESP32_CPU_FREQ=160 -D MQTT_WIFI_TX_POWER=WIFI_POWER_11dBm + -D WITH_SNMP=1 build_src_filter = ${heltec_v4_oled.build_src_filter} + + + + + + +<../examples/simple_room_server> lib_deps = ${heltec_v4_oled.lib_deps} @@ -341,6 +343,7 @@ lib_deps = arduino-libraries/NTPClient JChristensen/Timezone paulstoffregen/Time@1.6.1 + 0neblock/SNMP_Agent [env:heltec_v4_expansionkit_room_server_observer_mqtt] extends = heltec_v4_oled @@ -370,11 +373,13 @@ build_flags = -D MQTT_WIFI_TX_POWER=WIFI_POWER_11dBm -D ENV_PIN_SDA=4 -D ENV_PIN_SCL=3 + -D WITH_SNMP=1 build_src_filter = ${heltec_v4_oled.build_src_filter} + + + + + + +<../examples/simple_room_server> lib_deps = ${heltec_v4_oled.lib_deps} @@ -384,6 +389,7 @@ lib_deps = arduino-libraries/NTPClient JChristensen/Timezone paulstoffregen/Time@1.6.1 + 0neblock/SNMP_Agent [env:heltec_v4_room_server] extends = heltec_v4_oled diff --git a/variants/rak3112/platformio.ini b/variants/rak3112/platformio.ini index 775c50b5..e7222a22 100644 --- a/variants/rak3112/platformio.ini +++ b/variants/rak3112/platformio.ini @@ -187,10 +187,12 @@ build_flags = -D CONFIG_MBEDTLS_CERTIFICATE_BUNDLE=y -D ESP32_CPU_FREQ=160 -D MQTT_WIFI_TX_POWER=WIFI_POWER_11dBm + -D WITH_SNMP=1 build_src_filter = ${rak3112.build_src_filter} + + + + + +<../examples/simple_room_server> lib_deps = ${rak3112.lib_deps} @@ -200,6 +202,7 @@ lib_deps = arduino-libraries/NTPClient JChristensen/Timezone paulstoffregen/Time@1.6.1 + 0neblock/SNMP_Agent [env:RAK_3112_terminal_chat] extends = rak3112 diff --git a/variants/station_g2/platformio.ini b/variants/station_g2/platformio.ini index d41c062f..89219748 100644 --- a/variants/station_g2/platformio.ini +++ b/variants/station_g2/platformio.ini @@ -196,11 +196,13 @@ build_flags = ; -D MESH_DEBUG=1 -D CONFIG_MBEDTLS_CERTIFICATE_BUNDLE=y -D ESP32_CPU_FREQ=160 + -D WITH_SNMP=1 build_src_filter = ${Station_G2.build_src_filter} + + + + + + +<../examples/simple_room_server> lib_deps = ${Station_G2.lib_deps} @@ -210,6 +212,7 @@ lib_deps = arduino-libraries/NTPClient JChristensen/Timezone paulstoffregen/Time@1.6.1 + 0neblock/SNMP_Agent [env:Station_G2_room_server] extends = Station_G2 From 643a1ba41b0d7808c44eea514a29223a9318d834 Mon Sep 17 00:00:00 2001 From: agessaman Date: Wed, 22 Jul 2026 11:29:54 -0700 Subject: [PATCH 17/87] build.sh: put the published build number in observer asset filenames Stamp FIRMWARE_BUILD_NUMBER into FIRMWARE_VERSION_STRING (the output filename), so production assets become -v1.16.0.N-.bin and the web flasher's Version dropdown (parsed from the asset name by the /releases Worker) shows the true published build instead of the bare base version. The build-number suffix is computed once, up front, and now feeds BOTH the filename and the embedded -DFIRMWARE_VERSION (byte-for-byte unchanged, it already carried .N). Local builds without the counter still produce 3-segment names, so OTA and every filename parser stay backward-compatible. Co-Authored-By: Claude Opus 4.8 (1M context) --- build.sh | 37 +++++++++++++++++++++++-------------- 1 file changed, 23 insertions(+), 14 deletions(-) diff --git a/build.sh b/build.sh index 8ea83b40..bd4967fb 100755 --- a/build.sh +++ b/build.sh @@ -133,9 +133,24 @@ build_firmware() { exit 1 fi + # Observer build number: when CI provides FIRMWARE_BUILD_NUMBER (the per-base + # published-build counter), it becomes a 4th version component (e.g. .5 -> + # v1.16.0.5). Computed up front because it now feeds BOTH the filename and the + # embedded version. Local dev builds leave it unset → no 4th component. + BUILD_NUMBER_SUFFIX="" + if [ -n "$FIRMWARE_BUILD_NUMBER" ]; then + BUILD_NUMBER_SUFFIX=".${FIRMWARE_BUILD_NUMBER}" + fi + # set firmware version string (used for the output filename) - # e.g: v1.0.0-abcdef - FIRMWARE_VERSION_STRING="${FIRMWARE_VERSION}-${COMMIT_HASH}" + # e.g: v1.0.0-abcdef — or v1.16.0.5-abcdef with a build number. The build + # number is now IN the filename so the web flasher's Version dropdown (parsed + # from the asset name by the /releases Worker) shows the true published build, + # matching the embedded version that `ver` reports. Every filename parser + # (flasher gen-slim-manifests ASSET_RE, the /releases Worker VERSION_RE, + # flasher.js stale-URL recovery) accepts an optional 4th "." component + # between version and hash. + FIRMWARE_VERSION_STRING="${FIRMWARE_VERSION}${BUILD_NUMBER_SUFFIX}-${COMMIT_HASH}" # craft filename # e.g: RAK_4631_Repeater-v1.0.0-SHA @@ -143,23 +158,17 @@ build_firmware() { # Tag the *embedded* version for observer builds, e.g. v1.0.0-observer-abcdef, # so `ver`, the MQTT firmware_version/client_version, and SNMP all identify the - # fork. The filename above is intentionally left untagged: the env name already - # contains "observer", and the web flasher keys off that existing pattern. + # fork. The filename above carries the same version + build number but no + # variant tag: the env name already contains "observer", and the web flasher + # keys off that existing pattern. VARIANT_TAG="" case "$1" in *observer*) VARIANT_TAG="-observer" ;; esac - # Observer build number: when CI provides FIRMWARE_BUILD_NUMBER (the per-base - # published-build counter), append it as a 4th version component so the node - # reports e.g. v1.16.0.5-observer-abcdef and `ota check` can show how many - # builds behind it is. Local dev builds leave it unset → no 4th component. - # The *filename* (FIRMWARE_VERSION_STRING above) is deliberately left without - # the build number so assets stay -v-.bin. - BUILD_NUMBER_SUFFIX="" - if [ -n "$FIRMWARE_BUILD_NUMBER" ]; then - BUILD_NUMBER_SUFFIX=".${FIRMWARE_BUILD_NUMBER}" - fi + # Embedded version: base + build number (4th component) + variant tag + hash, + # e.g. v1.16.0.5-observer-abcdef, so the node reports its build and `ota check` + # can show how many builds behind it is. EMBEDDED_VERSION_STRING="${FIRMWARE_VERSION}${BUILD_NUMBER_SUFFIX}${VARIANT_TAG}-${COMMIT_HASH}" # add firmware version info to end of existing platformio build flags in environment vars. From cfff9d5cb7a86778b5e32ad5924683d63c357c08 Mon Sep 17 00:00:00 2001 From: agessaman Date: Fri, 24 Jul 2026 15:29:17 -0700 Subject: [PATCH 18/87] fix(mqtt): update eastidahomesh broker preset --- MQTT_IMPLEMENTATION.md | 2 +- src/helpers/MQTTPresets.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/MQTT_IMPLEMENTATION.md b/MQTT_IMPLEMENTATION.md index ba0fe5d2..cb94c628 100644 --- a/MQTT_IMPLEMENTATION.md +++ b/MQTT_IMPLEMENTATION.md @@ -109,7 +109,7 @@ The MQTT bridge uses a slot-based architecture with up to 6 concurrent connectio | `ctmesh` | mqtt.ctmesh.org:1883 | Username/password (fixed in firmware) | Plain MQTT | | `chimesh` | wss://mqtt.chimesh.org:443 | JWT (Ed25519) | WSS | | `meshat.se` | meshcore-mqtt.meshat.se:443 | JWT (Ed25519) | WSS | -| `eastidahomesh` | wss://broker.eastidahomesh.net:443 | None | WSS | +| `eastidahomesh` | mqtt://live.eastidahomesh.com:1883 | None | Plain MQTT | | `coloradomesh` | wss://mqtt.meshcore.coloradomesh.org:1883 | JWT (Ed25519) | WSS | | `dutchmeshcore-1` | collector1.dutchmeshcore.nl:443 | JWT (Ed25519) | WSS | | `dutchmeshcore-2` | collector2.dutchmeshcore.nl:443 | JWT (Ed25519) | WSS | diff --git a/src/helpers/MQTTPresets.h b/src/helpers/MQTTPresets.h index 1fed7f49..2a8d4bdd 100644 --- a/src/helpers/MQTTPresets.h +++ b/src/helpers/MQTTPresets.h @@ -153,7 +153,7 @@ static const MQTTPresetDef MQTT_PRESETS[MQTT_PRESET_COUNT] = { { "ctmesh", "mqtt://mqtt.ctmesh.org:1883", nullptr, nullptr, MQTT_AUTH_USERPASS, MQTT_TOPIC_MESHCORE, 0, true, 60, "meshdev", "large4cats"}, { "chimesh", "wss://mqtt.chimesh.org:443", "mqtt.chimesh.org", ISRG_ROOT_X1, MQTT_AUTH_JWT, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, { "meshat.se", "wss://meshcore-mqtt.meshat.se:443", "meshcore-mqtt.meshat.se", ISRG_ROOT_X1, MQTT_AUTH_JWT, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, - { "eastidahomesh", "wss://broker.eastidahomesh.net:443", nullptr, ISRG_ROOT_X1, MQTT_AUTH_NONE, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, + { "eastidahomesh", "mqtt://live.eastidahomesh.com:1883", nullptr, nullptr, MQTT_AUTH_NONE, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, { "coloradomesh", "wss://mqtt.meshcore.coloradomesh.org:443","mqtt.meshcore.coloradomesh.org", ISRG_ROOT_X1, MQTT_AUTH_JWT, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, { "dutchmeshcore-1", "wss://collector1.dutchmeshcore.nl:443/mqtt", "collector1.dutchmeshcore.nl", GTS_ROOT_R4, MQTT_AUTH_JWT, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, { "dutchmeshcore-2", "wss://collector2.dutchmeshcore.nl:443/mqtt", "collector2.dutchmeshcore.nl", GTS_ROOT_R4, MQTT_AUTH_JWT, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, From 8fc96f68f2646ea7af864ec029f03b1ab43f3dcd Mon Sep 17 00:00:00 2001 From: agessaman Date: Fri, 24 Jul 2026 15:31:19 -0700 Subject: [PATCH 19/87] fix(mqtt): update eastidahomesh broker preset --- MQTT_IMPLEMENTATION.md | 2 +- src/helpers/MQTTPresets.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/MQTT_IMPLEMENTATION.md b/MQTT_IMPLEMENTATION.md index 40e56fd3..f3d1b772 100644 --- a/MQTT_IMPLEMENTATION.md +++ b/MQTT_IMPLEMENTATION.md @@ -109,7 +109,7 @@ The MQTT bridge uses a slot-based architecture with up to 6 concurrent connectio | `ctmesh` | mqtt.ctmesh.org:1883 | Username/password (fixed in firmware) | Plain MQTT | | `chimesh` | wss://mqtt.chimesh.org:443 | JWT (Ed25519) | WSS | | `meshat.se` | meshcore-mqtt.meshat.se:443 | JWT (Ed25519) | WSS | -| `eastidahomesh` | wss://broker.eastidahomesh.net:443 | None | WSS | +| `eastidahomesh` | mqtt://live.eastidahomesh.com:1883 | None | Plain MQTT | | `coloradomesh` | wss://mqtt.meshcore.coloradomesh.org:1883 | JWT (Ed25519) | WSS | | `dutchmeshcore-1` | collector1.dutchmeshcore.nl:443 | JWT (Ed25519) | WSS | | `dutchmeshcore-2` | collector2.dutchmeshcore.nl:443 | JWT (Ed25519) | WSS | diff --git a/src/helpers/MQTTPresets.h b/src/helpers/MQTTPresets.h index bf263799..6264fa20 100644 --- a/src/helpers/MQTTPresets.h +++ b/src/helpers/MQTTPresets.h @@ -150,7 +150,7 @@ static const MQTTPresetDef MQTT_PRESETS[MQTT_PRESET_COUNT] = { { "ctmesh", "mqtt://mqtt.ctmesh.org:1883", nullptr, nullptr, MQTT_AUTH_USERPASS, MQTT_TOPIC_MESHCORE, 0, true, 60, "meshdev", "large4cats"}, { "chimesh", "wss://mqtt.chimesh.org:443", "mqtt.chimesh.org", ISRG_ROOT_X1, MQTT_AUTH_JWT, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, { "meshat.se", "wss://meshcore-mqtt.meshat.se:443", "meshcore-mqtt.meshat.se", ISRG_ROOT_X1, MQTT_AUTH_JWT, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, - { "eastidahomesh", "wss://broker.eastidahomesh.net:443", nullptr, ISRG_ROOT_X1, MQTT_AUTH_NONE, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, + { "eastidahomesh", "mqtt://live.eastidahomesh.com:1883", nullptr, nullptr, MQTT_AUTH_NONE, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, { "coloradomesh", "wss://mqtt.meshcore.coloradomesh.org:443","mqtt.meshcore.coloradomesh.org", ISRG_ROOT_X1, MQTT_AUTH_JWT, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, { "dutchmeshcore-1", "wss://collector1.dutchmeshcore.nl:443/mqtt", "collector1.dutchmeshcore.nl", GTS_ROOT_R4, MQTT_AUTH_JWT, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, { "dutchmeshcore-2", "wss://collector2.dutchmeshcore.nl:443/mqtt", "collector2.dutchmeshcore.nl", GTS_ROOT_R4, MQTT_AUTH_JWT, MQTT_TOPIC_MESHCORE, 0, true, 55, nullptr, nullptr }, From 5313aa01b1b0129ac60e96e14787d36b53764845 Mon Sep 17 00:00:00 2001 From: agessaman Date: Fri, 24 Jul 2026 16:33:12 -0700 Subject: [PATCH 20/87] ci: stop the docs sync from cancelling the observer firmware build MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit build-observer-firmwares.yml and sync-flasher-content.yml shared the `flasher-publish` concurrency group. A mixed code+docs commit queues both, and GitHub cancels any run left *pending* in an occupied group — cancel-in-progress: false only shields runs already in progress. On 8fc96f68 the build lost that race and was cancelled in 1s with zero jobs, so nothing was published. Give each workflow its own group, and make the flasher push tolerate the concurrent writes the shared group was guarding against: rebase onto the sibling's commit and retry, up to 3 attempts. Both workflows generate byte-identical docs and changelog content, so the loser's commit normally rebases away to nothing and the push reports up-to-date. --- .../workflows/build-observer-firmwares.yml | 29 ++++++++++++++++--- .github/workflows/sync-flasher-content.yml | 28 +++++++++++++++--- 2 files changed, 49 insertions(+), 8 deletions(-) diff --git a/.github/workflows/build-observer-firmwares.yml b/.github/workflows/build-observer-firmwares.yml index a33ac359..fb366277 100644 --- a/.github/workflows/build-observer-firmwares.yml +++ b/.github/workflows/build-observer-firmwares.yml @@ -26,10 +26,13 @@ on: - '.vscode/**' - '.claude/**' -# Serialize with sync-flasher-content.yml so the two workflows never push to the -# flasher repo at the same time (shared group name across both workflows). +# Own group: serialize builds against each other only. Do NOT share this group +# with sync-flasher-content.yml — a mixed code+docs commit triggers both, and a +# run that is still pending when its sibling is queued gets cancelled, which +# silently skipped the firmware build. Concurrent flasher pushes are instead made +# safe by the rebase-retry in "Commit & Push Flasher Config". concurrency: - group: flasher-publish + group: observer-firmware-build cancel-in-progress: false env: @@ -306,4 +309,22 @@ jobs: git config user.name "meshcore-bot" git config user.email "noreply@gessaman.com" git commit -m "Update observer firmware to ${{ steps.sha.outputs.short }} (build ${FIRMWARE_VERSION}.${{ needs.enumerate.outputs.build_number }})" - git push + # sync-flasher-content.yml can be pushing the same docs/changelog right + # now (a mixed code+docs commit triggers both), so a non-fast-forward is + # expected rather than fatal: rebase onto the sibling's commit and retry. + # If the rebase leaves nothing, git reports up-to-date and we exit clean. + br=$(git rev-parse --abbrev-ref HEAD) + for attempt in 1 2 3; do + if git push origin "HEAD:$br"; then + exit 0 + fi + echo "push rejected (attempt $attempt); rebasing onto origin/$br" + git fetch origin "$br" + git rebase "origin/$br" || { + git rebase --abort || true + echo "::error::flasher rebase conflicted; re-run this workflow to republish" + exit 1 + } + done + echo "::error::could not push flasher changes after 3 attempts" + exit 1 diff --git a/.github/workflows/sync-flasher-content.yml b/.github/workflows/sync-flasher-content.yml index c534c2f4..a5685a9c 100644 --- a/.github/workflows/sync-flasher-content.yml +++ b/.github/workflows/sync-flasher-content.yml @@ -20,10 +20,12 @@ on: - 'docs/**' - 'scripts/gen_changelog.py' -# Shared with build-observer-firmwares.yml so the two never push to the flasher -# repo concurrently (a mixed code+docs commit can trigger both). +# Own group: serialize doc syncs against each other only. Deliberately NOT shared +# with build-observer-firmwares.yml — a shared group let one workflow cancel the +# other's pending run (a mixed code+docs commit triggers both). Concurrent flasher +# pushes are handled by the rebase-retry in "Commit & Push Flasher Content". concurrency: - group: flasher-publish + group: flasher-docs-sync cancel-in-progress: false jobs: @@ -73,4 +75,22 @@ jobs: git config user.name "meshcore-bot" git config user.email "noreply@gessaman.com" git commit -am "Sync docs & changelog from MeshCore ${GITHUB_SHA::7}" - git push + # build-observer-firmwares.yml can be pushing the same docs/changelog + # right now (a mixed code+docs commit triggers both), so a non-fast-forward + # is expected rather than fatal: rebase onto its commit and retry. If the + # rebase leaves nothing, git reports up-to-date and we exit clean. + br=$(git rev-parse --abbrev-ref HEAD) + for attempt in 1 2 3; do + if git push origin "HEAD:$br"; then + exit 0 + fi + echo "push rejected (attempt $attempt); rebasing onto origin/$br" + git fetch origin "$br" + git rebase "origin/$br" || { + git rebase --abort || true + echo "::error::flasher rebase conflicted; re-run this workflow to resync" + exit 1 + } + done + echo "::error::could not push flasher changes after 3 attempts" + exit 1 From 7ea93a640e366f953cae2170ccfa1bf9d59221f7 Mon Sep 17 00:00:00 2001 From: agessaman Date: Fri, 24 Jul 2026 20:04:31 -0700 Subject: [PATCH 21/87] ci: rebase with --autostash on the flasher push retry A plain `git rebase` refuses to run when the checkout has unstaged changes, so the retry path could fail instead of retrying. The beta workflow stages a scoped subset on purpose (production's files must never ride along), which leaves the rest of the checkout dirty and hits this exactly; keep all three flasher pushes identical so it cannot resurface here either. Unstaged content stays unstaged, so nothing extra gets published. --- .github/workflows/build-observer-firmwares.yml | 4 +++- .github/workflows/sync-flasher-content.yml | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build-observer-firmwares.yml b/.github/workflows/build-observer-firmwares.yml index fb366277..35514591 100644 --- a/.github/workflows/build-observer-firmwares.yml +++ b/.github/workflows/build-observer-firmwares.yml @@ -320,7 +320,9 @@ jobs: fi echo "push rejected (attempt $attempt); rebasing onto origin/$br" git fetch origin "$br" - git rebase "origin/$br" || { + # --autostash so a dirty tree can never make the rebase refuse to run; + # nothing unstaged gets published either way. + git rebase --autostash "origin/$br" || { git rebase --abort || true echo "::error::flasher rebase conflicted; re-run this workflow to republish" exit 1 diff --git a/.github/workflows/sync-flasher-content.yml b/.github/workflows/sync-flasher-content.yml index a5685a9c..93ef1b34 100644 --- a/.github/workflows/sync-flasher-content.yml +++ b/.github/workflows/sync-flasher-content.yml @@ -86,7 +86,9 @@ jobs: fi echo "push rejected (attempt $attempt); rebasing onto origin/$br" git fetch origin "$br" - git rebase "origin/$br" || { + # --autostash so a dirty tree can never make the rebase refuse to run; + # nothing unstaged gets published either way. + git rebase --autostash "origin/$br" || { git rebase --abort || true echo "::error::flasher rebase conflicted; re-run this workflow to resync" exit 1 From c69186439717376f8172ef411ae1d9d3d4a91988 Mon Sep 17 00:00:00 2001 From: agessaman Date: Fri, 24 Jul 2026 20:04:55 -0700 Subject: [PATCH 22/87] ci: give the beta channel its own concurrency group MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Concurrency groups are repo-wide, not per-branch, so the beta build sharing `flasher-publish` with the production workflows meant an in-flight beta build forced a production build into the pending state — where the next run queued into that group cancelled it outright. That is what silently dropped the production build of the eastidahomesh preset fix: the beta build held the group from 22:29:24 to 22:37:34, production's build queued at 22:31:26 and went pending, and its docs sync evicted it a second later. Beta now uses its own group and, since the channels are no longer serialized against each other, its flasher push rebase-retries the same way production's does (with --autostash, because the scoped add here leaves the checkout dirty). Also refresh the two workflow copies this branch never received: they still triggered on the retired mqtt-bridge-implementation-flex branch, so they were dead on push while still sitting in the shared group. Taking production's current versions verbatim keeps them inert here and conflict-free on promotion. --- .../build-observer-firmwares-beta.yml | 33 +++++++-- .../workflows/build-observer-firmwares.yml | 69 ++++++++++++++----- .github/workflows/sync-flasher-content.yml | 32 +++++++-- 3 files changed, 108 insertions(+), 26 deletions(-) diff --git a/.github/workflows/build-observer-firmwares-beta.yml b/.github/workflows/build-observer-firmwares-beta.yml index 73c6fd40..ddb89647 100644 --- a/.github/workflows/build-observer-firmwares-beta.yml +++ b/.github/workflows/build-observer-firmwares-beta.yml @@ -39,10 +39,13 @@ on: - '.vscode/**' - '.claude/**' -# Shared with build-observer-firmwares.yml and sync-flasher-content.yml so the -# workflows never push to the flasher repo at the same time. +# Own group: serialize beta builds against each other only. Concurrency groups are +# repo-wide, not per-branch, so sharing one with the production workflows let an +# in-flight beta build force a production build into the pending state, where the +# next run queued into the group cancelled it outright. Concurrent flasher pushes +# are instead made safe by the rebase-retry in "Commit & Push Beta Artifacts". concurrency: - group: flasher-publish + group: observer-firmware-beta-build cancel-in-progress: false env: @@ -346,4 +349,26 @@ jobs: git config user.name "meshcore-bot" git config user.email "noreply@gessaman.com" git commit -m "Update BETA observer firmware to ${{ steps.sha.outputs.short }} (build ${FIRMWARE_VERSION}.${{ needs.enumerate.outputs.build_number }})" - git push + # The production workflows can be pushing to the flasher repo right now + # (no shared concurrency group any more), so a non-fast-forward is expected + # rather than fatal: rebase onto their commit and retry. The scoped add + # above keeps this commit inside the beta paths, so a conflict with + # production's files is not possible. + br=$(git rev-parse --abbrev-ref HEAD) + for attempt in 1 2 3; do + if git push origin "HEAD:$br"; then + exit 0 + fi + echo "push rejected (attempt $attempt); rebasing onto origin/$br" + git fetch origin "$br" + # --autostash: the scoped add above leaves unrelated edits in the + # checkout unstaged on purpose, and plain rebase refuses to run with a + # dirty tree. They stay unstaged, so they still never get published. + git rebase --autostash "origin/$br" || { + git rebase --abort || true + echo "::error::flasher rebase conflicted; re-run this workflow to republish" + exit 1 + } + done + echo "::error::could not push beta changes after 3 attempts" + exit 1 diff --git a/.github/workflows/build-observer-firmwares.yml b/.github/workflows/build-observer-firmwares.yml index 3c13bf0c..35514591 100644 --- a/.github/workflows/build-observer-firmwares.yml +++ b/.github/workflows/build-observer-firmwares.yml @@ -7,7 +7,7 @@ on: workflow_dispatch: push: branches: - - mqtt-bridge-implementation-flex + - observer-firmware # Only rebuild firmware when something that affects the binaries changes. # Docs, the changelog, the changelog generator, CI files, and repo-meta # files do not change firmware output — docs/changelog are handled by @@ -26,10 +26,13 @@ on: - '.vscode/**' - '.claude/**' -# Serialize with sync-flasher-content.yml so the two workflows never push to the -# flasher repo at the same time (shared group name across both workflows). +# Own group: serialize builds against each other only. Do NOT share this group +# with sync-flasher-content.yml — a mixed code+docs commit triggers both, and a +# run that is still pending when its sibling is queued gets cancelled, which +# silently skipped the firmware build. Concurrent flasher pushes are instead made +# safe by the rebase-retry in "Commit & Push Flasher Config". concurrency: - group: flasher-publish + group: observer-firmware-build cancel-in-progress: false env: @@ -38,6 +41,10 @@ env: FIRMWARE_VERSION: v1.16.0 # Rolling release tag that hosts the latest observer binaries. RELEASE_TAG: observer-mqtt-latest + # Download host serving RELEASE_TAG's assets (cloudflare-worker in the flasher + # repo). Baked into the slim OTA manifests' file URLs; must stay consistent + # with config.json's staticPath. + STATIC_PATH: https://observer-fw.gessaman.com jobs: @@ -185,6 +192,13 @@ jobs: # not published as release assets (and they'd break the hash-based prune). gh release upload "$RELEASE_TAG" $(find out -maxdepth 1 -type f ! -name '*.partsig') --clobber + # Keep the release body in sync with the repo's notes source: the + # /releases feed serves it as the flasher dropdown's changelog + # (config.json no longer carries per-version notes). Non-fatal — + # stale notes beat a red build whose binaries are already live. + gh release edit "$RELEASE_TAG" --notes-file firmware-notes.html \ + || echo "WARNING: release notes sync failed" >&2 + # Prune old assets, but RETAIN the most recent KEEP_BUILDS build hashes # (not just the current one). The flasher SPA reads config.json once at # page load and the embedded git short-hash is what links it to assets; @@ -222,22 +236,23 @@ jobs: token: ${{ secrets.FLASHER_DISPATCH_TOKEN }} path: flasher - - name: Update Flasher Config - run: | - python3 flasher/scripts/update-firmware.py \ - "${{ steps.sha.outputs.short }}" \ - "$GITHUB_WORKSPACE/firmware-notes.html" + # update-firmware.py is no longer called here: config.json's observer + # entries are github release defs (no embedded filenames to rewrite), the + # flasher's versions come from the Worker's /releases feed, and the + # changelog notes ride the release body (see the publish step). - name: Generate Slim Per-Variant Manifests + Persist Build Counter env: BUILD_NUMBER: ${{ needs.enumerate.outputs.build_number }} run: | # Derive the slim per-variant manifests (flasher/v/.json) that the - # firmware fetches for `ota check`/`ota update`, from the just-updated - # config.json (single source of truth), stamping this build's number. - # Then persist the counter so the next run increments from here. + # firmware fetches for `ota check`/`ota update`, from the build output + # in out/ — the assets actually uploaded to the release — stamping this + # build's number. Then persist the counter so the next run increments + # from here. python3 flasher/scripts/gen-slim-manifests.py \ - --config flasher/config.json \ + --bin-dir out \ + --static-path "$STATIC_PATH" \ --out-dir flasher/v \ --base-version "$FIRMWARE_VERSION" \ --build "$BUILD_NUMBER" \ @@ -283,9 +298,9 @@ jobs: - name: Commit & Push Flasher Config working-directory: flasher run: | - # Stage everything: update-firmware.py edits config.json in place, but the - # slim manifests (v/*.json) and the build counter can be NEW files, which - # `commit -am` would miss — so add -A and check the staged diff. + # Stage everything: the slim manifests (v/*.json) and the build counter + # can be NEW files, which `commit -am` would miss — so add -A and check + # the staged diff. (config.json is no longer rewritten per build.) git add -A if git diff --cached --quiet; then echo "No flasher changes to commit." @@ -294,4 +309,24 @@ jobs: git config user.name "meshcore-bot" git config user.email "noreply@gessaman.com" git commit -m "Update observer firmware to ${{ steps.sha.outputs.short }} (build ${FIRMWARE_VERSION}.${{ needs.enumerate.outputs.build_number }})" - git push + # sync-flasher-content.yml can be pushing the same docs/changelog right + # now (a mixed code+docs commit triggers both), so a non-fast-forward is + # expected rather than fatal: rebase onto the sibling's commit and retry. + # If the rebase leaves nothing, git reports up-to-date and we exit clean. + br=$(git rev-parse --abbrev-ref HEAD) + for attempt in 1 2 3; do + if git push origin "HEAD:$br"; then + exit 0 + fi + echo "push rejected (attempt $attempt); rebasing onto origin/$br" + git fetch origin "$br" + # --autostash so a dirty tree can never make the rebase refuse to run; + # nothing unstaged gets published either way. + git rebase --autostash "origin/$br" || { + git rebase --abort || true + echo "::error::flasher rebase conflicted; re-run this workflow to republish" + exit 1 + } + done + echo "::error::could not push flasher changes after 3 attempts" + exit 1 diff --git a/.github/workflows/sync-flasher-content.yml b/.github/workflows/sync-flasher-content.yml index e2bd5033..93ef1b34 100644 --- a/.github/workflows/sync-flasher-content.yml +++ b/.github/workflows/sync-flasher-content.yml @@ -14,16 +14,18 @@ on: workflow_dispatch: push: branches: - - mqtt-bridge-implementation-flex + - observer-firmware paths: - '**.md' - 'docs/**' - 'scripts/gen_changelog.py' -# Shared with build-observer-firmwares.yml so the two never push to the flasher -# repo concurrently (a mixed code+docs commit can trigger both). +# Own group: serialize doc syncs against each other only. Deliberately NOT shared +# with build-observer-firmwares.yml — a shared group let one workflow cancel the +# other's pending run (a mixed code+docs commit triggers both). Concurrent flasher +# pushes are handled by the rebase-retry in "Commit & Push Flasher Content". concurrency: - group: flasher-publish + group: flasher-docs-sync cancel-in-progress: false jobs: @@ -73,4 +75,24 @@ jobs: git config user.name "meshcore-bot" git config user.email "noreply@gessaman.com" git commit -am "Sync docs & changelog from MeshCore ${GITHUB_SHA::7}" - git push + # build-observer-firmwares.yml can be pushing the same docs/changelog + # right now (a mixed code+docs commit triggers both), so a non-fast-forward + # is expected rather than fatal: rebase onto its commit and retry. If the + # rebase leaves nothing, git reports up-to-date and we exit clean. + br=$(git rev-parse --abbrev-ref HEAD) + for attempt in 1 2 3; do + if git push origin "HEAD:$br"; then + exit 0 + fi + echo "push rejected (attempt $attempt); rebasing onto origin/$br" + git fetch origin "$br" + # --autostash so a dirty tree can never make the rebase refuse to run; + # nothing unstaged gets published either way. + git rebase --autostash "origin/$br" || { + git rebase --abort || true + echo "::error::flasher rebase conflicted; re-run this workflow to resync" + exit 1 + } + done + echo "::error::could not push flasher changes after 3 attempts" + exit 1 From df070833ba176a56aa15e5a9b488190cd1588c23 Mon Sep 17 00:00:00 2001 From: agessaman Date: Fri, 24 Jul 2026 20:17:12 -0700 Subject: [PATCH 23/87] chore: untrack the .build-wt-028a5dca build worktree MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit It was committed as a gitlink (mode 160000) with no .gitmodules entry, so every CI checkout's cleanup step ended with "No url found for submodule path '.build-wt-028a5dca'" and a git exit-128 warning, and `git submodule` commands failed in fresh clones. The worktree itself is removed locally; commit 028a5dca stays reachable from this branch, so no history is affected. Ignore the pattern so a throwaway build worktree cannot be committed again, and add it to paths-ignore since such a directory can never change the binaries — that also keeps this removal from triggering a pointless full rebuild. --- .build-wt-028a5dca | 1 - .github/workflows/build-observer-firmwares.yml | 3 +++ .gitignore | 3 +++ 3 files changed, 6 insertions(+), 1 deletion(-) delete mode 160000 .build-wt-028a5dca diff --git a/.build-wt-028a5dca b/.build-wt-028a5dca deleted file mode 160000 index 028a5dca..00000000 --- a/.build-wt-028a5dca +++ /dev/null @@ -1 +0,0 @@ -Subproject commit 028a5dcaddd69b4be80cc595d33d450be1c46e8b diff --git a/.github/workflows/build-observer-firmwares.yml b/.github/workflows/build-observer-firmwares.yml index 35514591..804139b7 100644 --- a/.github/workflows/build-observer-firmwares.yml +++ b/.github/workflows/build-observer-firmwares.yml @@ -25,6 +25,9 @@ on: - 'LICENSE' - '.vscode/**' - '.claude/**' + # Throwaway build worktrees never affect the binaries. + - '.build-wt-*' + - '.build-wt-*/**' # Own group: serialize builds against each other only. Do NOT share this group # with sync-flasher-content.yml — a mixed code+docs commit triggers both, and a diff --git a/.gitignore b/.gitignore index bee82048..6e7f5748 100644 --- a/.gitignore +++ b/.gitignore @@ -24,3 +24,6 @@ platformio.local.ini .claude/* .cursorrules src/helpers/esp32/WebConfigHtml.h +# Throwaway build worktrees; committing one adds a stray gitlink that makes every +# CI checkout warn "No url found for submodule path ... in .gitmodules". +.build-wt-*/ From 0c86e0dcdee3222fbe03e908223c5cb05cbab57d Mon Sep 17 00:00:00 2001 From: agessaman Date: Fri, 24 Jul 2026 20:18:20 -0700 Subject: [PATCH 24/87] chore: ignore throwaway build worktrees observer-firmware had a .build-wt- worktree committed as a gitlink, which made every CI checkout warn "No url found for submodule path ... in .gitmodules" and broke `git submodule` in fresh clones. This branch has no such entry; ignore the pattern so it stays that way, and skip it in both channels' paths-ignore since a build worktree can never change the binaries. The production build workflow is taken from observer-firmware verbatim, keeping the copy on this branch identical and conflict-free on promotion. --- .github/workflows/build-observer-firmwares-beta.yml | 3 +++ .github/workflows/build-observer-firmwares.yml | 3 +++ .gitignore | 5 ++++- 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/.github/workflows/build-observer-firmwares-beta.yml b/.github/workflows/build-observer-firmwares-beta.yml index ddb89647..3b27d4a0 100644 --- a/.github/workflows/build-observer-firmwares-beta.yml +++ b/.github/workflows/build-observer-firmwares-beta.yml @@ -38,6 +38,9 @@ on: - 'LICENSE' - '.vscode/**' - '.claude/**' + # Throwaway build worktrees never affect the binaries. + - '.build-wt-*' + - '.build-wt-*/**' # Own group: serialize beta builds against each other only. Concurrency groups are # repo-wide, not per-branch, so sharing one with the production workflows let an diff --git a/.github/workflows/build-observer-firmwares.yml b/.github/workflows/build-observer-firmwares.yml index 35514591..804139b7 100644 --- a/.github/workflows/build-observer-firmwares.yml +++ b/.github/workflows/build-observer-firmwares.yml @@ -25,6 +25,9 @@ on: - 'LICENSE' - '.vscode/**' - '.claude/**' + # Throwaway build worktrees never affect the binaries. + - '.build-wt-*' + - '.build-wt-*/**' # Own group: serialize builds against each other only. Do NOT share this group # with sync-flasher-content.yml — a mixed code+docs commit triggers both, and a diff --git a/.gitignore b/.gitignore index ae3815fb..d3d96b7e 100644 --- a/.gitignore +++ b/.gitignore @@ -24,4 +24,7 @@ ssl_certs/cacert.pem platformio.local.ini .cursor/* .claude/* -.cursorrules \ No newline at end of file +.cursorrules +# Throwaway build worktrees; committing one adds a stray gitlink that makes every +# CI checkout warn "No url found for submodule path ... in .gitmodules". +.build-wt-*/ From aba571ed7d005f0af736082e796ade8cc0dd5ce1 Mon Sep 17 00:00:00 2001 From: agessaman Date: Sun, 26 Jul 2026 14:03:04 -0700 Subject: [PATCH 25/87] feat(mqtt): enhance neighbor discovery with request management Add functionality to manage neighbor discovery requests in MyMesh. This includes handling the status of requests, updating the state upon transmission success or failure, and implementing a timeout mechanism for neighbor discovery queries. The changes improve the efficiency and reliability of neighbor discovery operations in MQTT-enabled environments. --- examples/simple_repeater/MyMesh.cpp | 191 ++++++++++++++++++++----- examples/simple_repeater/MyMesh.h | 29 ++-- examples/simple_room_server/MyMesh.cpp | 191 ++++++++++++++++++++----- examples/simple_room_server/MyMesh.h | 23 ++- src/Dispatcher.h | 1 + 5 files changed, 342 insertions(+), 93 deletions(-) diff --git a/examples/simple_repeater/MyMesh.cpp b/examples/simple_repeater/MyMesh.cpp index 2358581c..a8f3cb92 100644 --- a/examples/simple_repeater/MyMesh.cpp +++ b/examples/simple_repeater/MyMesh.cpp @@ -558,6 +558,18 @@ void MyMesh::logRx(mesh::Packet *pkt, int len, float score) { } void MyMesh::logTx(mesh::Packet *pkt, int len) { +#if defined(WITH_MQTT_NEIGHBORS) + if (neighbor_discover_active && pkt == neighbor_discover_request + && neighbor_discover_next < neighbor_discover_count) { + NeighborDiscoverEntry& entry = neighbor_discover[neighbor_discover_next]; + if (entry.status == ND_QUEUED) { + entry.status = ND_PENDING; + neighbor_discover_request = NULL; + neighbor_discover_until = futureMillis(neighborDiscoverQueryTimeoutMs()); + } + } +#endif + #ifdef WITH_MQTT_BRIDGE // MQTT bridge: always feed TX packets — bridge decides based on mqtt.tx setting if (bridge) bridge->sendPacket(pkt); @@ -587,6 +599,18 @@ void MyMesh::logTx(mesh::Packet *pkt, int len) { } void MyMesh::logTxFail(mesh::Packet *pkt, int len) { +#if defined(WITH_MQTT_NEIGHBORS) + if (neighbor_discover_active && pkt == neighbor_discover_request + && neighbor_discover_next < neighbor_discover_count) { + NeighborDiscoverEntry& entry = neighbor_discover[neighbor_discover_next]; + if (entry.status == ND_QUEUED) { + entry.status = ND_SEND_FAILED; + neighbor_discover_request = NULL; + neighbor_discover_until = 0; + } + } +#endif + if (_logging) { File f = openAppend(PACKET_LOG_FILE); if (f) { @@ -677,9 +701,9 @@ int MyMesh::searchPeersByHash(const uint8_t *hash) { // distinct from real ACL indices. if (neighbor_discover_active) { for (int i = 0; i < neighbor_discover_count && n < MAX_CLIENTS; i++) { - auto& nb = neighbours[neighbor_discover[i].neighbour_idx]; - if (acl.getClient(nb.id.pub_key, PUB_KEY_SIZE) != nullptr) continue; - if (nb.heard_timestamp > 0 && nb.id.isHashMatch(hash)) { + auto& entry = neighbor_discover[i]; + if (acl.getClient(entry.id.pub_key, PUB_KEY_SIZE) != nullptr) continue; + if (entry.heard_timestamp > 0 && entry.id.isHashMatch(hash)) { matching_peer_indexes[n++] = NEIGHBOR_DISCOVER_PEER_BASE + i; } } @@ -700,7 +724,7 @@ void MyMesh::getPeerSharedSecret(uint8_t *dest_secret, int peer_idx) { if (neighbor_discover_active && i >= NEIGHBOR_DISCOVER_PEER_BASE) { int oi = i - NEIGHBOR_DISCOVER_PEER_BASE; if (oi >= 0 && oi < neighbor_discover_count) { - self_id.calcSharedSecret(dest_secret, neighbours[neighbor_discover[oi].neighbour_idx].id); + self_id.calcSharedSecret(dest_secret, neighbor_discover[oi].id); return; } } @@ -757,8 +781,8 @@ void MyMesh::onPeerDataRecv(mesh::Packet *packet, uint8_t type, int sender_idx, // scope-query response from it lands here — match it against the overlay. if (neighbor_discover_active && type == PAYLOAD_TYPE_RESPONSE) { for (int oi = 0; oi < neighbor_discover_count; oi++) { - auto& nb = neighbours[neighbor_discover[oi].neighbour_idx]; - if (client->id.matches(nb.id) && handleNeighborDiscoverResponse(oi, data, len)) { + if (client->id.matches(neighbor_discover[oi].id) + && handleNeighborDiscoverResponse(oi, data, len)) { return; } } @@ -1041,10 +1065,12 @@ MyMesh::MyMesh(mesh::MainBoard &board, mesh::Radio &radio, mesh::MillisecondCloc #if defined(WITH_MQTT_NEIGHBORS) neighbor_discover_count = 0; + neighbor_discover_next = 0; neighbor_discover_active = false; neighbor_table_refresh_active = false; neighbor_table_refresh_periodic = false; neighbor_discover_until = 0; + neighbor_discover_request = NULL; next_neighbors_publish = 0; self_scopes_buf[0] = 0; #endif @@ -1806,7 +1832,11 @@ void MyMesh::getLocalScopes(char* buf, size_t len) { // Client side of the anon-regions request (the server side is handleAnonRegionsReq). // Inner payload: {tag(4)}{ANON_REQ_TYPE_REGIONS}{0x00 = zero-hop reply path}. -bool MyMesh::sendAnonRegionsReq(const mesh::Identity& target, uint32_t& tag) { +mesh::Packet* MyMesh::sendAnonRegionsReq(const mesh::Identity& target, uint32_t& tag) { + // RxReservePacketManager keeps a four-packet emergency floor. Preflight one + // extra free packet so its void queue API cannot silently shed this request. + if (_mgr->getFreeCount() < NEIGHBOR_DISCOVER_MIN_FREE_PACKETS) return NULL; + uint8_t secret[PUB_KEY_SIZE]; self_id.calcSharedSecret(secret, target); @@ -1817,9 +1847,33 @@ bool MyMesh::sendAnonRegionsReq(const mesh::Identity& target, uint32_t& tag) { inner[5] = 0x00; // request a zero-hop reply path mesh::Packet* pkt = createAnonDatagram(PAYLOAD_TYPE_ANON_REQ, self_id, target, secret, inner, sizeof(inner)); - if (!pkt) return false; + if (!pkt) return NULL; sendDirect(pkt, NULL, 0, 0); - return true; + return pkt; +} + +bool MyMesh::cancelNeighborDiscoverRequest() { + if (!neighbor_discover_request) return false; + for (int i = _mgr->getOutboundTotal() - 1; i >= 0; i--) { + if (_mgr->getOutboundByIdx(i) == neighbor_discover_request) { + mesh::Packet* pkt = _mgr->removeOutboundByIdx(i); + if (pkt) releasePacket(pkt); + neighbor_discover_request = NULL; + return true; + } + } + return false; +} + +// This timer starts after the request finishes transmitting. Allow the server +// delay, the responder's full CAD deferral window plus one maximum retry +// overshoot, and airtime for one priority-0 packet ahead of the response plus +// the response itself. The radio estimate scales with SF, bandwidth, coding +// rate, and preamble. +uint32_t MyMesh::neighborDiscoverQueryTimeoutMs() const { + uint32_t response_airtime = _radio->getEstAirtimeFor(MAX_PACKET_PAYLOAD + 2); + return SERVER_RESPONSE_DELAY + getCADFailMaxDuration() + 360UL + + response_airtime * 2UL; } // Match a RESPONSE against the pending overlay entry by tag; copy its scope @@ -1907,14 +1961,14 @@ void MyMesh::finishNeighborDiscover() { uint32_t now_secs = getRTCClock()->getCurrentTime(); for (int i = 0; i < neighbor_discover_count; i++) { - auto& nb = neighbours[neighbor_discover[i].neighbour_idx]; - mesh::Utils::toHex(pubkey_hex[i], nb.id.pub_key, PUB_KEY_SIZE); + auto& entry = neighbor_discover[i]; + mesh::Utils::toHex(pubkey_hex[i], entry.id.pub_key, PUB_KEY_SIZE); entries[i].pubkey_hex = pubkey_hex[i]; - entries[i].snr = nb.snr / 4.0f; - entries[i].heard_secs_ago = (nb.heard_timestamp > 0 && now_secs >= nb.heard_timestamp) - ? (now_secs - nb.heard_timestamp) : 0; - entries[i].scopes = neighbor_discover[i].scopes; - switch (neighbor_discover[i].status) { + entries[i].snr = entry.snr / 4.0f; + entries[i].heard_secs_ago = (entry.heard_timestamp > 0 && now_secs >= entry.heard_timestamp) + ? (now_secs - entry.heard_timestamp) : 0; + entries[i].scopes = entry.scopes; + switch (entry.status) { case ND_RESPONDED: entries[i].status = "responded"; break; case ND_SEND_FAILED: entries[i].status = "send_failed"; break; default: entries[i].status = "timeout"; break; @@ -1940,6 +1994,9 @@ void MyMesh::finishNeighborDiscover() { if (!json_buf) { neighbor_discover_active = false; neighbor_discover_count = 0; + neighbor_discover_next = 0; + neighbor_discover_until = 0; + neighbor_discover_request = NULL; if (_cli.getObserverPrefs()->mqtt_neighbors_enabled) { next_neighbors_publish = futureMillis(_cli.getObserverPrefs()->mqtt_neighbors_interval); } @@ -1969,27 +2026,67 @@ void MyMesh::finishNeighborDiscover() { neighbor_discover_active = false; neighbor_discover_count = 0; + neighbor_discover_next = 0; + neighbor_discover_until = 0; + neighbor_discover_request = NULL; if (_cli.getObserverPrefs()->mqtt_neighbors_enabled) { next_neighbors_publish = futureMillis(_cli.getObserverPrefs()->mqtt_neighbors_interval); } } -// Advance the scope-query phase; publish once all entries resolve or the window -// times out (stragglers marked ND_TIMEOUT). +// Advance the newest-first scope-query phase. Keep only one request in flight so +// its responder gets a clear reply opportunity and the packet pool stays free. void MyMesh::loopNeighborDiscover() { if (!neighbor_discover_active) return; - bool all_done = true; - for (int i = 0; i < neighbor_discover_count; i++) { - if (neighbor_discover[i].status == ND_PENDING) { all_done = false; break; } + if (neighbor_discover_next >= neighbor_discover_count) { + finishNeighborDiscover(); + return; } - if (!all_done && !millisHasNowPassed(neighbor_discover_until)) return; - if (!all_done) { - for (int i = 0; i < neighbor_discover_count; i++) { - if (neighbor_discover[i].status == ND_PENDING) neighbor_discover[i].status = ND_TIMEOUT; + + NeighborDiscoverEntry& entry = neighbor_discover[neighbor_discover_next]; + if (entry.status == ND_QUEUED) { + if (!millisHasNowPassed(neighbor_discover_until)) return; + if (cancelNeighborDiscoverRequest()) { + entry.status = ND_SEND_FAILED; + neighbor_discover_next++; + return; } + if (isCurrentOutbound(neighbor_discover_request)) { + neighbor_discover_until = futureMillis(neighborDiscoverQueryTimeoutMs()); + return; + } + neighbor_discover_request = NULL; // packet manager already shed it + entry.status = ND_SEND_FAILED; + neighbor_discover_next++; + return; + } + if (entry.status == ND_PENDING) { + if (!millisHasNowPassed(neighbor_discover_until)) return; + entry.status = ND_TIMEOUT; + neighbor_discover_next++; + return; + } + if (entry.status == ND_RESPONDED || entry.status == ND_SEND_FAILED) { + neighbor_discover_next++; + return; + } + if (entry.status != ND_UNSENT) { + neighbor_discover_next++; + return; + } + + uint32_t tag; + mesh::Packet* request = sendAnonRegionsReq(entry.id, tag); + if (request) { + entry.tag = tag; + entry.status = ND_QUEUED; + neighbor_discover_request = request; + neighbor_discover_until = futureMillis(NEIGHBOR_DISCOVER_QUEUE_TIMEOUT_MS); + } else { + entry.status = ND_SEND_FAILED; + neighbor_discover_next++; } - finishNeighborDiscover(); } // Shared precondition for starting a discovery: PSRAM present + bridge running. @@ -2001,8 +2098,8 @@ bool MyMesh::neighborDiscoverReady(char* reply) { return true; } -// Snapshot the neighbor table into the overlay and fire one anon-regions query -// per heard neighbour; arm the 30s scope-query window. +// Snapshot the neighbor table newest-first. loopNeighborDiscover() emits one +// anon-regions query at a time so hidden responders do not reply as a burst. bool MyMesh::startNeighborDiscover(char* reply) { if (neighbor_discover_active) { strcpy(reply, "Err - neighbor discover already active"); @@ -2016,26 +2113,44 @@ bool MyMesh::startNeighborDiscover(char* reply) { neighbor_discover_count = 0; for (int i = 0; i < MAX_NEIGHBOURS; i++) { if (neighbours[i].heard_timestamp > 0) { - neighbor_discover[neighbor_discover_count].neighbour_idx = (uint8_t)i; - neighbor_discover[neighbor_discover_count].scopes[0] = 0; - neighbor_discover[neighbor_discover_count].status = ND_PENDING; - uint32_t tag; - if (sendAnonRegionsReq(neighbours[i].id, tag)) { - neighbor_discover[neighbor_discover_count].tag = tag; - } else { - neighbor_discover[neighbor_discover_count].status = ND_SEND_FAILED; - } + NeighborDiscoverEntry& entry = neighbor_discover[neighbor_discover_count]; + entry.id = neighbours[i].id; + entry.heard_timestamp = neighbours[i].heard_timestamp; + entry.snr = neighbours[i].snr; + entry.scopes[0] = 0; + entry.tag = 0; + entry.status = ND_UNSENT; neighbor_discover_count++; } } + // Query the freshest/strongest entries first; pubkey makes ties deterministic. + for (int i = 1; i < neighbor_discover_count; i++) { + NeighborDiscoverEntry entry = neighbor_discover[i]; + int j = i; + while (j > 0) { + auto& rhs = neighbor_discover[j - 1]; + bool before = entry.heard_timestamp > rhs.heard_timestamp + || (entry.heard_timestamp == rhs.heard_timestamp && entry.snr > rhs.snr) + || (entry.heard_timestamp == rhs.heard_timestamp && entry.snr == rhs.snr + && memcmp(entry.id.pub_key, rhs.id.pub_key, PUB_KEY_SIZE) < 0); + if (!before) break; + neighbor_discover[j] = neighbor_discover[j - 1]; + j--; + } + neighbor_discover[j] = entry; + } + + neighbor_discover_next = 0; neighbor_discover_active = true; - neighbor_discover_until = futureMillis(NEIGHBOR_DISCOVER_TIMEOUT_MS); + neighbor_discover_until = 0; + neighbor_discover_request = NULL; if (neighbor_discover_count == 0) { finishNeighborDiscover(); strcpy(reply, "OK - neighbor discover started (0 neighbors, self only)"); } else { + loopNeighborDiscover(); // queue the first request now sprintf(reply, "OK - neighbor discover started (%u neighbors)", (unsigned)neighbor_discover_count); } return true; diff --git a/examples/simple_repeater/MyMesh.h b/examples/simple_repeater/MyMesh.h index 562ba93b..bfca869e 100644 --- a/examples/simple_repeater/MyMesh.h +++ b/examples/simple_repeater/MyMesh.h @@ -149,30 +149,38 @@ class MyMesh : public mesh::Mesh, public CommonCLICallbacks #if defined(WITH_MQTT_NEIGHBORS) // Neighbor-scope discovery: a snapshot of the neighbor table overlaid with an - // in-flight anon-regions query per neighbor, published to the MQTT neighbors - // topic once every neighbor has responded or the window times out. + // anon-regions query per neighbor, published to the MQTT neighbors topic once + // every neighbor has responded or timed out. enum NeighborDiscoverStatus : uint8_t { - ND_PENDING = 1, - ND_RESPONDED = 2, - ND_TIMEOUT = 3, - ND_SEND_FAILED = 4, + ND_UNSENT = 0, + ND_QUEUED = 1, + ND_PENDING = 2, + ND_RESPONDED = 3, + ND_TIMEOUT = 4, + ND_SEND_FAILED = 5, }; struct NeighborDiscoverEntry { - uint8_t neighbour_idx; // index into neighbours[] + mesh::Identity id; // immutable snapshot: neighbour table can change mid-pass + uint32_t heard_timestamp; + int8_t snr; // multiplied by 4 uint32_t tag; // anon-regions request tag we're waiting on char scopes[96]; // scope names from the response uint8_t status; // NeighborDiscoverStatus }; NeighborDiscoverEntry neighbor_discover[MAX_NEIGHBOURS]; uint8_t neighbor_discover_count; + uint8_t neighbor_discover_next; // newest-first entry currently being queried bool neighbor_discover_active; // scope-query phase in flight bool neighbor_table_refresh_active; // zero-hop table refresh (stage 1) in flight bool neighbor_table_refresh_periodic; // that refresh was kicked by the periodic timer - unsigned long neighbor_discover_until; // scope-query timeout deadline + unsigned long neighbor_discover_until; // current queue or response deadline + mesh::Packet* neighbor_discover_request; // request awaiting TX completion unsigned long next_neighbors_publish; // periodic publish deadline (0 = fire ASAP) char self_scopes_buf[96]; - bool sendAnonRegionsReq(const mesh::Identity& target, uint32_t& tag); + mesh::Packet* sendAnonRegionsReq(const mesh::Identity& target, uint32_t& tag); + bool cancelNeighborDiscoverRequest(); + uint32_t neighborDiscoverQueryTimeoutMs() const; bool neighborDiscoverReady(char* reply); bool startNeighborDiscover(char* reply); void loopNeighborDiscover(); @@ -182,7 +190,8 @@ class MyMesh : public mesh::Mesh, public CommonCLICallbacks // Overlay peer indices are offset by this base so onPeerDataRecv can tell a // discovery response apart from a normal ACL-client index. static const int NEIGHBOR_DISCOVER_PEER_BASE = 1000; - static const unsigned long NEIGHBOR_DISCOVER_TIMEOUT_MS = 30000; + static const unsigned long NEIGHBOR_DISCOVER_QUEUE_TIMEOUT_MS = 29000; + static const int NEIGHBOR_DISCOVER_MIN_FREE_PACKETS = 5; #endif void putNeighbour(const mesh::Identity& id, uint32_t timestamp, float snr); diff --git a/examples/simple_room_server/MyMesh.cpp b/examples/simple_room_server/MyMesh.cpp index 65ad8683..f161f68c 100644 --- a/examples/simple_room_server/MyMesh.cpp +++ b/examples/simple_room_server/MyMesh.cpp @@ -250,6 +250,18 @@ void MyMesh::logRx(mesh::Packet *pkt, int len, float score) { } } void MyMesh::logTx(mesh::Packet *pkt, int len) { +#if defined(WITH_MQTT_NEIGHBORS) + if (neighbor_discover_active && pkt == neighbor_discover_request + && neighbor_discover_next < neighbor_discover_count) { + NeighborDiscoverEntry& entry = neighbor_discover[neighbor_discover_next]; + if (entry.status == ND_QUEUED) { + entry.status = ND_PENDING; + neighbor_discover_request = NULL; + neighbor_discover_until = futureMillis(neighborDiscoverQueryTimeoutMs()); + } + } +#endif + #ifdef WITH_MQTT_BRIDGE // MQTT bridge: always feed TX packets — bridge decides based on mqtt.tx setting if (_prefs.bridge_enabled && bridge) bridge->sendPacket(pkt); @@ -273,6 +285,18 @@ void MyMesh::logTx(mesh::Packet *pkt, int len) { } } void MyMesh::logTxFail(mesh::Packet *pkt, int len) { +#if defined(WITH_MQTT_NEIGHBORS) + if (neighbor_discover_active && pkt == neighbor_discover_request + && neighbor_discover_next < neighbor_discover_count) { + NeighborDiscoverEntry& entry = neighbor_discover[neighbor_discover_next]; + if (entry.status == ND_QUEUED) { + entry.status = ND_SEND_FAILED; + neighbor_discover_request = NULL; + neighbor_discover_until = 0; + } + } +#endif + if (_logging) { File f = openAppend(PACKET_LOG_FILE); if (f) { @@ -426,12 +450,12 @@ int MyMesh::searchPeersByHash(const uint8_t *hash) { #if defined(WITH_MQTT_NEIGHBORS) if (neighbor_discover_active) { for (int i = 0; i < neighbor_discover_count && n < MAX_CLIENTS; i++) { - auto& nb = neighbours[neighbor_discover[i].neighbour_idx]; + auto& entry = neighbor_discover[i]; // ACL clients already have a matching peer entry and shared secret. Adding // a second overlay entry would decrypt first and intercept their normal // CLI/request traffic for the duration of discovery. - if (acl.getClient(nb.id.pub_key, PUB_KEY_SIZE) != nullptr) continue; - if (nb.heard_timestamp > 0 && nb.id.isHashMatch(hash)) { + if (acl.getClient(entry.id.pub_key, PUB_KEY_SIZE) != nullptr) continue; + if (entry.heard_timestamp > 0 && entry.id.isHashMatch(hash)) { matching_peer_indexes[n++] = NEIGHBOR_DISCOVER_PEER_BASE + i; } } @@ -451,7 +475,7 @@ void MyMesh::getPeerSharedSecret(uint8_t *dest_secret, int peer_idx) { if (neighbor_discover_active && i >= NEIGHBOR_DISCOVER_PEER_BASE) { int oi = i - NEIGHBOR_DISCOVER_PEER_BASE; if (oi >= 0 && oi < neighbor_discover_count) { - self_id.calcSharedSecret(dest_secret, neighbours[neighbor_discover[oi].neighbour_idx].id); + self_id.calcSharedSecret(dest_secret, neighbor_discover[oi].id); return; } } @@ -488,8 +512,8 @@ void MyMesh::onPeerDataRecv(mesh::Packet *packet, uint8_t type, int sender_idx, // scope-query response from it lands here — match it against the overlay. if (neighbor_discover_active && type == PAYLOAD_TYPE_RESPONSE) { for (int oi = 0; oi < neighbor_discover_count; oi++) { - auto& nb = neighbours[neighbor_discover[oi].neighbour_idx]; - if (client->id.matches(nb.id) && handleNeighborDiscoverResponse(oi, data, len)) { + if (client->id.matches(neighbor_discover[oi].id) + && handleNeighborDiscoverResponse(oi, data, len)) { return; } } @@ -860,10 +884,12 @@ MyMesh::MyMesh(mesh::MainBoard &board, mesh::Radio &radio, mesh::MillisecondCloc pending_discover_tag = 0; pending_discover_until = 0; neighbor_discover_count = 0; + neighbor_discover_next = 0; neighbor_discover_active = false; neighbor_table_refresh_active = false; neighbor_table_refresh_periodic = false; neighbor_discover_until = 0; + neighbor_discover_request = NULL; next_neighbors_publish = 0; self_scopes_buf[0] = 0; memset(neighbours, 0, sizeof(neighbours)); @@ -1653,7 +1679,11 @@ void MyMesh::getLocalScopes(char* buf, size_t len) { // Client side of the anon-regions request (the server side is handleAnonRegionsReq). // Inner payload: {tag(4)}{ANON_REQ_TYPE_REGIONS}{0x00 = zero-hop reply path}. -bool MyMesh::sendAnonRegionsReq(const mesh::Identity& target, uint32_t& tag) { +mesh::Packet* MyMesh::sendAnonRegionsReq(const mesh::Identity& target, uint32_t& tag) { + // RxReservePacketManager keeps a four-packet emergency floor. Preflight one + // extra free packet so its void queue API cannot silently shed this request. + if (_mgr->getFreeCount() < NEIGHBOR_DISCOVER_MIN_FREE_PACKETS) return NULL; + uint8_t secret[PUB_KEY_SIZE]; self_id.calcSharedSecret(secret, target); @@ -1664,9 +1694,33 @@ bool MyMesh::sendAnonRegionsReq(const mesh::Identity& target, uint32_t& tag) { inner[5] = 0x00; // request a zero-hop reply path mesh::Packet* pkt = createAnonDatagram(PAYLOAD_TYPE_ANON_REQ, self_id, target, secret, inner, sizeof(inner)); - if (!pkt) return false; + if (!pkt) return NULL; sendDirect(pkt, NULL, 0, 0); - return true; + return pkt; +} + +bool MyMesh::cancelNeighborDiscoverRequest() { + if (!neighbor_discover_request) return false; + for (int i = _mgr->getOutboundTotal() - 1; i >= 0; i--) { + if (_mgr->getOutboundByIdx(i) == neighbor_discover_request) { + mesh::Packet* pkt = _mgr->removeOutboundByIdx(i); + if (pkt) releasePacket(pkt); + neighbor_discover_request = NULL; + return true; + } + } + return false; +} + +// This timer starts after the request finishes transmitting. Allow the server +// delay, the responder's full CAD deferral window plus one maximum retry +// overshoot, and airtime for one priority-0 packet ahead of the response plus +// the response itself. The radio estimate scales with SF, bandwidth, coding +// rate, and preamble. +uint32_t MyMesh::neighborDiscoverQueryTimeoutMs() const { + uint32_t response_airtime = _radio->getEstAirtimeFor(MAX_PACKET_PAYLOAD + 2); + return SERVER_RESPONSE_DELAY + getCADFailMaxDuration() + 360UL + + response_airtime * 2UL; } // Match a RESPONSE against the pending overlay entry by tag; copy its scope @@ -1754,14 +1808,14 @@ void MyMesh::finishNeighborDiscover() { uint32_t now_secs = getRTCClock()->getCurrentTime(); for (int i = 0; i < neighbor_discover_count; i++) { - auto& nb = neighbours[neighbor_discover[i].neighbour_idx]; - mesh::Utils::toHex(pubkey_hex[i], nb.id.pub_key, PUB_KEY_SIZE); + auto& entry = neighbor_discover[i]; + mesh::Utils::toHex(pubkey_hex[i], entry.id.pub_key, PUB_KEY_SIZE); entries[i].pubkey_hex = pubkey_hex[i]; - entries[i].snr = nb.snr / 4.0f; - entries[i].heard_secs_ago = (nb.heard_timestamp > 0 && now_secs >= nb.heard_timestamp) - ? (now_secs - nb.heard_timestamp) : 0; - entries[i].scopes = neighbor_discover[i].scopes; - switch (neighbor_discover[i].status) { + entries[i].snr = entry.snr / 4.0f; + entries[i].heard_secs_ago = (entry.heard_timestamp > 0 && now_secs >= entry.heard_timestamp) + ? (now_secs - entry.heard_timestamp) : 0; + entries[i].scopes = entry.scopes; + switch (entry.status) { case ND_RESPONDED: entries[i].status = "responded"; break; case ND_SEND_FAILED: entries[i].status = "send_failed"; break; default: entries[i].status = "timeout"; break; @@ -1787,6 +1841,9 @@ void MyMesh::finishNeighborDiscover() { if (!json_buf) { neighbor_discover_active = false; neighbor_discover_count = 0; + neighbor_discover_next = 0; + neighbor_discover_until = 0; + neighbor_discover_request = NULL; if (_cli.getObserverPrefs()->mqtt_neighbors_enabled) { next_neighbors_publish = futureMillis(_cli.getObserverPrefs()->mqtt_neighbors_interval); } @@ -1816,27 +1873,67 @@ void MyMesh::finishNeighborDiscover() { neighbor_discover_active = false; neighbor_discover_count = 0; + neighbor_discover_next = 0; + neighbor_discover_until = 0; + neighbor_discover_request = NULL; if (_cli.getObserverPrefs()->mqtt_neighbors_enabled) { next_neighbors_publish = futureMillis(_cli.getObserverPrefs()->mqtt_neighbors_interval); } } -// Advance the scope-query phase; publish once all entries resolve or the window -// times out (stragglers marked ND_TIMEOUT). +// Advance the newest-first scope-query phase. Keep only one request in flight so +// its responder gets a clear reply opportunity and the packet pool stays free. void MyMesh::loopNeighborDiscover() { if (!neighbor_discover_active) return; - bool all_done = true; - for (int i = 0; i < neighbor_discover_count; i++) { - if (neighbor_discover[i].status == ND_PENDING) { all_done = false; break; } + if (neighbor_discover_next >= neighbor_discover_count) { + finishNeighborDiscover(); + return; } - if (!all_done && !millisHasNowPassed(neighbor_discover_until)) return; - if (!all_done) { - for (int i = 0; i < neighbor_discover_count; i++) { - if (neighbor_discover[i].status == ND_PENDING) neighbor_discover[i].status = ND_TIMEOUT; + + NeighborDiscoverEntry& entry = neighbor_discover[neighbor_discover_next]; + if (entry.status == ND_QUEUED) { + if (!millisHasNowPassed(neighbor_discover_until)) return; + if (cancelNeighborDiscoverRequest()) { + entry.status = ND_SEND_FAILED; + neighbor_discover_next++; + return; } + if (isCurrentOutbound(neighbor_discover_request)) { + neighbor_discover_until = futureMillis(neighborDiscoverQueryTimeoutMs()); + return; + } + neighbor_discover_request = NULL; // packet manager already shed it + entry.status = ND_SEND_FAILED; + neighbor_discover_next++; + return; + } + if (entry.status == ND_PENDING) { + if (!millisHasNowPassed(neighbor_discover_until)) return; + entry.status = ND_TIMEOUT; + neighbor_discover_next++; + return; + } + if (entry.status == ND_RESPONDED || entry.status == ND_SEND_FAILED) { + neighbor_discover_next++; + return; + } + if (entry.status != ND_UNSENT) { + neighbor_discover_next++; + return; + } + + uint32_t tag; + mesh::Packet* request = sendAnonRegionsReq(entry.id, tag); + if (request) { + entry.tag = tag; + entry.status = ND_QUEUED; + neighbor_discover_request = request; + neighbor_discover_until = futureMillis(NEIGHBOR_DISCOVER_QUEUE_TIMEOUT_MS); + } else { + entry.status = ND_SEND_FAILED; + neighbor_discover_next++; } - finishNeighborDiscover(); } // Shared precondition for starting a discovery: PSRAM present + bridge running. @@ -1848,8 +1945,8 @@ bool MyMesh::neighborDiscoverReady(char* reply) { return true; } -// Snapshot the neighbor table into the overlay and fire one anon-regions query -// per heard neighbour; arm the 30s scope-query window. +// Snapshot the neighbor table newest-first. loopNeighborDiscover() emits one +// anon-regions query at a time so hidden responders do not reply as a burst. bool MyMesh::startNeighborDiscover(char* reply) { if (neighbor_discover_active) { strcpy(reply, "Err - neighbor discover already active"); @@ -1863,26 +1960,44 @@ bool MyMesh::startNeighborDiscover(char* reply) { neighbor_discover_count = 0; for (int i = 0; i < MAX_NEIGHBOURS; i++) { if (neighbours[i].heard_timestamp > 0) { - neighbor_discover[neighbor_discover_count].neighbour_idx = (uint8_t)i; - neighbor_discover[neighbor_discover_count].scopes[0] = 0; - neighbor_discover[neighbor_discover_count].status = ND_PENDING; - uint32_t tag; - if (sendAnonRegionsReq(neighbours[i].id, tag)) { - neighbor_discover[neighbor_discover_count].tag = tag; - } else { - neighbor_discover[neighbor_discover_count].status = ND_SEND_FAILED; - } + NeighborDiscoverEntry& entry = neighbor_discover[neighbor_discover_count]; + entry.id = neighbours[i].id; + entry.heard_timestamp = neighbours[i].heard_timestamp; + entry.snr = neighbours[i].snr; + entry.scopes[0] = 0; + entry.tag = 0; + entry.status = ND_UNSENT; neighbor_discover_count++; } } + // Query the freshest/strongest entries first; pubkey makes ties deterministic. + for (int i = 1; i < neighbor_discover_count; i++) { + NeighborDiscoverEntry entry = neighbor_discover[i]; + int j = i; + while (j > 0) { + auto& rhs = neighbor_discover[j - 1]; + bool before = entry.heard_timestamp > rhs.heard_timestamp + || (entry.heard_timestamp == rhs.heard_timestamp && entry.snr > rhs.snr) + || (entry.heard_timestamp == rhs.heard_timestamp && entry.snr == rhs.snr + && memcmp(entry.id.pub_key, rhs.id.pub_key, PUB_KEY_SIZE) < 0); + if (!before) break; + neighbor_discover[j] = neighbor_discover[j - 1]; + j--; + } + neighbor_discover[j] = entry; + } + + neighbor_discover_next = 0; neighbor_discover_active = true; - neighbor_discover_until = futureMillis(NEIGHBOR_DISCOVER_TIMEOUT_MS); + neighbor_discover_until = 0; + neighbor_discover_request = NULL; if (neighbor_discover_count == 0) { finishNeighborDiscover(); strcpy(reply, "OK - neighbor discover started (0 neighbors, self only)"); } else { + loopNeighborDiscover(); // queue the first request now sprintf(reply, "OK - neighbor discover started (%u neighbors)", (unsigned)neighbor_discover_count); } return true; diff --git a/examples/simple_room_server/MyMesh.h b/examples/simple_room_server/MyMesh.h index c1f4ffe1..2b064d5f 100644 --- a/examples/simple_room_server/MyMesh.h +++ b/examples/simple_room_server/MyMesh.h @@ -145,29 +145,37 @@ class MyMesh : public mesh::Mesh, public CommonCLICallbacks uint32_t pending_discover_tag; unsigned long pending_discover_until; enum NeighborDiscoverStatus : uint8_t { - ND_PENDING = 1, - ND_RESPONDED = 2, - ND_TIMEOUT = 3, - ND_SEND_FAILED = 4, + ND_UNSENT = 0, + ND_QUEUED = 1, + ND_PENDING = 2, + ND_RESPONDED = 3, + ND_TIMEOUT = 4, + ND_SEND_FAILED = 5, }; struct NeighborDiscoverEntry { - uint8_t neighbour_idx; + mesh::Identity id; + uint32_t heard_timestamp; + int8_t snr; uint32_t tag; char scopes[96]; uint8_t status; }; NeighborDiscoverEntry neighbor_discover[MAX_NEIGHBOURS]; uint8_t neighbor_discover_count; + uint8_t neighbor_discover_next; bool neighbor_discover_active; bool neighbor_table_refresh_active; bool neighbor_table_refresh_periodic; unsigned long neighbor_discover_until; + mesh::Packet* neighbor_discover_request; unsigned long next_neighbors_publish; char self_scopes_buf[96]; void putNeighbour(const mesh::Identity& id, uint32_t timestamp, float snr); void sendNodeDiscoverReq(); - bool sendAnonRegionsReq(const mesh::Identity& target, uint32_t& tag); + mesh::Packet* sendAnonRegionsReq(const mesh::Identity& target, uint32_t& tag); + bool cancelNeighborDiscoverRequest(); + uint32_t neighborDiscoverQueryTimeoutMs() const; bool neighborDiscoverReady(char* reply); bool startNeighborDiscover(char* reply); void loopNeighborDiscover(); @@ -175,7 +183,8 @@ class MyMesh : public mesh::Mesh, public CommonCLICallbacks bool handleNeighborDiscoverResponse(int overlay_idx, const uint8_t* data, size_t len); void getLocalScopes(char* buf, size_t len); static const int NEIGHBOR_DISCOVER_PEER_BASE = 1000; - static const unsigned long NEIGHBOR_DISCOVER_TIMEOUT_MS = 30000; + static const unsigned long NEIGHBOR_DISCOVER_QUEUE_TIMEOUT_MS = 29000; + static const int NEIGHBOR_DISCOVER_MIN_FREE_PACKETS = 5; #endif #ifdef WITH_MQTT_BRIDGE MQTTBridge* bridge; diff --git a/src/Dispatcher.h b/src/Dispatcher.h index 67f706c3..7edb7c7f 100644 --- a/src/Dispatcher.h +++ b/src/Dispatcher.h @@ -213,6 +213,7 @@ public: uint32_t getNumRecvDirect() const { return n_recv_direct; } uint16_t getErrFlags() const { return _err_flags; } // Get error flags bool hasOutbound() const { return outbound != NULL; } + bool isCurrentOutbound(const Packet* packet) const { return outbound == packet; } void resetStats() { n_sent_flood = n_sent_direct = n_recv_flood = n_recv_direct = 0; _err_flags = 0; From 3c5b25c2cfe3d8cbc0885420a34af1244a61fa73 Mon Sep 17 00:00:00 2001 From: agessaman Date: Sun, 26 Jul 2026 19:10:51 -0700 Subject: [PATCH 26/87] feat(mqtt): enhance neighbor discovery JSON handling Introduce new methods to manage neighbor discovery JSON budget and entries in MyMesh. This includes tracking the number of queried and published neighbors, measuring JSON sizes, and handling truncation when the buffer limit is reached. These improvements optimize the neighbor discovery process and ensure efficient JSON message construction for MQTT communications. --- examples/simple_repeater/MyMesh.cpp | 97 +++++++++++--- examples/simple_repeater/MyMesh.h | 7 + examples/simple_room_server/MyMesh.cpp | 97 +++++++++++--- examples/simple_room_server/MyMesh.h | 7 + src/helpers/MQTTMessageBuilder.cpp | 24 +++- src/helpers/MQTTMessageBuilder.h | 13 +- src/helpers/MQTTPayloadBuilder.cpp | 95 +++++++++++--- src/helpers/MQTTPayloadBuilder.h | 18 ++- src/helpers/bridges/MQTTBridge.cpp | 4 +- .../test_mqtt_payload_builder.cpp | 123 +++++++++++++++++- 10 files changed, 423 insertions(+), 62 deletions(-) diff --git a/examples/simple_repeater/MyMesh.cpp b/examples/simple_repeater/MyMesh.cpp index a8f3cb92..6fabf045 100644 --- a/examples/simple_repeater/MyMesh.cpp +++ b/examples/simple_repeater/MyMesh.cpp @@ -564,6 +564,7 @@ void MyMesh::logTx(mesh::Packet *pkt, int len) { NeighborDiscoverEntry& entry = neighbor_discover[neighbor_discover_next]; if (entry.status == ND_QUEUED) { entry.status = ND_PENDING; + neighbor_discover_queried_count++; neighbor_discover_request = NULL; neighbor_discover_until = futureMillis(neighborDiscoverQueryTimeoutMs()); } @@ -1066,6 +1067,10 @@ MyMesh::MyMesh(mesh::MainBoard &board, mesh::Radio &radio, mesh::MillisecondCloc #if defined(WITH_MQTT_NEIGHBORS) neighbor_discover_count = 0; neighbor_discover_next = 0; + neighbor_discover_publish_count = 0; + neighbor_discover_queried_count = 0; + neighbor_discover_json_size = 0; + neighbor_discover_truncated = false; neighbor_discover_active = false; neighbor_table_refresh_active = false; neighbor_table_refresh_periodic = false; @@ -1073,6 +1078,7 @@ MyMesh::MyMesh(mesh::MainBoard &board, mesh::Radio &radio, mesh::MillisecondCloc neighbor_discover_request = NULL; next_neighbors_publish = 0; self_scopes_buf[0] = 0; + neighbor_discover_origin[0] = 0; #endif memset(default_scope.key, 0, sizeof(default_scope.key)); @@ -1876,6 +1882,57 @@ uint32_t MyMesh::neighborDiscoverQueryTimeoutMs() const { + response_airtime * 2UL; } +void MyMesh::resetNeighborDiscoverJsonBudget() { + getLocalScopes(self_scopes_buf, sizeof(self_scopes_buf)); + MQTTBridge::getEffectiveMqttOrigin( + &_prefs, _cli.getObserverPrefs(), + neighbor_discover_origin, sizeof(neighbor_discover_origin)); + + char self_pubkey_hex[65]; + mesh::Utils::toHex(self_pubkey_hex, self_id.pub_key, PUB_KEY_SIZE); + char timestamp[40]; + MQTTMessageBuilder::formatIsoTimestampForMqtt( + getRTCClock()->getCurrentTime(), 0, nullptr, timestamp, sizeof(timestamp)); + + neighbor_discover_publish_count = 0; + neighbor_discover_queried_count = 0; + neighbor_discover_truncated = false; + neighbor_discover_json_size = MQTTMessageBuilder::measureNeighborsMessageBase( + neighbor_discover_origin, self_pubkey_hex, timestamp, self_scopes_buf, + neighbor_discover_count); +} + +// Account for one terminal result. The base measurement reserves maximum-width +// progress metadata; UINT32_MAX likewise reserves the widest heard-age value. +// If this result cannot fit, stop before transmitting another scope request. +bool MyMesh::completeNeighborDiscoverEntry() { + NeighborDiscoverEntry& entry = neighbor_discover[neighbor_discover_next]; + char pubkey_hex[65]; + mesh::Utils::toHex(pubkey_hex, entry.id.pub_key, PUB_KEY_SIZE); + + MQTTMessageBuilder::NeighborsMessageEntry measured = { + pubkey_hex, + entry.snr / 4.0f, + UINT32_MAX, + entry.scopes, + entry.status == ND_RESPONDED ? "responded" + : (entry.status == ND_SEND_FAILED ? "send_failed" : "timeout") + }; + size_t added = MQTTMessageBuilder::measureNeighborsMessageEntry(measured); + if (neighbor_discover_publish_count > 0) added++; // array comma + + if (neighbor_discover_json_size + added >= MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE) { + neighbor_discover_truncated = true; + finishNeighborDiscover(); + return false; + } + + neighbor_discover_json_size += added; + neighbor_discover_publish_count++; + neighbor_discover_next++; + return true; +} + // Match a RESPONSE against the pending overlay entry by tag; copy its scope // string (payload after the 8-byte {tag}{clock} header) into the entry. bool MyMesh::handleNeighborDiscoverResponse(int overlay_idx, const uint8_t* data, size_t len) { @@ -1945,14 +2002,9 @@ struct NeighborsDocAllocator : ArduinoJson::Allocator { // Build the neighbors-table JSON and hand it to the bridge, then reschedule. void MyMesh::finishNeighborDiscover() { - getLocalScopes(self_scopes_buf, sizeof(self_scopes_buf)); - char self_pubkey_hex[65]; mesh::Utils::toHex(self_pubkey_hex, self_id.pub_key, PUB_KEY_SIZE); - char origin[32]; - MQTTBridge::getEffectiveMqttOrigin(&_prefs, _cli.getObserverPrefs(), origin, sizeof(origin)); - char timestamp[40]; MQTTMessageBuilder::formatIsoTimestampForMqtt(getRTCClock()->getCurrentTime(), 0, nullptr, timestamp, sizeof(timestamp)); @@ -1960,7 +2012,7 @@ void MyMesh::finishNeighborDiscover() { MQTTMessageBuilder::NeighborsMessageEntry entries[MAX_NEIGHBOURS]; uint32_t now_secs = getRTCClock()->getCurrentTime(); - for (int i = 0; i < neighbor_discover_count; i++) { + for (int i = 0; i < neighbor_discover_publish_count; i++) { auto& entry = neighbor_discover[i]; mesh::Utils::toHex(pubkey_hex[i], entry.id.pub_key, PUB_KEY_SIZE); entries[i].pubkey_hex = pubkey_hex[i]; @@ -1976,7 +2028,7 @@ void MyMesh::finishNeighborDiscover() { } // insertion sort: most useful first (JSON builder drops the tail on overflow) - for (int i = 1; i < neighbor_discover_count; i++) { + for (int i = 1; i < neighbor_discover_publish_count; i++) { MQTTMessageBuilder::NeighborsMessageEntry entry = entries[i]; int j = i; while (j > 0 && neighborPublishEntryComesBefore(entry, entries[j - 1])) { @@ -1995,6 +2047,10 @@ void MyMesh::finishNeighborDiscover() { neighbor_discover_active = false; neighbor_discover_count = 0; neighbor_discover_next = 0; + neighbor_discover_publish_count = 0; + neighbor_discover_queried_count = 0; + neighbor_discover_json_size = 0; + neighbor_discover_truncated = false; neighbor_discover_until = 0; neighbor_discover_request = NULL; if (_cli.getObserverPrefs()->mqtt_neighbors_enabled) { @@ -2010,9 +2066,11 @@ void MyMesh::finishNeighborDiscover() { JsonDocument doc; #endif int json_len = MQTTMessageBuilder::buildNeighborsMessage( - doc, origin, self_pubkey_hex, timestamp, self_scopes_buf, - entries, neighbor_discover_count, - json_buf, MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE); + doc, neighbor_discover_origin, self_pubkey_hex, timestamp, self_scopes_buf, + entries, neighbor_discover_publish_count, + json_buf, MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE, + neighbor_discover_count, neighbor_discover_queried_count, + neighbor_discover_truncated); if (json_len > 0 && bridge) { bridge->requestPublishNeighbors(json_buf, (size_t)json_len); @@ -2027,6 +2085,10 @@ void MyMesh::finishNeighborDiscover() { neighbor_discover_active = false; neighbor_discover_count = 0; neighbor_discover_next = 0; + neighbor_discover_publish_count = 0; + neighbor_discover_queried_count = 0; + neighbor_discover_json_size = 0; + neighbor_discover_truncated = false; neighbor_discover_until = 0; neighbor_discover_request = NULL; if (_cli.getObserverPrefs()->mqtt_neighbors_enabled) { @@ -2049,7 +2111,7 @@ void MyMesh::loopNeighborDiscover() { if (!millisHasNowPassed(neighbor_discover_until)) return; if (cancelNeighborDiscoverRequest()) { entry.status = ND_SEND_FAILED; - neighbor_discover_next++; + completeNeighborDiscoverEntry(); return; } if (isCurrentOutbound(neighbor_discover_request)) { @@ -2058,17 +2120,18 @@ void MyMesh::loopNeighborDiscover() { } neighbor_discover_request = NULL; // packet manager already shed it entry.status = ND_SEND_FAILED; - neighbor_discover_next++; + completeNeighborDiscoverEntry(); return; } if (entry.status == ND_PENDING) { if (!millisHasNowPassed(neighbor_discover_until)) return; entry.status = ND_TIMEOUT; - neighbor_discover_next++; + completeNeighborDiscoverEntry(); return; } - if (entry.status == ND_RESPONDED || entry.status == ND_SEND_FAILED) { - neighbor_discover_next++; + if (entry.status == ND_RESPONDED || entry.status == ND_SEND_FAILED + || entry.status == ND_TIMEOUT) { + completeNeighborDiscoverEntry(); return; } if (entry.status != ND_UNSENT) { @@ -2085,7 +2148,7 @@ void MyMesh::loopNeighborDiscover() { neighbor_discover_until = futureMillis(NEIGHBOR_DISCOVER_QUEUE_TIMEOUT_MS); } else { entry.status = ND_SEND_FAILED; - neighbor_discover_next++; + completeNeighborDiscoverEntry(); } } @@ -2109,7 +2172,6 @@ bool MyMesh::startNeighborDiscover(char* reply) { return false; // reply already set } - getLocalScopes(self_scopes_buf, sizeof(self_scopes_buf)); neighbor_discover_count = 0; for (int i = 0; i < MAX_NEIGHBOURS; i++) { if (neighbours[i].heard_timestamp > 0) { @@ -2142,6 +2204,7 @@ bool MyMesh::startNeighborDiscover(char* reply) { } neighbor_discover_next = 0; + resetNeighborDiscoverJsonBudget(); neighbor_discover_active = true; neighbor_discover_until = 0; neighbor_discover_request = NULL; diff --git a/examples/simple_repeater/MyMesh.h b/examples/simple_repeater/MyMesh.h index bfca869e..76fae199 100644 --- a/examples/simple_repeater/MyMesh.h +++ b/examples/simple_repeater/MyMesh.h @@ -170,6 +170,10 @@ class MyMesh : public mesh::Mesh, public CommonCLICallbacks NeighborDiscoverEntry neighbor_discover[MAX_NEIGHBOURS]; uint8_t neighbor_discover_count; uint8_t neighbor_discover_next; // newest-first entry currently being queried + uint8_t neighbor_discover_publish_count; // completed prefix that fits the JSON buffer + uint8_t neighbor_discover_queried_count; // requests confirmed transmitted + size_t neighbor_discover_json_size; + bool neighbor_discover_truncated; bool neighbor_discover_active; // scope-query phase in flight bool neighbor_table_refresh_active; // zero-hop table refresh (stage 1) in flight bool neighbor_table_refresh_periodic; // that refresh was kicked by the periodic timer @@ -177,10 +181,13 @@ class MyMesh : public mesh::Mesh, public CommonCLICallbacks mesh::Packet* neighbor_discover_request; // request awaiting TX completion unsigned long next_neighbors_publish; // periodic publish deadline (0 = fire ASAP) char self_scopes_buf[96]; + char neighbor_discover_origin[32]; mesh::Packet* sendAnonRegionsReq(const mesh::Identity& target, uint32_t& tag); bool cancelNeighborDiscoverRequest(); uint32_t neighborDiscoverQueryTimeoutMs() const; + bool completeNeighborDiscoverEntry(); + void resetNeighborDiscoverJsonBudget(); bool neighborDiscoverReady(char* reply); bool startNeighborDiscover(char* reply); void loopNeighborDiscover(); diff --git a/examples/simple_room_server/MyMesh.cpp b/examples/simple_room_server/MyMesh.cpp index f161f68c..d7409f1b 100644 --- a/examples/simple_room_server/MyMesh.cpp +++ b/examples/simple_room_server/MyMesh.cpp @@ -256,6 +256,7 @@ void MyMesh::logTx(mesh::Packet *pkt, int len) { NeighborDiscoverEntry& entry = neighbor_discover[neighbor_discover_next]; if (entry.status == ND_QUEUED) { entry.status = ND_PENDING; + neighbor_discover_queried_count++; neighbor_discover_request = NULL; neighbor_discover_until = futureMillis(neighborDiscoverQueryTimeoutMs()); } @@ -885,6 +886,10 @@ MyMesh::MyMesh(mesh::MainBoard &board, mesh::Radio &radio, mesh::MillisecondCloc pending_discover_until = 0; neighbor_discover_count = 0; neighbor_discover_next = 0; + neighbor_discover_publish_count = 0; + neighbor_discover_queried_count = 0; + neighbor_discover_json_size = 0; + neighbor_discover_truncated = false; neighbor_discover_active = false; neighbor_table_refresh_active = false; neighbor_table_refresh_periodic = false; @@ -892,6 +897,7 @@ MyMesh::MyMesh(mesh::MainBoard &board, mesh::Radio &radio, mesh::MillisecondCloc neighbor_discover_request = NULL; next_neighbors_publish = 0; self_scopes_buf[0] = 0; + neighbor_discover_origin[0] = 0; memset(neighbours, 0, sizeof(neighbours)); #endif @@ -1723,6 +1729,57 @@ uint32_t MyMesh::neighborDiscoverQueryTimeoutMs() const { + response_airtime * 2UL; } +void MyMesh::resetNeighborDiscoverJsonBudget() { + getLocalScopes(self_scopes_buf, sizeof(self_scopes_buf)); + MQTTBridge::getEffectiveMqttOrigin( + &_prefs, _cli.getObserverPrefs(), + neighbor_discover_origin, sizeof(neighbor_discover_origin)); + + char self_pubkey_hex[65]; + mesh::Utils::toHex(self_pubkey_hex, self_id.pub_key, PUB_KEY_SIZE); + char timestamp[40]; + MQTTMessageBuilder::formatIsoTimestampForMqtt( + getRTCClock()->getCurrentTime(), 0, nullptr, timestamp, sizeof(timestamp)); + + neighbor_discover_publish_count = 0; + neighbor_discover_queried_count = 0; + neighbor_discover_truncated = false; + neighbor_discover_json_size = MQTTMessageBuilder::measureNeighborsMessageBase( + neighbor_discover_origin, self_pubkey_hex, timestamp, self_scopes_buf, + neighbor_discover_count); +} + +// Account for one terminal result. The base measurement reserves maximum-width +// progress metadata; UINT32_MAX likewise reserves the widest heard-age value. +// If this result cannot fit, stop before transmitting another scope request. +bool MyMesh::completeNeighborDiscoverEntry() { + NeighborDiscoverEntry& entry = neighbor_discover[neighbor_discover_next]; + char pubkey_hex[65]; + mesh::Utils::toHex(pubkey_hex, entry.id.pub_key, PUB_KEY_SIZE); + + MQTTMessageBuilder::NeighborsMessageEntry measured = { + pubkey_hex, + entry.snr / 4.0f, + UINT32_MAX, + entry.scopes, + entry.status == ND_RESPONDED ? "responded" + : (entry.status == ND_SEND_FAILED ? "send_failed" : "timeout") + }; + size_t added = MQTTMessageBuilder::measureNeighborsMessageEntry(measured); + if (neighbor_discover_publish_count > 0) added++; // array comma + + if (neighbor_discover_json_size + added >= MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE) { + neighbor_discover_truncated = true; + finishNeighborDiscover(); + return false; + } + + neighbor_discover_json_size += added; + neighbor_discover_publish_count++; + neighbor_discover_next++; + return true; +} + // Match a RESPONSE against the pending overlay entry by tag; copy its scope // string (payload after the 8-byte {tag}{clock} header) into the entry. bool MyMesh::handleNeighborDiscoverResponse(int overlay_idx, const uint8_t* data, size_t len) { @@ -1792,14 +1849,9 @@ struct NeighborsDocAllocator : ArduinoJson::Allocator { // Build the neighbors-table JSON and hand it to the bridge, then reschedule. void MyMesh::finishNeighborDiscover() { - getLocalScopes(self_scopes_buf, sizeof(self_scopes_buf)); - char self_pubkey_hex[65]; mesh::Utils::toHex(self_pubkey_hex, self_id.pub_key, PUB_KEY_SIZE); - char origin[32]; - MQTTBridge::getEffectiveMqttOrigin(&_prefs, _cli.getObserverPrefs(), origin, sizeof(origin)); - char timestamp[40]; MQTTMessageBuilder::formatIsoTimestampForMqtt(getRTCClock()->getCurrentTime(), 0, nullptr, timestamp, sizeof(timestamp)); @@ -1807,7 +1859,7 @@ void MyMesh::finishNeighborDiscover() { MQTTMessageBuilder::NeighborsMessageEntry entries[MAX_NEIGHBOURS]; uint32_t now_secs = getRTCClock()->getCurrentTime(); - for (int i = 0; i < neighbor_discover_count; i++) { + for (int i = 0; i < neighbor_discover_publish_count; i++) { auto& entry = neighbor_discover[i]; mesh::Utils::toHex(pubkey_hex[i], entry.id.pub_key, PUB_KEY_SIZE); entries[i].pubkey_hex = pubkey_hex[i]; @@ -1823,7 +1875,7 @@ void MyMesh::finishNeighborDiscover() { } // insertion sort: most useful first (JSON builder drops the tail on overflow) - for (int i = 1; i < neighbor_discover_count; i++) { + for (int i = 1; i < neighbor_discover_publish_count; i++) { MQTTMessageBuilder::NeighborsMessageEntry entry = entries[i]; int j = i; while (j > 0 && neighborPublishEntryComesBefore(entry, entries[j - 1])) { @@ -1842,6 +1894,10 @@ void MyMesh::finishNeighborDiscover() { neighbor_discover_active = false; neighbor_discover_count = 0; neighbor_discover_next = 0; + neighbor_discover_publish_count = 0; + neighbor_discover_queried_count = 0; + neighbor_discover_json_size = 0; + neighbor_discover_truncated = false; neighbor_discover_until = 0; neighbor_discover_request = NULL; if (_cli.getObserverPrefs()->mqtt_neighbors_enabled) { @@ -1857,9 +1913,11 @@ void MyMesh::finishNeighborDiscover() { JsonDocument doc; #endif int json_len = MQTTMessageBuilder::buildNeighborsMessage( - doc, origin, self_pubkey_hex, timestamp, self_scopes_buf, - entries, neighbor_discover_count, - json_buf, MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE); + doc, neighbor_discover_origin, self_pubkey_hex, timestamp, self_scopes_buf, + entries, neighbor_discover_publish_count, + json_buf, MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE, + neighbor_discover_count, neighbor_discover_queried_count, + neighbor_discover_truncated); if (json_len > 0 && bridge) { bridge->requestPublishNeighbors(json_buf, (size_t)json_len); @@ -1874,6 +1932,10 @@ void MyMesh::finishNeighborDiscover() { neighbor_discover_active = false; neighbor_discover_count = 0; neighbor_discover_next = 0; + neighbor_discover_publish_count = 0; + neighbor_discover_queried_count = 0; + neighbor_discover_json_size = 0; + neighbor_discover_truncated = false; neighbor_discover_until = 0; neighbor_discover_request = NULL; if (_cli.getObserverPrefs()->mqtt_neighbors_enabled) { @@ -1896,7 +1958,7 @@ void MyMesh::loopNeighborDiscover() { if (!millisHasNowPassed(neighbor_discover_until)) return; if (cancelNeighborDiscoverRequest()) { entry.status = ND_SEND_FAILED; - neighbor_discover_next++; + completeNeighborDiscoverEntry(); return; } if (isCurrentOutbound(neighbor_discover_request)) { @@ -1905,17 +1967,18 @@ void MyMesh::loopNeighborDiscover() { } neighbor_discover_request = NULL; // packet manager already shed it entry.status = ND_SEND_FAILED; - neighbor_discover_next++; + completeNeighborDiscoverEntry(); return; } if (entry.status == ND_PENDING) { if (!millisHasNowPassed(neighbor_discover_until)) return; entry.status = ND_TIMEOUT; - neighbor_discover_next++; + completeNeighborDiscoverEntry(); return; } - if (entry.status == ND_RESPONDED || entry.status == ND_SEND_FAILED) { - neighbor_discover_next++; + if (entry.status == ND_RESPONDED || entry.status == ND_SEND_FAILED + || entry.status == ND_TIMEOUT) { + completeNeighborDiscoverEntry(); return; } if (entry.status != ND_UNSENT) { @@ -1932,7 +1995,7 @@ void MyMesh::loopNeighborDiscover() { neighbor_discover_until = futureMillis(NEIGHBOR_DISCOVER_QUEUE_TIMEOUT_MS); } else { entry.status = ND_SEND_FAILED; - neighbor_discover_next++; + completeNeighborDiscoverEntry(); } } @@ -1956,7 +2019,6 @@ bool MyMesh::startNeighborDiscover(char* reply) { return false; // reply already set } - getLocalScopes(self_scopes_buf, sizeof(self_scopes_buf)); neighbor_discover_count = 0; for (int i = 0; i < MAX_NEIGHBOURS; i++) { if (neighbours[i].heard_timestamp > 0) { @@ -1989,6 +2051,7 @@ bool MyMesh::startNeighborDiscover(char* reply) { } neighbor_discover_next = 0; + resetNeighborDiscoverJsonBudget(); neighbor_discover_active = true; neighbor_discover_until = 0; neighbor_discover_request = NULL; diff --git a/examples/simple_room_server/MyMesh.h b/examples/simple_room_server/MyMesh.h index 2b064d5f..ff60c6ce 100644 --- a/examples/simple_room_server/MyMesh.h +++ b/examples/simple_room_server/MyMesh.h @@ -163,6 +163,10 @@ class MyMesh : public mesh::Mesh, public CommonCLICallbacks NeighborDiscoverEntry neighbor_discover[MAX_NEIGHBOURS]; uint8_t neighbor_discover_count; uint8_t neighbor_discover_next; + uint8_t neighbor_discover_publish_count; + uint8_t neighbor_discover_queried_count; // requests confirmed transmitted + size_t neighbor_discover_json_size; + bool neighbor_discover_truncated; bool neighbor_discover_active; bool neighbor_table_refresh_active; bool neighbor_table_refresh_periodic; @@ -170,12 +174,15 @@ class MyMesh : public mesh::Mesh, public CommonCLICallbacks mesh::Packet* neighbor_discover_request; unsigned long next_neighbors_publish; char self_scopes_buf[96]; + char neighbor_discover_origin[32]; void putNeighbour(const mesh::Identity& id, uint32_t timestamp, float snr); void sendNodeDiscoverReq(); mesh::Packet* sendAnonRegionsReq(const mesh::Identity& target, uint32_t& tag); bool cancelNeighborDiscoverRequest(); uint32_t neighborDiscoverQueryTimeoutMs() const; + bool completeNeighborDiscoverEntry(); + void resetNeighborDiscoverJsonBudget(); bool neighborDiscoverReady(char* reply); bool startNeighborDiscover(char* reply); void loopNeighborDiscover(); diff --git a/src/helpers/MQTTMessageBuilder.cpp b/src/helpers/MQTTMessageBuilder.cpp index b9029544..b5dd826d 100644 --- a/src/helpers/MQTTMessageBuilder.cpp +++ b/src/helpers/MQTTMessageBuilder.cpp @@ -111,11 +111,31 @@ int MQTTMessageBuilder::buildNeighborsMessage( const NeighborsMessageEntry* neighbors, int neighbor_count, char* buffer, - size_t buffer_size + size_t buffer_size, + int total_neighbors, + int queried_neighbors, + bool truncated ) { return MQTTPayloadBuilder::buildNeighborsMessage( doc, origin, origin_id, timestamp, self_scopes, neighbors, neighbor_count, - buffer, buffer_size); + buffer, buffer_size, total_neighbors, queried_neighbors, truncated); +} + +size_t MQTTMessageBuilder::measureNeighborsMessageBase( + const char* origin, + const char* origin_id, + const char* timestamp, + const char* self_scopes, + int total_neighbors +) { + return MQTTPayloadBuilder::measureNeighborsMessageBase( + origin, origin_id, timestamp, self_scopes, total_neighbors); +} + +size_t MQTTMessageBuilder::measureNeighborsMessageEntry( + const NeighborsMessageEntry& neighbor +) { + return MQTTPayloadBuilder::measureNeighborsMessageEntry(neighbor); } int MQTTMessageBuilder::buildPacketJSON( diff --git a/src/helpers/MQTTMessageBuilder.h b/src/helpers/MQTTMessageBuilder.h index fc0c4705..2d45af86 100644 --- a/src/helpers/MQTTMessageBuilder.h +++ b/src/helpers/MQTTMessageBuilder.h @@ -167,8 +167,19 @@ public: const NeighborsMessageEntry* neighbors, int neighbor_count, char* buffer, - size_t buffer_size + size_t buffer_size, + int total_neighbors = -1, + int queried_neighbors = -1, + bool truncated = false ); + static size_t measureNeighborsMessageBase( + const char* origin, + const char* origin_id, + const char* timestamp, + const char* self_scopes, + int total_neighbors + ); + static size_t measureNeighborsMessageEntry(const NeighborsMessageEntry& neighbor); /** * Convert packet to JSON message diff --git a/src/helpers/MQTTPayloadBuilder.cpp b/src/helpers/MQTTPayloadBuilder.cpp index 5a40e502..77d9054c 100644 --- a/src/helpers/MQTTPayloadBuilder.cpp +++ b/src/helpers/MQTTPayloadBuilder.cpp @@ -185,6 +185,68 @@ int MQTTPayloadBuilder::buildRawMessage( return serializeComplete(root, buffer, buffer_size); } +static JsonArray buildNeighborsMessageBase( + JsonDocument& doc, + const char* origin, + const char* origin_id, + const char* timestamp, + const char* self_scopes, + int total_neighbors, + int queried_neighbors, + bool truncated +) { + doc.clear(); + JsonObject root = doc.to(); + root["timestamp"] = timestamp; + root["origin"] = origin; + root["origin_id"] = origin_id; + if (total_neighbors >= 0) { + root["total_neighbors"] = total_neighbors; + root["queried_neighbors"] = queried_neighbors >= 0 ? queried_neighbors : total_neighbors; + root["truncated"] = truncated; + } + + JsonObject self = root["self"].to(); + self["scopes"] = self_scopes ? self_scopes : ""; + return root["neighbors"].to(); +} + +static void addNeighborsMessageEntry( + JsonArray& arr, + const MQTTPayloadBuilder::NeighborsMessageEntry& neighbor +) { + JsonObject nb = arr.add(); + nb["pubkey"] = neighbor.pubkey_hex; + nb["snr"] = neighbor.snr; + nb["heard_secs_ago"] = neighbor.heard_secs_ago; + nb["scopes"] = neighbor.scopes ? neighbor.scopes : ""; + nb["status"] = neighbor.status; +} + +size_t MQTTPayloadBuilder::measureNeighborsMessageBase( + const char* origin, + const char* origin_id, + const char* timestamp, + const char* self_scopes, + int total_neighbors +) { + JsonDocument doc; + buildNeighborsMessageBase( + doc, origin, origin_id, timestamp, self_scopes, + total_neighbors, total_neighbors, false); + return measureJson(doc); +} + +size_t MQTTPayloadBuilder::measureNeighborsMessageEntry( + const NeighborsMessageEntry& neighbor +) { + JsonDocument doc; + JsonArray arr = doc.to(); + addNeighborsMessageEntry(arr, neighbor); + JsonObject entry = arr[0]; + return measureJson(entry); +} + int MQTTPayloadBuilder::buildNeighborsMessage( JsonDocument& doc, const char* origin, @@ -194,38 +256,31 @@ int MQTTPayloadBuilder::buildNeighborsMessage( const NeighborsMessageEntry* neighbors, int neighbor_count, char* buffer, - size_t buffer_size + size_t buffer_size, + int total_neighbors, + int queried_neighbors, + bool truncated ) { if (!buffer || buffer_size == 0) return 0; - doc.clear(); - JsonObject root = doc.to(); - root["timestamp"] = timestamp; - root["origin"] = origin; - root["origin_id"] = origin_id; - - JsonObject self = root["self"].to(); - self["scopes"] = self_scopes ? self_scopes : ""; - - JsonArray arr = root["neighbors"].to(); - if (measureJson(root) >= buffer_size) return 0; + JsonArray arr = buildNeighborsMessageBase( + doc, origin, origin_id, timestamp, self_scopes, + total_neighbors, queried_neighbors, truncated); + if (doc.overflowed() || arr.isNull() || measureJson(doc) >= buffer_size) return 0; for (int i = 0; i < neighbor_count; i++) { - JsonObject nb = arr.add(); - nb["pubkey"] = neighbors[i].pubkey_hex; - nb["snr"] = neighbors[i].snr; - nb["heard_secs_ago"] = neighbors[i].heard_secs_ago; - nb["scopes"] = neighbors[i].scopes ? neighbors[i].scopes : ""; - nb["status"] = neighbors[i].status; + addNeighborsMessageEntry(arr, neighbors[i]); + if (doc.overflowed()) return 0; // Entries arrive ordered most- to least-useful. Stop as soon as the next // one would fill the fixed publish buffer, dropping the remaining tail so // document growth stays bounded. - if (measureJson(root) >= buffer_size) { + if (measureJson(doc) >= buffer_size) { arr.remove(arr.size() - 1); + if (total_neighbors >= 0) doc["truncated"] = true; break; } } - return serializeComplete(root, buffer, buffer_size); + return serializeComplete(doc.as(), buffer, buffer_size); } diff --git a/src/helpers/MQTTPayloadBuilder.h b/src/helpers/MQTTPayloadBuilder.h index a6fa333d..f9e5c959 100644 --- a/src/helpers/MQTTPayloadBuilder.h +++ b/src/helpers/MQTTPayloadBuilder.h @@ -89,7 +89,21 @@ public: const NeighborsMessageEntry* neighbors, int neighbor_count, char* buffer, - size_t buffer_size + size_t buffer_size, + int total_neighbors = -1, + int queried_neighbors = -1, + bool truncated = false ); -}; + // Exact serialized-size components used by paced neighbor discovery. The + // base reserves the largest progress metadata values for this snapshot; + // callers add each entry size plus one byte for commas after the first. + static size_t measureNeighborsMessageBase( + const char* origin, + const char* origin_id, + const char* timestamp, + const char* self_scopes, + int total_neighbors + ); + static size_t measureNeighborsMessageEntry(const NeighborsMessageEntry& neighbor); +}; diff --git a/src/helpers/bridges/MQTTBridge.cpp b/src/helpers/bridges/MQTTBridge.cpp index 4102cbe5..63a04306 100644 --- a/src/helpers/bridges/MQTTBridge.cpp +++ b/src/helpers/bridges/MQTTBridge.cpp @@ -3314,8 +3314,10 @@ bool MQTTBridge::publishNeighbors() { // MeshRank slots reject non-packets by contract, so buildTopicForSlot // returns false for them here and the slot is skipped. if (buildTopicForSlot(i, MSG_NEIGHBORS, topic, sizeof(topic))) { + // Neighbor snapshots are periodically refreshed. Publish synchronously + // at QoS 0 to avoid the QoS 1 outbox, retaining where the broker allows. bool use_retain = _slots[i].preset ? _slots[i].preset->allow_retain : false; - if (publishToSlot(i, topic, _neighbors_json_buffer, use_retain, 1)) { + if (publishToSlot(i, topic, _neighbors_json_buffer, use_retain, 0)) { published = true; } } diff --git a/test/test_mqtt_payload_builder/test_mqtt_payload_builder.cpp b/test/test_mqtt_payload_builder/test_mqtt_payload_builder.cpp index d7f81130..4a1e40de 100644 --- a/test/test_mqtt_payload_builder/test_mqtt_payload_builder.cpp +++ b/test/test_mqtt_payload_builder/test_mqtt_payload_builder.cpp @@ -12,6 +12,12 @@ namespace { constexpr const char* kTimestamp = "2026-07-18T12:34:56.123456+00:00"; +struct RejectAllJsonAllocations : ArduinoJson::Allocator { + void* allocate(size_t) override { return nullptr; } + void deallocate(void*) override {} + void* reallocate(void*, size_t) override { return nullptr; } +}; + static int buildMinimalStatus(JsonDocument& scratch, char* buffer, size_t buffer_size) { return MQTTPayloadBuilder::buildStatusMessage( scratch, "DEN Repeater", "0123456789ABCDEF", "Heltec V3", "v1.16.0", @@ -237,7 +243,7 @@ TEST(MQTTPayloadBuilder, NeighborsMessageRoundTripsSelfAndEntries) { char buffer[1024]; int len = MQTTPayloadBuilder::buildNeighborsMessage( scratch, "DEN Repeater", "0123456789ABCDEF", kTimestamp, "DEN,APRS", - neighbors, 2, buffer, sizeof(buffer)); + neighbors, 2, buffer, sizeof(buffer), 5, 2, true); ASSERT_GT(len, 0); EXPECT_EQ(static_cast(len), strlen(buffer)); @@ -247,6 +253,9 @@ TEST(MQTTPayloadBuilder, NeighborsMessageRoundTripsSelfAndEntries) { EXPECT_STREQ("DEN Repeater", parsed["origin"].as()); EXPECT_STREQ("0123456789ABCDEF", parsed["origin_id"].as()); EXPECT_STREQ("DEN,APRS", parsed["self"]["scopes"].as()); + EXPECT_EQ(5, parsed["total_neighbors"].as()); + EXPECT_EQ(2, parsed["queried_neighbors"].as()); + EXPECT_TRUE(parsed["truncated"].as()); JsonArray arr = parsed["neighbors"].as(); ASSERT_EQ(2U, arr.size()); @@ -260,6 +269,116 @@ TEST(MQTTPayloadBuilder, NeighborsMessageRoundTripsSelfAndEntries) { EXPECT_STREQ("stale", arr[1]["status"].as()); } +TEST(MQTTPayloadBuilder, NeighborsMessageMeasurementsMatchCompletePayload) { + MQTTPayloadBuilder::NeighborsMessageEntry neighbors[] = { + {"0011223344556677", 9.75f, UINT32_MAX, "DEN,APRS", "responded"}, + {"8899AABBCCDDEEFF", -3.5f, UINT32_MAX, "", "timeout"}, + }; + + size_t measured = + MQTTPayloadBuilder::measureNeighborsMessageBase( + "DEN Repeater", "0123456789ABCDEF", kTimestamp, "DEN,APRS", 2) + + MQTTPayloadBuilder::measureNeighborsMessageEntry(neighbors[0]) + + 1 // array comma + + MQTTPayloadBuilder::measureNeighborsMessageEntry(neighbors[1]); + + JsonDocument scratch; + char buffer[1024]; + int len = MQTTPayloadBuilder::buildNeighborsMessage( + scratch, "DEN Repeater", "0123456789ABCDEF", kTimestamp, "DEN,APRS", + neighbors, 2, buffer, sizeof(buffer), 2, 2, false); + + ASSERT_GT(len, 0); + EXPECT_EQ(measured, static_cast(len)); +} + +TEST(MQTTPayloadBuilder, NeighborsMessageMeasuredPrefixStopsBeforeOverflow) { + MQTTPayloadBuilder::NeighborsMessageEntry neighbors[20]; + static char keys[20][17]; + static char long_scopes[96]; + memset(long_scopes, 'S', sizeof(long_scopes) - 1); + long_scopes[sizeof(long_scopes) - 1] = '\0'; + for (int i = 0; i < 20; i++) { + snprintf(keys[i], sizeof(keys[i]), "%016X", i); + neighbors[i] = { + keys[i], static_cast(i), UINT32_MAX, + long_scopes, "responded"}; + } + + constexpr size_t kBufferSize = 512; + size_t used = MQTTPayloadBuilder::measureNeighborsMessageBase( + "node", "id", kTimestamp, "DEN", 20); + int published = 0; + int queried = 0; + while (queried < 20) { + size_t added = + MQTTPayloadBuilder::measureNeighborsMessageEntry(neighbors[queried]) + + (published > 0 ? 1U : 0U); + queried++; + if (used + added >= kBufferSize) break; + used += added; + published++; + } + + ASSERT_GT(published, 0); + ASSERT_LT(published, 20); + ASSERT_EQ(published + 1, queried); // only the non-fitting candidate was extra + + JsonDocument scratch; + char buffer[kBufferSize]; + int len = MQTTPayloadBuilder::buildNeighborsMessage( + scratch, "node", "id", kTimestamp, "DEN", + neighbors, published, buffer, sizeof(buffer), + 20, queried, true); + + ASSERT_GT(len, 0); + EXPECT_LT(static_cast(len), sizeof(buffer)); + JsonDocument parsed; + ASSERT_FALSE(deserializeJson(parsed, buffer)); + EXPECT_EQ(static_cast(published), + parsed["neighbors"].as().size()); + EXPECT_EQ(20, parsed["total_neighbors"].as()); + EXPECT_EQ(queried, parsed["queried_neighbors"].as()); + EXPECT_TRUE(parsed["truncated"].as()); +} + +TEST(MQTTPayloadBuilder, NeighborsMessageFallbackMarksTruncated) { + MQTTPayloadBuilder::NeighborsMessageEntry neighbors[] = { + {"0011223344556677", 9.75f, 1, "DEN", "responded"}, + {"8899AABBCCDDEEFF", -3.5f, 2, "APRS", "responded"}, + }; + size_t first_only_size = + MQTTPayloadBuilder::measureNeighborsMessageBase( + "node", "id", kTimestamp, "DEN", 2) + + MQTTPayloadBuilder::measureNeighborsMessageEntry(neighbors[0]); + std::vector buffer(first_only_size + 1); + + JsonDocument scratch; + int len = MQTTPayloadBuilder::buildNeighborsMessage( + scratch, "node", "id", kTimestamp, "DEN", + neighbors, 2, buffer.data(), buffer.size(), + 2, 2, false); + + ASSERT_GT(len, 0); + JsonDocument parsed; + ASSERT_FALSE(deserializeJson(parsed, buffer.data())); + EXPECT_EQ(1U, parsed["neighbors"].as().size()); + EXPECT_TRUE(parsed["truncated"].as()); +} + +TEST(MQTTPayloadBuilder, NeighborsMessageFailsCleanlyOnAllocationFailure) { + MQTTPayloadBuilder::NeighborsMessageEntry neighbor = { + "0011223344556677", 9.75f, 1, "DEN", "responded"}; + RejectAllJsonAllocations allocator; + JsonDocument scratch(&allocator); + char buffer[512]; + + EXPECT_EQ(0, MQTTPayloadBuilder::buildNeighborsMessage( + scratch, "node", "id", kTimestamp, "DEN", + &neighbor, 1, buffer, sizeof(buffer), + 1, 1, false)); +} + TEST(MQTTPayloadBuilder, NeighborsMessageHandlesEmptyTableAndNullScopes) { JsonDocument scratch; char buffer[256]; @@ -304,6 +423,7 @@ TEST(MQTTPayloadBuilder, NeighborsMessageDropsTailWhenBufferFills) { ASSERT_FALSE(arr.isNull()); EXPECT_GT(arr.size(), 0U); EXPECT_LT(arr.size(), 20U); + EXPECT_FALSE(parsed["truncated"].is()); // Kept entries are the head of the input, in order. EXPECT_STREQ(keys[0], arr[0]["pubkey"].as()); } @@ -314,4 +434,3 @@ int main(int argc, char** argv) { ::testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); } - From e00b29b4b84d817ed99a9eddfc7896dcccfa3da1 Mon Sep 17 00:00:00 2001 From: agessaman Date: Mon, 27 Jul 2026 18:34:04 -0700 Subject: [PATCH 27/87] feat(mqtt): add per-slot packet filters for MQTT slots Introduce per-slot packet filters to allow users to specify which packet types are uploaded for each MQTT slot. This feature enhances the flexibility of the MQTT bridge by enabling users to configure allowlists for packet types, improving the efficiency of data transmissions. The implementation includes updates to the WebConfig interface, internal handling of packet filters, and necessary modifications to the MQTT preferences structure. --- MQTT_IMPLEMENTATION.md | 34 +++++ MQTT_INTERNALS.md | 9 +- scripts/webconfig_mock_server.py | 31 +++- src/helpers/CommonCLI_Observer.cpp | 23 +++ src/helpers/MQTTDefaults.h | 4 + src/helpers/MQTTPacketFilter.h | 126 +++++++++++++++ src/helpers/MQTTPrefsCodec.h | 6 + src/helpers/MQTTPrefsStorage.h | 15 +- src/helpers/WebConfigKeys.h | 1 + src/helpers/bridges/MQTTBridge.cpp | 72 +++++++-- src/helpers/bridges/MQTTBridge.h | 6 +- src/helpers/esp32/WebConfigServer.cpp | 10 ++ test/README.md | 1 + .../test_mqtt_packet_filter.cpp | 143 ++++++++++++++++++ .../test_mqtt_prefs_codec.cpp | 44 ++++++ .../test_webconfig_keys.cpp | 2 + webui/index.html | 5 +- 17 files changed, 510 insertions(+), 22 deletions(-) create mode 100644 src/helpers/MQTTPacketFilter.h create mode 100644 test/test_mqtt_packet_filter/test_mqtt_packet_filter.cpp diff --git a/MQTT_IMPLEMENTATION.md b/MQTT_IMPLEMENTATION.md index cb94c628..33dd877c 100644 --- a/MQTT_IMPLEMENTATION.md +++ b/MQTT_IMPLEMENTATION.md @@ -248,6 +248,7 @@ The MQTT bridge comes with the following defaults for fresh installs (unless ove - **Slot 1**: `analyzer-us` - **Slot 2**: `analyzer-eu` - **Slots 3-6**: `none` (disabled) +- **Per-slot packet filters**: `all` (every payload type is uploaded) - **WiFi SSID**: (blank — must be configured) - **WiFi Password**: (blank — optional for open networks) - **WiFi Power Save**: `none` (no power save) @@ -272,6 +273,7 @@ Each slot (1-6) supports the following commands: - `get mqttN.token` - Get per-slot token (e.g., MeshRank account token) - `get mqttN.topic` - Get custom topic template for slot N - `get mqttN.audience` - Get JWT audience for slot N (custom slots only) +- `get mqttN.filter` - Get the slot's packet-type allowlist (`all`, `none`, or numeric CSV) #### Set Commands - `set mqttN.preset ` - Set slot N to a built-in preset. Use any `name` from the [preset table](#slot-based-preset-system) (run `get mqtt.presets` on-device for the full list). Most presets need no further configuration; the exceptions are: @@ -287,9 +289,41 @@ Each slot (1-6) supports the following commands: - `set mqttN.topic