feat(mqtt-neighbors): support non-PSRAM observer boards

Neighbors publication was gated on BOARD_HAS_PSRAM. Removing that gate alone
was not enough: the feature built but was inert without PSRAM, because three
allocation sites asked for MALLOC_CAP_SPIRAM (which returns null with no
PSRAM), the bridge's persistent buffer was allocated inside a BOARD_HAS_PSRAM
block, and neighborDiscoverReady() rejected every pass at runtime on
psramFound(). The entry table also did not fit: finishNeighborDiscover put
pubkey_hex[50][65] plus entries[50] on the stack, a 4752-byte frame against
the mesh loop task's 8 KB.

- Gate on MAX_NEIGHBOURS plus PSRAM or an explicit per-variant
  MQTT_NEIGHBORS_WITHOUT_PSRAM opt-in.
- Move the entry table and its hex strings into one heap block sized to the
  pass; the frame drops from 4752 to 304 bytes.
- Prefer PSRAM and fall back to internal DRAM in the mesh-side allocations
  and the ArduinoJson pool; hoist the bridge's persistent buffer out of the
  BOARD_HAS_PSRAM block (psram_malloc already falls back).
- Keep the runtime psramFound() check only where the buffers are sized for
  PSRAM, so a board whose PSRAM failed to init still refuses.
- Size for internal DRAM without PSRAM: 4 KB text buffer and 20 entries per
  publish, keeping the pool to a single block and the peak near 13 KB rather
  than ~35 KB. Oversized tables truncate and report total_neighbors as before.

Enabled on the ESP32-S3 observer envs (Heltec V3/WSL3, RAK3112, Heltec
Tracker v1.1/v2). Left off for the classic ESP32 T-LoRa V2.1-1.6, which is
already limited to one active TLS slot.

Costs ~7.4 KB static DRAM on repeaters and ~9.6 KB on room servers. The
prefs layout is unchanged, so this is neutral for existing devices.
This commit is contained in:
agessaman committed 2026-08-03 10:29:54 -07:00
1 parent 34037f202d
commit a3a0a94dc4
11 files changed
+307 -191

No files matched your search

+17 -7
View File
@@ -172,6 +172,16 @@ Fresh installs default to slot 1 `analyzer-us`, slot 2 `analyzer-eu`, and slots
- Slots configured beyond what the device supports show as `(inactive)` in `get mqtt.status`
- Slot configuration is preserved in preferences — moving the firmware to a PSRAM device activates the rest
**Neighbors publication without PSRAM.** PSRAM boards get `WITH_MQTT_NEIGHBORS`
automatically. Non-PSRAM boards opt in per variant with
`-D MQTT_NEIGHBORS_WITHOUT_PSRAM=1`, which is set on the ESP32-S3 observer envs
(Heltec V3/WSL3, RAK3112, Heltec Tracker v1.1/v2). It costs ~7.4 KB of static DRAM
(~9.6 KB on room servers) and up to ~13 KB transiently per publish, and caps the
table at 20 entries — so the feature is deliberately **not** enabled on the classic
ESP32 T-LoRa V2.1–1.6 observer builds, which are already down to one active TLS
slot. Publishing peaks while the table is built, so keep the slot guidance above
in mind: the peak lands on the same internal heap the TLS stack draws from.
## Build Configuration
To build the MQTT bridge firmware:
@@ -209,7 +219,7 @@ pio run -e ThinkNode_M7_room_server_observer_mqtt
`ThinkNode_M7_companion_radio_ethernet` uses it, but the MQTT bridge's link
management is bound to the WiFi station API, so the observer envs uplink over WiFi.
See `UPSTREAM_BUGS.md` for the Ethernet gap. The board has PSRAM, so these builds
get the PSRAM-only neighbors publication (`WITH_MQTT_NEIGHBORS`).
get neighbors publication (`WITH_MQTT_NEIGHBORS`) automatically.
**TLora naming:** The env prefix `LilyGo_TLora_V2_1_1_6` is LilyGo’s **T-LoRa V2.1–1.6** board (SX1276); PlatformIO selects **`ttgo-lora32-v1`** (TTGO LoRa32 V1.0). **MQTT observer** envs extend a slim base **without** `sensor_base` so the image fits `min_spiffs`; **all other** `LilyGo_TLora_V2_1_1_6_*` targets still use optional I2C environmental sensors as before. The **`lilygo_tlora_c6`** variant is separate hardware (ESP32-C6).
@@ -500,8 +510,8 @@ These settings apply across all MQTT slots:
- `get mqtt.rx` - Get RX packet uplinking setting (on/off)
- `get mqtt.tx` - Get TX packet uplinking setting (on/off/advert)
- `get mqtt.interval` - Get status publish interval
- `get mqtt.neighbors` - Get periodic neighbors publishing setting (on/off; PSRAM only)
- `get mqtt.neighbors.interval` - Get neighbors publish interval in hours (PSRAM only)
- `get mqtt.neighbors` - Get periodic neighbors publishing setting (on/off; neighbors-enabled builds)
- `get mqtt.neighbors.interval` - Get neighbors publish interval in hours (neighbors-enabled builds)
- `get mqtt.ntp` - Get effective NTP server hostname
- `get mqtt.ntp.diag` - Probe every configured NTP server for connectivity (does not change the clock; serial console shows each server's reported time, LoRa shows a compact `<server> ok|fail` list)
- `get mqtt.owner` - Get owner public key (serial console only)
@@ -519,8 +529,8 @@ These settings apply across all MQTT slots:
- `advert` - Uplink only this node's own advert packets (self-originated)
- `off` - Disable TX packet uplinking
- `set mqtt.interval <minutes>` - Set status publish interval (1-60 minutes)
- `set mqtt.neighbors on|off` - Enable/disable periodic neighbors publishing (PSRAM only; read live, no restart)
- `set mqtt.neighbors.interval <hours>` - Set neighbors publish interval (12-336 hours, default 24; PSRAM only)
- `set mqtt.neighbors on|off` - Enable/disable periodic neighbors publishing (neighbors-enabled builds; read live, no restart)
- `set mqtt.neighbors.interval <hours>` - Set neighbors publish interval (12-336 hours, default 24; neighbors-enabled builds)
- `set mqtt.ntp <hostname>` - Set custom NTP server (validated with immediate sync); `none` reverts to default
- `set mqtt.owner <64-hex-char-public-key>` - Set owner public key
- `set mqtt.email <email>` - Set owner email address
@@ -698,7 +708,7 @@ Full packet data with RF characteristics and metadata.
Minimal raw packet data for map integration.
### Neighbors Topic: `meshcore/{IATA}/{DEVICE_PUBLIC_KEY}/neighbors`
Cached zero-hop repeater neighbors with SNR, last-heard age, and flood-allowed scopes. Published on `discover.scopes` or periodically when `mqtt.neighbors` is enabled (PSRAM observer builds only). Goes to every configured slot's `neighbors` topic at QoS 0, retained only where the preset allows it.
Cached zero-hop repeater neighbors with SNR, last-heard age, and flood-allowed scopes. Published on `discover.scopes` or periodically when `mqtt.neighbors` is enabled (observer builds with neighbors compiled in; non-PSRAM builds cap the table at 20 entries and set `truncated`). Goes to every configured slot's `neighbors` topic at QoS 0, retained only where the preset allows it.
Periodic publishing first runs a 60-second zero-hop neighbor refresh equivalent to `discover.neighbors`, then queries the refreshed table for scopes and publishes when the scope-query phase completes.
@@ -784,7 +794,7 @@ While `mqtt.neighbors` is on, `get mqtt.status` appends `nbr: <next>/<last>` —
}
```
### Neighbors Message (PSRAM observer builds)
### Neighbors Message (neighbors-enabled observer builds)
```json
{
"timestamp": "2024-01-01T12:00:00.000000+00:00",
+15 -4
View File
@@ -74,11 +74,22 @@ defined in `MQTTBridge.h`). It spans two subsystems and two cores:
the snapshot and `neighbours[]` — proof of reception, and the only thing that heals a
stamp taken before the clock was set. After all responses land or a 30 s window
expires, `finishNeighborDiscover()` builds the JSON with
`MQTTMessageBuilder::buildNeighborsMessage` into a transient PSRAM buffer and hands it
to the bridge. Ages that still span a clock epoch publish as `null` rather than a
`MQTTMessageBuilder::buildNeighborsMessage` into a transient buffer (PSRAM where
available, internal DRAM otherwise) and hands it to the bridge. The entry table and its
hex strings share one heap block sized to the pass — at `MAX_NEIGHBOURS` they reach
~4.5 KB, which does not fit the mesh loop task's 8 KB stack. Ages that still span a clock epoch publish as `null` rather than a
fabricated delta (`neighborHeardAgeUsable`, see `UPSTREAM_BUGS.md` #1).
- **Bridge side, handoff**: `requestPublishNeighbors(json, len)` (Core 1) memcpys into a
persistent ~10 KB PSRAM buffer (`NEIGHBORS_JSON_BUFFER_SIZE`) and sets
- **Buffer sizing**: `NEIGHBORS_JSON_BUFFER_SIZE` is 10 KB with PSRAM and 4 KB without,
since a non-PSRAM board pays for the persistent buffer, the transient build buffer and
the ArduinoJson pool out of the same internal DRAM each TLS slot needs ~40 KB of
(~13 KB peak instead of ~35 KB). The pool has its own budget
(`NEIGHBORS_DOC_POOL_BUDGET`) because ArduinoJson v7 hands out pool blocks in fixed
4096-byte chunks, so a table that just fits the text buffer can still need well over
it in pool — and a starved pool sets `doc.overflowed()`, which drops the entire publish
instead of truncating it. `NEIGHBORS_MAX_PUBLISH_ENTRIES` (20 without PSRAM) keeps the
pool inside a single block.
- **Bridge side, handoff**: `requestPublishNeighbors(json, len)` (Core 1) memcpys into the
persistent buffer (`NEIGHBORS_JSON_BUFFER_SIZE`, PSRAM where available) and sets
`_neighbors_publish_pending` with a release store; the MQTT task (`mqttTaskLoop`, Core 0)
consumes it with an acquire load, calls `publishNeighbors()`, and clears the flag. A
second snapshot is dropped while one is in flight. `publishNeighbors()` sends QoS 1,
+10 -5
View File
@@ -136,9 +136,10 @@ scopes and publishes the assembled table to the MQTT `neighbors` topic once.
**Usage:**
- `discover.scopes`
**Note:** Requires a PSRAM board with the MQTT bridge running. On non-PSRAM MQTT
builds it replies `Err - not supported (requires PSRAM)`. If a `discover.neighbors`
refresh is already in flight, the scope pass is queued behind it.
**Note:** Requires an MQTT observer build with the neighbors feature compiled in
(all PSRAM boards, plus non-PSRAM boards built with `MQTT_NEIGHBORS_WITHOUT_PSRAM`).
Elsewhere it replies `Err - neighbors not enabled in this build`. If a
`discover.neighbors` refresh is already in flight, the scope pass is queued behind it.
---
@@ -1160,8 +1161,12 @@ region save
**Default:** `off`
> **Note:** Requires a PSRAM board. On non-PSRAM MQTT builds this replies
> `Err - not supported (requires PSRAM)`. The setting is read live by the mesh
> **Note:** Requires a build with the neighbors feature compiled in (all PSRAM
> boards, plus non-PSRAM boards built with `MQTT_NEIGHBORS_WITHOUT_PSRAM`);
> elsewhere this replies `Err - neighbors not enabled in this build`. Non-PSRAM
> builds publish at most 20 neighbours per pass to bound internal-DRAM use, and
> set `truncated` with the true `total_neighbors` when the table is larger.
> The setting is read live by the mesh
> loop — no restart required; enabling it triggers a discovery on the next pass.
> While enabled, `get mqtt.status` gains a trailing `nbr: <next>/<last>` field
> (time to next publish, and how the last publish went).
+98 -79
View File
@@ -1651,7 +1651,7 @@ void MyMesh::handleCommand(uint32_t sender_timestamp, char *command, char *reply
}
#elif defined(WITH_MQTT_BRIDGE)
} else if (memcmp(command, "discover.scopes", 15) == 0) {
strcpy(reply, "Err - not supported (requires PSRAM)");
strcpy(reply, "Err - neighbors not enabled in this build");
#endif
} else{
_cli.handleCommand(sender_timestamp, command, reply); // common CLI commands
@@ -2021,34 +2021,61 @@ static bool neighborPublishEntryComesBefore(
}
#if defined(ESP_PLATFORM)
// Neighbors allocations prefer PSRAM where it exists and otherwise come from
// internal DRAM, so MQTT_NEIGHBORS_WITHOUT_PSRAM boards can build the table too.
#if defined(BOARD_HAS_PSRAM)
static const uint32_t kNeighborsAllocCaps = MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT;
#else
static const uint32_t kNeighborsAllocCaps = MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT;
#endif
static void* neighborsAlloc(size_t size) {
if (size == 0) return nullptr;
void* p = heap_caps_malloc(size, kNeighborsAllocCaps);
#if defined(BOARD_HAS_PSRAM)
if (!p) p = heap_caps_malloc(size, MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT);
#endif
return p;
}
static void neighborsFree(void* ptr) {
if (ptr) heap_caps_free(ptr);
}
// ArduinoJson v7 JsonDocument has no real capacity cap (DynamicJsonDocument(N)
// is a no-op shim). Keep the pool off internal DRAM and soft-cap peak growth to
// the publish buffer size. used only rises on allocate — conservative for this
// single-shot doc (overflow path removes+breaks, so no further growth after free).
// is a no-op shim), so soft-cap peak pool growth to NEIGHBORS_DOC_POOL_BUDGET.
// used only rises on allocate — conservative for this single-shot doc (overflow
// path removes+breaks, so no further growth after free).
struct NeighborsDocAllocator : ArduinoJson::Allocator {
size_t used = 0;
static const size_t kBudget = MQTTBridge::NEIGHBORS_DOC_POOL_BUDGET;
void* allocate(size_t size) override {
if (used >= kBudget || size > kBudget - used) return nullptr;
void* p = heap_caps_malloc(size, MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT);
void* p = neighborsAlloc(size);
if (p) used += size;
return p;
}
void deallocate(void* ptr) override {
heap_caps_free(ptr);
neighborsFree(ptr);
}
void* reallocate(void* ptr, size_t new_size) override {
size_t old_size = ptr ? heap_caps_get_allocated_size(ptr) : 0;
size_t next_used = (used >= old_size) ? (used - old_size) : 0;
if (next_used >= kBudget || new_size > kBudget - next_used) return nullptr;
void* p = heap_caps_realloc(ptr, new_size, MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT);
void* p = heap_caps_realloc(ptr, new_size, kNeighborsAllocCaps);
#if defined(BOARD_HAS_PSRAM)
if (!p) p = heap_caps_realloc(ptr, new_size, MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT);
#endif
if (p) used = next_used + new_size;
return p;
}
};
#else
static void* neighborsAlloc(size_t size) { return size ? malloc(size) : nullptr; }
static void neighborsFree(void* ptr) { free(ptr); }
#endif
// Build the neighbors-table JSON and hand it to the bridge, then reschedule.
@@ -2059,80 +2086,69 @@ void MyMesh::finishNeighborDiscover() {
char timestamp[40];
MQTTMessageBuilder::formatIsoTimestampForMqtt(getRTCClock()->getCurrentTime(), 0, nullptr, timestamp, sizeof(timestamp));
char pubkey_hex[MAX_NEIGHBOURS][65];
MQTTMessageBuilder::NeighborsMessageEntry entries[MAX_NEIGHBOURS];
uint32_t now_secs = getRTCClock()->getCurrentTime();
// The entry table plus one hex string each reaches ~4.5 KB at MAX_NEIGHBOURS,
// which does not fit the mesh loop task's 8 KB stack, so both share a single
// heap block sized to this pass. Publishing is skipped if either alloc fails.
const int publish_count = neighbor_discover_publish_count;
const size_t hex_size = PUB_KEY_SIZE * 2 + 1;
const size_t entries_bytes =
sizeof(MQTTMessageBuilder::NeighborsMessageEntry) * publish_count;
void* scratch = neighborsAlloc(entries_bytes + hex_size * publish_count);
char* json_buf = (char*)neighborsAlloc(MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE);
for (int i = 0; i < neighbor_discover_publish_count; i++) {
auto& entry = neighbor_discover[i];
mesh::Utils::toHex(pubkey_hex[i], entry.id.pub_key, PUB_KEY_SIZE);
entries[i].pubkey_hex = pubkey_hex[i];
entries[i].snr = entry.snr / 4.0f;
bool heard_known = neighborHeardAgeUsable(entry.heard_timestamp, now_secs);
entries[i].heard_unknown = !heard_known;
entries[i].heard_secs_ago = heard_known ? (now_secs - entry.heard_timestamp) : 0;
entries[i].scopes = entry.scopes;
switch (entry.status) {
case ND_RESPONDED: entries[i].status = "responded"; break;
case ND_SEND_FAILED: entries[i].status = "send_failed"; break;
default: entries[i].status = "timeout"; break;
if (json_buf && (scratch || publish_count == 0)) {
auto* entries = (MQTTMessageBuilder::NeighborsMessageEntry*)scratch;
char* pubkey_hex = (char*)scratch + entries_bytes;
uint32_t now_secs = getRTCClock()->getCurrentTime();
for (int i = 0; i < publish_count; i++) {
auto& entry = neighbor_discover[i];
char* hex = &pubkey_hex[i * hex_size];
mesh::Utils::toHex(hex, entry.id.pub_key, PUB_KEY_SIZE);
entries[i].pubkey_hex = hex;
entries[i].snr = entry.snr / 4.0f;
bool heard_known = neighborHeardAgeUsable(entry.heard_timestamp, now_secs);
entries[i].heard_unknown = !heard_known;
entries[i].heard_secs_ago = heard_known ? (now_secs - entry.heard_timestamp) : 0;
entries[i].scopes = entry.scopes;
switch (entry.status) {
case ND_RESPONDED: entries[i].status = "responded"; break;
case ND_SEND_FAILED: entries[i].status = "send_failed"; break;
default: entries[i].status = "timeout"; break;
}
}
// insertion sort: most useful first (JSON builder drops the tail on overflow)
for (int i = 1; i < publish_count; i++) {
MQTTMessageBuilder::NeighborsMessageEntry entry = entries[i];
int j = i;
while (j > 0 && neighborPublishEntryComesBefore(entry, entries[j - 1])) {
entries[j] = entries[j - 1];
j--;
}
entries[j] = entry;
}
#if defined(ESP_PLATFORM)
NeighborsDocAllocator doc_alloc;
JsonDocument doc(&doc_alloc);
#else
JsonDocument doc;
#endif
int json_len = MQTTMessageBuilder::buildNeighborsMessage(
doc, neighbor_discover_origin, self_pubkey_hex, timestamp, self_scopes_buf,
self_default_scope_buf, entries, publish_count,
json_buf, MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE,
neighbor_discover_count, neighbor_discover_queried_count,
neighbor_discover_truncated);
if (json_len > 0 && bridge) {
bridge->requestPublishNeighbors(json_buf, (size_t)json_len);
}
}
// insertion sort: most useful first (JSON builder drops the tail on overflow)
for (int i = 1; i < neighbor_discover_publish_count; i++) {
MQTTMessageBuilder::NeighborsMessageEntry entry = entries[i];
int j = i;
while (j > 0 && neighborPublishEntryComesBefore(entry, entries[j - 1])) {
entries[j] = entries[j - 1];
j--;
}
entries[j] = entry;
}
#if defined(ESP_PLATFORM)
char* json_buf = (char*)heap_caps_malloc(MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE, MALLOC_CAP_SPIRAM);
#else
char* json_buf = (char*)malloc(MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE);
#endif
if (!json_buf) {
neighbor_discover_active = false;
neighbor_discover_count = 0;
neighbor_discover_next = 0;
neighbor_discover_publish_count = 0;
neighbor_discover_queried_count = 0;
neighbor_discover_json_size = 0;
neighbor_discover_truncated = false;
neighbor_discover_until = 0;
neighbor_discover_request = NULL;
if (_cli.getObserverPrefs()->mqtt_neighbors_enabled) {
next_neighbors_publish = futureMillis(_cli.getObserverPrefs()->mqtt_neighbors_interval);
}
return;
}
#if defined(ESP_PLATFORM)
NeighborsDocAllocator doc_alloc;
JsonDocument doc(&doc_alloc);
#else
JsonDocument doc;
#endif
int json_len = MQTTMessageBuilder::buildNeighborsMessage(
doc, neighbor_discover_origin, self_pubkey_hex, timestamp, self_scopes_buf,
self_default_scope_buf, entries, neighbor_discover_publish_count,
json_buf, MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE,
neighbor_discover_count, neighbor_discover_queried_count,
neighbor_discover_truncated);
if (json_len > 0 && bridge) {
bridge->requestPublishNeighbors(json_buf, (size_t)json_len);
}
#if defined(ESP_PLATFORM)
heap_caps_free(json_buf);
#else
free(json_buf);
#endif
neighborsFree(scratch);
neighborsFree(json_buf);
neighbor_discover_active = false;
neighbor_discover_count = 0;
@@ -2204,9 +2220,12 @@ void MyMesh::loopNeighborDiscover() {
}
}
// Shared precondition for starting a discovery: PSRAM present + bridge running.
// Shared precondition for starting a discovery: usable buffers + bridge running.
// PSRAM builds size their neighbors buffers for PSRAM, so a board whose PSRAM
// failed to init must not silently spend that much internal DRAM here.
// MQTT_NEIGHBORS_WITHOUT_PSRAM builds are already sized for internal DRAM.
bool MyMesh::neighborDiscoverReady(char* reply) {
#if defined(ESP_PLATFORM)
#if defined(ESP_PLATFORM) && defined(BOARD_HAS_PSRAM)
if (!psramFound()) { strcpy(reply, "Err - PSRAM not available"); return false; }
#endif
if (!bridge || !bridge->isRunning()) { strcpy(reply, "Err - MQTT bridge not running"); return false; }
+98 -79
View File
@@ -1477,7 +1477,7 @@ void MyMesh::handleCommand(uint32_t sender_timestamp, char *command, char *reply
}
#elif defined(WITH_MQTT_BRIDGE)
} else if (memcmp(command, "discover.scopes", 15) == 0) {
strcpy(reply, "Err - not supported (requires PSRAM)");
strcpy(reply, "Err - neighbors not enabled in this build");
#endif
} else if (strncmp(command, "room.post", 9) == 0) {
char* msg = command + 9;
@@ -1905,34 +1905,61 @@ static bool neighborPublishEntryComesBefore(
}
#if defined(ESP_PLATFORM)
// Neighbors allocations prefer PSRAM where it exists and otherwise come from
// internal DRAM, so MQTT_NEIGHBORS_WITHOUT_PSRAM boards can build the table too.
#if defined(BOARD_HAS_PSRAM)
static const uint32_t kNeighborsAllocCaps = MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT;
#else
static const uint32_t kNeighborsAllocCaps = MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT;
#endif
static void* neighborsAlloc(size_t size) {
if (size == 0) return nullptr;
void* p = heap_caps_malloc(size, kNeighborsAllocCaps);
#if defined(BOARD_HAS_PSRAM)
if (!p) p = heap_caps_malloc(size, MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT);
#endif
return p;
}
static void neighborsFree(void* ptr) {
if (ptr) heap_caps_free(ptr);
}
// ArduinoJson v7 JsonDocument has no real capacity cap (DynamicJsonDocument(N)
// is a no-op shim). Keep the pool off internal DRAM and soft-cap peak growth to
// the publish buffer size. used only rises on allocate — conservative for this
// single-shot doc (overflow path removes+breaks, so no further growth after free).
// is a no-op shim), so soft-cap peak pool growth to NEIGHBORS_DOC_POOL_BUDGET.
// used only rises on allocate — conservative for this single-shot doc (overflow
// path removes+breaks, so no further growth after free).
struct NeighborsDocAllocator : ArduinoJson::Allocator {
size_t used = 0;
static const size_t kBudget = MQTTBridge::NEIGHBORS_DOC_POOL_BUDGET;
void* allocate(size_t size) override {
if (used >= kBudget || size > kBudget - used) return nullptr;
void* p = heap_caps_malloc(size, MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT);
void* p = neighborsAlloc(size);
if (p) used += size;
return p;
}
void deallocate(void* ptr) override {
heap_caps_free(ptr);
neighborsFree(ptr);
}
void* reallocate(void* ptr, size_t new_size) override {
size_t old_size = ptr ? heap_caps_get_allocated_size(ptr) : 0;
size_t next_used = (used >= old_size) ? (used - old_size) : 0;
if (next_used >= kBudget || new_size > kBudget - next_used) return nullptr;
void* p = heap_caps_realloc(ptr, new_size, MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT);
void* p = heap_caps_realloc(ptr, new_size, kNeighborsAllocCaps);
#if defined(BOARD_HAS_PSRAM)
if (!p) p = heap_caps_realloc(ptr, new_size, MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT);
#endif
if (p) used = next_used + new_size;
return p;
}
};
#else
static void* neighborsAlloc(size_t size) { return size ? malloc(size) : nullptr; }
static void neighborsFree(void* ptr) { free(ptr); }
#endif
// Build the neighbors-table JSON and hand it to the bridge, then reschedule.
@@ -1943,80 +1970,69 @@ void MyMesh::finishNeighborDiscover() {
char timestamp[40];
MQTTMessageBuilder::formatIsoTimestampForMqtt(getRTCClock()->getCurrentTime(), 0, nullptr, timestamp, sizeof(timestamp));
char pubkey_hex[MAX_NEIGHBOURS][65];
MQTTMessageBuilder::NeighborsMessageEntry entries[MAX_NEIGHBOURS];
uint32_t now_secs = getRTCClock()->getCurrentTime();
// The entry table plus one hex string each reaches ~4.5 KB at MAX_NEIGHBOURS,
// which does not fit the mesh loop task's 8 KB stack, so both share a single
// heap block sized to this pass. Publishing is skipped if either alloc fails.
const int publish_count = neighbor_discover_publish_count;
const size_t hex_size = PUB_KEY_SIZE * 2 + 1;
const size_t entries_bytes =
sizeof(MQTTMessageBuilder::NeighborsMessageEntry) * publish_count;
void* scratch = neighborsAlloc(entries_bytes + hex_size * publish_count);
char* json_buf = (char*)neighborsAlloc(MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE);
for (int i = 0; i < neighbor_discover_publish_count; i++) {
auto& entry = neighbor_discover[i];
mesh::Utils::toHex(pubkey_hex[i], entry.id.pub_key, PUB_KEY_SIZE);
entries[i].pubkey_hex = pubkey_hex[i];
entries[i].snr = entry.snr / 4.0f;
bool heard_known = neighborHeardAgeUsable(entry.heard_timestamp, now_secs);
entries[i].heard_unknown = !heard_known;
entries[i].heard_secs_ago = heard_known ? (now_secs - entry.heard_timestamp) : 0;
entries[i].scopes = entry.scopes;
switch (entry.status) {
case ND_RESPONDED: entries[i].status = "responded"; break;
case ND_SEND_FAILED: entries[i].status = "send_failed"; break;
default: entries[i].status = "timeout"; break;
if (json_buf && (scratch || publish_count == 0)) {
auto* entries = (MQTTMessageBuilder::NeighborsMessageEntry*)scratch;
char* pubkey_hex = (char*)scratch + entries_bytes;
uint32_t now_secs = getRTCClock()->getCurrentTime();
for (int i = 0; i < publish_count; i++) {
auto& entry = neighbor_discover[i];
char* hex = &pubkey_hex[i * hex_size];
mesh::Utils::toHex(hex, entry.id.pub_key, PUB_KEY_SIZE);
entries[i].pubkey_hex = hex;
entries[i].snr = entry.snr / 4.0f;
bool heard_known = neighborHeardAgeUsable(entry.heard_timestamp, now_secs);
entries[i].heard_unknown = !heard_known;
entries[i].heard_secs_ago = heard_known ? (now_secs - entry.heard_timestamp) : 0;
entries[i].scopes = entry.scopes;
switch (entry.status) {
case ND_RESPONDED: entries[i].status = "responded"; break;
case ND_SEND_FAILED: entries[i].status = "send_failed"; break;
default: entries[i].status = "timeout"; break;
}
}
// insertion sort: most useful first (JSON builder drops the tail on overflow)
for (int i = 1; i < publish_count; i++) {
MQTTMessageBuilder::NeighborsMessageEntry entry = entries[i];
int j = i;
while (j > 0 && neighborPublishEntryComesBefore(entry, entries[j - 1])) {
entries[j] = entries[j - 1];
j--;
}
entries[j] = entry;
}
#if defined(ESP_PLATFORM)
NeighborsDocAllocator doc_alloc;
JsonDocument doc(&doc_alloc);
#else
JsonDocument doc;
#endif
int json_len = MQTTMessageBuilder::buildNeighborsMessage(
doc, neighbor_discover_origin, self_pubkey_hex, timestamp, self_scopes_buf,
self_default_scope_buf, entries, publish_count,
json_buf, MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE,
neighbor_discover_count, neighbor_discover_queried_count,
neighbor_discover_truncated);
if (json_len > 0 && bridge) {
bridge->requestPublishNeighbors(json_buf, (size_t)json_len);
}
}
// insertion sort: most useful first (JSON builder drops the tail on overflow)
for (int i = 1; i < neighbor_discover_publish_count; i++) {
MQTTMessageBuilder::NeighborsMessageEntry entry = entries[i];
int j = i;
while (j > 0 && neighborPublishEntryComesBefore(entry, entries[j - 1])) {
entries[j] = entries[j - 1];
j--;
}
entries[j] = entry;
}
#if defined(ESP_PLATFORM)
char* json_buf = (char*)heap_caps_malloc(MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE, MALLOC_CAP_SPIRAM);
#else
char* json_buf = (char*)malloc(MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE);
#endif
if (!json_buf) {
neighbor_discover_active = false;
neighbor_discover_count = 0;
neighbor_discover_next = 0;
neighbor_discover_publish_count = 0;
neighbor_discover_queried_count = 0;
neighbor_discover_json_size = 0;
neighbor_discover_truncated = false;
neighbor_discover_until = 0;
neighbor_discover_request = NULL;
if (_cli.getObserverPrefs()->mqtt_neighbors_enabled) {
next_neighbors_publish = futureMillis(_cli.getObserverPrefs()->mqtt_neighbors_interval);
}
return;
}
#if defined(ESP_PLATFORM)
NeighborsDocAllocator doc_alloc;
JsonDocument doc(&doc_alloc);
#else
JsonDocument doc;
#endif
int json_len = MQTTMessageBuilder::buildNeighborsMessage(
doc, neighbor_discover_origin, self_pubkey_hex, timestamp, self_scopes_buf,
self_default_scope_buf, entries, neighbor_discover_publish_count,
json_buf, MQTTBridge::NEIGHBORS_JSON_BUFFER_SIZE,
neighbor_discover_count, neighbor_discover_queried_count,
neighbor_discover_truncated);
if (json_len > 0 && bridge) {
bridge->requestPublishNeighbors(json_buf, (size_t)json_len);
}
#if defined(ESP_PLATFORM)
heap_caps_free(json_buf);
#else
free(json_buf);
#endif
neighborsFree(scratch);
neighborsFree(json_buf);
neighbor_discover_active = false;
neighbor_discover_count = 0;
@@ -2088,9 +2104,12 @@ void MyMesh::loopNeighborDiscover() {
}
}
// Shared precondition for starting a discovery: PSRAM present + bridge running.
// Shared precondition for starting a discovery: usable buffers + bridge running.
// PSRAM builds size their neighbors buffers for PSRAM, so a board whose PSRAM
// failed to init must not silently spend that much internal DRAM here.
// MQTT_NEIGHBORS_WITHOUT_PSRAM builds are already sized for internal DRAM.
bool MyMesh::neighborDiscoverReady(char* reply) {
#if defined(ESP_PLATFORM)
#if defined(ESP_PLATFORM) && defined(BOARD_HAS_PSRAM)
if (!psramFound()) { strcpy(reply, "Err - PSRAM not available"); return false; }
#endif
if (!bridge || !bridge->isRunning()) { strcpy(reply, "Err - MQTT bridge not running"); return false; }
+2 -2
View File
@@ -289,7 +289,7 @@ bool CommonCLI::handleObserverSetCmd(uint32_t sender_timestamp, const char* conf
#elif defined(WITH_MQTT_BRIDGE)
} else if (memcmp(config, "mqtt.neighbors.interval ", 24) == 0 ||
memcmp(config, "mqtt.neighbors ", 15) == 0) {
strcpy(reply, "Err - not supported (requires PSRAM)");
strcpy(reply, "Err - neighbors not enabled in this build");
#endif
} else if (memcmp(config, "mqtt.ntp ", 9) == 0) {
const char* host = &config[9];
@@ -837,7 +837,7 @@ bool CommonCLI::handleObserverGetCmd(uint32_t sender_timestamp, const char* conf
#elif defined(WITH_MQTT_BRIDGE)
} else if (memcmp(config, "mqtt.neighbors.interval", 23) == 0 ||
memcmp(config, "mqtt.neighbors", 14) == 0) {
strcpy(reply, "Err - not supported (requires PSRAM)");
strcpy(reply, "Err - neighbors not enabled in this build");
#endif
} else if (memcmp(config, "mqtt.ntp.diag", 13) == 0 && (config[13] == '\0' || config[13] == ' ')) {
#ifdef ESP_PLATFORM
+15 -8
View File
@@ -727,18 +727,23 @@ void MQTTBridge::allocateRuntimeBuffers() {
_publish_json_buffer, PUBLISH_JSON_BUFFER_SIZE, psram_malloc));
_status_json_buffer = static_cast<char*>(MQTTRuntimeBufferLifecycle::allocateIfMissing(
_status_json_buffer, STATUS_JSON_BUFFER_SIZE, psram_malloc));
#if defined(WITH_MQTT_NEIGHBORS)
// Persistent neighbors JSON buffer. Unlike status/packet there is no stack
// fallback: the feature is PSRAM-gated, so a nullptr simply disables publishing
// (requestPublishNeighbors/publishNeighbors both no-op on nullptr).
_neighbors_json_buffer = static_cast<char*>(MQTTRuntimeBufferLifecycle::allocateIfMissing(
_neighbors_json_buffer, NEIGHBORS_JSON_BUFFER_SIZE, psram_malloc));
#endif
MQTT_DEBUG_PRINTLN("Runtime buffers: raw=%s publish=%s status=%s",
_last_raw_data ? "PSRAM" : "unavailable",
_publish_json_buffer ? "PSRAM" : "stack fallback",
_status_json_buffer ? "PSRAM" : "stack fallback");
#endif
#if defined(WITH_MQTT_NEIGHBORS)
// Persistent neighbors JSON buffer, heap-allocated on every board: too large to
// keep inline in the bridge object the way the non-PSRAM status/packet buffers
// are. psram_malloc() falls back to internal DRAM, so this works without PSRAM.
// Unlike status/packet there is no stack fallback — a nullptr simply disables
// publishing (requestPublishNeighbors/publishNeighbors both no-op on nullptr).
_neighbors_json_buffer = static_cast<char*>(MQTTRuntimeBufferLifecycle::allocateIfMissing(
_neighbors_json_buffer, NEIGHBORS_JSON_BUFFER_SIZE, psram_malloc));
MQTT_DEBUG_PRINTLN("Neighbors buffer: %s",
_neighbors_json_buffer ? "ready" : "unavailable");
#endif
}
void MQTTBridge::releaseRuntimeBuffers() {
@@ -749,13 +754,15 @@ void MQTTBridge::releaseRuntimeBuffers() {
_publish_json_buffer, psram_free));
_status_json_buffer = static_cast<char*>(MQTTRuntimeBufferLifecycle::release(
_status_json_buffer, psram_free));
#endif
#if defined(WITH_MQTT_NEIGHBORS)
// Paired with the unconditional allocation in allocateRuntimeBuffers().
_neighbors_json_buffer = static_cast<char*>(MQTTRuntimeBufferLifecycle::release(
_neighbors_json_buffer, psram_free));
_neighbors_publish_len = 0;
_neighbors_publish_pending.store(false, std::memory_order_release);
#endif
#endif
// Never pair a newly allocated raw buffer with metadata from a prior bridge
// run. This also makes non-PSRAM restarts discard their stale raw cache.
+32 -7
View File
@@ -38,11 +38,15 @@ class MeshSNMPAgent; // Forward declaration
#ifdef WITH_MQTT_BRIDGE
// Periodic neighbors publication is PSRAM-only: it needs a persistent ~10 KB JSON
// buffer plus a second transient one while the mesh builds the table, and it keys
// off the mesh neighbor cache (sized by MAX_NEIGHBOURS). Every neighbors-specific
// member, method, and code block in this bridge is gated on WITH_MQTT_NEIGHBORS.
#if defined(BOARD_HAS_PSRAM) && defined(MAX_NEIGHBOURS) && MAX_NEIGHBOURS > 0
// Periodic neighbors publication keys off the mesh neighbor cache (sized by
// MAX_NEIGHBOURS) and needs a persistent JSON buffer plus a second transient one
// while the mesh builds the table. PSRAM boards get it automatically; non-PSRAM
// boards must opt in per variant with MQTT_NEIGHBORS_WITHOUT_PSRAM, which spends
// internal DRAM the TLS stack also needs (see NEIGHBORS_JSON_BUFFER_SIZE). Every
// neighbors-specific member, method, and code block here is gated on
// WITH_MQTT_NEIGHBORS.
#if defined(MAX_NEIGHBOURS) && MAX_NEIGHBOURS > 0 && \
(defined(BOARD_HAS_PSRAM) || defined(MQTT_NEIGHBORS_WITHOUT_PSRAM))
#define WITH_MQTT_NEIGHBORS 1
#endif
@@ -513,17 +517,38 @@ public:
#if defined(WITH_MQTT_NEIGHBORS)
// Single source of truth for the neighbors JSON size, used by both the bridge's
// persistent buffer and the mesh's transient build buffer.
#if defined(BOARD_HAS_PSRAM)
static const size_t NEIGHBORS_JSON_BUFFER_SIZE = 10240;
#else
// Without PSRAM all three neighbors allocations (persistent buffer, transient
// build buffer, ArduinoJson pool) come out of internal DRAM, which each TLS
// slot also needs ~40 KB of. A quarter-size buffer keeps the peak near 13 KB
// instead of ~35 KB; the builder drops the tail and sets "truncated" once the
// next entry will not fit, so the table degrades instead of failing.
static const size_t NEIGHBORS_JSON_BUFFER_SIZE = 4096;
#endif
// The ArduinoJson pool is NOT bounded by the text buffer: v7 hands out pool
// blocks in fixed 4096-byte chunks, so a table that just fits the text buffer
// can still need well over it in pool. Budgeting the pool at the text size
// starves it, and a starved pool sets doc.overflowed() — which drops the whole
// publish instead of truncating. Measured need is 12541 B for 50 entries.
// publish instead of truncating. Measured need is 12541 B for 50 entries and
// 4286 B for 21, so allow a spare block over each.
#if defined(BOARD_HAS_PSRAM)
static const size_t NEIGHBORS_DOC_POOL_BUDGET = 16384;
#else
static const size_t NEIGHBORS_DOC_POOL_BUDGET = 12288;
#endif
// Entries per publish; the neighbour cache and text buffer are the real limit.
// Entries per publish. One 4096-byte pool block holds 21, so staying under
// that keeps the non-PSRAM pool to a single block (~4.3 KB rather than
// ~8.4 KB) and the whole publish peak near 13 KB of internal DRAM. PSRAM
// boards are limited only by the text buffer and the neighbour cache.
#if defined(BOARD_HAS_PSRAM)
static const int NEIGHBORS_MAX_PUBLISH_ENTRIES = MAX_NEIGHBOURS;
#else
static const int NEIGHBORS_MAX_PUBLISH_ENTRIES = 20;
#endif
// Called by the mesh (Core 1) once a neighbor-discovery pass has built the
// table JSON. Copies it into the persistent PSRAM buffer and raises the
@@ -185,6 +185,8 @@ build_flags =
-D ADMIN_PASSWORD='"password"'
-D MAX_NEIGHBOURS=50
-D WITH_MQTT_BRIDGE=1
; Non-PSRAM board: neighbors table costs ~4 KB of internal DRAM (see MQTTBridge.h).
-D MQTT_NEIGHBORS_WITHOUT_PSRAM=1
-D MAX_MQTT_BROKERS=3
-D MQTT_MAX_PACKET_SIZE=1024
-D MQTT_DEBUG=1
@@ -221,6 +223,8 @@ build_flags =
-D ADMIN_PASSWORD='"password"'
-D MAX_NEIGHBOURS=50
-D WITH_MQTT_BRIDGE=1
; Non-PSRAM board: neighbors table costs ~4 KB of internal DRAM (see MQTTBridge.h).
-D MQTT_NEIGHBORS_WITHOUT_PSRAM=1
-D MAX_MQTT_BROKERS=3
-D MQTT_MAX_PACKET_SIZE=1024
-D MQTT_DEBUG=1
@@ -257,6 +261,8 @@ build_flags =
-D ADMIN_PASSWORD='"password"'
-D ROOM_PASSWORD='"hello"'
-D WITH_MQTT_BRIDGE=1
; Non-PSRAM board: neighbors table costs ~4 KB of internal DRAM (see MQTTBridge.h).
-D MQTT_NEIGHBORS_WITHOUT_PSRAM=1
-D MAX_NEIGHBOURS=50
-D MAX_MQTT_BROKERS=3
-D MQTT_MAX_PACKET_SIZE=1024
@@ -294,6 +300,8 @@ build_flags =
-D ADMIN_PASSWORD='"password"'
-D ROOM_PASSWORD='"hello"'
-D WITH_MQTT_BRIDGE=1
; Non-PSRAM board: neighbors table costs ~4 KB of internal DRAM (see MQTTBridge.h).
-D MQTT_NEIGHBORS_WITHOUT_PSRAM=1
-D MAX_NEIGHBOURS=50
-D MAX_MQTT_BROKERS=3
-D MQTT_MAX_PACKET_SIZE=1024
+8
View File
@@ -123,6 +123,8 @@ build_flags =
-D ADMIN_PASSWORD='"password"'
-D MAX_NEIGHBOURS=50
-D WITH_MQTT_BRIDGE=1
; Non-PSRAM board: neighbors table costs ~4 KB of internal DRAM (see MQTTBridge.h).
-D MQTT_NEIGHBORS_WITHOUT_PSRAM=1
-D MAX_MQTT_BROKERS=3
-D MQTT_MAX_PACKET_SIZE=1024
-D MQTT_DEBUG=1
@@ -203,6 +205,8 @@ build_flags =
-D ADMIN_PASSWORD='"password"'
-D ROOM_PASSWORD='"hello"'
-D WITH_MQTT_BRIDGE=1
; Non-PSRAM board: neighbors table costs ~4 KB of internal DRAM (see MQTTBridge.h).
-D MQTT_NEIGHBORS_WITHOUT_PSRAM=1
-D MAX_NEIGHBOURS=50
-D MAX_MQTT_BROKERS=3
-D MQTT_MAX_PACKET_SIZE=1024
@@ -497,6 +501,8 @@ build_flags =
-D ADMIN_PASSWORD='"password"'
-D MAX_NEIGHBOURS=50
-D WITH_MQTT_BRIDGE=1
; Non-PSRAM board: neighbors table costs ~4 KB of internal DRAM (see MQTTBridge.h).
-D MQTT_NEIGHBORS_WITHOUT_PSRAM=1
-D MAX_MQTT_BROKERS=3
-D MQTT_MAX_PACKET_SIZE=1024
-D MQTT_DEBUG=1
@@ -546,6 +552,8 @@ build_flags =
-D ADMIN_PASSWORD='"password"'
-D ROOM_PASSWORD='"hello"'
-D WITH_MQTT_BRIDGE=1
; Non-PSRAM board: neighbors table costs ~4 KB of internal DRAM (see MQTTBridge.h).
-D MQTT_NEIGHBORS_WITHOUT_PSRAM=1
-D MAX_NEIGHBOURS=50
-D MAX_MQTT_BROKERS=3
-D MQTT_MAX_PACKET_SIZE=1024
+4
View File
@@ -113,6 +113,8 @@ build_flags =
-D ADMIN_PASSWORD='"password"'
-D MAX_NEIGHBOURS=50
-D WITH_MQTT_BRIDGE=1
; Non-PSRAM board: neighbors table costs ~4 KB of internal DRAM (see MQTTBridge.h).
-D MQTT_NEIGHBORS_WITHOUT_PSRAM=1
-D MAX_MQTT_BROKERS=3
-D MQTT_MAX_PACKET_SIZE=1024
-D MQTT_DEBUG=1
@@ -176,6 +178,8 @@ build_flags =
-D ADMIN_PASSWORD='"password"'
-D ROOM_PASSWORD='"hello"'
-D WITH_MQTT_BRIDGE=1
; Non-PSRAM board: neighbors table costs ~4 KB of internal DRAM (see MQTTBridge.h).
-D MQTT_NEIGHBORS_WITHOUT_PSRAM=1
-D MAX_NEIGHBOURS=50
-D MAX_MQTT_BROKERS=3
-D MQTT_MAX_PACKET_SIZE=1024