fix(mqtt-neighbors): report unusable heard ages as null

A neighbour heard before the clock is set carries the firmware's unset-clock
default (1715770351, 15 May 2024). Subtracting that from an NTP-synced clock
published ages of ~806 days for neighbours that had just answered a live scope
query, and a backwards clock step reported 0, i.e. "heard just now".

- finishNeighborDiscover() reports the age as unknown ("heard_secs_ago": null)
  when the stored stamp and the current clock come from different epochs, or
  when the clock has stepped backwards
- handleNeighborDiscoverResponse() re-stamps heard_timestamp on a zero-hop scope
  reply, in both the snapshot and the live table, so entries heal once per
  discovery cycle instead of waiting for the neighbour's next advert
- publish ordering places usable ages ahead of unknown ones so a poisoned stamp
  cannot displace fresh entries when the JSON buffer truncates
- document the null case, and the always-present total_neighbors /
  queried_neighbors / truncated fields the payload sample omitted
- add UPSTREAM_BUGS.md, tracking the monotonic-uptime fix to propose upstream
  plus the unclamped subtraction in the companion and CLI readouts

Containment only: upstream still stamps neighbours from the wall clock at
packet-reception time, which on a cold boot always precedes NTP.
This commit is contained in:
agessaman committed 2026-07-31 08:54:39 -07:00
1 parent d8fd3defc4
commit 5828a3c563
11 files changed
+296 -17

No files matched your search

+31 -3
View File
@@ -777,6 +777,9 @@ While `mqtt.neighbors` is on, `get mqtt.status` appends `nbr: <next>/<last>` —
"timestamp": "2024-01-01T12:00:00.000000+00:00",
"origin": "MeshCore-HOWL",
"origin_id": "A1B2C3D4E5F67890...",
"total_neighbors": 2,
"queried_neighbors": 2,
"truncated": false,
"self": { "scopes": "DEN,APRS", "default_scope": "*" },
"neighbors": [
{
@@ -785,13 +788,38 @@ While `mqtt.neighbors` is on, `get mqtt.status` appends `nbr: <next>/<last>` —
"heard_secs_ago": 42,
"scopes": "DEN,APRS",
"status": "responded"
},
{
"pubkey": "8899AABBCCDDEEFF...",
"snr": 12.5,
"heard_secs_ago": null,
"scopes": "DEN",
"status": "responded"
}
]
}
```
Entries are ordered most- to least-useful (most recently heard, then stronger
SNR); the tail is dropped if the payload would exceed the 10 KB publish buffer.
`status` is `responded`, `timeout`, or `send_failed` per neighbor.
Entries are ordered most- to least-useful (usable age first, then most recently
heard, then stronger SNR); the tail is dropped if the payload would exceed the
10 KB publish buffer. `status` is `responded`, `timeout`, or `send_failed` per
neighbor.
`total_neighbors` is the size of the neighbor-table snapshot the cycle started
from, `queried_neighbors` how many scope requests were confirmed transmitted, and
`truncated` whether the buffer filled before every entry fit. All three are
always present. The `neighbors` array can therefore be shorter than
`total_neighbors` — compare its length against that field rather than assuming
the table is complete.
`heard_secs_ago` is `null`, as in the second entry above, when the age cannot be
computed: the neighbor was last heard before the clock was set, so the stored
stamp and the current clock come from different epochs and their difference is
meaningless (see `UPSTREAM_BUGS.md` #1). **Consumers must treat `null` as
unknown, not as zero** — the key is always present, so a missing key means an
older firmware, and a `null` never means "heard just now". A neighbor that
answers the scope query has its stamp refreshed, so a `null` age normally clears
itself on the next publish cycle.
`self.default_scope` is the region name this node floods to by default (`region
default`); it is `*` when no default region is set, matching the unscoped flood
the radio actually performs in that case.
+7 -3
View File
@@ -69,10 +69,14 @@ defined in `MQTTBridge.h`). It spans two subsystems and two cores:
(`startNeighborDiscover`) fires one anon-regions scope query per heard neighbor,
overlaying them onto the peer-index space at `NEIGHBOR_DISCOVER_PEER_BASE` so their
`PAYLOAD_TYPE_RESPONSE` packets decrypt via `searchPeersByHash`/`getPeerSharedSecret`/
`onPeerDataRecv` even when the neighbor is not an ACL client. After all responses land
or a 30 s window expires, `finishNeighborDiscover()` builds the JSON with
`onPeerDataRecv` even when the neighbor is not an ACL client. A reply is zero-hop by
request, so `handleNeighborDiscoverResponse` also re-stamps `heard_timestamp` in both
the snapshot and `neighbours[]` — proof of reception, and the only thing that heals a
stamp taken before the clock was set. After all responses land or a 30 s window
expires, `finishNeighborDiscover()` builds the JSON with
`MQTTMessageBuilder::buildNeighborsMessage` into a transient PSRAM buffer and hands it
to the bridge.
to the bridge. Ages that still span a clock epoch publish as `null` rather than a
fabricated delta (`neighborHeardAgeUsable`, see `UPSTREAM_BUGS.md` #1).
- **Bridge side, handoff**: `requestPublishNeighbors(json, len)` (Core 1) memcpys into a
persistent ~10 KB PSRAM buffer (`NEIGHBORS_JSON_BUFFER_SIZE`) and sets
`_neighbors_publish_pending` with a release store; the MQTT task (`mqttTaskLoop`, Core 0)
+122
View File
@@ -0,0 +1,122 @@
# Upstream Bug Candidates
Bugs in code this fork inherits from upstream (`meshcore-dev/dev`), each with the
fix we would propose upstream. Any fork-local workaround is recorded so it can be
dropped once upstream lands a real fix.
Status: **open** (not reported) · **reported** (issue/PR filed) · **landed**
(upstream fixed, fork workaround removable).
---
## 1. Neighbour "heard" ages are derived from a steppable wall clock
**Status:** open
**Files:** `examples/simple_repeater/MyMesh.cpp`, `examples/simple_room_server/MyMesh.cpp`
**Impact:** wrong ages reported to companion apps, CLI, and (in this fork) MQTT. No crash.
### Symptom
A repeater published a neighbour age of 69681995 seconds — 806 days — for two of
three neighbours that had just answered a live query (capture abridged; see
`MQTT_IMPLEMENTATION.md` for the full payload shape):
```json
{"timestamp":"2026-07-30T22:59:09.000000+00:00","total_neighbors":3,
"neighbors":[
{"pubkey":"0CE5EA7C…","snr":12.5,"heard_secs_ago":57,"status":"responded"},
{"pubkey":"B0D17C59…","snr":12.5,"heard_secs_ago":69681995,"status":"responded"},
{"pubkey":"D19BEE16…","snr":12.25,"heard_secs_ago":69681995,"status":"responded"}]}
```
### Root cause
`heard_secs_ago` is a *relative* quantity computed by subtracting two samples of
an *absolute* wall clock that can step by years mid-session.
`putNeighbour()` stamps `heard_timestamp = getRTCClock()->getCurrentTime()` when a
zero-hop advert or node-discover reply arrives (`MyMesh.cpp:93`). On a cold boot
the radio is receiving within a second or two, while the clock is still at the
firmware's unset default — `1715770351`, 15 May 2024 (`src/helpers/ArduinoHelpers.h`,
`src/helpers/ESP32Board.h`). NTP (or an admin over the CLI) then steps the clock
forward by years, and every later reader subtracts the two:
```
publish epoch 1785452349 (2026-07-30T22:59:09Z)
− reported age 69681995
= stored heard_timestamp 1715770354 = unset-clock default + 3 s of uptime
```
So both stale entries were stamped 3 seconds after boot. The entries do not
self-heal: `putNeighbour()` is the only writer of `heard_timestamp`, so an entry
stays wrong until that specific neighbour is physically heard again. In the capture
above the two stale neighbours missed the node-discover refresh (responder-side
rate limit `discover_limiter`, a reply collision, or firmware without
`CTL_TYPE_NODE_DISCOVER_RESP`) yet still answered a direct query — hence
`"status":"responded"` next to a 2-year age.
Deferring the *read* until after time sync does not fix this. The bad value is
written at packet-reception time, which on any cold boot precedes WiFi and NTP.
### Affected readers
| Path | Location |
| --- | --- |
| Companion `REQ_TYPE_GET_NEIGHBOURS` binary reply | `MyMesh.cpp:404` |
| CLI `neighbors` text output | `MyMesh.cpp:1350` |
| MQTT neighbors JSON (fork-only) | `MyMesh.cpp:2036` |
### Proposed upstream fix
Stamp neighbours from a monotonic source and derive every age from that:
- add `uint32_t heard_millis` to `NeighbourInfo`, set from `millis()` in
`putNeighbour()` alongside the existing wall-clock stamp;
- compute every "seconds ago" readout as `(now_millis - heard_millis) / 1000`
using the unsigned-subtraction idiom already used by `millisHasNowPassed()` /
`futureMillis()`, which is immune to clock steps and stays correct across one
32-bit rollover;
- clamp the result to a documented maximum so an entry untouched for longer than
the 49.7-day `millis()` period cannot alias back to a small age;
- keep `heard_timestamp` for anything that needs an absolute instant — sort order
is equivalent under either field.
Cost: 4 bytes × `MAX_NEIGHBOURS` of RAM.
### Fork-local containment (in place)
Not a substitute for the above; it stops the fork from publishing a fabricated
number while upstream still stamps from the wall clock.
- `finishNeighborDiscover()` reports the age as unknown — `"heard_secs_ago":null`
— when the stored stamp predates `MQTTConnectionPolicy::kSyncedClockEpoch`
(2025-01-01, above the unset-clock default) while the current clock is past it,
or when the clock has stepped backwards. Previously a backwards clock reported
`0`, i.e. "heard just now".
- `handleNeighborDiscoverResponse()` re-stamps `heard_timestamp` on a zero-hop
scope reply, in both the snapshot and the live table. The reply is proof of
reception, so entries now heal once per discovery cycle instead of waiting for
the neighbour's next advert. This would have made 2 of the 3 entries above
correct.
- Publish ordering places known ages before unknown ones, so an unusable stamp
cannot masquerade as the freshest entry when the JSON buffer truncates.
---
## 2. Neighbour age subtraction is unclamped in the companion and CLI readouts
**Status:** open
**Files:** `examples/simple_repeater/MyMesh.cpp:404`, `:1350` (and the room-server twins)
**Impact:** age underflows to ~4.29e9 instead of a sane value.
Both readers subtract without checking ordering:
```cpp
uint32_t heard_seconds_ago = getRTCClock()->getCurrentTime() - neighbour->heard_timestamp;
```
Any state where `heard_timestamp > now` — clock stepped backwards by NTP or an
admin, or a stamp taken before a backwards step — wraps the unsigned result to
near `UINT32_MAX`. Distinct from #1 in that it is a missing guard rather than the
choice of clock: worth fixing on its own even before the monotonic change, by
clamping to `0` (or reporting "unknown", which is what the age actually is).
+40 -5
View File
@@ -2,6 +2,9 @@
#include <algorithm>
#include <stdlib.h> // for qsort()
#include <helpers/RxReservePacketManager.h>
#if defined(WITH_MQTT_NEIGHBORS)
#include <helpers/MQTTConnectionPolicy.h> // kSyncedClockEpoch
#endif
/* ------------------------------ Config -------------------------------- */
@@ -1967,15 +1970,46 @@ bool MyMesh::handleNeighborDiscoverResponse(int overlay_idx, const uint8_t* data
memcpy(entry.scopes, &data[8], scope_len);
entry.scopes[scope_len] = 0;
entry.status = ND_RESPONDED;
// A zero-hop reply is proof we heard this neighbour now, so re-stamp both the
// snapshot and the live table; a stamp taken before time sync heals here.
entry.heard_timestamp = getRTCClock()->getCurrentTime();
touchNeighbourHeard(entry.id, entry.heard_timestamp);
return true;
}
// Publish-ordering: most recently heard first, then stronger SNR, then pubkey.
// The JSON builder drops the tail if the buffer fills, so the head must be the
// most useful entries.
// Refresh a live neighbour's heard time only: a scope reply carries no advert
// timestamp or SNR to update.
void MyMesh::touchNeighbourHeard(const mesh::Identity& id, uint32_t heard_timestamp) {
#if MAX_NEIGHBOURS
for (int i = 0; i < MAX_NEIGHBOURS; i++) {
if (id.matches(neighbours[i].id)) {
neighbours[i].heard_timestamp = heard_timestamp;
return;
}
}
#endif
}
// A heard age is a wall-clock delta, so it only means something when both stamps
// share a clock epoch. An entry heard before the clock was set holds the unset
// default, which a synced clock turns into a ~2-year age; report those as
// unknown instead. See UPSTREAM_BUGS.md for the monotonic fix.
static bool neighborHeardAgeUsable(uint32_t heard_timestamp, uint32_t now_secs) {
if (heard_timestamp == 0 || now_secs < heard_timestamp) return false;
// Never synced: the stamp shares this clock's boot epoch, so the delta holds.
if (now_secs < MQTTConnectionPolicy::kSyncedClockEpoch) return true;
return heard_timestamp >= MQTTConnectionPolicy::kSyncedClockEpoch;
}
// Publish-ordering: usable ages first, then most recently heard, then stronger
// SNR, then pubkey. The JSON builder drops the tail if the buffer fills, so the
// head must be the most useful entries.
static bool neighborPublishEntryComesBefore(
const MQTTMessageBuilder::NeighborsMessageEntry& lhs,
const MQTTMessageBuilder::NeighborsMessageEntry& rhs) {
if (lhs.heard_unknown != rhs.heard_unknown) {
return !lhs.heard_unknown;
}
if (lhs.heard_secs_ago != rhs.heard_secs_ago) {
return lhs.heard_secs_ago < rhs.heard_secs_ago; // newer first
}
@@ -2033,8 +2067,9 @@ void MyMesh::finishNeighborDiscover() {
mesh::Utils::toHex(pubkey_hex[i], entry.id.pub_key, PUB_KEY_SIZE);
entries[i].pubkey_hex = pubkey_hex[i];
entries[i].snr = entry.snr / 4.0f;
entries[i].heard_secs_ago = (entry.heard_timestamp > 0 && now_secs >= entry.heard_timestamp)
? (now_secs - entry.heard_timestamp) : 0;
bool heard_known = neighborHeardAgeUsable(entry.heard_timestamp, now_secs);
entries[i].heard_unknown = !heard_known;
entries[i].heard_secs_ago = heard_known ? (now_secs - entry.heard_timestamp) : 0;
entries[i].scopes = entry.scopes;
switch (entry.status) {
case ND_RESPONDED: entries[i].status = "responded"; break;
+1
View File
@@ -195,6 +195,7 @@ class MyMesh : public mesh::Mesh, public CommonCLICallbacks
void loopNeighborDiscover();
void finishNeighborDiscover();
bool handleNeighborDiscoverResponse(int overlay_idx, const uint8_t* data, size_t len);
void touchNeighbourHeard(const mesh::Identity& id, uint32_t heard_timestamp);
void getLocalScopes(char* buf, size_t len);
// Overlay peer indices are offset by this base so onPeerDataRecv can tell a
// discovery response apart from a normal ACL-client index.
+38 -5
View File
@@ -1,6 +1,9 @@
#include "MyMesh.h"
#include <algorithm>
#include <helpers/RxReservePacketManager.h>
#if defined(WITH_MQTT_NEIGHBORS)
#include <helpers/MQTTConnectionPolicy.h> // kSyncedClockEpoch
#endif
#define ANON_REQ_TYPE_REGIONS 0x01 // client side of the anon-regions scope query (neighbors feature)
@@ -1853,15 +1856,44 @@ bool MyMesh::handleNeighborDiscoverResponse(int overlay_idx, const uint8_t* data
memcpy(entry.scopes, &data[8], scope_len);
entry.scopes[scope_len] = 0;
entry.status = ND_RESPONDED;
// A zero-hop reply is proof we heard this neighbour now, so re-stamp both the
// snapshot and the live table; a stamp taken before time sync heals here.
entry.heard_timestamp = getRTCClock()->getCurrentTime();
touchNeighbourHeard(entry.id, entry.heard_timestamp);
return true;
}
// Publish-ordering: most recently heard first, then stronger SNR, then pubkey.
// The JSON builder drops the tail if the buffer fills, so the head must be the
// most useful entries.
// Refresh a live neighbour's heard time only: a scope reply carries no advert
// timestamp or SNR to update.
void MyMesh::touchNeighbourHeard(const mesh::Identity& id, uint32_t heard_timestamp) {
for (int i = 0; i < MAX_NEIGHBOURS; i++) {
if (id.matches(neighbours[i].id)) {
neighbours[i].heard_timestamp = heard_timestamp;
return;
}
}
}
// A heard age is a wall-clock delta, so it only means something when both stamps
// share a clock epoch. An entry heard before the clock was set holds the unset
// default, which a synced clock turns into a ~2-year age; report those as
// unknown instead. See UPSTREAM_BUGS.md for the monotonic fix.
static bool neighborHeardAgeUsable(uint32_t heard_timestamp, uint32_t now_secs) {
if (heard_timestamp == 0 || now_secs < heard_timestamp) return false;
// Never synced: the stamp shares this clock's boot epoch, so the delta holds.
if (now_secs < MQTTConnectionPolicy::kSyncedClockEpoch) return true;
return heard_timestamp >= MQTTConnectionPolicy::kSyncedClockEpoch;
}
// Publish-ordering: usable ages first, then most recently heard, then stronger
// SNR, then pubkey. The JSON builder drops the tail if the buffer fills, so the
// head must be the most useful entries.
static bool neighborPublishEntryComesBefore(
const MQTTMessageBuilder::NeighborsMessageEntry& lhs,
const MQTTMessageBuilder::NeighborsMessageEntry& rhs) {
if (lhs.heard_unknown != rhs.heard_unknown) {
return !lhs.heard_unknown;
}
if (lhs.heard_secs_ago != rhs.heard_secs_ago) {
return lhs.heard_secs_ago < rhs.heard_secs_ago; // newer first
}
@@ -1919,8 +1951,9 @@ void MyMesh::finishNeighborDiscover() {
mesh::Utils::toHex(pubkey_hex[i], entry.id.pub_key, PUB_KEY_SIZE);
entries[i].pubkey_hex = pubkey_hex[i];
entries[i].snr = entry.snr / 4.0f;
entries[i].heard_secs_ago = (entry.heard_timestamp > 0 && now_secs >= entry.heard_timestamp)
? (now_secs - entry.heard_timestamp) : 0;
bool heard_known = neighborHeardAgeUsable(entry.heard_timestamp, now_secs);
entries[i].heard_unknown = !heard_known;
entries[i].heard_secs_ago = heard_known ? (now_secs - entry.heard_timestamp) : 0;
entries[i].scopes = entry.scopes;
switch (entry.status) {
case ND_RESPONDED: entries[i].status = "responded"; break;
+1
View File
@@ -189,6 +189,7 @@ class MyMesh : public mesh::Mesh, public CommonCLICallbacks
void loopNeighborDiscover();
void finishNeighborDiscover();
bool handleNeighborDiscoverResponse(int overlay_idx, const uint8_t* data, size_t len);
void touchNeighbourHeard(const mesh::Identity& id, uint32_t heard_timestamp);
void getLocalScopes(char* buf, size_t len);
static const int NEIGHBOR_DISCOVER_PEER_BASE = 1000;
static const unsigned long NEIGHBOR_DISCOVER_QUEUE_TIMEOUT_MS = 29000;
+4
View File
@@ -18,6 +18,10 @@ static const uint32_t kDefaultJwtLifetimeSecs = 86400UL;
static const uint32_t kMaxJwtStaggerSecs = 300UL;
static const uint32_t kMinimumValidEpoch = 1000000000UL;
static const uint32_t kJwtClockThreshold = 1735689600UL; // 2025-01-01 UTC
// A wall clock at or past this instant was set from a real source (NTP or an
// admin); anything earlier is the firmware's unset-clock default (1715770351,
// 15 May 2024), so a delta spanning the sync is meaningless.
static const uint32_t kSyncedClockEpoch = kJwtClockThreshold;
// Unsigned subtraction is intentionally used: it is the standard millis()
// idiom and remains correct across a single 32-bit counter rollover.
+5 -1
View File
@@ -220,7 +220,11 @@ static void addNeighborsMessageEntry(
JsonObject nb = arr.add<JsonObject>();
nb["pubkey"] = neighbor.pubkey_hex;
nb["snr"] = neighbor.snr;
nb["heard_secs_ago"] = neighbor.heard_secs_ago;
if (neighbor.heard_unknown) {
nb["heard_secs_ago"] = nullptr; // age unknown, not zero
} else {
nb["heard_secs_ago"] = neighbor.heard_secs_ago;
}
nb["scopes"] = neighbor.scopes ? neighbor.scopes : "";
nb["status"] = neighbor.status;
}
+5
View File
@@ -75,6 +75,11 @@ public:
uint32_t heard_secs_ago;
const char* scopes;
const char* status;
// True renders heard_secs_ago as JSON null, for a neighbour whose stored
// stamp cannot yield an age. No default initializer: the struct stays an
// aggregate for the device toolchain, and a zeroed tail means "age known",
// so size-measurement callers keep reserving the widest numeric value.
bool heard_unknown;
};
// Build neighbors-table JSON for the meshcore/{iata}/{device}/neighbors topic.
@@ -407,6 +407,7 @@ TEST(MQTTPayloadBuilder, NeighborsMessageDropsTailWhenBufferFills) {
neighbors[i].pubkey_hex = keys[i];
neighbors[i].snr = static_cast<float>(i);
neighbors[i].heard_secs_ago = static_cast<uint32_t>(i) * 10U;
neighbors[i].heard_unknown = false;
neighbors[i].scopes = "DEN";
neighbors[i].status = "active";
}
@@ -430,6 +431,47 @@ TEST(MQTTPayloadBuilder, NeighborsMessageDropsTailWhenBufferFills) {
EXPECT_STREQ(keys[0], arr[0]["pubkey"].as<const char*>());
}
TEST(MQTTPayloadBuilder, NeighborsMessageRendersUnknownHeardAgeAsNull) {
// A neighbour stamped before the clock was set has no usable age. The key
// stays present and null so a consumer cannot read it as "heard just now".
MQTTPayloadBuilder::NeighborsMessageEntry neighbors[2];
neighbors[0] = {"0011223344556677", 9.75f, 42, "DEN", "responded"};
neighbors[1] = {"8899AABBCCDDEEFF", -3.5f, 0, "DEN", "responded"};
neighbors[1].heard_unknown = true;
JsonDocument scratch;
char buffer[1024];
int len = MQTTPayloadBuilder::buildNeighborsMessage(
scratch, "node", "id", kTimestamp, "DEN", "*", neighbors, 2,
buffer, sizeof(buffer), 2, 2, false);
ASSERT_GT(len, 0);
JsonDocument parsed;
ASSERT_FALSE(deserializeJson(parsed, buffer));
JsonArray arr = parsed["neighbors"].as<JsonArray>();
ASSERT_EQ(2U, arr.size());
EXPECT_EQ(42U, arr[0]["heard_secs_ago"].as<uint32_t>());
EXPECT_FALSE(arr[0]["heard_secs_ago"].isNull());
EXPECT_TRUE(arr[1]["heard_secs_ago"].isNull());
EXPECT_NE(nullptr, strstr(buffer, "\"heard_secs_ago\":null"));
// The rest of the entry still renders.
EXPECT_STREQ("8899AABBCCDDEEFF", arr[1]["pubkey"].as<const char*>());
EXPECT_STREQ("responded", arr[1]["status"].as<const char*>());
}
TEST(MQTTPayloadBuilder, NeighborsMessageUnknownHeardAgeFitsMeasuredWidth) {
// Paced discovery measures each entry with a known UINT32_MAX age; a null age
// must never serialize wider than what that reserved.
MQTTPayloadBuilder::NeighborsMessageEntry measured = {
"0011223344556677", 9.75f, UINT32_MAX, "DEN", "responded"};
MQTTPayloadBuilder::NeighborsMessageEntry unknown = measured;
unknown.heard_unknown = true;
unknown.heard_secs_ago = 0;
EXPECT_LE(MQTTPayloadBuilder::measureNeighborsMessageEntry(unknown),
MQTTPayloadBuilder::measureNeighborsMessageEntry(measured));
}
} // namespace
int main(int argc, char** argv) {