Commit Graph
1408 Commits
Author SHA1 Message Date
agessaman 28dd599fa8 fix(observer): recover T-Beam Supreme display startup 2026-08-10 12:59:02 -07:00
agessaman 95f326954b merge: perf/memory-savings into observer-firmware-dev
Nine commits reducing the MQTT bridge's internal-DRAM footprint, plus four
fixes that rode with them (invalid path encodings, stale-JWT scan after a
clock correction, setup-retry interval measured from the failure, retried
setup consuming the reconnect allowance).

Touches no prefs surface -- nothing in NodePrefs, MQTTPrefs, or
ConfigSerializer -- so /prefs.json layout is unaffected and there is no
fleet config risk.

Soak evidence: every soak branch already contained this work in full. Device 1
has run it 69.6 h with 325,852 publishes, 0 errors and 0 reboots. The caveat
worth carrying: that long-duration evidence is all on the reduced-TLS
framework (OUT_CONTENT_LEN 4096). Device 3 is now soaking it on the stock
framework, which is where the allocation-ordering interaction with the full
16 KiB record buffers actually gets exercised.
2026-08-09 09:33:20 -07:00
agessaman f9968ac1a5 merge: upstream/dev into observer-firmware-dev (v1.17.0)
Picks up upstream MeshCore 1.17.0 and the dev commits that followed it.

Notable upstream content:
- 1.17.0 version/build-date bump in the example MyMesh headers.
- anon_req hardening: reply_path_len is now uint8_t with an isValidPathLen()
  bounds check and a 0xFF sentinel; reply_path_hash_size is gone.
- LR2021 support (Meshnology W12, Seeed MeshTracker X1) incl. side detectors,
  multi-SF and the new `extra.sf` get/set CLI.
- CustomLFS 0.2.2 -> 0.2.3 (GD25Q64C support).
- kiss_modem envs for several nRF52 variants; ThinkNode M6 GPS/flash fixes.

Conflicts resolved:
- platformio.ini: took upstream's CustomLFS 0.2.3, kept our lib_ignore for
  the vendored PsychicMqttClient.
- CommonCLI.h: kept both upstream's USE_LR2021 configSideDetectors() hook and
  our fault-alert callbacks.

NodePrefs gains upstream's extra_sf[4], which is not registered with
ConfigSerializer, so /prefs.json layout is unchanged.

Verified: 275/275 native host tests pass; Heltec_v3_repeater_observer_mqtt,
Heltec_v3_repeater and heltec_v4_repeater_observer_mqtt all build.
2026-08-09 06:32:53 -07:00
Adam Gessaman 2125917f15 Merge pull request #43 from agessaman/feat/webconfig-cli
Feat/webconfig cli
2026-08-08 17:20:24 -07:00
agessaman 6cfdc61baf fix(webconfig): define the in-class constants out of line
LilyGo_TLora_V2_1_1_6 observer builds failed to link:

  undefined reference to `WebConfigServer::MAX_BATCH'
    in handleStatus and handleCliPost

An in-class initialiser is only a declaration under C++11, which is what the
xtensa-esp32 toolchain builds with. Every previous use of MAX_BATCH was a
comparison, which reads the value and needs no symbol. Reporting it as
status.max_cmds, and naming it in the "too many commands" error, passes it to
ArduinoJson — which takes `const T&` — and binding a reference odr-uses it.

It linked on most targets because the compiler folded the reference away, and
failed on the ones where it did not. A cast at the two call sites would have
silenced it just as narrowly; defining the symbols is what stops the next use
from depending on the same luck. MAX_BODY and STOP_WARN_MS get the same
treatment for the same reason, before they are the next to be passed by
reference.
2026-08-08 17:13:00 -07:00
agessaman da37b6eb2d feat(mqtt): add idahomesh broker preset
Keep built-in preset names in parity with observer-firmware after #41.
2026-08-08 15:19:05 -07:00
agessaman 8abe26ba7b fix(webconfig): stop the CLI reading secrets, and enforce the setup password
Two findings from review, both real, both mine.

The CLI could read secrets the portal has never exposed. CommonCLI splits its
surface by CALLER, not by command: a serial caller (sender_timestamp 0, physical
access) reads secrets in plaintext, a remote one gets "******** (serial only)".
Its own comments say so — "Serial only (WiFi creds grant LAN access); remote
sees set/unset". execCommand passes 0, which is what makes `erase`, `stats-*`
and `set freq` reachable at all, and with it the terminal inherited the serial
console's plaintext answers for an HTTP request: `get prv.key` returned this
node's identity, `get wifi.pwd` the operator's network.

Worse in setup mode, which authenticates by proximity to an open AP — and `start
webconfig ap` can be run on an already-configured node, so the secrets are real
by then, not blank.

I had reasoned that the AP was the trust boundary either way because the wizard
can already rewrite these. That conflated two capabilities: replacing a WiFi
password does not reveal the current one, and replacing an identity does not
reveal the existing private key. /api/config has always masked these on read
(wcIsSecretKey); the CLI simply broke that rule. Now only the READ is masked —
the command surface stays whole — in CommonCLI's own words, keeping the
set/unset signal that is the useful part.

Onboarding could also skip the mandatory password. handleConfigPost refuses to
arm a reboot during initial setup without one; the CLI only warned in the
browser, which a pasted script or a direct POST ignores, so a node could reboot
onto the LAN still holding the factory credential. Same rule now applies at
POST. It is satisfied by a `password` command anywhere in the session rather
than only in the same request, so the natural two-step console flow still works
— the form batch always sends both together and never needed that memory.

wcIsSecretReadCommand lives in WebConfigKeys.h beside the rest of the secret
classification, pinned by three host tests: what must be masked, what must not,
and that only reads are touched. 17 keys + 24 batch tests pass; the audit checks
a masked read round-trips as masked.
2026-08-08 14:05:02 -07:00
agessaman 20826dcccf feat(webconfig): trim the displayed version to base, build and channel
The full embedded string is v1.16.0.5-observer-beta-dev-a1b2c3d. The -observer
tag is on every observer build and the commit is for machines, so neither tells
a person anything at a glance. Both the page header and the console banner now
show base + published build number + channel, paired with the build date the
way `ver` pairs them:

  v1.16.0.5-dev (6 Jun 2026)
  v1.16.0.5 (6 Jun 2026)

`ver` still prints the whole string, commit included, for when that is what you
need. The channel suffix follows the release filenames rather than the embedded
tag — build.sh writes FILENAME_CHANNEL_TAG "-dev" for the same builds it tags
"-observer-beta-dev" internally, so "-dev" is the name these already carry.

Carrying the build date meant /api/status had to report it; WebConfigServer now
takes FIRMWARE_BUILD_DATE alongside FIRMWARE_VERSION, from the same defines
`ver` reads.

A local build has neither build number nor channel to show, so the fact worth
knowing about it moves to the second line: "local build, OTA not configured".
build.sh deliberately leaves OTA_MANIFEST_BASE undefined there, and a bare
version number gives no hint that the node cannot update itself.
2026-08-08 09:01:29 -07:00
agessaman c831e599ec fix(webconfig): tighten CLI failure detection, reboot deferral and refusals
Five findings from review, all confirmed against the source.

Failure classification (P2). Testing replies for an "Err" prefix passed five
other shapes off as success: "Unknown command", "unknown config: x", "??: x",
"Can't find GPS", "(ERR: clock cannot go backwards)" and "File system erase:
Err". They rendered green, and worse, left _batch_all_ok true — so a queued
reboot went ahead after commands that had failed, defeating the gate entirely.

Rather than lengthen one guess, the two questions are now asked separately,
each erring safe:

  - colour asks "does this look like a failure", against every shape CommonCLI
    actually emits, enumerated in WebConfigBatch.h and pinned by a host test
    that uses the literal strings. Getting this wrong is cosmetic.
  - the reboot gate asks something narrower and answerable: "did every setting
    I asked for take". Only `set`/`password` gate it, and only on the "OK"
    prefix every setter keeps. Diagnostics no longer gate a reboot at all, so a
    harmless `memory` cannot strand one and no guess is made about "> value".

Reboot deferral (P2). CommonCLI dispatches on a six-byte prefix, so `reboot
now` and `rebooted` reach Board::reboot() too. Matching exactly meant those
variants skipped both the confirmation and the deferral and took the node down
mid-drain — the precise failure deferral exists to prevent. Both sides now
anchor the way the firmware dispatches, and the UI's risk matcher with them.

Three commands the portal cannot honestly serve are refused at POST with a
reason, and dropped from autocomplete, instead of running and lying:

  - `start ota` builds a second AsyncWebServer on port 80 with no bind check
    and answers "Started" regardless; the portal already holds that port, so it
    could only leak the allocation and inhibit sleep.
  - `clock sync` takes its time from the caller's timestamp, which a web
    request has none of, so CommonCLI always rejected it. `time <epoch>` works
    and remains offered.
  - bare `log` and `get acl` write their real output to Serial and hand back a
    stub the terminal showed as success; `log` also streams a whole file from
    the loop task, stalling the mesh and radio while it does.

The mock now emits the same failure shapes it used to fake as successes, so
these are reproducible off-hardware. 24 batch + 14 keys tests pass; audit
reports 119/119 answered, 0 missing, 4/4 refused with a reason.
2026-08-08 08:40:49 -07:00
agessaman d532e4ea86 fix(webconfig): correct reply classification and the missing MyMesh commands
Two things hardware turned up.

The whole terminal came back red. The endpoint decided a command had succeeded
by testing its reply for an "OK" prefix — the convention the config batch relies
on, and a safe one there because every allowlisted setter uses it. The CLI
reaches the whole surface, where success has no single shape: setters answer
"OK...", getters answer "> value", `erase` answers "File system erase: OK". Only
failure is uniform ("Err", "ERR:", "Error:"), so that is what the CLI now tests
for.

Colour was the visible half. The other half was worse: _batch_all_ok went false
the moment a sequence contained a `get`, so a script ending in `reboot` was told
some commands had failed and the reboot was withheld.

Replies are green now and red means the node said no, which is what red should
have meant all along. The "> " a getter prefixes its value with is dropped for
display — on the serial console it sets the value apart, here it collides with
the prompt glyph that means "you typed this". The mock emits that marker too;
had it done so from the start, this would have shown up before the flash.

Second: discover.neighbors and discover.scopes did not autocomplete, because
MyMesh::handleCommand intercepts a few commands before delegating to CommonCLI
and the table was built by reading CommonCLI alone. setperm, `get acl` and
`shutdown` were missing for the same reason.

The audit could not have caught that: it drove every command the table offered
and checked the mock answered, which only finds gaps in one direction. It now
also reads the command literals the firmware dispatches on — across CommonCLI,
CommonCLI_Observer and MyMesh — and fails on any the table does not offer. That
check found `shutdown` immediately.

122 commands, all answered, none missing. 22 batch + 14 keys tests pass.
2026-08-08 08:14:54 -07:00
agessaman d7109c185c feat(webconfig): implement /api/cli on the device
The terminal has been driving the mock since it was built. This is the firmware
side, so it works on hardware.

Same 202 + reqid + poll contract as a config save, for the same reason:
CommonCLI touches prefs, the radio and the filesystem, none of which may be
reached from the async_tcp task. Commands go into the deferred slot and tick()
drains them on the loop task. Unlike a save this is not allowlisted — reaching
what the serial console reaches is the point, and execCommand() already passes
sender_timestamp 0, so the terminal gets exactly the serial console's
privilege. Authentication is the boundary, as it is there.

The CLI shares the config batch's slot rather than owning a second MAX_BATCH
array: both drain on the loop task, both are single-slot, and a duplicate would
cost ~8 KB of permanently resident RAM. Sharing also makes a save and a CLI run
mutually exclusive, which they must be. Each reader checks the kind, so neither
can serve the other's results.

Three things the mock could not have taught us:

  - Board::reboot() does not return, so a drained `reboot` would take the node
    down before the client read a single result. It is answered rather than
    executed, and the batch arms the existing deferred-reboot path once the
    results have been read — withheld if any command failed, exactly as a save
    withholds one. clkreboot/poweroff/ota update do real work on the way down
    and cannot be faked, so they still drop the connection; the UI warns first.
  - `password <new>` echoes the new password in its reply. The config path
    already scrubbed that by key; a CLI entry has no key, so it is matched on
    the command. CLI commands are also kept out of the serial log entirely —
    the browser session and the serial console are different audiences.
  - MAX_BATCH is 24, not the 64 the page assumed. It is reported as
    status.max_cmds instead of hardcoded, so the cap cannot drift.

Results stream and page (kCliResultPage = 8), and "done" means the client has
been handed every result, not merely that execution finished — otherwise a
client that stops polling at "done" loses the last page. Commands are never
echoed back: they may carry a secret, and the client matches by index.

New decisions live in WebConfigBatch.h with the rest, covered by three host
tests. Builds clean for heltec_v4_repeater_observer_mqtt; 22 batch + 14 keys
tests pass; the CLI audit reports 119/119 against the updated mock.
2026-08-07 23:16:38 -07:00
agessaman 3b11540e15 feat(mqtt): add gomesh to broker presets table 2026-08-07 20:54:56 -07:00
Martynas Zilinskas 2436229591 Update MQTT preset for atvirastinklas to use GTS_ROOT_R4 certificate 2026-08-07 18:07:41 +03:00
Martynas Zilinskas da7b51f181 Add new MQTT preset for atvirastinklas. 2026-08-07 17:53:58 +03:00
Hacuchino-hash 3abe415cf7 Add RGB LED and haptic notifications for SenseCAP MeshTracker X1 2026-08-06 07:27:05 -05:00
agessaman 73faa30cfa feat(mqtt): increase built-in MQTT preset count to 31 2026-08-05 19:58:47 -07:00
Josh Clarke c0adb67033 Add new MQTT presets for okimesh 2026-08-05 22:37:03 -04:00
ripplebiz 84e3a67bd7 Merge pull request #3115 from oltaco/meshnology_w12
Add support for Meshnology W12 and LR2021 Wrapper
2026-08-05 15:04:00 +10:00
taco 9d4f93806d LR2021: auto-LDRO for side detectors 2026-08-05 14:23:32 +10:00
taco 36e77671a2 LR2021: fix for hardware cad and side detectors 2026-08-05 14:23:32 +10:00
taco 696a82d7c2 LR2021 multi-SF support (side detectors) 2026-08-05 14:23:32 +10:00
taco 7cc16366cb add Meshnology W12 support and LR2021 wrapper 2026-08-05 14:23:29 +10:00
agessaman 8d1a0eb333 fix(mqtt): reject invalid path encodings before serializing
canSerialize() validated payload_len and the destination size but not whether the
path encoding is one writePath() will actually emit. writePath() self-guards
against overrunning the path array, but it does so by writing nothing and
returning 0 — a correctness problem, not a safety one, because getRawLength()
still counts the path. An over-long or reserved encoding therefore passed the
size check and then serialized to a truncated frame that was published as the
packet.

Worst case is path_len 0xFF with no payload: 63 hops of 4 bytes counts as 254
bytes, inside the 255-byte buffer, while writeTo() emits just the 2-byte header.
The `raw` field would carry 4 hex chars presented as the frame. Reserved 4-byte
hash encodings passed too, producing frames Packet::readFrom() rejects.

Now gated on Packet::isValidPathLen(), which rejects the reserved 4-byte hash
size and any count * size above MAX_PATH_SIZE in one predicate. It is the same
check readFrom() applies to every received packet, and TX packets are built via
setPathHashSizeAndCount() with real hash sizes, so no decodable packet is turned
away.

Two tests added for the cases a destination-size check cannot reach. The existing
truncation test passed for the wrong reason -- its payload_len of 4 pushed
getRawLength() to 258 and tripped the size check, masking the hole -- so it is
split into the >0xFF truncation case and the counted-length-fits case, with the
254/2-byte asymmetry asserted explicitly so it cannot be masked again.

274/274 native tests; both observer envs and an nRF52 repeater build clean.
2026-08-04 14:06:32 -07:00
agessaman dbdf2d732f fix(mqtt): scan every slot for a stale JWT after a clock correction
The post-NTP-correction refresh looped to _max_active_slots, which is a count of
activation positions and never an index bound. The indices holding those
positions are not contiguous: a slot passed over by isSlotReady() -- or, since
the demand-driven work, by a failed setup -- leaves a higher index activated. On
a two-position board that meant slots 1 and 2 could be live while only indices 0
and 1 were scanned, so slot 3 kept a JWT issued against the pre-correction clock
until its own expiry or a reconnect regenerated it.

Now bounded by RUNTIME_MQTT_SLOTS. The existing guard already skips disabled,
non-JWT, and clientless slots, so widening the range cannot touch a slot that
was never set up.

Pre-existing (the loop predates the demand-driven work) and kept as its own
commit so it can be picked separately. Audited the other _max_active_slots uses:
all are count comparisons or log arguments, so this was the only misuse.
2026-08-04 14:04:39 -07:00
agessaman 53bae5684e fix(mqtt): measure the setup-retry interval from the failure
last_reconnect_attempt starts at zero and teardownSlot() re-zeroes it, so the
retry gate in maintainSlotConnections() reduced to "uptime >=
SLOT_SETUP_RETRY_INTERVAL". Past 60 s of uptime a failed setup was therefore
retried on the next maintenance pass rather than 60 s later -- in the same task
iteration for a live reconfigure, since reconfigure processing runs before
maintenance in the loop.

setupSlot() now stamps last_reconnect_attempt on each failure that represents a
real attempt (client allocation, and both JWT token paths), so all three callers
get the interval measured from the failure. The bounds check and the !enabled
early return are not attempts and stay unstamped. The reconnect ladder reads
this field only for slots with initial_connect_done set, which a failed setup
never sets, so reconnect timing is unaffected.

The retry path's own pre-call stamp is kept as a backstop for any future
false-returning path that does not stamp itself.
2026-08-04 14:04:08 -07:00
agessaman 6be84eb4eb fix(mqtt): let a retried setup consume the reconnect allowance
reconnect_attempted_this_cycle is computed once before the maintenance loop and
only maintainSlotConnection() was setting it. The deferred-setup retry armed the
15 s cross-slot guard via _last_slot_reconnect_ms but left the local flag false,
so a disconnected slot later in the same pass still saw "no reconnect yet" and
started a second TLS handshake. Two concurrent ~40 KB sessions is exactly the
contention the guard prevents, and it landed in the one situation where internal
heap is already known to be short -- a failed allocation is why the retry runs.

Set on success only: setupSlot() returns true only after client->connect(), so
success means a handshake was launched. A failed retry launches nothing and
continues to spend only setup_retry_this_cycle, which rate-limits the allocation
attempts without consuming the handshake allowance.
2026-08-04 14:03:21 -07:00
agessaman e242cc6fd3 fix(mqtt): address review of the demand-driven slot work
[P1] A failed setup no longer strands the slot. setupSlot() returns bool and the
startup loops count only successful activations, so a slot that fails on a client
allocation neither consumes an active-slot position (starving a later healthy
broker on capped hardware) nor sits dead forever: maintainSlotConnections()
previously skipped clientless slots and the reconnect ladder is gated on
initial_connect_done, so nothing retried it. It now retries an enabled but
unactivated slot on a 60 s timer, one per cycle, gated on the same
_slots_setup_done ordering so the NTP-deferred setup sequence is preserved.

[P2] JWT setup no longer proceeds without a usable token. Both the preset and
custom-audience paths returned after ignoring createSlotAuthToken()'s result,
then called connect() and latched initial_connect_done -- so the token-allocation
failure introduced by the previous commit produced an unauthenticated attempt
exactly when memory was exhausted. They now return false and let the retry path
handle it.

[P2] ensureSlotAuthToken() no longer clears an existing token. It cleared
unconditionally, so every renewal wiped the current token before JWTHelper ran;
a renewal that then failed left an empty password where the inline buffer used to
preserve working credentials (JWTHelper writes only on success). Only freshly
allocated buffers are initialised now.

[P2] Raw publications reuse the shared document. buildRawJSON() reached
MQTTPayloadBuilder::buildRawMessage(), which constructed its own default
JsonDocument and therefore malloc'd and freed an internal-heap variant pool per
message -- on the highest-rate topic. The document is threaded through both
builders and the bridge passes _json_scratch_doc.

[P3] The writeTo() guard validates the source fields, not just the destination.
A corrupt payload_len of MAX_PACKET_PAYLOAD + 1 still leaves getRawLength()
inside MAX_TRANS_UNIT, so writeTo() read past packet->payload. Sizing and
validation moved to a pure MQTTWireScratch header with host tests covering the
accept/reject edges, matching the MQTTPacketFilter/MQTTConnectionPolicy pattern.
Two findings fell out: MAX_PATH_SIZE one-byte hops is not encodable (the hop
count is 6 bits, so 64 & 63 == 0; 32 two-byte hops is the widest real path), and
a zero-payload packet serializes but does not survive readFrom() -- pinned as a
test because it constrains any future wire-only queue.

[P3] Corrected the pool-size comment: these targets are 32-bit, so
ARDUINOJSON_SLOT_ID_SIZE is 2 and a pool block is 128 slots / 1024 bytes, not
4096. The 4096 figure came from a pre-existing comment near
NEIGHBORS_DOC_POOL_BUDGET, which is left alone -- its byte measurements are
empirical and still stand, only the block-size attribution is wrong.

Activation is now centralized in activatedSlotCount()/canActivateSlot(), used by
both startup loops, the retry path, and applySlotPreset(). That closes the
pre-existing divergence where a live preset change called setupSlot() without
consulting _max_active_slots, letting a non-PSRAM board reach three concurrent
TLS sessions against a cap of two. BEHAVIOUR CHANGE: a reconfigure that would
exceed the cap now logs and leaves the slot inactive instead of connecting.
Reconfiguring an already-active slot still works, because teardownSlot() releases
its position first.

272/272 native tests pass (5 new); both observer envs and an nRF52 repeater build
clean. Flash 1593249 B non-PSRAM, 1555625 B PSRAM.
2026-08-04 10:13:04 -07:00
agessaman b93b6b1560 perf(mqtt): keep slot JWT tokens in PSRAM where available
ensureSlotAuthToken()/releaseSlotAuthToken() now use the bridge's existing
psram_malloc/psram_free rather than malloc/free, so a JWT slot's 768-byte token
comes out of PSRAM instead of internal DRAM. On PSRAM boards this is the larger
half of the auth_token work: the previous commit reclaims 768 B per slot that
never creates a token, while this reclaims 768 B per slot that does -- up to
3840 B of internal DRAM on a fully configured five-slot board, which is where
the mbedTLS working set is competing for space.

No effect on non-PSRAM boards: psram_malloc falls back to internal DRAM, so the
buffer stays exactly where the previous commit left it. Same for a
BOARD_HAS_PSRAM board whose PSRAM fails to initialise.

Safe to move because every access is a CPU copy on the bridge task, never DMA,
an ISR, or a cache-disabled window: JWTHelper memcpy's the token into this
buffer, and esp-mqtt copies it out of _mqtt_cfg into its own internal-DRAM
storage when connect() applies the config. This is unlike the PSRAM-backed MQTT
task stack that was tried and reverted for resetting Heltec V4 boards, where
the fault was PSRAM execution context rather than a plain buffer read.

Split from the previous commit so it can be reverted alone if hardware soak
shows any PSRAM-related instability on the JWT path.

flash  non-PSRAM 1592801 -> 1592865 B (+64)
           PSRAM 1555213 -> 1555237 B (+24)
267/267 native tests pass; both observer envs and an nRF52 repeater build clean.
2026-08-04 09:47:06 -07:00
agessaman 9141ad854f perf(mqtt): allocate slot JWT token buffers on demand
MQTTSlot carried an inline char auth_token[768] -- 768 of its 1192 bytes --
for every runtime slot, whether the slot was JWT, username/password, disabled,
or above the active-slot cap. Now a char* allocated by ensureSlotAuthToken()
from createSlotAuthToken(), the sole writer, which runs only once a slot is
confirmed to have a JWT audience.

Because _slots[] is a fixed member array, this shrinks the bridge object
unconditionally at construction and re-adds only what is used:

  MQTTSlot     1192 ->  428 B  (-764)
  MQTTBridge  non-PSRAM 12032 -> 9740 B  (-2292)
                  PSRAM 10080 -> 5496 B  (-4584)

The object is allocated as one contiguous block at boot, so a smaller boot-time
chunk is easier to satisfy and leaves a larger contiguous remainder; JWT slots
then take 768 B each as separate blocks. Net steady-state saving is 768 B per
slot that never creates a token -- 768 B on a fully configured board, up to
3840 B on a PSRAM board with one JWT slot. 22 of the 29 presets are
MQTT_AUTH_JWT, so this is mostly reclaiming unconfigured slots rather than
non-JWT ones.

Lifetime rules, which are the whole risk here:
- setCredentials() stores this pointer in _mqtt_cfg rather than copying, and
  esp-mqtt re-reads it whenever a later connect() re-applies a dirtied config.
  So the buffer is freed only alongside the client, in destroySlotClients(),
  after the delete. MQTTSlot::broker_uri already carries an "avoids dangling
  pointer" comment from this same hazard with setServer().
- teardownSlot() still clears the token to an empty string but keeps the
  buffer: the client survives teardown and does not clear cfg->password (only
  setupSlot()'s reconfigure branch does). Freeing there would dangle.
- Never freed per reconnect, preserving the churn-avoidance the inline buffer
  was there for.

Allocation goes through MQTTRuntimeBufferLifecycle (already host-tested) using
plain malloc, so the buffer stays in internal DRAM exactly where it was when
inline. Failure propagates through createSlotAuthToken()'s existing bool.

The two call sites that create a token and then test it (preset JWT in
setupSlot, and the custom-slot audience path) now null-check first; the five
other readers are all inside if (createSlotAuthToken(...)) success branches and
need no change. destroySlotClients() releases the token unconditionally rather
than after its client null-check, so a token can never outlive its slot.

flash  non-PSRAM 1592697 -> 1592801 B (+104)
           PSRAM 1555109 -> 1555213 B (+104)
267/267 native tests pass; both observer envs build clean.
2026-08-04 09:45:30 -07:00
agessaman c833da10e9 perf(mqtt): allocate slot clients on demand
initSlotClients() created a PsychicMqttClient for every one of
RUNTIME_MQTT_SLOTS at begin(), without consulting slot.enabled or the active
cap -- even though presets are applied and _max_active_slots is computed
earlier in the same function. RUNTIME_MQTT_SLOTS is deliberately cap+1
(6/5 with PSRAM, 3/2 without), so at least one client was always unusable,
and a lightly-configured board wasted several.

Replaced with ensureSlotClient(index), called from setupSlot() -- i.e. only
for a slot that is enabled, inside the cap, and ready to connect. A client
that never reaches setupSlot() is completely inert: the reconnect ladder is
gated on initial_connect_done, which only setupSlot() sets. Retained for the
bridge lifetime as before, so reconfigure/reconnect still reuse one mbedTLS
context; that context is built by connect(), not by the constructor, so
deferring costs nothing but the 1284-byte object.

Internal DRAM saved, by configured slot count:
  non-PSRAM (3 runtime / 2 cap)   1 configured  2568 B
                                 2-3           1284 B
  PSRAM     (6 runtime / 5 cap)   1 configured  6420 B
                                 2             5136 B
                                 5-6           1284 B
A BOARD_HAS_PSRAM board whose PSRAM fails to init gets cap 2 against 6
runtime slots, so it saves at least 5136 B -- on the board that just lost its
PSRAM.

Two gates used "client != nullptr" as a proxy for "configured". Under eager
allocation that conjunct was always true and therefore harmless; with lazy
allocation it would have made shouldQueuePacketType() drop every packet
received before the post-NTP-sync slot setup, which is precisely the window
the queue exists to cover. Both now key on slot.enabled, matching the
documented intent above eligiblePacketSlots() that a configured-but-
disconnected broker is still a target.

formatSlotDiagReply() gains the !isSlotReady() -> "wait" branch that
get mqtt.status and getSlotStatusSnapshot() already have; that state
previously fell through to "disc" and read as a network fault when it is
really a missing token/IATA/credential. "no client" now means only what its
name says: the slot is ready but the client could not be allocated. That
state is newly reachable, so the allocation uses new (std::nothrow) -- this
framework enables C++ exceptions, and a throwing new on exhaustion would
panic the node instead of degrading one slot.

flash  non-PSRAM 1592573 -> 1592697 B (+124)
           PSRAM 1555001 -> 1555109 B (+108)
267/267 native tests pass; both observer envs build clean.
2026-08-04 09:16:47 -07:00
agessaman c28548aecf perf(mqtt): tranche 1 memory reductions in the bridge
Five independent, behavior-preserving reductions. Measured on
Heltec_v3_repeater_observer_mqtt (non-PSRAM) and
ThinkNode_M7_repeater_observer_mqtt (PSRAM), 267/267 native tests green.

Drop the unused static-task bookkeeping. StaticTask_t _mqtt_task_tcb (344 B)
and StackType_t* _mqtt_task_stack were never used: there is no
xTaskCreateStatic call, the pointer was assigned nullptr immediately before
xTaskCreatePinnedToCore, and the two psram_free() calls on it were dead.

Size the wire-format scratch buffers from the protocol maximum. raw_hex[1024]
at three sites becomes 2*MAX_TRANS_UNIT+1, and raw_buf/reconstructed[512]
become MAX_TRANS_UNIT. Both writeTo() sites now check getRawLength() first:
writeTo() does not bounds-check and returns uint8_t, so the old 512-byte
buffers were the only thing absorbing a malformed payload_len, and the
post-hoc "raw_len > sizeof(buf)" test ran after the overrun.

Pass the already-known serialized length into publishToSlot() instead of
re-running strlen() per destination slot (up to 2 KB per packet per slot, and
NEIGHBORS_JSON_BUFFER_SIZE per neighbor snapshot). Same for the direct
publish in publishStatusToSlot().

Share one JSON buffer and one document across packet, raw, and status. All
publish paths serialize on the bridge task, so the separate status buffer and
document were never concurrent. Status keeps STATUS_JSON_BUFFER_SIZE as its
serialization ceiling, so which oversized status documents get dropped is
unchanged.

Route the document's pools through a PSRAM-preferring allocator. Under
ArduinoJson 7 StaticJsonDocument<N> is a deprecated empty subclass of
JsonDocument whose template argument only feeds capacity(); the object is
64 B and each pool block (4096 B here) came from plain malloc(), i.e. the
internal DRAM the mbedTLS working set needs. Mirrors NeighborsDocAllocator.
The old comment claiming an inline pool has been corrected.

Measured:
  sizeof(MQTTBridge)  non-PSRAM 13208 -> 12032 B  (-1176, internal heap)
                          PSRAM 10492 -> 10080 B  (-412, plus one fewer
                                                   768 B PSRAM allocation)
  stack, non-PSRAM    buildPacketJSON[FromRaw] 1264 -> 736 B
                      buildRawJSON             1136 -> 608 B
                      publishPacket             688 -> ~432 B
                      packetToHex               560 -> ~304 B
                      deepest publish chain    ~2.6 -> ~1.8 KB of 8 KB
  flash               1592513 -> 1592573 B (+60)
  static RAM          74656 B unchanged -- MQTTBridge is heap-allocated, so
                      these savings are internal heap, not the linker figure

Deferred from the review: the QueuedPacket wire-only redesign (reward is
1.56 KB on non-PSRAM only, and Packet::readFrom() rejects payload_len == 0),
demand-driven slot clients/JWT tokens, and pool retention across publishes --
JsonDocument::to<T>() always calls clear(), which destroys pools, so
"retain pools by clearing the root object" needs a string-pool lifetime
analysis first.
2026-08-03 22:15:12 -07:00
agessaman 45379ad742 fix(build): restore nRF52 builds broken by observer-only dependencies
Three fork-introduced dependencies had leaked into non-ESP32 builds, breaking
every nRF52 and RP2040 target since 7e4f75c9 (2026-04-10):

- The `memory` CLI command called ESP.getFreeHeap()/heap_caps_* unguarded from
  shared CommonCLI.cpp. ESP32 output is unchanged; other platforms now report
  newlib arena stats, omitting min-ever-free and largest-free-block rather than
  substituting numbers that mean something different.
- The vendored PsychicMqttClient (ESP-IDF esp-mqtt) was pulled in by the LDF on
  nRF52; nrf52_base now lib_ignores it, matching how RP2040 variants ignore BLE.
- JWTHelper.cpp and MQTTMessageBuilder.cpp are excluded in arduino_base, but 22
  variants re-glob helpers/*.cpp after inheriting it, which undoes the exclusion.
  Guarding the file contents on WITH_MQTT_BRIDGE is robust against any variant's
  filter, and matches helpers/esp32/WebConfigServer.cpp.

Verified: RAK4631 repeater + room server, Heltec T114, T1000-E, Wio WM1110,
Xiao nRF52 and ThinkNode M1 all build. ESP32 observer builds are byte-identical
to before (RAM and flash), and the native suite stays at 267/267.

RP2040 still fails separately: four boards declare the pre-force_ap
startOTAUpdate signature. Not addressed here.
2026-08-03 11:27:36 -07:00
agessaman 126a2564ed Merge remote-tracking branch 'upstream/dev' into observer-firmware-dev 2026-08-03 11:02:13 -07:00
agessaman a3a0a94dc4 feat(mqtt-neighbors): support non-PSRAM observer boards
Neighbors publication was gated on BOARD_HAS_PSRAM. Removing that gate alone
was not enough: the feature built but was inert without PSRAM, because three
allocation sites asked for MALLOC_CAP_SPIRAM (which returns null with no
PSRAM), the bridge's persistent buffer was allocated inside a BOARD_HAS_PSRAM
block, and neighborDiscoverReady() rejected every pass at runtime on
psramFound(). The entry table also did not fit: finishNeighborDiscover put
pubkey_hex[50][65] plus entries[50] on the stack, a 4752-byte frame against
the mesh loop task's 8 KB.

- Gate on MAX_NEIGHBOURS plus PSRAM or an explicit per-variant
  MQTT_NEIGHBORS_WITHOUT_PSRAM opt-in.
- Move the entry table and its hex strings into one heap block sized to the
  pass; the frame drops from 4752 to 304 bytes.
- Prefer PSRAM and fall back to internal DRAM in the mesh-side allocations
  and the ArduinoJson pool; hoist the bridge's persistent buffer out of the
  BOARD_HAS_PSRAM block (psram_malloc already falls back).
- Keep the runtime psramFound() check only where the buffers are sized for
  PSRAM, so a board whose PSRAM failed to init still refuses.
- Size for internal DRAM without PSRAM: 4 KB text buffer and 20 entries per
  publish, keeping the pool to a single block and the peak near 13 KB rather
  than ~35 KB. Oversized tables truncate and report total_neighbors as before.

Enabled on the ESP32-S3 observer envs (Heltec V3/WSL3, RAK3112, Heltec
Tracker v1.1/v2). Left off for the classic ESP32 T-LoRa V2.1-1.6, which is
already limited to one active TLS slot.

Costs ~7.4 KB static DRAM on repeaters and ~9.6 KB on room servers. The
prefs layout is unchanged, so this is neutral for existing devices.
2026-08-03 10:29:54 -07:00
agessaman 34037f202d fix(mqtt-neighbors): budget the JSON pool separately from the publish buffer
ArduinoJson v7 hands out document-pool blocks in fixed 4096-byte chunks, so
the pool is not bounded by the size of the text it serialises to. Budgeting
it at NEIGHBORS_JSON_BUFFER_SIZE starved it once the table grew: a 50-entry
table needs 12541 B of pool against the 10240 B cap, and a starved allocator
sets doc.overflowed(), which makes buildNeighborsMessage return 0 and drop
the entire publish rather than truncating the tail.

Repeaters with roughly 40 or more neighbours therefore published no neighbors
message at all, silently, while smaller tables published normally.

Give the pool its own NEIGHBORS_DOC_POOL_BUDGET and add an explicit
NEIGHBORS_MAX_PUBLISH_ENTRIES cap alongside the existing text-size check.
2026-08-03 10:29:32 -07:00
Huw Duddy 9d902e634a Merge pull request #3076 from MDamon/fix/lr1110-rx-timeout
LR1110: fix startReceive() passing an IRQ bit as the RX timeout 🤖🤖
2026-08-03 23:02:52 +10:00
Huw Duddy 5d940a1dc9 Clean up comments in startReceive method
Removed unnecessary comments regarding RX timeout and IRQ mask.
2026-08-03 22:57:34 +10:00
ripplebiz 626a82fd30 Merge pull request #2824 from NickDunklee/rak-advert-hw-encryption
feat: use nrf52 hardware crypto where we can
2026-08-02 15:35:33 +10:00
agessaman 5828a3c563 fix(mqtt-neighbors): report unusable heard ages as null
A neighbour heard before the clock is set carries the firmware's unset-clock
default (1715770351, 15 May 2024). Subtracting that from an NTP-synced clock
published ages of ~806 days for neighbours that had just answered a live scope
query, and a backwards clock step reported 0, i.e. "heard just now".

- finishNeighborDiscover() reports the age as unknown ("heard_secs_ago": null)
  when the stored stamp and the current clock come from different epochs, or
  when the clock has stepped backwards
- handleNeighborDiscoverResponse() re-stamps heard_timestamp on a zero-hop scope
  reply, in both the snapshot and the live table, so entries heal once per
  discovery cycle instead of waiting for the neighbour's next advert
- publish ordering places usable ages ahead of unknown ones so a poisoned stamp
  cannot displace fresh entries when the JSON buffer truncates
- document the null case, and the always-present total_neighbors /
  queried_neighbors / truncated fields the payload sample omitted
- add UPSTREAM_BUGS.md, tracking the monotonic-uptime fix to propose upstream
  plus the unclamped subtraction in the companion and CLI readouts

Containment only: upstream still stamps neighbours from the wall clock at
packet-reception time, which on a cold boot always precedes NTP.
2026-07-31 08:54:39 -07:00
agessaman 612c52132c merge: upstream/dev into observer-firmware-dev (2026-07-30, db232808)
Absorbs 106 upstream commits. Seven files conflicted; the substantive one
was upstream's new JSON ConfigSerializer (PR #2982), which replaces the
binary /com_prefs layout with /prefs.json and makes NodePrefs a
ConfigSerializer subclass.

Prefs migration
- Adopt upstream's ConfigSerializer. writeCommonPrefsImage() and its
  documented offsets (0-294) are deleted, along with the now-unreachable
  saveCommonPrefsImageAtomically()/CommonPrefsFileStore atomic rename path.
- Load order is /prefs.json, then /com_prefs, then /node_prefs. Upstream
  dropped the /node_prefs fallback; it is restored here so devices that
  never advanced past that filename keep their config.
- Legacy files are never removed, so migration cannot destroy its own
  source and a deferred or failed save simply retries next boot.
- /com_prefs is treated as a format migration only, not an "upgrade", so
  it does not trip the bridge.source tx->rx flip on existing nodes.
- The MQTTPrefsAtomicStore legacy gate is retained: the observer tail
  recovered from an old-format file still commits to /mqtt_prefs before
  /prefs.json is written.
- MQTTPrefs and /mqtt_prefs are untouched; savePrefs keeps its save_mqtt
  parameter and now returns upstream's bool.

Fixes to upstream code
- RadioPrefs::structure() bound both "rxgain" and "fem_rxgain" to
  rx_boosted_gain, so radio_fem_rxgain was never persisted. Bound to the
  correct field.
- discovery_mod_timestamp was dropped from structure(); it gates
  'since'-filtered DISCOVER replies and is set on every config change, so
  losing it would silently stop discovery responses after a reboot. Added
  as "disc_mod".

Merge artifacts repaired
- Restored bblanchon/ArduinoJson to the native test env; a clean but wrong
  auto-merge at the lib_deps block boundary dropped it and broke all 19
  host test suites.
- Migrated the fork's WebConfig UITask screens off the removed
  DisplayDriver::Color enum to upstream's UIColor element types.
- Removed duplicate getCADEnabled() definitions in companion MyMesh.cpp
  and simple_sensor SensorMesh.cpp that both sides had added.
- Dropped memset(&_prefs, 0, ...) in the four example meshes; NodePrefs now
  has a vtable. guard gains an initializer that memset used to provide.

Other resolutions
- simple_room_server keeps both the fork's discover.* commands and
  upstream's new room.post.
- docs/payloads.md taken from upstream, undoing content earlier merges had
  reverted (Control data section, split login tables).

Verified: 273/273 host tests pass across native and native_kiss_modem;
Heltec v3 repeater, repeater_observer_mqtt, room_server_observer_mqtt,
sensor, and companion_radio_ble all build clean.
2026-07-30 14:56:23 -07:00
Mike Damon 78723d2565 LR1110: fix startReceive() passing an IRQ bit as the RX timeout
CustomLR1110::startReceive() passed RADIOLIB_LR11X0_IRQ_PREAMBLE_DETECTED
(1<<4 = 16) as RadioLib's first argument, which is the RX *timeout*, not an
IRQ mask. At the LR11x0's 30.52us tick that armed the receiver for ~488us, so
it dropped out of RX before any packet could arrive and the node received
nothing at all -- while transmitting normally.

Symptoms on a SenseCAP T1000-E: tx_air_secs rising, rx_air_secs stuck at 0,
recv_errors 0, and the noise floor pinned at the -120 clamp because
getCurrentRSSI() never sampled a live receiver.

Pass RADIOLIB_LR11X0_RX_TIMEOUT_INF (continuous RX), which is what
LR11x0::startReceive() itself uses, keeping the PREAMBLE_DETECTED flag in the
reported IRQ flags as intended.

Introduced in ea5d7c8b ("LR1110: add PREAMBLE_DETECTED to reported irq flags").
Verified on two T1000-E units: with only the repeater fixed it began receiving
(last_rssi -29, SNR 17.0) while the unfixed companion stayed deaf; fixing both
brought up the link in each direction.
2026-07-30 12:14:39 -04:00
ripplebiz 57ddada3ba Merge pull request #3004 from axhoff/agent/preserve-utf8-advert-names
Preserve UTF-8 boundaries in advert names
2026-07-30 12:59:59 +10:00
agessaman 616fe96d1e feat(mqtt): publish all message types except raw to MeshRank
MeshRank slots previously took packets only. The broker tolerates status and
neighbors (and its maintainer intends to look at using the neighbors data), so
those now publish under meshrank/uplink/{token}/{device}/{type}, using the same
type suffixes as the MeshCore layout. Raw stays excluded: it is the
highest-volume topic and the broker does not consume it, so `set mqtt.raw on`
has no effect on a MeshRank slot.

All MeshRank gating funnels through the topic router, so relaxing that single
guard was sufficient - publishStatusToSlot, publishStatus, publishRaw,
publishNeighbors, and eligiblePacketSlots already skip slots that cannot form a
topic. A per-slot token is still required. Because eligiblePacketSlots resolves
topic support before serialising, raw JSON is never built for a MeshRank slot
rather than built and discarded.

observer-firmware sends raw to MeshRank; this supersedes that decision, so raw
must stay excluded when the branches merge. MeshRankTakesEveryTypeExceptRaw
fails if a merge re-enables it.
2026-07-29 15:51:28 -07:00
agessaman 8b16af3830 feat(mqtt): add default_scope to neighbor discovery JSON handling
Enhance the neighbor discovery JSON structure by introducing a
default_scope field, which indicates the region name this node
floods to by default. This change improves clarity in the
neighbor discovery process and aligns with the unscoped flood
behavior when no default region is set. Updates include
modifications to the MyMesh class and related message building
functions to accommodate the new field.
2026-07-29 15:00:32 -07:00
ripplebiz 6e39cff8a1 Merge pull request #3063 from oltaco/unify-irq-timeout
Unify IRQ timeout logic across all applicable radio types
2026-07-29 23:52:48 +10:00
ripplebiz 279ac78122 Merge pull request #3049 from liamcottle/feature/ethernet
Refactor Companion Interfaces + Add ThinkNode M7 Ethernet Support
2026-07-29 23:45:51 +10:00
taco 79eb7f5d8d LLCC68: add IRQ timeout logic 2026-07-29 23:34:01 +10:00
taco a3732e1c65 SX1268: add IRQ timeout logic 2026-07-29 23:34:01 +10:00
taco a5c323cc19 STM32WLx: add IRQ timeout logic 2026-07-29 23:33:51 +10:00
taco a7426de0bf LR1110: fix unnecessary IRQ clear 2026-07-29 23:18:15 +10:00