Build counters are per channel, so a cross-channel build number is not
comparable: a switch to a channel with a lower counter reported up to date
and never happened. Off the native channel, update unless the hash matches.
A slot waiting on a token/IATA/credential stays enabled but never attempts a
connection, so it held mqtt_slots_up below mqtt_slots_total (and down= at 1)
permanently -- a standing false alert on a node that is merely unconfigured.
slots_total and down= now count only ready slots, matching the CLI's "wait".
mqtt_connect_failures summed only enabled slots, so disabling a failing slot
made it drop. It now sums every slot, monotonic within a boot.
cf: moves after the error tail in mqttN.diag so a clamped 160-byte reply
loses the count rather than the refusal/TLS detail. A failed start now clears
the attempt flag instead of restoring a possibly stale one. Outage comments
corrected: the clock starts at a slot's first DISCONNECTED, not first connect.
reconnect() on a started client returns ESP_FAIL while an attempt is already
in progress. On a Heltec V3, a live reconfigure hit exactly that and the
attempt then connected, yet cf read 1. Count start_failures only when
connect() on a stopped client fails, and restore the attempt flag to its
prior value on any failed call so the in-progress attempt is still judged by
its own outcome.
connect()/reconnect() returning an error started no attempt, so no event
arrived and the failure went uncounted -- the low-heap case the counter exists
for. Counted now in a bridge-task-only start_failures, summed into cf: and
mqtt_connect_failures.
A renewal or corrected-clock bounce whose softDisconnect() times out gets its
old session's DISCONNECTED after the new attempt raised the flag. That session
is still marked connected until the handler runs, so the gate now also
requires !connected. An ignored late CONNECTED clears the flag too.
cf: no longer depends on dc being non-zero. Comments corrected for the
PSRAM-fallback stack cost and heap_largest's sampling point; the worst-case
test now models 6 slots.
The Starting-state gate raced the event task: client_state becomes Starting
only after connect()/reconnect() returns, so a fast failure could deliver
DISCONNECTED first and go uncounted, while a deliberate stop of an attempt in
flight was counted as a failure. A per-slot flag is now raised before the call
and cleared by onConnect, a failed start, or stopSlotClient().
Also stops zero-valued connect_failures/slots_breaker from creating an empty
stats object.
Two gaps remained after the first connection-health commit.
A slot retrying into a heap that cannot satisfy its TLS allocation moved no
counter at all. disconnect_count rose, but it also rises for dropped live
sessions, so it could not separate "actively failing to connect" from "was
connected, dropped once". Adds a per-slot connect_failures, incremented in
onDisconnect when the event arrives while the client is Starting -- meaning
it ended an attempt that never reached onConnect. Late events from a stopped
or quarantined client, and soft-disconnects of a live session, are in other
states and are not miscounted.
A slot parked by the circuit breaker also looked identical to one still
retrying, despite being materially worse: the breaker only probes every 30
minutes, so recovery is far slower and an operator may need to intervene.
Surfaces both:
- `mqttN.diag` gains `cf:<n>` beside the existing `dc:<n>`.
- the status payload gains `mqtt_connect_failures` (summed across slots) and
`mqtt_slots_breaker` (count of tripped slots), each emitted only when
non-zero so a healthy board's payload does not grow.
The counter is what distinguishes a retrying slot from a quiet one: while a
slot is down its publish counters cannot move by definition, so this is the
only value that shows the device is still trying.
Both suggested by adversarial review.
The worst-case test added with the connection-health fields did not model
production maxima -- it used a 28-byte model and a 24-byte firmware string,
while MQTTBridge permits _origin[32], _device_id[65], _board_model[64],
_firmware_version[64] and derives a 63-byte client version. isValidName()
rejects [ ] / \ : , ? * but permits '"', so a 31-character node name of
quotes is legal and every quote escapes to two bytes.
Measured at those real maxima:
without connection-health fields 762 bytes
with connection-health fields 851 bytes
buffer 768 bytes
So the 768-byte buffer already had only six bytes of headroom before this
work, and the four new fields overflow it. Overflow is silent:
serializeComplete() returns 0 and clears the buffer, and the bridge
publishes only a positive length, so a device would simply stop publishing
status with no error anywhere -- the same blindness the connection-health
fields exist to remove.
Raises STATUS_JSON_BUFFER_SIZE to 1024 and corrects the test to use the
actual maxima. Status serializes into the shared 2048-byte publish scratch
buffer, so this costs no extra RAM; only the PSRAM-allocation-failed stack
fallback grows by 256 bytes. A static_assert keeps the status ceiling within
the shared buffer.
Found by adversarial review of the preceding commit.
A slot that cannot establish TLS reported nothing. The per-slot counters
track publish successes and failures, and a slot that never connects never
attempts a publish -- so its ok count freezes and its error count stays at
zero for the entire outage. Observed on hardware: a slot sat dead for over
five hours while `sN=` showed 0 errors and the published status payload
carried no indication at all.
Two surfaces gain the missing signal:
- `get mqtt.stats` gains `down=<n>` (omitted when zero, like `filt=`) and
marks each disconnected slot with a trailing `!`. The marker goes after
the counts so existing "s(\d)=(\d+)/(\d+)" parsers keep matching.
- The published status payload gains `mqtt_slots_up`, `mqtt_slots_total`,
`heap_largest`, and `mqtt_outage_secs`. These ride on every healthy slot's
status message, because the slot that is down cannot report itself.
`mqtt_slots_up < mqtt_slots_total` is the condition to alert on.
`mqtt_outage_secs` is detail: it is present only when the outage has a known
start, so its absence does not imply health -- a slot that has never
connected since boot has no outage start time.
`heap_largest` is included because free heap alone does not predict a failed
connection. A slot needs one contiguous 16,384-byte block for its TLS record
buffers; measured immediately before an attempt this value predicted the
outcome in every observed case, while total free heap was 85 KB throughout a
multi-hour outage. A low value with every slot up is normal rather than a
fault -- established sessions already hold their buffers.
Publish-error semantics are unchanged: connect failures are not folded into
them.
Adds host coverage for the new fields, for the outage key's presence rule,
and for worst-case payload size. That last one matters because
serializeComplete() returns 0 and clears the buffer on overflow and the
bridge publishes only a positive length, so exceeding the 768-byte status
buffer would silently stop status publication -- the same blindness this
change exists to remove.
The reconstruction-less fallback path in buildPacketJSON (queued packet
with no captured raw frame that cannot be re-serialized) published
fabricated SNR 12.5 / RSSI -65 that consumers could not distinguish
from real radio readings. It now passes NAN / MQTT_RSSI_UNKNOWN, and
buildPacketMessage omits the SNR/RSSI keys entirely when no measurement
exists, mirroring the existing score omission.
Ports the fix from ACETyr's PR #13 (0a037040) on mqtt-bridge-implementation.
Co-Authored-By: ACETyr <35745290+ACETyr@users.noreply.github.com>
Codex review of the previous commit found two real problems.
The routes go live in startLanMode()/startSetupMode(), one tick before
probeBoardCommands() first runs, so a status request can land in between.
That made _board_cmds a genuine cross-task race despite the comment
claiming otherwise, and — worse — let /api/status answer "no board
commands" before the board had been asked. The page reads status once,
so a browser that loaded in that window hid working FEM controls for the
whole session.
The probe result is now published and read under _mux, and /api/status
OMITS board_cmds until the probe has run, which is not the same as
reporting an empty list. The page treats the absent field as "not known
yet" and re-polls with backoff; the audit waits for it the same way,
rather than auditing a board it only half knows.
Probing from the start path instead would close the window, but
Callbacks::execCommand is documented as loop-task-only and `start
webconfig` is itself mid-handleCommand, so that trades a race for
re-entrancy.
The audit never executed fan.lo or fan.hi, which have no getter and so no
round-trip: they were only ever matched as strings. They now run for real,
including the four rejection paths, and the mock grew --probe-delay so the
retry path is exercised rather than assumed.
The FEM commands moved out of CommonCLI into Board::handleCommand(), and
the T-Beam 1W fan control was added there too, but the portal never caught
up: `radio.fem.txgain` and the fan commands were missing from the terminal
table entirely, while `radio.fem.rxgain` was offered on every board even
though a board with no hook now answers "??:" rather than "unsupported".
Whether a node answers these is a property of the board, not of the build,
so the page cannot know from the firmware version. Ask the board instead:
probeBoardCommands() runs each candidate getter once on the loop task at
startup and keeps the ones that answer, which needs no per-variant list
because Board::handleCommand() already reports whether it handled a
command. /api/status names the survivors and the page hides everything
else, so adding a command to a variant means one entry in WC_BOARD_CMDS
rather than an edit per board.
The two FEM keys also become Radio-panel toggles, gated the same way;
their `set` reaches the board hook through the existing config batch, so
only the allowlist and /api/config needed to grow.
webconfig_cli_audit.py now scans variants/*/*Board.cpp for the boards that
actually build the portal, checks set-only keys in the reverse direction,
and verifies every gate is a command some board answers; the mock gained
--board-cmds so both shapes of board are testable. `stop ota` joins
NOT_OFFERED: `start ota` cannot run from the portal, so it has nothing to
stop.
Adds a built-in preset for the Polish MeshCore community broker at
mqtt.marwoj.net: MQTT over TLS on port 8883, username/password auth
with credentials embedded in firmware, and the standard MeshCore topic
layout. The Let's Encrypt chain anchors at the existing ISRG_ROOT_X1
constant, so no new CA certificate is needed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The preset table is fleet state, not a build detail. A slot's preset is stored
in /mqtt.json by name, and firmware that does not know a name does not merely
ignore it: MQTTPrefsSerializer repairs it to "none" and CommonCLI writes the
repaired file back to flash. A node that rolls back to a channel missing a
preset therefore loses that slot permanently, and re-upgrading does not bring
it back.
The parity gate compared preset names only, deliberately allowing URL, CA and
credential drift. It now requires src/helpers/MQTTPresets.h to be byte-identical
between the channels, which also catches that drift — the two channels are meant
to dial the same brokers.
That is only workable if the file holds no channel-specific code, so
mqttPresetEnforcesTokenExp() moves to the new MQTTPresetPolicy.h. It was the
sole difference between the two channels' copies, and with it moved they match
exactly today. Policy keyed off the table belongs there from now on; the table
itself stays pure data.
The older name-only comparison stays available without --exact for ad-hoc use,
and the checker's self-test now covers both modes, including that --exact
rejects a config-only change the name check waves through.
The hostname was built once in begin() from the node name, so `set name` left
the node advertising its old name to DHCP until the next reflash. It is now
rebuilt on rename through a new CommonCLICallbacks::onNodeNameChanged() hook,
and begin() shares the same helper.
What this cannot do is rename a live lease. IDF 4.4 states that a hostname
changed after the interface is up "would only be reflected once the interface
restarts/reconnects", and Arduino 2.x's WiFi.setHostname() does not touch a
running netif at all — it writes a static string that the netif reads at
bring-up. Forcing the issue would mean bouncing the link, which costs every
MQTT slot a reconnect; that is not a reasonable price for a rename, so the new
name lands at the next reconnect or boot instead.
Only automatic (Ethernet-preferred) links carry a hostname today, exactly as
before. Giving plain Wi-Fi observers one would change the DHCP identity of the
existing fleet and is a separate decision.
Follow-up to the DNS restore, from a second review round.
lwIP's DNS functions belong to the TCP/IP thread, and both halves were called
from elsewhere: the Ethernet snapshot from the Arduino event task, the Wi-Fi
snapshot and every restore from the MQTT task. Both now go through
esp_netif_tcpip_exec(), which is also what serializes the snapshot itself, so
it no longer needs atomics. Neither caller is the TCP/IP thread, so the call
cannot deadlock on itself.
The snapshot keeps the whole ip_addr_t instead of an IPv4 word. These builds
compile lwIP with IPv6 enabled, so reinterpreting a v6 server as v4 would have
installed four meaningless bytes as a resolver. RDNSS is disabled here, so a v6
server should not arise today; storing the tagged value means it stays correct
if that changes.
Restore now writes every slot, empty ones included, so returning to a
one-server network cannot leave the other medium's second server behind as a
fallback that only fails slowly. A lease that carried no resolver at all is not
recorded, so a medium that has never had DNS leaves the current resolver alone
instead of wiping it.
Two defects found by an adversarial review of the merge.
A link transition no longer stops a slot whose connect attempt is in flight.
Stopping one means esp_mqtt_client_stop(), which waits on the SDK's API mutex
and its task's stopped event with no bound; the client task notices only when
it returns from whatever transport call it is in. During teardown the bridge's
StopUnproven timeout contains that, but during a transition nothing does, so a
routine WiFi flap could freeze the sole MQTT worker for far longer than the 2 s
this path advertises, with the bridge's own stop handshake queued behind it.
The attempt is left to resolve instead: unlike a reconfigure, a transition
changes neither endpoint nor credentials, so an attempt that completes is
credited to the broker it actually reached and drops with the old route.
Failing back to Ethernet left the node using the WiFi network's DNS server.
lwIP keeps one global server list and IDF 4.4 has no per-interface retention,
so the medium that leased last owns DNS for every socket. Each link now
remembers the resolver its own DHCP lease installed and puts it back when it is
selected again; a medium that never held a lease leaves the current resolver
alone. Where the two networks are on different subnets this was a silent
outage: the link read as connected while every broker and NTP hostname failed
to resolve, until Ethernet's own DHCP renewal happened to fix it.
#54 lands first, so the network abstraction is reworked on top of the
per-slot client ownership model instead of alongside it.
Two resolutions are semantic, not textual:
- Link-transition teardown goes through the ownership API. It called
softDisconnect() on every started client, which bypassed client_state and
ignored the typed result. A slot that is still Starting now gets a real
stop: softDisconnect() returns immediately on a client that is not yet
connected, so its attempt would otherwise complete against the old route and
deliver a CONNECTED event indistinguishable from the new one's (F04). On a
medium switch the old route can still be briefly usable, so that is not
hypothetical. A connected client keeps the cheap bounded path, and a
quarantined one is left alone.
- NetworkLink::applyPowerPrefs() adopts WifiPowerSavePolicy. Its local
`2 ? MAX : NONE` mapping would have reintroduced F11 and read the new stored
value 3 (explicit `min`) as none. It also applies the setting when the link
starts already associated, which is the case the bridge used to cover.
The NTP probe from #54 keeps its validation and its one-attempt-per-server
bound, but resolves and gates on the selected link rather than on WiFi, so it
works on an Ethernet-preferred node.
- The room server checked canFlashAfterStop() only when the bridge was
running at OTA time. After a timed-out restart the bridge reads as
stopped while its unacknowledged task may still own TLS/client state,
so an OTA could erase and write flash under it. It now refuses while
the stop is unproven, after first reaping any late ack. The repeater
already gated unconditionally.
- MyMesh::loop() now reaps a late stop acknowledgement whenever it lands,
releasing the withheld queue and buffers, and restarts only when a
resume is pending and the bridge is enabled. Before, a bridge disabled
during StopUnproven kept those resources until re-enabled or rebooted.
pollLateStopAck() is public for this.
- The wrapper's destructor no longer stops an already-stopped client:
destroySlotClients() had just stopped it, so every shutdown logged five
spurious "esp_mqtt_client_stop failed: ESP_FAIL" errors.
Hardware (Heltec V4, 1 s test stop deadline): restart -> StopUnproven ->
`set bridge.enabled off`; the late ack was reaped ("releasing withheld
resources"), status read "not running", and `set bridge.enabled on`
started cleanly without a second release.
setBridgeState(false) returned early when the bridge was already down,
so `set bridge.enabled off` after a timed-out restart left the resume
pending and MQTT restarted against the operator's choice once the task
acknowledged. Disabling now clears the pending resume before that early
return, and the resume also requires bridge_enabled. The StopUnproven
log and `get mqtt.status` no longer say only "reboot to recover", since
a late ack now recovers on its own.
Hardware (Heltec V4, 5 live slots, 1 s test stop deadline): a restart
went StopUnproven, the task acknowledged 27 s later and the bridge
resumed with all slots reconnecting; with `set bridge.enabled off` in
between it stayed stopped until `set bridge.enabled on`.
The renewal bounce called reconnect()/connect() directly, so a successful
bounce never advanced applied_token_expires_at. The renewal decision
reads that value, so a renewed slot stayed due and, on a broker that
enforces exp, minted and reconnected every minute for the rest of the
token's life. Both the renewal bounce and the corrected-clock bounce now
go through reconnectSlotClient(), which starts a stopped client, refuses a
quarantined one, moves the slot to Starting and records the token in use.
The corrected-clock no-bounce case settles the renewal as the renewal
path already does.
- A stop that timed out into StopUnproven refused begin(), and only a
later begin() honoured the task's late acknowledgement. Nothing called
it, so restartBridge() (any `set mqtt...` restart) or an aborted OTA left
the observer offline for the rest of the boot. MyMesh now records that a
start was refused and restarts the bridge once stopAcknowledgedLate()
reports the ack. The OTA-abort alert no longer claims "bridge resumed"
when the restart was refused.
- refreshNTP() now runs the validated probe with one attempt per server.
With two attempts and a 1 s pause it blocked the MQTT task ~18 s every
hour on networks that drop UDP/123, where the old async SNTP cost
nothing. Loop comment and docs updated to match.
Stored 0 was the shipped default from 2026-01-02 to 2026-03-28, and every
association path has run it with power save off since. Reading it as
MIN_MODEM, as F11 did, would have put every node set up in that window
into modem sleep on its next association. 0 now reads as `none`, and
`set wifi.powersave min` stores a new value 3. /mqtt.json accepts 0..3;
older firmware repairs 3 to `none` on load. Binary snapshots are no longer
written, so 3 never reaches a legacy layout.
startLanMode() took the caller's IP, read before the route lock. A
Wi-Fi/Ethernet switch in between left WebConfig locked to the new link
while advertising the old link's address. It now reads the selected
link's address after lockSwitching(), as startOTAUpdate() already does.
The old route is already down or switched away when a transition fires,
so waiting the default 5 s per slot for a DISCONNECTED event only delays
recovery (up to 25 s across five slots). A client whose event is late is
aborted by keepalive and retried by the normal backoff loop, the same as
the token-renewal path's softDisconnect() + reconnect().
- Link down/switch edges now softDisconnect() every started slot instead
of disconnect(). The full stop could wait forever for a DISCONNECTED
event on the Core 0 MQTT task, and it destroyed and recreated every
slot's esp-mqtt task on each Wi-Fi drop, where the bridge previously
stopped none (measured: 62 s deauth, five slots, zero stops).
softDisconnect() is bounded and keeps the task; reconnectSlotClient()
then calls reconnect() on the new route. Rename the transition action
to disconnect_started_slots to match.
- Ethernet no-IP recovery and init retries rebuild the CH390 netif, so
they now run inside the route-switch mutation gate. An OTA/WebConfig
lock taken after maintain() samples the lock count can no longer have
the interface torn down underneath it.
- Guard the boot-time link bootstrap to observer ESP32 builds. It ran
unconditionally in MyMesh::begin(), breaking every non-observer
repeater/room server build (ESP32 and nRF52).
- Rename NetworkInterface -> NetworkLink (class, accessor, files).
Arduino-ESP32 3.x ships its own NetworkInterface class and header,
which broke the ESP32-C6 builds. Drop WiFi.setAutoConnect(), a no-op
on 2.x and removed in 3.x.
- Refresh stored Wi-Fi credentials every bridge tick so the STA
reconnect loop picks up `set wifi.ssid` / `set wifi.pwd` without a
reboot, as the bridge did before the link moved out of it. Skip
reconnects while the SSID is empty.
- Restore the "WiFi connected: <ip>" / "WiFi disconnected: reason N"
debug lines the bridge used to print.
- Alert on Ethernet only once it has held a lease this boot or when no
Wi-Fi is configured; Wi-Fi-only installs of an Ethernet-preferred
image keep Wi-Fi alerts instead of reporting "Ethernet down".
- Record wifi.setup_complete only for Ethernet LAN onboarding, so
Wi-Fi builds keep the SSID-based first-boot portal rule.
- Use seq_cst for the route-switch lock/mutation flag handshake.
- Docs: SNMP RSSI sentinel is -127; describe link-return vs medium-switch
reconnect behavior accurately; note runtime credential pickup.
- Test: unknown keys inside a known /mqtt.json group are ignored, which
keeps wifi.setup_complete downgrade-safe.
The major rev from 19.x to 20.x breaks Radiolab, so pin it for now. This should fix the failing wio-e5-mini_repeater builds.
ltoa is removed upstream, so it was replaced with snprintf. The p[] buffer is always 16 bytes. Copy/paste for why claude thinks this is fine below, but also if this is broken it wouldn't be new.
```
exp2 in [23, 30] (int_part = mantissa << (exp2 - 23)): this is where int_part can reach its true max, up to 2,147,483,520 — 10 digits. That's what the snprintf bound of 11 (10 digits + NUL) is sized for. But in this branch frac_part is never assigned anything other than its initial 0, so only *p++ = '0' runs for the fraction — 1 char, not 7.
exp2 in [0, 22] (int_part = mantissa >> (23 - exp2)): this is the only branch where frac_part can also be nonzero and enter the 7-digit loop. Here int_part < 2^23 by construction, so it's capped at 7 digits, never 10.
```
agessaman
Elektr0Vodka
ripplebiz
Wolfram Keil
Blake Latchford
fdlamotte
Ev Lbibass
Liam Cottle
MarekWoandClaude Opus 5
Jarosław Domański