reconnect() on a started client returns ESP_FAIL while an attempt is already
in progress. On a Heltec V3, a live reconfigure hit exactly that and the
attempt then connected, yet cf read 1. Count start_failures only when
connect() on a stopped client fails, and restore the attempt flag to its
prior value on any failed call so the in-progress attempt is still judged by
its own outcome.
connect()/reconnect() returning an error started no attempt, so no event
arrived and the failure went uncounted -- the low-heap case the counter exists
for. Counted now in a bridge-task-only start_failures, summed into cf: and
mqtt_connect_failures.
A renewal or corrected-clock bounce whose softDisconnect() times out gets its
old session's DISCONNECTED after the new attempt raised the flag. That session
is still marked connected until the handler runs, so the gate now also
requires !connected. An ignored late CONNECTED clears the flag too.
cf: no longer depends on dc being non-zero. Comments corrected for the
PSRAM-fallback stack cost and heap_largest's sampling point; the worst-case
test now models 6 slots.
The Starting-state gate raced the event task: client_state becomes Starting
only after connect()/reconnect() returns, so a fast failure could deliver
DISCONNECTED first and go uncounted, while a deliberate stop of an attempt in
flight was counted as a failure. A per-slot flag is now raised before the call
and cleared by onConnect, a failed start, or stopSlotClient().
Also stops zero-valued connect_failures/slots_breaker from creating an empty
stats object.
Two gaps remained after the first connection-health commit.
A slot retrying into a heap that cannot satisfy its TLS allocation moved no
counter at all. disconnect_count rose, but it also rises for dropped live
sessions, so it could not separate "actively failing to connect" from "was
connected, dropped once". Adds a per-slot connect_failures, incremented in
onDisconnect when the event arrives while the client is Starting -- meaning
it ended an attempt that never reached onConnect. Late events from a stopped
or quarantined client, and soft-disconnects of a live session, are in other
states and are not miscounted.
A slot parked by the circuit breaker also looked identical to one still
retrying, despite being materially worse: the breaker only probes every 30
minutes, so recovery is far slower and an operator may need to intervene.
Surfaces both:
- `mqttN.diag` gains `cf:<n>` beside the existing `dc:<n>`.
- the status payload gains `mqtt_connect_failures` (summed across slots) and
`mqtt_slots_breaker` (count of tripped slots), each emitted only when
non-zero so a healthy board's payload does not grow.
The counter is what distinguishes a retrying slot from a quiet one: while a
slot is down its publish counters cannot move by definition, so this is the
only value that shows the device is still trying.
Both suggested by adversarial review.
The worst-case test added with the connection-health fields did not model
production maxima -- it used a 28-byte model and a 24-byte firmware string,
while MQTTBridge permits _origin[32], _device_id[65], _board_model[64],
_firmware_version[64] and derives a 63-byte client version. isValidName()
rejects [ ] / \ : , ? * but permits '"', so a 31-character node name of
quotes is legal and every quote escapes to two bytes.
Measured at those real maxima:
without connection-health fields 762 bytes
with connection-health fields 851 bytes
buffer 768 bytes
So the 768-byte buffer already had only six bytes of headroom before this
work, and the four new fields overflow it. Overflow is silent:
serializeComplete() returns 0 and clears the buffer, and the bridge
publishes only a positive length, so a device would simply stop publishing
status with no error anywhere -- the same blindness the connection-health
fields exist to remove.
Raises STATUS_JSON_BUFFER_SIZE to 1024 and corrects the test to use the
actual maxima. Status serializes into the shared 2048-byte publish scratch
buffer, so this costs no extra RAM; only the PSRAM-allocation-failed stack
fallback grows by 256 bytes. A static_assert keeps the status ceiling within
the shared buffer.
Found by adversarial review of the preceding commit.
A slot that cannot establish TLS reported nothing. The per-slot counters
track publish successes and failures, and a slot that never connects never
attempts a publish -- so its ok count freezes and its error count stays at
zero for the entire outage. Observed on hardware: a slot sat dead for over
five hours while `sN=` showed 0 errors and the published status payload
carried no indication at all.
Two surfaces gain the missing signal:
- `get mqtt.stats` gains `down=<n>` (omitted when zero, like `filt=`) and
marks each disconnected slot with a trailing `!`. The marker goes after
the counts so existing "s(\d)=(\d+)/(\d+)" parsers keep matching.
- The published status payload gains `mqtt_slots_up`, `mqtt_slots_total`,
`heap_largest`, and `mqtt_outage_secs`. These ride on every healthy slot's
status message, because the slot that is down cannot report itself.
`mqtt_slots_up < mqtt_slots_total` is the condition to alert on.
`mqtt_outage_secs` is detail: it is present only when the outage has a known
start, so its absence does not imply health -- a slot that has never
connected since boot has no outage start time.
`heap_largest` is included because free heap alone does not predict a failed
connection. A slot needs one contiguous 16,384-byte block for its TLS record
buffers; measured immediately before an attempt this value predicted the
outcome in every observed case, while total free heap was 85 KB throughout a
multi-hour outage. A low value with every slot up is normal rather than a
fault -- established sessions already hold their buffers.
Publish-error semantics are unchanged: connect failures are not folded into
them.
Adds host coverage for the new fields, for the outage key's presence rule,
and for worst-case payload size. That last one matters because
serializeComplete() returns 0 and clears the buffer on overflow and the
bridge publishes only a positive length, so exceeding the 768-byte status
buffer would silently stop status publication -- the same blindness this
change exists to remove.
The reconstruction-less fallback path in buildPacketJSON (queued packet
with no captured raw frame that cannot be re-serialized) published
fabricated SNR 12.5 / RSSI -65 that consumers could not distinguish
from real radio readings. It now passes NAN / MQTT_RSSI_UNKNOWN, and
buildPacketMessage omits the SNR/RSSI keys entirely when no measurement
exists, mirroring the existing score omission.
Ports the fix from ACETyr's PR #13 (0a037040) on mqtt-bridge-implementation.
Co-Authored-By: ACETyr <35745290+ACETyr@users.noreply.github.com>
Drop BLE_TX_POWER=0 from the companion envs so they use the default of
4 dBm, like most other variants. The lower power was carried over from
the original Base Duo PR and real-world range is better at the default.
- rename the board class and files to MuziBaseBoard
- uppercase the build flags (MUZI_BASE, MUZI_BASE_SUPERIO)
- name the superIO envs muzi_base_{duo,uno}_superIO_companion_radio_ble so
build.sh picks them up
- set MAX_CONTACTS to 350 like the other variants
- move the pin setup from initVariant() into MuziBaseBoard::begin()
Codex review of the previous commit found two real problems.
The routes go live in startLanMode()/startSetupMode(), one tick before
probeBoardCommands() first runs, so a status request can land in between.
That made _board_cmds a genuine cross-task race despite the comment
claiming otherwise, and — worse — let /api/status answer "no board
commands" before the board had been asked. The page reads status once,
so a browser that loaded in that window hid working FEM controls for the
whole session.
The probe result is now published and read under _mux, and /api/status
OMITS board_cmds until the probe has run, which is not the same as
reporting an empty list. The page treats the absent field as "not known
yet" and re-polls with backoff; the audit waits for it the same way,
rather than auditing a board it only half knows.
Probing from the start path instead would close the window, but
Callbacks::execCommand is documented as loop-task-only and `start
webconfig` is itself mid-handleCommand, so that trades a race for
re-entrancy.
The audit never executed fan.lo or fan.hi, which have no getter and so no
round-trip: they were only ever matched as strings. They now run for real,
including the four rejection paths, and the mock grew --probe-delay so the
retry path is exercised rather than assumed.
The FEM commands moved out of CommonCLI into Board::handleCommand(), and
the T-Beam 1W fan control was added there too, but the portal never caught
up: `radio.fem.txgain` and the fan commands were missing from the terminal
table entirely, while `radio.fem.rxgain` was offered on every board even
though a board with no hook now answers "??:" rather than "unsupported".
Whether a node answers these is a property of the board, not of the build,
so the page cannot know from the firmware version. Ask the board instead:
probeBoardCommands() runs each candidate getter once on the loop task at
startup and keeps the ones that answer, which needs no per-variant list
because Board::handleCommand() already reports whether it handled a
command. /api/status names the survivors and the page hides everything
else, so adding a command to a variant means one entry in WC_BOARD_CMDS
rather than an edit per board.
The two FEM keys also become Radio-panel toggles, gated the same way;
their `set` reaches the board hook through the existing config batch, so
only the allowlist and /api/config needed to grow.
webconfig_cli_audit.py now scans variants/*/*Board.cpp for the boards that
actually build the portal, checks set-only keys in the reverse direction,
and verifies every gate is a command some board answers; the mock gained
--board-cmds so both shapes of board are testable. `stop ota` joins
NOT_OFFERED: `start ota` cannot run from the portal, so it has nothing to
stop.
Two gaps found while tracing a ThinkNode M7 report of a missing OTA
manifest. The M7 itself was healthy in both channels; these are what the
investigation turned up around it.
Pass --prune to gen-slim-manifests.py in both workflows. The generator only
ever added, so an env that stopped being built kept a manifest pointing at a
release asset the KEEP_BUILDS=2 prune later deleted — a 404 for `ota update`
on any node still running it. The flasher-side commit adds the flag along
with an empty-input guard and --prune-limit, which fails the run rather than
pruning when a build looks like it silently lost envs.
Add "Verify production channel is baked in", which production lacked while
beta has had it from the start. The manifest base is a compile-time -D, so a
build that lost it or picked up the other channel's is invisible until a node
runs `ota check` — and the failure mode is production hardware OTA-ing itself
onto beta. Checked against real published binaries: passes the production M7
build, rejects the beta one.
OTA_MANIFEST_BASE_URL is now stated explicitly in the production env: block,
equal to build.sh's default, so the verify step asserts against the value the
build was actually handed instead of a second hardcoded copy that could drift.
Neither change triggers a build: .github/** is in both workflows' paths-ignore.
Adds a built-in preset for the Polish MeshCore community broker at
mqtt.marwoj.net: MQTT over TLS on port 8883, username/password auth
with credentials embedded in firmware, and the standard MeshCore topic
layout. The Let's Encrypt chain anchors at the existing ISRG_ROOT_X1
constant, so no new CA certificate is needed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
agessaman
Liam Cottle
Scott Powell
Wolfram Keil
Blake Latchford
fdlamotte
Wessel Nieboer
Ev Lbibass
TJ Downes
MarekWoandClaude Opus 5