The fallback configured a server, waited 500 ms, and accepted any plausible
system clock as proof that server had answered. It usually has not answered
that fast — and the device usually already holds valid time, from an earlier
sync or the RTC — so the first server in the list was credited unconditionally,
the walk stopped there, _last_ntp_sync was refreshed, and an unreachable host
was logged as the source. On the `set mqtt.ntp` validation path, where the
single-server walk exists so a typo fails fast, that reported a bad server as OK.
Poll sntp_get_sync_status() for SNTP_SYNC_STATUS_COMPLETED instead, which is the
layer's own statement that a packet arrived. The status is one-shot — reading
COMPLETED clears it — so a result left by an earlier sync would latch on the
first poll; clear it before the loop.
An implausible epoch after a completed sync now moves to the next server rather
than spinning out the remaining attempts against a server that has answered.
The manifest said "rebuild these for every espressif32 platform bump" and nothing
enforced it. mbedtls_4k.py verified the staged archives against the manifest's own
hashes, which proves the pair agrees with itself and nothing more: bump the
platform without rebuilding and every check still passes while the link takes
mbedTLS built against a different IDF. That fails at runtime on struct-layout
drift, not at the link, which is the failure the mechanism claimed to prevent.
Fingerprint the framework's own mbedTLS archives — the ones ours displace — as
stock: lines in the manifest and check them before the build. If the framework
moves, the staged pair is stale by construction and the build stops with the
replacement hashes printed ready to paste. Stronger than comparing a version
string: framework-arduinoespressif32 versions independently of the platform, and
its archives are what actually has to match.
The lib directory is resolved by trying the layouts espressif32 has used rather
than hardcoding one, and failing closed if none holds all four archives. The fetch
script ignores the new lines; its known-arches hint skips them so they cannot be
reported as architectures.
esp_mqtt_client_reconnect() is honoured only from MQTT_STATE_WAIT_RECONNECT, so
the post-correction path minted a fresh token, staged it, and then asked a
connected client to reconnect — a request esp-mqtt refuses. The slot kept running
on the token the clock correction had just proven stale, and recovery became
broker-driven rather than the clean reconnect this code intends.
Split the three states the path can find: a stopped or waiting client goes through
reconnectSlotClient() as before, and a live one has its transport closed first.
Only where the broker enforces exp, though — waev leaves live sessions alone past
expiry, so bouncing it would spend the 16 KiB contiguous handshake that the rest of
this branch exists to avoid.
Not a regression: the base branch called client->reconnect() directly at the same
site. On ESP32 the block is reachable from the WiFi-reconnect resync and the CLI
forced sync; the hourly refresh uses refreshNTP(), which does not carry it.
RadioLibWrapper::setTxPower() called setOutputPower() and nothing else. On
LR2021 that writes the PA config and TxParams, which are standby-only commands,
and it never re-arms the receiver.
This only affects LR2021 boards. recvRaw() has:
#if defined(USE_LR2021)
state = STATE_RX; // LR2021 stays in Rx after readData
#else
state = STATE_IDLE; // need another startReceive()
#endif
so on SX126x the next recvRaw() re-arms Rx anyway and the write is harmless,
while on LR2021 startReceive() is never called again on its own. A 'set tx'
issued while the radio was listening could therefore stop reception until the
next reboot. Observed a few times on an LR2021 repeater; recovery required a
power cycle.
Dispatcher's stuck-radio check does not catch it: isInRecvMode() reports the
wrapper's own `state` flag rather than the chip's actual mode, so it still
believes the radio is in Rx, and it only raises ERR_EVENT_STARTRX_TIMEOUT
without attempting recovery.
Fix: on LR2021, drop to standby via idle() before writing the PA config and let
checkRecv() re-arm Rx - the same pattern resetAGC() and
applySideDetectorConfig() already use. Other radios are left untouched.
Affects meshtracker_x1 and meshnology_w12 (both USE_LR2021).
Tested on hardware with an LR2021 repeater: six consecutive 'set tx' changes
(15/18/21/14/19/14), after which the radio kept receiving and forwarding
traffic (rawrx/rxpkts increasing, no missed IRQs). Noise-floor sampling also
kept updating, which only happens while state == STATE_RX.
The contact sync streams up to MAX_CONTACTS frames back to back, but
ArduinoSerialInterface never checked whether the stream could take
them: isWriteBusy() returned false unconditionally, so MyMesh's pacing
gate had no effect, and writeFrame() called write() without looking at
availableForWrite(). On ESP32 (HWCDC, 256 byte TX ring) a host that
stalls for a moment makes the driver drop queued bytes silently, which
tears a frame in half - and since the framing is length prefixed with
no checksum and no resync marker, the client stays desynchronised for
the rest of the session. Reported as 'the app disconnects while
syncing contacts' on devices with a large contact list.
Add opt-in flow control: report busy until a whole frame fits, and
drop frames as a unit instead of tearing them. Enable it for the
companion USB interface and give HWCDC a bigger TX buffer with a short
write timeout, mirroring what kiss_modem already does.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ArduinoSerialInterface::isConnected() always returned true, so any
companion build with a USB interface believes a client is attached from
boot onwards. Two user visible effects:
- new message notifications never fire, because MyMesh only notifies
the UI (display, buzzer) while no client is connected
- on builds that also expose BLE the home screen shows '< Connected >'
instead of the BLE pairing PIN, so a freshly flashed device cannot be
paired at all
Add an optional connection check callback and wire it up per platform:
native USB-CDC (TinyUSB on nRF52/RP2040/ESP32 with USB_MODE=0) exposes
real DTR through (bool)Serial. The ESP32 USB-Serial-JTAG peripheral has
no DTR concept - it reports 'connected' as soon as the host enumerated
the device - so fall back to frame activity there. Plain UARTs keep the
previous assume-connected behaviour.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
connect() logged "MQTT client started." unconditionally, so a failing
esp_mqtt_client_start() looked identical to a successful one. That is the one
state a later reconnect() cannot recover from, which made it the worst possible
line to be wrong.
reconnectSlotClient() checks isStarted() and calls connect() instead of
reconnect() when the client was stopped, but only the post-NTP stale-token
path used it. The ordinary backoff ladder and the circuit-breaker probe called
slot.client->reconnect() directly, and esp_mqtt_client_reconnect() is a no-op
on a client that is not started.
Two ways in. connect() sets _started only when esp_mqtt_client_start() returns
ESP_OK while setupSlot() sets initial_connect_done unconditionally, so a start
failure under heap pressure stranded the slot. More routinely, the WiFi-drop
handler calls disconnect() on every connected slot, which clears _started —
after that the ladder issued no-ops forever and the slot never came back.
Not caught by the soaks: the log line the guard prints can only come from the
NTP path, so a stranded slot and a slot that never entered the state produce
identical logs. Observed reconnects were broker-side drops with WiFi up, which
leave the client started.
The renewal-bounce path keeps its own isStarted() branch — it needs
softDisconnect(), which the helper does not do.
Every ordinary backoff reconnect and every circuit-breaker probe minted a
fresh JWT and re-applied credentials, with no check of whether the existing
token was still valid. setCredentials() always dirties the esp-mqtt config, so
reconnect() then called esp_mqtt_set_config() as well. On a flapping broker
that is a signing plus a configuration-copy cycle on every retry, and these
observers see ~38 genuine reconnects/day per slot.
The no-bounce renewal change (27bd05a1) only stopped the proactive renewal
from tearing down a live session; it left this retry path untouched, which is
why a soak shows renewals neither firing nor failing for hours while drops
continue — each reconnect silently re-mints and pushes the expiry out.
Reuse the credentials when their validity is provable and refresh them
otherwise. canReuseJwtForReconnect() lives with the other policy predicates so
it is host-testable, and it establishes current_time < token_expires_at before
subtracting: token_expires_at is unsigned, so an already-expired token would
otherwise wrap to ~4e9 seconds and read as valid for decades. The
>= kMinimumValidEpoch term also rejects the 0 that a failed renewal writes.
Minting stays the default for every uncertain case — unsynced clock, missing or
insane expiry, empty token, or an expiry inside kJwtReconnectSafetyMarginSecs
(60 s), which covers the handshake itself.
Two paths still always mint, deliberately:
- The circuit-breaker probe. It is the recovery of last resort for a slot
that has already failed repeatedly, quite possibly on auth, and it runs
once per 30 minutes — so a fresh token there costs nothing worth counting
against keeping that path guaranteed-clean.
- Any slot whose last error was a broker refusal. Before this change, minting
on every retry accidentally recovered from server-side credential
invalidation: key rotation, revocation, broker clock skew, or an audience
change after a reconfigure. Reuse would have retried a rejected credential
until it neared expiry — up to 24 h for every preset that leaves
token_lifetime at the default. onError already detects
MQTT_ERROR_TYPE_CONNECTION_REFUSED and only logged it; it now also sets a
per-slot force-mint flag, cleared on a successful connect and wherever the
credentials it referred to are blanked. The flag is volatile because the
esp-mqtt callback sets it and the bridge loop consumes it.
The reconnect log line reports the decision and its outcome — REUSE, MINT with
a reason, and OK/FAILED for the mint — because a silently failed mint is the
case most likely to end in an auth refusal. It never prints the token.
Host tests cover the reuse boundary: exact margin, already-expired, expiry 0,
sub-epoch expiry, empty token, unsynced clock, and the force-mint override.
The comment claimed the (WS_BUFFER_SIZE + 1) padding makes an oversized
upgrade response "fail cleanly (Upgrade header not found)". Reading
release/v4.4 transport_ws.c against release/v5.3 shows it does not.
v4.4's response loop is
} while (NULL == strstr(ws->buffer, "\r\n\r\n") && header_len < WS_BUFFER_SIZE);
so it also exits when the buffer fills without the terminator, and the code
then looks for "Sec-WebSocket-Accept:" and returns 0 if it is present. That
header appears early in a response, so an oversized header block yields a
BOGUS SUCCESS rather than a clean failure: the unread remainder stays queued
on the socket and is delivered as the first post-upgrade read, where the
deframer parses HTTP bytes as a WebSocket frame header.
Observed on hardware 2026-08-12 on a Heltec V4: a Cloudflare Page Shield CSP
report-uri header pushed the 101 response past the buffer, and the tail of
that header ("csp-reporting.cloudflare.com/cdn-cgi/script_monitor/report?")
reached the MQTT layer as payload, surfacing as
"Invalid MSG_TYPE response: 3" (0x35 = '5', high nibble 3).
The padding's real and only value is preventing the one-byte overflow of the
heap canary, which is still worth having. Narrow the comment to that claim and
record where the parser fix has to come from: IDF 5.2+ requires the "\r\n\r\n"
delimiter, memmoves the bytes following it, and fails cleanly when the buffer
fills. It cannot be patched here, since transport_ws.c ships precompiled in
libtcp_transport.a on Arduino 2.x.
No functional change.
(cherry picked from commit 9894e65e8ad961704d430a12a065baffda353f50)
waev's operator confirmed on 2026-08-11 that their servers do not disconnect a
client when its JWT passes exp — a 60-minute token can hold a session open for
hours. The renewal path assumed the opposite, in as many words: the comment at
the bounce called the renewal buffer "the ONLY margin between 'device
re-authenticates' and 'broker enforces exp and FIN-closes the session
mid-stream' — observed on the waev preset".
That premise made waev expensive, because waev is the only preset with a short
token_lifetime (3300 s; every other is 0, meaning the 24 h default). It was
therefore the only slot bouncing often: measured every ~47 minutes, about 30
times a day per device. And the bounce's re-handshake is where contiguous
internal DRAM goes — one renewal traced on hardware took the largest free block
from 27,124 to 16,372 B, below the 16,384 B mbedTLS inbound record buffer, after
which that slot could not re-handshake at all. The teardown and the credential
update cost nothing; the handshake costs everything.
So for a broker that leaves live sessions alone, refresh the credentials in place
and let the next genuine reconnect use them. That path already existed for the
"token renewed but old one still valid" case; this just stops treating imminent
expiry as a reason to tear down a healthy connection.
mqttPresetEnforcesTokenExp() defaults to true and is keyed by preset name rather
than a new struct field: adding a field would mean re-ordering a dozen positional
initialisers, where a mistake is silent, and the wrong default costs an outage
rather than a re-handshake. Custom and audience-only slots have no preset and are
treated as enforcing.
Our own logs already argued against the premise and we had not noticed: across 14
multi-device outages (10 hitting all four devices) the drops landed within ~3 s of
each other, on devices whose independent boot times gave them independent token
issue times. Independent expiries cannot align that tightly, so exp enforcement
was never a good explanation for them.
Unverified on hardware yet — the operator's statement is second-hand. Next: apply
to one board only and confirm the session survives past exp, that a later
reconnect still authenticates, and that the ~47-minute 27,124<->16,372
oscillation stops.
(cherry picked from commit 27bd05a17b9303b158feec7dab60af2fe128f5ce)
Three defects found reviewing the preceding commits.
1. reconnectSlotClient() stranded a STOPPED client, reintroducing the very bug
this branch fixes. It only rebuilt when isStarted() was true and otherwise
fell through to reconnect(), which is a documented no-op on a stopped client
— so nothing restarted it, at any rung, including the breaker probe. The
WiFi-transition teardown reaches exactly this state: it calls the hard
disconnect(), clearing _started while initial_connect_done stays set, so
after WiFi returned the slot could never come back. Now a stopped client is
started with connect() before the rebuild/reuse decision is considered.
The post-NTP credential refresh had the same exposure — it called
reconnect() directly — so it now goes through the helper too, still reusing
the transport since its fault is stale credentials, not the transport.
2. Allocating the neighbors buffer on first use let a stopped bridge allocate.
A neighbour discovery started before a stop can complete after it, and
neither caller rechecks bridge state, so requestPublishNeighbors() would
allocate 4 KB after releaseRuntimeBuffers() had already run and strand
_neighbors_publish_pending with no task to consume it. end() then returns
early on !_initialized, retaining the buffer until a later begin/end or a
reboot. Guarded on isRunning(), the same flag end() checks.
The release/acquire handoff itself was confirmed sound: the allocation and
copy precede the release store, and the task loop reads the pointer only
after its acquire load, so a half-published pointer is not observable.
3. The post-link map check failed open, contradicting the fail-closed claim in
its own commit message. A missing map, an unrecognised map format, or a
partial archive list each warned and passed; and it hardcoded firmware.map
while the post-action target used ${PROGNAME}, so a renamed program could
inspect a stale or absent file and still succeed. All four now fail the
build, and it requires every one of the four archives to appear rather than
at least one.
Rebuilt Heltec_v3_repeater_observer_mqtt, Heltec_v3_repeater and
heltec_v4_repeater_observer_mqtt; the opt-in path still reports all 4 archives
linked from .mbedtls-4k/.
(cherry picked from commit 5b5f076e5e165997e8050f2be061c7c67340fcf7)
The reduced-TLS work was validated on hardware but only reachable through
PLATFORMIO_BUILD_FLAGS pointing at an absolute path in a developer's home
directory, so nothing outside that machine could reproduce it.
Distribute the archives as a release asset instead of committing them: ~6 MB
per architecture, and they must be rebuilt for every espressif32 bump, so
committing would grow history permanently and go stale without any signal.
scripts/mbedtls_4k_manifest.txt per-arch sha256 of each archive
scripts/fetch_mbedtls_4k.sh fetch into .mbedtls-4k/<arch>/, verify
scripts/mbedtls_4k.py pre-build wiring and post-link proof
Off by default. The script is attached to esp32_base but returns immediately
unless MESHCORE_REDUCED_TLS=1, so ordinary builds need no artifact and are
byte-for-byte unaffected — confirmed by building with it absent.
Both ways this can fail silently produce a firmware that looks fine and lacks
the change, so the opt-in path refuses to guess:
- a -L at a missing or partial directory: the linker ignores an unusable
search path and resolves mbedTLS from the framework. Now a hard error.
- archives left over from an earlier platform version: now a sha256
mismatch against the manifest, naming both hashes.
- a -L that is present but outranked, leaving the flag inert: after the
link, firmware.map must resolve every libmbed*.a into .mbedtls-4k/, or
the build fails and prints the offending paths.
That last check earned its place immediately — it caught its own first
implementation comparing a relative map path against an absolute one, and an
earlier build flag in this investigation was accepted by the compiler while
no source read it. A flag reaching the compiler proves nothing about the link.
Verified all four paths on Heltec_v3_repeater_observer_mqtt: default build
unaffected; opted in with archives present links all four from .mbedtls-4k/
and says so; archives absent fails with a fetch hint; a single appended byte
fails on sha256.
Also removes platformio.local.ini.hold, which held the superseded approach of
pointing platform_packages at a whole custom framework. That installs over the
shared framework package and changes mbedTLS for every other ESP32 project on
the machine; the -L path keeps the change scoped to one env.
Note the inbound record buffer stays at 16 KiB, so this lowers per-connection
footprint by ~12 KiB but does not move the contiguous allocation a handshake
needs. It buys headroom, not a lower floor.
(cherry picked from commit a87faff6ff170c328fdd0550f4b4dd9089aa2ea0)
allocateRuntimeBuffers() took NEIGHBORS_JSON_BUFFER_SIZE unconditionally on
every board built WITH_MQTT_NEIGHBORS, whether or not mqtt.neighbors was ever
turned on. On a non-PSRAM board that is 4 KB of internal DRAM held for the
bridge's lifetime by a node that may never publish a neighbours snapshot.
Gating the existing allocation on the pref would not work: mqtt.neighbors is
read live by the mesh loop with no bridge restart, so enabling it at runtime
would find no buffer and silently publish nothing. Allocate on first use
instead, in requestPublishNeighbors(), which is reached only when something
actually wants to publish — periodic or a manual discovery.
Publishing the pointer across cores is safe with the existing handshake: the
allocation precedes the release store on _neighbors_publish_pending, and the
task loop reads the pointer only after its matching acquire load, so the
pointer cannot be observed half-published. A failed allocation drops that one
snapshot and retries on the next, rather than disabling neighbours for the
bridge's lifetime as the eager path did.
(cherry picked from commit e6da052a93f8765824d0fb4bd0c704ca3ed3d294)
The scheduled JWT bounce called PsychicMqttClient::disconnect(), which ends
with esp_mqtt_client_stop(). That ends the client task and returns its 6 KiB
stack to the heap at the moment the TLS teardown vacates two 16 KiB mbedTLS
record buffers, so the stack lands in that hole and the next handshake cannot
reuse it. On non-PSRAM boards the largest free block then ratchets down 16 KiB
at a time while total free heap stays flat.
Soak evidence from a Heltec V3 on 8d1a0eb3: 43 of 60 disconnects had no
preceding transport error, i.e. they were this proactive bounce rather than a
broker FIN, and two of the three max_alloc steps landed within 5 s of one.
Losing a whole TLS session later returned exactly 16,384 bytes of contiguity.
softDisconnect() closes the transport without the stop, so the task and its
stack stay put across the handshake. The bounce uses it plus reconnect(), and
falls back to connect() when the client really is stopped, since reconnect()
is a silent no-op in that state.
Also corrects a comment claiming the mbedTLS context survives a transport
close: only the esp-mqtt client object does.
(cherry picked from commit 10cf5cf48fb009e751e25b37fcc1f3d1256ddbbc)
The real Arduino.h includes stdlib.h, so ConfigSerializer.cpp reaches atoi,
atol and atof through it and compiles on device. The mock supplied only
cstdint, cmath and Stream.h, leaving those undeclared — and since the native
env compiles ConfigSerializer.cpp into every suite via build_src_filter, all
21 suites errored rather than just its own.
Fixing the mock keeps src/ identical to upstream and covers any other source
relying on the same transitive include.
pio test -e native: 297 test cases, 297 succeeded.
The rolling-release body is what the flasher dropdown serves as changelog.
Drop the stale v1.16.0 experimental blurb and describe the current observer
surface: MeshCore 1.17.1, web config, in-channel OTA, and neighbors.
The rolling-release body is what the flasher dropdown serves as changelog.
Drop the stale v1.16.0 experimental blurb and describe the current observer
surface: MeshCore 1.17.1, web config, in-channel OTA, and neighbors.
Promotes the v1.17.1 upstream sync and observer-firmware-dev work onto the
production channel.
Conflict in SH1106Display.cpp: kept the observer-firmware-dev begin() that
always runs Adafruit init and falls back across the 0x3C/0x3D SA0 pair,
rather than the earlier production DISPLAY_ADDRESS_ALT-only probe.
Both channels move together so a node comparing its embedded base against
its own channel's manifest does not read as permanently behind.
Resets the per-base build counter to N=1 on both channels. Nodes still on
v1.17.0.N take the diff_base branch in ESP32Board's OTA comparison, so the
counter going backwards does not strand them.
Picks up upstream MeshCore 1.17.1.
Notable upstream content:
- 1.17.1 version/build-date bump in the example MyMesh headers.
- nRF52: combine radio entropy with CC310 RNG.
- Companion FEM prefs: load/save of fem_ properties commented out until they
can be set from the client.
- Scoped reply routing: replies no longer dropped when flood.max.unscoped is
low (RoutingPolicy + unit tests).
- nRF52 unused-pin sweep (T1, T-Echo Lite, MeshPocket).
No conflicts.
agessaman
mikecarper
Fedor Kallay
rootandClaude Fable 5
Kevin Le