rich/MeshCore
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Record shared ESP32 MQTT regression and hardware validation

Browse Source
This commit is contained in:
mikecarper committed 2026-10-10 15:52:04 -07:00
1 parent d11bf3615d
commit 584fc9a4d7
1 file changed
+96 -18
Download Patch File Download Diff File Expand all files Collapse all files
docs/test-results/mqtt_tls_v4_regression.md
+96 -18
View File
@@ -21,7 +21,8 @@ The shipped ESP32-S3 AES driver allocates DMA-capable internal bounce buffers
for unaligned external-memory records, up to 1,600 bytes per buffer. Ample
PSRAM or an aggregate free-heap reading therefore does not establish that an
AES operation can succeed. The supplied log identifies an allocation failure
in this driver; the exact V4 heap history has not been captured in this run.
in this driver. The same allocation error was reproduced on the lab V4
with the older firmware and five enabled WSS presets.
The shipped Arduino 2.0.17/IDF 4.4.7 TLS allocator was checked in its archives:
TLS allocations use the configurable `mbedtls_calloc` function, so the
@@ -43,9 +44,12 @@ room-server and companion images that enable it.
the actual pending attempt across SDK tasks, rather than assuming a
five-second startup delay or 15-second reconnect gap finished a handshake.
- Reserve 32 KiB free DMA-capable internal memory and an 8 KiB largest block
before starting an SDK client task. This includes a 16 KiB startup allowance
above the 16 KiB AES reserve. Retained, running clients need 16 KiB free and
a 4 KiB largest block to reconnect. Pin SDK client tasks to their existing
before initializing a new SDK client. This includes a 16 KiB startup allowance
above the 16 KiB AES reserve. Restarting an initialized, stopped client needs
24 KiB free and an 8 KiB largest block: its buffers/configuration remain
allocated, while the 6 KiB task stack and task control block are recreated.
Retained, running clients need 16 KiB free and a 4 KiB largest block to
reconnect. Pin SDK client tasks to their existing
6,144-byte default stack; the separate bridge worker remains 8 KiB.
TLS without working PSRAM
additionally needs 60 KiB free and a 17 KiB largest block. These are
@@ -57,11 +61,19 @@ room-server and companion images that enable it.
- Require byte-addressable internal memory for string/JSON/queue buffer
fallbacks. The internal-memory capability alone also admits instruction RAM
that only supports aligned 32-bit access.
- Use uniform 896-byte MQTT buffers for all ESP32 presets. The SDK does not
resize initialized buffers through a config update, so a retained client
must already have room when an operator changes from password to JWT
authentication. This costs additional buffer memory for non-JWT presets on
internal-only boards, while preserving safe client ownership.
- Count broker QoS acknowledgments separately from accepted writes and expose
them as `pub_ack` in `get mqttN.diag`. This lets hardware tests verify
delivery without treating an expired outbox entry as an acknowledgment.
- Add DMA free/largest readings to `get mqtt.stats` and diagnostic logs.
Correct the operator-facing TLS error labels against SDK constants.
- Charge the existing 49,152-byte WiFi stack reserve for every MQTT image,
including profiles without compiled WiFi credentials. Multiple WiFi features
charge it once. Saved runtime credentials can activate the station.
Existing slot limits, certificate verification, authentication and backoff
remain in effect. Memory pressure defers additional slots instead of consuming
@@ -76,27 +88,52 @@ PSRAM and internal-only modes, fragmented heap, exhausted PSRAM, delayed
connection callbacks, retry timing and bounded replies. They do not execute
an ESP32 AES peripheral or qualify physical WiFi behavior.
All 43 focused Python MQTT tests, 1,794 native tests and 31 memory-budget tests
passed. The focused MQTT tests are also connected to the GitHub unit-test
workflow.
All 46 focused Python MQTT tests and 33 memory-budget tests passed after the
final reconnect/buffer changes. The 1,794-test native core suite passed after
the initial TLS/admission fix, before the subsequent MQTT-only refinements.
The focused MQTT tests are also connected to the GitHub unit-test workflow.
The new memory tests include a linked-ELF boundary that the previous
WiFi-omitted policy incorrectly accepted.
Three Full MQTT repeater builds with USB packet logging passed on 10 October
2026 at revision `7acc63746`. Their build logs report:
Full MQTT repeater and room-server builds with USB packet logging passed on
10 October 2026 at revision `937f8610c`. The V4 Full Companion build passed at `d11bf3615`. The RC32 build at the
same revision also passed using Arduino 3.3.11 and ESP-IDF 5.5.5, exercising
the newer SDK in an actual firmware build. The only intervening change corrected the host-side
WiFi memory accounting. Their build logs report:
| Build | Application flash bytes | Available internal heap bytes | Required heap bytes | Largest internal region bytes |
| --- | ---: | ---: | ---: | ---: |
| Heltec V4 | 2,182,125 | 205,080 | 198,432 | 172,312 |
| Heltec V3 | 2,125,149 | 211,296 | 209,184 | 178,528 |
| Station G2 | 2,017,685 | 201,280 | 200,480 | 168,512 |
| Heltec V4 | 2,182,265 | 205,080 | 198,432 | 172,312 |
| Heltec V3 | 2,125,377 | 211,296 | 209,184 | 178,528 |
| Station G2 | 2,017,953 | 201,280 | 200,480 | 168,512 |
| Heltec V3 room server | 2,104,057 | 201,992 | 196,016 | 169,224 |
| Heltec RC32 without display, IDF 5 | 2,287,033 | 195,688 | 194,336 | 162,920 |
| Heltec V4 Full Companion | 2,418,405 | 233,880 | 194,336 | 201,112 |
Application flash is the PlatformIO size-check reading. Internal heap is
linked capacity before dynamic allocation, not free heap measured after boot.
All three RAM gates passed the startup policy; they do not qualify maximum
These RAM gates passed the startup policy; they do not qualify maximum
TLS/WSS/outbox load. The default MQTT startup allowance remains 24,576 bytes,
with runtime admission for additional connections. See
[firmware memory checks](../research/firmware_memory_budget.md#mqtt-startup-minimum-and-runtime-qualification)
for the separate maximum-slot estimate and qualification limits.
Two additional Full builds compiled and linked the shared MQTT code but were
held by the unchanged RAM safety gates:
| Full build held | Available internal bytes | Required bytes | Shortfall bytes |
| --- | ---: | ---: | ---: |
| T-Beam SX1276 repeater, original ESP32 | 179,736 | 192,560 | 12,824 |
| Heltec V3 Companion | 182,264 | 198,432 | 16,168 |
Both already enabled WiFi through other build flags, so the corrected
MQTT-only WiFi detection did not increase either requirement. Their capacity
shortfalls require separate memory work before those Full artifacts can be
qualified; neither gate was lowered and neither failed artifact was published.
The shared firmware fixes still apply to their MQTT code. The original ESP32
compile check and IDF 4/5 tests cover the non-V4 source paths; physical testing
in this report is limited to the V4.
Live TLS 1.2 certificate chains from 12 audited preset endpoints were
verified with the exact CA PEMs extracted from `MQTTPresets.h` using a native
host build of the pinned mbedTLS 2.28.7 submodule from ESP-IDF 4.4.7, commit
@@ -114,8 +151,49 @@ require a CA replacement for the reported AES allocation errors. The verifier
used the native default mbedTLS configuration; this establishes pinned X.509
chain and trust-anchor behavior, not an ESP32 hardware handshake.
MercerWoodMesh is now reachable and live V4 testing is in progress. Live flash,
MQTT publication, reconnect and soak outcomes remain pending. The hardware
check must exercise the reported WSS presets together, verify rising publish
counters and DMA headroom, then include a broker/WiFi reconnection and
power-saving idle run with no reset or AES allocation error.
## Physical V4 results
MercerWoodMesh's Heltec V4 was tested over its native USB console using the
existing SlowFi network. The Pi's WiFi configuration was not changed. A full
16 MiB private flash backup was made before testing. Test credentials and raw
console logs are not included in the public results.
| Check | Result |
| --- | --- |
| Older firmware `c5fe8a0e`, WAEV and Cascadia | Both connected for the 150-second trial; accepted writes increased with no outbox backlog. The older firmware has no PUBACK counter, so accepted writes alone do not establish broker delivery. |
| Older firmware, five WSS presets | Reproduced `esp-aes: Failed to allocate memory`, `0x8018` and `0x8006`. All five slots eventually disconnected. This is a failing baseline. |
| Final firmware `937f8610c`, five presets enabled | WAEV, Cascadia and Analyzer US connected within 46 seconds. Analyzer EU and MeshMapper waited without allocating SDK clients because DMA memory was below the cold-start admission threshold. No AES allocation, TLS timeout/write error or CPU reset was observed during this trial. |
| Final firmware, power saving on, 180.005 seconds without console commands | All three connected brokers stayed healthy. Their `pub_ack` counters increased from 2/1/1 to 5/5/5. Uptime advanced from 57 to 258 seconds including surrounding queries. DMA heap stayed at 22,776 free / 22,516 largest bytes, with no outbox backlog, drops, AES allocation error, TLS timeout/write error or CPU reset. |
| Cooperative MQTT stop | The stop request completed and `get mqtt.stopping` returned off. The radio console remained responsive. |
The three-connection measurements included about 22.7 KiB free DMA-capable
heap with a 22.0 KiB largest block. Outbox activity briefly consumed additional
memory. Three brokers were qualified in these conditions; this does not
qualify five simultaneous WSS connections or a long-duration loaded-radio
soak. Additional enabled slots wait for headroom rather than exhausting the
memory needed by active TLS connections.
The final power-saving trial recorded core error bitmask `0` before and after
the idle interval; bitmask `8` appeared after the subsequent MQTT stop. An
earlier fixed-firmware trial also recorded bitmask `8`. This is the configured
radio watchdog flag on a quiet channel, also observed in earlier V4 console
qualification; it is not eight MQTT errors or evidence of a CPU reset. This
run does not claim zero radio watchdog events.
The attempted SSID interruption returned "WiFi SSID saved; reboot to apply"
and did not disconnect the station. It is therefore not counted as a physical
WiFi-loss recovery test. Regression tests exercise the production WiFi-loss
handler and connection scheduler with retained SDK clients, including the
restart memory allowance and serialized reconnect attempts.
A Pi-wide USB hub reset interrupted the older five-broker baseline. Several
radios disconnected together and subsequently reappeared. That interruption
is retained as a host-lab failure, not attributed to a V4 firmware crash.
After the final live check, the original NVS and filesystem backup regions
were restored and their writes verified, preserving the final firmware. All
52 saved preference checks matched and the public-key hash was unchanged.
Comparison removes console echo/prompt framing and treats the old blank
`get wifi.ssid` response as the new `(not configured)` response. MQTT and
power saving were both returned to their original off state. No release assets
were uploaded as part of this test.