Preserve MQTT TLS connection time and internal DMA headroom

docs/research/firmware_memory_budget.md

@@ -13,6 +13,8 @@ The check reserves room for enabled runtime allocations as well as static
 data. A firmware image fitting its board's reported RAM total is insufficient:
 the display, packet pool, USB, Bluetooth workers and WiFi can allocate after
 startup. The T096 Full 1.17.1.5 report exposed this distinction.
+MQTT uses an explicit startup minimum plus runtime connection admission. A
+passing build report does not qualify its maximum TLS/WSS/outbox workload.
 
 ## What is counted
 
@@ -38,7 +40,9 @@ keeps runtime UART enablement in the budget beside MQTT and ESP-NOW.
 T-LoRa V2.1-1.6 cannot fit all three transports: the qualified linked triple
 image had 190,488 available internal bytes against 196,984 required bytes
 under the earlier allocation estimates. The corrected equivalent requirement
-is 193,400 bytes, still 2,912 bytes above capacity.
+was 193,400 bytes with the earlier flat MQTT allowance, still 2,912 bytes above
+capacity. Its configured 16 KiB MQTT worker stack adds another 8 KiB under the
+current startup policy; these figures describe historical linked checks.
 Its ordinary release retains separate UART/ESP-NOW and MQTT/ESP-NOW Full
 images; routing capacities remain unchanged.
 A 160x80 ST7735
@@ -50,6 +54,53 @@ Headless and OLED devices use their own smaller totals. The JSON lists each
 component and checks the largest available region against the largest planned
 single allocation.
 
+### MQTT startup minimum and runtime qualification
+
+The `mqtt_startup_allowance` component keeps a 24,576-byte internal startup
+minimum for the default 8,192-byte MQTTBridge worker stack and two active
+clients. Each SDK client task defaults to 6,144 bytes and its maximum configured
+receive/transmit buffers total 1,792 bytes. These stacks and buffers total
+24,064 bytes for two clients, leaving 512 bytes within the existing allowance.
+This is a minimum allowance, not a complete allocation bound for the clients,
+transport objects, TLS records, certificate processing, WebSockets or outboxes.
+The independent 16 KiB transient margin remains in the policy.
+
+`MQTT_TASK_STACK_SIZE` above 8,192 bytes raises the startup requirement by its
+actual increase and raises the required contiguous allocation when needed.
+Smaller stacks and fewer runtime slots do not lower the existing allowance.
+The default runtime slot array has six entries on PSRAM builds and three on
+internal-only builds. The bridge can expose five active connections when PSRAM
+is present and two otherwise; `MQTT_RUNTIME_SLOT_COUNT` can restrict that
+capacity. A PSRAM board with unavailable PSRAM uses the two-connection cap.
+
+The report's separate `mqtt.maximum_slot_stack_buffer_allowance_bytes` records
+48,384 bytes for five exposed slots or 24,576 bytes for two, with larger worker
+stacks added to either. The five-slot figure extends the minimum by 7,936 bytes
+for each additional client task and buffer pair. It is not added to the startup
+gate: extra clients allocate lazily, and every connection attempt must pass
+runtime admission. It still excludes the maximum TLS/WSS/outbox workload.
+`maximum_runtime_load_qualified` and `physical_validation_performed` are false
+in these build reports. A `passed` startup check must not be presented as
+qualification of five simultaneous PSRAM connections or two internal TLS
+connections.
+
+The `mqtt` report also names the production runtime admission thresholds.
+Each attempt requires at least 16 KiB of DMA-capable internal free heap and a
+4 KiB contiguous DMA block. A TLS attempt without PSRAM requires at least
+60 KiB free and a 17,408-byte contiguous block. PSRAM allocation fallback must
+leave the 16 KiB DMA reserve available. These thresholds describe free heap at
+the time of admission; they are not additional linked capacity and the TLS
+contiguous threshold is not a startup allocation requirement. Available linked
+internal heap cannot establish how much DMA-capable heap remains or how it is
+fragmented after boot. Only runtime observations can establish that.
+
+Maximum-load qualification still needs physical boot and sustained reconnect,
+publish and configuration-change tests with the selected number of brokers,
+transport schemes, certificates and enabled UART/browser/OTA features. Record
+DMA free heap, largest DMA block and allocation failures throughout that
+workload. Host admission tests and passing ELF checks provide engineering
+evidence without claiming that hardware validation has been performed.
+
 The W12 Full MQTT profile remains unqualified: the current linked image has
 182,768 internal bytes available against 198,432 required, including the
 unchanged 16 KiB transient margin. Its standard repeater compiles and passes

docs/test-results/mqtt_tls_v4_regression.md

@@ -0,0 +1,82 @@
+<!-- meshcore-hosted-doc-link:start -->
+<p class="meshcore-hosted-doc-link"><a href="https://mikecarper.github.io/MeshCore/test-results/mqtt_tls_v4_regression/">View this page on MeshCore Docs</a>.</p>
+<!-- meshcore-hosted-doc-link:end -->
+
+# ESP32 MQTT TLS connection regression
+
+## Report and diagnosis
+
+The Heltec V4 report contains two separate failures:
+
+- `0x8006`: ESP-TLS connection timeout opening the WSS brokers
+  `mqtt.waev.app:443` and `mqtt-v1.cascadiamesh.org:443`.
+- `esp-aes: Failed to allocate memory`, followed by TLS write failure
+  `0x8018` and resend errors.
+
+The firmware configured a 2,500 ms network timeout to bound synchronous
+publishes. ESP-MQTT also uses that setting for connection establishment,
+including TLS and the WebSocket upgrade. It was not a write-only setting.
+
+The shipped ESP32-S3 AES driver allocates DMA-capable internal bounce buffers
+for unaligned external-memory records, up to 1,600 bytes per buffer. Ample
+PSRAM or an aggregate free-heap reading therefore does not establish that an
+AES operation can succeed. The supplied log identifies an allocation failure
+in this driver; the exact V4 heap history has not been captured in this run.
+
+The shipped Arduino 2.0.17/IDF 4.4.7 TLS allocator was checked in its archives:
+TLS allocations use the configurable `mbedtls_calloc` function, so the
+existing PSRAM redirect takes effect. Retaining a PsychicMqttClient does not
+retain its TLS records: ESP-IDF frees/recreates the TLS session on reconnect.
+
+References: [ESP-IDF 4.4.7 AES implementation](https://github.com/espressif/esp-idf/blob/v4.4.7/components/mbedtls/port/aes/dma/esp_aes.c)
+and [ESP-TLS errors](https://github.com/espressif/esp-idf/blob/v4.4.7/components/esp-tls/esp_tls_errors.h).
+
+## Firmware changes
+
+These changes apply to the shared ESP32 MQTT bridge, including repeater,
+room-server and companion images that enable it.
+
+- Restore the SDK's 10,000 ms network timeout. A stalled synchronous publish
+  can now wait up to that same timeout; the radio loop runs separately.
+  Cooperative shutdown also includes that deadline in its stop budget.
+- Schedule initial connections through the normal maintenance loop. Check
+  the actual pending attempt across SDK tasks, rather than assuming a
+  five-second startup delay or 15-second reconnect gap finished a handshake.
+- Defer a new connection if DMA-capable internal memory has less than
+  16 KiB free or a largest block below 4 KiB. TLS without working PSRAM
+  additionally needs 60 KiB free and a 17 KiB largest block. These are
+  conservative admission bounds, not a promise that future allocations succeed.
+- Prefer PSRAM for TLS allocations and reject an internal fallback that
+  would consume the 16 KiB DMA reserve. Check allocation-size overflow.
+- Keep renewal able to close an old TLS connection before checking memory
+  for its replacement. Stop an in-flight SDK client on reconfiguration.
+- Require byte-addressable internal memory for string/JSON/queue buffer
+  fallbacks. The internal-memory capability alone also admits instruction RAM
+  that only supports aligned 32-bit access.
+- Add DMA free/largest readings to `get mqtt.stats` and diagnostic logs.
+  Correct the operator-facing TLS error labels against SDK constants.
+
+Existing slot limits, certificate verification, authentication and backoff
+remain in effect. Memory pressure defers additional slots instead of consuming
+memory needed by existing connections and the radio.
+
+## Validation and limits
+
+The regression tests execute the production optimizer, adapter setters,
+connection admission, TLS allocator fallback, scheduler and diagnostics with
+controlled SDK calls and heap readings. They cover IDF 4/5 config layouts,
+PSRAM and internal-only modes, fragmented heap, exhausted PSRAM, delayed
+connection callbacks, retry timing and bounded replies. They do not execute
+an ESP32 AES peripheral or qualify physical WiFi behavior.
+
+All 1,794 native tests and the 31 memory-budget tests passed. The focused
+MQTT tests are also connected to the GitHub unit-test workflow.
+
+The VM reached both reported broker TLS endpoints with certificate
+verification enabled. That is an endpoint check, not an ESP32 MQTT test.
+
+MercerWoodMesh reported offline and its SSH connection timed out during this
+run. No live V4 flash, MQTT publication or soak result is claimed. The next
+hardware check must exercise the reported WSS presets together, verify rising
+publish counters and DMA headroom, then include a broker/WiFi reconnection
+and power-saving idle run with no reset or AES allocation error.