Protect MQTT publishing headroom and credit replies before disconnect

docs/WiFi.md

@@ -94,6 +94,13 @@ normally use no more than two TLS/WSS brokers; some classic ESP32 targets are
 reliable with only one. Excess configured slots remain saved but show as
 inactive in `get mqtt.status`.
 
+Five active brokers on a PSRAM board is an upper limit, not a qualified load
+for every board. Adding a connection while another broker is connected must
+leave contiguous memory for packet publishing as well as SDK startup
+allocations. A saved slot can wait for memory while existing connections
+continue forwarding. Recovery of the first broker uses a separate bootstrap
+bound so this extra reservation cannot strand every disconnected slot.
+
 The bridge has a bounded packet queue for broker or WiFi outages and reconnects
 automatically. A non-PSRAM build holds 6 packets and a PSRAM build holds 50.
 When the queue is full, the oldest item is replaced; after five minutes with no
@@ -111,7 +118,8 @@ settings remain enabled. Failed requests, a WiFi association, and NTP replies
 alone do not restart the outage timer or keep a failed MQTT uplink awake.
 
 After 30 minutes, the firmware opens a recovery window lasting up to two
-minutes. A broker acknowledgment or a connection that stays up for 30 seconds
+minutes. A new broker acknowledgment, including one immediately followed by a
+disconnect, or a connection that stays up for 30 seconds
 returns the node to normal operation. Otherwise WiFi is suspended for another
 30 minutes. An explicit management request can ask for an earlier probe.
 Automatic NTP requests have a 15-second timeout; failed requests wait 30 minutes
@@ -841,6 +849,7 @@ For an MQTT observer:
 get wifi.status
 get mqtt.enabled
 get mqtt.status
+get mqtt.stats
 get mqtt1.diag
 get mqtt2.diag
 ```
@@ -850,6 +859,17 @@ configured slot's `pub_ack` count in `get mqttN.diag`. Accepted publishes are
 not proof of delivery: a QoS 1 publish is confirmed only when the broker
 acknowledges it. A quiet radio channel alone is not a failed MQTT connection.
 
+Capture these outputs before rebooting, then repeat `mqtt.stats` and the slot
+diagnostics after 60 seconds. `(ok)` reports connection state, not delivery.
+In `mqtt.stats`, `drops=A/B` separates outbox-cap drops from packet publication
+attempts skipped for memory pressure; a growing `B` can explain missing packet
+reports while broker connections still look healthy. `DNS failed (0x8001)`
+records a hostname lookup failure, which can follow an earlier disconnect; it
+does not identify the reason for that first disconnect. `wifi.status` showing
+`suspended` means the shared outage cooldown is active, not that WiFi settings
+were erased. These diagnostics do not reveal passwords or tokens, but WiFi
+status can include the SSID or local IP address. Redact them before posting.
+
 The shared MQTT bridge recovers failed transports on its owning task, including
 errors without a subsequent disconnect callback and connection attempts that
 never finish. It also watches accepted QoS 1 publishes for missing broker

docs/research/firmware_memory_budget.md

@@ -119,19 +119,34 @@ qualification of five simultaneous PSRAM connections or two internal TLS
 connections.
 
 The `mqtt` report also names the production runtime admission thresholds.
-A cold SDK start requires at least 32 KiB of DMA-capable internal free heap
-and an 8 KiB contiguous DMA block. This includes a separate 16 KiB allowance
-for its SDK task stack/TCB, receive/transmit/reassembly buffers and transport
-or configuration objects, alongside the 16 KiB AES reserve. Those ordinary
-SDK allocations occur after admission and do not use the TLS allocation guard.
-A retained initialized SDK handle with a stopped task requires 24 KiB free and
-an 8 KiB contiguous DMA block. Its separate 8 KiB restart allowance covers the
-pinned 6 KiB task stack plus TCB and allocator overhead. SDK stop retains its
-receive/transmit buffers, configuration, transport objects and wrapper
-reassembly buffer, so restarting that handle does not need to reserve their
-first-initialization allocation again. A wrapper whose SDK initialization has
-not completed still receives the full cold allowance. This distinction also
-applies after WiFi loss stops the SDK tasks but keeps their handles.
+Connection admission and the packet-publishing guard share a contiguous-block
+floor: 16,000 bytes with detected PSRAM, or 8,000 bytes without it. Admitting
+another broker while one is connected reserves its startup allocations beside
+that floor, protecting existing packet forwarding. The first connection after
+boot or an outage can bootstrap without that extra reservation, so the
+additional-broker reserve cannot block every disconnected slot's recovery.
+
+A cold SDK start requires at least 32 KiB of DMA-capable internal free heap.
+The separate 16 KiB allowance covers its SDK task stack/TCB,
+receive/transmit/reassembly buffers and transport or configuration objects,
+alongside the 16 KiB AES reserve. Those ordinary allocations occur after
+admission and do not use the TLS allocation guard. Bootstrap needs a contiguous
+16,000-byte block with PSRAM or 8,192 bytes without it. With another connected
+broker, the contiguous minimum includes its publication floor plus the startup
+allowance: 32,384 bytes with PSRAM or 24,384 without it, before the additional
+internal TLS requirement below.
+
+A retained initialized SDK handle with a stopped task requires 24 KiB free.
+Its separate 8 KiB restart allowance covers the pinned 6 KiB task stack plus
+TCB and allocator overhead. Bootstrap uses the same 16,000/8,192-byte block
+minima as a cold start. With another connected broker, its contiguous minimum
+is 24,192 bytes with PSRAM or 16,192 without it, before the internal TLS
+requirement below. SDK stop retains its receive/transmit buffers,
+configuration, transport objects and wrapper reassembly buffer, so restarting
+that handle does not need their first-initialization allocation again. A
+wrapper whose SDK initialization has not completed still receives the full
+cold allowance. This distinction also applies after WiFi loss stops SDK tasks
+but keeps their handles.
 
 All ESP32 clients use 896-byte receive, transmit and wrapper reassembly buffers
 from first initialization. Applying configuration cannot grow the retained SDK
@@ -143,15 +158,22 @@ and cold admission covers the ordinary allocations. WiFi recovery and explicit
 reconfiguration continue to retain clients.
 
 A warm reconnect keeps its running task and buffers; it requires at least
-16 KiB free and a 4 KiB contiguous DMA block. The per-client SDK stack is
-independent of `MQTT_TASK_STACK_SIZE`, which sizes the existing bridge worker.
-A TLS attempt without PSRAM requires at least
-60 KiB free and a 17,408-byte contiguous block. PSRAM allocation fallback must
-leave the 16 KiB DMA reserve available. These thresholds describe free heap at
-the time of admission; they are not additional linked capacity and the TLS
-contiguous threshold is not a startup allocation requirement. Available linked
-internal heap cannot establish how much DMA-capable heap remains or how it is
-fragmented after boot. Only runtime observations can establish that.
+16 KiB free and the same 16,000/8,000-byte publication block. The per-client SDK
+stack is independent of `MQTT_TASK_STACK_SIZE`, which sizes the existing bridge
+worker. TLS without detected PSRAM retains its 60 KiB free-heap and 17,408-byte
+record-block floors for bootstrap. With another connected broker, admission
+reserves that record beside the publication block and ordinary startup
+allocations: 41,792 contiguous bytes for a cold start, 33,600 for a stopped
+task, or 25,408 for a warm reconnect. Both admission and publication use
+detected PSRAM, including a board whose compiled-in PSRAM support failed to
+initialize.
+
+PSRAM allocation fallback must leave the 16 KiB DMA reserve available. These
+conservative thresholds describe free heap at admission; they are not extra
+linked capacity or a guarantee against later fragmentation or allocations by
+other tasks. Available linked internal heap cannot establish how much
+DMA-capable heap remains or how it is fragmented after boot. Only runtime
+observations can establish that.
 
 Maximum-load qualification still needs physical boot and sustained reconnect,
 publish and configuration-change tests with the selected number of brokers,

docs/test-results/mqtt_tls_v4_regression.md

@@ -265,3 +265,51 @@ measured with three live brokers or qualification of maximum simultaneous load.
 No new physical three-hour V4 soak or reporter-side retest has been performed
 as part of this repair. These tests establish the repaired recovery behavior,
 not the cause of the reporter's hardware failure or website-side availability.
+
+## Station G3 follow-up: early failures and network suspension
+
+A user supplied a failed-state console capture from `1.18.0.1` with five WSS
+presets configured. It shows all five slots disconnected and WiFi suspended
+with the 30-minute recovery schedule. The displayed uptime and first-disconnect
+ages place the first two failures roughly 28 and 89 seconds after boot, before
+the reported seven-to-eight-minute loss of reporting.
+
+The slot diagnostics record repeated completed transport recoveries and DNS
+lookup failures (`0x8001`). This establishes active recovery followed by the
+outage cooldown, rather than an indefinitely frozen worker in this capture.
+The DNS errors are the latest failures, not proof of the original disconnect
+cause. The stats also record 345 packet publication attempts skipped for
+memory pressure (`drops=0/345`), an empty SDK outbox and a growing bridge queue.
+These skips are attempts, including retries, rather than 345 distinct packets.
+Heap readings after clients were stopped do not establish their earlier
+connected-state headroom.
+
+Two source issues were identified:
+
+- Connection admission could permit another SDK client below the contiguous
+  memory threshold required by packet publishing. When another slot is
+  connected, the shared admission policy now protects that publishing
+  threshold and accounts for allocations made during SDK startup or restart.
+  It defers additional connections under pressure. First-broker recovery has
+  a separate bootstrap bound, retaining a way to reconnect when all slots
+  have disconnected. Neither path lowers the publishing safety threshold or
+  erases slots; admission bounds do not guarantee future allocations.
+- A new PUBACK followed by DISCONNECTED between worker passes was consumed
+  without crediting the successful delivery to the outage timer. The worker
+  now credits that reply once, even if the connection has since dropped.
+  CONNACK-only flapping and old acknowledgment counts cannot renew the timer.
+
+These repairs do not establish why the user's DNS resolution failed. Physical
+G3 reproduction and a reporter-side retest remain necessary before calling
+that reported failure resolved. The shared changes apply to all ESP32 roles
+that include this MQTT bridge.
+
+Validation passed: 85 MQTT Python host cases, four Companion MQTT host cases
+and 57 firmware RAM policy cases. The new disconnected-PUBACK regression
+fails with the old condition and passes with the fix, including timer wrap,
+disabled slots and one-time acknowledgment consumption. Admission tests use
+the captured 93,024-byte free DMA heap and 20,468-byte largest block: one
+stopped broker can attempt recovery, while an additional broker waits when
+another slot is connected. The Station G3 Full repeater compiled and passed
+capability, dual-OTA partition and linked RAM checks. These are host/build
+checks, not a physical five-broker G3 soak.