Protect MQTT publishing headroom and credit replies before disconnect

docs/WiFi.md

@@ -94,6 +94,13 @@ normally use no more than two TLS/WSS brokers; some classic ESP32 targets are
 reliable with only one. Excess configured slots remain saved but show as
 inactive in `get mqtt.status`.
 
+Five active brokers on a PSRAM board is an upper limit, not a qualified load
+for every board. Adding a connection while another broker is connected must
+leave contiguous memory for packet publishing as well as SDK startup
+allocations. A saved slot can wait for memory while existing connections
+continue forwarding. Recovery of the first broker uses a separate bootstrap
+bound so this extra reservation cannot strand every disconnected slot.
+
 The bridge has a bounded packet queue for broker or WiFi outages and reconnects
 automatically. A non-PSRAM build holds 6 packets and a PSRAM build holds 50.
 When the queue is full, the oldest item is replaced; after five minutes with no
@@ -111,7 +118,8 @@ settings remain enabled. Failed requests, a WiFi association, and NTP replies
 alone do not restart the outage timer or keep a failed MQTT uplink awake.
 
 After 30 minutes, the firmware opens a recovery window lasting up to two
-minutes. A broker acknowledgment or a connection that stays up for 30 seconds
+minutes. A new broker acknowledgment, including one immediately followed by a
+disconnect, or a connection that stays up for 30 seconds
 returns the node to normal operation. Otherwise WiFi is suspended for another
 30 minutes. An explicit management request can ask for an earlier probe.
 Automatic NTP requests have a 15-second timeout; failed requests wait 30 minutes
@@ -841,6 +849,7 @@ For an MQTT observer:
 get wifi.status
 get mqtt.enabled
 get mqtt.status
+get mqtt.stats
 get mqtt1.diag
 get mqtt2.diag
 ```
@@ -850,6 +859,17 @@ configured slot's `pub_ack` count in `get mqttN.diag`. Accepted publishes are
 not proof of delivery: a QoS 1 publish is confirmed only when the broker
 acknowledges it. A quiet radio channel alone is not a failed MQTT connection.
 
+Capture these outputs before rebooting, then repeat `mqtt.stats` and the slot
+diagnostics after 60 seconds. `(ok)` reports connection state, not delivery.
+In `mqtt.stats`, `drops=A/B` separates outbox-cap drops from packet publication
+attempts skipped for memory pressure; a growing `B` can explain missing packet
+reports while broker connections still look healthy. `DNS failed (0x8001)`
+records a hostname lookup failure, which can follow an earlier disconnect; it
+does not identify the reason for that first disconnect. `wifi.status` showing
+`suspended` means the shared outage cooldown is active, not that WiFi settings
+were erased. These diagnostics do not reveal passwords or tokens, but WiFi
+status can include the SSID or local IP address. Redact them before posting.
+
 The shared MQTT bridge recovers failed transports on its owning task, including
 errors without a subsequent disconnect callback and connection attempts that
 never finish. It also watches accepted QoS 1 publishes for missing broker

docs/research/firmware_memory_budget.md

@@ -119,19 +119,34 @@ qualification of five simultaneous PSRAM connections or two internal TLS
 connections.
 
 The `mqtt` report also names the production runtime admission thresholds.
-A cold SDK start requires at least 32 KiB of DMA-capable internal free heap
-and an 8 KiB contiguous DMA block. This includes a separate 16 KiB allowance
-for its SDK task stack/TCB, receive/transmit/reassembly buffers and transport
-or configuration objects, alongside the 16 KiB AES reserve. Those ordinary
-SDK allocations occur after admission and do not use the TLS allocation guard.
-A retained initialized SDK handle with a stopped task requires 24 KiB free and
-an 8 KiB contiguous DMA block. Its separate 8 KiB restart allowance covers the
-pinned 6 KiB task stack plus TCB and allocator overhead. SDK stop retains its
-receive/transmit buffers, configuration, transport objects and wrapper
-reassembly buffer, so restarting that handle does not need to reserve their
-first-initialization allocation again. A wrapper whose SDK initialization has
-not completed still receives the full cold allowance. This distinction also
-applies after WiFi loss stops the SDK tasks but keeps their handles.
+Connection admission and the packet-publishing guard share a contiguous-block
+floor: 16,000 bytes with detected PSRAM, or 8,000 bytes without it. Admitting
+another broker while one is connected reserves its startup allocations beside
+that floor, protecting existing packet forwarding. The first connection after
+boot or an outage can bootstrap without that extra reservation, so the
+additional-broker reserve cannot block every disconnected slot's recovery.
+
+A cold SDK start requires at least 32 KiB of DMA-capable internal free heap.
+The separate 16 KiB allowance covers its SDK task stack/TCB,
+receive/transmit/reassembly buffers and transport or configuration objects,
+alongside the 16 KiB AES reserve. Those ordinary allocations occur after
+admission and do not use the TLS allocation guard. Bootstrap needs a contiguous
+16,000-byte block with PSRAM or 8,192 bytes without it. With another connected
+broker, the contiguous minimum includes its publication floor plus the startup
+allowance: 32,384 bytes with PSRAM or 24,384 without it, before the additional
+internal TLS requirement below.
+
+A retained initialized SDK handle with a stopped task requires 24 KiB free.
+Its separate 8 KiB restart allowance covers the pinned 6 KiB task stack plus
+TCB and allocator overhead. Bootstrap uses the same 16,000/8,192-byte block
+minima as a cold start. With another connected broker, its contiguous minimum
+is 24,192 bytes with PSRAM or 16,192 without it, before the internal TLS
+requirement below. SDK stop retains its receive/transmit buffers,
+configuration, transport objects and wrapper reassembly buffer, so restarting
+that handle does not need their first-initialization allocation again. A
+wrapper whose SDK initialization has not completed still receives the full
+cold allowance. This distinction also applies after WiFi loss stops SDK tasks
+but keeps their handles.
 
 All ESP32 clients use 896-byte receive, transmit and wrapper reassembly buffers
 from first initialization. Applying configuration cannot grow the retained SDK
@@ -143,15 +158,22 @@ and cold admission covers the ordinary allocations. WiFi recovery and explicit
 reconfiguration continue to retain clients.
 
 A warm reconnect keeps its running task and buffers; it requires at least
-16 KiB free and a 4 KiB contiguous DMA block. The per-client SDK stack is
-independent of `MQTT_TASK_STACK_SIZE`, which sizes the existing bridge worker.
-A TLS attempt without PSRAM requires at least
-60 KiB free and a 17,408-byte contiguous block. PSRAM allocation fallback must
-leave the 16 KiB DMA reserve available. These thresholds describe free heap at
-the time of admission; they are not additional linked capacity and the TLS
-contiguous threshold is not a startup allocation requirement. Available linked
-internal heap cannot establish how much DMA-capable heap remains or how it is
-fragmented after boot. Only runtime observations can establish that.
+16 KiB free and the same 16,000/8,000-byte publication block. The per-client SDK
+stack is independent of `MQTT_TASK_STACK_SIZE`, which sizes the existing bridge
+worker. TLS without detected PSRAM retains its 60 KiB free-heap and 17,408-byte
+record-block floors for bootstrap. With another connected broker, admission
+reserves that record beside the publication block and ordinary startup
+allocations: 41,792 contiguous bytes for a cold start, 33,600 for a stopped
+task, or 25,408 for a warm reconnect. Both admission and publication use
+detected PSRAM, including a board whose compiled-in PSRAM support failed to
+initialize.
+
+PSRAM allocation fallback must leave the 16 KiB DMA reserve available. These
+conservative thresholds describe free heap at admission; they are not extra
+linked capacity or a guarantee against later fragmentation or allocations by
+other tasks. Available linked internal heap cannot establish how much
+DMA-capable heap remains or how it is fragmented after boot. Only runtime
+observations can establish that.
 
 Maximum-load qualification still needs physical boot and sustained reconnect,
 publish and configuration-change tests with the selected number of brokers,

docs/test-results/mqtt_tls_v4_regression.md

@@ -265,3 +265,51 @@ measured with three live brokers or qualification of maximum simultaneous load.
 No new physical three-hour V4 soak or reporter-side retest has been performed
 as part of this repair. These tests establish the repaired recovery behavior,
 not the cause of the reporter's hardware failure or website-side availability.
+
+## Station G3 follow-up: early failures and network suspension
+
+A user supplied a failed-state console capture from `1.18.0.1` with five WSS
+presets configured. It shows all five slots disconnected and WiFi suspended
+with the 30-minute recovery schedule. The displayed uptime and first-disconnect
+ages place the first two failures roughly 28 and 89 seconds after boot, before
+the reported seven-to-eight-minute loss of reporting.
+
+The slot diagnostics record repeated completed transport recoveries and DNS
+lookup failures (`0x8001`). This establishes active recovery followed by the
+outage cooldown, rather than an indefinitely frozen worker in this capture.
+The DNS errors are the latest failures, not proof of the original disconnect
+cause. The stats also record 345 packet publication attempts skipped for
+memory pressure (`drops=0/345`), an empty SDK outbox and a growing bridge queue.
+These skips are attempts, including retries, rather than 345 distinct packets.
+Heap readings after clients were stopped do not establish their earlier
+connected-state headroom.
+
+Two source issues were identified:
+
+- Connection admission could permit another SDK client below the contiguous
+  memory threshold required by packet publishing. When another slot is
+  connected, the shared admission policy now protects that publishing
+  threshold and accounts for allocations made during SDK startup or restart.
+  It defers additional connections under pressure. First-broker recovery has
+  a separate bootstrap bound, retaining a way to reconnect when all slots
+  have disconnected. Neither path lowers the publishing safety threshold or
+  erases slots; admission bounds do not guarantee future allocations.
+- A new PUBACK followed by DISCONNECTED between worker passes was consumed
+  without crediting the successful delivery to the outage timer. The worker
+  now credits that reply once, even if the connection has since dropped.
+  CONNACK-only flapping and old acknowledgment counts cannot renew the timer.
+
+These repairs do not establish why the user's DNS resolution failed. Physical
+G3 reproduction and a reporter-side retest remain necessary before calling
+that reported failure resolved. The shared changes apply to all ESP32 roles
+that include this MQTT bridge.
+
+Validation passed: 85 MQTT Python host cases, four Companion MQTT host cases
+and 57 firmware RAM policy cases. The new disconnected-PUBACK regression
+fails with the old condition and passes with the fix, including timer wrap,
+disabled slots and one-time acknowledgment consumption. Admission tests use
+the captured 93,024-byte free DMA heap and 20,468-byte largest block: one
+stopped broker can attempt recovery, while an additional broker waits when
+another slot is connected. The Station G3 Full repeater compiled and passed
+capability, dual-OTA partition and linked RAM checks. These are host/build
+checks, not a physical five-broker G3 soak.

scripts/check_firmware_ram.py

@@ -67,6 +67,7 @@ def mqtt_requirements(defines):
     # slots are lazy and each attempt must pass the production admission guard.
     startup = MQTT_STARTUP_ALLOWANCE + max(0, worker_stack - MQTT_WORKER_STACK_BYTES)
     active = min(slots, 5 if psram else 2)
+    publish_contiguous = 16000 if psram else 8000
     potential = startup + max(0, active - 2) * (MQTT_CLIENT_STACK_BYTES + MQTT_CLIENT_BUFFER_BYTES)
     return {
         "budget_scope": "startup minimum with runtime connection admission",
@@ -82,18 +83,28 @@ def mqtt_requirements(defines):
         "maximum_slot_stack_buffer_allowance_bytes": potential,
         "cold_start_allocation_allowance_bytes": 16384,
         "cold_start_dma_admission_free_bytes": 32768,
-        "cold_start_dma_admission_contiguous_bytes": 8192,
+        "cold_start_dma_admission_contiguous_bytes": max(publish_contiguous, 8192),
+        "additional_broker_cold_start_dma_admission_contiguous_bytes": publish_contiguous + 16384,
         "restart_allocation_allowance_bytes": 8192,
         "restart_dma_admission_free_bytes": 24576,
-        "restart_dma_admission_contiguous_bytes": 8192,
+        "restart_dma_admission_contiguous_bytes": max(publish_contiguous, 8192),
+        "additional_broker_restart_dma_admission_contiguous_bytes": publish_contiguous + 8192,
         "dma_admission_free_bytes": 16384,
-        "dma_admission_contiguous_bytes": 4096,
+        "dma_admission_contiguous_bytes": publish_contiguous,
+        "publish_contiguous_bytes": publish_contiguous,
+        "publish_contiguous_without_psram_bytes": 8000,
+        "publish_contiguous_with_psram_bytes": 16000,
         "tls_internal_admission_free_bytes": 61440,
+        "tls_record_allocation_bytes": 17408,
+        "additional_broker_cold_start_tls_internal_admission_contiguous_bytes": 8000 + 16384 + 17408,
+        "additional_broker_restart_tls_internal_admission_contiguous_bytes": 8000 + 8192 + 17408,
         "tls_internal_admission_contiguous_bytes": 17408,
+        "additional_broker_tls_internal_admission_contiguous_bytes": 8000 + 17408,
         "maximum_runtime_load_qualified": False,
         "physical_validation_performed": False,
         "limitations": "Maximum TLS/WSS/outbox workload is not covered by the startup allowance; "
-                       "runtime free DMA heap and fragmentation must satisfy connection admission.",
+                       "runtime free DMA heap and fragmentation must satisfy connection admission; "
+                       "additional brokers reserve publication headroom beside startup allocations.",
     }
 
 

src/helpers/MQTTConnectionAdmission.h

@@ -11,6 +11,11 @@ namespace MQTTConnectionAdmission {
 
 static const size_t kDmaReserveBytes = 16384;
 static const size_t kDmaLargestBytes = 4096;
+// Keep the packet-publishing guard and connection admission on the same floor.
+// A new optional broker must not consume the contiguous block that existing
+// brokers need to continue forwarding packets.
+static const size_t kPsramPublishLargestBytes = 16000;
+static const size_t kInternalPublishLargestBytes = 8000;
 // A stopped SDK client needs a new internal task stack/TCB. First starts also
 // allocate receive/transmit/reassembly buffers and SDK transport/config objects
 // after admission. Budget that ordinary allocation separately from AES headroom.
@@ -33,26 +38,46 @@ enum class ClientMemoryState : uint8_t {
   Started,
 };
 
+static inline size_t publishLargestBytes(bool psram) {
+  return psram ? kPsramPublishLargestBytes : kInternalPublishLargestBytes;
+}
+
+static inline bool hasPublishHeadroom(bool psram, size_t largest) {
+  return largest >= publishLargestBytes(psram);
+}
+
 static inline bool canStart(bool psram, bool tls, ClientMemoryState state,
-                            size_t dma_free, size_t dma_largest) {
+                            size_t dma_free, size_t dma_largest,
+                            bool protect_publish = false) {
   size_t allocation_allowance;
-  size_t largest_required;
+  size_t sdk_largest;
   switch (state) {
     case ClientMemoryState::Uninitialized:
       allocation_allowance = kColdStartAllowanceBytes;
-      largest_required = kColdStartLargestBytes;
+      sdk_largest = kColdStartLargestBytes;
       break;
     case ClientMemoryState::Stopped:
       allocation_allowance = kRestartAllowanceBytes;
-      largest_required = kRestartLargestBytes;
+      sdk_largest = kRestartLargestBytes;
       break;
     case ClientMemoryState::Started:
       allocation_allowance = 0;
-      largest_required = kDmaLargestBytes;
+      sdk_largest = kDmaLargestBytes;
       break;
     default:
       return false;
   }
+  const size_t publication_floor = publishLargestBytes(psram);
+  const size_t tls_record = tls && !psram ? kTlsRecordAllocationBytes : 0;
+  // Bootstrap/recover one broker without requiring spare capacity for another
+  // live connection: retained SDK buffers can fragment the heap after an outage.
+  // With a live peer, conservatively allow startup/TLS allocations to consume
+  // the largest block while leaving existing forwarding's publication floor.
+  // Other tasks and TLS allocations can still change/fragment this snapshot;
+  // these are admission bounds, not an allocation guarantee.
+  size_t largest_required = publication_floor > sdk_largest ? publication_floor : sdk_largest;
+  if (tls_record > largest_required) largest_required = tls_record;
+  if (protect_publish) largest_required = publication_floor + allocation_allowance + tls_record;
   const size_t free_required = kDmaReserveBytes + allocation_allowance;
   if (dma_free < free_required || dma_largest < largest_required) {
     return false;

src/helpers/bridges/MQTTBridge.cpp

@@ -1671,10 +1671,10 @@ bool MQTTBridge::serviceNetworkPower(uint32_t now) {
         && acknowledgments != _network_publish_acks[i];
     _network_publish_acks[i] = acknowledgments;
     // A flapping CONNACK must not buy another full outage window forever.
-    // A PUBACK proves delivery immediately; a quiet connection must stay up
-    // for 30 seconds, leaving time to prove recovery within the probe window.
-    if (_slots[i].enabled && _slots[i].connected
-        && (broker_replied || (now - static_cast<uint32_t>(_slots[i].connected_at_ms)
+    // A PUBACK proves delivery even if DISCONNECTED arrives before this pass.
+    // Consume it once; a quiet connection must still stay up for 30 seconds.
+    if (_slots[i].enabled && (broker_replied || (_slots[i].connected
+        && now - static_cast<uint32_t>(_slots[i].connected_at_ms)
                 >= mesh::wifi::kNetworkHealthyMs))) {
       useful_network = true;
     }
@@ -2381,9 +2381,16 @@ bool MQTTBridge::canStartSlotConnection(int index) const {
       ? ClientMemoryState::Uninitialized
       : (slot.client->isStarted() ? ClientMemoryState::Started : ClientMemoryState::Stopped);
   const uint32_t caps = MALLOC_CAP_INTERNAL | MALLOC_CAP_DMA;
+  bool protect_publish = false;
+  for (int i = 0; i < RUNTIME_MQTT_SLOTS; ++i) {
+    if (i != index && _slots[i].enabled && _slots[i].connected) {
+      protect_publish = true;
+      break;
+    }
+  }
   return MQTTConnectionAdmission::canStart(
       ESP.getPsramSize() > 0, tls, memory_state, heap_caps_get_free_size(caps),
-      heap_caps_get_largest_free_block(caps));
+      heap_caps_get_largest_free_block(caps), protect_publish);
 #else
   return true;
 #endif
@@ -4503,11 +4510,6 @@ bool MQTTBridge::publishPacket(mesh::Packet* packet, bool is_tx,
   // at PUBLISH_JSON_BUFFER_SIZE (2 KB); the threshold includes frame overhead
   // and transient TCP buffers.
   #ifdef ESP32
-  #if defined(BOARD_HAS_PSRAM)
-  static const size_t PUBLISH_SKIP_MAX_ALLOC_THRESHOLD = 16000;
-  #else
-  static const size_t PUBLISH_SKIP_MAX_ALLOC_THRESHOLD = 8000;
-  #endif
   unsigned long now = millis();
   // Re-sample max-alloc at most once per interval and cache the verdict.
   // getMaxAllocHeap() walks the heap free-list, so it must not run per packet.
@@ -4519,10 +4521,13 @@ bool MQTTBridge::publishPacket(mesh::Packet* packet, bool is_tx,
   if (now - _last_memory_check > 5000) {
     _last_memory_check = now;
     size_t max_alloc = ESP.getMaxAllocHeap();
-    _memory_pressure = (max_alloc < PUBLISH_SKIP_MAX_ALLOC_THRESHOLD);
+    // Match connection admission's detected-PSRAM decision, including a board
+    // whose optional PSRAM failed initialization.
+    const bool psram = ESP.getPsramSize() > 0;
+    _memory_pressure = !MQTTConnectionAdmission::hasPublishHeadroom(psram, max_alloc);
     if (_memory_pressure) {
       MQTT_DEBUG_PRINTLN("MQTT: memory pressure, skipping publishes (Max alloc: %d, threshold: %d, skipped: %d)",
-                         (int)max_alloc, (int)PUBLISH_SKIP_MAX_ALLOC_THRESHOLD, _skipped_publishes);
+                         (int)max_alloc, (int)MQTTConnectionAdmission::publishLargestBytes(psram), _skipped_publishes);
     }
   }
   if (_memory_pressure) {

test/test_firmware_ram.py

@@ -607,10 +607,12 @@ int main() {
                 self.assertEqual(profile["maximum_slot_stack_buffer_allowance_bytes"], potential)
                 self.assertEqual(profile["cold_start_allocation_allowance_bytes"], 16384)
                 self.assertEqual(profile["cold_start_dma_admission_free_bytes"], 32768)
-                self.assertEqual(profile["cold_start_dma_admission_contiguous_bytes"], 8192)
+                self.assertEqual(profile["cold_start_dma_admission_contiguous_bytes"],
+                                 16000 if active == 5 else 8192)
                 self.assertEqual(profile["restart_allocation_allowance_bytes"], 8192)
                 self.assertEqual(profile["restart_dma_admission_free_bytes"], 24576)
-                self.assertEqual(profile["restart_dma_admission_contiguous_bytes"], 8192)
+                self.assertEqual(profile["restart_dma_admission_contiguous_bytes"],
+                                 16000 if active == 5 else 8192)
                 self.assertFalse(profile["maximum_runtime_load_qualified"])
                 self.assertFalse(profile["physical_validation_performed"])
         ordinary = ram.requirements("ESP32_PLATFORM", {"BOARD_HAS_PSRAM": 1}, "v4_companion")
@@ -707,16 +709,31 @@ int main() {
             ("client_task_stack_bytes", "kClientTaskStackBytes"),
             ("client_task_priority", "kClientTaskPriority"),
             ("cold_start_allocation_allowance_bytes", "kColdStartAllowanceBytes"),
-            ("cold_start_dma_admission_contiguous_bytes", "kColdStartLargestBytes"),
             ("restart_allocation_allowance_bytes", "kRestartAllowanceBytes"),
-            ("restart_dma_admission_contiguous_bytes", "kRestartLargestBytes"),
             ("dma_admission_free_bytes", "kDmaReserveBytes"),
-            ("dma_admission_contiguous_bytes", "kDmaLargestBytes"),
+            ("publish_contiguous_without_psram_bytes", "kInternalPublishLargestBytes"),
+            ("publish_contiguous_with_psram_bytes", "kPsramPublishLargestBytes"),
             ("tls_internal_admission_free_bytes", "kTlsInternalFreeBytes"),
-            ("tls_internal_admission_contiguous_bytes", "kTlsRecordAllocationBytes"),
+            ("tls_record_allocation_bytes", "kTlsRecordAllocationBytes"),
         ):
             with self.subTest(key=key):
                 self.assertIn(f"{name} = {profile[key]};", source)
+        for psram in (False, True):
+            flags = {"WITH_MQTT_BRIDGE": 1}
+            if psram:
+                flags["BOARD_HAS_PSRAM"] = 1
+            actual = ram.requirements("ESP32_PLATFORM", flags, "v4_companion")["mqtt"]
+            floor = 16000 if psram else 8000
+            self.assertEqual(actual["publish_contiguous_bytes"], floor)
+            self.assertEqual(actual["dma_admission_contiguous_bytes"], floor)
+            self.assertEqual(actual["cold_start_dma_admission_contiguous_bytes"], max(floor, 8192))
+            self.assertEqual(actual["additional_broker_cold_start_dma_admission_contiguous_bytes"], floor + 16384)
+            self.assertEqual(actual["restart_dma_admission_contiguous_bytes"], max(floor, 8192))
+            self.assertEqual(actual["additional_broker_restart_dma_admission_contiguous_bytes"], floor + 8192)
+            self.assertEqual(actual["additional_broker_cold_start_tls_internal_admission_contiguous_bytes"], 41792)
+            self.assertEqual(actual["additional_broker_restart_tls_internal_admission_contiguous_bytes"], 33600)
+            self.assertEqual(actual["tls_internal_admission_contiguous_bytes"], 17408)
+            self.assertEqual(actual["additional_broker_tls_internal_admission_contiguous_bytes"], 25408)
         self.assertEqual(profile["cold_start_dma_admission_free_bytes"],
                          profile["dma_admission_free_bytes"] + profile["cold_start_allocation_allowance_bytes"])
         self.assertEqual(profile["restart_dma_admission_free_bytes"],

test/test_mqtt_connection_admission.py

@@ -14,6 +14,7 @@ POLICY_PREAMBLE = r'''
 #include <cassert>
 #include <cstddef>
 #include <cstdint>
+#include <initializer_list>
 #include "helpers/MQTTConnectionAdmission.h"
 #include "helpers/MQTTTransportHealth.h"
 namespace Admission = MQTTConnectionAdmission;
@@ -62,6 +63,7 @@ struct Client {
 };
 struct MQTTBridge {
   struct MQTTSlot {
+    bool enabled = true, connected = false;
     Client* client = nullptr;
     const Preset* preset = nullptr;
     char broker_uri[128] = {}, host[64] = "broker";
@@ -97,46 +99,112 @@ class MqttConnectionAdmissionTest(unittest.TestCase):
         self.compile_run(POLICY_PREAMBLE + r'''
 int main() {
   // A large aggregate heap cannot compensate for a fragmented DMA region.
-  assert(!Admission::canStart(true, true, MemoryState::Started, 100000, 4095));
-  assert(!Admission::canStart(true, true, MemoryState::Started, 16383, 20000));
-  assert(Admission::canStart(true, true, MemoryState::Started, 16384, 4096));
-  assert(Admission::canStart(false, false, MemoryState::Started, 16384, 4096));
-  assert(!Admission::canStart(false, false, MemoryState::Started, 16383, 4096));
-  assert(!Admission::canStart(false, false, MemoryState::Started, 16384, 4095));
+  assert(!Admission::canStart(true, true, MemoryState::Started, 100000, 15999, true));
+  assert(!Admission::canStart(true, true, MemoryState::Started, 16383, 16000, true));
+  assert(Admission::canStart(true, true, MemoryState::Started, 16384, 16000, true));
+  assert(Admission::canStart(false, false, MemoryState::Started, 16384, 8000, true));
+  assert(!Admission::canStart(false, false, MemoryState::Started, 16383, 8000, true));
+  assert(!Admission::canStart(false, false, MemoryState::Started, 16384, 7999, true));
 
-  // Without PSRAM, TLS also needs a full record allocation and session budget.
+  // Internal TLS must leave a publication block beside its new TLS record and
+  // ordinary SDK startup allocations, even for a retained running task.
   const MemoryState states[] = {MemoryState::Uninitialized, MemoryState::Stopped,
                                 MemoryState::Started};
-  for (MemoryState state : states) {
-    assert(!Admission::canStart(false, true, state, 61439, 20000));
-    assert(!Admission::canStart(false, true, state, 100000, 17407));
-    assert(Admission::canStart(false, true, state, 61440, 17408));
+  const size_t internal_tls_largest[] = {41792, 33600, 25408};
+  for (size_t i = 0; i < 3; ++i) {
+    assert(!Admission::canStart(false, true, states[i], 61439, 100000, true));
+    assert(!Admission::canStart(false, true, states[i], 100000, internal_tls_largest[i] - 1, true));
+    assert(Admission::canStart(false, true, states[i], 61440, internal_tls_largest[i], true));
   }
-  assert(Admission::canStart(true, false, MemoryState::Started, 16384, 4096));
-  assert(!Admission::canStart(true, false, static_cast<MemoryState>(255), SIZE_MAX, SIZE_MAX));
+  assert(Admission::canStart(true, false, MemoryState::Started, 16384, 16000, true));
+  assert(!Admission::canStart(true, false, static_cast<MemoryState>(255), SIZE_MAX, SIZE_MAX, true));
 
-  // A new SDK task needs its stack and ordinary startup allocations in
-  // addition to the DMA reserve, including plaintext and PSRAM-backed TLS.
   assert(Admission::kClientTaskStackBytes == 6144);
   assert(Admission::kClientTaskPriority == 5);
   assert(Admission::kColdStartAllowanceBytes == 16384);
-  assert(Admission::kColdStartLargestBytes == 8192);
   assert(Admission::kRestartAllowanceBytes == 8192);
-  assert(Admission::kRestartLargestBytes == 8192);
   for (int psram = 0; psram <= 1; ++psram) {
+    const size_t floor = psram ? 16000 : 8000;
     for (int tls = 0; tls <= 1; ++tls) {
       if (!psram && tls) continue;  // Larger TLS thresholds checked above.
-      assert(!Admission::canStart(psram, tls, MemoryState::Uninitialized, 32767, 20000));
-      assert(!Admission::canStart(psram, tls, MemoryState::Uninitialized, 100000, 8191));
-      assert(Admission::canStart(psram, tls, MemoryState::Uninitialized, 32768, 8192));
-      // An initialized handle retains its ordinary SDK buffers across a stop.
-      // Only the replacement task/TCB is charged beside the DMA reserve.
-      assert(!Admission::canStart(psram, tls, MemoryState::Stopped, 24575, 20000));
-      assert(!Admission::canStart(psram, tls, MemoryState::Stopped, 100000, 8191));
-      assert(Admission::canStart(psram, tls, MemoryState::Stopped, 24576, 8192));
+      assert(!Admission::canStart(psram, tls, MemoryState::Uninitialized, 32767, 100000, true));
+      assert(!Admission::canStart(psram, tls, MemoryState::Uninitialized, 100000, floor + 16383, true));
+      assert(Admission::canStart(psram, tls, MemoryState::Uninitialized, 32768, floor + 16384, true));
+      // An initialized handle retains its buffers, but recreates its task.
+      assert(!Admission::canStart(psram, tls, MemoryState::Stopped, 24575, 100000, true));
+      assert(!Admission::canStart(psram, tls, MemoryState::Stopped, 100000, floor + 8191, true));
+      assert(Admission::canStart(psram, tls, MemoryState::Stopped, 24576, floor + 8192, true));
     }
   }
 }
+''')
+
+    def test_additional_broker_cannot_spend_live_packet_publishing_headroom(self):
+        self.compile_run(POLICY_PREAMBLE + r'''
+int main() {
+  for (bool psram : {false, true}) {
+    const size_t floor = psram ? 16000 : 8000;
+    assert(!Admission::hasPublishHeadroom(psram, floor - 1));
+    assert(Admission::hasPublishHeadroom(psram, floor));
+    // Previously admitted: 8 KiB largest was enough to start another client,
+    // even though publishing on a PSRAM board was already suppressed there.
+    assert(!Admission::canStart(psram, false, MemoryState::Uninitialized, 100000, 8192, true));
+    assert(!Admission::canStart(psram, true, MemoryState::Stopped, 100000, 8192, true));
+    const size_t allowances[] = {16384, 8192, 0};
+    const MemoryState states[] = {MemoryState::Uninitialized, MemoryState::Stopped,
+                                  MemoryState::Started};
+    for (size_t i = 0; i < 3; ++i) {
+      const size_t minimum = floor + allowances[i];
+      assert(!Admission::canStart(psram, false, states[i], 100000, minimum - 1, true));
+      assert(Admission::canStart(psram, false, states[i], 100000, minimum, true));
+      // Spending the complete bounded SDK startup allowance still leaves a
+      // block accepted by the exact production packet-publishing policy.
+      assert(Admission::hasPublishHeadroom(psram, minimum - allowances[i]));
+    }
+  }
+}
+''')
+
+    def test_actual_packet_publish_guard_uses_detected_psram_and_throttles_sampling(self):
+        source = (ROOT / "src/helpers/bridges/MQTTBridge.cpp").read_text()
+        start = source.index("  #ifdef ESP32\n", source.index("bool MQTTBridge::publishPacket("))
+        end = source.index("  #endif\n", start) + len("  #endif\n")
+        guard = source[start:end]
+        self.compile_run(POLICY_PREAMBLE + r'''
+#define ESP32 1
+#define BOARD_HAS_PSRAM 1
+#define MQTT_DEBUG_PRINTLN(...) ((void)0)
+static unsigned long now_ms = 6000;
+unsigned long millis() { return now_ms; }
+struct {
+  size_t psram = 2097152, largest = 16000, queries = 0;
+  size_t getPsramSize() const { return psram; }
+  size_t getMaxAllocHeap() { ++queries; return largest; }
+} ESP;
+struct Bridge {
+  unsigned long _last_memory_check = 0;
+  bool _memory_pressure = false;
+  int _skipped_publishes = 0;
+  bool publish() {
+''' + guard + r'''
+    return true;
+  }
+};
+int main() {
+  Bridge bridge;
+  assert(bridge.publish() && ESP.queries == 1);
+  ESP.largest = 15999;
+  now_ms += 5001;
+  assert(!bridge.publish() && bridge._skipped_publishes == 1 && ESP.queries == 2);
+  assert(!bridge.publish() && bridge._skipped_publishes == 2 && ESP.queries == 2);
+  now_ms += 5001; ESP.largest = 16000;
+  assert(bridge.publish() && ESP.queries == 3);
+  // Compiling BOARD_HAS_PSRAM does not establish that PSRAM initialized.
+  now_ms += 5001; ESP.psram = 0; ESP.largest = 8000;
+  assert(bridge.publish() && ESP.queries == 4);
+  now_ms += 5001; ESP.largest = 7999;
+  assert(!bridge.publish() && bridge._skipped_publishes == 3);
+}
 ''')
 
     def test_internal_fallback_preserves_dma_reserve_and_rejects_fragmentation(self):
@@ -292,35 +360,35 @@ int main() {
   assert(!bridge.canStartSlotConnection(0));
   dma_free = 100000; dma_largest = 17407;
   assert(!bridge.canStartSlotConnection(0));
-  ESP.size = 2097152; dma_free = 16384; dma_largest = 4096;
+  ESP.size = 2097152; dma_free = 16384; dma_largest = 16000;
   assert(!bridge.canStartSlotConnection(0));
   Client client;
   bridge._slots[0].client = &client;
   assert(!bridge.canStartSlotConnection(0));  // Wrapper allocated; SDK not initialized.
-  dma_free = 32767; dma_largest = 20000;
+  dma_free = 32767; dma_largest = 16000;
   assert(!bridge.canStartSlotConnection(0));
-  dma_free = 32768; dma_largest = 8191;
+  dma_free = 32768; dma_largest = 15999;
   assert(!bridge.canStartSlotConnection(0));
-  dma_largest = 8192;
+  dma_largest = 16000;
   assert(bridge.canStartSlotConnection(0));
 
   // An initialized but stopped handle needs its task again, not new SDK buffers.
   client.initialized = true; dma_free = 24575; dma_largest = 20000;
   assert(!bridge.canStartSlotConnection(0));
-  dma_free = 24576; dma_largest = 8191;
+  dma_free = 24576; dma_largest = 15999;
   assert(!bridge.canStartSlotConnection(0));
-  dma_largest = 8192;
+  dma_largest = 16000;
   assert(bridge.canStartSlotConnection(0));
 
   // A warm reconnect retains the SDK task/buffers and uses the smaller floor.
-  client.started = true; dma_free = 16384; dma_largest = 4096;
+  client.started = true; dma_free = 16384; dma_largest = 16000;
   assert(bridge.canStartSlotConnection(0));
-  dma_largest = 4095;
+  dma_largest = 15999;
   assert(!bridge.canStartSlotConnection(0));
 
   // An explicit plaintext URI overrides a TLS port. The effective URI wins
   // over the raw host; a preset's URI wins over both custom fields.
-  ESP.size = 0; dma_free = 16384; dma_largest = 4096;
+  ESP.size = 0; dma_free = 16384; dma_largest = 16000;
   strcpy(bridge._slots[0].host, "mqtt://broker:8883");
   assert(bridge.canStartSlotConnection(0));
   strcpy(bridge._slots[0].broker_uri, "wss://broker/mqtt");
@@ -334,6 +402,32 @@ int main() {
   ESP.size = 2097152;
   assert(bridge.canStartSlotConnection(0));
 
+  // Reproduce the reporter's failed-state heap after all SDK tasks stopped.
+  // One broker must be able to recover instead of waiting forever for a
+  // contiguous 24,192-byte block pinned apart by retained SDK buffers.
+  client.started = false;
+  dma_free = 93024; dma_largest = 20468;
+  assert(bridge.canStartSlotConnection(0));
+  // Once another broker is connected, an additional task must leave its
+  // publication headroom beside the restart allowance.
+  bridge._slots[1].connected = true;
+  assert(!bridge.canStartSlotConnection(0));
+  dma_largest = 24191;
+  assert(!bridge.canStartSlotConnection(0));
+  dma_largest = 24192;
+  assert(bridge.canStartSlotConnection(0));
+  // Health reconciliation removing the other failed connection restores
+  // bootstrap admission. The target never counts itself as a live peer.
+  dma_largest = 20468;
+  bridge._slots[1].connected = false;
+  bridge._slots[0].connected = true;
+  assert(bridge.canStartSlotConnection(0));
+  bridge._slots[0].connected = false;
+  bridge._slots[1].enabled = false;
+  bridge._slots[1].connected = true;
+  assert(bridge.canStartSlotConnection(0));
+  bridge._slots[1].connected = false;
+
   // Every pending TLS attempt, including the target's own attempt, blocks all
   // other starts until an event callback clears it. No heap walk is needed.
   for (int pending = 0; pending < RUNTIME_MQTT_SLOTS; ++pending) {

test/test_mqtt_handshake_scheduler.py

@@ -54,7 +54,7 @@ struct esp_mqtt_error_codes {
 constexpr uint32_t MALLOC_CAP_INTERNAL = 1, MALLOC_CAP_DMA = 2;
 static uint32_t clock_ms = 1000;
 static unsigned long wall_seconds = 1800000000;
-static size_t dma_free = 100000, dma_largest = 20000;
+static size_t dma_free = 100000, dma_largest = 100000;
 static unsigned long s_wifi_connected_at = 0;
 uint32_t millis() { return clock_ms; }
 unsigned long fake_time(void*) { return wall_seconds; }
@@ -155,7 +155,7 @@ struct PsychicMqttClient {
   void softDisconnect() {
     ++bounces; live = false;
     // Releasing the old internal TLS session creates room for its replacement.
-    dma_free += 44000; dma_largest = 20000;
+    dma_free += 44000; dma_largest = 50000;
   }
   void setCredentials(const char*, const char*) { ++credentials; }
 };
@@ -421,11 +421,11 @@ int main() {
   bridge.maintainSlotConnections();
   assert(bridge.setups.empty() && bridge._slots[0].last_reconnect_attempt == 0);
   assert(bridge._slots[0].enabled && bridge.activatedSlotCount() == 0);
-  dma_free = 32768; dma_largest = 8191;
+  dma_free = 32768; dma_largest = 15999;
   bridge.maintainSlotConnections();
   assert(bridge.setups.empty() && bridge._slots[0].last_reconnect_attempt == 0);
   assert(!bridge.hasPendingSlotConnection() && bridge.activatedSlotCount() == 0);
-  dma_largest = 8192;
+  dma_largest = 16000;
   bridge.maintainSlotConnections();
   assert(bridge.setups == std::vector<int>{0} && bridge.activatedSlotCount() == 1);
   assert(bridge.clients[0].starts == 1 && bridge.clients[0].reconnects == 0);
@@ -438,7 +438,7 @@ int main() {
   MQTTBridge bridge;
   for (int i = 2; i < RUNTIME_MQTT_SLOTS; ++i) bridge._slots[i].enabled = false;
   clock_ms = 70000;
-  dma_free = 16384; dma_largest = 4096;
+  dma_free = 16384; dma_largest = 16000;
   auto& stopped = bridge._slots[0];
   auto& retained = bridge._slots[1];
   stopped.client = &bridge.clients[0];
@@ -462,7 +462,7 @@ int main() {
   // with no extra delay introduced by the earlier memory-only deferral.
   bridge.complete(1);
   clock_ms += 15000;
-  dma_free = 24576; dma_largest = 8192;
+  dma_free = 24576; dma_largest = 24192;
   bridge.maintainSlotConnections();
   assert(stopped.client->starts == 1 && stopped.client->reconnects == 0);
   assert(stopped.client->isStarted() && bridge._slot_attempt_pending[0]);
@@ -536,14 +536,14 @@ int main() {
   // headroom is inadequate. Neither failure consumes retry/backoff state.
   dma_free = 24575;
   bridge.maintainSlotConnections();
-  dma_free = 24576; dma_largest = 8191;
+  dma_free = 24576; dma_largest = 15999;
   bridge.maintainSlotConnections();
   for (int i = 0; i < 3; ++i) {
     assert(bridge.clients[i].starts == 0 && bridge._slots[i].last_reconnect_attempt == 0);
     assert(bridge._slots[i].reconnect_backoff == 0 && bridge._slots[i].start_failures == 0);
   }
 
-  dma_free = 42376; dma_largest = 22516;
+  dma_free = 42376; dma_largest = 24192;
   bridge.maintainSlotConnections();
   assert(bridge.clients[0].starts == 1 && dma_free == 35840);
   assert(bridge.hasPendingSlotConnection());
@@ -837,10 +837,10 @@ int main() {
   bridge.maintainSlotConnections();
   assert(slot.client->starts == 0 && slot.reconnect_backoff == 3);
   assert(slot.last_reconnect_attempt == 70000 && slot.start_failures == 0);
-  dma_free = 24576; dma_largest = 8191;
+  dma_free = 24576; dma_largest = 15999;
   bridge.maintainSlotConnections();
   assert(slot.client->starts == 0 && slot.reconnect_backoff == 3);
-  dma_largest = 8192;
+  dma_largest = 16000;
   bridge.maintainSlotConnections();
   assert(slot.client->starts == 1 && bridge._slot_attempt_pending[0]);
   assert(slot.reconnect_backoff == 4 && slot.max_backoff_failures == 1);

test/test_mqtt_network_timeout.py

@@ -104,7 +104,7 @@ RECONFIGURE_PREAMBLE = r'''
 #define pdMS_TO_TICKS(value) (value)
 constexpr int RUNTIME_MQTT_SLOTS = 2, MQTT_AUTH_JWT = 1;
 constexpr uint32_t MALLOC_CAP_INTERNAL = 1, MALLOC_CAP_DMA = 2;
-static size_t dma_free = 100000, dma_largest = 20000;
+static size_t dma_free = 100000, dma_largest = 100000;
 size_t heap_caps_get_free_size(uint32_t caps) {
   assert(caps == (MALLOC_CAP_INTERNAL | MALLOC_CAP_DMA)); return dma_free;
 }
@@ -228,7 +228,7 @@ int main() {
   bridge.applySlotPreset(0, "plain");
   assert(slot.client && !slot.client->isStarted() && !slot.initial_connect_done);
   assert(init_calls == 1 && destroy_calls == 0);
-  dma_free = 24576;
+  dma_free = 24576; dma_largest = 24192;
   assert(bridge.setupSlot(0));
   assert(slot.client->_buffer_capacity == 896 && init_calls == 1);
   const int destructions_before_guard = destroy_calls;

test/test_mqtt_wifi_network_power.py

@@ -612,6 +612,41 @@ static void publishAcknowledgmentProvesRecoveryImmediately() {
           "an old PUBACK repeatedly renewed a disconnected broker");
 }
 
+static void disconnectedPublishAcknowledgmentRenewsGraceOnce() {
+  for (uint32_t start : {100U, UINT32_MAX - 10000U}) {
+    resetEnvironment();
+    MQTTBridge bridge;
+    require(bridge.serviceNetworkPower(start), "initial outage did not run");
+    require(!bridge.serviceNetworkPower(start + grace), "outage did not suspend");
+    const uint32_t resumed = start + grace + interval;
+    require(bridge.serviceNetworkPower(resumed), "probe did not resume");
+    const uint32_t replied = resumed + probe - 1U;
+    // The SDK delivered PUBACK and then DISCONNECTED between owner passes.
+    // A real broker reply still proves service; a CONNACK by itself does not.
+    bridge.clients[0].publish_acks = 1U;
+    require(!bridge._slots[0].connected, "fixture unexpectedly connected");
+    require(bridge.serviceNetworkPower(replied), "new PUBACK was discarded");
+    require(bridge.serviceNetworkPower(resumed + probe),
+            "disconnect hid PUBACK and retained the expired probe deadline");
+    require(bridge.serviceNetworkPower(replied + grace - 1U),
+            "disconnected PUBACK did not renew normal grace");
+    require(!bridge.serviceNetworkPower(replied + grace),
+            "unchanged PUBACK kept a failed service awake forever");
+  }
+  resetEnvironment();
+  {
+    MQTTBridge bridge;
+    bridge.serviceNetworkPower(100U);
+    bridge._slots[0].enabled = false;
+    bridge.clients[0].publish_acks = 1U;
+    require(!bridge.serviceNetworkPower(100U + grace),
+            "a disabled broker renewed the outage budget");
+    bridge._slots[0].enabled = true;
+    require(!bridge.serviceNetworkPower(101U + grace),
+            "enabling a broker replayed its already consumed PUBACK");
+  }
+}
+
 static void quietConnectionHasExactThirtySecondHealthBoundary() {
   resetEnvironment();
   {
@@ -743,6 +778,7 @@ int main(int argc, char** argv) {
     else if (name == "channel-repair") deferredEspNowChannelRepair();
     else if (name == "healthy") stableBrokerCountsButFlappingDoesNot();
     else if (name == "puback") publishAcknowledgmentProvesRecoveryImmediately();
+    else if (name == "disconnected-puback") disconnectedPublishAcknowledgmentRenewsGraceOnce();
     else if (name == "quiet-boundary") quietConnectionHasExactThirtySecondHealthBoundary();
     else if (name == "ntp-retry") sntpFailureAndBusyLeaseRetries();
     else if (name == "ntp-rotation") everyNtpServerGetsTheLeadingTimeoutSlot();
@@ -891,6 +927,9 @@ class MqttWiFiNetworkPowerTests(unittest.TestCase):
     def test_publish_ack_proves_recovery_immediately_and_is_consumed_once(self):
         self.run_case("puback")
 
+    def test_puback_followed_by_disconnect_renews_grace_once_across_wrap(self):
+        self.run_case("disconnected-puback")
+
     def test_quiet_broker_health_requires_exactly_thirty_seconds(self):
         self.run_case("quiet-boundary")