Files
MeshCore/MQTT_OWNERSHIP.md
T

192 lines
10 KiB
Markdown

# MQTT Bridge Cross-Core Ownership Model
> **Historical design record, status reconciled 2026-08-18.** The ownership
> analysis was written during Phases 4-5. File/line references describe that
> snapshot and must be re-located before editing current code. Cooperative
> shutdown and the OTA barrier are implemented and hardware-characterized;
> immutable status publication and replacement of the remaining volatile
> NTP/reconfigure handshakes are still deferred.
This document is the Phase 4 deliverable from `STABILITY_TESTABILITY_HANDOFF.md`:
it records **one owner for each mutable runtime domain**, maps every place a
non-owner reads owned state across cores today, and states the target primitive
for each. It is paired with the fork-owned lifecycle test seam
(`src/helpers/MQTTLifecycle.h`, `test/test_mqtt_lifecycle/`).
**Status:** ownership model documented and the lifecycle/teardown test seam
landed (Phase 4). **Phase 5 (branch `phase5/cooperative-mqtt-shutdown`) has now
implemented the cooperative shutdown and the OTA barrier -- hazard Section 4 below.**
Still **deferred** (carried to Phase 5b / Phase 6): publishing a plain-data
snapshot and repointing the Section 1/Section 2 consumers, and replacing the Section 3 `volatile`
handshakes with a command channel. This document is a design record; Sections
1-3 describe hazards that still exist conceptually, while Section 4 is resolved.
Line
references are against the tree at the time of writing and should be re-verified
before editing.
## Execution contexts
- **MQTT task -- Core 0** (`"MQTTBridge"`, `xTaskCreatePinnedToCore(..., MQTT_TASK_CORE=0)`,
entry `mqttTask` -> `mqttTaskLoop`, `MQTTBridge.cpp:923`, `:991`). Owns all
WiFi / MQTT / NTP I/O and every mutation of `_slots[]`, connection state, and
NTP state.
- **Loop task -- Core 1** (`MyMesh::loop()`). Runs the CLI, WebConfig `tick()`,
and `AlertReporter::onLoop()`. `examples/simple_repeater/MyMesh.cpp:1529`
notes the bridge loop is *not* called here -- it lives on Core 0.
- **Producer / radio context.** Stages raw radio bytes before queue handoff
(`storeRawRadioData` -> `_staged_*`, consumed by `queuePacket`; both Core 1, in
guaranteed sequence, `MQTTBridge.h:232-239`).
- **Async TCP context.** WebConfig request parsing and immutable response
handoff only.
## Ownership model (one owner per mutable domain)
| Domain | Owner | Notes |
|--------|-------|-------|
| MQTT clients, slot connection state, publish counters, packet drain, NTP I/O | **MQTT task (Core 0)** | `_slots[]`, `PsychicMqttClient`s, `_ntp_client`, `_last_raw_*` |
| CLI execution, preference persistence, WebConfig batch draining, bridge lifecycle requests | **Loop task (Core 1)** | issues start/stop/reconfigure, drains the WebConfig batch |
| Packet staging before queue handoff | **Producer / radio (Core 1)** | `_staged_*`, no lock needed (sequential) |
| Request parse + immutable response | **Async TCP** | must not touch mutable bridge state |
The rule that follows: **loop/WebConfig/CLI/AlertReporter code must not directly
inspect mutable MQTT slot objects or client counters.** The MQTT task must
publish a plain-data snapshot they can read instead.
## Remaining cross-core hazards after the minimal Phase 5 change
All of the following run on **Core 1** and read state mutated by **Core 0**
without a lock or a published snapshot.
### 1. Diagnostic reads of live `_slots[]` via the singleton
Four `static` accessors reach the live object through the file-scope
`s_mqtt_bridge_instance` (`MQTTBridge.cpp:201`, set at `begin()` end `:849`,
nulled first in `end()` `:858` -- a plain, non-atomic pointer):
- `getSlotStatusSnapshot` (`:299`) -- despite its name, built on demand from
live `_slots[slot_index]`. **Refined premise:** the returned `name`/`state`
`const char*`s point at static rodata (string literals / `MQTT_PRESETS[]`),
so they are *not* dangling. The real hazards are (a) reading the
`slot.preset` **pointer value**, which Core 0 can null/reassign mid-read
(`applySlotPreset`, `teardownSlot`), and (b) `slot.client->getPublishOk()`,
a live client pointer Core 0 can `delete` during teardown (UAF window).
- `formatMqttStatusReply` (`:207`), `formatMqttStatsReply` (`:263`),
`formatSlotDiagReply` (`:397`) -- same singleton + live `_slots[]` reads.
Consumers: CLI (`CommonCLI_Observer.cpp:726, :728, :791`) and the app-layer
stats JSON (`examples/simple_repeater/MyMesh.cpp:1423`, mirrored in
`examples/simple_room_server/MyMesh.cpp:1050`). **Refined premise:**
`WebConfigServer.cpp` itself reads only a compile-time constant
(`getMaxActiveSlots`, `:479`); the live-bridge web reads are in `buildStatsJson`
in the `MyMesh.cpp` app layer.
### 2. Instance reads that survive `end()`
`AlertReporter` (Core 1, `onLoop`, `AlertReporter.cpp:208`) reads live `_slots[]`
via instance methods it reaches through its own `_bridge` pointer
(`AlertReporter.cpp:281` `isSlotEnabledAndAttempted`, `:285`
`getSlotCurrentOutageStartMs`, `:296`/`:311` `getSlotPresetName`). The stats
JSON also calls `bridge->getQueueSize()` (`MyMesh.cpp:1419`).
**Refined premise (teardown hazard):** `end()` nulls only
`s_mqtt_bridge_instance`. The app's `bridge` pointer and `AlertReporter::_bridge`
are **not** cleared by `end()`, so these instance reads can touch a torn-down
bridge. (`getConnectedBrokers()` at `MQTTBridge.cpp:3676` is defined but has
zero consumers.)
### 3. `volatile` cross-core handshakes
Plain `volatile` flags (no atomics/barriers), Core 1 sets the request, Core 0
clears/processes and writes a "done" flag last, Core 1 spins:
- `_slot_reconfigure_pending[]` -- set `MQTTBridge.cpp:2057` (Core 1), read/clear
`:1118` (Core 0).
- `_ntp_force_{requested,done,result}` -- request `:3298`, process `:1063-1068`,
spin-wait `:3308-3315`.
- `_ntp_diag_{requested,done}` (+ non-volatile `_ntp_diag_results[]`) -- request
`:3344`, process `:1073-1076`, spin-wait `:3347`.
The two blocking waiters (`requestForcedNtpSync` `:3298`, `ntpDiag` `:3344`) spin
on Core 1 while `end()` could tear down the singleton/task concurrently.
### 4. Abrupt teardown, no start guard -- RESOLVED in Phase 5
Original hazard (retained for context): `end()` used
`vTaskDelete(_mqtt_task_handle)` to kill the task wherever it was (possibly
mid-`_slots[]` mutation or inside mbedTLS), then ran slot/client cleanup *after*
deletion on the caller's context -- the OTA teardown heap-panic path. `begin()`
had no double-call guard, and lifecycle state was a single `_initialized` bool.
**Phase 5 resolution** (branch `phase5/cooperative-mqtt-shutdown`):
- `end()` now requests a cooperative stop through `MQTTLifecycle::Coordinator`.
The MQTT task (Core 0) tears down its own clients where the mbedTLS contexts
live, acknowledges via `_stop_acked`, and self-terminates; `end()` waits for
the ack before freeing the queue/buffers. The blind `vTaskDelete` survives
only as the bounded-timeout fallback, which sets a dirty latch that withholds
OTA flashing (`canFlashAfterStop()`).
- `begin()` has an idempotent double-call guard and drives the Coordinator to
`Running`; the lifecycle state now lives in the tested state machine, not a
bare bool.
- The OTA barrier gates `simple_repeater`'s deferred flash on a clean stop.
The shutdown path and timeout were subsequently hardware-characterized. The
flat 8-second placeholder was replaced by a per-stop budget of 5 seconds plus
8 seconds per enabled slot. A two-WSS-slot stop that previously timed out
acknowledged cleanly in about 11.7 seconds within its 21-second budget. The
residual Section 1/Section 2 instance-pointer reads remain deferred, so those
consumers can still touch live or torn-down bridge state as tracked above.
## Remaining target primitives
- **Task notifications or a command queue** for one-way lifecycle / reconfigure
/ NTP requests -- replacing every `volatile` flag in Section 3.
- **Immutable published snapshots** for WebConfig, CLI diagnostics, and alerting
-- the MQTT task publishes a plain-data `SlotStatusSnapshot` (owned char
buffers + scalars, no live pointers) that Section 1/Section 2 consumers read. This also
removes the "instance pointer survives `end()`" hazard because consumers stop
dereferencing the live bridge.
- **Atomics** only for truly independent scalar state.
- **A mutex** only where ownership transfer or snapshot publication cannot
express the operation cleanly.
## Lifecycle contract (the test seam)
`src/helpers/MQTTLifecycle.h` encodes the cooperative lifecycle implemented by
Phase 5 as a pure state machine plus a narrow injected `Ops` seam
(clock / task control / resource owner / OTA barrier). The invariants proven by
`test/test_mqtt_lifecycle/` and preserved by the production wiring are:
- `Stopped -> Starting -> Running -> StopRequested -> Stopping -> Stopped`.
- Idempotent start and stop; safe restart only from `Stopped` (`mayRestart`).
- New connects/publishes/retries/reconfigurations cease once a stop is requested
(`acceptsNewWork`).
- Resources are released **only** after a stop acknowledgment (or the reviewed
timeout fallback) -- never mid-run.
- A late/stale callback consults `mayTouchOwnedState()` and is a no-op once the
owner has released resources.
- Bounded stop timeout -> reviewed fallback (models replacing the abrupt
`vTaskDelete`).
- **OTA barrier:** `mayBeginFlash()` is true only after a **clean** stop
acknowledgment; a timed-out stop leaves flashing blocked so OTA aborts rather
than writing under uncertain ownership.
### Hardware characterization result
WSS teardown was measured as roughly 5-6 seconds per slot and is sequential.
Production now sets the coordinator timeout to `5 s + 8 s * enabled_slots`.
Representative non-PSRAM and PSRAM start/stop matrices found no downward heap
or largest-block trend; multi-day soak, task-stack high-water, and exhaustive
callback ordering remain outside that run.
## Implementation status after Phase 5
- Publishing the plain-data snapshot and repointing the Section 1/Section 2 consumers at it.
- Replacing the Section 3 `volatile` handshakes with a command queue / task
notifications.
- **Completed:** the cooperative shutdown state machine in `MQTTBridge`, the
`begin()` double-call guard, slot-scaled timeout, and OTA teardown barrier.
Historical note: `MQTTBridge.cpp` was intentionally untouched during Phase 4;
Phase 5 later made the planned lifecycle changes as one reviewable unit.