MeshCore/docs/usb_logging_watchdog.md

<!-- meshcore-hosted-doc-link:start -->
<p class="meshcore-hosted-doc-link"><a href="https://mikecarper.github.io/MeshCore/usb_logging_watchdog/">View this page on MeshCore Docs</a>.</p>
<!-- meshcore-hosted-doc-link:end -->

# USB logging watchdog

Current source includes a saved USB logging watchdog in Companion, Repeater,
Room Server, and Sensor. It repairs the radio's USB connection; it cannot
restart a Pi, restore SSH, or prove MQTT or disk delivery. Supported backends
are ESP32 native TinyUSB/HWCDC and nRF52 TinyUSB. An external USB-to-UART
adapter is not an observable native-USB backend.
Unsupported targets compile out recovery timers and statefile transactions;
their watchdog commands report unsupported rather than saving an unusable mode.
Management still reports the logging master with USB support false.

## Commands and defaults

```text
get usb.watchdog
get usb.watchdog.last
set usb.watchdog auto
set usb.watchdog on
set usb.watchdog off
```

Setters accept exactly one lowercase mode, save it before applying, and do not
reboot immediately. They do not enable `usb.logging` or `usb.debug`.

| Mode | Behavior |
| --- | --- |
| `auto` | Default for missing watchdog state, including upgrades. Requires 14 continuous healthy logging-client days before saving `on`, with the retry tier reset to one hour. A previously confirmed client can receive USB-only recovery before promotion, but Auto never reboots the MCU. |
| `on` | Explicitly arms physical USB recovery and eventual MCU reboot. A confirmed stats client also gets USB-only recovery if its stats polls stop; missing polls alone cannot authorize MCU reboot. |
| `off` | No watchdog probes, automatic USB recovery, qualification, or MCU reset. Packet logging remains independent. |

Existing valid saved On/Off choices win. Auto qualification is RAM-only and
restarts on reboot, USB disconnect, observed stall, expired client lease,
logging master Off, or an explicit mode change. A node that has never confirmed
a logging client does not cycle USB in Auto merely because no cable is attached.

## Confirming a logging client

The existing `meshcoretomqtt` five-minute `stats-core`, `stats-radio`, and
`stats-packets` polls renew a 15-minute lease. Only complete, exact stats
commands received on the actual USB logging endpoint count; BLE, network,
LoRa, management reports, and ordinary framed Companion traffic do not.
No new heartbeat command is required. A cable, open TTY, DTR, or kernel USB
ACK alone cannot qualify Auto.

`logger=1` requires the USB logging master, an observed host and reader, a fresh
stats lease, and no physical TX stall. Any program polling those commands can
qualify: this is not application identification or proof that packets were
parsed, published to MQTT, or written to disk. A one-time manual stats query
can also establish the lease; use watchdog Off when later USB-only reconnection
would be unwanted.

On nRF52 Full Companion, only the dedicated logging CDC (`*-if02`) establishes
the lease. That endpoint accepts bounded stats-poll markers without replying
with statistics or exposing arbitrary CLI/configuration commands. Queries and
configuration still use primary CDC (`*-if00`), whose stats queries do not
prove a logging reader. Existing single-port bridges needing identity/CLI
replies still require separate control-port support.

## Recovery timing and safety

For a physical host/reader outage, or pending TX with no real completion,
software USB recovery is requested after five minutes. Re-enumeration follows
after another 60 seconds if health is not restored. These are one staged
attempt per outage, not a repeated five-minute reboot loop. Busy backends retry
at most once every five seconds without consuming an attempted stage.

After a confirmed client's stats lease expires, USB-only repair is due five
minutes later: normally 20 minutes after its last stats poll. This applies in
Auto and, once a client is confirmed during that boot, On. On with no stats
client observed remains compatible with non-stats log readers. A gap in stats
polls does not by itself reboot a healthy USB transport.

MCU reboot requires On, a physical USB fault lasting the entire saved tier,
and the USB recovery grace periods. Tiers grow after reset authorization:
1, 2, 4, 8, 16, 32, 64, 128, then 168 hours (one week). An older app-only
stats outage cannot shorten a later physical fault's interval. Ten minutes
of healthy monitored operation resets a raised tier to one hour.

Native TX-completion events are the progress evidence. Queueing, purging,
reinitializing, or accepting the optional bounded minute probe is not proof
of recovery. The watchdog never waits for a host. USB connection reports,
especially HWCDC's host/reader flags, are SDK approximations and do not reliably
identify an open TTY process.

Recovery waits for active Binary USB clients, mOTA/update owners, unsaved
contacts, radio tests/calibration/temporary-radio operations, functional CLI
work, and selected/in-flight radio transmissions to clear. Stale ASCII output
from a disconnected or stalled host does not indefinitely prevent recovery.
Before MCU reset, the next tier must commit and pass readback; the firmware
then rechecks transport health and ownership. Re-enumeration preserves the
configured USB descriptors, but can disconnect host applications and lose
best-effort packet records.

## Persistence and status

The independent, versioned, CRC-protected `/usb_wdg` statefile does not
change existing role preference offsets. Unreadable/corrupt state or a volatile
fallback filesystem disables automatic recovery/promotion. Failed commits
remain fail-closed; an explicit setter can retry or repair this statefile on
durable storage, applying only after complete verification. It cannot make
volatile storage durable. A verified backup repairs a conclusively missing
primary after an interrupted rename; unreadable live state is not replaced
automatically.

Current writes use the 33-byte UW2 format, including the latest watchdog event.
Valid 20-byte UW1 files still load with their saved mode, tier, and counters;
they upgrade on the next legitimate event/configuration save, not merely on
boot. Each attempted software recovery or re-enumeration saves its event and
counters once. Probes, idle ticks, ownership/backend deferrals, and status
reads do not write flash. A failed save leaves the live event marked unsaved
and automatic actions fail closed until explicit repair.

### Last-event review

```text
get usb.watchdog.last
```

This returns the latest event only, not an unbounded history. New events replace
older ones; exporting periodic management reports can retain observations
off-node. Mode changes preserve the record. Erasing the filesystem or restoring
a backup can remove or roll back it.

The event includes an advisory Unix epoch from the node's RTC, uptime seconds
at that recorded boot, a saturating event sequence, action, reason bitmask, and
durability status. Uptime extends the millisecond clock across rollover.
RTC time is not verified UTC: default clocks can start at a plausible fixed
date, and a later time correction does not rewrite old events. Epoch zero
is allowed and can mean no usable timestamp source. Use a synchronized node
clock for meaningful calendar times; sequence and recorded uptime remain useful when
clock time is wrong.

Reason bits are `1` host absent, `2` reader absent, `4` physical TX stalled,
and `8` previously confirmed stats client inactive. Physical failures can set
multiple bits; a stats-only expiry uses bit 8 and cannot authorize an MCU reset.
Actions are `soft-recovery`, `reenumerate`, `reboot-requested`, and
`reboot-cancelled`. Software actions mean the backend reported an attempt, not
that USB was successfully repaired. A reboot request is persisted with the
next backoff tier before authorization; health/ownership are rechecked after
the commit. A subsequent veto records cancellation immediately in RAM. If OTA
or another owner makes storage unsafe, its durable save waits for a safe
application tick; power loss before that save can leave the earlier reboot
intent on disk. Neither the intent record
nor the existing reboot counter proves a completed physical reset. Power loss
or a hardware main-loop watchdog can occur without any opportunity to save
an exact trigger time; this record covers the staged USB logging watchdog.

`get usb.watchdog` includes these compact fields:

| Field | Meaning |
| --- | --- |
| `usb`, `log`, `host`, `reader`, `logger` | Backend supported; logging master; observed physical USB host/reader; confirmed healthy stats client. |
| `stall`, `stage`, `step` | Physical TX stall; stage 0 idle, 1 software wait, 2 re-enumeration wait, 3 reset pending; saved tier index 0-8. |
| `retry`, `idle`, `auto` | Saved MCU interval, current monitored fault age, and continuous Auto qualification age, in seconds. Retry is not an action countdown. |
| `defer`, `fs` | Ownership/backend deferral and verified durable persistence readiness. |
| `rec`, `boot` | Attempted USB recoveries and durable reset authorizations. Attempts save the event and recovery count together; a failed save can leave the live recovery count unsaved. The reset count can include a later safety/health veto; it is not a count of completed physical reboots. |

The [MGR2 management report](management_reports.md) exports the USB status and
the same latest-event record as a frozen, authenticated public snapshot. Updated
Python and browser decoders retain released MGR1 support; older decoder copies
must be updated to read MGR2. Events appear on the normal management schedule, not
as immediate alerts. Reading status does not renew the logging-client lease.

Concurrent readers receive one coherent status publication, including its tier,
counters, and latest event. Reads never wait for a writer. If three bounded
attempts cannot obtain a snapshot, status conservatively reports deferred
recovery and no verified persistence, ages, counters, or event. It preserves
published hardware/logging/mode intent and protects a possibly active reader.
This temporary fallback cannot authorize a reset. The separate awake hint keeps
an already durable watchdog policy serviced while publication is busy.

## Verification

The two qualification stages below precede the subsequent upstream USB/GPS
and repeater trace merge. Their firmware byte margins describe those earlier
images, not the current merged source.

Initial watchdog qualification on 2026-10-03, before the latest-event extension,
passed 109 USB regression tests, 143 native test cases, 11 management protocol
tests, and the browser decoder checks.
Tests execute the production policy, runtime, marker parser, role ownership
guards, and native USB boundaries with simulated clock, filesystem, and FIFO
faults. The 14-day qualification uses an advanced test clock, not a hardware
soak. Bounded console output, unread hosts, session races, and exact stats
markers are covered.

Those initial compiled checks passed Station G2 Full Companion, Repeater, and Room Server;
XIAO nRF52 Full Companion and Repeater (full and reduced OTA qualifications);
and ordinary RAK 3x72 STM32 USB Companion. The STM32 image retains all CLI
features and its 32 KiB filesystem, but has only 368 bytes of app flash margin;
its unsupported USB watchdog machinery is compiled out.

Latest-event verification on the same date passed 116 USB regression tests,
15 management tests with fatal AddressSanitizer/UndefinedBehaviorSanitizer
checks, and the browser decoder tests. Coverage includes timestamp callbacks,
clock corrections and rollover, legacy-state migration, commit failures,
reboot-intent cancellation, deferred cancellation saves, and authenticated
cross-language decoding of the development report formats. The extension was
called MGR3 during those pre-release checks, then assigned to MGR2 because the
earlier USB-only MGR2 layout had never been released. Current decoders support
released MGR1 and the finalized MGR2 layout; this reassignment was not a migration
of deployed MGR2 reports. The full native suite passed 1,713 cases
during implementation; the final packet validator also passed a fresh 116-case
routing suite and 5,855,168 previous/current validator comparisons under fatal
sanitizers.

The extension compiled in standard Station G2 Full Companion and Repeater,
XIAO nRF52 Repeater, and RAK 3x72 USB Companion. The normal `build.sh` STM32
recipe at version `1.17.1.9` retains all CLI features and the 32 KiB filesystem,
with 372 bytes of app flash margin; longer version strings or future changes
can still exceed that tight budget. No features or filesystem space were cut
to fit this extension. This is not an all-board release qualification or a
physical USB recovery test. No hardware was flashed, and verification artifacts
stayed outside `out`.

The subsequent USB/GPS/repeater-trace merge passed all 1,718 native cases,
133 USB regression methods, 25 management/event/watchdog methods plus the
browser decoder, and 10 GPS/trace/persistence methods. Companion preference
tests exercise six platform/layout variants under fatal sanitizers. The
compact final preference writer also matches the previous full image for
256 tail-value sets per variant and preserves committed files through every
partial-tail write/readback boundary and rename failure.

Compile checks for that merge passed standard XIAO S3 Full Companion, Station
G2 Repeater and Room Server, native-USB XIAO S3 USB Companion with all five
session/progress hooks linked, and XIAO nRF52 Repeater. The final compact writer
passed both the bare STM32 CI recipe and the normal `1.17.1.9` recipe; the
latter uses 229,020 of 229,376 app flash bytes (356 bytes of margin), retaining
all features and the 32 KiB filesystem. The bare recipe is tighter still.
These checks do not qualify every board or physical USB recovery. No device
was flashed and existing `out` artifacts were not changed.

Finalizing the event-capable layout as MGR2 passed 1,719 native cases, 16
management tests with real C++/Python/browser cryptography, the standalone
browser decoder checks, and the bare STM32 Companion compile. Coverage retains
MGR1, rejects retired formats and old prototype ciphertext, and verifies that
invalid management sends release their packet without queueing it.