Record V4 Bluetooth recovery and pending physical USB reconnection

docs/test-results/hardware_validation_checklist.md

@@ -41,14 +41,31 @@ The passed Room logging-on trial does not qualify Full logging-on.
   One subsequent identity-guarded reset-only request to exact parent-hub port 2
   returned success; status changed `0x101` to `0x103` after two seconds, but the
   target still did not enumerate and sibling T096 remained unchanged. No hub
-  power change or broader reset followed. Operator physical V4 RESET and a
-  verified return remain pending. ModemManager was restored active at 09:24:34
-  UTC and all four checked host services were active; device return remained
-  unverified.
+  power change or broader hub reset followed. BLE diagnostic identity passed and
+  uptime 3959→3961 with zero errors established continued MCU operation through
+  the incident. The watchdog's volatile stage 2 and original SPIFFS namespace
+  absence of watchdog files establish two current-boot attempts; last event
+  reason `0x08` means client inactivity, not successful USB repair. ASCII
+  `stats-core` renews its 900-second lease; framed BLE does not. No evidence
+  selects the earlier lease-clear path or firmware/physical cause. A guarded
+  warm-restart request passed exact identity/flag gates, durable intent and
+  disconnect/cleanup checks, but USB remained absent at 09:45:07 with no new
+  kernel entries. Post-restart read-only trial 01 verified exact identity,
+  uptime 34→36 seconds, zero core errors and all owned cleanup checks; it did not
+  pair or create another bond. The unchanged last watchdog journal belongs to
+  the previous boot. USB remained absent at 09:48; physical reconnection of only
+  the V4 USB cable and verified USB recovery remain pending. No further software
+  or hub recovery is planned. Battery-backed USB reconnection does not qualify
+  cold boot. All four checked host services, including ModemManager, were active
+  by 09:48.
 - [ ] **Current V4 settings restoration:** original NVS/SPIFFS readback and
-  returned-state checks passed before the native trials. Subsequent temporary
-  power saving on, USB logging on and MQTT off remained **unrestored** at 09:22,
-  pending USB recovery. The earlier preservation PASS is not a final-state check.
+  returned-state checks passed before the native trials. BLE restoration trial
+  01 passed at 09:41:23: power saving off, logging off, MQTT on, WiFi off and
+  MQTT not running, with immediate/final readbacks and zero error flags.
+  Those values persisted through the verified warm reboot. A new owned BLE bond
+  was created by diagnostic 01 and is retained; authoritative original NVS/SPIFFS
+  restoration and full saved-state comparison await USB return.
+  The earlier preservation PASS is not a final-state check.
 - [ ] **Full V4 MQTT/WSS soak:** the pinned recovery-guarded runner is staged
   but has **not launched**. Require real PUBACK/heap evidence and original-data
   restoration before qualification.
@@ -56,7 +73,9 @@ The passed Room logging-on trial does not qualify Full logging-on.
 Complete CI runs [39110179910](https://github.com/mikecarper/MeshCore/actions/runs/39110179910)
 at `5fd33a230` and [39109551854](https://github.com/mikecarper/MeshCore/actions/runs/39109551854)
 at `7a5a7993c` are green. The native-console/TTY cleanup follow-up passed 35
-offline checks; its new CI result was pending at the authoring checkpoint.
+offline checks; [run 39115389840](https://github.com/mikecarper/MeshCore/actions/runs/39115389840)
+at `76220d5d7` was in progress at 09:46 UTC, with ten of eleven jobs green and
+only the host test job still running.
 The completed runs do not qualify those follow-up changes.
 
 ## PR17–21 follow-up qualification, 2026-10-18 UTC

docs/test-results/hardware_validation_mercerwood_1_18_0_1_2026-10-18.md

@@ -8,9 +8,11 @@ This campaign is in progress. It is not release approval. Results below bind
 specific images and physical devices; successful compilation and older bench
 results do not qualify unexecuted hardware cases.
 
-At the 09:22 UTC checkpoint, the V4 Full logging-on trial had failed and its USB
-endpoint remained absent. Recovery and restoration of subsequent temporary
-settings were still in progress. The staged MQTT/WSS soak had not launched.
+At the 09:46 UTC recovery checkpoint, the V4 Full logging-on trial remained
+failed/unqualified and USB was still absent after a verified guarded warm
+restart. BLE verified that the three restored temporary flags persisted.
+The new owned diagnostic bond remains; authoritative original-data restoration
+and USB recovery remain open. The staged MQTT/WSS soak has not launched.
 
 ## Firmware and preservation
 
@@ -81,9 +83,12 @@ the original data readback and those returned settings, not every unexposed
 preference or battery-backed cold boot. At that restoration checkpoint, the Full
 runtime had WiFi off, an unset SSID, MQTT enabled but not running, and unavailable
 ESP-NOW; temporary test changes require their own restoration. Later native-console
-tests enabled power saving and USB logging and disabled MQTT. Those temporary flags remained
-**unrestored** at 09:22 UTC, pending recovery of the missing USB endpoint.
-The earlier NVS/SPIFFS readback PASS does not establish the current settings.
+tests enabled power saving and USB logging and disabled MQTT. Those flags were
+unrestored at 09:22 UTC, then passed identity-bound BLE restoration and readback
+at 09:41:23: power saving off, USB logging off and MQTT on, with WiFi off and
+MQTT not running. The diagnostic created a new owned BLE bond. Complete original
+NVS/SPIFFS restoration has not been repeated after that bond; the earlier
+partition/readback PASS does not establish current byte-identical saved state.
 
 ## Executed Companion checks
 
@@ -350,18 +355,68 @@ The run ended at 09:13:42 with `FileNotFoundError` while TTY cleanup reopened
 the old path. That cleanup error masked the primary trial failure, which cannot
 be reconstructed from the old safe report. No post-idle reply, reopen check or
 qualified idle interval was obtained. USB recovery was still in progress at
-the checkpoint; no firmware, reset or physical-fault cause was proven.
+the 09:22 checkpoint; no firmware, reset or physical-fault cause was proven.
 The passed Room logging-on case does not qualify this Full image.
 
 A subsequent identity-guarded, reset-only request to the exact parent hub's
 port 2 returned success. Port status changed from `0x101` to `0x103` after two
 seconds, with zero change bits: connected, powered and then enabled. The target
 still did not enumerate; the sibling T096 was unchanged. No hub power change or
-broader reset followed. A physical V4 RESET was requested from the operator;
-recovery, temporary-setting restoration and the unstarted WSS soak remain
-pending that action and verified device return.
+broader hub reset followed. A physical V4 RESET was requested from the operator.
 ModemManager was restored to active at 09:24:34 UTC; all four checked host
-services were active. This does not establish device return or settings recovery.
+services were active at that checkpoint.
+
+The first read-only BLE diagnostic passed exact board, complete ec9ebc09 version
+and public-key checks. At approximately 09:35 UTC, core uptime advanced from
+3959 to 3961 seconds with zero error flags, versus the pre-idle 2076 seconds.
+This establishes continued MCU uptime through the USB disappearance rather than
+an MCU reset in that interval; it does not establish a working USB connection.
+
+The live USB watchdog reported Auto, stage 2, two recovery attempts and zero
+reboots. Its persisted last event was re-enumeration, sequence 2, uptime 2532
+seconds, reason `0x08` (`CLIENT_INACTIVE`). Its RTC epoch is advisory and does
+not date this incident reliably. Offline extraction of the exact original
+SPIFFS active namespace found no `usb_wdg`, `usb_wdg.bak` or `usb_wdg.tmp` files.
+The journal therefore was not imported by the original-data restoration.
+Policy stage is volatile and starts at zero, so the evidence establishes two
+current-boot USB recovery attempts. These are requested actions, not proof of
+successful repair or proof that they caused the observed kernel errors. The
+later sticky TX-stall observation is separate from the event's earlier reason.
+
+Source tracing confirms actual single-TTY ASCII `stats-core` renews the
+900-second logging-client lease; framed USB/BLE stats and CLI getters do not.
+Natural expiry after the fresh pre-idle ASCII poll cannot explain a fault
+starting roughly 97 seconds later. A bus-reset/session-generation change,
+observed host/reader loss or logging-off can clear that lease earlier. No
+pre-failure evidence selects one of these paths. This is watchdog-progress
+evidence, without a firmware or physical-fault cause assignment.
+
+BLE flag-restoration trial 01 passed at 09:41:23 UTC. Each of the three original
+flag values was acknowledged and read back immediately and again at the end;
+WiFi off and MQTT not running also matched. Uptime advanced from 4325 to 4333
+seconds with zero error flags, and owned BLE cleanup passed. This restores those
+flags, not the whole saved-state image or the new diagnostic bond.
+
+Guarded warm-restart trial 01 then passed exact identity and original-flag gates,
+durably recorded intent, sent the shipping `0x13` reboot request and observed
+disconnect. Pre-request uptime was 4556 seconds with zero error flags; owned
+BLE cleanup passed. USB was still absent at 09:45:07 and the kernel had no new
+entries.
+
+The post-restart read-only trial 01 passed exact identity with uptime 34→36
+seconds and zero core error flags, proving the deliberate warm reboot. It did
+not pair or create another bond. Persisted power saving off, logging off, MQTT
+on, WiFi off and MQTT not running all matched, and every owned cleanup check
+passed. The last watchdog journal was unchanged: re-enumeration, reason `0x08`,
+uptime 2532 seconds and sequence 2 refer to the previous boot, not a new event.
+
+By 09:48 UTC all four checked host services, including ModemManager, were active;
+USB was still absent. No additional software or hub recovery is planned. The
+operator was asked to unplug and reconnect only the V4 USB cable; physical
+reconnection and verified USB recovery remain pending. Battery-backed USB
+reconnection will not establish cold boot. The new diagnostic BLE bond is retained;
+authoritative original-data restoration and complete saved-state comparison
+await USB return. Full logging-on remains failed/unqualified and WSS is unstarted.
 
 For Full, `tools/hil/official_console_acceptance.py` uses explicit
 `--terminal-mode companion`. It sends the shipping terminal-start control
@@ -413,9 +468,12 @@ both that run and the preceding 7a5a7993 run completed with all jobs green.
 The native-console cleanup follow-up preserves the primary JS/transport
 failure separately from browser/TTY cleanup failures, reports missing or
 replaced endpoints, and cannot qualify a cleanup failure. The native-console
-and Linux TTY suites passed 35 offline checks. Its new CI result was pending at
-the authoring checkpoint; the completed runs do not qualify these follow-up
-changes or rerun the failed hardware case.
+and Linux TTY suites passed 35 offline checks. The follow-up workflow at
+`76220d5d7` was in progress
+([run 39115389840](https://github.com/mikecarper/MeshCore/actions/runs/39115389840))
+at 09:46 UTC: ten of eleven jobs were green and only the host test job remained
+running. The earlier completed runs do not qualify those changes or rerun the
+failed hardware case.
 Documentation route/link checks also passed in the CI-pinned renderer environment.
 
 ## Remaining release gates
@@ -429,9 +487,12 @@ Documentation route/link checks also passed in the CI-pinned renderer environmen
   V4 OLED has one runtime slot despite six saved configurations. Commit
   ec9ebc091 exposes the existing read-only diagnostic counter on Full. The
   recovery-guarded soak is staged and pinned but **not launched**.
-- Recover the V4 USB endpoint, restore and verify its temporary power/logging/MQTT
-  flags, and diagnose/retest the failed Full logging-on idle case. Retain the
-  Full logging-off PASS and the two Room logging-mode passes as separate results.
+- Recover the V4 USB endpoint, repeat authoritative original-data restoration
+  and complete saved-state readbacks, and diagnose the failed Full logging-on
+  idle case before another attempt. The three temporary flags passed BLE
+  restoration and warm-reboot persistence checks; the new owned bond remains
+  outside the earlier original-partition restoration. Retain the Full
+  logging-off PASS and the two Room logging-mode passes as separate results.
 - Complete two-source recovery-AP throttle checks beyond the passed existing-admin
   authentication and browser cases. Execute migration credentials and both source
   OTA slots, actual browser upload, interruption/recovery, privileged replay