diff --git a/docs/test-results/hardware_validation_checklist.md b/docs/test-results/hardware_validation_checklist.md index faf7b8318..6fe9c564d 100644 --- a/docs/test-results/hardware_validation_checklist.md +++ b/docs/test-results/hardware_validation_checklist.md @@ -41,14 +41,31 @@ The passed Room logging-on trial does not qualify Full logging-on. One subsequent identity-guarded reset-only request to exact parent-hub port 2 returned success; status changed `0x101` to `0x103` after two seconds, but the target still did not enumerate and sibling T096 remained unchanged. No hub - power change or broader reset followed. Operator physical V4 RESET and a - verified return remain pending. ModemManager was restored active at 09:24:34 - UTC and all four checked host services were active; device return remained - unverified. + power change or broader hub reset followed. BLE diagnostic identity passed and + uptime 3959→3961 with zero errors established continued MCU operation through + the incident. The watchdog's volatile stage 2 and original SPIFFS namespace + absence of watchdog files establish two current-boot attempts; last event + reason `0x08` means client inactivity, not successful USB repair. ASCII + `stats-core` renews its 900-second lease; framed BLE does not. No evidence + selects the earlier lease-clear path or firmware/physical cause. A guarded + warm-restart request passed exact identity/flag gates, durable intent and + disconnect/cleanup checks, but USB remained absent at 09:45:07 with no new + kernel entries. Post-restart read-only trial 01 verified exact identity, + uptime 34→36 seconds, zero core errors and all owned cleanup checks; it did not + pair or create another bond. The unchanged last watchdog journal belongs to + the previous boot. USB remained absent at 09:48; physical reconnection of only + the V4 USB cable and verified USB recovery remain pending. No further software + or hub recovery is planned. Battery-backed USB reconnection does not qualify + cold boot. All four checked host services, including ModemManager, were active + by 09:48. - [ ] **Current V4 settings restoration:** original NVS/SPIFFS readback and - returned-state checks passed before the native trials. Subsequent temporary - power saving on, USB logging on and MQTT off remained **unrestored** at 09:22, - pending USB recovery. The earlier preservation PASS is not a final-state check. + returned-state checks passed before the native trials. BLE restoration trial + 01 passed at 09:41:23: power saving off, logging off, MQTT on, WiFi off and + MQTT not running, with immediate/final readbacks and zero error flags. + Those values persisted through the verified warm reboot. A new owned BLE bond + was created by diagnostic 01 and is retained; authoritative original NVS/SPIFFS + restoration and full saved-state comparison await USB return. + The earlier preservation PASS is not a final-state check. - [ ] **Full V4 MQTT/WSS soak:** the pinned recovery-guarded runner is staged but has **not launched**. Require real PUBACK/heap evidence and original-data restoration before qualification. @@ -56,7 +73,9 @@ The passed Room logging-on trial does not qualify Full logging-on. Complete CI runs [39110179910](https://github.com/mikecarper/MeshCore/actions/runs/39110179910) at `5fd33a230` and [39109551854](https://github.com/mikecarper/MeshCore/actions/runs/39109551854) at `7a5a7993c` are green. The native-console/TTY cleanup follow-up passed 35 -offline checks; its new CI result was pending at the authoring checkpoint. +offline checks; [run 39115389840](https://github.com/mikecarper/MeshCore/actions/runs/39115389840) +at `76220d5d7` was in progress at 09:46 UTC, with ten of eleven jobs green and +only the host test job still running. The completed runs do not qualify those follow-up changes. ## PR17–21 follow-up qualification, 2026-10-18 UTC diff --git a/docs/test-results/hardware_validation_mercerwood_1_18_0_1_2026-10-18.md b/docs/test-results/hardware_validation_mercerwood_1_18_0_1_2026-10-18.md index c94be6fc2..5c4ac5cad 100644 --- a/docs/test-results/hardware_validation_mercerwood_1_18_0_1_2026-10-18.md +++ b/docs/test-results/hardware_validation_mercerwood_1_18_0_1_2026-10-18.md @@ -8,9 +8,11 @@ This campaign is in progress. It is not release approval. Results below bind specific images and physical devices; successful compilation and older bench results do not qualify unexecuted hardware cases. -At the 09:22 UTC checkpoint, the V4 Full logging-on trial had failed and its USB -endpoint remained absent. Recovery and restoration of subsequent temporary -settings were still in progress. The staged MQTT/WSS soak had not launched. +At the 09:46 UTC recovery checkpoint, the V4 Full logging-on trial remained +failed/unqualified and USB was still absent after a verified guarded warm +restart. BLE verified that the three restored temporary flags persisted. +The new owned diagnostic bond remains; authoritative original-data restoration +and USB recovery remain open. The staged MQTT/WSS soak has not launched. ## Firmware and preservation @@ -81,9 +83,12 @@ the original data readback and those returned settings, not every unexposed preference or battery-backed cold boot. At that restoration checkpoint, the Full runtime had WiFi off, an unset SSID, MQTT enabled but not running, and unavailable ESP-NOW; temporary test changes require their own restoration. Later native-console -tests enabled power saving and USB logging and disabled MQTT. Those temporary flags remained -**unrestored** at 09:22 UTC, pending recovery of the missing USB endpoint. -The earlier NVS/SPIFFS readback PASS does not establish the current settings. +tests enabled power saving and USB logging and disabled MQTT. Those flags were +unrestored at 09:22 UTC, then passed identity-bound BLE restoration and readback +at 09:41:23: power saving off, USB logging off and MQTT on, with WiFi off and +MQTT not running. The diagnostic created a new owned BLE bond. Complete original +NVS/SPIFFS restoration has not been repeated after that bond; the earlier +partition/readback PASS does not establish current byte-identical saved state. ## Executed Companion checks @@ -350,18 +355,68 @@ The run ended at 09:13:42 with `FileNotFoundError` while TTY cleanup reopened the old path. That cleanup error masked the primary trial failure, which cannot be reconstructed from the old safe report. No post-idle reply, reopen check or qualified idle interval was obtained. USB recovery was still in progress at -the checkpoint; no firmware, reset or physical-fault cause was proven. +the 09:22 checkpoint; no firmware, reset or physical-fault cause was proven. The passed Room logging-on case does not qualify this Full image. A subsequent identity-guarded, reset-only request to the exact parent hub's port 2 returned success. Port status changed from `0x101` to `0x103` after two seconds, with zero change bits: connected, powered and then enabled. The target still did not enumerate; the sibling T096 was unchanged. No hub power change or -broader reset followed. A physical V4 RESET was requested from the operator; -recovery, temporary-setting restoration and the unstarted WSS soak remain -pending that action and verified device return. +broader hub reset followed. A physical V4 RESET was requested from the operator. ModemManager was restored to active at 09:24:34 UTC; all four checked host -services were active. This does not establish device return or settings recovery. +services were active at that checkpoint. + +The first read-only BLE diagnostic passed exact board, complete ec9ebc09 version +and public-key checks. At approximately 09:35 UTC, core uptime advanced from +3959 to 3961 seconds with zero error flags, versus the pre-idle 2076 seconds. +This establishes continued MCU uptime through the USB disappearance rather than +an MCU reset in that interval; it does not establish a working USB connection. + +The live USB watchdog reported Auto, stage 2, two recovery attempts and zero +reboots. Its persisted last event was re-enumeration, sequence 2, uptime 2532 +seconds, reason `0x08` (`CLIENT_INACTIVE`). Its RTC epoch is advisory and does +not date this incident reliably. Offline extraction of the exact original +SPIFFS active namespace found no `usb_wdg`, `usb_wdg.bak` or `usb_wdg.tmp` files. +The journal therefore was not imported by the original-data restoration. +Policy stage is volatile and starts at zero, so the evidence establishes two +current-boot USB recovery attempts. These are requested actions, not proof of +successful repair or proof that they caused the observed kernel errors. The +later sticky TX-stall observation is separate from the event's earlier reason. + +Source tracing confirms actual single-TTY ASCII `stats-core` renews the +900-second logging-client lease; framed USB/BLE stats and CLI getters do not. +Natural expiry after the fresh pre-idle ASCII poll cannot explain a fault +starting roughly 97 seconds later. A bus-reset/session-generation change, +observed host/reader loss or logging-off can clear that lease earlier. No +pre-failure evidence selects one of these paths. This is watchdog-progress +evidence, without a firmware or physical-fault cause assignment. + +BLE flag-restoration trial 01 passed at 09:41:23 UTC. Each of the three original +flag values was acknowledged and read back immediately and again at the end; +WiFi off and MQTT not running also matched. Uptime advanced from 4325 to 4333 +seconds with zero error flags, and owned BLE cleanup passed. This restores those +flags, not the whole saved-state image or the new diagnostic bond. + +Guarded warm-restart trial 01 then passed exact identity and original-flag gates, +durably recorded intent, sent the shipping `0x13` reboot request and observed +disconnect. Pre-request uptime was 4556 seconds with zero error flags; owned +BLE cleanup passed. USB was still absent at 09:45:07 and the kernel had no new +entries. + +The post-restart read-only trial 01 passed exact identity with uptime 34→36 +seconds and zero core error flags, proving the deliberate warm reboot. It did +not pair or create another bond. Persisted power saving off, logging off, MQTT +on, WiFi off and MQTT not running all matched, and every owned cleanup check +passed. The last watchdog journal was unchanged: re-enumeration, reason `0x08`, +uptime 2532 seconds and sequence 2 refer to the previous boot, not a new event. + +By 09:48 UTC all four checked host services, including ModemManager, were active; +USB was still absent. No additional software or hub recovery is planned. The +operator was asked to unplug and reconnect only the V4 USB cable; physical +reconnection and verified USB recovery remain pending. Battery-backed USB +reconnection will not establish cold boot. The new diagnostic BLE bond is retained; +authoritative original-data restoration and complete saved-state comparison +await USB return. Full logging-on remains failed/unqualified and WSS is unstarted. For Full, `tools/hil/official_console_acceptance.py` uses explicit `--terminal-mode companion`. It sends the shipping terminal-start control @@ -413,9 +468,12 @@ both that run and the preceding 7a5a7993 run completed with all jobs green. The native-console cleanup follow-up preserves the primary JS/transport failure separately from browser/TTY cleanup failures, reports missing or replaced endpoints, and cannot qualify a cleanup failure. The native-console -and Linux TTY suites passed 35 offline checks. Its new CI result was pending at -the authoring checkpoint; the completed runs do not qualify these follow-up -changes or rerun the failed hardware case. +and Linux TTY suites passed 35 offline checks. The follow-up workflow at +`76220d5d7` was in progress +([run 39115389840](https://github.com/mikecarper/MeshCore/actions/runs/39115389840)) +at 09:46 UTC: ten of eleven jobs were green and only the host test job remained +running. The earlier completed runs do not qualify those changes or rerun the +failed hardware case. Documentation route/link checks also passed in the CI-pinned renderer environment. ## Remaining release gates @@ -429,9 +487,12 @@ Documentation route/link checks also passed in the CI-pinned renderer environmen V4 OLED has one runtime slot despite six saved configurations. Commit ec9ebc091 exposes the existing read-only diagnostic counter on Full. The recovery-guarded soak is staged and pinned but **not launched**. -- Recover the V4 USB endpoint, restore and verify its temporary power/logging/MQTT - flags, and diagnose/retest the failed Full logging-on idle case. Retain the - Full logging-off PASS and the two Room logging-mode passes as separate results. +- Recover the V4 USB endpoint, repeat authoritative original-data restoration + and complete saved-state readbacks, and diagnose the failed Full logging-on + idle case before another attempt. The three temporary flags passed BLE + restoration and warm-reboot persistence checks; the new owned bond remains + outside the earlier original-partition restoration. Retain the Full + logging-off PASS and the two Room logging-mode passes as separate results. - Complete two-source recovery-AP throttle checks beyond the passed existing-admin authentication and browser cases. Execute migration credentials and both source OTA slots, actual browser upload, interruption/recovery, privileged replay