diff --git a/docs/test-results/hardware_validation_checklist.md b/docs/test-results/hardware_validation_checklist.md index a34e5ec18..faf7b8318 100644 --- a/docs/test-results/hardware_validation_checklist.md +++ b/docs/test-results/hardware_validation_checklist.md @@ -14,6 +14,51 @@ when its log identifies the physical device, firmware artifact, artifact hash, command result, and cold/warm boot outcome. Do not infer success from a tool's exit code when the tool has a documented false-success mode. +## Mercerwood candidate campaign, 2026-10-18 UTC + +The [current hardware campaign](hardware_validation_mercerwood_1_18_0_1_2026-10-18.md) +records actual candidate flashes, identity-bound USB/BLE and warm-contact +checks, live Room authentication/browser results, and shipping Morse failures. +It retains the original populated-board stack failures and the d52e4e93 +repeated-read/browser retests that passed. The matching-profile trials established +LoRa reception and paced live-session mailbox operation; durable replay, +final-role testing and other release gates remain open. These observations do +not constitute release approval. + +The ec9ebc09 Full V4 logging-off native-console trial 03 passed 600.444 seconds +with zero idle writes, complete replies/reopen and zero core error flags. +The passed Room logging-on trial does not qualify Full logging-on. + +- [ ] **Full V4 logging-on idle/reopen:** trial 01 started at 09:02:25 UTC, + then lost target USB at 09:11:14. The kernel's preceding 09:11:13 + `disabled by hub (EMI?), re-enabling...` message is not proof of EMI; + descriptor/address `-71` errors and failed enumeration continued through + 09:11:16. The run ended at 09:13:42 with a TTY cleanup `FileNotFoundError` + masking its primary failure. No post-idle reply or qualified idle was obtained; + by-id/path remained absent at 09:22 and recovery was in progress. No firmware + or reset cause is established. The harness cleanup fix passed 35 offline + native-console/TTY checks and preserves primary and cleanup failures separately. + One subsequent identity-guarded reset-only request to exact parent-hub port 2 + returned success; status changed `0x101` to `0x103` after two seconds, but the + target still did not enumerate and sibling T096 remained unchanged. No hub + power change or broader reset followed. Operator physical V4 RESET and a + verified return remain pending. ModemManager was restored active at 09:24:34 + UTC and all four checked host services were active; device return remained + unverified. +- [ ] **Current V4 settings restoration:** original NVS/SPIFFS readback and + returned-state checks passed before the native trials. Subsequent temporary + power saving on, USB logging on and MQTT off remained **unrestored** at 09:22, + pending USB recovery. The earlier preservation PASS is not a final-state check. +- [ ] **Full V4 MQTT/WSS soak:** the pinned recovery-guarded runner is staged + but has **not launched**. Require real PUBACK/heap evidence and original-data + restoration before qualification. + +Complete CI runs [39110179910](https://github.com/mikecarper/MeshCore/actions/runs/39110179910) +at `5fd33a230` and [39109551854](https://github.com/mikecarper/MeshCore/actions/runs/39109551854) +at `7a5a7993c` are green. The native-console/TTY cleanup follow-up passed 35 +offline checks; its new CI result was pending at the authoring checkpoint. +The completed runs do not qualify those follow-up changes. + ## PR17–21 follow-up qualification, 2026-10-18 UTC The merged source checkpoint is `0a24279a3`. This audit adds host regressions, @@ -44,11 +89,13 @@ and `esp32_s3_4mb_partition_migrator_usb`, including their application-size and runtime-RAM gates. This covers representative chip families and Wi-Fi, identity-only LoRa, Expander and USB code paths, rather than every board alias. -### Remaining physical acceptance checks +### Physical acceptance checks and subsequent campaign results Record the exact clean source revision, firmware SHA-256, device identity, observed result, and cold/warm boot outcome for each check. Record password classes and success/failure, never the passwords themselves. +Checked entries below are later, candidate-specific campaign results; they do +not change the host-only scope of the earlier `0a24279a3` audit. - [ ] **Migration credentials:** on cable-recoverable test hardware, join with a distinct saved 8–15-character printable ASCII admin password. Reject the @@ -61,18 +108,32 @@ classes and success/failure, never the passwords themselves. identity/supported settings, final version, and cold-boot persistence. Exercise supported interruption/retry points on sacrificial hardware; the partition-table sector write is not atomic. -- [ ] **WebConfig:** use two distinct source addresses on LAN and on a - saved-network recovery AP. One address's five failures must not lock the - other out; exhausting the shared budget must reject even a correct new - login. An existing authorized session must remain usable. Verify recovery - at the six-second refill and 30-second address-lock boundaries. Clients - sharing one address also share its lock. -- [ ] **Room browser:** use separate real browser tabs/tokens for five - different wrong passwords. Nonempty credentials, including an already-open - admin tab, must wait for the shared window to expire; public empty-password - read-only access must remain available when enabled. Repeated stale-tab - passwords must not consume a new failure on every poll. Check visible error, - retry and unsaved-edit behavior against the real board. +- [x] **WebConfig LAN and Room password windows:** the complete two-source + LAN suite passed on d52e4e93 in 161.67 seconds. It covered per-address/shared + limits, existing-session access, refill/recovery, Room duplicate-password + suppression and the shared window, including nonempty credential rejection + and continued public empty-password read-only access. No temporary lockout + remained. These runs use guarded timing windows; they do not establish exact + millisecond boundary behavior or saved-network recovery-AP acceptance. +- [ ] **WebConfig saved-network recovery AP:** repeat the two-source checks + on the actual recovery AP. One address's five failures must not lock the + other out; shared-budget exhaustion must reject even a correct new login; + an existing session must remain usable. Verify refill/address-lock recovery + and remember that clients sharing one address also share its lock. +- [x] **Room browser:** actual Chromium on d52e4e93 passed real admin form + login, five distinct wrong passwords across two tokens, visible automatic + throttle-error polling, production Connect retry, and preserved unsent draft + and Room boot identity. The browser used unmodified native fetch and issued + no compose/write requests. This completes the previously open real-board + error/retry/draft case for this candidate. +- [x] **Room populated-board repeated reads:** d52e4e93 completed twenty + live index/article reads of the owned 2048-byte UTF-8 article with unchanged + content, boot identity and continuous uptime. Earlier 652/d3e05abc stack + failures remain historical evidence; subsequent candidates must retain this + regression check. A later sticky radio-watchdog flag `8` is consistent with + the five-minute no-successful-RX policy and does not establish an HTTP cause + or qualify radio reception. Original-filesystem restoration was independently + verified after the later Full Companion handoff. - [ ] **Guest sessions:** churn transient guest identities, then admit a new admin and an explicitly assigned writer. Power-cycle and verify privileged login replay floors remain authoritative. Document the expected captured @@ -90,9 +151,11 @@ classes and success/failure, never the passwords themselves. sleep does not latch unrelated live pins. A successful build and simulated pad retention do not establish physical latch or current behavior. -Remaining automated follow-ups are browser throttle-error/retry/draft cases -and an executed configuration-restore-to-AP credential handoff. The host AP -fixture supplies pre/post-restore files; it does not run an entire migration. +The later d52e4e93 campaign completes the real-board browser +throttle-error/retry/draft and reproduced repeated-read cases above. An executed +configuration-restore-to-AP credential handoff remains open. The host AP fixture +supplies pre/post-restore files; it does not run an entire migration. The current +hardware campaign remains in progress and is not release qualification. ## ESP32 web console and browser OTA release gates @@ -135,7 +198,8 @@ successful build or simulated peripheral is not a hardware pass. raw binary request. Confirm identity and settings were preserved, then restore temporary device settings and the test host's network connection. -Keep the host regressions (`test_official_web_console.py`, +Keep the host regressions (`test_official_console_acceptance.py`, +`test_official_web_console.py`, `test_wifi_ota_start.py`, `test_wifi_ota_lifecycle.py`, and `test_async_web_lifecycle.py`) alongside these gates. They cover parsing, sleep policy, startup errors, network ownership, and diff --git a/docs/test-results/hardware_validation_mercerwood_1_18_0_1_2026-10-18.md b/docs/test-results/hardware_validation_mercerwood_1_18_0_1_2026-10-18.md new file mode 100644 index 000000000..c94be6fc2 --- /dev/null +++ b/docs/test-results/hardware_validation_mercerwood_1_18_0_1_2026-10-18.md @@ -0,0 +1,451 @@ + +

View this page on MeshCore Docs.

+ + +# Mercerwood 1.18.0.1 hardware campaign, 2026-10-18 UTC + +This campaign is in progress. It is not release approval. Results below bind +specific images and physical devices; successful compilation and older bench +results do not qualify unexecuted hardware cases. + +At the 09:22 UTC checkpoint, the V4 Full logging-on trial had failed and its USB +endpoint remained absent. Recovery and restoration of subsequent temporary +settings were still in progress. The staged MQTT/WSS soak had not launched. + +## Firmware and preservation + +The initial eight-image Full matrix built successfully from clean source +`652289ee823bbed22918fe584f9d49ccb2ec230f`, with release-length version +`v1.18.0.1-halo-keymind-cascade-dev-652289ee`: V4 Room Server, V4.3 Companion +FEM-off, RAK4631, T096 FEM-on, MeshPocket, T1000-E, SenseCAP Indicator LoRa, +and Xiao S3 WIO. Room fixes were separately built and readback-verified at +`d3e05abce308b52261e86c9297029b4bce3cc4a1` and +`d52e4e93475e0960328131e2e65cc6b992aa4dbb`. The diagnostic-enabled Full +V4.3 FEM-off and Indicator LoRa images built successfully at +`ec9ebc0914d61a02fb21f23444768ec5f847e7ec`. +The later MeshTower V2 SD Full build and transfer are recorded separately below. + +Before ESP32 application changes, complete private flash backups were made +through the pinned real flasher's identity/reset/finish path. Application-only +writes checked silicon MAC, USB identity, capacity, active OTA slot, image +integrity and full application readback. Partition tables and OTA metadata +were verified unchanged. Original NVS/filesystem restoration is a separate +operation; an application-only flash does not restore settings changed by tests. +The Indicator retains its existing noncanonical 8 MiB partition table. + +nRF52 updates used application-only DFU. The RAK3401 service radio and +Indicator RP2040 display coprocessor were excluded. No root USB hub power +operation or concurrent PlatformIO process was used. + +The flashed 652289ee artifacts below were built from source +`652289ee823bbed22918fe584f9d49ccb2ec230f`. USB identity labels are the first +12 hexadecimal characters of SHA256 of the UTF-8 USB serial as reported; +the serial-less CH340 Indicator was bound privately by its physical USB path +and public device identity. These hashes bind the executed images, not +unexecuted release qualification. ESP32 hashes cover application-only BINs; +nRF52 hashes cover the application DFU ZIPs actually transferred. + +| Target | Source | USB serial hash | Artifact | SHA256 | +| --- | --- | --- | --- | --- | +| `heltec_v4_room_server` | `652289ee8` | `af0a529cb0cc` | Application BIN | `4e370dd285813ce330aae13a9a8891795145d3bbb7adef84ff4dbb19926cb677` | +| `RAK_4631_companion_radio_full` | `652289ee8` | `b11fbd38286f` | Application DFU ZIP | `64940ae543491831a20da2176e50e44a1519917bd42c4615601ddc0c7bf82364` | +| `Heltec_t096_companion_radio_full_femon` | `652289ee8` | `ab7bc0653271` | Application DFU ZIP | `0a44dbc7a39fb65007713aa61c004b8bb790035c1d6c343ba150a0a77f855dfa` | +| `Mesh_pocket_companion_radio_full` | `652289ee8` | `c5ba0587de70` | Application DFU ZIP | `eb13c05521f2545719db809c5367ed31de22e9aeadc0c0992366a6ab770570c5` | +| `t1000e_companion_radio_full` | `652289ee8` | `5244f0b64696` | Application DFU ZIP | `7a6003afa50720e2c97cfbb625692143f66b4e7de063de63106fecd5ff4fc5fb` | +| `SenseCapIndicator-LoRa_companion_radio_full` | `652289ee8` | No USB serial | Application BIN | `6cb1405fb2a6c3f4889506f184751d7576ad23159c303f84c054e4ad6b124b03` | +| `Xiao_S3_WIO_companion_radio_full` | `652289ee8` | `b00e0958f2ff` | Application BIN | `59c62ff6d985aa42740b6c26dcc30efe50c04b747d94d566e4ca4a8cd1e42e47` | +| `Heltec_tower_v2_sdcard_repeater_lora_ota_no_external_sensors` | `652289ee8` | `030f24857801` | Application DFU ZIP | `0d20f4486023a13427eb09087da70aacac5562848371a919631ab6a76f7888d1` | + +The same V4's subsequent Room application hashes were +`54962d21088f1c87c3956620181a3499102e074d2e28791003d1b0479f64c7af` +for d3e05abc and +`7753f5f4161a6631d7f3abe44a99d70d5b892a94ad3cb69d313d54e44c608c87` +for d52e4e93. Their archived exact ELF hashes are respectively +`57d5b38fcc454d1ccc05763d3d8c6fa047904c42ca761be3a0393f14e799d1eb` +and `5db0fb151ee388ffab66ccce11b0d137907ff6450530f7812d35383c8c6aa785`. + +After the Room campaign, the V4.3 Full FEM-off ec9ebc09 application passed +identity-bound application-only flashing and complete readback. Its 2,545,080-byte +BIN SHA256 is +`cb1e167a222e7a8d0ca253f7e3272561a29ba6753f4173eb89c79405b2bdad2f`. +The separate restoration then wrote only the original NVS and SPIFFS partitions +from the pinned pre-campaign full backup. Both partitions matched their original +bytes on readback; the active application, partition table and OTA metadata +were independently verified unchanged before runtime handoff. + +A fresh Full snapshot matched all original `SELF_INFO` bytes, complete contact +records and all 40 canonical channels. Returned `DEVICE_INFO` settings matched +after excluding only the runtime active PIN and independently verified new +build/version metadata. There were no new core error flags. This qualifies +the original data readback and those returned settings, not every unexposed +preference or battery-backed cold boot. At that restoration checkpoint, the Full +runtime had WiFi off, an unset SSID, MQTT enabled but not running, and unavailable +ESP-NOW; temporary test changes require their own restoration. Later native-console +tests enabled power saving and USB logging and disabled MQTT. Those temporary flags remained +**unrestored** at 09:22 UTC, pending recovery of the missing USB endpoint. +The earlier NVS/SPIFFS readback PASS does not establish the current settings. + +## Executed Companion checks + +| Physical hardware | Candidate | USB queries/reopens | Owned contact add/update/delete | Add/delete survive warm reboots | BLE bonded hold/reconnect | +| --- | --- | --- | --- | --- | --- | +| RAK4631 | Full 652289ee | PASS: 100/10 | PASS | PASS | PASS: two 10-second sessions | +| T096 | Full FEM-on 652289ee | PASS: 100/10 | PASS | PASS | PASS: two 10-second sessions | +| MeshPocket | Full 652289ee | PASS: 100/10 | PASS | PASS | PASS: two 10-second sessions | +| T1000-E | Full 652289ee | PASS: 100/10 | PASS | PASS | FAIL/unqualified: four shipping attempts | +| Xiao S3 WIO | Full 652289ee | PASS: 100/10 | Not attempted: 350 occupied contacts | Not tested | Not tested | +| SenseCAP Indicator | LoRa Full 652289ee | Identity/readback and USB WiFi/BLE mode cycle PASS | Not tested | Not tested | Not tested | + +Warm tests used a confirmed-absent random contact, verified complete readback, +performed the shipping guarded reboot, verified persistence, removed only the +recognized owned record, then rebooted and verified original state. They do +not prove cold boot. Display boards legitimately regenerate the active BLE +PIN when the saved PIN selects that policy; comparisons exclude only those +four runtime bytes and retain the private raw frames. Original contacts, +canonical channels, public identity, name and the other returned +`SELF_INFO`/`DEVICE_INFO` settings fields are checked. This comparison does +not inspect every persisted preference. + +The Xiao retained all 350 contact keys and all 40 channels. A known contact's +advertisement and modification timestamps advanced during live mesh traffic; +this was not a contact edit by the runner. No existing Xiao contact was removed +to make room for a test record. + +The Indicator LoRa Full mode cycle passed on 2026-10-18 UTC. USB checks +verified the exact board, complete candidate version and public identity, +then proved warm reboots into WiFi and BLE modes with the expected selected, +boot and live service states. The original BLE selector was restored and +another warm reboot verified. All 40 channels, the empty contact baseline, +returned binary settings, saved radio/RXPS/radio2 configuration and queried +Bluetooth preferences matched the original state, with no new core error +flags. The runner opened seven USB sessions and issued 510 framed requests; +it issued no RF message commands or partition/RP2040 writes. These checks do +not qualify an external TCP or bonded BLE session, font download/retention, +fresh NTP, visible rendering or cold boot. + +The ESP-NOW Full build at `ec9ebc091` failed the mandatory internal RAM gate: +233,688 bytes available versus 251,168 required, a 17,480-byte shortfall. +Its native 480px WiFi canvas policy remains unchanged. Both ESP-NOW by +WiFi/BLE combinations remain unqualified; the normal Reduced ESP-NOW USB +image lacks those secondary transports and cannot substitute for their +Full acceptance. + +T1000-E BLE initially resolves services and returns identity/core data, and +one attempt returned all five read-only getters. Later writes/reads fail with +backend service-discovery, ATT `0x0e`, or timeout errors. USB remains healthy. +The probe's final `disconnected` event is cleanup, not proof of which side +terminated a connection. These observations do not establish a root cause. + +A fourth existing-bond trial captured HCI traffic. It completed encryption and +service discovery at MTU 247 and validated the complete version and public key. +The `get name` ATT Write Request (`0x12`) received its Write Response (`0x13`), +then no reply notification appeared. About 2.01 seconds later the controller +reported Connection Timeout (`0x08`), matching the negotiated two-second +supervision timeout and preceding the client's eight-second reply deadline. +No host HCI Disconnect command appeared in the capture. This trial therefore +lost the link before client cleanup. BLE uptime was 2720 seconds and later +USB uptime was 2993 seconds, consistent with the 272.70-second host interval; +both reported zero core error flags. This excludes a full MCU reboot in that +observed window, without identifying a firmware or RF cause. The post-trial +BlueZ state was disconnected and both owned processes closed. Negotiated link +parameters were a 30 ms interval, latency four and 2000 ms supervision timeout; +no PHY/data-length-change event appeared in the saved capture. + +A separate diagnostic build enabled the existing RAM trace and disabled debug +printing, retaining the same source, version and connection parameters. Its +application DFU ZIP SHA256 is +`713765d9145db2aec03059a87280a065d4ae68e1a23ee4dd56a4d16f37b4f8c1`; +the exact ELF SHA256 is +`18442324b87deef9af658d2997b668ee10912929af4be9bacf9c473bc70d2169`. +This trial failed during GATT setup before any Companion request was sent. +The MCU initiated its documented no-app-data recovery exactly 15 seconds after +security completed. All 14 recorded stack dispatches exited, with a maximum +2.93 ms dispatch and 103 ms main-loop gap. All 62 trace records were retained, +with no ring wrap or recorded contention drops. HCI recorded the peripheral +termination and no host Disconnect command. This is a different failure stage +from the shipping post-getter supervision timeout; concurrent host Chromium +startup is a possible, unproven confounder. It does not identify a production +firmware fix. No old post-USB core metrics were reused for this trial. + +The original shipping application was then restored through application-only +DFU. Identity, complete contact records, all 40 channels and all returned device +settings matched the immediately preceding baseline, including the active PIN; +core error flags were zero. An explicit read confirmed trace support absent. +The diagnostic image does not count as shipping BLE qualification. + +GPS was enabled for a bounded 180-second local-only T1000-E telemetry window. +No location appeared. The exact GPS switch was restored, sync cadence was +unchanged, and packet TX counters did not advance. Outdoor acquisition, +fresh-fix age, stale/no-fix behavior and sleep current remain unqualified. + +## MeshTower V2 SD + +The MeshTower V2 SD Full image from `652289ee8` passed identity-bound, +application-only USB DFU and initial SD mount/list/space checks. All 16 returned +getter checks matched the original journals, including public identity, +radio/power/logging settings, `OTAFIX2.4.11-preview.2`, and InternalFS geometry +at `0xED000+28 KiB`. Ten consecutive SD space/list/core triplets completed all +30 replies: uptime advanced from 119 to 125 seconds, core error flags and +queue length remained zero, and the original directory metadata and reported +32.0 KiB used/959.6 MiB free stayed unchanged. The directory contained the same +single 4-byte `/mota/cache.off` marker; no file contents were exported. Original +firmware had error flags 2 before the update; the new boot is evaluated +separately. These checks qualify returned values and read-only SD access only. +They do not qualify card-absent fallback, cold initialization, interruption or +power-loss recovery, LoRa OTA transfer/install/serving, or a bootloader update. +No SD format/erase or deliberate LoRa OTA transmission was performed. + +## Room authentication and browser + +On the original 652289ee Room image, actual LAN requests from two separately +bound source addresses passed per-address failure isolation, shared-budget +exhaustion/refill, existing-session access, no token refund on success, and +post-window recovery. The Room password checks passed duplicate-password +suppression, five-unique-failure admission, the global 30-second window, +nonempty admin/guest rejection during lockout, and continued public blank +read-only access. Public guest wire role is `0`; role `1` denotes explicit +ACL read-only permission. An earlier runner used the wrong expectation; +its failed sample is retained separately. + +The complete two-source-address LAN suite was repeated on d52e4e93 and passed +all WebConfig and Room cases in 161.67 seconds, with no temporary lockout left. +These network checks use guarded timing windows, not exact millisecond +boundary measurements. + +The actual saved-network recovery AP passed anonymous protected-config denial, +wrong-password rejection, existing-admin login/config read and logout. Status +required `needs_setup=false` and `needs_password=false`, distinguishing recovery +from initial password setup. The Pi used an owned temporary WLAN profile while +Ethernet management/default route stayed unchanged, then restored its original +WLAN profile and deleted only the temporary profile. + +The real Room browser also passed throttle-error polling, Connect retry and +draft preservation over this AP. The first AP browser attempt was an inconclusive +harness failure: the portal's favicon redirect was outside the strict request +allowlist. A narrow setup-mode redirect-chain check fixed that observer; the +second run passed with native fetch unchanged and zero compose/write requests. +Its unused icon follow-up failed outside the SSH-forwarded origin and is +reported separately from completed API response bodies. The AP BSSID was +measured, and the observed AP was open while protected settings still required +the saved admin password. AP two-source throttle testing remains unexecuted. + +An actual Room LoRa trial verified a temporary matching +910.525 MHz/62.5 kHz/SF7/CR5 profile before transmission. Shared guest-password +login returned non-admin RW2 permission, denied admin-only `ver`, and returned +the board index. The next `mail settings` request had local send admission but +no decoded reply before its 35-second deadline; no delivered reply was discarded +by the runner's type/prefix matching. This is an incomplete RF request/reply +trial, not a demonstrated mailbox authorization fault. Source inspection confirms +active sessions may read their own settings. Source contacts/returned settings, +logout and the Room's original saved/temporary radio state were restored. + +Post-trial counters showed 18 received packets (14 flood/four direct), six +transmits, one receive error and no outbound packet, with continuous MCU uptime. +This establishes reception on the temporary matching profile. It does not +identify which packet was lost or qualify mailbox mutation/retry behavior. + +A separate admin trial with an explicit two-second gap after each peer request +passed login, exact version/board reads, complete mailbox settings, self-send, +body readback, acknowledgement and both exact request replays. It required an +empty own inbox and provisioned only its unchanged closed policy. The first +replay returned the original message ID; replay after acknowledgement also +returned that ID without re-enqueueing. Final reads verified an empty queue and +the original mode, delivery and complete allow list. Source contacts/returned +settings, logout and the Room radio lease were restored. This qualifies the +explicitly paced live-session sequence, not unpaced reliability or durable +replay across reboot/cache eviction. Owner/revision/retired-receipt metadata +persisted until the separate original-filesystem restoration described above; +policy preservation alone does not mean byte-identical filesystem contents. +The earlier incomplete trial remains a failed sample. + +Actual Chromium/native-fetch testing exposed a loop-task stack overflow while +loading `board.index`. The saved core matches the 652 Room ELF and shows +`processRoomRequest -> loadRoomBoard -> recover -> filePresence -> snprintf` +reaching the stack-bottom watchpoint. Browser 202/429 response bodies also +were not consumed under their request deadline. + +Commit d3e05abc separates operation scratch frames and drains response bodies +under a bounded deadline. Actual browser runs with an empty board and with +an owned 2048-byte UTF-8 article both passed login, automatic throttle-error +polling, production Connect retry, preserved unsent drafts and unchanged +boot/browser identity. Every completed 202 response in these runs was drained. + +A subsequent populated-board stress run completed five whole article reads, +then triggered another panic during the sixth. Its fresh core positively +matches d3e05abc, reaches the loop stack bottom, and reconstructs the populated +index/filesystem/flash-cache/context-switch chain. The initial fix therefore +does **not** qualify repeated populated-board operations. Further stack +reduction and live retesting were required. + +Commit d52e4e93 clears index state in place, recovers directly into the output, +uses a checked heap allocation for web index metadata, and separates read/write +scratch frames. Host fault and allocation-failure checks pass, including +preserving an existing 2 KiB article on allocation failure. Stack-frame analysis +shows a substantial reduction, but its estimated margin is not a hardware pass. +The original owned article was recovered through exact title/version/content +checks into a new private intent after flashing this candidate. Twenty complete +live index/article reads passed with unchanged content, boot identity and +continuous uptime. This qualifies the reproduced repeated-read case; +original-filesystem restoration was verified separately after the Full handoff. + +The populated-board Chromium check also passed on d52e4e93: exact served HTML, +real admin form login, visible automatic throttle-error polling, Connect retry, +and unchanged unsent draft and Room boot identity. It used unmodified native +fetch, observed zero compose/write requests, and closed its owned browser. + +The later CLI preflight showed sticky error bit `8` +(`ERR_EVENT_RADIO_WATCHDOG`), versus zero shortly after the d52e4e93 boot. +At observation the radio reported RX, zero received packets/errors, one direct +transmit, no outbound packet, and noise floor -73 dBm. There was no MCU uptime +regression. The HTTP/content pass does not establish absence of radio recovery +events. +Source inspection shows this Full Room build enables the five-minute +successful-RX-silence watchdog even when MQTT is off. Its flag is set before +the soft re-arm outcome is known. With zero received packets, that policy is +consistent with the observed transition; it is not proof that HTTP caused a +fault. Here, with `recv=0`, `last_rx=0s ago` is the formatter's zero-timestamp +fallback, not recent-RX evidence. + +## Native official USB console + +The unmodified official `console.js` from commit +`45a5257bf792b11a83f4c51c18d4f312addfb677`, SHA256 +`107d3ad867915aea8b2454c324327397a0f9e2f0469656046054011bc64475f9`, +ran through Chromium's actual Linux Web Serial on the d52e4e93 V4 Room image. +Power saving was on; WebConfig WiFi, MQTT and ESP-NOW were off. + +With USB logging off, 600.35 seconds of uninterrupted idle transmitted zero +commands/keepalives. The first complete CRLF reply arrived in 303.2 ms; after +close/reopen, the reply arrived in 35.7 ms. MCU uptime advanced continuously +and the preexisting sticky error flags remained `8`. Native streams, owned +Chromium and the original TTY settings were all closed/restored. This is a +console lifecycle pass, not a current-consumption measurement. + +With USB logging on, the same image also passed 600.38 seconds with zero +commands/keepalives. The first reply took 164.5 ms and the reopened reply took +25.2 ms. Uptime remained continuous, sticky flags stayed `8`, and all streams, +owned Chromium and original TTY settings closed/restored. Power saving and USB +logging were then restored to their original off values. Earlier +startup/framing harness failures remain separate failed samples and do not +qualify idle. These passes bind the Room image; final Full Companion remains +a separate role-specific qualification. + +The ec9ebc09 Full Companion image subsequently passed logging-off trial 03: +600.444 seconds, zero idle writes, a 275 ms first reply and an 18.1 ms reopened +reply. Uptime advanced from 1284 to 1885 to 1888 seconds; error flags stayed +zero. Both closes confirmed return to binary mode, and streams, owned browser +and TTY settings closed/restored. + +Full logging-on trial 01 started at 09:02:25 UTC with baseline uptime 2076 +seconds and zero core error flags. Its nominal 600-second idle began around +09:03:30 UTC. At 09:11:13 the kernel reported the target hub port as +`disabled by hub (EMI?), re-enabling...`; the parenthetical is the kernel's +tentative wording, not a demonstrated EMI cause. Target USB path `1-1.2.1.2` +disconnected at 09:11:14, followed by repeated descriptor/address errors `-71`. +The kernel attempted a port power cycle at 09:11:15 and reported inability to +enumerate at 09:11:16. Both the bound by-id and physical-path endpoints remained +absent at 09:22. + +The run ended at 09:13:42 with `FileNotFoundError` while TTY cleanup reopened +the old path. That cleanup error masked the primary trial failure, which cannot +be reconstructed from the old safe report. No post-idle reply, reopen check or +qualified idle interval was obtained. USB recovery was still in progress at +the checkpoint; no firmware, reset or physical-fault cause was proven. +The passed Room logging-on case does not qualify this Full image. + +A subsequent identity-guarded, reset-only request to the exact parent hub's +port 2 returned success. Port status changed from `0x101` to `0x103` after two +seconds, with zero change bits: connected, powered and then enabled. The target +still did not enumerate; the sibling T096 was unchanged. No hub power change or +broader reset followed. A physical V4 RESET was requested from the operator; +recovery, temporary-setting restoration and the unstarted WSS soak remain +pending that action and verified device return. +ModemManager was restored to active at 09:24:34 UTC; all four checked host +services were active. This does not establish device return or settings recovery. + +For Full, `tools/hil/official_console_acceptance.py` uses explicit +`--terminal-mode companion`. It sends the shipping terminal-start control +through the unmodified official module on each open and confirms terminal-stop +on each logging-off close. Those four setup writes are counted separately from +the zero writes during idle. Full's labeled logging-status reply is checked +against the exact live mode. An earlier attempt omitted this handshake and +timed out before idle; another failed during owned Chromium startup before +serial opened. Both remain separate failed harness samples. + +## Shipping Morse results + +The unchanged Full RAK4631/MeshPocket images were tested at 902.1 MHz with +requested chip TX power at most -9 dBm, three different messages, 25/60 WPM, +both directions and two repeats. No HIL threshold, timing or sampling override +was enabled. All 24 decoded messages mismatched. Thirty-second no-transmitter +controls produced 20 characters on RAK and four on Pocket. + +Wanted peaks across all 24 trials ranged from -50.5 to -46.0 dBm; the 25 WPM +subset ranged from -50.5 to -47.5 dBm. Maximum observed valid sample gaps were +2-3 ms. Quiet RSSI excursions reached -110.5 dBm on RAK and +-92.5 dBm on Pocket, above their roughly -126/-125 dBm floors. This supports +an automatic-detection qualification failure, without identifying external RF +interference versus receiver-generated excursions. The reported `bad` counter +is consecutive read failures, not a cumulative guarantee of no SPI errors. +All decoded capture lengths matched their firmware headers and reached the +end marker, excluding host paging truncation in these trials. Source inspection +shows automatic squelch can admit sustained quiet excursions and corrupt mark +or gap boundaries; RSSI extrema alone do not establish those pulse durations. + +A separate five-minute Pocket quiet capture expired automatically after +300.40 seconds, retained seven false characters, and returned the same capture +on repeated firmware paging. Lifecycle/retrieval passed; silence detection +failed. Original settings were read back after both campaigns. Historical +successful HIL fixed-threshold trials do not qualify the shipping detector. + +## Host checks and CI + +The production Room fix passed the complete workflow at d52e4e93 +([run 39107275083](https://github.com/mikecarper/MeshCore/actions/runs/39107275083)). +A later tooling run exposed a host fixture that invoked real `fuser` against +its fake device path. The stricter ownership guard correctly refused that +diagnostic error. Commit 7a5a7993 makes the no-owner fixture deterministic and +retains real refusal tests for busy/error cases; its complete workflow passed +([run 39109551854](https://github.com/mikecarper/MeshCore/actions/runs/39109551854)). +The later AP browser-observer workflow also passed at `5fd33a230` +([run 39110179910](https://github.com/mikecarper/MeshCore/actions/runs/39110179910)); +both that run and the preceding 7a5a7993 run completed with all jobs green. +The native-console cleanup follow-up preserves the primary JS/transport +failure separately from browser/TTY cleanup failures, reports missing or +replaced endpoints, and cannot qualify a cleanup failure. The native-console +and Linux TTY suites passed 35 offline checks. Its new CI result was pending at +the authoring checkpoint; the completed runs do not qualify these follow-up +changes or rerun the failed hardware case. +Documentation route/link checks also passed in the CI-pinned renderer environment. + +## Remaining release gates + +- Resolve and retest T1000-E BLE; retain repeated populated-board stress as a + regression gate for subsequent Room candidates. +- Retain actual LoRa receive evidence alongside the Room HTTP results; the + quiet-radio watchdog flag is not a successful-RX or recovery-outcome counter. +- Diagnose shipping Morse noise rejection on a controlled RF setup. +- Execute the real MQTT/WSS soak with broker PUBACK and heap evidence; Full + V4 OLED has one runtime slot despite six saved configurations. Commit + ec9ebc091 exposes the existing read-only diagnostic counter on Full. The + recovery-guarded soak is staged and pinned but **not launched**. +- Recover the V4 USB endpoint, restore and verify its temporary power/logging/MQTT + flags, and diagnose/retest the failed Full logging-on idle case. Retain the + Full logging-off PASS and the two Room logging-mode passes as separate results. +- Complete two-source recovery-AP throttle checks beyond the passed existing-admin + authentication and browser cases. Execute migration credentials and both source + OTA slots, actual browser upload, interruption/recovery, privileged replay + persistence and transient guest churn. +- Execute contact-storage fault cases, LoRa full/delta OTA and SD recovery. +- Complete Indicator external WiFi/BLE endpoint, font, NTP and visible-rendering + checks beyond the passed LoRa mode/boot-state cycle. Resolve the ESP-NOW Full + build RAM gate before qualifying its two secondary transport combinations. +- Obtain physical power-removal and outdoor GPS assistance. USB reset and + software reboot do not establish battery-backed cold boot or sleep current. +- C6, STM32 and standalone RP2040 LoRa hardware are absent from this bench; + the Indicator RP2040 peripheral is not a substitute. +- Freeze the final source, run the canonical release build/package matrix and + finish release notes. This campaign binds several exact development images. + +Private raw snapshots, flash images, credentials, BLE PINs, inbox traffic and +cores remain outside the repository in the dated campaign directories.