rich/MeshCore
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Record Mercerwood candidate hardware results and open release gates

Browse Source
This commit is contained in:
mikecarper committed 2026-10-18 02:27:11 -07:00
1 parent 3ca724e73e
commit cfb9d166c4
2 files changed
+532 -17
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/test-results/hardware_validation_checklist.md
+81 -17
View File
@@ -14,6 +14,51 @@ when its log identifies the physical device, firmware artifact, artifact hash,
command result, and cold/warm boot outcome. Do not infer success from a tool's
exit code when the tool has a documented false-success mode.
## Mercerwood candidate campaign, 2026-10-18 UTC
The [current hardware campaign](hardware_validation_mercerwood_1_18_0_1_2026-10-18.md)
records actual candidate flashes, identity-bound USB/BLE and warm-contact
checks, live Room authentication/browser results, and shipping Morse failures.
It retains the original populated-board stack failures and the d52e4e93
repeated-read/browser retests that passed. The matching-profile trials established
LoRa reception and paced live-session mailbox operation; durable replay,
final-role testing and other release gates remain open. These observations do
not constitute release approval.
The ec9ebc09 Full V4 logging-off native-console trial 03 passed 600.444 seconds
with zero idle writes, complete replies/reopen and zero core error flags.
The passed Room logging-on trial does not qualify Full logging-on.
- [ ] **Full V4 logging-on idle/reopen:** trial 01 started at 09:02:25 UTC,
then lost target USB at 09:11:14. The kernel's preceding 09:11:13
`disabled by hub (EMI?), re-enabling...` message is not proof of EMI;
descriptor/address `-71` errors and failed enumeration continued through
09:11:16. The run ended at 09:13:42 with a TTY cleanup `FileNotFoundError`
masking its primary failure. No post-idle reply or qualified idle was obtained;
by-id/path remained absent at 09:22 and recovery was in progress. No firmware
or reset cause is established. The harness cleanup fix passed 35 offline
native-console/TTY checks and preserves primary and cleanup failures separately.
One subsequent identity-guarded reset-only request to exact parent-hub port 2
returned success; status changed `0x101` to `0x103` after two seconds, but the
target still did not enumerate and sibling T096 remained unchanged. No hub
power change or broader reset followed. Operator physical V4 RESET and a
verified return remain pending. ModemManager was restored active at 09:24:34
UTC and all four checked host services were active; device return remained
unverified.
- [ ] **Current V4 settings restoration:** original NVS/SPIFFS readback and
returned-state checks passed before the native trials. Subsequent temporary
power saving on, USB logging on and MQTT off remained **unrestored** at 09:22,
pending USB recovery. The earlier preservation PASS is not a final-state check.
- [ ] **Full V4 MQTT/WSS soak:** the pinned recovery-guarded runner is staged
but has **not launched**. Require real PUBACK/heap evidence and original-data
restoration before qualification.
Complete CI runs [39110179910](https://github.com/mikecarper/MeshCore/actions/runs/39110179910)
at `5fd33a230` and [39109551854](https://github.com/mikecarper/MeshCore/actions/runs/39109551854)
at `7a5a7993c` are green. The native-console/TTY cleanup follow-up passed 35
offline checks; its new CI result was pending at the authoring checkpoint.
The completed runs do not qualify those follow-up changes.
## PR17–21 follow-up qualification, 2026-10-18 UTC
The merged source checkpoint is `0a24279a3`. This audit adds host regressions,
@@ -44,11 +89,13 @@ and `esp32_s3_4mb_partition_migrator_usb`, including their application-size and
runtime-RAM gates. This covers representative chip families and Wi-Fi,
identity-only LoRa, Expander and USB code paths, rather than every board alias.
### Remaining physical acceptance checks
### Physical acceptance checks and subsequent campaign results
Record the exact clean source revision, firmware SHA-256, device identity,
observed result, and cold/warm boot outcome for each check. Record password
classes and success/failure, never the passwords themselves.
Checked entries below are later, candidate-specific campaign results; they do
not change the host-only scope of the earlier `0a24279a3` audit.
- [ ] **Migration credentials:** on cable-recoverable test hardware, join with
a distinct saved 8–15-character printable ASCII admin password. Reject the
@@ -61,18 +108,32 @@ classes and success/failure, never the passwords themselves.
identity/supported settings, final version, and cold-boot persistence.
Exercise supported interruption/retry points on sacrificial hardware; the
partition-table sector write is not atomic.
- [ ] **WebConfig:** use two distinct source addresses on LAN and on a
saved-network recovery AP. One address's five failures must not lock the
other out; exhausting the shared budget must reject even a correct new
login. An existing authorized session must remain usable. Verify recovery
at the six-second refill and 30-second address-lock boundaries. Clients
sharing one address also share its lock.
- [ ] **Room browser:** use separate real browser tabs/tokens for five
different wrong passwords. Nonempty credentials, including an already-open
admin tab, must wait for the shared window to expire; public empty-password
read-only access must remain available when enabled. Repeated stale-tab
passwords must not consume a new failure on every poll. Check visible error,
retry and unsaved-edit behavior against the real board.
- [x] **WebConfig LAN and Room password windows:** the complete two-source
LAN suite passed on d52e4e93 in 161.67 seconds. It covered per-address/shared
limits, existing-session access, refill/recovery, Room duplicate-password
suppression and the shared window, including nonempty credential rejection
and continued public empty-password read-only access. No temporary lockout
remained. These runs use guarded timing windows; they do not establish exact
millisecond boundary behavior or saved-network recovery-AP acceptance.
- [ ] **WebConfig saved-network recovery AP:** repeat the two-source checks
on the actual recovery AP. One address's five failures must not lock the
other out; shared-budget exhaustion must reject even a correct new login;
an existing session must remain usable. Verify refill/address-lock recovery
and remember that clients sharing one address also share its lock.
- [x] **Room browser:** actual Chromium on d52e4e93 passed real admin form
login, five distinct wrong passwords across two tokens, visible automatic
throttle-error polling, production Connect retry, and preserved unsent draft
and Room boot identity. The browser used unmodified native fetch and issued
no compose/write requests. This completes the previously open real-board
error/retry/draft case for this candidate.
- [x] **Room populated-board repeated reads:** d52e4e93 completed twenty
live index/article reads of the owned 2048-byte UTF-8 article with unchanged
content, boot identity and continuous uptime. Earlier 652/d3e05abc stack
failures remain historical evidence; subsequent candidates must retain this
regression check. A later sticky radio-watchdog flag `8` is consistent with
the five-minute no-successful-RX policy and does not establish an HTTP cause
or qualify radio reception. Original-filesystem restoration was independently
verified after the later Full Companion handoff.
- [ ] **Guest sessions:** churn transient guest identities, then admit a new
admin and an explicitly assigned writer. Power-cycle and verify privileged
login replay floors remain authoritative. Document the expected captured
@@ -90,9 +151,11 @@ classes and success/failure, never the passwords themselves.
sleep does not latch unrelated live pins. A successful build and simulated
pad retention do not establish physical latch or current behavior.
Remaining automated follow-ups are browser throttle-error/retry/draft cases
and an executed configuration-restore-to-AP credential handoff. The host AP
fixture supplies pre/post-restore files; it does not run an entire migration.
The later d52e4e93 campaign completes the real-board browser
throttle-error/retry/draft and reproduced repeated-read cases above. An executed
configuration-restore-to-AP credential handoff remains open. The host AP fixture
supplies pre/post-restore files; it does not run an entire migration. The current
hardware campaign remains in progress and is not release qualification.
## ESP32 web console and browser OTA release gates
@@ -135,7 +198,8 @@ successful build or simulated peripheral is not a hardware pass.
raw binary request. Confirm identity and settings were preserved, then restore temporary
device settings and the test host's network connection.
Keep the host regressions (`test_official_web_console.py`,
Keep the host regressions (`test_official_console_acceptance.py`,
`test_official_web_console.py`,
`test_wifi_ota_start.py`, `test_wifi_ota_lifecycle.py`, and
`test_async_web_lifecycle.py`) alongside these gates.
They cover parsing, sleep policy, startup errors, network ownership, and
docs/test-results/hardware_validation_mercerwood_1_18_0_1_2026-10-18.md
+451
View File
@@ -0,0 +1,451 @@
<!-- meshcore-hosted-doc-link:start -->
<p class="meshcore-hosted-doc-link"><a href="https://mikecarper.github.io/MeshCore/test-results/hardware_validation_mercerwood_1_18_0_1_2026-10-18/">View this page on MeshCore Docs</a>.</p>
<!-- meshcore-hosted-doc-link:end -->
# Mercerwood 1.18.0.1 hardware campaign, 2026-10-18 UTC
This campaign is in progress. It is not release approval. Results below bind
specific images and physical devices; successful compilation and older bench
results do not qualify unexecuted hardware cases.
At the 09:22 UTC checkpoint, the V4 Full logging-on trial had failed and its USB
endpoint remained absent. Recovery and restoration of subsequent temporary
settings were still in progress. The staged MQTT/WSS soak had not launched.
## Firmware and preservation
The initial eight-image Full matrix built successfully from clean source
`652289ee823bbed22918fe584f9d49ccb2ec230f`, with release-length version
`v1.18.0.1-halo-keymind-cascade-dev-652289ee`: V4 Room Server, V4.3 Companion
FEM-off, RAK4631, T096 FEM-on, MeshPocket, T1000-E, SenseCAP Indicator LoRa,
and Xiao S3 WIO. Room fixes were separately built and readback-verified at
`d3e05abce308b52261e86c9297029b4bce3cc4a1` and
`d52e4e93475e0960328131e2e65cc6b992aa4dbb`. The diagnostic-enabled Full
V4.3 FEM-off and Indicator LoRa images built successfully at
`ec9ebc0914d61a02fb21f23444768ec5f847e7ec`.
The later MeshTower V2 SD Full build and transfer are recorded separately below.
Before ESP32 application changes, complete private flash backups were made
through the pinned real flasher's identity/reset/finish path. Application-only
writes checked silicon MAC, USB identity, capacity, active OTA slot, image
integrity and full application readback. Partition tables and OTA metadata
were verified unchanged. Original NVS/filesystem restoration is a separate
operation; an application-only flash does not restore settings changed by tests.
The Indicator retains its existing noncanonical 8 MiB partition table.
nRF52 updates used application-only DFU. The RAK3401 service radio and
Indicator RP2040 display coprocessor were excluded. No root USB hub power
operation or concurrent PlatformIO process was used.
The flashed 652289ee artifacts below were built from source
`652289ee823bbed22918fe584f9d49ccb2ec230f`. USB identity labels are the first
12 hexadecimal characters of SHA256 of the UTF-8 USB serial as reported;
the serial-less CH340 Indicator was bound privately by its physical USB path
and public device identity. These hashes bind the executed images, not
unexecuted release qualification. ESP32 hashes cover application-only BINs;
nRF52 hashes cover the application DFU ZIPs actually transferred.
| Target | Source | USB serial hash | Artifact | SHA256 |
| --- | --- | --- | --- | --- |
| `heltec_v4_room_server` | `652289ee8` | `af0a529cb0cc` | Application BIN | `4e370dd285813ce330aae13a9a8891795145d3bbb7adef84ff4dbb19926cb677` |
| `RAK_4631_companion_radio_full` | `652289ee8` | `b11fbd38286f` | Application DFU ZIP | `64940ae543491831a20da2176e50e44a1519917bd42c4615601ddc0c7bf82364` |
| `Heltec_t096_companion_radio_full_femon` | `652289ee8` | `ab7bc0653271` | Application DFU ZIP | `0a44dbc7a39fb65007713aa61c004b8bb790035c1d6c343ba150a0a77f855dfa` |
| `Mesh_pocket_companion_radio_full` | `652289ee8` | `c5ba0587de70` | Application DFU ZIP | `eb13c05521f2545719db809c5367ed31de22e9aeadc0c0992366a6ab770570c5` |
| `t1000e_companion_radio_full` | `652289ee8` | `5244f0b64696` | Application DFU ZIP | `7a6003afa50720e2c97cfbb625692143f66b4e7de063de63106fecd5ff4fc5fb` |
| `SenseCapIndicator-LoRa_companion_radio_full` | `652289ee8` | No USB serial | Application BIN | `6cb1405fb2a6c3f4889506f184751d7576ad23159c303f84c054e4ad6b124b03` |
| `Xiao_S3_WIO_companion_radio_full` | `652289ee8` | `b00e0958f2ff` | Application BIN | `59c62ff6d985aa42740b6c26dcc30efe50c04b747d94d566e4ca4a8cd1e42e47` |
| `Heltec_tower_v2_sdcard_repeater_lora_ota_no_external_sensors` | `652289ee8` | `030f24857801` | Application DFU ZIP | `0d20f4486023a13427eb09087da70aacac5562848371a919631ab6a76f7888d1` |
The same V4's subsequent Room application hashes were
`54962d21088f1c87c3956620181a3499102e074d2e28791003d1b0479f64c7af`
for d3e05abc and
`7753f5f4161a6631d7f3abe44a99d70d5b892a94ad3cb69d313d54e44c608c87`
for d52e4e93. Their archived exact ELF hashes are respectively
`57d5b38fcc454d1ccc05763d3d8c6fa047904c42ca761be3a0393f14e799d1eb`
and `5db0fb151ee388ffab66ccce11b0d137907ff6450530f7812d35383c8c6aa785`.
After the Room campaign, the V4.3 Full FEM-off ec9ebc09 application passed
identity-bound application-only flashing and complete readback. Its 2,545,080-byte
BIN SHA256 is
`cb1e167a222e7a8d0ca253f7e3272561a29ba6753f4173eb89c79405b2bdad2f`.
The separate restoration then wrote only the original NVS and SPIFFS partitions
from the pinned pre-campaign full backup. Both partitions matched their original
bytes on readback; the active application, partition table and OTA metadata
were independently verified unchanged before runtime handoff.
A fresh Full snapshot matched all original `SELF_INFO` bytes, complete contact
records and all 40 canonical channels. Returned `DEVICE_INFO` settings matched
after excluding only the runtime active PIN and independently verified new
build/version metadata. There were no new core error flags. This qualifies
the original data readback and those returned settings, not every unexposed
preference or battery-backed cold boot. At that restoration checkpoint, the Full
runtime had WiFi off, an unset SSID, MQTT enabled but not running, and unavailable
ESP-NOW; temporary test changes require their own restoration. Later native-console
tests enabled power saving and USB logging and disabled MQTT. Those temporary flags remained
**unrestored** at 09:22 UTC, pending recovery of the missing USB endpoint.
The earlier NVS/SPIFFS readback PASS does not establish the current settings.
## Executed Companion checks
| Physical hardware | Candidate | USB queries/reopens | Owned contact add/update/delete | Add/delete survive warm reboots | BLE bonded hold/reconnect |
| --- | --- | --- | --- | --- | --- |
| RAK4631 | Full 652289ee | PASS: 100/10 | PASS | PASS | PASS: two 10-second sessions |
| T096 | Full FEM-on 652289ee | PASS: 100/10 | PASS | PASS | PASS: two 10-second sessions |
| MeshPocket | Full 652289ee | PASS: 100/10 | PASS | PASS | PASS: two 10-second sessions |
| T1000-E | Full 652289ee | PASS: 100/10 | PASS | PASS | FAIL/unqualified: four shipping attempts |
| Xiao S3 WIO | Full 652289ee | PASS: 100/10 | Not attempted: 350 occupied contacts | Not tested | Not tested |
| SenseCAP Indicator | LoRa Full 652289ee | Identity/readback and USB WiFi/BLE mode cycle PASS | Not tested | Not tested | Not tested |
Warm tests used a confirmed-absent random contact, verified complete readback,
performed the shipping guarded reboot, verified persistence, removed only the
recognized owned record, then rebooted and verified original state. They do
not prove cold boot. Display boards legitimately regenerate the active BLE
PIN when the saved PIN selects that policy; comparisons exclude only those
four runtime bytes and retain the private raw frames. Original contacts,
canonical channels, public identity, name and the other returned
`SELF_INFO`/`DEVICE_INFO` settings fields are checked. This comparison does
not inspect every persisted preference.
The Xiao retained all 350 contact keys and all 40 channels. A known contact's
advertisement and modification timestamps advanced during live mesh traffic;
this was not a contact edit by the runner. No existing Xiao contact was removed
to make room for a test record.
The Indicator LoRa Full mode cycle passed on 2026-10-18 UTC. USB checks
verified the exact board, complete candidate version and public identity,
then proved warm reboots into WiFi and BLE modes with the expected selected,
boot and live service states. The original BLE selector was restored and
another warm reboot verified. All 40 channels, the empty contact baseline,
returned binary settings, saved radio/RXPS/radio2 configuration and queried
Bluetooth preferences matched the original state, with no new core error
flags. The runner opened seven USB sessions and issued 510 framed requests;
it issued no RF message commands or partition/RP2040 writes. These checks do
not qualify an external TCP or bonded BLE session, font download/retention,
fresh NTP, visible rendering or cold boot.
The ESP-NOW Full build at `ec9ebc091` failed the mandatory internal RAM gate:
233,688 bytes available versus 251,168 required, a 17,480-byte shortfall.
Its native 480px WiFi canvas policy remains unchanged. Both ESP-NOW by
WiFi/BLE combinations remain unqualified; the normal Reduced ESP-NOW USB
image lacks those secondary transports and cannot substitute for their
Full acceptance.
T1000-E BLE initially resolves services and returns identity/core data, and
one attempt returned all five read-only getters. Later writes/reads fail with
backend service-discovery, ATT `0x0e`, or timeout errors. USB remains healthy.
The probe's final `disconnected` event is cleanup, not proof of which side
terminated a connection. These observations do not establish a root cause.
A fourth existing-bond trial captured HCI traffic. It completed encryption and
service discovery at MTU 247 and validated the complete version and public key.
The `get name` ATT Write Request (`0x12`) received its Write Response (`0x13`),
then no reply notification appeared. About 2.01 seconds later the controller
reported Connection Timeout (`0x08`), matching the negotiated two-second
supervision timeout and preceding the client's eight-second reply deadline.
No host HCI Disconnect command appeared in the capture. This trial therefore
lost the link before client cleanup. BLE uptime was 2720 seconds and later
USB uptime was 2993 seconds, consistent with the 272.70-second host interval;
both reported zero core error flags. This excludes a full MCU reboot in that
observed window, without identifying a firmware or RF cause. The post-trial
BlueZ state was disconnected and both owned processes closed. Negotiated link
parameters were a 30 ms interval, latency four and 2000 ms supervision timeout;
no PHY/data-length-change event appeared in the saved capture.
A separate diagnostic build enabled the existing RAM trace and disabled debug
printing, retaining the same source, version and connection parameters. Its
application DFU ZIP SHA256 is
`713765d9145db2aec03059a87280a065d4ae68e1a23ee4dd56a4d16f37b4f8c1`;
the exact ELF SHA256 is
`18442324b87deef9af658d2997b668ee10912929af4be9bacf9c473bc70d2169`.
This trial failed during GATT setup before any Companion request was sent.
The MCU initiated its documented no-app-data recovery exactly 15 seconds after
security completed. All 14 recorded stack dispatches exited, with a maximum
2.93 ms dispatch and 103 ms main-loop gap. All 62 trace records were retained,
with no ring wrap or recorded contention drops. HCI recorded the peripheral
termination and no host Disconnect command. This is a different failure stage
from the shipping post-getter supervision timeout; concurrent host Chromium
startup is a possible, unproven confounder. It does not identify a production
firmware fix. No old post-USB core metrics were reused for this trial.
The original shipping application was then restored through application-only
DFU. Identity, complete contact records, all 40 channels and all returned device
settings matched the immediately preceding baseline, including the active PIN;
core error flags were zero. An explicit read confirmed trace support absent.
The diagnostic image does not count as shipping BLE qualification.
GPS was enabled for a bounded 180-second local-only T1000-E telemetry window.
No location appeared. The exact GPS switch was restored, sync cadence was
unchanged, and packet TX counters did not advance. Outdoor acquisition,
fresh-fix age, stale/no-fix behavior and sleep current remain unqualified.
## MeshTower V2 SD
The MeshTower V2 SD Full image from `652289ee8` passed identity-bound,
application-only USB DFU and initial SD mount/list/space checks. All 16 returned
getter checks matched the original journals, including public identity,
radio/power/logging settings, `OTAFIX2.4.11-preview.2`, and InternalFS geometry
at `0xED000+28 KiB`. Ten consecutive SD space/list/core triplets completed all
30 replies: uptime advanced from 119 to 125 seconds, core error flags and
queue length remained zero, and the original directory metadata and reported
32.0 KiB used/959.6 MiB free stayed unchanged. The directory contained the same
single 4-byte `/mota/cache.off` marker; no file contents were exported. Original
firmware had error flags 2 before the update; the new boot is evaluated
separately. These checks qualify returned values and read-only SD access only.
They do not qualify card-absent fallback, cold initialization, interruption or
power-loss recovery, LoRa OTA transfer/install/serving, or a bootloader update.
No SD format/erase or deliberate LoRa OTA transmission was performed.
## Room authentication and browser
On the original 652289ee Room image, actual LAN requests from two separately
bound source addresses passed per-address failure isolation, shared-budget
exhaustion/refill, existing-session access, no token refund on success, and
post-window recovery. The Room password checks passed duplicate-password
suppression, five-unique-failure admission, the global 30-second window,
nonempty admin/guest rejection during lockout, and continued public blank
read-only access. Public guest wire role is `0`; role `1` denotes explicit
ACL read-only permission. An earlier runner used the wrong expectation;
its failed sample is retained separately.
The complete two-source-address LAN suite was repeated on d52e4e93 and passed
all WebConfig and Room cases in 161.67 seconds, with no temporary lockout left.
These network checks use guarded timing windows, not exact millisecond
boundary measurements.
The actual saved-network recovery AP passed anonymous protected-config denial,
wrong-password rejection, existing-admin login/config read and logout. Status
required `needs_setup=false` and `needs_password=false`, distinguishing recovery
from initial password setup. The Pi used an owned temporary WLAN profile while
Ethernet management/default route stayed unchanged, then restored its original
WLAN profile and deleted only the temporary profile.
The real Room browser also passed throttle-error polling, Connect retry and
draft preservation over this AP. The first AP browser attempt was an inconclusive
harness failure: the portal's favicon redirect was outside the strict request
allowlist. A narrow setup-mode redirect-chain check fixed that observer; the
second run passed with native fetch unchanged and zero compose/write requests.
Its unused icon follow-up failed outside the SSH-forwarded origin and is
reported separately from completed API response bodies. The AP BSSID was
measured, and the observed AP was open while protected settings still required
the saved admin password. AP two-source throttle testing remains unexecuted.
An actual Room LoRa trial verified a temporary matching
910.525 MHz/62.5 kHz/SF7/CR5 profile before transmission. Shared guest-password
login returned non-admin RW2 permission, denied admin-only `ver`, and returned
the board index. The next `mail settings` request had local send admission but
no decoded reply before its 35-second deadline; no delivered reply was discarded
by the runner's type/prefix matching. This is an incomplete RF request/reply
trial, not a demonstrated mailbox authorization fault. Source inspection confirms
active sessions may read their own settings. Source contacts/returned settings,
logout and the Room's original saved/temporary radio state were restored.
Post-trial counters showed 18 received packets (14 flood/four direct), six
transmits, one receive error and no outbound packet, with continuous MCU uptime.
This establishes reception on the temporary matching profile. It does not
identify which packet was lost or qualify mailbox mutation/retry behavior.
A separate admin trial with an explicit two-second gap after each peer request
passed login, exact version/board reads, complete mailbox settings, self-send,
body readback, acknowledgement and both exact request replays. It required an
empty own inbox and provisioned only its unchanged closed policy. The first
replay returned the original message ID; replay after acknowledgement also
returned that ID without re-enqueueing. Final reads verified an empty queue and
the original mode, delivery and complete allow list. Source contacts/returned
settings, logout and the Room radio lease were restored. This qualifies the
explicitly paced live-session sequence, not unpaced reliability or durable
replay across reboot/cache eviction. Owner/revision/retired-receipt metadata
persisted until the separate original-filesystem restoration described above;
policy preservation alone does not mean byte-identical filesystem contents.
The earlier incomplete trial remains a failed sample.
Actual Chromium/native-fetch testing exposed a loop-task stack overflow while
loading `board.index`. The saved core matches the 652 Room ELF and shows
`processRoomRequest -> loadRoomBoard -> recover -> filePresence -> snprintf`
reaching the stack-bottom watchpoint. Browser 202/429 response bodies also
were not consumed under their request deadline.
Commit d3e05abc separates operation scratch frames and drains response bodies
under a bounded deadline. Actual browser runs with an empty board and with
an owned 2048-byte UTF-8 article both passed login, automatic throttle-error
polling, production Connect retry, preserved unsent drafts and unchanged
boot/browser identity. Every completed 202 response in these runs was drained.
A subsequent populated-board stress run completed five whole article reads,
then triggered another panic during the sixth. Its fresh core positively
matches d3e05abc, reaches the loop stack bottom, and reconstructs the populated
index/filesystem/flash-cache/context-switch chain. The initial fix therefore
does **not** qualify repeated populated-board operations. Further stack
reduction and live retesting were required.
Commit d52e4e93 clears index state in place, recovers directly into the output,
uses a checked heap allocation for web index metadata, and separates read/write
scratch frames. Host fault and allocation-failure checks pass, including
preserving an existing 2 KiB article on allocation failure. Stack-frame analysis
shows a substantial reduction, but its estimated margin is not a hardware pass.
The original owned article was recovered through exact title/version/content
checks into a new private intent after flashing this candidate. Twenty complete
live index/article reads passed with unchanged content, boot identity and
continuous uptime. This qualifies the reproduced repeated-read case;
original-filesystem restoration was verified separately after the Full handoff.
The populated-board Chromium check also passed on d52e4e93: exact served HTML,
real admin form login, visible automatic throttle-error polling, Connect retry,
and unchanged unsent draft and Room boot identity. It used unmodified native
fetch, observed zero compose/write requests, and closed its owned browser.
The later CLI preflight showed sticky error bit `8`
(`ERR_EVENT_RADIO_WATCHDOG`), versus zero shortly after the d52e4e93 boot.
At observation the radio reported RX, zero received packets/errors, one direct
transmit, no outbound packet, and noise floor -73 dBm. There was no MCU uptime
regression. The HTTP/content pass does not establish absence of radio recovery
events.
Source inspection shows this Full Room build enables the five-minute
successful-RX-silence watchdog even when MQTT is off. Its flag is set before
the soft re-arm outcome is known. With zero received packets, that policy is
consistent with the observed transition; it is not proof that HTTP caused a
fault. Here, with `recv=0`, `last_rx=0s ago` is the formatter's zero-timestamp
fallback, not recent-RX evidence.
## Native official USB console
The unmodified official `console.js` from commit
`45a5257bf792b11a83f4c51c18d4f312addfb677`, SHA256
`107d3ad867915aea8b2454c324327397a0f9e2f0469656046054011bc64475f9`,
ran through Chromium's actual Linux Web Serial on the d52e4e93 V4 Room image.
Power saving was on; WebConfig WiFi, MQTT and ESP-NOW were off.
With USB logging off, 600.35 seconds of uninterrupted idle transmitted zero
commands/keepalives. The first complete CRLF reply arrived in 303.2 ms; after
close/reopen, the reply arrived in 35.7 ms. MCU uptime advanced continuously
and the preexisting sticky error flags remained `8`. Native streams, owned
Chromium and the original TTY settings were all closed/restored. This is a
console lifecycle pass, not a current-consumption measurement.
With USB logging on, the same image also passed 600.38 seconds with zero
commands/keepalives. The first reply took 164.5 ms and the reopened reply took
25.2 ms. Uptime remained continuous, sticky flags stayed `8`, and all streams,
owned Chromium and original TTY settings closed/restored. Power saving and USB
logging were then restored to their original off values. Earlier
startup/framing harness failures remain separate failed samples and do not
qualify idle. These passes bind the Room image; final Full Companion remains
a separate role-specific qualification.
The ec9ebc09 Full Companion image subsequently passed logging-off trial 03:
600.444 seconds, zero idle writes, a 275 ms first reply and an 18.1 ms reopened
reply. Uptime advanced from 1284 to 1885 to 1888 seconds; error flags stayed
zero. Both closes confirmed return to binary mode, and streams, owned browser
and TTY settings closed/restored.
Full logging-on trial 01 started at 09:02:25 UTC with baseline uptime 2076
seconds and zero core error flags. Its nominal 600-second idle began around
09:03:30 UTC. At 09:11:13 the kernel reported the target hub port as
`disabled by hub (EMI?), re-enabling...`; the parenthetical is the kernel's
tentative wording, not a demonstrated EMI cause. Target USB path `1-1.2.1.2`
disconnected at 09:11:14, followed by repeated descriptor/address errors `-71`.
The kernel attempted a port power cycle at 09:11:15 and reported inability to
enumerate at 09:11:16. Both the bound by-id and physical-path endpoints remained
absent at 09:22.
The run ended at 09:13:42 with `FileNotFoundError` while TTY cleanup reopened
the old path. That cleanup error masked the primary trial failure, which cannot
be reconstructed from the old safe report. No post-idle reply, reopen check or
qualified idle interval was obtained. USB recovery was still in progress at
the checkpoint; no firmware, reset or physical-fault cause was proven.
The passed Room logging-on case does not qualify this Full image.
A subsequent identity-guarded, reset-only request to the exact parent hub's
port 2 returned success. Port status changed from `0x101` to `0x103` after two
seconds, with zero change bits: connected, powered and then enabled. The target
still did not enumerate; the sibling T096 was unchanged. No hub power change or
broader reset followed. A physical V4 RESET was requested from the operator;
recovery, temporary-setting restoration and the unstarted WSS soak remain
pending that action and verified device return.
ModemManager was restored to active at 09:24:34 UTC; all four checked host
services were active. This does not establish device return or settings recovery.
For Full, `tools/hil/official_console_acceptance.py` uses explicit
`--terminal-mode companion`. It sends the shipping terminal-start control
through the unmodified official module on each open and confirms terminal-stop
on each logging-off close. Those four setup writes are counted separately from
the zero writes during idle. Full's labeled logging-status reply is checked
against the exact live mode. An earlier attempt omitted this handshake and
timed out before idle; another failed during owned Chromium startup before
serial opened. Both remain separate failed harness samples.
## Shipping Morse results
The unchanged Full RAK4631/MeshPocket images were tested at 902.1 MHz with
requested chip TX power at most -9 dBm, three different messages, 25/60 WPM,
both directions and two repeats. No HIL threshold, timing or sampling override
was enabled. All 24 decoded messages mismatched. Thirty-second no-transmitter
controls produced 20 characters on RAK and four on Pocket.
Wanted peaks across all 24 trials ranged from -50.5 to -46.0 dBm; the 25 WPM
subset ranged from -50.5 to -47.5 dBm. Maximum observed valid sample gaps were
2-3 ms. Quiet RSSI excursions reached -110.5 dBm on RAK and
-92.5 dBm on Pocket, above their roughly -126/-125 dBm floors. This supports
an automatic-detection qualification failure, without identifying external RF
interference versus receiver-generated excursions. The reported `bad` counter
is consecutive read failures, not a cumulative guarantee of no SPI errors.
All decoded capture lengths matched their firmware headers and reached the
end marker, excluding host paging truncation in these trials. Source inspection
shows automatic squelch can admit sustained quiet excursions and corrupt mark
or gap boundaries; RSSI extrema alone do not establish those pulse durations.
A separate five-minute Pocket quiet capture expired automatically after
300.40 seconds, retained seven false characters, and returned the same capture
on repeated firmware paging. Lifecycle/retrieval passed; silence detection
failed. Original settings were read back after both campaigns. Historical
successful HIL fixed-threshold trials do not qualify the shipping detector.
## Host checks and CI
The production Room fix passed the complete workflow at d52e4e93
([run 39107275083](https://github.com/mikecarper/MeshCore/actions/runs/39107275083)).
A later tooling run exposed a host fixture that invoked real `fuser` against
its fake device path. The stricter ownership guard correctly refused that
diagnostic error. Commit 7a5a7993 makes the no-owner fixture deterministic and
retains real refusal tests for busy/error cases; its complete workflow passed
([run 39109551854](https://github.com/mikecarper/MeshCore/actions/runs/39109551854)).
The later AP browser-observer workflow also passed at `5fd33a230`
([run 39110179910](https://github.com/mikecarper/MeshCore/actions/runs/39110179910));
both that run and the preceding 7a5a7993 run completed with all jobs green.
The native-console cleanup follow-up preserves the primary JS/transport
failure separately from browser/TTY cleanup failures, reports missing or
replaced endpoints, and cannot qualify a cleanup failure. The native-console
and Linux TTY suites passed 35 offline checks. Its new CI result was pending at
the authoring checkpoint; the completed runs do not qualify these follow-up
changes or rerun the failed hardware case.
Documentation route/link checks also passed in the CI-pinned renderer environment.
## Remaining release gates
- Resolve and retest T1000-E BLE; retain repeated populated-board stress as a
regression gate for subsequent Room candidates.
- Retain actual LoRa receive evidence alongside the Room HTTP results; the
quiet-radio watchdog flag is not a successful-RX or recovery-outcome counter.
- Diagnose shipping Morse noise rejection on a controlled RF setup.
- Execute the real MQTT/WSS soak with broker PUBACK and heap evidence; Full
V4 OLED has one runtime slot despite six saved configurations. Commit
ec9ebc091 exposes the existing read-only diagnostic counter on Full. The
recovery-guarded soak is staged and pinned but **not launched**.
- Recover the V4 USB endpoint, restore and verify its temporary power/logging/MQTT
flags, and diagnose/retest the failed Full logging-on idle case. Retain the
Full logging-off PASS and the two Room logging-mode passes as separate results.
- Complete two-source recovery-AP throttle checks beyond the passed existing-admin
authentication and browser cases. Execute migration credentials and both source
OTA slots, actual browser upload, interruption/recovery, privileged replay
persistence and transient guest churn.
- Execute contact-storage fault cases, LoRa full/delta OTA and SD recovery.
- Complete Indicator external WiFi/BLE endpoint, font, NTP and visible-rendering
checks beyond the passed LoRa mode/boot-state cycle. Resolve the ESP-NOW Full
build RAM gate before qualifying its two secondary transport combinations.
- Obtain physical power-removal and outdoor GPS assistance. USB reset and
software reboot do not establish battery-backed cold boot or sleep current.
- C6, STM32 and standalone RP2040 LoRa hardware are absent from this bench;
the Indicator RP2040 peripheral is not a substitute.
- Freeze the final source, run the canonical release build/package matrix and
finish release notes. This campaign binds several exact development images.
Private raw snapshots, flash images, credentials, BLE PINs, inbox traffic and
cores remain outside the repository in the dated campaign directories.