diff --git a/docs/test-results/hardware_validation_ota_1_18_0_1_2026-10-20.md b/docs/test-results/hardware_validation_ota_1_18_0_1_2026-10-20.md index 82657624a..7177f7e53 100644 --- a/docs/test-results/hardware_validation_ota_1_18_0_1_2026-10-20.md +++ b/docs/test-results/hardware_validation_ota_1_18_0_1_2026-10-20.md @@ -4,25 +4,30 @@ # OTA hardware qualification, 20-21 October 2026 UTC -Status: IN PROGRESS. Results below describe completed measurements only. +Status: COMPLETED for the listed OTA cases. Earlier failures and unresolved +observations remain recorded below; this is not a claim that every board or +transport has passed all release qualification. The operator authorized erasing these test nodes without backups. MeshPocket -remains excluded. No bootloader or SoftDevice payload is part of this campaign. +remains excluded. No nRF52 bootloader or SoftDevice payload is part of this +campaign. Temporary ESP32 bridge role changes use the board's merged image, +including its ESP32 bootloader and partitions, under the authorized erase. -Completed hardware OTA results use artifacts pinned to +The initial hardware OTA results use artifacts pinned to `8b5660b00b88d8abaae094859bc1232f7ce886fd`. That commit changes documentation only relative to firmware source `0b207a1ec3a32eb0a8ad6b8643db211de4921cd4`. Separate OTA-check version strings produce distinct image hashes for controlled update tests. The common numeric version is `0x01120001`; same-version updates are explicitly authorized for these tests. They are not claimed as numeric version upgrades. +Later source, CPU-sleep and one-hop results identify their own commits below. | Device and role | Transport | Result | | --- | --- | --- | | Heltec V4.3 Full Companion | Production browser WiFi OTA | PASS: one upload installed the exact intended image; postboot identity, version, core, radio and original flags verified | -| RAK4631 unified repeater | Bluetooth application DFU; QSPI full LoRa OTA | Transfers and eventual postchecks PASS; initial USB version response remains under investigation | +| RAK4631 unified repeater | Bluetooth application DFU; QSPI full LoRa OTA; two-frequency bridged one-hop LoRa delta | Transfers and eventual postchecks PASS, including one-hop installation; initial USB version response remains under investigation | | Seeed XIAO nRF52 repeater | Bluetooth application DFU; external QSPI full LoRa OTA | Transfers and eventual postchecks PASS; initial USB version response remains under investigation | | RAK3401 unified repeater and gateway | Bluetooth application DFU; internal LoRa delta; gateway restoration | PASS: Bluetooth update and repeater service, internal LoRa delta apply, normal-radio restoration and gateway service handback | -| MeshTower SD repeater | Signed SD-backed LoRa delta and full update | Delta transfer, signature verification and postboot checks PASS; full update pending | +| MeshTower SD repeater | Signed SD-backed LoRa delta and full update | PASS: signed SD delta and full-image transfer, installation and LoRa postboot checks | ## V4 WiFi OTA @@ -157,6 +162,59 @@ is not treated as missing command support or a first-attempt pass. The successful image, role, settings and core checks above describe the later independent verification. +## Two-frequency ESP-NOW bridged one-hop LoRa update + +The physical one-hop test PASSED. The V4 source ran `2bd4cb4d1` and the +RAK4631 received an application-only delta to `bd4524318`. Its setup, +discovery, download, progress checks, install command, postboot proof and +settings restoration all used LoRa. The receiving RAK's USB and Bluetooth +interfaces were never opened for this case. + +The measured path was V4 at 909.5 MHz to a V3 bridge, ESP-NOW to a XIAO +ESP32 bridge, then LoRa at 908.5 MHz to the RAK4631. Both LoRa segments used +500 kHz bandwidth, SF7 and CR5. The two bridges copied received packets before +appending their own hop, with one-hop limits. Disabling the far bridge's +ESP-NOW link made the target's full-key query time out; re-enabling and +verifying the same configuration restored that query. This counterfactual +confirms that the two-frequency bridge was required. + +The ordinary file-data session populated the source catalog and its automatic +discovery burst. No separate announcement command or control channel was +used. The first asynchronous catalog query requested discovery; the next +query found the intended manifest. Download completed **62/62 blocks**. +The one install command received an on-air acknowledgement verifying the +unsigned QSPI delta. Subsequent normal-channel LoRa reads proved exact +installed body `D2073EC88FF4E459`, the full `dev-bd452431` version and zero +core errors. Original target policies, receive power saving, output path and +the source's saved contact path were restored. The package SHA256 is +`f0a7319570394e23f06c9a901838be3cbaf5ae520d2743607828977e5fa3a0e9`. + +A passive source USB capture wrote zero bytes to the radio and recorded +47 matching update requests with one hop through the measured near bridge. +This is source-side packet evidence, not an independent RF capture of the +whole chain. The separate frequencies, bridge counterfactual and on-air +install/postboot proofs establish the complete case. The integrated private +receipt SHA256 is +`c6d6b2e2d3aefec128303b0ef5e2d8a0be52c38f03bc17ecf3032f4118489cc6`. + +Preparation failures remain recorded separately: a historical channel-busy +flag prompted one documented LoRa restart, and several bridge receive-power +configuration attempts returned a generic busy/unsupported rejection before +a fresh configuration check and acknowledged attempt succeeded. No cause +for those rejections or the historical busy flag is claimed fixed. + +Both temporary ESP32 bridges were returned to Full Companion images built +from `ebb59d482`. Exact runtime bodies were `E297CF6E342C8332` (V3) and +`4EEB12667E5E554B` (XIAO ESP32), with matching Binary/CLI identity and zero +core errors. The V3's first postflash Binary handshake timed out; a later +independent read passed without a reset or another flash. Its role reply +required a narrow correction to the bench parser's nested prompt handling. +The XIAO's positive-readiness check also rejected a valid prompt-prefixed +acknowledgement before sending any Binary request. After correcting that +check, its first Binary handshake and complete CLI checks passed without +another flash, reset or settings change. Original failed workflow receipts +remain retained. These bench corrections are not claimed as firmware fixes. + ## Findings and fixes An authenticated nRF52 application handoff does not preserve a custom @@ -222,8 +280,9 @@ retries lost remote clock reads within the existing bounded operation budget. Validation passed 671 tests, including nine new clock-read regressions; one optional PyPI integration test was skipped. Canonical Full and Reduced RAK4631 and RAK3401 pairs, plus Full V3 and XIAO S3 WIO images, built -successfully from this commit. These newly built images have not yet been -flashed; their build success does not replace the earlier hardware results. +successfully from this commit. The ESP32 images were subsequently installed +as the temporary one-hop bridges. Build success alone does not qualify an +update or replace the earlier hardware results. Repeater Bluetooth provides an update service; Companion terminal/sync functionality is not claimed for repeater builds. The RAK4631 reports a 2 MiB @@ -254,8 +313,8 @@ During RAK4631 discovery, source receive power saving was on and discovery timed out. After disabling that setting under a restoration intent, a fresh advert was received. This sequence records an observation; it does not establish a receive-power-saving firmware defect. Source receive power -saving remains off under the owned restoration intent while the remaining -LoRa cases run; final source restoration is pending. +saving stayed off under the owned restoration intent while the remaining +LoRa cases ran. Exact final source restoration is recorded below. ## MeshTower signed SD updates and remaining failures @@ -290,8 +349,8 @@ clock-read retry fix above addresses the host's handling of lost reads. A second full-image attempt at SF7 and 500 kHz bandwidth passed the full-key and status proofs, but all three `ota self` proof attempts failed. Safe temporary-lease expiry and return to the normal radio were verified. No -firmware was transferred in either full-image attempt; full apply and -postchecks remain pending. +firmware was transferred in either of those full-image attempts; full apply +and postchecks were still pending at that point. A local USB snapshot afterward confirmed body `F1909AD4C46915FD`, 1133 seconds uptime, core error bitmask `0x02` and an empty queue. Radio diagnostics @@ -318,21 +377,136 @@ fresh identity/settings check followed by one new, acknowledged restoration attempt proved the original configuration. No particular runtime guard is identified as the cause of that rejection. +A later signed-full retry used the ordinary production file-data session and +source catalog readiness, with every receiver command on LoRa. Its first +attempt stopped before new data because an old failed receive session +required an explicit replacement choice. The one-minute temporary-channel +cleanup command lost its acknowledgement. The updater then proved identity +on the temporary channel and restored the source, but did not prove that the +receiver had returned to its normal channel. The next attempt could not +reach the receiver on the normal channel. These are failed workflow records. + +A bounded, LoRa-only recovery found the exact Tower still running body +`F1909AD4C46915FD` on the temporary channel. One command restored its normal +radio; its acknowledgement was lost and the command was not replayed. +Subsequent authenticated normal-channel reads proved the full identity, +unchanged body, inactive temporary mode, normal radio and 0 dBm TX. No target +USB, Bluetooth, reset or flash operation was used. A historical core bitmask +`0x06` was retained: channel-busy timeout and radio outside receive for over +eight seconds. It is a bitmask, not a count of six errors. The recovery receipt +SHA256 is +`bc44880091e0024966fb6b2d9fd77b21556a9899e9c9995cf772dc14ae1bd57e`. + +After that recovery, a fresh ordinary updater attempt passed its temporary +channel rehearsal and natural return, explicitly replaced only the old failed +receive session, and downloaded the signed full image **279/279 blocks in +831 seconds**. The install acknowledgement verified a trusted signer and a +full image on SD. Normal-channel LoRa postboot reads proved body +`718365A6E08823C7`, bootloader apply OK and the full `dev-ebb59d48` version. +The first matching body response arrived during the 60-second postboot probe. +A lost optional statistics reply required a bounded getter retry. The +570682-byte package SHA256 is +`4f92525f7500ad74b7f1895a62c4626a02a049b1a3978906ae8eea8617393bde`. +The production runner exited successfully; its private result SHA256 is +`de1bb4c4fa7af7fe627c00d96c94d1f7244141556c29271754379ad8f7fd7ca9`. +All receiver setup, download, install and postboot commands used LoRa; its USB +and Bluetooth interfaces were never opened for this attempt. The source was +fed through its ordinary file-data session, with no extra announcement or +separate receiver control channel. + +An independent, authenticated LoRa postcheck then verified the new full body +and version, repeater role, normal radio/TX, inactive temporary mode, zero +core error bitmask and an empty transmit queue. It removed only the owned +fourth test signing key and verified that the original three key prefixes +remained in their original order. The CLI exposes prefixes, so this is not +an independent full-byte readback of those three keys. The postcheck repeated +the identity, radio and core checks after removal and closed its controller. +Lost getter/status replies used bounded read-only retries; all failed attempts +remain recorded. Its private summary SHA256 is +`02964ab9a2f511f96055ef6c99c59d32bb091d21dbf5c61dbe9de5ac00555659`. + +The passive source-only capture recorded 405398 bytes and wrote zero USB +bytes. Its SHA256 is +`6558971dea7e9cdc221e3e196d34f5ae00427300932c766e3fb7798d3332d194`. +The capture was stopped and its serial port closed before the source's +original logging-off setting was restored and verified. This capture is +source-side evidence; the receiver's installed-image and runtime proofs +above came from LoRa. + +The host cleanup defect is separate from the radio transfer: temporary-channel +identity alone cannot establish that a lost relative-lease command executed. +Cleanup must retain its recovery obligation until a full-key proof and +inactive temporary mode are observed on the normal channel. The same +requirement applies independently to controlled relay participants. + +Commit [c5c67f8f7](https://github.com/mikecarper/MeshCore/commit/c5c67f8f7) +fixes that host ownership defect. It persists a private recovery record for +each full-key-bound target and controlled relay before arming, records a +one-shot shortening attempt before sending, and requires each participant's +full identity plus inactive temporary mode on the normal channel before +retiring its obligation. A shared Full Companion source must also report its +actual temporary mode inactive; its saved Binary radio settings alone are +insufficient. Lost relative-lease acknowledgements are not replayed. Unresolved +records survive interruption without passwords, and manual read-only recovery +is documented in the [LoRa OTA automation guide](../lora_ota_automation.md). +The regression cases cover delivered and undelivered lost acknowledgements, +independent relay failures, mismatched keys, active temporary profiles, +interruptions, existing recovery records, resolved-password redaction and the +explicit no-install staging path. These are host changes; accepted unchanged +firmware was not flashed again to validate them. + +## Final source restoration and handback + +After all transfers and the Tower signing-key cleanup, two fresh USB cleanup +attempts received an explicit `Error: radio busy; retry` while restoring the +source's original RXPS setting. Both attempts stopped and retained their raw +failed records. This firmware rejects that command before updating or saving +RXPS preferences; a missing or ambiguous reply would not permit the same +continuation. Fresh read-only diagnostics showed normal receive operation, +zero core errors, an empty queue and RXPS still off. They did not identify +which radio reconfiguration guard rejected those particular commands. + +A bounded continuation used the production updater's varying retry cadence +for this exact explicit busy response, opening a fresh console after each +rejection. Its second attempt received the exact positive acknowledgement +and exact original configuration readback: on, level 8, preamble 16, +39657 microseconds receive and 15975 microseconds sleep. It did not retry a +lost or malformed acknowledgement, reset the radio or change firmware. +The continuation receipt SHA256 is +`4253c91dbf2eca91e7b2716367e27e95a7dd1b952008b9ef6bb036fd673d3c62`. + +A separate final cleanup skipped the already restored RXPS setting, restored +all five original saved flags, and verified normal radio/TX, idle detached OTA +file service, original hop settings and strict USB Binary handback. It disabled +the test WiFi runtime before restoring MQTT. The completed private summary +SHA256 is +`1c07fc8301f883eb3a33201924ca77ce4d2d02573b9a27a785f84547a06a8165`. +The hash-bound completed receipt retired the host's RXPS restoration intent. +Owned test tunnels, target leases and the cold-idle hold were closed; the +root hardware session had no remaining FLOCK descriptors before it exited. +The already restored RAK3401 gateway service was left running. + ## Limits and completion -The MeshTower signed full-image LoRa case and final source cleanup remain -pending. The live cold-idle RAK preflight check is complete. Newly requested -two-frequency ESP-NOW bridged one-hop OTA testing remains pending. Latest -RAK3401 CPU sleep, post-idle LoRa and gateway restoration passed. Receiver -update setup, progress, installation -and post-install verification in the new radio campaign must use LoRa; -receiver USB commands cannot substitute for failed radio communication. -The RAK3401's current-image pause snapshot recorded historical channel-busy -bitmask `0x02`; it was retained without clearing statistics or claiming a -CPU-sleep result from that historical snapshot. The first-command USB bootstrap observation, -Tower SF5 preflight failure, later full-image self-proof failures and separate -USB open failure remain unresolved. The Tower signed delta, RAK4631 and XIAO -nRF52 full QSPI LoRa updates and their later postchecks are complete; the -RAK3401 gateway handback is complete for the earlier pinned image. +The listed physical OTA cases and their restoration are complete: V4 WiFi, +RAK3401 internal delta, RAK4631 and XIAO nRF52 external QSPI full images, +Tower signed SD delta/full, and the two-frequency ESP-NOW bridged one-hop +update. Latest RAK3401 CPU sleep, post-idle LoRa and gateway handback passed. +Each case records its exact firmware commit and body; these results do not +qualify all 13 radios against every later commit. + +Receiver setup, progress, installation and post-install verification in the +new radio campaign used LoRa. Receiver USB commands did not substitute for +failed radio communication. Earlier local-control diagnostics remain +explicitly identified. The RAK3401's pre-fix historical channel-busy bitmask +`0x02` was retained without clearing statistics or claiming a sleep result +from that snapshot. + +The first-command USB bootstrap observation, Tower SF5 preflight failure, +earlier full-image self-proof failures, separate USB open failure and +historical radio-busy flags remain unexplained. Later full-image transfer and +postchecks passed under the stated conditions; this does not establish their +underlying causes. Source restoration's explicit busy replies were handled +with bounded retries; no radio guard defect is claimed fixed by that action. Private raw responses, keys, PINs and device identities stay outside the repo. Earlier accepted Companion tests are not repeated without a relevant change.