rich/MeshCore
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: harden LoRa OTA chain and storage

Browse Source
This commit is contained in:
mikecarper committed 2026-08-29 06:58:48 -07:00
1 parent 400aa710a2
commit b40d2e6cb2
29 files changed
+7497 -528
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/lora_ota_automation.md
+89 -26
View File
@@ -262,6 +262,22 @@ runner can restore those periods but cannot reconstruct an unreported saved
level. A radio change later recalculates from a preserved saved minimum, so
moving back to a slower tuple returns to the operator's saved level.
The OTA source has a stricter policy than the destination. For every source
with a managed serial or TCP CLI, the runner reads and retains its exact RXPS
preference, using the legacy fixed-period query only when the detailed query is
unavailable. It then disables RXPS and verifies the readback before any target
discovery or TempRadio change. Source RXPS stays off through catalog serving,
download, installation, and post-install identity verification. Cleanup first
proves that the source has returned to its normal radio, then restores and
verifies the exact saved level/preamble or fixed-period state once. A source
whose RXPS state cannot be read, disabled, or restored safely fails closed.
If current firmware explicitly rejects an RXPS disable or restore with `radio
busy; retry`, the runner retries that idempotent mutation at staggered
210–378 ms intervals. All 32 delays are distinct and contribute about 9.4
seconds of waits; source-CLI command round-trip time is additional. This avoids
repeatedly sampling one radio phase while retaining a strict attempt cap. Other
rejections are not replayed.
## 4. Run an ESP32 update
The ZIP can contain a compatible ready `.mota` or the exact board-and-role
@@ -427,7 +443,9 @@ later as a catalog timeout.
If the source is already on the exact TempRadio tuple through a scheduled or
manual operation, `--source-already-temp` lets a TCP source run without a raw
CLI link. The script cannot verify, extend, or shorten that source window, so
leave a comfortable time margin.
leave a comfortable time margin. It also cannot inspect or change that
unmanaged source's RXPS state; disable source RXPS yourself before starting
the run and restore it only after returning the source to its normal radio.
Use `--controller-baud` or `--source-baud` only for a build whose corresponding
interface is genuinely configured to another speed.
@@ -458,7 +476,9 @@ Useful controls:
- `--no-install` downloads and verifies the image but leaves it staged. By
default the runner then schedules the target, relays, and a script-configured
source back to their normal radios. Combining it with
`--leave-controller-radio` deliberately preserves the TempRadio topology.
`--leave-controller-radio` deliberately preserves the destination, relays,
and separate controller on the TempRadio topology; a managed source is still
returned to normal so its exact RXPS setting can be restored.
If the version gate required RXPS off, it stays off while that topology is
preserved; use `target-rxps-settings.json` to restore it only after sending
`normalradio`.
@@ -467,10 +487,14 @@ Useful controls:
downloading or staged on the target. Without it, that update is preserved.
- `--source-shares-controller` is for a Full Companion whose USB Binary API is
the controller while its TCP port `5001` is the source. It verifies that the
source's port-`5000` public key equals the controller key. The Binary API
first moves the shared physical radio, port `5002` then enables the local OTA
egress gate, and cleanup sends `normalradio` before restoring the saved
Binary radio tuple.
source's port-`5000` public key equals the controller key. Port `5002` uses a
bounded local `tempradio` override to move the shared physical radio without
overwriting its saved normal tuple; the Binary API remains the authenticated
transport. Cleanup sends local `normalradio`, proves that override inactive,
and then reasserts the saved Binary tuple. It cannot be combined with
`--leave-controller-radio`,
because exact source RXPS restoration requires that shared physical radio to
be back on its verified normal tuple.
- `--require-system-watchdog-off` checks `get system.watchdog` immediately
before every `ota install` transmission and refuses installation unless the
destination reports `> off`. Use it for nRF52 chains whose bootloader cannot
@@ -511,12 +535,14 @@ the destination.
hashes, Merkle root, full-image hash where applicable, identity fields,
signature, codec, base, and the firmware's 1024-byte maximum block size.
Version-3 bootloader packages are refused before any target state changes.
4. Save the controller's normal radio tuple, read every participant's version,
save the destination's RXPS state, select the qualified RXPS policy, and
show the confirmation prompt.
5. Apply and verify that RXPS policy, then start TempRadio on the target,
far-to-near relays, and the source;
finally switch the controller to the same tuple and read it back. The runner
4. Read and save a managed source's exact RXPS preference, disable and verify
source RXPS, save the controller's normal radio tuple, read every
participant's version, save the destination's RXPS state, select the
qualified destination policy, and show the confirmation prompt.
5. Apply and verify the destination RXPS policy, then start TempRadio on the
target, far-to-near relays, and source. A separate controller is moved and
read back through Binary; a shared Full Companion instead schedules its
bounded local override while Binary remains the transport. The runner
rejects a TempRadio window that cannot cover setup, seeder startup,
discovery, the transfer timeout, final polling, and install checks.
6. Start `motatool serve`, discover the exact eight-hex manifest ID, request
@@ -527,14 +553,17 @@ the destination.
7. Recheck that exact ID, give the target a short final TempRadio safety window,
and request `ota install`. Then shorten each relay's TempRadio window so the
normal multi-hop route returns, stop the seeder, shorten the source window,
restore the controller, and probe `ota self` at 10 and 20 seconds instead of
sleeping for 90 seconds. The exact new body hash is the readiness signal;
restore the controller, and probe `ota self` every 10 seconds through the
configured readiness window (five minutes by default). The exact new body
hash is the readiness signal;
only then does the runner require the exact package version. A relayed run
continues the 10-second probes through the mandatory relay-return window. A
source supplied with `--source-already-temp` is never modified.
`--leave-controller-radio` moves the controller back to TempRadio only after
this normal-channel verification. Restore the destination's exact original
RXPS setting after normal-channel identity is proven.
RXPS setting after normal-channel identity is proven. A managed source stays
RXPS-off through that verification and its exact setting is restored only
after its own TempRadio state is proven inactive.
Remote replies are matched only after queued messages have been drained and
only when they come from the intended contact and fit the command. A ready
@@ -565,26 +594,54 @@ budget. If a bounded window expires, rerun the same package after the nodes
return to their normal channel; the manifest-ID check resumes its partial
download without replacing it.
The working directory is retained and printed at exit. It contains the exact
The working directory is created before a managed source can be changed, then
retained and printed at exit. It contains the exact
served mOTA, `motatool-serve.log`, extracted build inputs when needed, and
`controller-radio.txt`. When the destination started with RXPS enabled it also
contains protected `target-rxps-settings.json` for manual recovery. It contains
no saved admin password.
`controller-radio.txt`. A managed source also gets a protected
`source-rxps-settings.json` containing its exact original preference and
idempotent restore command. Its contents and directory entry are flushed before
RXPS is disabled. When the destination
started with RXPS enabled, protected `target-rxps-settings.json` records its
manual recovery state. The RAK3401 chain points every nested step at one
chain-root source record, so a rerun after host power loss does not adopt the
temporary RXPS-off state as the original. A retained record is accepted only
for the same managed CLI endpoint. These files contain no saved admin password.
After exact source restoration, a standalone run atomically retires its record;
the chain retains its shared record between steps and retires it only after the
verified endpoint restoration completes.
## Interruption and recovery
Ctrl-C stops the seeder, detaches its serial folder, makes one best-effort
request to shorten a source TempRadio window started by the script, and attempts
to restore the controller. The target and relays remain on TempRadio only until
their bounded windows end; rebooting also restores their saved radio settings.
A normal cleanup restores the destination's exact RXPS periods. If that remote
restore cannot be confirmed, use `target-rxps-settings.json` after the target
returns to its normal channel.
Ctrl-C stops the seeder, detaches its serial folder, makes a bounded attempt to
shorten a source TempRadio window started by the script, and attempts to
restore the controller. For a managed source it leaves RXPS off until the
source is proven back on its normal radio, then restores and verifies the
saved source preference. A transient success-path restore failure remains
armed for one more idempotent cleanup attempt. The target and relays remain on
TempRadio only until their bounded windows end; rebooting also restores their
saved radio settings. A normal cleanup restores the destination's exact RXPS
periods. If that remote restore cannot be confirmed, use
`target-rxps-settings.json` after the target returns to its normal channel.
A partial download remains safe. Once the target is reachable again (after its
TempRadio window ends, or after putting the controller back on that tuple),
rerunning the same package recognizes its manifest ID and resumes the existing
session instead of clearing it.
When a chained run has already proved the newly running body hash, its retained
previous package can briefly report `verifying staged blocks` after TempRadio
reactivates the OTA manager. The runner waits only through the configured
discovery timeout, keeps checking source liveness, and accepts only the same
manifest becoming `ready to install` or the manager becoming idle. A changed
ID, failed or incomplete state, or timeout stops the chain. It then proves the
exact installed body again. If that same session is still attached and ready,
the runner detaches it with `ota cancel`; if the manager has become idle, it
sends no cancel. An ordinary-channel `no download` status proves only that the
manager is idle, not that persistent staging was erased. The runner therefore
does not issue or describe an IDLE cancel as durable cleanup. The next chain
transition explicitly re-adopts and proves the expected previous MID before detaching it;
after the final install OTAFIX has consumed the approval word, so any retained
container is inert and is replaced by the next valid pull.
A hard process kill or host power loss cannot run cleanup. Recover a serial
controller using the tuple saved in the printed work directory:
@@ -598,6 +655,12 @@ $radio = (Get-Content '.\meshcore-lora-ota-...\controller-radio.txt' -Raw).Trim(
meshcli -s COM7 set radio $radio
```
For a managed source, first return it to its ordinary radio, then inspect
`source-rxps-settings.json` and issue its exact `restore_command` through the
same serial or TCP-console endpoint recorded in that file. The command is
idempotent; confirm the full setting with `get radio.rxps.config` before
resuming an update.
If you stop during final confirmation, reconnect on the node's normal channel
and run `ota self` and `ver`. A completed run returns success only when
`ota self` reports a valid new body hash and `ver` exactly matches the package;
docs/ota_protocol.md
+10 -1
View File
@@ -824,7 +824,7 @@ ota rescue install <base_hash16> internal-flash nRF52 only: recover from failed
ota bootloader [status] capable allowlisted nRF52 repeater: installed BL identity/caps + staged confirmation
ota bootloader install <MID8> <HASH16>
explicitly verify/arm one complete trusted v3 package; never automatic
ota cancel | drop | stop drop the current fetch session (frees the slot)
ota cancel | drop | stop drop the fetch; durably invalidate device staging, or retain a folder partial for resume
ota announce | adv serve self + send a beacon now
ota self | id print this firmware's EndF (body/image size, base_hash)
ota qspi | storage QSPI nRF52 only: JEDEC/SR1/stage/latched storage error (read-only)
@@ -834,6 +834,15 @@ ota key | keys [add|rm <hex>] trusted signer allowlist ; bare = list
ota dev ... bring-up helpers (stage/recv/serve/resume <MID8>/verify)
```
For a device-backed pull, current firmware returns success only after the
persistent store can no longer be reopened; flash/SD/QSPI I/O or readback
failure is reported as an error even though the in-memory manager session was
dropped. For a `folder` pull, cancellation detaches the live transfer but
deliberately leaves the host `.part` file available for a later resume.
If the shared receive engine is currently performing the MeshTower SD
auto-archive capture, cancellation likewise detaches that archive transfer and
retains its `.part` file; it does not erase the unrelated manual-install store.
---
<a id="12-apply-bootloader-contract"></a>
docs/ota_user_guide.md
+12 -1
View File
@@ -212,6 +212,15 @@ falls back to flash. To **stop** a download you no longer want:
ota cancel
```
For a download staged on the node, success means current firmware also
invalidated the persistent flash/SD/QSPI copy; it reports an error if that
media operation cannot be verified. For a `folder` capture, cancel only
detaches the live session and retains the host partial so it can be resumed or
removed on the host deliberately. On a MeshTower SD auto-archive capture,
cancel also detaches the live archive session and retains its card partial for
the archive service to resume later; use `ota cache off` to stop new archive
captures.
### 4. Install a downloaded update
Once `ota status` shows the download is **ready to install**:
@@ -292,7 +301,9 @@ complete target inventory, storage layouts, and safety contract.
### 5. If something goes wrong
- A download that stalls or gets interrupted just **resumes** later, or you can `ota cancel` and try again.
- A download that stalls or gets interrupted just **resumes** later. Use
`ota cancel` to durably discard device staging; a folder capture keeps its
host partial for a later resume.
- A legacy app-only internal-flash **nRF52** that still runs but reports `no EndF` can use the pre-provisioned rescue path
if its physical EndF is intact and only app-side validation is failing. Fetch the exact `[rescue]`
in-place delta with an explicit acknowledgement, obtain its 16-hex-digit `base_hash` from the package
docs/rak3401_mota_chain.md
+235 -97
View File
@@ -1,24 +1,32 @@
# RAK3401 1W repeater compact LoRa update chain
> Status: physically qualified on 19-Aug-2026. All nine exact transitions
> completed on a deployed RAK3401 1W with its existing bootloader, using a
> Heltec V4 source at 909.950 MHz / 500 kHz / SF5 / CR5. Every post-boot EndF
> hash matched. The same files also pass independent reconstruction, container
> verification, and both deployed Preview 6 and current OTAFIX simulators.
> Status: **unreleased candidate with physically passed package transitions.**
> On 29-Aug-2026 all ten exact package transitions completed on the target
> RAK3401 with its deployed `0.9.2-OTAFIX2.4` bootloader and a Heltec V4 source at
> 909.950 MHz / 500 kHz / SF5 / CR5. Every intermediate EndF hash matched.
> Independent SWD readback then matched the endpoint application, original
> bootloader, SoftDevice/MBR, and UICR byte-for-byte. The host runner received
> cleanup/recovery fixes after that test and has not had a new clean end-to-end
> physical rerun. Multi-hop and alternate-bandwidth estimates remain planning
> data, not physical qualification.
Use this asset:
The candidate is a local qualification artifact, not a GitHub release:
```text
RAK3401-update-chain-v1.16.7-c1caa5ad-to-v1.17.1.02-e742333a.zip
RAK3401-update-chain-v1.16.7-c1caa5ad-to-v1.17.1.5-fd98bc90.zip
```
- ZIP SHA-256: `9f80eef191b88833bf4d2e4fea559cf5233ca53f9266ba310d447f37fa445f3a`
- inner `SHA256SUMS.txt` SHA-256: `73d96e23237896a3e342fe736be12d94087a813bf09ad609fb55330bbe586055`
- endpoint image SHA-256: `2784e4b645bc3dc198de0b8b18d3d7369cd02eca61cd71c46a51b61854da5345`
- endpoint EndF body hash: `4BB1526BF647547D`
- candidate ZIP SHA-256: `c0b33f4568985e8b2b8dc99411295907212cf2bad21764b6333d5e0ba298fd61`
- inner `SHA256SUMS.txt` SHA-256: `3f8c4af8096b96a4aa6506825c387cc8a06f74d5213a29c9387bd11689546881`
- endpoint image SHA-256: `31c182c888ceb1135e5afb2376610d93cee2e807b556c838e07fd4486c79d095`
- endpoint EndF body hash: `9179B98432895924`
Until that asset is uploaded to the release tag pinned by the runner, pass its
local path with `--bundle`.
Always pass the exact local path with `--bundle`. The runner has no download
URL for this candidate. `--verify-only` remains the safe default. The hidden
`--accept-test-candidate` override is intentionally absent from `--help` and
remains required until this exact artifact is published with its qualification
record; direct physical success does not silently turn a local file into a
released deployment asset.
## Exact destination
@@ -29,29 +37,32 @@ target:
- hardware: `RAK_3401`
- role: `RAK_3401_repeater_lora_ota_no_external_sensors`
- start: `v1.16.7.0-c1caa5ad`, EndF `71F4026CBE4B8B74`
- endpoint source: MeshCore `e742333a0ee393b0d55d2414f25b28f2f67e1ea1`
- endpoint label: `v1.17.1.02-halo-keymind-cascade-dev-e742333a`
- endpoint packed version: `0x01110102` (EndF tools render this as `1.17.1.2`)
- endpoint source: MeshCore `fd98bc90f7fb25d5956facd550e227b217abbe95`
- endpoint label: `v1.17.1.5-halo-keymind-cascade-dev-fd98bc90`
- endpoint packed version: `0x01110105`
- deployment target key: `63d8df6387eaffd2e25db7d2a8ad967a65202182a48d681d7e7a9260f917280d`
### Historical endpoint GPS limitation
### Retained and omitted hardware support
This chain's exact `RAK_3401_repeater_lora_ota_no_external_sensors` endpoint
compiles with `ENV_INCLUDE_GPS` removed. It will not detect, configure, or read
a RAK12501 GPS module. GPS requires the ordinary full-sensor
`RAK_3401_repeater` firmware and the RAK12501 must be installed in sensor slot
A. Do not use slot D with the RAK13302 1 W radio: the GPS reset/PPS signals
would overlap the radio's BUSY/DIO1 signals. The full-sensor build is a
different target and is not an endpoint of this compact OTA chain.
Despite the legacy `no_external_sensors` target name, the endpoint retains the
RAK12501 GPS provider and the common INA3221, INA219, INA226, and INA260 I2C
voltage/current monitors. Install RAK12501 in sensor slot A. Do not use slot D
with the RAK13302 1 W radio because the GPS reset/PPS signals overlap the
radio's BUSY/DIO1 wiring.
This limitation describes the immutable released chain endpoint identified
above. Newly built RAK3401 OTA repeater firmware retains RAK12501 support in
slot A, but that newer image is not retroactively part of this exact chain.
The reduced profile omits optional external environmental and ranging devices:
AHTX0, BME280, BMP280, SHTC3, SHT4x, LPS22HB, MLX90614, VL53L0X, BME680,
BMP085, RAK12035, and BME680 BSEC. Board radio, display, buttons, battery
telemetry, GPS, and the INA monitor family remain. No additional “remove I2C
temperature” image is needed: the optional I2C temperature/environment drivers
are already in the omitted set, while removing the remaining I2C/INA support
would discard desired voltage/current monitoring and create another firmware
identity without helping this route.
Do not use the chain on another target ID, hardware family, starting image, or
firmware body hash. The runner checks all four.
## Why the old bootloader works
## Why Preview 5 and the old staging ceiling work
The deployed bootloader is not changed and no package uses the newer expanded
`0xED000` ceiling. Each package remains bottom-aligned below the old
@@ -66,31 +77,43 @@ The old bootloader already reads and validates that encoded detools geometry
before its first destructive write. The previous application receiver was the
limiting part: it reserved a fixed `0x98000` workspace before accepting a
download. Step 1 therefore remains byte-for-byte identical to the physically
passed 89,844-byte package. Step 2 is a 46,363-byte fixed-workspace package
passed 89,844-byte package. Step 2 is a 46,364-byte fixed-workspace package
that installs the compatibility receiver. Later receivers stage above the
real EndF image extent and let the bootloader enforce each package's selected
workspace.
An exhaustive search tested every page-aligned workspace relevant to a route
shorter than nine packages: 9,430 candidate geometries across 272 possible
layer-skipping edges. None fit. The 61 edges on the shortest-path graph were
then swept across another 2,296 workspace choices; 840 nine-package routes
were compared by total transfer size. The selected route is the byte-minimum
shortest route:
Every package in this candidate is an ordinary application container using
format ABI 2 and in-place codec 2. The historical bridge applications scan the
bootloader capability marker byte-by-byte, so they recognize the released
Preview 5 RAK4631 marker at absolute address `0xFCDEA` even though it is two
bytes off a word boundary. The `fd98bc90` endpoint deliberately retains that
exact Preview 5 application-update fallback. Its separately compiled internal
bootloader-update feature remains unavailable on Preview 5 because privileged
self-update requires a current ABI 3 marker with the exact storage profile.
Installing this chain therefore does not replace the bootloader, and the final
application can still accept a future valid ABI-2/codec-2 application update.
The schema-2 exhaustive search checked 17,402 candidate geometries and found
121 feasible forward edges in the declared 33-image inventory. Fourteen
ten-package routes tied for the minimum package count; the selected route has
the smallest total container size among them. That minimum applies to the
declared inventory, not every conceivable firmware image. The selected route
is:
| Step | From | To | Workspace | Package | Margin |
|---:|---|---|---:|---:|---:|
| 1 | 1.16.7.0 | 1.16.7.9 | `0x98000` | 89,844 | 0 |
| 2 | 1.16.7.9 | 1.16.7.10 | `0x98000` | 46,363 | 40,960 |
| 3 | 1.16.7.10 | 1.16.8.7 | `0x91000` | 106,030 | 12,288 |
| 2 | 1.16.7.9 | 1.16.7.10 | `0x98000` | 46,364 | 40,960 |
| 3 | 1.16.7.10 | 1.16.8.7 | `0x91000` | 106,029 | 12,288 |
| 4 | 1.16.8.7 | 1.16.9.105 | `0x89000` | 149,927 | 0 |
| 5 | 1.16.9.105 | 1.16.9.110 | `0x8E000` | 111,307 | 16,384 |
| 6 | 1.16.9.110 | 1.16.9.111 | `0x8D000` | 70,679 | 61,440 |
| 7 | 1.16.9.111 | 1.16.9.116 | `0x89000` | 143,441 | 4,096 |
| 8 | 1.16.9.116 | 1.16.9.118 | `0x81000` | 98,188 | 86,016 |
| 9 | 1.16.9.118 | 1.17.1.02 | `0x7D000` | 186,385 | 12,288 |
| 5 | 1.16.9.105 | 1.16.9.110 | `0x8E000` | 111,312 | 16,384 |
| 6 | 1.16.9.110 | 1.16.9.113 | `0x8F000` | 120,624 | 4,096 |
| 7 | 1.16.9.113 | 1.16.9.117 | `0x8B000` | 139,637 | 0 |
| 8 | 1.16.9.117 | 1.16.10.0 | `0x7E000` | 174,202 | 20,480 |
| 9 | 1.16.10.0 | 1.17.1.3 | `0x6B000` | 271,569 | 0 |
| 10 | 1.17.1.3 | 1.17.1.5 | `0x74000` | 173,334 | 61,440 |
Total mOTA transfer data is 1,002,164 bytes. `ROUTE.json`, `CHAIN.csv`, and
Total mOTA transfer data is 1,382,842 bytes. `ROUTE.json`, `CHAIN.csv`, and
`validation-results.json` in the bundle pin the exact geometry and image hash
for every transition.
@@ -105,17 +128,27 @@ No password or device is needed:
```bash
python3 tools/lora_ota/rak3401_mota_chain.py \
--bundle /path/to/RAK3401-update-chain-v1.16.7-c1caa5ad-to-v1.17.1.02-e742333a.zip \
--bundle /path/to/RAK3401-update-chain-v1.16.7-c1caa5ad-to-v1.17.1.5-fd98bc90.zip \
--work-dir ./rak3401-mota-chain-work \
--motatool /path/to/motatool \
--verify-only
```
The runner checks the outer ZIP, complete inner checksum coverage, all nine
manifests, continuity, image anchors, final recovery image, and every container
with `motatool`.
When given the ZIP, the runner checks its pinned outer hash and binds the
extraction cache to that exact archive. An explicitly supplied extracted root
starts at the pinned inner checksum list. In either case it checks complete
inner coverage, all ten manifests, continuity, image anchors, both recovery
images, and every container with `motatool`. Passing this command means offline
and simulator qualification only; it does not unlock live use.
## Direct recoverable bench run
## Current guarded reproduction command
These are the same direct-link arguments used for the physical transition run,
but the current host runner also contains later cleanup and recovery fixes.
That revised runner has not been cleanly rerun through all ten transitions.
Keep local USB/SWD recovery available while the artifact remains unreleased.
The hidden override bypasses only the candidate-status block; all identity,
checksum, bootloader, route, watchdog, and post-boot gates still run.
Restore the test start locally with
`recovery/test-start/RAK3401-test-start-v1.16.7-c1caa5ad.uf2`, then run:
@@ -124,7 +157,7 @@ Restore the test start locally with
export MESHCORE_ADMIN_PASSWORD='password'
python3 tools/lora_ota/rak3401_mota_chain.py \
--bundle /path/to/RAK3401-update-chain-v1.16.7-c1caa5ad-to-v1.17.1.02-e742333a.zip \
--bundle /path/to/RAK3401-update-chain-v1.16.7-c1caa5ad-to-v1.17.1.5-fd98bc90.zip \
--work-dir ./rak3401-mota-chain-work \
--controller-serial /dev/ttyACM0 \
--source-tcp 192.168.1.51:5001 \
@@ -135,6 +168,7 @@ python3 tools/lora_ota/rak3401_mota_chain.py \
--ota-hops 0 \
--legacy-full-airtime \
--motatool /path/to/motatool \
--accept-test-candidate \
--yes
```
@@ -145,23 +179,76 @@ zero and restores it at the endpoint. Use that option only where the selected
frequency and local duty-cycle rules permit a full transmit budget; omit it
otherwise.
### Measured direct-link timing
The first `ready to install` transfer ages were:
| Step | Blocks | Transfer |
|---:|---:|---:|
| 1 | 88 | 598 s |
| 2 | 45 | 42 s |
| 3 | 103 | 81 s |
| 4 | 146 | 105 s |
| 5 | 109 | 76 s |
| 6 | 118 | 94 s |
| 7 | 136 | 102 s |
| 8 | 170 | 127 s |
| 9 | 264 | 187 s |
| 10 | 169 | 193 s |
The physically observed bulk-transfer total for those exact packages is 1,605
seconds (26m45s). The log came from the pre-cleanup-fix runner revision. Step 1
was measured before the host began forcing source RXPS off and is therefore a
conservative outlier. A new complete run with that fix is projected at roughly
18–19 minutes of bulk transfer, but that projection is not a substitute for
the measured total. Allow about 70–75 minutes direct at BW500 for validation,
ten transfers, installs/reboots, retained-store checks, and final restoration.
Step 10's longer per-block time was not a V4 TempRadio delay: the source RXPS
readback was off. Its v1.17.1.3 requester uses non-overlapping adaptive flights
and a five-second recovery floor and logged more block reloads than step 8.
The V4 itself intentionally schedules its local radio handoff after 1.5 seconds
so the normal-channel command reply can drain; the host waits three seconds and
then checks the live TempRadio state.
For planning only, scale the conservative measured 26m45s bulk baseline by the
qualified adaptive-preamble airtime and by each additional relay transmission:
| Bandwidth, SF5/CR5 | Direct / 0 relays | 1 relay | 2 relays |
|---:|---:|---:|---:|
| 500 kHz | 26m45s | 53m30s | 1h20m15s |
| 250 kHz | 46m51s | 1h33m42s | 2h20m33s |
| 125 kHz | 1h27m03s | 2h54m07s | 4h21m11s |
| 62.5 kHz | 2h54m07s | 5h48m15s | 8h42m23s |
These are transfer-only estimates, not measured alternate-bandwidth or relay
results. Packet loss, relay contention, discovery, administrative commands,
staged verification, and ten reboot cycles add time. The bandwidth factors are
not simple powers of two because the adaptive wire preamble is 128 symbols at
500 kHz, 64 at 250 kHz, and 32 at 125/62.5 kHz.
## Manual operation
The pinned runner is the safer method. If it cannot be used, follow every
check below and keep a written copy of each original setting. Do not skip a
step even when a later package appears in `ota ls`.
The automated runner is preferred because it binds every transition to the
expected MID and body hash. The manual details below document the checks used
during qualification and recovery; they are not permission to skip the
runner's candidate gate. Keep a written copy of each original setting and
never skip a step even when a later package appears in `ota ls`.
### 1. Verify and extract the asset
```bash
sha256sum RAK3401-update-chain-v1.16.7-c1caa5ad-to-v1.17.1.02-e742333a.zip
unzip RAK3401-update-chain-v1.16.7-c1caa5ad-to-v1.17.1.02-e742333a.zip
cd RAK3401-update-chain-v1.16.7-c1caa5ad-to-v1.17.1.02-e742333a
sha256sum RAK3401-update-chain-v1.16.7-c1caa5ad-to-v1.17.1.5-fd98bc90.zip
unzip RAK3401-update-chain-v1.16.7-c1caa5ad-to-v1.17.1.5-fd98bc90.zip
cd RAK3401-update-chain-v1.16.7-c1caa5ad-to-v1.17.1.5-fd98bc90
sha256sum -c SHA256SUMS.txt
for package in motas/*.mota; do motatool verify "$package" || exit 1; done
```
The outer hash must be `9f80eef191b88833bf4d2e4fea559cf5233ca53f9266ba310d447f37fa445f3a`.
The outer hash must be
`c0b33f4568985e8b2b8dc99411295907212cf2bad21764b6333d5e0ba298fd61`,
and the SHA-256 of the extracted `SHA256SUMS.txt` must be
`3f8c4af8096b96a4aa6506825c387cc8a06f74d5213a29c9387bd11689546881`.
Do not continue after any checksum or verification error.
### 2. Record and prepare the destination
@@ -218,13 +305,12 @@ tests use `hops 0`: `hops 1` makes the target echo each source response despite
there being no intermediate relay, increasing half-duplex loss and legacy
three-second retries.
The exact compact chain never qualifies for fast RXPS: its bridge and endpoint
versions predate the v1.17.1.5 adaptive-preamble contract. Keep RXPS off for
this historical chain whether using SF5/BW250 or the faster direct-bench
SF5/BW500 tuple. For a later update whose target, controller, source, and
relays are all v1.17.1.5 or newer, use the generic LoRa OTA runner; it can
retain RXPS at the qualified level-8/preamble-64 boundary for SF5/BW250 or the
level-8/preamble-128 boundary for SF5/BW500.
The exact candidate cannot use fast RXPS throughout because its historical
bridge applications predate the v1.17.1.5 adaptive-preamble contract. Keep
RXPS off for all ten steps whether using SF5/BW250 or the faster direct-bench
SF5/BW500 tuple. After the final endpoint is proven, a later update whose
target, controller, source, and relays are all v1.17.1.5 or newer can use the
generic LoRa OTA runner and its qualified adaptive-preamble boundaries.
Put the source on the identical TempRadio tuple. A current ASCII-first Full
Companion recognizes `motatool`'s initial `ota folder on` line directly:
@@ -243,30 +329,48 @@ every step so the source emits a fresh catalog advert. Leave it running during
the download and stop it with Ctrl-C only after the destination reports `ready
to install`.
### 3. Install all nine packages in order
### 3. Install all ten packages in order
| Step | Manifest ID | Before hash | Expected version | After hash |
|---:|---|---|---|---|
| 1 | `C147BCEF` | `71F4026CBE4B8B74` | `1.16.7.9` | `42BC53A64288E845` |
| 2 | `C510B628` | `42BC53A64288E845` | `1.16.7.10` | `6F6E51D63BF3E93F` |
| 3 | `8CAD7045` | `6F6E51D63BF3E93F` | `1.16.8.7` | `F04DA8DB515E5C6D` |
| 4 | `8B1EEFF6` | `F04DA8DB515E5C6D` | `1.16.9.105` | `3894A4D7B11ED282` |
| 5 | `687E3BEA` | `3894A4D7B11ED282` | `1.16.9.110` | `CDC5CA630881DA15` |
| 6 | `0D59A34B` | `CDC5CA630881DA15` | `1.16.9.111` | `D0CC4A9E135B9A4D` |
| 7 | `2C5C2082` | `D0CC4A9E135B9A4D` | `1.16.9.116` | `66272A0E7FEF2773` |
| 8 | `994AB743` | `66272A0E7FEF2773` | `1.16.9.118` | `8E00019AA08E00BF` |
| 9 | `FE986948` | `8E00019AA08E00BF` | `1.17.1.02` | `4BB1526BF647547D` |
| 2 | `BEB578FC` | `42BC53A64288E845` | `1.16.7.10` | `1AD2AA8B6C478DA6` |
| 3 | `6D8EF814` | `1AD2AA8B6C478DA6` | `1.16.8.7` | `D709A25308BC1831` |
| 4 | `40AC4CEB` | `D709A25308BC1831` | `1.16.9.105` | `457BEDA5E6406C81` |
| 5 | `F6C8890B` | `457BEDA5E6406C81` | `1.16.9.110` | `E76BFA205634CAB8` |
| 6 | `82405E89` | `E76BFA205634CAB8` | `1.16.9.113` | `65328FC8A1FBED2D` |
| 7 | `80F40DB6` | `65328FC8A1FBED2D` | `1.16.9.117` | `AF7532E13337FADD` |
| 8 | `DBF6310D` | `AF7532E13337FADD` | `1.16.10.0` | `22002359BBDFA76E` |
| 9 | `C2979E08` | `22002359BBDFA76E` | `1.17.1.3` | `FE65A6135A1E7B3F` |
| 10 | `319DEC92` | `FE65A6135A1E7B3F` | `1.17.1.5` | `9179B98432895924` |
For each row, first prove that `ota self` exactly matches its **Before hash**.
Then run:
For each row, first prove that `ota self` exactly matches its **Before hash**,
then inspect the manager before changing it:
```text
ota status
```
Proceed to discovery only after that status is reconciled. When the manager is
idle, run:
```text
ota cancel
ota ls
ota pull MANIFEST_ID flash
ota status
```
If the first status reports this row's MID as partial, use `ota pull` with that
same MID to resume it. If this row's MID is already ready, do not pull or
cancel it; continue with the install checks below. If status reports the
immediately previous row's MID as `verifying staged blocks`, wait through the
bounded discovery window; require either that exact MID to become `ready to
install` or the manager to become `no download`, and re-prove the current
**Before hash**. Cancel only the same previous MID while it is visibly
attached and ready. If it has become idle, send no cancel. Stop on any other
MID, failed or incomplete state, ambiguous reply, or timeout. Never issue a
blind `ota cancel`: it can discard a resumable or unrelated session.
Repeat `ota status` at a restrained interval until it says `ready to install`.
If the manifest is initially absent, wait for a fresh source advert, run
`ota ls` again, or restart `motatool`; do not pull a different ID. Stop the
@@ -277,19 +381,35 @@ get system.watchdog
ota install
```
The watchdog reply must still be `> off`. Allow up to 90 seconds for the USB
port to disappear and return. Then run `ver`, `ota self`, and `ota help`.
The watchdog reply must still be `> off`. Allow the current default five-minute
readiness window for the USB/LoRa identity to return; automation probes every
10 seconds and returns early as soon as the exact body is visible. Then run
`ver`, `ota self`, and `ota help`.
Require the row's exact **After hash**, require `bootloader: apply OK`, and
require `rescue install <hash16>` in the help before exposing that bridge to
the next package. Clear only the now-proven retained download with `ota cancel`,
reapply the transfer guardrails and TempRadio tuple, restart `motatool`, and
continue with the next row.
the next package. This rescue-help gate applies to steps 1-9; step 10 is the
endpoint and is not exposed to another package in this chain. A normal-radio
`ota status` response of `no download` proves only that the manager is idle;
legacy internal-flash firmware does not durably erase a retained container by
accepting an IDLE `ota cancel`. Do not claim otherwise. When the same expected
manifest reports `verifying staged blocks` after the next TempRadio starts,
wait only for the bounded discovery window and require that it become `ready
to install` or `no download`; stop on a changed ID, failed/incomplete state, or
timeout. Prove the exact running body before cancelling an attached completed
session. OTAFIX consumes the approval word during a successful install, so a
retained endpoint container is inert and a later valid pull replaces it.
Reapply the transfer
guardrails and TempRadio tuple, restart `motatool`, and continue with the next
row.
If the board boots but `ota self` says the EndF is invalid, stop. Only when
`ota status` still identifies that row's complete staged package may the
guarded `ota rescue install BEFORE_HASH` command be used. It is not a force
option and must use that row's exact Before hash. If the board does not boot,
recover the documented start/recovery UF2 locally over USB.
recover the documented start/recovery UF2 locally over USB. The fd98 endpoint
deliberately disables rescue on its shared internal bootloader-update build;
if its EndF is invalid, use local DFU/SWD rather than attempting another LoRa
package.
### 4. Intermediate relays and restoration
@@ -307,7 +427,7 @@ window. The source's manifest-fragment gap independently follows active packet
airtime and duty spacing, clamped to 100-1000 ms. These two delays solve
different problems.
After step 9 is proven, restore each saved destination and relay value exactly,
After step 10 is proven, restore each saved destination and relay value exactly,
including `rxdelay`, RXPS, CPU power saving, `af`, `ota config hops`, and relay
timing. Current full-parser repeater firmware accepts `normalradio` and restores
the saved tuple after replying on the temporary channel:
@@ -316,7 +436,7 @@ the saved tuple after replying on the temporary channel:
normalradio
```
The historical endpoint in this pinned bundle predates that command. If it
The fd98 endpoint supports that command. If a recovery build unexpectedly
replies `Unknown command`, shorten its lease to one minute with the same tuple:
```text
@@ -335,20 +455,26 @@ A Full Companion source is the exception: its local TCP console supports
`normalradio` and the automated runner uses it before restoring the shared
Binary API radio tuple.
Final success requires version `v1.17.1.02-halo-keymind-cascade-dev-e742333a`,
body hash `4BB1526BF647547D`, target `2FA509C1`, hardware `RAK_3401`, and
Final lab success requires version
`v1.17.1.5-halo-keymind-cascade-dev-fd98bc90`, body hash
`9179B98432895924`, target `2FA509C1`, hardware `RAK_3401`, and
`get system.watchdog` reporting `> on`. If a relay cannot be restored before
its TempRadio lease ends, wait for it to return to the normal channel and
restore its saved `rxdelay` and `txdelay` there.
## Two-relay deployment
For two intermediate relays, list them farthest-to-nearest and use three OTA
hops. Run a non-mutating preflight first:
The exact direct package transitions above were physically observed; this
post-run host-runner revision and multi-hop operation are not cleanly
end-to-end qualified. A controlled, recoverable multi-hop lab can exercise two
intermediate relays by
listing them farthest-to-nearest and using three OTA hops. Run a non-mutating
preflight first; even preflight requires the hidden candidate override because
the artifact remains unreleased and the command connects to live devices:
```bash
python3 tools/lora_ota/rak3401_mota_chain.py \
--bundle /path/to/RAK3401-update-chain-v1.16.7-c1caa5ad-to-v1.17.1.02-e742333a.zip \
--bundle /path/to/RAK3401-update-chain-v1.16.7-c1caa5ad-to-v1.17.1.5-fd98bc90.zip \
--work-dir ./rak3401-mota-chain-work \
--controller-serial /dev/ttyACM0 \
--source-tcp SOURCE_IP:5001 \
@@ -360,6 +486,7 @@ python3 tools/lora_ota/rak3401_mota_chain.py \
--temp-radio 909.950,250,5,5,120 \
--ota-hops 3 \
--motatool /path/to/motatool \
--accept-test-candidate \
--preflight-only
```
@@ -370,16 +497,27 @@ legal at the deployment location.
## Rescue and completion
The guarded `ota rescue install <base_body_hash>` command is present after
step 1. It is not a force command: it refuses a valid normal EndF, a base
mismatch, wrong target/hardware, or invalid payload. A failure before step 1
completes still requires local USB recovery because the deployed start image
predates the rescue command.
step 1 and remains available through the step-9 bridge. It is not a force
command: it refuses a valid normal EndF, a base mismatch, wrong
target/hardware, or invalid payload. A failure before step 1 completes still
requires local USB recovery because the deployed start image predates the
rescue command. The final fd98 endpoint uses the stricter shared-slot profile;
its invalid-EndF recovery is local DFU/SWD, not the rescue command.
Before the first mutation, the runner saves the destination's RXPS periods,
CPU power-saving state, RX flood delay, airtime factor, and OTA hop reach in
the persistent work directory. It reads the destination, controller, source,
and relay versions before applying the same RXPS policy as the generic runner.
The pinned historical chain therefore verifies `radio.rxps off` after every
the persistent work directory, flushing both the file and its directory entry
before changing a guardrail. It also saves the managed source's exact RXPS
preference in the chain-root `source-rxps-settings.json` before disabling it.
Every step reuses that protected record, so a rerun after a killed host process
restores the chain-start preference instead of treating the temporary off state
as original. The record is accepted only for the same CLI endpoint. The runner
retires that chain-root record only after it has proved the source is back on
its normal radio and restored the exact preference at the verified endpoint.
The runner
reads the destination, controller, source, and relay versions before applying
the same RXPS policy as the generic runner.
The mixed historical chain therefore verifies `radio.rxps off` after every
bridge reboot; a future all-v1.17.1.5-or-newer SF5/BW250 chain would instead
keep RXPS on under the qualified level-8/preamble-64 boundary. It also verifies
`powersaving off` and `rxdelay 0`, plus `af 0` when
@@ -398,8 +536,8 @@ the relay leaves TempRadio.
The runner also keeps the watchdog off across the chain, rechecks identity and
OTA reach before every package, requires the exact post-boot EndF hash, and
only re-enables the watchdog after step 9. Success requires endpoint body hash
`4BB1526BF647547D`, target `2FA509C1`, hardware `RAK_3401`, normal radio
only re-enables the watchdog after step 10. Lab success requires endpoint body
hash `9179B98432895924`, target `2FA509C1`, hardware `RAK_3401`, normal radio
`910.525 MHz / 62.5 kHz / SF7 / CR5`, and the watchdog verified on.
## External-radio startup and manual USB recovery