Add corrected RAK3401 mOTA chain runner

This commit is contained in:
mikecarper committed 2026-08-12 20:03:25 -07:00
1 parent 3cd9c84893
commit 38333e02f0
5 files changed
+1642 -8

No files matched your search

+15
View File
@@ -1,5 +1,10 @@
# Scripted LoRa OTA from start to finish
The dedicated [RAK3401 chain report](rak3401_mota_chain.md) records the
physical failure of the withdrawn 26-step migration and the corrected
v1.17.01 test candidate. Its runner blocks the withdrawn chain and requires an
explicit lab-only gate for the corrected chain until physical testing passes.
[`tools/lora_ota/lora_ota.sh`](../tools/lora_ota/lora_ota.sh) and
[`tools/lora_ota/lora_ota.ps1`](../tools/lora_ota/lora_ota.ps1) automate a
MeshCore LoRa firmware update from a release `.zip` or ready `.mota`. They
@@ -365,6 +370,16 @@ Useful controls:
- `--allow-non-upgrade` deliberately permits the same or an older version.
- `--replace-active-download` deliberately discards a different update already
downloading or staged on the target. Without it, that update is preserved.
- `--source-shares-controller` is for a Full Companion whose USB Binary API is
the controller while its TCP port `5001` is the source. It verifies that the
source's port-`5000` public key equals the controller key. The Binary API
first moves the shared physical radio, port `5002` then enables the local OTA
egress gate, and cleanup sends `normalradio` before restoring the saved
Binary radio tuple.
- `--require-system-watchdog-off` checks `get system.watchdog` immediately
before every `ota install` transmission and refuses installation unless the
destination reports `> off`. Use it for nRF52 chains whose bootloader cannot
service an application watchdog inherited across reset.
- `--work-dir PATH` chooses a new, non-existent work directory.
- `--meshcli PATH` and `--motatool PATH` select binaries not on `PATH`.
+313
View File
@@ -0,0 +1,313 @@
# RAK3401 1W repeater LoRa update chain
> Status: the corrected v1.17.01 26-package release candidate has passed
> complete offline reconstruction, independent container verification, and
> both stable OTAFIX 2.4 and Preview 6 C simulations. It still needs its first
> complete physical RAK3401 run and is gated behind
> `--accept-test-candidate`; do not use it in production yet.
This procedure records the exact 26-step update chain from the release
[`rak3401-mota-v1.16.07-c1caa5ad-to-v1.17.01-c96bdd6e`](https://github.com/mikecarper/MeshCore/releases/tag/rak3401-mota-v1.16.07-c1caa5ad-to-v1.17.01-c96bdd6e).
It is intentionally specific to the following destination firmware:
- target ID: `2FA509C1`
- hardware: `RAK_3401`
- role: `RAK_3401_repeater_lora_ota_no_external_sensors`
- starting version: `1.16.7.0`, build `c1caa5ad`
- starting EndF body hash: `71F4026CBE4B8B74`
- destination key used for this deployment:
`63d8df6387eaffd2e25db7d2a8ad967a65202182a48d681d7e7a9260f917280d`
Do not use this chain on another node, and do not start it on another RAK3401.
The artifact checks do not replace the pending complete physical-board run.
Use this candidate only for the next recoverable lab test.
## Corrected test-candidate bundle
The replacement rebuilds steps 6 through 14 so both `Utils::sha256` overloads
use software SHA-256; hardware AES and HMAC stay enabled. Temporary steps 14
through 25 have monotonic EndF versions `1.16.9.110` through `1.16.9.121`.
The final endpoint is a real v1.17.01 build from `c96bdd6e`, with both its
runtime string and EndF version set to `1.17.1.0`. No v1.17.02 endpoint is used.
```text
/home/mesh/git/MeshCore/out-rak3401-mota-v1.17.01-corrected/RAK3401-update-chain-v1.16.7-c1caa5ad-to-v1.17.01-c96bdd6e.zip
```
ZIP SHA-256:
`693f08187e42cce72124f01328983965726bfbbb3fef80de503f06c4cbe9256a`.
The largest package is 88,280 bytes, leaving 1,832 bytes below the staging
limit. All 26 deltas reconstruct with both zero-filled and 0xFF-filled
workspaces. The endpoint has whole-image SHA-256
`e923f6209e0f071a5862ffbc7690e0b355c1e74829067e0bfdacd0de6c316065`
and EndF body hash `AC267B02E055F42E`.
The candidate includes the exact v1.16.7 test-start UF2/Nordic DFU ZIP, a
diagnostic corrected step-6 image, and the final recovery UF2/ZIP. Restore the
lab node locally to the included test-start image before rerunning the chain.
The bundle's `RUNBOOK.md` contains the direct and two-hop commands.
## Withdrawn v1.17.02-chain smoke-test result
The lab topology used a Heltec V4.3 OLED ESP32 Full Companion on
`/dev/ttyACM0` and a direct LoRa path to `RAK3401 1W Repeater`. The remote
admin password is the lowercase default `password`, not `Password`.
Steps 1 through 5 completed and passed an exact post-boot version and EndF
body-hash check. Step 6 then did all of the following successfully:
- downloaded all 79 blocks of mOTA ID `7C8BC884`;
- verified the watchdog was off immediately before install;
- returned `verified (unsigned); applying`;
- booted `v1.16.08.7-halo-keymind-cascade-b-0c4ed3ac`;
- retained a soft-reset boot record consistent with OTAFIX apply success.
The resulting bridge returns `ERR no EndF (firmware lacks the trailer?)` from
`ota self`. The release package is not corrupt: independently applying its
delta reconstructs the expected 524,204-byte image with whole-image SHA-256
`61ced8b63953c614748c2fa1b04c2c01e8eb6626a604f6ef95fd2594d6d8ce71`
and valid EndF body hash `A0130F9BD6760D47`.
The regression is in the intermediate application. This is the first bridge
in the chain built with the old CC310 hardware-crypto SHA path. That path
passes the memory-mapped application body to CC310 and ignores a failed return
code instead of falling back to software SHA. The preceding v1.16.08 image
uses software SHA, while later code contains the required checked fallback.
The normal nRF52 install gate correctly refuses step 7 when it cannot validate
the running body. Forcing the bootloader trigger would bypass an application
safety gate and is not an acceptable recovery.
The lab RAK remains alive on v1.16.08.7. Its normal radio is restored to
`910.525,62.5,7,5`, and `system.watchdog` is restored and verified `on`.
## Recover the lab RAK
Recovery now requires a local data path to the RAK3401 itself. The Heltec
controller cable cannot recover the remote RAK, and this host currently has no
Bluetooth adapter.
1. Connect the RAK3401 itself to this host by USB.
2. Enter its exact-board UF2 bootloader, using a local `uf2reset` command or
the board's double-reset gesture.
3. Flash the final RAK3401 UF2 from the corrected bundle's `recovery/final/`
directory:
`RAK_3401_repeater_lora_ota_no_external_sensors-ota-v1.17.01-halo-keymind-cascade-dev-c96bdd6e.uf2`.
4. After it boots, require version `v1.17.1`, target `2FA509C1`, hardware
`RAK_3401`, a valid `ota self` reply, normal radio settings, and watchdog
`on`.
Do not send step 7, force an apply, or issue a remote recovery reboot. Once the
RAK USB cable is attached, the port and UF2 mount can be detected and the
recovery completed without guessing device paths.
## Intended runner safety behavior
Stable OTAFIX 2.4 cannot feed a system watchdog inherited by the bootloader.
The dormant live path implements the release's watchdog sequence:
1. Persist `system.watchdog off` without issuing a normal reboot.
2. Wait 90 seconds for the already-running watchdog to reset the node.
3. Require the node to report `> off`.
4. Prove another 90 seconds of stable uptime.
5. Enforce the requested `ota config hops` reach before every chain step.
6. Recheck `> off` immediately before every `ota install` command.
7. Re-enable the watchdog only after the exact step 26 image boots and passes
both version and EndF body-hash verification.
These controls are active for the corrected candidate. They cannot repair the
old bridge whose own EndF check is broken. The candidate must pass a complete
physical-board run before its explicit lab-only gate is removed.
An interrupted download is resumable. Avoid interrupting power or radio
coverage while `ota install` is applying a package. Physical USB recovery is
the fallback if an apply is interrupted at the wrong instant.
## Host and radio requirements
Install:
- Python 3.9 or newer
- `meshcli` 1.6.0 or newer
- a current `motatool`
- a Full Companion or OTA-enabled repeater to seed the files
The ESP32 Full Companion serves mOTA data through its dedicated TCP port
`5001`; its USB port remains the Binary API controller. Its local OTA and
TempRadio console is TCP port `5002`. For a Heltec Full Companion at
`192.168.1.51`, use these three connections together:
```text
controller: /dev/ttyACM0
mOTA source: 192.168.1.51:5001
source CLI: 192.168.1.51:5002
```
Before a deployment, verify that USB and TCP port `5000` report the same
Companion public key:
```bash
meshcli -s /dev/ttyACM0 -j -c off infos
meshcli -t 192.168.1.51 -p 5000 -j -c off infos
```
The tested Heltec key is
`8ce031bc322f3cf77376eeacfe8800b30d8c5dfe927f3636d801f3455a2dd4cf`.
Change the IP address for a different deployment.
The source, controller, destination, and every intermediate relay must support
the temporary tuple `909.950,250,7,5`. Confirm that this frequency and
bandwidth are legal at the deployment location. All nodes return to their
ordinary radio after each step.
## Verify without touching a radio
The chain runner pins the release ZIP SHA-256 to
`693f08187e42cce72124f01328983965726bfbbb3fef80de503f06c4cbe9256a`.
It also pins and verifies the inner checksum list, checks complete checksum
coverage, parses every manifest, verifies the chain continuity, and runs
`motatool verify` on all 26 containers.
It downloads the release automatically when `--bundle` is omitted:
```bash
python3 tools/lora_ota/rak3401_mota_chain.py \
--work-dir ./rak3401-mota-chain-work \
--motatool /path/to/motatool \
--verify-only
```
An already-downloaded asset can be supplied with `--bundle`. Only the pinned
ZIP or its extracted root is accepted.
## Corrected-release live preflight
The remote-admin password for the tested RAK is the lowercase default
`password`. Prefer the environment variable so it is not placed in the child
process command line:
```bash
export MESHCORE_ADMIN_PASSWORD='password'
python3 tools/lora_ota/rak3401_mota_chain.py \
--work-dir ./rak3401-mota-chain-work \
--controller-serial /dev/ttyACM0 \
--source-tcp 192.168.1.51:5001 \
--source-cli-tcp 192.168.1.51:5002 \
--source-shares-controller \
--ota-hops 3 \
--accept-test-candidate \
--motatool /path/to/motatool \
--preflight-only
```
This validates the corrected bundle, Full Companion identity, source, and
destination without changing radio or watchdog settings. Omitting
`--accept-test-candidate` exits before connecting to a device. The withdrawn
v1.17.02 chain is also recognized and always blocked before device access.
## Direct lab template
Use only on a locally recoverable test RAK3401. Keep the work directory for
resume, and retain the explicit `--accept-test-candidate` lab gate.
```bash
export MESHCORE_ADMIN_PASSWORD='password'
python3 tools/lora_ota/rak3401_mota_chain.py \
--work-dir ./rak3401-mota-chain-work \
--controller-serial /dev/ttyACM0 \
--source-tcp 192.168.1.51:5001 \
--source-cli-tcp 192.168.1.51:5002 \
--source-shares-controller \
--ota-hops 3 \
--accept-test-candidate \
--motatool /path/to/motatool \
--yes
```
The intended runner logs each isolated step attempt below `steps/`, records every
verified transition in `progress.jsonl`, and prints the source log path. Rerun
the exact command after a host restart or recoverable RF failure. Its default
60-second status interval deliberately leaves airtime for the low-priority OTA
transfer; avoid shortening it on a busy or relayed mesh.
## Relayed lab template
Every intermediate relay must enter the same TempRadio tuple. Add one
`--relay` argument for each intermediate repeater, ordered from the destination
side back toward the controller. In other words, list the farthest relay first
and the nearest relay last. A bare relay name uses
`MESHCORE_ADMIN_PASSWORD`; append `=PASSWORD` only when that relay has a
different password.
The word "hop" is sometimes counted two ways:
- `controller -> relay -> destination` is two RF links but has one
intermediate relay, so pass one `--relay`.
- `controller -> near relay -> far relay -> destination` has two intermediate
relays, so pass two `--relay` values, `far` first and `near` second.
For one intermediate relay:
```bash
python3 tools/lora_ota/rak3401_mota_chain.py \
--work-dir ./rak3401-mota-chain-work \
--controller-serial /dev/ttyACM0 \
--source-tcp 192.168.1.51:5001 \
--source-cli-tcp 192.168.1.51:5002 \
--source-shares-controller \
--ota-hops 3 \
--relay 'Intermediate Repeater' \
--accept-test-candidate \
--motatool /path/to/motatool \
--yes
```
For two intermediate relays:
```bash
python3 tools/lora_ota/rak3401_mota_chain.py \
--work-dir ./rak3401-mota-chain-work \
--controller-serial /dev/ttyACM0 \
--source-tcp 192.168.1.51:5001 \
--source-cli-tcp 192.168.1.51:5002 \
--source-shares-controller \
--ota-hops 3 \
--relay 'Far Repeater' \
--relay 'Near Repeater' \
--accept-test-candidate \
--motatool /path/to/motatool \
--yes
```
Do not begin a production deployment until the corrected bundle passes a
direct physical smoke test from start to endpoint. Run `--preflight-only` with
the same relay arguments first. The runner sends
the destination to TempRadio, then the relays farthest-to-nearest while their
remaining normal routes still exist. It checks that the 120-minute TempRadio
window covers the selected relay count and all configured timeouts. The
starting build may report `hops=0` after the watchdog-driven reset, so the
runner re-enforces `--ota-hops 3` before every step. That setting accepts OTA
packets received through up to three intermediate repeaters.
For a lossy multi-hop path, increase `--reply-timeout`,
`--discovery-timeout`, or `--transfer-timeout-minutes` as needed. If their sum
no longer fits the TempRadio window, also increase the minutes in
`--temp-radio`; the runner rejects an unsafe combination before connecting.
## Completion checks
Success requires all of the following:
- version `v1.17.1.0`
- target ID `2FA509C1`
- hardware `RAK_3401`
- final EndF body hash `AC267B02E055F42E`
- `get system.watchdog` returning `> on`
Do not treat a successful transfer alone as completion. Each temporary bridge
is a complete bootable image, but the RAK should not be left on a bridge
version between planned maintenance sessions.
See [LoRa OTA automation](lora_ota_automation.md) for the generic single-step
workflow and source topology details.