rich/MeshCore
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add compressed self-serve OTA and resilient native USB sleep

Browse Source 
Use a bounded uzlib-derived block encoder with the existing strict tinf decoder, qualify infrastructure roles, budget temporary RAM, and cache running-image metadata. Add self-serve CLI and policy, USB host-loss sleep grace, hardware test harnesses, regression tests, and CI coverage.
This commit is contained in:
mikecarper committed 2026-10-03 01:14:23 -07:00
1 parent 43f474a806
commit f0199b3111
34 files changed
+5437 -40
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/run-unit-tests.yml
+10 -2
View File
@@ -166,17 +166,25 @@ jobs:
run: |
python3 -B -m unittest discover -s test -p 'test_profile_*.py' -v
python3 -B -m unittest discover -s tools/hil -p 'test_*.py' -v
python3 -B test/test_hil_esp32_self_serve_ota.py -v
python3 -B test/test_hil_esp32_usb_host_reboot.py -v
- name: Verify ESP32 static DRAM budget
run: |
python3 -B test/test_esp32_dram.py
python3 -B test/test_elrs_power.py
python3 -B test/test_ota_heap_context.py
python3 -B test/test_ota_self_serve_policy.py -v
python3 -B test/test_ota_self_metadata_cache.py -v
python3 -B test/test_ota_device_deflate_encoder.py -v
python3 -B test/test_ota_device_deflate_role.py -v
python3 -B test/test_client_acl_spiffs.py
python3 -B test/test_client_acl_cli.py -v
- name: Verify ESP32 USB sleep and G3 button wake
run: python3 -B test/test_esp32_usb_sleep.py
- name: Verify ESP32 USB reboot sleep grace and G3 button wake
run: |
python3 -B test/test_usb_host_sleep_policy.py
python3 -B test/test_esp32_usb_sleep.py
- name: Verify GPS status distinguishes user settings from receiver sleep
run: |
docs/ota_user_guide.md
+57
View File
@@ -571,6 +571,61 @@ offers it to *its* neighbours too. So a new firmware spreads outward node-to-nod
hammering the one node that had it first. Discovery remains background traffic; an actual transfer is
primary traffic for the duration of its TempRadio maintenance window.
### Share the firmware already running on a node
OTA-enabled ESP32 and external-storage nRF52 repeater, room-server, and sensor
builds automatically offer their running application during a TempRadio window.
Both source and receiver need overlapping windows on the same radio settings.
Adaptive RAK builds offer it only when the application has selected a usable,
bootloader-matched external store. Internal-only nRF52 nodes do not automatically
offer full images; they still need a computer-built delta to receive an update.
Start the overlapping TempRadio windows before explicitly publishing the source.
Boards with an on-demand OTA workspace release it when the window ends, so the
source is available only during the window. To check or publish it:
```
ota serve status
ota serve self
```
`ota serve self` selects the running application as the primary source and
announces it. It replaces any manually staged primary image; attached host folders
remain available. Automatic offering never replaces a manually staged primary.
The receiver uses the normal `ota ls`, `ota get <mid8> flash`, and `ota install`
commands. It must have compatible hardware, enough staging/application space,
and the required OTA-capable bootloader. ESP32 reads the running A or B app
partition; nRF52 reads only its application region. Private keys, ACLs, node names,
and saved radio settings are not part of the shared image and are not copied from
the source. Existing update paths preserve the receiver's saved data.
The generated package is **full and unsigned**, streamed directly from flash
without keeping the whole image in RAM. This does not enable automatic download
or installation, bypass trust checks, or generate deltas on-device. Trusted-only
automatic installation rejects it. SD-backed nRF52 receivers also require a
trusted signed application package for manual installation: those sources can
export their image, but installation there needs a computer-packaged, signed
`.mota`. Source-only Companions continue to offer host-provided files instead.
Eligible repeater, room-server, and sensor builds can compress their own-image
blocks on the serving radio with a small, bounded raw-DEFLATE encoder. The
receiver uses the existing `tinf` decoder; no second decoder is added. Each
2 KiB block is independent, and incompressible blocks, unsupported receivers,
or insufficient encoder memory fall back to uncompressed transfer. Only the
application image through its EndF trailer is shared, not unused partition
space. This remains a full image, not a device-generated delta.
The radio does not run the computer packager's Zopfli encoder. Host-generated
packages retain their stronger precompressed-block path. Allow a maintenance
window long enough for the full image. ESP32 caches successfully read running-image metadata
for the current boot and partition geometry, so repeated `ota status` requests
do not rescan the application and interrupt radio servicing.
For an exact delta base from an internal-only nRF52, the existing diagnostic
`ota dev serve self` export, followed by `ota announce`, remains available during
TempRadio for capture onto a computer; it does not make that node capable of
receiving a full image.
---
## Where firmware files come from
@@ -603,6 +658,8 @@ that only contains what changed). You get them by:
| Trust a signer | `ota key add <hex>` |
| Relay a folder (gateway) | `ota folder on` + the seeder daemon |
| List what I'm offering | `ota folder` |
| Check whether I'm sharing my running firmware | `ota serve status` |
| Share my running firmware (ESP32 / external nRF52) | `ota serve self` |
(Older names still work too: `neighbors`/`updates` = `ls`, `pull` = `get`, `applydelta`/`apply` = `install`, `drop`/`stop` = `cancel`.)
platformio.ini
+2
View File
@@ -85,6 +85,7 @@ monitor_filters = esp32_exception_decoder
extra_scripts =
pre:scripts/generate_webconfig_html.py
pre:scripts/meshcore_image_identity.py
pre:scripts/ota_device_deflate_role.py
pre:scripts/portable_esp32_link.py
pre:scripts/esp32_ota_heap_context.py
merge-bin.py
@@ -153,6 +154,7 @@ platform_packages =
framework-arduinoadafruitnrf52 @ https://github.com/meshcore-dev/Adafruit_nRF52_Arduino#d541301665b40959682252911e57b11df3ee651a
platformio/toolchain-gccarmnoneeabi@^1.140201.0
extra_scripts = create-uf2.py
pre:scripts/ota_device_deflate_role.py
pre:scripts/nrf52_usb_power_fix.py
pre:scripts/nrf52_internal_flash_fix.py
pre:scripts/nrf52_ble_cccd_fix.py
scripts/check_firmware_ram.py
+23 -1
View File
@@ -106,6 +106,28 @@ def requirements(platform, defines, target):
parts["room_flood_rule_table"] = 31 * 200 + 16
if "ENABLE_OTA" in defines and "OTA_HEAP_CONTEXT" in defines:
parts["ota_context"] = 16384 + 16
# Match OtaDeflateConfig.h's compile eligibility and build-role override.
# Adaptive RAK may qualify only at runtime; reserve its worst eligible case.
# The shared role hook qualifies sources, not environment names. An
# infrastructure recipe can inherit a Companion-named custom environment.
device_deflate = ("ENABLE_OTA" in defines and "OTA_SEEDER_ONLY" not in defines
and integer(defines, "MESHCORE_OTA_DEVICE_DEFLATE", 0) != 0
and (platform == "ESP32_PLATFORM" or
(platform == "NRF52_PLATFORM" and any(name in defines for name in
("OTA_QSPI_STORE", "OTA_SD_STORE", "OTA_RAK_AUTO_STORE")))))
if device_deflate:
hash_bits = integer(defines, "MESHCORE_OTA_DEFLATE_HASH_BITS", 9)
if not 7 <= hash_bits <= 10:
raise ValueError("device DEFLATE hash must have 128..1024 entries")
# The encoder uses input offsets, not pointers: sizeof(uint16_t) per
# bucket on every platform. This allocation is freed before radio TX.
parts["ota_encoder_workspace"] = (1 << hash_bits) * 2 + 16
if platform == "NRF52_PLATFORM":
# The running nRF52840 app is <1 MiB: <=512 2-KiB leaves. Self-serve
# holds its leaf cache plus max(leaves,2048) proof/output scratch.
# Unlike ESP32's existing source scratch reserve these allocations
# were formerly covered only by the general transient margin.
parts["ota_self_source_scratch"] = 2 * 2048 + 2 * 16
if display and display != "NullDisplayDriver":
parts["display_pixels_and_driver"] = display_heap
screen_budget = integer(defines, "MESH_COMPANION_SCREEN_STARTUP_BYTES", 0)
@@ -137,7 +159,7 @@ def requirements(platform, defines, target):
if "expanded_message_previews" in parts:
largest = max(largest, 8192 + parts["expanded_message_previews"])
for name in ("client_table", "flood_filter_table", "room_flood_rule_table",
"ota_context", "screen_objects_and_history"):
"ota_context", "screen_objects_and_history", "ota_encoder_workspace"):
largest = max(largest, parts.get(name, 0))
return {"required_heap_bytes": required, "required_contiguous_bytes": largest,
"components": parts, "display": display, "full_companion": full}
scripts/ota_device_deflate_role.py
+94
View File
@@ -0,0 +1,94 @@
"""Enable bounded on-device OTA encoding only for qualified infrastructure.
Inspect the EFFECTIVE ordered source filter, never the environment name. This
pre-script contributes one shared compiler definition, so the C encoder and all
C++ OTA translation units agree. A custom recipe that omits this hook keeps
OtaDeflateConfig.h's fail-closed default. Companions keep host-precompressed OTA.
"""
import importlib.util
from pathlib import Path
SCRIPT_DIR = Path(__file__).resolve().parent if "__file__" in globals() else Path("scripts").resolve()
SPEC = importlib.util.spec_from_file_location(
"ota_device_deflate_source_policy", SCRIPT_DIR / "nrf52_flash_trim.py")
SOURCE_POLICY = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(SOURCE_POLICY)
DEFINITION = "MESHCORE_OTA_DEVICE_DEFLATE"
ROLE_SOURCES = (
"../examples/simple_repeater/main.cpp",
"../examples/simple_room_server/main.cpp",
"../examples/simple_sensor/main.cpp",
)
EXCLUDED_SOURCES = (
"../examples/companion_radio/main.cpp",
"../examples/companion_radio/MyMesh.cpp",
"../examples/simple_secure_chat/main.cpp",
"../examples/kiss_modem/main.cpp",
"../examples/partition_expander/main.cpp",
"../examples/esp32_partition_migrator/main.cpp",
"../examples/esp32_partition_legacy_seed/main.cpp",
)
def enabled(values, name):
value = values.get(name)
return value is not None and str(value).strip("\"'() ") not in ("", "0")
def bool_override(value):
# -U leaves the header's default zero and is an explicit opt-out as well.
if value is None:
return 0
text = str(value).strip("\"'() ")
if text not in ("0", "1"):
raise ValueError(DEFINITION + " override must be 0 or 1")
return int(text)
def role_policy(platform, mcu, src_filter, flags=(), cppdefines=()):
"""Return a common define or reject an unsafe explicit enable override."""
values = SOURCE_POLICY.flag_defines(flags, cppdefines)
selected = SOURCE_POLICY.source_selected
known_roles = sum(selected(src_filter, source) for source in ROLE_SOURCES)
excluded = any(selected(src_filter, source) for source in EXCLUDED_SOURCES)
seeder = values.get("OTA_SEEDER_ONLY") is not None
companion = any(enabled(values, name) for name in values
if name.startswith("COMPANION_"))
ota_enabled = enabled(values, "ENABLE_OTA") and not enabled(values, "DISABLE_LORA_OTA")
esp32 = (platform == "espressif32" and str(mcu).lower().startswith("esp32")
and enabled(values, "ESP32_PLATFORM"))
nrf52 = (platform == "nordicnrf52" and str(mcu).lower().startswith("nrf52")
and enabled(values, "NRF52_PLATFORM")
and any(enabled(values, store)
for store in ("OTA_QSPI_STORE", "OTA_SD_STORE", "OTA_RAK_AUTO_STORE")))
qualified = (known_roles == 1 and not excluded and not seeder and not companion
and ota_enabled and (esp32 or nrf52))
if DEFINITION in values:
override = bool_override(values[DEFINITION])
if override and not qualified:
raise ValueError(DEFINITION + "=1 requires an OTA infrastructure source role "
"and eligible ESP32/external-nRF52 storage")
return {} # Preserve an explicit valid definition/opt-out without duplicates.
return {DEFINITION: 1} if qualified else {}
def install(build_env):
platform = build_env.PioPlatform().name
mcu = build_env.BoardConfig().get("build.mcu", "")
src_filter = build_env.get("SRC_FILTER")
if src_filter is None:
src_filter = build_env.GetProjectOption("build_src_filter", "")
definitions = role_policy(platform, mcu, src_filter,
build_env.get("BUILD_FLAGS", ()),
build_env.get("CPPDEFINES", ()))
if definitions:
build_env.AppendUnique(CPPDEFINES=list(definitions.items()))
print("Device OTA DEFLATE: qualified infrastructure role; bounded per-block encoder")
if "Import" in globals():
Import("env")
install(env)
src/Mesh.cpp
+4 -3
View File
@@ -521,10 +521,11 @@ void __attribute__((noinline)) Mesh::serviceLoopMaintenance() {
bool in_burst = _ota_announce_count < OTA_ANNOUNCE_BURST;
uint32_t mins = oc.manager.advert_mins(); // periodic cadence in minutes; 0 = disabled (boot burst only)
if (in_burst || mins != 0) {
// Install-capable nodes add their running firmware to the served set.
// Seeder-only nodes advertise only host-provided folder entries.
// Full-image platforms offer the running app directly from flash. Keep a
// manually staged primary source intact; host folder entries coexist.
// Internal-only nRF52 diagnostic exports remain explicitly selectable.
#if !defined(OTA_SEEDER_ONLY)
if (!oc.serving) oc.serving = ota::ota_serve_self(oc, 0);
if (oc.self_serve_supported && !oc.serving) oc.serving = ota::ota_serve_self(oc, 0);
#endif
oc.manager.announce();
if (_ota_announce_count < 250) _ota_announce_count++;
src/helpers/ESP32Board.h
+35 -6
View File
@@ -17,6 +17,13 @@
#include <driver/rtc_io.h>
#include "ESP32TrueRandom.h"
#include "UsbLogging.h"
#include "UsbHostSleepPolicy.h"
#ifndef MESH_ESP32_USB_HOST_LOSS_SLEEP_GRACE_MS
#define MESH_ESP32_USB_HOST_LOSS_SLEEP_GRACE_MS 120000UL
#endif
static_assert(MESH_ESP32_USB_HOST_LOSS_SLEEP_GRACE_MS <= 0x7fffffffUL,
"USB host-loss sleep grace must fit a bounded millis interval");
#if defined(ARDUINO_USB_CDC_ON_BOOT) && ARDUINO_USB_CDC_ON_BOOT && \
(!defined(ARDUINO_USB_MODE) || !ARDUINO_USB_MODE)
@@ -32,6 +39,9 @@ class ESP32Board : public mesh::MainBoard {
protected:
uint8_t startup_reason;
bool inhibit_sleep = false;
#if MESH_ESP32_USB_CONSOLE_COOPERATIVE
mesh::UsbHostSleepPolicy usb_host_sleep_policy;
#endif
#if defined(LIGHTWEIGHT_WIFI_OTA)
void* ota_server = nullptr;
#else
@@ -104,7 +114,17 @@ public:
// Native USB loses its connection in light sleep. An enumerated host
// still needs USB serviced when no terminal asserts CDC DTR (for example,
// after a Pi closes its serial port). Guard every caller here.
bool keep_awake = inhibit_sleep || isUsbHostConnected() || isRadioTestActive();
// Sample even when another blocker is active, so an open console/OTA does
// not leave the last positive native USB host signal stale.
const bool usb_host_connected = isUsbHostConnected();
bool keep_awake = inhibit_sleep || usb_host_connected || isRadioTestActive();
#if MESH_ESP32_USB_CONSOLE_COOPERATIVE
// HWCDC loses its SOF host signal after about 5 ms. A warm Pi reboot,
// especially on a USB 1.1 bus, must not put us in a 30-second light sleep
// while the host is returning and trying to enumerate this same radio.
keep_awake = keep_awake || usb_host_sleep_policy.shouldKeepAwake(
millis(), MESH_ESP32_USB_HOST_LOSS_SLEEP_GRACE_MS);
#endif
#if MESH_USB_LOGGING_AVAILABLE
// A live logging stream must also remain available before a host opens
// it and across host disconnects. Compiled-out logging is not a blocker.
@@ -234,6 +254,11 @@ public:
bool isUsbDataConnected() override {
#if defined(ARDUINO_USB_CDC_ON_BOOT) && ARDUINO_USB_CDC_ON_BOOT
#if MESH_ESP32_USB_CONSOLE_COOPERATIVE
// Role loops may never call sleep() while a CDC client is open. Keep the
// native-host signal history current here too; host detection never calls us.
(void)isUsbHostConnected();
#endif
return (bool)Serial;
#else
return false;
@@ -241,17 +266,21 @@ public:
}
bool isUsbHostConnected() override {
bool host_connected = false;
#if defined(ARDUINO_USB_CDC_ON_BOOT) && ARDUINO_USB_CDC_ON_BOOT
#if defined(ARDUINO_USB_MODE) && ARDUINO_USB_MODE
return Serial.isPlugged();
host_connected = Serial.isPlugged();
#elif defined(CONFIG_TINYUSB_ENABLED) && CONFIG_TINYUSB_ENABLED
return (bool)USB;
host_connected = (bool)USB;
#else
return (bool)Serial;
host_connected = (bool)Serial;
#endif
#else
return false;
#endif
#if MESH_ESP32_USB_CONSOLE_COOPERATIVE
usb_host_sleep_policy.observe(host_connected, millis());
#endif
// The grace is a sleep policy, not proof that a USB host is still present.
return host_connected;
}
void setInhibitSleep(bool inhibit) {
src/helpers/UsbHostSleepPolicy.h
+37
View File
@@ -0,0 +1,37 @@
#pragma once
#include <stdint.h>
namespace mesh {
// Keep a native USB radio awake through a short host reboot after a positive
// native USB host signal. No reported host means no grace. Some cores initially
// report a host before their first SOF check; that can arm only this bounded
// window, never an indefinite charger-only/battery sleep inhibitor.
class UsbHostSleepPolicy {
uint32_t _last_host_seen_ms = 0;
bool _host_present = false;
bool _loss_grace_armed = false;
public:
void observe(bool host_present, uint32_t now) {
_host_present = host_present;
if (host_present) {
_last_host_seen_ms = now;
_loss_grace_armed = true;
}
}
bool shouldKeepAwake(uint32_t now, uint32_t grace_ms) {
if (_host_present) return true;
if (!_loss_grace_armed) return false;
if (static_cast<uint32_t>(now - _last_host_seen_ms) < grace_ms) {
return true;
}
// Disarm after expiry so another millis() cycle cannot revive an old signal.
_loss_grace_armed = false;
return false;
}
};
} // namespace mesh
src/helpers/ota/OtaCli.cpp
+33 -9
View File
@@ -4,6 +4,7 @@
#include "FolderMotaStore.h" // `ota pull <id> folder` destination (set_mid on the connected folder store)
#include "OtaVerify.h"
#include "OtaSelf.h"
#include "OtaByteIO.h"
#include "OtaTargets.h" // ota_target_env_name(): human-readable name for a target_id (no string on the wire)
#if defined(NRF52_PLATFORM)
#include "OtaBlInfo.h" // installed bootloader application/update capability views
@@ -183,30 +184,30 @@ bool handle_ota_command(const char* command, char* reply, mesh::MainBoard& board
#endif
#elif defined(NRF52_PLATFORM) && defined(OTA_RAK_AUTO_STORE)
strcpy(reply,
"OTA: status | stats | ls | get <id> flash | install | bootloader | cancel | announce | self | storage | folder | config | key");
"OTA: status | stats | ls | get <id> flash | install | bootloader | cancel | announce | self | serve | storage | folder | config | key");
#elif defined(NRF52_PLATFORM) && defined(OTA_QSPI_STORE)
#if defined(OTA_QSPI_BOOTLOADER_UPDATE)
strcpy(reply,
"OTA: status | stats | ls | get <id> flash | install | bootloader | cancel | announce | self | "
"OTA: status | stats | ls | get <id> flash | install | bootloader | cancel | announce | self | serve | "
"qspi | folder | config | key");
#else
strcpy(reply,
"OTA: status | stats | ls | get <id> flash | install | cancel | announce | self | qspi | "
"OTA: status | stats | ls | get <id> flash | install | cancel | announce | self | serve | qspi | "
"folder | config | key");
#endif
#elif defined(NRF52_PLATFORM) && defined(OTA_SD_STORE)
#if defined(OTA_SD_BOOTLOADER_UPDATE)
strcpy(reply,
"OTA: status | stats | ls | get <id> flash | install | bootloader | cancel | announce | self | "
"OTA: status | stats | ls | get <id> flash | install | bootloader | cancel | announce | self | serve | "
"folder | cache | config | key");
#else
strcpy(reply,
"OTA: status | stats | ls | get <id> flash | install | cancel | announce | self | folder | "
"OTA: status | stats | ls | get <id> flash | install | cancel | announce | self | serve | folder | "
"cache | config | key");
#endif
#else
snprintf(reply, 160,
"OTA: status | stats | ls | get <id> flash | install | cancel | announce | self | folder | "
"OTA: status | stats | ls | get <id> flash | install | cancel | announce | self | serve | folder | "
"cache | config | key. `ota ls [page]`; folder [validate].");
#endif
@@ -731,6 +732,28 @@ bool handle_ota_command(const char* command, char* reply, mesh::MainBoard& board
fstate_char(fs), midhx);
}
// ---- public own-image source: reads the app, never saved identity/settings ----
} else if (is_cmd(a, "serve", &rest)) {
if (*rest == 0 || strcmp(rest, "status") == 0) {
const bool own_image = c.manager.servingPrimaryManifest(c.serve_self_manifest);
snprintf(reply, 160,
"Own firmware: default:%s serving:%s | full unsigned | transfers require TempRadio",
c.self_serve_supported ? "on" : "off", own_image ? "on" : "off");
} else if (strcmp(rest, "self") != 0) {
strcpy(reply, "ERR usage: ota serve [self|status]");
} else if (!c.self_serve_supported) {
strcpy(reply, "ERR self serve requires ESP32 or external-storage nRF52; host sources use ota folder");
} else if (!ota_serve_self(c, 0)) {
strcpy(reply, "ERR serve self (invalid running image or insufficient RAM)");
} else {
char midhx[9]; mesh::Utils::toHex(midhx, c.serve_self_manifest + 20, 4);
uint32_t image = rd_u32le(c.serve_self_manifest + 11);
c.manager.announce();
snprintf(reply, 160,
"OK serving own fw mid=%s (%u B, flash-backed, unsigned); transfers require TempRadio",
midhx, (unsigned)image);
}
// ---- broadcast our tiny beacon so peers discover us. If not already serving, set up flash-backed
// self-serve first (so we're a real, fetchable source of our own running firmware). ----
} else if (is_cmd(a, "announce|adv", &rest)) {
@@ -739,7 +762,7 @@ bool handle_ota_command(const char* command, char* reply, mesh::MainBoard& board
snprintf(reply, 160, "OK beacon sent (serving=%u host mOTA)",
(unsigned)c.manager.servedCount());
#else
if (!c.serving) c.serving = ota_serve_self(c, 0);
if (c.self_serve_supported && !c.serving) c.serving = ota_serve_self(c, 0);
c.manager.announce();
sprintf(reply, "OK beacon sent (serving=%s)", c.serving ? "self fw" : "nothing");
#endif
@@ -957,7 +980,7 @@ bool handle_ota_command(const char* command, char* reply, mesh::MainBoard& board
if (strcmp(p, "on") == 0) {
#if defined(OTA_FOLDER_SERIAL)
#if !defined(OTA_SEEDER_ONLY)
if (!c.serving) c.serving = ota_serve_self(c, 0); // keep serving our own fw alongside the folder
if (c.self_serve_supported && !c.serving) c.serving = ota_serve_self(c, 0);
#endif
char m2[120]; c.attach_folder(m2, sizeof(m2)); c.manager.announce();
strncpy(reply, m2, 159); reply[159] = 0;
@@ -969,7 +992,8 @@ bool handle_ota_command(const char* command, char* reply, mesh::MainBoard& board
#if defined(OTA_SEEDER_ONLY)
strcpy(reply, "OK folder detached (serving nothing)");
#else
strcpy(reply, "OK folder detached (still serving own fw)");
strcpy(reply, c.serving ? "OK folder detached (primary source retained)"
: "OK folder detached (no primary source)");
#endif
} else if (*p == 0) { // status + list served entries (* = our own fw)
uint16_t offered = 0, advertised = 0;
src/helpers/ota/OtaContext.h
+16
View File
@@ -12,6 +12,7 @@
#include "OtaFormat.h"
#include "OtaByteIO.h"
#include "OtaSelf.h" // ota_self_firmware() - prefer self-describing EndF identity at begin()
#include "OtaSelfServePolicy.h"
#include "OtaBlInfo.h" // bootloader OTA-apply capability marker (nRF52); cached after first read
// Storage policy for the mOTA context. A "dynamic" context is created on demand
@@ -144,6 +145,10 @@ struct OtaContext {
#endif
uint32_t serve_expected = 0; // size declared by `ota stage`
bool serving = false; // manager.serve() succeeded
// Capture application-storage eligibility once. An adaptive nRF52 may later
// switch to internal flash to stage a bootloader without changing its normal
// ability to offer its running application.
bool self_serve_supported = false;
// flash-backed self-serve: cached merkle leaves (heap, freed on re-serve) + assembled manifest of our
// own running firmware. The payload is read from flash per block; only the metadata is held in RAM.
// serve_self_proof is the proof-gen working buffer (>= block_count*4) - sized to OUR image's block
@@ -713,6 +718,17 @@ struct OtaContext {
manager.set_accept_bootloader(false);
#endif
manager.set_fetch_store(&fetch_store);
#if defined(NRF52_PLATFORM) && defined(OTA_RAK_AUTO_STORE) && !defined(OTA_SEEDER_ONLY)
self_serve_supported = ota_self_serve_supported(fetch_store.usesExternal());
#else
self_serve_supported = ota_self_serve_supported();
#endif
#if MESHCORE_OTA_DEVICE_DEFLATE
// Register only on qualified full-image nodes (adaptive RAK eligibility
// is the application backend latched above). Host precompressed blocks
// still take precedence in OtaManager; receiver checks are unchanged.
manager.set_transport_deflate_encoder(self_serve_supported ? ota_transport_deflate : nullptr);
#endif
#if defined(NRF52_PLATFORM) && defined(OTA_SD_STORE)
sd_cache.attach(fetch_store);
manager.set_archive_interest(true); // default on; the SD marker can turn it off at first mount
src/helpers/ota/OtaDeflate.cpp
+26
View File
@@ -4,6 +4,9 @@
extern "C" {
#include "tinf/tinf.h"
#if MESHCORE_OTA_DEVICE_DEFLATE
#include "tinydeflate/deflate.h"
#endif
}
namespace mesh {
@@ -22,6 +25,29 @@ bool ota_transport_inflate(void* context, const uint8_t* src, uint16_t src_len,
return true;
}
#if MESHCORE_OTA_DEVICE_DEFLATE
bool ota_transport_deflate(void* context, const uint8_t* src, uint16_t src_len,
uint8_t* dst, uint16_t dst_cap, uint16_t* dst_len) {
(void)context;
if (dst_len) *dst_len = 0;
if (!src || src_len < 3 || src_len > MESHCORE_OTA_DEFLATE_BLOCK_MAX ||
!dst || !dst_cap || !dst_len) return false;
// Proof/output scratch is disjoint from the manager's raw input. Reject
// accidental aliases without overflowing a pointer-range endpoint.
const uintptr_t source = reinterpret_cast<uintptr_t>(src);
const uintptr_t output = reinterpret_cast<uintptr_t>(dst);
if ((output >= source && output - source < src_len) ||
(source > output && source - output < dst_cap)) return false;
// Abort as soon as the stream cannot save a byte. Never need a worst-case
// expanded-output buffer, and leave the original raw input intact on failure.
const uint16_t limit = dst_cap < src_len ? dst_cap : (uint16_t)(src_len - 1);
const uint16_t produced = meshcore_deflate_raw(src, src_len, dst, limit);
if (!produced || produced >= src_len) return false;
*dst_len = produced;
return true;
}
#endif
} // namespace ota
} // namespace mesh
src/helpers/ota/OtaDeflate.h
+11
View File
@@ -1,6 +1,7 @@
#pragma once
#include <stdint.h>
#include "OtaDeflateConfig.h"
namespace mesh {
namespace ota {
@@ -12,5 +13,15 @@ namespace ota {
bool ota_transport_inflate(void* context, const uint8_t* src, uint16_t src_len,
uint8_t* dst, uint16_t dst_cap, uint16_t* dst_len);
#if MESHCORE_OTA_DEVICE_DEFLATE
// Compact fixed-Huffman raw RFC1951 encoder; independent blocks of at most
// 2048 bytes, with disjoint input/output. Returns false/zero length on invalid
// arguments, OOM, output overflow, or no size saving, selecting raw OTA DATA.
// 512 uint16_t hash offsets (1024 bytes) are allocated only during the call.
// No shared state: repeated/reordered/concurrent blocks encode identically.
bool ota_transport_deflate(void* context, const uint8_t* src, uint16_t src_len,
uint8_t* dst, uint16_t dst_cap, uint16_t* dst_len);
#endif
} // namespace ota
} // namespace mesh
src/helpers/ota/OtaDeflateConfig.h
+27
View File
@@ -0,0 +1,27 @@
#pragma once
// Device-side compression is useful only where peers can install full own-images.
// Source-only Companions retain the host's stronger precompressed-block path;
// internal-only nRF52 receivers do not pay the encoder's flash or RAM cost.
#ifndef MESHCORE_OTA_DEVICE_DEFLATE
#if defined(OTA_TRANSPORT_DEFLATE_TEST)
#define MESHCORE_OTA_DEVICE_DEFLATE 1
#else
#define MESHCORE_OTA_DEVICE_DEFLATE 0
#endif
#endif
// Production's shared pre-script positively qualifies the effective source
// filter and platform/storage, and supplies this flag to EVERY translation
// unit. Without qualification the default stays off, including Companions,
// receiver-only targets, unknown roles and custom recipes missing the hook.
// Independent 2 KiB blocks need only 16-bit input offsets. 512 entries use
// 1024 bytes, allocated for one encode call and released before radio TX.
#ifndef MESHCORE_OTA_DEFLATE_HASH_BITS
#define MESHCORE_OTA_DEFLATE_HASH_BITS 9
#endif
#if MESHCORE_OTA_DEFLATE_HASH_BITS < 7 || MESHCORE_OTA_DEFLATE_HASH_BITS > 10
#error "Device DEFLATE hash must have 128..1024 entries"
#endif
#define MESHCORE_OTA_DEFLATE_BLOCK_MAX 2048u
src/helpers/ota/OtaManager.h
+5
View File
@@ -333,6 +333,11 @@ public:
// Stop serving the primary caller-owned image while preserving attached sources.
// Call this before releasing or overwriting the primary image's backing buffer.
void clear_primary();
// Identify the actual primary backing manifest, not just the view0 catalog
// slot: that slot can also contain a manually staged image of another build.
bool servingPrimaryManifest(const uint8_t* manifest) const {
return _view0.valid && _view0.m.manifest_start == manifest;
}
uint8_t servedCount() const { return _n_serve; } // total mOTAs we offer (own fw + folder)
// Per-source enumeration accounting. `offered` is what the attached host/source reported; `advertised`
// is what fit in this build's serve registry after validation/deduplication/capacity limits.
src/helpers/ota/OtaSelf.cpp
+40 -13
View File
@@ -27,10 +27,25 @@ namespace ota {
// Scan the running app partition for the firmware's EndF trailer using esp_partition_read (stable
// across IDF versions - no mmap). Same rule as find_self_firmware(): the marker's absolute offset
// must equal its stored body_len, which uniquely identifies the running firmware's own trailer.
// The active app is immutable until reboot: OTA writes its inactive peer. Cache only a successful
// read, keyed to the current running partition's geometry, so status polling never rescans the app.
bool ota_self_firmware(SelfFwInfo& out) {
struct MetadataCache {
bool valid = false;
uint32_t address = 0;
uint32_t size = 0;
SelfFwInfo info;
};
static MetadataCache cache;
out = SelfFwInfo();
const esp_partition_t* p = esp_ota_get_running_partition();
if (!p) return false;
if (!p) { cache.valid = false; return false; }
if (cache.valid && cache.address == p->address && cache.size == p->size) {
out = cache.info;
return true;
}
// Missing/changed partitions and failed scans must not resurrect a previous app's metadata.
cache.valid = false;
const uint32_t CH = 512;
uint8_t buf[CH + ENDF_LEN]; // overlap so a marker spanning a chunk edge is still seen
@@ -43,20 +58,24 @@ bool ota_self_firmware(SelfFwInfo& out) {
if (memcmp(buf + i, ENDF_MAGIC, 4) != 0) continue;
uint32_t body_len = rd_u32le(buf + i + 4);
if (body_len != base + i) continue; // must sit immediately after a body of that length
// Read the complete fixed trailer before publishing or caching ANY metadata. A transient
// failure here must remain retryable, not become a valid entry with missing identity fields.
uint8_t tr[ENDF_LEN];
if (body_len > p->size - ENDF_LEN ||
esp_partition_read(p, body_len, tr, ENDF_LEN) != ESP_OK) return false;
if (memcmp(tr, ENDF_MAGIC, 4) != 0 || rd_u32le(tr + 4) != body_len) return false;
out.valid = true;
out.endf_offset = body_len;
out.body_len = body_len;
out.image_len = body_len + ENDF_LEN;
memcpy(out.body_hash, buf + i + 8, 8);
// Fixed 56-byte trailer: re-read it whole at the marker (it may straddle the chunk window, so the
// identity fields aren't reliably in `buf`) and pull identity from constant offsets (docs Section 2).
uint8_t tr[ENDF_LEN];
if (body_len + ENDF_LEN <= p->size &&
esp_partition_read(p, body_len, tr, ENDF_LEN) == ESP_OK) {
out.fw_version = (uint32_t)tr[16] | ((uint32_t)tr[17]<<8) | ((uint32_t)tr[18]<<16) | ((uint32_t)tr[19]<<24);
out.target_id = (uint32_t)tr[20] | ((uint32_t)tr[21]<<8) | ((uint32_t)tr[22]<<16) | ((uint32_t)tr[23]<<24);
memcpy(out.hw_id, tr + 24, 32); out.hw_id[32] = 0;
}
memcpy(out.body_hash, tr + 8, 8);
out.fw_version = rd_u32le(tr + 16);
out.target_id = rd_u32le(tr + 20);
memcpy(out.hw_id, tr + 24, 32); out.hw_id[32] = 0;
cache.address = p->address;
cache.size = p->size;
cache.info = out;
cache.valid = true;
return true;
}
}
@@ -159,7 +178,15 @@ bool ota_serve_self(OtaContext& c, uint32_t fw_version) {
c.serving = false;
free(c.serve_self_leaves); free(c.serve_self_proof);
c.serve_self_leaves = (uint8_t*)malloc((size_t)bc * 4);
c.serve_self_proof = (uint8_t*)malloc((size_t)bc * 4); // proof-gen working buffer (sized to OUR image)
size_t proof_size = (size_t)bc * 4;
#if MESHCORE_OTA_DEVICE_DEFLATE
// Proof generation and transport encoding use this buffer at different
// times. Small nRF52 images can have <2 KiB of leaves: keep enough room for
// a whole encoded block without allocating a second input/output buffer.
// Receiver-only/Companion diagnostic raw exports retain the old allocation.
if (proof_size < BS) proof_size = BS;
#endif
c.serve_self_proof = (uint8_t*)malloc(proof_size);
if (!c.serve_self_leaves || !c.serve_self_proof) {
free(c.serve_self_leaves); free(c.serve_self_proof);
c.serve_self_leaves = c.serve_self_proof = nullptr;
@@ -200,7 +227,7 @@ bool ota_serve_self(OtaContext& c, uint32_t fw_version) {
memcpy(m + 57, out_hw, strlen(out_hw) < 32 ? strlen(out_hw) : 32); // hw_id[32] (NUL-padded by memset)
memcpy(m + MOTA_OFF_APPROVAL, APPROVAL_NOT, 4); // approval (fetching device's apply-gate handles it)
c.serving = c.manager.serve_self(m, MOTA_MFL, c.serve_self_leaves, bc,
c.serve_self_proof, (size_t)bc * 4, self_read_cb, nullptr);
c.serve_self_proof, proof_size, self_read_cb, nullptr);
return c.serving;
}
#endif
src/helpers/ota/OtaSelf.h
+4 -3
View File
@@ -3,15 +3,16 @@
#include "FirmwareInfo.h"
// Device-side accessor for the running firmware's own image (to read its EndF trailer).
// Per-platform: ESP32 memory-maps the running app partition; other platforms TBD (nRF52 uses the
// bootloader-apply path, so its app-region wiring lands with that work). Not compiled on the native
// host - the portable scan logic in FirmwareInfo.{h,cpp} is what gets unit-tested there.
// ESP32 reads the running A/B app partition; nRF52 reads its memory-mapped app
// region. The portable scan logic in FirmwareInfo.{h,cpp} is host-testable.
namespace mesh {
namespace ota {
// Locate this firmware's EndF trailer in its own flash image. Returns false if unsupported on this
// platform or no valid EndF is present (e.g. firmware built without the EndF build hook).
// ESP32 caches successful metadata for this boot while the running partition address/size match;
// missing partitions, changed geometry and read failures are never cached. OTA writes inactive flash.
bool ota_self_firmware(SelfFwInfo& out);
// Read `len` bytes of the running firmware image at offset `off` (ESP32: running partition via
src/helpers/ota/OtaSelfServePolicy.h
+28
View File
@@ -0,0 +1,28 @@
#pragma once
namespace mesh { namespace ota {
// Default own-image offering is for full-image platforms. Internal-only nRF52
// nodes retain the explicit diagnostic export used to capture delta bases, but
// do not automatically offer a full image their ordinary peers cannot install.
// Adaptive RAK builds must use the qualified application backend, not merely
// the presence of a NOR chip (or the temporary bootloader staging backend).
inline bool ota_self_serve_supported(bool external_application_storage = false) {
#if defined(OTA_SEEDER_ONLY)
(void)external_application_storage;
return false; // Companion sources offer host packages
#elif defined(ESP32_PLATFORM)
(void)external_application_storage;
return true;
#elif defined(NRF52_PLATFORM) && defined(OTA_RAK_AUTO_STORE)
return external_application_storage;
#elif defined(NRF52_PLATFORM) && (defined(OTA_QSPI_STORE) || defined(OTA_SD_STORE))
(void)external_application_storage;
return true;
#else
(void)external_application_storage;
return false;
#endif
}
} }
src/helpers/ota/OtaTinf.c
+6
View File
@@ -4,6 +4,7 @@
*/
#include "../CompanionReaderConfig.h"
#include "../ui/Nrf52FontConfig.h"
#include "OtaDeflateConfig.h"
#if defined(ENABLE_OTA) || defined(OTA_TRANSPORT_DEFLATE_TEST) || COMPANION_FEATURE_READER \
|| MESHCORE_FONT_DEFLATE
#if defined(__GNUC__) && !defined(__clang__)
@@ -14,6 +15,11 @@
#endif
#define MESHCORE_TINF_IMPLEMENTATION 1
#include "tinf/tinflate.c"
#if MESHCORE_OTA_DEVICE_DEFLATE
// One true-C translation unit for the existing strict decoder and the small
// bounded encoder. Do not bring uzlib's second decoder or wrapper sources in.
#include "tinydeflate/deflate.c"
#endif
#if defined(__GNUC__) && !defined(__clang__)
#pragma GCC pop_options
#endif
src/helpers/ota/tinydeflate/LICENSE
+65
View File
@@ -0,0 +1,65 @@
Introduction
------------
uzlib as a whole is licensed under the terms of Zlib license, listed in
the next section. It consists of substantial works of individuals whose
copyrights listed in the next session. Some portions of the uzlib codebase
originally were licensed under different license(s), however compatible
with the Zlib license. Such license(s) are listed at the end of this file.
License
-------
uzlib - Deflate/Zlib-compatible LZ77 compression/decompression library
Copyright (c) 2003 Joergen Ibsen
Copyright (c) 1997-2014 Simon Tatham
Copyright (c) 2014-2020 Paul Sokolovsky
This software is provided 'as-is', without any express or implied
warranty. In no event will the authors be held liable for any damages
arising from the use of this software.
Permission is granted to anyone to use this software for any purpose,
including commercial applications, and to alter it and redistribute it
freely, subject to the following restrictions:
1. The origin of this software must not be misrepresented; you must
not claim that you wrote the original software. If you use this
software in a product, an acknowledgment in the product
documentation would be appreciated but is not required.
2. Altered source versions must be plainly marked as such, and must
not be misrepresented as being the original software.
3. This notice may not be removed or altered from any source
distribution.
-------
Original license for src/defl_static.c:
PuTTY is copyright 1997-2014 Simon Tatham.
Portions copyright Robert de Bath, Joris van Rantwijk, Delian
Delchev, Andreas Schultz, Jeroen Massar, Wez Furlong, Nicolas Barry,
Justin Bradford, Ben Harris, Malcolm Smith, Ahmad Khalifa, Markus
Kuhn, Colin Watson, and CORE SDI S.A.
Permission is hereby granted, free of charge, to any person
obtaining a copy of this software and associated documentation files
(the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge,
publish, distribute, sublicense, and/or sell copies of the Software,
and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE COPYRIGHT HOLDERS BE LIABLE
FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
src/helpers/ota/tinydeflate/README.meshcore.txt
+30
View File
@@ -0,0 +1,30 @@
Compact bounded device raw-DEFLATE encoder
=========================================
Derived from https://github.com/pfalcon/uzlib
Revision: 6d60d651a4499a64f2e5b21b4cc08d98cb84b5c1
Original files: src/genlz77.c, src/defl_static.c
Only the greedy single-probe LZ77/fixed-Huffman encoder is integrated. The
existing strict tinf decoder is reused; no second decoder, zlib/gzip wrapper,
checksum, or dynamic-Huffman implementation is added. The compressor is
compiled as C through ../OtaTinf.c with the existing local size optimization.
MeshCore modifications are marked in deflate.c. Hash buckets hold 16-bit
offsets+1 into the current <=2048-byte input instead of pointers. 512 buckets
need 1024 bytes, calloc'ed for one call then freed (OOM falls back to raw).
Hash bits may be 7..10; any override is also counted by the runtime RAM gate.
There is no persistent dictionary, no shared mutable state, and no input copy.
Output uses the caller's disjoint proof scratch, bounded at every byte write;
overflow/incompressible blocks return failure and the untouched raw input is
sent instead. The state is small on the stack (24 bytes on 32-bit targets),
with no recursion. Match searches stop at 258 bytes; all tail reads are bounded.
Repeated/reordered blocks produce identical bytes, independent of addresses.
Only eligible non-seeder ESP32/external-storage nRF52 targets compile the
encoder. Runtime qualification also gates adaptive RAK boards. Host supplied
precompressed blocks retain priority, and source-only Companions/internal-only
nRF52 receivers retain their existing decoder and raw path without this code.
The original Zlib and PuTTY MIT-compatible licenses are preserved in LICENSE
and the source notices. Modified source is not represented as upstream.
src/helpers/ota/tinydeflate/deflate.c
+384
View File
@@ -0,0 +1,384 @@
/*
Routines in this file are based on:
Zlib (RFC1950 / RFC1951) compression for PuTTY.
PuTTY is copyright 1997-2014 Simon Tatham.
Portions copyright Robert de Bath, Joris van Rantwijk, Delian
Delchev, Andreas Schultz, Jeroen Massar, Wez Furlong, Nicolas Barry,
Justin Bradford, Ben Harris, Malcolm Smith, Ahmad Khalifa, Markus
Kuhn, Colin Watson, and CORE SDI S.A.
Permission is hereby granted, free of charge, to any person
obtaining a copy of this software and associated documentation files
(the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge,
publish, distribute, sublicense, and/or sell copies of the Software,
and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE COPYRIGHT HOLDERS BE LIABLE
FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
#include <stdlib.h>
#include <stdint.h>
#include <string.h>
#include <assert.h>
#include "deflate.h"
#include "../OtaDeflateConfig.h"
/* MeshCore modifications (not unmodified upstream):
* Only the genlz77 + fixed-Huffman encoder is retained. Output is bounded,
* no realloc is used, and hash buckets are 16-bit offsets into ONE <=2KiB
* input block. The hash table is calloc'ed for one call and freed afterwards.
* No mutable global state, wrapper/checksum, or second decoder is included.
*/
struct meshcore_deflate {
uint8_t *outbuf;
uint16_t outlen, outsize;
uint32_t outbits;
unsigned noutbits;
int failed;
uint16_t *hash_table; /* input offset + 1; zero is the empty sentinel */
};
/* ----------------------------------------------------------------------
* Zlib compression. We always use the static Huffman tree option.
* Mostly this is because it's hard to scan a block in advance to
* work out better trees; dynamic trees are great when you're
* compressing a large file under no significant time constraint,
* but when you're compressing little bits in real time, things get
* hairier.
*
* I suppose it's possible that I could compute Huffman trees based
* on the frequencies in the _previous_ block, as a sort of
* heuristic, but I'm not confident that the gain would balance out
* having to transmit the trees.
*/
static void outbits(struct meshcore_deflate *out, uint32_t bits, unsigned nbits)
{
if (out->failed) return;
assert(out->noutbits + nbits <= 32);
out->outbits |= bits << out->noutbits;
out->noutbits += nbits;
while (out->noutbits >= 8) {
if (out->outlen >= out->outsize) {
out->failed = 1;
return;
}
out->outbuf[out->outlen++] = (unsigned char) (out->outbits & 0xFF);
out->outbits >>= 8;
out->noutbits -= 8;
}
}
static const unsigned char mirrorbytes[256] = {
0x00, 0x80, 0x40, 0xc0, 0x20, 0xa0, 0x60, 0xe0,
0x10, 0x90, 0x50, 0xd0, 0x30, 0xb0, 0x70, 0xf0,
0x08, 0x88, 0x48, 0xc8, 0x28, 0xa8, 0x68, 0xe8,
0x18, 0x98, 0x58, 0xd8, 0x38, 0xb8, 0x78, 0xf8,
0x04, 0x84, 0x44, 0xc4, 0x24, 0xa4, 0x64, 0xe4,
0x14, 0x94, 0x54, 0xd4, 0x34, 0xb4, 0x74, 0xf4,
0x0c, 0x8c, 0x4c, 0xcc, 0x2c, 0xac, 0x6c, 0xec,
0x1c, 0x9c, 0x5c, 0xdc, 0x3c, 0xbc, 0x7c, 0xfc,
0x02, 0x82, 0x42, 0xc2, 0x22, 0xa2, 0x62, 0xe2,
0x12, 0x92, 0x52, 0xd2, 0x32, 0xb2, 0x72, 0xf2,
0x0a, 0x8a, 0x4a, 0xca, 0x2a, 0xaa, 0x6a, 0xea,
0x1a, 0x9a, 0x5a, 0xda, 0x3a, 0xba, 0x7a, 0xfa,
0x06, 0x86, 0x46, 0xc6, 0x26, 0xa6, 0x66, 0xe6,
0x16, 0x96, 0x56, 0xd6, 0x36, 0xb6, 0x76, 0xf6,
0x0e, 0x8e, 0x4e, 0xce, 0x2e, 0xae, 0x6e, 0xee,
0x1e, 0x9e, 0x5e, 0xde, 0x3e, 0xbe, 0x7e, 0xfe,
0x01, 0x81, 0x41, 0xc1, 0x21, 0xa1, 0x61, 0xe1,
0x11, 0x91, 0x51, 0xd1, 0x31, 0xb1, 0x71, 0xf1,
0x09, 0x89, 0x49, 0xc9, 0x29, 0xa9, 0x69, 0xe9,
0x19, 0x99, 0x59, 0xd9, 0x39, 0xb9, 0x79, 0xf9,
0x05, 0x85, 0x45, 0xc5, 0x25, 0xa5, 0x65, 0xe5,
0x15, 0x95, 0x55, 0xd5, 0x35, 0xb5, 0x75, 0xf5,
0x0d, 0x8d, 0x4d, 0xcd, 0x2d, 0xad, 0x6d, 0xed,
0x1d, 0x9d, 0x5d, 0xdd, 0x3d, 0xbd, 0x7d, 0xfd,
0x03, 0x83, 0x43, 0xc3, 0x23, 0xa3, 0x63, 0xe3,
0x13, 0x93, 0x53, 0xd3, 0x33, 0xb3, 0x73, 0xf3,
0x0b, 0x8b, 0x4b, 0xcb, 0x2b, 0xab, 0x6b, 0xeb,
0x1b, 0x9b, 0x5b, 0xdb, 0x3b, 0xbb, 0x7b, 0xfb,
0x07, 0x87, 0x47, 0xc7, 0x27, 0xa7, 0x67, 0xe7,
0x17, 0x97, 0x57, 0xd7, 0x37, 0xb7, 0x77, 0xf7,
0x0f, 0x8f, 0x4f, 0xcf, 0x2f, 0xaf, 0x6f, 0xef,
0x1f, 0x9f, 0x5f, 0xdf, 0x3f, 0xbf, 0x7f, 0xff,
};
typedef struct {
uint8_t extrabits;
uint8_t min, max;
} len_coderecord;
typedef struct {
uint8_t code, extrabits;
uint16_t min, max;
} dist_coderecord;
#define TO_LCODE(x, y) x - 3, y - 3
#define FROM_LCODE(x) (x + 3)
static const len_coderecord lencodes[] = {
{0, TO_LCODE(3, 3)},
{0, TO_LCODE(4, 4)},
{0, TO_LCODE(5, 5)},
{0, TO_LCODE(6, 6)},
{0, TO_LCODE(7, 7)},
{0, TO_LCODE(8, 8)},
{0, TO_LCODE(9, 9)},
{0, TO_LCODE(10, 10)},
{1, TO_LCODE(11, 12)},
{1, TO_LCODE(13, 14)},
{1, TO_LCODE(15, 16)},
{1, TO_LCODE(17, 18)},
{2, TO_LCODE(19, 22)},
{2, TO_LCODE(23, 26)},
{2, TO_LCODE(27, 30)},
{2, TO_LCODE(31, 34)},
{3, TO_LCODE(35, 42)},
{3, TO_LCODE(43, 50)},
{3, TO_LCODE(51, 58)},
{3, TO_LCODE(59, 66)},
{4, TO_LCODE(67, 82)},
{4, TO_LCODE(83, 98)},
{4, TO_LCODE(99, 114)},
{4, TO_LCODE(115, 130)},
{5, TO_LCODE(131, 162)},
{5, TO_LCODE(163, 194)},
{5, TO_LCODE(195, 226)},
{5, TO_LCODE(227, 257)},
{0, TO_LCODE(258, 258)},
};
static const dist_coderecord distcodes[] = {
{0, 0, 1, 1},
{1, 0, 2, 2},
{2, 0, 3, 3},
{3, 0, 4, 4},
{4, 1, 5, 6},
{5, 1, 7, 8},
{6, 2, 9, 12},
{7, 2, 13, 16},
{8, 3, 17, 24},
{9, 3, 25, 32},
{10, 4, 33, 48},
{11, 4, 49, 64},
{12, 5, 65, 96},
{13, 5, 97, 128},
{14, 6, 129, 192},
{15, 6, 193, 256},
{16, 7, 257, 384},
{17, 7, 385, 512},
{18, 8, 513, 768},
{19, 8, 769, 1024},
{20, 9, 1025, 1536},
{21, 9, 1537, 2048},
};
static void deflate_literal(struct meshcore_deflate *out, unsigned char c)
{
if (c <= 143) {
/* 0 through 143 are 8 bits long starting at 00110000. */
outbits(out, mirrorbytes[0x30 + c], 8);
} else {
/* 144 through 255 are 9 bits long starting at 110010000. */
outbits(out, 1 + 2 * mirrorbytes[0x90 - 144 + c], 9);
}
}
static void deflate_match(struct meshcore_deflate *out, int distance, int len)
{
const dist_coderecord *d;
const len_coderecord *l;
int i, j, k;
int lcode;
while (len > 0) {
int thislen;
/*
* We can transmit matches of lengths 3 through 258
* inclusive. So if len exceeds 258, we must transmit in
* several steps, with 258 or less in each step.
*
* Specifically: if len >= 261, we can transmit 258 and be
* sure of having at least 3 left for the next step. And if
* len <= 258, we can just transmit len. But if len == 259
* or 260, we must transmit len-3.
*/
thislen = (len > 260 ? 258 : len <= 258 ? len : len - 3);
len -= thislen;
/*
* Binary-search to find which length code we're
* transmitting.
*/
i = -1;
j = sizeof(lencodes) / sizeof(*lencodes);
while (1) {
assert(j - i >= 2);
k = (j + i) / 2;
if (thislen < FROM_LCODE(lencodes[k].min))
j = k;
else if (thislen > FROM_LCODE(lencodes[k].max))
i = k;
else {
l = &lencodes[k];
break; /* found it! */
}
}
lcode = l - lencodes + 257;
/*
* Transmit the length code. 256-279 are seven bits
* starting at 0000000; 280-287 are eight bits starting at
* 11000000.
*/
if (lcode <= 279) {
outbits(out, mirrorbytes[(lcode - 256) * 2], 7);
} else {
outbits(out, mirrorbytes[0xc0 - 280 + lcode], 8);
}
/*
* Transmit the extra bits.
*/
if (l->extrabits)
outbits(out, thislen - FROM_LCODE(l->min), l->extrabits);
/*
* Binary-search to find which distance code we're
* transmitting.
*/
i = -1;
j = sizeof(distcodes) / sizeof(*distcodes);
while (1) {
assert(j - i >= 2);
k = (j + i) / 2;
if (distance < distcodes[k].min)
j = k;
else if (distance > distcodes[k].max)
i = k;
else {
d = &distcodes[k];
break; /* found it! */
}
}
/*
* Transmit the distance code. Five bits starting at 00000.
*/
outbits(out, mirrorbytes[d->code * 8], 5);
/*
* Transmit the extra bits.
*/
if (d->extrabits)
outbits(out, distance - d->min, d->extrabits);
}
}
static void deflate_start_block(struct meshcore_deflate *out)
{
outbits(out, 1, 1); /* Final block */
outbits(out, 1, 2); /* Static huffman block */
}
static void deflate_finish_block(struct meshcore_deflate *out)
{
outbits(out, 0, 7); /* close block */
outbits(out, 0, 7); /* Make sure all bits are flushed */
}
/*
* genlz77 - Generic LZ77 compressor
*
* Copyright (c) 2014 by Paul Sokolovsky
*
* This software is provided 'as-is', without any express
* or implied warranty. In no event will the authors be
* held liable for any damages arising from the use of
* this software.
*
* Permission is granted to anyone to use this software
* for any purpose, including commercial applications,
* and to alter it and redistribute it freely, subject to
* the following restrictions:
*
* 1. The origin of this software must not be
* misrepresented; you must not claim that you
* wrote the original software. If you use this
* software in a product, an acknowledgment in the
* product documentation would be appreciated
* but is not required.
*
* 2. Altered source versions must be plainly marked
* as such, and must not be misrepresented as being
* the original software.
*
* 3. This notice may not be removed or altered from
* any source distribution.
*/
/*
* Match finder derived from uzlib src/genlz77.c.
* MeshCore modification: offsets replace pointers, reset for every block.
* Compare only inside the current input, including safe 0..2-byte handling.
*/
#define DEFLATE_HASH_SIZE (1u << MESHCORE_OTA_DEFLATE_HASH_BITS)
static unsigned deflate_hash(const uint8_t *p) {
const uint32_t v = ((uint32_t)p[0] << 16) | ((uint32_t)p[1] << 8) | p[2];
return ((v >> (24 - MESHCORE_OTA_DEFLATE_HASH_BITS)) - v) & (DEFLATE_HASH_SIZE - 1u);
}
uint16_t meshcore_deflate_raw(const uint8_t *src, uint16_t src_len,
uint8_t *dst, uint16_t dst_cap) {
struct meshcore_deflate out = {0};
const uint8_t *base = src, *top;
if (!src || src_len < 3 || src_len > MESHCORE_OTA_DEFLATE_BLOCK_MAX ||
!dst || !dst_cap) return 0;
out.hash_table = (uint16_t *)calloc(DEFLATE_HASH_SIZE, sizeof(uint16_t));
if (!out.hash_table) return 0;
out.outbuf = dst;
out.outsize = dst_cap;
deflate_start_block(&out);
top = base + src_len - 3; /* length checked before any backwards pointer */
while (src < top && !out.failed) {
const unsigned hash = deflate_hash(src);
const unsigned offset = (unsigned)(src - base);
const uint16_t stored = out.hash_table[hash];
out.hash_table[hash] = (uint16_t)(offset + 1u);
if (stored && (unsigned)(stored - 1u) < offset &&
!memcmp(src, base + stored - 1u, 3)) {
const uint8_t *subs = base + stored - 1u;
const unsigned distance = (unsigned)(src - subs);
const uint8_t *match = subs + 3;
int len = 3;
src += 3;
while (src < top && len < 258 && *src == *match) {
++src; ++match; ++len;
}
deflate_match(&out, (int)distance, len);
} else {
deflate_literal(&out, *src++);
}
}
top = base + src_len;
while (src < top && !out.failed) deflate_literal(&out, *src++);
if (!out.failed) deflate_finish_block(&out);
free(out.hash_table);
return out.failed ? 0 : out.outlen;
}
src/helpers/ota/tinydeflate/deflate.h
+18
View File
@@ -0,0 +1,18 @@
#ifndef MESHCORE_TINY_DEFLATE_H
#define MESHCORE_TINY_DEFLATE_H
#include <stdint.h>
#ifdef __cplusplus
extern "C" {
#endif
// Modified uzlib compressor: bounded caller-owned raw RFC1951 output only.
// Returns zero on OOM/output overflow/invalid parameters. No decoder, container
// headers, checksum, persistent dictionary, or dynamic-Huffman tree is included.
uint16_t meshcore_deflate_raw(const uint8_t* src, uint16_t src_len,
uint8_t* dst, uint16_t dst_cap);
#ifdef __cplusplus
}
#endif
#endif
test/test_esp32_usb_serial_hygiene.py
+3 -1
View File
@@ -263,7 +263,9 @@ class Esp32UsbSerialHygieneTest(unittest.TestCase):
host_check = board[start : board.index("void setInhibitSleep", start)]
self.assertIn("ARDUINO_USB_MODE", host_check)
self.assertIn("return Serial.isPlugged();", host_check)
self.assertIn("host_connected = Serial.isPlugged();", host_check)
self.assertIn("usb_host_sleep_policy.observe(host_connected, millis());", host_check)
self.assertIn("return host_connected;", host_check)
def test_hwcdc_retries_tx_kick_after_transient_sof_loss(self):
logging = source("src/helpers/UsbLogging.cpp")
test/test_esp32_usb_sleep.py
+123 -2
View File
@@ -16,6 +16,13 @@ HARNESS = r'''
#include <stdexcept>
#include <iostream>
#include <array>
#include <helpers/UsbHostSleepPolicy.h>
#ifndef MESH_ESP32_USB_HOST_LOSS_SLEEP_GRACE_MS
#define MESH_ESP32_USB_HOST_LOSS_SLEEP_GRACE_MS 120000UL
#endif
static uint32_t mock_millis = 1000;
static uint32_t millis() { return mock_millis; }
static void require(bool ok, const char* message) {
if (!ok) throw std::runtime_error(message);
@@ -97,6 +104,9 @@ struct MainBoard {
};
struct ESP32Board : MainBoard {
bool inhibit_sleep = false;
#if MESH_ESP32_USB_CONSOLE_COOPERATIVE
mesh::UsbHostSleepPolicy usb_host_sleep_policy;
#endif
uint32_t irq = 48; // Station G3 LoRa DIO1
uint32_t getIRQGpio() { return irq; }
@METHODS@
@@ -133,9 +143,11 @@ int main() {
require(!board.isUsbHostConnected(), "UART build invented a USB host");
#endif
// USB power without an enumerated host must still permit battery saving.
// After the bounded host-reboot grace, disconnected USB power must still
// permit battery saving. A fresh never-seen host is tested separately.
attachHost(false);
Serial.terminal_open = false;
mock_millis += MESH_ESP32_USB_HOST_LOSS_SLEEP_GRACE_MS;
board.sleep(30);
require(sleep_calls == 1 && timer_us == 30000000ULL,
"disconnected board did not retain its scheduled sleep");
@@ -149,6 +161,7 @@ int main() {
board.sleep(30);
require(sleep_calls == 1, "reattached closed terminal failed to inhibit sleep");
attachHost(false);
mock_millis += MESH_ESP32_USB_HOST_LOSS_SLEEP_GRACE_MS;
#endif
board.inhibit_sleep = true;
@@ -245,9 +258,117 @@ class Esp32UsbSleepTest(unittest.TestCase):
compiler, "-std=c++17", "-Wall", "-Wextra",
f"-DARDUINO_USB_MODE={mode}",
f"-DARDUINO_USB_CDC_ON_BOOT={cdc}",
f"-DMESH_ESP32_USB_CONSOLE_COOPERATIVE={int(bool(cdc))}",
f"-DMOMENTARY_BUTTON_WAKE_FROM_SLEEP={button}",
f"-DMESH_USB_LOGGING_AVAILABLE={logging}", "-DPIN_USER_BTN=38",
"-DCONFIG_TINYUSB_ENABLED=1", str(cpp), "-o", str(binary),
"-DCONFIG_TINYUSB_ENABLED=1", "-I", str(ROOT / "src"),
str(cpp), "-o", str(binary),
], capture_output=True, text=True)
self.assertEqual(compiled.returncode, 0, compiled.stderr)
result = subprocess.run([str(binary)], capture_output=True, text=True)
self.assertEqual(result.returncode, 0, result.stderr)
def test_native_host_reboot_grace_without_logging(self):
methods = "\n".join(board_method(signature) for signature in (
"void sleep(uint32_t secs) override",
"bool isUsbDataConnected() override",
"bool isUsbHostConnected() override",
))
harness = HARNESS.split("int main()", 1)[0] + r'''
int main() {
try {
// Charger-only/battery startup must not create a host-reboot grace.
ESP32Board cold;
attachHost(false);
Serial.terminal_open = false;
mock_millis = 0;
require(!cold.isUsbHostConnected(), "cold boot invented a host");
cold.sleep(30);
require(sleep_calls == 1, "never-seen host disabled power saving");
mock_millis = 121000;
cold.sleep(30);
require(sleep_calls == 2, "charger-only boot later armed a grace");
ESP32Board board;
attachHost(true);
Serial.terminal_open = true;
mock_millis = UINT32_MAX - 60000U;
// Only the data getter runs while the terminal is open: the role loop can
// skip board.sleep(), but must still remember its actual USB host.
require(board.isUsbDataConnected() == bool(ARDUINO_USB_CDC_ON_BOOT),
"data getter changed its raw connection meaning");
attachHost(false);
Serial.terminal_open = false;
require(!board.isUsbHostConnected(), "grace falsely reported a present host");
const unsigned before = sleep_calls;
mock_millis += 6U; // Beyond HWCDC's five-millisecond SOF loss detector.
board.sleep(30);
#if MESH_ESP32_USB_CONSOLE_COOPERATIVE
require(sleep_calls == before, "SOF loss allowed immediate native USB sleep");
mock_millis += MESH_ESP32_USB_HOST_LOSS_SLEEP_GRACE_MS - 7U;
require(!board.isUsbDataConnected(), "closed data getter changed under grace");
board.sleep(30);
require(sleep_calls == before, "native grace ended early across millis wrap");
++mock_millis;
board.sleep(30);
require(sleep_calls == before + 1, "expired native grace never restored sleep");
// Disconnected polls must not rearm the window, even after a full millis
// cycle appears to bring the old timestamp near again.
mock_millis = UINT32_MAX - 59999U;
board.sleep(30);
require(sleep_calls == before + 2, "expired host revived after clock wrap");
// A returning closed terminal is a real host and refreshes the grace.
attachHost(true);
mock_millis = 100;
require(board.isUsbHostConnected(), "returning host was not observed");
attachHost(false);
mock_millis += MESH_ESP32_USB_HOST_LOSS_SLEEP_GRACE_MS - 1U;
board.sleep(30);
require(sleep_calls == before + 2, "reconnect did not renew native grace");
++mock_millis;
board.sleep(30);
require(sleep_calls == before + 3, "renewed native grace never expired");
// Observation must also continue while inhibit_sleep short-circuits the
// old expression, or a long OTA can miss its last connected host entirely.
ESP32Board inhibited;
inhibited.inhibit_sleep = true;
attachHost(true);
mock_millis = 200000;
inhibited.sleep(30);
inhibited.inhibit_sleep = false;
attachHost(false);
mock_millis += 6U;
const unsigned after = sleep_calls;
inhibited.sleep(30);
require(sleep_calls == after, "OTA inhibitor hid the host observation");
#else
require(sleep_calls == before + 1, "non-native board gained a USB grace");
#endif
} catch (const std::exception& e) {
std::cerr << e.what() << '\n';
return 1;
}
}
'''
self.assertIn("mesh::UsbHostSleepPolicy usb_host_sleep_policy;",
(ROOT / "src/helpers/ESP32Board.h").read_text())
with tempfile.TemporaryDirectory(prefix="meshcore-usb-reboot-") as temp:
cpp = Path(temp) / "reboot.cpp"
cpp.write_text(harness.replace("@METHODS@", methods))
for mode, cdc, grace in ((0, 1, 120000), (1, 1, 120000),
(1, 1, 1000), (1, 0, 120000)):
with self.subTest(usb_mode=mode, cdc=cdc, grace=grace):
binary = Path(temp) / f"reboot-{mode}-{cdc}-{grace}"
compiled = subprocess.run([
os.environ.get("CXX", "c++"), "-std=c++17", "-Wall", "-Wextra",
f"-DARDUINO_USB_MODE={mode}", f"-DARDUINO_USB_CDC_ON_BOOT={cdc}",
f"-DMESH_ESP32_USB_CONSOLE_COOPERATIVE={int(bool(cdc))}",
f"-DMESH_ESP32_USB_HOST_LOSS_SLEEP_GRACE_MS={grace}U",
"-DMOMENTARY_BUTTON_WAKE_FROM_SLEEP=0", "-DMESH_USB_LOGGING_AVAILABLE=0",
"-DCONFIG_TINYUSB_ENABLED=1", "-I", str(ROOT / "src"),
str(cpp), "-o", str(binary),
], capture_output=True, text=True)
self.assertEqual(compiled.returncode, 0, compiled.stderr)
result = subprocess.run([str(binary)], capture_output=True, text=True)
test/test_firmware_ram.py
+46
View File
@@ -66,6 +66,52 @@ def esp_fixture(path, modern=False, fragmented=False):
class FirmwareRamTest(unittest.TestCase):
def test_device_encoder_budget_matches_storage_and_role_gates(self):
matrix = [
("ESP32_PLATFORM", {"ENABLE_OTA": 1, "MESHCORE_OTA_DEVICE_DEFLATE": 1}, "v4_repeater", True),
("ESP32_PLATFORM", {"ENABLE_OTA": 1, "MESHCORE_OTA_DEVICE_DEFLATE": 1}, "v4_room_server", True),
("ESP32_PLATFORM", {"ENABLE_OTA": 1, "MESHCORE_OTA_DEVICE_DEFLATE": 1}, "v4_sensor", True),
("ESP32_PLATFORM", {"ENABLE_OTA": 1, "MESHCORE_OTA_DEVICE_DEFLATE": 1}, "v4_companion_alias", True),
("ESP32_PLATFORM", {"ENABLE_OTA": 1}, "v4_repeater", False),
("ESP32_PLATFORM", {"ENABLE_OTA": 1}, "v4_companion_radio_usb", False),
("ESP32_PLATFORM", {"ENABLE_OTA": 1, "OTA_SEEDER_ONLY": 1}, "v4_repeater", False),
("ESP32_PLATFORM", {"ENABLE_OTA": 1, "MESHCORE_OTA_DEVICE_DEFLATE": 0}, "v4_repeater", False),
("ESP32_PLATFORM", {}, "v4_repeater", False),
("NRF52_PLATFORM", {"ENABLE_OTA": 1, "OTA_FLASH_STORE": 1}, "rak3401_repeater", False),
("NRF52_PLATFORM", {"ENABLE_OTA": 1, "OTA_QSPI_STORE": 1, "MESHCORE_OTA_DEVICE_DEFLATE": 1}, "xiao_repeater", True),
("NRF52_PLATFORM", {"ENABLE_OTA": 1, "OTA_SD_STORE": 1, "MESHCORE_OTA_DEVICE_DEFLATE": 1}, "tower_room_server", True),
("NRF52_PLATFORM", {"ENABLE_OTA": 1, "OTA_RAK_AUTO_STORE": 1, "MESHCORE_OTA_DEVICE_DEFLATE": 1}, "rak4631_sensor", True),
("NRF52_PLATFORM", {"ENABLE_OTA": 1, "OTA_QSPI_STORE": 1}, "xiao_companion_radio_full", False),
("STM32_PLATFORM", {"ENABLE_OTA": 1, "OTA_SD_STORE": 1}, "wio_repeater", False),
]
for platform, defines, target, enabled in matrix:
with self.subTest(platform=platform, defines=defines, target=target):
components = ram.requirements(platform, defines, target)["components"]
self.assertEqual("ota_encoder_workspace" in components, enabled)
if enabled:
self.assertEqual(components["ota_encoder_workspace"], 1024 + 16)
self.assertEqual("ota_self_source_scratch" in components,
enabled and platform == "NRF52_PLATFORM")
if enabled and platform == "NRF52_PLATFORM":
self.assertEqual(components["ota_self_source_scratch"], 4096 + 32)
def test_device_encoder_hash_override_is_budgeted_and_bounded(self):
for bits in (7, 8, 9, 10):
with self.subTest(bits=bits):
policy = ram.requirements("NRF52_PLATFORM", {
"ENABLE_OTA": 1, "OTA_QSPI_STORE": 1,
"MESHCORE_OTA_DEVICE_DEFLATE": 1,
"MESHCORE_OTA_DEFLATE_HASH_BITS": bits,
}, "xiao_repeater")
self.assertEqual(policy["components"]["ota_encoder_workspace"],
(1 << bits) * 2 + 16)
for bits in (6, 11):
with self.subTest(bits=bits), self.assertRaisesRegex(ValueError, "DEFLATE hash"):
ram.requirements("ESP32_PLATFORM", {
"ENABLE_OTA": 1, "MESHCORE_OTA_DEFLATE_HASH_BITS": bits,
"MESHCORE_OTA_DEVICE_DEFLATE": 1,
}, "v4_repeater")
def test_sh1107_framebuffer_is_budgeted(self):
policy = ram.requirements("NRF52_PLATFORM", {
"DISPLAY_CLASS": "SH1107Display",
test/test_hil_esp32_self_serve_ota.py
+915
View File
@@ -0,0 +1,915 @@
#!/usr/bin/env python3
"""Offline serial-framing and safety tests for the two-V4 own-flash harness."""
import contextlib
import copy
import importlib.util
import io
import json
from pathlib import Path
from types import SimpleNamespace
import sys
import tempfile
import unittest
from unittest import mock
ROOT = Path(__file__).resolve().parents[1]
HIL_DIR = ROOT / "tools" / "hil"
def load_module(name, path):
spec = importlib.util.spec_from_file_location(name, path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
SOAK = load_module("self_serve_test_soak", HIL_DIR / "s3_memory_soak.py")
with mock.patch.dict(sys.modules, {"s3_memory_soak": SOAK}):
HIL = load_module("self_serve_test_harness", HIL_DIR / "esp32_self_serve_ota.py")
class FakeClock:
def __init__(self):
self.now = 0.0
def monotonic(self):
return self.now
def sleep(self, seconds):
self.now += seconds
class ScriptedSerial:
"""Deliver timed USB chunks; writes are captured, never sent to hardware."""
def __init__(self, clock, response):
self.clock = clock
self.response = response
self.pending = []
self.writes = []
self.closed = False
@property
def in_waiting(self):
if self.pending and self.pending[0][0] <= self.clock.now:
return len(self.pending[0][1])
return 0
def read(self, size):
if not self.in_waiting:
self.clock.sleep(.05)
if self.in_waiting:
due, data = self.pending[0]
chunk, remaining = data[:size], data[size:]
if remaining:
self.pending[0] = (due, remaining)
else:
self.pending.pop(0)
return chunk
return b""
def write(self, data):
self.writes.append(data)
self.pending.extend((self.clock.now + delay, chunk)
for delay, chunk in self.response)
return len(data)
def flush(self):
pass
def close(self):
self.closed = True
class FramingTests(unittest.TestCase):
def setUp(self):
self.clock = FakeClock()
self.stack = contextlib.ExitStack()
self.addCleanup(self.stack.close)
self.stack.enter_context(mock.patch.object(HIL.time, "monotonic", self.clock.monotonic))
self.stack.enter_context(mock.patch.object(HIL.time, "sleep", self.clock.sleep))
def device(self, response):
device = HIL.V4Device("A", "441BF669CF98")
stream = ScriptedSerial(self.clock, response)
device.stream = stream
return device, stream
def test_late_setter_ack_uses_full_deadline_and_one_write(self):
device, stream = self.device([(8, b"set radio2 x\r\n -> OK - radio2\r\n")])
self.assertEqual(device.checked("set radio2 909.5,500,6,5,rx", timeout=30), "OK - radio2")
self.assertEqual(stream.writes, [b"set radio2 909.5,500,6,5,rx\r"])
self.assertLess(self.clock.now, 30)
def test_missing_marker_closes_transport_without_retry_or_secret_in_error(self):
device, stream = self.device([])
secret = "AB" * 64
with self.assertRaises(HIL.TestFailure) as raised:
device.checked("set prv.key " + secret, timeout=30)
self.assertEqual(len(stream.writes), 1)
self.assertTrue(stream.closed)
self.assertIsNone(device.stream)
self.assertIn("set prv.key", str(raised.exception))
self.assertNotIn(secret, str(raised.exception))
self.assertLessEqual(self.clock.now, 30.2)
def test_split_marker_and_key_packets_are_joined_without_logging_secret(self):
secret = "AB" * 64
device, stream = self.device([
(.1, b"get prv.key\r\n -"),
(.2, b"> > " + secret[:64].encode()),
(.3, secret[64:].encode() + b"\r\n"),
])
self.assertEqual(HIL.key_token(device.command("get prv.key"), 128), secret)
self.assertEqual(stream.writes, [b"get prv.key\r"])
def test_embedded_debug_arrow_does_not_end_wait_before_eight_second_ack(self):
device, stream = self.device([
(.1, b"debug packet [01 -> 02]\r\n"),
(8, b"set radio2 x\r\n -> OK - radio2\r\n"),
])
self.assertEqual(device.checked("set radio2 909.5,500,6,5,rx", timeout=30), "OK - radio2")
self.assertEqual(len(stream.writes), 1)
self.assertLess(self.clock.now, 30)
def test_marker_at_100ms_and_complete_reply_at_800ms_do_not_resend(self):
device, stream = self.device([
(.1, b"set radio2 x\r\n -> "),
(.8, b"OK - radio2\r\n"),
])
self.assertEqual(device.checked("set radio2 909.5,500,6,5,rx", timeout=30), "OK - radio2")
self.assertEqual(len(stream.writes), 1)
self.assertGreaterEqual(self.clock.now, .8)
def test_empty_or_unterminated_reply_fails_boundedly_without_resend_or_raw_reply_leak(self):
secret = "AB" * 64
for tail in (b"", b"\r\n", secret.encode()):
with self.subTest(empty=not tail, terminated=tail.endswith(b"\r\n")):
started = self.clock.now
device, stream = self.device([(.1, b"get prv.key\r\n -> " + tail)])
close = mock.Mock(wraps=device.close)
device.close = close
with self.assertRaisesRegex(HIL.TestFailure, "CLI reply was incomplete for get prv.key") as raised:
device.command("get prv.key", timeout=2)
self.assertNotIn(secret, str(raised.exception))
self.assertEqual(len(stream.writes), 1)
close.assert_called_once_with()
self.assertTrue(stream.closed)
self.assertIsNone(device.stream)
self.assertLessEqual(self.clock.now - started, 2.2)
class PreparationTests(unittest.TestCase):
def devices(self):
return [SimpleNamespace(name=name, checked=mock.Mock(return_value="OK"),
reboot=mock.Mock()) for name in ("A", "B")]
def test_every_prepare_mutation_uses_30s_and_diagnostic_labels_only(self):
devices = self.devices()
report = SimpleNamespace(event=mock.Mock())
keys = ["AB" * 32, "CD" * 32]
with mock.patch.object(HIL.time, "sleep") as sleep:
HIL.prepare(devices, keys, "0123456789ABCDEF", report)
for device in devices:
self.assertEqual(device.checked.call_count, 10)
for call in device.checked.call_args_list:
self.assertEqual(call.kwargs, {"timeout": 30})
device.reboot.assert_called_once_with("0123456789ABCDEF")
sleep.assert_called_once_with(6)
events = "\n".join(call.args[0] for call in report.event.call_args_list)
for key in keys:
self.assertNotIn(key, events)
self.assertNotIn(HIL.LAB_RADIO, events)
self.assertIn("Preparing A: set radio2", events)
def test_timed_out_mutation_is_never_retried_or_followed_by_reboot(self):
devices = self.devices()
devices[0].checked.side_effect = ["OK", "OK", "OK", HIL.TestFailure("marker missing")]
report = SimpleNamespace(event=mock.Mock())
with mock.patch.object(HIL.time, "sleep") as sleep:
with self.assertRaisesRegex(HIL.TestFailure, "marker missing"):
HIL.prepare(devices, ["AB" * 32, "CD" * 32], "0123456789ABCDEF", report)
self.assertEqual(devices[0].checked.call_count, 4)
self.assertEqual(devices[0].checked.call_args.args,
("set radio2 " + HIL.LAB_SECONDARY,))
devices[1].checked.assert_not_called()
for device in devices:
device.reboot.assert_not_called()
sleep.assert_not_called()
def test_command_labels_exclude_setting_values_and_ota_ids(self):
cases = {
"set prv.key " + "AB" * 64: "set prv.key",
"setperm " + "CD" * 32 + " 1": "setperm",
"set name secret-node-name": "set name",
"ota get DEADBEEF flash": "ota get",
"get acl 128": "get acl",
"tempradio 909.5,500,5,5,120": "tempradio",
}
for command, label in cases.items():
with self.subTest(label=label):
self.assertEqual(HIL.command_label(command), label)
class LabActivationTests(unittest.TestCase):
PRIMARY = "909.500,500.000,5,5,32; 0d1h59m (119 min) left"
SECONDARY = "909.500,500.000,5,5,rxtx,32 (auto); timing self-test pending"
POLICY = "autofetch=off autoinstall=off hops=0"
def setUp(self):
self.clock = FakeClock()
self.stack = contextlib.ExitStack()
self.addCleanup(self.stack.close)
self.sleep = self.stack.enter_context(
mock.patch.object(HIL.time, "sleep", side_effect=self.clock.sleep))
self.stack.enter_context(mock.patch.object(HIL.time, "monotonic", self.clock.monotonic))
def device(self, name, ready_at=0, secondary=None, policy=None, getter_delays=None):
def checked(command, timeout=5):
if command == "ota config":
return self.POLICY if policy is None else policy
return "OK"
def command(command, timeout=5):
self.clock.sleep((getter_delays or {}).get(command, 0))
if self.clock.now < ready_at:
text = "off" if command == "get tempradio" else "909.500,500.000,6,5,rx,32"
elif command == "get tempradio":
text = self.PRIMARY
else:
text = self.SECONDARY if secondary is None else secondary
return "\r\n -> > " + text + "\r\n"
return SimpleNamespace(name=name, checked=mock.Mock(side_effect=checked),
command=mock.Mock(side_effect=command))
def test_lab_tuple_is_strict_about_frequency_bandwidth_sf_cr_and_secondary_mode(self):
self.assertTrue(HIL.lab_profile_matches(self.PRIMARY))
self.assertTrue(HIL.lab_profile_matches(self.SECONDARY, secondary=True))
for text in ("off", "910.525,500.000,5,5,32", "909.500,250.000,5,5,32",
"909.500,500.000,6,5,32", "909.500,500.000,5,7,32"):
with self.subTest(text=text):
self.assertFalse(HIL.lab_profile_matches(text))
with self.assertRaises(HIL.TestFailure):
HIL.require_lab_profile(text)
for text in (self.PRIMARY, "909.500,500.000,5,5,rx,32",
"909.500,500.000,6,5,rxtx,32"):
with self.subTest(secondary=text):
self.assertFalse(HIL.lab_profile_matches(text, secondary=True))
def test_arm_writes_each_mutation_once_with_30s_before_live_verification(self):
devices = [self.device("A"), self.device("B")]
HIL.arm_lab(devices)
expected = ["ota config autofetch off", "ota config autoinstall off", "ota config hops 0",
"set tempradio2 909.5,500,5,5,rxtx,120", "tempradio 909.5,500,5,5,120"]
for device in devices:
writes = [call for call in device.checked.call_args_list if call.args[0] != "ota config"]
self.assertEqual([call.args[0] for call in writes], expected)
self.assertTrue(all(call.kwargs == {"timeout": 30} for call in writes))
self.assertEqual([call.args[0] for call in device.command.call_args_list],
["get tempradio", "get tempradio2"])
device.checked.assert_any_call("ota config")
self.sleep.assert_not_called()
def test_arm_polls_delayed_primary_and_secondary_activation_without_reapplying(self):
devices = [self.device("A", ready_at=2), self.device("B", ready_at=4)]
HIL.arm_lab(devices)
self.assertEqual(self.clock.now, 4)
self.assertEqual(self.sleep.call_count, 8)
for device in devices:
self.assertGreater(device.command.call_count, 2)
self.assertEqual(device.checked.call_count, 6)
writes = [call.args[0] for call in device.checked.call_args_list
if call.args[0] != "ota config"]
self.assertEqual(len(writes), len(set(writes)))
def test_arm_never_accepts_saved_sf6_rx_as_live_sf5_rxtx_and_times_out(self):
devices = [self.device("A", secondary="909.500,500.000,6,5,rx,32"), self.device("B")]
with self.assertRaisesRegex(HIL.TestFailure, "A: temporary lab profiles were not activated"):
HIL.arm_lab(devices)
self.assertEqual(self.clock.now, 30)
self.assertEqual(devices[0].checked.call_count, 5)
devices[1].command.assert_not_called()
for device in devices:
self.assertNotIn("ota config", [call.args[0] for call in device.checked.call_args_list])
self.assertEqual(device.checked.call_count, 5)
def test_arm_rejects_auto_policy_even_when_both_live_profiles_match(self):
devices = [self.device("A", policy="autofetch=on autoinstall=off hops=0"), self.device("B")]
with self.assertRaisesRegex(HIL.TestFailure, "Manual-only one-hop OTA policy"):
HIL.arm_lab(devices)
devices[1].command.assert_not_called()
self.sleep.assert_not_called()
def test_activation_getter_can_take_eight_seconds_without_short_read_timeout(self):
device = self.device("A", getter_delays={"get tempradio": 8})
HIL.arm_lab([device])
self.assertEqual(self.clock.now, 8)
self.assertEqual(device.command.call_args_list,
[mock.call("get tempradio", timeout=30), mock.call("get tempradio2", timeout=22)])
self.assertEqual(device.checked.call_count, 6)
def test_each_board_has_its_own_30_second_activation_deadline(self):
devices = [self.device(name, getter_delays={"get tempradio": 20}) for name in ("A", "B")]
HIL.arm_lab(devices)
self.assertEqual(self.clock.now, 40)
for device in devices:
self.assertEqual(device.command.call_args_list,
[mock.call("get tempradio", timeout=30), mock.call("get tempradio2", timeout=10)])
def test_matching_profiles_received_at_expired_deadline_are_not_accepted(self):
device = self.device("A", getter_delays={"get tempradio": 8, "get tempradio2": 22})
with self.assertRaisesRegex(HIL.TestFailure, "not activated in time"):
HIL.arm_lab([device])
self.assertEqual(self.clock.now, 30)
self.assertEqual(device.checked.call_count, 5)
self.assertNotIn("ota config", [call.args[0] for call in device.checked.call_args_list])
class NormalModeTests(unittest.TestCase):
def setUp(self):
self.clock = FakeClock()
self.trace = []
self.stack = contextlib.ExitStack()
self.addCleanup(self.stack.close)
self.stack.enter_context(mock.patch.object(HIL.time, "monotonic", self.clock.monotonic))
self.stack.enter_context(mock.patch.object(HIL.time, "sleep", self.clock.sleep))
def device(self, name, active=True, ready_at=0, ota_active=False):
state = SimpleNamespace(primary_off=not active, secondary_off=not active)
def command(command, timeout=5):
self.trace.append((name, "read", command))
if command == "ota status":
text = "OTA | fw lab | download: pulling 1/3 | serving:1" if ota_active else \
"OTA | fw lab | no download | serving:1"
elif command == "get tempradio":
text = "off" if state.primary_off and self.clock.now >= ready_at else LabActivationTests.PRIMARY
elif command == "get tempradio2":
text = "off" if state.secondary_off and self.clock.now >= ready_at else LabActivationTests.SECONDARY
else:
raise AssertionError("Unexpected normal-mode getter: " + command)
return "\r\n -> > " + text + "\r\n"
def checked(command, timeout=5):
self.trace.append((name, "write", command))
if command == "normalradio":
state.primary_off = True
elif command == "set tempradio2 off":
state.secondary_off = True
else:
raise AssertionError("Unexpected normal-mode mutation: " + command)
return "OK"
return SimpleNamespace(name=name, serial_number=ContinuationTests.SERIALS[ord(name)-ord("A")],
state=state, command=mock.Mock(side_effect=command),
checked=mock.Mock(side_effect=checked), close=mock.Mock())
def test_disarm_checks_both_receivers_before_any_write_and_never_resends(self):
devices = [self.device("A", ready_at=1), self.device("B", ready_at=2)]
HIL.disarm_lab(devices)
first_write = next(n for n, row in enumerate(self.trace) if row[1] == "write")
self.assertEqual(self.trace[:first_write], [("A", "read", "ota status"),
("B", "read", "ota status")])
for device in devices:
self.assertEqual(device.checked.call_args_list,
[mock.call("normalradio", timeout=30), mock.call("set tempradio2 off", timeout=30)])
self.assertGreater(device.command.call_count, 3)
self.assertEqual(self.clock.now, 2)
def test_active_download_on_either_peer_prevents_every_disarm_write(self):
for active in ("A", "B"):
with self.subTest(active=active):
devices = [self.device(name, ota_active=name == active) for name in ("A", "B")]
with self.assertRaisesRegex(HIL.TestFailure, "cannot disarm a live OTA download"):
HIL.disarm_lab(devices)
for device in devices:
device.checked.assert_not_called()
def test_disarm_activation_timeout_fails_closed_without_repeated_mutations(self):
devices = [self.device("A", ready_at=100), self.device("B")]
with self.assertRaisesRegex(HIL.TestFailure, "baseline deadline"):
HIL.disarm_lab(devices)
self.assertEqual(self.clock.now, 30)
for device in devices:
self.assertEqual(device.checked.call_count, 2)
for call in devices[0].command.call_args_list:
if call.args[0] != "ota status":
self.assertTrue(0 < call.kwargs["timeout"] <= 5)
def test_normal_profile_validation_is_read_only_and_rejects_either_overlay(self):
for primary, secondary in ((True, False), (False, True)):
with self.subTest(primary=primary, secondary=secondary):
device = self.device("A", active=False)
device.state.primary_off = not primary
device.state.secondary_off = not secondary
with self.assertRaisesRegex(HIL.TestFailure, "inactive temporary profiles"):
HIL.require_normal_profiles([device])
device.checked.assert_not_called()
devices = [self.device("A", active=False), self.device("B", active=False)]
HIL.require_normal_profiles(devices)
for device in devices:
device.checked.assert_not_called()
def run_main(self, action="run", start_source="b", rounds=2, active=True):
devices = [self.device("A", active=active), self.device("B", active=active)]
report = SimpleNamespace(data={"rounds": []}, save=mock.Mock(), event=mock.Mock())
apps = {"A": 0, "B": 1}
directions = []
def snapshot(device):
self.assertTrue(device.state.primary_off and device.state.secondary_off)
return {"private_key_sha256": device.name * 64, "settings": {"advert.interval": "0"}}
def finish_round(number, source, receiver, peers, expected_hash, baseline, args, output, continuing):
self.assertIsNone(continuing)
directions.append((source.name, receiver.name))
output.data["rounds"].append({
"source_slot": {"inactive_address": 0x10000 if apps[source.name] else 0x310000},
"receiver_slot_before": {"inactive_address": 0x10000 if apps[receiver.name] else 0x310000},
})
apps[receiver.name] = 1 - apps[receiver.name]
with mock.patch.object(HIL, "Report", return_value=report), \
mock.patch.object(HIL, "V4Device", side_effect=devices), \
mock.patch.object(HIL, "preflight", return_value=[]), \
mock.patch.object(HIL, "prepare") as prepare, \
mock.patch.object(HIL, "snapshot", side_effect=snapshot) as snapshots, \
mock.patch.object(HIL, "run_round", side_effect=finish_round) as run:
result = HIL.main([action, "--serial-a", ContinuationTests.SERIALS[0],
"--serial-b", ContinuationTests.SERIALS[1],
"--expected-body-hash16", ContinuationTests.HASH,
"--rounds", str(rounds), "--start-source", start_source,
"--report", "unused.json"])
prepare.assert_not_called()
return result, report, devices, directions, snapshots, run
def test_fresh_run_takes_normal_advert_baseline_then_starts_from_b(self):
code, report, devices, directions, snapshots, _run = self.run_main()
self.assertEqual(code, 0)
self.assertEqual(directions, [("B", "A"), ("A", "B")])
self.assertEqual(report.data["baseline_mode"], HIL.BASELINE_MODE)
self.assertEqual(snapshots.call_count, 2)
self.assertTrue(all(value["settings"]["advert.interval"] == "0"
for value in report.data["baseline"].values()))
for device in devices:
self.assertEqual(device.checked.call_count, 2)
# Both new sources run appB; original A-appA transfer is needed to
# complete source coverage across the separate real-hardware reports.
self.assertFalse(report.data["both_source_slots_covered"])
self.assertTrue(report.data["both_destination_slots_covered"])
source_slots = {row["source_slot"]["inactive_address"] for row in report.data["rounds"]}
self.assertEqual(len(source_slots | {0x310000}), 2)
def test_three_rounds_starting_b_measure_both_source_and_destination_slots(self):
code, report, _devices, directions, _snapshots, _run = self.run_main(rounds=3)
self.assertEqual(code, 0)
self.assertEqual(directions, [("B", "A"), ("A", "B"), ("B", "A")])
self.assertTrue(report.data["both_source_slots_covered"])
self.assertTrue(report.data["both_destination_slots_covered"])
def test_probe_never_disarms_and_fails_before_snapshot_if_temp_is_active(self):
code, report, devices, _directions, snapshots, run = self.run_main(action="probe", active=True)
self.assertEqual(code, 1)
self.assertIn("inactive temporary profiles", report.data["failure"])
snapshots.assert_not_called()
run.assert_not_called()
for device in devices:
device.checked.assert_not_called()
def test_probe_accepts_normal_profile_getters_without_any_mutation(self):
code, report, devices, _directions, snapshots, run = self.run_main(action="probe", active=False)
self.assertEqual(code, 0)
self.assertEqual(report.data["baseline_mode"], HIL.BASELINE_MODE)
self.assertEqual(snapshots.call_count, 2)
run.assert_not_called()
for device in devices:
device.checked.assert_not_called()
class ContinuationTests(unittest.TestCase):
HASH = "0123456789ABCDEF"
MID = "DEADBEEF"
SERIALS = ("441BF669CF98", "441BF669C9C0")
@staticmethod
def raw(text):
return "\r\n -> > " + text + "\r\n"
def fixture(self):
state = SimpleNamespace(
clock=FakeClock(), downtime=0,
receiver_reconnected=False,
info={"body_bytes": 4000, "image_bytes": 4112, "body_hash16": self.HASH},
source_slot={"inactive_address": 0x310000, "size_bytes": 0x300000},
receiver_slot={"inactive_address": 0x310000, "size_bytes": 0x300000},
primary=LabActivationTests.PRIMARY, secondary=LabActivationTests.SECONDARY,
serve_status="default:on serving:on", serve_mid=self.MID, stats_suffix="",
statuses=[("pulling", 1, 3, self.MID), ("pulling", 2, 3, self.MID),
("ready to install", 3, 3, self.MID)],
baseline={
"A": {"private_key_sha256": "aa", "acl": [], "settings": {
"name": "SelfOTA_A", "advert.interval": "0", "flood.advert.interval": "0"}},
"B": {"private_key_sha256": "bb", "acl": [], "settings": {
"name": "SelfOTA_B", "advert.interval": "0", "flood.advert.interval": "0"}},
},
)
state.entry = {
"round": 1, "source": "A", "receiver": "B", "result": "RUNNING",
"source_slot": copy.deepcopy(state.source_slot),
"receiver_slot_before": copy.deepcopy(state.receiver_slot),
"source_self": copy.deepcopy(state.info), "manifest_id": self.MID,
"progress": [{"done": 1, "total": 3, "elapsed_seconds": 10,
"utc": "2001-09-09T01:46:40Z"}],
}
state.after_snapshot = copy.deepcopy(state.baseline["B"])
state.snapshot_calls = []
def slot_reply(slot):
return self.raw("inactive slot addr=0x%x size=%u" %
(slot["inactive_address"], slot["size_bytes"]))
def source_command(command):
if command == "ota self":
return self.raw("self body=%u image=%u base_hash=%s" %
(state.info["body_bytes"], state.info["image_bytes"],
state.info["body_hash16"]))
if command == "ota dev apply slot":
return slot_reply(state.source_slot)
if command == "get tempradio":
return self.raw(state.primary)
if command == "get tempradio2":
return self.raw(state.secondary)
raise AssertionError("Unexpected source command: " + command)
def source_checked(command, timeout=5):
if command == "ota serve status":
return state.serve_status
if command == "ota stats":
return "OTA | fw lab id=" + state.serve_mid + state.stats_suffix
raise AssertionError("Continuation must not mutate source: " + command)
state.receiver_slot_reads = 0
def receiver_command(command):
if command == "ota dev apply slot":
state.receiver_slot_reads += 1
return slot_reply(state.receiver_slot if state.receiver_slot_reads == 1 else
dict(state.receiver_slot, inactive_address=0x10000))
if command == "get tempradio":
return self.raw("off" if state.receiver_reconnected else state.primary)
if command == "get tempradio2":
return self.raw("off" if state.receiver_reconnected else state.secondary)
if command == "ota status":
status, done, total, mid = state.statuses.pop(0)
return self.raw("download: %s %u/%u pace=1.0x id=%s" % (status, done, total, mid))
raise AssertionError("Unexpected receiver command: " + command)
def receiver_checked(command, timeout=5):
if command != "ota install":
raise AssertionError("Continuation must not restart receiver: " + command)
self.assertIn("install_requested_utc", state.entry)
self.assertIn("install_requested_utc", state.saved_entries[-1])
return "OK installing"
state.source = SimpleNamespace(name="A", serial_number=self.SERIALS[0],
command=mock.Mock(side_effect=source_command),
checked=mock.Mock(side_effect=source_checked), close=mock.Mock())
state.receiver = SimpleNamespace(name="B", serial_number=self.SERIALS[1],
command=mock.Mock(side_effect=receiver_command),
checked=mock.Mock(side_effect=receiver_checked), close=mock.Mock(),
reconnect=mock.Mock())
def reconnect(expected_hash):
state.receiver_reconnected = True
return copy.deepcopy(state.info)
state.receiver.reconnect.side_effect = reconnect
state.saved_entries = []
state.report = SimpleNamespace(
data={"rounds": [state.entry]}, event=mock.Mock(),
save=mock.Mock(side_effect=lambda: state.saved_entries.append(copy.deepcopy(state.entry))))
state.args = SimpleNamespace(timeout=10, stall_timeout=5, poll_interval=1)
return state
def run_continuation(self, state):
def snapshot(receiver):
self.assertIs(receiver, state.receiver)
self.assertTrue(state.receiver_reconnected)
self.assertEqual([call.args[0] for call in receiver.command.call_args_list[-2:]],
["get tempradio", "get tempradio2"])
state.snapshot_calls.append(receiver.name)
return state.after_snapshot
with mock.patch.object(HIL.time, "monotonic", state.clock.monotonic), \
mock.patch.object(HIL.time, "sleep", state.clock.sleep), \
mock.patch.object(HIL.time, "time", return_value=1000000000 + state.downtime), \
mock.patch.object(HIL, "arm_lab") as arm, \
mock.patch.object(HIL, "snapshot", side_effect=snapshot):
HIL.run_round(1, state.source, state.receiver, [state.source, state.receiver], self.HASH,
state.baseline, state.args, state.report, continuing=state.entry)
arm.assert_not_called()
def test_valid_continuation_preserves_progress_and_never_restarts_rf_session(self):
state = self.fixture()
self.run_continuation(state)
self.assertEqual(state.entry["result"], "PASS")
self.assertTrue(state.entry["preservation"]["equal"])
self.assertTrue(state.entry["intermediate_rf_progress_observed"])
self.assertEqual([p["done"] for p in state.entry["progress"]], [1, 2, 3])
self.assertEqual(state.entry["download_seconds"], 11)
self.assertIn("install_requested_utc", state.entry)
self.assertEqual([call.args[0] for call in state.source.checked.call_args_list],
["ota serve status", "ota stats"])
state.receiver.checked.assert_called_once_with("ota install", timeout=30)
state.receiver.reconnect.assert_called_once_with(self.HASH)
self.assertEqual(len(state.report.data["rounds"]), 1)
def test_mismatched_live_continuation_fails_before_install(self):
def mutate(state, case):
if case == "hash":
state.info["body_hash16"] = "FFFFFFFFFFFFFFFF"
elif case == "source slot":
state.source_slot["inactive_address"] = 0x10000
elif case == "receiver slot":
state.receiver_slot["inactive_address"] = 0x10000
elif case == "source geometry":
state.info["body_bytes"] += 1
elif case == "direction":
state.entry["receiver"] = "A"
elif case == "completed":
state.entry["result"] = "PASS"
elif case == "installing":
state.entry["install_reply"] = "OK"
elif case == "install requested":
state.entry["install_requested_utc"] = "2001-09-09T01:46:40Z"
elif case == "invalid manifest":
state.entry["manifest_id"] = "not-a-mid"
elif case == "serving manifest":
state.serve_mid = "AAAAAAAA"
elif case == "receive id cannot prove serving id":
state.serve_mid = "AAAAAAAA"
state.stats_suffix = " | download id=" + self.MID
elif case == "not serving":
state.serve_status = "default:on serving:off"
elif case == "receive manifest":
state.statuses[0] = ("pulling", 1, 3, "AAAAAAAA")
elif case == "receive geometry":
state.statuses[0] = ("pulling", 1, 4, self.MID)
elif case == "lost progress":
state.statuses[0] = ("pulling", 0, 3, self.MID)
elif case == "primary profile":
state.primary = "910.525,500.000,5,5,32"
elif case == "secondary profile":
state.secondary = "909.500,500.000,6,5,rx,32"
for case in ("hash", "source slot", "receiver slot", "source geometry", "direction",
"completed", "installing", "install requested", "invalid manifest", "serving manifest",
"receive id cannot prove serving id", "not serving",
"receive manifest", "receive geometry", "lost progress", "primary profile",
"secondary profile"):
with self.subTest(case=case):
state = self.fixture()
mutate(state, case)
with self.assertRaises(HIL.TestFailure):
self.run_continuation(state)
state.receiver.checked.assert_not_called()
state.receiver.reconnect.assert_not_called()
def test_live_adoption_count_cannot_regress_on_first_followup_poll(self):
state = self.fixture()
state.statuses = [("pulling", 2, 3, self.MID), ("pulling", 1, 3, self.MID)]
with self.assertRaisesRegex(HIL.TestFailure, "completed-block count moved backwards"):
self.run_continuation(state)
state.receiver.checked.assert_not_called()
def test_geometry_is_rechecked_after_initial_adoption(self):
state = self.fixture()
state.statuses = [("pulling", 1, 3, self.MID), ("pulling", 2, 4, self.MID)]
with self.assertRaisesRegex(HIL.TestFailure, "block geometry changed"):
self.run_continuation(state)
state.receiver.checked.assert_not_called()
def test_observer_downtime_is_included_in_elapsed_transfer_pace(self):
state = self.fixture()
state.downtime = 7
self.run_continuation(state)
self.assertEqual(state.entry["download_seconds"], 18)
self.assertEqual(state.entry["progress"][-1]["elapsed_seconds"], 18)
def test_transport_interruption_after_install_send_leaves_persisted_intent(self):
state = self.fixture()
state.receiver.checked.side_effect = HIL.TestFailure("CLI reply marker was not received for ota install")
with self.assertRaisesRegex(HIL.TestFailure, "marker was not received"):
self.run_continuation(state)
self.assertIn("install_requested_utc", state.entry)
self.assertNotIn("install_reply", state.entry)
self.assertIn("install_requested_utc", state.saved_entries[-1])
state.receiver.checked.assert_called_once_with("ota install", timeout=30)
state.receiver.reconnect.assert_not_called()
def test_post_install_temp_overlay_fails_before_saved_setting_snapshot(self):
state = self.fixture()
state.receiver.reconnect.side_effect = None
state.receiver.reconnect.return_value = copy.deepcopy(state.info)
with self.assertRaisesRegex(HIL.TestFailure, "inactive temporary profiles"):
self.run_continuation(state)
self.assertEqual(state.snapshot_calls, [])
def test_post_install_normal_snapshot_rechecks_original_saved_advert_interval(self):
state = self.fixture()
state.after_snapshot["settings"]["advert.interval"] = "60"
with self.assertRaisesRegex(HIL.TestFailure, "saved settings changed across OTA install"):
self.run_continuation(state)
self.assertEqual(state.snapshot_calls, ["B"])
self.assertEqual(state.entry["preservation"], {"equal": False, "changed_fields": ["settings"]})
def previous_report(self, state):
return {"schema": 1, "action": "run", "expected_body_hash16": self.HASH,
"baseline_mode": HIL.BASELINE_MODE, "first_source": "a",
"lab_radio": HIL.LAB_RADIO, "requested_rounds": 1, "result": "ABORTED",
"baseline": copy.deepcopy(state.baseline), "rounds": [copy.deepcopy(state.entry)],
"devices": {name: {"identity": {"usb_serial": serial}}
for name, serial in zip(("A", "B"), self.SERIALS)}}
def call_main(self, state, previous):
report = SimpleNamespace(data={"rounds": []}, save=mock.Mock(), event=mock.Mock())
live = copy.deepcopy(state.baseline)
for value in live.values():
value["settings"]["advert.interval"] = "60"
def preflight(devices, expected_hash, output):
output.data["devices"] = copy.deepcopy(previous["devices"])
return ["AB" * 32, "CD" * 32]
def finish_round(*args):
args[-1]["result"] = "PASS"
with tempfile.TemporaryDirectory() as folder:
previous_path = Path(folder) / "previous.json"
previous_path.write_text(json.dumps(previous), encoding="utf-8")
with mock.patch.object(HIL, "Report", return_value=report), \
mock.patch.object(HIL, "V4Device", side_effect=[state.source, state.receiver]), \
mock.patch.object(HIL, "preflight", side_effect=preflight), \
mock.patch.object(HIL, "prepare") as prepare, \
mock.patch.object(HIL, "disarm_lab") as disarm, \
mock.patch.object(HIL, "snapshot", side_effect=[live["A"], live["B"]]), \
mock.patch.object(HIL, "run_round", side_effect=finish_round) as run:
result = HIL.main(["run", "--serial-a", self.SERIALS[0], "--serial-b", self.SERIALS[1],
"--expected-body-hash16", self.HASH, "--rounds", "1",
"--report", str(Path(folder) / "new.json"),
"--continue-report", str(previous_path)])
prepare.assert_not_called()
disarm.assert_not_called()
return result, report, run
def test_main_accepts_matching_baseline_for_interrupted_or_timed_out_observer(self):
for result in ("ABORTED", "FAIL"):
with self.subTest(result=result):
state = self.fixture()
previous = self.previous_report(state)
previous["result"] = result
if result == "FAIL":
previous["failure"] = "OTA download exceeded its per-round timeout"
code, report, run = self.call_main(state, previous)
self.assertEqual(code, 0)
self.assertEqual(report.data["result"], "PASS")
self.assertEqual(report.data["baseline"], previous["baseline"])
self.assertEqual(run.call_count, 1)
self.assertIs(run.call_args.args[-1], report.data["rounds"][0])
self.assertIn("continued_from", report.data)
def test_live_continuation_defers_only_local_advert_and_keeps_all_other_settings_strict(self):
saved = self.fixture().baseline["A"]
original = copy.deepcopy(saved)
live = copy.deepcopy(saved)
live["settings"]["advert.interval"] = "60"
self.assertTrue(HIL.live_snapshot_matches_saved(saved, live))
self.assertEqual(saved, original)
for key in ("name", "radio", "radio2", "flood.advert.interval"):
with self.subTest(key=key):
changed = copy.deepcopy(live)
changed["settings"][key] = "changed"
self.assertFalse(HIL.live_snapshot_matches_saved(saved, changed))
for value in ("0", "59", "61"):
with self.subTest(local_advert=value):
changed = copy.deepcopy(live)
changed["settings"]["advert.interval"] = value
self.assertFalse(HIL.live_snapshot_matches_saved(saved, changed))
def test_main_rejects_changed_key_acl_name_or_radio_settings_before_any_round(self):
for field in ("private_key_sha256", "acl", "name", "radio", "radio2"):
with self.subTest(field=field):
state = self.fixture()
previous = self.previous_report(state)
if field in ("private_key_sha256", "acl"):
previous["baseline"]["A"][field] = "changed"
else:
previous["baseline"]["A"]["settings"][field] = "changed"
code, report, run = self.call_main(state, previous)
self.assertEqual(code, 1)
self.assertEqual(report.data["failure"], "Identity, ACL or settings changed before continuation")
run.assert_not_called()
def test_main_rejects_invalid_definition_identity_or_pending_round(self):
for case in ("schema", "hash", "radio", "round count", "baseline mode", "first source", "not interrupted",
"other failure", "physical identity", "empty progress", "completed round"):
with self.subTest(case=case):
state = self.fixture()
previous = self.previous_report(state)
if case == "schema":
previous["schema"] = 2
elif case == "hash":
previous["expected_body_hash16"] = "FFFFFFFFFFFFFFFF"
elif case == "radio":
previous["lab_radio"] = "910.525,500,5,5"
elif case == "round count":
previous["requested_rounds"] = 3
elif case == "baseline mode":
previous.pop("baseline_mode")
elif case == "first source":
previous["first_source"] = "b"
elif case == "not interrupted":
previous["result"] = "RUNNING"
elif case == "other failure":
previous.update(result="FAIL", failure="Some other failure")
elif case == "physical identity":
previous["devices"]["A"]["identity"]["usb_serial"] = "OTHER"
elif case == "empty progress":
previous["rounds"][0]["progress"] = []
elif case == "completed round":
previous["rounds"][0]["result"] = "PASS"
code, report, run = self.call_main(state, previous)
self.assertEqual(code, 1)
self.assertEqual(report.data["result"], "FAIL")
run.assert_not_called()
def test_continuation_refuses_prepare_or_probe_before_creating_devices(self):
for action, extra in (("run", ["--prepare"]), ("prepare", []), ("probe", [])):
with self.subTest(action=action), mock.patch.object(HIL, "Report") as report, \
mock.patch.object(HIL, "V4Device") as devices, \
contextlib.redirect_stderr(io.StringIO()):
with self.assertRaises(SystemExit) as raised:
HIL.main([action, "--serial-a", self.SERIALS[0], "--serial-b", self.SERIALS[1],
"--expected-body-hash16", self.HASH, "--report", "unused.json",
"--continue-report", "previous.json"] + extra)
self.assertEqual(raised.exception.code, 2)
report.assert_not_called()
devices.assert_not_called()
class ConnectionTests(unittest.TestCase):
def test_native_usb_session_holds_dtr_and_keeps_rts_low(self):
device = HIL.V4Device("A", "44:1b:f6:69:cf:98")
serial_instance = mock.Mock()
serial_module = SimpleNamespace(Serial=mock.Mock(return_value=serial_instance))
port = SimpleNamespace(device="/dev/ttyACM9", vid=0x303A, pid=0x1001)
with mock.patch.dict(sys.modules, {"serial": serial_module}), \
mock.patch.object(HIL, "find_port", return_value=port), \
mock.patch.object(device, "read", return_value=""), \
mock.patch.object(device, "_command", side_effect=[
"\r\n -> Heltec V4.3 OLED\r\n", "\r\n -> > repeater\r\n"]):
device.connect()
self.assertIs(device.stream, serial_instance)
self.assertTrue(serial_instance.dtr)
self.assertFalse(serial_instance.rts)
serial_instance.open.assert_called_once_with()
self.assertEqual(device.identity["usb_serial"], "441BF669CF98")
self.assertEqual(device.connections, 1)
def reconnect_case(self, startup_error):
device = HIL.V4Device("A", "441BF669CF98")
clock = FakeClock()
reply = "\r\n -> self body=1600000 image=1600200 base_hash=0123456789ABCDEF\r\n"
with mock.patch.object(device, "close") as close, \
mock.patch.object(device, "connect", side_effect=[startup_error, None]) as connect, \
mock.patch.object(device, "command", return_value=reply) as command, \
mock.patch.object(HIL.time, "monotonic", clock.monotonic), \
mock.patch.object(HIL.time, "sleep", clock.sleep):
result = device.reconnect("0123456789ABCDEF", timeout=10)
self.assertEqual(result["body_hash16"], "0123456789ABCDEF")
self.assertEqual(connect.call_count, 2)
self.assertEqual(close.call_count, 2)
command.assert_called_once_with("ota self")
def test_reconnect_retries_missing_startup_marker(self):
self.reconnect_case(HIL.TestFailure("A: CLI reply marker was not received for board"))
def test_reconnect_never_retries_wrong_identity_role_or_image_hash(self):
for error in ("USB identity is not a Heltec V4", "Board firmware role is not Repeater",
"Running firmware hash changed after reconnect"):
with self.subTest(error=error):
device = HIL.V4Device("A", "441BF669CF98")
with mock.patch.object(device, "close"), \
mock.patch.object(device, "connect", side_effect=HIL.TestFailure(error)) as connect, \
mock.patch.object(HIL.time, "sleep") as sleep:
with self.assertRaisesRegex(HIL.TestFailure, error):
device.reconnect("0123456789ABCDEF", timeout=10)
connect.assert_called_once_with()
sleep.assert_not_called()
if __name__ == "__main__":
unittest.main()
test/test_hil_esp32_usb_host_reboot.py
+622
View File
@@ -0,0 +1,622 @@
#!/usr/bin/env python3
"""Offline safety and evidence tests; no serial devices or host reboot calls."""
import contextlib
import copy
import importlib.util
import json
import os
from pathlib import Path
from types import SimpleNamespace
import sys
import tempfile
import unittest
from unittest import mock
ROOT = Path(__file__).resolve().parents[1]
HIL_DIR = ROOT / "tools" / "hil"
def load_module(name, path):
spec = importlib.util.spec_from_file_location(name, path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
SOAK = load_module("usb_reboot_test_soak", HIL_DIR / "s3_memory_soak.py")
with mock.patch.dict(sys.modules, {"s3_memory_soak": SOAK}):
OTA = load_module("usb_reboot_test_ota", HIL_DIR / "esp32_self_serve_ota.py")
with mock.patch.dict(sys.modules, {"esp32_self_serve_ota": OTA}):
HIL = load_module("usb_reboot_test_harness", HIL_DIR / "esp32_usb_host_reboot.py")
HASH = "0123456789ABCDEF"
SERIALS = ("441BF669CF98", "441BF669C9C0")
CORE = {"battery_mv": 4000, "uptime_secs": 200, "errors": 0, "queue_len": 0}
DIAG = {"state": 1, "state_name": "RX", "recv": 1, "sent": 0, "errors": 0,
"err_flags": 0, "outbound": False, "last_rx_secs": 3}
DIAG_TEXT = "state=1(RX), recv=1, sent=0, errors=0, err_flags=0, outbound=no, last_rx=3s ago"
STATE = {
"radio": "909.500,500.000,5,5", "radio2": "off", "tempradio": "off",
"tempradio2": "off", "powersaving": "on",
"radio2.status": "off; RX=1,0 TX=0,0 switches=0 errors=0 max=0us preamble=16,16",
"usb.logging": {"available": False, "enabled": False},
}
def probe():
data = {
"schema": 1, "evidence_kind": HIL.KIND, "action": "probe", "result": "PASS",
"expected_body_hash16": HASH, "host_reboot_performed_by_tool": False,
"minimum_board_age_seconds": 180,
"pi": {"boot_id": "11111111-1111-1111-1111-111111111111",
"dwc_otg_speed": 1, "cmdline_sha256": "a" * 64,
"model": "Raspberry Pi", "kernel": "same"},
"devices": {},
}
for name, serial in zip(("A", "B"), SERIALS):
data["devices"][name] = {
"identity": {"usb_serial": serial, "vid": 0x303A, "pid": 0x1001,
"role": "Repeater", "board": "Heltec V4", "endpoint": "/dev/ttyACM0"},
"self": {"body_hash16": HASH, "body_bytes": 2000, "image_bytes": 2100},
"inactive_slot": {"inactive_address": 4096, "size_bytes": 5000},
"usb": {"usb_serial": serial, "speed_mbps": 12}, "ota_idle": True,
"ota_policy": {"autofetch": "off", "autoinstall": "off"},
"core": copy.deepcopy(CORE), "radio_diag": copy.deepcopy(DIAG),
"radio_state": copy.deepcopy(STATE),
"snapshot": {"private_key_sha256": "b" * 64, "public_key": "C" * 64,
"acl": [], "settings": {"name": name, "radio2": {"mode": "off"},
"advert.interval": "0",
"flood.advert.interval": "0"}},
"reset_reason": "Power on reset", "observed_unix_seconds": 1000.,
"sample_uncertainty_seconds": .2,
}
return data
def post_probe():
data = probe()
data["pi"]["boot_id"] = "22222222-2222-2222-2222-222222222222"
for item in data["devices"].values():
item["core"]["uptime_secs"] += 90
item["observed_unix_seconds"] += 90
return data
class ParserTests(unittest.TestCase):
def rejected(self, fn, *args):
with self.assertRaises(HIL.TestFailure):
fn(*args)
def test_exact_core_json(self):
self.assertEqual(HIL.parse_core(json.dumps(CORE)), CORE)
for data in ("{", "{}", "[]", json.dumps(dict(CORE, queue_len=True)),
json.dumps(dict(CORE, queue_len=-1)), json.dumps(dict(CORE, extra=1)),
json.dumps(dict(CORE, queue_len=2**32))):
with self.subTest(data=data):
self.rejected(HIL.parse_core, data)
def test_radio_diagnostics_reject_truncation_and_unsolicited_tail(self):
self.assertEqual(HIL.parse_radio_diag(DIAG_TEXT), DIAG)
for data in ("state=1(RX)", DIAG_TEXT + " asynchronous output",
DIAG_TEXT.replace("outbound=no", "outbound=maybe"),
DIAG_TEXT.replace("recv=1", "recv=4294967296")):
with self.subTest(data=data):
self.rejected(HIL.parse_radio_diag, data)
def test_quiet_requires_empty_queue_receive_and_no_outbound(self):
HIL.require_quiet(CORE, DIAG)
self.rejected(HIL.require_quiet, dict(CORE, queue_len=1), DIAG)
self.rejected(HIL.require_quiet, CORE, dict(DIAG, state=3, state_name="TX_WAIT"))
self.rejected(HIL.require_quiet, CORE, dict(DIAG, outbound=True))
def test_logging_accepts_only_exact_compiled_out_response(self):
device = SimpleNamespace(name="A")
for text, expected in (
("> off", {"available": True, "enabled": False}),
("> on", {"available": True, "enabled": True}),
("Error: unknown setting: usb.logging", {"available": False, "enabled": False})):
device.command = mock.Mock(return_value="\n -> " + text)
self.assertEqual(HIL.logging_state(device), expected)
for text in ("Error: invalid", "Error: unknown setting: usb.logging extra", "unavailable", ""):
device.command = mock.Mock(return_value="\n -> " + text)
self.rejected(HIL.logging_state, device)
def test_only_idle_ota_is_accepted(self):
device = SimpleNamespace(name="A", command=mock.Mock(
return_value="\n -> OTA | thing | no download | serving:on"))
HIL.require_idle(device)
for text in ("OTA | download: complete 1/1 | serving:on",
"OTA | download: failed 1/1 | serving:on", "no download",
"OTA | download: receiving 1/2 | serving:on"):
device.command.return_value = "\n -> " + text
self.rejected(HIL.require_idle, device)
def test_automatic_ota_policy_must_already_be_off(self):
device = SimpleNamespace(name="A", command=mock.Mock(
return_value="\n -> ota config: autofetch=off autoinstall=off checkpoint=16"))
self.assertEqual(HIL.manual_ota_policy(device),
{"autofetch": "off", "autoinstall": "off"})
for text in ("ota config: autofetch=any autoinstall=off",
"ota config: autofetch=off autoinstall=trusted",
"ota config: autofetch=off", "Error: unknown command",
"ota config: autofetch=off autofetch=off autoinstall=off"):
device.command.return_value = "\n -> " + text
self.rejected(HIL.manual_ota_policy, device)
def test_running_hash_and_slot_geometry_are_checked_before_policy(self):
device = SimpleNamespace(name="A", connect=mock.Mock(),
identity=probe()["devices"]["A"]["identity"])
replies = {
"ota self": "\n -> self body=2000 image=2100 base_hash=" + HASH,
"ota dev apply slot": "\n -> inactive slot addr=0x1000 size=5000",
"ota config": "\n -> ota config: autofetch=off autoinstall=off",
}
device.command = mock.Mock(side_effect=lambda command: replies[command])
self.assertEqual(HIL.verified_device(device, HASH)["self"]["body_hash16"], HASH)
device.command.reset_mock()
self.rejected(HIL.verified_device, device, "FEDCBA9876543210")
self.assertEqual(device.command.call_args_list, [mock.call("ota self")])
replies["ota dev apply slot"] = "\n -> inactive slot addr=0x1000 size=1000"
self.rejected(HIL.verified_device, device, HASH)
def test_temporary_profile_fields_are_strict_and_diagnostic_tails_not_retained(self):
device = SimpleNamespace(command=mock.Mock(side_effect=lambda command, **kwargs:
"\n -> > " + ("909.500,500.000,5,5,32" if command == "get tempradio"
else "909.500,500.000,5,5,rxtx,32 (auto); diagnostic tail")))
state = HIL.temporary_profiles(device)
self.assertEqual(state["tempradio2"], "909.500,500.000,5,5,rxtx,32 (auto)")
self.assertNotIn("diagnostic tail", json.dumps(state))
for value in ("Error: unknown setting", "unavailable", "60", "1,2,3", "off extra"):
device.command = mock.Mock(return_value="\n -> > " + value)
self.rejected(HIL.temporary_profiles, device)
class ComparisonTests(unittest.TestCase):
def test_continued_uptime_and_renumbered_tty_pass(self):
old, new = probe(), post_probe()
new["devices"]["A"]["identity"]["endpoint"] = "/dev/ttyACM7"
HIL.validate_probe(old, HASH, SERIALS)
HIL.validate_probe(new, HASH, SERIALS)
result = HIL.compare_probes(old, new, 10)
self.assertFalse(result["devices"]["A"]["board_reboot_observed"])
self.assertEqual(result["devices"]["A"]["board_uptime_gain_seconds"], 90)
def test_host_must_reboot_without_configuration_or_kernel_change(self):
for key, value in (
("boot_id", probe()["pi"]["boot_id"]), ("dwc_otg_speed", 2),
("cmdline_sha256", "changed"), ("kernel", "changed"), ("model", "changed")):
with self.subTest(key=key):
new = post_probe()
new["pi"][key] = value
with self.assertRaises(HIL.TestFailure):
HIL.compare_probes(probe(), new, 10)
def test_radio_reboot_or_changed_settings_hash_slot_identity_fail(self):
for field, value in (
("reset_reason", "Software reset"), ("inactive_slot", {}), ("snapshot", {}),
("self", {})):
with self.subTest(field=field):
new = post_probe()
new["devices"]["A"][field] = value
with self.assertRaises(HIL.TestFailure):
HIL.compare_probes(probe(), new, 10)
for uptime in (10, 200, 400):
new = post_probe()
new["devices"]["A"]["core"]["uptime_secs"] = uptime
with self.assertRaises(HIL.TestFailure):
HIL.compare_probes(probe(), new, 10)
for field, value in (("usb_serial", "other"), ("pid", 0x1002), ("role", "Companion")):
new = post_probe()
new["devices"]["A"]["identity"][field] = value
with self.assertRaises(HIL.TestFailure):
HIL.compare_probes(probe(), new, 10)
new = post_probe()
new["devices"]["A"]["radio_state"]["radio"] = "910.525,500,5,5"
with self.assertRaises(HIL.TestFailure):
HIL.compare_probes(probe(), new, 10)
def test_probe_rejects_sleep_inhibitors_and_boot_grace(self):
for field, value in (
("powersaving", "off"), ("radio2", "909.5,500,5,5,rx"),
("tempradio", "pending"), ("tempradio2", "pending"),
("radio2.status", "radio2 change pending"),
("usb.logging", {"available": True, "enabled": True})):
with self.subTest(field=field):
data = probe()
data["devices"]["A"]["radio_state"][field] = value
with self.assertRaises(HIL.TestFailure):
HIL.validate_probe(data, HASH, SERIALS)
for field, value in (("uptime_secs", 180), ("queue_len", 1)):
data = probe()
data["devices"]["A"]["core"][field] = value
with self.assertRaises(HIL.TestFailure):
HIL.validate_probe(data, HASH, SERIALS)
def test_probe_hash_physical_serial_role_native_speed_are_required(self):
data = probe()
data["devices"]["A"]["usb"]["speed_mbps"] = 480
with self.assertRaises(HIL.TestFailure):
HIL.validate_probe(data, HASH, SERIALS)
with self.assertRaises(HIL.TestFailure):
HIL.validate_probe(probe(), "FEDCBA9876543210", SERIALS)
with self.assertRaises(HIL.TestFailure):
HIL.validate_probe(probe(), HASH, tuple(reversed(SERIALS)))
def test_probe_rejects_invalid_time_or_secret_digest(self):
for field, value in (("observed_unix_seconds", float("nan")),
("sample_uncertainty_seconds", 6)):
data = probe()
data["devices"]["A"][field] = value
with self.assertRaises(HIL.TestFailure):
HIL.validate_probe(data, HASH, SERIALS)
data = probe()
data["devices"]["A"]["snapshot"]["private_key_sha256"] = "raw private key"
with self.assertRaises(HIL.TestFailure):
HIL.validate_probe(data, HASH, SERIALS)
class FakeClock:
def __init__(self):
self.now = 0.
def monotonic(self):
return self.now
def sleep(self, seconds):
self.now += seconds
class FakeDevice:
def __init__(self, name, ledger):
self.name = name
self.ledger = ledger
self.active = False
self.rebooted = False
self.busy_polls = 0
self.logging_available = False
self.fail_write = False
self.autofetch = "off"
self.temp_primary = True
self.temp_secondary = True
self.saved_advert = "0"
self.saved_flood_advert = "0"
self.long_temp = False
self.secondary_mode = "rx"
def command(self, command, timeout=5):
self.ledger.append((self.name, "read", command))
if command == "ota status":
text = "OTA | download: receiving 1/2 | serving:on" if self.active else \
"OTA | thing | no download | serving:on"
elif command == "ota config":
text = "ota config: autofetch=" + self.autofetch + " autoinstall=off"
elif command == "stats-core":
text = json.dumps(CORE)
elif command == "stats-radio-diag":
text = DIAG_TEXT
if self.busy_polls:
self.busy_polls -= 1
text = text.replace("1(RX)", "3(TX_WAIT)")
elif command == "get usb.logging":
text = "> off" if self.logging_available else "Error: unknown setting: usb.logging"
elif command == "get tempradio":
text = "> 909.500,500.000,5,5,120" if self.temp_primary else "> off"
elif command == "get tempradio2":
text = "> 909.500,500.000,5,5,rxtx,120" if self.temp_secondary else "> off"
elif command == "get powersaving":
text = "> on"
else:
raise AssertionError("Unexpected fake read command")
return "\n -> " + text
def checked(self, command, timeout=5):
self.ledger.append((self.name, "write", command, timeout))
if self.fail_write:
raise HIL.TestFailure("Acknowledgement timed out")
if command == "normalradio":
self.temp_primary = False
elif command == "set tempradio2 off":
self.temp_secondary = False
elif command == "set radio2 off":
self.secondary_mode = "off"
return "OK - acknowledged"
def reboot(self, expected_hash):
self.ledger.append((self.name, "reboot", expected_hash))
self.rebooted = True
class PreparationTests(unittest.TestCase):
def setUp(self):
self.ledger = []
self.devices = [FakeDevice(name, self.ledger) for name in ("A", "B")]
self.report = SimpleNamespace(data={"devices": {}}, save=mock.Mock(), event=mock.Mock())
self.clock = FakeClock()
self.stack = contextlib.ExitStack()
self.addCleanup(self.stack.close)
self.stack.enter_context(mock.patch.object(HIL.time, "monotonic", self.clock.monotonic))
self.stack.enter_context(mock.patch.object(HIL.time, "sleep", self.clock.sleep))
self.stack.enter_context(mock.patch.object(HIL, "native_usb_state", return_value={"speed_mbps": 12}))
self.stack.enter_context(mock.patch.object(HIL, "verified_device", side_effect=self.verify))
self.stack.enter_context(mock.patch.object(HIL, "snapshot", side_effect=self.saved))
def verify(self, device, expected_hash):
self.ledger.append((device.name, "verify", expected_hash))
result = copy.deepcopy(probe()["devices"][device.name])
result["ota_policy"] = HIL.manual_ota_policy(device)
return result
def saved(self, device):
self.ledger.append((device.name, "snapshot"))
result = copy.deepcopy(probe()["devices"][device.name]["snapshot"])
result["settings"]["radio2"] = {"mode": device.secondary_mode}
result["settings"]["advert.interval"] = "60" if device.temp_primary else device.saved_advert
result["settings"]["flood.advert.interval"] = (
"3" if device.temp_primary and device.long_temp else device.saved_flood_advert)
return result
def test_both_preflights_precede_first_write_and_reboots_preserve_identity(self):
HIL.prepare_devices(self.devices, HASH, self.report)
first_write = next(i for i, row in enumerate(self.ledger) if row[1] == "write")
self.assertIn(("A", "snapshot"), self.ledger[:first_write])
self.assertIn(("B", "snapshot"), self.ledger[:first_write])
for device in self.devices:
writes = [row for row in self.ledger if row[0] == device.name and row[1] == "write"]
self.assertEqual([row[2] for row in writes],
["normalradio", "set tempradio2 off", "set radio2 off", "set powersaving on"])
self.assertTrue(all(row[3] == 30 for row in writes))
self.assertTrue(device.rebooted)
self.assertTrue(self.report.data["board_reboots_intentionally_performed"])
def test_raw_temporary_getter_and_normalized_saved_baselines_are_both_retained(self):
HIL.prepare_devices(self.devices, HASH, self.report)
entry = self.report.data["devices"]["A"]
self.assertEqual(entry["before_prepare"]["snapshot"]["settings"]["advert.interval"], "60")
self.assertEqual(entry["before_prepare"]["core"], CORE)
self.assertEqual(entry["before_prepare"]["radio_diag"], DIAG)
self.assertEqual(entry["normalized_saved_snapshot"]["settings"]["advert.interval"], "0")
self.assertEqual(entry["normalized_saved_snapshot"]["settings"]["radio2"], {"mode": "rx"})
self.assertEqual(entry["after_prepare"]["snapshot"]["settings"]["advert.interval"], "0")
self.assertEqual(entry["normalization_differences"]["advert.interval"],
{"live_before": "60", "saved_after_normalradio": "0",
"reason": "documented temporary radio timing getter override"})
self.assertFalse(any("advert.interval" in row[2]
for row in self.ledger if row[1] == "write"))
# Both normalized snapshots must be taken before either saved radio2
# setting is changed; clearing temporary overlays is a separate phase.
first_saved_write = next(i for i, row in enumerate(self.ledger)
if row[1] == "write" and row[2] == "set radio2 off")
for name in ("A", "B"):
self.assertEqual(self.ledger[:first_saved_write].count((name, "snapshot")), 2)
def test_normalization_does_not_assume_saved_advert_is_zero(self):
self.devices[0].saved_advert = "72"
self.devices[0].saved_flood_advert = "7"
self.devices[0].long_temp = True
HIL.prepare_devices(self.devices, HASH, self.report)
entry = self.report.data["devices"]["A"]
self.assertEqual(entry["normalized_saved_snapshot"]["settings"]["advert.interval"], "72")
self.assertEqual(entry["after_prepare"]["snapshot"]["settings"]["advert.interval"], "72")
self.assertEqual(entry["normalization_differences"]["flood.advert.interval"]
["saved_after_normalradio"], "7")
def test_unrelated_normalization_change_fails_before_persistent_settings_or_reboot(self):
original = self.saved
def changed(device):
result = original(device)
if not device.temp_primary:
result["settings"]["name"] = "changed unexpectedly"
return result
with mock.patch.object(HIL, "snapshot", side_effect=changed):
with self.assertRaisesRegex(HIL.TestFailure, "unrelated saved setting"):
HIL.prepare_devices(self.devices, HASH, self.report)
self.assertFalse(any(row[1] == "reboot" or row[1:3] == ("write", "set radio2 off")
for row in self.ledger))
def test_advert_change_without_original_temporary_profile_is_not_ignored(self):
self.devices[0].temp_primary = False
original = self.saved
calls = {}
def changed(device):
result = original(device)
calls[device.name] = calls.get(device.name, 0) + 1
if calls[device.name] >= 2:
result["settings"]["advert.interval"] = "2"
return result
with mock.patch.object(HIL, "snapshot", side_effect=changed):
with self.assertRaisesRegex(HIL.TestFailure, "unrelated saved setting"):
HIL.prepare_devices(self.devices, HASH, self.report)
self.assertFalse(any(device.rebooted for device in self.devices))
def test_arbitrary_advert_change_after_reboot_still_fails(self):
original = self.saved
def changed(device):
result = original(device)
if device.rebooted:
result["settings"]["advert.interval"] = "2"
return result
with mock.patch.object(HIL, "snapshot", side_effect=changed):
with self.assertRaisesRegex(HIL.TestFailure, "unrelated saved setting"):
HIL.prepare_devices(self.devices, HASH, self.report)
self.assertEqual(self.report.data["devices"]["A"]["normalized_saved_snapshot"]
["settings"]["advert.interval"], "0")
def test_identity_change_during_normalization_is_rejected(self):
original = self.saved
def changed(device):
result = original(device)
if not device.temp_primary:
result["private_key_sha256"] = "d" * 64
return result
with mock.patch.object(HIL, "snapshot", side_effect=changed):
with self.assertRaisesRegex(HIL.TestFailure, "identity or ACL"):
HIL.prepare_devices(self.devices, HASH, self.report)
self.assertFalse(any(device.rebooted for device in self.devices))
def test_pending_temporary_restoration_timeout_never_changes_saved_radio2_or_reboots(self):
original = self.devices[0].checked
def stays_temporary(command, timeout=5):
result = original(command, timeout)
self.devices[0].temp_primary = True
return result
self.devices[0].checked = stays_temporary
with self.assertRaisesRegex(HIL.TestFailure, "restoration timed out"):
HIL.prepare_devices(self.devices, HASH, self.report)
self.assertFalse(any(row[1] == "reboot" or row[1:3] == ("write", "set radio2 off")
for row in self.ledger))
self.assertLessEqual(self.clock.now, 30)
def test_logging_off_only_if_available(self):
self.devices[0].logging_available = True
HIL.prepare_devices(self.devices, HASH, self.report)
self.assertIn(("A", "write", "set usb.logging off", 30), self.ledger)
self.assertNotIn(("B", "write", "set usb.logging off", 30), self.ledger)
def test_any_active_download_refuses_all_writes(self):
self.devices[1].active = True
with self.assertRaises(HIL.TestFailure):
HIL.prepare_devices(self.devices, HASH, self.report)
self.assertFalse(any(row[1] in ("write", "reboot") for row in self.ledger))
def test_auto_download_policy_refuses_all_writes(self):
self.devices[1].autofetch = "any"
with self.assertRaises(HIL.TestFailure):
HIL.prepare_devices(self.devices, HASH, self.report)
self.assertFalse(any(row[1] in ("write", "reboot") for row in self.ledger))
def test_failed_setter_is_not_retried_and_neither_radio_reboots(self):
self.devices[0].fail_write = True
with self.assertRaises(HIL.TestFailure):
HIL.prepare_devices(self.devices, HASH, self.report)
self.assertEqual([row for row in self.ledger if row[1] == "write"],
[("A", "write", "normalradio", 30)])
self.assertFalse(any(device.rebooted for device in self.devices))
def test_transient_beacon_tx_waits_without_mutations(self):
self.devices[0].busy_polls = 1
result = HIL.wait_quiet(self.devices)
self.assertEqual(set(result), {"A", "B"})
self.assertGreater(self.clock.now, 0)
self.assertFalse(any(row[1] in ("write", "reboot") for row in self.ledger))
def test_persistent_busy_state_has_bounded_timeout(self):
self.devices[0].busy_polls = 100000
with self.assertRaises(HIL.TestFailure):
HIL.wait_quiet(self.devices, timeout=2)
self.assertLessEqual(self.clock.now, 2)
self.assertFalse(any(row[1] in ("write", "reboot") for row in self.ledger))
def test_new_download_during_quiet_poll_fails_without_writes(self):
self.devices[0].active = True
with self.assertRaises(HIL.TestFailure):
HIL.wait_quiet(self.devices)
self.assertFalse(any(row[1] in ("write", "reboot") for row in self.ledger))
def test_download_started_after_quiet_sample_refuses_first_mutation(self):
def newly_active(devices):
self.devices[1].active = True
return {device.name: {"core": CORE, "radio_diag": DIAG} for device in devices}
with mock.patch.object(HIL, "wait_quiet", side_effect=newly_active):
with self.assertRaises(HIL.TestFailure):
HIL.prepare_devices(self.devices, HASH, self.report)
self.assertFalse(any(row[1] in ("write", "reboot") for row in self.ledger))
def test_download_started_after_settings_refuses_board_reboots(self):
original = self.devices[1].checked
def active_after_settings(command, timeout=5):
result = original(command, timeout)
if command == "set powersaving on":
self.devices[1].active = True
return result
self.devices[1].checked = active_after_settings
with self.assertRaises(HIL.TestFailure):
HIL.prepare_devices(self.devices, HASH, self.report)
self.assertFalse(any(row[1] == "reboot" for row in self.ledger))
class FilesystemEvidenceTests(unittest.TestCase):
def test_new_report_is_final_read_only_and_cannot_be_overwritten(self):
args = SimpleNamespace(action="probe", expected_body_hash16=HASH, rounds=0, timeout=30,
poll_interval=0, min_age=180, uptime_tolerance=10)
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "report.json"
report = HIL.EvidenceReport(path, args)
report.data["result"] = "PASS"
report.seal()
self.assertEqual(json.loads(path.read_text())["result"], "PASS")
if os.name != "nt":
self.assertEqual(path.stat().st_mode & 0o777, 0o400)
with self.assertRaises(HIL.TestFailure):
report.save()
with self.assertRaises(HIL.TestFailure):
HIL.EvidenceReport(path, args)
os.chmod(path, 0o600) # Permit temporary-fixture cleanup on Windows.
def test_report_creation_race_cannot_replace_existing_evidence(self):
args = SimpleNamespace(action="probe", expected_body_hash16=HASH, rounds=0, timeout=30,
poll_interval=0, min_age=180, uptime_tolerance=10)
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "report.json"
path.write_text("existing evidence", encoding="utf-8")
# Simulate another process creating the report after the initial
# exists() check. The atomic hard-link creation still rejects it.
with mock.patch.object(Path, "exists", return_value=False):
with self.assertRaises(HIL.TestFailure):
HIL.EvidenceReport(path, args)
self.assertEqual(path.read_text(encoding="utf-8"), "existing evidence")
def test_host_records_speed_and_hash_not_raw_cmdline(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
paths = {
"proc/sys/kernel/random/boot_id": probe()["pi"]["boot_id"],
"proc/uptime": "200.1 300.2",
"proc/cmdline": "dwc_otg.speed=1 secret_argument=not-for-report",
"proc/device-tree/model": "Raspberry Pi 3 Model B Rev 1.2\0",
}
for relative, value in paths.items():
path = root / relative
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(value, encoding="ascii")
result = HIL.pi_state(root)
self.assertEqual(result["dwc_otg_speed"], 1)
self.assertNotIn("secret_argument", json.dumps(result))
(root / "proc/cmdline").write_text("dwc_otg.speed=0", encoding="ascii")
with self.assertRaises(HIL.TestFailure):
HIL.pi_state(root)
def test_native_usb_serial_full_speed_gate(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
device_path = root / "sys/class/tty/ttyACM0/device"
device_path.mkdir(parents=True)
for name, value in {"idVendor": "303a", "idProduct": "1001",
"serial": SERIALS[0], "speed": "12"}.items():
(device_path / name).write_text(value, encoding="ascii")
device = SimpleNamespace(endpoint="/dev/ttyACM0", serial_number=SERIALS[0])
self.assertEqual(HIL.native_usb_state(device, root)["speed_mbps"], 12)
for name, value in (("speed", "480"), ("speed", "unknown"),
("idProduct", "1002"), ("serial", SERIALS[1])):
original = (device_path / name).read_text(encoding="ascii")
(device_path / name).write_text(value, encoding="ascii")
with self.assertRaises(HIL.TestFailure):
HIL.native_usb_state(device, root)
(device_path / name).write_text(original, encoding="ascii")
def test_load_report_requires_successful_probe(self):
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "probe.json"
path.write_text(json.dumps(probe()), encoding="utf-8")
self.assertEqual(HIL.load_probe(path, HASH, SERIALS)["result"], "PASS")
data = probe()
data["result"] = "RUNNING"
path.write_text(json.dumps(data), encoding="utf-8")
with self.assertRaises(HIL.TestFailure):
HIL.load_probe(path, HASH, SERIALS)
if __name__ == "__main__":
unittest.main()
test/test_ota_device_deflate_encoder.py
+507
View File
@@ -0,0 +1,507 @@
#!/usr/bin/env python3
"""Exercise the real on-device transport codec without PlatformIO or hardware."""
from pathlib import Path
import os
import random
import shutil
import subprocess
import tempfile
import unittest
import zlib
from test_t096_full_memory import method
ROOT = Path(__file__).resolve().parents[1]
CODEC_RUNNER = r'''
#include <helpers/ota/OtaDeflate.h>
#include <cassert>
#include <cstdint>
#include <cstdlib>
#include <cstring>
#include <iomanip>
#include <iostream>
#include <sstream>
#include <string>
#include <vector>
using namespace mesh::ota;
static bool fail_allocation = false;
static void* active_allocation = nullptr;
static unsigned allocation_calls = 0;
extern "C" void* __real_calloc(size_t, size_t);
extern "C" void __real_free(void*);
extern "C" void* __wrap_calloc(size_t n, size_t size) {
++allocation_calls;
if (fail_allocation) return nullptr;
assert(!active_allocation);
active_allocation = __real_calloc(n, size);
return active_allocation;
}
extern "C" void __wrap_free(void* pointer) {
if (pointer && pointer == active_allocation) active_allocation = nullptr;
__real_free(pointer);
}
static std::vector<uint8_t> unhex(const std::string& hex) {
std::vector<uint8_t> bytes;
if (hex == "-") return bytes;
assert(hex.size() % 2 == 0);
for (size_t i = 0; i < hex.size(); i += 2)
bytes.push_back(static_cast<uint8_t>(std::stoul(hex.substr(i, 2), nullptr, 16)));
return bytes;
}
static std::string hex(const uint8_t* bytes, unsigned len) {
if (!len) return "-";
std::ostringstream out;
for (unsigned i = 0; i < len; ++i)
out << std::hex << std::setw(2) << std::setfill('0') << unsigned(bytes[i]);
return out.str();
}
static void invalid_arguments() {
uint8_t source[32] = {}, destination[32] = {};
uint16_t length = 0xBEEF;
assert(!ota_transport_deflate(nullptr, nullptr, 16, destination, 32, &length));
assert(length == 0);
length = 0xBEEF;
assert(!ota_transport_deflate(nullptr, source, 0, destination, 32, &length));
assert(length == 0);
length = 0xBEEF;
assert(!ota_transport_deflate(nullptr, source, 16, nullptr, 32, &length));
assert(length == 0);
length = 0xBEEF;
assert(!ota_transport_deflate(nullptr, source, 16, destination, 0, &length));
assert(length == 0);
assert(!ota_transport_deflate(nullptr, source, 16, destination, 32, nullptr));
length = 0xBEEF;
assert(!ota_transport_deflate(nullptr, source, 2049, destination, 32, &length));
assert(length == 0);
for (int displacement : {-8, 0, 1, 8}) {
uint8_t shared[128];
memset(shared, 0x71, sizeof(shared));
uint8_t expected[128]; memcpy(expected, shared, sizeof(shared));
length = 0xBEEF;
assert(!ota_transport_deflate(nullptr, shared + 32, 32,
shared + 32 + displacement, 32, &length));
assert(length == 0 && memcmp(expected, shared, sizeof(shared)) == 0);
}
assert(!active_allocation);
}
int main() {
std::string operation, input;
unsigned capacity = 0;
while (std::cin >> operation) {
if (operation == "invalid") {
invalid_arguments(); std::cout << "OK\n"; continue;
}
assert(std::cin >> input >> capacity);
assert(capacity <= 65535);
auto bytes = unhex(input);
assert(bytes.size() <= 65535);
std::vector<uint8_t> source(bytes.size() + 32, 0xC3);
if (!bytes.empty()) memcpy(source.data() + 16, bytes.data(), bytes.size());
const auto original = source;
std::vector<uint8_t> destination(capacity + 32, 0xA5);
uint16_t produced = 0xBEEF;
fail_allocation = operation == "oom";
allocation_calls = 0;
bool ok;
if (operation == "decode")
ok = ota_transport_inflate(nullptr, source.data() + 16, bytes.size(),
destination.data() + 16, capacity, &produced);
else
ok = ota_transport_deflate(nullptr, source.data() + 16, bytes.size(),
destination.data() + 16, capacity, &produced);
fail_allocation = false;
assert(!active_allocation);
assert(source == original);
assert(produced <= capacity);
if (!ok) assert(produced == 0);
if (operation == "oom") assert(!ok && allocation_calls > 0);
for (unsigned i = 0; i < 16; ++i) {
assert(destination[i] == 0xA5);
assert(destination[16 + capacity + i] == 0xA5);
}
std::cout << ok << ' ' << produced << ' '
<< hex(destination.data() + 16, produced) << '\n';
}
}
'''
MANAGER_RUNNER = r'''
#include <helpers/ota/OtaContext.h>
#include <helpers/ota/OtaByteIO.h>
#include <SHA256.h>
#include <cassert>
#include <deque>
#include <iostream>
#include <map>
#include <string>
#include <vector>
using namespace mesh::ota;
#define OTA_SELF_LEAVES_MAX 65536u
static std::vector<uint8_t> firmware;
static std::vector<size_t> allocations;
static void* self_malloc(size_t bytes) {
allocations.push_back(bytes);
return malloc(bytes);
}
namespace mesh { namespace ota {
bool ota_self_firmware(SelfFwInfo& info) {
info = SelfFwInfo(); info.valid = true; info.image_len = firmware.size();
info.target_id = 123; info.fw_version = 2;
strcpy(info.hw_id, "test");
return true;
}
bool ota_self_read(uint32_t off, uint8_t* buf, uint32_t len) {
if (uint64_t(off) + len > firmware.size()) return false;
memcpy(buf, firmware.data() + off, len);
return true;
}
static bool self_read_cb(void*, uint32_t off, uint8_t* buf, uint32_t len) {
return ota_self_read(off, buf, len);
}
#define malloc self_malloc
#include "self_serve.h"
#undef malloc
} }
struct Message { OtaManager* destination; std::vector<uint8_t> data; };
struct Route { OtaManager* destination; bool source; };
static std::deque<Message> messages;
static unsigned data_packets = 0, deflated_packets = 0, raw_v2_packets = 0, legacy_packets = 0;
static bool drop_one = false, dropped = false;
static std::map<unsigned, std::vector<uint8_t>> representations;
static bool send(void* context, const uint8_t* bytes, uint16_t len, bool) {
const Route& route = *static_cast<Route*>(context);
if (route.source && len && bytes[0] == OTA_DATA) {
++data_packets;
DataMsg data;
assert(decode_data(bytes, len, data));
if (data.frag_off & OTA_DATA_V2_MARK) {
uint8_t fragment; uint16_t length; bool deflated;
const unsigned raw_length = std::min<size_t>(2048, firmware.size() - data.block_idx*2048);
assert(ota_data_v2_unpack_extended(data.frag_off, raw_length, fragment, length, deflated));
assert(data.data_len > 4 && data.data_len <= 175);
const std::vector<uint8_t> identity(data.data, data.data + 4);
if (representations.count(data.block_idx)) assert(representations[data.block_idx] == identity);
else representations[data.block_idx] = identity;
if (deflated) ++deflated_packets; else ++raw_v2_packets;
if (drop_one && !dropped && data.block_idx == 0 && fragment == 1) {
dropped = true; return true;
}
} else {
++legacy_packets;
assert(data.data_len > 0 && data.data_len <= 160);
}
}
messages.push_back({route.destination, {bytes, bytes + len}});
return true;
}
static bool reject_encoder(void*, const uint8_t*, uint16_t, uint8_t* out,
uint16_t cap, uint16_t* len) {
if (cap) memset(out, 0x42, cap);
*len = 0; return false;
}
int main(int argc, char** argv) {
assert(argc == 2);
const std::string scenario = argv[1];
const unsigned tail = scenario == "tail1" ? 1 : scenario == "tail2" ? 2 : 360;
firmware.resize(2048 + tail);
uint32_t rng = 0xDEAD1234;
for (unsigned i = 0; i < firmware.size(); ++i) {
rng ^= rng << 13; rng ^= rng >> 17; rng ^= rng << 5;
firmware[i] = scenario == "incompressible" ? uint8_t(rng) :
scenario == "retry" ? uint8_t(rng & 15) : uint8_t(i % 32);
}
OtaContext source;
OtaManager receiver;
OtaStoreRam<8192> store;
Route to_receiver{&receiver, true}, to_source{&source.manager, false};
source.begin(123, send, &to_receiver, "test");
#if defined(OTA_SEEDER_ONLY) || defined(NRF52_PLATFORM)
// These fixture variants are raw diagnostic exports, not automatically
// offered full own-images: Companion source-only or internal-only nRF52.
assert(!source.self_serve_supported);
#else
assert(source.self_serve_supported);
#endif
allocations.clear();
assert(ota_serve_self(source, 2));
#if MESHCORE_OTA_DEVICE_DEFLATE
assert(allocations.size() == 2 && allocations[0] == 8 && allocations[1] == 2048);
#else
// No encoder means no extra output allocation. The original proof working
// area remains exactly block_count*4 even for a tiny two-block own-image.
assert(allocations.size() == 2 && allocations[0] == 8 && allocations[1] == 8);
#endif
if (scenario == "rejected") source.manager.set_transport_deflate_encoder(reject_encoder);
receiver.begin(123, send, &to_source);
receiver.set_accept_full(true);
receiver.set_fetch_store(&store);
if (scenario != "rawv2" && scenario != "legacy")
receiver.set_transport_deflate_decoder(ota_transport_inflate);
if (scenario == "legacy") receiver.set_wire_v2_enabled(false);
drop_one = scenario == "retry";
receiver.pull(source.serve_self_manifest + 20, 123);
for (unsigned step = 1; step <= 10000 && receiver.fetchState() != OtaManager::COMPLETE; ++step) {
source.manager.set_clock(step*20); receiver.set_clock(step*20);
source.manager.loop(); receiver.loop();
source.manager.serviceEgress(); receiver.serviceEgress();
unsigned delivered = 0;
while (!messages.empty()) {
assert(++delivered < 100);
Message message = std::move(messages.front()); messages.pop_front();
message.destination->on_message(message.data.data(), message.data.size());
}
assert(receiver.fetchState() != OtaManager::FAILED);
}
assert(receiver.fetchState() == OtaManager::COMPLETE);
MotaManifest staged;
assert(mota_parse(store.data(), store.staged_size(), staged));
assert(staged.payload_size == firmware.size());
assert(memcmp(staged.payload, firmware.data(), firmware.size()) == 0);
if (scenario == "legacy") {
assert(legacy_packets > 0 && !raw_v2_packets && !deflated_packets);
assert(data_packets == 13 + (tail + 159)/160);
} else if (scenario == "rawv2" || scenario == "incompressible" || scenario == "rejected" ||
scenario == "disabled") {
assert(raw_v2_packets > 0 && !legacy_packets && !deflated_packets);
assert(data_packets == 12 + (tail + 170)/171);
} else {
assert(deflated_packets > 0 && !legacy_packets);
// Deliberately lossy runs include retransmissions; compare clean geometry
// only for the loss-free cases, not actual TX counts with a dropped packet.
if (scenario != "retry") assert(data_packets < 12 + (tail + 170)/171);
if (tail < 3) assert(raw_v2_packets == 1);
if (scenario == "retry") assert(dropped);
}
std::cout << scenario << ": " << data_packets << " DATA, " << deflated_packets
<< " compressed; strict decoder + Merkle + original bytes OK\n";
}
'''
class DeviceDeflateEncoderTest(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.directory = tempfile.TemporaryDirectory(prefix="ota-device-deflate-")
cls.path = Path(cls.directory.name)
source = cls.path / "codec.cpp"
source.write_text(CODEC_RUNNER)
flags = [] if os.name == "nt" else ["-fsanitize=address,undefined", "-fno-omit-frame-pointer"]
tinf = cls.path / "tinf.o"
compiled = subprocess.run([
shutil.which("cc") or "gcc", "-std=c99", "-Os", *flags,
"-DOTA_TRANSPORT_DEFLATE_TEST=1", "-c",
str(ROOT / "src/helpers/ota/OtaTinf.c"), "-o", str(tinf),
], capture_output=True, text=True)
if compiled.returncode:
cls.directory.cleanup()
raise AssertionError(compiled.stderr)
cls.binary = cls.path / "codec.exe"
compiled = subprocess.run([
"c++", "-std=c++17", "-Os", *flags, "-DOTA_TRANSPORT_DEFLATE_TEST=1",
"-I", str(ROOT / "src"), str(source),
str(ROOT / "src/helpers/ota/OtaDeflate.cpp"), str(tinf),
"-Wl,--wrap=calloc", "-Wl,--wrap=free", "-o", str(cls.binary),
], capture_output=True, text=True)
if compiled.returncode:
cls.directory.cleanup()
raise AssertionError(compiled.stderr)
@classmethod
def tearDownClass(cls):
cls.directory.cleanup()
def exchange(self, requests):
lines = [f"{operation} {data.hex() or '-'} {capacity}"
for operation, data, capacity in requests]
result = subprocess.run([str(self.binary)], input="\n".join(lines) + "\n",
capture_output=True, text=True, timeout=30)
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
replies = result.stdout.splitlines()
self.assertEqual(len(replies), len(requests))
decoded = []
for reply in replies:
success, length, encoded = reply.split()
data = b"" if encoded == "-" else bytes.fromhex(encoded)
self.assertEqual(len(data), int(length))
decoded.append((success == "1", data))
return decoded
def roundtrips(self, blocks, require_savings=False):
encoded = self.exchange([("encode", block, len(block)) for block in blocks])
decode_requests, expected = [], []
for block, (success, stream) in zip(blocks, encoded):
if require_savings:
self.assertTrue(success, f"No saving for {len(block)} byte patterned block")
if not success:
self.assertEqual(stream, b"")
continue
self.assertLess(len(stream), len(block))
inflater = zlib.decompressobj(wbits=-15)
self.assertEqual(inflater.decompress(stream) + inflater.flush(), block)
self.assertTrue(inflater.eof)
self.assertEqual(inflater.unused_data, b"")
self.assertEqual(inflater.unconsumed_tail, b"")
decode_requests.append(("decode", stream, len(block)))
expected.append(block)
if decode_requests:
for block, (success, decoded) in zip(expected, self.exchange(decode_requests)):
self.assertTrue(success)
self.assertEqual(decoded, block)
return encoded
def test_firmware_like_patterns_roundtrip_both_existing_decoders(self):
patterns = [b"\x00" * 2048, b"\xff" * 2048,
(b"\x36\x41\x00\x22\xa1\xff\x0c\x3d" * 256),
(b"MeshCore repeater ota self target radio2 TempRadio\x00" * 48)[:2048],
bytes(range(256)) * 8,
(b"\x00\xff\x10\x00\xff\x20" * 342)[:2048]]
self.roundtrips(patterns, require_savings=True)
def test_all_small_and_fragment_boundary_tails_are_bounded(self):
sizes = list(range(0, 33)) + [170, 171, 172, 341, 342, 343, 359, 360, 361,
1023, 1024, 1025, 1880, 1881, 1882, 2047, 2048]
blocks = [(b"abcd" * 512)[:size] for size in sizes]
result = self.roundtrips(blocks)
for size, (success, encoded) in zip(sizes, result):
if size < 3:
self.assertFalse(success)
self.assertEqual(encoded, b"")
def test_random_and_low_entropy_blocks_roundtrip_or_fall_back(self):
rng = random.Random(0x90955055)
blocks = []
for length in [1, 2, 3, 17, 171, 360, 1024, 2048]:
for _ in range(8):
blocks.append(rng.randbytes(length))
blocks.append(bytes(rng.randrange(8) for _ in range(length)))
encoded = self.roundtrips(blocks)
self.assertGreater(sum(success for success, _ in encoded), 0)
self.assertGreater(sum(not success for success, _ in encoded), 0)
def test_output_capacity_exact_boundary_and_canaries(self):
rng = random.Random(4711)
block = bytes(rng.randrange(16) for _ in range(2048))
success, stream = self.roundtrips([block], require_savings=True)[0]
self.assertTrue(success)
caps = sorted({0, 1, 2, 3, 7, 8, 15, 16, 31, 32, 63, 64, 127, 128,
255, 256, len(stream)-1, len(stream), len(stream)+1, 2048, 4096})
for cap, (success, bounded) in zip(caps, self.exchange([
("encode", block, cap) for cap in caps])):
if cap < len(stream):
self.assertFalse(success)
self.assertEqual(bounded, b"")
else:
self.assertTrue(success)
self.assertEqual(bounded, stream)
def test_incompressible_full_block_returns_raw_fallback_without_heap_leak(self):
block = random.Random(12).randbytes(2048)
for success, stream in self.exchange([("encode", block, 2048)] * 32):
self.assertFalse(success)
self.assertEqual(stream, b"")
def test_deterministic_independent_blocks_and_allocation_failure_recovery(self):
first = (b"MeshCore\x00OTA\x00" * 256)[:2048]
second = (bytes(range(64)) * 32)
requests = [("encode", first, 2048), ("encode", second, 2048),
("oom", first, 2048), ("encode", first, 2048),
("encode", second, 2048), ("encode", first, 2048)]
result = self.exchange(requests)
self.assertEqual(result[0], result[3])
self.assertEqual(result[0], result[5])
self.assertEqual(result[1], result[4])
self.assertEqual(result[2], (False, b""))
for request, (success, stream) in zip(requests, result):
if success:
self.assertEqual(zlib.decompress(stream, wbits=-15), request[1])
def test_invalid_and_overlapping_arguments_fail_before_writing(self):
result = subprocess.run([str(self.binary)], input="invalid\n", capture_output=True,
text=True, timeout=30)
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
self.assertEqual(result.stdout, "OK\n")
def test_existing_decoder_stays_strict_for_encoded_streams(self):
block = b"\x00" * 2048
_, stream = self.roundtrips([block], require_savings=True)[0]
requests = [("decode", stream[:-1], 2048), ("decode", stream + b"\x00", 2048),
("decode", stream, 2047), ("decode", stream, 2049)]
for success, output in self.exchange(requests):
self.assertFalse(success)
self.assertEqual(output, b"")
def test_packet_savings_count_headers_and_final_tail(self):
def packets(length):
return (length + 170) // 171
self.assertEqual(packets(2048), 12)
self.assertEqual(packets(1882), 12)
self.assertEqual(packets(1881), 11)
self.assertEqual(packets(360), 3)
self.assertEqual(packets(342), 2)
self.assertEqual(packets(171), 1)
self.assertEqual(811 * packets(2048) + packets(360), 9735)
blocks = [b"\x00" * 2048, b"\xff" * 360]
for block, (success, stream) in zip(blocks, self.roundtrips(blocks, True)):
self.assertTrue(success)
self.assertGreater(packets(len(block)) - packets(len(stream)), 0)
self.assertGreater(len(block) + 15*packets(len(block)) -
len(stream) - 15*packets(len(stream)), 0)
class DeviceDeflateManagerTest(unittest.TestCase):
def compile_and_run(self, defines, scenarios):
with tempfile.TemporaryDirectory(prefix="ota-device-wire-") as directory:
path = Path(directory)
(path / "self_serve.h").write_text(method(
(ROOT / "src/helpers/ota/OtaSelf.cpp").read_text(),
"bool ota_serve_self(OtaContext& c, uint32_t fw_version)"))
source = path / "manager.cpp"
source.write_text(MANAGER_RUNNER)
flags = [] if os.name == "nt" else ["-fsanitize=address,undefined", "-fno-omit-frame-pointer"]
tinf = path / "tinf.o"
result = subprocess.run([
shutil.which("cc") or "gcc", "-std=c99", "-Os", *flags, *defines,
"-c", str(ROOT / "src/helpers/ota/OtaTinf.c"), "-o", str(tinf),
], capture_output=True, text=True)
self.assertEqual(result.returncode, 0, result.stderr)
binary = path / "manager.exe"
sources = ["OtaManager.cpp", "OtaProtocol.cpp", "MotaContainer.cpp",
"MerkleTree.cpp", "OtaDeflate.cpp"]
result = subprocess.run([
"c++", "-std=c++17", "-Os", *flags, *defines,
"-I", str(ROOT / "src"), "-I", str(ROOT / "test/mocks"),
str(source), *[str(ROOT / "src/helpers/ota" / name) for name in sources],
str(ROOT / "src/Utils.cpp"), str(tinf), "-o", str(binary),
], capture_output=True, text=True)
self.assertEqual(result.returncode, 0, result.stderr)
for scenario in scenarios:
with self.subTest(scenario=scenario):
result = subprocess.run([str(binary), scenario], capture_output=True,
text=True, timeout=30)
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
def test_real_self_source_negotiation_fallback_small_scratch_and_retry(self):
# Match the positively qualified production recipe: the common
# pre-script enables this flag uniformly for all translation units.
self.compile_and_run(["-DENABLE_OTA=1", "-DESP32_PLATFORM=1",
"-DMESHCORE_OTA_DEVICE_DEFLATE=1"],
("small", "tail1", "tail2", "rawv2", "legacy",
"incompressible", "rejected", "retry"))
def test_disabled_internal_nrf52_and_companion_keep_original_proof_allocation(self):
for defines in (
["-DENABLE_OTA=1", "-DESP32_PLATFORM=1", "-DMESHCORE_OTA_DEVICE_DEFLATE=0"],
["-DENABLE_OTA=1", "-DNRF52_PLATFORM=1"],
["-DENABLE_OTA=1", "-DESP32_PLATFORM=1", "-DOTA_SEEDER_ONLY=1"],
["-DENABLE_OTA=1", "-DNRF52_PLATFORM=1", "-DOTA_SEEDER_ONLY=1"],
):
with self.subTest(defines=defines):
self.compile_and_run(defines, ("disabled",))
if __name__ == "__main__":
unittest.main()
test/test_ota_device_deflate_role.py
+194
View File
@@ -0,0 +1,194 @@
"""Offline effective-source qualification; never invokes PlatformIO/hardware."""
import importlib.util
import configparser
from pathlib import Path
import re
from types import SimpleNamespace
import unittest
ROOT = Path(__file__).resolve().parents[1]
SPEC = importlib.util.spec_from_file_location(
"ota_device_deflate_role", ROOT / "scripts/ota_device_deflate_role.py")
POLICY = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(POLICY)
DEFINE = POLICY.DEFINITION
class DeviceDeflateRoleTests(unittest.TestCase):
def policy(self, **updates):
options = {
"platform": "espressif32", "mcu": "esp32s3",
"src_filter": "+<*.cpp> +<helpers/ota/*.cpp> +<../examples/simple_repeater>",
"flags": "-DENABLE_OTA=1 -DESP32_PLATFORM -DOTA_FLASH_STORE=1",
}
options.update(updates)
return POLICY.role_policy(**options)
def test_all_three_infrastructure_roles_on_classic_and_s3(self):
for role in ("simple_repeater", "simple_room_server", "simple_sensor"):
for mcu in ("esp32", "esp32s3", "esp32s2", "esp32c3", "esp32c6"):
with self.subTest(role=role, mcu=mcu):
self.assertEqual(self.policy(mcu=mcu, src_filter="+<../examples/" + role + "/*.cpp>"),
{DEFINE: 1})
def test_external_nrf52_and_adaptive_rak_are_compile_eligible(self):
for store in ("OTA_QSPI_STORE", "OTA_SD_STORE", "OTA_RAK_AUTO_STORE"):
with self.subTest(store=store):
self.assertEqual(self.policy(platform="nordicnrf52", mcu="nrf52840",
flags="-DENABLE_OTA -DNRF52_PLATFORM -D" + store),
{DEFINE: 1})
def test_internal_nrf52_and_non_ota_or_other_platforms_excluded(self):
for updates in (
{"platform": "nordicnrf52", "mcu": "nrf52840",
"flags": "-DENABLE_OTA -DNRF52_PLATFORM -DOTA_FLASH_STORE"},
{"platform": "ststm32", "mcu": "stm32f411"},
{"platform": "raspberrypi", "mcu": "rp2040"},
{"flags": "-DESP32_PLATFORM -UENABLE_OTA"},
{"flags": "-DESP32_PLATFORM -DENABLE_OTA -DDISABLE_LORA_OTA=1"},
{"platform": "espressif32", "mcu": "nrf52840"},
{"flags": "-DENABLE_OTA -DNRF52_PLATFORM -DOTA_QSPI_STORE"}):
with self.subTest(updates=updates):
self.assertEqual(self.policy(**updates), {})
def test_source_role_not_environment_names_selects_encoder(self):
for role in ("companion_radio", "simple_secure_chat", "kiss_modem",
"partition_expander", "esp32_partition_migrator",
"esp32_partition_legacy_seed", "unknown"):
with self.subTest(role=role):
self.assertEqual(self.policy(src_filter="+<../examples/" + role + "/*.cpp>"), {})
self.assertEqual(self.policy(src_filter="+<*>"), {})
def test_any_selected_excluded_role_beats_infrastructure_include(self):
for source in POLICY.EXCLUDED_SOURCES:
self.assertEqual(self.policy(
src_filter="+<../examples/simple_repeater> +<" + source + ">"), {})
def test_effective_ordered_exclusion_and_later_reinclude(self):
source = "../examples/simple_repeater"
self.assertEqual(self.policy(src_filter="+<" + source + "> -<" + source + ">"), {})
self.assertEqual(self.policy(src_filter="+<" + source + "> -<" + source
+ "> +<" + source + "/*.cpp>"), {DEFINE: 1})
self.assertEqual(self.policy(src_filter="+<../examples/simple_repeater> "
"+<../examples/companion_radio> "
"-<../examples/companion_radio>"), {DEFINE: 1})
def test_mixed_infrastructure_roles_fail_closed(self):
self.assertEqual(self.policy(src_filter="+<../examples/simple_repeater> "
"+<../examples/simple_room_server>"), {})
def test_companion_and_seeder_flags_exclude_even_misnamed_infrastructure(self):
for flag in ("OTA_SEEDER_ONLY=1", "OTA_SEEDER_ONLY=0", "COMPANION_RADIO_FULL=1",
"COMPANION_FEATURE_USB_MOTA_SOURCE=1"):
self.assertEqual(self.policy(flags="-DENABLE_OTA -DESP32_PLATFORM -D" + flag), {})
def test_explicit_boolean_opt_out_or_eligible_enable_is_preserved(self):
for flags in ("-D" + DEFINE + "=0", "-U" + DEFINE, "-D" + DEFINE + "=1"):
self.assertEqual(self.policy(flags="-DENABLE_OTA -DESP32_PLATFORM " + flags), {})
self.assertEqual(self.policy(cppdefines=[(DEFINE, 0)]), {})
self.assertEqual(self.policy(flags="-DENABLE_OTA -DESP32_PLATFORM -D" + DEFINE + "=0",
cppdefines=[(DEFINE, 1)]), {})
def test_unsafe_enable_and_nonboolean_overrides_are_rejected(self):
for source in ("companion_radio", "unknown", "kiss_modem"):
with self.subTest(source=source):
with self.assertRaises(ValueError):
self.policy(src_filter="+<../examples/" + source + ">",
flags="-DENABLE_OTA -DESP32_PLATFORM -D" + DEFINE + "=1")
for value in ("2", "-1", "true", "yes", "garbage"):
with self.assertRaises(ValueError):
self.policy(flags="-DENABLE_OTA -DESP32_PLATFORM -D" + DEFINE + "=" + value)
def test_unset_or_zero_storage_flags_do_not_enable_internal_nrf(self):
self.assertEqual(self.policy(platform="nordicnrf52", mcu="nrf52840",
flags="-DENABLE_OTA -DNRF52_PLATFORM -DOTA_QSPI_STORE=0"), {})
self.assertEqual(self.policy(platform="nordicnrf52", mcu="nrf52840",
flags="-DENABLE_OTA -DNRF52_PLATFORM -UOTA_QSPI_STORE"), {})
def test_flag_and_filter_lists_and_windows_slashes(self):
self.assertEqual(self.policy(flags=["-D ENABLE_OTA=1", "-D ESP32_PLATFORM"],
src_filter=["+<*.cpp>", "+<..\\examples\\simple_sensor\\*.cpp>"]),
{DEFINE: 1})
def test_install_uses_effective_filter_and_emits_one_common_definition(self):
class Env(dict):
def PioPlatform(self):
return SimpleNamespace(name="espressif32")
def BoardConfig(self):
return {"build.mcu": "esp32s3"}
def AppendUnique(self, **options):
self["appended"] = options
def GetProjectOption(self, name, default):
raise AssertionError("Effective SRC_FILTER must take precedence")
env = Env(SRC_FILTER="+<../examples/simple_sensor>",
BUILD_FLAGS="-DENABLE_OTA -DESP32_PLATFORM", CPPDEFINES=[])
POLICY.install(env)
self.assertEqual(env["appended"], {"CPPDEFINES": [(DEFINE, 1)]})
env["SRC_FILTER"] = ""
del env["appended"]
POLICY.install(env)
self.assertNotIn("appended", env)
def test_recipe_hooks_and_default_header_are_fail_closed(self):
ini = (ROOT / "platformio.ini").read_text(encoding="utf-8")
for name in ("esp32_base", "nrf52_base"):
section = ini.split("[" + name + "]", 1)[1].split("\n[", 1)[0]
self.assertIn("pre:scripts/ota_device_deflate_role.py", section)
header = (ROOT / "src/helpers/ota/OtaDeflateConfig.h").read_text(encoding="utf-8")
self.assertIn("#define MESHCORE_OTA_DEVICE_DEFLATE 0", header)
self.assertNotIn("defined(ESP32_PLATFORM)", header)
def test_real_representative_recipe_inheritance(self):
# Read the recipes without calling PlatformIO (which must remain
# single-process). Resolve only static section/option interpolation.
config = configparser.ConfigParser(interpolation=None, strict=False,
inline_comment_prefixes=(";",))
config.read([str(ROOT / "platformio.ini")]
+ [str(path) for path in sorted((ROOT / "variants").glob("*/platformio.ini"))],
encoding="utf-8")
def option(section, key):
if config.has_option(section, key):
raw = config.get(section, key)
else:
parents = config.get(section, "extends", fallback="").split(",")
raw = next((option(parent.strip(), key) for parent in parents
if parent.strip() and option(parent.strip(), key)), "")
return re.sub(r"\$\{([^}]+)\}",
lambda match: option(*match[1].rsplit(".", 1)), raw)
cases = (
("Heltec_v2_repeater", "esp32", True),
("heltec_v4_repeater", "esp32s3", True),
("heltec_v4_room_server", "esp32s3", True),
("Xiao_nrf52_repeater", "nrf52840", True),
("RAK_3401_repeater_rak13302_w25q16_lora_ota", "nrf52840", True),
("RAK_3401_repeater_unified_lora_ota", "nrf52840", True),
("RAK_3401_repeater_lora_ota_no_external_sensors", "nrf52840", False),
("heltec_v4_companion_radio_usb", "esp32s3", False),
("heltec_v4_companion_radio_ble", "esp32s3", False),
("heltec_v4_companion_radio_wifi_femon", "esp32s3", False),
("Xiao_nrf52_companion_radio_usb", "nrf52840", False),
("Xiao_nrf52_companion_radio_ble", "nrf52840", False),
("Xiao_nrf52_kiss_modem", "nrf52840", False),
)
for name, mcu, expected in cases:
with self.subTest(name=name):
section = "env:" + name
platform = option(section, "platform")
platform = "espressif32" if "espressif32" in platform else platform
result = POLICY.role_policy(platform, mcu, option(section, "build_src_filter"),
option(section, "build_flags"))
self.assertEqual(result, {DEFINE: 1} if expected else {})
self.assertIn("pre:scripts/ota_device_deflate_role.py",
option(section, "extra_scripts"))
if __name__ == "__main__":
unittest.main()
test/test_ota_self_metadata_cache.py
+282
View File
@@ -0,0 +1,282 @@
#!/usr/bin/env python3
"""Run the real ESP32 self-metadata accessor against host-only partition stubs."""
from pathlib import Path
import os
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
def method(text, signature):
"""Extract a real method, as in test_t096_full_memory, without ESP-IDF headers."""
start = text.index(signature)
end = text.index("{", start) + 1
depth = 1
while depth:
depth += (text[end] == "{") - (text[end] == "}")
end += 1
return text[start:end]
HARNESS = r'''
#include <helpers/ota/FirmwareInfo.h>
#include <helpers/ota/OtaByteIO.h>
#include <cassert>
#include <cstring>
#include <iostream>
#include <vector>
struct esp_partition_t { uint32_t address; uint32_t size; };
static esp_partition_t running = {0x10000, 0x400000};
static bool present = true;
static std::vector<uint8_t> flash;
static unsigned reads = 0;
static uint64_t bytes_read = 0;
static unsigned fail_call = 0;
static bool fail_trailer = false;
static bool corrupt_trailer_read = false;
constexpr int ESP_OK = 0;
const esp_partition_t* esp_ota_get_running_partition() {
return present ? &running : nullptr;
}
int esp_partition_read(const esp_partition_t* p, uint32_t off, void* dst, uint32_t len) {
++reads;
if (reads == fail_call || (fail_trailer && len == mesh::ota::ENDF_LEN)) return -1;
assert(p == &running && uint64_t(off) + len <= p->size);
assert(uint64_t(off) + len <= flash.size());
memcpy(dst, flash.data() + off, len);
if (corrupt_trailer_read && len == mesh::ota::ENDF_LEN) static_cast<uint8_t*>(dst)[0] = '?';
bytes_read += len;
return ESP_OK;
}
namespace mesh { namespace ota {
@ACCESSOR@
} }
using namespace mesh::ota;
static void image(uint32_t body, uint8_t tag) {
assert(uint64_t(body) + ENDF_LEN <= running.size);
flash.assign(running.size, 0xA5);
uint8_t* trailer = flash.data() + body;
memcpy(trailer, ENDF_MAGIC, 4);
wr_u32le(trailer + 4, body);
memset(trailer + 8, tag, 8);
wr_u32le(trailer + 16, 0x01110100u + tag);
wr_u32le(trailer + 20, 0xA0000000u + tag);
memset(trailer + 24, 'A' + tag, 32);
}
static void expect_info(const SelfFwInfo& info, uint32_t body, uint8_t tag) {
assert(info.valid && info.body_len == body && info.endf_offset == body);
assert(info.image_len == body + ENDF_LEN);
for (uint8_t value : info.body_hash) assert(value == tag);
assert(info.fw_version == 0x01110100u + tag && info.target_id == 0xA0000000u + tag);
for (unsigned i = 0; i < 32; ++i) assert(info.hw_id[i] == 'A' + tag);
assert(info.hw_id[32] == 0);
}
static void expect_empty(const SelfFwInfo& info) {
assert(!info.valid && !info.body_len && !info.image_len && !info.endf_offset);
assert(!info.fw_version && !info.target_id);
for (uint8_t value : info.body_hash) assert(value == 0);
for (char value : info.hw_id) assert(value == 0);
}
static void invalidate() {
present = false;
SelfFwInfo out;
assert(!ota_self_firmware(out));
expect_empty(out);
present = true;
}
int main(int argc, char** argv) {
assert(argc == 2);
const std::string which = argv[1];
SelfFwInfo out;
if (which == "production") {
// 812 logical 2-KiB OTA blocks, like the current full V4 image.
constexpr uint32_t body = 1661216;
image(body, 1);
assert(ota_self_firmware(out));
expect_info(out, body, 1);
assert(reads >= 3244 && reads <= 3250);
const unsigned initial_reads = reads;
const uint64_t initial_bytes = bytes_read;
for (unsigned i = 0; i < 50; ++i) {
out = SelfFwInfo();
assert(ota_self_firmware(out));
expect_info(out, body, 1);
}
assert(reads == initial_reads && bytes_read == initial_bytes);
std::cout << initial_reads << " partition reads, " << initial_bytes
<< " bytes on first scan; zero extra reads for 50 cached calls\n";
} else if (which == "address") {
image(4096, 1);
assert(ota_self_firmware(out));
const unsigned first = reads;
running.address = 0x410000;
image(8192, 2);
assert(ota_self_firmware(out));
expect_info(out, 8192, 2);
assert(reads > first);
const unsigned second = reads;
running.address = 0x10000;
image(1024, 3);
assert(ota_self_firmware(out));
expect_info(out, 1024, 3);
assert(reads > second);
} else if (which == "size") {
image(4096, 1);
assert(ota_self_firmware(out));
const unsigned first = reads;
running.size = 0x200000;
image(1024, 2);
assert(ota_self_firmware(out));
expect_info(out, 1024, 2);
assert(reads > first);
running.size = ENDF_LEN - 1;
assert(!ota_self_firmware(out));
expect_empty(out);
running.size = 0x200000;
image(512, 3);
assert(ota_self_firmware(out));
expect_info(out, 512, 3);
} else if (which == "null") {
image(4096, 1);
assert(ota_self_firmware(out));
const unsigned first = reads;
present = false;
assert(!ota_self_firmware(out));
expect_empty(out);
assert(reads == first);
present = true;
image(1024, 2);
assert(ota_self_firmware(out));
expect_info(out, 1024, 2);
assert(reads > first);
} else if (which == "read_failure") {
image(4096, 1);
fail_call = 1;
assert(!ota_self_firmware(out));
expect_empty(out);
fail_call = 0;
assert(ota_self_firmware(out));
expect_info(out, 4096, 1);
const unsigned complete = reads;
assert(ota_self_firmware(out) && reads == complete);
} else if (which == "trailer_failure") {
image(4096, 1);
fail_trailer = true;
assert(!ota_self_firmware(out));
expect_empty(out);
const unsigned first = reads;
assert(!ota_self_firmware(out));
expect_empty(out);
assert(reads == first * 2);
fail_trailer = false;
assert(ota_self_firmware(out));
expect_info(out, 4096, 1);
const unsigned complete = reads;
assert(ota_self_firmware(out) && reads == complete);
} else if (which == "invalid") {
running.size = 8192;
image(4096, 1);
wr_u32le(flash.data() + 4096 + 4, 4095); // Marker with the wrong absolute body length.
assert(!ota_self_firmware(out));
expect_empty(out);
const unsigned first = reads;
assert(!ota_self_firmware(out));
expect_empty(out);
assert(reads == first * 2);
image(4096, 2);
assert(ota_self_firmware(out));
expect_info(out, 4096, 2);
invalidate();
flash.assign(running.size, 0xA5); // No trailer at all remains retryable.
assert(!ota_self_firmware(out));
expect_empty(out);
image(1024, 3);
assert(ota_self_firmware(out));
expect_info(out, 1024, 3);
} else if (which == "reread_mismatch") {
image(4096, 1);
corrupt_trailer_read = true;
assert(!ota_self_firmware(out));
expect_empty(out);
const unsigned first = reads;
corrupt_trailer_read = false;
assert(ota_self_firmware(out));
expect_info(out, 4096, 1);
assert(reads > first);
} else if (which == "edges") {
// Exercise the overlap/chunk boundary and trailer ending at the partition boundary.
for (uint32_t body : {0u, 1u, 511u, 512u, 513u, 1023u, 4095u, 4096u}) {
invalidate();
running.size = body + ENDF_LEN;
image(body, 1);
assert(ota_self_firmware(out));
expect_info(out, body, 1);
const unsigned complete = reads;
assert(ota_self_firmware(out) && reads == complete);
}
} else {
assert(false && "unknown test case");
}
}
'''
class OtaSelfMetadataCacheTest(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.temp = tempfile.TemporaryDirectory(prefix="meshcore-self-metadata-")
cls.addClassCleanup(cls.temp.cleanup)
cls.binary = Path(cls.temp.name) / ("self_metadata.exe" if os.name == "nt" else "self_metadata")
accessor = method((ROOT / "src/helpers/ota/OtaSelf.cpp").read_text(encoding="utf-8"),
"bool ota_self_firmware(SelfFwInfo& out)")
source = HARNESS.replace("@ACCESSOR@", accessor)
built = subprocess.run(["c++", "-std=c++11", "-Wall", "-Wextra", "-Werror",
"-I", str(ROOT / "src"), "-x", "c++", "-", "-o", str(cls.binary)],
input=source, text=True, capture_output=True)
if built.returncode:
raise AssertionError(built.stderr)
def run_case(self, case):
result = subprocess.run([str(self.binary), case], text=True, capture_output=True)
self.assertEqual(result.returncode, 0, result.stderr)
return result.stdout
def test_production_sized_image_is_scanned_once(self):
self.assertIn("zero extra reads for 50 cached calls", self.run_case("production"))
def test_a_b_partition_address_changes_require_a_new_scan(self):
self.run_case("address")
def test_partition_size_change_and_shrink_invalidate_cache(self):
self.run_case("size")
def test_null_partition_clears_out_and_previous_success(self):
self.run_case("null")
def test_initial_read_failure_is_retryable(self):
self.run_case("read_failure")
def test_whole_trailer_read_failure_never_publishes_partial_info(self):
self.run_case("trailer_failure")
def test_invalid_or_missing_trailers_are_not_cached(self):
self.run_case("invalid")
def test_whole_trailer_reread_mismatch_is_not_cached(self):
self.run_case("reread_mismatch")
def test_chunk_overlap_and_partition_boundary(self):
self.run_case("edges")
if __name__ == "__main__":
unittest.main()
test/test_ota_self_serve_policy.py
+386
View File
@@ -0,0 +1,386 @@
#!/usr/bin/env python3
"""Exercise automatic self-serving eligibility without hardware or PlatformIO."""
from pathlib import Path
import os
import subprocess
import tempfile
import unittest
import test_ota_heap_context as heap_test
from test_t096_full_memory import method
ROOT = Path(__file__).resolve().parents[1]
class OtaSelfServePolicyTest(unittest.TestCase):
def compile_and_run(self, path, source, defines=(), scenarios=((),)):
binary = path / "self-serve-policy.exe"
sanitizers = [] if os.name == "nt" else ["-fsanitize=address,undefined"]
result = subprocess.run([
"c++", "-std=c++17", *sanitizers,
*["-D" + define for define in defines],
"-I", str(ROOT / "src"), str(source), "-o", str(binary),
], text=True, capture_output=True)
self.assertEqual(result.returncode, 0, result.stderr)
for scenario in scenarios:
result = subprocess.run([str(binary), *scenario], text=True,
capture_output=True)
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
def test_real_platform_and_storage_policy_matrix(self):
# The adaptive RAK flag takes precedence over a generic QSPI/SD flag:
# fitted storage is not usable until its application backend is matched.
matrix = [
((), False, False),
(("ESP32_PLATFORM=1",), True, True),
(("ESP32_PLATFORM=1", "OTA_FLASH_STORE=1"), True, True),
(("NRF52_PLATFORM=1",), False, False),
(("NRF52_PLATFORM=1", "OTA_FLASH_STORE=1"), False, False),
(("NRF52_PLATFORM=1", "OTA_QSPI_STORE=1"), True, True),
(("NRF52_PLATFORM=1", "OTA_SD_STORE=1"), True, True),
(("NRF52_PLATFORM=1", "OTA_RAK_AUTO_STORE=1"), False, True),
(("NRF52_PLATFORM=1", "OTA_RAK_AUTO_STORE=1",
"OTA_QSPI_STORE=1"), False, True),
(("NRF52_PLATFORM=1", "OTA_RAK_AUTO_STORE=1",
"OTA_SD_STORE=1"), False, True),
(("STM32_PLATFORM=1", "OTA_QSPI_STORE=1"), False, False),
(("RP2040_PLATFORM=1", "OTA_SD_STORE=1"), False, False),
]
# Source-only Companions must advertise only their host-provided files,
# even when their architecture and fitted storage would otherwise qualify.
matrix += [(defines + ("OTA_SEEDER_ONLY=1", "COMPANION_RADIO_FULL=1"),
False, False) for defines, _, _ in matrix if defines]
with tempfile.TemporaryDirectory(prefix="ota-self-policy-") as directory:
path = Path(directory)
source = path / "test.cpp"
source.write_text(r'''
#include <helpers/ota/OtaSelfServePolicy.h>
#include <cassert>
int main() {
using mesh::ota::ota_self_serve_supported;
assert(ota_self_serve_supported() == bool(EXPECT_INTERNAL));
assert(ota_self_serve_supported(false) == bool(EXPECT_INTERNAL));
assert(ota_self_serve_supported(true) == bool(EXPECT_EXTERNAL));
}
''')
for defines, internal, external in matrix:
with self.subTest(defines=defines):
self.compile_and_run(path, source, defines + (
"EXPECT_INTERNAL=" + str(int(internal)),
"EXPECT_EXTERNAL=" + str(int(external)),
))
def test_begin_latches_application_capability_before_backend_changes(self):
begin = method((ROOT / "src/helpers/ota/OtaContext.h").read_text(),
"void begin(uint32_t target_id, OtaSend send,")
with tempfile.TemporaryDirectory(prefix="ota-self-begin-") as directory:
path = Path(directory)
(path / "begin.h").write_text(begin)
source = path / "test.cpp"
source.write_text(r'''
#include <helpers/ota/OtaSelfServePolicy.h>
#include <cstdint>
#include <cassert>
#include <cstring>
using namespace mesh::ota;
typedef bool (*OtaSend)(void*, const uint8_t*, uint16_t, bool);
struct SelfFwInfo { bool valid = false; uint32_t target_id = 0, fw_version = 0; char hw_id[33] = {}; };
static bool ota_self_firmware(SelfFwInfo& info) { info = SelfFwInfo(); return false; }
static int ota_transport_inflate;
enum { CODEC_DETOOLS_INPLACE = 1, CODEC_DETOOLS_SEQUENTIAL = 2 };
struct Store {
bool external = false;
unsigned probes = 0;
bool usesExternal() { ++probes; return external; }
};
struct Manager {
static const uint8_t AUTOFETCH_OFF = 0;
bool accept_full = false;
void begin(uint32_t, OtaSend, void*) {}
void set_transport_deflate_decoder(int) {}
void set_auto_version_floor(uint32_t, bool) {}
void set_accept_full(bool value) { accept_full = value; }
void set_autofetch(uint8_t) {}
void set_apply_codec(int) {}
void set_apply_codec2(int) {}
void set_accept_bootloader(bool) {}
void set_fetch_store(Store*) {}
void set_archive_interest(bool) {}
};
using OtaManager = Manager;
struct Context {
Manager manager;
Store fetch_store;
bool self_serve_supported = false, fetch_to_folder = false;
static const uint8_t AUTOINSTALL_OFF = 0;
uint8_t autoinstall = AUTOINSTALL_OFF;
char hw_id[33] = {};
struct Cache { void attach(Store&) {} } sd_cache;
#include "begin.h"
};
static bool send(void*, const uint8_t*, uint16_t, bool) { return true; }
int main() {
Context c;
assert(!c.self_serve_supported);
c.begin(123, send, nullptr, "test");
assert(c.self_serve_supported == bool(EXPECT_INTERNAL));
// The adaptive store switches to internal flash for bootloader packages.
// Eligibility describes the application storage chosen at begin, not that
// transient fetch destination; there must be no live backend re-probe here.
c.fetch_store.external = true;
assert(c.self_serve_supported == bool(EXPECT_INTERNAL));
c.begin(123, send, nullptr, "test");
assert(c.self_serve_supported == bool(EXPECT_EXTERNAL));
unsigned probes = c.fetch_store.probes;
c.fetch_store.external = false;
assert(c.self_serve_supported == bool(EXPECT_EXTERNAL));
assert(c.fetch_store.probes == probes);
}
''')
for defines, internal, external in [
(("ESP32_PLATFORM=1",), True, True),
(("NRF52_PLATFORM=1", "OTA_FLASH_STORE=1"), False, False),
(("NRF52_PLATFORM=1", "OTA_QSPI_STORE=1"), True, True),
(("NRF52_PLATFORM=1", "OTA_SD_STORE=1"), True, True),
(("NRF52_PLATFORM=1", "OTA_RAK_AUTO_STORE=1"), False, True),
(("ESP32_PLATFORM=1", "OTA_SEEDER_ONLY=1"), False, False),
]:
with self.subTest(defines=defines):
self.compile_and_run(path, source, defines + (
"EXPECT_INTERNAL=" + str(int(internal)),
"EXPECT_EXTERNAL=" + str(int(external)),
))
def test_registration_and_temp_radio_service_keep_lazy_workspace(self):
context = (ROOT / "src/helpers/ota/OtaContext.h").read_text()
begin = method(context, "void begin(uint32_t target_id, OtaSend send,")
self.assertRegex(context, r"bool\s+self_serve_supported\s*=\s*false")
# Eligibility is cheap registration only. Scanning/hashing the image
# and claiming its leaf/proof buffers stay at the announcement boundary.
self.assertNotIn("ota_serve_self(", begin)
self.assertNotIn("malloc(", begin)
self.assertNotIn("ensureServeBuffer(", begin)
service = method(context, "inline void ota_service_temp_radio_context(")
with tempfile.TemporaryDirectory(prefix="ota-self-lazy-") as directory:
path = Path(directory)
(path / "service.h").write_text(service)
source = path / "test.cpp"
source.write_text(r'''
#include <cassert>
#include <cstddef>
static unsigned acquisitions = 0, releases = 0;
static bool ota_acquire_context(char*, size_t) { ++acquisitions; return false; }
static void ota_release_context_if_idle(bool temporary_radio_active) {
assert(!temporary_radio_active); ++releases;
}
#include "service.h"
int main() {
ota_service_temp_radio_context(false);
assert(acquisitions == 0 && releases == 1);
ota_service_temp_radio_context(true);
assert(acquisitions == 1 && releases == 1); // failed acquisition is soft
ota_service_temp_radio_context(false);
assert(acquisitions == 1 && releases == 2);
}
''')
self.compile_and_run(path, source)
for name in ("simple_repeater/MyMesh.cpp", "simple_room_server/MyMesh.cpp",
"simple_sensor/SensorMesh.cpp"):
with self.subTest(role=name):
role = (ROOT / "examples" / name).read_text()
self.assertIn("#if defined(ENABLE_OTA) && OTA_DYNAMIC_CONTEXT\n"
" mesh::ota::ota_service_temp_radio_context(isAnyTempRadioActive());",
role)
def test_automatic_self_init_is_gated_after_temp_radio_admission(self):
mesh = (ROOT / "src/Mesh.cpp").read_text()
auto_serve = "if (oc.self_serve_supported && !oc.serving)"
self.assertIn(auto_serve, mesh)
active = mesh.index("const bool ota_active = isAnyTempRadioActive();")
inactive = method(mesh[active:], "if (!ota_active)")
self.assertIn("return;", inactive)
self.assertLess(active + len(inactive), mesh.index(auto_serve))
cli = (ROOT / "src/helpers/ota/OtaCli.cpp").read_text()
for signature in ('else if (is_cmd(a, "announce|adv", &rest))',
'else if (is_cmd(a, "folder|fold", &rest))'):
with self.subTest(command=signature):
branch = method(cli, signature)
self.assertIn("c.self_serve_supported && !c.serving", branch)
self.assertIn("ota_serve_self(c, 0)", branch)
# Internal-only devices retain the explicit delta-base export diagnostic;
# Companion staging/export remains covered by its existing seeder guard.
dev = method(cli, 'else if (strncmp(d, "serve self", 10) == 0)')
self.assertIn("ota_serve_self(c, 0)", dev)
self.assertNotIn("self_serve_supported", dev)
def test_public_serve_command_checks_running_manifest_and_reports_failures(self):
cli = (ROOT / "src/helpers/ota/OtaCli.cpp").read_text()
match = method(cli, "static bool is_cmd(")
serve = method(cli, 'else if (is_cmd(a, "serve", &rest))').removeprefix("else ")
announce = method(cli, 'else if (is_cmd(a, "announce|adv", &rest))').removeprefix("else ")
with tempfile.TemporaryDirectory(prefix="ota-self-command-") as directory:
path = Path(directory)
(path / "command.h").write_text(
match + '\nstatic bool command(const char* a, char* reply, Context& c) {\n'
'const char* rest = a;\n' + serve + '\n' + announce +
'\nreturn true;\n}\n')
source = path / "test.cpp"
source.write_text(r'''
#include <helpers/ota/OtaByteIO.h>
#include <cassert>
#include <cstdio>
#include <cstring>
#include <initializer_list>
using namespace mesh::ota;
namespace mesh { struct Utils {
static void toHex(char* out, const uint8_t* bytes, size_t count) {
static const char hex[] = "0123456789abcdef";
for (size_t i = 0; i < count; ++i) {
out[i * 2] = hex[bytes[i] >> 4]; out[i * 2 + 1] = hex[bytes[i] & 15];
}
out[count * 2] = 0;
}
}; }
struct OtaManager {
struct ServeEntry { bool is_self = false; } entries[3];
uint8_t count = 0;
unsigned announcements = 0;
const uint8_t* primary_manifest = nullptr;
uint8_t servedCount() const { return count; }
const ServeEntry* servedEntry(uint8_t index) const {
return index < count ? entries + index : nullptr;
}
bool servingPrimaryManifest(const uint8_t* manifest) const {
return primary_manifest && primary_manifest == manifest;
}
void announce() { ++announcements; }
};
struct Context {
OtaManager manager;
bool self_serve_supported = false, serving = false;
uint8_t serve_self_manifest[197] = {};
// Source publication must not modify automatic installation or trust policy.
uint8_t autoinstall = 0, autofetch = 0, signer_count = 1;
};
static unsigned self_calls = 0;
static bool self_ok = true;
static bool ota_serve_self(Context& c, uint32_t) {
++self_calls;
if (!self_ok) return false;
c.serving = true;
c.manager.count = 1; c.manager.entries[0].is_self = true;
c.manager.primary_manifest = c.serve_self_manifest;
wr_u32le(c.serve_self_manifest + 11, 3000);
wr_u32le(c.serve_self_manifest + 20, 0x78563412);
return true;
}
#include "command.h"
int main() {
Context c;
char reply[160] = {}, plain[160] = {};
auto call = [&](const char* text) {
assert(command(text, reply, c));
assert(c.autoinstall == 0 && c.autofetch == 0 && c.signer_count == 1);
};
call("serve");
assert(strstr(reply, "default:off serving:off"));
assert(self_calls == 0 && c.manager.announcements == 0);
strcpy(plain, reply); call("serve status"); assert(!strcmp(plain, reply));
call("serve self");
assert(!strncmp(reply, "ERR", 3) && self_calls == 0);
c.self_serve_supported = true;
c.serving = true; c.manager.count = 2; c.manager.entries[0].is_self = true;
uint8_t manually_staged_manifest[197] = {};
c.manager.primary_manifest = manually_staged_manifest;
call("serve status");
assert(strstr(reply, "default:on serving:off")); // RAM primary is not own fw
c.serving = false; c.manager.primary_manifest = c.serve_self_manifest;
call("serve status");
assert(strstr(reply, "default:on serving:on")); // exact own manifest pointer
c.manager.count = 0; c.manager.primary_manifest = nullptr;
for (const char* bad : {"serve self extra", "serve selfx", "serve on", "serve status x"}) {
call(bad);
assert(!strncmp(reply, "ERR usage:", 10));
assert(self_calls == 0 && c.manager.announcements == 0);
}
self_ok = false;
call("serve self");
assert(!strncmp(reply, "ERR", 3) && self_calls == 1);
assert(c.manager.announcements == 0);
self_ok = true;
call("serve self");
assert(!strncmp(reply, "OK", 2) && self_calls == 2 && c.serving);
assert(strstr(reply, "mid=12345678") && strstr(reply, "3000 B"));
assert(strstr(reply, "flash-backed") && strstr(reply, "unsigned"));
assert(strstr(reply, "TempRadio") && c.manager.announcements == 1);
unsigned calls = self_calls;
call("serve status");
assert(strstr(reply, "serving:on") && self_calls == calls);
// An explicit announcement is still allowed on ineligible devices, but it
// must not implicitly initialize a full-image source there.
c.self_serve_supported = false; c.serving = false; c.manager.count = 0;
call("announce");
assert(self_calls == calls && c.manager.announcements == 2);
c.self_serve_supported = true;
call("adv");
#if defined(OTA_SEEDER_ONLY)
assert(self_calls == calls && !c.serving);
#else
assert(self_calls == calls + 1 && c.serving);
#endif
assert(c.manager.announcements == 3);
}
''')
for defines in ((), ("OTA_SEEDER_ONLY=1",)):
with self.subTest(defines=defines):
self.compile_and_run(path, source, defines)
def test_real_manager_distinguishes_ram_primary_from_running_image(self):
with tempfile.TemporaryDirectory(prefix="ota-self-primary-") as directory:
path = Path(directory)
source = path / "test.cpp"
(path / "vectors.h").write_text('#include "' +
(ROOT / "test/test_ota/mota_vectors.h").as_posix() + '"\n')
source.write_text(r'''
#include <helpers/ota/OtaContext.h>
#include <cassert>
#include <cstring>
#include "vectors.h"
using namespace mesh::ota;
namespace mesh { namespace ota {
bool ota_self_firmware(SelfFwInfo& info) { info = SelfFwInfo(); return false; }
} }
static bool send(void*, const uint8_t*, uint16_t, bool) { return true; }
static bool read(void*, uint32_t, uint8_t*, uint32_t) { return true; }
int main() {
OtaManager manager;
manager.begin(EXP_TARGET_ID, send, nullptr);
MotaManifest parsed;
assert(mota_parse(MOTA_VEC, MOTA_VEC_LEN, parsed));
uint8_t own_manifest[MOTA_MFL], proof[4096];
memcpy(own_manifest, parsed.manifest_start, sizeof(own_manifest));
assert(!manager.servingPrimaryManifest(own_manifest));
assert(manager.serve(MOTA_VEC, MOTA_VEC_LEN));
assert(manager.servedCount() == 1 && manager.servedEntry(0)->is_self);
// Both primaries have identical manifest bytes and is_self, but only one
// points into the context's own running-image manifest buffer.
assert(manager.servingPrimaryManifest(parsed.manifest_start));
assert(!manager.servingPrimaryManifest(own_manifest));
assert(manager.serve_self(own_manifest, sizeof(own_manifest), parsed.leaves,
parsed.block_count, proof, sizeof(proof), read, nullptr));
assert(manager.servedCount() == 1 && manager.servedEntry(0)->is_self);
assert(manager.servingPrimaryManifest(own_manifest));
assert(!manager.servingPrimaryManifest(parsed.manifest_start));
manager.clear_primary();
assert(!manager.servingPrimaryManifest(own_manifest));
assert(!manager.servingPrimaryManifest(parsed.manifest_start));
}
''')
# This helper compiles the real manager/container/context into a
# native fixture, never invokes PlatformIO or hardware interfaces.
heap_test.OtaHeapTest().compile_and_run(path, source)
if __name__ == "__main__":
unittest.main()
test/test_usb_host_sleep_policy.py
+94
View File
@@ -0,0 +1,94 @@
#!/usr/bin/env python3
"""Execute the pure native USB host-loss sleep policy without Arduino/USB."""
from pathlib import Path
import os
import shutil
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
HARNESS = r'''
#include <helpers/UsbHostSleepPolicy.h>
#include <iostream>
#include <stdexcept>
static void require(bool condition, const char* why) {
if (!condition) throw std::runtime_error(why);
}
int main() {
try {
constexpr uint32_t grace = 120000U;
for (uint32_t now : {0U, 1U, 120001U, UINT32_MAX}) {
mesh::UsbHostSleepPolicy cold;
cold.observe(false, now);
require(!cold.shouldKeepAwake(now, grace), "never-seen host armed grace");
}
for (uint32_t start : {0U, 100U, UINT32_MAX - 60000U}) {
mesh::UsbHostSleepPolicy policy;
policy.observe(true, start);
require(policy.shouldKeepAwake(start + grace + 1U, grace),
"present host was allowed to sleep");
policy.observe(false, start + 6U);
require(policy.shouldKeepAwake(start + 6U, grace), "five-ms SOF loss lost grace");
// Negative observations are not permission to renew the loss deadline.
policy.observe(false, start + grace - 1U);
require(policy.shouldKeepAwake(start + grace - 1U, grace), "grace ended early");
policy.observe(false, start + grace);
require(!policy.shouldKeepAwake(start + grace, grace), "grace never expired");
policy.observe(false, start + 1U);
require(!policy.shouldKeepAwake(start + 1U, grace), "old host revived after wrap");
// A new actual host cancels absence and arms a fresh bounded deadline.
const uint32_t returned_at = start + grace + 10U;
policy.observe(true, returned_at);
require(policy.shouldKeepAwake(returned_at, grace), "reconnect remained asleep");
policy.observe(true, returned_at + 500U);
policy.observe(false, returned_at + 506U);
require(policy.shouldKeepAwake(returned_at + 500U + grace - 1U, grace),
"last positive observation did not refresh grace");
require(!policy.shouldKeepAwake(returned_at + 500U + grace, grace),
"renewed grace was unbounded");
}
mesh::UsbHostSleepPolicy configurable;
configurable.observe(true, 200U);
configurable.observe(false, 201U);
require(configurable.shouldKeepAwake(1199U, 1000U), "custom grace ended early");
require(!configurable.shouldKeepAwake(1200U, 1000U), "custom grace never expired");
configurable.observe(true, 3000U);
require(configurable.shouldKeepAwake(3000U, 0U), "zero grace hid a live host");
configurable.observe(false, 3000U);
require(!configurable.shouldKeepAwake(3000U, 0U), "zero grace did not disable hold");
} catch (const std::exception& e) {
std::cerr << e.what() << '\n';
return 1;
}
}
'''
class UsbHostSleepPolicyTest(unittest.TestCase):
def test_host_loss_reconnect_expiry_and_clock_rollover(self):
compiler = os.environ.get("CXX", "c++")
self.assertIsNotNone(shutil.which(compiler), "a C++17 compiler is required")
with tempfile.TemporaryDirectory(prefix="meshcore-usb-host-policy-") as temp:
cpp = Path(temp) / "policy.cpp"
binary = Path(temp) / "policy"
cpp.write_text(HARNESS)
compiled = subprocess.run([
compiler, "-std=c++17", "-Wall", "-Wextra", "-Werror",
"-I", str(ROOT / "src"), str(cpp), "-o", str(binary),
], capture_output=True, text=True)
self.assertEqual(compiled.returncode, 0, compiled.stderr)
result = subprocess.run([str(binary)], capture_output=True, text=True)
self.assertEqual(result.returncode, 0, result.stderr)
if __name__ == "__main__":
unittest.main()
tools/hil/esp32_self_serve_ota.py
+747
View File
@@ -0,0 +1,747 @@
#!/usr/bin/env python3
"""Two Heltec V4 own-flash LoRa OTA test, without a host firmware seeder.
Run on the Linux hardware gateway after flashing both identified boards with
the same lab-default repeater image. This tool never erases a device, supplies
an image over USB, or opens a host .mota file. USB carries CLI control only.
Example:
python3 esp32_self_serve_ota.py run --prepare \
--serial-a 441BF669CF98 --serial-b 441BF669C9C0 \
--expected-body-hash16 0123456789ABCDEF --report /tmp/self-ota/report.json
The three default rounds cover source reads from both running A/B slots and
receiver installs in both directions. Private keys are read locally solely to
compare SHA-256 digests: raw replies and raw key bytes are never logged.
"""
import argparse
from datetime import datetime
import hashlib
import json
import os
from pathlib import Path
import re
import sys
import tempfile
import time
from s3_memory_soak import Device
LAB_RADIO = "909.5,500,5,5"
LAB_SECONDARY = "909.5,500,6,5,rx"
BASELINE_MODE = "normal; no temporary radio profiles"
HEX16 = re.compile(r"^[0-9A-Fa-f]{16}$")
SNAPSHOT_COMMANDS = (
"get name", "get radio", "get radio2", "get radio2.cross",
"get tx.reply", "get tx", "get radio.rxps.config", "get radio.rxgain",
"get radio.fem.txgain", "get radio.fem.rxgain", "get advert.interval",
"get flood.advert.interval", "get lat", "get lon",
)
class TestFailure(RuntimeError):
"""A bounded, non-secret failure description suitable for the report."""
def require(condition, message):
if not condition:
raise TestFailure(message)
def utc():
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
def normalized_serial(value):
return re.sub(r"[^A-Za-z0-9]", "", str(value or "")).upper()
def command_label(command):
# Diagnostic verbs/settings only, never arguments, keys or raw replies.
words = command.split()
label = words[0]
if words[0] in ("get", "set", "ota") and len(words) > 1 \
and re.fullmatch(r"[a-z][a-z0-9.]*", words[1]):
label += " " + words[1]
return label
def reply_text(raw):
# Drop command echoes and unsolicited startup/debug output. Only the CLI
# reply suffix is used, and callers never retain the private-key suffix.
match = re.search(r"(?:^|\n)\s*->\s*", raw)
require(match is not None, "CLI reply marker was not received")
return raw[match.end():].replace("\r", "").strip()
def value_reply(raw):
text = reply_text(raw)
require(bool(text), "CLI reply was empty")
first = text.splitlines()[0].strip()
return re.sub(r"^>\s*", "", first)
def key_token(raw, digits):
# Do not interpolate raw/text into any failure message or exception.
text = value_reply(raw)
require(re.fullmatch(r"[0-9A-Fa-f]{%d}" % digits, text) is not None,
"Local identity reply has an invalid bounded key length")
return text.upper()
def parse_self(raw):
text = value_reply(raw)
match = re.search(r"self body=(\d+) image=(\d+) base_hash=([0-9A-Fa-f]{16})\b", text)
require(match is not None, "Running image lacks a valid OTA self identity")
return {"body_bytes": int(match[1]), "image_bytes": int(match[2]),
"body_hash16": match[3].upper()}
def parse_slot(raw):
text = value_reply(raw)
match = re.search(r"inactive slot addr=0x([0-9A-Fa-f]+) size=(\d+)\b", text)
require(match is not None, "Board does not expose an inactive A/B app slot")
return {"inactive_address": int(match[1], 16), "size_bytes": int(match[2])}
def saved_secondary(text):
# get radio2 appends measured timing/recommended-preamble diagnostics. An
# automatic preamble is recomputed at each boot, not a saved setting; do
# not mistake a different self-test result for changed configuration.
if text == "off":
return {"mode": "off"}
match = re.match(r"^([0-9.]+),([0-9.]+),(\d+),(\d+),(rx|rxtx),(\d+)(\s+\(auto\))?(?:;|$)", text)
require(match is not None, "Saved radio2 reply has invalid profile fields")
return {"frequency_mhz": float(match[1]), "bandwidth_khz": float(match[2]),
"sf": int(match[3]), "cr": int(match[4]), "mode": match[5],
"preamble": "auto" if match[7] else int(match[6])}
def lab_profile_matches(text, secondary=False):
prefix = r"^909\.500,500\.000,5,5,rxtx," if secondary else r"^909\.500,500\.000,5,5,"
return re.match(prefix, text) is not None
def require_lab_profile(text, secondary=False):
require(lab_profile_matches(text, secondary),
"Active temporary radio profile does not match the lab frequency/modem tuple")
def parse_progress(raw, mid):
text = value_reply(raw)
match = re.search(r"download:\s*(.*?)\s+(\d+)/(\d+)\s+.*?id=([0-9A-Fa-f]{8})\b", text)
if not match:
require("no download" not in text, "Receive session unexpectedly became idle")
raise TestFailure("OTA status lacks complete download progress fields")
require(match[4].upper() == mid, "Receive session selected a different manifest")
state, done, total = match[1], int(match[2]), int(match[3])
require(total >= 0 and 0 <= done <= total, "Invalid OTA block progress")
require(total > 0 or state == "starting", "Non-starting download has zero blocks")
require(not state.startswith("failed"), "Firmware reported a failed OTA download")
return {"state": state, "done": done, "total": total, "status": text}
def find_port(serial_number):
from serial.tools import list_ports
matches = [p for p in list_ports.comports()
if normalized_serial(p.serial_number) == serial_number]
require(len(matches) == 1, "Expected exactly one matching physical USB serial")
port = matches[0]
require(port.vid == 0x303A, "Matched USB device does not have ESP32 VID 303A")
return port
class V4Device(Device):
"""Reuse the soak transport, with held native-HWCDC DTR on Linux as well."""
def __init__(self, name, serial_number):
self.serial_number = normalized_serial(serial_number)
require(bool(self.serial_number), "A nonempty physical USB serial is required")
super().__init__(name, "", "Heltec V4")
self.is_serial = True
self.identity = None
def connect(self):
if self.stream:
return
import serial
port = find_port(self.serial_number)
self.endpoint = port.device
stream = serial.Serial()
stream.port, stream.baudrate, stream.timeout = self.endpoint, 115200, .05
stream.write_timeout = 5
stream.dtr, stream.rts = True, False
stream.open()
self.stream = stream
self.connections += 1
self.read(time.monotonic() + 1)
board = value_reply(self._command("board"))
require(re.search(r"Heltec\s+V4\b", board, re.I) is not None,
"USB identity is not a Heltec V4")
role = value_reply(self._command("get role"))
require(role.lower() == "repeater", "Board firmware role is not Repeater")
self.identity = {"usb_serial": self.serial_number, "vid": port.vid,
"pid": port.pid, "board": board, "role": role,
"endpoint": self.endpoint}
def _command(self, command, timeout=5):
# A fresh bounded request cannot consume a stale previous CLI reply.
self.read(time.monotonic() + .05)
self.write(command.encode("ascii") + b"\r")
deadline = time.monotonic() + timeout
result, complete = "", False
while time.monotonic() < deadline:
# A debug line may contain an embedded arrow; only the CLI marker
# at line start counts. USB can split the marker and reply itself.
result += self.read(min(deadline, time.monotonic() + .1))
marker = re.search(r"(?:^|\n)[ \t]*->[ \t]*", result)
if marker and re.match(r"[^\r\n]+(?:\r\n|\n|\r)", result[marker.end():]):
complete = True
break
if complete:
# ACL is a multi-line, maximum-160-byte reply. Wait for the tail,
# not merely the marker (USB may split key bytes across packets).
result += self.read(min(deadline, time.monotonic() + .35))
elif re.search(r"(?:^|\n)[ \t]*->[ \t]*", result):
raise TestFailure(self.name + ": CLI reply was incomplete for " + command_label(command))
else:
raise TestFailure(self.name + ": CLI reply marker was not received for "
+ command_label(command))
return result
def write(self, data):
require(self.stream.write(data) == len(data), "USB command write was incomplete")
self.stream.flush()
def checked(self, command, timeout=5):
text = reply_text(self.command(command, timeout))
require(not re.match(r"(?:ERR|Error|Err)\b", text, re.I),
self.name + ": command rejected: " + command.split()[0])
# Only bounded protocol rows can extend a multi-line CLI reply. Never
# retain unrelated asynchronous logs in our JSON report.
lines = text.splitlines()
if command.startswith("get acl "):
rows = [line for line in lines[1:]
if re.fullmatch(r"[0-9A-Fa-f]{2} [0-9A-Fa-f]{64}", line)]
return "\n".join([lines[0]] + rows[:2])
if command == "ota ls" or command.startswith("ota ls "):
rows = [line for line in lines[1:]
if re.match(r"^\s+\d+\) [0-9A-Fa-f]{8} ", line)]
return "\n".join([lines[0]] + rows[:2])
return lines[0]
def reconnect(self, expected_hash, timeout=90):
self.close()
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
self.connect()
info = parse_self(self.command("ota self"))
require(info["body_hash16"] == expected_hash,
"Running firmware hash changed after reconnect")
return info
except TestFailure as error:
self.close()
# Missing enumeration/startup responses can recover. Wrong
# board/role/hash is a safety failure, never a retry candidate.
if "exactly one" not in str(error) and "marker" not in str(error):
raise
except (OSError, IOError):
self.close()
time.sleep(1)
raise TestFailure(self.name + ": exact-serial reconnect timed out")
def reboot(self, expected_hash):
with self.lock:
self.connect()
self.write(b"reboot\r") # This command intentionally has no reply.
self.close()
time.sleep(3)
self.reconnect(expected_hash)
class Report:
def __init__(self, path, args):
self.path = Path(path)
require(not self.path.exists(), "Report path already exists; select a new report")
self.path.parent.mkdir(parents=True, exist_ok=True)
self.data = {"schema": 1, "started_utc": utc(), "result": "RUNNING",
"action": args.action, "lab_radio": LAB_RADIO,
"expected_body_hash16": args.expected_body_hash16.upper(),
"requested_rounds": args.rounds, "devices": {}, "rounds": [],
"first_source": getattr(args, "start_source", "a"),
"observer_timeout_seconds": args.timeout,
"observer_poll_interval_seconds": args.poll_interval,
"events": [], "firmware_transport": "LoRa, own running app flash",
"host_firmware_payload_supplied": False}
self.save()
def save(self):
# Owner-only report and atomic replace; no secrets or raw CLI logs.
fd, temporary = tempfile.mkstemp(prefix=self.path.name + ".", suffix=".tmp",
dir=self.path.parent)
temp = Path(temporary)
try:
with os.fdopen(fd, "w", encoding="utf-8") as stream:
json.dump(self.data, stream, indent=2)
stream.write("\n")
stream.flush()
os.fsync(stream.fileno())
os.replace(temp, self.path)
finally:
if temp.exists():
temp.unlink()
def event(self, text):
entry = {"utc": utc(), "message": text}
self.data["events"].append(entry)
self.save()
print(entry["utc"] + " " + text, flush=True)
def acl_snapshot(device):
rows, pages = [], 1
for page in range(1, 129):
text = device.checked("get acl " + str(page))
if page == 1 and text == "ACL: empty":
return []
match = re.match(r"ACL (\d+)/(\d+)\s*", text)
require(match is not None and int(match[1]) == page,
"ACL snapshot returned invalid pagination")
pages = int(match[2])
require(1 <= pages <= 128, "ACL snapshot page count exceeded bound")
current = re.findall(r"(?:^|\n)([0-9A-Fa-f]{2}) ([0-9A-Fa-f]{64})(?=\n|$)", text)
require(1 <= len(current) <= 2, "ACL page contained truncated or missing rows")
require(page == pages or len(current) == 2, "Non-final ACL page lost a complete row")
rows.extend({"permissions": int(perm, 16), "public_key": key.upper()}
for perm, key in current)
if page == pages:
return sorted(rows, key=lambda row: (row["public_key"], row["permissions"]))
raise TestFailure("ACL snapshot did not finish")
def snapshot(device):
private_hex = key_token(device.command("get prv.key"), 128)
private_digest = hashlib.sha256(bytes.fromhex(private_hex)).hexdigest()
del private_hex
data = {"private_key_sha256": private_digest,
"public_key": key_token(device.command("get public.key"), 64),
"acl": acl_snapshot(device), "settings": {}}
for command in SNAPSHOT_COMMANDS:
# Unsupported FEM controls are legitimate on boards without a switch;
# record their bounded first-line response and require it stays equal.
text = value_reply(device.command(command))
if command not in ("get radio.fem.txgain", "get radio.fem.rxgain"):
require(not re.match(r"(?:ERR|Error|Err)\b", text, re.I),
"Required saved-setting snapshot command was rejected")
data["settings"][command[4:]] = saved_secondary(text) if command == "get radio2" else text
return data
def preflight(devices, expected_hash, report):
for device in devices:
device.connect()
info = parse_self(device.command("ota self"))
require(info["body_hash16"] == expected_hash,
device.name + ": unexpected running firmware body hash")
require(info["image_bytes"] > info["body_bytes"] > 0,
"Running image geometry is invalid")
slot = parse_slot(device.command("ota dev apply slot"))
require(info["image_bytes"] <= slot["size_bytes"], "Own image will not fit the A/B slot")
report.data["devices"][device.name] = {"identity": device.identity,
"self": info, "initial_slot": slot}
report.save()
require(devices[0].serial_number != devices[1].serial_number,
"The two radios must have different physical USB serials")
public_keys = [key_token(d.command("get public.key"), 64) for d in devices]
require(public_keys[0] != public_keys[1], "Radios have duplicated mesh identities")
report.event("Both exact USB identities, V4 repeater roles, and image hashes verified")
return public_keys
def prepare(devices, public_keys, expected_hash, report):
for index, device in enumerate(devices):
commands = (
"set name SelfOTA_" + device.name,
"setperm " + public_keys[1-index] + " 1",
"set radio " + LAB_RADIO,
"set radio2 " + LAB_SECONDARY,
"set radio2.cross off", "set tx.reply auto", "set tx -5",
"ota config autofetch off", "ota config autoinstall off", "ota config hops 0",
)
for command in commands:
report.event("Preparing " + device.name + ": " + command_label(command))
# SPIFFS transactions and dual-radio timing calibration can exceed
# the short read-only CLI deadline. Do not resend a timed-out write.
text = device.checked(command, timeout=30)
require(text.startswith("OK"), device.name + ": preparation was not acknowledged")
report.event("Lab settings, distinct names, and nonempty ACLs seeded; allowing lazy writes")
time.sleep(6)
for device in devices:
device.reboot(expected_hash)
report.event("Both boards rebooted after saved-setting persistence")
def arm_lab(devices):
for device in devices:
# The secondary CLI accepts only rx/rxtx tuples; bare tempradio2 off
# restores the saved SF6 profile. Overlay the exact same primary lab
# tuple instead, leaving the saved secondary configuration untouched.
# Use rxtx because tx.reply auto answers on the receive profile and
# cross is deliberately off; an rx-only overlay would drop replies.
commands = ("ota config autofetch off", "ota config autoinstall off", "ota config hops 0",
"set tempradio2 " + LAB_RADIO + ",rxtx,120", "tempradio " + LAB_RADIO + ",120")
for command in commands:
require(device.checked(command, timeout=30).startswith("OK"),
"Lab maintenance/policy command was not acknowledged")
# Both commands intentionally schedule changes. Wait for published live
# profiles, rather than treating a fixed delay as evidence of activation.
for device in devices:
deadline = time.monotonic() + 30
while True:
remaining = deadline - time.monotonic()
require(remaining > 0, device.name + ": temporary lab profiles were not activated in time")
primary = value_reply(device.command("get tempradio", timeout=remaining))
remaining = deadline - time.monotonic()
require(remaining > 0, device.name + ": temporary lab profiles were not activated in time")
secondary = value_reply(device.command("get tempradio2", timeout=remaining))
require(time.monotonic() < deadline,
device.name + ": temporary lab profiles were not activated in time")
if lab_profile_matches(primary) and lab_profile_matches(secondary, True):
break
require(time.monotonic() < deadline,
device.name + ": temporary lab profiles were not activated in time")
time.sleep(.5)
config = device.checked("ota config")
require("autofetch=off" in config and "autoinstall=off" in config and "hops=0" in config,
"Manual-only one-hop OTA policy was not applied")
def require_normal_profiles(devices):
for device in devices:
require(value_reply(device.command("get tempradio")) == "off"
and value_reply(device.command("get tempradio2")) == "off",
device.name + ": saved-setting snapshot requires inactive temporary profiles")
def disarm_lab(devices):
# Getters for advert intervals describe EFFECTIVE TempRadio timing, not
# saved preferences. Take the baseline with both temporary profiles off.
# Qualify every receiver as idle before clearing any maintenance window.
for device in devices:
require(" | no download | " in value_reply(device.command("ota status")),
device.name + ": cannot disarm a live OTA download for a baseline")
for device in devices:
for command in ("normalradio", "set tempradio2 off"):
require(device.checked(command, timeout=30).startswith("OK"),
"Normal-mode baseline command was not acknowledged")
deadline = time.monotonic() + 30
for device in devices:
while True:
remaining = deadline - time.monotonic()
require(remaining > 0, "Temporary profiles did not stop before the baseline deadline")
primary = value_reply(device.command("get tempradio", timeout=min(5, remaining)))
remaining = deadline - time.monotonic()
require(remaining > 0, "Temporary profiles did not stop before the baseline deadline")
secondary = value_reply(device.command("get tempradio2", timeout=min(5, remaining)))
require(time.monotonic() < deadline, "Temporary profiles did not stop before the baseline deadline")
if primary == secondary == "off":
break
time.sleep(.5)
def live_snapshot_matches_saved(original, current):
# Our fixed 120-minute window forces ONLY the local advert getter to60.
# Do not stop a live RF transfer to read saved timing. Defer that one field
# until the post-install normal-mode snapshot; keep every other field strict.
if current["settings"].get("advert.interval") != "60":
return False
for key in original:
if key == "settings":
old = {k: v for k, v in original[key].items() if k != "advert.interval"}
new = {k: v for k, v in current[key].items() if k != "advert.interval"}
if old != new:
return False
elif original[key] != current.get(key):
return False
return set(original) == set(current)
def discover(receiver, mid, timeout=60):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
first = receiver.checked("ota ls")
if first.startswith("No updates seen yet"):
time.sleep(2)
continue
header = re.search(r"Updates 1/(\d+)\b", first)
require(header is not None, "Receiver catalog has an invalid page header")
pages = int(header[1])
require(1 <= pages <= 128, "Receiver catalog exceeds page bound")
catalog = [first]
catalog.extend(receiver.checked("ota ls " + str(page)) for page in range(2, pages+1))
if any(re.search(r"\b" + mid + r"\b", row, re.I) for row in catalog):
return catalog
time.sleep(2)
raise TestFailure("Receiver did not discover the source's own-flash manifest over LoRa")
def run_round(number, source, receiver, devices, expected_hash, baseline, args, report,
continuing=None):
entry = continuing or {"round": number, "source": source.name, "receiver": receiver.name,
"started_utc": utc(), "result": "RUNNING", "progress": []}
if continuing is None:
report.data["rounds"].append(entry)
report.save()
arm_lab(devices)
source_info = parse_self(source.command("ota self"))
require(source_info["body_hash16"] == expected_hash, "Source hash is not the expected build")
source_slot = parse_slot(source.command("ota dev apply slot"))
receiver_slot = parse_slot(receiver.command("ota dev apply slot"))
if continuing is not None:
# Adopt only an already-started download. Never reissue cancel/get or
# install based solely on a saved report; verify the live session and
# physical devices before observing it with a larger bounded deadline.
require(entry.get("source") == source.name and entry.get("receiver") == receiver.name,
"Continuation device direction does not match")
require("install_reply" not in entry and "install_requested_utc" not in entry
and entry.get("result") == "RUNNING",
"Continuation cannot adopt a completed or installing round")
require(entry.get("source_slot") == source_slot and entry.get("receiver_slot_before") == receiver_slot,
"Application slots changed before continuation")
require(entry.get("source_self") == source_info, "Source image changed before continuation")
mid = entry.get("manifest_id", "")
require(re.fullmatch(r"[0-9A-F]{8}", mid) is not None, "Continuation has no valid manifest ID")
for device in devices:
require_lab_profile(value_reply(device.command("get tempradio")))
require_lab_profile(value_reply(device.command("get tempradio2")), True)
own_status = source.checked("ota serve status")
own_stats = source.checked("ota stats")
own_mid = re.match(r"^OTA\s*\|\s*fw\s+\S+\s+id=([0-9A-Fa-f]{8})\b", own_stats)
require("default:on serving:on" in own_status and own_mid is not None
and own_mid[1].upper() == mid,
"Original own-flash manifest is no longer being served")
progress = parse_progress(receiver.command("ota status"), mid)
require(progress["total"] == (source_info["image_bytes"] + 2047) // 2048,
"Continued download has unexpected block geometry")
require(progress["done"] >= entry["progress"][-1]["done"],
"Continued download lost recorded block progress")
report.event("Round %d: continuing live MID %s at %u/%u blocks without restarting RF" %
(number, mid, progress["done"], progress["total"]))
else:
entry["source_slot"], entry["receiver_slot_before"] = source_slot, receiver_slot
clear = receiver.checked("ota cancel", timeout=30)
require("OTA receive slot confirmed clear" in clear, "Receiver persistent OTA slot was not cleared")
require("no download" in receiver.checked("ota status"), "Receiver did not return to idle")
own = source.checked("ota serve self", timeout=30)
match = re.search(r"OK serving own fw mid=([0-9A-Fa-f]{8})\b", own)
require(match is not None and "flash-backed, unsigned" in own,
"Source did not acknowledge a flash-backed own image")
mid = match[1].upper()
entry["manifest_id"], entry["source_self"] = mid, source_info
own_status = source.checked("ota serve status")
require("default:on serving:on" in own_status, "Source own-image serving is not active")
entry["source_serve_status"] = own_status
entry["catalog"] = discover(receiver, mid)
start_reply = receiver.checked("ota get " + mid + " flash")
require("OK pulling mid=" + mid in start_reply and "-> flash" in start_reply,
"Receiver did not start a new full OTA flash download")
report.event("Round %d: %s own flash -> %s, MID %s" % (number, source.name, receiver.name, mid))
started, last_progress = time.monotonic(), time.monotonic()
prior_elapsed = entry["progress"][-1]["elapsed_seconds"] if continuing is not None else 0
if continuing is not None:
# RF keeps running while its observer is stopped. Include that gap in
# pace/download duration rather than reporting an artificially fast run.
recorded = datetime.fromisoformat(entry["progress"][-1]["utc"].replace("Z", "+00:00"))
prior_elapsed += max(0, time.time() - recorded.timestamp())
previous = progress["done"] if continuing is not None else -1
observed_intermediate = any(0 < p["done"] < p["total"] for p in entry["progress"])
ready = False
last_log = 0.0
while time.monotonic() - started < args.timeout:
progress = parse_progress(receiver.command("ota status"), mid)
done, total = progress["done"], progress["total"]
require(total == (source_info["image_bytes"] + 2047) // 2048
or (progress["state"] == "starting" and total == 0),
"OTA download block geometry changed")
if 0 < done < total:
observed_intermediate = True
if done > previous:
last_progress = time.monotonic()
require(done >= previous, "OTA completed-block count moved backwards")
previous = done
if not entry["progress"] or done != entry["progress"][-1]["done"]:
entry["progress"].append({"utc": utc(), "elapsed_seconds": round(prior_elapsed+time.monotonic()-started, 2),
**progress})
report.save()
if time.monotonic() - last_log >= 30:
elapsed = prior_elapsed+time.monotonic()-started
rate = done/elapsed if elapsed else 0
report.event("Round %d: %u/%u blocks, %.2f blocks/s, %s" %
(number, done, total, rate, progress["state"]))
last_log = time.monotonic()
if progress["state"] == "ready to install":
require(done == total, "Ready state does not include every firmware block")
ready = True
break
require(time.monotonic()-last_progress < args.stall_timeout,
"OTA download stalled beyond the configured progress timeout")
time.sleep(args.poll_interval)
require(ready, "OTA download exceeded its per-round timeout")
require(observed_intermediate, "No intermediate RF block progress was observed; cannot prove transfer")
entry["download_seconds"] = round(prior_elapsed+time.monotonic()-started, 2)
entry["intermediate_rf_progress_observed"] = True
entry["ready_status"] = progress["status"]
report.save()
# Persist intent BEFORE sending. A transport interruption after transmission
# must not be misclassified as a safe, never-installed continuation.
entry["install_requested_utc"] = utc()
report.save()
install = receiver.checked("ota install", timeout=30)
require(install.startswith("OK"), "Verified fetched image was not accepted for install")
entry["install_reply"] = install
report.event("Round %d: all blocks received; guarded OTA install accepted" % number)
receiver.close()
time.sleep(4)
entry["receiver_self_after"] = receiver.reconnect(expected_hash)
after_slot = parse_slot(receiver.command("ota dev apply slot"))
before_slot = entry["receiver_slot_before"]
require(after_slot["inactive_address"] != before_slot["inactive_address"],
"Receiver did not switch its running A/B application slot")
require(after_slot["size_bytes"] == before_slot["size_bytes"], "OTA changed app partition geometry")
entry["receiver_slot_after"] = after_slot
require_normal_profiles([receiver])
after = snapshot(receiver)
changes = [key for key in baseline[receiver.name] if baseline[receiver.name][key] != after[key]]
entry["snapshot_after"] = after
entry["preservation"] = {"equal": not changes, "changed_fields": changes}
report.save()
require(not changes, "Receiver identity/ACL/saved settings changed across OTA install")
entry["result"], entry["finished_utc"] = "PASS", utc()
report.event("Round %d PASS: hash, A/B slot flip, key digest, ACL, and settings preserved" % number)
def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("action", choices=("probe", "prepare", "run"), nargs="?", default="run")
parser.add_argument("--serial-a", required=True, help="Exact physical USB serial, not a tty path")
parser.add_argument("--serial-b", required=True, help="Exact physical USB serial, not a tty path")
parser.add_argument("--expected-body-hash16", required=True, help="16 hex digits from this build's EndF")
parser.add_argument("--report", required=True, help="New JSON report path (must not already exist)")
parser.add_argument("--prepare", action="store_true", help="Seed lab saved settings/ACLs and reboot before run")
parser.add_argument("--rounds", type=int, default=3)
parser.add_argument("--start-source", choices=("a", "b"), default="a",
help="Which physical radio serves the first round")
parser.add_argument("--timeout", type=float, default=5400, help="Maximum seconds per download observation")
parser.add_argument("--continue-report", help="Adopt a live, uninstalled download from an interrupted report")
parser.add_argument("--stall-timeout", type=float, default=120, help="Maximum seconds without block progress")
parser.add_argument("--poll-interval", type=float, default=15)
args = parser.parse_args(argv)
if not HEX16.fullmatch(args.expected_body_hash16):
parser.error("--expected-body-hash16 must contain exactly 16 hexadecimal digits")
if args.rounds < 1 or args.timeout <= 0 or args.stall_timeout <= 0 or args.poll_interval <= 0:
parser.error("Round count and timeout/interval values must be positive")
if normalized_serial(args.serial_a) == normalized_serial(args.serial_b):
parser.error("Two distinct physical USB serials are required")
expected_hash = args.expected_body_hash16.upper()
if args.continue_report and (args.action != "run" or args.prepare):
parser.error("--continue-report requires run without --prepare")
devices, report = [], None
try:
report = Report(args.report, args)
devices = [V4Device("A", args.serial_a), V4Device("B", args.serial_b)]
public_keys = preflight(devices, expected_hash, report)
if args.action == "prepare" or (args.action == "run" and args.prepare):
prepare(devices, public_keys, expected_hash, report)
if not args.continue_report:
if args.action == "run":
disarm_lab(devices)
else:
require_normal_profiles(devices)
baseline = {device.name: snapshot(device) for device in devices}
require(baseline["A"]["private_key_sha256"] != baseline["B"]["private_key_sha256"],
"Radios have duplicated private identities")
continuing = None
first_round = 1
if args.continue_report:
previous_report = json.loads(Path(args.continue_report).read_text(encoding="utf-8"))
require(previous_report.get("schema") == 1 and previous_report.get("action") == "run"
and previous_report.get("expected_body_hash16") == expected_hash
and previous_report.get("lab_radio") == LAB_RADIO
and previous_report.get("requested_rounds") == args.rounds
and previous_report.get("first_source", "a") == args.start_source
and previous_report.get("baseline_mode") == BASELINE_MODE,
"Continuation report does not match this test definition")
require(previous_report.get("result") == "ABORTED" or
(previous_report.get("result") == "FAIL" and previous_report.get("failure") ==
"OTA download exceeded its per-round timeout"),
"Only an interrupted or timed-out download report can be continued")
require(all(live_snapshot_matches_saved(previous_report["baseline"][name], baseline[name])
for name in ("A", "B")),
"Identity, ACL or settings changed before continuation")
for device in devices:
require(previous_report["devices"][device.name]["identity"]["usb_serial"] == device.serial_number,
"Continuation physical USB identities do not match")
prior_rounds = previous_report.get("rounds", [])
require(bool(prior_rounds) and len(prior_rounds) <= args.rounds,
"Continuation report has no pending round")
require(all(r.get("result") == "PASS" and r.get("round") == n
for n, r in enumerate(prior_rounds[:-1], 1)),
"Continuation has invalid completed rounds")
continuing = prior_rounds[-1]
require(continuing.get("result") == "RUNNING" and continuing.get("round") == len(prior_rounds)
and bool(continuing.get("progress")), "Continuation has no observed uninstalled RF download")
first_round = len(prior_rounds)
report.data["rounds"] = prior_rounds
# Retain the original SAVED baseline, not the effective TempRadio
# getter values just observed while the interrupted transfer runs.
baseline = previous_report["baseline"]
report.data["continued_from"] = str(Path(args.continue_report).resolve())
report.event("Adopting interrupted observer report; original RF transfer remains active")
report.data["baseline"] = baseline
report.data["baseline_mode"] = BASELINE_MODE
report.save()
if args.action == "run":
for number in range(first_round, args.rounds+1):
ordered = devices if args.start_source == "a" else list(reversed(devices))
source, receiver = ordered if number % 2 else list(reversed(ordered))
run_round(number, source, receiver, devices, expected_hash, baseline, args, report,
continuing if number == first_round else None)
# Never report A/B coverage from just a single successful transfer.
sources = {entry["source_slot"]["inactive_address"] for entry in report.data["rounds"]}
destinations = {entry["receiver_slot_before"]["inactive_address"] for entry in report.data["rounds"]}
report.data["both_source_slots_covered"] = len(sources) == 2
report.data["both_destination_slots_covered"] = len(destinations) == 2
if args.rounds >= 3:
require(len(sources) == 2 and len(destinations) == 2,
"Three-round run did not exercise both source and destination A/B slots")
report.data["result"] = "PASS"
report.data["finished_utc"] = utc()
report.event(args.action.upper() + " PASS")
return 0
except KeyboardInterrupt:
if report:
report.data["result"], report.data["failure"] = "ABORTED", "Operator interrupted the test"
report.data["finished_utc"] = utc()
report.save()
return 130
except Exception as error:
# Third-party serial exceptions could include a raw buffer in their
# message. Only our explicitly non-secret failures are reportable.
message = str(error) if isinstance(error, TestFailure) else type(error).__name__
if report:
report.data["result"], report.data["failure"] = "FAIL", message
report.data["finished_utc"] = utc()
report.event("FAIL: " + message)
else:
print("FAIL: " + message, file=sys.stderr)
return 1
finally:
for device in devices:
device.close()
if __name__ == "__main__":
sys.exit(main())
tools/hil/esp32_usb_host_reboot.py
+563
View File
@@ -0,0 +1,563 @@
#!/usr/bin/env python3
"""Qualified two-V4 USB host-reboot evidence; this tool NEVER reboots the host.
Run prepare after the OTA test to restore saved primary radio settings, disable
radio2/logging, enable powersaving, and reboot the two exact radios. Wait more
than three minutes before probe. After the operator reboots the Pi, probe again
to a NEW report, then compare the two probes offline.
All actions require both physical USB serials and the expected running-image
hash. Preparation sends each setting once, with a bounded acknowledgement
deadline. Only normal USB CLI commands are used; no bootloader or flash access.
Private keys are retained solely as SHA-256 digests through the OTA helper.
Final reports are owner-readable only and new paths are required.
OTA autofetch/autoinstall must already be off. Comparison assumes the Pi wall
clock remains stable across reboot; retain an independent operator timestamp
if clock synchronization or a large clock correction is suspected.
"""
import argparse
import hashlib
import json
import math
import os
from pathlib import Path
import platform
import re
import sys
import tempfile
import time
from esp32_self_serve_ota import (
HEX16, Report, TestFailure, V4Device, normalized_serial, parse_self,
parse_slot, require, snapshot, utc, value_reply,
)
KIND = "esp32_usb_host_warm_reboot"
CORE_KEYS = {"battery_mv", "uptime_secs", "errors", "queue_len"}
RADIO_DIAG = re.compile(
r"state=(\d+)\((RX|IDLE|TX_WAIT|INT_READY|\?)\), recv=(\d+), sent=(\d+), "
r"errors=(\d+), err_flags=(\d+), outbound=(yes|no), last_rx=(\d+)s ago"
)
RADIO2_OFF = re.compile(
r"off; RX=\d+,\d+ TX=\d+,\d+ switches=\d+ errors=\d+ "
r"max=\d+us preamble=\d+,\d+"
)
MAX_REPORT_BYTES = 4 * 1024 * 1024
class EvidenceReport(Report):
"""Reuse atomic owner-only reporting, adding no-clobber creation/sealing."""
def __init__(self, path, args):
self.created = False
self.sealed = False
super().__init__(path, args)
self.data.update(
evidence_kind=KIND, firmware_transport="none; USB CLI evidence only",
host_reboot_performed_by_tool=False, minimum_board_age_seconds=args.min_age,
uptime_tolerance_seconds=args.uptime_tolerance,
)
self.save()
def save(self):
require(not self.sealed, "A finalized evidence report cannot be modified")
if self.created:
return super().save()
fd, temporary = tempfile.mkstemp(prefix=self.path.name + ".", suffix=".tmp",
dir=self.path.parent)
temporary = Path(temporary)
try:
with os.fdopen(fd, "w", encoding="utf-8") as stream:
json.dump(self.data, stream, indent=2)
stream.write("\n")
stream.flush()
os.fsync(stream.fileno())
# Atomic no-clobber: a simultaneous creator must not be overwritten.
os.link(temporary, self.path)
self.created = True
except FileExistsError:
raise TestFailure("Report path already exists; select a new report") from None
finally:
temporary.unlink(missing_ok=True)
def seal(self):
self.data["finished_utc"] = utc()
self.save()
os.chmod(self.path, 0o400)
self.sealed = True
def parse_core(text):
try:
data = json.loads(text)
except (ValueError, TypeError):
raise TestFailure("stats-core is not complete JSON") from None
require(isinstance(data, dict) and set(data) == CORE_KEYS,
"stats-core has unexpected fields")
require(all(type(value) is int and 0 <= value <= 0xFFFFFFFF
for value in data.values()), "stats-core has invalid integer fields")
return data
def parse_radio_diag(text):
match = RADIO_DIAG.fullmatch(text)
require(match is not None, "stats-radio-diag has incomplete or unexpected fields")
values = [int(match[index]) for index in (1, 3, 4, 5, 6, 8)]
require(all(value <= 0xFFFFFFFF for value in values),
"stats-radio-diag integer exceeded its bound")
return dict(zip(("state", "recv", "sent", "errors", "err_flags", "last_rx_secs"),
values), state_name=match[2], outbound=match[7] == "yes")
def require_quiet(core, diag):
require(core["queue_len"] == 0, "Radio has a nonempty outbound queue")
require(diag["state"] == 1 and diag["state_name"] == "RX" and not diag["outbound"],
"Radio is not in idle receive with no outbound packet")
def require_idle(device, timeout=5):
text = value_reply(device.command("ota status", timeout=timeout))
require(" | no download | " in text, device.name + ": OTA session is not idle")
def wait_quiet(devices, timeout=30):
"""Wait out short beacons, never wait through or mutate an active download."""
deadline = time.monotonic() + timeout
while True:
samples = {}
for device in devices:
def query(command):
remaining = deadline - time.monotonic()
require(remaining > 0, "Bounded idle-receive observation timed out")
return value_reply(device.command(command, timeout=min(5, remaining)))
status = query("ota status")
require(" | no download | " in status, device.name + ": OTA session is not idle")
started = time.time()
core = parse_core(query("stats-core"))
finished = time.time()
diag = parse_radio_diag(query("stats-radio-diag"))
samples[device.name] = {
"core": core, "radio_diag": diag,
"observed_unix_seconds": (started + finished) / 2,
"sample_uncertainty_seconds": (finished - started) / 2,
}
if all(item["core"]["queue_len"] == 0 and item["radio_diag"]["state"] == 1
and item["radio_diag"]["state_name"] == "RX"
and not item["radio_diag"]["outbound"] for item in samples.values()):
return samples
remaining = deadline - time.monotonic()
require(remaining > 0, "Bounded idle-receive observation timed out")
time.sleep(min(.5, remaining))
def logging_state(device):
text = value_reply(device.command("get usb.logging"))
if text == "Error: unknown setting: usb.logging":
return {"available": False, "enabled": False}
require(text in ("on", "off"), device.name + ": USB logging state is unavailable")
return {"available": True, "enabled": text == "on"}
def manual_ota_policy(device):
text = value_reply(device.command("ota config"))
values = {}
for key in ("autofetch", "autoinstall"):
found = re.findall(r"\b" + key + r"=([a-z]+)\b", text)
require(len(found) == 1 and found[0] == "off",
device.name + ": automatic OTA policy must already be off")
values[key] = found[0]
return values
def verified_device(device, expected_hash):
device.connect()
info = parse_self(device.command("ota self"))
require(info["body_hash16"] == expected_hash,
device.name + ": running firmware does not match the expected hash")
require(info["image_bytes"] > info["body_bytes"] > 0,
device.name + ": running image geometry is invalid")
slot = parse_slot(device.command("ota dev apply slot"))
require(info["image_bytes"] <= slot["size_bytes"], "Image does not fit the inactive slot")
return {"identity": dict(device.identity), "self": info, "inactive_slot": slot,
"ota_policy": manual_ota_policy(device)}
def native_usb_state(device, root=Path("/")):
endpoint = Path(device.endpoint).name
require(re.fullmatch(r"ttyACM\d+", endpoint) is not None,
"Qualified V4 is not a Linux native USB serial endpoint")
tty = root / "sys/class/tty" / endpoint / "device"
resolved = tty.resolve(strict=True)
parents = (resolved,) + tuple(resolved.parents)
for parent in parents:
if not all((parent / field).is_file()
for field in ("idVendor", "idProduct", "serial", "speed")):
continue
serial = normalized_serial((parent / "serial").read_text(encoding="ascii").strip())
require(serial == device.serial_number, "Native USB sysfs serial does not match")
vid = (parent / "idVendor").read_text(encoding="ascii").strip().lower()
pid = (parent / "idProduct").read_text(encoding="ascii").strip().lower()
require(vid == "303a" and pid == "1001",
"Qualified V4 is not the expected native USB/JTAG interface")
speed = (parent / "speed").read_text(encoding="ascii").strip()
require(speed in ("12", "12.0"), "Native USB link is not full-speed 12 Mbps")
return {"usb_serial": serial, "speed_mbps": 12, "sysfs_path": str(parent),
"vid": vid, "pid": pid}
raise TestFailure("Native USB device attributes were not found in sysfs")
def pi_state(root=Path("/")):
boot_id = (root / "proc/sys/kernel/random/boot_id").read_text(encoding="ascii").strip()
require(re.fullmatch(r"[0-9a-fA-F]{8}(?:-[0-9a-fA-F]{4}){3}-[0-9a-fA-F]{12}",
boot_id) is not None, "Pi boot identity is invalid")
uptime = float((root / "proc/uptime").read_text(encoding="ascii").split()[0])
require(math.isfinite(uptime) and uptime >= 0, "Pi uptime is invalid")
cmdline = (root / "proc/cmdline").read_text(encoding="ascii").strip()
speed_tokens = [word for word in cmdline.split() if word.startswith("dwc_otg.speed=")]
require(speed_tokens == ["dwc_otg.speed=1"], "Pi USB 1.1 host configuration is not retained")
model = (root / "proc/device-tree/model").read_bytes().rstrip(b"\0").decode("ascii")
require("Raspberry Pi" in model and len(model) <= 128, "Host is not a qualified Raspberry Pi")
return {"boot_id": boot_id.lower(), "uptime_seconds": uptime, "model": model,
"kernel": platform.release(), "dwc_otg_speed": 1,
# Do not copy arbitrary kernel arguments (which may contain secrets).
"cmdline_sha256": hashlib.sha256(cmdline.encode("ascii")).hexdigest()}
def capture_device(device, expected_hash, min_age):
result = verified_device(device, expected_hash)
require_idle(device)
result["snapshot"] = snapshot(device)
state = {key: value_reply(device.command("get " + key))
for key in ("radio", "radio2", "tempradio", "tempradio2", "powersaving",
"radio2.status")}
require(re.fullmatch(r"[0-9.]+,[0-9.]+,\d+,\d+", state["radio"]) is not None,
"Saved primary radio tuple is invalid")
require(state["radio2"] == "off" and RADIO2_OFF.fullmatch(state["radio2.status"]),
device.name + ": radio2 is not disabled in both saved and current state")
require(state["tempradio"] == "off" and state["tempradio2"] == "off",
device.name + ": a temporary radio profile is still active or pending")
require(state["powersaving"] == "on", device.name + ": powersaving is not enabled")
state["usb.logging"] = logging_state(device)
require(not state["usb.logging"]["enabled"], device.name + ": USB logging remains active")
# Firmware exposes saved primary settings, not an instantaneous tuple.
state["primary_current_inferred_from"] = "saved tuple; no temporary profile; radio2 off; RX"
result["radio_state"] = state
powerlog = value_reply(device.command("powerlog"))
match = re.fullmatch(r"Last reset reason: ([ -~]{1,100})", powerlog)
require(match is not None, "powerlog lacks a bounded reset reason")
result["reset_reason"] = match[1]
sample = wait_quiet([device])[device.name]
core, diag = sample["core"], sample["radio_diag"]
require(core["uptime_secs"] > min_age,
device.name + ": board is still within the post-boot sleep grace period")
require_quiet(core, diag)
# Check OTA again after all telemetry so an intervening download fails closed.
require_idle(device)
require(manual_ota_policy(device) == result["ota_policy"],
"OTA policy changed during the probe")
result.update(sample, ota_idle=True, usb=native_usb_state(device))
return result
def temporary_profiles(device, deadline=None):
state = {}
for key in ("tempradio", "tempradio2"):
remaining = 5 if deadline is None else deadline - time.monotonic()
require(remaining > 0, "Temporary profile restoration timed out")
value = value_reply(device.command("get " + key, timeout=min(5, remaining)))
# Secondary replies append timing recommendations. Retain only the
# bounded profile fields, never arbitrary diagnostic/log tails.
if key == "tempradio2":
value = value.split(";", 1)[0]
pattern = (r"[0-9.]+,[0-9.]+,\d+,\d+,\d+" if key == "tempradio"
else r"[0-9.]+,[0-9.]+,\d+,\d+,(?:rx|rxtx),\d+(?: \(auto\))?")
require(value == "off" or len(value) <= 100 and re.fullmatch(pattern, value),
"Temporary radio state has invalid profile fields")
state[key] = value
return state
def wait_no_temporary_profiles(devices, timeout=30):
deadline = time.monotonic() + timeout
while True:
states = {}
for device in devices:
remaining = deadline - time.monotonic()
require(remaining > 0, "Temporary profile restoration timed out")
require_idle(device, timeout=min(5, remaining))
states[device.name] = temporary_profiles(device, deadline)
if all(all(value == "off" for value in state.values()) for state in states.values()):
return states
remaining = deadline - time.monotonic()
require(remaining > 0, "Temporary profile restoration timed out")
time.sleep(min(.5, remaining))
def normalized_snapshot_differences(raw, saved, primary_was_temporary, name):
require(all(raw[key] == saved[key] for key in ("private_key_sha256", "public_key", "acl")),
name + ": temporary restoration changed identity or ACL")
old, new = raw["settings"], saved["settings"]
require(set(old) == set(new), name + ": normalized saved-setting fields changed")
differences = {}
for key, value in old.items():
if value == new[key]:
continue
# These getters intentionally report runtime timing overrides, not
# persisted prefs. Restoring normal radio exposes the saved values;
# never set an assumed advert interval to make a comparison pass.
masks = {"advert.interval": ("60", 510), "flood.advert.interval": ("3", 255)}
expected = masks.get(key)
require(primary_was_temporary and expected is not None and value == expected[0],
name + ": temporary restoration changed an unrelated saved setting")
require(re.fullmatch(r"0|[1-9]\d*", new[key]) is not None
and int(new[key]) <= expected[1],
name + ": normalized advert interval is invalid")
differences[key] = {"live_before": value, "saved_after_normalradio": new[key],
"reason": "documented temporary radio timing getter override"}
return differences
def prepare_devices(devices, expected_hash, report):
before = {}
# Qualify BOTH radios before issuing the first write to either.
for device in devices:
before[device.name] = verified_device(device, expected_hash)
require_idle(device)
before[device.name]["usb"] = native_usb_state(device)
before[device.name]["temporary_profiles"] = temporary_profiles(device)
before[device.name]["snapshot"] = snapshot(device)
before[device.name]["snapshot_interpretation"] = (
"raw live CLI getters; advert timing may be temporarily overridden")
before[device.name]["logging"] = logging_state(device)
report.data["devices"] = {name: {"before_prepare": value} for name, value in before.items()}
report.save()
# Both devices must be quiet in one bounded observation round, after the
# potentially long ACL snapshots and before the first mutation.
quiet = wait_quiet(devices)
for name, sample in quiet.items():
report.data["devices"][name]["before_prepare"].update(sample)
report.save()
for device in devices:
for peer in devices:
require_idle(peer) # Refuse a newly active session before this group.
manual_ota_policy(peer)
# Clear only runtime overlays first. Snapshot getters for advert
# intervals otherwise report the temporary policy rather than prefs.
for command in ("normalradio", "set tempradio2 off"):
require(device.checked(command, timeout=30).startswith("OK"),
device.name + ": temporary restoration was not acknowledged")
report.event(device.name + ": temporary restoration acknowledged once")
restored = wait_no_temporary_profiles(devices)
normalized = {}
for device in devices:
require_idle(device)
manual_ota_policy(device)
saved = snapshot(device)
entry = report.data["devices"][device.name]
entry["normalized_saved_snapshot"] = saved
entry["temporary_profiles_after_normalization"] = restored[device.name]
report.save() # Preserve observed values even if invariant checks fail.
entry["normalization_differences"] = normalized_snapshot_differences(
before[device.name]["snapshot"], saved,
before[device.name]["temporary_profiles"]["tempradio"] != "off", device.name)
normalized[device.name] = saved
report.save()
for device in devices:
for peer in devices:
require_idle(peer)
manual_ota_policy(peer)
commands = ["set radio2 off", "set powersaving on"]
if before[device.name]["logging"]["available"]:
commands.append("set usb.logging off")
for command in commands:
require(device.checked(command, timeout=30).startswith("OK"),
device.name + ": preparation setting was not acknowledged")
report.event(device.name + ": idle-test settings acknowledged once")
time.sleep(6) # Existing commands persist settings; allow pending/lazy commits.
for device in devices:
for peer in devices:
require_idle(peer)
manual_ota_policy(peer)
device.reboot(expected_hash)
after = verified_device(device, expected_hash)
require_idle(device)
after["snapshot"] = snapshot(device)
old, new = normalized[device.name], after["snapshot"]
require(all(old[key] == new[key] for key in ("private_key_sha256", "public_key", "acl")),
device.name + ": preparation changed identity or ACL")
require(set(old["settings"]) == set(new["settings"])
and all(value == new["settings"].get(key)
for key, value in old["settings"].items() if key != "radio2"),
device.name + ": preparation changed an unrelated saved setting")
require(new["settings"]["radio2"] == {"mode": "off"},
device.name + ": radio2 off was not persisted")
require(all(value_reply(device.command("get " + key)) == expected
for key, expected in (("tempradio", "off"), ("tempradio2", "off"),
("powersaving", "on"))),
device.name + ": idle-test settings did not survive reboot")
require(not logging_state(device)["enabled"],
device.name + ": USB logging was not disabled after reboot")
report.data["devices"][device.name]["after_prepare"] = after
report.save()
report.data["board_reboots_intentionally_performed"] = True
report.event("Both radios rebooted; wait over 180 seconds, then create a pre-host-reboot probe")
def validate_probe(data, expected_hash, serials):
require(isinstance(data, dict) and data.get("schema") == 1
and data.get("evidence_kind") == KIND and data.get("action") == "probe"
and data.get("result") == "PASS", "Input is not a successful reboot probe")
require(data.get("expected_body_hash16") == expected_hash, "Probe expected hash differs")
require(data.get("host_reboot_performed_by_tool") is False, "Probe host action is unexpected")
require(data["pi"]["dwc_otg_speed"] == 1, "Probe host speed differs")
require(set(data["devices"]) == {"A", "B"}, "Probe does not contain exactly both radios")
for name, serial in zip(("A", "B"), serials):
item = data["devices"][name]
ident = item["identity"]
require(ident["usb_serial"] == serial and ident["vid"] == 0x303A
and ident["pid"] == 0x1001 and ident["role"].lower() == "repeater"
and re.search(r"Heltec\s+V4\b", ident["board"], re.I),
"Probe physical identity or role differs")
require(item["self"]["body_hash16"] == expected_hash
and item["usb"]["usb_serial"] == serial and item["usb"]["speed_mbps"] == 12,
"Probe image or native USB identity differs")
require(item["ota_idle"] is True, "Probe OTA session was not idle")
require(item["ota_policy"] == {"autofetch": "off", "autoinstall": "off"},
"Probe automatic OTA policy was not disabled")
core = parse_core(json.dumps(item["core"]))
require(core["uptime_secs"] > max(180, data["minimum_board_age_seconds"]),
"Probe was captured during the boot grace period")
require_quiet(core, item["radio_diag"])
state = item["radio_state"]
require(state["radio2"] == "off" and RADIO2_OFF.fullmatch(state["radio2.status"])
and state["tempradio"] == "off" and state["tempradio2"] == "off"
and state["powersaving"] == "on"
and state["usb.logging"]["enabled"] is False, "Probe sleep inhibitors are present")
require(re.fullmatch(r"[0-9a-f]{64}", item["snapshot"]["private_key_sha256"]) is not None,
"Probe identity digest is malformed")
require(math.isfinite(item["observed_unix_seconds"])
and 0 <= item["sample_uncertainty_seconds"] <= 5,
"Probe observation timestamp is invalid")
def load_probe(path, expected_hash, serials):
source = Path(path)
require(source.is_file() and source.stat().st_size <= MAX_REPORT_BYTES,
"Probe report is absent or exceeded its size bound")
with source.open(encoding="utf-8") as stream:
data = json.load(stream)
validate_probe(data, expected_hash, serials)
return data
def compare_probes(before, after, tolerance):
require(before["pi"]["boot_id"] != after["pi"]["boot_id"],
"Pi boot identity did not change; a host reboot was not demonstrated")
require(before["pi"]["dwc_otg_speed"] == after["pi"]["dwc_otg_speed"] == 1
and before["pi"]["cmdline_sha256"] == after["pi"]["cmdline_sha256"],
"Pi USB host configuration changed")
require(before["pi"]["model"] == after["pi"]["model"]
and before["pi"]["kernel"] == after["pi"]["kernel"], "Pi hardware/kernel changed")
compared = {}
for name in ("A", "B"):
old, new = before["devices"][name], after["devices"][name]
require(all(old["identity"][key] == new["identity"][key]
for key in ("usb_serial", "vid", "pid", "board", "role")),
name + ": physical radio identity changed")
require(all(old[key] == new[key]
for key in ("self", "inactive_slot", "snapshot", "reset_reason", "ota_policy")),
name + ": firmware, A/B slot, identity, saved settings or reset reason changed")
require(all(old["radio_state"][key] == new["radio_state"][key]
for key in ("radio", "radio2", "tempradio", "tempradio2",
"powersaving", "usb.logging")),
name + ": saved/current radio or logging settings changed")
elapsed = new["observed_unix_seconds"] - old["observed_unix_seconds"]
uptime_gain = new["core"]["uptime_secs"] - old["core"]["uptime_secs"]
require(math.isfinite(elapsed) and elapsed > 0 and uptime_gain > 0,
name + ": board uptime did not continue across the host reboot")
uncertainty = old["sample_uncertainty_seconds"] + new["sample_uncertainty_seconds"]
require(abs(uptime_gain - elapsed) <= tolerance + uncertainty,
name + ": board uptime gain is inconsistent with the elapsed wall time")
compared[name] = {"usb_serial": new["identity"]["usb_serial"],
"body_hash16": new["self"]["body_hash16"],
"wall_interval_seconds": elapsed, "board_uptime_gain_seconds": uptime_gain,
"sample_uncertainty_seconds": uncertainty,
"uptime_tolerance_seconds": tolerance, "board_reboot_observed": False,
"identity_settings_slots_unchanged": True, "native_usb_speed_mbps": 12}
return {"pi_boot_id_before": before["pi"]["boot_id"], "pi_boot_id_after": after["pi"]["boot_id"],
"devices": compared, "wall_clock_stability_required": True,
"interpretation": "Exact USB CLI recovered with uptime-consistent board continuity"}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("action", choices=("prepare", "probe", "compare"))
parser.add_argument("--serial-a", required=True)
parser.add_argument("--serial-b", required=True)
parser.add_argument("--expected-body-hash16", required=True)
parser.add_argument("--report", required=True, help="new owner-only evidence report path")
parser.add_argument("--min-age", type=int, default=180)
parser.add_argument("--uptime-tolerance", type=float, default=10)
parser.add_argument("--before", help="successful pre-host-reboot probe report")
parser.add_argument("--after", help="successful post-host-reboot probe report")
args = parser.parse_args()
serials = tuple(normalized_serial(value) for value in (args.serial_a, args.serial_b))
if not all(re.fullmatch(r"[0-9A-F]{12}", serial) for serial in serials) or serials[0] == serials[1]:
parser.error("two distinct complete 12-hex physical USB serials are required")
if not HEX16.fullmatch(args.expected_body_hash16):
parser.error("expected image body hash must contain 16 hex digits")
if args.min_age < 180 or not math.isfinite(args.uptime_tolerance) or not 1 <= args.uptime_tolerance <= 60:
parser.error("minimum age must be at least 180s; uptime tolerance must be 1..60s")
if args.action == "compare":
if not args.before or not args.after or Path(args.before).resolve() == Path(args.after).resolve():
parser.error("compare needs two different successful probe paths")
elif args.before or args.after:
parser.error("before/after paths are only valid for compare")
args.expected_body_hash16 = args.expected_body_hash16.upper()
args.rounds, args.timeout, args.poll_interval = 0, 30, 0
report, devices = None, []
try:
report = EvidenceReport(args.report, args)
if args.action == "compare":
before = load_probe(args.before, args.expected_body_hash16, serials)
after = load_probe(args.after, args.expected_body_hash16, serials)
report.data["comparison"] = compare_probes(before, after, args.uptime_tolerance)
else:
report.data["pi"] = pi_state()
devices = [V4Device(name, serial) for name, serial in zip(("A", "B"), serials)]
if args.action == "prepare":
prepare_devices(devices, args.expected_body_hash16, report)
else:
for device in devices:
report.data["devices"][device.name] = capture_device(
device, args.expected_body_hash16, args.min_age)
report.save()
validate_probe(dict(report.data, result="PASS"), args.expected_body_hash16, serials)
require(pi_state()["boot_id"] == report.data["pi"]["boot_id"],
"Host rebooted while a probe/preparation was being captured")
report.data["result"] = "PREPARED" if args.action == "prepare" else "PASS"
report.event("Reboot evidence " + report.data["result"])
report.seal()
return 0
except (Exception, KeyboardInterrupt) as error:
# Never print unexpected exception text, which may contain raw serial data.
reason = str(error) if isinstance(error, TestFailure) else type(error).__name__
print("Reboot evidence failed: " + reason, file=sys.stderr)
if report is not None and not report.sealed:
report.data.update(result="ABORTED" if isinstance(error, KeyboardInterrupt) else "FAIL",
failure=reason)
report.seal()
return 130 if isinstance(error, KeyboardInterrupt) else 1
finally:
for device in devices:
try:
device.close()
except Exception:
pass
if __name__ == "__main__":
sys.exit(main())