Record targeted RAK build, radio and CPU sleep passes [skip ci]

2026-10-22 23:21:14 -07:00
1 parent 302e441e2b
commit 9b62b43995
@@ -75,6 +75,7 @@ Completed validation reports, experiments, and test guides are collected here. O

 ## Build, storage, memory, and checklists

+- [RAK4631 Full/Reduced RAM repair and RAK3401 Room Server radio/sleep qualification, 2026-10-22](rak_release_fixes_2026-10-22.md)
 - [Shared battery ADC calibration validation, 2026-10-22](battery_calibration_2026-10-22.md)
 - [Heltec V2 BLE RAM reserve repair, 2026-10-22](heltec_v2_ble_memory_2026-10-22.md)
 - [ESP32 Bluetooth shutdown and radio2 CPU idle, 2026-10-21](esp32_ble_radio2_sleep_2026-10-21.md)
@@ -4,6 +4,11 @@

 # RAK3401 all firmware CPU sleep checks, 21 October 2026

+Follow-up: the [targeted RAK release checks](rak_release_fixes_2026-10-22.md)
+record a new Full Room Server PASS at `adb11e7b4`: three physical LoRa round
+trips, 79.31% sampled CPU sleep and zero post-check errors. The original failed
+capture below remains unchanged; its RF cause was not established.
+
 All 15 applicable images were installed and measured sequentially on the same
 RAK3401. **All 13 profiles that support power saving passed the CPU sleep
 capture with PS on.** Twelve also passed every runtime check. Full Room Server
@@ -0,0 +1,127 @@
+<!-- meshcore-hosted-doc-link:start -->
+<p class="meshcore-hosted-doc-link"><a href="https://mikecarper.github.io/MeshCore/test-results/rak_release_fixes_2026-10-22/">View this page on MeshCore Docs</a>.</p>
+<!-- meshcore-hosted-doc-link:end -->
+
+# RAK release fixes and targeted hardware checks, 22 October 2026
+
+Status: **RAK4631 Full/Reduced Repeater builds and physical radio checks PASS;
+RAK3401 Full Room Server radio, CPU sleep and health checks PASS.**
+All six selected RAK release packages passed their existing capacity,
+capability, Bluetooth DFU and packaging gates.
+
+Firmware and hardware results are bound to
+[`adb11e7b4`](https://github.com/mikecarper/MeshCore/commit/adb11e7b41f2d3c561bc6aba4ebf6c92a45623bd).
+The later `302e441e2` changes only Mini-selected CLI messages; selecting the
+non-Mini branches reproduces the prior RAK source byte for byte. Unchanged RAK
+checks were retained. The [machine-readable receipt](rak_release_fixes_2026-10-22.safe.json)
+includes package hashes, RAM requirements, radio counters and CPU observations.
+
+## RAM repair
+
+The fresh starting source, `e34e83e6b`, failed both RAK4631 Repeater profiles:
+Full had 81,668 internal bytes available against 85,152 required; Reduced had
+84,680 against 85,152. These are separate from the earlier Full Companion
+flash repair.
+
+Repeater rule targets cannot contain both a scope and a region. One tagged
+32-byte name replaces their two separate names, reducing each actual rule
+from 332 to 300 bytes. All 63 slots remain, saving 2,016 allocated heap bytes.
+FPF6/7/8 file formats retain their existing field widths and validation.
+
+Ordinary nRF52 SSD1306 builds now use the existing source-enforced 2 KiB
+allocation bound for this same concrete driver, correcting its previous
+4 KiB estimate. This is a 2,048-byte accounting correction, distinct from
+the rule-table saving. Stacks, packet pools, allocator/transient reserves,
+sensor features and admission boundaries remain unchanged.
+
+| Target/profile | Available internal bytes | Required heap bytes | Margin | Result |
+| --- | ---: | ---: | ---: | --- |
+| RAK4631 Repeater Full | 81,676 | 81,088 | 588 | PASS |
+| RAK4631 Repeater Reduced | 84,680 | 81,088 | 3,592 | PASS |
+| RAK3401 Room Server Full | 153,940 | 58,416 | 95,524 | PASS |
+| RAK3401 Room Server Reduced | 156,928 | 58,416 | 98,512 | PASS |
+| RAK3401 unified Repeater Full | 78,364 | 74,176 | 4,188 | PASS |
+| RAK3401 unified Repeater Reduced | 81,368 | 74,176 | 7,192 | PASS |
+
+These are canonical `build.sh build-firmware` Full/Reduced pairs with Cascade
+settings, `--radio-preset usa-cascade-fixed` and `--require-ota`. The canonical
+RAK4631 Repeater and Room Server packages retain their normalized internal-OTA
+target identities. PlatformIO commands ran sequentially.
+
+## Channel-busy repair
+
+A proven dispatcher defect retained a profile's channel-busy timestamp after
+its last waiting packet was cancelled or retired. A later unrelated packet
+could inherit the old age and immediately force a transmit with error `0x02`.
+
+New work now retires an unused wait before joining that profile. Queued work,
+retained driver retries and foreign/direct-OTA owners preserve their live
+backoff. `begin()` resets previous CAD state. No RAM fields or idle-loop scans
+were added. Genuine sustained contention still raises the original timeout.
+
+The original failed Room Server capture lacked enough RF data to prove this
+was its cause. Both old installed repeaters also had latched `0x02` flags.
+The new software regressions and finite new-image hardware results are the
+evidence for this repair; historical failures remain recorded.
+
+## Physical results
+
+Application-only DFU installed the source-bound images on the Mercerwood Pi.
+USB reopen checks confirmed role, version, EndF body identity and continuous
+uptime. No chip/data erase, filesystem backup, bootloader or SoftDevice update occurred.
+
+The RAK Tag was the independent LoRa witness. Each case used three fresh
+eight-byte nonces, exact routed trace replies and advancing target RX/TX
+counters. USB carried commands and observations; the request and reply crossed
+the physical radio link. Settings were 917 MHz, SF7, BW62.5, CR5, preamble32,
+with two-minute temporary radio leases. The RAK targets retained 2 dBm TX
+power; the Tag used its saved 22 dBm. The excluded 910.525 MHz channel was
+not used.
+
+| Installed image | Exact-nonce round trips | Core errors before/after | Result |
+| --- | ---: | --- | --- |
+| RAK4631 Reduced Repeater | 3/3 | 0 / 0 | PASS |
+| RAK4631 Full Repeater | 3/3 | 0 / 0 | PASS |
+| RAK3401 Full Room Server | 3/3 | 0 / 0 | PASS |
+
+After radio cleanup, the Room Server completed one 90.009-second nonhalting
+SWD capture with power saving on: 6,374 sleeping observations among 8,035
+samples, or **79.31% weighted sampled sleep**. There were no sampled resets,
+halts, steps, interrupt masking or lockups; debug controls stayed unchanged.
+Maximum read time was 3 ms and maximum sample gap 16 ms. Uptime advanced
+continuously; the post-check had zero errors and an empty queue. This measures
+sampled CPU states, not electrical current or exact sleep residency.
+
+The first Room preparation stopped on an unsupported `ota bootloader status`
+getter before any RF or CPU capture. Its failure and repeater restoration are
+retained. The corrected room-only follow-up treats that optional status page
+as unavailable while retaining mandatory bootloader-version and application
+identity checks. It did not repeat the already-passing RAK4631 tests.
+
+Final handback leaves both RAKs on patched Full Repeater images, saved 917 MHz
+settings and power saving on. The RAK3401 gateway service is active again with
+its original configuration and exact USB owner; final core errors were zero.
+
+## Regression checks and CI
+
+- 62 packet-manager cases, including four stale-wait reproductions that failed
+  before their fixes; genuine timeout and live-owner backoff still pass.
+- Three Meshtext and nine direct-OTA sanitizer cases; five receive-contract
+  cases; startup-watchdog and radio-idle regressions.
+- 55 RAM tests, 21 flood test methods and two OLED checks, including native
+  sanitizers, ARM layout, all 63 rules, FPF6/7/8 compatibility, full-length
+  names, room imports, fault recovery and malformed-file rejection.
+- Actual CI now builds both RAK4631 Repeater profiles and both RAK3401 Room
+  Server profiles, rather than relying on a Full Companion build alone.
+
+The shared CAD fix initially exposed a 44-byte Mini Repeater flash overflow.
+Mini-only CLI text was shortened without removing commands, values or checks.
+Its exact release-wrapper build now passes at 245,696 of 245,760 application
+bytes; RAM passes at 43,920 available against 32,288 required. Four replay
+integration and eight Mini-profile tests passed. Other platforms select their
+unchanged messages.
+
+The updated [CI run](https://github.com/mikecarper/MeshCore/actions/runs/39909897061)
+was still running when this report was prepared. These targeted checks do not
+claim a new all-board campaign, RS232 cable traffic, physical OTA installation,
+maximum-load operation or a long RF soak.
@@ -0,0 +1,349 @@
+{
+  "schema": 1,
+  "status": "targeted_checks_passed",
+  "firmware_source": "adb11e7b41f2d3c561bc6aba4ebf6c92a45623bd",
+  "mini_only_followup": "302e441e2b0baae4d87bf6c1c87785c762944c7a",
+  "builds": [
+    {
+      "artifact": "RAK_4631_repeater_lora_ota_no_external_sensors-full-ota-v1.18.0.1-halo-keymind-cascade-dev-adb11e7b.zip",
+      "zip_sha256": "06c8e8df88f35deed3785dde37674ee5973a91cdba95e6f32c108f2df2a579a7",
+      "available_internal_bytes": 81676,
+      "required_heap_bytes": 81088,
+      "margin_bytes": 588,
+      "required_contiguous_bytes": 19168,
+      "capabilities": [
+        "profile.auto",
+        "sensor.profile.full",
+        "bridge.rs232",
+        "ota.update.bluetooth",
+        "ota.update.lora",
+        "cli.retry_preset",
+        "telemetry.history",
+        "ota.cli",
+        "sensor.gps",
+        "sensor.ina219",
+        "sensor.ina226",
+        "sensor.ina260",
+        "sensor.ina3221",
+        "logging.usb.packets",
+        "logging.usb.control",
+        "bridge.rs232.merged"
+      ],
+      "passed": true
+    },
+    {
+      "artifact": "RAK_4631_repeater_lora_ota_no_external_sensors-reduced-ota-v1.18.0.1-halo-keymind-cascade-dev-adb11e7b.zip",
+      "zip_sha256": "84221bbb430e47028ea47dc11e9ac7ac2f62568d2ea134e7325b7ff54943bc03",
+      "available_internal_bytes": 84680,
+      "required_heap_bytes": 81088,
+      "margin_bytes": 3592,
+      "required_contiguous_bytes": 19168,
+      "capabilities": [
+        "profile.auto",
+        "sensor.profile.reduced",
+        "bridge.rs232",
+        "ota.update.bluetooth",
+        "ota.update.lora",
+        "cli.retry_preset",
+        "telemetry.history",
+        "ota.cli",
+        "sensor.gps",
+        "sensor.ina219",
+        "sensor.ina226",
+        "sensor.ina260",
+        "sensor.ina3221",
+        "logging.usb.packets",
+        "logging.usb.control",
+        "bridge.rs232.merged"
+      ],
+      "passed": true
+    },
+    {
+      "artifact": "RAK_3401_room_server_lora_ota_no_external_sensors-full-ota-v1.18.0.1-halo-keymind-cascade-dev-adb11e7b.zip",
+      "zip_sha256": "0f5cb79b19f2e9bebb05e6c7ce68cce7cad0e8093b45c3ec910e476c6f102908",
+      "available_internal_bytes": 153940,
+      "required_heap_bytes": 58416,
+      "margin_bytes": 95524,
+      "required_contiguous_bytes": 10256,
+      "capabilities": [
+        "profile.auto",
+        "sensor.profile.full",
+        "ota.update.bluetooth",
+        "ota.update.lora",
+        "cli.retry_preset",
+        "telemetry.history",
+        "ota.cli",
+        "sensor.gps",
+        "sensor.ina219",
+        "sensor.ina226",
+        "sensor.ina260",
+        "sensor.ina3221",
+        "logging.usb.packets",
+        "logging.usb.control"
+      ],
+      "passed": true
+    },
+    {
+      "artifact": "RAK_3401_room_server_lora_ota_no_external_sensors-reduced-ota-v1.18.0.1-halo-keymind-cascade-dev-adb11e7b.zip",
+      "zip_sha256": "00fb2075cd4ac74d5b5a121dcf63c7fdcd91743405ab14727daa67fe95e4942a",
+      "available_internal_bytes": 156928,
+      "required_heap_bytes": 58416,
+      "margin_bytes": 98512,
+      "required_contiguous_bytes": 10256,
+      "capabilities": [
+        "profile.auto",
+        "sensor.profile.reduced",
+        "ota.update.bluetooth",
+        "ota.update.lora",
+        "cli.retry_preset",
+        "telemetry.history",
+        "ota.cli",
+        "sensor.gps",
+        "sensor.ina219",
+        "sensor.ina226",
+        "sensor.ina260",
+        "sensor.ina3221",
+        "logging.usb.packets",
+        "logging.usb.control"
+      ],
+      "passed": true
+    },
+    {
+      "artifact": "RAK_3401_repeater_unified_lora_ota-full-ota-v1.18.0.1-halo-keymind-cascade-dev-adb11e7b.zip",
+      "zip_sha256": "de9e8a316a6d5806698892622521dba8430c874d4507391c62ccc90e0fc29914",
+      "available_internal_bytes": 78364,
+      "required_heap_bytes": 74176,
+      "margin_bytes": 4188,
+      "required_contiguous_bytes": 14304,
+      "capabilities": [
+        "profile.auto",
+        "sensor.profile.full",
+        "ota.update.bluetooth",
+        "ota.update.lora",
+        "cli.retry_preset",
+        "telemetry.history",
+        "ota.cli",
+        "sensor.gps",
+        "sensor.ina219",
+        "sensor.ina226",
+        "sensor.ina260",
+        "sensor.ina3221",
+        "logging.usb.packets",
+        "logging.usb.control"
+      ],
+      "passed": true
+    },
+    {
+      "artifact": "RAK_3401_repeater_unified_lora_ota-reduced-ota-v1.18.0.1-halo-keymind-cascade-dev-adb11e7b.zip",
+      "zip_sha256": "7d5cbe203e9115d7d3a82289b16730d67f39531a5c9d12630cea5890c464d71c",
+      "available_internal_bytes": 81368,
+      "required_heap_bytes": 74176,
+      "margin_bytes": 7192,
+      "required_contiguous_bytes": 14304,
+      "capabilities": [
+        "profile.auto",
+        "sensor.profile.reduced",
+        "ota.update.bluetooth",
+        "ota.update.lora",
+        "cli.retry_preset",
+        "telemetry.history",
+        "ota.cli",
+        "sensor.gps",
+        "sensor.ina219",
+        "sensor.ina226",
+        "sensor.ina260",
+        "sensor.ina3221",
+        "logging.usb.packets",
+        "logging.usb.control"
+      ],
+      "passed": true
+    }
+  ],
+  "rf": {
+    "rak4631_reduced": {
+      "passed": true,
+      "rounds_passed": 3,
+      "expected_rounds": 3,
+      "radio": {
+        "frequency_mhz": 917,
+        "bw_khz": 62.5,
+        "sf": 7,
+        "cr": 5,
+        "preamble": 32,
+        "temporary_lease_minutes": 2,
+        "target_tx_dbm": 2,
+        "witness_tx_dbm": 22
+      },
+      "core_before": {
+        "battery_mv": 4239,
+        "uptime_secs": 8,
+        "errors": 0,
+        "queue_len": 1
+      },
+      "core_after": {
+        "battery_mv": 4266,
+        "uptime_secs": 16,
+        "errors": 0,
+        "queue_len": 0
+      },
+      "receipt_sha256": "f15a342bdc1897a74f5c5106027d45f1436ddfe4a767e34cb4d5dcb6fa3937bd",
+      "application": {
+        "target": "RAK_4631_repeater_lora_ota_no_external_sensors",
+        "version": "v1.18.0.1-halo-keymind-cascade-dev-adb11e7b",
+        "application_sha256": "489d3bae98d7bda2c3f2b600d5bd79116685bc0a2ceb41c0d25fc7d0d3f0233d",
+        "body_hash": "427AFAF40AF4904C",
+        "zip_sha256": "84221bbb430e47028ea47dc11e9ac7ac2f62568d2ea134e7325b7ff54943bc03",
+        "runtime_role": "repeater",
+        "powersaving": "on"
+      }
+    },
+    "rak4631_full": {
+      "passed": true,
+      "rounds_passed": 3,
+      "expected_rounds": 3,
+      "radio": {
+        "frequency_mhz": 917,
+        "bw_khz": 62.5,
+        "sf": 7,
+        "cr": 5,
+        "preamble": 32,
+        "temporary_lease_minutes": 2,
+        "target_tx_dbm": 2,
+        "witness_tx_dbm": 22
+      },
+      "core_before": {
+        "battery_mv": 4283,
+        "uptime_secs": 8,
+        "errors": 0,
+        "queue_len": 1
+      },
+      "core_after": {
+        "battery_mv": 4236,
+        "uptime_secs": 16,
+        "errors": 0,
+        "queue_len": 0
+      },
+      "receipt_sha256": "f38a76152bacb4a267b916383912806f4055908ba763d0dc747b6c9398e3a739",
+      "application": {
+        "target": "RAK_4631_repeater_lora_ota_no_external_sensors",
+        "version": "v1.18.0.1-halo-keymind-cascade-dev-adb11e7b",
+        "application_sha256": "518fb91c1c053183e20e2927e278ba287718f355fd4de34a8488e84b338d6f31",
+        "body_hash": "D511EA44AFEACBA4",
+        "zip_sha256": "06c8e8df88f35deed3785dde37674ee5973a91cdba95e6f32c108f2df2a579a7",
+        "runtime_role": "repeater",
+        "powersaving": "on"
+      }
+    },
+    "rak3401_room_full": {
+      "passed": true,
+      "rounds_passed": 3,
+      "expected_rounds": 3,
+      "radio": {
+        "frequency_mhz": 917,
+        "bw_khz": 62.5,
+        "sf": 7,
+        "cr": 5,
+        "preamble": 32,
+        "temporary_lease_minutes": 2,
+        "target_tx_dbm": 2,
+        "witness_tx_dbm": 22
+      },
+      "core_before": {
+        "battery_mv": 4247,
+        "uptime_secs": 8,
+        "errors": 0,
+        "queue_len": 1
+      },
+      "core_after": {
+        "battery_mv": 4215,
+        "uptime_secs": 13,
+        "errors": 0,
+        "queue_len": 0
+      },
+      "receipt_sha256": "39edf68662db0080eec1013b31d2e17c8db0f701809d7899c05ba162267f0dd5",
+      "application": {
+        "target": "RAK_3401_room_server_lora_ota_no_external_sensors",
+        "version": "v1.18.0.1-halo-keymind-cascade-dev-adb11e7b",
+        "application_sha256": "31418a1a7d76bab0ee9b21b81dbdeaa8eb2123be1defeb0a4a3b95f6e664fac7",
+        "body_hash": "99642F2A9CB51F73",
+        "zip_sha256": "0f5cb79b19f2e9bebb05e6c7ce68cce7cad0e8093b45c3ec910e476c6f102908",
+        "runtime_role": "room_server",
+        "powersaving": "on"
+      }
+    }
+  },
+  "room_sleep": {
+    "passed": true,
+    "cpu_sleep_passed": true,
+    "weighted_sleep_fraction": 0.7930892728181768,
+    "sampling": {
+      "passed": true,
+      "failures": [],
+      "sample_count": 8035,
+      "counts": {
+        "C_DEBUGEN": 0,
+        "C_HALT": 0,
+        "C_STEP": 0,
+        "C_MASKINTS": 0,
+        "S_HALT": 0,
+        "S_SLEEP": 6374,
+        "S_LOCKUP": 0,
+        "S_RESET_ST": 0
+      },
+      "elapsed_ms": 90009,
+      "observed_ms": 90005.0,
+      "weighted_sleep_ms": 71382.0,
+      "weighted_sleep_fraction": 0.7930892728181768,
+      "unweighted_sleep_fraction": 0.7932794026135657,
+      "max_gap_ms": 16.0,
+      "max_read_ms": 3,
+      "baseline_debug_ctrl_hex": "0000",
+      "baseline_C_DEBUGEN": false,
+      "initial_sticky_reset": true,
+      "final_DHCSR": "01050000",
+      "openocd_version": [
+        "Open On-Chip Debugger 0.12.0+dev-snapshot (2025-07-16-14:15)"
+      ],
+      "qualification": "sampled CPU sleep observations, not exact residency or electrical current",
+      "firmware_identity_must_be_bound_separately": true
+    },
+    "core_before": {
+      "battery_mv": 4184,
+      "uptime_secs": 14,
+      "errors": 0,
+      "queue_len": 0
+    },
+    "core_after": {
+      "battery_mv": 4190,
+      "uptime_secs": 110,
+      "errors": 0,
+      "queue_len": 0
+    },
+    "ps_after": "on",
+    "capture_sha256": "0bd8925efa1115c6df88347c9a75b0570dd7e9ec1772a16c36b0e9f1896d476a",
+    "diagnostic_boolean_restored": true
+  },
+  "first_room_preparation": {
+    "stopped_before_cpu_capture": true,
+    "failure": "USB getter rejected: ota bootloader status",
+    "gateway_restored": true
+  },
+  "final_handback": {
+    "service_restored": true,
+    "core": {
+      "battery_mv": 4250,
+      "uptime_secs": 9,
+      "errors": 0,
+      "queue_len": 1
+    },
+    "version": "v1.18.0.1-halo-keymind-cascade-dev-adb11e7b (Build: 23-Oct-2026)",
+    "role": "repeater",
+    "radio": "917.0,62.5,7,5",
+    "powersaving": "on"
+  },
+  "mini": {
+    "application_bytes": 245696,
+    "application_limit_bytes": 245760,
+    "passed": true
+  }
+}
@@ -4,6 +4,10 @@

 # RS-232 bridge diagnostics, 22 October 2026

+Follow-up: the [RAK RAM repair and hardware checks](rak_release_fixes_2026-10-22.md)
+close the Full/Reduced build failure below at `adb11e7b4`. Physical RS232 cable
+and line-driver testing remain outside that follow-up's scope.
+
 Status: **host tests and ESP32 Full build PASS**. The RAK4631 Reduced
 sensor/LoRa OTA build passes, but its Full sensor build fails the RAM gate.
 The same Full failure occurs on the previous source, before these diagnostics.