diff --git a/docs/test-results/rak3401_radio2_fast_switch_2026-10-23.md b/docs/test-results/rak3401_radio2_fast_switch_2026-10-23.md new file mode 100644 index 000000000..b42432cdc --- /dev/null +++ b/docs/test-results/rak3401_radio2_fast_switch_2026-10-23.md @@ -0,0 +1,166 @@ + +

View this page on MeshCore Docs.

+ + +# RAK3401 radio2 fast switching, 23 October 2026 + +Status: **PARTIAL. Switching improved; the 500 us maximum goal is not met.** +The production candidate retains buffered 8 MHz SPI on SPI1/SPIM2. Private +16 MHz experiments failed radio initialization or returned incorrect radio +readback. They are not production defaults. **Fresh witnessed RF delivery +qualification failed: the scanner received 195 of 200 test packets.** + +## Timing checkpoints + +These are hardware measurements of the RX-to-RX profile-switch path, including +admission, standby, changed radio parameters and restarting receive. They are +not bare PLL-lock times. The production budget uses the sampled maximum plus +a rounded-up 10% margin; it is not a guarantee against a larger future sample. + +| Source | Change | Sampled maximum, us | Guarded budget, us | +| --- | --- | ---: | ---: | +| `adb11e7b` | Original 2 MHz, byte-at-a-time SPI | 1069 | 1176 | +| `99979655` | Buffered 2 MHz SPI | 855 | 941 | +| `258b37df` | Buffered 2 MHz plus modem/IQ caches | 702 | 773 | +| `b8b7880a` | Buffered 8 MHz plus caches | 550 | 605 | +| `1b97c089a` | Normal 8 MHz production image, longer RF campaign | 611 | 673 | + +The ordinary nRF52 switch budget uses the 32,768 Hz RTC2 clock, approximately +30.52 us per tick. A separate 64 MHz DWT instrumentation build from +`2629c158` captured 586 successful switches in each direction: + +| Direction | Minimum, us | Mean, us | Maximum, us | +| --- | ---: | ---: | ---: | +| Primary to secondary | 410 | 415.643 | 555 | +| Secondary to primary | 409 | 414.614 | 550 | + +Both directions recorded zero busy admissions and zero failed switches in +that capture. DWT stage counters, later interrupt observers and experimental +guards add work. Times from differently instrumented builds are not directly +comparable with each other or with production timing. A mean below 500 us +does not satisfy the requested maximum. + +## Why 16 MHz is not enabled + +The pure Repeater failed startup at 16 MHz on SPIM3 with RXDELAY settings +0, 1 and 2. Read-only SWD captures showed completed SPI frames, SoftDevice +disabled, and no observed fault-register evidence in those snapshots. This +does not support blaming those particular failures on Bluetooth or USB +enumeration. It also does not establish a hard clock limit for the pins. + +A narrower diagnostic initialized the radio successfully at 8 MHz, then +issued 16 read-only GetPacketType commands at each of 8, 16 and 8 MHz. No +radio writes or resets occurred between phases. The controller, pins and +explicit anomaly-198 workaround remained the same; the per-frame SoftDevice +safety check remained active. + +| RXDELAY | Initial 8 MHz correct / 16 | 16 MHz correct / 16 | Restored 8 MHz correct / 16 | +| ---: | ---: | ---: | ---: | +| 2 | 16 | 4 | 16 | +| 1 | 16 | 3 | 16 | +| 0 | 16 | 2 | 16 | + +The first two raw response bytes, `A2,A2`, are normal for these reads. The +failure is the final packet-type byte differing from the expected LoRa value +`01`. Restoring 8 MHz restored all 16 correct reads in every trial. This +isolates a reproducible fast-read problem after successful initialization; +it does not identify whether its cause is signal timing, the controller or +another part of the SPI path. No 16 MHz shipping claim is made. + +## Experimental scheduling and interrupt controls + +Task-priority elevation, bounded USB interrupt deferral and their explicitly +enabled combination were slower in the measured builds. All remain private, +default-off HIL options. The combined experiment did not achieve the 500 us +maximum goal. + +One combined worst-hop capture reported zero activity for all seven observed +interrupt/scheduler sources. Shared TWI0/SPIM0 coverage was then added in +`aa480ae79`; the exact linked vectors and scheduler call were verified before +the physical test. Both directions completed 591 successful switches with no +failed switches; the reverse direction also recorded 12 busy admissions. +The slowest forward switch took 659.109 us and +again recorded zero activity, now across all eight observed sources. The +reverse maximum was 575.234 us. SysTick ENABLE and TICKINT were both off when +read after stopping the test. These observations rule out attribution to +the recorded sources for that forward hop, without proving a particular +memory, peripheral or CPU-stall cause. Observer overhead and inclusive handler +timing also prevent treating these counters as a complete or additive +explanation of elapsed time. +The observer excludes architectural exception paths and the experiment's +RTC2 deadline handler; the USB guard recorded no deadline expiry. The retained +worst sample is selected by integer microseconds, so equal-microsecond ties +can differ in raw cycles. + +## Production scope and validation + +The RAK3401 production path uses buffered SPI1/SPIM2 at 8 MHz. It does not +enable the global nrfx SPIM3 anomaly-198 workaround, which is unsafe with an +enabled SoftDevice. The newly buffered, unqualified nRF default-SPI/SPIM3 +constructor rollout was restored to its prior defaults in `4e513b19d`. +Previously buffered LR1110 targets retain their prior constructors. Shared +cache improvements and the ESP32/RP2040 improvements remain. SenseCAP +Indicator is excluded, and MeshPocket has not been physically tested in this +campaign. + +Focused native checks passed for buffered transfers and constructor/bus +selection, cache invalidation, CPU-clock policy transitions, timing captures, +SoftDevice refusal, read-probe timeout/restoration/recovery decisions, and +default-off experiment gates. Relevant native fixtures execute the actual +helpers; sanitizer checks cover the diagnostic paths. These checks do not +replace hardware qualification. + +Cross-architecture build receipts include RAK3401, RAK4631, ThinkNode M4, +Heltec V2/V3, ESP32-C6 and Pico W targets. They cover campaign checkpoints, +not every board, firmware role or final-source image. The raw +`Station_G2_repeater` PlatformIO target exceeded its default 1,310,720-byte +application limit both at the baseline and after the changes; that +pre-existing overflow remains a separate build limitation. No all-builds or +all-boards release qualification is claimed here. + +## Fresh witnessed RF test + +The normal production image from `1b97c089a` was installed and its running +body hash matched the retained Full ELF/package. Two fixed receivers witnessed +the same 200 fresh, one-shot test packets, with no test-packet retransmission. +The settled recommendation was 32/192 symbols; the test selected 32/200. + +| Receiver | Primary received / sent | Secondary received / sent | +| --- | ---: | ---: | +| RAK3401 scanning both profiles | 96 / 100 | 99 / 100 | +| Fixed primary witness | 100 / 100 | - | +| Fixed secondary witness | - | 100 / 100 | + +There were no duplicate test packets, wrong-profile receipts, collector drops, +new core error flags, watchdog recoveries or switch-command errors. The RAK +recorded one additional receive error. The five-minute nonhalting CPU-status +capture passed its integrity gates: 27,134 samples, maximum gap 17 ms, no CPU +halt/debug enable, reset or lockup in the observation window. USB was active, +so this does not qualify CPU sleep or electrical power consumption. + +The gateway and RAK settings were restored. The sender's original transmit +power restoration initially returned a radio-busy error; a separate settings- +only cleanup continuation was required. The original failed receipt was +preserved. Packet delivery qualification and the 500 us goal both remain +failed; an offline audit independently reproduced the same five missing +packets. + +## Remaining work + +The [earlier witnessed RF run](rak3401_radio2_sf5_bw500_p384_2026-10-23.md) +lost three primary packets while its fixed witness received all 100. A +[recommended-preamble run](rak3401_radio2_recommended_preamble_2026-10-23.md) +also had scanner losses. The new test confirms that scanner packet loss +remains after these speed improvements. Faster switching alone has not fixed +delivery. + +The additional interrupt observer has completed its physical test. The normal +production image was built separately from `1b97c089a`, with all HIL helpers +absent and SPI1/SPIM2 selected; the later observer change affects HIL only. +Fresh witnessed RF testing of that production image is complete and failed. +The remaining work is to fix packet loss, explain and reduce the remaining +long switching intervals, and requalify the changed implementation. The +500 us maximum remains unmet. + +Machine-readable [sanitized results](rak3401_radio2_fast_switch_2026-10-23.safe.json) +record the same scope and failures. diff --git a/docs/test-results/rak3401_radio2_fast_switch_2026-10-23.safe.json b/docs/test-results/rak3401_radio2_fast_switch_2026-10-23.safe.json new file mode 100644 index 000000000..b748963fd --- /dev/null +++ b/docs/test-results/rak3401_radio2_fast_switch_2026-10-23.safe.json @@ -0,0 +1,183 @@ +{ + "schema": 1, + "date": "2026-10-23", + "board": "RAK3401 with external SX1262", + "role": "repeater", + "status": "partial", + "maximum_switch_goal_us": 500, + "performance_goal_passed": false, + "production_source": "1b97c089a735ebe5ddb2f39937c685994f85850e", + "production": { + "spi_controller": "SPI1/SPIM2", + "spi_hz": 8000000, + "buffered_transfers": true, + "iq_modem_cache": true, + "experimental_priority_guard": false, + "experimental_usb_irq_guard": false, + "global_spim3_anomaly198_workaround": false, + "hil_symbols_present": false + }, + "profiles": { + "primary": { + "frequency_mhz": 917, + "bandwidth_khz": 62.5, + "sf": 7, + "cr": 5, + "preamble": 32 + }, + "secondary": { + "frequency_mhz": 918, + "bandwidth_khz": 500, + "sf": 5, + "cr": 5, + "preamble": 200 + } + }, + "timing_checkpoints": [ + { + "source": "adb11e7b", + "sampled_maximum_us": 1069, + "guarded_budget_us": 1176 + }, + { + "source": "99979655", + "sampled_maximum_us": 855, + "guarded_budget_us": 941 + }, + { + "source": "258b37df", + "sampled_maximum_us": 702, + "guarded_budget_us": 773 + }, + { + "source": "b8b7880a", + "sampled_maximum_us": 550, + "guarded_budget_us": 605 + }, + { + "source": "1b97c089a", + "sampled_maximum_us": 611, + "guarded_budget_us": 673, + "scope": "longer normal production RF campaign" + } + ], + "spi16_read_probe": { + "softdevice_enabled": false, + "reads_per_phase": 16, + "phase_hz": [ + 8000000, + 16000000, + 8000000 + ], + "correct_packet_types_by_rxdelay": { + "2": [ + 16, + 4, + 16 + ], + "1": [ + 16, + 3, + 16 + ], + "0": [ + 16, + 2, + 16 + ] + }, + "radio_writes_or_resets_between_phases": false, + "qualified": false + }, + "combined_irq_experiment": { + "source": "aa480ae79e46bf63e10927de0327df0271b89505", + "abi": 2, + "enabled_in_production": false, + "successful_switches_per_direction": [ + 591, + 591 + ], + "busy_admissions_per_direction": [ + 0, + 12 + ], + "failed_switches_per_direction": [ + 0, + 0 + ], + "worst_forward_us": 659.109375, + "worst_forward_observed_irq_or_scheduler_calls": 0, + "observed_sources": [ + "rtc1", + "gpiote", + "uart0", + "uart1", + "power_clock", + "usb", + "scheduler", + "twim0_spim0" + ], + "cause_established": false, + "systick_enabled_after_stop": false, + "systick_interrupt_enabled_after_stop": false + }, + "witnessed_rf": { + "status": "failed", + "qualification_passed": false, + "source": "1b97c089a735ebe5ddb2f39937c685994f85850e", + "test_packets_sent": 200, + "test_packet_retries": 0, + "scanner": { + "primary_received": 96, + "primary_sent": 100, + "secondary_received": 99, + "secondary_sent": 100 + }, + "fixed_primary_witness": { + "received": 100, + "sent": 100 + }, + "fixed_secondary_witness": { + "received": 100, + "sent": 100 + }, + "duplicates": 0, + "wrong_profile_receipts": 0, + "collector_drops": 0, + "new_core_error_flags": 0, + "new_receive_errors": 1, + "watchdog_recoveries": 0, + "switch_command_errors": 0, + "guarded_budget_before_rf_us": 673, + "guarded_budget_after_rf_us": 673, + "selected_preambles": [ + 32, + 200 + ], + "settled_recommended_preambles": [ + 32, + 192 + ], + "cpu_capture": { + "duration_ms": 300004, + "samples": 27134, + "integrity_passed": true, + "maximum_gap_ms": 17.0, + "halt_debug_reset_lockup_observed": false, + "sleep_qualified": false + }, + "gateway_restored": true, + "sender_cleanup_initially_failed": true, + "sender_cleanup_continuation_passed": true, + "original_failed_receipt_preserved": true, + "report_sha256": "cb7bd8c44a9e26cb131c2c4f123cd323163d7504f7f4cbfd3ed63f0fb5d87b89", + "swd_log_sha256": "e78301c78fda379e6f4b11f6095c4b969e4a7a1e81e25a1d34147f0c99b08e5d", + "cleanup_continuation_sha256": "b1929df2d2c6bc31b966773aa357f63778647c8ee67718a88204cbb031a3666d" + }, + "scope_limits": [ + "No all-board or release qualification", + "No MeshPocket physical testing", + "Indicator excluded", + "New unqualified nRF SPIM3 constructor rollout restored to prior defaults" + ] +} diff --git a/docs/test-results/rak3401_radio2_sf5_bw500_p384_2026-10-23.md b/docs/test-results/rak3401_radio2_sf5_bw500_p384_2026-10-23.md index b32d6ad0f..25c3a77d2 100644 --- a/docs/test-results/rak3401_radio2_sf5_bw500_p384_2026-10-23.md +++ b/docs/test-results/rak3401_radio2_sf5_bw500_p384_2026-10-23.md @@ -114,3 +114,7 @@ were removed, RAK live radio2 status returned to off, original TX power and USB logging were restored, and the gateway service reacquired its radio with its configuration unchanged. The independent offline audit reproduced the three missing primary packets and reported no measurement confounds. + +See the [fast-switch follow-up](rak3401_radio2_fast_switch_2026-10-23.md) +for buffered SPI measurements, the 16 MHz readback failure and subsequent +qualification results.