diff --git a/docs/test-results/hardware_validation_checklist.md b/docs/test-results/hardware_validation_checklist.md index 685a3e0b5..50d867015 100644 --- a/docs/test-results/hardware_validation_checklist.md +++ b/docs/test-results/hardware_validation_checklist.md @@ -14,6 +14,34 @@ when its log identifies the physical device, firmware artifact, artifact hash, command result, and cold/warm boot outcome. Do not infer success from a tool's exit code when the tool has a documented false-success mode. +## Known-failure follow-up, 2026-10-20/21 UTC + +The [follow-up report](hardware_validation_known_failures_1_18_0_1_2026-10-20.md) +binds builds, host tests and hardware observations to their actual sources. +It preserves the earlier failures and the accepted fleet results. + +- [x] **Current V4 Full logging-on idle/reopen:** shipping source `0b207a1ec` + passed case 3: 600.0913 seconds, zero idle writes, 105.3/20.5 ms post-idle + reply/reopen, continuous uptime and zero core errors. Watchdog recoveries + stayed 2 -> 2 and boot count 0 -> 0. Original settings, strict binary + handback, TTY/browser/stream cleanup passed. Complete application readback + matched the shipping image; partition table/OTA metadata were unchanged. +- [x] **Morse receive setting floor:** configured reception now requires at + least 15 WPM, with default 25; TX at 1-14 WPM remains available. The operator + accepts false letters for now. Earlier quiet/message failures remain recorded; + complete-message accuracy and an absolute physical speed cutoff are not + inferred from setting validation. + +The independent eight-board nRF52 test-bootloader deployment and original-app +handoffs are complete with documented USB concurrency and RAK secondary-data +preservation limits. This is separate from final-source Companion qualification. +The nRF52 testing hold is released; MeshPocket remains excluded. + +The current V4 differs from the physical board in the 18 October failure below. +Its follow-up cannot establish that earlier USB disappearance's cause or erase +the failed historical measurement. Remaining release gates keep their latest +documented status. + ## Mercerwood candidate campaign, 2026-10-18 UTC The [current hardware campaign](hardware_validation_mercerwood_1_18_0_1_2026-10-18.md) diff --git a/docs/test-results/hardware_validation_known_failures_1_18_0_1_2026-10-20.md b/docs/test-results/hardware_validation_known_failures_1_18_0_1_2026-10-20.md new file mode 100644 index 000000000..1ab161ea9 --- /dev/null +++ b/docs/test-results/hardware_validation_known_failures_1_18_0_1_2026-10-20.md @@ -0,0 +1,134 @@ + +

View this page on MeshCore Docs.

+ + +# Known-failure follow-up, 20-21 October 2026 UTC + +Status: **V4 USB PASS**; Morse false letters are an accepted limitation. +Final-source CI remains in progress. This scoped follow-up preserves failed +measurements and the operator's limited Morse acceptance. It does not grant broader release approval. The current V4 is a +different physical board from the 18 October campaign. + +Final source: `0b207a1ec3a32eb0a8ad6b8643db211de4921cd4`. +[Safe evidence summary](hardware_validation_known_failures_1_18_0_1_2026-10-20.safe.json) +records source bindings, evidence digests and explicit pending fields. + +## Builds and host verification + +| Check | Source | Result | +| --- | --- | --- | +| RAK4631 Full | `0b207a1ec` | PASS: 712432/712704 application bytes, 272 bytes free | +| T096 Full, FEM on | `0b207a1ec` | Shipping build/package PASS | +| V4.3 Full, FEM off | `0b207a1ec` | Shipping build/package PASS: 2559496 application bytes | +| Focused Morse host suite | `0b207a1ec` | 111 methods PASS | +| Targeted USB host suite | `7278eacb9`, included in final source | 45 methods PASS, including 12 HWCDC TX backport suite methods with actual ISR scenarios | +| Earlier unchanged RadioProfile suite | `fc72943f3` | 17 methods PASS; not rerun for final source | + +RAK Full retains all 13 feature capabilities plus `profile.full`; its build +used INA260 1.5.4 and MLX90614 2.1.7. Builds and host tests do not qualify RF +behavior or physical USB stability. In +[CI run 39523154379](https://github.com/mikecarper/MeshCore/actions/runs/39523154379), +all 11 non-main jobs are reported green; the main host job is **PENDING**. +A completed workflow pass is not yet claimed. + +## V4 application and native USB + +The earlier `4773ac4219666d0f5b4c2bff5eb72aa053014662` application passed full +readback with unchanged partition table and OTA metadata. An initial private +adapter's ordinary hard reset left native S3 in forced download mode despite +correct readback. The pinned flasher's masked forced-download clear and +watchdog reset restored the application. Its adapter guidance was clarified in +meshfirmware commit [4ce4f44](https://github.com/mikecarper/meshfirmware/commit/4ce4f44); +the flasher already had the correct sequence. + +The second logging-on native-console trial on `4773ac42` passed 600.1086 +seconds with zero idle command writes. The observed idle phase started at +2026-10-21 00:13:03.667 UTC. First reply/reopen took 124/21.8 ms; uptime advanced +378 -> 979 -> 981 seconds. Identity, settings restoration, strict binary handback +and cleanup passed, with zero core errors. However, watchdog recoveries rose +from 0 to 2; the last reason was client inactivity (`0x08`). This is a transport +PASS, not a clean watchdog qualification. The earlier browser startup failure +and its successful restoration remain recorded separately. + +Fix `7278eacb9131d613da719fee814fefcdad3adf23`, included in final source, +requires 100 ms of continuous presence loss before invalidating the valid +client lease and recognizes the host-backed physical reader. It preserves +bus-reset, stall and true-unplug recovery. The finding does not conclusively +attribute both earlier events to that path. + +Ordinary shipping `0b207a1ec` was restored successfully at +2026-10-21 01:21:32 UTC. All 2559496 application bytes matched SHA256 +`04eed20ee0f964fef2a5a0f74f4aafe4b11e0a71b011ff757efc96c03a2cfb14`; +partition table and OTA metadata stayed unchanged. Runtime source/identity, +absence of the raw diagnostic marker, strict binary handback and clean core +state passed. This was an application-only update. + +Final logging-on native-console case 3 on this shipping image: **PASS**. +The idle lasted 600.0913 seconds with zero command writes; post-idle reply and +reopen took 105.3/20.5 ms. Uptime advanced 172 -> 773 -> 775 seconds, with +zero core errors. Watchdog recoveries stayed 2 -> 2 and boot count 0 -> 0. +The watchdog's `boot` field counts reset authorizations, not all physical boots; +the continuous uptime separately establishes no MCU restart during this case. +Identity, original preferences, strict binary handback, TTY restoration and +owned browser/stream cleanup passed. The wrapper finished at +2026-10-21 01:34:33 UTC without errors. This qualifies the current board and +artifact for this finite logging-on idle/reopen case; it does not establish +the cause of the different 18 October board's USB disappearance. + +## Morse measurements and accepted limitation + +These shipping controls use automatic detection without HIL threshold overrides. + +| No intentional TX, 30 seconds | RAK4631 | T096 | +| --- | --- | --- | +| Unchanged `08650f64` | FAIL: 3 false characters | FAIL: 1 false character | +| Adaptive `4773ac42` | PASS: 0 characters in this control | UNQUALIFIED: 6 retained characters and command timeout during external-actor contention | + +The contended T096 observation is not a clean detector pass/failure attribution. +The RAK control alone does not qualify complete messages or other board families. + +An unchanged `08650f64` V3 transmitted to a shipping `4773ac42` V4. The quiet +control came first: **FAIL**, 26 false characters in 300.820761 seconds. +Three messages at 25/60 WPM, two repeats, then produced **4/12 exact passes and +8 failures**. Both preferences and strict binary handbacks passed; the wrapper +reported no errors. This one-way campaign does not replace the original two-way +24-message matrix. + +The operator accepted false letters for now and stopped further detector/gate +work. That accepts the false-letter limitation; it does not establish complete +message accuracy or turn the failed measurements into passes. Further Morse +experiments are not a mandatory gate in this follow-up. The receive setting +minimum is 15 WPM, default 25; TX at 1-14 WPM remains available. Hand-timing +tolerance remains, so setting validation is not an absolute physical speed cutoff. + +Separate raw diagnostic captures used production RF settings and automatic +squelch. Capture integrity, both preference restorations and both strict binary +handbacks passed. These instrumented captures do not qualify shipping accuracy. +The ordinary shipping image is restored; raw lab details remain private. + +## External nRF52 bootloader handback + +The operator reported completion of their separate bootloader testing. Its +deployment record was also read: **COMPLETE_WITH_KNOWN_LIMITATIONS**, eight live +INFO/original-application handoff proofs and gateway service handback PASS. +It uses unpublished test bootloader `TEST_0x02040B7E`, source +`b07255d4cdc601f1313e4cbe020caad3b5544821`, not a latest-stable release. +The record reports no publication or push by that deployment. + +Limits remain: scheduled concurrent USB control failed with one EPIPE and 31 +successful requests; 11 unique bulk halt-clear cycles passed. RAK secondary-data +preservation is uncertain after earlier manual scratch overlap. Identity and +seven settings match, without a complete immediately prior contact/channel +baseline. These records do not qualify final-source `0b207a1ec` applications. +The nRF52 hold is released; no new nRF52 campaign was started. MeshPocket remains +excluded from all further testing in this session. + +## Remaining scope + +Current CI completion and final release artifact-matrix binding remain pending. +The scoped V4 logging-on hardware case is complete. The [20 October fleet qualification](hardware_validation_mercerwood_1_18_0_1_2026-10-20.md) +retains its accepted USB/BLE/RF/contact coverage at its actual sources and +conditions. This report neither reopens those closures nor extends them to +unmeasured artifacts. Historical failures in the +[18 October report](hardware_validation_mercerwood_1_18_0_1_2026-10-18.md) +remain visible. Private device identities, addresses and raw responses are omitted. diff --git a/docs/test-results/hardware_validation_known_failures_1_18_0_1_2026-10-20.safe.json b/docs/test-results/hardware_validation_known_failures_1_18_0_1_2026-10-20.safe.json new file mode 100644 index 000000000..c3f356b40 --- /dev/null +++ b/docs/test-results/hardware_validation_known_failures_1_18_0_1_2026-10-20.safe.json @@ -0,0 +1,256 @@ +{ + "schema": 1, + "report_date": "2026-10-20", + "campaign_dates_utc": [ + "2026-10-20", + "2026-10-21" + ], + "status": "V4_USB_PASS_MORSE_LIMITATION_ACCEPTED_CI_PENDING", + "release_approval": false, + "source_commit": "0b207a1ec3a32eb0a8ad6b8643db211de4921cd4", + "scope": { + "current_v4_differs_from_18_october_board": true, + "nrf52_hold_released_by_operator": true, + "new_nrf52_final_source_hardware_qualification_claimed": false, + "mesh_pocket_future_testing_excluded": true, + "existing_fleet_closures_reopened": false, + "device_identifiers_and_host_addresses_omitted": true + }, + "builds": [ + { + "target": "RAK_4631_companion_radio_full", + "source_commit": "0b207a1ec3a32eb0a8ad6b8643db211de4921cd4", + "status": "PASS", + "application_bytes": 712432, + "application_limit_bytes": 712704, + "free_bytes": 272, + "feature_capability_count": 13, + "profile_full_marker": true, + "ina260_library_version": "1.5.4", + "mlx90614_library_version": "2.1.7", + "artifact_filename": "RAK_4631_companion_radio_full-v1.18.0.1-halo-keymind-cascade-dev-0b207a1e.zip", + "artifact_sha256": "3126f6eae26a640985bfaeed68fbc2c65f6f99857b326d6d086a35aec8b30f13", + "artifact_bytes": 713315 + }, + { + "target": "Heltec_t096_companion_radio_full_femon", + "source_commit": "0b207a1ec3a32eb0a8ad6b8643db211de4921cd4", + "status": "PASS", + "artifact_filename": "Heltec_t096_companion_radio_full_femon-v1.18.0.1-halo-keymind-cascade-dev-0b207a1e.zip", + "artifact_sha256": "95bcc8c2ba4456a646b21928589c50e17ebd26f8bd866dd01e6862c55ed4d85f", + "artifact_bytes": 646782 + }, + { + "target": "heltec_v4_3_companion_radio_full_femoff", + "source_commit": "0b207a1ec3a32eb0a8ad6b8643db211de4921cd4", + "status": "PASS", + "application_bytes": 2559496, + "artifact_sha256": "04eed20ee0f964fef2a5a0f74f4aafe4b11e0a71b011ff757efc96c03a2cfb14", + "instrumentation_hil": false, + "artifact_filename": "heltec_v4_3_companion_radio_full_femoff-v1.18.0.1-halo-keymind-cascade-dev-0b207a1e.bin", + "artifact_bytes": 2559496 + } + ], + "host_checks": { + "morse": { + "source_commit": "0b207a1ec3a32eb0a8ad6b8643db211de4921cd4", + "status": "PASS", + "methods": 111, + "elapsed_seconds": 58.472, + "log_sha256": "90935b6e37090a425931c98b528d7f56cbd1c9639ca5181db59ec6158e48d27e" + }, + "targeted_usb": { + "source_commit": "7278eacb9131d613da719fee814fefcdad3adf23", + "included_in_final_source": true, + "status": "PASS", + "methods": 45, + "included_hwcdc_tx_backport_suite_methods": 12, + "backport_suite_includes_actual_isr_scenarios": true + }, + "radio_profile": { + "source_commit": "fc72943f3bbdcb2d587a39441c82a461d5df68e4", + "status": "PASS", + "methods": 17, + "unchanged_after_run": true, + "rerun_for_final_source": false + } + }, + "ci": { + "run_id": 39523154379, + "url": "https://github.com/mikecarper/MeshCore/actions/runs/39523154379", + "status": "PENDING", + "non_main_jobs_reported_green": 11, + "main_host_job": "PENDING", + "completed_workflow_pass_claimed": false + }, + "v4_native_console": { + "historical_logging_on_case2": { + "source_commit": "4773ac4219666d0f5b4c2bff5eb72aa053014662", + "status": "TRANSPORT_PASS_WATCHDOG_NOT_QUALIFIED", + "observed_idle_started_utc": "2026-10-21T00:13:03.667Z", + "idle_seconds": 600.1086, + "idle_command_writes": 0, + "first_post_idle_reply_ms": 124, + "reopened_reply_ms": 21.8, + "uptime_seconds": [ + 378, + 979, + 981 + ], + "core_error_flags": 0, + "watchdog_recoveries_before": 0, + "watchdog_recoveries_after": 2, + "last_watchdog_reason": "0x08", + "last_watchdog_reason_meaning": "client_inactivity", + "identity_settings_binary_handback_cleanup": "PASS", + "native_safe_evidence_sha256": "8f5f9e9060ab1643575caf121bd1ee0cf703cecf0e02548f5b9c6c7f85701adf", + "earlier_browser_startup_failure_preserved": true + }, + "presence_fix": { + "source_commit": "7278eacb9131d613da719fee814fefcdad3adf23", + "included_in_final_source": true, + "continuous_presence_loss_debounce_ms": 100, + "host_backed_physical_reader_recognized": true, + "bus_reset_stall_and_true_unplug_recovery_preserved": true, + "both_historical_events_conclusively_attributed": false + }, + "ordinary_shipping_restore": { + "source_commit": "0b207a1ec3a32eb0a8ad6b8643db211de4921cd4", + "status": "PASS", + "finished_utc": "2026-10-21T01:21:32.396048Z", + "application_only": true, + "application_bytes": 2559496, + "artifact_and_readback_sha256": "04eed20ee0f964fef2a5a0f74f4aafe4b11e0a71b011ff757efc96c03a2cfb14", + "whole_application_readback_match": true, + "partition_table_unchanged": true, + "otadata_unchanged": true, + "runtime_source_identity_and_binary_handback": "PASS", + "raw_diagnostic_marker_absent": true, + "core_error_flags": 0, + "summary_sha256": "ee6a5fa3dbbf8c902e2b2f418831730ef6d91ca42d314dba5b3dd67ef59a1203" + }, + "final_logging_on_case3": { + "source_commit": "0b207a1ec3a32eb0a8ad6b8643db211de4921cd4", + "status": "PASS", + "required_idle_seconds": 600, + "required_unchanged_recovery_and_boot_counters": true, + "measured_idle_seconds": 600.0913000000119, + "recovery_counters": { + "before": 2, + "after": 2 + }, + "boot_counters": { + "before": 0, + "after": 0 + }, + "identity_uptime_settings_and_binary_handback": "PASS", + "safe_evidence_sha256": "75f6bfa7738269d4574f63d6efa6a6795282dd7b6ba6f2694a1eaa5e1f0097ac", + "idle_command_writes": 0, + "summary_sha256": "f674b83d34a64e5a5ae94411303f42ad7448b3c4a1d31a9723b8086e6e66dc48", + "first_post_idle_reply_ms": 105.30000001192093, + "reopened_reply_ms": 20.5, + "uptime_seconds": [ + 172, + 773, + 775 + ], + "core_error_flags": 0, + "original_preferences_restored": true, + "original_tty_restored": true, + "native_streams_and_owned_browser_closed": true, + "finished_utc": "2026-10-21T01:34:33.510254+00:00" + } + }, + "shipping_morse": { + "automatic_detection": true, + "hil_threshold_overrides": false, + "baseline_quiet_30_seconds": { + "source_commit": "08650f644b680a5dfaacb95142e3cef8529da9d2", + "status": "FAIL", + "rak_false_characters": 3, + "t096_false_characters": 1 + }, + "adaptive_quiet_30_seconds": { + "source_commit": "4773ac4219666d0f5b4c2bff5eb72aa053014662", + "rak_status": "PASS_CONTROL_ONLY", + "rak_false_characters": 0, + "t096_status": "UNQUALIFIED_CONTENDED", + "t096_retained_characters": 6, + "t096_first_command_timeout": true, + "t096_detector_failure_attributed": false + }, + "esp32_one_way_campaign": { + "transmitter_source_commit": "08650f644b680a5dfaacb95142e3cef8529da9d2", + "receiver_source_commit": "4773ac4219666d0f5b4c2bff5eb72aa053014662", + "status": "FAIL", + "quiet_control_first": true, + "quiet_seconds": 300.82076147598855, + "quiet_false_characters": 26, + "messages": 3, + "rates_wpm": [ + 25, + 60 + ], + "repeats": 2, + "exact_message_passes": 4, + "exact_message_failures": 8, + "substitutes_for_original_two_way_24_trial_matrix": false, + "both_preferences_restored": true, + "both_strict_binary_handbacks": "PASS", + "wrapper_errors": [], + "summary_sha256": "40b5d555c177d81ea11a1fa71e73dafb0e015ee4e25d146a82bf986873bc3405" + }, + "operator_acceptance": { + "scope": "current_false_letters_limitation", + "accepted": true, + "further_detector_gate_work_stopped": true, + "mandatory_additional_morse_experiments": false, + "complete_message_accuracy_acceptance_or_qualification_inferred": false, + "failed_measurements_relabelled": false + }, + "final_source_settings": { + "receive_minimum_configured_wpm": 15, + "default_wpm": 25, + "transmit_1_through_14_wpm_available": true, + "hand_timing_tolerance_retained": true, + "absolute_physical_speed_cutoff_claimed": false + }, + "separate_raw_diagnostics": { + "source_commit": "0b207a1ec3a32eb0a8ad6b8643db211de4921cd4", + "instrumentation_hil": true, + "production_rf_settings": true, + "automatic_squelch": true, + "capture_integrity": "PASS", + "both_preferences_restored": "PASS", + "both_strict_binary_handbacks": "PASS", + "ordinary_shipping_image_restored": true, + "shipping_accuracy_qualification": false, + "further_analysis_stopped_at_operator_request": true, + "completed_capture_summary_sha256": "b1d06e6ed98273e29da28601f9a65b273a40d0f295e400f93fc99d12c90c9687" + } + }, + "external_nrf52_bootloader_deployment": { + "evidence_basis": "operator_report_and_independently_read_deployment_record", + "status": "COMPLETE_WITH_KNOWN_LIMITATIONS", + "source_commit": "b07255d4cdc601f1313e4cbe020caad3b5544821", + "test_bootloader_version": "TEST_0x02040B7E", + "latest_stable_release_claimed": false, + "live_info_and_original_application_handoffs": 8, + "gateway_service_handback": "PASS", + "published_or_pushed_by_deployment": false, + "scheduled_concurrent_control": { + "status": "FAIL", + "epipe_errors": 1, + "successful_requests": 31 + }, + "unique_bulk_halt_clear_cycles_passed": 11, + "rak_secondary_data_preservation": "UNCERTAIN_AFTER_EARLIER_MANUAL_SCRATCH_OVERLAP", + "rak_identity_and_seven_settings_match": true, + "rak_complete_immediately_pre_scratch_contact_channel_baseline_available": false, + "final_source_application_qualification_claimed": false, + "installed_safe_record_sha256": "297cdfa58fb7d0420694422b680ee25de7d267aa81436911389ea9a766c3110a" + }, + "pending": [ + "current_ci_completion_and_final_release_artifact_matrix_binding" + ] +} diff --git a/docs/test-results/hardware_validation_mercerwood_1_18_0_1_2026-10-20.md b/docs/test-results/hardware_validation_mercerwood_1_18_0_1_2026-10-20.md index 10876a0cc..e236a788c 100644 --- a/docs/test-results/hardware_validation_mercerwood_1_18_0_1_2026-10-20.md +++ b/docs/test-results/hardware_validation_mercerwood_1_18_0_1_2026-10-20.md @@ -23,6 +23,13 @@ Seven active nRF52 boards retain the source `88d39510140ce6894cf8e74b93cbd28d7387fa21` bootloader baseline; Companion application-only updates transferred no bootloader or SoftDevice data. +That bootloader baseline describes this campaign's measurements. A subsequent +independent deployment restored the original applications after installing an +unpublished test bootloader on eight nRF52 boards. Its remaining USB stress +and RAK data-preservation limits are recorded in the +[known-failure follow-up](hardware_validation_known_failures_1_18_0_1_2026-10-20.md). +This does not repeat or extend the source-bound qualifications below. + | Board | Installed source | Required USB / BLE / RF / contacts | Overall | | --- | --- | --- | --- | | Heltec V4.3, FEM off | 0865 | USB/BLE/contacts PASS; RF retained ABF | PASS |