diff --git a/.github/workflows/run-unit-tests.yml b/.github/workflows/run-unit-tests.yml index 9c79d294..18f277d1 100644 --- a/.github/workflows/run-unit-tests.yml +++ b/.github/workflows/run-unit-tests.yml @@ -271,7 +271,9 @@ jobs: python3 -B test/test_esp32_full_power_lifecycle.py - name: Verify official web console framing and power-saving idle sessions - run: python3 -B test/test_official_web_console.py -v + run: | + python3 -B test/test_official_web_console.py -v + python3 -B test/test_hil_linux_web_serial.py -v - name: Verify official setup GUI initial time command deadline run: python3 -B test/test_official_config_gui.py -v diff --git a/docs/research/index.md b/docs/research/index.md index f7524f43..dfbc3972 100644 --- a/docs/research/index.md +++ b/docs/research/index.md @@ -10,6 +10,7 @@ Engineering investigations, design notes, diagnostic methods, and historical upd ## Diagnostics and update-chain studies +- [Linux Chromium Web Serial after pyserial](linux_chromium_web_serial_tty.md) - [nRF52 Companion Bluetooth diagnostic firmware](nrf52_bluetooth_debug.md) - [RAK3401 compact LoRa update chain](rak3401_mota_chain.md) - [RAK3401 1.16.07 to 1.17.1.7 with legacy bootloader](rak3401_1.17.1.7_legacy24.md) diff --git a/docs/research/linux_chromium_web_serial_tty.md b/docs/research/linux_chromium_web_serial_tty.md new file mode 100644 index 00000000..b6cd48ab --- /dev/null +++ b/docs/research/linux_chromium_web_serial_tty.md @@ -0,0 +1,85 @@ +# Linux Chromium Web Serial after pyserial + +A Station G2 browser timeout on firmware source +`1e421ff4fc6a3b56d43db7a18aab94a72129b527` was traced to the Linux host's +inherited TTY read settings. Chromium `139.0.7258.127` closed its receive stream +with `NetworkError` / device lost before the first `ver` request. The radio was +still running and an exclusive pyserial query answered in 21.330 ms without a +reset or replug. This finding explains this controlled host failure; it does +not establish that every reported USB hang has the same cause. + +## Why an idle port looked disconnected + +pyserial normally sets `VMIN=0, VTIME=0` and performs its timeout handling with +`select`. These kernel TTY attributes survive closing the port. With those +settings, an idle nonblocking Linux `read()` can return zero bytes. +[pyserial's POSIX implementation](https://github.com/pyserial/pyserial/blob/v3.5/serial/serialposix.py) +sets these values and closes without restoring the previous attributes. + +In the exact +[Chromium 139 serial implementation](https://github.com/chromium/chromium/blob/139.0.7258.127/services/device/serial/serial_io_handler_posix.cc#L152), +port configuration preserves the prior `VMIN` and `VTIME`. Its +[receive handler](https://github.com/chromium/chromium/blob/139.0.7258.127/services/device/serial/serial_io_handler_posix.cc#L300) +interprets a zero-byte read as device loss. Setting `VMIN=1, VTIME=0` makes an +idle nonblocking read return `EAGAIN`, so the browser waits for data instead. + +Upstream Chromium fixed this in +[commit 45ccdf6e](https://github.com/chromium/chromium/commit/45ccdf6e7af9b56473ab9588c44d9048a8aaf7fb) +on 2026-09-18, under bug 559872992. The change explicitly configures both read +settings and adds a real Linux pseudoterminal regression for inherited settings. +The presence of this fix upstream does not prove a particular installed browser +contains it; the Pi's tested Chromium 139 predates the fix. + +## Controlled G2 comparison + +The same firmware, native Pi Chromium, and unmodified official flasher console +were used for both attempts. Console source SHA-256 was +`107d3ad867915aea8b2454c324327397a0f9e2f0469656046054011bc64475f9`. +Instrumentation observed and forwarded existing native API calls. + +| Attempt | Host settings and observation | Result | +| --- | --- | --- | +| 203 | `VMIN=0, VTIME=0`. The receive stream was already lost before Enter. The native five-byte `ver\r\n` write nevertheless completed in 28.1 ms. | FAIL. No displayed reply by the unchanged 5,000 ms deadline. | +| 204 | Only host `VMIN` changed to 1; `VTIME` stayed 0. The receive stream remained open. | PASS. First ordinary `ver` reply was displayed in 154.1 ms. | + +No firmware or website patch, modem-signal change, reset, replug, added command +retry, keepalive, or longer deadline was used to obtain the passing comparison. +Before this comparison, `stats-core` reported uptime 1,728 seconds and error +bitmask 2; this is evidence of continued operation, not an error-free radio +qualification. The comparison does not qualify a fresh flash, a long idle +window, another host OS/browser, or every USB transport. + +Safe receipt names and hashes are retained for the comparison: + +| Receipt | SHA-256 | +| --- | --- | +| `stock-browser-diagnostic-203-safe.json` | `e41b0f009b8dce42ad9ee23ad843032b97a47fc2cfc5385aefbf093ac46a9788` | +| `controlled-browser-host-setting-safe.json` | `88527690ca8d8740fd415831123c45ac3249134aec3929f7fd2b0224ddbe8fce` | +| `stock-browser-diagnostic-204-safe.json` | `6ba3249a23364a589653a0a1a4abb7e453a1cb2f095cec6138f3cebc171d5f39` | + +## Hardware-test preflight + +The tracked +[Linux Web Serial helper](https://github.com/mikecarper/MeshCore/blob/keymindCascade/tools/hil/linux_web_serial.py) +changes and verifies only `VMIN=1, VTIME=0` immediately before a native browser +test. It requires an explicit absolute TTY path and restores the complete +original termios attributes after the browser closes, including on a Python +exception or interrupt. Failed preparation also restores the original state. +It sends no serial data and makes no DTR/RTS or USB-reset calls. + +Pause other serial clients and finish all pyserial preflight queries before +entering the context. Another opener can overwrite the read settings, and a +second reader can consume the radio's reply. The helper closes its preparation +descriptor before the trial, so it does not keep the port open across the +stock browser's Disconnect action. Controllers must close their native reader, +writer and port in their own `finally` cleanup before the context exits, and +route termination into that cleanup. If the endpoint is replaced or disappears, +restoration fails explicitly rather than applying old attributes to a different +device. + +Record browser version, TTY settings, native stream state/error, native write +completion, raw receive-byte count and rendered reply separately. A zero-byte +console display alone does not prove that the USB endpoint received no bytes. +For example, the stock console buffers input until a complete CRLF line arrives. +See the [HIL instructions](https://github.com/mikecarper/MeshCore/blob/keymindCascade/test/README.md#linux-native-web-serial-preflight) +for the context-manager example and the hardware-free pseudoterminal test. diff --git a/test/README.md b/test/README.md index 7d110e36..a310169c 100644 --- a/test/README.md +++ b/test/README.md @@ -62,6 +62,36 @@ With a sandboxed browser that cannot read `/tmp`, set `TMPDIR` to a writable directory visible to that browser before running it. The Bluetooth settings contracts and WebConfig suites also run in the unit-test GitHub workflow. +### Linux native Web Serial preflight + +After the last pyserial query, prepare the explicit radio TTY before a native +Chromium Web Serial trial. Some Linux Chromium builds inherit `VMIN=0` from +pyserial and mistake an idle read for device loss. Pause other serial clients +first; another opener can change these settings or consume replies. + +```python +from tools.hil.linux_web_serial import prepared_linux_web_serial + +with prepared_linux_web_serial("/dev/serial/by-id/-if00") as host: + # Run the native browser controller here and record host with its receipt. + # Its finally cleanup must close the browser reader, writer and port. + # Controllers must route termination signals into that same cleanup. + ... +``` + +The helper changes/checks only `VMIN=1, VTIME=0`, closes its descriptor before +the trial, and restores all original termios attributes on context exit or +failed preparation. It does not call DTR/RTS, reset USB, send data, change +firmware, extend a deadline, or retry a command. Endpoint replacement and +failed readback are explicit failures. The real Linux pseudoterminal regression +uses no radio and is included in the unit-test workflow: + +```sh +python3 -B test/test_hil_linux_web_serial.py -v +``` + +See [the scoped G2 comparison and Chromium source evidence](../docs/research/linux_chromium_web_serial_tty.md). + The bootloader-version regression compiles the production reader with a C++17 `g++` (or `CXX`) compiler. Its offline cases include the MeshTower V2 SD 2.4.6 missing-UF2-text failure, OTAFIX metadata versus cached base-version precedence, diff --git a/test/test_hil_linux_web_serial.py b/test/test_hil_linux_web_serial.py new file mode 100644 index 00000000..f2f2c2cb --- /dev/null +++ b/test/test_hil_linux_web_serial.py @@ -0,0 +1,180 @@ +#!/usr/bin/env python3 +"""Actual Linux pseudoterminal tests; never opens a physical radio.""" + +import copy +import errno +import importlib.util +import os +from pathlib import Path +import pty +import select +import termios +import unittest +from unittest import mock + + +ROOT = Path(__file__).resolve().parents[1] +SPEC = importlib.util.spec_from_file_location( + "hil_linux_web_serial", ROOT / "tools/hil/linux_web_serial.py") +HIL = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(HIL) + + +class NativeTTYTests(unittest.TestCase): + def setUp(self): + self.master, slave = pty.openpty() + self.path = os.ttyname(slave) + self.assertTrue(self.path.startswith("/dev/pts/")) + attrs = termios.tcgetattr(slave) + attrs[3] &= ~(termios.ICANON | termios.ECHO | termios.ISIG) + attrs[1] &= ~termios.OPOST + attrs[6][termios.VMIN] = 0 + attrs[6][termios.VTIME] = 0 + termios.tcsetattr(slave, termios.TCSANOW, attrs) + self.original = termios.tcgetattr(slave) + os.close(slave) + self.addCleanup(os.close, self.master) + + def attributes(self): + fd = os.open(self.path, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK) + try: + return termios.tcgetattr(fd) + finally: + os.close(fd) + + def test_inherited_zero_read_is_fixed_and_exact_attributes_restored(self): + fd = os.open(self.path, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK) + self.assertEqual(os.read(fd, 16), b"") + os.close(fd) + with HIL.prepared_linux_web_serial(self.path) as evidence: + self.assertEqual(evidence["vmin_before"], 0) + fd = os.open(self.path, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK) + try: + expected = copy.deepcopy(self.original) + expected[6][termios.VMIN] = 1 + self.assertEqual(termios.tcgetattr(fd), expected) + with self.assertRaises(BlockingIOError) as raised: + os.read(fd, 16) + self.assertEqual(raised.exception.errno, errno.EAGAIN) + os.write(self.master, b"x") + readable, _, _ = select.select([fd], [], [], 1.0) + self.assertEqual(readable, [fd]) + self.assertEqual(os.read(fd, 16), b"x") + finally: + os.close(fd) + self.assertEqual(self.attributes(), self.original) + + def test_test_failure_restores_all_attributes(self): + with self.assertRaisesRegex(RuntimeError, "browser failed"): + with HIL.prepared_linux_web_serial(self.path): + fd = os.open(self.path, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK) + try: + attrs = termios.tcgetattr(fd) + attrs[0] ^= termios.IXOFF + attrs[6][termios.VTIME] = 7 + termios.tcsetattr(fd, termios.TCSANOW, attrs) + finally: + os.close(fd) + raise RuntimeError("browser failed") + self.assertEqual(self.attributes(), self.original) + + def test_interrupted_test_restores_attributes(self): + with self.assertRaises(KeyboardInterrupt): + with HIL.prepared_linux_web_serial(self.path): + raise KeyboardInterrupt() + self.assertEqual(self.attributes(), self.original) + + def test_failed_preparation_readback_restores_before_yield(self): + original_apply = HIL._apply_and_check + + def fail_after_normalization(fd, attrs): + original_apply(fd, attrs) + if attrs[6][termios.VMIN] == 1: + raise RuntimeError("simulated preparation readback failure") + + with mock.patch.object(HIL, "_apply_and_check", side_effect=fail_after_normalization): + with self.assertRaisesRegex(RuntimeError, "preparation readback failure"): + with HIL.prepared_linux_web_serial(self.path): + self.fail("failed preflight must not yield") + self.assertEqual(self.attributes(), self.original) + + def test_ignored_write_is_rejected_by_actual_readback(self): + with mock.patch.object(termios, "tcsetattr") as setter: + with self.assertRaisesRegex(RuntimeError, "readback differs"): + with HIL.prepared_linux_web_serial(self.path): + self.fail("ignored normalization must not yield") + self.assertEqual(setter.call_count, 1) + self.assertEqual(self.attributes(), self.original) + + def test_already_normalized_tty_needs_no_attribute_write(self): + fd = os.open(self.path, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK) + attrs = termios.tcgetattr(fd) + attrs[6][termios.VMIN] = 1 + termios.tcsetattr(fd, termios.TCSANOW, attrs) + os.close(fd) + with mock.patch.object(termios, "tcsetattr", wraps=termios.tcsetattr) as setter: + with HIL.prepared_linux_web_serial(self.path): + pass + setter.assert_not_called() + self.assertEqual(self.attributes(), attrs) + + def test_canonical_attribute_representation_is_preserved(self): + fd = os.open(self.path, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK) + attrs = termios.tcgetattr(fd) + attrs[3] |= termios.ICANON + termios.tcsetattr(fd, termios.TCSANOW, attrs) + original = termios.tcgetattr(fd) + os.close(fd) + with HIL.prepared_linux_web_serial(self.path) as evidence: + self.assertEqual(evidence["vmin_before"], 0) + expected = copy.deepcopy(original) + expected[6][termios.VMIN] = b"\x01" + self.assertEqual(self.attributes(), expected) + self.assertEqual(self.attributes(), original) + + def test_no_second_descriptor_is_held_during_the_trial(self): + opened = [] + original_open = HIL._open_tty + + def capture(path): + fd = original_open(path) + opened.append(fd) + return fd + + with mock.patch.object(HIL, "_open_tty", side_effect=capture): + with HIL.prepared_linux_web_serial(self.path): + with self.assertRaises(OSError): + os.fstat(opened[0]) + self.assertEqual(self.attributes(), self.original) + + def test_replaced_endpoint_is_not_modified(self): + other_master, other_slave = pty.openpty() + other_path = os.ttyname(other_slave) + self.addCleanup(os.close, other_master) + self.addCleanup(os.close, other_slave) + attrs = termios.tcgetattr(other_slave) + original_open = HIL._open_tty + calls = 0 + + def substitute_restore(path): + nonlocal calls + calls += 1 + return original_open(path if calls == 1 else other_path) + + with mock.patch.object(HIL, "_open_tty", side_effect=substitute_restore): + with self.assertRaisesRegex(RuntimeError, "endpoint changed"): + with HIL.prepared_linux_web_serial(self.path): + pass + self.assertEqual(termios.tcgetattr(other_slave), attrs) + + def test_explicit_absolute_tty_required(self): + with self.assertRaisesRegex(ValueError, "absolute TTY"): + with HIL.prepared_linux_web_serial("ttyACM0"): + self.fail("relative device must not be opened") + with self.assertRaisesRegex(ValueError, "must identify a TTY"): + with HIL.prepared_linux_web_serial("/dev/null"): + self.fail("non-TTY must not be yielded") + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/hil/linux_web_serial.py b/tools/hil/linux_web_serial.py new file mode 100644 index 00000000..ec4a66c1 --- /dev/null +++ b/tools/hil/linux_web_serial.py @@ -0,0 +1,90 @@ +"""Scope Linux TTY read settings to a native Web Serial hardware test. + +Call after the last pyserial session, while other serial clients are paused. +The caller must close its browser streams before leaving the context, including +on failure. No descriptor is held across the trial, so a stock browser close +retains its normal last-close behavior. This helper sends no serial data and +does not set DTR/RTS, reset USB, or change firmware. +""" + +from contextlib import contextmanager +import copy +import os +from pathlib import Path +import sys +import termios + + +def _identity(fd): + stat = os.fstat(fd) + return stat.st_dev, stat.st_ino, stat.st_rdev + + +def _open_tty(path): + fd = os.open(path, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK) + if not os.isatty(fd): + os.close(fd) + raise ValueError("The explicit path must identify a TTY") + return fd + + +def _apply_and_check(fd, attributes): + if termios.tcgetattr(fd) != attributes: + termios.tcsetattr(fd, termios.TCSANOW, attributes) + if termios.tcgetattr(fd) != attributes: + raise RuntimeError("TTY attribute readback differs from the requested settings") + + +@contextmanager +def prepared_linux_web_serial(port): + """Set/check VMIN=1, VTIME=0 for one test; restore exact prior termios. + + Requires an explicit absolute path. A replaced endpoint is rejected before + restoration rather than changing a different device. Preparation failures + also restore the original attributes. Browser/serial ownership and signal + cleanup remain the caller's responsibility; close native streams first. + """ + if sys.platform != "linux": + raise RuntimeError("This preflight is for Linux native Web Serial only") + path = os.fspath(port) + if not Path(path).is_absolute(): + raise ValueError("An explicit absolute TTY path is required") + + fd = _open_tty(path) + original = None + try: + identity = _identity(fd) + original = termios.tcgetattr(fd) + desired = copy.deepcopy(original) + # Python represents these slots as bytes in canonical mode and ints + # in noncanonical mode. Preserve that representation for exact checks. + desired[6][termios.VMIN] = ( + b"\x01" if isinstance(original[6][termios.VMIN], bytes) else 1) + desired[6][termios.VTIME] = ( + b"\x00" if isinstance(original[6][termios.VTIME], bytes) else 0) + try: + _apply_and_check(fd, desired) + except BaseException: + _apply_and_check(fd, original) + raise + finally: + os.close(fd) + + def cc_number(value): + return value[0] if isinstance(value, bytes) else value + + try: + yield { + "vmin_before": cc_number(original[6][termios.VMIN]), + "vtime_before": cc_number(original[6][termios.VTIME]), + "vmin_during_test": 1, + "vtime_during_test": 0, + } + finally: + fd = _open_tty(path) + try: + if _identity(fd) != identity: + raise RuntimeError("TTY endpoint changed; original attributes were not applied") + _apply_and_check(fd, original) + finally: + os.close(fd)