Organize test results and historical release docs
This commit is contained in:
mikecarper committed
1 parent
d44910865c
commit
e96918cd91
130 files changed
+8975
-7736
No files matched your search
@@ -1,233 +1,5 @@
|
||||
# ESP32-S3 OTA memory experiment
|
||||
# Document moved
|
||||
|
||||
This experiment compares the current static S3 OTA workspace (A) with
|
||||
`OTA_HEAP_CONTEXT=1` (B), with WiFi and a real TLS MQTT connection enabled.
|
||||
It is isolated on `experiment/s3-ota-memory-soak`; published recipes retain
|
||||
their existing defaults. A short stress test cannot establish months of uptime.
|
||||
<script>window.location.replace(new URL("../test-results/s3_memory_soak_validation/" + window.location.hash, window.location.href));</script>
|
||||
|
||||
## Hardware and scope
|
||||
|
||||
| Board | Role | Flash | PSRAM | Lab identity |
|
||||
|---|---|---:|---:|---|
|
||||
| Heltec V4.3 OLED, standard V4 | Observer/MQTT repeater | 16 MiB | 2 MiB QSPI | USB MAC `44:1B:F6:69:CF:98` |
|
||||
| Original SenseCAP Indicator LoRa | Full companion plus experimental MQTT | 8 MiB | 8 MiB octal | WiFi MAC `D8:3B:DA:75:23:AC` |
|
||||
| Seeed XIAO ESP32-S3 with WIO LoRa | Observer/MQTT repeater | 8 MiB | 8 MiB octal | MAC `28:84:85:B4:09:80`, Windows COM31 |
|
||||
|
||||
This is not an R8 test. Both Indicator variants use the same 320x320 canvas;
|
||||
the native 480x480 Full-plus-MQTT configuration is not qualified by this run.
|
||||
One custom TLS MQTT slot per board publishes only to the isolated Pi broker.
|
||||
These results do not qualify five concurrent slots, JWT renewal, heavy RF
|
||||
traffic, every OTA receive/apply path, or months of operation.
|
||||
|
||||
## Reproducible builds
|
||||
|
||||
Base commit: `bc0fb2c335938c97220de51098cea6162f2f0fe0`, following PR #7's merge.
|
||||
Copy `tools/hil/s3_memory_soak.ini` to ignored `platformio.local.ini` and build
|
||||
the six named environments sequentially, with `MESHCORE_ESP32_FULL_BUILD=1`.
|
||||
Never run multiple PlatformIO processes in this checkout. Archive each image,
|
||||
ELF, map, partitions, bootloader, and memory report before the next build.
|
||||
|
||||
The private broker's public CA must be in `ssl_certs` while generating these
|
||||
experimental images. Remove it and `platformio.local.ini` afterward. Private
|
||||
keys stay on the Pi. WiFi credentials and the Windows logger token are in
|
||||
private local files; do not commit them or the generated certificate bundle.
|
||||
Broker certificate verification remains enabled.
|
||||
|
||||
The V4 and XIAO lab recipes also enable `MESH_SOAK_WIFI_KEY_OVERRIDE` and
|
||||
require the ignored local header `tools/hil/S3SoakWiFiKey.h`. To reproduce
|
||||
diagnostics using ordinary WiFi credentials, remove that define from the lab
|
||||
recipe; the private header is then unnecessary. Published recipes do not
|
||||
enable either lab flag.
|
||||
|
||||
| Image | Application bytes | Link-time available internal bytes | SHA-256 |
|
||||
|---|---:|---:|---|
|
||||
| V4 A | 1,910,392 | 237,360 | `68c2749e0ea40db91d953f6355010bc602f85424526ca6acbe1cad7fd11c8630` |
|
||||
| V4 B | 1,911,256 | 252,760 | `6847ad4ad6bac2e558b8aed67de37417a497972391f13a71e0a42b5c9704dcff` |
|
||||
| Indicator A | 2,313,256 | 244,248 | `3d0ac27f95ddc54a32b59a3892d157713dcded285dbeb98d1bdfdb1ce0b951ab` |
|
||||
| Indicator B | 2,314,136 | 255,496 | `83045d7ec48befc43f3460c08c68b71da037a5476e14dd59def5706fc5a1b07e` |
|
||||
| XIAO A | 1,845,000 | 238,448 | `6fcc657c5ccbc5dd3d53a0cf1ceebde54479d59d329f8625e7cb7141a27d9ed8` |
|
||||
| XIAO B | 1,845,912 | 253,864 | `2b233e42e0c515b152ce83ca8fe38b1f35780970bb6350dbadf2591b6a2699a1` |
|
||||
|
||||
All six firmware builds passed their RAM/flash gates. The linked internal
|
||||
capacity gain is 15,400 bytes on V4, 11,248 bytes on Indicator, and 15,416 bytes
|
||||
on XIAO. This is
|
||||
capacity returned while idle, not a reduction in peak OTA allocation needs.
|
||||
Both A/B variants keep the existing MQTT client reuse policy and internal
|
||||
MQTT task stack; this experiment does not move that stack into PSRAM.
|
||||
|
||||
## Instrumentation
|
||||
|
||||
`MESH_SOAK_DIAGNOSTICS` exposes local-only `get soak`, `soak ota-cycle`, and
|
||||
`soak wifi-drop` commands. They report free/largest/minimum internal heap,
|
||||
free PSRAM, OTA context size/presence, WiFi state, and uptime. No published
|
||||
recipe enables these controls.
|
||||
|
||||
The lab network has a valid 64-digit raw WiFi key. Production MQTT preferences
|
||||
currently accept only 63 characters. The V4 experiment uses a separate,
|
||||
test-only NVS key through `MESH_SOAK_WIFI_KEY_OVERRIDE`, without changing the
|
||||
production binary preference layout. This does not fix released raw-key input.
|
||||
The network's original nine-character passphrase was subsequently recovered
|
||||
from the connected Windows profile; XIAO uses it through ordinary settings.
|
||||
The override is compiled into both XIAO images but no override key is stored.
|
||||
|
||||
The Pi logger owns one persistent V4 USB handle: opening this particular USB
|
||||
device resets it, even with DTR/RTS disabled. Firmware installations and logger
|
||||
restarts therefore begin new uptime runs. Indicator telemetry uses TCP 5002.
|
||||
The logger distinguishes the 49.7-day `millis()` rollover from a reboot.
|
||||
|
||||
Passive samples omit `get mqtt.stats`: its outbox query can block behind the
|
||||
MQTT client lock during a TLS handshake and stall the CLI. The logger reads
|
||||
`get mqtt.status` and separately counts messages received through a verified
|
||||
TLS subscription. It stores topic counts, not MQTT payloads or credentials.
|
||||
|
||||
Two exploratory rapid-flap runs used an overly short 90-second recovery cutoff.
|
||||
The first also used the blocking outbox probe. Preserve those pilot results;
|
||||
do not count them as successful tests. Production MQTT backoff deliberately
|
||||
grows to five minutes and clears after two stable connected minutes, so rapid
|
||||
flapping is not equivalent to repeated independent connection tests.
|
||||
|
||||
## Results and continuing soak
|
||||
|
||||
V4 and Indicator both passed the matched A/B workload: 200 explicit OTA context
|
||||
acquire/release attempts, four MQTT bridge restarts, one WiFi reconnection and
|
||||
one five-second private-broker outage per image. An initial bridge restart
|
||||
preconditions each run. No unexpected uptime resets occurred in these runs.
|
||||
A was paused after its four bridge restarts to investigate the display report,
|
||||
then resumed for the network faults; its elapsed duration includes that pause.
|
||||
|
||||
After the workload, with WiFi and one TLS MQTT slot connected:
|
||||
|
||||
| Board | A free internal bytes | B free internal bytes | Gain | A largest block | B largest block |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| Standard V4 | 95,396 | 111,040 | 15,644 (15.28 KiB) | 81,908 | 98,292 |
|
||||
| Indicator | 107,168 | 118,520 | 11,352 (11.09 KiB) | 94,196 | 106,484 |
|
||||
| XIAO (same saved settings, cold boot) | 151,028 | 166,444 | 15,416 (15.05 KiB) | 139,252 | 147,444 |
|
||||
|
||||
Free heap varies with network activity; link-time capacity is the cleaner
|
||||
measurement of storage moved out of static RAM. PSRAM remained about 2.01 MB
|
||||
on V4 and 6.33 MB on Indicator. In each B run the OTA context returned to absent
|
||||
after the allocation workload. Neither board's largest block shrank through
|
||||
those 200 allocation cycles or the reconnect workload. WiFi/MQTT recovery took
|
||||
about 4.3 seconds on A and 3.8 seconds on B; broker recovery took about 19.5 and
|
||||
21.8 seconds respectively. These are single observed recovery times.
|
||||
|
||||
Workspace recreation has an observed latency cost. The 200 command cycles
|
||||
took approximately 189 seconds on V4 B and 379 seconds on Indicator B, versus
|
||||
32 and 15 seconds on A. This includes command transport, OTA initialization,
|
||||
and ordinary firmware service, so it is not an allocator microbenchmark.
|
||||
The static comparator already owns an initialized workspace; the dynamic
|
||||
variant recreates it. This experiment does not claim faster OTA startup, and
|
||||
this cost should be checked against real session use before a default change.
|
||||
Code inspection shows that `OtaContext::begin()` calls `ota_self_firmware()`,
|
||||
which scans the running ESP32 application for its EndF identity on each call.
|
||||
All six experimental images contain valid EndF trailers. Caching immutable
|
||||
boot identity is a candidate for a separate change, not part of these results.
|
||||
|
||||
The real WiFi mOTA folder endpoint was tested separately on Indicator: 20
|
||||
empty-folder sessions on A and 40 on B. All completed and B released its context.
|
||||
During B's first batch, free internal RAM went from 118,552 to 116,316 bytes,
|
||||
and its largest block from 106,484 to 102,388. A second batch ended at 116,308
|
||||
bytes with the same 102,388-byte largest block. This retained allocation/layout
|
||||
change is not explained by the now-absent OTA context; record it separately and
|
||||
watch it during the soak. It did not keep growing in the second batch.
|
||||
|
||||
The original A folder script also tried V4, whose observer recipe has no TCP
|
||||
5001 folder source. Its overall result therefore says `passed: false`, despite
|
||||
Indicator completing all 20 sessions. That endpoint mismatch is preserved in
|
||||
the evidence; it is not an OTA failure on V4. The corrected script targets only
|
||||
the Indicator. These empty-folder checks do not qualify a full firmware transfer.
|
||||
|
||||
XIAO passed 200 context cycles, four bridge restarts and one WiFi reconnect
|
||||
per image, with no unexpected reset. Its largest block stayed unchanged within
|
||||
each workload. WiFi/TLS recovered in about six seconds for both. This local
|
||||
test did not inject a separate broker outage or exercise a TCP folder endpoint.
|
||||
Verified subscriptions on the Pi confirmed actual TLS publications for both.
|
||||
|
||||
The first XIAO A run followed initial device configuration and finished with
|
||||
126,140 internal bytes free, which would have overstated the OTA saving. It is
|
||||
preserved as `xiao-live/A-configuration-stress.json`. Reflashing A without an
|
||||
erase, retaining exactly the saved settings used by B, and rerunning the whole
|
||||
workload produced the 151,028-byte baseline above. Its resulting gain exactly
|
||||
matches the 15,416-byte linked-capacity change. The earlier larger gap is not
|
||||
credited to OTA storage; the precise retained allocation was not isolated.
|
||||
|
||||
The Pi stores evidence under `/home/mikec/hwtest/runs/s3-memory-soak`:
|
||||
|
||||
- `telemetry.jsonl`: append-only memory, WiFi/MQTT state, uptime, transport errors.
|
||||
- `latest.json`: latest samples plus broker-side received-message counts.
|
||||
- `A-stress.json` / `B-stress.json`: matched stress results and phase boundaries.
|
||||
- `A-folder.json` / `B-folder.json`: real WiFi mOTA empty-folder session checks.
|
||||
- `B-repeat-folder.json`: the extra batch checking the initial allocation change.
|
||||
|
||||
`meshcore-memory-soak.service` holds the USB connection and samples each minute.
|
||||
`meshcore-soak-broker.service` supplies the private TLS broker. Both are enabled
|
||||
on the Pi. The logger never reboots a board to hide a firmware failure, but
|
||||
reopening the V4 USB transport can itself reset it; inspect transport counts,
|
||||
service restarts, and uptime together when diagnosing any interruption.
|
||||
The final passive phase started at `2026-09-11T07:18:07Z`. Installing the final
|
||||
logger revision (which serializes concurrent samples) deliberately restarted
|
||||
the service and therefore reset V4 once. That begins its long-soak uptime run;
|
||||
Indicator retained its uptime. Both then reported WiFi/TLS connected, absent
|
||||
OTA contexts, and actual broker-side publications.
|
||||
|
||||
The local XIAO logger owns COM31 and writes under
|
||||
`C:\git\MeshCore\out\s3-memory-soak\xiao-live`. Its control endpoint is
|
||||
loopback-only with a private token. It runs while this Windows session remains
|
||||
alive; it is not a Pi service and will not survive a computer shutdown. A
|
||||
separate verified TLS subscription on the Pi confirmed XIAO publications for
|
||||
both images. The Pi logger's per-device counts cover V4 and Indicator only.
|
||||
The final XIAO B passive phase was verified at `2026-09-11T07:21:21Z`, with
|
||||
166,508 free internal bytes, a 155,636-byte largest block, WiFi/TLS connected,
|
||||
and no active OTA context. The largest block differs from the post-stress
|
||||
sample because this is a fresh boot, which illustrates why comparisons must
|
||||
use the same workload history. A Pi subscription received its publication two
|
||||
seconds later.
|
||||
|
||||
Read-only checks on the Pi:
|
||||
|
||||
```sh
|
||||
systemctl is-active meshcore-memory-soak meshcore-soak-broker
|
||||
cat /home/mikec/hwtest/runs/s3-memory-soak/latest.json
|
||||
```
|
||||
|
||||
On Windows, read `out/s3-memory-soak/xiao-live/latest.json`. Keep the computer
|
||||
awake for continuous local samples. Neither logger automatically reboots a
|
||||
radio in response to a failed sample.
|
||||
|
||||
The public lab CA was removed from `ssl_certs`, and the temporary local PIO
|
||||
configuration and generated test trust bundle were removed after building.
|
||||
Archived experimental firmware and public evidence remain in ignored
|
||||
`out/s3-memory-soak`; no test CA enters the published build configuration.
|
||||
|
||||
No published default has changed. Leave the experimental B images running to
|
||||
collect longer evidence before deciding whether to expand S3 heap allocation.
|
||||
More idle internal RAM gives WiFi/TLS useful headroom; it does not itself prove
|
||||
stability or eliminate the need to allocate the OTA workspace during use.
|
||||
|
||||
## Physical display checks
|
||||
|
||||
The user reported unresponsive screens during MQTT restart testing. The V4
|
||||
lit after temporarily choosing `on`; after restoring battery and USB profiles
|
||||
to `button`, 15 seconds, the user confirmed that pressing the button wakes it.
|
||||
The synchronous reconnect/diagnostic work may have delayed UI service, but the
|
||||
exact cause of the original visual report was not established.
|
||||
|
||||
The T096 is a separate nRF52840 board, USB identity `651F8E496197F882`. It was
|
||||
reflashed with current Full firmware and briefly instrumented. The trace showed
|
||||
the radio, sensor and UI loop advancing; the display had initialized but was
|
||||
marked off. Its functional CDC0 was in Binary mode after a host-session close.
|
||||
Sending a newline-delimited `+++MESHCORE-TERM-START` restored ASCII control.
|
||||
With display mode set to `on`, the user confirmed the normal screen appeared.
|
||||
Temporary tracing was removed by reinstalling the non-instrumented image, and
|
||||
battery/USB profiles were restored to `button-pairing`, 15 seconds. The user
|
||||
then confirmed that its button wakes the normal screen after timeout.
|
||||
|
||||
T096 recovery firmware: `v1.17.1-t096-check`, Full/shared OTA queue, `-Os` size
|
||||
profile, available linked internal RAM 99,892 bytes (73,728 required). ZIP SHA-256
|
||||
`f5cfcb001efda559ae9b4e7b105ac9766a604808f5fd708b7c533f8a11c64578`.
|
||||
No device-wide filesystem erase was needed for this recovery. T096 does not
|
||||
count as an S3 A/B test.
|
||||
|
||||
Host validation: `python -m unittest discover -s tools/hil -p test_s3_memory_soak.py`.
|
||||
The five checks cover diagnostic filtering, missing data, resets, and counter
|
||||
rollover. They are not substitutes for the hardware evidence.
|
||||
This document is now in [Test Results](test-results/s3_memory_soak_validation.md).
|
||||
Reference in new issue
Block a user