build(mqtt): make the reduced-TLS mbedTLS archives shippable, opt-in and verified

The reduced-TLS work was validated on hardware but only reachable through
PLATFORMIO_BUILD_FLAGS pointing at an absolute path in a developer's home
directory, so nothing outside that machine could reproduce it.

Distribute the archives as a release asset instead of committing them: ~6 MB
per architecture, and they must be rebuilt for every espressif32 bump, so
committing would grow history permanently and go stale without any signal.

  scripts/mbedtls_4k_manifest.txt  per-arch sha256 of each archive
  scripts/fetch_mbedtls_4k.sh      fetch into .mbedtls-4k/<arch>/, verify
  scripts/mbedtls_4k.py            pre-build wiring and post-link proof

Off by default. The script is attached to esp32_base but returns immediately
unless MESHCORE_REDUCED_TLS=1, so ordinary builds need no artifact and are
byte-for-byte unaffected — confirmed by building with it absent.

Both ways this can fail silently produce a firmware that looks fine and lacks
the change, so the opt-in path refuses to guess:

  - a -L at a missing or partial directory: the linker ignores an unusable
    search path and resolves mbedTLS from the framework. Now a hard error.
  - archives left over from an earlier platform version: now a sha256
    mismatch against the manifest, naming both hashes.
  - a -L that is present but outranked, leaving the flag inert: after the
    link, firmware.map must resolve every libmbed*.a into .mbedtls-4k/, or
    the build fails and prints the offending paths.

That last check earned its place immediately — it caught its own first
implementation comparing a relative map path against an absolute one, and an
earlier build flag in this investigation was accepted by the compiler while
no source read it. A flag reaching the compiler proves nothing about the link.

Verified all four paths on Heltec_v3_repeater_observer_mqtt: default build
unaffected; opted in with archives present links all four from .mbedtls-4k/
and says so; archives absent fails with a fetch hint; a single appended byte
fails on sha256.

Also removes platformio.local.ini.hold, which held the superseded approach of
pointing platform_packages at a whole custom framework. That installs over the
shared framework package and changes mbedTLS for every other ESP32 project on
the machine; the -L path keeps the change scoped to one env.

Note the inbound record buffer stays at 16 KiB, so this lowers per-connection
footprint by ~12 KiB but does not move the contiguous allocation a handshake
needs. It buys headroom, not a lower floor.

(cherry picked from commit a87faff6ff170c328fdd0550f4b4dd9089aa2ea0)

.gitignore

@@ -32,3 +32,7 @@ platformio.local.ini
 .build-wt-*/
 .wt-*/
 scripts/__pycache__/*
+
+# Reduced-TLS mbedTLS archives, ~6 MB per arch, fetched by scripts/fetch_mbedtls_4k.sh.
+# Not committed because they must be rebuilt for every espressif32 platform bump.
+.mbedtls-4k/

docs/mbedtls-tls-footprint.md

@@ -136,6 +136,36 @@ fix was `rm -rf` the package and `pio pkg install` to re-download stock. A prepe
 search path keeps the change scoped to one env, because the linker takes each archive member
 from the first archive that satisfies an undefined symbol.
 
+### How the archives are distributed
+
+They are not committed: ~6 MB per architecture, and they have to be rebuilt for every
+`platformio/espressif32` bump, so committing them would grow history permanently and go
+stale silently. Instead they are published as a release asset and fetched on demand:
+
+```
+scripts/fetch_mbedtls_4k.sh esp32s3            # download + verify against the manifest
+MESHCORE_REDUCED_TLS=1 pio run -e Heltec_v3_repeater_observer_mqtt
+```
+
+- `scripts/mbedtls_4k_manifest.txt` — per-arch sha256 of each archive. **Update it on every
+  platform bump**, together with the published asset.
+- `scripts/fetch_mbedtls_4k.sh` — downloads into `.mbedtls-4k/<arch>/` (gitignored) and
+  verifies. `MBEDTLS_4K_LOCAL=<dir>` copies from a local build tree instead of downloading.
+- `scripts/mbedtls_4k.py` — wired into `esp32_base.extra_scripts`, but a **no-op unless
+  `MESHCORE_REDUCED_TLS=1`**, so ordinary builds need no artifact and behave as before.
+
+The opt-in path is deliberately loud, because both ways this can go wrong produce a
+firmware that looks correct and silently lacks the change:
+
+| failure | what happens without a guard | guard |
+|---|---|---|
+| directory missing or partial | linker ignores an unusable `-L` and resolves mbedTLS from the framework | pre-build: hard error |
+| archives stale after a platform bump | links the wrong build | pre-build: sha256 vs manifest |
+| `-L` present but outranked | framework archives win, flag is inert | post-link: `firmware.map` must resolve every `libmbed*.a` to `.mbedtls-4k/` |
+
+That last one is the reason the post-link check exists rather than trusting the flag: a
+build flag reaching the compiler proves nothing about what got linked.
+
 ## How to verify it worked
 
 1. `strings`/`grep` the new `sdkconfig.h` for the four settings.

platformio.ini

@@ -64,6 +64,8 @@ platform = platformio/espressif32@6.11.0
 monitor_filters = esp32_exception_decoder
 extra_scripts =
   pre:scripts/generate_webconfig_html.py
+; No-op unless MESHCORE_REDUCED_TLS=1; see docs/mbedtls-tls-footprint.md.
+  pre:scripts/mbedtls_4k.py
   merge-bin.py
 build_flags = ${arduino_base.build_flags}
   -D ESP32_PLATFORM

platformio.local.ini.hold

@@ -1,15 +0,0 @@
-; Local-only override (gitignored) pointing the Heltec V3 observer env at a custom
-; framework whose mbedTLS archives were rebuilt with an asymmetric TLS record buffer:
-;   CONFIG_MBEDTLS_ASYMMETRIC_CONTENT_LEN=y
-;   CONFIG_MBEDTLS_SSL_IN_CONTENT_LEN=16384   (unchanged — a peer may send a 16 KiB record)
-;   CONFIG_MBEDTLS_SSL_OUT_CONTENT_LEN=4096   (was 16384)
-;
-; Expected: ~12 KiB less internal DRAM per TLS connection, ~24 KiB across two broker slots.
-; Built from the shipped sdkconfig verbatim plus those three lines, so the archives differ
-; only by this change. See docs/mbedtls-tls-footprint.md.
-;
-; Absolute path, hence local-only: not committable.
-
-[env:Heltec_v3_repeater_observer_mqtt]
-platform_packages =
-  framework-arduinoespressif32 @ file:///Users/adam/framework-arduinoespressif32-tlsfix

scripts/fetch_mbedtls_4k.sh

@@ -0,0 +1,72 @@
+#!/usr/bin/env bash
+# Fetch the reduced-TLS mbedTLS archives into .mbedtls-4k/<arch>/.
+#
+# These archives are built from the shipped sdkconfig plus three lines
+# (CONFIG_MBEDTLS_ASYMMETRIC_CONTENT_LEN=y, IN_CONTENT_LEN 16384,
+# OUT_CONTENT_LEN 4096) and save ~12 KiB of internal DRAM per TLS connection.
+# See docs/mbedtls-tls-footprint.md for the rationale and the build recipe.
+#
+# They are not committed: ~6 MB per architecture, and they must be rebuilt for
+# every platform bump, so they are published as a release asset keyed on the
+# espressif32 platform version instead.
+#
+#   scripts/fetch_mbedtls_4k.sh [arch]        # default: esp32s3
+#
+# Set MBEDTLS_4K_LOCAL to skip the download and copy from a local build tree:
+#   MBEDTLS_4K_LOCAL=~/mbedtls-4k-esp32s3/staged scripts/fetch_mbedtls_4k.sh
+set -euo pipefail
+
+ARCH="${1:-esp32s3}"
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+DEST="$REPO_ROOT/.mbedtls-4k/$ARCH"
+MANIFEST="$REPO_ROOT/scripts/mbedtls_4k_manifest.txt"
+BASE_URL="${MBEDTLS_4K_BASE_URL:-https://github.com/agessaman/MeshCore/releases/download/mbedtls-4k}"
+
+if [ ! -f "$MANIFEST" ]; then
+  echo "error: missing $MANIFEST" >&2
+  exit 1
+fi
+
+# Manifest lines: <arch> <sha256> <filename>. Blank lines and # comments ignored.
+expected="$(awk -v a="$ARCH" '$1 == a && $0 !~ /^#/ {print $2"  "$3}' "$MANIFEST")"
+if [ -z "$expected" ]; then
+  echo "error: no manifest entries for arch '$ARCH'" >&2
+  echo "known arches: $(awk '$0 !~ /^#/ && NF {print $1}' "$MANIFEST" | sort -u | tr '\n' ' ')" >&2
+  exit 1
+fi
+
+mkdir -p "$DEST"
+
+if [ -n "${MBEDTLS_4K_LOCAL:-}" ]; then
+  echo "copying from $MBEDTLS_4K_LOCAL"
+  while read -r _sha name; do
+    cp "$MBEDTLS_4K_LOCAL/$name" "$DEST/$name"
+  done <<< "$expected"
+else
+  TARBALL="mbedtls-4k-$ARCH.tar.gz"
+  echo "downloading $BASE_URL/$TARBALL"
+  tmp="$(mktemp -d)"
+  trap 'rm -rf "$tmp"' EXIT
+  curl -fsSL "$BASE_URL/$TARBALL" -o "$tmp/$TARBALL"
+  tar -xzf "$tmp/$TARBALL" -C "$tmp"
+  while read -r _sha name; do
+    # Accept the archive whether or not the tarball has a leading directory.
+    found="$(find "$tmp" -name "$name" -type f | head -1)"
+    if [ -z "$found" ]; then
+      echo "error: $name missing from $TARBALL" >&2
+      exit 1
+    fi
+    cp "$found" "$DEST/$name"
+  done <<< "$expected"
+fi
+
+# Verify every archive against the manifest. A wrong or truncated archive would
+# otherwise link silently and produce a firmware without the reduced buffers.
+cd "$DEST"
+if command -v shasum >/dev/null 2>&1; then
+  echo "$expected" | shasum -a 256 -c -
+else
+  echo "$expected" | sha256sum -c -
+fi
+
+echo "ok: $ARCH archives verified in $DEST"

scripts/mbedtls_4k.py

@@ -0,0 +1,144 @@
+"""Link the reduced-TLS mbedTLS archives, and prove they were actually linked.
+
+Opt in per build with MESHCORE_REDUCED_TLS=1. Off by default, so an ordinary build
+needs no 6 MB artifact and behaves exactly as before.
+
+    MESHCORE_REDUCED_TLS=1 pio run -e Heltec_v3_repeater_observer_mqtt
+
+The archives lower the mbedTLS outbound record buffer from 16 KiB to 4 KiB, saving
+~12 KiB of internal DRAM per TLS connection on non-PSRAM observers. The inbound
+buffer stays at 16 KiB, so the contiguous allocation a handshake needs is unchanged
+— this buys headroom, it does not move that floor. See docs/mbedtls-tls-footprint.md.
+
+Two failure modes this guards against, both of which produce a firmware that looks
+fine and silently lacks the change:
+
+  - a -L pointing at a missing or partial directory. The linker ignores an
+    unusable search path and quietly resolves mbedTLS from the framework instead.
+  - archives that do not match the manifest, e.g. left over from an earlier
+    platform version.
+
+So the opt-in path verifies every archive by sha256 before the build, and after the
+link re-reads firmware.map to confirm every libmbed*.a came from our directory.
+"""
+Import("env")
+
+import hashlib
+import os
+import sys
+
+REQUIRED = ("libmbedcrypto.a", "libmbedtls_2.a", "libmbedtls.a", "libmbedx509.a")
+
+
+def _fail(msg):
+    print("\n*** reduced-TLS build failed ***", file=sys.stderr)
+    print(msg, file=sys.stderr)
+    print(
+        "\nFetch the archives with:  scripts/fetch_mbedtls_4k.sh <arch>"
+        "\nOr build without them by unsetting MESHCORE_REDUCED_TLS.",
+        file=sys.stderr,
+    )
+    env.Exit(1)
+
+
+def _sha256(path):
+    h = hashlib.sha256()
+    with open(path, "rb") as fh:
+        for chunk in iter(lambda: fh.read(1 << 20), b""):
+            h.update(chunk)
+    return h.hexdigest()
+
+
+def _manifest(project_dir, arch):
+    path = os.path.join(project_dir, "scripts", "mbedtls_4k_manifest.txt")
+    if not os.path.isfile(path):
+        _fail("missing scripts/mbedtls_4k_manifest.txt")
+    wanted = {}
+    with open(path) as fh:
+        for line in fh:
+            line = line.strip()
+            if not line or line.startswith("#"):
+                continue
+            parts = line.split()
+            if len(parts) == 3 and parts[0] == arch:
+                wanted[parts[2]] = parts[1]
+    return wanted
+
+
+if os.environ.get("MESHCORE_REDUCED_TLS", "") not in ("1", "true", "yes"):
+    Return()
+
+project_dir = env.subst("$PROJECT_DIR")
+arch = env.BoardConfig().get("build.mcu", "")
+if not arch:
+    _fail("could not determine board MCU, so cannot pick an archive set")
+
+staged = os.path.join(project_dir, ".mbedtls-4k", arch)
+if not os.path.isdir(staged):
+    _fail("no archives for %s at %s" % (arch, staged))
+
+wanted = _manifest(project_dir, arch)
+if not wanted:
+    _fail("manifest has no entries for arch '%s'" % arch)
+
+for name in REQUIRED:
+    archive = os.path.join(staged, name)
+    if not os.path.isfile(archive):
+        _fail("missing %s" % archive)
+    if name not in wanted:
+        _fail("%s is not in the manifest for %s" % (name, arch))
+    actual = _sha256(archive)
+    if actual != wanted[name]:
+        _fail(
+            "%s does not match the manifest\n  expected %s\n  actual   %s\n"
+            "Rebuild it for this platform version, or re-run the fetch script."
+            % (archive, wanted[name], actual)
+        )
+
+# Prepend so these satisfy mbedTLS symbols ahead of the framework's own copies:
+# the linker takes each archive member from the first archive that resolves it.
+env.Prepend(LIBPATH=[staged])
+print("reduced-TLS: linking mbedTLS from %s (verified)" % staged)
+
+
+def _verify_map(source, target, env):
+    """Confirm every mbedTLS archive in the link came from our directory."""
+    map_path = os.path.join(env.subst("$BUILD_DIR"), "firmware.map")
+    if not os.path.isfile(map_path):
+        print("reduced-TLS: WARNING no firmware.map, cannot confirm the link",
+              file=sys.stderr)
+        return
+    # The map records whatever the linker was given, which for a -L hit is a path
+    # relative to the linker's cwd (the project dir). Resolve before comparing, or
+    # every one of our own archives reads as stray.
+    staged_real = os.path.realpath(staged)
+    stray = set()
+    seen = set()
+    with open(map_path, errors="replace") as fh:
+        for line in fh:
+            for token in line.split():
+                if "libmbed" not in token or ".a" not in token:
+                    continue
+                path = token.split("(")[0]
+                base = os.path.basename(path)
+                if not base.startswith("libmbed") or not base.endswith(".a"):
+                    continue
+                seen.add(base)
+                resolved = os.path.realpath(os.path.join(project_dir, path))
+                if os.path.dirname(resolved) != staged_real:
+                    stray.add(path)
+    if stray:
+        print("\n*** reduced-TLS: archives linked from the WRONG place ***",
+              file=sys.stderr)
+        for path in sorted(stray):
+            print("  " + path, file=sys.stderr)
+        env.Exit(1)
+    if not seen:
+        print("reduced-TLS: WARNING firmware.map names no mbedTLS archive",
+              file=sys.stderr)
+        return
+    print("reduced-TLS: confirmed %d archives linked from %s"
+          % (len(seen), staged))
+
+
+env.AddPostAction("$BUILD_DIR/${PROGNAME}.elf", _verify_map)

scripts/mbedtls_4k_manifest.txt

@@ -0,0 +1,12 @@
+# Reduced-TLS mbedTLS archives (OUT_CONTENT_LEN 4096, IN_CONTENT_LEN 16384).
+# Format: <arch> <sha256> <filename>
+#
+# Rebuild these for every espressif32 platform bump — the archives must match the
+# rest of the framework they link against. Built as of platformio/espressif32@6.11.0
+# from the recipe in docs/mbedtls-tls-footprint.md.
+#
+# Fetch with: scripts/fetch_mbedtls_4k.sh esp32s3
+esp32s3 01629f635b33ffa2c1fdfcd8ac52327cd3a92e8d4e7c9a32d92974e0cfdfe398 libmbedcrypto.a
+esp32s3 07e7a09847589fefc35bc8d2f739d556535d535acd7185a9824b2af8edd5b05a libmbedtls_2.a
+esp32s3 e12bcc8d76a368e819987f266e73c265178d6b6675f05d0e29014bb543c402af libmbedtls.a
+esp32s3 c1e10324e19f6d7763737f72d9032cec832bdfce60c7a7d381a5fcfc7dcad573 libmbedx509.a