Commit Graph
15 Commits
Author SHA1 Message Date
Jean-Pierre MartineauandClaude Opus 5.5 c3c18b1c85 Formatting for the map's memory budget and its tests
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 17:16:08 -04:00
Jean-Pierre MartineauandClaude Opus 5.5 db6370407a A map moved faster than it can draw no longer freezes the PicoCalc
Panning or zooming the map quickly on the PicoCalc could stop the whole app
for many seconds. A watchdog run on the device found two causes behind it,
plus the port walk fixed in the previous commit.

Memory. A decoded tile holds 1 to 4.6 MB of RAM on the device (65 bytes a
point), not the 0.9 MB the code assumed. The source kept up to 24 of them,
counted as tiles, and the map screen kept 8 or more of its own on top, so a
few pans and a zoom out filled a 104 MB device. Swap is on the SD card, so
the process then stopped while the kernel read its code back in: 16,000
major page faults and a 15 s freeze in one run. Now the source holds decoded
tiles to a byte budget, one eighth of the machine's RAM (12.8 MB on the
PicoCalc, at most 128 MB on a desktop), weighed by mvt.resident_bytes. That
estimate was fitted against tracemalloc on the device's 157 cached tiles: 1%
under to 9% over there, and 6% under to 21% over on a 64-bit desktop. The
screen keeps only the tiles in view, and takes any tile already in RAM
straight from the source on the paint (BasemapSource.resident), with no trip
through a thread. The prefetcher warms the disk cache (BasemapSource.warm)
instead of filling the RAM with guesses, and passes over any tile already
written down. Tile loads take turns, two at a time, and a tile the view has
left before its turn comes is dropped, not loaded.

CPU. fill_polygon asked every edge about every scanline, which at z7 means
coastlines with tens of thousands of edges: 6.8 s of a 7.6 s frame. Each
edge is now filed under only the rows it crosses, and draws the same dots
(checked against the old fill on 3,000 random polygons, and in a test).
_draw_tile also tests each ring's bounds before projecting it, and skips a
ring wholly outside the view. That changes no dot: 1,500 frames from the
device's tiles match rendering with no culling at all. The one frame that
differed from HEAD was HEAD's own building cull, which dropped a footprint
whose fill rounds onto the canvas edge.

Measured on the PicoCalc with the profile-picocalc map_burst tour (keys sent
at 10 a second, from a pinned z10 view, real radio and database). The new
code ran first, so the baseline had the warmer tile cache:

  pans, catch-up after a burst     HEAD 1.0-1.8 s      now 0.4-0.6 s
  longest silence, whole tour      HEAD 11.6 s         now 0.6 s
  whole-process freezes            HEAD 8.0, 3.1, 2.1, 11.6 s
                                   now one, 1.6 s (idle pages swapped back in)
  swap at its peak                 HEAD 25 MB          now 7 MB
  warm raster, z7 / z8 / z9        HEAD 7.6 / 1.9 / 1.8 s
                                   now 0.57 / 0.29 / 0.81 s
  warm raster, z10 / z13           HEAD 0.49 / 0.56 s  now 0.26 / 0.26 s

Still open: z9 spends 0.23 s drawing lines; a prefetch decode of a heavy z7
tile is still 1.3 to 1.7 s on a worker thread; and pages swapped out while
the map sits idle can still cost a second or two to come back.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 17:15:19 -04:00
Jean-Pierre MartineauandClaude Opus 5 2e40603a64 Satisfy the lint gates the macOS work walked past
Import order, a docstring reworded rather than have "macOS" capitalised
into something that is not the product's name, two over-long test lines,
and _tls_context moved below the logger it uses so the module keeps its
shape: docstring, imports, constants, logger, then code.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-19 02:34:25 -04:00
Jean-Pierre MartineauandClaude Opus 5 730230461c The map draws no ground when the build has no certificates to trust
On a Mac the map showed nodes floating over nothing: overlays come from the
local database and arrived fine, while every base tile silently failed.

A frozen build can have no trust store. The default TLS context asks OpenSSL for
one, and on macOS and Linux that is a filesystem path baked into the
interpreter's own build -- which need not exist on a machine that never
installed that Python, which is every machine a PyInstaller bundle lands on.
Each tile fetch ended in CERTIFICATE_VERIFY_FAILED, and since the fetcher cannot
tell a refusal from empty terrain, the map drew what it had. Windows is the
exception that hid this: its default context reads the OS certificate store, so
the binary there always worked, and so did every checkout on every platform.

So the context now keeps the default when that default actually loaded
certificates, and otherwise points it at the bundle the operating system ships.
Those are the trust decisions the machine's administrator has made and the OS
keeps current; carrying our own copy would freeze a snapshot of them into every
release, and certifi -- the obvious way to do that -- is MPL-2.0, which this
project's own dependency policy refuses. `tests/test_notices.py` said so, which
is what that gate is for.

Verification is not weakened anywhere. With no bundle to be found the fetch
still verifies and still fails, because a map tile is not worth teaching the app
to skip certificate checks, and an unverified fetcher would be one import away
from being reused where it matters.

Confirmed against the 0.3.6 arm64 build on a Mac: handing the same binary the
OS bundle turned one hard failure into a fetched TileJSON and 7.3 MB of tiles.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 23:28:59 -04:00
Jean-Pierre MartineauandClaude Fable 5.1 c9f5a71ce7 Every Python file names its licence with an SPDX identifier
Apache-2.0 does not require per-file headers, but a file lifted out of the tree carried no marking at all. One comment line, first (or right after a shebang), on every .py under meshterm/, tests/, packaging/ and scripts/, plus the extensionless SPI bridge; tests/test_spdx.py keeps a new file from landing without one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-13 13:09:18 -04:00
Jean-Pierre MartineauandClaude Fable 5.1 ffd2649a35 Shared logic sits in the layer that reads it, and the CLI answers --version
A pre-release architecture and QA pass, applied as one change because the moves
share files and only the whole set was gate-tested (2480 passed, ruff clean).

Three new modules pull logic down out of layers that should not own it:
core/channel_probe.py (ChannelSlot, read_channel_slots, probe_channel_slots)
so services/device_state.py no longer imports a screen; services/markers.py
(gather_markers, with MapMarker a core/models dataclass) so two screens stop
importing a tool; core/tracing.py (trace_timeout, path_hash_flags) so
core/connection.py no longer imports services. The routing-prefix width is
DeviceState.routing_prefix_bytes(), replacing five private cross-module imports.

One definition per thing: is_path_hash, the NodeResolver/NameKeyResolver
aliases, TILE_RETRY_SECONDS, haversine_km, marks.SELF_MARK, MENTION; nine
private names that crossed module boundaries got public ones and menus.__all__
lists what other modules import. A preference's default is written once
(ALERT_CAP, DONE_CAP, DEFAULT_VIEW_FRACTION read the registry), the stranded
PREFERENCES doc comment sits on its subject, and three docstrings stop naming
questionary, a library dropped long ago.

The command line gains --version (eager, exit 0, documented, tested) and drops
the -p short form on trace, trace-path and tx-optimize: the root lifts -p as
--profile before a leaf ever sees it, so the leaf's could never work, and a
derived test now checks every leaf against every root short option.
tx-optimize with a one-hop --path is a usage error, exit 2, because nothing
transmitted is not a device failure. Log handlers are closed before they are
cleared (197 unclosed-file warnings gone), chat's fire-and-forget tasks keep a
reference, coroutines use get_running_loop(), the never-awaited MapScreen._load
in a test is awaited, and the CLI sweep passes --no-ble to devices so a test
run stops turning on the Bluetooth radio three times (the suite is 40 % faster).
The about page lists every dependency, and the release skill bumps the README's
download names, which still said 0.2.8.

apply_ops stays in tools/: it and its helpers build report blocks and open two
screens, so moving it to core/ would invert core over ui. The widgets CLAUDE.md
names by their private spellings (_format_age, _NODE_GLYPHS) keep them until
CLAUDE.md is edited alongside.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-12 19:19:45 -04:00
Jean-Pierre MartineauandClaude Opus 5 92d31a3191 Every file is spelled the way the formatter spells it
`ruff format .` across the repo: 153 files, no behaviour change. This is the
second gate CONTRIBUTING.md promises, and until now it reported 154 files needing
reformat — so a first contributor ran the command the guide told them to run and
could not tell their own diff from the background.

Landed on its own, with nothing else in it, because a whole-repo reformat is only
safe to skip past in blame if it is provably nothing but layout. It was checked
that way rather than assumed: every changed file was formatted in memory and
compared by full `ast.dump()`, and 152 of 153 come out structurally identical
(the one exception was a docstring opening on a quote character, fixed in the
commit before this one). The string and comment token streams were diffed
separately, so a quote flip or an edited comment could not hide inside a
re-layout — five quote-style normalisations and sixteen implicit-concatenation
joins, all value-identical, and one commented-out line gaining the space after
its `#`. `meshterm specimen` is byte-identical on both platforms, and the suite
is 1888 green, gallery gate and `test_theme16` included.

What actually changed, in order of how much of the diff it is: wrapped calls and
signatures rejoin onto one line where they fit inside the hundred columns; the
magic trailing comma expands what it is there to expand; complex slices gain
their spaces (`[len(noise) :]`); a few redundant parens go. The tables that are
laid out to be read were fenced in the previous commit and are untouched here.

The aligned trailing-comment column is gone from about ninety lanes that were not
fenced — including the `heat.*` ladder in `MESH_THEME`, which cannot be fenced
where it sits. Every comment stays on the entry it describes, so nothing there
reads wrong; it just no longer lines up.

Add this commit to `.git-blame-ignore-revs`, which follows immediately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-07 01:07:19 -04:00
Jean-Pierre MartineauandClaude Opus 5 8d8c8b0f2b The basemap user agent is one an operator can actually answer
`MeshTerm/0.1 (+https://github.com/; mesh node map)` went out on every
OpenFreeMap tile request and was wrong twice over: the URL is a bare host
nobody can visit, and `0.1` was a hand-written number that had already
drifted from the package's `0.1.0` and would drift again on the next bump.
OpenFreeMap issues no API key, so this header is the entire handle a tile
operator has on a client — the difference between being emailed about
traffic and being blocked for it.

The URL now points at the real repository, and the version is the package's
own `__version__`, which is what every other surface that shows a version
already reads (the About page, the splash wordmark, the menu mark, a trace
export). That is the single source of truth here, so reusing it is what
keeps the string honest.

Deliberately *not* `importlib.metadata.version("meshterm")`, despite that
being the usual advice: MeshTerm is run straight from a checkout at least as
often as from a wheel — this repo's own venv has no `meshterm` distribution
installed, nor does the PicoCalc deploy — so the metadata lookup would raise
`PackageNotFoundError` in exactly the setups it is developed and tested in,
and a fallback around it would be a second version literal to keep in step
with the first. Reading the package attribute has no failure mode at all,
which is also why the value stays a module constant rather than becoming a
function: it never changes, computing it cannot raise, and the tile fetch
path is no place to discover either.

Three tests in the basemap section of `test_map.py`: the string is exactly
the repo, the live version and the descriptive tail; it rides the header of
the request that actually leaves; and the module still imports with a real
version in it when the metadata lookup is made to fail outright, which is
the regression guard against someone reaching for `importlib.metadata` later
without a fallback.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-07 00:48:41 -04:00
Jean-Pierre MartineauandClaude Opus 5 71b16ec7b7 The annotations, imports, and dead locals answer to ruff
`ruff check --fix` over the whole tree, ahead of putting the linter in CI:
1977 mechanical fixes, overwhelmingly `Optional[X]` → `X | None` (1186 in the
package alone) and the quoted annotations that `from __future__ import
annotations` already made unnecessary, plus import sorting and unused imports.
Nothing here changes behaviour — the suite is 1885 green before and after.

Four findings were real rather than stylistic and are fixed by hand: a dead
`cells` range in `MapCanvas`, a dead slot dict in the heat-ladder test, a
`ProgressScreen` task handle nobody read, and a tab loop binding a `label` it
never used. The two `F821`s were quoted forward references to names imported
inside their function bodies — now declared under `TYPE_CHECKING`, which keeps
`test_performance`'s local import (it guards the startup import graph) while
letting the linter resolve the name.

412 findings remain and are deliberately untouched: they need a decision about
the rule, not an edit to the code.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-06 17:26:12 -04:00
Jean-Pierre MartineauandClaude Opus 5 f5d0fb3677 Map: silence about a tile is not the source saying it's empty
A tile fetch that gets no answer was being remembered as one that did.
BasemapSource is careful about the difference — a timeout is never written
to disk, precisely so the next pan can ask again — and both callers then
threw it away, filing every None as "fetched, empty/absent" and never
asking again for the life of the screen. The same latch sat one level up:
_resolve() marked itself done whether or not it had resolved anything, so
one failed TileJSON round-trip made `available` False, and _ensure_tiles
skipped every fetch, for the whole session.

That is a desktop no-op and a PicoCalc bug. The device brings its Wi-Fi up
some forty seconds into the boot, well after an app started alongside it
reaches the menu and warms the tile source — so the map answers "offline"
before the answer can be true and draws the entire session from whatever
the cache already held. JP, 2026-08-18: black tiles at the two highest
zooms, where one z14 tile is the whole screen and a hole is the picture.
VE2RAD_FH_50m's tile (14/4842/5861) was simply absent from the device
cache; nothing there had been written in ten days of using the map, and it
fetched in 1.4s the moment it was asked for.

So: `available` reports what is known and never itself goes to the network
(it is read on the paint path), a failed resolve stands for 30s rather than
for ever, `answered_empty` lets a caller tell the two Nones apart, and a
tile nobody answered about waits out a 20s cooldown instead of a restart.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 20:28:34 -04:00
Jean-Pierre MartineauandClaude Fable 5 57386fe879 Repaint: we already composed the frame - stop having it read back
Every paint handed prompt_toolkit a finished ANSI frame and let it parse
that back into fragments, write all ~1400 cells into a fresh Char grid,
diff the grid, and re-emit ANSI. Measured on the PicoCalc that round trip
is 70-125 ms, and it is paid in full whether the frame changed by one row,
by all of them, or not at all - the 2 s header tick recomposes an
identical frame and still pays it.

Since the frame is already rows of text, FastRenderer compares this
frame's rows against the last frame's and rewrites only the ones that
differ. Opt-in for now (MESHTERM_FASTRENDER=1) and deliberately narrow:
a floating dialog or the busy overlay is placed by pt's float containers,
so those paints fall straight through to the stock renderer.

The erase has to lead the row, never follow it. A row that exactly fills
53 columns leaves the cursor in the last column with wrap pending, and an
erase-to-end there clears the character just written - the last cell of
every row went missing until the clear moved in front.

Alongside it, three costs that were pure waste on this platform:

  * The redraw was postponed up to max_render_postpone_time before it
    even started - a flat 3 ms on the Lyra, 10 ms on desktop, on every
    key. That default guards a UI whose own output floods it with
    invalidations; here they are keystrokes and a 2 s tick.
  * The splash rendered its body twice per paint - once to measure the
    box, once to fill it - and ran the corner glow ungated, walking every
    line to hand back the identical list on a console with no truecolor
    to blend. compose_dialog had the same ungated glow.
  * A tile's decoded layers were re-read from the SD card on every frame
    (~96 ms each), so panning one dot re-read every tile that had not
    moved. They are held in RAM now, bounded at a viewport plus its ring.

On the device, driven over the real radio: main menu 115 -> 40 ms,
Contacts 155 -> 73 ms, Dashboard 213 -> 117 ms, leaving a screen
375 -> 103 ms. Console readback confirms the frames are byte-identical.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 01:29:24 -04:00
Jean-Pierre MartineauandClaude Opus 5 504ee37b9a Tile cache: write the decoding down, not just the bytes
The layer narrowing cut what a tile costs to parse; this stops us paying
it twice. Decoding is a pure function of the bytes and the layer set, but
the cache only held the bytes, so every session re-parsed the same tiles
from scratch -- and parsing is the single most expensive thing this app
does.

A decoded/ sidecar now sits beside the raw tiles holding the parsed
layers, via marshal: stdlib, C-speed, and it understands the plain
tuples and scalars a decoded tile reduces to. Deliberately not pickle --
this reads a file off disk, and marshal cannot be talked into importing a
module or calling a constructor.

The sidecar is derived data throughout, and the design leans on that. It
is never what makes a tile "known", so losing it costs only the decode it
was saving. Every way it can fail to be what this build expects -- bumped
wire version, a Python whose marshal differs, a truncated file -- reads
as a cache miss rather than an error. It carries a stamp of the layer set
it was decoded under, so widening DRAWN_LAYERS re-decodes instead of
serving a narrower blob. And because it runs ~2.4x the size of the bytes
it came from, it is the half of the cache that gets a budget: 64 MB, LRU
by mtime, swept once per 8 MB written rather than per tile, since the
walk is slow on the SD card the PicoCalc runs from.

Measured on the device over the 50 real cached tiles: a first session
costs 126 s and a later one 13 s, 9x, with all 50 tiles restoring
identically. Disk goes 7.8 MB -> 7.8 + 19.0 MB. Serialising adds ~13% to
the first decode.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 00:45:21 -04:00
Jean-Pierre MartineauandClaude Opus 5 4e7bcacc37 Tile decode: don't pay for the layers the map never draws
Profiling on the Lyra found vector-tile decode dominating everything else
in the app by about two orders of magnitude: a median cached tile took
1236 ms of pure CPU, the largest 3.4 s, against 16 ms of SD read for a
whole viewport. Decoded tiles are memoised for the session, so this is a
first-pan cost -- but it runs on a GIL-bound worker thread, so it starves
the render loop for those seconds rather than running beside it.

Roughly 40% of that work was thrown away. A planet tile ships building,
housenumber, poi and mountain_peak; _draw_tile looks up ten layer names
and none of them is those. So decode_tile now takes the set of layers the
caller actually wants, reads each layer's identity with a cheap header
walk, and decodes the features only for the ones asked for. Unwanted
layers are still returned, named and empty -- which matters, because the
tile source decides whether a cache entry is real by whether the bytes
parsed into layers, and a tile whose every layer we skip must not read as
blank and get its perfectly good cache entry pruned and re-downloaded
every session. That test moves off "has features" and onto "parsed into
layers", which is sound because the other two ways a tile isn't real --
empty and corrupt -- decode to nothing and raise respectively.

Only the decode narrows; whole tiles are still cached, so drawing more
later costs a re-decode and never a re-download.

Measured on the device across the 46 real cached tiles: 183 s -> 51 s,
3.6x, with the drawn layers byte-identical on every one. The median tile
is 1259 -> 353 ms, the largest 3545 -> 759 ms.

_Reader.skip stopped slicing out the bytes it steps over while it was
here, which is most of what makes the header walk cheap.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 00:04:59 -04:00
Jean-Pierre MartineauandClaude Opus 5 649a55b558 Tile cache: a flaky link's silence is not an empty tile
Every fetch failure — offline, DNS, timeout, reset, 5xx — collapsed into
None and then into b"", which the loader wrote to disk as "this tile is
legitimately empty". One 12-second Wi-Fi stall on the PicoCalc therefore
poisoned that tile for good: every later pan and every later session read
the zero-byte marker and drew a black square. A body cut short mid-link
did the same by another door, cached and undecodable ever after. The PC's
network simply never flaked long enough to trip it.

So the cache now holds only tiles that decode to real geometry. A GET
returns whether the source actually answered: a 2xx body or a 404 is an
answer, everything else is silence and is never written down, so the next
pan asks again. A body short of its Content-Length is truncation, not a
tile. A tile the source really calls empty is remembered in memory for
the session alone — no repeat requests while the map is open, no
permanent hole on disk.

Cached entries that no longer decode are pruned on read, so a cache
already poisoned this way repairs itself as you pan over it. Writes go
through a temp file and os.replace, since the map and a mini-map share
one source and can fetch the same tile from their own threads.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 22:52:18 -04:00
Jean-Pierre MartineauandClaude Opus 4.8 2ec18138a7 Rename project from MeshTools to MeshTerm
Rename the package directory, imports, brand strings, env-var prefixes,
console-script entry point, and the config/data dir (~/.config/meshterm).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 19:40:49 -04:00