Files
MeshTerm/meshterm/core/contact_store.py
T
Jean-Pierre MartineauandClaude Opus 5.5 6b13755c2b Every store writes only the fields its records know
The admin and remote stores round-tripped the raw JSON they read, so any
field a past version wrote survived forever. Both now read into typed
records (AdminCredential; _RemoteNode holding CachedValue settings) and
write out of them, keeping the file's existing spelling. The settings store
keeps only keys device_config still registers: it offers remembered values
back to the radio, so an unregistered key must not reach a restore.

Every persisted record now declares a RETIRED set (all empty: the history
holds no removed fields), device_config gains its own RETIRED registry, and
test_retired pins each store's records so the sweep cannot go vacuous.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 22:07:48 -04:00

442 lines
20 KiB
Python

# SPDX-License-Identifier: Apache-2.0
"""Persistence for contacts heard through MeshTerm, so they survive a device that forgets them.
Most companions keep their contact table in firmware, so MeshTerm reads contacts live from the
device and never has to remember them. A firmware-less radio bridge is the exception: it holds
its contact table only in RAM, so every contact is gone when the bridge process restarts — the
Contacts screen and the chat recipient list come up empty (or only as sparse as the adverts
heard so far this session), even though you were messaging those nodes last time.
This store is the durable backup for exactly that case. Every time MeshTerm reads the device's
contacts it records them here, keyed by the device's own public key so two radios keep separate
sets. When a list is then drawn, :func:`merge_contacts` unions the device's live contacts with
any remembered contact the device isn't currently reporting — so a forgetful device still shows
the nodes you know, and they stay messageable: a direct message addresses a node by a prefix of
its public key, which the remembered contact carries, so nothing has to be written back onto the
device to reach them.
The union is inert where it isn't needed: a firmware radio always reports its whole table, so
every remembered contact is already present and nothing is added — no device-type check required.
Recording never *removes* a contact on a device's empty read, so a bridge that just restarted
doesn't wipe the memory of what it knew.
A contact can also be **archived**, which is the opposite arrangement and the one the
Contacts sweep uses (see :mod:`~meshterm.core.contact_score`): the contact is deleted from
the *device*, freeing a slot in a companion's finite contact table, and kept here with an
:attr:`~RememberedContact.archived_at` stamp so nothing about it is actually lost.
:func:`merge_contacts` skips an archived contact — without that the union above would put it
straight back into every list, indistinguishable from a live one, and the sweep would appear
to have done nothing. What survives an archive is everything except the device row: the
node's whole reception history (untouched — it lives in the SQLite database, not here), its
direct-message transcript (keyed by key prefix, not by a contact row), and its full public
key, which is exactly what :meth:`~meshterm.core.connection.Device.add_contact` needs to put
it back. So a restore is one write, and the chat screen already performs it on demand when a
send is rejected for an unknown recipient (see
:class:`~meshterm.core.connection.ContactNotOnDeviceError`).
A live contact can also be **locked**, which is the reader's own word that it stays: a
locked contact is never a sweep candidate (see
:data:`~meshterm.core.contact_score.PROTECT_LOCKED`) and its page offers no Archive. The flag
lives here rather than on the device because the firmware has no such field — it is
MeshTerm's claim about the contact, kept beside the other one it makes (the archive stamp),
and a device read never clears it.
The one place to be careful is a firmware-less bridge, whose contact table is RAM-only and
for which this store *is* the memory: archiving there removes the contact from the lists for
real, and only a restore brings it back.
Like the other operator state (mutes, remembered channels, saved settings), this is global
machine state in a small JSON file (``<config_dir>/contacts.json``), not the per-invocation
SQLite database. Reads are served from memory after the first load; a write happens only when the
contact set actually changes, and flushes atomically.
"""
from __future__ import annotations
import json
from dataclasses import dataclass, replace
from pathlib import Path
from typing import ClassVar
from .atomicwrite import write_atomically
from .models import Contact, advert_time
def _norm(pubkey: str) -> str:
"""Normalise a public key (device or contact) to the lowercase hex used as a key."""
return (pubkey or "").lower().removeprefix("0x")
def _opt_int(value: object) -> int | None:
"""Coerce a JSON value to ``int``, or ``None`` if absent/unparseable."""
if value is None:
return None
try:
return int(value)
except (TypeError, ValueError):
return None
def _opt_float(value: object) -> float | None:
"""Coerce a JSON value to ``float``, or ``None`` if absent/unparseable."""
if value is None:
return None
try:
return float(value)
except (TypeError, ValueError):
return None
@dataclass(frozen=True)
class RememberedContact:
"""One contact MeshTerm remembers for a device: enough to list it and address a message.
Attributes:
public_key: The contact's full public key (lowercase hex) — how a direct message
addresses it, so it is what makes a remembered contact messageable.
name: The node's advertised name.
node_type: The advert type (see the ``NODE_TYPE_*`` constants), so the chat picker's
direct-messageable filter keeps behaving as it would for a live contact.
last_advert: Unix seconds of the node's most recent advert when last heard, for the
list's last-heard column; ``None`` when unknown.
lat: Last advertised latitude, if it shared one.
lon: Last advertised longitude, if it shared one.
archived_at: Unix seconds when this contact was swept off the device, or ``None``
while it is a live contact. An archived contact is remembered in full but kept
*out* of :func:`merge_contacts`, so it stops occupying a device slot without
being forgotten — and the stamp is what lets a list say how long ago it went.
locked: Whether the reader locked this contact against archiving. MeshTerm's own
state, not the device's, so :meth:`ContactStore.remember_all` carries it across
every fresh read of the contact rather than resetting it.
"""
public_key: str
name: str
node_type: int | None = None
last_advert: int | None = None
lat: float | None = None
lon: float | None = None
archived_at: int | None = None
locked: bool = False
#: Fields this record once held and must never hold again under another meaning (see
#: :data:`meshterm.core.preferences.RETIRED` for why a name is never reused).
RETIRED: ClassVar[frozenset[str]] = frozenset()
@property
def archived(self) -> bool:
"""Whether this contact has been swept off the device but kept here."""
return self.archived_at is not None
@classmethod
def from_contact(cls, contact: Contact) -> RememberedContact:
"""Distil a live :class:`~meshterm.core.models.Contact` into the fields we persist."""
epoch = int(contact.last_seen.timestamp()) if contact.last_seen else None
return cls(
public_key=_norm(contact.public_key),
name=contact.name,
node_type=contact.node_type,
last_advert=epoch,
lat=contact.lat,
lon=contact.lon,
)
def to_contact(self) -> Contact:
"""Rebuild a :class:`~meshterm.core.models.Contact` for the merged list.
The learned route is dropped (``route_hops=None``): a remembered contact floods until
the device relearns a path from received traffic, exactly as a freshly-heard one does.
The stored epoch re-enters through :func:`~meshterm.core.models.advert_time`, so a
future-stamped advert remembered before its sender's clock was corrected reads as
unknown rather than resurfacing as "heard in the future".
"""
last_seen = advert_time(self.last_advert)
return Contact(
name=self.name,
public_key=self.public_key,
key_prefix=self.public_key[:12],
last_seen=last_seen,
node_type=self.node_type,
lat=self.lat,
lon=self.lon,
route_hops=None,
)
class ContactStore:
"""Reads and writes the per-device set of remembered contacts, memory-first.
Interact through :meth:`contacts` (a device's remembered contacts) and :meth:`remember_all`
(record the contacts just read from a device — an upsert that never drops one on absence).
The backing map is loaded once on first access and kept in memory; a mutation persists the
whole map atomically, and only when something actually changed.
"""
def __init__(self, path: Path) -> None:
"""Open the store against a JSON file location.
Args:
path: Path to the JSON state file (created lazily on the first remembered contact).
"""
self._path = path
self._devices: dict[str, dict[str, RememberedContact]] | None = None
@property
def _state(self) -> dict[str, dict[str, RememberedContact]]:
"""The device -> {contact key -> remembered contact} map, loaded on first access."""
if self._devices is None:
self._devices = self._load()
return self._devices
def _load(self) -> dict[str, dict[str, RememberedContact]]:
"""Parse the file into the device map, or empty on a missing/corrupt file."""
try:
data = json.loads(self._path.read_text(encoding="utf-8"))
except (OSError, ValueError):
return {}
if not isinstance(data, dict):
return {}
devices: dict[str, dict[str, RememberedContact]] = {}
for pubkey, entries in (data.get("devices") or {}).items():
if not isinstance(entries, list):
continue
contacts: dict[str, RememberedContact] = {}
for entry in entries:
contact = _contact_from_json(entry)
if contact is not None:
contacts[contact.public_key] = contact
if contacts:
devices[_norm(str(pubkey))] = contacts
return devices
def contacts(self, device_pubkey: str) -> list[RememberedContact]:
"""The contacts remembered for a device (a copy, sorted by name), empty if none."""
remembered = self._state.get(_norm(device_pubkey), {})
return sorted(remembered.values(), key=lambda c: (c.name.lower(), c.public_key))
def remember_all(self, device_pubkey: str, contacts: list[Contact]) -> None:
"""Record the contacts just read from a device, upserting by public key.
A contact new to the device, or one whose fields changed (a rename, a fresher advert),
is stored; an unchanged one is left alone. A contact the device is *no longer* reporting
is **kept** — so a forgetful device's empty read never erases what it once knew. Persists
once, only if anything changed.
Args:
device_pubkey: The device's own public key.
contacts: The contacts just read from that device.
"""
dev = _norm(device_pubkey)
if not dev:
return
current = dict(self._state.get(dev, {}))
changed = False
for contact in contacts:
if not contact.public_key:
continue # unaddressable — nothing to remember it by
remembered = RememberedContact.from_contact(contact)
known = current.get(remembered.public_key)
if known is not None and known.locked:
# The lock is ours, not the device's: a fresh read knows nothing about it.
remembered = replace(remembered, locked=True)
if known != remembered:
current[remembered.public_key] = remembered
changed = True
if changed:
self._state[dev] = current
self._save()
def archived(self, device_pubkey: str) -> list[RememberedContact]:
"""The contacts archived off this device, most recently archived first.
The Contacts screen's ``Archived`` section. Ordered by when each was swept rather
than by name, because that is the question the section answers — *what did the last
sweep take?* — and a fresh sweep's work should be at the top of it.
"""
remembered = self._state.get(_norm(device_pubkey), {}).values()
return sorted(
(c for c in remembered if c.archived),
key=lambda c: (-(c.archived_at or 0), c.name.lower()),
)
def archive(self, device_pubkey: str, contact: Contact, *, when: int) -> None:
"""Mark one contact archived — remembered in full, but no longer merged into lists.
The store half of a sweep: the device half is
:meth:`~meshterm.core.connection.Device.remove_contact`, and this is what keeps the
removal from being a loss. The contact is *upserted* first, so archiving one the
store had never recorded (a live-only contact on a firmware radio, which is the
usual case) still remembers everything needed to put it back.
Args:
device_pubkey: The device's own public key.
contact: The contact being swept.
when: Unix seconds to stamp the archive with.
"""
dev = _norm(device_pubkey)
if not dev or not contact.public_key:
return # unaddressable — there would be nothing to restore it by
entry = replace(RememberedContact.from_contact(contact), archived_at=int(when))
current = dict(self._state.get(dev, {}))
if current.get(entry.public_key) == entry:
return
current[entry.public_key] = entry
self._state[dev] = current
self._save()
def restore(self, device_pubkey: str, contact_pubkey: str) -> RememberedContact | None:
"""Clear one contact's archived mark, returning what was archived.
Only the *store* side: the caller writes the contact back onto the device (see
:meth:`~meshterm.core.connection.Device.add_contact`) and calls this once that
succeeded, so a failed write never leaves a contact listed as live on a device that
doesn't hold it.
Args:
device_pubkey: The device's own public key.
contact_pubkey: The contact to un-archive.
Returns:
The record as it was archived, or ``None`` if no archived contact matched.
"""
dev = _norm(device_pubkey)
key = _norm(contact_pubkey)
contacts = self._state.get(dev) or {}
entry = contacts.get(key)
if entry is None or not entry.archived:
return None
current = dict(contacts)
current[key] = replace(entry, archived_at=None)
self._state[dev] = current
self._save()
return entry
def locked_keys(self, device_pubkey: str) -> frozenset[str]:
"""The full keys (lowercase hex) of every contact locked on this device."""
remembered = self._state.get(_norm(device_pubkey), {}).values()
return frozenset(c.public_key for c in remembered if c.locked)
def is_locked(self, device_pubkey: str, contact_pubkey: str) -> bool:
"""Whether one contact is locked against archiving on this device."""
entry = self._state.get(_norm(device_pubkey), {}).get(_norm(contact_pubkey))
return entry is not None and entry.locked
def set_locked(self, device_pubkey: str, contact: Contact, locked: bool) -> None:
"""Lock or unlock one contact, upserting it so a contact the store never saw can be.
Only a live contact is locked — the lock exists to keep a contact *off* the archive
path, so locking an archived one would claim something about a contact that has
already gone. Persists only a real change.
Args:
device_pubkey: The device's own public key.
contact: The contact to lock or unlock, addressed by its full key.
locked: The state to leave it in.
"""
dev = _norm(device_pubkey)
if not dev or not contact.public_key:
return # unaddressable — nothing to hang the flag on
current = dict(self._state.get(dev, {}))
key = _norm(contact.public_key)
known = current.get(key) or RememberedContact.from_contact(contact)
if known.archived or known.locked == locked:
return
current[key] = replace(known, locked=locked)
self._state[dev] = current
self._save()
def forget(self, device_pubkey: str, contact_pubkey: str) -> None:
"""Drop one remembered contact; persist only a real change."""
dev = _norm(device_pubkey)
contacts = self._state.get(dev)
key = _norm(contact_pubkey)
if not contacts or key not in contacts:
return
contacts = {k: v for k, v in contacts.items() if k != key}
if contacts:
self._state[dev] = contacts
else:
del self._state[dev]
self._save()
def _save(self) -> None:
"""Persist the whole device map atomically (a crash mid-write keeps the old file)."""
data = {
"devices": {
pubkey: [
_contact_to_json(c)
for c in sorted(contacts.values(), key=lambda c: (c.name.lower(), c.public_key))
]
for pubkey, contacts in sorted(self._state.items())
if contacts
}
}
write_atomically(self._path, json.dumps(data, indent=2))
def _contact_to_json(contact: RememberedContact) -> dict:
"""Serialise one remembered contact, omitting the fields it doesn't carry."""
entry: dict = {"public_key": contact.public_key, "name": contact.name}
if contact.node_type is not None:
entry["node_type"] = contact.node_type
if contact.last_advert is not None:
entry["last_advert"] = contact.last_advert
if contact.lat is not None:
entry["lat"] = contact.lat
if contact.lon is not None:
entry["lon"] = contact.lon
if contact.archived_at is not None:
entry["archived_at"] = contact.archived_at
if contact.locked:
entry["locked"] = True
return entry
def _contact_from_json(entry: object) -> RememberedContact | None:
"""Parse one stored contact entry, or ``None`` if it is malformed (no key or name)."""
if not isinstance(entry, dict):
return None
pubkey = _norm(str(entry.get("public_key", "")))
name = entry.get("name")
if not pubkey or not isinstance(name, str):
return None
return RememberedContact(
public_key=pubkey,
name=name,
node_type=_opt_int(entry.get("node_type")),
last_advert=_opt_int(entry.get("last_advert")),
lat=_opt_float(entry.get("lat")),
lon=_opt_float(entry.get("lon")),
archived_at=_opt_int(entry.get("archived_at")),
locked=entry.get("locked") is True,
)
def merge_contacts(store: ContactStore, device_pubkey: str, live: list[Contact]) -> list[Contact]:
"""Union a device's live contacts with any remembered ones it isn't currently reporting.
Live contacts pass through unchanged and first; a remembered contact whose key the device
already reports is left to the live entry (the fresher of the two), so nothing is
duplicated. A firmware radio reports its whole table, so this adds nothing there; a
forgetful device gets the missing contacts back, rebuilt from what was remembered.
Args:
store: The contact store to read remembered contacts from.
device_pubkey: The device's own public key.
live: The contacts the device is currently reporting.
Returns:
The live contacts followed by the remembered contacts the device is missing.
"""
present = {_norm(c.public_key) for c in live if c.public_key}
extra = [
remembered.to_contact()
for remembered in store.contacts(device_pubkey)
# An archived contact is deliberately withheld: it was swept off the device to free
# a slot, and merging it back would undo the sweep in the only place anyone looks.
if remembered.public_key
and not remembered.archived
and remembered.public_key not in present
]
return list(live) + extra