Jean-Pierre MartineauandClaude Opus 5
01ca242c98
The NODE lane suited the two classes that name a node and left every other one reading a dash. On this mesh that is half the traffic: the direct messages, requests, responses, returned paths, acks and traces that relay past us naming no origin. So the lane asks each class its own question instead, and the header says SUBJECT — the word that stays true down the whole column. An advert or a telemetry frame is still about its node. A channel text is about its channel, named by the key whose MAC confirms the frame — never by the fingerprint alone, which 256 buckets make no proof at all; a channel we hold no key for reads as that fingerprint and says so. A direct message, request, response, returned path or anonymous request is about the two nodes it travels between, drawn sender → recipient on the hop-sequence widget, so endpoints read exactly as route hops do: resolved names, key-derived hues, our own node white the moment a frame is addressed to us. Too wide, and the sender drops to its hash — the frame is addressed, so the addressee is the half worth the cells. An ack is about the message it answers, a trace about its tag. And an unsigned channel post is filed under its channel rather than the bare word "channel", which named nothing. Half of that was unreadable before because nobody had opened the frame. The companion hands every overheard packet to the library, which decodes the body for exactly two classes — advert and channel text — and leaves the rest as bytes. Those bytes are not opaque: MeshCore addresses a frame in the first few of them, and core/frames.py now reads them, class by class, against the layouts the protocol's own docs/payloads.md publishes. The trace tag is the little-endian uint32 the firmware writes, so it reads as the same number a trace reply reports; an ack's checksum keeps wire order, the form the companion reports its own delivery acks in. Two checks guard every decode, because a wrong slice yields not an error but a plausible-looking hash: the frame's payload version must be v1 (the header reserves two bits for a v2 that widens both hash and MAC), and the body must be long enough for the layout its class promises. Either failing yields nothing at all. The feed opens seeded from history, so any of this that lived only in a live event's raw payload would leave the opening screen blank — the same problem the channel-text crypto trio had. Same answer: three columns (dest/src/tag), written on capture and restored to the very keys the event used, so a replayed frame and a fresh one read identically all the way up. Older rows carry NULL until re-heard. The viewer gained the matching rows: to, from, tag, acks, and a channel for a datagram — whose body is not text, but whose MAC still names it. Verified against the MeshCore protocol documentation and firmware source, then corroborated on the air: every frame captured off the radio has a body length that decomposes exactly into its documented layout, an anonymous request's 51 bytes (1 + 32 + 2 + one AES block) included. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
256 lines
12 KiB
Python
256 lines
12 KiB
Python
"""SQLite database bootstrap and schema.
|
|
|
|
The schema is intentionally generic: a central ``runs`` table records every tool
|
|
execution (with arguments and a JSON summary), and measurement tables reference it so
|
|
new tools can persist data without schema churn.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import sqlite3
|
|
from pathlib import Path
|
|
|
|
SCHEMA_VERSION = 13
|
|
|
|
_SCHEMA = """
|
|
CREATE TABLE IF NOT EXISTS schema_meta (
|
|
key TEXT PRIMARY KEY,
|
|
value TEXT NOT NULL
|
|
);
|
|
|
|
-- Small key/value store for UI state that should survive across sessions (e.g. the last
|
|
-- map viewport), independent of any tool run. New keys need no schema change.
|
|
CREATE TABLE IF NOT EXISTS app_state (
|
|
key TEXT PRIMARY KEY,
|
|
value TEXT NOT NULL
|
|
);
|
|
|
|
-- One row per tool execution; the spine that measurements hang off of.
|
|
CREATE TABLE IF NOT EXISTS runs (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
tool TEXT NOT NULL,
|
|
profile TEXT,
|
|
args_json TEXT NOT NULL DEFAULT '{}',
|
|
status TEXT NOT NULL DEFAULT 'running', -- running | ok | error
|
|
summary_json TEXT,
|
|
started_at TEXT NOT NULL,
|
|
finished_at TEXT
|
|
);
|
|
|
|
-- Aggregated outcome of a single path trace, linked to its run.
|
|
CREATE TABLE IF NOT EXISTS traces (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
run_id INTEGER NOT NULL REFERENCES runs(id) ON DELETE CASCADE,
|
|
target TEXT NOT NULL,
|
|
success INTEGER NOT NULL,
|
|
hop_count INTEGER NOT NULL DEFAULT 0,
|
|
min_snr REAL,
|
|
round_trip_ms REAL,
|
|
tx_power INTEGER,
|
|
created_at TEXT NOT NULL
|
|
);
|
|
|
|
-- Per-hop SNR for a trace.
|
|
CREATE TABLE IF NOT EXISTS trace_hops (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
trace_id INTEGER NOT NULL REFERENCES traces(id) ON DELETE CASCADE,
|
|
hop_index INTEGER NOT NULL,
|
|
node TEXT,
|
|
snr REAL NOT NULL
|
|
);
|
|
|
|
-- One robust sample per (tx_power) tried during TX-power optimization.
|
|
CREATE TABLE IF NOT EXISTS tx_samples (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
run_id INTEGER NOT NULL REFERENCES runs(id) ON DELETE CASCADE,
|
|
target TEXT NOT NULL,
|
|
tx_power INTEGER NOT NULL,
|
|
median_min_snr REAL,
|
|
success_rate REAL,
|
|
samples INTEGER NOT NULL,
|
|
created_at TEXT NOT NULL
|
|
);
|
|
|
|
-- A candidate path evaluated during path optimization.
|
|
CREATE TABLE IF NOT EXISTS path_candidates (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
run_id INTEGER NOT NULL REFERENCES runs(id) ON DELETE CASCADE,
|
|
target TEXT NOT NULL,
|
|
path_json TEXT NOT NULL,
|
|
bottleneck_snr REAL,
|
|
success_rate REAL,
|
|
median_rtt_ms REAL,
|
|
created_at TEXT NOT NULL
|
|
);
|
|
|
|
-- One entry of a remote repeater's neighbour table, fetched over the mesh (v8). Each row
|
|
-- is a link the repeater reported hearing directly, with SNR measured at the repeater;
|
|
-- refetching appends a new snapshot and readers take the latest row per pair.
|
|
CREATE TABLE IF NOT EXISTS neighbour_reports (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
run_id INTEGER NOT NULL REFERENCES runs(id) ON DELETE CASCADE,
|
|
repeater TEXT NOT NULL, -- canonical id of the repeater that was asked
|
|
neighbour TEXT NOT NULL, -- reported neighbour's key-prefix hex, as replied
|
|
snr REAL, -- dB, measured at the repeater
|
|
heard_at TEXT, -- when the repeater last heard the neighbour
|
|
fetched_at TEXT NOT NULL
|
|
);
|
|
|
|
-- One record-holding walk in the trophy case (v11), scored from a trace that came
|
|
-- home (see services/records). Records live per
|
|
-- (category, width_bytes) because the per-hop hash width bounds both the walk's maximum
|
|
-- length (the path field is 64 bytes) and its collision odds — a 1-byte record is not
|
|
-- comparable to a 4-byte one. Standalone rows (no run_id): the walks' traces are already
|
|
-- persisted under their own runs, and a record must outlive history pruning. app_version
|
|
-- stamps the discoverer so future schema/scoring migrations can tell eras apart.
|
|
CREATE TABLE IF NOT EXISTS discovered_paths (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
category TEXT NOT NULL,
|
|
width_bytes INTEGER NOT NULL,
|
|
spec TEXT NOT NULL, -- the transmitted hex spec, comma-separated
|
|
route_json TEXT NOT NULL, -- canonical walked node ids, aligned with spec
|
|
score REAL NOT NULL,
|
|
stats_json TEXT NOT NULL,
|
|
app_version TEXT NOT NULL,
|
|
discovered_at TEXT NOT NULL
|
|
);
|
|
|
|
-- One packet overheard while passively monitoring the mesh (advert/telemetry/...).
|
|
CREATE TABLE IF NOT EXISTS observations (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
run_id INTEGER NOT NULL REFERENCES runs(id) ON DELETE CASCADE,
|
|
node TEXT, -- 12-hex canonical id (grouped/joined on)
|
|
public_key TEXT, -- the node's full public key, when the advert carried one
|
|
name TEXT,
|
|
kind TEXT NOT NULL DEFAULT 'advert',
|
|
node_type INTEGER, -- advert type (repeater/chat/room/...), when carried
|
|
snr REAL,
|
|
rssi REAL,
|
|
lat REAL,
|
|
lon REAL,
|
|
path TEXT, -- 'packet' rows: comma-separated relay-hop hex hashes
|
|
observed_at TEXT NOT NULL,
|
|
chan_hash TEXT, -- overheard channel frames: channel-hash fingerprint (hex)
|
|
cipher_mac TEXT, -- …its 2-byte MAC (hex)
|
|
crypted TEXT, -- …its ciphertext (hex), so a stored channel text still decrypts
|
|
payload_typename TEXT, -- 'packet' rows: the frame's payload class (GRP_TXT/TRACE/…)
|
|
dest TEXT, -- …the recipient's key hash, for an addressed class (hex)
|
|
src TEXT, -- …the sender's key hash, or its whole key (anon request)
|
|
tag TEXT -- …the frame's own token: an ack's checksum, a trace's tag
|
|
);
|
|
|
|
-- One chat message, sent or received, on a channel or with a contact. Unlike the other
|
|
-- measurement tables these are not tied to a single tool run (they arrive unsolicited via
|
|
-- the event hub), so run_id is nullable and detaches rather than cascades.
|
|
CREATE TABLE IF NOT EXISTS messages (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
run_id INTEGER REFERENCES runs(id) ON DELETE SET NULL,
|
|
outbound INTEGER NOT NULL DEFAULT 0,
|
|
is_channel INTEGER NOT NULL DEFAULT 0,
|
|
channel_id TEXT, -- a channel's slot-independent identity (its history key)
|
|
channel_idx INTEGER, -- the slot it went out on / arrived on (reference only)
|
|
peer TEXT,
|
|
peer_name TEXT,
|
|
text TEXT NOT NULL,
|
|
snr REAL,
|
|
acked INTEGER,
|
|
created_at TEXT NOT NULL
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_traces_run ON traces(run_id);
|
|
CREATE INDEX IF NOT EXISTS idx_trace_hops_trace ON trace_hops(trace_id);
|
|
CREATE INDEX IF NOT EXISTS idx_tx_samples_run ON tx_samples(run_id);
|
|
CREATE INDEX IF NOT EXISTS idx_neighbour_reports_pair ON neighbour_reports(repeater, neighbour);
|
|
CREATE INDEX IF NOT EXISTS idx_discovered_paths_cat ON discovered_paths(category, width_bytes);
|
|
CREATE INDEX IF NOT EXISTS idx_observations_run ON observations(run_id);
|
|
CREATE INDEX IF NOT EXISTS idx_observations_node ON observations(node);
|
|
CREATE INDEX IF NOT EXISTS idx_messages_peer ON messages(peer);
|
|
"""
|
|
|
|
|
|
def connect(db_path: Path) -> sqlite3.Connection:
|
|
"""Open (creating if needed) the SQLite database and ensure the schema exists.
|
|
|
|
Args:
|
|
db_path: Filesystem location of the database. Parent directories are created.
|
|
|
|
Returns:
|
|
An open connection with ``Row`` factory and foreign keys enabled.
|
|
"""
|
|
db_path.parent.mkdir(parents=True, exist_ok=True)
|
|
conn = sqlite3.connect(db_path)
|
|
conn.row_factory = sqlite3.Row
|
|
conn.execute("PRAGMA foreign_keys = ON;")
|
|
conn.executescript(_SCHEMA)
|
|
_migrate(conn)
|
|
conn.execute(
|
|
"INSERT OR REPLACE INTO schema_meta(key, value) VALUES ('version', ?)",
|
|
(str(SCHEMA_VERSION),),
|
|
)
|
|
conn.commit()
|
|
return conn
|
|
|
|
|
|
def _migrate(conn: sqlite3.Connection) -> None:
|
|
"""Bring an existing database's tables up to the current schema.
|
|
|
|
``executescript(_SCHEMA)`` only *creates* missing tables (``IF NOT EXISTS``); it cannot
|
|
add a column to a table an older version already created. Column additions are applied
|
|
here, guarded so the migration is idempotent and safe to run on every open. Indexes on
|
|
added columns are created here too (not in ``_SCHEMA``), so ``executescript`` never
|
|
references a column an older database hasn't grown yet.
|
|
"""
|
|
message_cols = {row["name"] for row in conn.execute("PRAGMA table_info(messages)")}
|
|
if "channel_id" not in message_cols:
|
|
# v3 -> v4: channel history moved from being keyed by slot index to a
|
|
# slot-independent channel identity. Existing rows are backfilled lazily at runtime
|
|
# (see Repository.backfill_channel_ids) once the device's channels can be read.
|
|
conn.execute("ALTER TABLE messages ADD COLUMN channel_id TEXT")
|
|
conn.execute(
|
|
"CREATE INDEX IF NOT EXISTS idx_messages_channel_id ON messages(channel_id)"
|
|
)
|
|
|
|
observation_cols = {row["name"] for row in conn.execute("PRAGMA table_info(observations)")}
|
|
if "node_type" not in observation_cols:
|
|
# v4 -> v5: observations gained the transmitting node's advert type, so the map can
|
|
# tell repeaters from leaf nodes. Older rows simply carry NULL (type unknown).
|
|
conn.execute("ALTER TABLE observations ADD COLUMN node_type INTEGER")
|
|
if "path" not in observation_cols:
|
|
# v6 -> v7: observations gained the relay path of RX-logged packets, the passive
|
|
# evidence the mesh topology graph is built from. Older rows carry NULL (no path).
|
|
conn.execute("ALTER TABLE observations ADD COLUMN path TEXT")
|
|
if "chan_hash" not in observation_cols:
|
|
# v8 -> v9: overheard channel-text (GRP_TXT) frames keep the fields the packet
|
|
# viewer needs to decrypt them — the channel-hash fingerprint, the 2-byte MAC, and
|
|
# the ciphertext — so a channel we hold the key for stays readable even when the feed
|
|
# is seeded from stored history (a live frame carries them in its raw payload; a
|
|
# replayed one had nowhere to keep them). Older rows carry NULL (never decryptable).
|
|
conn.execute("ALTER TABLE observations ADD COLUMN chan_hash TEXT")
|
|
conn.execute("ALTER TABLE observations ADD COLUMN cipher_mac TEXT")
|
|
conn.execute("ALTER TABLE observations ADD COLUMN crypted TEXT")
|
|
if "payload_typename" not in observation_cols:
|
|
# v9 -> v10: 'packet' RX-log rows keep their payload class (GRP_TXT, TRACE, PATH,
|
|
# ACK, …), the one identifying thing a relayed flood carries when it names no
|
|
# origin node — so the dashboard feed can read "channel text" / "trace" instead of
|
|
# a bare "?" even when seeded from history. Older rows carry NULL (class unknown).
|
|
conn.execute("ALTER TABLE observations ADD COLUMN payload_typename TEXT")
|
|
if "dest" not in observation_cols:
|
|
# v12 -> v13: 'packet' rows keep what the frame addressed — the recipient's key
|
|
# hash, the sender's (a hash, or the whole key an anonymous request carries), and
|
|
# the token a tokened class stands on: an ack's checksum, a trace's tag. Decoded
|
|
# from the frame body (see meshterm.core.frames), which only a live event carries,
|
|
# so — exactly like the channel-text crypto trio above — a replayed row would
|
|
# otherwise have nowhere to keep it and the feed's subject lane would go blank the
|
|
# moment it was seeded from history. Older rows carry NULL until re-heard.
|
|
conn.execute("ALTER TABLE observations ADD COLUMN dest TEXT")
|
|
conn.execute("ALTER TABLE observations ADD COLUMN src TEXT")
|
|
conn.execute("ALTER TABLE observations ADD COLUMN tag TEXT")
|
|
if "public_key" not in observation_cols:
|
|
# v11 -> v12: adverts carry the transmitting node's full public key, but only its
|
|
# 12-hex prefix was ever stored as the node id. Keep that prefix as the canonical id
|
|
# (traces, contacts, and the topology graph all match on it) and record the whole key
|
|
# alongside it, so a hash lane can show more than the twelve stored digits. Older rows
|
|
# carry NULL until re-heard — the raw payloads the past keys arrived in were never
|
|
# persisted, so history can't be completed from itself.
|
|
conn.execute("ALTER TABLE observations ADD COLUMN public_key TEXT")
|