Docstrings and comments of meshterm/persistence are in ASD-STE100
The same rewrite as for the core package. Two docstrings now agree with the code: node_names does one aggregate scan (the query is NOT INDEXED), and the log file is plain text, not structured logging. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
ffe31dac36
commit
aff99b633f
4 files changed
+956
-835
No files matched your search
@@ -1,2 +1,2 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Persistence layer: SQLite schema, repository, and structured logging."""
|
||||
"""The persistence layer: the SQLite schema, the repository, and the log setup."""
|
||||
+139
-118
@@ -1,9 +1,9 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""SQLite database bootstrap and schema.
|
||||
"""The SQLite database: its start-up and its schema.
|
||||
|
||||
The schema is intentionally generic: a central ``runs`` table records every tool
|
||||
execution (with arguments and a JSON summary), and measurement tables reference it so
|
||||
new tools can persist data without schema churn.
|
||||
The schema is generic on purpose. A central ``runs`` table stores each run of a tool,
|
||||
with its arguments and a JSON summary. The measurement tables refer to that table. Thus a
|
||||
new tool can store data, and the schema does not have to change again and again.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -184,39 +184,43 @@ CREATE INDEX IF NOT EXISTS idx_messages_peer ON messages(peer);
|
||||
|
||||
|
||||
def _tune(conn: sqlite3.Connection) -> None:
|
||||
"""Apply the durability/throughput pragmas, on the safe side of every trade-off.
|
||||
"""Apply the pragmas for durability and throughput, with the safe choice in each trade-off.
|
||||
|
||||
MeshTerm writes constantly and in tiny pieces — every packet the monitor hears becomes
|
||||
an ``observations`` row — and its most constrained host (the PicoCalc) keeps its
|
||||
database on an SD card, where SQLite's default of a full ``fsync`` per transaction
|
||||
costs milliseconds *each*. Three pragmas move that cost without moving the risk:
|
||||
MeshTerm writes all the time, in very small parts: each packet that the monitor hears
|
||||
becomes a row in ``observations``. The host with the most limits (the PicoCalc) keeps
|
||||
its database on an SD card. There, the SQLite default (a full ``fsync`` for each
|
||||
transaction) costs milliseconds for each transaction. Three pragmas make that cost
|
||||
smaller, and they do not make the risk larger:
|
||||
|
||||
* ``journal_mode=WAL`` — writers append to a log instead of rewriting the rollback
|
||||
journal, which turns each insert's several synchronous seeks into one append, and
|
||||
lets a reader (a screen redrawing) run against a writer (the monitor recording)
|
||||
instead of blocking on it. This one is *persistent*: it lives in the database file,
|
||||
so it survives into every later connection once set.
|
||||
* ``synchronous=NORMAL`` — the actual latency win, but **only requested once WAL is
|
||||
confirmed engaged**. Under WAL, NORMAL risks losing the last few transactions to a
|
||||
power cut and nothing worse; under the rollback journal it can leave the file
|
||||
*corrupt*. A PicoCalc running on a battery pack loses power for real, so on any
|
||||
filesystem that refused WAL this stays at the default ``FULL`` — slow and intact
|
||||
beats fast and unreadable.
|
||||
* ``temp_store=MEMORY`` — sorts and temporary b-trees stay in RAM rather than landing
|
||||
on the card. This workload's temporaries are small (screen-sized query results), so
|
||||
the memory is bounded and the card sees strictly less traffic.
|
||||
* ``journal_mode=WAL``: a writer appends to a log, instead of a new write of the
|
||||
rollback journal. Thus each insert does one append, instead of several synchronous
|
||||
seeks. Also, a reader (a screen that paints) can run at the same time as a writer
|
||||
(the monitor, which stores what it hears), and the reader does not wait for the
|
||||
writer. This pragma is persistent: the database file keeps it. Thus, after it is set
|
||||
one time, it applies to each later connection.
|
||||
* ``synchronous=NORMAL``: this pragma is the one that decreases the latency. But
|
||||
MeshTerm **asks for it only after it confirms that WAL is on**. With WAL, the risk of
|
||||
NORMAL is the loss of the last few transactions at a power failure, and nothing
|
||||
worse. With the rollback journal, NORMAL can make the file corrupt. A PicoCalc on a
|
||||
battery pack does lose power in real use. Thus, on a filesystem that refused WAL,
|
||||
this pragma stays at the default ``FULL``. A slow database that is intact is better
|
||||
than a fast database that MeshTerm cannot read.
|
||||
* ``temp_store=MEMORY``: sorts and temporary b-trees stay in RAM, instead of on the
|
||||
card. The temporary data of this workload is small (query results of the size of a
|
||||
screen). Thus the memory has a limit, and the card always gets less traffic, never
|
||||
more.
|
||||
|
||||
WAL can legitimately be unavailable — a read-only mount, or a filesystem without the
|
||||
shared-memory primitive it needs — and SQLite reports that by *returning* the mode it
|
||||
actually left the database in rather than raising, so the result is inspected instead
|
||||
of trusted. Either way this is best-effort tuning: a database that will not take the
|
||||
pragmas is still a perfectly working database, so nothing here is allowed to fail the
|
||||
open.
|
||||
WAL can be unavailable for a correct reason: a read-only mount, or a filesystem that
|
||||
does not have the shared-memory primitive that WAL must have. SQLite does not raise an
|
||||
error for this. It returns the mode that the database is in after the request. Thus
|
||||
this function checks the result, and does not assume that the change occurred. In all
|
||||
cases, this tuning is best effort. A database that does not accept the pragmas still
|
||||
works correctly. Thus nothing here can cause the open to fail.
|
||||
|
||||
Args:
|
||||
conn: The freshly opened connection, before any schema work.
|
||||
conn: The connection, immediately after it opens and before any work on the schema.
|
||||
"""
|
||||
# Returns the resulting mode as a row — "wal" only if the switch actually took.
|
||||
# The pragma returns the result mode as a row. It is "wal" only if the change occurred.
|
||||
try:
|
||||
mode = conn.execute("PRAGMA journal_mode = WAL;").fetchone()
|
||||
except sqlite3.Error: # pragma: no cover - filesystem-dependent
|
||||
@@ -225,19 +229,20 @@ def _tune(conn: sqlite3.Connection) -> None:
|
||||
conn.execute("PRAGMA synchronous = NORMAL;")
|
||||
try:
|
||||
conn.execute("PRAGMA temp_store = MEMORY;")
|
||||
except sqlite3.Error: # pragma: no cover - defensive; no known failure mode
|
||||
except sqlite3.Error: # pragma: no cover - defensive. No failure mode is known.
|
||||
pass
|
||||
|
||||
|
||||
def connect(db_path: Path) -> sqlite3.Connection:
|
||||
"""Open (creating if needed) the SQLite database and ensure the schema exists.
|
||||
"""Open or create the SQLite database, and make sure that the schema exists.
|
||||
|
||||
Args:
|
||||
db_path: Filesystem location of the database. Parent directories are created.
|
||||
db_path: The location of the database in the filesystem. The function creates the
|
||||
parent directories.
|
||||
|
||||
Returns:
|
||||
An open connection with ``Row`` factory, foreign keys enabled, and the
|
||||
durability/throughput pragmas applied (see :func:`_tune`).
|
||||
An open connection with the ``Row`` factory, with foreign keys turned on, and with
|
||||
the pragmas for durability and throughput applied (refer to :func:`_tune`).
|
||||
"""
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = sqlite3.connect(db_path)
|
||||
@@ -255,122 +260,138 @@ def connect(db_path: Path) -> sqlite3.Connection:
|
||||
|
||||
|
||||
def _migrate(conn: sqlite3.Connection) -> None:
|
||||
"""Bring an existing database's tables up to the current schema.
|
||||
"""Change the tables of an existing database to the current schema.
|
||||
|
||||
``executescript(_SCHEMA)`` only *creates* missing tables (``IF NOT EXISTS``); it cannot
|
||||
add a column to a table an older version already created. Column additions are applied
|
||||
here, guarded so the migration is idempotent and safe to run on every open. Indexes on
|
||||
added columns are created here too (not in ``_SCHEMA``), so ``executescript`` never
|
||||
references a column an older database hasn't grown yet.
|
||||
``executescript(_SCHEMA)`` only creates the tables that are missing
|
||||
(``IF NOT EXISTS``). It cannot add a column to a table that an older version created.
|
||||
Thus this function adds the new columns. A check guards each addition, so the
|
||||
migration is idempotent and safe to run at each open. This function also creates the
|
||||
indexes on the added columns (not ``_SCHEMA``). Thus ``executescript`` never refers to
|
||||
a column that an older database does not have yet.
|
||||
"""
|
||||
message_cols = {row["name"] for row in conn.execute("PRAGMA table_info(messages)")}
|
||||
if "channel_id" not in message_cols:
|
||||
# v3 -> v4: channel history moved from being keyed by slot index to a
|
||||
# slot-independent channel identity. Existing rows are backfilled lazily at runtime
|
||||
# (see Repository.backfill_channel_ids) once the device's channels can be read.
|
||||
# v3 -> v4: the key of the channel history changed from the slot index to a channel
|
||||
# identity that does not depend on the slot. MeshTerm fills in the existing rows
|
||||
# later, at runtime (refer to Repository.backfill_channel_ids), when it can read
|
||||
# the channels of the device.
|
||||
conn.execute("ALTER TABLE messages ADD COLUMN channel_id TEXT")
|
||||
conn.execute("CREATE INDEX IF NOT EXISTS idx_messages_channel_id ON messages(channel_id)")
|
||||
|
||||
observation_cols = {row["name"] for row in conn.execute("PRAGMA table_info(observations)")}
|
||||
if "node_type" not in observation_cols:
|
||||
# v4 -> v5: observations gained the transmitting node's advert type, so the map can
|
||||
# tell repeaters from leaf nodes. Older rows simply carry NULL (type unknown).
|
||||
# v4 -> v5: observations got the advert type of the node that transmits. Thus the
|
||||
# map can tell repeaters from leaf nodes. Older rows have NULL (the type is
|
||||
# unknown).
|
||||
conn.execute("ALTER TABLE observations ADD COLUMN node_type INTEGER")
|
||||
if "path" not in observation_cols:
|
||||
# v6 -> v7: observations gained the relay path of RX-logged packets, the passive
|
||||
# evidence the mesh topology graph is built from. Older rows carry NULL (no path).
|
||||
# v6 -> v7: observations got the relay path of the packets in the RX log. This path
|
||||
# is the passive evidence from which MeshTerm builds the graph of the mesh
|
||||
# topology. Older rows have NULL (no path).
|
||||
conn.execute("ALTER TABLE observations ADD COLUMN path TEXT")
|
||||
if "chan_hash" not in observation_cols:
|
||||
# v8 -> v9: overheard channel-text (GRP_TXT) frames keep the fields the packet
|
||||
# viewer needs to decrypt them — the channel-hash fingerprint, the 2-byte MAC, and
|
||||
# the ciphertext — so a channel we hold the key for stays readable even when the feed
|
||||
# is seeded from stored history (a live frame carries them in its raw payload; a
|
||||
# replayed one had nowhere to keep them). Older rows carry NULL (never decryptable).
|
||||
# v8 -> v9: overheard channel-text (GRP_TXT) packets keep the fields that the packet
|
||||
# viewer must have to decrypt them: the channel-hash fingerprint, the 2-byte MAC,
|
||||
# and the ciphertext. Thus a channel for which we have the key stays readable, also
|
||||
# when the feed gets its first rows from stored history. (A live packet has these
|
||||
# fields in its raw payload. A replayed packet had no place to keep them.) Older
|
||||
# rows have NULL (MeshTerm can never decrypt them).
|
||||
conn.execute("ALTER TABLE observations ADD COLUMN chan_hash TEXT")
|
||||
conn.execute("ALTER TABLE observations ADD COLUMN cipher_mac TEXT")
|
||||
conn.execute("ALTER TABLE observations ADD COLUMN crypted TEXT")
|
||||
if "payload_typename" not in observation_cols:
|
||||
# v9 -> v10: 'packet' RX-log rows keep their payload class (GRP_TXT, TRACE, PATH,
|
||||
# ACK, …), the one identifying thing a relayed flood carries when it names no
|
||||
# origin node — so the dashboard feed can read "channel text" / "trace" instead of
|
||||
# a bare "?" even when seeded from history. Older rows carry NULL (class unknown).
|
||||
# v9 -> v10: 'packet' rows from the RX log keep their payload class (GRP_TXT, TRACE,
|
||||
# PATH, ACK, …). When a relayed flood does not name its origin node, the class is
|
||||
# the only thing that identifies the flood. Thus the dashboard feed can show
|
||||
# "channel text" or "trace" instead of only "?", also when it gets its first rows
|
||||
# from history. Older rows have NULL (the class is unknown).
|
||||
conn.execute("ALTER TABLE observations ADD COLUMN payload_typename TEXT")
|
||||
if "dest" not in observation_cols:
|
||||
# v12 -> v13: 'packet' rows keep what the frame addressed — the recipient's key
|
||||
# hash, the sender's (a hash, or the whole key an anonymous request carries), and
|
||||
# the token a tokened class stands on: an ack's checksum, a trace's tag. Decoded
|
||||
# from the frame body (see meshterm.core.frames), which only a live event carries,
|
||||
# so — exactly like the channel-text crypto trio above — a replayed row would
|
||||
# otherwise have nowhere to keep it and the feed's subject lane would go blank the
|
||||
# moment it was seeded from history. Older rows carry NULL until re-heard.
|
||||
# v12 -> v13: 'packet' rows keep the addresses of the packet: the hash of the
|
||||
# recipient's key, the sender's hash (or the full key that an anonymous request
|
||||
# carries), and the token of a class that has one (the checksum of an ack, the tag
|
||||
# of a trace). MeshTerm decodes these values from the packet body (refer to
|
||||
# meshterm.core.frames), and only a live event has the body. Thus, the same as
|
||||
# the three channel-text crypto fields above, a replayed row has no other place to
|
||||
# keep these values. Without them, the subject lane of the feed becomes blank when
|
||||
# the feed gets its first rows from history. Older rows have NULL until the packet
|
||||
# is heard again.
|
||||
conn.execute("ALTER TABLE observations ADD COLUMN dest TEXT")
|
||||
conn.execute("ALTER TABLE observations ADD COLUMN src TEXT")
|
||||
conn.execute("ALTER TABLE observations ADD COLUMN tag TEXT")
|
||||
if "public_key" not in observation_cols:
|
||||
# v11 -> v12: adverts carry the transmitting node's full public key, but only its
|
||||
# 12-hex prefix was ever stored as the node id. Keep that prefix as the canonical id
|
||||
# (traces, contacts, and the topology graph all match on it) and record the whole key
|
||||
# alongside it, so a hash lane can show more than the twelve stored digits. Older rows
|
||||
# carry NULL until re-heard — the raw payloads the past keys arrived in were never
|
||||
# persisted, so history can't be completed from itself.
|
||||
# v11 -> v12: an advert has the full public key of the node that transmits it, but
|
||||
# MeshTerm stored only its key prefix (12 hex digits) as the node id. Keep that
|
||||
# prefix as the canonical id (traces, contacts, and the topology graph all match on
|
||||
# it). Store the full key next to it, so that a key lane can show more than the
|
||||
# twelve stored digits. Older rows have NULL until the node is heard again.
|
||||
# MeshTerm did not store the raw payloads that brought the past keys, so the
|
||||
# history cannot complete itself.
|
||||
conn.execute("ALTER TABLE observations ADD COLUMN public_key TEXT")
|
||||
if "trace_snrs" not in observation_cols:
|
||||
# v13 -> v14: a trace's header path field is not relay hashes but one signed SNR
|
||||
# byte per hop it traversed (see meshterm.core.frames.trace_link_snrs), so it gets
|
||||
# a column of its own rather than sitting in `path` pretending to be adjacency.
|
||||
# Live frames carried the readings in their raw payload; without somewhere to keep
|
||||
# them a replayed row would show a trace's links blank, the same gap the crypto
|
||||
# trio and the addressing columns above were added to close.
|
||||
# v13 -> v14: the path field in the header of a trace does not have relay hashes.
|
||||
# It has one signed SNR byte for each hop that the trace went through (refer to
|
||||
# meshterm.core.frames.trace_link_snrs). Thus it gets its own column, instead of a
|
||||
# place in `path`, where it looks like adjacency data but is not. Live packets had
|
||||
# the readings in their raw payload. Without a place to keep them, a replayed row
|
||||
# shows the links of a trace as blank. This is the same gap that the crypto fields
|
||||
# and the address columns above close.
|
||||
conn.execute("ALTER TABLE observations ADD COLUMN trace_snrs TEXT")
|
||||
if "route" not in observation_cols:
|
||||
# v14 -> v15: a frame's route type, which says what its `path` field *means* — and
|
||||
# without it the field was being read one way for packets that meant it two.
|
||||
# v14 -> v15: the route type of a packet, which tells what its `path` field means.
|
||||
# Without the route type, MeshTerm read the field in one way, but packets use it in
|
||||
# two ways.
|
||||
#
|
||||
# A FLOOD packet accumulates: every relay appends its hash, so the path is the route
|
||||
# the packet actually travelled to reach us. A DIRECT packet is the opposite — the
|
||||
# path is a routing *instruction* the sender wrote and the relays consume, so it
|
||||
# describes where the packet is going, not where it has been, and an empty one means
|
||||
# "route used up", not "arrived in zero hops".
|
||||
# A FLOOD packet collects hops: each relay appends its hash, so the path is the
|
||||
# route that the packet went through to get to us. A DIRECT packet is the opposite.
|
||||
# Its path is a routing instruction that the sender wrote and that the relays
|
||||
# consume. Thus it tells where the packet goes, not where it was. An empty path
|
||||
# means "route used up", not "arrived in zero hops".
|
||||
#
|
||||
# Reading every path as a travelled route therefore drew a direct-routed message as
|
||||
# having reached us out of nowhere, and drew our own outgoing route as if it were an
|
||||
# inbound one (JP, 2026-09-02). Older rows carry NULL and are read as unknown rather
|
||||
# than assumed flooded; the raw headers they arrived in were never persisted, so this
|
||||
# cannot be filled in from history.
|
||||
# MeshTerm read each path as a route that the packet went through. Thus it drew a
|
||||
# direct-routed message as if it came to us from nowhere, and it drew our own
|
||||
# outgoing route as if it were an incoming route (JP, 2026-09-02). Older rows have
|
||||
# NULL. MeshTerm reads them as unknown, and does not assume that they are floods.
|
||||
# MeshTerm did not store the raw headers that they came with, so the history
|
||||
# cannot supply this value.
|
||||
conn.execute("ALTER TABLE observations ADD COLUMN route TEXT")
|
||||
if "transport_code" not in observation_cols:
|
||||
# v15 -> v16: a scoped flood's region. A TC_FLOOD frame carries a transport code —
|
||||
# an HMAC of its own payload keyed on the region it was sent into — and nothing
|
||||
# else names the region, so which one it was can only be answered by trying the
|
||||
# region names known (see meshterm.core.regions). Names arrive late: a repeater
|
||||
# tells us what it carries long after we overheard traffic scoped to it. So the
|
||||
# code is kept with the exact bytes it was computed over, and a row heard today
|
||||
# resolves against a name learned next week. The body is kept for TC_FLOOD rows
|
||||
# alone — a plain flood has no code, a direct frame is never region-filtered — so
|
||||
# the price is paid only by the traffic it explains. Older rows carry NULL: the
|
||||
# raw frames they arrived in were never persisted, so their scope is lost.
|
||||
# v15 -> v16: the region of a scoped flood. A TC_FLOOD packet has a transport code:
|
||||
# an HMAC of its own payload, keyed on the region that it was sent into. Nothing
|
||||
# else names the region. Thus the only way to find the region is to try each known
|
||||
# region name (refer to meshterm.core.regions). The names come late: a repeater
|
||||
# tells us which regions it carries a long time after we overheard traffic scoped
|
||||
# to them. Thus MeshTerm keeps the code with the exact bytes from which it was
|
||||
# computed, and a row heard today can resolve against a name learned next week.
|
||||
# Only TC_FLOOD rows keep the body: a plain flood has no code, and a direct packet
|
||||
# is never filtered by region. Thus only the traffic that the body explains pays
|
||||
# its cost. Older rows have NULL: MeshTerm did not store the raw packets that they
|
||||
# came in, so their scope is lost.
|
||||
conn.execute("ALTER TABLE observations ADD COLUMN transport_code TEXT")
|
||||
conn.execute("ALTER TABLE observations ADD COLUMN scope_body TEXT")
|
||||
if "scope" not in message_cols:
|
||||
# v16 -> v17: the scope a message we sent went out under. A channel can be given a
|
||||
# region its messages are flooded into (a scope is a *setting on the companion*,
|
||||
# made just before the send — see Device.send_channel_in_scope), and the scope can
|
||||
# change between one message and the next, or be dropped for one resend. So the
|
||||
# transcript cannot work out afterwards what a message went out under from what the
|
||||
# channel's scope is *now*; it is kept with the message, the moment it is sent.
|
||||
# Its first reader is the message paths dialog: a scoped message nothing relayed is
|
||||
# most often one no repeater in earshot carries the region of, and saying so needs
|
||||
# the region. Stored as the bare name, '*' for a flood sent unscoped, and NULL for
|
||||
# anything else — inbound messages, direct messages, and every row sent before this
|
||||
# column existed, whose scope was never known and cannot be recovered.
|
||||
# v16 -> v17: the scope of a message that we sent. A channel can have a region into
|
||||
# which its messages are flooded. (A scope is a setting on the companion, made
|
||||
# immediately before the send: refer to Device.send_channel_in_scope.) The scope
|
||||
# can change from one message to the next, or it can be removed for one resend.
|
||||
# Thus, later, the transcript cannot find the scope of a message from the scope
|
||||
# that the channel has now. MeshTerm keeps the scope with the message, at the time
|
||||
# that it sends the message. The first code that reads it is the message paths
|
||||
# dialog. When no repeater relayed a scoped message, the usual cause is that no
|
||||
# repeater in radio range carries its region. To tell the user this, the dialog
|
||||
# must have the region. The stored value is the bare name, '*' for a flood sent
|
||||
# unscoped, and NULL for all other rows: incoming messages, direct messages, and
|
||||
# each row sent before this column existed. The scope of these rows was never
|
||||
# known, and MeshTerm cannot recover it.
|
||||
conn.execute("ALTER TABLE messages ADD COLUMN scope TEXT")
|
||||
if "author" not in message_cols:
|
||||
# v17 -> v18: who wrote a room post. A room server relays its members' posts as
|
||||
# direct messages from *itself*, so the row's peer is the room and the member who
|
||||
# wrote the post is a second fact the wire carries beside it — the first four bytes
|
||||
# of their key, which the room signs the post with. Stored as that lowercase hex;
|
||||
# NULL for every other message, a room's own command-line replies included, which
|
||||
# is what tells a post from a reply in the one conversation the two share. Rows
|
||||
# from before this column are NULL too: a post recorded then lost its author on
|
||||
# the way in, and nothing on disk can bring it back.
|
||||
# v17 -> v18: the author of a room post. A room server relays the posts of its
|
||||
# members as direct messages from itself. Thus the peer of the row is the room, and
|
||||
# the member who wrote the post is a second fact that the packet carries next to
|
||||
# it: the first four bytes of the member's key, with which the room signs the post.
|
||||
# MeshTerm stores that key prefix as lowercase hex. The value is NULL for all other
|
||||
# messages, which includes the command-line replies of a room. This NULL is how
|
||||
# MeshTerm tells a post from a reply in the one conversation that the two share.
|
||||
# Rows from before this column are NULL too: when MeshTerm stored such a post, the
|
||||
# author was lost on the way in, and nothing on disk can bring it back.
|
||||
conn.execute("ALTER TABLE messages ADD COLUMN author TEXT")
|
||||
@@ -1,23 +1,23 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Logging setup: a plain text file you can open, and a console that stays out of the way.
|
||||
"""Logging setup: a plain text file that you can open, and a console that does not interfere.
|
||||
|
||||
Two sinks. The console handler is Rich-formatted and shows what a person watching the
|
||||
terminal should see. The file handler writes ordinary log lines to
|
||||
``<config_dir>/meshterm.log`` — one record per line, timestamp, level, logger, message,
|
||||
with a traceback indented under the line that raised it.
|
||||
There are two sinks. The console handler uses Rich formatting. It shows the records that
|
||||
are for a person who watches the terminal. The file handler writes ordinary log lines to
|
||||
``<config_dir>/meshterm.log``: one record on each line, with the timestamp, the level,
|
||||
the logger, and the message. A traceback is indented under the line that raised it.
|
||||
|
||||
Plain text on purpose. The file used to be JSON Lines, on the theory that something might
|
||||
replay it; nothing ever did, and the one job it actually has is being opened by a person
|
||||
who has hit a problem — often the person reporting it, who then pastes the last twenty
|
||||
lines into an issue. That reader has a text editor, not a JSON parser.
|
||||
The file is plain text on purpose. It was JSON Lines before, because some program might
|
||||
replay it one day. No program ever did. The only job of the file is this: a person who
|
||||
has a problem opens it. Often this person reports the problem, and pastes the last twenty
|
||||
lines into an issue. That person has a text editor, not a JSON parser.
|
||||
|
||||
The file rotates. It used to be a single :class:`~logging.FileHandler`, which grows
|
||||
without limit: a real one reached 11MB over ten weeks, which is both a lot of disk on a
|
||||
handheld and an unreasonable thing to attach to a bug report.
|
||||
The file rotates. Before, it was a single :class:`~logging.FileHandler`, which grows with
|
||||
no limit. A real log file got to 11MB in ten weeks. That is a large quantity of disk
|
||||
space on a handheld, and it is too large to attach to a bug report.
|
||||
|
||||
How much is written is a preference — ``log_level``, defaulting to ``WARNING``, so the
|
||||
file holds the problems rather than a narration of a working session. Turn it down to
|
||||
``INFO`` or ``DEBUG`` when chasing something.
|
||||
A preference sets how much MeshTerm writes: ``log_level``, with the default ``WARNING``.
|
||||
Thus the file has the problems, not a full account of a session that works. When you
|
||||
look for the cause of a problem, set it lower, to ``INFO`` or ``DEBUG``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -31,24 +31,26 @@ from rich.logging import RichHandler
|
||||
|
||||
_LOGGER_NAME = "meshterm"
|
||||
|
||||
#: The log file, in the config directory beside the data it describes.
|
||||
#: The log file, in the config directory next to the data that it describes.
|
||||
LOG_FILENAME = "meshterm.log"
|
||||
|
||||
#: Keep this much before rolling over, and this many rolled files. Roughly a fortnight of
|
||||
#: a chatty session at DEBUG, and small enough that the whole set can be attached to an
|
||||
#: issue without apology.
|
||||
#: The size at which the file rolls over, and the number of rolled files to keep. This is
|
||||
#: approximately two weeks of a session with much traffic at DEBUG. Also, the full set is
|
||||
#: small enough to attach to an issue with no problem.
|
||||
_MAX_BYTES = 2 * 1024 * 1024
|
||||
_BACKUP_COUNT = 3
|
||||
|
||||
#: ``2026-09-07 14:23:11 WARNING meshterm.core.connection: link lost, retrying``
|
||||
#: Level is padded so the messages line up; the logger name says which part spoke.
|
||||
#: The level is padded, so that the messages align. The logger name tells which part of
|
||||
#: MeshTerm wrote the record.
|
||||
_LINE_FORMAT = "%(asctime)s %(levelname)-8s %(name)s: %(message)s"
|
||||
_TIME_FORMAT = "%Y-%m-%d %H:%M:%S"
|
||||
|
||||
|
||||
#: The levels the ``log_level`` preference offers, by their own names. Spelled out rather
|
||||
#: than taken from ``logging.getLevelName``, whose string-to-number direction is deprecated,
|
||||
#: or ``getLevelNamesMapping``, which arrived in 3.11 and this package supports 3.10.
|
||||
#: The levels that the ``log_level`` preference offers, by their own names. They are
|
||||
#: written out here, instead of taken from ``logging.getLevelName`` or
|
||||
#: ``getLevelNamesMapping``. The string-to-number direction of ``getLevelName`` is
|
||||
#: deprecated. ``getLevelNamesMapping`` came in Python 3.11, and this package supports 3.10.
|
||||
_LEVELS: dict[str, int] = {
|
||||
"ERROR": logging.ERROR,
|
||||
"WARNING": logging.WARNING,
|
||||
@@ -58,37 +60,39 @@ _LEVELS: dict[str, int] = {
|
||||
|
||||
|
||||
def level_from_name(name: str, *, fallback: int = logging.WARNING) -> int:
|
||||
"""Turn a ``log_level`` preference value into a level number.
|
||||
"""Change a value of the ``log_level`` preference into a level number.
|
||||
|
||||
Args:
|
||||
name: A level name, in any case.
|
||||
fallback: Used when the name is unrecognised — a hand-edited ``preferences.toml``
|
||||
with a typo in it should lose some log detail, not stop the app from starting.
|
||||
name: A level name, in any letter case.
|
||||
fallback: The level to use when the name is not known. A typo in a
|
||||
``preferences.toml`` that a person edited must cause less log detail. It must
|
||||
not prevent the start of the app.
|
||||
|
||||
Returns:
|
||||
The matching level number.
|
||||
The level number that matches.
|
||||
"""
|
||||
return _LEVELS.get(name.strip().upper(), fallback)
|
||||
|
||||
|
||||
def log_path(log_dir: Path) -> Path:
|
||||
"""Return the log file's path, for an error message that wants to name it."""
|
||||
"""Return the path of the log file, for an error message that names it."""
|
||||
return log_dir / LOG_FILENAME
|
||||
|
||||
|
||||
def log_file_for(level: int) -> Path | None:
|
||||
"""Return the file a record at ``level`` is written to, or ``None`` if none takes it.
|
||||
"""Return the file that gets a record at ``level``, or ``None`` if no file gets it.
|
||||
|
||||
For a message that wants to say "the full error is in …": it may say so only when the
|
||||
record really went there, and the ``log_level`` preference decides that — a reader who
|
||||
turned the file down to ``ERROR`` has no warnings in it to be sent looking for.
|
||||
This function is for a message that says "the full error is in …". The message can say
|
||||
this only when the record went to that file, and the ``log_level`` preference controls
|
||||
that. If a user set the level of the file to ``ERROR``, the file has no warnings. Thus
|
||||
the message must not tell the user to look for them there.
|
||||
|
||||
Args:
|
||||
level: The level the record was logged at.
|
||||
level: The level at which the record was logged.
|
||||
|
||||
Returns:
|
||||
The log file's path, or ``None`` when logging isn't configured or the file's level
|
||||
is above ``level``.
|
||||
The path of the log file, or ``None`` when logging is not configured or the level
|
||||
of the file is above ``level``.
|
||||
"""
|
||||
for handler in logging.getLogger(_LOGGER_NAME).handlers:
|
||||
if isinstance(handler, logging.FileHandler) and handler.level <= level:
|
||||
@@ -107,11 +111,11 @@ def configure_logging(
|
||||
"""Configure and return the application logger.
|
||||
|
||||
Args:
|
||||
console: The Rich console the handler should render to.
|
||||
log_dir: Directory for the ``meshterm.log`` file.
|
||||
level: Console log level.
|
||||
file_level: File log level, from the ``log_level`` preference.
|
||||
quiet: When ``True``, suppress console output (file logging continues).
|
||||
console: The Rich console to which the handler renders.
|
||||
log_dir: The directory for the ``meshterm.log`` file.
|
||||
level: The log level of the console.
|
||||
file_level: The log level of the file, from the ``log_level`` preference.
|
||||
quiet: When ``True``, write nothing to the console. Logging to the file continues.
|
||||
|
||||
Returns:
|
||||
The configured ``meshterm`` logger.
|
||||
@@ -122,12 +126,12 @@ def configure_logging(
|
||||
logger.propagate = False
|
||||
|
||||
if not quiet:
|
||||
# markup=False: a log line carries whatever the device said — a node name, a
|
||||
# firmware error — and Rich would read `[...]` in it as a style tag. That made the
|
||||
# *error path* the one most likely to fail: reporting a fault on a node called
|
||||
# `[/]Bob` raised MarkupError from inside the handler, replacing the diagnostic
|
||||
# with a traceback about the diagnostic. No log call in the package marks up its
|
||||
# message, so parsing them buys nothing.
|
||||
# markup=False: a log line can have any text that the device sent (a node name, a
|
||||
# firmware error). With markup on, Rich reads `[...]` in such text as a style tag.
|
||||
# Because of this, the error path was the path most likely to fail. A report of a
|
||||
# fault on a node with the name `[/]Bob` raised MarkupError in the handler, and a
|
||||
# traceback about the diagnostic replaced the diagnostic. No log call in the
|
||||
# package uses markup in its message, so to parse the messages gives no benefit.
|
||||
rich_handler = RichHandler(
|
||||
console=console, rich_tracebacks=True, show_path=False, markup=False
|
||||
)
|
||||
@@ -150,31 +154,34 @@ def configure_logging(
|
||||
return logger
|
||||
|
||||
|
||||
#: Third-party loggers that must never write to the terminal — their records would print
|
||||
#: straight over the full-screen TUI. Routed to the file handler instead so they stay
|
||||
#: captured for debugging. ``meshcore`` is the notable offender: it calls
|
||||
#: ``logging.basicConfig(level=INFO)`` on import (e.g. "INFO:meshcore:Serial Connection
|
||||
#: started"). ``asyncio`` is included so its default exception handler — the one that
|
||||
#: reports stray background-task errors once prompt_toolkit's screen-dumping handler is
|
||||
#: disabled — logs to the file instead of the console.
|
||||
#: Third-party loggers that must never write to the terminal, because their records print
|
||||
#: over the full-screen TUI. This module routes them to the file handler instead, so that
|
||||
#: the records stay available for debugging. ``meshcore`` is the main cause: it calls
|
||||
#: ``logging.basicConfig(level=INFO)`` when it is imported (for example,
|
||||
#: "INFO:meshcore:Serial Connection started"). ``asyncio`` is in the list so that its
|
||||
#: default exception handler logs to the file instead of the console. That handler reports
|
||||
#: stray errors of background tasks after the prompt_toolkit handler that dumps errors on
|
||||
#: the screen is turned off.
|
||||
_LIBRARY_LOGGERS: tuple[str, ...] = ("meshcore", "asyncio")
|
||||
|
||||
|
||||
def _quiet_library_console(file_handler: logging.Handler) -> None:
|
||||
"""Keep noisy third-party libraries off the terminal (they corrupt the TUI).
|
||||
"""Keep the output of noisy third-party libraries off the terminal (it corrupts the TUI).
|
||||
|
||||
The ``meshcore`` client calls ``logging.basicConfig(level=INFO)`` when it is imported,
|
||||
which installs a stream handler on the *root* logger that prints its INFO lines straight
|
||||
over the screen. Two guards prevent that:
|
||||
The ``meshcore`` client calls ``logging.basicConfig(level=INFO)`` when it is imported.
|
||||
This call installs a stream handler on the root logger, and that handler prints its
|
||||
INFO lines over the screen. Two guards prevent this problem:
|
||||
|
||||
1. Give the root logger a :class:`~logging.NullHandler` so ``basicConfig`` — which only
|
||||
acts when the root has no handlers — becomes a no-op and never adds its stream handler.
|
||||
2. Take each known library logger off propagation and attach only ``file_handler``, so
|
||||
its records are still captured to the log but never reach the console, even if some
|
||||
other code path installs a root stream handler anyway.
|
||||
1. Give the root logger a :class:`~logging.NullHandler`. ``basicConfig`` acts only
|
||||
when the root has no handlers. Thus it becomes a no-op and never adds its stream
|
||||
handler.
|
||||
2. Turn off propagation for each known library logger, and attach only
|
||||
``file_handler``. Thus its records still go to the log, but they never get to the
|
||||
console, also if some other code path installs a stream handler on the root logger.
|
||||
|
||||
Args:
|
||||
file_handler: The application's file handler to also capture library records to.
|
||||
file_handler: The file handler of the application. It also gets the library
|
||||
records.
|
||||
"""
|
||||
root = logging.getLogger()
|
||||
if not any(isinstance(h, logging.NullHandler) for h in root.handlers):
|
||||
@@ -189,18 +196,19 @@ def _quiet_library_console(file_handler: logging.Handler) -> None:
|
||||
|
||||
|
||||
def _drop_handlers(logger: logging.Logger) -> None:
|
||||
"""Detach ``logger``'s handlers, closing each on the way out.
|
||||
"""Detach the handlers of ``logger``, and close each one.
|
||||
|
||||
Clearing the list alone drops the *reference* and leaves the file open: every
|
||||
reconfigure — a reconnect in the app, each in-process CLI invocation under test — leaked
|
||||
another descriptor on ``meshterm.log``, which on Windows is also a lock on a file the
|
||||
rotation then wants to rename. A handler that refuses to close is dropped anyway: the
|
||||
point of this call is that the old handlers stop receiving records.
|
||||
If the function only clears the list, it removes the reference but leaves the file
|
||||
open. Each new configuration (a reconnect in the app, or each in-process CLI run under
|
||||
test) leaked one more descriptor on ``meshterm.log``. On Windows, such a descriptor is
|
||||
also a lock on a file that the rotation must rename later. A handler that fails to
|
||||
close is removed all the same, because the purpose of this call is that the old
|
||||
handlers get no more records.
|
||||
"""
|
||||
for handler in list(logger.handlers):
|
||||
try:
|
||||
handler.close()
|
||||
except Exception: # noqa: BLE001 - a handler we are discarding cannot fail the app
|
||||
except Exception: # noqa: BLE001 - a handler that we discard must not stop the app
|
||||
pass
|
||||
logger.handlers.clear()
|
||||
|
||||
@@ -209,7 +217,8 @@ def get_logger(name: str | None = None) -> logging.Logger:
|
||||
"""Return a child of the application logger.
|
||||
|
||||
Args:
|
||||
name: Optional child name; when ``None`` the root app logger is returned.
|
||||
name: An optional child name. When it is ``None``, the function returns the root
|
||||
logger of the app.
|
||||
|
||||
Returns:
|
||||
The requested logger.
|
||||
|
||||
+735
-644
File diff suppressed because it is too large.
Load diff
Reference in new issue
Block a user