Docstrings and comments of meshterm/persistence are in ASD-STE100

The same rewrite as for the core package. Two docstrings now agree with
the code: node_names does one aggregate scan (the query is NOT INDEXED),
and the log file is plain text, not structured logging.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jean-Pierre MartineauandClaude Opus 5.5 committed 2026-10-06 03:29:03 -04:00
1 parent ffe31dac36
commit aff99b633f
4 files changed
+956 -835

No files matched your search

+1 -1
View File
@@ -1,2 +1,2 @@
# SPDX-License-Identifier: Apache-2.0
"""Persistence layer: SQLite schema, repository, and structured logging."""
"""The persistence layer: the SQLite schema, the repository, and the log setup."""
+139 -118
View File
@@ -1,9 +1,9 @@
# SPDX-License-Identifier: Apache-2.0
"""SQLite database bootstrap and schema.
"""The SQLite database: its start-up and its schema.
The schema is intentionally generic: a central ``runs`` table records every tool
execution (with arguments and a JSON summary), and measurement tables reference it so
new tools can persist data without schema churn.
The schema is generic on purpose. A central ``runs`` table stores each run of a tool,
with its arguments and a JSON summary. The measurement tables refer to that table. Thus a
new tool can store data, and the schema does not have to change again and again.
"""
from __future__ import annotations
@@ -184,39 +184,43 @@ CREATE INDEX IF NOT EXISTS idx_messages_peer ON messages(peer);
def _tune(conn: sqlite3.Connection) -> None:
"""Apply the durability/throughput pragmas, on the safe side of every trade-off.
"""Apply the pragmas for durability and throughput, with the safe choice in each trade-off.
MeshTerm writes constantly and in tiny pieces — every packet the monitor hears becomes
an ``observations`` row — and its most constrained host (the PicoCalc) keeps its
database on an SD card, where SQLite's default of a full ``fsync`` per transaction
costs milliseconds *each*. Three pragmas move that cost without moving the risk:
MeshTerm writes all the time, in very small parts: each packet that the monitor hears
becomes a row in ``observations``. The host with the most limits (the PicoCalc) keeps
its database on an SD card. There, the SQLite default (a full ``fsync`` for each
transaction) costs milliseconds for each transaction. Three pragmas make that cost
smaller, and they do not make the risk larger:
* ``journal_mode=WAL`` — writers append to a log instead of rewriting the rollback
journal, which turns each insert's several synchronous seeks into one append, and
lets a reader (a screen redrawing) run against a writer (the monitor recording)
instead of blocking on it. This one is *persistent*: it lives in the database file,
so it survives into every later connection once set.
* ``synchronous=NORMAL`` — the actual latency win, but **only requested once WAL is
confirmed engaged**. Under WAL, NORMAL risks losing the last few transactions to a
power cut and nothing worse; under the rollback journal it can leave the file
*corrupt*. A PicoCalc running on a battery pack loses power for real, so on any
filesystem that refused WAL this stays at the default ``FULL`` — slow and intact
beats fast and unreadable.
* ``temp_store=MEMORY`` — sorts and temporary b-trees stay in RAM rather than landing
on the card. This workload's temporaries are small (screen-sized query results), so
the memory is bounded and the card sees strictly less traffic.
* ``journal_mode=WAL``: a writer appends to a log, instead of a new write of the
rollback journal. Thus each insert does one append, instead of several synchronous
seeks. Also, a reader (a screen that paints) can run at the same time as a writer
(the monitor, which stores what it hears), and the reader does not wait for the
writer. This pragma is persistent: the database file keeps it. Thus, after it is set
one time, it applies to each later connection.
* ``synchronous=NORMAL``: this pragma is the one that decreases the latency. But
MeshTerm **asks for it only after it confirms that WAL is on**. With WAL, the risk of
NORMAL is the loss of the last few transactions at a power failure, and nothing
worse. With the rollback journal, NORMAL can make the file corrupt. A PicoCalc on a
battery pack does lose power in real use. Thus, on a filesystem that refused WAL,
this pragma stays at the default ``FULL``. A slow database that is intact is better
than a fast database that MeshTerm cannot read.
* ``temp_store=MEMORY``: sorts and temporary b-trees stay in RAM, instead of on the
card. The temporary data of this workload is small (query results of the size of a
screen). Thus the memory has a limit, and the card always gets less traffic, never
more.
WAL can legitimately be unavailable — a read-only mount, or a filesystem without the
shared-memory primitive it needs — and SQLite reports that by *returning* the mode it
actually left the database in rather than raising, so the result is inspected instead
of trusted. Either way this is best-effort tuning: a database that will not take the
pragmas is still a perfectly working database, so nothing here is allowed to fail the
open.
WAL can be unavailable for a correct reason: a read-only mount, or a filesystem that
does not have the shared-memory primitive that WAL must have. SQLite does not raise an
error for this. It returns the mode that the database is in after the request. Thus
this function checks the result, and does not assume that the change occurred. In all
cases, this tuning is best effort. A database that does not accept the pragmas still
works correctly. Thus nothing here can cause the open to fail.
Args:
conn: The freshly opened connection, before any schema work.
conn: The connection, immediately after it opens and before any work on the schema.
"""
# Returns the resulting mode as a row — "wal" only if the switch actually took.
# The pragma returns the result mode as a row. It is "wal" only if the change occurred.
try:
mode = conn.execute("PRAGMA journal_mode = WAL;").fetchone()
except sqlite3.Error: # pragma: no cover - filesystem-dependent
@@ -225,19 +229,20 @@ def _tune(conn: sqlite3.Connection) -> None:
conn.execute("PRAGMA synchronous = NORMAL;")
try:
conn.execute("PRAGMA temp_store = MEMORY;")
except sqlite3.Error: # pragma: no cover - defensive; no known failure mode
except sqlite3.Error: # pragma: no cover - defensive. No failure mode is known.
pass
def connect(db_path: Path) -> sqlite3.Connection:
"""Open (creating if needed) the SQLite database and ensure the schema exists.
"""Open or create the SQLite database, and make sure that the schema exists.
Args:
db_path: Filesystem location of the database. Parent directories are created.
db_path: The location of the database in the filesystem. The function creates the
parent directories.
Returns:
An open connection with ``Row`` factory, foreign keys enabled, and the
durability/throughput pragmas applied (see :func:`_tune`).
An open connection with the ``Row`` factory, with foreign keys turned on, and with
the pragmas for durability and throughput applied (refer to :func:`_tune`).
"""
db_path.parent.mkdir(parents=True, exist_ok=True)
conn = sqlite3.connect(db_path)
@@ -255,122 +260,138 @@ def connect(db_path: Path) -> sqlite3.Connection:
def _migrate(conn: sqlite3.Connection) -> None:
"""Bring an existing database's tables up to the current schema.
"""Change the tables of an existing database to the current schema.
``executescript(_SCHEMA)`` only *creates* missing tables (``IF NOT EXISTS``); it cannot
add a column to a table an older version already created. Column additions are applied
here, guarded so the migration is idempotent and safe to run on every open. Indexes on
added columns are created here too (not in ``_SCHEMA``), so ``executescript`` never
references a column an older database hasn't grown yet.
``executescript(_SCHEMA)`` only creates the tables that are missing
(``IF NOT EXISTS``). It cannot add a column to a table that an older version created.
Thus this function adds the new columns. A check guards each addition, so the
migration is idempotent and safe to run at each open. This function also creates the
indexes on the added columns (not ``_SCHEMA``). Thus ``executescript`` never refers to
a column that an older database does not have yet.
"""
message_cols = {row["name"] for row in conn.execute("PRAGMA table_info(messages)")}
if "channel_id" not in message_cols:
# v3 -> v4: channel history moved from being keyed by slot index to a
# slot-independent channel identity. Existing rows are backfilled lazily at runtime
# (see Repository.backfill_channel_ids) once the device's channels can be read.
# v3 -> v4: the key of the channel history changed from the slot index to a channel
# identity that does not depend on the slot. MeshTerm fills in the existing rows
# later, at runtime (refer to Repository.backfill_channel_ids), when it can read
# the channels of the device.
conn.execute("ALTER TABLE messages ADD COLUMN channel_id TEXT")
conn.execute("CREATE INDEX IF NOT EXISTS idx_messages_channel_id ON messages(channel_id)")
observation_cols = {row["name"] for row in conn.execute("PRAGMA table_info(observations)")}
if "node_type" not in observation_cols:
# v4 -> v5: observations gained the transmitting node's advert type, so the map can
# tell repeaters from leaf nodes. Older rows simply carry NULL (type unknown).
# v4 -> v5: observations got the advert type of the node that transmits. Thus the
# map can tell repeaters from leaf nodes. Older rows have NULL (the type is
# unknown).
conn.execute("ALTER TABLE observations ADD COLUMN node_type INTEGER")
if "path" not in observation_cols:
# v6 -> v7: observations gained the relay path of RX-logged packets, the passive
# evidence the mesh topology graph is built from. Older rows carry NULL (no path).
# v6 -> v7: observations got the relay path of the packets in the RX log. This path
# is the passive evidence from which MeshTerm builds the graph of the mesh
# topology. Older rows have NULL (no path).
conn.execute("ALTER TABLE observations ADD COLUMN path TEXT")
if "chan_hash" not in observation_cols:
# v8 -> v9: overheard channel-text (GRP_TXT) frames keep the fields the packet
# viewer needs to decrypt them — the channel-hash fingerprint, the 2-byte MAC, and
# the ciphertext — so a channel we hold the key for stays readable even when the feed
# is seeded from stored history (a live frame carries them in its raw payload; a
# replayed one had nowhere to keep them). Older rows carry NULL (never decryptable).
# v8 -> v9: overheard channel-text (GRP_TXT) packets keep the fields that the packet
# viewer must have to decrypt them: the channel-hash fingerprint, the 2-byte MAC,
# and the ciphertext. Thus a channel for which we have the key stays readable, also
# when the feed gets its first rows from stored history. (A live packet has these
# fields in its raw payload. A replayed packet had no place to keep them.) Older
# rows have NULL (MeshTerm can never decrypt them).
conn.execute("ALTER TABLE observations ADD COLUMN chan_hash TEXT")
conn.execute("ALTER TABLE observations ADD COLUMN cipher_mac TEXT")
conn.execute("ALTER TABLE observations ADD COLUMN crypted TEXT")
if "payload_typename" not in observation_cols:
# v9 -> v10: 'packet' RX-log rows keep their payload class (GRP_TXT, TRACE, PATH,
# ACK, …), the one identifying thing a relayed flood carries when it names no
# origin node — so the dashboard feed can read "channel text" / "trace" instead of
# a bare "?" even when seeded from history. Older rows carry NULL (class unknown).
# v9 -> v10: 'packet' rows from the RX log keep their payload class (GRP_TXT, TRACE,
# PATH, ACK, …). When a relayed flood does not name its origin node, the class is
# the only thing that identifies the flood. Thus the dashboard feed can show
# "channel text" or "trace" instead of only "?", also when it gets its first rows
# from history. Older rows have NULL (the class is unknown).
conn.execute("ALTER TABLE observations ADD COLUMN payload_typename TEXT")
if "dest" not in observation_cols:
# v12 -> v13: 'packet' rows keep what the frame addressed — the recipient's key
# hash, the sender's (a hash, or the whole key an anonymous request carries), and
# the token a tokened class stands on: an ack's checksum, a trace's tag. Decoded
# from the frame body (see meshterm.core.frames), which only a live event carries,
# so — exactly like the channel-text crypto trio above — a replayed row would
# otherwise have nowhere to keep it and the feed's subject lane would go blank the
# moment it was seeded from history. Older rows carry NULL until re-heard.
# v12 -> v13: 'packet' rows keep the addresses of the packet: the hash of the
# recipient's key, the sender's hash (or the full key that an anonymous request
# carries), and the token of a class that has one (the checksum of an ack, the tag
# of a trace). MeshTerm decodes these values from the packet body (refer to
# meshterm.core.frames), and only a live event has the body. Thus, the same as
# the three channel-text crypto fields above, a replayed row has no other place to
# keep these values. Without them, the subject lane of the feed becomes blank when
# the feed gets its first rows from history. Older rows have NULL until the packet
# is heard again.
conn.execute("ALTER TABLE observations ADD COLUMN dest TEXT")
conn.execute("ALTER TABLE observations ADD COLUMN src TEXT")
conn.execute("ALTER TABLE observations ADD COLUMN tag TEXT")
if "public_key" not in observation_cols:
# v11 -> v12: adverts carry the transmitting node's full public key, but only its
# 12-hex prefix was ever stored as the node id. Keep that prefix as the canonical id
# (traces, contacts, and the topology graph all match on it) and record the whole key
# alongside it, so a hash lane can show more than the twelve stored digits. Older rows
# carry NULL until re-heard — the raw payloads the past keys arrived in were never
# persisted, so history can't be completed from itself.
# v11 -> v12: an advert has the full public key of the node that transmits it, but
# MeshTerm stored only its key prefix (12 hex digits) as the node id. Keep that
# prefix as the canonical id (traces, contacts, and the topology graph all match on
# it). Store the full key next to it, so that a key lane can show more than the
# twelve stored digits. Older rows have NULL until the node is heard again.
# MeshTerm did not store the raw payloads that brought the past keys, so the
# history cannot complete itself.
conn.execute("ALTER TABLE observations ADD COLUMN public_key TEXT")
if "trace_snrs" not in observation_cols:
# v13 -> v14: a trace's header path field is not relay hashes but one signed SNR
# byte per hop it traversed (see meshterm.core.frames.trace_link_snrs), so it gets
# a column of its own rather than sitting in `path` pretending to be adjacency.
# Live frames carried the readings in their raw payload; without somewhere to keep
# them a replayed row would show a trace's links blank, the same gap the crypto
# trio and the addressing columns above were added to close.
# v13 -> v14: the path field in the header of a trace does not have relay hashes.
# It has one signed SNR byte for each hop that the trace went through (refer to
# meshterm.core.frames.trace_link_snrs). Thus it gets its own column, instead of a
# place in `path`, where it looks like adjacency data but is not. Live packets had
# the readings in their raw payload. Without a place to keep them, a replayed row
# shows the links of a trace as blank. This is the same gap that the crypto fields
# and the address columns above close.
conn.execute("ALTER TABLE observations ADD COLUMN trace_snrs TEXT")
if "route" not in observation_cols:
# v14 -> v15: a frame's route type, which says what its `path` field *means* — and
# without it the field was being read one way for packets that meant it two.
# v14 -> v15: the route type of a packet, which tells what its `path` field means.
# Without the route type, MeshTerm read the field in one way, but packets use it in
# two ways.
#
# A FLOOD packet accumulates: every relay appends its hash, so the path is the route
# the packet actually travelled to reach us. A DIRECT packet is the opposite — the
# path is a routing *instruction* the sender wrote and the relays consume, so it
# describes where the packet is going, not where it has been, and an empty one means
# "route used up", not "arrived in zero hops".
# A FLOOD packet collects hops: each relay appends its hash, so the path is the
# route that the packet went through to get to us. A DIRECT packet is the opposite.
# Its path is a routing instruction that the sender wrote and that the relays
# consume. Thus it tells where the packet goes, not where it was. An empty path
# means "route used up", not "arrived in zero hops".
#
# Reading every path as a travelled route therefore drew a direct-routed message as
# having reached us out of nowhere, and drew our own outgoing route as if it were an
# inbound one (JP, 2026-09-02). Older rows carry NULL and are read as unknown rather
# than assumed flooded; the raw headers they arrived in were never persisted, so this
# cannot be filled in from history.
# MeshTerm read each path as a route that the packet went through. Thus it drew a
# direct-routed message as if it came to us from nowhere, and it drew our own
# outgoing route as if it were an incoming route (JP, 2026-09-02). Older rows have
# NULL. MeshTerm reads them as unknown, and does not assume that they are floods.
# MeshTerm did not store the raw headers that they came with, so the history
# cannot supply this value.
conn.execute("ALTER TABLE observations ADD COLUMN route TEXT")
if "transport_code" not in observation_cols:
# v15 -> v16: a scoped flood's region. A TC_FLOOD frame carries a transport code —
# an HMAC of its own payload keyed on the region it was sent into — and nothing
# else names the region, so which one it was can only be answered by trying the
# region names known (see meshterm.core.regions). Names arrive late: a repeater
# tells us what it carries long after we overheard traffic scoped to it. So the
# code is kept with the exact bytes it was computed over, and a row heard today
# resolves against a name learned next week. The body is kept for TC_FLOOD rows
# alone — a plain flood has no code, a direct frame is never region-filtered — so
# the price is paid only by the traffic it explains. Older rows carry NULL: the
# raw frames they arrived in were never persisted, so their scope is lost.
# v15 -> v16: the region of a scoped flood. A TC_FLOOD packet has a transport code:
# an HMAC of its own payload, keyed on the region that it was sent into. Nothing
# else names the region. Thus the only way to find the region is to try each known
# region name (refer to meshterm.core.regions). The names come late: a repeater
# tells us which regions it carries a long time after we overheard traffic scoped
# to them. Thus MeshTerm keeps the code with the exact bytes from which it was
# computed, and a row heard today can resolve against a name learned next week.
# Only TC_FLOOD rows keep the body: a plain flood has no code, and a direct packet
# is never filtered by region. Thus only the traffic that the body explains pays
# its cost. Older rows have NULL: MeshTerm did not store the raw packets that they
# came in, so their scope is lost.
conn.execute("ALTER TABLE observations ADD COLUMN transport_code TEXT")
conn.execute("ALTER TABLE observations ADD COLUMN scope_body TEXT")
if "scope" not in message_cols:
# v16 -> v17: the scope a message we sent went out under. A channel can be given a
# region its messages are flooded into (a scope is a *setting on the companion*,
# made just before the send — see Device.send_channel_in_scope), and the scope can
# change between one message and the next, or be dropped for one resend. So the
# transcript cannot work out afterwards what a message went out under from what the
# channel's scope is *now*; it is kept with the message, the moment it is sent.
# Its first reader is the message paths dialog: a scoped message nothing relayed is
# most often one no repeater in earshot carries the region of, and saying so needs
# the region. Stored as the bare name, '*' for a flood sent unscoped, and NULL for
# anything else — inbound messages, direct messages, and every row sent before this
# column existed, whose scope was never known and cannot be recovered.
# v16 -> v17: the scope of a message that we sent. A channel can have a region into
# which its messages are flooded. (A scope is a setting on the companion, made
# immediately before the send: refer to Device.send_channel_in_scope.) The scope
# can change from one message to the next, or it can be removed for one resend.
# Thus, later, the transcript cannot find the scope of a message from the scope
# that the channel has now. MeshTerm keeps the scope with the message, at the time
# that it sends the message. The first code that reads it is the message paths
# dialog. When no repeater relayed a scoped message, the usual cause is that no
# repeater in radio range carries its region. To tell the user this, the dialog
# must have the region. The stored value is the bare name, '*' for a flood sent
# unscoped, and NULL for all other rows: incoming messages, direct messages, and
# each row sent before this column existed. The scope of these rows was never
# known, and MeshTerm cannot recover it.
conn.execute("ALTER TABLE messages ADD COLUMN scope TEXT")
if "author" not in message_cols:
# v17 -> v18: who wrote a room post. A room server relays its members' posts as
# direct messages from *itself*, so the row's peer is the room and the member who
# wrote the post is a second fact the wire carries beside it — the first four bytes
# of their key, which the room signs the post with. Stored as that lowercase hex;
# NULL for every other message, a room's own command-line replies included, which
# is what tells a post from a reply in the one conversation the two share. Rows
# from before this column are NULL too: a post recorded then lost its author on
# the way in, and nothing on disk can bring it back.
# v17 -> v18: the author of a room post. A room server relays the posts of its
# members as direct messages from itself. Thus the peer of the row is the room, and
# the member who wrote the post is a second fact that the packet carries next to
# it: the first four bytes of the member's key, with which the room signs the post.
# MeshTerm stores that key prefix as lowercase hex. The value is NULL for all other
# messages, which includes the command-line replies of a room. This NULL is how
# MeshTerm tells a post from a reply in the one conversation that the two share.
# Rows from before this column are NULL too: when MeshTerm stored such a post, the
# author was lost on the way in, and nothing on disk can bring it back.
conn.execute("ALTER TABLE messages ADD COLUMN author TEXT")
+81 -72
View File
@@ -1,23 +1,23 @@
# SPDX-License-Identifier: Apache-2.0
"""Logging setup: a plain text file you can open, and a console that stays out of the way.
"""Logging setup: a plain text file that you can open, and a console that does not interfere.
Two sinks. The console handler is Rich-formatted and shows what a person watching the
terminal should see. The file handler writes ordinary log lines to
``<config_dir>/meshterm.log`` — one record per line, timestamp, level, logger, message,
with a traceback indented under the line that raised it.
There are two sinks. The console handler uses Rich formatting. It shows the records that
are for a person who watches the terminal. The file handler writes ordinary log lines to
``<config_dir>/meshterm.log``: one record on each line, with the timestamp, the level,
the logger, and the message. A traceback is indented under the line that raised it.
Plain text on purpose. The file used to be JSON Lines, on the theory that something might
replay it; nothing ever did, and the one job it actually has is being opened by a person
who has hit a problem — often the person reporting it, who then pastes the last twenty
lines into an issue. That reader has a text editor, not a JSON parser.
The file is plain text on purpose. It was JSON Lines before, because some program might
replay it one day. No program ever did. The only job of the file is this: a person who
has a problem opens it. Often this person reports the problem, and pastes the last twenty
lines into an issue. That person has a text editor, not a JSON parser.
The file rotates. It used to be a single :class:`~logging.FileHandler`, which grows
without limit: a real one reached 11MB over ten weeks, which is both a lot of disk on a
handheld and an unreasonable thing to attach to a bug report.
The file rotates. Before, it was a single :class:`~logging.FileHandler`, which grows with
no limit. A real log file got to 11MB in ten weeks. That is a large quantity of disk
space on a handheld, and it is too large to attach to a bug report.
How much is written is a preference — ``log_level``, defaulting to ``WARNING``, so the
file holds the problems rather than a narration of a working session. Turn it down to
``INFO`` or ``DEBUG`` when chasing something.
A preference sets how much MeshTerm writes: ``log_level``, with the default ``WARNING``.
Thus the file has the problems, not a full account of a session that works. When you
look for the cause of a problem, set it lower, to ``INFO`` or ``DEBUG``.
"""
from __future__ import annotations
@@ -31,24 +31,26 @@ from rich.logging import RichHandler
_LOGGER_NAME = "meshterm"
#: The log file, in the config directory beside the data it describes.
#: The log file, in the config directory next to the data that it describes.
LOG_FILENAME = "meshterm.log"
#: Keep this much before rolling over, and this many rolled files. Roughly a fortnight of
#: a chatty session at DEBUG, and small enough that the whole set can be attached to an
#: issue without apology.
#: The size at which the file rolls over, and the number of rolled files to keep. This is
#: approximately two weeks of a session with much traffic at DEBUG. Also, the full set is
#: small enough to attach to an issue with no problem.
_MAX_BYTES = 2 * 1024 * 1024
_BACKUP_COUNT = 3
#: ``2026-09-07 14:23:11 WARNING meshterm.core.connection: link lost, retrying``
#: Level is padded so the messages line up; the logger name says which part spoke.
#: The level is padded, so that the messages align. The logger name tells which part of
#: MeshTerm wrote the record.
_LINE_FORMAT = "%(asctime)s %(levelname)-8s %(name)s: %(message)s"
_TIME_FORMAT = "%Y-%m-%d %H:%M:%S"
#: The levels the ``log_level`` preference offers, by their own names. Spelled out rather
#: than taken from ``logging.getLevelName``, whose string-to-number direction is deprecated,
#: or ``getLevelNamesMapping``, which arrived in 3.11 and this package supports 3.10.
#: The levels that the ``log_level`` preference offers, by their own names. They are
#: written out here, instead of taken from ``logging.getLevelName`` or
#: ``getLevelNamesMapping``. The string-to-number direction of ``getLevelName`` is
#: deprecated. ``getLevelNamesMapping`` came in Python 3.11, and this package supports 3.10.
_LEVELS: dict[str, int] = {
"ERROR": logging.ERROR,
"WARNING": logging.WARNING,
@@ -58,37 +60,39 @@ _LEVELS: dict[str, int] = {
def level_from_name(name: str, *, fallback: int = logging.WARNING) -> int:
"""Turn a ``log_level`` preference value into a level number.
"""Change a value of the ``log_level`` preference into a level number.
Args:
name: A level name, in any case.
fallback: Used when the name is unrecognised — a hand-edited ``preferences.toml``
with a typo in it should lose some log detail, not stop the app from starting.
name: A level name, in any letter case.
fallback: The level to use when the name is not known. A typo in a
``preferences.toml`` that a person edited must cause less log detail. It must
not prevent the start of the app.
Returns:
The matching level number.
The level number that matches.
"""
return _LEVELS.get(name.strip().upper(), fallback)
def log_path(log_dir: Path) -> Path:
"""Return the log file's path, for an error message that wants to name it."""
"""Return the path of the log file, for an error message that names it."""
return log_dir / LOG_FILENAME
def log_file_for(level: int) -> Path | None:
"""Return the file a record at ``level`` is written to, or ``None`` if none takes it.
"""Return the file that gets a record at ``level``, or ``None`` if no file gets it.
For a message that wants to say "the full error is in …": it may say so only when the
record really went there, and the ``log_level`` preference decides that — a reader who
turned the file down to ``ERROR`` has no warnings in it to be sent looking for.
This function is for a message that says "the full error is in …". The message can say
this only when the record went to that file, and the ``log_level`` preference controls
that. If a user set the level of the file to ``ERROR``, the file has no warnings. Thus
the message must not tell the user to look for them there.
Args:
level: The level the record was logged at.
level: The level at which the record was logged.
Returns:
The log file's path, or ``None`` when logging isn't configured or the file's level
is above ``level``.
The path of the log file, or ``None`` when logging is not configured or the level
of the file is above ``level``.
"""
for handler in logging.getLogger(_LOGGER_NAME).handlers:
if isinstance(handler, logging.FileHandler) and handler.level <= level:
@@ -107,11 +111,11 @@ def configure_logging(
"""Configure and return the application logger.
Args:
console: The Rich console the handler should render to.
log_dir: Directory for the ``meshterm.log`` file.
level: Console log level.
file_level: File log level, from the ``log_level`` preference.
quiet: When ``True``, suppress console output (file logging continues).
console: The Rich console to which the handler renders.
log_dir: The directory for the ``meshterm.log`` file.
level: The log level of the console.
file_level: The log level of the file, from the ``log_level`` preference.
quiet: When ``True``, write nothing to the console. Logging to the file continues.
Returns:
The configured ``meshterm`` logger.
@@ -122,12 +126,12 @@ def configure_logging(
logger.propagate = False
if not quiet:
# markup=False: a log line carries whatever the device said — a node name, a
# firmware error — and Rich would read `[...]` in it as a style tag. That made the
# *error path* the one most likely to fail: reporting a fault on a node called
# `[/]Bob` raised MarkupError from inside the handler, replacing the diagnostic
# with a traceback about the diagnostic. No log call in the package marks up its
# message, so parsing them buys nothing.
# markup=False: a log line can have any text that the device sent (a node name, a
# firmware error). With markup on, Rich reads `[...]` in such text as a style tag.
# Because of this, the error path was the path most likely to fail. A report of a
# fault on a node with the name `[/]Bob` raised MarkupError in the handler, and a
# traceback about the diagnostic replaced the diagnostic. No log call in the
# package uses markup in its message, so to parse the messages gives no benefit.
rich_handler = RichHandler(
console=console, rich_tracebacks=True, show_path=False, markup=False
)
@@ -150,31 +154,34 @@ def configure_logging(
return logger
#: Third-party loggers that must never write to the terminal — their records would print
#: straight over the full-screen TUI. Routed to the file handler instead so they stay
#: captured for debugging. ``meshcore`` is the notable offender: it calls
#: ``logging.basicConfig(level=INFO)`` on import (e.g. "INFO:meshcore:Serial Connection
#: started"). ``asyncio`` is included so its default exception handler — the one that
#: reports stray background-task errors once prompt_toolkit's screen-dumping handler is
#: disabled — logs to the file instead of the console.
#: Third-party loggers that must never write to the terminal, because their records print
#: over the full-screen TUI. This module routes them to the file handler instead, so that
#: the records stay available for debugging. ``meshcore`` is the main cause: it calls
#: ``logging.basicConfig(level=INFO)`` when it is imported (for example,
#: "INFO:meshcore:Serial Connection started"). ``asyncio`` is in the list so that its
#: default exception handler logs to the file instead of the console. That handler reports
#: stray errors of background tasks after the prompt_toolkit handler that dumps errors on
#: the screen is turned off.
_LIBRARY_LOGGERS: tuple[str, ...] = ("meshcore", "asyncio")
def _quiet_library_console(file_handler: logging.Handler) -> None:
"""Keep noisy third-party libraries off the terminal (they corrupt the TUI).
"""Keep the output of noisy third-party libraries off the terminal (it corrupts the TUI).
The ``meshcore`` client calls ``logging.basicConfig(level=INFO)`` when it is imported,
which installs a stream handler on the *root* logger that prints its INFO lines straight
over the screen. Two guards prevent that:
The ``meshcore`` client calls ``logging.basicConfig(level=INFO)`` when it is imported.
This call installs a stream handler on the root logger, and that handler prints its
INFO lines over the screen. Two guards prevent this problem:
1. Give the root logger a :class:`~logging.NullHandler` so ``basicConfig`` — which only
acts when the root has no handlers — becomes a no-op and never adds its stream handler.
2. Take each known library logger off propagation and attach only ``file_handler``, so
its records are still captured to the log but never reach the console, even if some
other code path installs a root stream handler anyway.
1. Give the root logger a :class:`~logging.NullHandler`. ``basicConfig`` acts only
when the root has no handlers. Thus it becomes a no-op and never adds its stream
handler.
2. Turn off propagation for each known library logger, and attach only
``file_handler``. Thus its records still go to the log, but they never get to the
console, also if some other code path installs a stream handler on the root logger.
Args:
file_handler: The application's file handler to also capture library records to.
file_handler: The file handler of the application. It also gets the library
records.
"""
root = logging.getLogger()
if not any(isinstance(h, logging.NullHandler) for h in root.handlers):
@@ -189,18 +196,19 @@ def _quiet_library_console(file_handler: logging.Handler) -> None:
def _drop_handlers(logger: logging.Logger) -> None:
"""Detach ``logger``'s handlers, closing each on the way out.
"""Detach the handlers of ``logger``, and close each one.
Clearing the list alone drops the *reference* and leaves the file open: every
reconfigure — a reconnect in the app, each in-process CLI invocation under test — leaked
another descriptor on ``meshterm.log``, which on Windows is also a lock on a file the
rotation then wants to rename. A handler that refuses to close is dropped anyway: the
point of this call is that the old handlers stop receiving records.
If the function only clears the list, it removes the reference but leaves the file
open. Each new configuration (a reconnect in the app, or each in-process CLI run under
test) leaked one more descriptor on ``meshterm.log``. On Windows, such a descriptor is
also a lock on a file that the rotation must rename later. A handler that fails to
close is removed all the same, because the purpose of this call is that the old
handlers get no more records.
"""
for handler in list(logger.handlers):
try:
handler.close()
except Exception: # noqa: BLE001 - a handler we are discarding cannot fail the app
except Exception: # noqa: BLE001 - a handler that we discard must not stop the app
pass
logger.handlers.clear()
@@ -209,7 +217,8 @@ def get_logger(name: str | None = None) -> logging.Logger:
"""Return a child of the application logger.
Args:
name: Optional child name; when ``None`` the root app logger is returned.
name: An optional child name. When it is ``None``, the function returns the root
logger of the app.
Returns:
The requested logger.
File diff suppressed because it is too large. Load diff