No log keeps a password or a secret: the log file, the run log, or the remote history

Passwords, channel secrets, pairing PINs, and private keys reached the DEBUG log and
the runs table in plain text: every chat run stored its channel secret inside the
Conversation repr, and --password, --secret, channel links, and config set device_pin
were stored as typed. meshterm.core.secrets now hides each one as <secret> by its
shape (a secret-named key, record, set operation, or dataclass field; bytes; a Secret;
a remote `password`/`set guest.password` command; name=value in text), and by value
for each secret MeshTerm has read or sent. Both log handlers and the runs table go
through it, always. The remote CLI history keeps `password ` without its value. Schema
v20 hides the secrets that older versions stored.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jean-Pierre MartineauandClaude Opus 5.5 committed 2026-10-18 00:20:22 -04:00
1 parent 9e579424b4
commit 1d983013d7
13 files changed
+896 -60

No files matched your search

+8
View File
@@ -115,6 +115,14 @@ radios from two directories in the same shell.
| `meshterm.log` | The log file. |
| `.lock` | The lock for a single instance. |
**No log keeps a password or a secret.** The log file, the run log in the database (the
arguments and the result of each command), and the remote command history of
`remote.json` write `<secret>` in the place of a password, a channel secret, a pairing
PIN, or a private key. This is always true, at each log level and with or without the
**Anonymize** preference. The history keeps `password ` without its value, so ↑ still
recalls the start of the command. `admin.json` keeps the admin passwords that you asked
MeshTerm to remember, because that is its purpose.
**`--db` alone does not isolate a run.** It moves the database. The database is most of
the data, but it does not hold the identity. The contact caches, the channel caches, the
outbox, the stored admin passwords, and the remembered device are all in the config
+2
View File
@@ -281,6 +281,8 @@ and in quoted text.
| save | Do an explicit save: the user's Save, or the device's save command | |
| record | One stored data item | |
| delete | Remove stored data permanently | purge (except the Purge feature) |
| secret | A value that unlocks something: a password, a channel secret, a pairing PIN, a private key. No log keeps one: it is hidden as `<secret>` | credential (except in a name) |
| hide (a secret) | Replace a secret with `<secret>` in a log or in the run log | redact, scrub |
| personal detail | A value that identifies the user or the equipment of the user, but is not a secret: the account name in a home folder, the address of a radio, the address of a host | PII, private data |
| mask | Replace most characters of a personal detail, and keep its first and last characters: `j*********u`, `DA:xx:xx:xx:7C:7C` | redact, scrub, obfuscate |
| anonymize | Mask each personal detail in a text: the `anonymize` preference does it to the log and the diagnostics | |
+2 -1
View File
@@ -21,7 +21,8 @@ MeshTerm also writes each failure to the log file, `~/.meshterm/meshterm.log` (r
[Where MeshTerm keeps its state](../cli/README.md#where-meshterm-keeps-its-state)). Attach
the log file to a bug report. To get more detail, set the **Log detail** preference to
`DEBUG` (`meshterm preferences set log_level DEBUG`), and run the command again. The log
can show your account name and the address of your radio. To mask them as MeshTerm writes
never holds a password or a secret, at any level. It can show your account name and the
address of your radio. To mask them as MeshTerm writes
each line, turn on the **Anonymize** preference (`meshterm preferences set anonymize on`).
Sometimes MeshTerm meets a failure that it has no name for. Then the message is the error
+12 -1
View File
@@ -58,6 +58,7 @@ from .regions import WILDCARD as REGION_WILDCARD
from .regions import normalize as normalize_region
from .regions import parse_region_list, region_key
from .regions import validate as validate_region
from .secrets import remember as remember_secret
from .tracing import path_hash_flags, trace_timeout
from .transmit_lock import TransmitLock
@@ -2105,6 +2106,9 @@ class MeshCoreDevice(Device):
from .anonymize import remember_host
remember_host(host)
# The pairing PIN is a secret. No log may repeat it (refer to
# meshterm.core.secrets), so it is registered before the first connect.
remember_secret(pin)
self._mc = None # type: ignore[var-annotated] # meshcore.MeshCore
#: The ``bleak.BleakClient`` of a BLE link. MeshTerm keeps it because ``meshcore``
#: releases its own reference at the moment the peripheral disappears. Refer to
@@ -3197,7 +3201,9 @@ class MeshCoreDevice(Device):
result = await mc.commands.send_device_query()
except Exception: # noqa: BLE001 - older firmware has no query: unknown, not fatal
return {}
return dict(getattr(result, "payload", {}) or {})
payload = dict(getattr(result, "payload", {}) or {})
remember_secret(payload.get("ble_pin")) # the PIN with which a phone pairs
return payload
#: The maximum silence of the contacts stream before MeshTerm considers a read as
#: failed (seconds). The library's own ``get_contacts`` arms one future for the *full*
@@ -3467,6 +3473,9 @@ class MeshCoreDevice(Device):
"""
from meshcore import EventType
# Each password that MeshTerm sends is a secret that no log may repeat (refer to
# meshterm.core.secrets): an admin password, a room password, or a guess.
remember_secret(password)
mc = self._require()
pub = self._node_pubkey(node)
loop = asyncio.get_running_loop()
@@ -4384,6 +4393,7 @@ class MeshCoreDevice(Device):
raise DeviceCommandError(f"channel read for slot {index} returned slot {got}")
if not payload.get("channel_name"):
return None
remember_secret(payload.get("channel_secret"))
return payload
async def _channel_event(self, index: int): # noqa: ANN202 - a meshcore Event
@@ -4573,6 +4583,7 @@ class MeshCoreDevice(Device):
key = payload.get("private_key")
if key is None:
raise RuntimeError("device did not return a private key (export may be disabled)")
remember_secret(key)
return key.hex() if isinstance(key, (bytes, bytearray)) else str(key)
async def import_private_key(self, key_hex: str) -> None: # noqa: D102
+29 -6
View File
@@ -145,7 +145,7 @@ def _parse_typed(spec: SettingSpec, raw: Any, snapshot: dict | None) -> Any:
return True
if low in _FALSE:
return False
raise DeviceConfigError(f"{spec.key}: expected a boolean, got {raw!r}")
raise DeviceConfigError(f"{spec.key}: expected a boolean{_got(spec, repr(raw))}")
# Numeric (int / enum / float): the try wraps only the conversion. Thus this handler
# does not catch and wrap again a DeviceConfigError from the range and enum checks
@@ -158,7 +158,9 @@ def _parse_typed(spec: SettingSpec, raw: Any, snapshot: dict | None) -> Any:
else: # int or enum
value = int(text, 0) if isinstance(raw, str) else int(raw)
except (ValueError, TypeError) as exc:
raise DeviceConfigError(f"{spec.key}: invalid {spec.value_type} value {raw!r}") from exc
raise DeviceConfigError(
f"{spec.key}: invalid {spec.value_type} value{_got(spec, repr(raw), before=' ')}"
) from exc
if (
spec.value_type == "enum"
@@ -167,15 +169,35 @@ def _parse_typed(spec: SettingSpec, raw: Any, snapshot: dict | None) -> Any:
and value not in spec.choices
):
allowed = ", ".join(str(k) for k in spec.choices)
raise DeviceConfigError(f"{spec.key}: must be one of {allowed}, got {value}")
raise DeviceConfigError(f"{spec.key}: must be one of {allowed}{_got(spec, value)}")
if spec.minimum is not None and value < spec.minimum:
raise DeviceConfigError(f"{spec.key}: must be >= {spec.minimum}, got {value}")
raise DeviceConfigError(f"{spec.key}: must be >= {spec.minimum}{_got(spec, value)}")
maximum = effective_maximum(spec, snapshot)
if maximum is not None and value > maximum:
raise DeviceConfigError(f"{spec.key}: must be <= {maximum:g}, got {value}")
raise DeviceConfigError(f"{spec.key}: must be <= {maximum:g}{_got(spec, value)}")
return value
def _got(spec: SettingSpec, shown: Any, *, before: str = ", got ") -> str:
"""The end of an error message that repeats the refused value, or nothing for a secret.
An error goes to the log and to the run log, and the value of a secret setting (the
pairing PIN) must not (refer to :mod:`meshterm.core.secrets`). Thus the message of a
secret setting states the rule and not the value.
Args:
spec: The setting that refused the value.
shown: The refused value, as the message writes it.
before: The text before the value: ``", got "``, or ``" "`` for "value X".
Returns:
``", got 1234"``, ``" '12a'"``, or ``""`` for a secret setting.
"""
from .secrets import names_secret
return "" if names_secret(spec.key) else f"{before}{shown}"
def effective_maximum(spec: SettingSpec, snapshot: dict | None) -> float | None:
"""The inclusive maximum for ``spec``: the device value if it is known, else the static one.
@@ -430,7 +452,8 @@ def _pin_rule(value: Any, snapshot: dict | None) -> str | None:
"""The firmware accepts only a pairing PIN of zero (no PIN) or of exactly six digits."""
if value == 0 or 100000 <= value <= 999999:
return None
return f"must be 0 (no PIN) or six digits, got {value}"
# No "got": the PIN is a secret, and the message goes to the log (refer to _got).
return "must be 0 (no PIN) or six digits"
def _repeat_rule(value: Any, snapshot: dict | None) -> str | None:
+25 -3
View File
@@ -28,6 +28,8 @@ from typing import ClassVar
from .admin_store import admin_key
from .atomicwrite import write_atomically
from .models import Contact, utcnow
from .secrets import names_secret, without_secret
from .secrets import remember as remember_secret
#: How many command-line entries the store keeps for each node (newest last).
HISTORY_CAP = 100
@@ -121,7 +123,11 @@ class _RemoteNode:
if value is not None:
settings[str(key)] = value
raw_history = record.get("history")
history = [str(c) for c in raw_history] if isinstance(raw_history, list) else []
# A history that an earlier version wrote can hold a password. The record keeps
# the command without it, so the next write removes it from the file.
history = (
[without_secret(str(c)) for c in raw_history] if isinstance(raw_history, list) else []
)
return cls(settings=settings, history=history)
@property
@@ -169,11 +175,20 @@ class RemoteStore:
return dict(self._node(node).settings)
def remember_setting(self, node: Contact, key: str, value: str) -> None:
"""Cache the value of one setting for ``node``, with the time now."""
"""Cache the value of one setting for ``node``, with the time now.
The value of a setting that is a secret (``guest.password``, ``bridge.secret``) is
also registered, so that no log can repeat it (refer to
:mod:`meshterm.core.secrets`).
"""
if names_secret(key):
remember_secret(value)
self._put(node, key, CachedValue(value=value, read_at=utcnow()))
def remember_discovered(self, node: Contact, key: str, value: str) -> None:
"""Cache a setting that is not in the catalog, which ``node`` proved that it has."""
if names_secret(key):
remember_secret(value)
self._put(node, key, CachedValue(value=value, read_at=utcnow(), discovered=True))
def remember_unsupported(self, node: Contact, key: str) -> None:
@@ -209,10 +224,17 @@ class RemoteStore:
return list(self._node(node).history)
def append_history(self, node: Contact, command: str) -> None:
"""Add one sent command to the history of the node (not a repeat of the last command)."""
"""Add one sent command to the history of the node (not a repeat of the last command).
A command that sets a password or a secret is kept without its value
(:func:`~meshterm.core.secrets.without_secret`): ``password hunter2`` is kept as
``password``, with the space after it. Thus ↑ recalls the start of the command, and
the file never holds the secret.
"""
command = command.strip()
if not command:
return
command = without_secret(command)
records = self._load()
record = records.setdefault(admin_key(node), _RemoteNode())
if record.history and record.history[-1] == command:
+383
View File
@@ -0,0 +1,383 @@
# SPDX-License-Identifier: Apache-2.0
"""Keep each password and each secret out of each log: the log file and the run log.
MeshTerm keeps a record of what it does in two places: the log file (``meshterm.log``) and
the ``runs`` table, which stores the parameters and the summary of each run. Neither place
may ever hold a password, a channel secret, a pairing PIN, or a private key (JP,
2026-10-18: "NEVER log passwords or secrets!"). Each one is written as :data:`SECRET_TEXT`
instead. This rule does not depend on a preference, and the ``anonymize`` preference
(refer to :mod:`meshterm.core.anonymize`) adds to it and does not replace it.
A secret gets into a record in many shapes, and this module has one rule for each shape.
No rule depends on a list of the tools or of their parameters. Thus a tool that is added
later is covered on the day that it arrives:
- **A named value.** A value is a secret when its name says so (:func:`names_secret`):
a mapping key (``{"password": …}``), a dataclass field (``Conversation.secret``), the
``key`` of a record (``{"key": "device_pin", "value": …}``), the name in an operation
(``("set", "device_pin", "123456")``), or the name before ``=`` in a text
(``…&secret=8f…``, ``secret=b'…'``).
- **A wrapped value.** A tool can wrap a value in :class:`Secret` when nothing names it.
- **Bytes.** The parameters of a run carry bytes only for keys and secrets. Thus each
``bytes`` value is a secret.
- **A remote command.** ``password <new>`` and ``set guest.password <new>`` carry the
secret as their last word.
- **A known value in free text.** Each value that a rule above hid, and each secret that
MeshTerm reads (a stored password, a channel secret, a pairing PIN), is registered
(:func:`remember`). :func:`hide` then finds it in any text: an error message that
repeats it, or a frame that a library writes in hex.
:func:`loggable` applies the rules to a structure (the parameters and the summary of a
run). :func:`hide` applies them to a text (a log record).
"""
from __future__ import annotations
import re
import threading
from collections.abc import Mapping
from dataclasses import fields, is_dataclass
from datetime import date, datetime
from enum import Enum
from pathlib import PurePath
from typing import Any
#: What a log and the run log show in the place of a secret.
SECRET_TEXT = "<secret>"
#: The words that make a name the name of a secret. A name matches when one of its words
#: is in this set. The words of a name are the parts between the characters that are not
#: letters or digits. Thus ``device_pin``, ``guest.password``, and ``bridge.secret`` match,
#: and ``pinned`` and ``spin`` do not. ``prv`` is the private key of a repeater
#: (``prv.key``). ``key`` alone is not a secret word, because most keys in MeshTerm are
#: public keys.
SECRET_WORDS = frozenset(
{
"password",
"passwd",
"passphrase",
"secret",
"pin",
"private",
"privkey",
"prv",
"psk",
"token",
"credential",
"credentials",
}
)
#: The entries of a record whose ``key`` names a secret, and which hold its value.
_RECORD_VALUES = frozenset({"value", "previous", "default", "old", "new"})
#: The entries of a record that name the thing that the record is about.
_RECORD_NAMES = ("key", "setting")
#: The shortest text that :func:`remember` registers. A shorter value matches too much
#: ordinary text in a log (``abc``, ``123``). The structural rules still hide it where it
#: has a name.
_MIN_REMEMBERED = 4
#: The shortest hex value that :func:`hide` finds inside a longer run of hex. A channel
#: secret has 32 hex digits. With 16 or more, a match by chance is not probable.
_MIN_HEX_SUBSTRING = 16
_WORD_SPLIT = re.compile(r"[^a-z0-9]+")
_HEX = re.compile(r"[0-9a-fA-F]+")
#: ``name = value`` in a text: a URL query, a ``repr`` of an object, or a ``key=value``
#: log line. A name with no quotes takes ``=``. A quoted name (a ``repr`` of a dict, or
#: JSON) also takes ``:``. Prose ("device_pin: must be six digits") is not an assignment.
#: The value is a quoted literal (also a ``bytes`` literal), or a run of characters up to
#: the end of the value.
_ASSIGNMENT = re.compile(
r"(?P<name>(?<![\w.-])[A-Za-z_][\w.-]*)"
r"(?P<sep>\s*=\s*|['\"]\s*[=:]\s*)"
r"(?P<value>b?'(?:[^'\\]|\\.)*'|b?\"(?:[^\"\\]|\\.)*\"|<secret>|[^\s'\",;)}\]&<>]+)"
)
#: A remote command that sets a password: ``password <new>``.
_PASSWORD_COMMAND = re.compile(r"^(?P<head>\s*password\s+)(?P<value>\S.*)$", re.IGNORECASE)
#: A remote command that sets a setting: ``set <name> <value>``.
_SET_COMMAND = re.compile(r"^(?P<head>\s*set\s+(?P<name>\S+)\s+)(?P<value>\S.*)$", re.IGNORECASE)
_known: set[str] = set()
_known_pattern: re.Pattern[str] | None = None
_known_lock = threading.Lock()
class Secret:
"""A value that no log may keep, for example a private key or a password.
:meth:`~meshterm.tools.base.Tool.execute` stores the parameters of each run in the
``runs`` table, and logs them at debug level. A tool that has a secret that nothing
names (a key in a tuple) puts it in this wrapper. :func:`loggable` and each text form
of this object show :data:`SECRET_TEXT`, and the tool reads :attr:`value` (refer to
:func:`reveal`).
The class does not derive from ``str``, because ``json.dumps`` writes the value of a
``str`` subclass and never calls its ``default`` for it.
Attributes:
value: The secret itself.
"""
__slots__ = ("value",)
def __init__(self, value: str) -> None:
"""Keep ``value`` out of each text form of this object, and register it."""
self.value = value
remember(value)
def __repr__(self) -> str:
""":data:`SECRET_TEXT`, never the value."""
return SECRET_TEXT
__str__ = __repr__
def reveal(value: Any) -> Any:
"""The value inside a :class:`Secret`, or ``value`` itself if it is not a secret."""
return value.value if isinstance(value, Secret) else value
def names_secret(name: str) -> bool:
"""Whether ``name`` is the name of a secret (refer to :data:`SECRET_WORDS`).
Args:
name: A parameter name, a field name, a setting key, or a word of a text.
Returns:
``True`` when one of the words of ``name`` is a secret word.
"""
return not SECRET_WORDS.isdisjoint(_WORD_SPLIT.split(name.lower()))
def remember(value: Any) -> None:
"""Register a secret, so that :func:`hide` finds it in any text after this call.
A value of bytes registers as hex, the form in which a log writes bytes. A value that
is too short to find safely, or that is a secret word (a password of ``password``), is
not registered: each mention of the word in the log would become the mark. Each other
rule still hides such a value where it has a name.
Args:
value: The secret: text, a number (a PIN), or bytes.
"""
global _known_pattern
if value is None or isinstance(value, bool):
return
if isinstance(value, (bytes, bytearray, memoryview)):
text = bytes(value).hex()
else:
text = str(value).strip()
if len(text) < _MIN_REMEMBERED or text == SECRET_TEXT or text.lower() in SECRET_WORDS:
return
with _known_lock:
if text in _known:
return
_known.add(text)
_known_pattern = _compile_known(_known)
def hide(text: str) -> str:
"""Replace each secret in ``text`` with :data:`SECRET_TEXT`.
Args:
text: A log record, an error message, a remote command, or a URL.
Returns:
The text with each secret that a rule finds replaced.
"""
match = _PASSWORD_COMMAND.match(text)
if match is not None:
_remember_text(match.group("value"))
return match.group("head") + SECRET_TEXT
match = _SET_COMMAND.match(text)
if match is not None and names_secret(match.group("name")):
_remember_text(match.group("value"))
return match.group("head") + SECRET_TEXT
text = _ASSIGNMENT.sub(_hide_assignment, text)
known = _known_pattern
if known is not None:
text = known.sub(SECRET_TEXT, text)
return text
def without_secret(command: str) -> str:
"""A remote command with its secret value removed, for a history to recall.
The history of the remote command line keeps ``password `` for ``password hunter2``.
Thus ↑ recalls the start of the command, and the user types the new value again. If
the history kept :data:`SECRET_TEXT`, a recall and Enter would set the password to
that text.
Args:
command: A command as the user typed it.
Returns:
The command up to its secret value, or the command itself when it has none.
"""
for pattern in (_PASSWORD_COMMAND, _SET_COMMAND):
match = pattern.match(command)
if match is not None and (pattern is _PASSWORD_COMMAND or names_secret(match["name"])):
return match.group("head")
return command
def loggable(value: Any) -> Any:
"""A copy of ``value`` that a log can keep: each secret in it is :data:`SECRET_TEXT`.
The copy has the same shape: mappings stay mappings, and lists and tuples keep their
types. A dataclass becomes an object whose text form is the ``repr`` of the dataclass
with its secret fields hidden, so the log line looks as it did. Each text goes through
:func:`hide`. Each value that a rule hides is registered (:func:`remember`). Thus the
same value is also hidden when an error message repeats it later in the run.
Args:
value: The parameters or the summary of a run, or any part of them.
Returns:
The copy, ready for a log line or for ``json.dumps(..., default=str)``.
"""
if isinstance(value, Secret):
remember(value.value)
return SECRET_TEXT
if isinstance(value, (bytes, bytearray, memoryview)):
return _hidden(value)
if isinstance(value, str):
return hide(value)
if value is None or isinstance(value, (bool, int, float, Enum, PurePath, date, datetime)):
return value
if isinstance(value, Mapping):
return _loggable_mapping(value)
if is_dataclass(value) and not isinstance(value, type):
items = []
for f in fields(value):
item = getattr(value, f.name)
items.append((f.name, _hidden(item) if names_secret(f.name) else loggable(item)))
return _Shown(type(value).__name__, items)
if isinstance(value, (list, tuple, set, frozenset)):
items = _loggable_sequence(list(value))
return tuple(items) if isinstance(value, tuple) else items
return _Shown(hide(repr(value)))
class _Shown:
"""An object in a log, with its secrets hidden: its ``repr`` and its ``str`` are the same.
``json.dumps(..., default=str)`` writes an object as its ``str``, and a log line writes
a value in a dict as its ``repr``. Both forms of the original object can hold the
secret, so both forms of this object are the hidden form.
"""
__slots__ = ("_text",)
def __init__(self, name: str, items: list[tuple[str, Any]] | None = None) -> None:
"""Build the form ``Name(a=1, b=<secret>)``, or keep ``name`` as the whole text."""
if items is None:
self._text = name
else:
inner = ", ".join(f"{field}={_repr(value)}" for field, value in items)
self._text = f"{name}({inner})"
def __repr__(self) -> str:
"""The hidden form."""
return self._text
__str__ = __repr__
def _repr(value: Any) -> str:
"""The ``repr`` of a value in a hidden form, with :data:`SECRET_TEXT` not in quotes."""
return SECRET_TEXT if value == SECRET_TEXT else repr(value)
def _hidden(value: Any) -> Any:
""":data:`SECRET_TEXT` for a value that a name or a type makes secret, after it is registered.
An absent value (``None``, or an empty text) stays as it is. That a password was not
given is a fact that a maintainer can use, and it hides nothing.
"""
if value is None or value == "" or value == b"":
return value
remember(reveal(value))
return SECRET_TEXT
def _loggable_mapping(value: Mapping) -> dict:
""":func:`loggable` for a mapping: hide each value that its key or its record names secret."""
record_names = [value.get(name) for name in _RECORD_NAMES]
secret_record = any(isinstance(name, str) and names_secret(name) for name in record_names)
shown: dict = {}
for key, item in value.items():
named = names_secret(str(key)) or (secret_record and key in _RECORD_VALUES)
shown[key] = _hidden(item) if named else loggable(item)
return shown
def _loggable_sequence(items: list) -> list:
""":func:`loggable` for a sequence, and for an operation that sets a secret.
An operation that sets a value names the setting before the value:
``("set", "device_pin", "123456")``. Thus, in a sequence of three items that starts
with ``set`` and whose second item names a secret, the third item is the secret. No
other sequence has a value after a name. A list of setting names
(``["name", "device_pin", "radio_freq"]``) stays as it is.
"""
shown = [loggable(item) for item in items]
verb, name = (items[0], items[1]) if len(items) == 3 else (None, None)
if isinstance(verb, str) and verb.lower() == "set" and isinstance(name, str):
if names_secret(name):
shown[2] = _hidden(items[2])
return shown
def _hide_assignment(match: re.Match[str]) -> str:
"""Hide the value of ``name = value`` when the name is the name of a secret."""
value = match.group("value")
if not names_secret(match.group("name")) or value in (SECRET_TEXT, "None", "null"):
return match.group(0)
if value in ("''", '""', "b''", 'b""'):
return match.group(0)
_remember_text(value)
return match.group("name") + match.group("sep") + SECRET_TEXT
def _remember_text(value: str) -> None:
"""Register a value that a text rule found, without the quotes of a literal."""
text = value.strip()
if len(text) >= 2 and text[-1] in "'\"" and text.lstrip("b")[:1] == text[-1]:
text = text.lstrip("b")[1:-1]
remember(text)
def _compile_known(known: set[str]) -> re.Pattern[str]:
"""The pattern that finds each registered secret.
A long hex value matches also inside a longer run of hex, in any letter case, because a
library writes a whole frame as one run of hex. Each other value matches only as a
whole word, so a PIN of ``123456`` does not hide a part of ``71234567``.
"""
parts: list[str] = []
for text in sorted(known, key=len, reverse=True):
if len(text) >= _MIN_HEX_SUBSTRING and _HEX.fullmatch(text):
parts.append(f"(?i:{re.escape(text)})")
else:
parts.append(rf"(?<![\w]){re.escape(text)}(?![\w])")
return re.compile("|".join(parts))
__all__ = [
"SECRET_TEXT",
"SECRET_WORDS",
"Secret",
"hide",
"loggable",
"names_secret",
"remember",
"reveal",
"without_secret",
]
+32 -1
View File
@@ -12,7 +12,7 @@ import json
import sqlite3
from pathlib import Path
SCHEMA_VERSION = 19
SCHEMA_VERSION = 20
_SCHEMA = """
CREATE TABLE IF NOT EXISTS schema_meta (
@@ -404,6 +404,37 @@ def _migrate(conn: sqlite3.Connection) -> None:
# that the run log writes as "<secret>" (refer to meshterm.tools.base.Secret).
# This step removes the keys that the earlier versions stored, one time.
_forget_imported_keys(conn)
if stored is not None and int(stored["value"]) < 20:
# v19 -> v20: no secret of any kind in the run log. The parameters and summaries
# of the runs held channel secrets (in each chat run, inside its Conversation),
# room passwords, admin passwords, and pairing PINs. The run log now stores each
# one as "<secret>" (refer to meshterm.core.secrets). This step hides the secrets
# that the earlier versions stored, one time.
_hide_stored_secrets(conn)
def _hide_stored_secrets(conn: sqlite3.Connection) -> None:
"""Hide each secret in the stored arguments and summaries of the runs.
The rules are the rules that the run log applies to a new run
(:func:`~meshterm.core.secrets.loggable`). The stored JSON has lost the types of the
objects: a ``Conversation`` is the text of its ``repr``. The text rules find the
secret in that text (``secret=b'...'``). A row that does not change is not written.
"""
from ..core.secrets import loggable
rows = conn.execute("SELECT id, args_json, summary_json FROM runs").fetchall()
for row in rows:
for column in ("args_json", "summary_json"):
stored = row[column]
if not stored:
continue
try:
hidden = json.dumps(loggable(json.loads(stored)), default=str)
except ValueError:
continue
if hidden != json.dumps(json.loads(stored)):
conn.execute(f"UPDATE runs SET {column} = ? WHERE id = ?", (hidden, row["id"]))
def _forget_imported_keys(conn: sqlite3.Connection) -> None:
+59 -11
View File
@@ -19,7 +19,11 @@ A preference sets how much MeshTerm writes: ``log_level``, with the default ``WA
Thus the file has the problems, not a full account of a session that works. When you
look for the cause of a problem, set it lower, to ``INFO`` or ``DEBUG``.
A second preference, ``anonymize``, masks the personal details of each line before it
No log ever holds a password or a secret. Both handlers hide each secret that
:mod:`meshterm.core.secrets` finds in a record, and the file formatter does it again on
the traceback. This does not depend on a preference.
A preference, ``anonymize``, masks the personal details of each line before it
goes into the file: the account name in a home folder, the address of a radio, and the
address of a host (refer to :mod:`meshterm.core.anonymize`). The mask applies as each line
is written, so the file never holds what it masks. A line that was written before the
@@ -142,6 +146,7 @@ def configure_logging(
console=console, rich_tracebacks=True, show_path=False, markup=False
)
rich_handler.setLevel(level)
rich_handler.addFilter(_HIDE_SECRETS)
logger.addHandler(rich_handler)
log_dir.mkdir(parents=True, exist_ok=True)
@@ -152,6 +157,7 @@ def configure_logging(
encoding="utf-8",
)
file_handler.setLevel(file_level)
file_handler.addFilter(_HIDE_SECRETS)
file_handler.setFormatter(_FileFormatter(_LINE_FORMAT, datefmt=_TIME_FORMAT))
logger.addHandler(file_handler)
@@ -160,23 +166,65 @@ def configure_logging(
return logger
class _FileFormatter(logging.Formatter):
"""The line format of the file, with the personal details masked when the user asks.
class _HideSecrets(logging.Filter):
"""Hide each password and secret in the message of a record, for every handler.
The formatter masks the whole formatted text: the message, and the traceback under
it. A traceback holds the most home folders, because each frame names its file. The
formatter reads the ``anonymize`` preference at each record, so a change on the
Preferences page applies to the next line. The mask is a function of the text only,
so it does not change the record. The console handler gets the same record and shows
it as it is, because the console belongs to the user who reads it.
A filter on a handler sees each record that the handler gets, also the records of a
child logger and of the libraries in :data:`_LIBRARY_LOGGERS`. A filter on a logger
sees only the records of that logger. The filter writes the hidden message back into
the record, so each handler after it gets the hidden form. A second pass finds nothing
more, so the order of the handlers does not matter.
The text of an exception can repeat a secret too. The console handler draws a
traceback from the exception itself, and not from text that a filter can change. Thus,
when the traceback of a record holds a secret, the filter adds the hidden traceback to
the message as text and removes the exception from the record. Only such a record
loses the drawn traceback of the console.
"""
def filter(self, record: logging.LogRecord) -> bool:
"""Replace the message of ``record`` with its hidden form, and keep the record."""
from ..core.secrets import hide
try:
message = record.getMessage()
except Exception: # noqa: BLE001 - a bad format is the problem of the handler
return True
hidden = hide(message)
if record.exc_info and record.exc_info[0] is not None:
traceback = logging.Formatter().formatException(record.exc_info)
hidden_traceback = hide(traceback)
if hidden_traceback != traceback:
hidden = f"{hidden}\n{hidden_traceback}"
record.exc_info, record.exc_text = None, None
if hidden != message or record.args:
record.msg, record.args = hidden, None
return True
_HIDE_SECRETS = _HideSecrets()
class _FileFormatter(logging.Formatter):
"""The line format of the file, with no secret, and anonymized when the user asks.
The formatter works on the whole formatted text: the message, and the traceback under
it. :class:`_HideSecrets` already hid the message. The text of an exception in a
traceback can repeat a secret too, so the formatter hides secrets again here, always.
Then it masks the personal details if the ``anonymize`` preference is on. A traceback
holds the most home folders, because each frame names its file. The formatter reads
the preference at each record, so a change on the Preferences page applies to the
next line. The mask does not change the record. The console handler gets the same
record and shows it unmasked, because the console belongs to the user who reads it.
"""
def format(self, record: logging.LogRecord) -> str:
"""Format ``record`` as one line (and its traceback), masked if ``anonymize`` is on."""
"""Format ``record`` as one line (and its traceback), with no secret in it."""
from ..core.anonymize import anonymize
from ..core.preferences import current
from ..core.secrets import hide
text = super().format(record)
text = hide(super().format(record))
return anonymize(text) if current().anonymize else text
+11 -2
View File
@@ -30,6 +30,7 @@ from ..core.models import (
utcnow,
)
from ..core.regions import SCOPED_ROUTES, raw_scope_body
from ..core.secrets import loggable
from . import db
if TYPE_CHECKING:
@@ -371,6 +372,10 @@ class Repository:
) -> int:
"""Store the start of a tool run.
The run log never keeps a secret. The arguments go through
:func:`~meshterm.core.secrets.loggable`, so a password, a channel secret, or a PIN
is stored as ``<secret>``.
Args:
tool: The name of the tool.
args: The arguments with which the tool was called (they must be
@@ -382,10 +387,11 @@ class Repository:
Returns:
The primary key of the new run.
"""
stored = json.dumps(loggable(args), default=str)
cur = self._conn.execute(
"INSERT INTO runs (tool, profile, args_json, status, started_at) "
"VALUES (?, ?, ?, 'running', ?)",
(tool, profile, json.dumps(args, default=str), (at or utcnow()).isoformat()),
(tool, profile, stored, (at or utcnow()).isoformat()),
)
self._conn.commit()
return int(cur.lastrowid)
@@ -400,6 +406,9 @@ class Repository:
) -> None:
"""Mark a run as finished.
The summary goes through :func:`~meshterm.core.secrets.loggable`, as the arguments
do in :meth:`start_run`. An error message that repeats a secret is hidden too.
Args:
run_id: The run to update.
status: The final status (``ok`` or ``error``).
@@ -410,7 +419,7 @@ class Repository:
"UPDATE runs SET status = ?, summary_json = ?, finished_at = ? WHERE id = ?",
(
status,
json.dumps(summary, default=str) if summary is not None else None,
json.dumps(loggable(summary), default=str) if summary is not None else None,
(at or utcnow()).isoformat(),
run_id,
),
+10 -35
View File
@@ -16,6 +16,13 @@ from typing import TYPE_CHECKING, Any
from ..core import exitcodes
# The wrapper for a secret parameter, and its reader. They live with the other rules that
# keep secrets out of the logs (refer to meshterm.core.secrets). The tools import them
# from here, next to Tool and ToolResult.
from ..core.secrets import Secret as Secret
from ..core.secrets import loggable
from ..core.secrets import reveal as reveal
if TYPE_CHECKING: # do not import typer or the context at module load, for a fast start
import typer
@@ -23,40 +30,6 @@ if TYPE_CHECKING: # do not import typer or the context at module load, for a fa
from ..ui.report import Report
class Secret:
"""A parameter of a run that the run log must not keep, for example a private key.
:meth:`Tool.execute` stores the parameters of each run in the ``runs`` table, and logs
them at debug level. A secret in the parameters was thus stored as plain text: the key
of ``config import-key`` stayed in the database. A secret goes into the parameters in
this wrapper. The JSON of the run log and the log line show ``<secret>``, and the tool
reads :attr:`value`.
The class does not derive from ``str``, because ``json.dumps`` writes the value of a
``str`` subclass and never calls its ``default`` for it.
Attributes:
value: The secret itself.
"""
__slots__ = ("value",)
def __init__(self, value: str) -> None:
"""Keep ``value`` out of each text form of this object."""
self.value = value
def __repr__(self) -> str:
"""``<secret>``, never the value."""
return "<secret>"
__str__ = __repr__
def reveal(value: Any) -> Any:
"""The value inside a :class:`Secret`, or ``value`` itself if it is not a secret."""
return value.value if isinstance(value, Secret) else value
@dataclass(slots=True)
class ToolResult:
"""The result of a run of a tool.
@@ -202,7 +175,9 @@ class Tool(ABC):
from ..core.selection import DeviceSelectionError
run_id = ctx.repo.start_run(self.name, params, ctx.profile_name)
ctx.log.debug("run %s start: tool=%s params=%s", run_id, self.name, params)
# The copy with each secret hidden: a password, a channel secret, or a PIN in the
# parameters must never reach the log (refer to meshterm.core.secrets).
ctx.log.debug("run %s start: tool=%s params=%s", run_id, self.name, loggable(params))
try:
result = await self.run(ctx, {**params, "_run_id": run_id})
except (
+17
View File
@@ -191,6 +191,23 @@ def _isolate_config_dir(tmp_path_factory: pytest.TempPathFactory) -> Iterator[No
os.environ["MESHTERM_HOME"] = previous
@pytest.fixture(autouse=True)
def _empty_registries(monkeypatch: pytest.MonkeyPatch) -> None:
"""Each test starts with no registered secret and no registered host.
The two registries are process-wide on purpose: a secret that MeshTerm read once must
stay hidden for the rest of the run. In a suite, that means that a password one test
used ("hello", the default room password) hides the same word in each later test that
looks for it. ``monkeypatch`` puts the shared objects back after the test.
"""
from meshterm.core import anonymize, secrets
monkeypatch.setattr(secrets, "_known", set())
monkeypatch.setattr(secrets, "_known_pattern", None)
monkeypatch.setattr(anonymize, "_hosts", {})
monkeypatch.setattr(anonymize, "_host_pattern", None)
@pytest.fixture(autouse=True)
def _reset_platform() -> Iterator[None]:
"""Each test starts and ends on :data:`~meshterm.platforms.REGULAR`.
+306
View File
@@ -0,0 +1,306 @@
# SPDX-License-Identifier: Apache-2.0
"""No log ever holds a password or a secret: the log file, the run log, or the remote history.
JP, 2026-10-18: "NEVER log passwords or secrets!" The rule is a property of the places that
keep records, and not a habit of each tool. Thus the most important test here does not
look at the rules one by one. It runs each command that takes a secret, with a planted
value, through the real CLI at the DEBUG log level, and then searches the log file and the
``runs`` table for each planted value. A new command that takes a secret is covered when
it is added to that list. The other tests hold each rule and each shape on its own, so
that a failure says which rule broke.
"""
from __future__ import annotations
import io
import json
import logging
import sqlite3
from pathlib import Path
import pytest
from rich.console import Console
from typer.testing import CliRunner
from meshterm.core.device_config import DeviceConfigError, get_spec, parse_value
from meshterm.core.models import Contact, Conversation
from meshterm.core.remote_store import RemoteStore
from meshterm.core.secrets import (
SECRET_TEXT,
Secret,
hide,
loggable,
names_secret,
remember,
without_secret,
)
from meshterm.persistence import db
from meshterm.persistence.logging import configure_logging, get_logger, log_path
from meshterm.persistence.repository import Repository
#: Planted values. Each one is distinct, so a failure names the path that leaked it.
_PASSWORD = "plantedpw-7c1e"
_HEX = "5ec2e7a1" * 4
# --- names ----------------------------------------------------------------------------
@pytest.mark.parametrize(
"name", ["password", "device_pin", "guest.password", "bridge.secret", "prv.key", "ble_pin"]
)
def test_a_name_with_a_secret_word_names_a_secret(name: str) -> None:
"""A secret word anywhere in the name, between any separators, makes the name secret."""
assert names_secret(name)
@pytest.mark.parametrize("name", ["pinned", "spin", "public_key", "key_prefix", "name", "to"])
def test_a_name_that_only_contains_the_letters_is_not_secret(name: str) -> None:
"""A word must be whole: ``pinned`` is not ``pin``, and a public key is not a secret."""
assert not names_secret(name)
# --- structures -----------------------------------------------------------------------
def test_each_shape_of_a_secret_in_the_parameters_is_hidden() -> None:
"""A key, a record, an operation, a dataclass field, a wrapper, and bytes: each is hidden."""
params = {
"room": "Lakeside",
"password": _PASSWORD,
"ops": [("set", "device_pin", 246813), ("import_key", Secret(_HEX))],
"changes": [{"key": "device_pin", "previous": 135792, "value": 246813}],
"set_channel": (5, "#lakeside", bytes.fromhex(_HEX)),
"conversation": Conversation(
label="#lakeside", is_channel=True, secret=bytes.fromhex(_HEX)
),
}
shown = json.dumps(loggable(params), default=str)
for value in (_PASSWORD, _HEX, "246813", "135792", "xc2"):
assert value not in shown
assert "Lakeside" in shown # what is not a secret stays
assert "Conversation(label='#lakeside'" in shown
assert "secret=<secret>" in shown
def test_a_list_of_setting_names_is_not_an_operation() -> None:
"""Only ``(verb, name, value)`` hides its value. A list of names has no value in it."""
names = ["name", "device_pin", "radio_freq"]
assert loggable({"settings": names}) == {"settings": names}
assert loggable(("get", "device_pin")) == ("get", "device_pin")
def test_an_absent_secret_stays_absent() -> None:
"""A password that was not given is a fact that hides nothing. It stays ``None``."""
assert loggable({"password": None, "secret": ""}) == {"password": None, "secret": ""}
assert "secret=None" in str(loggable(Conversation(label="Bob", is_channel=False)))
# --- text -----------------------------------------------------------------------------
@pytest.mark.parametrize(
"text,expected",
[
("password hunter2-new", f"password {SECRET_TEXT}"),
("set guest.password hello there", f"set guest.password {SECRET_TEXT}"),
("set radio 869.5,250,11,5", "set radio 869.5,250,11,5"),
(
f"meshcore://channel/add?name=Lakeside&secret={_HEX}",
f"meshcore://channel/add?name=Lakeside&secret={SECRET_TEXT}",
),
(
"Conversation(secret=b'\\x8b3\\x87', contact=None)",
"Conversation(secret=<secret>, contact=None)",
),
("{'password': 'abcd1234'}", "{'password': <secret>}"),
('{"pin": "998877"}', '{"pin": <secret>}'),
("Conversation(secret=None)", "Conversation(secret=None)"),
("device_pin: must be 0 (no PIN) or six digits", None),
("BLE ProvidePin pairing for DD:F3:76:8B:0B:0B: status=19 ok=False", None),
],
)
def test_a_text_hides_each_secret_that_its_form_shows(text: str, expected: str | None) -> None:
"""A remote command, a URL, a ``repr``, and JSON all show which value is the secret."""
assert hide(text) == (text if expected is None else expected)
def test_a_known_secret_is_hidden_in_any_text() -> None:
"""A registered secret is hidden where nothing names it: in an error, or inside a hex frame."""
remember(_PASSWORD)
remember(bytes.fromhex(_HEX))
assert hide(f"login refused: {_PASSWORD} is wrong") == f"login refused: {SECRET_TEXT} is wrong"
frame = "Unknown stats type: 66, data: 1842" + _HEX.upper() + "00ff"
assert _HEX.upper() not in hide(frame)
def test_a_short_or_common_value_is_not_registered() -> None:
"""A value of three characters, or a secret word, would hide ordinary text everywhere."""
remember("abc")
remember("password")
assert hide("abc password") == "abc password"
def test_a_registered_pin_does_not_hide_part_of_a_longer_number() -> None:
"""A PIN is found as a whole word only, so it cannot cut a timestamp or a count."""
remember(246813)
assert hide("pin 246813 and 12468139") == f"pin {SECRET_TEXT} and 12468139"
# --- the remote command line ----------------------------------------------------------
def test_the_remote_history_keeps_the_command_without_its_secret(tmp_path: Path) -> None:
"""↑ recalls ``password ``, ready to type, and the file never holds the password."""
store = RemoteStore(tmp_path / "remote.json")
node = Contact(name="Hilltop", public_key="3d" * 32)
store.append_history(node, f"password {_PASSWORD}")
store.append_history(node, f"set guest.password {_PASSWORD}")
store.append_history(node, "set name Hilltop")
assert store.history(node) == ["password ", "set guest.password ", "set name Hilltop"]
assert _PASSWORD not in (tmp_path / "remote.json").read_text(encoding="utf-8")
assert without_secret("ver") == "ver"
# --- the device settings --------------------------------------------------------------
def test_a_refused_pin_is_not_repeated_in_the_error() -> None:
"""The error of a secret setting states the rule. Each other setting still shows the value."""
with pytest.raises(DeviceConfigError) as refused:
parse_value(get_spec("device_pin"), "24681x")
assert "24681x" not in str(refused.value)
with pytest.raises(DeviceConfigError) as short:
parse_value(get_spec("device_pin"), "24681")
assert "24681" not in str(short.value)
with pytest.raises(DeviceConfigError) as other:
parse_value(get_spec("radio_sf"), "13")
assert "13" in str(other.value)
# --- the places that keep records -----------------------------------------------------
def test_neither_log_handler_writes_a_secret(tmp_path: Path) -> None:
"""The file, its traceback, and the console all get the hidden form of a record."""
console = Console(file=io.StringIO(), width=200, record=True, color_system=None)
configure_logging(console, tmp_path, file_level=logging.DEBUG, level=logging.DEBUG)
log = get_logger("core.connection")
log.debug("run 1 start: params=%s", {"password": _PASSWORD})
try:
raise ValueError(f"login with {_PASSWORD} refused")
except ValueError:
log.exception("login failed")
for handler in get_logger().handlers:
handler.flush()
written = log_path(tmp_path).read_text(encoding="utf-8")
assert _PASSWORD not in written
assert SECRET_TEXT in written
assert _PASSWORD not in console.export_text()
def test_the_run_log_stores_no_secret(tmp_path: Path) -> None:
"""The arguments and the summary of a run go into the ``runs`` table hidden."""
repo = Repository(tmp_path / "test.db")
run = repo.start_run("rooms", {"cli_action": "join", "password": _PASSWORD})
repo.finish_run(run, "error", {"error": f"refused {_PASSWORD}"})
repo.close()
conn = sqlite3.connect(tmp_path / "test.db")
args, summary = conn.execute("SELECT args_json, summary_json FROM runs").fetchone()
conn.close()
assert _PASSWORD not in args + summary
assert SECRET_TEXT in args
def test_the_migration_hides_the_secrets_that_older_versions_stored(tmp_path: Path) -> None:
"""A database at version 19 loses each stored secret, one time, when it opens."""
path = tmp_path / "old.db"
db.connect(path).close()
conn = sqlite3.connect(path)
conversation = "Conversation(label='#x', is_channel=True, secret=b'\\x5e\\xc2', contact=None)"
leaked = json.dumps({"conversation": conversation})
conn.execute(
"INSERT INTO runs (tool, args_json, summary_json, status, started_at) "
"VALUES ('chat', ?, ?, 'ok', '2026-10-01T00:00:00')",
(leaked, json.dumps({"password": _PASSWORD})),
)
conn.execute("UPDATE schema_meta SET value = '19' WHERE key = 'version'")
conn.commit()
conn.close()
db.connect(path).close()
conn = sqlite3.connect(path)
args, summary = conn.execute("SELECT args_json, summary_json FROM runs").fetchone()
conn.close()
assert "\\\\x5e" not in args and "secret=<secret>" in args
assert _PASSWORD not in summary
# --- the whole program ----------------------------------------------------------------
#: Each command that takes a secret, with a planted value in it, and the planted values.
#: The commands run against the simulator. Some of them fail (a room that the simulator
#: does not have), and that is fine: a failure is logged too, and with the parameters.
_COMMANDS: list[tuple[list[str], tuple[str, ...]]] = [
(["repeater-admin", "Yagi-Repeater", "ver", "--password", "plantedpw-a1"], ("plantedpw-a1",)),
(["repeater-admin", "Yagi-Repeater", "password", "plantedpw-a2"], ("plantedpw-a2",)),
(
["repeater-admin", "Yagi-Repeater", "set", "guest.password", "plantedpw-a3"],
("plantedpw-a3",),
),
(["rooms", "join", "Lakeside BBS", "--password", "plantedpw-r1"], ("plantedpw-r1",)),
(
["tx-optimize", "--path", "a1,b2", "--min", "20", "--max", "20", "--samples", "1"]
+ ["--no-apply", "--password", "plantedpw-t1"],
("plantedpw-t1",),
),
(["channels", "add", "5", "Planted", "--secret", "a1" * 16], ("a1" * 16,)),
(["channels", "join", "6", "Planted2", "b2" * 16], ("b2" * 16,)),
(
["channels", "import", "7", "meshcore://channel/add?name=P3&secret=" + "c3" * 16],
("c3" * 16,),
),
(["config", "channel", "5", "Planted4", "--secret", "d4" * 16], ("d4" * 16,)),
(["config", "set", "device_pin", "246813"], ("246813",)),
(["config", "set", "device_pin", "97531"], ("97531",)),
]
def test_no_command_writes_a_planted_secret_to_any_log(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Each command with a secret, at the DEBUG level: no planted value in the log or the runs.
This is the test that holds the promise. The rules above can each be correct and a
path between them can still leak, as the parameters of a chat run did. Thus this test
asks the two places that keep records, and not the rules.
"""
home = tmp_path / "home"
monkeypatch.setenv("MESHTERM_HOME", str(home))
runner = CliRunner()
assert runner.invoke(app_(), ["preferences", "set", "log_level", "DEBUG"]).exit_code == 0
planted: list[str] = []
for args, values in _COMMANDS:
runner.invoke(app_(), ["--mock", *args])
planted.extend(values)
for handler in get_logger().handlers:
handler.flush()
log = log_path(home).read_text(encoding="utf-8")
assert log.count("start: tool=") >= len(_COMMANDS), "the commands did not run"
conn = sqlite3.connect(home / "meshterm-mock.db")
runs = "\n".join(
f"{args} {summary}"
for args, summary in conn.execute("SELECT args_json, summary_json FROM runs")
)
conn.close()
for value in planted:
assert value not in log, f"{value} is in the log"
assert value not in runs, f"{value} is in the runs table"
def app_():
"""The CLI application, imported late so that the environment of the test applies."""
from meshterm.cli import app
return app