A read-logs skill turns the log into incidents before anyone reasons about it
The log misleads read raw: a count hides its span (204 identical records that
were three seconds), a failed CLI run is logged twice, the suite's runs from
before 59f10f9 sit in the real file, and the runs table keeps UTC while the
log keeps local time. logscan.py groups tracebacks, warnings and the DEBUG
failures that lost their stack, each with first and last; folds the CLI's
reports onto the run they repeat; hides fixture and other-database runs and
says how many; and joins the runs table by run id. SKILL.md is the method:
pin the question, fetch the log (this machine, the PicoCalc, an attachment),
summary then around/run/show, the reading traps, and checking HEAD before
calling a finding live.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
10f48dbffc
commit
7afd231bc7
2 files changed
+800
No files matched your search
@@ -0,0 +1,168 @@
|
||||
---
|
||||
name: read-logs
|
||||
description: Investigate a MeshTerm problem from its log — find what went wrong, when, and where in the code, and say whether it is still live on HEAD. Use when JP says to check, read or look at the logs, asks why something crashed, froze, disconnected or did something odd, pastes an error or a log excerpt, or reports a bug from the PicoCalc or another machine.
|
||||
---
|
||||
|
||||
# Reading the logs to find an issue
|
||||
|
||||
The log is evidence, not a diagnosis. The job runs **symptom → the records that show it →
|
||||
the code that wrote them → whether HEAD still does it**. Stop at the diagnosis and propose
|
||||
the fix; make it only when JP asks for it.
|
||||
|
||||
## What there is to read
|
||||
|
||||
| Source | Where | Clock |
|
||||
|---|---|---|
|
||||
| `meshterm.log`, rotated at 2 MB into `.1`–`.3` | `$MESHTERM_HOME`, else `~/.meshterm/` | local, of the machine that wrote it |
|
||||
| `meshterm.log.jsonl` — the log before 2026-09-07 15:00, JSON Lines | same directory; nothing reads it but `--legacy` | local |
|
||||
| the `runs` table — a row per tool run: `status` ok / error / running, `summary_json.error` | `meshterm.db`, same directory | **UTC** |
|
||||
|
||||
The **run id** joins the log to the table, never the timestamp.
|
||||
|
||||
How much the file holds is the `log_level` preference (**Log detail** on the Preferences
|
||||
page), default **WARNING**. JP's machine runs at DEBUG; a bug report from anyone else
|
||||
probably doesn't. At WARNING there are no `run N start`/`ok` records and no quiet failures,
|
||||
so a missing record proves nothing. The summary says so when it sees no DEBUG/INFO at all.
|
||||
When the question needs the narration, ask for Log detail at DEBUG (it relaunches) and a
|
||||
reproduction.
|
||||
|
||||
## 1. Pin the question
|
||||
|
||||
Before opening anything, settle four things:
|
||||
|
||||
- what went wrong: the screen or command, and the words on it
|
||||
- roughly when, in local time: "just now" is `--since 30m`
|
||||
- which machine: this one, the PicoCalc, or an attached file
|
||||
- menu or CLI
|
||||
|
||||
A window turns five thousand records into forty.
|
||||
|
||||
## 2. Get the log
|
||||
|
||||
- **This machine:** nothing to fetch.
|
||||
- **The PicoCalc:** read `.dev.env` for the `$DEV_PICOCALC_*` values, then copy the set into
|
||||
the scratchpad and point `--home` at it:
|
||||
```
|
||||
scp -i $DEV_PICOCALC_SSH_KEY "$DEV_PICOCALC_USER@$DEV_PICOCALC_HOST:.meshterm/meshterm.log*" <scratchpad>/picocalc/
|
||||
```
|
||||
Bring `meshterm.db` too only when the runs table matters; it is tens of MB. Those records
|
||||
are in the device's own local time.
|
||||
- **A pasted or attached log:** save it to the scratchpad and pass `--log <file>`.
|
||||
Repeat `--log` for a rotated set, oldest first.
|
||||
|
||||
## 3. Summary first
|
||||
|
||||
```
|
||||
python .claude/skills/read-logs/logscan.py summary --since "2026-09-11 03:00"
|
||||
```
|
||||
|
||||
Stdlib only, so any Python runs it, the PicoCalc's included. It prints, in order:
|
||||
|
||||
- **header**: files, span, window, records kept and how many test records it hid, the
|
||||
level mix
|
||||
- **Activity**: blocks of records split by silences (`--gap`, default 20 min), each with
|
||||
its warning and error count, which is where "when did it go wrong" usually shows first
|
||||
- **Tracebacks**, grouped by the exception they ended on, with first → last and the
|
||||
record they hang under; the CLI's second copy of a failed run is folded in
|
||||
- **Warnings and errors without a traceback**, grouped with first → last
|
||||
- **Quiet failures**: DEBUG/INFO records reporting a failure, which a WARNING-and-up
|
||||
read never sees
|
||||
- **Runs table**: status counts, the latest errors, and runs never finished
|
||||
|
||||
Time forms everywhere: `30m`, `2h`, `3d`, `03:54` (today), `2026-09-11 03:54[:10]`.
|
||||
|
||||
## 4. Then narrow
|
||||
|
||||
```
|
||||
logscan.py around "2026-09-11 03:54:47" --minutes 2 # every level, both sides of a moment
|
||||
logscan.py run 3379 # its runs row, its records, what ran beside it
|
||||
logscan.py show --level WARNING --grep "BLE|link" # verbatim records, filtered
|
||||
```
|
||||
|
||||
`--grep` searches message and traceback, case-insensitive. `--fixtures` brings the hidden
|
||||
test records back. `--no-db` skips the table.
|
||||
|
||||
## Reading it right
|
||||
|
||||
- **The logger name rarely says who spoke.** Nearly everything logs through `ctx.log`, the
|
||||
bare `meshterm` logger, so the component is the message's own prefix: `devstate:`,
|
||||
`courier:`, `event hub:`, `chat:`, `run N`. Only `meshterm.core.connection`,
|
||||
`meshterm.core.discovery`, `meshcore` (the library) and `asyncio` are loggers of their
|
||||
own.
|
||||
- **A count means nothing without its span.** 204× `event hub: subscriber raised: [Errno
|
||||
22]` reads as chronic, and it was three seconds on 09-08. Read first → last before
|
||||
calling anything recurring.
|
||||
- **A run's life** is `run N start: tool=… params=…` followed by exactly one ending
|
||||
(`tools/base.py`):
|
||||
- `run N ok`, at DEBUG.
|
||||
- `run N aborted: …`, at DEBUG. An *expected* refusal: a bad argument, no device, a
|
||||
device error. It has no traceback, and the CLI also logs the one line it printed as
|
||||
`<tool> failed: …` at WARNING.
|
||||
- `run N lost the device connection: …`, at WARNING.
|
||||
- `run N failed: …`, at ERROR with a traceback. The CLI logs the same exception again as
|
||||
`<tool> raised an unhandled error`.
|
||||
|
||||
`the interactive session raised an unhandled error` is the TUI itself dying.
|
||||
- **A run left `running` in the table** never reached an ending. The process was killed,
|
||||
the window closed, it died without a traceback, or it is still running now. A background
|
||||
`chat`/`monitor` row (`{"mode": "background"}`) left open means that *session* didn't
|
||||
tear down. A foreground tool left open died mid-command. `run N` on it shows what
|
||||
followed its start.
|
||||
- **Quiet failures have no stack.** Background loops log `"…: %s", exc` at DEBUG
|
||||
(`devstate: … failed`, `battery poller: read failed`, `courier: pass failed`, `event hub:
|
||||
subscriber raised`). The message survives and where it was raised does not:
|
||||
`[Errno 22] Invalid argument` names no file. Grep the literal prefix to find the call
|
||||
site and read what its `try` covers. Or reproduce with a temporary `exc_info=True` on
|
||||
that call, and never commit it.
|
||||
- **A connection has a lifecycle, a session doesn't.** No record marks the app starting.
|
||||
A link opens as `meshcore: BLE Connection started`, then `event hub started`, then the
|
||||
services (`passive monitor recording`, `advert scheduler started`, `courier started`, …).
|
||||
It closes with `event hub stopped`. A stop/start pair seconds apart is a reconnect, so
|
||||
read the record just before it (`BLE write failed`, `Client is not connected`).
|
||||
- **Radio refusals are usually a symptom.** A storm of `ERR_CODE_BAD_STATE` from contact
|
||||
refreshes and battery polls while an admin login or another exchange is in flight means
|
||||
the companion was busy. Look for what held it, not at the refresh.
|
||||
- **An `asyncio` record is a real bug.** prompt_toolkit's loop handler is disabled, so a
|
||||
stray background-task exception (`Task exception was never retrieved`) lands in the file
|
||||
and nowhere else. No screen ever showed it.
|
||||
- **Test noise before 2026-09-08 02:52.** Until 59f10f9 the suite wrote into the real
|
||||
`~/.meshterm/meshterm.log`. The summary hides it by fingerprint, and hides runs whose ids
|
||||
came from another database, and says how many records went. Anything on this machine
|
||||
from before then that still looks synthetic probably is. A hand-run `--mock` session
|
||||
still writes into the real log *and* the real database, so run `purge-test-data` after
|
||||
one.
|
||||
|
||||
## From record to code
|
||||
|
||||
1. **Find the call site.** Grep the message's literal text, the part before its first
|
||||
formatted value, under `meshterm/`. Read the `try` it sits in.
|
||||
2. **Check it is still live.** A traceback from last week may already be fixed, and
|
||||
finding the name in the source proves nothing: the attribute an `AttributeError`
|
||||
complains of usually still exists, on a different class. Read the frame's own line at
|
||||
HEAD, and use `git log -L` on that function (or `git log -S "<literal>"`) to see whether
|
||||
it changed after the record's date. A fixed one gets named with its commit, not fixed
|
||||
twice.
|
||||
3. **Reproduce a live one.** If it doesn't need hardware, reproduce it (the `verify`
|
||||
skill; `--mock` needs `purge-test-data` after). If it needs the radio, tell JP what to
|
||||
do and at what Log detail.
|
||||
|
||||
## Report
|
||||
|
||||
- **Symptom**, in JP's words.
|
||||
- **Evidence**: the few decisive records, quoted with their timestamps, not the summary
|
||||
dump.
|
||||
- **Cause**: `file:line` and the mechanism.
|
||||
- **Status**: live on HEAD, fixed in `<hash>`, or unknown without a reproduction (and what
|
||||
that reproduction needs).
|
||||
- **Fix**: proposed, not applied, unless JP asked.
|
||||
|
||||
Log excerpts carry node and channel names, and the legacy `.jsonl` carries tool `params`
|
||||
logged verbatim, **admin passwords among them**. Quote only what a finding needs, and never
|
||||
put log text anywhere public (an issue, Discord) without JP reading it first.
|
||||
|
||||
## Keeping the script honest
|
||||
|
||||
- `RECORD` in `logscan.py` parses `_LINE_FORMAT` from `meshterm/persistence/logging.py`.
|
||||
Change one, change the other.
|
||||
- `FIXTURE` carries `MockDevice`'s identities. Keep them in step with
|
||||
`.claude/skills/purge-test-data/purge.py`.
|
||||
@@ -0,0 +1,632 @@
|
||||
"""Read MeshTerm's log the way an investigation needs it: grouped, windowed, joined to runs.
|
||||
|
||||
MeshTerm writes ``<config_dir>/meshterm.log`` (see ``meshterm/persistence/logging.py``):
|
||||
one record per line — ``2026-09-07 14:23:11 WARNING meshterm: message`` — with a
|
||||
traceback indented under the record that raised it, rotated at 2MB into ``.1``..``.3``.
|
||||
Before 2026-09-07 it was JSON Lines, and that file is still beside it as
|
||||
``meshterm.log.jsonl`` (``--legacy``).
|
||||
|
||||
Read raw, the file misleads in predictable ways, and this script takes them off the table
|
||||
before any reasoning starts:
|
||||
|
||||
- **A count without its spread.** Two hundred identical records inside one hour are one
|
||||
incident, not a chronic fault — so every group carries its first and last time.
|
||||
- **The same failure twice.** A tool that raises logs ``run N failed`` with a traceback, and
|
||||
the CLI then logs ``<tool> raised an unhandled error`` with the *same* traceback. The
|
||||
second is folded into the first.
|
||||
- **Test runs.** Until ``tests/conftest.py`` isolated ``MESHTERM_HOME`` the suite wrote into
|
||||
the developer's real log, and ``--mock`` sessions still can. Records carrying their
|
||||
fingerprints, and runs numbered by another database, are hidden unless ``--fixtures``
|
||||
asks for them, and the summary says how many went.
|
||||
- **Two clocks.** Log records are the writing machine's *local* time; the ``runs`` table is
|
||||
UTC. Both carry the run id, which is the join — the table's times are converted to this
|
||||
machine's local time for display only.
|
||||
|
||||
Stdlib only, so it runs anywhere MeshTerm does, the PicoCalc included.
|
||||
|
||||
Usage::
|
||||
|
||||
logscan.py [summary] the whole picture, grouped
|
||||
logscan.py show --level WARNING matching records, verbatim
|
||||
logscan.py around "2026-09-11 03:54" --minutes 3
|
||||
logscan.py run 3379 one run's records, its neighbours, its runs row
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sqlite3
|
||||
import sys
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, time, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
#: ``_LINE_FORMAT`` / ``_TIME_FORMAT`` in meshterm/persistence/logging.py, as a parser.
|
||||
RECORD = re.compile(
|
||||
r"^(\d{4}-\d\d-\d\d \d\d:\d\d:\d\d) (DEBUG|INFO|WARNING|ERROR|CRITICAL)\s+(\S+): (.*)$"
|
||||
)
|
||||
TIME_FORMAT = "%Y-%m-%d %H:%M:%S"
|
||||
LEVELS = {"DEBUG": 10, "INFO": 20, "WARNING": 30, "ERROR": 40, "CRITICAL": 50}
|
||||
|
||||
#: Test-suite and simulator fingerprints: the CLI-contract fixtures (``config get
|
||||
#: nosuchkey``/``no_such_key``, ``--port COM99``/``NOPE1``/``xx``, ``chat --to Nobody``, the
|
||||
#: ``[/]Bob`` contact planted to prove markup escaping), the discovery tests' BLE address,
|
||||
#: and ``MockDevice``'s identities — keep the last in step with
|
||||
#: ``.claude/skills/purge-test-data/purge.py``. Every one of them reached the real log only
|
||||
#: before 59f10f9 (2026-09-08 02:52 local) moved the suite into its own ``MESHTERM_HOME``;
|
||||
#: after that, only a ``--mock`` session run by hand can add more.
|
||||
FIXTURE = re.compile(
|
||||
r"nosuchkey|no_such_key|COM99|NOPE\d*|'Nobody'|'nope'|'xx'|\[/\]|AA:BB:CC:DD:EE:FF"
|
||||
r"|\bAlice\b|Yagi-Repeater|Local-Repeater|Observer-Bot"
|
||||
r"|MockCompanion|\bhub-1\b|\brelay-[12]\b|a1b2c3d4|b2c3d4e5|c3d4e5f6|d4e5f6a7"
|
||||
)
|
||||
|
||||
RUN_REF = re.compile(r"\brun (\d+)\b")
|
||||
RUN_START = re.compile(r"^run (\d+) start: tool=([\w-]+)")
|
||||
RUN_FAILED = re.compile(r"^run (\d+) failed: ")
|
||||
RUN_END = re.compile(r"^run (\d+) (?:ok|aborted|failed|lost)\b")
|
||||
#: ``tools/base.py``'s endings that the CLI goes on to report in a line of its own.
|
||||
RUN_REPORTED = re.compile(r"^run (\d+) (?:aborted|failed|lost)\b")
|
||||
#: ``cli.py``'s two reports of a run that ended badly: the one line it printed on stderr
|
||||
#: (``_reported``), and the traceback of one it could not explain.
|
||||
CLI_LINE = re.compile(r"^([\w-]+) failed: ")
|
||||
CLI_COPY = re.compile(r"^([\w-]+) raised an unhandled error$")
|
||||
|
||||
#: A DEBUG/INFO record that reports a failure. Most of the package's background loops log
|
||||
#: ``"...: %s", exc`` at DEBUG — the message survives, the stack does not — so these are
|
||||
#: the failures a WARNING-and-up read never sees.
|
||||
QUIET = re.compile(
|
||||
r"fail|raised|error|could not|couldn't|timed out|timeout|rejected|refused|lost|skipped"
|
||||
r"|invalid|unexpected",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Record:
|
||||
"""One log record and the lines hanging under it (a traceback, a wrapped message)."""
|
||||
|
||||
when: datetime
|
||||
level: str
|
||||
logger: str
|
||||
message: str
|
||||
extra: list[str] = field(default_factory=list)
|
||||
#: The run-ending record this one reports again — the CLI's line or traceback for it.
|
||||
twin_of: Record | None = None
|
||||
|
||||
@property
|
||||
def text(self) -> str:
|
||||
"""The message and everything under it, for searching."""
|
||||
return "\n".join([self.message, *self.extra])
|
||||
|
||||
@property
|
||||
def rank(self) -> int:
|
||||
"""The level as a number, for ``--level``; an unknown level ranks lowest."""
|
||||
return LEVELS.get(self.level, 0)
|
||||
|
||||
def run_id(self) -> int | None:
|
||||
"""The run this record speaks about, if it names one."""
|
||||
m = RUN_REF.search(self.message)
|
||||
return int(m[1]) if m else None
|
||||
|
||||
def exception(self) -> str | None:
|
||||
"""The line a traceback ended on — the exception itself — or ``None``."""
|
||||
if not any(line.startswith("Traceback") for line in self.extra):
|
||||
return None
|
||||
tail = [
|
||||
line
|
||||
for line in self.extra
|
||||
if line.strip()
|
||||
and not line.startswith(
|
||||
(" ", "\t", "Traceback", "During handling", "The above exception")
|
||||
)
|
||||
]
|
||||
return tail[-1] if tail else None
|
||||
|
||||
def render(self, mark: str = "") -> str:
|
||||
"""The record as the file spelt it, behind an optional two-cell ``mark``."""
|
||||
stamp = self.when.strftime(TIME_FORMAT)
|
||||
head = f"{mark}{stamp} {self.level:<8} {self.logger}: {self.message}"
|
||||
return "\n".join([head, *self.extra])
|
||||
|
||||
|
||||
# --- loading --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def default_home() -> Path:
|
||||
"""``meshterm.core.config.default_config_dir``, without importing the package."""
|
||||
env = os.environ.get("MESHTERM_HOME", "").strip()
|
||||
return Path(env).expanduser() if env else Path.home() / ".meshterm"
|
||||
|
||||
|
||||
def log_files(home: Path, *, legacy: bool) -> list[Path]:
|
||||
"""Every log file in ``home``, oldest first: ``.jsonl``, ``.3``, ``.2``, ``.1``, live."""
|
||||
rotated = sorted(
|
||||
(p for p in home.glob("meshterm.log.*") if p.suffix[1:].isdigit()),
|
||||
key=lambda p: int(p.suffix[1:]),
|
||||
reverse=True,
|
||||
)
|
||||
files = [*rotated]
|
||||
if (home / "meshterm.log").exists():
|
||||
files.append(home / "meshterm.log")
|
||||
if legacy and (home / "meshterm.log.jsonl").exists():
|
||||
files.insert(0, home / "meshterm.log.jsonl")
|
||||
return files
|
||||
|
||||
|
||||
def load(files: list[Path]) -> tuple[list[Record], int]:
|
||||
"""Parse ``files`` in order. Returns the records and the count of orphan lines.
|
||||
|
||||
An orphan is a line before a file's first record — rotation cut a traceback away from
|
||||
the record it hangs under. It cannot be placed, so it is counted rather than guessed at.
|
||||
"""
|
||||
records: list[Record] = []
|
||||
orphans = 0
|
||||
for path in files:
|
||||
if path.suffix == ".jsonl":
|
||||
records.extend(_load_jsonl(path))
|
||||
continue
|
||||
with path.open(encoding="utf-8", errors="replace") as fh:
|
||||
here: list[Record] = []
|
||||
for raw in fh:
|
||||
line = raw.rstrip("\r\n")
|
||||
m = RECORD.match(line)
|
||||
if m:
|
||||
when = datetime.strptime(m[1], TIME_FORMAT)
|
||||
here.append(Record(when, m[2], m[3], m[4]))
|
||||
elif here:
|
||||
here[-1].extra.append(line)
|
||||
elif line.strip():
|
||||
orphans += 1
|
||||
records.extend(here)
|
||||
for r in records:
|
||||
while r.extra and not r.extra[-1].strip():
|
||||
r.extra.pop()
|
||||
_pair_cli_reports(records)
|
||||
return records, orphans
|
||||
|
||||
|
||||
def _load_jsonl(path: Path) -> list[Record]:
|
||||
"""The pre-2026-09-07 JSON Lines log: ``ts``/``level``/``logger``/``msg``, and ``exc``.
|
||||
|
||||
A ``msg`` may carry newlines (asyncio's ``Task was destroyed…`` puts the task on a
|
||||
second line), which the text log would have indented under the record; they go to
|
||||
``extra`` here for the same reason, so a group is keyed on the first line alone.
|
||||
"""
|
||||
out: list[Record] = []
|
||||
with path.open(encoding="utf-8", errors="replace") as fh:
|
||||
for raw in fh:
|
||||
try:
|
||||
obj = json.loads(raw)
|
||||
when = datetime.fromisoformat(obj["ts"]).replace(tzinfo=None)
|
||||
except (ValueError, KeyError, TypeError):
|
||||
continue
|
||||
first, *more = str(obj.get("msg", "")).splitlines() or [""]
|
||||
extra = [f" {line}" for line in more]
|
||||
for key, value in obj.items():
|
||||
if key in ("ts", "level", "logger", "msg"):
|
||||
continue
|
||||
text = str(value)
|
||||
extra.extend(text.splitlines() if "Traceback" in text else [f" {key}: {text}"])
|
||||
level = str(obj.get("level", "?"))
|
||||
out.append(Record(when, level, str(obj.get("logger", "?")), first, extra))
|
||||
return out
|
||||
|
||||
|
||||
def _pair_cli_reports(records: list[Record]) -> None:
|
||||
"""Point each of the CLI's reports of a run at the run-ending record it repeats.
|
||||
|
||||
``tools/base.py`` ends a bad run with ``run N aborted``/``lost``/``failed``; ``cli.py``
|
||||
then reports the same run once more, naming the tool rather than the run — a
|
||||
``<tool> failed: …`` line, or a ``<tool> raised an unhandled error`` traceback. The
|
||||
report carries no run id, so it is matched to the latest bad ending of a run of that
|
||||
tool within a few seconds. That is what lets a test run's report be hidden with the
|
||||
run, and a traceback be counted once.
|
||||
"""
|
||||
tools: dict[int, str] = {}
|
||||
ended: dict[str, Record] = {}
|
||||
for r in records:
|
||||
if m := RUN_START.match(r.message):
|
||||
tools[int(m[1])] = m[2]
|
||||
elif m := RUN_REPORTED.match(r.message):
|
||||
tool = tools.get(int(m[1]))
|
||||
if tool is not None:
|
||||
ended[tool] = r
|
||||
elif m := CLI_LINE.match(r.message) or CLI_COPY.match(r.message):
|
||||
end = ended.get(m[1])
|
||||
if end is not None and r.when - end.when <= timedelta(seconds=5):
|
||||
r.twin_of = end
|
||||
|
||||
|
||||
def drop_fixtures(records: list[Record]) -> tuple[list[Record], int]:
|
||||
"""Remove test/simulator records: fingerprinted ones, and every record of their runs.
|
||||
|
||||
A run is a fixture when its start record carries a fingerprint, or when its id is lower
|
||||
than one this log has already started. One database hands out increasing ids, so a
|
||||
``run 1`` after ``run 3212`` was written by *another* database — the suite's temporary
|
||||
one, or a ``--db`` run pointed at a scratch file. Ids repeat across databases, so a
|
||||
record belongs to the latest start of its id, never to every run that ever had it.
|
||||
|
||||
A record naming a run whose start this log never saw (a background recorder opens its
|
||||
run without one; ``log_level`` above DEBUG drops them all) is foreign only when it is
|
||||
far below the highest id — a real background run can trail the foreground ids of its
|
||||
own session by a little, never by a thousand.
|
||||
"""
|
||||
highest = 0
|
||||
flagged: dict[int, bool] = {}
|
||||
dropped: set[int] = set()
|
||||
kept: list[Record] = []
|
||||
for r in records:
|
||||
rid = r.run_id()
|
||||
if m := RUN_START.match(r.message):
|
||||
rid = int(m[1])
|
||||
foreign = bool(FIXTURE.search(r.message)) or rid < highest
|
||||
flagged[rid] = foreign
|
||||
if not foreign:
|
||||
highest = max(highest, rid)
|
||||
if rid is not None and rid in flagged:
|
||||
by_run = flagged[rid]
|
||||
else:
|
||||
by_run = rid is not None and highest - rid > 1000
|
||||
if by_run or FIXTURE.search(r.text) or (r.twin_of is not None and id(r.twin_of) in dropped):
|
||||
dropped.add(id(r))
|
||||
else:
|
||||
kept.append(r)
|
||||
return kept, len(records) - len(kept)
|
||||
|
||||
|
||||
# --- time -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def parse_when(text: str, now: datetime) -> datetime:
|
||||
"""``30m`` / ``2h`` / ``3d`` ago, ``03:54`` today, or ``2026-09-11[ 03:54[:10]]``."""
|
||||
text = text.strip()
|
||||
if m := re.fullmatch(r"(\d+(?:\.\d+)?)([mhd])", text):
|
||||
unit = {"m": "minutes", "h": "hours", "d": "days"}[m[2]]
|
||||
return now - timedelta(**{unit: float(m[1])})
|
||||
if m := re.fullmatch(r"(\d\d?):(\d\d)(?::(\d\d))?", text):
|
||||
return datetime.combine(now.date(), time(int(m[1]), int(m[2]), int(m[3] or 0)))
|
||||
return datetime.fromisoformat(text)
|
||||
|
||||
|
||||
def utc_to_local(stamp: str | None) -> datetime | None:
|
||||
"""A ``runs`` timestamp (UTC, RFC 3339) in this machine's local time, naive."""
|
||||
if not stamp:
|
||||
return None
|
||||
when = datetime.fromisoformat(stamp)
|
||||
if when.tzinfo is None:
|
||||
when = when.replace(tzinfo=timezone.utc)
|
||||
return when.astimezone().replace(tzinfo=None)
|
||||
|
||||
|
||||
def short(when: datetime | None) -> str:
|
||||
"""A time narrow enough to sit in a grouped row: month-day and clock, no year."""
|
||||
return when.strftime("%m-%d %H:%M:%S") if when else "-"
|
||||
|
||||
|
||||
# --- grouping -------------------------------------------------------------------------
|
||||
|
||||
|
||||
def shape(message: str) -> str:
|
||||
"""A message's first line with its variable parts blanked, so repeats group together."""
|
||||
s = (message.splitlines() or [""])[0]
|
||||
s = re.sub(r"\b[0-9a-fA-F]{6,}\b", "<hex>", s)
|
||||
s = re.sub(r"\d+(?:\.\d+)?", "#", s)
|
||||
return s if len(s) <= 150 else s[:149] + "…"
|
||||
|
||||
|
||||
class Groups:
|
||||
"""Count, first time and last time per key."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
"""Start empty."""
|
||||
self.items: dict[object, list] = {}
|
||||
|
||||
def add(self, key: object, r: Record) -> None:
|
||||
"""Count ``r`` under ``key`` and stretch the key's span to cover it."""
|
||||
g = self.items.setdefault(key, [0, r.when, r.when])
|
||||
g[0] += 1
|
||||
g[2] = r.when
|
||||
|
||||
def rows(self, top: int) -> list[tuple[object, list]]:
|
||||
"""The ``top`` biggest groups, largest first, earliest first among equals."""
|
||||
ranked = sorted(self.items.items(), key=lambda kv: (-kv[1][0], kv[1][1]))
|
||||
return ranked[:top]
|
||||
|
||||
|
||||
def blocks(records: list[Record], gap: timedelta) -> list[list[Record]]:
|
||||
"""Split into runs of activity separated by at least ``gap`` of silence."""
|
||||
out: list[list[Record]] = []
|
||||
for r in records:
|
||||
if not out or r.when - out[-1][-1].when >= gap:
|
||||
out.append([])
|
||||
out[-1].append(r)
|
||||
return out
|
||||
|
||||
|
||||
# --- the runs table -------------------------------------------------------------------
|
||||
|
||||
|
||||
def open_db(path: Path | None) -> sqlite3.Connection | None:
|
||||
"""Open the database read-only, or ``None`` when there is no usable ``runs`` table."""
|
||||
if path is None or not path.exists():
|
||||
return None
|
||||
try:
|
||||
con = sqlite3.connect(f"file:{path.resolve().as_posix()}?mode=ro", uri=True)
|
||||
con.execute("SELECT 1 FROM runs LIMIT 1")
|
||||
return con
|
||||
except sqlite3.Error:
|
||||
return None
|
||||
|
||||
|
||||
def _error_of(summary_json: str | None) -> str:
|
||||
"""The ``error`` a failed run's summary recorded."""
|
||||
try:
|
||||
return str(json.loads(summary_json or "{}").get("error", ""))
|
||||
except (ValueError, AttributeError):
|
||||
return summary_json or ""
|
||||
|
||||
|
||||
def runs_section(
|
||||
con: sqlite3.Connection,
|
||||
since: datetime | None,
|
||||
until: datetime | None,
|
||||
fixtures: bool,
|
||||
top: int,
|
||||
) -> None:
|
||||
"""Print the window's runs: status counts, the latest errors, and runs never finished."""
|
||||
rows = []
|
||||
for rid, tool, status, args_json, summary_json, started, finished in con.execute(
|
||||
"SELECT id, tool, status, args_json, summary_json, started_at, finished_at FROM runs"
|
||||
):
|
||||
start = utc_to_local(started)
|
||||
if start is None or (since and start < since) or (until and start > until):
|
||||
continue
|
||||
if not fixtures and FIXTURE.search(f"{args_json} {summary_json}"):
|
||||
continue
|
||||
rows.append((rid, tool, status, start, utc_to_local(finished), args_json, summary_json))
|
||||
print("\n== Runs table (meshterm.db; UTC shown as this machine's local time)")
|
||||
if not rows:
|
||||
print(" no runs in the window")
|
||||
return
|
||||
counts = Counter(r[2] for r in rows)
|
||||
print(" " + " · ".join(f"{k} {v}" for k, v in counts.most_common()))
|
||||
errors = [r for r in rows if r[2] == "error"][-top:]
|
||||
if errors:
|
||||
print(" errors, latest last — a raised error the CLI reported cleanly has no traceback:")
|
||||
for rid, tool, _, start, _, _, summary in errors:
|
||||
print(f" #{rid:<6} {short(start)} {tool:<14} {_error_of(summary)[:100]}")
|
||||
open_rows = [r for r in rows if r[2] == "running"][-top:]
|
||||
if open_rows:
|
||||
print(" never finished — the process died, was killed, or is still running now:")
|
||||
for rid, tool, _, start, _, args, _ in open_rows:
|
||||
print(f" #{rid:<6} {short(start)} {tool:<14} {args[:80]}")
|
||||
|
||||
|
||||
# --- modes ----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def cmd_summary(
|
||||
records: list[Record],
|
||||
args: argparse.Namespace,
|
||||
window: tuple[datetime | None, datetime | None],
|
||||
hidden: int,
|
||||
orphans: int,
|
||||
files: list[Path],
|
||||
con: sqlite3.Connection | None,
|
||||
) -> None:
|
||||
"""Print the grouped picture: activity, tracebacks, warnings, quiet failures, runs."""
|
||||
since, until = window
|
||||
print("files " + ", ".join(f"{p.name} ({p.stat().st_size // 1024} KB)" for p in files))
|
||||
if records:
|
||||
print(f"span {records[0].when} → {records[-1].when}")
|
||||
if since or until:
|
||||
print(f"window {since or '…'} → {until or '…'}")
|
||||
note = (
|
||||
f" · {hidden} test, mock and other-database records hidden (--fixtures)" if hidden else ""
|
||||
)
|
||||
if orphans:
|
||||
note += f" · {orphans} orphan lines (rotation cut)"
|
||||
print(f"records {len(records)}{note}")
|
||||
levels = Counter(r.level for r in records)
|
||||
print("levels " + " · ".join(f"{k} {levels[k]}" for k in LEVELS if levels[k]))
|
||||
if levels and not levels["DEBUG"] and not levels["INFO"]:
|
||||
print(
|
||||
" no DEBUG/INFO at all: log_level is probably WARNING — silence proves nothing"
|
||||
)
|
||||
if not records:
|
||||
return
|
||||
|
||||
print(f"\n== Activity (a new block after {args.gap:g} min of silence; latest {args.top})")
|
||||
for chunk in blocks(records, timedelta(minutes=args.gap))[-args.top :]:
|
||||
lv = Counter(r.level for r in chunk)
|
||||
errors = lv["ERROR"] + lv["CRITICAL"]
|
||||
bad = (f" ⚠ {lv['WARNING']}" if lv["WARNING"] else "") + (
|
||||
f" ✗ {errors}" if errors else ""
|
||||
)
|
||||
print(f" {short(chunk[0].when)} → {short(chunk[-1].when)} {len(chunk):>5} records{bad}")
|
||||
|
||||
problems, tracebacks, quiet = Groups(), Groups(), Groups()
|
||||
copies: Counter = Counter()
|
||||
for r in records:
|
||||
exc = r.exception()
|
||||
if r.twin_of is not None and exc and exc == r.twin_of.exception():
|
||||
copies[(exc, shape(r.twin_of.message))] += 1
|
||||
continue
|
||||
if exc:
|
||||
tracebacks.add((exc, shape(r.message)), r)
|
||||
elif r.rank >= LEVELS["WARNING"]:
|
||||
problems.add((r.level, r.logger, shape(r.message)), r)
|
||||
elif QUIET.search(r.message) and not RUN_START.match(r.message):
|
||||
quiet.add((r.logger, shape(r.message)), r)
|
||||
|
||||
print("\n== Tracebacks (grouped by the exception they ended on)")
|
||||
if not tracebacks.items:
|
||||
print(" none")
|
||||
for (exc, under), (n, first, last) in tracebacks.rows(args.top):
|
||||
folded = copies[(exc, under)]
|
||||
tail = f" (+{folded} CLI copies folded)" if folded else ""
|
||||
print(f" {n:>4}× {short(first)} → {short(last)} {exc[:120]}")
|
||||
print(f" under: {under}{tail}")
|
||||
|
||||
print("\n== Warnings and errors without a traceback")
|
||||
if not problems.items:
|
||||
print(" none")
|
||||
for (level, logger, msg), (n, first, last) in problems.rows(args.top):
|
||||
print(f" {n:>4}× {short(first)} → {short(last)} {level:<7} {logger}: {msg}")
|
||||
|
||||
print("\n== Quiet failures (DEBUG/INFO that report a failure — message kept, stack lost)")
|
||||
if not quiet.items:
|
||||
print(" none (or log_level is above DEBUG)")
|
||||
for (logger, msg), (n, first, last) in quiet.rows(args.top):
|
||||
print(f" {n:>4}× {short(first)} → {short(last)} {logger}: {msg}")
|
||||
|
||||
if con is not None:
|
||||
runs_section(con, since, until, args.fixtures, min(args.top, 10))
|
||||
|
||||
|
||||
def cmd_show(records: list[Record], args: argparse.Namespace) -> None:
|
||||
"""Print the filtered records verbatim — the last ``--limit`` of them."""
|
||||
picked = records[-args.limit :] if args.limit else records
|
||||
if len(picked) < len(records):
|
||||
print(f"… {len(records) - len(picked)} earlier records not shown (--limit)")
|
||||
for r in picked:
|
||||
print(r.render())
|
||||
|
||||
|
||||
def cmd_around(records: list[Record], centre: datetime, args: argparse.Namespace) -> None:
|
||||
"""Print every record within ``--minutes`` of ``centre``; the nearest 30s are marked."""
|
||||
span = timedelta(minutes=args.minutes)
|
||||
print(f"-- {centre - span} → {centre + span}")
|
||||
for r in records:
|
||||
if centre - span <= r.when <= centre + span:
|
||||
print(r.render("» " if abs(r.when - centre) < timedelta(seconds=30) else " "))
|
||||
|
||||
|
||||
def cmd_run(
|
||||
all_records: list[Record],
|
||||
records: list[Record],
|
||||
run_id: int,
|
||||
args: argparse.Namespace,
|
||||
con: sqlite3.Connection | None,
|
||||
) -> None:
|
||||
"""Print one run: its ``runs`` row, then every record from its start to its end.
|
||||
|
||||
The run's own records are marked; the rest are what the process was doing beside it.
|
||||
A run with no end record gets ``--minutes`` after its start instead — the run died or
|
||||
is still going, and what came next is the evidence either way.
|
||||
"""
|
||||
if con is not None:
|
||||
row = con.execute(
|
||||
"SELECT tool, profile, status, args_json, summary_json, started_at, finished_at"
|
||||
" FROM runs WHERE id = ?",
|
||||
(run_id,),
|
||||
).fetchone()
|
||||
if row:
|
||||
tool, profile, status, args_json, summary_json, started, finished = row
|
||||
start, end = utc_to_local(started), utc_to_local(finished)
|
||||
took = f" ({(end - start).total_seconds():.1f}s)" if start and end else ""
|
||||
print(f"run #{run_id} {tool} status={status} profile={profile or '-'}")
|
||||
print(f" started {start} finished {end or 'never'}{took} (local)")
|
||||
print(f" args {args_json}")
|
||||
print(f" summary {(summary_json or '-')[:300]}")
|
||||
else:
|
||||
print(f"run #{run_id}: no row in the runs table")
|
||||
own = [r for r in all_records if r.run_id() == run_id]
|
||||
if not own:
|
||||
print(f"no log record names run {run_id} — its tool logs nothing at this log_level")
|
||||
return
|
||||
start = own[0].when
|
||||
ended = any(RUN_END.match(r.message) for r in own)
|
||||
end = own[-1].when if ended else max(own[-1].when, start + timedelta(minutes=args.minutes))
|
||||
if not ended:
|
||||
print(f" no end record: the {args.minutes:g} min after its start follow")
|
||||
slack = timedelta(seconds=2)
|
||||
window = [r for r in records if start - slack <= r.when <= end + slack]
|
||||
if args.limit and len(window) > args.limit:
|
||||
print(f" {len(window)} records in the window; the first {args.limit} (--limit)")
|
||||
window = window[: args.limit]
|
||||
ids = {id(r) for r in own}
|
||||
for r in window:
|
||||
print(r.render("» " if id(r) in ids else " "))
|
||||
|
||||
|
||||
# --- entry ----------------------------------------------------------------------------
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
"""Parse the arguments, load and filter the log, and run the chosen mode."""
|
||||
ap = argparse.ArgumentParser(description="Read MeshTerm's log: grouped, windowed, joined.")
|
||||
ap.add_argument(
|
||||
"mode", nargs="?", default="summary", choices=("summary", "show", "around", "run")
|
||||
)
|
||||
ap.add_argument("target", nargs="?", help="a time for `around`, a run id for `run`")
|
||||
ap.add_argument("--home", type=Path, help="config dir (default $MESHTERM_HOME or ~/.meshterm)")
|
||||
ap.add_argument(
|
||||
"--log", type=Path, action="append", help="read this file; repeatable, oldest first"
|
||||
)
|
||||
ap.add_argument("--db", type=Path, help="runs table source (default <home>/meshterm.db)")
|
||||
ap.add_argument("--no-db", action="store_true", help="skip the runs table")
|
||||
ap.add_argument(
|
||||
"--legacy", action="store_true", help="include the pre-09-07 meshterm.log.jsonl"
|
||||
)
|
||||
ap.add_argument("--since", help="30m | 2h | 3d | 03:54 | 2026-09-11 03:54 (local)")
|
||||
ap.add_argument("--until", help="same forms as --since")
|
||||
ap.add_argument("--level", help="minimum level: DEBUG INFO WARNING ERROR")
|
||||
ap.add_argument("--grep", help="regex over message and traceback, case-insensitive")
|
||||
ap.add_argument("--fixtures", action="store_true", help="keep test-suite and --mock records")
|
||||
ap.add_argument("--minutes", type=float, default=5, help="`around` half-width; open-run tail")
|
||||
ap.add_argument("--gap", type=float, default=20, help="minutes of silence that start a block")
|
||||
ap.add_argument("--top", type=int, default=25, help="rows per summary section")
|
||||
ap.add_argument("--limit", type=int, default=200, help="most records `show`/`run` print")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
if hasattr(sys.stdout, "reconfigure"):
|
||||
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
|
||||
|
||||
home = (args.home or default_home()).expanduser()
|
||||
files = [f for f in (args.log or log_files(home, legacy=args.legacy)) if f.exists()]
|
||||
if not files:
|
||||
print(f"no log files found (looked in {home})", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
now = datetime.now()
|
||||
since = parse_when(args.since, now) if args.since else None
|
||||
until = parse_when(args.until, now) if args.until else None
|
||||
|
||||
all_records, orphans = load(files)
|
||||
records, hidden = (all_records, 0) if args.fixtures else drop_fixtures(all_records)
|
||||
minimum = LEVELS.get((args.level or "").upper(), 0)
|
||||
grep = re.compile(args.grep, re.IGNORECASE) if args.grep else None
|
||||
records = [
|
||||
r
|
||||
for r in records
|
||||
if (not since or r.when >= since)
|
||||
and (not until or r.when <= until)
|
||||
and r.rank >= minimum
|
||||
and (not grep or grep.search(r.text))
|
||||
]
|
||||
|
||||
con = open_db(None if args.no_db else (args.db or home / "meshterm.db"))
|
||||
|
||||
if args.mode == "summary":
|
||||
cmd_summary(records, args, (since, until), hidden, orphans, files, con)
|
||||
elif args.mode == "show":
|
||||
cmd_show(records, args)
|
||||
elif args.mode == "around":
|
||||
if not args.target:
|
||||
ap.error("around needs a time")
|
||||
cmd_around(records, parse_when(args.target, now), args)
|
||||
else:
|
||||
if not (args.target or "").isdigit():
|
||||
ap.error("run needs a run id")
|
||||
cmd_run(all_records, records, int(args.target), args, con)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in new issue
Block a user