Packets: the frames a neighbour sends were being thrown away

Two silent drops in the RX-log mapper, and between them a device sitting
beside this one could trace all day and never appear in the feed.

A frame that crossed no relays was discarded as having "no topology
content". It has the most there is: nothing stood between us, so we heard
the transmitter itself. Only an advert survived, and only because it
carries an origin key — every trace, message, request and ack from an
adjacent node went into the bin. Now a frame is kept unless the library
could not even give it a payload class, and the empty path says what it
means: zero hops crossed.

The other is a trace reading its own header wrong. Every class keeps
relay hashes in the path field; a trace keeps one signed SNR byte per hop
it has traversed. Read as hashes those bytes resolved to whichever nodes
shared their leading digits and entered the topology graph as links that
were never observed. The population says so plainly: across 1604 stored
traces, all 2221 path entries land inside the LoRa SNR band when read as
signed bytes, spanning -10.5 to +15.0 dB, where hashes would put five in
six outside it. So a trace now yields no hops, and its readings — which
are worth having — ride alongside as trace_snrs.

The 1604 rows already stored keep their bad paths; a one-shot cleanup
follows separately.
This commit is contained in:
Jean-Pierre Martineau committed 2026-08-22 13:57:06 -04:00
1 parent c19711c463
commit 6217a7dc58
3 files changed
+158 -9

No files matched your search

+35 -8
View File
@@ -23,7 +23,7 @@ from typing import TYPE_CHECKING, Callable, Optional
from .channels import CHANNEL_SLOT_PROBE_CAP
from .events import MeshEvent
from .frames import frame_addressing
from .frames import frame_addressing, trace_link_snrs
from .models import (
NODE_TYPE_CHAT,
NODE_TYPE_REPEATER,
@@ -2857,8 +2857,18 @@ def packet_observation_from_event(event) -> Optional[Observation]: # noqa: ANN0
The originating node is only knowable when the payload class reveals it: the library
decodes adverts inline (``adv_key``/``adv_name``), so those carry an origin; other
packet classes are recorded origin-less — their path (plus our reception of its last
relay) is still adjacency evidence. Frames that carry neither an origin nor any path
teach us nothing about topology and map to ``None``.
relay) is still adjacency evidence.
A frame that crossed no relays at all is kept too, with an empty path. Far from
teaching us nothing, it is the strongest adjacency evidence there is — we heard the
transmitter directly — and it is the whole of what an adjacent device puts on the air,
so discarding it made a neighbour's traffic invisible. Only a frame the library could
not even assign a payload class to is dropped.
One class needs its header read differently from the rest: a ``TRACE``'s ``path``
field carries per-hop SNR readings rather than relay hashes (see
:func:`~meshterm.core.frames.trace_link_snrs`), so it yields no hops and its readings
are merged in as ``trace_snrs`` instead.
Origin-less does not mean featureless, though: what the frame *addresses* is decoded
out of its undecoded body (:func:`~meshterm.core.frames.frame_addressing`) and merged
@@ -2870,28 +2880,45 @@ def packet_observation_from_event(event) -> Optional[Observation]: # noqa: ANN0
event: A meshcore ``RX_LOG_DATA`` event (anything exposing a ``payload`` mapping).
Returns:
The parsed :class:`Observation` (``kind="packet"``), or ``None`` for frames with
no topology content or an unparsable path.
The parsed :class:`Observation` (``kind="packet"``), or ``None`` for a frame with
no decodable payload class or an unparsable path.
"""
payload = dict(getattr(event, "payload", {}) or {})
typename = payload.get("payload_typename")
if not typename or typename == "UNK":
return None # the library's sentinel for a frame too short to have a class at all
path_len = _as_int(payload.get("path_len")) or 0
hash_size = _as_int(payload.get("path_hash_size")) or 1
path_hex = str(payload.get("path") or "").lower().removeprefix("0x")
hops: list[str] = []
if path_len > 0:
# A trace's path field is SNR readings, not relay hashes (see
# :func:`~meshterm.core.frames.trace_link_snrs`), so it contributes no hops — its
# readings are recovered below and kept beside the frame instead.
if path_len > 0 and typename != "TRACE":
width = hash_size * 2
hops = [path_hex[i * width : (i + 1) * width] for i in range(path_len)]
if any(len(h) != width for h in hops):
return None # a truncated path would fabricate adjacency between wrong nodes
origin = payload.get("adv_key")
if not origin and not hops:
return None # neither endpoint nor relays: no topology content
# A frame with no origin and no relays is not featureless — it is the strongest
# adjacency evidence the mesh produces: we heard the transmitter *directly*, with
# nothing in between. Dropping it made every non-advert frame from an adjacent node
# invisible, which is why a companion device sitting beside this one could trace all
# day and never appear in the feed. What it carries — its class, what it addresses,
# how well it was heard — is recorded exactly as a relayed frame's is, with an empty
# path standing for the zero hops it crossed.
#
# What the frame addresses — the recipient, the sender, the channel, the token it
# carries — read out of the body the library leaves undecoded for every class but
# advert and channel text (see :mod:`~meshterm.core.frames`). Merged in under its own
# keys so a class that names no origin node still says what it is *about*.
payload.update(frame_addressing(payload))
readings = trace_link_snrs(payload)
if readings is not None:
# Kept beside the frame, not in ``path``: a trace's hop readings say how well each
# leg was heard, and nothing at all about who relayed it.
payload["trace_snrs"] = readings
ident = str(origin).lower().removeprefix("0x") if origin else None
node = ident[:12] if ident else None
public_key = ident if ident and len(ident) > 12 else None # keep the whole adv_key
+54
View File
@@ -24,6 +24,11 @@ class body layout
``GRP_TXT``/``GRP_DATA`` ``[channel hash:1][MAC:2][ciphertext]``
=========================== ==========================================================
One class also breaks the rule the *header* follows. Every other frame's ``path`` field is
the list of relay hashes it has crossed; a ``TRACE``'s is a list of **signed SNR bytes**,
one per hop traversed — see :func:`trace_link_snrs`, which is what reads it, and which
exists because reading those bytes as hashes fabricated adjacency in the topology graph.
Endpoints are named by a *hash* — the leading byte of the node's public key, the same
one-byte identity the relay path's hops use — so they resolve through the app's ordinary
node resolver and land on the same names, hues and collisions as every hop does. An
@@ -160,3 +165,52 @@ def frame_addressing(payload: Mapping[str, Any]) -> dict[str, str]:
"crypted": body[hash_w + _MAC :].hex(),
}
return {}
def trace_link_snrs(payload: Mapping[str, Any]) -> list[float] | None:
"""Recover the per-hop link readings a ``TRACE`` frame collected, from its path field.
A trace is the one class whose header ``path`` field does not hold relay hashes. The
firmware grows it by **one signed SNR byte per hop the packet traverses** — the
reading the relaying node heard its predecessor at — which is how a trace reply can
report a whole route's link quality without a second field, and what the meshcore
parser's ``# Beware of traces where pathes are mixed`` warns about. Read as hashes it
is not merely useless but actively false: the bytes resolve to whichever nodes happen
to share those leading digits and enter the topology graph as adjacency that was never
observed.
The distinction is measurable, not theoretical. Across a captured population of 1,604
traces (2,221 path entries), every entry read as a signed byte over four lands inside
the LoRa SNR band, spanning −10.5 to +15.0 dB — where relay hashes, being uniform over
the byte, would put roughly five in six outside it.
Args:
payload: The frame's raw payload as the RX-log event carried it (needs
``payload_typename``, ``path_len`` and the hex ``path``).
Returns:
One SNR in dB per traversed hop, in the order the packet walked them, or ``None``
when the frame is not a trace or its path field is unreadable at the announced
length. A trace nobody has relayed yet correctly yields an empty list.
"""
if payload.get("payload_typename") != "TRACE":
return None
try:
hop_count = int(payload.get("path_len") or 0)
except (TypeError, ValueError):
return None
if hop_count < 0:
return None
raw = str(payload.get("path") or "").lower().removeprefix("0x")
try:
readings = bytes.fromhex(raw)
except ValueError:
return None
if len(readings) < hop_count:
return None # short of what it announced: the tail would be invented, not read
return [_snr_db(b) for b in readings[:hop_count]]
def _snr_db(byte: int) -> float:
"""Decode one wire SNR byte: a signed value in quarter-decibels."""
return (byte - 256 if byte >= 128 else byte) / 4.0
+69 -1
View File
@@ -13,7 +13,7 @@ from datetime import timedelta
from pathlib import Path
from meshterm.core.connection import packet_observation_from_event
from meshterm.core.frames import frame_addressing
from meshterm.core.frames import frame_addressing, trace_link_snrs
from meshterm.core.models import Observation, utcnow
from meshterm.persistence.repository import Repository
@@ -135,3 +135,71 @@ def test_stored_frames_remember_what_they_addressed(tmp_path: Path) -> None:
for key, value in decoded.items():
assert read_back[typename][key] == value, (typename, key)
repo.close()
def test_a_trace_path_is_link_readings_not_relay_hashes() -> None:
"""The one class whose header path field means something else entirely.
A trace grows its path by one signed SNR byte per hop, so reading it as hashes both
invents adjacency and throws away the readings. Bytes here span the wire's signed
range: +13.25 dB, then two negative legs.
"""
payload = _frame("TRACE", bytes(9), path_len=3, path_hash_size=1, path="35eeef")
assert trace_link_snrs(payload) == [13.25, -4.5, -4.25]
def test_only_a_trace_reads_its_path_that_way() -> None:
"""Every other class keeps hashes there, so nothing else may be decoded as readings."""
for typename in ("TEXT_MSG", "ADVERT", "GRP_TXT", "ACK", "PATH"):
assert trace_link_snrs(_frame(typename, bytes(9), path_len=1, path="35")) is None
def test_an_unrelayed_trace_has_readings_for_no_hops() -> None:
"""Nobody has forwarded it yet, so there is nothing to have measured it — not a fault."""
assert trace_link_snrs(_frame("TRACE", bytes(9), path_len=0, path="")) == []
def test_a_trace_path_shorter_than_announced_is_not_invented() -> None:
"""Three hops promised, one byte delivered: the missing readings stay missing."""
assert trace_link_snrs(_frame("TRACE", bytes(9), path_len=3, path="35")) is None
def test_a_traces_readings_reach_the_observation_and_its_hops_do_not() -> None:
"""The readings ride in the raw payload; ``path`` stays empty, having no hops to hold.
Storing those bytes as a path fed the topology graph links that were never observed —
whichever nodes happened to share the leading digits of an SNR reading.
"""
obs = packet_observation_from_event(
_Event(payload_typename="TRACE", pkt_payload=bytes.fromhex("5f3c2a10") + bytes(5),
path_len=3, path_hash_size=1, path="35eeef", snr=13.75)
)
assert obs is not None and obs.raw is not None
assert obs.path == ""
assert obs.raw["trace_snrs"] == [13.25, -4.5, -4.25]
assert obs.raw["trace_tag"] == "102a3c5f"
def test_a_frame_heard_straight_off_its_sender_is_kept() -> None:
"""Zero relays is the strongest adjacency evidence there is, not the absence of any.
This is the whole of what a device sitting beside this one puts on the air: nothing
has relayed it, so it names no repeater, and only an advert carries an origin key.
Dropping it made a neighbour's every trace, message and ack invisible.
"""
for typename in ("TRACE", "TEXT_MSG", "ACK", "REQ", "GRP_TXT"):
obs = packet_observation_from_event(
_Event(payload_typename=typename, pkt_payload=bytes(24),
path_len=0, path_hash_size=1, path="", snr=9.25, rssi=-61)
)
assert obs is not None, typename
assert obs.path == "" and obs.snr == 9.25
assert (obs.raw or {})["payload_typename"] == typename
def test_a_frame_with_no_class_at_all_is_still_dropped() -> None:
"""The library's sentinel for a frame too short to parse teaches nothing about anything."""
assert packet_observation_from_event(
_Event(payload_typename="UNK", pkt_payload=b"", path_len=0, path="")
) is None
assert packet_observation_from_event(_Event(snr=6.0)) is None