richandClaude Opus 5.5 e2e2279be2
CI / test-and-build (3.10) (push) Waiting to run
CI / test-and-build (3.11) (push) Waiting to run
CI / test-and-build (3.12) (push) Waiting to run
CI / test-and-build (3.13) (push) Waiting to run
Keep the spool's file list in memory instead of rescanning it
Every oldest(), stats() and put() globbed and sorted the whole spool
directory, so draining a backlog cost O(N^2 log N). After an outage on
a Pi 3B+ the writer thread sat at 100% CPU for minutes working through
3,300 batches. The directory is now scanned once at startup and the
names, sizes and total are tracked as batches are added and removed;
draining 3,300 batches drops from 59s to 0.24s on a laptop.

Bump to 0.1.6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 10:42:24 -04:00
2026-10-02 10:34:40 -04:00
2026-10-04 15:55:16 -04:00

openHop Repeater → InfluxDB plugin

An openHop Repeater plugin that pushes the repeater's data to a remote InfluxDB (v1, v2 or v3), ready for Grafana dashboards. A matching Grafana dashboard ships in dashboards/.

openHop Repeater ──REST /api/*────────► metrics     (every 30 s)
                 ──WS   /ws/packets───► packets     (real time, gap backfill)
                 ──SSE  /api/logs_stream► logs      (real time, resumable)
                 ──REST /api/*────────► snapshots   (every 5 min)
                 ──REST history APIs──► history     (full import, then every 5 min)
                                          │
                     batch + gzip + retry + disk spool
                                          ▼
                                InfluxDB ──► Grafana
Plugin id Entrypoint Distribution
community.influxdb openhop-influxdb openhop-influxdb-plugin

What is shipped

Every point is tagged repeater=<node name> (override with repeater_tag), so several repeaters can share one bucket.

Measurement Source Tags Fields
openhop_repeater_*, openhop_system_* (≈120 families) /api/stats, /api/packet_stats, /api/packet_type_stats, /api/route_stats, /api/lbt_diagnostics, /api/noise_floor_stats, /api/neighbor_links the metric's labels value
openhop_packet /ws/packets + /api/bulk_packets type, type_name, route, route_name, transmitted, is_duplicate, dropped, drop_reason, rx_radio_id, tx_radio_id, src_pubkey (ADVERT only: the advertiser's public key) rssi, snr, score, length, payload_length, tx_delay_ms, airtime_ms, lbt_attempts, lbt_channel_busy, hop_count, original_path, forwarded_path, packet_hash, src_hash, dst_hash, path_hash, …
openhop_log /api/logs_stream level, logger message, log_id
openhop_advert /api/stats neighbors pubkey, contact_type, is_repeater, zero_hop node_name, rssi, snr, advert_count, first_seen, last_seen, last_seen_age_seconds, latitude, longitude
openhop_radio /api/stats radios + radio_stack (one point per radio; air settings inherit from the top-level radio: like the Repeater does) radio_id, radio_type, default frequency_hz, bandwidth_hz, spreading_factor, coding_rate, tx_power_dbm, preamble_length, configured
openhop_gps /api/gps state fix, satellites, DOP, speed, latitude, longitude, altitude_m
openhop_identities, openhop_acl, openhop_rooms, openhop_db, openhop_mqtt, openhop_crc_errors, openhop_advert_rate_limit matching /api/* endpoints item for list payloads every numeric value, flattened (a_b_c)
openhop_advert_received /api/adverts_by_contact_type (every advert the Repeater stored) pubkey, contact_type, is_repeater, zero_hop, new_neighbor node_name, rssi, snr, route_type, advert_count, first_seen, last_seen, latitude, longitude
openhop_noise_floor /api/noise_floor_history — noise_floor_dbm (each stored sample)
openhop_crc_error /api/crc_error_history — count (each stored CRC error event)
openhop_rrd /api/rrd_data (imported once) resolution (1m, 5m, 1h) rx_count_per_second, tx_count_per_second, drop_count_per_second, type_N_per_second, avg_rssi, avg_snr, avg_length, avg_score, neighbor_count
openhop_exporter the plugin itself stream lines written, errors, spool size, connection state

The metric families come from the official openhop-prometheus-plugin collector (MIT), vendored under src/openhop_influxdb_plugin/vendor/. Its metric names and labels are reused as InfluxDB measurement names and tags.

Notes on the schema:

  • Numbers are always floats. InfluxDB rejects a whole batch when a field changes type, and Repeater JSON mixes ints and floats.
  • Hashes are fields, not tags, to keep series cardinality low. pubkey on openhop_advert and peer_hash on neighbour links are tags. Expect one series per heard node.
  • Per-radio data comes from packets. On a multi-radio Repeater the counters, noise floor, radio status and LBT stats in /api/stats cover the whole stack (noise floor and status are the default radio's). Only packet records say which radio received (rx_radio_id) and sent (tx_radio_id) them, so the dashboard's Per radio row is built from openhop_packet.
  • Which radio hears which node comes from adverts. Only ADVERT packets name their sender (src_pubkey). Nodes heard by radio lists each radio/node pair. Node coverage by radio has one row per node and a column per radio, and highlights nodes heard on only one radio. Both count only the first copy of each advert (is_duplicate = false). The Repeater dedups across all its radios, so a later copy, such as another repeater re-flooding what we forwarded, is not counted as a receive on the radio that heard it. Adverts are flooded, so set Advert hops to Direct only to count only adverts heard straight from the node rather than through a relay.
  • Packets are idempotent. A packet seen live and again during backfill gets the same timestamp and tags, so InfluxDB overwrites it instead of duplicating it.

Privacy

  • Off by default: packet payload (include_payload) and raw packet bytes (include_raw_packet).
  • On by default, because the goal is "all data": node and repeater GPS coordinates (include_locations). Turn it off to drop all lat/lon/alt.
  • The generic snapshot endpoints (identities, ACL, rooms, DB, MQTT, …) contribute only numeric values, so names, keys and other strings in those payloads never leave the repeater. The only free-text values shipped are packet hashes and paths, advert node_name, and log message.
  • Tokens and passwords are never logged. --check-config prints them as ***.

Requirements

  • An openHop Repeater with the plugin manager (Repeater ≥ 0.3.0; validated against source 13eb8b2).
  • A Repeater API token, created in the Repeater web UI. Current Repeater tokens are administrator-equivalent, so the plugin config holds an admin credential. Treat config.json accordingly.
  • A reachable InfluxDB:
    • v2: URL, org, bucket and an API token with write access to the bucket.
    • v3 (Core/Enterprise/Cloud): URL, database name in influx_bucket, and a token.
    • v1 (1.8 and 1.11 tested): URL, influx_database, optional influx_retention_policy, and influx_username/influx_password if auth is enabled. influx_token set to username:password also works on 1.8+. InfluxDB 1.x does not create databases on write, so the plugin issues CREATE DATABASE once on the first "database not found" error (influx_create_database, default on). That needs an admin user; with a write-only user, create the database yourself and grant WRITE on it.

Install

  1. Download openhop_influxdb_plugin-<version>-py3-none-any.whl (or build it with ./build_wheel.sh).

  2. In the Repeater UI go to Plugins → Upload and select the wheel. A locally uploaded plugin starts disabled.

  3. Open Settings for InfluxDB Exporter and fill in at least the following:

    {
      "repeater_api_token": "<repeater API token>",
      "influx_url": "https://influx.example.com:8086",
      "influx_version": "v2",
      "influx_org": "home",
      "influx_bucket": "openhop",
      "influx_token": "<influx write token>"
    }
    

    For InfluxDB 1.x use this instead:

    {
      "repeater_api_token": "<repeater API token>",
      "influx_url": "http://influx.example.com:8086",
      "influx_version": "v1",
      "influx_database": "openhop",
      "influx_username": "openhop",
      "influx_password": "<password>"
    }
    
  4. Enable the plugin. Check Logs and the runtime status (GET /api/plugins/runtime?id=community.influxdb). Each stream should show running and the writer should show lines_written increasing.

  5. In Grafana, add an InfluxDB data source using the InfluxQL query language (database = your bucket or 1.x database; for 1.x with auth, set user and password; for v2, add header Authorization: Token <read token>). Then import dashboards/openhop-overview.json.

The plugin and Repeater usually run on the same host, so the default repeater_url of http://127.0.0.1:8000 works. In the Docker image this also holds, because the plugin manager runs in the same container.

Configuration reference

Key Default Meaning
repeater_url http://127.0.0.1:8000 Repeater web/API base URL
repeater_api_token "" Sent as X-API-Key (a leading Bearer is stripped)
repeater_verify_tls true Verify Repeater TLS certificates
repeater_tag "" Value of the repeater tag; empty uses the node name from /api/stats
influx_url "" InfluxDB base URL. Empty means the plugin idles and reports "not configured"
influx_version v2 v1, v2 or v3
influx_token "" Authorization: Token …
influx_org "" v2 organisation (required for v2)
influx_bucket openhop v2 bucket / v3 database
influx_database / influx_retention_policy openhop / "" v1 target
influx_username / influx_password "" v1 basic auth
influx_create_database true v1 only: run CREATE DATABASE if the database is missing (needs admin)
influx_verify_tls true Verify InfluxDB TLS certificates
stream_metrics / stream_packets / stream_logs / stream_snapshots / stream_history true Enable each stream
metrics_interval_seconds 30 Metrics poll interval (5–3600)
snapshot_interval_seconds 300 Snapshot poll interval (30–86400)
request_timeout_seconds 10 HTTP timeout for Repeater and InfluxDB calls
initial_backfill_hours "all" On first start, import the Repeater's stored history: "all", a number of hours, or 0 for none
backfill_hours "all" Window for a manual re-import (--backfill default, backfill_request_id)
backfill_request_id "" Set to any new value and save with restart to re-import backfill_hours of history once
log_min_level INFO Minimum Repeater log level to ship
include_payload / include_raw_packet false Ship packet payload hex / raw packet hex
include_locations true Ship GPS and advert coordinates
batch_size 5000 Max lines per InfluxDB write
flush_interval_seconds 5 Max time lines wait before being written
spool_max_mb 50 Disk spool cap for undeliverable batches (0 disables spooling)

Restart the plugin after changing settings (the Settings dialog can do this).

Importing history

On its first start the plugin imports everything the Repeater has stored (initial_backfill_hours, default "all"), alongside the live streams:

Stored on the Repeater How far back Imported as
Every packet (SQLite) storage.retention.sqlite_cleanup_days (default 31 days) openhop_packet
Every advert received same retention openhop_advert_received
Noise-floor samples same retention openhop_noise_floor
CRC error events same retention openhop_crc_error
Aggregated traffic/RF (RRD) 1-min steps for 7 days, 5-min for 30 days, hourly for 1 year openhop_rrd
Logs only the in-memory buffer openhop_log (sent on connect)

Counters such as total RX/forwarded have no history on the Repeater; their series start when the plugin starts (the RRD import covers the past year at lower resolution).

After the import, adverts, noise floor and CRC errors keep flowing every snapshot_interval_seconds, and packets keep flowing live. Each source resumes from its own cursor in state-influxdb.json and only advances it after a whole window has shipped, so an interrupted import is redone rather than left with a gap. Packets are paged by timestamp, so even a month of history on a Pi imports with bounded memory.

Re-importing is safe at any time: points keep their original timestamps and tags, so InfluxDB overwrites them instead of duplicating. Two ways:

  • From the Repeater UI (no shell needed): in the plugin's Settings, set backfill_request_id to any new value (for example today's date), optionally set backfill_hours, and save with restart. The import runs once in the background; GET /api/plugins/runtime?id=community.influxdb shows backfill → done.
  • From a shell: openhop-influxdb --backfill (everything) or --backfill 72 (last 72 hours). It flushes and exits; the running service's cursors are untouched.

The overwrite relies on identical tags. Version 0.1.2 added the src_pubkey tag to ADVERT packets, so ADVERT points written by 0.1.1 or earlier get no tag on re-import; they would land in a second series and be counted twice. The Nodes heard by radio and Node coverage by radio panels fill in from the upgrade onwards. To fill in history as well, first delete the old ADVERT points for the window the Repeater still holds, then re-import it. For example, for the last 7 days:

  • InfluxDB 1.x: DELETE FROM "openhop_packet" WHERE "type_name" = 'ADVERT' AND time > now() - 7d
  • InfluxDB 2.x: influx delete --bucket openhop --start <7 days ago, RFC3339> --stop <now> --predicate '_measurement="openhop_packet" AND type_name="ADVERT"'

then run openhop-influxdb --backfill 168.

Reliability

  • Independent streams. Each stream runs in its own thread with exponential reconnect backoff, so one failing stream never stops the others.
  • Write failures. If InfluxDB is down, slow or returns 5xx/429, batches go to a gzip spool in $OPENHOP_PLUGIN_DATA/spool/ and are replayed oldest-first when it recovers. Retry-After is honoured. HTTP 413 splits the batch. HTTP 400/422 (bad data) drops only that batch and logs the reason.
  • Packet gaps. The newest shipped packet timestamp is persisted in state-influxdb.json. After any disconnect or restart, the plugin backfills the gap from /api/bulk_packets.
  • Log resume. The log stream resumes with Last-Event-ID and resynchronises when the Repeater restarts and its log ids reset.
  • Clean shutdown. SIGTERM flushes (or spools) queued data within the plugin manager's 5 s stop budget.
  • Bad config. Invalid or incomplete config keeps the process alive and reports the problem in runtime.json and the logs, instead of crash-looping into FAILED.

CLI

openhop-influxdb --check-config          # validate config.json, print it redacted
openhop-influxdb --once --dry-run        # one collection cycle, print line protocol
openhop-influxdb --once                  # one collection cycle, write to InfluxDB
openhop-influxdb --backfill [HOURS|all]  # import stored Repeater history, then exit
openhop-influxdb                         # run as a service (what the plugin manager does)

The data directory is $OPENHOP_PLUGIN_DATA (default ./devdata), or set it with --data-dir.

Development

python3 -m venv .venv && .venv/bin/pip install -e '.[dev]'
.venv/bin/ruff check src tests scripts && .venv/bin/pytest -q
.venv/bin/python scripts/build_dashboard.py      # regenerate the Grafana dashboard
./build_wheel.sh

# Validate the wheel with the real Repeater plugin parser
git clone --no-checkout https://github.com/openhop-dev/openhop_repeater.git repeater-source
git -C repeater-source checkout --detach 13eb8b2ea8b1cdb4a07ed6e282dc99e3aa8a5a8b
PYTHONPATH=repeater-source .venv/bin/python scripts/validate_wheel.py dist/*.whl

Local InfluxDB + Grafana

docker compose -f dev/docker-compose.yml up -d    # InfluxDB 2 on :8086, Grafana on :3000
# Add InfluxDB 1.8 on :8087 (admin / openhop-admin); set INFLUXDB1_TAG=1.11 for 1.11
docker compose -f dev/docker-compose.yml --profile v1 up -d
mkdir -p devdata && cp dev/config.example.json devdata/config.json   # then edit it
.venv/bin/openhop-influxdb --data-dir devdata --once --dry-run | head
.venv/bin/openhop-influxdb --data-dir devdata

Grafana (admin/admin) comes pre-provisioned with data sources for both InfluxDB versions and the openHop Repeater dashboard (pick the data source at the top).

No repeater handy? dev/fake_repeater.py serves the REST API, the /ws/packets WebSocket and the /api/logs_stream SSE feed with live synthetic data:

.venv/bin/python dev/fake_repeater.py --port 8765 --token dev-token
# add --radios 2 for a two-radio bridge stack (SP0.0 / SP0.1)
# then use "repeater_url": "http://127.0.0.1:8765", "repeater_api_token": "dev-token"

The fake keeps 35 days of packets, 30 days of adverts, 31 days of noise samples and a year of RRD data, so the history import can be exercised end to end.

Note: --dry-run and --backfill never save resume points, so a later service run still performs its own first-start import.

Licence

MIT. Includes MIT-licensed code from openhop-prometheus-plugin; see NOTICE.

S
Description
openHop Repeater plugin to feed metrics into InfluxDB
Readme MIT
340 KiB
0 Stars 1 Watchers 0 Forks
Languages
Python 99.8%
Shell 0.2%