Support private text with one known public key

This commit is contained in:
mikecarper committed 2026-09-28 00:19:31 -07:00
1 parent fd7b93c52d
commit ea4917b322
7 files changed
+377 -8

No files matched your search

+29
View File
@@ -242,6 +242,35 @@ catalog; bytes `0x2C`-`0x31` are parked and `0x35` is unused.
The sections below detail the most common frames. Refer to the source named
above for command bodies that are not expanded here.
### Private text when only the sender knows the recipient
Clients continue to send an ordinary `CMD_SEND_TXT_MSG` (`0x02`) with
`TXT_TYPE_PLAIN`; no client protocol change is needed. For a chat contact
without a recent confirmed DM, Companion firmware first sends a signed,
encrypted [one-key introduction](payloads.md#companion-one-key-dm-introduction)
that includes its full public key, then queues the normal private text after
the introduction. It sends the introduction again on an application retry,
which recovers if the recipient removed the sender contact. An ACK confirms
that the recipient can decrypt normal DMs, so subsequent first attempts omit
the extra packet until the sender reboots or the small recent-ACK cache is
replaced.
The receiving Companion verifies the introduction signature and adds the
sender as a chat contact even when advert auto-add is off. It emits the usual
`PUSH_CODE_NEW_ADVERT` contact frame before the normal private-message frame,
so existing clients can associate the six-byte message prefix with the full
key. A recipient running older firmware still needs the sender contact before
it can decrypt a normal DM. This flow uses a second radio packet for the
initial DM; `RESP_CODE_SENT` includes that scheduling delay in its timeout.
For a two-radio regression test, run `tools/hil/one_key_dm.py` with the USB
serial paths for `--sender` and `--recipient`. `--reset-contact` removes the
sender from recipient contacts, and `--zero-hop` sets the sender's route to
zero hop when the devices are nearby. The test checks contact creation,
message waiting, and the matching ACK without draining the offline queue.
`--invalid-signature-first` also checks that a decryptable introduction with
an invalid Ed25519 signature does not add a contact.
`CMD_RUN_CLI_COMMAND` is followed by the local CLI text without a terminating
NUL. The device returns `RESP_CODE_CLI_REPLY` (`0x1D`) followed by the reply
text. This is separate from sending a remote on-air CLI command with
+23
View File
@@ -231,6 +231,29 @@ same-timestamp mismatches are rejected.
| cipher MAC | 2 | MAC for encrypted data in next field |
| ciphertext | rest of payload | encrypted message, see below for details |
### Companion one-key DM introduction
A Companion may send this request immediately before a normal private text
packet when it has not received an ACK from that chat contact, and on each
application retry. It lets the recipient learn the sender's full public key
without a broadcast advert. The ciphertext contains:
| Field | Size (bytes) | Description |
|---|---:|---|
| tag | 4 | unique sender timestamp |
| marker | 4 | ASCII `DMK1` |
| name | 1–32 | sender name followed by NUL |
| signature | 64 | Ed25519 signature by the sender |
The signed bytes are the recipient's full public key followed by the tag,
marker, and NUL-terminated name. The recipient verifies that signature against
the 32-byte sender key in the anonymous request header before creating a chat
contact. A receiver that does not implement `DMK1` ignores the introduction;
the following normal text packet still works when it already has the sender's
key. The recipient's manual advert auto-add setting does not suppress a
verified, addressed `DMK1` introduction. The sender's full public key remains
visible in the radio packet header, as with other anonymous requests.
### Room server login
| Field | Size (bytes) | Description |
+117 -2
View File
@@ -217,6 +217,11 @@ static constexpr uint8_t DEFAULT_FEM_RX_GAIN = 1;
#define CONTACT_PAGE_WRITE_GAP 100
#define EXPECTED_ACK_RETRY_RECHECK_MILLIS 1000
// The encrypted body of an ANON_REQ starts with its usual four-byte tag.
// A distinct marker after that tag prevents a contact introduction from being
// mistaken for a login or another anonymous request.
static constexpr uint8_t ONE_KEY_INTRO_MARKER[] = {'D', 'M', 'K', '1'};
static bool save_filter(const ContactInfo& c);
#ifndef DEFAULT_MULTI_ACKS
@@ -990,6 +995,7 @@ ContactInfo* MyMesh::processAck(const uint8_t *data) {
// NOTE: the same ACK can be received multiple times!
ContactInfo* contact = expected_ack_table[i].contact;
if (contact != NULL && contact->type == ADV_TYPE_CHAT) rememberOneKeyAck(*contact);
clearExpectedAck(expected_ack_table[i]);
expireExpectedAcks();
return contact;
@@ -998,6 +1004,108 @@ ContactInfo* MyMesh::processAck(const uint8_t *data) {
return checkConnectionsAck(data);
}
bool MyMesh::hasOneKeyAck(const ContactInfo& contact) const {
for (uint8_t i = 0; i < one_key_acked_count; ++i) {
if (memcmp(one_key_acked_keys[i], contact.id.pub_key, PUB_KEY_SIZE) == 0) return true;
}
return false;
}
void MyMesh::rememberOneKeyAck(const ContactInfo& contact) {
if (hasOneKeyAck(contact)) return;
memcpy(one_key_acked_keys[one_key_acked_next], contact.id.pub_key, PUB_KEY_SIZE);
if (one_key_acked_count < ONE_KEY_ACKED_PEERS) ++one_key_acked_count;
one_key_acked_next = (one_key_acked_next + 1) % ONE_KEY_ACKED_PEERS;
}
uint32_t MyMesh::sendOneKeyIntroduction(const ContactInfo& contact) {
// ANON_REQ carries the sender's full key in its outer header. It therefore
// reaches a recipient that cannot yet decrypt a normal text packet, while
// old Companion firmware simply ignores this unfamiliar request body.
uint8_t body[4 + sizeof(ONE_KEY_INTRO_MARKER) + 32 + SIGNATURE_SIZE];
const uint32_t tag = getRTCClock()->getCurrentTimeUnique();
memcpy(body, &tag, sizeof(tag));
memcpy(body + 4, ONE_KEY_INTRO_MARKER, sizeof(ONE_KEY_INTRO_MARKER));
char fallback_name[20];
snprintf(fallback_name, sizeof(fallback_name), "Unknown %02X%02X%02X",
self_id.pub_key[0], self_id.pub_key[1], self_id.pub_key[2]);
const char* name = _prefs.node_name[0] ? _prefs.node_name : fallback_name;
const size_t name_len = strnlen(name, 31);
memcpy(body + 4 + sizeof(ONE_KEY_INTRO_MARKER), name, name_len);
body[4 + sizeof(ONE_KEY_INTRO_MARKER) + name_len] = 0;
const size_t signed_body_len = 4 + sizeof(ONE_KEY_INTRO_MARKER) + name_len + 1;
uint8_t signed_message[PUB_KEY_SIZE + 4 + sizeof(ONE_KEY_INTRO_MARKER) + 32];
memcpy(signed_message, contact.id.pub_key, PUB_KEY_SIZE);
memcpy(signed_message + PUB_KEY_SIZE, body, signed_body_len);
self_id.sign(body + signed_body_len, signed_message,
PUB_KEY_SIZE + signed_body_len);
mesh::Packet* intro = createAnonDatagram(
PAYLOAD_TYPE_ANON_REQ, self_id, contact.id,
contact.getSharedSecret(self_id), body,
signed_body_len + SIGNATURE_SIZE);
if (intro == NULL) return 0;
const uint32_t airtime = getTransmitAirtime(intro);
uint32_t delay;
bool sent;
if (contact.out_path_len == OUT_PATH_UNKNOWN) {
delay = calcFloodTimeoutMillisFor(airtime) / 2;
sent = sendFloodScoped(contact, intro);
} else {
const uint32_t hops = (contact.out_path_len & 63) + 1;
delay = (airtime * 3 + 500) * hops;
sent = sendDirect(intro, contact.getPath(), contact.out_path_len);
}
// Scheduling the text after the introduction also lets the ordinary text
// path work on older recipients which already know this sender.
return sent ? delay : 0;
}
void MyMesh::onAnonDataRecv(mesh::Packet* packet, const uint8_t* secret,
const mesh::Identity& sender, uint8_t* data,
size_t len) {
(void)secret;
if (packet->getPayloadType() != PAYLOAD_TYPE_ANON_REQ
|| len < 4 + sizeof(ONE_KEY_INTRO_MARKER) + 2 + SIGNATURE_SIZE
|| memcmp(data + 4, ONE_KEY_INTRO_MARKER,
sizeof(ONE_KEY_INTRO_MARKER)) != 0
|| !canMutateContacts()) return;
const char* name = reinterpret_cast<const char*>(data + 4 + sizeof(ONE_KEY_INTRO_MARKER));
const size_t available = min(size_t(32), len - 4 - sizeof(ONE_KEY_INTRO_MARKER) - SIGNATURE_SIZE);
const char* end = static_cast<const char*>(memchr(name, 0, available));
if (end == NULL || end == name || size_t(end - name) >= sizeof(ContactInfo::name)) return;
const size_t signed_body_len = 4 + sizeof(ONE_KEY_INTRO_MARKER) + size_t(end - name) + 1;
if (signed_body_len + SIGNATURE_SIZE > len) return;
uint8_t signed_message[PUB_KEY_SIZE + 4 + sizeof(ONE_KEY_INTRO_MARKER) + 32];
memcpy(signed_message, self_id.pub_key, PUB_KEY_SIZE);
memcpy(signed_message + PUB_KEY_SIZE, data, signed_body_len);
if (!sender.verify(data + signed_body_len, signed_message,
PUB_KEY_SIZE + signed_body_len)) return;
if (lookupPersistentContactByPubKey(sender.pub_key, PUB_KEY_SIZE) != NULL) return;
ContactInfo contact;
memcpy(contact.id.pub_key, sender.pub_key, PUB_KEY_SIZE);
StrHelper::strncpy(contact.name, name, sizeof(contact.name));
contact.type = ADV_TYPE_CHAT;
contact.out_path_len = OUT_PATH_UNKNOWN;
contact.lastmod = getRTCClock()->getCurrentTime();
if (!addContact(contact)) {
onContactsFull();
return;
}
ContactInfo* added = lookupPersistentContactByPubKey(sender.pub_key, PUB_KEY_SIZE);
if (added == NULL || !scheduleContactWrite(*added)) {
if (added != NULL) removeContact(*added);
return;
}
// A prior anonymous service request may occupy a transient slot for this
// key. Remove it so regular text lookup finds the durable contact first.
ContactInfo* transient = lookupTransientContactByPubKey(sender.pub_key, PUB_KEY_SIZE);
if (transient != NULL) clearTransientContact(*transient);
onDiscoveredContact(*added, true, OUT_PATH_UNKNOWN, NULL);
}
void MyMesh::queueMessage(const ContactInfo &from, uint8_t txt_type, mesh::Packet *pkt,
uint32_t sender_timestamp, const uint8_t *extra,
int extra_len, const char *text,
@@ -4605,6 +4713,7 @@ void MyMesh::handleCmdFrame(size_t len) {
int result;
uint32_t expected_ack;
uint32_t one_key_delay = 0;
if (txt_type == TXT_TYPE_CLI_DATA || txt_type == TXT_TYPE_CLI_COMMAND) {
const uint32_t logical_request_id = msg_timestamp;
msg_timestamp = getRTCClock()->getCurrentTimeUnique(); // Use node's RTC instead of app timestamp to avoid tripping replay protection
@@ -4623,8 +4732,14 @@ void MyMesh::handleCmdFrame(size_t len) {
msg_timestamp = getRTCClock()->getCurrentTimeUnique();
}
replacement_entry = findPendingTextMessage(text_fingerprint, msg_timestamp);
if (recipient->type == ADV_TYPE_CHAT
&& (attempt != 0 || !hasOneKeyAck(*recipient))
&& tlen <= MAX_TEXT_LEN
&& (attempt <= 3 || tlen <= MAX_TEXT_LEN - 2)) {
one_key_delay = sendOneKeyIntroduction(*recipient);
}
result = sendMessage(*recipient, msg_timestamp, attempt, text, expected_ack, est_timeout,
packet_retry_key, NULL, text_fingerprint);
packet_retry_key, NULL, text_fingerprint, one_key_delay);
if (result != MSG_SEND_FAILED && is_room_post) {
room_message_timestamps.remember(text_fingerprint, app_timestamp,
msg_timestamp);
@@ -4655,7 +4770,7 @@ void MyMesh::handleCmdFrame(size_t len) {
// ownership lives in Mesh and ends only on an ACK, retry completion,
// or a successfully queued semantic replacement.
clearExpectedAck(entry, false);
entry.msg_sent = _ms->getMillis(); // add to circular table
entry.msg_sent = _ms->getMillis() + one_key_delay; // scheduled radio send
entry.expires_at = futureMillis(est_timeout);
entry.ack = expected_ack;
entry.message_timestamp = msg_timestamp;
+12
View File
@@ -317,6 +317,9 @@ protected:
void onContactVisit(const ContactInfo& contact) override;
#endif
ContactInfo* processAck(const uint8_t *data) override;
void onAnonDataRecv(mesh::Packet* packet, const uint8_t* secret,
const mesh::Identity& sender, uint8_t* data,
size_t len) override;
void queueMessage(const ContactInfo &from, uint8_t txt_type, mesh::Packet *pkt, uint32_t sender_timestamp,
const uint8_t *extra, int extra_len, const char *text,
bool terminal_command_reply=false,
@@ -681,6 +684,15 @@ private:
};
#define EXPECTED_ACK_TABLE_SIZE 8
AckTableEntry expected_ack_table[EXPECTED_ACK_TABLE_SIZE]; // circular table
// A recent ACK proves that peer can decrypt our normal text packets. Retry
// attempts still send the introduction in case its contact was later erased.
static constexpr uint8_t ONE_KEY_ACKED_PEERS = 8;
uint8_t one_key_acked_keys[ONE_KEY_ACKED_PEERS][PUB_KEY_SIZE] = {};
uint8_t one_key_acked_count = 0;
uint8_t one_key_acked_next = 0;
bool hasOneKeyAck(const ContactInfo& contact) const;
void rememberOneKeyAck(const ContactInfo& contact);
uint32_t sendOneKeyIntroduction(const ContactInfo& contact);
mesh::LogicalMessageCache<ROOM_MESSAGE_TIMESTAMP_CACHE_SIZE> room_message_timestamps;
int next_ack_idx;
unsigned long next_ack_expiry;
+6 -5
View File
@@ -570,7 +570,8 @@ mesh::Packet* BaseChatMesh::composeMsgPacket(const ContactInfo& recipient, uint3
int BaseChatMesh::sendMessage(const ContactInfo& recipient, uint32_t timestamp, uint8_t attempt, const char* text,
uint32_t& expected_ack, uint32_t& est_timeout, uint8_t* packet_hash,
const uint8_t* replace_retry_key,
const uint8_t* message_retry_key) {
const uint8_t* message_retry_key,
uint32_t delay_millis) {
mesh::Packet* pkt = composeMsgPacket(recipient, timestamp, attempt, text, expected_ack);
if (pkt == NULL) return MSG_SEND_FAILED;
if (packet_hash != NULL) {
@@ -588,12 +589,12 @@ int BaseChatMesh::sendMessage(const ContactInfo& recipient, uint32_t timestamp,
int rc;
bool sent;
if (recipient.out_path_len == OUT_PATH_UNKNOWN) {
sent = sendFloodScoped(recipient, pkt);
est_timeout = calcFloodTimeoutMillisFor(t);
sent = sendFloodScoped(recipient, pkt, delay_millis);
est_timeout = delay_millis + calcFloodTimeoutMillisFor(t);
rc = MSG_SEND_SENT_FLOOD;
} else {
sent = sendDirect(pkt, recipient.getPath(), recipient.out_path_len);
est_timeout = calcDirectTimeoutMillisFor(t, recipient.out_path_len);
sent = sendDirect(pkt, recipient.getPath(), recipient.out_path_len, delay_millis);
est_timeout = delay_millis + calcDirectTimeoutMillisFor(t, recipient.out_path_len);
rc = MSG_SEND_SENT_DIRECT;
}
if (!sent) {
+2 -1
View File
@@ -199,7 +199,8 @@ public:
int sendMessage(const ContactInfo& recipient, uint32_t timestamp, uint8_t attempt, const char* text,
uint32_t& expected_ack, uint32_t& est_timeout, uint8_t* packet_hash = NULL,
const uint8_t* replace_retry_key = NULL,
const uint8_t* message_retry_key = NULL);
const uint8_t* message_retry_key = NULL,
uint32_t delay_millis = 0);
int sendCommandData(const ContactInfo& recipient, uint32_t timestamp,
uint8_t attempt, uint8_t txt_type, const char* text,
uint32_t& est_timeout,
+188
View File
@@ -0,0 +1,188 @@
#!/usr/bin/env python3
"""Exercise a private DM when only its sender knows the recipient's key.
This test changes the two radios' contact lists and transmits one LoRa DM.
It leaves the recipient's offline message queue untouched. Use two nearby
Companion USB radios on the same radio profile, with no other client attached.
"""
from __future__ import annotations
import argparse
import json
import struct
import time
import serial
from esp32_companion_serial_stress import (
DeviceFrameReader,
FrameTimeout,
TransportCounters,
encode_host_frame,
make_app_start_spec,
validate_app_start_response,
)
class Link:
def __init__(self, port: str) -> None:
self.port = serial.Serial(port, 115200, timeout=0.1, write_timeout=2)
self.reader = DeviceFrameReader(TransportCounters())
self.pushes: list[bytes] = []
def close(self) -> None:
self.port.close()
def read(self, seconds: float) -> bytes | None:
try:
frame = self.reader.read_frame(self.port, seconds)
except FrameTimeout:
return None
if frame[0] >= 0x80:
self.pushes.append(frame)
return frame
def request(self, payload: bytes, expected: tuple[int, ...], seconds: float = 5) -> bytes:
self.port.write(encode_host_frame(payload))
self.port.flush()
until = time.monotonic() + seconds
while time.monotonic() < until:
frame = self.read(min(0.5, until - time.monotonic()))
if frame is None or frame[0] >= 0x80:
continue
if frame[0] not in expected:
raise RuntimeError(
f"unexpected Companion response {frame[0]}"
+ (f", error {frame[1]}" if frame[0] == 1 and len(frame) > 1 else "")
)
return frame
raise TimeoutError(f"Companion request {payload[0]} timed out")
def start(self) -> tuple[bytes, dict]:
# Some USB Companions need a fresh CDC open after DFU activation.
for attempt in range(3):
try:
frame = self.request(make_app_start_spec("OneKeyDM-HIL").payload,
(5,), seconds=5)
break
except TimeoutError:
if attempt == 2:
raise
self.port.close()
time.sleep(1)
self.port.open()
self.reader = DeviceFrameReader(TransportCounters())
return frame[4:36], validate_app_start_response(frame)
def contact(self, key: bytes) -> bytes | None:
frame = self.request(bytes([30]) + key, (1, 3))
return frame if frame[0] == 3 else None
def run(sender_port: str, recipient_port: str, reset_contact: bool,
zero_hop: bool, invalid_signature_first: bool) -> dict:
sender = Link(sender_port)
recipient = Link(recipient_port)
try:
sender_key, sender_info = sender.start()
recipient_key, recipient_info = recipient.start()
radio_fields = ("frequency_khz", "bandwidth_hz", "spreading_factor", "coding_rate")
if any(sender_info[field] != recipient_info[field] for field in radio_fields):
raise RuntimeError("radios do not use the same frequency, bandwidth, SF, and CR")
if recipient.contact(sender_key) is not None:
if not reset_contact:
raise RuntimeError("recipient already knows sender; pass --reset-contact to remove it")
recipient.request(bytes([15]) + sender_key, (0,))
if recipient.contact(sender_key) is not None:
raise RuntimeError("recipient still knows sender")
contact = sender.contact(recipient_key)
if contact is None:
advert = recipient.request(bytes([17]), (11,))[1:]
sender.request(bytes([18]) + advert, (0,))
for _ in range(10):
time.sleep(0.25)
contact = sender.contact(recipient_key)
if contact is not None:
break
if contact is None:
raise RuntimeError("sender could not import recipient's signed advert")
if zero_hop:
# Contact response and CMD_ADD_UPDATE_CONTACT share the same body.
update = bytearray(contact)
update[0] = 9
update[35] = 0 # zero path hashes
update[36:100] = bytes(64)
sender.request(bytes(update), (0,))
contact = sender.contact(recipient_key)
if contact is None:
raise RuntimeError("sender contact vanished")
if invalid_signature_first:
# CMD_SEND_ANON_REQ prepends the four-byte tag itself. This has a
# valid ECDH envelope but an invalid Ed25519 identity signature.
bogus = b"DMK1forged\0" + bytes(64)
sender.request(bytes([0x39]) + recipient_key + bogus, (6,), seconds=20)
time.sleep(2)
if recipient.contact(sender_key) is not None:
raise AssertionError("invalid introduction created a contact")
timestamp = int(time.time())
text = f"one-key DM HIL {timestamp}".encode()
message = bytes([2, 0, 0]) + struct.pack("<I", timestamp) + recipient_key[:6] + text
sent = sender.request(message, (6,), seconds=20)
expected_ack = sent[2:6]
timeout_ms = struct.unpack("<I", sent[6:10])[0]
until = time.monotonic() + min(timeout_ms / 1000 + 5, 120)
while time.monotonic() < until:
for link in (sender, recipient):
link.read(0.1)
confirmed = any(
frame[0] == 0x82 and frame[1:5] == expected_ack
for frame in sender.pushes
)
waiting = any(frame[0] == 0x83 for frame in recipient.pushes)
if confirmed and waiting:
break
confirmed = any(
frame[0] == 0x82 and frame[1:5] == expected_ack for frame in sender.pushes
)
waiting = any(frame[0] == 0x83 for frame in recipient.pushes)
learned = recipient.contact(sender_key) is not None
result = {
"sender_prefix": sender_key[:6].hex(),
"recipient_prefix": recipient_key[:6].hex(),
"route": "direct" if sent[1] == 0 else "flood",
"ack_confirmed": confirmed,
"recipient_message_waiting": waiting,
"recipient_learned_sender": learned,
"timeout_ms": timeout_ms,
}
if invalid_signature_first:
result["invalid_signature_rejected"] = True
if not (confirmed and waiting and learned):
raise AssertionError(json.dumps(result, sort_keys=True))
return result
finally:
sender.close()
recipient.close()
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--sender", required=True, help="sender Companion USB serial path")
parser.add_argument("--recipient", required=True, help="recipient Companion USB serial path")
parser.add_argument("--reset-contact", action="store_true",
help="remove the sender from recipient contacts before the test")
parser.add_argument("--zero-hop", action="store_true",
help="set a direct zero-hop path; use only when both radios are nearby")
parser.add_argument("--invalid-signature-first", action="store_true",
help="verify an encrypted introduction with a bad signature is rejected")
args = parser.parse_args()
print(json.dumps(run(args.sender, args.recipient, args.reset_contact,
args.zero_hop, args.invalid_signature_first),
sort_keys=True))