Author SHA1 Message Date
rich fa9f44abbd List upstream changes since the previous release in the release notes
build / build (push) Successful in 26s
Upstream commits are linked to GitHub and split into code and build files
versus docs/tests/CI only, using the fingerprint's pattern (now passed in
build.env). The manifest records patch hashes so the next release can say
which Hazelwood patches were added, changed or removed.
2026-10-30 15:15:32 -04:00
rich 8d0dfec9f7 Skip the build when upstream changes no firmware code
build / build (push) Successful in 16m3s
ci-build.sh hashes the patched source tree without *.md, docs/, test/,
.github/ and other files no build reads, plus the targets and itself.
The workflow skips the build when the newest release's manifest has the
same code fingerprint. Comparing binaries can't do this: the -hz.U in
the version label counts every upstream commit.
2026-10-30 14:54:04 -04:00
rich eb9c3e67e6 Add patch 0005: -hz.U.L build identifier with upstream commit count
build / build (push) Successful in 16m25s
2026-10-28 22:01:29 -04:00
rich 4819dc277e Drop the actions/cache step and add the force input (completes a6858dd)
build / build (push) Successful in 21s
2026-10-28 19:15:34 -04:00
rich a6858ddfe7 Keep PlatformIO toolchains on a runner volume; add a force option to manual runs 2026-10-28 19:15:13 -04:00
9 changed files with 224 additions and 22 deletions

No files matched your search

+18 -12
View File
@@ -1,12 +1,19 @@
# Build upstream keymindCascade + patches/ and publish a release, but only
# when that combination hasn't been released yet. Runs daily so LoRa OTA
# deltas between consecutive releases stay small.
# when that combination hasn't been released yet and its firmware code differs
# from the newest release (upstream commits touching only docs, tests or CI
# are skipped). Runs daily so LoRa OTA deltas between consecutive releases
# stay small.
name: build
on:
schedule:
- cron: "17 7 * * *"
workflow_dispatch:
inputs:
force:
description: "Build even if this source is already released or its code is unchanged (a release is never replaced)"
type: boolean
default: false
push:
branches: [main]
paths: ["patches/**", "scripts/**", ".gitea/workflows/**"]
@@ -33,12 +40,17 @@ jobs:
scripts/ci-build.sh --check-only
cat out/build.env >> "$GITHUB_OUTPUT"
- name: Skip if already released
- name: Skip if already released or the code is unchanged
id: released
run: |
if python3 scripts/publish-release.py --exists "${{ steps.source.outputs.RELEASE_TAG }}"; then
if [ "${{ inputs.force }}" = "true" ]; then
echo "skip=false" >> "$GITHUB_OUTPUT"
elif python3 scripts/publish-release.py --exists "${{ steps.source.outputs.RELEASE_TAG }}"; then
echo "Release ${{ steps.source.outputs.RELEASE_TAG }} exists; nothing to build."
echo "skip=true" >> "$GITHUB_OUTPUT"
elif python3 scripts/publish-release.py --unchanged "${{ steps.source.outputs.CODE_FINGERPRINT }}"; then
echo "Upstream ${{ steps.source.outputs.UPSTREAM_COMMIT }} changes no firmware code since the newest release; nothing to build."
echo "skip=true" >> "$GITHUB_OUTPUT"
else
echo "skip=false" >> "$GITHUB_OUTPUT"
fi
@@ -50,14 +62,8 @@ jobs:
"$HOME/pio-venv/bin/pip" install --quiet platformio==6.2.0 intelhex==2.3.0 zopfli==0.4.3
echo "$HOME/pio-venv/bin" >> "$GITHUB_PATH"
- name: Restore PlatformIO toolchains
if: steps.released.outputs.skip != 'true'
uses: actions/cache@v4
with:
path: ~/.platformio
key: pio-${{ hashFiles('work/MeshCore/platformio.ini', 'work/MeshCore/variants/rak4631/platformio.ini', 'work/MeshCore/variants/heltec_v3/platformio.ini') }}
restore-keys: pio-
# ~/.platformio is a persistent volume on the runner VM (see the runner's
# container.options), so toolchains are only downloaded when they change.
- name: Build
if: steps.released.outputs.skip != 'true'
run: scripts/ci-build.sh
+16 -3
View File
@@ -12,11 +12,20 @@ Gitea releases. The Hazelwood mesh collector deploys them to the nodes.
`./local_build.sh build-firmware <env>` for each target.
2. `scripts/publish-release.py` converts every nRF52 `.uf2` to the `.hex`
delta base LoRa OTA needs, writes `manifest.json` (version, upstream and
source commits, OTA target id, and each image's `base_hash`), and creates
the release `build-<source commit 8>` with all artifacts.
source commits, OTA target id, each image's `base_hash`, and patch
hashes), and creates the release `build-<source commit 8>` with all
artifacts. The release notes list the upstream commits since the previous
release, linked to GitHub and split into code and build files versus
docs/tests/CI only (the fingerprint's rule below), plus a full-diff link and
which Hazelwood patches were added, changed or removed.
3. `.gitea/workflows/build.yml` runs daily, on manual dispatch and on pushes
that change patches or scripts. It skips the build when the release for
that source commit already exists.
that source commit already exists, or when the code fingerprint matches
the newest release's: a hash of the patched source tree without `*.md`,
`docs/`, `test/`, `.github/` and other files no build reads, plus the
targets and `ci-build.sh`. Upstream commits touching only those files
therefore publish nothing. (Comparing the binaries wouldn't work: the
`-hz.U` in the version label counts every upstream commit.)
Targets: `RAK_4631_repeater_unified_lora_ota` (trimmed, see patch 0004) and
`Heltec_v3_companion_radio_full_nomqtt`. Override with `TARGETS=...`.
@@ -41,3 +50,7 @@ fails and no release is published. `scripts/check_firmware_capabilities.py`
## Runner
`vm-gitrunner-p01` (Debian 13 VM, gitea-runner, Docker). One job at a time.
PlatformIO toolchains persist in `/var/lib/gitea-runner/platformio` on the VM,
mounted into every job as `/root/.platformio`. A manual run with **force**
rebuilds an already released or unchanged source (useful after runner changes); it never
replaces the existing release.
@@ -1,7 +1,7 @@
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: "Rich O." <rich@rickelobe.com>
Date: Tue, 13 Oct 2026 21:31:31 -0400
Subject: [PATCH 1/4] Add Hazelwood local build wrapper and Heltec no-MQTT
Subject: [PATCH 1/5] Add Hazelwood local build wrapper and Heltec no-MQTT
companion env
local_build.sh wraps build.sh with the Hazelwood defaults (919.5 MHz /
@@ -1,7 +1,7 @@
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: "Rich O." <rich@rickelobe.com>
Date: Tue, 13 Oct 2026 21:40:55 -0400
Subject: [PATCH 2/4] Count only firmware-code commits in the -hz.N build
Subject: [PATCH 2/5] Count only firmware-code commits in the -hz.N build
identifier
Merge commits and commits that touch only the build-setup files
@@ -1,7 +1,7 @@
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: "Rich O." <rich@rickelobe.com>
Date: Wed, 28 Oct 2026 10:41:32 -0400
Subject: [PATCH 3/4] Apply BLE_PIN_CODE only to companion builds
Subject: [PATCH 3/5] Apply BLE_PIN_CODE only to companion builds
---
local_build.sh | 10 ++++++++--
@@ -1,7 +1,7 @@
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: "Rich O." <rich@rickelobe.com>
Date: Wed, 28 Oct 2026 16:39:31 -0400
Subject: [PATCH 4/4] Trim unused features from the Hazelwood RAK4631 repeater
Subject: [PATCH 4/5] Trim unused features from the Hazelwood RAK4631 repeater
build
Drop display, GPS, INA, RS232, KISS host and every sensor driver except
@@ -0,0 +1,64 @@
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: "Rich O." <rich@rickelobe.com>
Date: Wed, 28 Oct 2026 22:00:48 -0400
Subject: [PATCH 5/5] Add upstream commits since the tag to the -hz build
identifier
Builds on newer upstream code kept the same -hz.N label because N counted
only local commits, so KM-Test's v1.18.0.2-hz.1 and the CI release of a
later upstream were indistinguishable. The identifier is now -hz.U.L: U
upstream commits past the version tag, L local firmware-code commits.
---
local_build.sh | 18 +++++++++++++-----
local_version.py | 2 +-
2 files changed, 14 insertions(+), 6 deletions(-)
diff --git a/local_build.sh b/local_build.sh
index ff30faeb4..73d66f912 100755
--- a/local_build.sh
+++ b/local_build.sh
@@ -68,9 +68,12 @@ cd "$(dirname "$0")"
# VERSION follows the fork's own numbering: the highest vX.Y.Z.W tag reachable
# from HEAD (e.g. v1.17.1.10 from rak4631-full-companion-v1.17.1.10-preview3).
# Local code changes add an identifier; build settings (untracked files) don't:
-# -hz.N N of your own commits that change firmware code relative to the
-# maintainer's fork (upstream/keymindCascade; origin is the personal
-# Gitea copy, so @{upstream} would hide them)
+# -hz.U.L U upstream commits past the vX.Y.Z.W tag(s) in the maintainer's fork
+# (upstream/keymindCascade; origin is the personal Gitea copy, so
+# @{upstream} would hide them) and L of your own commits that change
+# firmware code. U grows with every upstream pull, so builds on newer
+# upstream code sort higher even when the tag and L are unchanged.
+# LoRa OTA sees only vX.Y.Z.W; compare U.L to decide an upgrade.
# .dirty uncommitted edits to tracked firmware code
# Merge commits and the build-setup files below don't count: they change
# settings, not code. Edit LOCAL_FW_SETUP_PATHS if you add more setup files.
@@ -90,8 +93,13 @@ LOCAL_FW_SETUP_PATHS=":(exclude)local_build.sh :(exclude)local_version.py :(excl
# shellcheck disable=SC2086 # intentional word-splitting of the pathspec list
local_commits=$(git rev-list --count --no-merges "${LOCAL_FW_UPSTREAM_REF}..HEAD" \
-- . $LOCAL_FW_SETUP_PATHS 2>/dev/null || echo 0)
-if [ "$local_commits" -gt 0 ]; then
- local_id="-hz.${local_commits}"
+# Every tag carrying this version is excluded, so U counts from the newest of them.
+upstream_base=$(git merge-base HEAD "${LOCAL_FW_UPSTREAM_REF}" 2>/dev/null || git rev-parse HEAD)
+# shellcheck disable=SC2046 # one ^tag argument per tag
+upstream_commits=$(git rev-list --count "$upstream_base" \
+ $(git tag --merged "$upstream_base" | grep -F "v${fork_version}" | sed 's/^/^/') 2>/dev/null || echo 0)
+if [ "$upstream_commits" -gt 0 ] || [ "$local_commits" -gt 0 ]; then
+ local_id="-hz.${upstream_commits}.${local_commits}"
fi
# shellcheck disable=SC2086
if [ -n "$(git status --porcelain --untracked-files=no -- . $LOCAL_FW_SETUP_PATHS)" ]; then
diff --git a/local_version.py b/local_version.py
index fbd6c069f..cd6a84079 100644
--- a/local_version.py
+++ b/local_version.py
@@ -4,7 +4,7 @@
#
# repeater/room/sensor: hazelwood_repeater v1.17.1.10 (core v1.17.1)
# companion: v1.17.1.10 hazelwood (core v1.17.1)
-# with local code changes the version gains -hz.N and/or .dirty (local_build.sh)
+# past the tag the version gains -hz.U.L and/or .dirty (local_build.sh)
#
# Companions keep the version first because the app's device-info frame
# carries only the first 19 characters (companion MyMesh.cpp strzcpy(..., 20)).
+20
View File
@@ -40,11 +40,31 @@ export GIT_COMMITTER_NAME="Hazelwood CI" GIT_COMMITTER_EMAIL="ci@hazelwood.inval
git -C "$src" am --quiet --committer-date-is-author-date "$here"/patches/*.patch
source_commit=$(git -C "$src" rev-parse HEAD)
# Fingerprint of everything that can change the binaries: the patched source
# tree minus files no build reads, plus the targets and this script. Upstream
# commits that touch only docs, tests or CI keep it, and the workflow then
# skips the build (publish-release.py --unchanged). The version label can't be
# compared instead: -hz.U counts every upstream commit, so it changes anyway.
# publish-release.py uses the same pattern to sort the release notes' changes.
noncode_re='(.*\.md|(docs|test|\.github|\.cursor|\.devcontainer|\.meshlog-review)(/.*)?)'
code_fingerprint=$(
{
# ls-tree takes no exclude pathspecs; lines are "<mode> <type> <id>\t<path>".
git -C "$src" ls-tree -r "$source_commit" | grep -vE $'\t'"$noncode_re\$"
echo "TARGETS=$TARGETS"
cat "$here/scripts/ci-build.sh"
} | sha256sum | cut -c1-16
)
{
echo "UPSTREAM_URL=$UPSTREAM_URL"
echo "UPSTREAM_COMMIT=$upstream_commit"
echo "SOURCE_COMMIT=$source_commit"
echo "CODE_FINGERPRINT=$code_fingerprint"
echo "RELEASE_TAG=build-${source_commit:0:8}"
echo "TARGETS=$TARGETS"
echo "SOURCE_DIR=$src"
echo "NONCODE_RE=$noncode_re"
} > "$OUT_DIR/build.env"
cat "$OUT_DIR/build.env"
[ "$check_only" -eq 1 ] && exit 0
+102 -3
View File
@@ -9,6 +9,8 @@ Usage:
publish-release.py OUT_DIR --manifest-only write manifest.json and .hex files
publish-release.py OUT_DIR --publish also create the release
publish-release.py --exists TAG exit 0 if the release exists, 1 if not
publish-release.py --unchanged FINGERPRINT exit 0 if the newest release has this
code fingerprint (ci-build.sh), 1 if not
Publishing reads GITEA_SERVER_URL, GITEA_REPOSITORY and GITEA_TOKEN.
"""
@@ -18,7 +20,9 @@ import argparse
import hashlib
import json
import os
import re
import struct
import subprocess
import sys
import urllib.error
import urllib.parse
@@ -27,6 +31,8 @@ import uuid
from datetime import datetime, timezone
from pathlib import Path
HERE = Path(__file__).resolve().parent.parent
MAX_CHANGES = 100 # commits listed per section of the release notes
UF2_MAGIC = (0x0A324655, 0x9E5D5157)
UF2_NOT_MAIN_FLASH = 0x00000001
ENDF_TRAILER = 56
@@ -85,6 +91,9 @@ def build_manifest(out_dir: Path) -> tuple[dict, list[Path]]:
"firmware_version": env.get("FIRMWARE_VERSION", ""),
"upstream_commit": env["UPSTREAM_COMMIT"],
"source_commit": env["SOURCE_COMMIT"],
"code_fingerprint": env.get("CODE_FINGERPRINT", ""),
# So the next release can say which patches changed.
"patches": {p.name: sha256_file(p) for p in sorted((HERE / "patches").glob("*.patch"))},
"built_at": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
"targets": {},
}
@@ -133,6 +142,91 @@ def release_exists(tag: str) -> bool:
raise
def newest_manifest() -> dict | None:
"""manifest.json of the newest release, or None if there is none."""
repo = os.environ["GITEA_REPOSITORY"]
releases = [r for r in api("GET", f"/repos/{repo}/releases?limit=50") if not r.get("draft")]
if not releases:
return None
# Not created_at: releases from one tag push can share it. The id breaks ties.
newest = max(releases, key=lambda r: (r.get("published_at") or r.get("created_at", ""), r.get("id", 0)))
if not any(a["name"] == "manifest.json" for a in newest.get("assets", [])):
return None
url = (os.environ["GITEA_SERVER_URL"].rstrip("/") + f"/{repo}/releases/download/"
+ urllib.parse.quote(newest["tag_name"]) + "/manifest.json")
req = urllib.request.Request(url)
req.add_header("Authorization", "token " + os.environ["GITEA_TOKEN"])
with urllib.request.urlopen(req, timeout=60) as resp:
return json.loads(resp.read())
def newest_fingerprint() -> str | None:
"""code_fingerprint of the newest release, or None if it has none."""
manifest = newest_manifest()
if manifest is None:
return None
print(f"newest release {manifest.get('release_tag')}: code fingerprint "
f"{manifest.get('code_fingerprint') or 'none'}")
return manifest.get("code_fingerprint") or None
def git(src: str, *args: str) -> subprocess.CompletedProcess:
return subprocess.run(["git", "-C", src, *args], capture_output=True, text=True)
def upstream_commits(src: str, old: str, new: str, noncode: re.Pattern) -> list[tuple[str, str, bool]]:
"""(commit, subject, touches firmware code) for each non-merge commit old..new, newest first."""
out = git(src, "log", "--no-merges", "--name-only", "--format=%x00%H%x09%s", f"{old}..{new}")
out.check_returncode()
commits = []
for record in out.stdout.split("\0")[1:]:
head, _, files = record.partition("\n")
sha, _, subject = head.partition("\t")
paths = [f for f in files.splitlines() if f]
commits.append((sha, subject, any(not noncode.fullmatch(f) for f in paths)))
return commits
def changes_section(manifest: dict, env: dict[str, str], prev: dict | None) -> list[str]:
"""Release-notes lines: upstream commits and patch changes since the previous release."""
if prev is None:
return ["### Changes", "", "First release.", ""]
lines = [f"### Changes since {prev.get('firmware_version', '')} ({prev.get('release_tag', '')})", ""]
web = env.get("UPSTREAM_URL", "").removesuffix(".git")
old, new = prev.get("upstream_commit", ""), manifest["upstream_commit"]
src = env.get("SOURCE_DIR", "")
def link(sha: str) -> str:
return f"[`{sha[:9]}`]({web}/commit/{sha})" if web.startswith("https://") else f"`{sha[:9]}`"
if old == new:
lines += ["No upstream changes.", ""]
elif not (src and old and git(src, "merge-base", "--is-ancestor", old, new).returncode == 0):
lines += [f"Upstream `{old[:9]}` is not an ancestor of `{new[:9]}` (history rewritten?); "
"commits can't be listed.", ""]
else:
commits = upstream_commits(src, old, new, re.compile(env["NONCODE_RE"]))
for title, code in (("Code and build files", True), ("Docs, tests and CI only", False)):
group = [c for c in commits if c[2] == code]
if not group:
continue
lines.append(f"**{title}** ({len(group)})")
lines += [f"- {link(sha)} {subject}" for sha, subject, _ in group[:MAX_CHANGES]]
if len(group) > MAX_CHANGES:
lines.append(f"- … and {len(group) - MAX_CHANGES} more")
lines.append("")
if web.startswith("https://") and old and old != new:
lines += [f"[Full upstream diff]({web}/compare/{old}...{new})", ""]
if "patches" in prev:
before, after = prev["patches"], manifest["patches"]
notes = ([f"added `{n}`" for n in after if n not in before]
+ [f"changed `{n}`" for n in after if n in before and before[n] != after[n]]
+ [f"removed `{n}`" for n in before if n not in after])
lines += ["**Hazelwood patches:** " + ("; ".join(notes) if notes else "unchanged"), ""]
return lines
def upload(release_id: int, path: Path) -> None:
repo = os.environ["GITEA_REPOSITORY"]
boundary = uuid.uuid4().hex
@@ -144,7 +238,7 @@ def upload(release_id: int, path: Path) -> None:
f"multipart/form-data; boundary={boundary}")
def publish(manifest: dict, assets: list[Path]) -> None:
def publish(manifest: dict, assets: list[Path], env: dict[str, str]) -> None:
repo = os.environ["GITEA_REPOSITORY"]
tag = manifest["release_tag"]
if release_exists(tag):
@@ -152,13 +246,15 @@ def publish(manifest: dict, assets: list[Path]) -> None:
return
lines = [f"Firmware **{manifest['firmware_version']}**", "",
f"- upstream: `{manifest['upstream_commit']}`",
f"- source (upstream + patches): `{manifest['source_commit']}`", ""]
f"- source (upstream + patches): `{manifest['source_commit']}`",
f"- code fingerprint: `{manifest['code_fingerprint'] or 'none'}`", ""]
for target, info in manifest["targets"].items():
lines.append(f"**{target}** (OTA target {info['target_id']})")
for a in info["artifacts"]:
if "base_hash" in a:
lines.append(f"- {a.get('variant', '')} base_hash `{a['base_hash']}` ({a['body_len']} B)")
lines.append("")
lines += changes_section(manifest, env, newest_manifest())
payload = {
"tag_name": tag,
"target_commitish": os.environ.get("GITHUB_SHA", ""),
@@ -181,16 +277,19 @@ def main() -> int:
mode.add_argument("--manifest-only", action="store_true")
mode.add_argument("--publish", action="store_true")
mode.add_argument("--exists", metavar="TAG")
mode.add_argument("--unchanged", metavar="FINGERPRINT")
args = parser.parse_args()
if args.exists:
return 0 if release_exists(args.exists) else 1
if args.unchanged is not None:
return 0 if args.unchanged and newest_fingerprint() == args.unchanged else 1
if args.out_dir is None:
parser.error("OUT_DIR is required")
manifest, assets = build_manifest(args.out_dir)
print(json.dumps({t: [a for a in i["artifacts"] if "base_hash" in a]
for t, i in manifest["targets"].items()}, indent=2))
if args.publish:
publish(manifest, assets)
publish(manifest, assets, read_env(args.out_dir / "build.env"))
return 0