7 Commits
Author SHA1 Message Date
rich 172c53f46d Build from upstream's latest full release instead of the branch tip
build / build (push) Successful in 20s
ci-build.sh now defaults to UPSTREAM_REF=latest-release, resolved through
GitHub's releases/latest (which skips pre-releases and drafts); the
keymindCascade tip was 693 commits and 13 pre-releases past
v1.18.0.1-halo-keymind-cascade-dev-03214a16. A branch, tag or commit still
works, and build.env records UPSTREAM_TAG.

The workflow skips the build when the newest release's upstream commit isn't
an ancestor of the new one (publish-release.py --behind), so going from the
pre-release-based builds back to 03214a16 doesn't make -hz.U go backwards.
Builds resume once upstream publishes a full release past c12050f.
2026-11-05 21:18:11 -05:00
rich 07e4b7d170 Trim radio presets and OTA diagnostics from the RAK repeater; keep -hz.U increasing
build / build (push) Successful in 15m50s
Upstream 54d5d97c (build-e31012c3) grew the RAK4631 repeater image by 8.4 KB and
its delta from build-25f0a3ec to 369 KB, past the 232 KB in-place limit. Without
named radio presets and the optional OTA transfer diagnostics the delta is 163 KB.

-hz.U counted from the newest tag on the version, so the new
w12-v1.18.0.2-preview3 tag reset it from 146 to 5 and the collector saw the
release as older. U now counts from the earliest tag on the version (483 here).
2026-11-01 09:59:19 -05:00
rich fa9f44abbd List upstream changes since the previous release in the release notes
build / build (push) Successful in 26s
Upstream commits are linked to GitHub and split into code and build files
versus docs/tests/CI only, using the fingerprint's pattern (now passed in
build.env). The manifest records patch hashes so the next release can say
which Hazelwood patches were added, changed or removed.
2026-10-30 15:15:32 -04:00
rich 8d0dfec9f7 Skip the build when upstream changes no firmware code
build / build (push) Successful in 16m3s
ci-build.sh hashes the patched source tree without *.md, docs/, test/,
.github/ and other files no build reads, plus the targets and itself.
The workflow skips the build when the newest release's manifest has the
same code fingerprint. Comparing binaries can't do this: the -hz.U in
the version label counts every upstream commit.
2026-10-30 14:54:04 -04:00
rich eb9c3e67e6 Add patch 0005: -hz.U.L build identifier with upstream commit count
build / build (push) Successful in 16m25s
2026-10-28 22:01:29 -04:00
rich 4819dc277e Drop the actions/cache step and add the force input (completes a6858dd)
build / build (push) Successful in 21s
2026-10-28 19:15:34 -04:00
rich a6858ddfe7 Keep PlatformIO toolchains on a runner volume; add a force option to manual runs 2026-10-28 19:15:13 -04:00
9 changed files with 304 additions and 36 deletions

No files matched your search

+23 -13
View File
@@ -1,12 +1,20 @@
# Build upstream keymindCascade + patches/ and publish a release, but only
# when that combination hasn't been released yet. Runs daily so LoRa OTA
# deltas between consecutive releases stay small.
# Build upstream's latest full (non-pre-release) release + patches/ and publish
# a release, but only when that combination hasn't been released yet, doesn't
# go back past the upstream commit of the newest release, and its firmware
# code differs from the newest release (upstream commits touching only docs,
# tests or CI are skipped). Runs daily so new upstream releases are picked up
# promptly.
name: build
on:
schedule:
- cron: "17 7 * * *"
workflow_dispatch:
inputs:
force:
description: "Build even if this source is already released, is older than the newest release's upstream, or its code is unchanged (a release is never replaced)"
type: boolean
default: false
push:
branches: [main]
paths: ["patches/**", "scripts/**", ".gitea/workflows/**"]
@@ -33,12 +41,20 @@ jobs:
scripts/ci-build.sh --check-only
cat out/build.env >> "$GITHUB_OUTPUT"
- name: Skip if already released
- name: Skip if already released, older upstream, or the code is unchanged
id: released
run: |
if python3 scripts/publish-release.py --exists "${{ steps.source.outputs.RELEASE_TAG }}"; then
if [ "${{ inputs.force }}" = "true" ]; then
echo "skip=false" >> "$GITHUB_OUTPUT"
elif python3 scripts/publish-release.py --exists "${{ steps.source.outputs.RELEASE_TAG }}"; then
echo "Release ${{ steps.source.outputs.RELEASE_TAG }} exists; nothing to build."
echo "skip=true" >> "$GITHUB_OUTPUT"
elif python3 scripts/publish-release.py --behind "${{ steps.source.outputs.SOURCE_DIR }}" "${{ steps.source.outputs.UPSTREAM_COMMIT }}"; then
echo "Upstream ${{ steps.source.outputs.UPSTREAM_TAG }} ${{ steps.source.outputs.UPSTREAM_COMMIT }} is older than the newest release's; waiting for a newer upstream release."
echo "skip=true" >> "$GITHUB_OUTPUT"
elif python3 scripts/publish-release.py --unchanged "${{ steps.source.outputs.CODE_FINGERPRINT }}"; then
echo "Upstream ${{ steps.source.outputs.UPSTREAM_COMMIT }} changes no firmware code since the newest release; nothing to build."
echo "skip=true" >> "$GITHUB_OUTPUT"
else
echo "skip=false" >> "$GITHUB_OUTPUT"
fi
@@ -50,14 +66,8 @@ jobs:
"$HOME/pio-venv/bin/pip" install --quiet platformio==6.2.0 intelhex==2.3.0 zopfli==0.4.3
echo "$HOME/pio-venv/bin" >> "$GITHUB_PATH"
- name: Restore PlatformIO toolchains
if: steps.released.outputs.skip != 'true'
uses: actions/cache@v4
with:
path: ~/.platformio
key: pio-${{ hashFiles('work/MeshCore/platformio.ini', 'work/MeshCore/variants/rak4631/platformio.ini', 'work/MeshCore/variants/heltec_v3/platformio.ini') }}
restore-keys: pio-
# ~/.platformio is a persistent volume on the runner VM (see the runner's
# container.options), so toolchains are only downloaded when they change.
- name: Build
if: steps.released.outputs.skip != 'true'
run: scripts/ci-build.sh
+22 -5
View File
@@ -6,17 +6,30 @@ Gitea releases. The Hazelwood mesh collector deploys them to the nodes.
## How it works
1. `scripts/ci-build.sh` clones upstream at the tip of `keymindCascade`,
applies `patches/*.patch` with fixed committer data (so the same upstream
1. `scripts/ci-build.sh` clones upstream at its latest full GitHub release
(pre-releases are skipped; set `UPSTREAM_REF=keymindCascade` for the
branch tip), applies `patches/*.patch` with fixed committer data (so the same upstream
commit and patches always give the same source commit), and runs
`./local_build.sh build-firmware <env>` for each target.
2. `scripts/publish-release.py` converts every nRF52 `.uf2` to the `.hex`
delta base LoRa OTA needs, writes `manifest.json` (version, upstream and
source commits, OTA target id, and each image's `base_hash`), and creates
the release `build-<source commit 8>` with all artifacts.
source commits, OTA target id, each image's `base_hash`, and patch
hashes), and creates the release `build-<source commit 8>` with all
artifacts. The release notes list the upstream commits since the previous
release, linked to GitHub and split into code and build files versus
docs/tests/CI only (the fingerprint's rule below), plus a full-diff link and
which Hazelwood patches were added, changed or removed.
3. `.gitea/workflows/build.yml` runs daily, on manual dispatch and on pushes
that change patches or scripts. It skips the build when the release for
that source commit already exists.
that source commit already exists, when the newest release's upstream
commit isn't an ancestor of this one (so `-hz.U` never goes backwards,
e.g. when the newest release was built from a pre-release), or when the
code fingerprint matches
the newest release's: a hash of the patched source tree without `*.md`,
`docs/`, `test/`, `.github/` and other files no build reads, plus the
targets and `ci-build.sh`. Upstream commits touching only those files
therefore publish nothing. (Comparing the binaries wouldn't work: the
`-hz.U` in the version label counts every upstream commit.)
Targets: `RAK_4631_repeater_unified_lora_ota` (trimmed, see patch 0004) and
`Heltec_v3_companion_radio_full_nomqtt`. Override with `TARGETS=...`.
@@ -41,3 +54,7 @@ fails and no release is published. `scripts/check_firmware_capabilities.py`
## Runner
`vm-gitrunner-p01` (Debian 13 VM, gitea-runner, Docker). One job at a time.
PlatformIO toolchains persist in `/var/lib/gitea-runner/platformio` on the VM,
mounted into every job as `/root/.platformio`. A manual run with **force**
rebuilds an already released or unchanged source (useful after runner changes); it never
replaces the existing release.
@@ -1,7 +1,7 @@
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: "Rich O." <rich@rickelobe.com>
Date: Tue, 13 Oct 2026 21:31:31 -0400
Subject: [PATCH 1/4] Add Hazelwood local build wrapper and Heltec no-MQTT
Subject: [PATCH 1/5] Add Hazelwood local build wrapper and Heltec no-MQTT
companion env
local_build.sh wraps build.sh with the Hazelwood defaults (919.5 MHz /
@@ -1,7 +1,7 @@
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: "Rich O." <rich@rickelobe.com>
Date: Tue, 13 Oct 2026 21:40:55 -0400
Subject: [PATCH 2/4] Count only firmware-code commits in the -hz.N build
Subject: [PATCH 2/5] Count only firmware-code commits in the -hz.N build
identifier
Merge commits and commits that touch only the build-setup files
@@ -1,7 +1,7 @@
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: "Rich O." <rich@rickelobe.com>
Date: Wed, 28 Oct 2026 10:41:32 -0400
Subject: [PATCH 3/4] Apply BLE_PIN_CODE only to companion builds
Subject: [PATCH 3/5] Apply BLE_PIN_CODE only to companion builds
---
local_build.sh | 10 ++++++++--
@@ -1,7 +1,7 @@
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: "Rich O." <rich@rickelobe.com>
Date: Wed, 28 Oct 2026 16:39:31 -0400
Subject: [PATCH 4/4] Trim unused features from the Hazelwood RAK4631 repeater
Subject: [PATCH 4/5] Trim unused features from the Hazelwood RAK4631 repeater
build
Drop display, GPS, INA, RS232, KISS host and every sensor driver except
@@ -13,20 +13,20 @@ check_firmware_capabilities.py accepts capabilities listed in
MESHCORE_LOCAL_OMIT_CAPABILITIES as deliberately omitted, and fails if a
listed one is still present.
---
local_build.sh | 17 +++++++++++++++++
local_build.sh | 20 ++++++++++++++++++++
scripts/check_firmware_capabilities.py | 19 +++++++++++++++++++
2 files changed, 36 insertions(+)
2 files changed, 39 insertions(+)
diff --git a/local_build.sh b/local_build.sh
index 5f0987a0d..ff30faeb4 100755
index 5f0987a0d..839950b90 100755
--- a/local_build.sh
+++ b/local_build.sh
@@ -37,6 +37,23 @@ case "${2:-}" in
@@ -37,6 +37,26 @@ case "${2:-}" in
*companion*) LOCAL_BUILD_FLAGS="${LOCAL_BUILD_FLAGS} -DBLE_PIN_CODE=123456" ;;
esac
+# Hazelwood RAK4631 repeaters (KM-Test, Treebeard): no display, GPS, INA, RS232
+# or KISS host; Treebeard's only sensor is a RAK1901 (SHTC3). Dropping the rest
+# Hazelwood RAK4631 repeaters (KM-Test, Treebeard): no display, GPS, INA, RS232,
+# KISS host or radio presets; Treebeard's only sensor is a RAK1901 (SHTC3). Dropping the rest
+# shrinks the image by ~100 KB, which leaves more in-place room for LoRa OTA
+# deltas. The env name must stay as is: it is the OTA target id (05F5FFAE).
+# Upstream qualification still expects these features from the env name, so
@@ -34,6 +34,9 @@ index 5f0987a0d..ff30faeb4 100755
+case "${2:-}" in
+ RAK_4631_repeater_unified_lora_ota)
+ LOCAL_BUILD_FLAGS="${LOCAL_BUILD_FLAGS} -UDISPLAY_CLASS -UENV_INCLUDE_GPS -UWITH_RS232_BRIDGE -UMESH_ENABLE_HOST_KISS"
+ # Named radio presets and the optional OTA transfer diagnostics (CLI) aren't
+ # used here; leaving them out keeps the image, and so its OTA deltas, smaller.
+ LOCAL_BUILD_FLAGS="${LOCAL_BUILD_FLAGS} -DMESH_ENABLE_RADIO_PRESETS=0 -DOTA_TRANSFER_DIAGNOSTICS=0"
+ for sensor in AHTX0 BME280 BME680 BME680_BSEC BMP085 BMP280 INA219 INA226 INA260 INA3221 LPS22HB MLX90614 RAK12035 SHT4X VL53L0X; do
+ LOCAL_BUILD_FLAGS="${LOCAL_BUILD_FLAGS} -UENV_INCLUDE_${sensor}"
+ done
@@ -0,0 +1,76 @@
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: "Rich O." <rich@rickelobe.com>
Date: Wed, 28 Oct 2026 22:00:48 -0400
Subject: [PATCH 5/5] Add upstream commits since the tag to the -hz build
identifier
Builds on newer upstream code kept the same -hz.N label because N counted
only local commits, so KM-Test's v1.18.0.2-hz.1 and the CI release of a
later upstream were indistinguishable. The identifier is now -hz.U.L: U
upstream commits past the version tag, L local firmware-code commits.
U counts from the earliest tag carrying the version, so a later tag on the
same version (w12-v1.18.0.2-preview3 after the alphas) does not reset it and
make newer builds sort as older.
---
local_build.sh | 26 +++++++++++++++++++++-----
local_version.py | 2 +-
2 files changed, 22 insertions(+), 6 deletions(-)
diff --git a/local_build.sh b/local_build.sh
index 839950b90..edfb3a027 100755
--- a/local_build.sh
+++ b/local_build.sh
@@ -71,9 +71,12 @@ cd "$(dirname "$0")"
# VERSION follows the fork's own numbering: the highest vX.Y.Z.W tag reachable
# from HEAD (e.g. v1.17.1.10 from rak4631-full-companion-v1.17.1.10-preview3).
# Local code changes add an identifier; build settings (untracked files) don't:
-# -hz.N N of your own commits that change firmware code relative to the
-# maintainer's fork (upstream/keymindCascade; origin is the personal
-# Gitea copy, so @{upstream} would hide them)
+# -hz.U.L U upstream commits past the first vX.Y.Z.W tag in the maintainer's fork
+# (upstream/keymindCascade; origin is the personal Gitea copy, so
+# @{upstream} would hide them) and L of your own commits that change
+# firmware code. U grows with every upstream pull, so builds on newer
+# upstream code sort higher even when the tag and L are unchanged.
+# LoRa OTA sees only vX.Y.Z.W; compare U.L to decide an upgrade.
# .dirty uncommitted edits to tracked firmware code
# Merge commits and the build-setup files below don't count: they change
# settings, not code. Edit LOCAL_FW_SETUP_PATHS if you add more setup files.
@@ -93,8 +96,21 @@ LOCAL_FW_SETUP_PATHS=":(exclude)local_build.sh :(exclude)local_version.py :(excl
# shellcheck disable=SC2086 # intentional word-splitting of the pathspec list
local_commits=$(git rev-list --count --no-merges "${LOCAL_FW_UPSTREAM_REF}..HEAD" \
-- . $LOCAL_FW_SETUP_PATHS 2>/dev/null || echo 0)
-if [ "$local_commits" -gt 0 ]; then
- local_id="-hz.${local_commits}"
+# U counts from the earliest tag carrying this version (the one furthest back), so
+# it only grows: a later tag on the same version (e.g. a new preview) doesn't reset it.
+upstream_base=$(git merge-base HEAD "${LOCAL_FW_UPSTREAM_REF}" 2>/dev/null || git rev-parse HEAD)
+version_tags=$(git tag --merged "$upstream_base" | grep -F "v${fork_version}" || true)
+if [ -z "$version_tags" ]; then
+ upstream_commits=$(git rev-list --count "$upstream_base" 2>/dev/null || echo 0)
+else
+ upstream_commits=0
+ for tag in $version_tags; do
+ n=$(git rev-list --count "$upstream_base" "^$tag" 2>/dev/null || echo 0)
+ [ "$n" -gt "$upstream_commits" ] && upstream_commits=$n
+ done
+fi
+if [ "$upstream_commits" -gt 0 ] || [ "$local_commits" -gt 0 ]; then
+ local_id="-hz.${upstream_commits}.${local_commits}"
fi
# shellcheck disable=SC2086
if [ -n "$(git status --porcelain --untracked-files=no -- . $LOCAL_FW_SETUP_PATHS)" ]; then
diff --git a/local_version.py b/local_version.py
index fbd6c069f..cd6a84079 100644
--- a/local_version.py
+++ b/local_version.py
@@ -4,7 +4,7 @@
#
# repeater/room/sensor: hazelwood_repeater v1.17.1.10 (core v1.17.1)
# companion: v1.17.1.10 hazelwood (core v1.17.1)
-# with local code changes the version gains -hz.N and/or .dirty (local_build.sh)
+# past the tag the version gains -hz.U.L and/or .dirty (local_build.sh)
#
# Companions keep the version first because the app's device-info frame
# carries only the first 19 characters (companion MyMesh.cpp strzcpy(..., 20)).
+40 -5
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Build the Hazelwood firmware: upstream keymindCascade tip + patches/*.patch,
# Build the Hazelwood firmware: upstream's latest full release + patches/*.patch,
# then ./local_build.sh for each target. Writes everything to $OUT_DIR and a
# build.env with the identifiers publish-release.py and the workflow need.
#
@@ -8,14 +8,17 @@
#
# Environment (all optional):
# UPSTREAM_URL default https://github.com/mikecarper/MeshCore.git
# UPSTREAM_REF default keymindCascade (branch, tag or commit)
# UPSTREAM_REF default latest-release: the newest GitHub release of the
# upstream repo that isn't a pre-release or draft. Or a
# branch, tag or commit (keymindCascade = the branch tip,
# pre-release commits included).
# TARGETS space-separated PlatformIO envs (default: see below)
# WORK_DIR default ./work OUT_DIR default ./out
set -euo pipefail
here=$(cd "$(dirname "$0")/.." && pwd)
UPSTREAM_URL=${UPSTREAM_URL:-https://github.com/mikecarper/MeshCore.git}
UPSTREAM_REF=${UPSTREAM_REF:-keymindCascade}
UPSTREAM_REF=${UPSTREAM_REF:-latest-release}
TARGETS=${TARGETS:-"RAK_4631_repeater_unified_lora_ota Heltec_v3_companion_radio_full_nomqtt"}
WORK_DIR=${WORK_DIR:-$here/work}
OUT_DIR=${OUT_DIR:-$here/out}
@@ -29,8 +32,19 @@ mkdir -p "$WORK_DIR" "$OUT_DIR"
# Blobless clone keeps history and tags (local_build.sh needs both) without
# downloading every historical file.
git clone --quiet --filter=blob:none --no-checkout "$UPSTREAM_URL" "$src"
git -C "$src" fetch --quiet --tags origin "$UPSTREAM_REF"
upstream_commit=$(git -C "$src" rev-parse FETCH_HEAD)
upstream_tag=
fetch_ref=$UPSTREAM_REF
if [ "$UPSTREAM_REF" = latest-release ]; then
# GitHub's releases/latest skips pre-releases and drafts.
repo_path=${UPSTREAM_URL#https://github.com/}
repo_path=${repo_path%.git}
upstream_tag=$(curl -fsSL --max-time 30 "https://api.github.com/repos/$repo_path/releases/latest" \
| python3 -c 'import json, sys; print(json.load(sys.stdin)["tag_name"])')
fetch_ref=refs/tags/$upstream_tag
echo "latest upstream release: $upstream_tag"
fi
git -C "$src" fetch --quiet --tags origin "$fetch_ref"
upstream_commit=$(git -C "$src" rev-parse "FETCH_HEAD^{commit}")
git -C "$src" checkout --quiet --detach "$upstream_commit"
# Fixed committer identity and dates make the patched commit ids depend only
@@ -40,11 +54,32 @@ export GIT_COMMITTER_NAME="Hazelwood CI" GIT_COMMITTER_EMAIL="ci@hazelwood.inval
git -C "$src" am --quiet --committer-date-is-author-date "$here"/patches/*.patch
source_commit=$(git -C "$src" rev-parse HEAD)
# Fingerprint of everything that can change the binaries: the patched source
# tree minus files no build reads, plus the targets and this script. Upstream
# commits that touch only docs, tests or CI keep it, and the workflow then
# skips the build (publish-release.py --unchanged). The version label can't be
# compared instead: -hz.U counts every upstream commit, so it changes anyway.
# publish-release.py uses the same pattern to sort the release notes' changes.
noncode_re='(.*\.md|(docs|test|\.github|\.cursor|\.devcontainer|\.meshlog-review)(/.*)?)'
code_fingerprint=$(
{
# ls-tree takes no exclude pathspecs; lines are "<mode> <type> <id>\t<path>".
git -C "$src" ls-tree -r "$source_commit" | grep -vE $'\t'"$noncode_re\$"
echo "TARGETS=$TARGETS"
cat "$here/scripts/ci-build.sh"
} | sha256sum | cut -c1-16
)
{
echo "UPSTREAM_URL=$UPSTREAM_URL"
echo "UPSTREAM_TAG=$upstream_tag"
echo "UPSTREAM_COMMIT=$upstream_commit"
echo "SOURCE_COMMIT=$source_commit"
echo "CODE_FINGERPRINT=$code_fingerprint"
echo "RELEASE_TAG=build-${source_commit:0:8}"
echo "TARGETS=$TARGETS"
echo "SOURCE_DIR=$src"
echo "NONCODE_RE=$noncode_re"
} > "$OUT_DIR/build.env"
cat "$OUT_DIR/build.env"
[ "$check_only" -eq 1 ] && exit 0
+130 -3
View File
@@ -9,6 +9,11 @@ Usage:
publish-release.py OUT_DIR --manifest-only write manifest.json and .hex files
publish-release.py OUT_DIR --publish also create the release
publish-release.py --exists TAG exit 0 if the release exists, 1 if not
publish-release.py --unchanged FINGERPRINT exit 0 if the newest release has this
code fingerprint (ci-build.sh), 1 if not
publish-release.py --behind SRC COMMIT exit 0 if COMMIT would move upstream backwards
(the newest release's upstream commit isn't
an ancestor of it), 1 if not
Publishing reads GITEA_SERVER_URL, GITEA_REPOSITORY and GITEA_TOKEN.
"""
@@ -18,7 +23,9 @@ import argparse
import hashlib
import json
import os
import re
import struct
import subprocess
import sys
import urllib.error
import urllib.parse
@@ -27,6 +34,8 @@ import uuid
from datetime import datetime, timezone
from pathlib import Path
HERE = Path(__file__).resolve().parent.parent
MAX_CHANGES = 100 # commits listed per section of the release notes
UF2_MAGIC = (0x0A324655, 0x9E5D5157)
UF2_NOT_MAIN_FLASH = 0x00000001
ENDF_TRAILER = 56
@@ -85,6 +94,9 @@ def build_manifest(out_dir: Path) -> tuple[dict, list[Path]]:
"firmware_version": env.get("FIRMWARE_VERSION", ""),
"upstream_commit": env["UPSTREAM_COMMIT"],
"source_commit": env["SOURCE_COMMIT"],
"code_fingerprint": env.get("CODE_FINGERPRINT", ""),
# So the next release can say which patches changed.
"patches": {p.name: sha256_file(p) for p in sorted((HERE / "patches").glob("*.patch"))},
"built_at": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
"targets": {},
}
@@ -133,6 +145,113 @@ def release_exists(tag: str) -> bool:
raise
def newest_manifest() -> dict | None:
"""manifest.json of the newest release, or None if there is none."""
repo = os.environ["GITEA_REPOSITORY"]
releases = [r for r in api("GET", f"/repos/{repo}/releases?limit=50") if not r.get("draft")]
if not releases:
return None
# Not created_at: releases from one tag push can share it. The id breaks ties.
newest = max(releases, key=lambda r: (r.get("published_at") or r.get("created_at", ""), r.get("id", 0)))
if not any(a["name"] == "manifest.json" for a in newest.get("assets", [])):
return None
url = (os.environ["GITEA_SERVER_URL"].rstrip("/") + f"/{repo}/releases/download/"
+ urllib.parse.quote(newest["tag_name"]) + "/manifest.json")
req = urllib.request.Request(url)
req.add_header("Authorization", "token " + os.environ["GITEA_TOKEN"])
with urllib.request.urlopen(req, timeout=60) as resp:
return json.loads(resp.read())
def newest_fingerprint() -> str | None:
"""code_fingerprint of the newest release, or None if it has none."""
manifest = newest_manifest()
if manifest is None:
return None
print(f"newest release {manifest.get('release_tag')}: code fingerprint "
f"{manifest.get('code_fingerprint') or 'none'}")
return manifest.get("code_fingerprint") or None
def git(src: str, *args: str) -> subprocess.CompletedProcess:
return subprocess.run(["git", "-C", src, *args], capture_output=True, text=True)
def upstream_behind(src: str, new: str) -> bool:
"""True if building upstream `new` would go back past the newest release's upstream.
Keeps -hz.U increasing when the upstream ref moves to an older commit, e.g.
the latest full release while the newest build came from a pre-release.
"""
manifest = newest_manifest()
old = (manifest or {}).get("upstream_commit")
if not old:
return False
result = git(src, "merge-base", "--is-ancestor", old, new)
if result.returncode == 0:
return False
if result.returncode == 1:
print(f"newest release {manifest.get('release_tag')} is built on upstream {old[:9]}, "
f"which {new[:9]} doesn't contain")
return True
# The old commit is missing from the clone (history rewritten?): don't block.
print(f"warning: can't compare upstream {old[:9]} with {new[:9]}: {result.stderr.strip()}")
return False
def upstream_commits(src: str, old: str, new: str, noncode: re.Pattern) -> list[tuple[str, str, bool]]:
"""(commit, subject, touches firmware code) for each non-merge commit old..new, newest first."""
out = git(src, "log", "--no-merges", "--name-only", "--format=%x00%H%x09%s", f"{old}..{new}")
out.check_returncode()
commits = []
for record in out.stdout.split("\0")[1:]:
head, _, files = record.partition("\n")
sha, _, subject = head.partition("\t")
paths = [f for f in files.splitlines() if f]
commits.append((sha, subject, any(not noncode.fullmatch(f) for f in paths)))
return commits
def changes_section(manifest: dict, env: dict[str, str], prev: dict | None) -> list[str]:
"""Release-notes lines: upstream commits and patch changes since the previous release."""
if prev is None:
return ["### Changes", "", "First release.", ""]
lines = [f"### Changes since {prev.get('firmware_version', '')} ({prev.get('release_tag', '')})", ""]
web = env.get("UPSTREAM_URL", "").removesuffix(".git")
old, new = prev.get("upstream_commit", ""), manifest["upstream_commit"]
src = env.get("SOURCE_DIR", "")
def link(sha: str) -> str:
return f"[`{sha[:9]}`]({web}/commit/{sha})" if web.startswith("https://") else f"`{sha[:9]}`"
if old == new:
lines += ["No upstream changes.", ""]
elif not (src and old and git(src, "merge-base", "--is-ancestor", old, new).returncode == 0):
lines += [f"Upstream `{old[:9]}` is not an ancestor of `{new[:9]}` (history rewritten?); "
"commits can't be listed.", ""]
else:
commits = upstream_commits(src, old, new, re.compile(env["NONCODE_RE"]))
for title, code in (("Code and build files", True), ("Docs, tests and CI only", False)):
group = [c for c in commits if c[2] == code]
if not group:
continue
lines.append(f"**{title}** ({len(group)})")
lines += [f"- {link(sha)} {subject}" for sha, subject, _ in group[:MAX_CHANGES]]
if len(group) > MAX_CHANGES:
lines.append(f"- … and {len(group) - MAX_CHANGES} more")
lines.append("")
if web.startswith("https://") and old and old != new:
lines += [f"[Full upstream diff]({web}/compare/{old}...{new})", ""]
if "patches" in prev:
before, after = prev["patches"], manifest["patches"]
notes = ([f"added `{n}`" for n in after if n not in before]
+ [f"changed `{n}`" for n in after if n in before and before[n] != after[n]]
+ [f"removed `{n}`" for n in before if n not in after])
lines += ["**Hazelwood patches:** " + ("; ".join(notes) if notes else "unchanged"), ""]
return lines
def upload(release_id: int, path: Path) -> None:
repo = os.environ["GITEA_REPOSITORY"]
boundary = uuid.uuid4().hex
@@ -144,7 +263,7 @@ def upload(release_id: int, path: Path) -> None:
f"multipart/form-data; boundary={boundary}")
def publish(manifest: dict, assets: list[Path]) -> None:
def publish(manifest: dict, assets: list[Path], env: dict[str, str]) -> None:
repo = os.environ["GITEA_REPOSITORY"]
tag = manifest["release_tag"]
if release_exists(tag):
@@ -152,13 +271,15 @@ def publish(manifest: dict, assets: list[Path]) -> None:
return
lines = [f"Firmware **{manifest['firmware_version']}**", "",
f"- upstream: `{manifest['upstream_commit']}`",
f"- source (upstream + patches): `{manifest['source_commit']}`", ""]
f"- source (upstream + patches): `{manifest['source_commit']}`",
f"- code fingerprint: `{manifest['code_fingerprint'] or 'none'}`", ""]
for target, info in manifest["targets"].items():
lines.append(f"**{target}** (OTA target {info['target_id']})")
for a in info["artifacts"]:
if "base_hash" in a:
lines.append(f"- {a.get('variant', '')} base_hash `{a['base_hash']}` ({a['body_len']} B)")
lines.append("")
lines += changes_section(manifest, env, newest_manifest())
payload = {
"tag_name": tag,
"target_commitish": os.environ.get("GITHUB_SHA", ""),
@@ -181,16 +302,22 @@ def main() -> int:
mode.add_argument("--manifest-only", action="store_true")
mode.add_argument("--publish", action="store_true")
mode.add_argument("--exists", metavar="TAG")
mode.add_argument("--unchanged", metavar="FINGERPRINT")
mode.add_argument("--behind", nargs=2, metavar=("SRC", "COMMIT"))
args = parser.parse_args()
if args.exists:
return 0 if release_exists(args.exists) else 1
if args.behind:
return 0 if upstream_behind(*args.behind) else 1
if args.unchanged is not None:
return 0 if args.unchanged and newest_fingerprint() == args.unchanged else 1
if args.out_dir is None:
parser.error("OUT_DIR is required")
manifest, assets = build_manifest(args.out_dir)
print(json.dumps({t: [a for a in i["artifacts"] if "base_hash" in a]
for t, i in manifest["targets"].items()}, indent=2))
if args.publish:
publish(manifest, assets)
publish(manifest, assets, read_env(args.out_dir / "build.env"))
return 0