Eleven minutes, zero humans: Building a self-healing Kubernetes upgrade pipeline on Kairos

Once upon a time, upgrading a Kubernetes control plane meant staying awake for it.

SSH into every node. Run the upgrade by hand. Watch etcd health the whole time, hoping quorum holds through every reboot.

This week I found out whether the pipeline I built actually ends that story.

The Roots of the Platform

This mgmt cluster is where everything starts.

It was bootstrapped with OpenTofu with three control plane nodes, K3s HA, Cilium CNI, all provisioned as code before a single workload ever ran. I think of it as the root of a platform that’s going to keep growing: more tooling, more workloads, more clusters hanging off it over time.

Roots need to be solid before you build on top of them. And in 2026, with the pace of CVEs landing across the stack, the thing I wanted most from this root wasn’t more features. It was a genuinely simple, genuinely reliable upgrade process. Something I could trust to run without me needing to relearn the steps every time a patch shipped. I wanted to use the most possible tooling from Cloud Native Computing Foundation (CNCF) to avoid any vendor lock-in and build upon my Golden Kubestronaut knowledge.

That’s what the rest of this is about.

The Setup

Three control plane nodes, k3s HA, running Kairos Hadron an immutable Linux distribution built around A/B partition upgrades and cosign-signed images. Kairos doesn’t patch in place. It writes a new OS image to an inactive partition and reboots into it. Rollback is just booting the old partition again.

That’s a strong supply-chain story. But it only matters if the upgrade actually happens. My previous version of this pipeline had a subtle but dangerous bug: concurrency: 0 in the upgrade spec, which I assumed meant “one node at a time.” It means all nodes simultaneously. Three control plane nodes rebooted at once during a homelab test. etcd quorum survived. Pure luck, not design.

Fixed to concurrency: 1. That fix is the backbone of everything below.

The Full Pipeline

Six tools, each with one job:

Gitea — self-hosted git, running on its own Kairos + k3s cluster, with a Gitea Actions runner executing CI. This is where every manifest, policy, and upgrade spec lives.

Renovate — watches quay.io/kairos/hadron on Quay for new tags. When upstream publishes a release, Renovate opens a PR bumping two lines: the image tag and the metadata.name of the upgrade CR.

Kyverno — a ClusterPolicy admission gate that rejects any upgrade CR whose image doesn’t match quay.io/kairos/hadron:*. This blocks misconfiguration and typosquatting before it ever reaches a node.

Cosign — verifies the image signature against the upstream GitHub Actions OIDC identity. Not just “is this the right tag,” but “did the real CI pipeline actually produce this artifact.”

ArgoCD — GitOps from Argo Project . Detects the merged PR as drift, applies the new manifest. No human runs kubectl apply.

kairos-operator — the actual upgrade executor. Watches for NodeOpUpgrade custom resources, cordons one node, pulls the image, writes the new A/B slot, reboots, waits for rejoin, moves to the next node.

What Actually Happened

Real timestamps from the July 7th upgrade to Hadron v0.4.0:

Time

Event / Status

11:40:20

Baseline — 3 nodes Ready, kernel 7.0.10-hadron

11:42:00

PR #9 merged (Renovate bump, 2-line diff)

11:42:30

ArgoCD detects OutOfSync

11:42:37

ArgoCD Synced — old CR deleted, new CR created

11:42:43

Preflight: image pull starts (324 MB)

11:43:09

Image pulled in 25.5s

11:43:11

mgmt-cp-0 cordoned

11:43:33

Upgrade job completed — A/B slot written, reboot triggered

~11:47

mgmt-cp-0 Ready — kernel 7.1.0-hadron confirmed

~11:47

mgmt-cp-1 cordoned, upgraded, rebooted

~11:49

mgmt-cp-1 Ready — kernel 7.1.0-hadron

~11:49

mgmt-cp-2 cordoned, upgraded, rebooted

11:51:36

mgmt-cp-2 Ready — kernel 7.1.0-hadron. All done.

Total wall-clock time: 11 minutes. Human intervention after merge: zero. etcd quorum broken: never. Workload disruption: none.

The two-line diff that triggered all of this:

--- a/upgrades/mgmt/hadron-upgrade.yaml +++ b/upgrades/mgmt/hadron-upgrade.yaml @@ -11,7 +11,7 @@ metadata: - name: hadron-mgmt-v0-3-0 + name: hadron-mgmt-v0-4-0 spec: - image: quay.io/kairos/hadron:v0.3.0-standard-amd64-generic-v4.1.1-k3s-v1.35.5-k3s1 + image: quay.io/kairos/hadron:v0.4.0-standard-amd64-generic-v4.1.2-k3s-v1.35.5-k3s1

Why the Name Has to Change

NodeOpUpgrade is a one-shot custom resource. kairos-operator marks it complete and never reprocesses it. Patching spec.image on an existing CR does nothing — the operator already considers that object finished. Renovate has to bump metadata.name in the same commit, which forces ArgoCD to delete the old CR and create a genuinely new one. That’s the actual trigger.

The Bug I Didn’t Expect

This pipeline wasn’t flawless on the first real run.

Renovate’s custom regex manager used a field called extractVersionTemplate to convert the dash-formatted version in the CR name (v0-3-0) into a comparable SemVer. That field doesn’t exist in Renovate’s custom manager schema. It silently did nothing.

Result: Renovate correctly bumped spec.image to the new tag, but left metadata.name completely unchanged. kairos-operator saw no new CR — because there wasn’t one — and did nothing. The pipeline looked broken, but it was actually just half-executed.

The fix was switching to currentValueTemplate, which properly converts v0-3-0 into v0.3.0 so Renovate’s version comparison logic works correctly, then reconstructs the dash format for the new tag.

This is exactly why the human review step still exists in this pipeline. Not to execute the upgrade — the automation does that. To catch the one place where automation quietly does the wrong thing while looking like it did the right thing.

Where This Goes Next

The gateway cluster — a separate single-node Kairos deployment running Netbird — is now wired into the same ArgoCD GitOps loop as mgmt. Same pipeline, same automated upgrade path, no separate process to maintain.

Next up: a CI dry-run stage running kairos-agent upgrade –recovery against the new image before the PR ever merges, catching failures before they touch a live node at all.

Do you know the name of the Immutable OS which can operate at scale? It’s Kairos!