Engineering report: kzero v1.1.1
kzero is a Go CLI for out-of-band Kubernetes maintenance: ordered down, up, and reset pipelines from YAML on a bastion or automation host. It is not in-cluster GitOps. The GHCR image is distroless (binary only); the host still needs kubectl on PATH (and helm when the profile uses shell-path release.* steps).
v1.1.1 (2026-09-02) is a security patch on the 1.1.x line (Go 1.26.6, no schema changes). Functional items below are anchored on v1.1.0 unless noted.
In the Hermes operator stack, kzero orchestrates maintenance; groot (read-only diagnostics archive) captures cluster evidence before or after a window — complementary, not a substitute. See §8.
1. When the control plane is “sick”
In-cluster reconcilers (Argo CD, Flux, operators) depend on the same API they are trying to heal. When etcd, admission webhooks, or the scheduler degrade, bastion-first tooling keeps a control path that does not require scheduling new pods in the sick cluster. See deployment models.
2. Why ad-hoc Bash breaks down
Typical maintenance scripts suffer from structural limits in production:
- No native idempotency — re-running after a partial failure is guesswork.
- No first-class dry-run — simulating impact means hand-rolling
echobranches per command. - Fragile error handling — transient API loss (HTTP/2 mid-stream) aborts the script or is ignored.
- Weak audit trail — no structured step log, exit taxonomy, or built-in notify hooks.
kzero doctor also checks the configured shell interpreter when hooks are present (/bin/sh vs bash).
3. Example: imperative Bash maintenance
A common window: scale ingest to zero, suspend a CronJob, run a cleanup Job, Helm upgrade, restore scale.
#!/bin/bash
set -eo pipefail
kubectl scale deployment/data-ingest --replicas=0 -n production
kubectl rollout status deployment/data-ingest -n production
kubectl patch cronjob/daily-reconcile -p '{"spec":{"suspend":true}}' -n production
kubectl delete job/storage-cleanup -n production --ignore-not-found
kubectl apply -f storage-cleanup-job.yaml
kubectl wait --for=condition=complete job/storage-cleanup --timeout=300s -n production
helm upgrade --install core-api ./charts/core-api --values prod-values.yaml -n production
kubectl scale deployment/data-ingest --replicas=3 -n production
kubectl patch cronjob/daily-reconcile -p '{"spec":{"suspend":false}}' -n productionIf kubectl wait fails on a transient blip, the cluster can stay half-down with no safe resume story.
4. Imperative → declarative mapping
| Bash / kubectl | kzero v1.1.x | Operator benefit |
|---|---|---|
kubectl scale deployment … --replicas=0 |
deployment.<ns>/<name> in pipelines.down |
Ordered steps; optional post hooks to wait for pod drain |
kubectl patch cronjob … suspend |
cronjob.<ns>/<name> |
Suspend on down, resume on up (native step) |
kubectl delete job + apply + wait |
job.<ns>/<name> |
Delete on down; create from manifest: on up with wait_for_complete |
helm upgrade --install … |
release.<ns>/<name> |
Helm SDK v4 when run.execution is native/auto (chart manifest under helm.workspace) |
Ad-hoc kubectl logs / evidence scrape |
groot collect (optional hooks.pre-down) |
Read-only .tar.gz archive before kzero mutates state |
| Manual pre-checks | kzero doctor, kzero analyze |
API, RBAC hints, YAML plan before mutation |
| Custom retry loops | Engine retry + run.api_watchdog |
Cancels active step if API stays unreachable past fail_after |
| Ad-hoc Slack curl | notify.* |
Optional exit 4 with notify.require_delivery |
Full contract: SPECIFICATIONS.md.
5. Equivalent kzero profile (valid YAML)
Same scenario as §3, using compact step refs (not invented nested blocks):
schema_version: "1.0"
cluster:
name: production-maintenance
environment: production
helm:
workspace: ./helm-assets # core-api.yaml for native Helm SDK
notify:
require_delivery: true
webhook:
enabled: true
url: "https://hooks.slack.com/services/…" # env or secret manager in prod
hooks:
pre-down: ./hooks/groot-capture.sh # optional: groot collect before mutations
on-error: ./hooks/groot-capture.sh # optional: second bundle if pipeline fails
run:
mode: dry-run # switch to live after doctor/analyze/diff
execution: native # default when omitted; Helm SDK for release.*
api_watchdog:
enabled: true
fail_after: 30s
pipelines:
down:
- deployment.production/data-ingest
- cronjob.production/daily-reconcile
- job.production/storage-cleanup
up:
- job.production/storage-cleanup:
manifest: ./jobs/storage-cleanup.yaml
wait_for_complete: true
timeout: 5m
- release.production/core-api
- deployment.production/data-ingest:
replicas: 3
wait_for_ready: true
- cronjob.production/daily-reconcileOn down, workloads scale to 0 automatically. On up, replicas: 3 restores ingest. More patterns: pvc-statefulset-data-strategy.md, pipeline-order-and-integrity.md.
Operator copies: kzero-selfhosted/run/examples.
6. Safe maintenance workflow
Run gates before flipping run.mode to live. On production bastions, capture evidence with groot first (or via hooks.pre-down above):
curl -fsSL https://get.kzero.hermesrodriguez.com/install.sh | sh
kzero --print-sample-config > ./kzero.yaml # edit for your cluster
# Optional but recommended: read-only archive while the cluster is still fully observable
groot collect -c ./groot.yaml -o ./evidence/pre-down-$(date +%Y%m%d-%H%M).tar.gz
kzero doctor -c ./kzero.yaml
kzero analyze -c ./kzero.yaml
kzero diff -c ./kzero.yaml --phase down # exit 2 if cluster already drifted
kzero down -c ./kzero.yaml # dry-run while run.mode: dry-run
# … maintenance work …
kzero up -c ./kzero.yaml
kzero diff -c ./kzero.yaml --phase up| Step | Purpose |
|---|---|
groot collect (optional) |
Read-only logs/events/API snapshot → .tar.gz for RCA and tickets |
doctor |
API reachability, binary paths, RBAC hints, shell interpreter |
analyze |
Static plan + optional object existence checks |
diff |
Desired vs live for --phase up|down; exit 2 on drift |
down / up / reset |
Live execution; api_watchdog cancels on prolonged API loss |
Cron / CI wrapper example:
kzero diff --config ./kzero.yaml --phase up || exit 2
kzero reset --config ./kzero.yamlSee diff cookbook.
7. Exit codes (automation)
| Code | Meaning | Typical action |
|---|---|---|
| 0 | Success | Continue pipeline |
| 1 | Config / validation | Fail CI; fix YAML |
| 2 | Kubernetes error or diff drift | Stop; inspect cluster |
| 3 | Executor / hook abort | On-call; may be partial state |
| 4 | Notify delivery failed | When notify.require_delivery: true |
8. Hermes ecosystem: groot complements kzero
kzero changes cluster state (scale, Helm, Jobs, PVCs). groot is read-only: groot collect bundles pod logs, events, and selected API snapshots into one .tar.gz for incident response, RCA, and compliance. It does not replace kzero — it preserves what the cluster looked like before you run down / up.
| Repo | Role |
|---|---|
| hrodrig/groot | CLI: collect, validate, inspect; optional S3/GCS/SFTP upload |
| groot-selfhosted | Bastion cron, Helm CronJob, operator playbooks |
| groot-trigger | In-cluster HTTP → Job that runs groot collect on demand |
| groot-share (gfs) | VPS catalog: ingest, list, download, retention for archives |
Landing: groot.hermesrodriguez.com. Same maintainer family as pgwd (Postgres connection watchdog) and gghstats (GitHub traffic analytics) — each tool covers one operator job.
Exit codes 0–4 in kzero follow the same wrapper-friendly pattern as groot exitcode.
9. v1.1.0 capabilities (unchanged in v1.1.1)
kzero diff— drift gate before destructive work.- Native
job/cronjobsteps — batch lifecycle without shell wrappers. - Helm SDK v4 — native
release.*without hosthelmwhen execution is native/auto. - Supply chain — Cosign-signed releases, SPDX/CycloneDX SBOMs, Grype gate on GHCR.
v1.1.1 only: Go 1.26.6 stdlib fixes — pull ghcr.io/hrodrig/kzero:v1.1.1; no YAML migration.
10. Comparison matrix
| Capability | kzero v1.1.1 | GitOps | Ansible | Shell scripts |
|---|---|---|---|---|
| Architecture | Out-of-band | In-cluster | External SSH | External |
| Maintenance | Declarative pipelines | Continuous reconcile | Playbooks | Imperative |
| API-degraded resilience | api_watchdog | API-dependent | Medium | Low |
| Simulation | analyze, dry-run, diff | Sync preview | check-mode | Rare |
| Supply chain | Cosign, SBOM, Grype | Varies | Varies | None |
11. Scope and links
kzero is a discrete maintenance orchestrator — scheduled resets, migration windows, disaster bring-up — not a replacement for Argo CD or Flux on the happy path. Pair it with groot when you need an evidence bundle around the same window.
| Resource | Link |
|---|---|
| SPEC | SPECIFICATIONS.md |
| Sample config | configs/kzero.sample.yml |
| Changelog | CHANGELOG.md |
| kzero (maintenance) | hrodrig/kzero · kzero-selfhosted |
| groot (diagnostics archive) | hrodrig/groot · groot-selfhosted |
| groot ecosystem | groot-trigger · groot-share |
| Operator examples | kzero-selfhosted/run/examples |