Development

Prerequisites

Go (see go.mod for the version)
Docker, for building the image
A Kubernetes cluster to test against — bring your own: kind, k3d, minikube, OrbStack, or a real one
No UniFi hardware required; see Running without a UDM

Repository layout

api/v1alpha1/          Automation CRD types
internal/engine/       provider-agnostic core: state store, transition detection
internal/events/       normalized Event and Observation models
internal/providers/    provider implementations (currently unifi)
internal/actions/      outbound edge actions: the destination allowlist, the
                       HTTP transport, templating, notification transports
internal/controller/   the reconciler and the UniFi poller
charts/reactor/        Helm chart (`make manifests` regenerates its templated CRD)
hack/mock-unifi/       mock UniFi API serving the captured payloads
hack/dev/              demo Automations used by `make dev-hello`
testdata/unifi/        real captured API responses — the parsers' ground truth

internal/actions/ is provider-agnostic for the same reason the engine is: a notification action must have no idea what wan means. It is also where Reactor’s outbound reach is bounded — read the package comment before changing anything in it, and SECURITY.md for why the bounds are where they are.

The engine must never contain provider-specific logic. Providers translate vendor reality into normalized state; the engine only ever sees that. Keeping the seam clean is what lets new providers arrive without touching the core — see Adding a provider for the contract and a walkthrough of the UniFi one.

Everyday commands

make test     # unit tests + envtest (controller-runtime)
make lint     # golangci-lint, same config CI runs
make build    # compile the manager binary
make help     # every target

CI runs lint, tests, e2e, and a manifest-drift check, so make manifests generate output must be committed.

End-to-end tests

Three suites, each in its own throwaway Kind cluster, each its own CI job:

make test-e2e        # the manager comes up under the kustomize manifests
make test-reaction   # reactions, restarts, and arbitration against a real API server
make test-lifecycle  # helm uninstall and the upgrade from the crds/ packaging

The two new ones install the Helm chart and point it at a rehearsed UniFi console running inside the cluster, then assert on what happened to real workloads: replicas, the reactor.robbeverhelst.com/* annotations, status.targets[], and the Ready and Applied conditions. They cover the things that cannot be reached from a unit test — converging on a state that changed while the operator was down, two Automations arbitrating one workload, and what helm uninstall leaves behind.

Each target creates its Kind cluster, runs the suite, and deletes the cluster whether or not it passed. Every kubectl and helm call inside the suites names its cluster explicitly and refuses to address anything that is not a local Kind context — they install cluster-wide RBAC and delete CRDs, so an unpinned command is not a failed test but an outage.

The suites reach the mock over a fixed node port mapped to the host by test/e2e/kind-config.yaml, which is why they create their own clusters rather than reusing one you already have.

make manifests also regenerates charts/reactor/templates/crds.yaml via hack/sync-chart-crds.sh. The CRD is a chart template deliberately: Helm installs a chart’s crds/ directory on first install only and never upgrades it, so every later schema change would ship silently broken. Don’t hand-edit the chart’s copy — the tests in test/chart/ fail when it drifts from config/crd/bases, and they need helm on your PATH to run at all.

Running against a cluster

Targets act on DEV_CONTEXT, which defaults to your current kubectl context. Pass it explicitly — these targets install an operator with cluster-wide RBAC, and the current context can change between two commands in the same target.

make dev-context DEV_CONTEXT=kind-reactor   # print what you're about to hit

make dev-deploy \
  DEV_CONTEXT=kind-reactor \
  UNIFI_URL=https://192.168.1.1 \
  UNIFI_API_KEY=<key>

make dev-hello DEV_CONTEXT=kind-reactor     # demo workloads + Automations
make dev-clean DEV_CONTEXT=kind-reactor     # remove it all

dev-deploy builds the image, creates the credentials Secret, and installs the local chart. Local images work on clusters that share the host Docker daemon (OrbStack, k3d with the shared registry); with kind, load the image first via kind load docker-image.

Running without a UDM

make dev-mock serves the captured payloads from testdata/ on :9443 and lets you drive state transitions by hand:

make dev-mock

curl -X POST http://localhost:9443/flip                        # WAN primary <-> backup
curl -X POST 'http://localhost:9443/ups?mode=battery&level=80' # power outage
curl -X POST 'http://localhost:9443/ups?level=5'               # battery critical
curl -X POST 'http://localhost:9443/ups?mode=mains&level=100'  # power restored
curl -X POST 'http://localhost:9443/ups?present=false'         # UPS drops off the console
curl -X POST 'http://localhost:9443/ups?runtime=150'           # minutes of runtime left
curl -X POST 'http://localhost:9443/ups?runtime=0'             # the UPS offers no estimate
curl -X POST 'http://localhost:9443/ups?output=850'            # a heavy load on the same budget

curl -X POST 'http://localhost:9443/internet?status=error'     # no internet, link unchanged
curl -X POST 'http://localhost:9443/internet?present=false'    # the www subsystem vanishes
curl -X POST 'http://localhost:9443/quality?availability=97'   # the live uplink gets flaky
curl -X POST 'http://localhost:9443/quality?latency=400'       # ...or just slow
curl -X POST 'http://localhost:9443/quality?reset=true'        # back to the capture

curl http://localhost:9443/device                              # what the capture holds, and each device's key
curl -X POST 'http://localhost:9443/device?name=ups-2u&state=offline'    # a device dies
curl -X POST 'http://localhost:9443/device?name=ups-2u&rename=Rack+UPS'  # ...or is renamed
curl -X POST 'http://localhost:9443/device?reset=true'         # back to the capture

curl -X POST 'http://localhost:9443/wifi?disconnected=1'       # one access point drops
curl -X POST 'http://localhost:9443/wifi?disconnected=3'       # all of them: wifi error

curl -X POST 'http://localhost:9443/firmware?upgradable=true'  # an update is waiting
curl -X POST 'http://localhost:9443/temperature?celsius=82'    # a device runs hot
curl -X POST 'http://localhost:9443/poe?watts=55&budget=60'    # the PoE budget fills up
curl -X POST 'http://localhost:9443/poe?silent=true'           # a powered port reports no wattage
curl -X POST 'http://localhost:9443/poe?port=7&name=re-patched' # ...and the write path's identity check

curl http://localhost:9443/outlets                             # every outlet, and which relay group it is in
curl -X POST 'http://localhost:9443/outlets?outlet=5&state=off'          # one outlet opens
curl -X POST 'http://localhost:9443/outlets?switching=group&outlet=5&state=off'   # ...and takes 5-8 with it
curl -X POST 'http://localhost:9443/outlets?outlet=5&label=nas'          # key becomes outlet.nas
curl -X POST 'http://localhost:9443/outlets?reset=true'        # back to the capture

/poe drives both halves of the PoE story, because the mock has one synthetic switch and one port_table and they are the same one: watts/budget/silent move what the poe state key measures, while port/name/uplink/poe break the identity checks the unifi.poe.cycle action makes. That switch is adopted and online, so it is part of the fleet devices counts and is addressable as mock-switch through /device.

/firmware, /temperature and /poe serve fields no capture contains. The committed records carry no upgrade flags, no thermals and no port_table, so those three endpoints render the shape UniFi’s API documents — including poe_power as a string, which is the form most likely to break a parser. Driving them exercises the derivation; it does not confirm a console reports any of it. Until one does, the mock’s honest default is what the captures show: those keys are simply absent, and present=false puts each back to that state. See the capture notes.

Per-device keys are opt-in in Reactor (unifi.devices.perDeviceKeys), so device.<name> will not appear until you ask for it — devices is published either way. A device is addressed by the slug of the name it was captured under even after rename=, which is what makes the rename rehearsal reversible: renaming makes the old key vanish, and the reconciler holds the last known state rather than treating it as a recovery.

/outlets is the one endpoint here that rehearses a question nobody has asked the hardware. Everything it serves is captured — index, name, relay_state and relay_group are all in stat-device-ups.json — but whether a UniFi UPS switches an outlet or a relay group is unknown, and switching= is the mock imitating each answer in turn. With switching=individual, asking for outlet 5 moves outlet 5; with switching=group the same request takes outlets 5–8, because they share relay_group: 2. Rehearse both, because a parser that only ever saw one of them is a parser tested against a guess.

Reactor never writes an outlet — /outlets is the mock’s own dev surface, not a UniFi endpoint Reactor calls. Switching is #23, and the experiment that decides it is H1 on #60: with the operator watching, toggle one outlet by hand in the UniFi UI and read whether one relay_state flips or four. outlet=5&label=nas rehearses the other half of that visit — naming the outlets, which moves the key off the index.

/wifi drives the wlan subsystem’s AP counts, because that is what wifi is derived from. ?status= moves the console’s own wording without moving the counts, which is how you rehearse the disagreement Reactor reports rather than silently resolving.

present=false removes the UPS from the device list rather than reporting a value for it, so the ups keys vanish entirely. That is the case an Automation has to distinguish from “the outage ended”, and the one the reconciler answers with StateKeyUnavailable. /internet?present=false does the same for the www health subsystem.

runtime=0 is the narrower version of the same idea: the UPS is still there and still reports charge, but offers no runtime estimate, so ups.runtime alone disappears while ups and ups.battery stay. Per-key degradation is meant to be that granular.

/internet is the rehearsal you cannot reach through /flip or /wan at all, and that is the point of the key: the link stays up, the uplink is unchanged, and there is no internet. /quality drives the live uplink’s uptime_stats — availability as a percentage and latency in milliseconds, which on real hardware are averages over the console’s 24-hour uptime window and here move instantly. Both follow whichever uplink /wan says is live.

The statuses /internet will serve — warning and error, which map to degraded and down — have never been seen on a real console’s www subsystem. Rehearsing them shows what Reactor does with them; it does not confirm a console ever sends them. See the capture notes.

Rehearsing a failover that has never been observed

/flip moves every WAN signal at once, which is what the wan mapping assumes a failover looks like. That assumption has never been checked against real hardware (#34), so the mock can also render the other plausible shapes — because a parser tested against one hypothesis is only tested against one hypothesis:

curl http://localhost:9443/wan          # current state, and what each variant means

curl -X POST 'http://localhost:9443/wan?link=backup&variant=is-uplink-pinned'
curl -X POST 'http://localhost:9443/wan?link=primary'

Variant	What it says a failover looks like	What Reactor should do
`clean`	every signal moves together	report `wan: backup`, quietly
`is-uplink-only`	only `is_uplink` moves	report `backup`, and log that `uplink.name` disagrees
`is-uplink-pinned`	`is_uplink` means “configured as primary” and never moves	report `primary` through a failover — the silent failure, so it logs loudly instead
`both-uplinks`	both ports claim `is_uplink`	fall back to `uplink.name` rather than guessing
`no-uplink`	neither claims it, mid-switchover	fall back to `uplink.name` instead of dropping the key

Add &isp=<name> to rehearse the carrier changing too; the default is an obviously synthetic one, because the real backup carrier has never been seen.

The same hypotheses are asserted in internal/providers/unifi/wan_test.go, derived from the committed capture in code. Neither the mock nor those tests produce a fixture: settling which hypothesis is real needs hardware, and the capture runbook is the procedure for it.

Point the operator at it with UNIFI_URL=http://<your-host>:9443 UNIFI_API_KEY=mock. Use a LAN address rather than localhost so the pod can reach your machine.

The mock also answers the Integration API’s info endpoint, which is what Reactor’s compatibility guard reads at startup. It serves the captured version by default; pass a different one to rehearse the warning without owning the hardware that would produce it:

go run ./hack/mock-unifi -network-version 11.0.0
# INFO This UniFi Network version is newer than anything Reactor has been tested against ...

Webhook fast path

The receiver turns a UniFi Alarm Manager delivery into an immediate re-observation. It is off by default, and everything about it degrades to poll-only:

make dev-deploy DEV_CONTEXT=kind-reactor \
  UNIFI_URL=http://<your-host>:9443 UNIFI_API_KEY=mock \
  HELM_EXTRA_ARGS="--set unifi.webhook.enabled=true"

The receiver needs a Secret with the shared secret it will demand from every delivery:

kubectl -n reactor-system create secret generic unifi-reactor-webhook \
  --from-literal=UNIFI_WEBHOOK_TOKEN="$(openssl rand -hex 32)"

Send a delivery by hand — the payload is never read, so any body will do:

curl -X POST http://<receiver>:9090/webhooks/unifi \
  -H 'Authorization: Bearer <token>' \
  -H 'Content-Type: application/json' \
  -d @hack/dev/webhook-delivery.json

The logs show the delivery and the observation it caused, in that order:

kubectl -n reactor-system logs deploy/reactor | grep -E 'webhook delivery|state observed'

hack/dev/webhook-delivery.json is synthetic, not a capture. Nothing in Reactor parses a delivery, so a stand-in is enough to drive the path.

Rehearsing self-registration

make dev-mock also mocks the Alarm Manager API, so the registration path can be exercised without touching a real console. Run the mock, start Reactor with unifi.webhook.registration.enabled=true, and then have the mock fire a delivery at whatever rule Reactor registered:

make dev-webhook

The mock’s alarm responses are built from the Alarm Manager API notes, not captured from a console. Registration succeeding against the mock proves Reactor sends what those notes describe. It does not prove a real console accepts it.

Capturing real deliveries

hack/webhook-logger.mjs dumps incoming requests verbatim to testdata/unifi/webhooks/raw/ (gitignored). Raw records contain every header, including the Authorization header carrying Reactor’s own shared secret, so nothing goes from there into testdata/ by hand:

node hack/webhook-logger.mjs 8080

./hack/sanitize-webhook-capture.sh --paths testdata/unifi/webhooks/raw/<file>.json
./hack/sanitize-webhook-capture.sh testdata/unifi/webhooks/raw/<file>.json \
  internet-disconnected alarm.trigger,alarm.title

The first command prints every field path in the body; the second keeps the ones you name and discards the rest, along with every header but content-type. Same allowlist discipline as hack/capture-unifi.sh, and hack/verify-testdata.sh rejects leftover credential material in testdata/unifi/webhooks/ as a safety net.

Captured payloads

Parsers are written and tested against real responses in testdata/unifi/, never against assumed formats. Capture them with:

UNIFI_URL=https://192.168.1.1 UNIFI_API_KEY=<key> ./hack/capture-unifi.sh

The script keeps an explicit allowlist of fields and discards everything else, then replaces the few remaining sensitive values with placeholders. Supporting a new field means adding it to the allowlist in that script, deliberately.

This is allowlist rather than denylist for a reason: stat/device returns whole device records containing management keys, syslog keys, and adoption identifiers, and the parser needs a dozen fields out of hundreds. An earlier version of these fixtures stripped the sensitive fields someone thought of instead of keeping only the needed ones, and a live credential reached this repository’s history as a result.

make test runs hack/verify-testdata.sh, which rejects unredacted secret fields, routable IPs, and real MACs. That is the safety net; the capture script is the mechanism.

hack/webhook-logger.mjs dumps incoming webhook deliveries verbatim to testdata/unifi/webhooks/raw/ (gitignored) when capturing from a real Alarm Manager. Apply the same allowlist discipline before committing any of it.

Metrics

make dev-deploy leaves the metrics endpoint off, the same as a chart install. Turn it on and read it without a Prometheus:

make dev-deploy DEV_CONTEXT=kind-reactor \
  UNIFI_URL=http://<your-host>:9443 UNIFI_API_KEY=mock \
  HELM_EXTRA_ARGS="--set metrics.enabled=true --set metrics.secure=false"

kubectl -n reactor-system port-forward deploy/reactor 8443:8443
curl -s localhost:8443/metrics | grep '^reactor_'

metrics.secure=false is a development convenience: the real posture is HTTPS behind the API server’s authn/authz filter, and a scrape needs a bearer token.

Driving make dev-mock is the fastest way to see the decision-layer series move. POST /flip produces a reactor_state_transitions_total increment, a reactor_state_info flip, an action, and a reactor_reaction_latency_seconds observation, in that order.

New metrics go in internal/metrics/, never in a controller or a provider — the definitions live in one file so the label decisions are reviewable in one place. Read the package comment before adding a label: the rule is that a label whose value set comes from the outside world does not go in. reactor_state_info enforces that structurally, by publishing only the keys whose provider declared a closed value set via StateVocabulary.

Releasing

Releases are cut entirely by CI from a tag; nothing is published from a developer machine.

git tag v0.3.0 && git push origin v0.3.0

That builds the multi-arch image, packages the chart with version/appVersion taken from the tag, pushes both to GHCR, and attaches install.yaml to a GitHub Release with generated notes. Image and chart versions always move together.

Both artifacts are signed by cosign keyless signing, using the workflow’s OIDC token — nothing to configure, no key anywhere. The image also gets an SBOM and build provenance. SECURITY.md has the cosign verify invocations.

Use conventional commits (feat:, fix:, docs:, …) — they drive the generated release notes.