Skip to content

An action did not happen, or happened twice

4. onExit fired — or did not — when you did not expect it

Section titled “4. onExit fired — or did not — when you did not expect it”

onExit declares the level you want while the condition does not hold, not a script that runs at the moment of exit. Most surprises come from reading it the second way.

It fired and you expected nothing. A key you match on changed value. With when: {ups: on-battery, ups.battery: critical}, a battery recovering from 8% to 15% moves ups.battery from critical to low, the condition stops holding, and the reversal applies — even though the power is still out. That is the semantics working; the fix is matching only the keys you actually care about.

It fired the moment you suspended the Automation. That is the same semantics again: suspending takes the policy out of force, so the target is arbitrated as if the Automation were gone, and the reversal applies even though the condition still holds — status.matching stays true and says so. See pausing an automation for why that matches what deleting does, and for the reversal: None patch that freezes the workload instead.

It did not fire and you expected it to. Three candidates, in order:

  1. The condition never stopped holding. Check status.matching and status.observedState.
  2. A key went missing rather than changing, so state is held — see §2.
  3. Another Automation still claims the same target and its claim wins. See §7.

You omitted onExit and the workload came back anyway. From v1, an Automation with no onExit restores the target’s recorded baseline when it stops matching, rather than leaving the workload wherever it was put. A workload that never comes back was judged the worse default. To get the old behaviour — Reactor sets it and never touches it again — set spec.reversal: None explicitly.

The reversal field and baseline restore land with the target-ownership change. On v0.3.0, an omitted onExit leaves the target untouched on exit.


12. An edge action did not happen — or happened too often

Section titled “12. An edge action did not happen — or happened too often”

Edge actions never fail the Automation — the desired-state action is the thing that had to happen, and a notification or a restart is not — so Ready stays True and the answer is in status.edgeActions and in the resource’s Events:

Terminal window
kubectl -n media get automation notify-on-failover -o jsonpath='{.status.edgeActions}' | jq
kubectl -n media describe automation notify-on-failover | grep EdgeAction
reason containsWhat it meansFix
outbound actions are disabled on this installactions.allowedDestinations is empty, which is the defaultAdd the destination to the chart value
is not allowed by this installThe destination is not on the allowlist. The message names it as scheme://host:portAdd exactly that, port included
refusing to dial ... loopback / link-localThe host resolved to an address that is refused whatever the allowlist saysNot a misconfiguration to work around; see SECURITY.md
refusing to follow a redirect toThe endpoint answered with a redirect. Redirects name a destination the allowlist never approvedPoint the action at the final URL
reading secret ... not foundNo credential Secret of that name in the Automation’s namespaceCreate it there; there is no cross-namespace read
has no "url" keyThe Secret exists but carries no destinationAdd url to it
responded 4xxThe endpoint rejected it. A 401 or 403 is a credential problemCheck the Secret’s url and authorization
rendering templateA message referenced a field or state key that does not exist{{ .State.wan }} errors on a typo, by design
has no "authorization" keyhomeassistant.service found no token, and refuses to collect a 401 to find outAdd authorization=Bearer <long-lived-token> to the Secret
did not render to a JSON objecthomeAssistant.data rendered to a list, a bare string or nothingService data is an object: {"entity_id": "light.hall"}
set no "SID" cookieqBittorrent rejected the username or password. It answers a wrong one with 200 OK and no cookie, not a 401Check the username and password keys in the Secret
needs both a "username" and a "password"qbittorrent.* found only one of themBoth are required; an instance that bypasses authentication is an http.request, not this action
responded 404 Not Found on /torrents/pauseqBittorrent 5.0 deprecated pause/resume in favour of stop/start; an instance that removed them answers 404Report it — Reactor uses the compatible names deliberately

An empty status.edgeActions when you expected one means the action never fired rather than failed. That is one of:

  • Nothing transitioned. Edge actions fire on a change of status.matching, not on every reconcile. kubectl get automation shows the current value.
  • The workload could not be scaled. A transition whose desired-state action failed is not committed, so nothing is announced until the retry succeeds — deliberately, so no message says a workload was paused while it is still running. Look at status.lastExecution.
  • The automation is suspended. spec.suspend: true sends nothing, the same way a deleted one does not.
  • It was a deletion. Deleting an Automation is not a state transition and fires no edge action.

A destination is only ever reported as scheme://host:port. That is not a truncation bug: for every notification transport the path is the credential, so it is kept out of status, logs and Events on purpose. A kubernetes.restart reports its target as Kind/namespace/name in the same field.

kubernetes.restart is the only action where a repeat is harmful, so this has exactly one cause worth looking for: the state key driving it is flapping.

Terminal window
# how many times has this key actually changed value?
# increase(reactor_state_transitions_total{key="wan"}[1h])
kubectl -n media describe automation restart-on-recovery | grep -E 'StateEntered|StateExited'

A pair of StateEntered / StateExited Events every poll is a flapping signal, not a restart bug. The engine only acts on transitions, so a steady condition never restarts anything twice — but each flap is a genuine transition and therefore a genuine rollout.

The fix is debounce, and it belongs on the key rather than on the automation, so that every automation still sees one settled value:

Terminal window
helm upgrade reactor oci://ghcr.io/robbeverhelst/charts/reactor \
--namespace reactor-system --reuse-values \
--set unifi.debounce.keys.wan=3

Each extra sample costs one pollInterval of reaction time. The shipped default of 1 is chosen for kubernetes.scale, where a flap is harmless; a key that drives a restart should be raised above it. spec.suspend: true stops the restarts immediately while you decide.

Two things that are not the cause: a reconcile without a transition (edge actions do not fire), and a retry (a restart is attempted exactly once per transition and never retried).