Background Agents on Kubernetes

The agent harness (mcp-s-harness) runs Willow background agents inside your own Kubernetes cluster. You install one Helm chart, register it in Willow as a custom agent platform, and from then on every agent you create in Willow gets a dedicated pod in your cluster.

Nothing about an agent is configured in the cluster. Its prompt, model, tools, skills, and rules are defined in Willow and pushed to the harness on sync; the harness only materializes them into Kubernetes objects. That is what keeps the agent's permission boundary and audit trail identical to any other Willow traffic.

Agent pods reach the Willow gateway to call tools, so the harness works with all three Willow deployment models — SaaS, Hybrid, and On-Prem. Paired with a hybrid or on-prem gateway, no agent traffic leaves your network at all.

This page is for whoever installs and operates the harness. For the dashboard side — choosing this platform for an agent and registering it as an agent type — see Run Agents in Your Own Cluster.

How it works

Components

Component	What it is	Where it runs
Control plane	A TypeScript/Fastify service. Receives Willow's webhooks, renders agent files, applies Kubernetes objects, owns sessions and the event log.	One Deployment in the release namespace
Agent runtime	A Python/FastAPI image wrapping a deepagents graph, built from the mounted config at startup.	One Deployment per agent, in the agents namespace
PostgreSQL	Agents, sessions, the append-only event log, and config revisions for the control plane; the conversation checkpointer for agent pods.	Your database

The control plane's Kubernetes permissions are a namespaced Role over the agents namespace covering deployments, services, configmaps, and secrets. There are no cluster-wide permissions and no access to any other namespace.

What a synced agent looks like

Each agent becomes four objects named agent-<slug>, all labeled mcp-s-harness/agent: <slug>. The slug is immutable in Willow, so those names are stable for the agent's lifetime.

Object	Contents
ConfigMap	The rendered agent definition: `AGENTS.md`, `mcp.json`, `harness.yaml`, and skill files
Secret	`WILLOW_AGENT_TOKEN` (the agent's gateway credential) and `MODEL_API_KEY`
Deployment	One replica of the runtime image, with the ConfigMap mounted at `/agent-config`
Service	ClusterIP on port 8000, reachable only by the control plane

The Deployment scales to zero replicas when the agent's status in Willow is not active, so disabling an agent in the dashboard stops its pod without deleting anything.

The pod template also carries two annotations, mcp-s-harness/config-hash and mcp-s-harness/secrets-hash, which force a rollout when the rendered files or the resolved secret values change. The secrets hash matters more than it looks: the runtime reads its token and model key as environment variables, which are fixed once a container starts, so updating the Secret alone would leave a rotated key sitting unused. The annotation stores only a digest, never the values, because anyone with read access to the namespace can see it.

The sync lifecycle

Selecting Deploy in Willow sends the full agent snapshot along with an agent_token — the agent's gateway credential. The control plane records the agent as deploying, renders its files, applies the objects, and marks it ready. If applying fails, the agent is left in error with the failure message, readable from GET /agents.

Deleting the agent in Willow arrives on the same endpoint as {"event": "agent.delete"}, because a custom-managed platform type only configures a sync URL and a message URL. Deletion is idempotent: an agent that is already gone still returns 204, since Willow drops its own row regardless of the answer.

The message lifecycle

Two things are worth noticing. The acknowledgement is fast and the run is not — the control plane answers Willow as soon as the run has started, then consumes the pod's event stream in the background, which is what keeps it inside Willow's 20-second delivery timeout. And every tool call goes back through the Willow gateway, so the agent pod holds no third-party API keys and guards, policies, and audit logging apply as they do everywhere else.

Events are appended to a log in Postgres rather than kept on the pod, so history survives a restart and reading a conversation never touches an agent pod. Their types line up with Willow's own session-event model — user.message, agent.message, agent.tool_use, agent.tool_result, run.completed, run.error — and are available as a list, as a live SSE stream, or straight from the database.

The reconcile loop

Beyond responding to Willow, the control plane re-applies every known agent from its stored snapshot on a timer (reconcileIntervalSeconds, 300 by default; 0 disables it).

Applying is a no-op when nothing changed, so this is cheap. Its real job is picking up what no webhook announces: a rotated secret, a new runtime image, or an edited harness configuration. Without it, agents would keep whatever values they were handed until Willow happened to sync them again, which may be never. One agent failing to reconcile does not stop the others.

Configuration layers

Harness configuration is merged from up to three layers, later ones winning:

For most installations the first layer is the only one and the other two stay off. Whatever the source, GET /config/effective shows what the harness is actually using, and invalid revisions are rejected rather than applied. See the configuration reference for the schema.

Prerequisites

A Kubernetes cluster with an ingress controller, reachable from Willow over HTTPS.
A PostgreSQL database the cluster can reach. The control plane stores agents, sessions, and events here, and agent pods use it as their conversation checkpointer. Schema migrations run automatically on startup.
Pull access to quay.io/webrix. If your cluster doesn't have it, create an image pull secret (see Image pull secrets).
An API key for the model provider your agents will use (for example Anthropic).
A shared bearer token you generate yourself. Willow sends it on every call to the harness, and the harness rejects anything else.

Network requirements

Inbound to the harness host (HTTPS / 443) — from Willow, for sync and message delivery. If your ingress restricts inbound traffic by IP and you are on Willow SaaS, allow the Willow egress NAT IP 3.130.252.122.

Outbound from the agents namespace (HTTPS / 443):

To your Willow run gateway, for MCP tool calls.
To your model provider's API endpoint.

Valid TLS required on the harness endpoint

Willow calls the harness over HTTPS and will not accept a self-signed certificate. Terminate TLS at your ingress with a publicly trusted certificate — cert-manager with Let's Encrypt, an ACM certificate on an AWS load balancer, or a proxy such as Cloudflare.

Don't buffer or time out the SSE stream

The harness exposes a live session stream at GET /agents/:agentId/sessions/:sessionId/stream as Server-Sent Events. A proxy that buffers responses or drops idle connections will break it — usually at the ingress default of 60 seconds. The chart's default ingress annotations already handle this for ingress-nginx:

nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-buffering: "off"

On Istio or Envoy, set a generous route timeout and don't enable a response buffer filter. On a cloud load balancer, raise the idle timeout well above 60 seconds.

Step 1 — Choose an image tag

The chart deliberately ships no default image tags, and helm install fails until you name a build:

images.controlPlane.tag is required: pin a build tag, never a moving one

This is on purpose. A moving tag like latest fails quietly rather than loudly: rebuilding the image leaves the pod spec unchanged, so Helm rolls nothing out, and with pullPolicy: IfNotPresent a node that already cached the tag keeps serving the old layer while another node runs the new one. Pin a concrete build for both images and change it deliberately.

Step 2 — Provide the secrets

The harness needs two secrets to start (a third, a GitHub token, is only needed for GitOps configuration):

Secret	Purpose	Where it comes from
`WILLOW_WEBHOOK_TOKEN`	The shared bearer token Willow sends on sync and message calls.	You invent it. Generate a random string and enter the same value here and in Willow.
`MODEL_API_KEY`	The model provider key handed to each agent pod.	Your model provider's console — Anthropic, OpenAI, or Google.

Neither side issues the webhook token, so there is no ordering problem: generate it before you install, and use the same value in both places.

openssl rand -hex 32

There is no Generate Secret button here

This differs from the hybrid and on-prem gateway setup, where the admin app mints the secret for you. The Bearer token field on a custom agent type is a plain input that stores whatever you type, so the harness is the side that defines the value.

The one credential nobody handles is the agent's own gateway token. Willow generates it and ships it inside the sync payload, and the control plane writes it straight into the agent's Secret.

One model key for the whole harness

MODEL_API_KEY is a single value shared by every agent, and the runtime decides which provider env var to set by reading the prefix of the model string — anthropic:, openai:, google_genai:, or google:. A model with no prefix is assumed to be Anthropic.

If an administrator selects a model from a different provider than the key you installed, that agent fails on its first run with an authentication error from the new provider, which gives no hint that the key is simply the wrong one. Either standardize on one provider, or agree with your Willow administrators on which models are allowed.

Pick one of three providers with secrets.provider:

Provider	How values arrive	Best for
`env` (default)	A Kubernetes Secret, either rendered by the chart, one you bring yourself, or one synced by the external-secrets operator.	Most installations.
`vault`	Read from HashiCorp Vault (KV v2) using the Kubernetes auth method.	Organizations already standardized on Vault.
`aws`	Read from AWS Secrets Manager via IRSA.	AWS-native clusters.

For a real installation, prefer external-secrets over inline values, so no token is ever written into values.yaml or a shell history:

secrets:
  provider: env
  env:
    externalSecret:
      secretName: my-org/mcp-s-harness-secret
      secretStoreName: aws-secretsmanager
      secretStoreKind: ClusterSecretStore

The backend entry is a single JSON object whose keys land verbatim in the Secret:

{
  "WILLOW_WEBHOOK_TOKEN": "<shared-token>",
  "MODEL_API_KEY": "<model-provider-key>"
}

Create the backend entry before the first install

If it's missing, the operator can't produce the Secret and the only visible symptom is the control plane stuck in CreateContainerConfigError until helm --wait times out. The real reason is on the ExternalSecret: kubectl describe externalsecret -n <namespace>.

See Secret providers for the Vault and AWS variants.

Step 3 — Create values.yaml

images:
  controlPlane:
    tag: "<build>"
  runtime:
    tag: "<build>"

imagePullSecrets:
  - name: webrix-registry

ingress:
  enabled: true
  className: nginx
  host: harness.your-domain.com
  tls:
    - hosts: ["harness.your-domain.com"]
      secretName: harness-tls

willow:
  # The Willow run gateway your agents call tools through. On hybrid or on-prem,
  # this is your own run URL, so agent traffic never leaves your network.
  runUrl: https://run.mcp-s.com

agents:
  namespace: mcp-s-agents
  createNamespace: true

database:
  existingSecret: harness-db
  secretKey: DATABASE_URL

secrets:
  provider: env
  env:
    externalSecret:
      secretName: my-org/mcp-s-harness-secret

defaults:
  model: anthropic:claude-sonnet-4-6

defaults.model is a fallback only. When an agent's platform configuration in Willow names a model, that wins.

For evaluation, you can skip the external secret store and pass values inline instead:

database:
  url: postgres://user:password@host:5432/harness
secrets:
  env:
    willowWebhookToken: "<shared-token>"
    modelApiKey: "<model-provider-key>"

The full list of values is in the configuration reference.

Step 4 — Install

helm upgrade --install mcp-s-harness ./helm/mcp-s-harness \
  --namespace mcp-s-harness \
  --create-namespace \
  -f values.yaml \
  --wait

The chart creates the control-plane Deployment, Service, and Ingress; a ServiceAccount with a namespaced Role over the agents namespace only (no cluster-wide permissions); and the agents namespace itself when agents.createNamespace is on.

Verify the control plane is healthy before moving on:

kubectl get pods -n mcp-s-harness
curl https://harness.your-domain.com/healthz

/healthz answers as soon as the process is up. /readyz additionally checks the database and returns 503 database unavailable if the connection string is wrong — check it too:

curl -i https://harness.your-domain.com/readyz

Step 5 — Register the platform in Willow

Hand your Willow administrator the harness hostname and the shared token. They register it under Manage → Machine Users → Background Agents → Settings → Add custom type, with the sync endpoint at https://harness.your-domain.com/willow/sync and the message endpoint at https://harness.your-domain.com/willow/message. The full walkthrough is in Run Agents in Your Own Cluster.

The token is not optional in practice

The harness enforces the bearer token whenever willow.webhook_auth resolves to a value, which the chart always configures. If the token registered in Willow doesn't match WILLOW_WEBHOOK_TOKEN, every sync and message call returns 401 Unauthorized.

Step 6 — Create and sync an agent

An administrator creates a background agent on the new platform type, gives it a system prompt and its tools and skills, and selects Deploy.

Willow POSTs the full agent snapshot to /willow/sync. The control plane renders the agent's files, writes a ConfigMap, Secret, Deployment, and Service named agent-<slug> into the agents namespace, and answers with the harness's own id as external_agent_id.

Watch the pod appear:

kubectl get pods -n mcp-s-agents -w
# agent-release-notes-writer-xxxxx   1/1   Running

Confirm the harness recorded it, using the shared token:

curl https://harness.your-domain.com/agents \
  -H "Authorization: Bearer <shared-token>"

{
  "data": [
    {
      "id": "…",
      "willow_agent_id": "…",
      "slug": "release-notes-writer",
      "harness_type": "deepagents",
      "status": "ready"
    }
  ]
}

status is ready once the workload is applied, disabled when the agent is toggled off in Willow, and error with an error message when the deploy failed.

Now use Test on the agent's Overview tab in Willow, or fire one of its triggers. The message lands on /willow/message, the control plane maps the session_key to a conversation thread on the pod, and the run's events stream back.

Verifying an end-to-end run

Tail the agent pod while you trigger it:

kubectl logs -n mcp-s-agents -l mcp-s-harness/agent=release-notes-writer -f

At startup, look for agent <slug> ready (model=<model>). If tool calls fail, the problem is almost always the gateway leg rather than the harness — confirm the rendered mcp.json points at the right gateway:

kubectl get configmap agent-release-notes-writer -n mcp-s-agents -o jsonpath='{.data.mcp\.json}'

Day-two operations

Upgrades

Bump the image tags and re-run helm upgrade. The control plane rolls immediately. Agent pods pick up a new runtime image on their next reconcile, which happens on a timer (reconcileIntervalSeconds, 300 by default) as well as on every sync from Willow.

That reconcile loop is also what delivers rotated secrets. Secret values are resolved fresh each pass and hashed into a pod annotation, so a rotated model key triggers a rollout of the agent pods on its own — without it, agents would keep the value they were given until Willow happened to sync them again, which may be never.

Removing an agent

Delete the agent in Willow. Willow POSTs {"event": "agent.delete"} to the sync endpoint (a custom-managed type only configures two URLs, so deletion reuses sync_url), and the control plane removes all four Kubernetes objects. The call is idempotent: an agent that is already gone still returns 204.

To tear one down without touching Willow, call DELETE /agents/:agentId with the shared token. Note that Willow will recreate it on the next sync, since Willow remains the source of truth.

Uninstalling

helm uninstall mcp-s-harness -n mcp-s-harness
kubectl delete namespace mcp-s-agents

Agent workloads live in the agents namespace and are created by the control plane at runtime, not by Helm, so they are not removed by helm uninstall.

Troubleshooting

Knowing which leg failed narrows the search quickly:

Leg	Symptom when it breaks	Where to look
Willow → control plane	Deploy or trigger fails in the dashboard	Ingress, TLS, and the shared bearer token
Control plane → Kubernetes	Agent status is `error`	Control-plane logs; RBAC and the agents namespace
Pod startup	`CrashLoopBackOff`	Agent pod logs; model key and checkpointer database
Agent pod → gateway	The run starts but tool calls fail	The rendered `mcp.json` and the agent token
Control plane → your client	The event stream stalls or drops	Proxy buffering and idle timeouts on the SSE route

Symptom	Cause	Fix
`helm install` fails with "images.controlPlane.tag is required"	No image tag pinned	Set `images.controlPlane.tag` and `images.runtime.tag` to a concrete build
Control plane stuck in `CreateContainerConfigError`	The Secret doesn't exist — usually a missing entry in the external secrets backend	`kubectl describe externalsecret -n <namespace>` for the real reason
`/readyz` returns `503 database unavailable`	Wrong or unreachable `DATABASE_URL`	Check the database secret and that the cluster can reach Postgres
Every sync and message call returns `401`	Token mismatch	The bearer token on the Willow custom type must equal `WILLOW_WEBHOOK_TOKEN`
Sync returns `400 Invalid agent.sync payload`	The endpoint received something other than an `agent.sync` event	Confirm the Sync endpoint URL ends in `/willow/sync` and the Message endpoint in `/willow/message`
Message returns `404 Agent not synced to this harness`	The agent was never deployed to the platform	Select Deploy on the agent in Willow
Message returns `409 Agent is not ready`	The agent is disabled in Willow, or its last deploy failed	Re-enable it, or check `error` on `GET /agents`
Agent status is `error` after deploy	The control plane could not apply the Kubernetes objects	Check control-plane logs; usually RBAC or a wrong `agents.namespace`
Agent pod is `CrashLoopBackOff`	Missing model key, or an unreachable checkpointer database	`kubectl logs` on the agent pod; confirm the DB secret exists in the agents namespace
A run fails with an authentication error from the model provider	The agent's model in Willow is from a different provider than `MODEL_API_KEY`	Check the `model` field on the agent ConfigMap's `harness.yaml` against the key you installed
Tool calls fail inside a run	Gateway URL or agent token wrong	Inspect the rendered `mcp.json` on the agent ConfigMap; re-deploy from Willow to reissue the token
The session stream connects then drops at ~60s	A proxy read/idle timeout on the SSE route	Raise `proxy-read-timeout` (or the load balancer idle timeout) and keep buffering off

Image pull secrets

If your cluster doesn't already have access to quay.io/webrix:

kubectl create secret docker-registry webrix-registry \
  --namespace mcp-s-harness \
  --docker-server=quay.io \
  --docker-username=<robot-username> \
  --docker-password=<robot-token>

Then reference it under imagePullSecrets in your values. The agents namespace needs the same secret if it pulls from the private registry.

What to do next

Agent Harness Configuration Reference — every Helm value, the configuration schema, and the control-plane API
Run Agents in Your Own Cluster — the dashboard side, for administrators
Background Agents — creating and managing agents in Willow
Harness API: New Sessions and Messages — the wire contract Willow speaks
Set up guards to constrain what an agent can do

Background Agents on Kubernetes

How it works

Components

What a synced agent looks like

The sync lifecycle

The message lifecycle

The reconcile loop

Configuration layers

Prerequisites

Network requirements

Step 1 — Choose an image tag

Step 2 — Provide the secrets

Step 3 — Create values.yaml

Step 4 — Install

Step 5 — Register the platform in Willow

Step 6 — Create and sync an agent

Verifying an end-to-end run

Day-two operations

Upgrades

Removing an agent

Uninstalling

Troubleshooting

Image pull secrets

What to do next

On this page

Was this helpful?

How it works​

Components​

What a synced agent looks like​

The sync lifecycle​

The message lifecycle​

The reconcile loop​

Configuration layers​

Prerequisites​

Network requirements​

Step 1 — Choose an image tag​

Step 2 — Provide the secrets​

Step 3 — Create values.yaml​

Step 4 — Install​

Step 5 — Register the platform in Willow​

Step 6 — Create and sync an agent​

Verifying an end-to-end run​

Day-two operations​

Upgrades​

Removing an agent​

Uninstalling​

Troubleshooting​

Image pull secrets​

What to do next​

On this page

Was this helpful?

How it works

Components

What a synced agent looks like

The sync lifecycle

The message lifecycle

The reconcile loop

Configuration layers

Prerequisites

Network requirements

Step 1 — Choose an image tag

Step 2 — Provide the secrets

Step 3 — Create values.yaml

Step 4 — Install

Step 5 — Register the platform in Willow

Step 6 — Create and sync an agent

Verifying an end-to-end run

Day-two operations

Upgrades

Removing an agent

Uninstalling

Troubleshooting

Image pull secrets

What to do next