Production Sizing and Hardening

The On-Prem and Hybrid guides get you to a working release. This page is what changes when that release has to carry real traffic: how big to make the cluster, how much to give each service, when to let them scale, and the few production settings the chart deliberately leaves to you.

Everything here is a value the chart already exposes per service — resources, autoscaling, rollingUpdate, nodeSelector, serviceAccount.annotations, image — so none of it requires a chart change.

Hybrid runs one service, not four

On Hybrid only run (and optionally slack-background-agent) runs in your cluster. Read the run rows and skip the rest — the sizing and scaling advice for run is identical in both topologies, because it is the same image doing the same work.

---

Nodes

One node pool is enough. Willow's services are ordinary web and API workloads with no special scheduling needs, and at the size a single organization runs, splitting them across pools adds operational surface without buying anything.

Instance types

Pick a general-purpose family with a 4 GiB per vCPU ratio. That matches the aggregate request profile below, so you are not paying for memory you cannot fill or running out of it at half the CPU.

Cloud	Recommended
AWS	m7i, m7g, m6i
Azure	Dsv5, Dasv5
GCP	n2-standard, t2d-standard

Willow's images are published for both amd64 and arm64, so AWS Graviton (m7g) works and is usually the cheaper option.

Don't put run on burstable instances

t3, t4g, and Azure B-series accrue CPU credits and throttle hard once they run out. A tool call that suddenly takes ten times as long, only during your busy hour, and only on some nodes, is the symptom — and it looks exactly like a slow upstream API. Burstable instances are fine for a lab; they are not fine for production.

How many nodes

Add up the requests from the table below for the services you actually enable, then add roughly 1 vCPU and 2 GiB per node for your ingress controller, metrics-server, CNI, and kubelet. Remember that each pod's request must fit on a single node — the scheduler cannot split a 2 vCPU request across two half-idle nodes.

For a full on-prem install of the four core services, that arithmetic lands here:

Starting point	Nodes	Carries
One replica of each service	2 × 8 vCPU / 32 GiB	~6.5 vCPU / 10 GiB of requests, plus room for one node to go away
Autoscaling enabled, minimum 2 replicas each	3 × 8 vCPU / 32 GiB	~13 vCPU / 20 GiB at the floor, with headroom to scale out

Hybrid is far smaller: run alone is ~2 vCPU / 2 GiB per replica, so two modest nodes cover it with room to scale.

Start there, watch it for a week, and grow from what you measure rather than from this table:

kubectl top pods -n <namespace>                  # actual usage vs the requests you set
kubectl describe node <node> | sed -n '/Allocated resources/,/Events/p'

Give nodes at least 50 GiB of disk. Stdio MCP servers install their package (npx -y example-mcp) at call time, and those installs land in the node's ephemeral storage. A node that fills up evicts pods with DiskPressure, which does not look like a disk problem from the Willow side.

Pinning a service to specific nodes

If your security policy requires a service to run on dedicated nodes, every deployment accepts nodeSelector, tolerations, and affinity:

deployments:
  run:
    nodeSelector:
      willow.ai/pool: run
    tolerations:
      - key: willow.ai/pool
        operator: Equal
        value: run
        effect: NoSchedule

---

Per-service CPU and memory

These come from load testing the platform at sustained tool-call volume. They are a starting point, not a ceiling — measure your own steady state and adjust.

Service	CPU request	Memory request	CPU limit	Memory limit
app	750m	2Gi	1	2Gi
connect	1.5	1Gi	2	1Gi
run	2	2Gi	2.5	2Gi
db-service	2	5Gi	2.5	5Gi
prompt-injection (if enabled)	1.5	4Gi	2	4Gi
slack-background-agent (if enabled)	300m	256Mi	500m	512Mi

deployments:
  run:
    resources:
      requests:
        cpu: "2"
        memory: 2Gi
      limits:
        cpu: "2.5"
        memory: 2Gi

Three things about the shape of those numbers:

• The memory limit equals the memory request. Memory is not compressible: a pod that exceeds its limit is OOM-killed, not throttled. Setting the two equal means the scheduler reserves exactly what the pod is allowed to use, so a busy neighbour can never be the reason your pod dies.
• The CPU limit is slightly above the request. The request is a reservation the scheduler must satisfy on a single node, so it tracks steady-state usage; the limit supplies burst headroom. A wide gap between them is what makes latency unpredictable — the pod runs fast until it is throttled at an arbitrary moment.
• The CPU request is what the HPA measures against. See the warning in Scaling before you raise it.

db-service is the memory-hungry one because of its query pools and result buffering, not because of request concurrency. Give it the 5Gi even when its CPU looks idle.

---

Scaling and autoscaling

Autoscaling is off by default — each service runs at its fixed replicas count. Enable it per service.

Prerequisites: metrics-server must be installed in the cluster, and each autoscaled service must define a CPU request (all the chart defaults do).

Recommended HPA settings

Service	Min	Max	Target CPU	Why this target
run	2	10	60%	Tool calls are bursty and a cold MCP client is expensive, so scale out early
connect	2	6	70%	OAuth flows and dashboard traffic
app	2	5	80%	Admin console; low volume, tolerant of a queue
db-service	2	5	50%	Scaling it is also scaling database connections — see below

deployments:
  run:
    autoscaling:
      enabled: true
      minReplicas: 2
      maxReplicas: 10
      targetCPUUtilizationPercentage: 60
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 30
        scaleDown:
          stabilizationWindowSeconds: 300

The behavior block matters as much as the target. Scaling up after 30 seconds means a traffic spike is absorbed rather than queued; scaling down only after 300 seconds of sustained low usage stops the deployment oscillating, which on run would throw away warm MCP clients every few minutes.

Keep the CPU request and the HPA target in step

The HPA measures usage as a percentage of the request, not of the node. Raise run's request to 4 CPU without touching targetCPUUtilizationPercentage: 60 and a pod now has to sustain 2.4 cores before a single replica is added — so the service saturates and never scales. Whenever you change a request, re-check the target.

Scaling db-service also scales your database connections

Each db-service pod opens up to three connection pools of DB_POOL_MAX (default 10) connections each, so peak connections are roughly maxReplicas × 30. At maxReplicas: 5 that is 150 against the in-cluster PostgreSQL's default max_connections of 200 — before anything else in your estate connects to the same database. Raise max_connections, lower DB_POOL_MAX, or keep maxReplicas low. Raising maxReplicas on its own is how you get connection-refused errors under exactly the load you scaled up for.

Memory-based autoscaling is deliberately not enabled

Node/V8 and model workloads hold memory high regardless of load, so a memory target scales up and then never scales back down. Opt in per service with targetMemoryUtilizationPercentage only if you know your workload's memory actually tracks its traffic.

slack-background-agent must stay at one replica. It holds a Slack Socket Mode connection, and a second replica processes every event twice. Leave autoscaling.enabled: false on it.

Rollouts without dropped requests

One setting, on every service you care about:

deployments:
  run:
    rollingUpdate:
      maxSurge: 50%
      maxUnavailable: 0

maxUnavailable: 0 means capacity only ever goes up during a deploy: new pods must become Ready before any old pod is removed. Without it, a rollout can take you below the replica count you sized for at exactly the moment you are also restarting everything.

Optional — share rate-limit state across replicas

Only relevant once a service runs more than one replica. The HTTP rate limiter keeps its per-org counters in memory, so with N replicas each org effectively gets N times its configured limit. If that matters to you, enable the bundled Redis and the chart wires REDIS_URL into every service automatically:

redis:
  enabled: true

The services fail open to their in-memory behavior if Redis is unreachable, so this is safe to add to a running deployment and safe to leave off. For production, a managed Redis (ElastiCache, Azure Cache, Memorystore) is preferable to the in-cluster instance, which ships with no persistence and no auth.

---

Hardening

Most of Willow's security posture is set in the install guides. Two things are not, and both are worth doing before you carry real traffic.

Pin your images

The chart defaults every image tag to latest, and its own logic then sets imagePullPolicy: Always for that tag. A pod restart can therefore silently pick up a different build than its neighbours are running.

deployments:
  run:
    image:
      repository: your-registry.example.com/webrix/mcp-s-run
      tag: "1.0.62"

Mirror the images into your own registry rather than pulling from quay.io/webrix at runtime. That removes a runtime dependency on an external registry, lets you scan images before they reach the cluster, and is required for air-gapped clusters anyway. See Custom Image Pull Secrets.

Workload identity

Only relevant if a service needs cloud permissions — IAM database authentication, or the decrypt-only KMS role in Write-Only KMS. When it does, give that service its own cloud identity on its own service account rather than granting the permission to the node role, where every pod on the node inherits it:

deployments:
  db-service:
    serviceAccount:
      create: true
      name: willow-db-service
      annotations:
        eks.amazonaws.com/role-arn: "arn:aws:iam::<ACCOUNT>:role/willow-db-service"
        # AKS:  azure.workload.identity/client-id: "<CLIENT_ID>"
        # GKE:  iam.gke.io/gcp-service-account: "<SA>@<PROJECT>.iam.gserviceaccount.com"

Already covered elsewhere

These belong on the same checklist, but they live in the install guides:

Control	Where
Replace the four placeholder secrets before first install	On-Prem Step 5
Store secrets in Kubernetes secrets, not values.yaml	Working with Custom Secrets
Rotate ENCRYPTION_KEY off the chart default	Rotating the Encryption Key
Keep the admin hostname off the public internet until SSO is configured	On-Prem Step 11
Publicly-trusted TLS on the run hostname	TLS and Inbound Reachability
Restrict which upstream hosts tools may reach	Egress Allowlist
Restrict which source IPs may reach the gateway	IP Access Filter

---

Putting it together

A production values.yaml fragment for run, combining everything above:

deployments:
  run:
    image:
      repository: your-registry.example.com/webrix/mcp-s-run
      tag: "1.0.62"
    resources:
      requests:
        cpu: "2"
        memory: 2Gi
      limits:
        cpu: "2.5"
        memory: 2Gi
    autoscaling:
      enabled: true
      minReplicas: 2
      maxReplicas: 10
      targetCPUUtilizationPercentage: 60
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 30
        scaleDown:
          stabilizationWindowSeconds: 300
    rollingUpdate:
      maxSurge: 50%
      maxUnavailable: 0

Verify it renders the way you expect before installing:

helm template willow willow/webrix-helm -f values.yaml \
  | grep -A6 -E "resources:|minReplicas"

After the rollout, confirm the cluster agrees:

kubectl get pods -n <namespace> -o wide          # which node each pod landed on
kubectl get hpa -n <namespace>                   # TARGETS must show a real percentage, not <unknown>
kubectl top pods -n <namespace>                  # actual usage vs the requests you set

An HPA showing <unknown> for its target means metrics-server is missing or not reporting — the autoscaler is configured but doing nothing.