On-Prem Deployment

Run the entire Willow platform inside your own Kubernetes cluster. The runtime, admin console, connect service, and database all run on your infrastructure — fully isolated from Willow SaaS, with no call-home requirement.

Why On-Prem?

Complete data isolation — tool execution, authentication, audit logs, and configuration never leave your network
No external dependencies at runtime — suitable for air-gapped environments and networks with strict egress controls
Compliance — satisfies the strictest data residency and sovereignty requirements (HIPAA, FedRAMP, financial services, internal security policy)
Full administrative control — you own the deployment lifecycle: upgrades, scaling, backups, and configuration

How It Works

All Willow microservices are deployed into your Kubernetes cluster with a single Helm chart:

┌──────────────────────────────────────────────────────────┐
│  Your Kubernetes cluster                                  │
│                                                          │
│  ┌──────────┐    ┌─────────────┐    ┌─────────────────┐  │
│  │   app     │──▶│  db-service  │◀──│     connect     │  │
│  │  (admin)  │   └──────┬──────┘    │  (dashboard,    │  │
│  └──────────┘          │           │   OAuth)        │  │
│                        ▼           └─────────────────┘  │
│                 ┌────────────┐            ▲             │
│                 │ PostgreSQL │            │             │
│                 └────────────┘     ┌──────┴──────┐      │
│  MCP clients ─────────────────────▶│     run     │      │
│  (Claude, Cursor)                  │ (tool exec) │      │
│                                    └─────────────┘      │
└──────────────────────────────────────────────────────────┘
                  Managed entirely by you

The Helm chart deploys:

app — the administrative console for managing your Willow instance
connect — the dashboard UI and OAuth / external tool authentication service
run — the MCP runtime that executes tool calls inside your network
db-service — the database access layer
PostgreSQL (optional) — in-cluster database when webrix-postgresql.enabled: true
Ingress, Service Accounts & RBAC — routing and the permissions required to operate

Your users connect their AI assistants (Claude, Cursor, or any MCP-compatible client) directly to the on-prem run endpoint. Authentication is handled by the on-prem connect service, which integrates with your existing SSO provider.

Setup Guide

Prerequisites

Kubernetes cluster — EKS, GKE, AKS, OpenShift, or any conformant distribution (v1.23+ recommended)
kubectl — configured to access your cluster
Helm — v3+
Domain name — with the ability to configure DNS records, and an ingress controller running in your cluster

Network Requirements

On-Prem has no call-home requirement — once deployed, it operates entirely within your network boundary. The only outbound traffic you may need to allow is:

During install — access to the image registry (quay.io/webrix) to pull images. For air-gapped clusters, mirror the images to your internal registry (see Custom Image Pull Secrets).
At runtime — only to the third-party APIs your tools actually call (GitHub, Slack, Jira, etc.). If you use AI-powered guardrails, also to your configured LLM provider.

Step 1 — Add the Helm Repository

helm repo add willow https://webrix-ai.github.io/webrix-helm
helm repo update

Verify the repository was added:

helm search repo willow

Prefer an OCI registry?

The chart is also published as an OCI artifact to oci://ghcr.io/webrix-ai/charts. If your organization standardizes on OCI registries, you can skip helm repo add entirely and reference the chart directly by its oci:// URL in Step 3 — it's the exact same chart, just distributed as an OCI artifact.

# No `helm repo add` needed — the oci:// URL is self-contained
helm show chart oci://ghcr.io/webrix-ai/charts/webrix-helm

Step 2 — Create values.yaml

Create a values.yaml to configure your installation. This overrides the chart defaults to match your environment. The minimal configuration below runs the full suite with an in-cluster PostgreSQL database:

global:
  # Base domain for your services. Default subdomains:
  #   app     → willow-admin.<host>
  #   connect → willow-dashboard.<host>
  #   run     → willow.<host>
  domain:
    host: "<YOUR_DOMAIN>" # e.g. example.com

  # Optional — required only if you use AI-powered guardrails
  OPENAI_API_KEY: ""

# In-cluster database. Deploys the webrix-postgres chart and wires
# db-service to it automatically.
webrix-postgresql:
  enabled: true

Choosing your database

The chart supports in-cluster PostgreSQL or an external database:

webrix-postgresql.enabled: true — the webrix-postgres chart (recommended default shown above)
externalDatabase.url.* — bring your own database (see External Database)

Enable only one.

All configuration values

You can view every available value, setting, and default in the Helm chart repository: github.com/webrix-ai/webrix-helm

Configuration reference

Value	Purpose
`global.domain.host`	Your base domain. Used to build the ingress hostnames for each service.
`global.OPENAI_API_KEY`	Optional. Required only if you use AI-powered guardrails.
`webrix-postgresql.enabled`	Deploys the in-cluster webrix-postgres database. Recommended for in-cluster deployments.
`externalDatabase.url.*`	Bring your own database. Disable both in-cluster options. See External Database.

Step 3 — Install

Deploy with helm upgrade --install. Replace <namespace> with your desired namespace (e.g. willow):

helm upgrade --install willow willow/webrix-helm \
  --namespace <namespace> \
  --create-namespace \
  -f values.yaml \
  --wait

Or, if you're using the OCI registry instead of helm repo add, reference the chart by its oci:// URL (no repo needed):

helm upgrade --install willow oci://ghcr.io/webrix-ai/charts/webrix-helm \
  --namespace <namespace> \
  --create-namespace \
  -f values.yaml \
  --wait

Installation typically takes 2–5 minutes depending on your cluster.

Step 4 — Verify

Pod health:

kubectl get pods -n <namespace>

Expected output:

NAME                        READY   STATUS    RESTARTS   AGE
app-xxxxxxxxxx-xxxxx         1/1     Running   0          2m
connect-xxxxxxxxxx-xxxxx     1/1     Running   0          2m
run-xxxxxxxxxx-xxxxx         1/1     Running   0          2m
db-service-xxxxxxxxxx-xxxxx  1/1     Running   0          2m
webrix-postgresql-0          1/1     Running   0          2m

Services and ingress:

kubectl get services -n <namespace>
kubectl get ingress -n <namespace>

Logs for a specific component:

kubectl logs -n <namespace> deployment/run --tail=100 -f

Step 5 — Set Up DNS and Access Your Services

Point DNS A records (or CNAMEs) at your ingress controller's external IP. Find it with:

kubectl get svc -n ingress-nginx  # Adjust namespace for your ingress controller

Create the records:

willow-admin.<domain>      → <ingress-ip>
willow-dashboard.<domain>  → <ingress-ip>
willow.<domain>            → <ingress-ip>

Once DNS resolves, your services are available at:

App — https://willow-admin.<domain> — administrative console
Dashboard — https://willow-dashboard.<domain> — analytics and OAuth flows
Run — https://willow.<domain> — MCP runtime endpoint your AI assistants connect to

Advanced

External Database

To use your own PostgreSQL, use externalDatabase.url:

externalDatabase:
  url:
    # Provide the DATABASE_URL via one of the following:
    secretName: "willow-db-url" # existing Secret with a DATABASE_URL key (recommended)
    # clearText: ""             # plain-text URL, stored in a ConfigMap (not recommended)

Prefer secretName (or sealedSecret) over clearText so credentials never land in plaintext values or a ConfigMap. Create the secret first:

kubectl create secret generic willow-db-url \
  --namespace <namespace> \
  --from-literal=DATABASE_URL="postgres://user:password@db-host:5432/willow?sslmode=require"

Working with Custom Secrets

You'll typically mount custom Kubernetes secrets when you need to provide:

External database credentials — when using db_provider: "external"
Third-party API keys & SSO secrets — e.g. your SSO client secret, OPENAI_API_KEY, etc.

Step 1 — Create the secret:

kubectl create secret generic app-api-keys \
  --namespace <namespace> \
  --from-literal=AUTH_OKTA_SECRET=xxxxxxxxxxxxx \
  --from-literal=OPENAI_API_KEY=sk-xxxxxxxxxxxxx

Verify it was created:

kubectl get secret app-api-keys -n <namespace>

Step 2 — Reference it in values.yaml:

deployments:
  app:
    secretName: "app-api-keys"

All key-value pairs from the secret are mounted as environment variables in the app deployment. You can also use global.secretName to share a secret across all services, or sealedSecrets for encrypted secret management.

Step 3 — Redeploy:

helm upgrade willow willow/webrix-helm \
  --namespace <namespace> \
  -f values.yaml \
  --wait

Multiple Gateways (Internal + External Access)

Some organizations need two separate access points to run — for example, one for VPN users (internal network) and one for non-VPN users (public internet). This is achieved with additional Ingress resources that use a different ingress controller. Both route to the same pods — no resource duplication.

Prerequisites: two ingress controllers in your cluster, each with a different ingressClassName (e.g. nginx-internal and nginx-external).

deployments:
  run:
    ingress:
      enabled: true
      className: "nginx-internal" # VPN users
      subdomain: "willow"
      path: "/"
      pathType: "ImplementationSpecific"
    ingresses:
      external:
        enabled: true
        className: "nginx-external" # Non-VPN users
        subdomain: "willow-ext"
        path: "/"
        pathType: "ImplementationSpecific"

This creates two Ingress resources for run:

Ingress Name	Hostname	Ingress Controller
`run`	`willow.<domain>`	`nginx-internal` (VPN)
`run-external`	`willow-ext.<domain>`	`nginx-external` (Public)

Point each hostname to its respective controller's IP:

willow.<domain>      → <internal-ingress-ip>
willow-ext.<domain>  → <external-ingress-ip>

Autoscaling (HPA)

Each service can scale automatically with a HorizontalPodAutoscaler. Autoscaling is off by default (services run at their fixed replicas count) — enable it per service under deployments.<service>.autoscaling:

deployments:
  run:
    autoscaling:
      enabled: true
      minReplicas: 2
      maxReplicas: 10
      targetCPUUtilizationPercentage: 70

Requirements: the metrics-server must be installed in your cluster, and each autoscaled service must define CPU resource requests (the chart's defaults already do).

Notes:

CPU-based by default. Memory-based autoscaling is intentionally disabled — Node/V8 and model workloads hold memory high regardless of load, so a memory target tends to scale up and never scale back down. Opt in per service with targetMemoryUtilizationPercentage if you understand the trade-off.
db-service connection limits. Each db-service pod opens up to ~10 PostgreSQL connections. Keep its maxReplicas low enough that peak connections (maxReplicas × 10) stay well under your database's max_connections. Raise both together, not just maxReplicas.
Advanced tuning. You can set a custom scaling behavior block per service to control scale-up/scale-down rates.

Custom Image Pull Secrets

If your cluster doesn't already have access to the quay.io/webrix registry, create an image pull secret:

kubectl create secret docker-registry webrix-registry \
  --namespace <namespace> \
  --docker-server=quay.io \
  --docker-username=<robot-username> \
  --docker-password=<robot-token> \
  --docker-email=unused@webrix.io

The chart references webrix-registry by default. To use a different name, set global.imagePullSecrets in your values.yaml.

Rotating the Encryption Key

ENCRYPTION_KEY is the master key db-service uses to encrypt all secrets at rest (OAuth tokens, API keys, webhook secrets, etc.) via envelope encryption: each secret gets its own random data key (DEK), and only that small DEK is wrapped by the master key. Rotating the master key means re-wrapping those DEKs — the secrets themselves are never re-encrypted or even decrypted in bulk.

Zero downtime

New and old keys are both accepted for decryption during the rotation window, so this can be done with a normal rolling deploy — no maintenance window required.

Step 1 — Generate a new key

openssl rand -base64 32

Step 2 — Deploy with both keys configured

Set ENCRYPTION_KEY to the new key and ENCRYPTION_KEY_PREVIOUS to the old one, then redeploy (helm upgrade, as in Step 3 — Install). From this point, all newly-encrypted data uses the new key; existing data is still readable via the old one.

deployments:
  db-service:
    env:
      ENCRYPTION_KEY: "<new-key>"
      ENCRYPTION_KEY_PREVIOUS: "<old-key>"

Use a Kubernetes secret rather than plaintext values.yaml for both values — see Working with Custom Secrets. ENCRYPTION_KEY_PREVIOUS accepts a comma-separated list if more than one retired key needs to stay readable.

Step 3 — Run the rewrap job

In the admin app, go to On-Prem → Encryption Key and click Rotate now. The job runs in the background on the server, so you can navigate away while it works; the card reports progress and the final result. Only one rotation runs at a time, across every db-service replica — they take a cluster-wide lock, so extra clicks can't start a second pass.

The button is only enabled once ENCRYPTION_KEY_PREVIOUS is configured (Step 2) — without a retired key there is nothing to rotate from.

This walks every table known to store encrypted data and re-wraps any row still on the old key. It's idempotent and safe to re-run — rows already on the new key are skipped. If it encounters a row it cannot verifiably re-wrap, it stops and reports the row rather than risk corrupting it (see "If the status card reports unreadable payloads" below).

Step 4 — Verify and clean up

Click Check status on the same card. This is a read-only pass that classifies every stored secret by the key that wrapped it, and reports:

Result	Meaning
All on current key (`Pending rewrap: 0`)	Rotation is complete — safe to proceed
Rotation incomplete (`Pending rewrap: > 0`)	Some data is still on a retired key — run Rotate now again
Action required (`Unreadable: > 0`)	Some data is wrapped by a key that is not configured — see below
Not checked yet	No scan has run against the keys currently configured

A scan result only ever describes the keys that were configured when it ran, so changing ENCRYPTION_KEY or ENCRYPTION_KEY_PREVIOUS resets the card to Not checked yet. Always re-run Check status after a redeploy — a green result from a previous rotation says nothing about the current one.

Do not destroy old key material until this reports zero pending and zero unreadable

ENCRYPTION_KEY_PREVIOUS is the only thing keeping un-rewrapped data readable. Removing it (or destroying the old key) while anything is still pending makes that data permanently unrecoverable.

Once it reports All on current key, remove ENCRYPTION_KEY_PREVIOUS from your values.yaml/secret, redeploy once more, and destroy the old key material.

If the status card reports unreadable payloads

This means data was encrypted with a key that is no longer configured — usually because ENCRYPTION_KEY was replaced without setting ENCRYPTION_KEY_PREVIOUS. The card lists the missing key id(s) and an example location. Add the missing key(s) to ENCRYPTION_KEY_PREVIOUS (comma-separated), redeploy, and check again. Rotation deliberately refuses to touch these rows, so the data is still recoverable as long as you have the old key.

TLS / Custom CA

If your network uses TLS inspection with a private certificate authority, add the CA certificate so the services trust internal endpoints:

global:
  caCertificate: |
    -----BEGIN CERTIFICATE-----
    MIIDxTCCAq2gAwIBAgI...
    -----END CERTIFICATE-----

The chart mounts the certificate and sets NODE_EXTRA_CA_CERTS automatically.

Troubleshooting

Symptom	Cause	Fix
Pods stuck in `Pending`	Insufficient cluster resources	Check node capacity: `kubectl describe pod <pod> -n <namespace>`
`ImagePullBackOff`	No registry access	Verify the `webrix-registry` pull secret (see Custom Image Pull Secrets)
Pod crashloops on startup	Configuration error in `values.yaml`	Review pod logs: `kubectl logs <pod> -n <namespace>`
Cannot access services	DNS not pointing to ingress	Verify DNS records resolve to the ingress controller's external IP
Ingress returns 404 / no host	Ingress controller not installed	Ensure an ingress controller is running and the `className` matches
TLS certificate errors	Certificate not configured / trusted	Check certificate config; for private CAs see TLS / Custom CA
Database connection failures	Wrong credentials or unreachable DB	In-cluster: `kubectl logs webrix-postgresql-0 -n <namespace>`. External: verify the `DATABASE_URL` secret values and network connectivity

Inspecting Deployments

kubectl get pods -n <namespace>
kubectl describe pod <pod-name> -n <namespace>

# Component logs
kubectl logs -n <namespace> deployment/app        --tail=100 -f
kubectl logs -n <namespace> deployment/connect     --tail=100 -f
kubectl logs -n <namespace> deployment/run         --tail=100 -f
kubectl logs -n <namespace> deployment/db-service  --tail=100 -f

Helm Operations

# List releases
helm list -n <namespace>

# Release history
helm history willow -n <namespace>

# Roll back to a previous revision
helm rollback willow <revision> -n <namespace>

Getting Help

If you continue to experience issues, contact Willow support with:

Kubernetes version: kubectl version
Helm version: helm version
Pod status and logs
Your values.yaml (redact sensitive information)

On-Prem Deployment

Why On-Prem?

How It Works

Setup Guide

Prerequisites

Network Requirements

Step 1 — Add the Helm Repository

Step 2 — Create values.yaml

Configuration reference

Step 3 — Install

Step 4 — Verify

Step 5 — Set Up DNS and Access Your Services

Advanced

External Database

Working with Custom Secrets

Multiple Gateways (Internal + External Access)

Autoscaling (HPA)

Custom Image Pull Secrets

Rotating the Encryption Key

TLS / Custom CA

Troubleshooting

Inspecting Deployments

Helm Operations

Getting Help

On this page

Was this helpful?

Why On-Prem?​

How It Works​

Setup Guide​

Prerequisites​

Network Requirements​

Step 1 — Add the Helm Repository​

Step 2 — Create values.yaml​

Configuration reference​

Step 3 — Install​

Step 4 — Verify​

Step 5 — Set Up DNS and Access Your Services​

Advanced​

External Database​

Working with Custom Secrets​

Multiple Gateways (Internal + External Access)​

Autoscaling (HPA)​

Custom Image Pull Secrets​

Rotating the Encryption Key​

TLS / Custom CA​

Troubleshooting​

Inspecting Deployments​

Helm Operations​

Getting Help​

On this page

Was this helpful?

Why On-Prem?

How It Works

Setup Guide

Prerequisites

Network Requirements

Step 1 — Add the Helm Repository

Step 2 — Create values.yaml

Configuration reference

Step 3 — Install

Step 4 — Verify

Step 5 — Set Up DNS and Access Your Services

Advanced

External Database

Working with Custom Secrets

Multiple Gateways (Internal + External Access)

Autoscaling (HPA)

Custom Image Pull Secrets

Rotating the Encryption Key

TLS / Custom CA

Troubleshooting

Inspecting Deployments

Helm Operations

Getting Help