Hybrid Deployment

Run MCP tool execution on your own infrastructure while keeping management on Willow SaaS. Only the run service is deployed on-prem — the admin app, connect, and db-service stay on SaaS.

Why Hybrid?

Data stays on-prem — tool calls (API keys, database queries, internal data) execute inside your network and never leave it
Zero management overhead — updates, database, SSO, and admin UI are all managed by Willow
Compliance — satisfies data residency and network isolation requirements without a full on-prem deployment
Simple operations — one stateless pod to run, no database to manage

How It Works

┌──────────────────────────────────────────────────────────┐
│  Willow SaaS                                              │
│                                                          │
│  ┌──────────┐    ┌─────────────┐    ┌─────────────────┐  │
│  │   Admin   │──▶│  db-service  │◀──│     connect     │  │
│  │   App     │   └──────┬──────┘    └──┬──────────────┘  │
│  └──────────┘           │              │                 │
│              gateway_settings.url   /api/on-prem-        │
│              (SaaS → on-prem)       db-service/*         │
│                         │           (on-prem → SaaS)     │
└─────────────────────────┼──────────────┼─────────────────┘
                          │              ▲
                          ▼              │
┌─────────────────────────┼──────────────┼─────────────────┐
│  Your Kubernetes cluster│              │                  │
│                    ┌────┴──────────────┴───┐              │
│                    │         run           │              │
│  MCP clients ────▶ │  (tool execution,     │              │
│  (Claude, Cursor)  │   MCP protocol)       │              │
│                    └──────────────────────┘              │
└──────────────────────────────────────────────────────────┘

On-prem run → SaaS: run reaches db-service through connect's authenticated proxy (/api/on-prem-db-service/*).

SaaS → on-prem run: db-service calls run for proxy-MCP tool listing and guard evaluation using the org's gateway URL.

MCP clients → on-prem run: Users connect directly to the on-prem run endpoint. OAuth authentication is handled by the SaaS connect service.

Setup Guide

Prerequisites

A Kubernetes cluster with an ingress controller
An organization on Willow SaaS
DNS pointing your run subdomain to the cluster ingress (e.g. willow.your-domain.com)

Network Requirements

The on-prem run service communicates with Willow SaaS in both directions. Make sure the following traffic is allowed:

Outbound from your cluster (HTTPS / 443):

To Willow SaaS (*.withwillow.ai) — db-service proxy and OAuth discovery
To the third-party APIs your tools call (GitHub, Slack, Jira, etc.)
If you enable AWS KMS integration: to the AWS KMS endpoint (kms.<region>.amazonaws.com)

Inbound to willow.<YOUR_DOMAIN> (HTTPS / 443):

From your end users (MCP clients such as Claude and Cursor)
From Willow SaaS, for tool listing and guard evaluation. If your ingress restricts inbound by IP, allow the Willow SaaS egress IP range. NAT IP: 3.136.98.54.

Step 1 — Configure Gateway in the Admin App

Log in to the Willow admin app (app.withwillow.ai)
Go to Settings → On-Prem
Click Configure Gateway
Set the Runtime URL to your on-prem run's public URL (e.g. https://willow.your-domain.com)
Click Generate Secret and copy the generated secret value — you'll use it in the next step

This stores the gateway URL and an encrypted auth secret on the SaaS side so that SaaS services can communicate with your on-prem run, and vice versa.

Step 2 — Create values.yaml

Replace the placeholders with your values:

<ORG_SLUG> — your organization slug (visible in your admin app URL or settings page)
<YOUR_DOMAIN> — your domain where the ingress is exposed

global:
  domain:
    host: "<YOUR_DOMAIN>"
  org: "<ORG_SLUG>"
  OPENAI_API_KEY: ""

deployments:
  app:
    enabled: false
  connect:
    enabled: false
  db-service:
    enabled: false
  run:
    enabled: true
    env:
      AUTH_SECRET: "<gateway-secret-from-admin-app>"
      PORT: "3000"
      LOG_LEVEL: "info"
      DB_SERVICE_URL: "https://<ORG_SLUG>.withwillow.ai/api/on-prem-db-service"
      CONNECT_URL: "https://<ORG_SLUG>.withwillow.ai"

Configuration reference

Value	Purpose
`global.domain.host`	Your domain. Used to build the run ingress hostname (`willow.<domain>`) and the service's `BASE_URL`.
`global.org`	Your org slug. Must match the SaaS org exactly. Used for org identification across all run operations.
`global.OPENAI_API_KEY`	Optional. Required only if you use AI-powered guardrails.
`DB_SERVICE_URL`	How run reaches db-service — proxied through connect at `/api/on-prem-db-service/*`.
`CONNECT_URL`	The SaaS connect URL for your org. Used for MCP OAuth discovery so clients can authenticate.

Auto-generated values

The following are generated automatically from your configuration — do not set them manually:

BASE_URL — derived from willow.<global.domain.host>
ON_PREM — defaults to true
ORG — derived from global.org

Step 3 — Install

helm repo add webrix https://webrix-ai.github.io/webrix-helm
helm repo update

helm upgrade --install webrix webrix/webrix-helm \
  --namespace <namespace> \
  --create-namespace \
  -f values.yaml \
  --wait

Step 4 — Verify

Pod health:

kubectl get pods -n <namespace>
# Expected: run-xxx   1/1   Running

Logs:

kubectl logs -n <namespace> -l app=run --tail=50

Look for Server listening at http://0.0.0.0:3000 and 200 responses on /healthz and /readyz.

Connectivity to SaaS:

kubectl exec -n <namespace> deploy/run -- \
  wget -qO- --header="Authorization: <AUTH_SECRET>" \
  "https://<ORG_SLUG>.withwillow.ai/api/on-prem-db-service/healthz"

A JSON response confirms run can reach db-service through the SaaS proxy.

MCP OAuth discovery:

curl https://willow.<YOUR_DOMAIN>/.well-known/oauth-protected-resource

Verify:

resource points to your on-prem run URL (https://willow.<YOUR_DOMAIN>)
authorization_servers points to SaaS connect (https://<ORG_SLUG>.withwillow.ai)

Tools tab in the admin app:

Open any MCP server in the admin app and check the Tools tab. Tools should load without errors.

TLS / Custom CA

If your network uses TLS inspection with a private certificate authority, add the CA certificate so run trusts SaaS endpoints:

global:
  caCertificate: |
    -----BEGIN CERTIFICATE-----
    MIIDxTCCAq2gAwIBAgI...
    -----END CERTIFICATE-----

The chart mounts the certificate and sets NODE_EXTRA_CA_CERTS automatically.

Advanced

Using a Kubernetes Secret for the Auth Secret

If you need to create or recreate the gateway auth secret manually (e.g. rotating secrets, automation pipelines), create a Kubernetes secret with the gateway secret from the admin app:

kubectl create secret generic webrix-secrets \
  --namespace <namespace> \
  --from-literal=AUTH_SECRET='<gateway-secret-from-admin-app>'

Reference it in your values.yaml:

deployments:
  run:
    secretName: "webrix-secrets"

Values from the secret override the ConfigMap, so AUTH_SECRET from the secret takes precedence over global.dbAuthSecret. You can also use global.secretName to share a secret across all services, or sealedSecrets for encrypted secret management.

Custom Image Pull Secrets

If your cluster doesn't already have access to the quay.io/webrix registry, create an image pull secret:

kubectl create secret docker-registry webrix-registry \
  --namespace <namespace> \
  --docker-server=quay.io \
  --docker-username=<robot-username> \
  --docker-password=<robot-token> \
  --docker-email=unused@webrix.io

The chart references webrix-registry by default. To use a different secret name, set global.imagePullSecrets in your values.yaml.

AWS KMS Integration (Write-Only KMS)

Optional

Most hybrid deployments don't need this. Enable it only if your security policy requires that Willow SaaS can never decrypt your secrets.

By default, the SaaS db-service decrypts tokens and returns them to your on-prem run over the authenticated channel. With Write-Only KMS, you bring your own AWS KMS key: Willow SaaS encrypts secrets with it but can never decrypt them — only your on-prem run service can. Secrets stay encrypted at rest in SaaS Postgres in the existing EncryptedPayload format, and plaintext exists only inside your cluster.

The permission split

The two sides get different permissions on the same customer-owned key. This is the part that's easy to get wrong — your run service does not need encrypt permissions.

Who	Where it runs	Permissions on your KMS key
Your run service	Your EKS (via IRSA)	`kms:Decrypt` (+ `kms:DescribeKey`) — decrypt only
Willow SaaS db-service	Willow cloud	`kms:GenerateDataKey` + `kms:Encrypt` (+ `kms:DescribeKey`) — encrypt only, never Decrypt

When an OAuth token expires, your run service decrypts the refresh token locally, refreshes it with the provider directly, and sends the new tokens back to the SaaS db-service to re-encrypt and store — so run never needs GenerateDataKey.

Step 1 — Create a KMS key in your AWS account

Create a symmetric encryption key (or reuse an existing one) in the same AWS region you'll configure on the run service. Note its key ARN, e.g. arn:aws:kms:us-east-1:<YOUR_ACCOUNT_ID>:key/<KEY_ID>.

Step 2 — Create the run IRSA role (Decrypt only)

Create an IAM role trusted by your EKS cluster's OIDC provider and bound to the run service account. Attach a decrypt-only policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RunDecryptOnly",
      "Effect": "Allow",
      "Action": ["kms:Decrypt", "kms:DescribeKey"],
      "Resource": "arn:aws:kms:<YOUR_REGION>:<YOUR_ACCOUNT_ID>:key/<KEY_ID>"
    }
  ]
}

Step 3 — Grant Willow SaaS encrypt access

Does a Willow account need access to my key? Yes — but only to encrypt. There are two ways to grant it; pick one.

Option A — Provide scoped IAM credentials to Willow (no cross-account access needed). In the admin app's External KMS dialog (Step 4), you enter an AWS access key / secret for an IAM user in your own account that has kms:GenerateDataKey + kms:Encrypt on the key. Willow stores these encrypted and uses them to encrypt. With this option, no Willow AWS account touches your key directly.

Option B — Cross-account key policy. Grant Willow's SaaS principal (AWS account 992382826040) encrypt access via your key policy. Use this if you'd rather not hand over static IAM credentials.

{
  "Sid": "AllowWillowSaaSEncryptOnly",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::992382826040:role/<WILLOW_SAAS_PRINCIPAL>"
  },
  "Action": ["kms:GenerateDataKey", "kms:Encrypt", "kms:DescribeKey"],
  "Resource": "*"
}

Confirm the exact principal ARN

The exact IAM principal ARN on the Willow side depends on the SaaS configuration. Ask your Willow contact for the precise ARN before applying Option B — don't assume it.

Also make sure your key policy allows your own run role to decrypt (from Step 2), since the key is in your account:

{
  "Sid": "AllowRunDecryptOnly",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::<YOUR_ACCOUNT_ID>:role/<YOUR_RUN_IRSA_ROLE>"
  },
  "Action": ["kms:Decrypt", "kms:DescribeKey"],
  "Resource": "*"
}

Step 4 — Configure External KMS in the admin app

In the admin app, go to Settings → On-Prem and edit your gateway
Click Configure KMS
Enter the same KMS key ARN and AWS region you configured on the run service
If you chose Option A, enter the IAM access key / secret with GenerateDataKey + Encrypt permissions

This tells the SaaS db-service to route encryption for this org through your key.

Step 5 — Point the run service at the key

Add the KMS settings and IRSA service account annotation to your values.yaml:

deployments:
  run:
    serviceAccount:
      create: true
      name: webrix-run
      annotations:
        # The decrypt-only IRSA role from Step 2
        eks.amazonaws.com/role-arn: "arn:aws:iam::<YOUR_ACCOUNT_ID>:role/<YOUR_RUN_IRSA_ROLE>"
    env:
      # Enables local KMS decryption on the run service
      KMS_KEY_ID: "arn:aws:kms:<YOUR_REGION>:<YOUR_ACCOUNT_ID>:key/<KEY_ID>"
      AWS_REGION: "<YOUR_REGION>"

caution

Do not set ENCRYPTION_KEY (that selects the static-key on-prem mode instead of KMS), and do not set AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY — let IRSA supply credentials to the run pod.

Once KMS_KEY_ID is set, the run service automatically requests encrypted payloads from the SaaS db-service and decrypts them locally — no additional flag is required.

Troubleshooting

Symptom	Cause	Fix
404 on tool listing	`DB_SERVICE_URL` missing `/api/` prefix	URL must be `https://<org>.withwillow.ai/api/on-prem-db-service`
401 from SaaS proxy	Auth secret mismatch	Verify `AUTH_SECRET` in `webrix-secrets` matches the gateway secret from the admin app
401 on inbound calls to run	Gateway not configured on SaaS	Configure gateway in admin app → Settings → On-Prem
MCP clients can't authenticate	Wrong `CONNECT_URL`	Must be `https://<org>.withwillow.ai`
MCP clients can't connect	DNS or ingress issue	Verify `willow.<domain>` resolves and the ingress controller is healthy
Pod crashloops on startup	Missing `AUTH_SECRET`	Ensure the `webrix-secrets` Kubernetes secret exists with the `AUTH_SECRET` key
Wrong org data	`global.org` doesn't match SaaS	Must be the exact org slug shown in the admin app
Extra pods running	Other services not disabled	Set `app`, `connect`, `db-service` to `enabled: false`
Secrets fail to decrypt on run (KMS mode)	Run role lacks `kms:Decrypt`, wrong `AWS_REGION`, or key policy missing run principal	Verify the IRSA role, `KMS_KEY_ID`/`AWS_REGION`, and the key policy grant from the AWS KMS section

Hybrid Deployment

Why Hybrid?

How It Works

Setup Guide

Prerequisites

Network Requirements

Step 1 — Configure Gateway in the Admin App

Step 2 — Create values.yaml

Configuration reference

Step 3 — Install

Step 4 — Verify

TLS / Custom CA

Advanced

Using a Kubernetes Secret for the Auth Secret

Custom Image Pull Secrets

AWS KMS Integration (Write-Only KMS)

The permission split

Step 1 — Create a KMS key in your AWS account

Step 2 — Create the run IRSA role (Decrypt only)

Step 3 — Grant Willow SaaS encrypt access

Step 4 — Configure External KMS in the admin app

Step 5 — Point the run service at the key

Troubleshooting

On this page

Was this helpful?

Why Hybrid?​

How It Works​

Setup Guide​

Prerequisites​

Network Requirements​

Step 1 — Configure Gateway in the Admin App​

Step 2 — Create values.yaml​

Configuration reference​

Step 3 — Install​

Step 4 — Verify​

TLS / Custom CA​

Advanced​

Using a Kubernetes Secret for the Auth Secret​

Custom Image Pull Secrets​

AWS KMS Integration (Write-Only KMS)​

The permission split​

Step 1 — Create a KMS key in your AWS account​

Step 2 — Create the run IRSA role (Decrypt only)​

Step 3 — Grant Willow SaaS encrypt access​

Step 4 — Configure External KMS in the admin app​

Step 5 — Point the run service at the key​

Troubleshooting​

On this page

Was this helpful?

Why Hybrid?

How It Works

Setup Guide

Prerequisites

Network Requirements

Step 1 — Configure Gateway in the Admin App

Step 2 — Create values.yaml

Configuration reference

Step 3 — Install

Step 4 — Verify

TLS / Custom CA

Advanced

Using a Kubernetes Secret for the Auth Secret

Custom Image Pull Secrets

AWS KMS Integration (Write-Only KMS)

The permission split

Step 1 — Create a KMS key in your AWS account

Step 2 — Create the run IRSA role (Decrypt only)

Step 3 — Grant Willow SaaS encrypt access

Step 4 — Configure External KMS in the admin app

Step 5 — Point the run service at the key

Troubleshooting