Hybrid Deployment

Run MCP tool execution on your own infrastructure while keeping management on Willow SaaS. Only the run service is deployed on-prem — the admin app, connect, and db-service stay on SaaS.

Why Hybrid?

Data stays on-prem — tool calls (API keys, database queries, internal data) execute inside your network and never leave it
Zero management overhead — updates, database, SSO, and admin UI are all managed by Willow
Compliance — satisfies data residency and network isolation requirements without a full on-prem deployment
Simple operations — one stateless pod to run, no database to manage

How It Works

┌──────────────────────────────────────────────────────────┐
│  Willow SaaS                                              │
│                                                          │
│  ┌──────────┐    ┌─────────────┐    ┌─────────────────┐  │
│  │   Admin   │──▶│  db-service  │◀──│     connect     │  │
│  │   App     │   └──────┬──────┘    └──┬──────────────┘  │
│  └──────────┘           │              │                 │
│              gateway_settings.url   /api/on-prem-        │
│              (SaaS → on-prem)       db-service/*         │
│                         │           (on-prem → SaaS)     │
└─────────────────────────┼──────────────┼─────────────────┘
                          │              ▲
                          ▼              │
┌─────────────────────────┼──────────────┼─────────────────┐
│  Your Kubernetes cluster│              │                  │
│                    ┌────┴──────────────┴───┐              │
│                    │         run           │              │
│  MCP clients ────▶ │  (tool execution,     │              │
│  (Claude, Cursor)  │   MCP protocol)       │              │
│                    └──────────────────────┘              │
└──────────────────────────────────────────────────────────┘

On-prem run → SaaS: run reaches db-service through connect's authenticated proxy (/api/on-prem-db-service/*).

SaaS → on-prem run (recommended): db-service calls run for live tool listing, tool testing, guard evaluation, condition lookups, and MCP server setup, using the org's gateway URL. Tool execution and runtime guard enforcement never depend on it, so the platform keeps working without it — but most of the admin experience degrades. See What You Lose Without Inbound before deciding to block it, and Isolated Mode if you must.

MCP clients → on-prem run: Users connect directly to the on-prem run endpoint. OAuth authentication is handled by the SaaS connect service by default; you can optionally serve the OAuth flow from your own run gateway so the code exchange and token encryption stay inside your network — see On-Prem OAuth (Run-Hosted Connect).

Setup Guide

Prerequisites

A Kubernetes cluster with an ingress controller
An organization on Willow SaaS
DNS pointing your run subdomain to the cluster ingress (e.g. willow.your-domain.com)

Network Requirements

The on-prem run service communicates with Willow SaaS in both directions. Make sure the following traffic is allowed:

Outbound from your cluster (HTTPS / 443):

To Willow SaaS (*.withwillow.ai) — db-service proxy and OAuth discovery
To the third-party APIs your tools call (GitHub, Slack, Jira, etc.)
If you enable AWS KMS integration: to the AWS KMS endpoint (kms.<region>.amazonaws.com)

Inbound to willow.<YOUR_DOMAIN> (HTTPS / 443):

From your end users (MCP clients such as Claude and Cursor) — required.
From Willow SaaS — recommended. Allow the Willow SaaS egress IP for your org's region:
- *.withwillow.ai (US): 3.130.252.122
- *.eu.withwillow.ai (EU): 3.120.156.158
This is a single, stable NAT egress IP per region — one static address, not a range, and it does not change with releases or scaling. Allowing it is the difference between a fully working admin console and a partly manual one; see What You Lose Without Inbound. If your policy genuinely forbids it, see Isolated Mode.
From your managed background agent platform, if you use one. Agents on the Claude, AWS Bedrock AgentCore, Cursor, or Willow Agents platforms run outside your network and connect to run as ordinary MCP clients, so they need inbound of their own. Allowlisting Willow's NAT IP does not cover them, because the traffic originates from the platform's cloud rather than from Willow SaaS. If you cannot open inbound for a vendor cloud, run agents inside your own cluster with the agent harness.

Valid TLS required on the run endpoint

Willow SaaS (tool listing / guard evaluation) and MCP OAuth clients require the run endpoint to be served over valid, publicly-trusted HTTPS. Terminate TLS at your ingress with a real certificate — e.g. cert-manager + Let's Encrypt, an ACM certificate on an AWS load balancer, or a proxy such as Cloudflare. A self-signed certificate will cause SaaS→run calls and OAuth discovery to fail.

Allow long-lived streaming (SSE) on the run route — don't buffer it or time it out

The MCP protocol uses a streaming server→client channel: the client opens a GET /mcp Server-Sent Events stream (content-type: text/event-stream) that stays open and must be flushed to the client immediately. If any proxy in front of run buffers the response, or times out the idle stream, MCP clients hang on a cold connection and fail (commonly at ~60s), even though tool listing itself completes in ~1s on the server. Requests made with plain curl tools/list look fast because they don't hold the SSE stream open — so this is easy to misdiagnose as a server problem.

There are two independent knobs to get right — buffering and read timeout:

NGINX Ingress: buffering is already off by default in ingress-nginx, so the more common culprit is the default 60s proxy-read-timeout, which drops an idle SSE stream. Raise the timeouts (and keep buffering off) on the run Ingress:
```
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-buffering: "off" # default, set explicitly if your ConfigMap turned it on
```
Istio / Envoy: Envoy streams by default — do not enable a response buffer filter on the run route, and set a generous route timeout (e.g. 0s to disable, or a large value) so long-lived streams aren't cut.
Cloud load balancers (ALB/NLB, GCLB, etc.): raise the idle timeout well above your client's timeout (the AWS ALB default is 60s), and don't front the run hostname with a buffering layer.
Cloudflare / other CDNs: ensure the run hostname is not proxied through a caching/buffering layer, or exempt text/event-stream from buffering.

To verify, open the stream directly and watch that headers arrive within a second or two and the connection stays open past 60s:

curl -N -H "Accept: text/event-stream" https://willow.<YOUR_DOMAIN>/mcp

If that stalls before headers arrive, a proxy is buffering; if it connects fast but drops at ~60s, a read/idle timeout is too low.

What You Lose Without Inbound

Blocking SaaS→run inbound is safe for your data path and costly for everything else. Nothing below affects a tool call made by a real user through an MCP client — but all of it is part of how admins build, test, and maintain the platform day to day.

What	Impact without inbound
Live tool listing (integration Tools tab)	Falls back to the tools last synced on the integration. New or renamed tools on an upstream MCP server won't appear until someone re-syncs from an in-network browser.
Tool refresh / re-sync	No longer happens server-side. An admin has to open the MCP server in the admin app from a browser inside your network so the browser can collect directly and push results back.
Testing a tool (Test Run in the admin console)	Unavailable. Admins cannot verify a connector, its credentials, or a parameter mapping from the console, and have to test through a real MCP client instead.
Guard playground (build-time guard testing)	Unavailable. Guard rules must be written without a dry run and validated only once they are live. Runtime enforcement is unaffected.
Condition option pickers	Access-rule and policy conditions that populate their values live from the connector cannot load options. Admins must type values by hand and get no validation that they exist.
Adding a new MCP server	The setup wizard's reachability probe and OAuth discovery both run through `run`. The server still saves, but it finishes with a "not reachable from gateway" warning, collects no tools, and cannot auto-detect its auth requirements.
Gateway health	The admin app cannot report whether your gateway is up. You lose Willow-side monitoring of your own runtime and find out about an outage from your users.
Managed background agents	Agents hosted outside your network cannot reach the gateway at all. See the note below — this one is not solved by allowlisting Willow's NAT IP.

Background agents need their own inbound

Willow registers your gateway URL with the agent platform at sync time, and the agent runtime then calls it directly as an MCP client. For agents on Claude, AWS Bedrock AgentCore, Cursor, or Willow Agents, that runtime lives in the platform's cloud, not in Willow SaaS — so Willow's NAT egress IP does not cover it, and an ingress locked to that IP will still block every agent tool call.

If you want background agents without opening inbound to a vendor cloud, deploy the agent harness in your own cluster. Agents then run beside run, and the harness can even be configured to poll Willow rather than receive inbound.

Isolated Mode (no inbound from Willow SaaS)

A last resort, not a default

Isolated mode exists for organizations whose policy forbids any inbound from a SaaS provider. It is a permanently degraded configuration: read What You Lose Without Inbound first. If your objection is to opening a broad range, note that the allowlist is a single static IP — most customers who start in isolated mode end up allowlisting it later.

If you truly cannot allow it, open inbound to willow.<YOUR_DOMAIN> only from your MCP clients and skip the Willow SaaS egress allowlist entirely. This is a network posture, not a separate build — the Helm values are identical to the standard setup below.

What still works:

MCP clients → run: tool discovery and tool execution
Runtime guard enforcement on every tool call — guards are evaluated locally inside run, so blocking/redaction/warnings all apply
MCP OAuth authentication — driven by SaaS connect; the end user's browser reaches connect, so no SaaS→run inbound is needed
run → SaaS (db-service proxy) for config, token exchange, and logging — this is outbound from your cluster
Audit logging of every tool call

Mark the gateway as isolated. In Settings → Gateway, turn on Isolated gateway on the external gateway entry. This is optional but recommended once you have committed to the posture: Willow then never opens a connection to your gateway, so admin screens stop waiting on calls that will time out and instead offer the in-network browser flow immediately. The Unreachable health badge is replaced by an Isolated badge, since an unreachable warning would otherwise read as a fault when it is the configured behavior.

Turning it off again is a toggle, not a redeploy — allowlist the region's egress IP (see Network Requirements) and switch Isolated gateway off to restore every feature in the table above.

Step 1 — Configure Gateway in the Admin App

Log in to the Willow admin app (app.withwillow.ai)
Go to Settings → On-Prem
Click Configure Gateway
Set the Runtime URL to your on-prem run's public URL (e.g. https://willow.your-domain.com)
Click Generate Secret and copy the generated secret value — you'll use it in the next step

This stores the gateway URL and an encrypted auth secret on the SaaS side so that SaaS services can communicate with your on-prem run, and vice versa.

Step 2 — Create values.yaml

Replace the placeholders with your values:

<ORG_SLUG> — your organization slug (visible in your admin app URL or settings page)
<YOUR_DOMAIN> — your domain where the ingress is exposed

global:
  domain:
    host: "<YOUR_DOMAIN>"
  org: "<ORG_SLUG>"
  OPENAI_API_KEY: ""

deployments:
  app:
    enabled: false
  connect:
    enabled: false
  db-service:
    enabled: false
  run:
    enabled: true
    ingress:
      # Public host = <subdomain>.<domain>. The chart default is "webrix";
      # set it to "willow" so the run endpoint is https://willow.<YOUR_DOMAIN>
      # (matching the URLs referenced throughout this guide).
      subdomain: "willow"
    env:
      AUTH_SECRET: "<gateway-secret-from-admin-app>"
      PORT: "3000"
      LOG_LEVEL: "info"
      DB_SERVICE_URL: "https://<ORG_SLUG>.withwillow.ai/api/on-prem-db-service"
      CONNECT_URL: "https://<ORG_SLUG>.withwillow.ai"

Configuration reference

Value	Purpose
`global.domain.host`	Your domain. Combined with the run `ingress.subdomain` to build the run ingress hostname (`<subdomain>.<domain>`) and the service's `BASE_URL`.
`global.org`	Your org slug. Must match the SaaS org exactly. Used for org identification across all run operations.
`global.OPENAI_API_KEY`	Optional. Required only if you use AI-powered guardrails.
`DB_SERVICE_URL`	How run reaches db-service — proxied through connect at `/api/on-prem-db-service/*`.
`CONNECT_URL`	The SaaS connect URL for your org. Used for MCP OAuth discovery so clients can authenticate.

Auto-generated values

The following are generated automatically from your configuration — do not set them manually:

BASE_URL — derived from the run ingress host, <deployments.run.ingress.subdomain>.<global.domain.host>
ON_PREM — defaults to true
ORG — derived from global.org

Step 3 — Install

helm repo add webrix https://webrix-ai.github.io/webrix-helm
helm repo update

helm upgrade --install webrix webrix/webrix-helm \
  --namespace <namespace> \
  --create-namespace \
  -f values.yaml \
  --wait

Prefer an OCI registry?

The chart is also published as an OCI artifact to oci://ghcr.io/webrix-ai/charts. If your organization standardizes on OCI registries, skip helm repo add and reference the chart directly by its oci:// URL — it's the exact same chart:

helm upgrade --install webrix oci://ghcr.io/webrix-ai/charts/webrix-helm \
  --namespace <namespace> \
  --create-namespace \
  -f values.yaml \
  --wait

Step 4 — Verify

Pod health:

kubectl get pods -n <namespace>
# Expected: run-xxx   1/1   Running

Logs:

kubectl logs -n <namespace> -l app=run --tail=50

Look for Server listening at http://0.0.0.0:3000 and 200 responses on /healthz and /readyz.

Connectivity to SaaS:

kubectl exec -n <namespace> deploy/run -- \
  wget -qO- --header="Authorization: <AUTH_SECRET>" \
  "https://<ORG_SLUG>.withwillow.ai/api/on-prem-db-service/healthz"

A JSON response confirms run can reach db-service through the SaaS proxy.

MCP OAuth discovery:

curl https://willow.<YOUR_DOMAIN>/.well-known/oauth-protected-resource

Verify:

resource points to your on-prem run URL (https://willow.<YOUR_DOMAIN>)
authorization_servers points to SaaS connect (https://<ORG_SLUG>.withwillow.ai)

Tools tab in the admin app:

Open any MCP server in the admin app and check the Tools tab. Tools should load without errors.

TLS / Custom CA

If your network uses TLS inspection with a private certificate authority, add the CA certificate so run trusts SaaS endpoints:

global:
  caCertificate: |
    -----BEGIN CERTIFICATE-----
    MIIDxTCCAq2gAwIBAgI...
    -----END CERTIFICATE-----

The chart mounts the certificate and sets NODE_EXTRA_CA_CERTS automatically.

Advanced

Using a Kubernetes Secret for the Auth Secret

If you need to create or recreate the gateway auth secret manually (e.g. rotating secrets, automation pipelines), create a Kubernetes secret with the gateway secret from the admin app:

kubectl create secret generic webrix-secrets \
  --namespace <namespace> \
  --from-literal=AUTH_SECRET='<gateway-secret-from-admin-app>'

Reference it in your values.yaml:

deployments:
  run:
    secretName: "webrix-secrets"

Values from the secret override the ConfigMap, so AUTH_SECRET from the secret takes precedence over global.dbAuthSecret. You can also use global.secretName to share a secret across all services, or sealedSecrets for encrypted secret management.

Custom Image Pull Secrets

If your cluster doesn't already have access to the quay.io/webrix registry, create an image pull secret:

kubectl create secret docker-registry webrix-registry \
  --namespace <namespace> \
  --docker-server=quay.io \
  --docker-username=<robot-username> \
  --docker-password=<robot-token> \
  --docker-email=unused@webrix.io

The chart references webrix-registry by default. To use a different secret name, set global.imagePullSecrets in your values.yaml.

AWS KMS Integration (Write-Only KMS)

Optional

Most hybrid deployments don't need this. Enable it only if your security policy requires that Willow SaaS can never decrypt your secrets.

By default, the SaaS db-service decrypts tokens and returns them to your on-prem run over the authenticated channel. With Write-Only KMS, you bring your own AWS KMS key: Willow SaaS encrypts secrets with it but can never decrypt them — only your on-prem run service can. Secrets stay encrypted at rest in SaaS Postgres in the existing EncryptedPayload format, and plaintext exists only inside your cluster.

The permission split

The two sides get different permissions on the same customer-owned key. This is the part that's easy to get wrong — your run service does not need encrypt permissions.

Who	Where it runs	Permissions on your KMS key
Your run service	Your EKS (via IRSA)	`kms:Decrypt` (+ `kms:DescribeKey`) — decrypt only
Willow SaaS db-service	Willow cloud	`kms:GenerateDataKey` + `kms:Encrypt` (+ `kms:DescribeKey`) — encrypt only, never Decrypt

When an OAuth token expires, your run service decrypts the refresh token locally, refreshes it with the provider directly, and sends the new tokens back to the SaaS db-service to re-encrypt and store — so run never needs GenerateDataKey.

Step 1 — Create a KMS key in your AWS account

Create a symmetric encryption key (or reuse an existing one) in the same AWS region you'll configure on the run service. Note its key ARN, e.g. arn:aws:kms:us-east-1:<YOUR_ACCOUNT_ID>:key/<KEY_ID>.

Step 2 — Create the run IRSA role (Decrypt only)

Create an IAM role trusted by your EKS cluster's OIDC provider and bound to the run service account. Use a trust policy like this — replace <OIDC_ID> with your cluster's OIDC provider ID (EKS → your cluster → Overview → OpenID Connect provider URL), and set the sub to match the run service account (see the note below):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<YOUR_ACCOUNT_ID>:oidc-provider/oidc.eks.<YOUR_REGION>.amazonaws.com/id/<OIDC_ID>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.<YOUR_REGION>.amazonaws.com/id/<OIDC_ID>:aud": "sts.amazonaws.com",
          "oidc.eks.<YOUR_REGION>.amazonaws.com/id/<OIDC_ID>:sub": "system:serviceaccount:<namespace>:webrix-run"
        }
      }
    }
  ]
}

Attach a decrypt-only permissions policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RunDecryptOnly",
      "Effect": "Allow",
      "Action": ["kms:Decrypt", "kms:DescribeKey"],
      "Resource": "arn:aws:kms:<YOUR_REGION>:<YOUR_ACCOUNT_ID>:key/<KEY_ID>"
    }
  ]
}

The trust sub must match the run service account name

The sub condition must equal system:serviceaccount:<namespace>:<run-service-account>. The run service account name is whatever you set in deployments.run.serviceAccount.name (Step 5) — e.g. webrix-run. If you leave serviceAccount.name unset, the chart defaults it to run-sa, so the sub must then be system:serviceaccount:<namespace>:run-sa. A mismatch here is the most common cause of sts:AssumeRoleWithWebIdentity and decrypt (AccessDenied) failures.

Step 3 — Grant Willow SaaS encrypt access

Does a Willow account need access to my key? Yes — but only to encrypt. There are two ways to grant it; pick one.

Option A — Provide scoped IAM credentials to Willow (no cross-account access needed). In the admin app's External KMS dialog (Step 4), you enter an AWS access key / secret for an IAM user in your own account that has kms:GenerateDataKey + kms:Encrypt on the key. Willow stores these encrypted and uses them to encrypt. With this option, no Willow AWS account touches your key directly.

Option B — Cross-account key policy. Grant Willow's SaaS principal (AWS account 992382826040) encrypt access via your key policy. Use this if you'd rather not hand over static IAM credentials.

{
  "Sid": "AllowWillowSaaSEncryptOnly",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::992382826040:role/<WILLOW_SAAS_PRINCIPAL>"
  },
  "Action": ["kms:GenerateDataKey", "kms:Encrypt", "kms:DescribeKey"],
  "Resource": "*"
}

Confirm the exact principal ARN

The exact IAM principal ARN on the Willow side depends on the SaaS configuration. Ask your Willow contact for the precise ARN before applying Option B — don't assume it.

Also make sure your key policy allows your own run role to decrypt (from Step 2), since the key is in your account:

{
  "Sid": "AllowRunDecryptOnly",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::<YOUR_ACCOUNT_ID>:role/<YOUR_RUN_IRSA_ROLE>"
  },
  "Action": ["kms:Decrypt", "kms:DescribeKey"],
  "Resource": "*"
}

Step 4 — Configure External KMS in the admin app

In the admin app, go to Settings → On-Prem and edit your gateway
Click Configure KMS
Enter the same KMS key ARN and AWS region you configured on the run service
If you chose Option A, enter the IAM access key / secret with GenerateDataKey + Encrypt permissions

This tells the SaaS db-service to route encryption for this org through your key.

Step 5 — Point the run service at the key

Add the KMS settings and IRSA service account annotation to your values.yaml:

deployments:
  run:
    serviceAccount:
      create: true
      # SA name. Must match the trust policy `sub` from Step 2
      # (system:serviceaccount:<namespace>:webrix-run). If omitted, the chart
      # defaults the name to "run-sa".
      name: webrix-run
      annotations:
        # The decrypt-only IRSA role from Step 2
        eks.amazonaws.com/role-arn: "arn:aws:iam::<YOUR_ACCOUNT_ID>:role/<YOUR_RUN_IRSA_ROLE>"
    env:
      # Enables local KMS decryption on the run service
      KMS_KEY_ID: "arn:aws:kms:<YOUR_REGION>:<YOUR_ACCOUNT_ID>:key/<KEY_ID>"
      AWS_REGION: "<YOUR_REGION>"

caution

Do not set ENCRYPTION_KEY (that selects the static-key on-prem mode instead of KMS), and do not set AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY — let IRSA supply credentials to the run pod.

Once KMS_KEY_ID is set, the run service automatically requests encrypted payloads from the SaaS db-service and decrypts them locally — no additional flag is required.

Troubleshooting

Symptom	Cause	Fix
404 on tool listing	`DB_SERVICE_URL` missing `/api/` prefix	URL must be `https://<org>.withwillow.ai/api/on-prem-db-service`
401 from SaaS proxy	Auth secret mismatch	Verify `AUTH_SECRET` in `webrix-secrets` matches the gateway secret from the admin app
401 on inbound calls to run	Gateway not configured on SaaS	Configure gateway in admin app → Settings → On-Prem
Gateway shows Unreachable in admin app; Tools tab won't list live	No inbound from Willow SaaS to run — either Isolated Mode or the SaaS egress IP isn't allowlisted (note: `3.136.98.54` from older docs is stale and no longer used)	Allowlist the region's SaaS egress IP — `3.130.252.122` (US, `.withwillow.ai`) or `3.120.156.158`* (EU, `.eu.withwillow.ai`) — to restore live listing and the rest of the admin console. Tool calls and runtime guards keep working either way; if you are staying isolated on purpose, turn on Isolated gateway* in Settings → Gateway so the badge reads Isolated instead
Test Run, guard playground, condition pickers, or tool refresh fail with a gateway error	Same missing SaaS→run inbound as the row above. These have no fallback: only tool listing can be driven from an in-network browser	Allowlist the region's SaaS egress IP. See What You Lose Without Inbound for the full list
Background agent tool calls fail, but end-user MCP clients work	The agent runtime is hosted by Claude, AWS, Cursor, or Willow Agents and reaches your gateway from the platform's cloud — Willow's NAT egress IP does not cover it	Allow inbound from your agent platform, or run agents in your own cluster with the agent harness
MCP clients can't authenticate	Wrong `CONNECT_URL`	Must be `https://<org>.withwillow.ai`
MCP clients can't connect	DNS or ingress issue	Verify `willow.<domain>` resolves and the ingress controller is healthy
MCP cold start hangs ~60s, then times out (`Daemon request timeout`, or MCP error `-32001 Request timed out`); warm/repeat calls succeed	A proxy in front of `run` drops or buffers the long-lived `GET /mcp` SSE stream — usually a too-low read/idle timeout (ingress-nginx defaults to 60s), sometimes response buffering	Raise `proxy-read-timeout`/`proxy-send-timeout` (or the LB idle timeout) and keep response buffering off on the run route (see SSE callout above). Confirm with `curl -N -H "Accept: text/event-stream" .../mcp` — if headers stall it's buffering; if it drops at ~60s it's a timeout
Pod crashloops on startup	Missing `AUTH_SECRET`	Ensure the `webrix-secrets` Kubernetes secret exists with the `AUTH_SECRET` key
Wrong org data	`global.org` doesn't match SaaS	Must be the exact org slug shown in the admin app
Extra pods running	Other services not disabled	Set `app`, `connect`, `db-service` to `enabled: false`
Secrets fail to decrypt on run (KMS mode)	Run role lacks `kms:Decrypt`, wrong `AWS_REGION`, or key policy missing run principal	Verify the IRSA role, `KMS_KEY_ID`/`AWS_REGION`, and the key policy grant from the AWS KMS section
`AssumeRoleWithWebIdentity` `AccessDenied` on run (KMS mode)	IRSA trust policy `sub` doesn't match the run service account name	Set the trust `sub` to `system:serviceaccount:<namespace>:<sa>`, where `<sa>` is `deployments.run.serviceAccount.name` (or `run-sa` if unset)

Hybrid Deployment

Why Hybrid?

How It Works

Setup Guide

Prerequisites

Network Requirements

What You Lose Without Inbound

Isolated Mode (no inbound from Willow SaaS)

Step 1 — Configure Gateway in the Admin App

Step 2 — Create values.yaml

Configuration reference

Step 3 — Install

Step 4 — Verify

TLS / Custom CA

Advanced

Using a Kubernetes Secret for the Auth Secret

Custom Image Pull Secrets

AWS KMS Integration (Write-Only KMS)

The permission split

Step 1 — Create a KMS key in your AWS account

Step 2 — Create the run IRSA role (Decrypt only)

Step 3 — Grant Willow SaaS encrypt access

Step 4 — Configure External KMS in the admin app

Step 5 — Point the run service at the key

Troubleshooting

On this page

Was this helpful?

Why Hybrid?​

How It Works​

Setup Guide​

Prerequisites​

Network Requirements​

What You Lose Without Inbound​

Isolated Mode (no inbound from Willow SaaS)​

Step 1 — Configure Gateway in the Admin App​

Step 2 — Create values.yaml​

Configuration reference​

Step 3 — Install​

Step 4 — Verify​

TLS / Custom CA​

Advanced​

Using a Kubernetes Secret for the Auth Secret​

Custom Image Pull Secrets​

AWS KMS Integration (Write-Only KMS)​

The permission split​

Step 1 — Create a KMS key in your AWS account​

Step 2 — Create the run IRSA role (Decrypt only)​

Step 3 — Grant Willow SaaS encrypt access​

Step 4 — Configure External KMS in the admin app​

Step 5 — Point the run service at the key​

Troubleshooting​

On this page

Was this helpful?

Why Hybrid?

How It Works

Setup Guide

Prerequisites

Network Requirements

What You Lose Without Inbound

Isolated Mode (no inbound from Willow SaaS)

Step 1 — Configure Gateway in the Admin App

Step 2 — Create values.yaml

Configuration reference

Step 3 — Install

Step 4 — Verify

TLS / Custom CA

Advanced

Using a Kubernetes Secret for the Auth Secret

Custom Image Pull Secrets

AWS KMS Integration (Write-Only KMS)

The permission split

Step 1 — Create a KMS key in your AWS account

Step 2 — Create the run IRSA role (Decrypt only)

Step 3 — Grant Willow SaaS encrypt access

Step 4 — Configure External KMS in the admin app

Step 5 — Point the run service at the key

Troubleshooting