TLS and Inbound Reachability

Getting a Willow endpoint working from the outside world is four separate things, and they fail independently:

DNS — the hostname resolves to your ingress controller's address
Network — a TCP connection to port 443 on that address is accepted
TLS — a publicly trusted certificate is served for that hostname
Routing — the ingress sends the request to a running pod

This page covers 2 and 3, which are the two that most often stall a deployment, and gives you a way to tell them apart. Use it from Hybrid Step 4 or On-Prem Step 4, and again from the verification step of either guide when something doesn't respond.

Nothing here is Willow-specific. If you already operate public HTTPS services on this cluster, most of it will be familiar and you can skip to Choosing a certificate.

Test it correctly first

Almost every long debugging session on this starts with a test that lied. Two rules:

Run the test from outside your own network. A laptop on the corporate Wi-Fi, a jump box inside the VPC, or a shell inside the cluster all reach the endpoint by a path your users and Willow SaaS do not have. "It works for me" from inside the network is not evidence that the endpoint is public. Use a machine on a home connection, or tether to a phone with Wi-Fi off.

Don't use -k or -L. They hide the two most common failures. -k suppresses exactly the certificate error you're trying to find, and -L follows a redirect that Willow's health check will not follow.

# The exact shape of the check Willow SaaS performs
curl -sS -w '\nhttp=%{http_code} redirect=%{redirect_url}\n' \
  --max-time 5 --max-redirs 0 \
  https://<your-run-hostname>/healthz

A pass is http=200 with an empty redirect and a body of {"status":"ok"}. The body matters: Willow rejects a 200 that isn't run-shaped, because that is the ingress answering rather than the run pod. Everything else maps to a specific layer:

What you see	What it means	Where to go
`Could not resolve host`	DNS record missing or removed	Fix DNS first — nothing else is testable
`Connection timed out`, `http=000`	Packets are being dropped by a firewall, security group, or load balancer with no rule	Inbound network
`Connection refused` (fast, not a timeout)	The address is reachable but nothing is listening on 443	Inbound network — usually a missing load balancer rule
`SSL certificate problem`, `self-signed certificate`	Network is fine. The certificate is missing, self-signed, or from a private CA	Choosing a certificate
`http=301` or `http=302`	A redirect is in the way. Willow does not follow redirects	Remove the HTTP-to-HTTPS redirect or auth layer on this path
`http=404`	Network and TLS are fine. The ingress has no rule for this hostname	Hostname mismatches
`http=503`	Ingress is working but has no healthy backend pod	No healthy backend
`http=200` with `{"status":"ok"}`	Working	Continue with the verification step in your deployment guide
`http=200` with an empty or HTML body	The ingress answered, not the run pod	Hostname mismatches — Gateway Settings shows Not the run service

Timeout and refused mean different things

This distinction saves hours. A machine that is reachable with nothing listening sends back a TCP reset immediately, and you see connection refused within milliseconds. A timeout means the packets vanished — something is silently discarding them, which is what firewalls and security groups do by default. If you see a timeout, stop looking at the cluster and start looking at the network.

Match your ingress controller

Read this before choosing a certificate — the ingress controller you run determines both the class name the chart must use and how certificates are supplied.

The chart defaults deployments.<service>.ingress.className to "nginx". That value must match an IngressClass that actually exists in your cluster. If it doesn't, the ingress object is created and then ignored by every controller: nothing is programmed, and the hostname returns a 404 or doesn't connect at all, with no error anywhere in the Willow release to explain it.

Check what your cluster actually has:

kubectl get ingressclass

Self-installed ingress-nginx

The chart's default. className: "nginx" (or whatever your installation named it — confirm with the command above), and certificates come from cert-manager or a secret you create yourself, as described below.

AKS application routing add-on

Don't enable this add-on just to attach a Key Vault certificate

If the cluster already has a self-installed ingress-nginx, leave className: nginx and install the certificate as a Kubernetes TLS secret (Option A). Turning the add-on on in addition creates a second NGINX controller and a second public IP, and it is easy to send Willow's ingress to the wrong one.

If your AKS cluster uses Azure's managed NGINX add-on, three things differ from a self-installed ingress-nginx, and getting any one wrong produces a silent failure:

The class is webapprouting.kubernetes.azure.com, not nginx.
Certificates come from Azure Key Vault via an annotation, not from cert-manager. Don't add a cert-manager.io/cluster-issuer annotation on these hosts.
The TLS secretName is not free-form. The add-on requires it to be keyvault- followed by the ingress resource name. The chart names each ingress after its service, so the value is keyvault-run for the run service, and keyvault-app / keyvault-connect for the other two on an on-prem deployment.

deployments:
  run:
    ingress:
      className: "webapprouting.kubernetes.azure.com"
      annotations:
        # Use the unversioned certificate URI so rotations are picked up automatically
        kubernetes.azure.com/tls-cert-keyvault-uri: "https://<vault-name>.vault.azure.net/certificates/<cert-name>"
      tls:
        - hosts:
            - "willow.<YOUR_DOMAIN>"
          # Must be "keyvault-" + the ingress name, which the chart sets to the service name
          secretName: "keyvault-run"

The add-on also requires the Azure Key Vault provider for Secrets Store CSI Driver to be enabled on the cluster. See Microsoft's application routing add-on documentation for enabling it and for loading the certificate into Key Vault.

Verify streaming separately on this add-on

The add-on restricts which NGINX snippet annotations it will accept. The chart's streaming settings (proxy-read-timeout, proxy-send-timeout, proxy-buffering) are standard ingress-nginx annotations rather than snippets, but confirm they took effect by running the SSE check from your deployment guide rather than assuming it.

Other controllers

For AWS Load Balancer Controller, Azure Application Gateway Ingress Controller, Traefik, or anything else, set className to the class that controller registers, and follow that controller's own documentation for TLS — several of them take the certificate from the cloud load balancer rather than from a Kubernetes secret, in which case ingress.tls stays empty and Option D applies.

On OpenShift, the chart can create a Route instead of an Ingress — see deployments.<service>.route in the chart values.

Choosing a certificate

Willow's ingress hostnames must serve a publicly trusted certificate. Both Willow SaaS and MCP clients such as Claude and Cursor reject anything else, and neither offers a way to skip verification. A self-signed certificate or one issued by your organization's internal CA will not work, even though it may look fine in a browser on a managed corporate laptop where that CA is already trusted.

The Helm chart ships no certificate configuration — ingress.tls is empty by default for every service. If you install without setting it, your ingress controller serves its own built-in fake certificate and every client rejects it.

Pick one of these. They are listed in the order that causes the fewest problems.

Option	Use when	Inbound needed to issue?
A. A certificate you already have	Your organization has a wildcard for the domain, or a normal certificate procurement process	No
B. cert-manager with a DNS-01 challenge	You want automatic renewal and your DNS provider is supported	No
C. cert-manager with an HTTP-01 challenge	You want automatic renewal and the hostname is already reachable from the internet	Yes
D. A certificate on the cloud load balancer	You already terminate TLS at an ALB, Application Gateway, or Cloud Load Balancer	No

HTTP-01 cannot work while inbound is blocked

This is the single most common way a deployment gets stuck. An HTTP-01 challenge requires Let's Encrypt to reach your hostname on port 80 from the public internet. If your firewall hasn't been opened yet, the challenge can never validate, and the Certificate sits at Ready: False indefinitely with no useful error. It looks like cert-manager is broken; it isn't.

If your inbound network isn't open yet, use Option A or Option B, or fix Inbound network first and then use Option C.

Option A — Install a certificate you already have

The fastest path, and usually available: most organizations that own a domain already have a wildcard certificate for it.

You need two files: the full chain (your certificate followed by any intermediate certificates) and the private key. A leaf certificate on its own is a common mistake — it produces an "incomplete chain" error on some clients and works on others, which makes it hard to diagnose.

kubectl create secret tls <secret-name> \
  --namespace <namespace> \
  --cert=fullchain.pem \
  --key=privkey.pem

The secret name must match exactly what your values.yaml references, and the secret must be in the same namespace as the Willow release, not the ingress controller's namespace. For a hybrid deployment that is:

deployments:
  run:
    ingress:
      tls:
        - hosts:
            - "willow.<YOUR_DOMAIN>"
          secretName: "willow-run-tls" # must match the secret you created

Do not add a cert-manager.io/cluster-issuer annotation when supplying your own certificate — cert-manager would try to take over the secret.

Verify the file is what you think it is before creating the secret:

# Does it cover your hostname? Check both Subject and the SAN list
openssl x509 -in fullchain.pem -noout -subject -ext subjectAltName -dates

# Does the key actually match the certificate? These two must print the same hash
openssl x509 -in fullchain.pem -noout -modulus | openssl md5
openssl rsa  -in privkey.pem   -noout -modulus | openssl md5

A wildcard certificate for *.example.com covers willow.example.com but not willow.internal.example.com — wildcards match one label only. This catches people using a subdomain-of-a-subdomain, which is a common pattern for externally-facing hostnames.

Renewal is manual with this option. Re-run the same kubectl create secret tls with --dry-run=client -o yaml | kubectl apply -f - to replace it, then the ingress controller picks it up within seconds — no pod restart needed.

Option B — cert-manager with DNS-01

Works with no inbound access at all, because validation happens through your DNS provider's API rather than by connecting to your server. This makes it the right choice when the firewall isn't open yet, and the only choice for a cluster that will never accept inbound from the internet.

Install cert-manager and configure a DNS-01 ClusterIssuer for your provider using the official cert-manager documentation — the setup is specific to your DNS provider and we won't duplicate it here. Then reference the issuer:

deployments:
  run:
    ingress:
      annotations:
        cert-manager.io/cluster-issuer: "letsencrypt-dns01"
      tls:
        - hosts:
            - "willow.<YOUR_DOMAIN>"
          secretName: "willow-run-tls"

Option C — cert-manager with HTTP-01

The most common setup, and the one shown in both deployment guides. It requires that Inbound network is already working on port 80.

Install cert-manager per the official installation guide and create an HTTP-01 ClusterIssuer, then:

deployments:
  run:
    ingress:
      annotations:
        cert-manager.io/cluster-issuer: "letsencrypt-prod"
      tls:
        - hosts:
            - "willow.<YOUR_DOMAIN>"
          secretName: "willow-run-tls"

Helm merges annotation keys with the chart's defaults, so adding the issuer annotation keeps the chart's streaming annotations in place.

Option D — Terminate TLS at the cloud load balancer

If you already run an AWS ALB, Azure Application Gateway, or Google Cloud Load Balancer in front of the cluster, the certificate lives there and the chart's ingress.tls stays empty. Two things to watch: the load balancer must not buffer responses or use a short idle timeout, or MCP streaming breaks (see the streaming warning in your deployment guide), and the certificate is managed in that service rather than in Kubernetes — so kubectl get certificate will show nothing and that's expected.

Verifying and debugging certificate issuance

Once installed, check what is actually being served — from outside your network, so you see what clients see:

echo | openssl s_client -connect <hostname>:443 -servername <hostname> 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

If the issuer says something like Kubernetes Ingress Controller Fake Certificate, the ingress has no certificate for this host and is falling back to its placeholder. The secret either doesn't exist, is in the wrong namespace, or has a different name than the ingress references.

The ingress controller says so explicitly in its logs, and this is the fastest way to confirm:

kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller --tail=200 \
  | grep -iE "certificate|ssl"

A line like local SSL certificate <namespace>/<secret-name> was not found. Using default certificate names the exact secret it expected.

If you're using cert-manager, walk the chain of objects it creates. Each one explains the next:

kubectl get certificate,certificaterequest,order,challenge -n <namespace>
kubectl describe certificate <secret-name> -n <namespace>

Symptom	Cause	Fix
`Certificate` stuck `Ready: False`, `Challenge` pending forever	HTTP-01 can't be reached from the internet	Fix Inbound network, or switch to DNS-01
No `Certificate` object at all	The `cert-manager.io/cluster-issuer` annotation didn't reach the ingress	`kubectl get ingress <name> -n <namespace> -o yaml` and check the annotations
`Issuer not found`	The `ClusterIssuer` name is wrong, or an `Issuer` was created instead of a `ClusterIssuer`	They're different resources — the annotation must match the kind you created
Certificate issued but clients still reject it	Ingress serving a different host, or an incomplete chain	Check the SAN list covers the exact hostname; confirm you supplied the full chain
`too many certificates already issued`	Let's Encrypt rate limit, usually from repeated failed installs	Wait out the limit, or use their staging issuer while debugging
Works in a browser, fails from `curl` and MCP clients	Certificate is from a private CA your laptop trusts	Must be publicly trusted — see Choosing a certificate

Inbound network

Symptom: the hostname resolves, but connections time out or are refused. The cluster looks completely healthy.

Work outward from the cluster. Each check rules out a layer, and the first failure tells you where to stop.

1. Does the ingress controller have the right address and ports?

kubectl get svc -n ingress-nginx -o wide

You need TYPE=LoadBalancer, an EXTERNAL-IP that matches what your hostname resolves to, and PORT(S) listing both 80:3xxxx/TCP and 443:3xxxx/TCP. Note the high-numbered node ports — you'll match against them later.

If EXTERNAL-IP differs from your DNS record, that's the bug. If 443 isn't listed, the ingress controller was installed without an HTTPS port.

2. Is anything restricting source addresses inside the cluster?

kubectl get svc ingress-nginx-controller -n ingress-nginx -o yaml \
  | grep -A 10 -E 'loadBalancerSourceRanges|annotations:'

If loadBalancerSourceRanges is present, only those addresses can connect and everyone else is dropped silently — which perfectly imitates a firewall problem. This is also where cloud-specific annotations live; an annotation marking the load balancer as internal would explain a private EXTERNAL-IP.

3. Does the ingress answer from inside the cluster?

kubectl run conn-test --rm -it --image=busybox --restart=Never -- \
  wget -qO- --timeout=5 --header="Host: <your-run-hostname>" \
  http://<CLUSTER-IP-from-step-1>/healthz

The Host header is required, not optional. An ingress controller routes by hostname, so a request to the bare cluster IP matches no rule and returns 404 from the default backend — which reads as a broken cluster when the only thing missing was the header.

If this works, the cluster is fine and the problem is definitively in your cloud provider's network layer. If it doesn't, see No healthy backend.

4. Check your cloud provider's network layer

This is where a healthy-looking cluster and an unreachable endpoint usually part ways.

Azure (AKS)

Four things can drop the traffic, and two security groups can apply at once — this is the detail that most often gets missed. AKS automatically creates an allow rule in the security group attached to the node resource group, but not in any security group your platform team attached to the subnet. Both are evaluated.

RG=<your-resource-group>
CLUSTER=<your-cluster-name>
NODE_RG=$(az aks show -g "$RG" -n "$CLUSTER" --query nodeResourceGroup -o tsv)

# a. Load balancer rules - need frontend 80 and 443, ProvisioningState Succeeded
# AKS puts every public LoadBalancer service on one Standard LB named "kubernetes".
# Confirm the name first if this returns nothing (a private-only cluster has
# "kubernetes-internal" instead).
az network lb list -g "$NODE_RG" --query "[].name" -o tsv
az network lb rule list -g "$NODE_RG" --lb-name kubernetes -o table

# b. The node security group (AKS manages this one)
NSG=$(az network nsg list -g "$NODE_RG" --query "[0].name" -o tsv)
az network nsg rule list -g "$NODE_RG" --nsg-name "$NSG" -o table

# c. The subnet's own security group and route table (AKS does NOT manage these)
SUBNET=$(az aks show -g "$RG" -n "$CLUSTER" --query "agentPoolProfiles[0].vnetSubnetId" -o tsv)
az network vnet subnet show --ids "$SUBNET" \
  --query "{nsg:networkSecurityGroup.id, routeTable:routeTable.id}" -o yaml

If step c returns a security group, list its rules and add an inbound allow if one is missing:

az network nsg rule list --ids <nsg-id-from-above> -o table

az network nsg rule create \
  -g <resource-group> --nsg-name <nsg-name> \
  -n allow-willow-inbound --priority 200 \
  --direction Inbound --access Allow --protocol Tcp \
  --source-address-prefixes Internet \
  --destination-address-prefixes <your-public-ip> \
  --destination-port-ranges 80 443

Azure evaluates rules by priority with lower numbers winning, so a permissive rule at priority 500 is overridden by a deny at 100.

If step c returns a route table, list its routes:

az network route-table show --ids <route-table-id> \
  --query "routes[].{name:name, prefix:addressPrefix, nextHop:nextHopType, ip:nextHopIpAddress}" -o table

A 0.0.0.0/0 route with next hop VirtualAppliance means egress is forced through a firewall. Inbound connections arrive at the load balancer correctly, but the reply is routed out through the firewall instead of back the way it came, the client never receives it, and the connection times out. Everything in Kubernetes and in the load balancer looks perfectly healthy. Your network team has to exempt the load balancer's return traffic or configure the firewall to handle it.

Using a pre-created public IP

If the ingress service has a service.beta.kubernetes.io/azure-pip-name annotation, the address was created by hand rather than by AKS. AKS then needs the Network Contributor role on the resource group holding that IP, or it will assign the address but silently fail to create the load balancer and security group rules that carry traffic to it — an endpoint that looks fully configured and accepts nothing.

PRINCIPAL=$(az aks show -g "$RG" -n "$CLUSTER" --query "identity.principalId" -o tsv)
PIP_RG=$(az network public-ip list --query "[?name=='<pip-name>'].resourceGroup" -o tsv)
az role assignment list --assignee "$PRINCIPAL" --resource-group "$PIP_RG" -o table

AWS (EKS)

# Which load balancer backs the service, and is it internet-facing?
aws elbv2 describe-load-balancers \
  --query "LoadBalancers[].{name:LoadBalancerName, scheme:Scheme, dns:DNSName, state:State.Code}" \
  --output table

# Do its listeners cover 443?
aws elbv2 describe-listeners --load-balancer-arn <arn> \
  --query "Listeners[].{port:Port, protocol:Protocol}" --output table

# Is the target group healthy? Unhealthy targets are blackholed silently
aws elbv2 describe-target-health --target-group-arn <arn> --output table

Common causes: scheme is internal rather than internet-facing; the node security group has no inbound rule for the node port range from the load balancer's security group; or the public subnets are missing the kubernetes.io/role/elb=1 tag, which makes the AWS Load Balancer Controller place the load balancer in private subnets.

Google Cloud (GKE)

# Firewall rules covering the load balancer health checks and traffic
gcloud compute firewall-rules list \
  --filter="direction=INGRESS AND allowed[].ports~'443'" \
  --format="table(name, sourceRanges.list(), allowed[].map().firewall_rule().list())"

# Forwarding rules - is anything actually listening on the address?
gcloud compute forwarding-rules list --format="table(name, IPAddress, portRange, target)"

Common causes: a restrictive VPC firewall with no ingress allow for 0.0.0.0/0 on 443, or a private cluster whose load balancer was created as internal.

5. Re-test from outside

After each change, re-run the test from the top of this page. Progressing from a timeout to a certificate error is success — it means traffic now flows and you've moved to the TLS layer.

No healthy backend

Symptom: the endpoint answers, but returns 503. Or the ingress controller logs repeat:

Service "<namespace>/run" does not have any active Endpoint.

This means no Ready pod matches the service's selector. The ingress is correct and has nothing to forward to.

kubectl get pods -n <namespace>
kubectl get endpoints run -n <namespace>
kubectl describe pod <pod-name> -n <namespace>

The most common causes are ImagePullBackOff because the cluster has no credentials for quay.io/webrix, and Pending because no node has capacity for the pod's CPU or memory request. Both are covered in the install step of your deployment guide.

An important consequence: while a pod is in ImagePullBackOff, the endpoint returns 503 rather than failing loudly, and a cert-manager HTTP-01 challenge against that host will also fail. Fixing the image pull can resolve what looks like two unrelated certificate and networking problems at once.

Hostname mismatches

Symptom: the endpoint answers with 404 from the ingress controller's default backend, even though the pod is healthy.

The ingress routes on the Host header, so it only serves the exact hostname configured in the chart. The chart builds that hostname as <deployments.<service>.ingress.subdomain>.<global.domain.host>, and that same hostname becomes the service's BASE_URL, which is used for MCP OAuth discovery.

Three values must agree, and a mismatch in any pair produces a different confusing failure:

Must match	Where it's set	If it doesn't match
The DNS record	Your DNS provider	Connection fails or reaches the wrong address
The ingress host	`ingress.subdomain` + `global.domain.host`	`404` from the ingress default backend
The registered gateway URL (hybrid only)	Settings → Gateway Settings in the Willow admin app	Gateway shows Unreachable even though the endpoint works

# What hostname is the ingress actually serving?
kubectl get ingress -n <namespace> -o custom-columns=NAME:.metadata.name,HOSTS:.spec.rules[*].host

# What does the service believe its own address is?
curl https://<hostname>/.well-known/oauth-protected-resource

The resource field in that last response must be your real hostname. If it isn't, BASE_URL is wrong and MCP client authentication will fail later even after the gateway badge turns green.

Check for a second installation

If the ingress controller logs lines like ignoring ingress run in namespace default different from the namespace watched <namespace>, the chart was installed more than once, into different namespaces. The ignored copy is inert, but it means two people may be looking at different releases. helm list -A and kubectl get ingress -A show what actually exists.

Symptom index

Symptom	Section
Hostname doesn't resolve	Fix DNS in your deployment guide's DNS step first
Connection times out from outside, works from inside	Inbound network
Connection refused immediately	Inbound network — missing load balancer rule
Certificate error, or "fake certificate" served	Choosing a certificate
`Certificate` stuck at `Ready: False`	Verifying and debugging certificate issuance
Certificate works in a browser, rejected by clients	Choosing a certificate — private CA
`404` from the endpoint	Hostname mismatches
`503` from the endpoint	No healthy backend
`301` or `302` from the endpoint	Test it correctly first — remove the redirect
Endpoint works, but MCP clients hang and time out	The streaming (SSE) warning in your deployment guide

TLS and Inbound Reachability

Test it correctly first

Match your ingress controller

Self-installed ingress-nginx

AKS application routing add-on

Other controllers

Choosing a certificate

Option A — Install a certificate you already have

Option B — cert-manager with DNS-01

Option C — cert-manager with HTTP-01

Option D — Terminate TLS at the cloud load balancer

Verifying and debugging certificate issuance

Inbound network

1. Does the ingress controller have the right address and ports?

2. Is anything restricting source addresses inside the cluster?

3. Does the ingress answer from inside the cluster?

4. Check your cloud provider's network layer

Azure (AKS)

AWS (EKS)

Google Cloud (GKE)

5. Re-test from outside

No healthy backend

Hostname mismatches

Symptom index

On this page

Was this helpful?

Test it correctly first​

Match your ingress controller​

Self-installed ingress-nginx​

AKS application routing add-on​

Other controllers​

Choosing a certificate​

Option A — Install a certificate you already have​

Option B — cert-manager with DNS-01​

Option C — cert-manager with HTTP-01​

Option D — Terminate TLS at the cloud load balancer​

Verifying and debugging certificate issuance​

Inbound network​

1. Does the ingress controller have the right address and ports?​

2. Is anything restricting source addresses inside the cluster?​

3. Does the ingress answer from inside the cluster?​

4. Check your cloud provider's network layer​

Azure (AKS)​

AWS (EKS)​

Google Cloud (GKE)​

5. Re-test from outside​

No healthy backend​

Hostname mismatches​

Symptom index​

On this page

Was this helpful?

Test it correctly first

Match your ingress controller

Self-installed ingress-nginx

AKS application routing add-on

Other controllers

Choosing a certificate

Option A — Install a certificate you already have

Option B — cert-manager with DNS-01

Option C — cert-manager with HTTP-01

Option D — Terminate TLS at the cloud load balancer

Verifying and debugging certificate issuance

Inbound network

1. Does the ingress controller have the right address and ports?

2. Is anything restricting source addresses inside the cluster?

3. Does the ingress answer from inside the cluster?

4. Check your cloud provider's network layer

Azure (AKS)

AWS (EKS)

Google Cloud (GKE)

5. Re-test from outside

No healthy backend

Hostname mismatches

Symptom index