OTLP Router on-prem

OTLP Router is an open-source OpenTelemetry endpoint for AI agents, by Willow. Claude Code, Codex, Gemini CLI and any OpenTelemetry SDK send to it once, and it copies each payload to every destination you configure: your on-prem Willow, Splunk, Datadog, Grafana or your own collector. Each destination gets its own filters, so prompts, emails and secrets can be removed before they leave the router.

Use it when you want one place that decides what each tool receives, instead of configuring every agent with several exporters. If all you need is your own collector in front of Willow, see OTLP proxy instead.

 Developer machines                       Your Kubernetes cluster
┌──────────────────┐   OTLP/HTTP   ┌──────────────┐  OTLP/JSON  ┌──────────────────┐
│ Claude Code      │──────────────▶│ OTLP Router  │────────────▶│ Willow connect   │
│ Codex, Gemini CLI│   :4318       │  per-target  │             │ (/otlp)          │
│ OTel SDKs        │               │  filters     │────────────▶│ SIEM, APM, ...   │
└──────────────────┘               └──────────────┘             └──────────────────┘

Install it with Helm on Kubernetes (steps 1 to 5 below), or as a single container on a VM with Docker (step 1, then Run with Docker).

Before you start

• Willow on-prem running. OTLP data is written to your Willow log store: PostgreSQL by default, or your own ClickHouse.
• Your connect URL, the connect hostname from your on-prem install, for example https://willow-dashboard.example.com.
• Kubernetes and Helm 3.8 or later, or Docker on a single host. The chart is published as an OCI artifact.
• An ingress controller. You need one hostname for agents to send to (for example otel.example.com) and, optionally, one for the admin UI. Keep the admin UI on an internal ingress class.
• Image access. Either egress to ghcr.io, or a mirror in your own registry (see Air-gapped clusters).

Artifact	Location
Helm chart	oci://ghcr.io/webrix-ai/charts/otlp-router
Container image	ghcr.io/webrix-ai/otlp-router

Step 1: Get your Willow OTLP token

The router authenticates to Willow with your organization's OTLP token.

1. In the Willow admin console, open Monitor > Analytics > Data sources.
2. Copy the token from any source card. The cards show it as Authorization: Bearer <org token>; you need only the part after Bearer .

Step 2: Create the router's Secret

Keep credentials in a Kubernetes Secret rather than in Helm values, so they never appear in helm get values or in your Git repository.

kubectl create namespace observability

kubectl -n observability create secret generic otlp-router-secrets \
  --from-literal=admin-password="$(openssl rand -base64 24)" \
  --from-literal=session-secret="$(openssl rand -base64 48)" \
  --from-literal=willow-token="<org token from step 1>"

Key	Required	Used for
admin-password	Yes	The admin UI login, user admin
session-secret	Yes	Signing admin UI sessions
willow-token	For the Willow target	Sent to Willow as Authorization: Bearer. The pods read it as WILLOW_OTLP_TOKEN
ingest-tokens	No	Comma-separated tokens agents may send, in addition to tokens created in the UI. See Step 5

Step 3: Write your values

Save this as otlp-router-values.yaml and replace the hostnames and ingress classes with yours.

publicUrl: https://otel.example.com

auth:
  existingSecret: otlp-router-secrets

willow:
  enabled: true
  region: custom
  connectUrl: https://willow-dashboard.example.com

ingress:
  otlp:
    enabled: true
    className: nginx
    host: otel.example.com
    tls:
      - hosts: [otel.example.com]
        secretName: otel-tls
  admin:
    enabled: true
    className: nginx-internal
    host: otel-admin.example.com
    tls:
      - hosts: [otel-admin.example.com]
        secretName: otel-admin-tls

• willow.region: custom with connectUrl points the Willow target at your on-prem connect service instead of Willow SaaS. The router sends to <connectUrl>/otlp/v1/<signal>; paste the URL without a path.
• The Willow target starts with filters that drop prompt events and redact secrets. Change them in the UI, or set willow.filters in values.
• The OTLP ingress forwards only /v1/* to port 4318 (OTLP/HTTP). OTLP/gRPC on 4317 is available in-cluster at otlp-router.observability.svc:4317. To expose gRPC outside the cluster, set service.type: LoadBalancer or add your own gRPC ingress.
• Every option is listed in the chart's values.yaml.

Step 4: Install

helm install otlp-router oci://ghcr.io/webrix-ai/charts/otlp-router \
  --version 0.2.0 \
  --namespace observability \
  -f otlp-router-values.yaml

Verify:

kubectl -n observability rollout status deploy/otlp-router

# 401 means the router is reachable and enforcing tokens
curl -s -o /dev/null -w "%{http_code}\n" -X POST \
  -H 'Content-Type: application/json' -d '{"resourceLogs":[]}' \
  https://otel.example.com/v1/logs

Then open https://otel-admin.example.com and sign in as admin with the admin-password from your Secret. Without an admin ingress, use kubectl -n observability port-forward svc/otlp-router 8080 and open http://localhost:8080.

Step 5: Point your agents at the router

The router rejects all telemetry until it has at least one ingest token. Pick one of the two setups below.

Option A: keep Willow's agent setup

If Willow already configures your agents (Claude Code policies, the Codex block, the Scan Agent), send that traffic through the router:

1. Add your org token to the router's accepted tokens, because agents configured by Willow send it as their Authorization header:

   kubectl -n observability patch secret otlp-router-secrets \
     -p "{\"stringData\":{\"ingest-tokens\":\"<org token from step 1>\"}}"
   kubectl -n observability rollout restart deploy/otlp-router

2. In Willow, open Admin > Settings > Log Settings, turn on OTLP proxy and enter https://otel.example.com as the base URL. See OTLP proxy.

Agents pick up the new endpoint from their next policy or config refresh. The router passes X-Device-Id, X-User-Email and X-Willow-Platform through to Willow, so device and user attribution keep working.

Option B: router-issued tokens

1. In the router UI, open Tokens and create a token, for example "Engineering laptops". It is shown once.

2. Open Agents for copy-paste setup for each agent, with your endpoint and token filled in. For Claude Code:

   export CLAUDE_CODE_ENABLE_TELEMETRY=1
   export OTEL_METRICS_EXPORTER=otlp OTEL_LOGS_EXPORTER=otlp
   export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
   export OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.example.com
   export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <router token>"

Rotate tokens from the same page. The old token keeps working for the grace period you choose.

Run with Docker

Use Docker to run the router on a single VM, or to try it out. It needs no config file: settings come from environment variables, and the targets and tokens you create in the UI are kept in a Docker volume.

1. Save the settings in an env file, for example otlp-router.env, readable only by you:

   OTLP_ROUTER_ADMIN_PASSWORD=<admin password>
   OTLP_ROUTER_SESSION_SECRET=<random string, e.g. from openssl rand -base64 48>
   OTLP_ROUTER_PUBLIC_URL=https://otel.example.com
   OTLP_ROUTER_WILLOW_CONNECT_URL=https://willow-dashboard.example.com
   WILLOW_OTLP_TOKEN=<org token from step 1>

2. Start the router:

   docker run -d --name otlp-router --restart unless-stopped \
     --env-file otlp-router.env \
     -p 4317:4317 -p 4318:4318 -p 127.0.0.1:8080:8080 \
     -v otlp-router:/data \
     ghcr.io/webrix-ai/otlp-router:0.2.0

   The admin UI on port 8080 is published on 127.0.0.1 only. Reach it from your machine with ssh -L 8080:localhost:8080 <host> and open http://localhost:8080.

3. Verify:

   docker logs otlp-router
   
   # 401 means the router is reachable and enforcing tokens
   curl -s -o /dev/null -w "%{http_code}\n" -X POST \
     -H 'Content-Type: application/json' -d '{"resourceLogs":[]}' \
     http://localhost:4318/v1/logs

4. Point your agents at the router as in Step 5. For Option A, add OTLP_ROUTER_INGEST_TOKENS=<org token from step 1> to the env file instead of patching a Secret, then recreate the container.

The router serves plain HTTP. Put a TLS reverse proxy (nginx, Caddy, your load balancer) in front of port 4318 so agents send to https://otel.example.com.

Variable	Sets
OTLP_ROUTER_ADMIN_PASSWORD	The admin UI password. Required, unless you set OTLP_ROUTER_ADMIN_PASSWORD_HASH to a bcrypt hash instead
OTLP_ROUTER_ADMIN_USERNAME	The admin UI username. Defaults to admin
OTLP_ROUTER_SESSION_SECRET	Signs admin UI sessions. Without it, everyone is signed out when the container restarts
OTLP_ROUTER_INGEST_TOKENS	Comma-separated tokens agents may send, in addition to tokens created in the UI
OTLP_ROUTER_PUBLIC_URL	The address agents send to, shown in the UI's setup snippets
OTLP_ROUTER_WILLOW_CONNECT_URL	Adds a Willow target that sends to your on-prem connect URL
OTLP_ROUTER_WILLOW_ORG	Adds a Willow target for a Willow SaaS org instead. Use with OTLP_ROUTER_WILLOW_REGION set to us (default) or eu
WILLOW_OTLP_TOKEN	Your org's OTLP token. Required when either Willow variable is set

The Willow target is added only on first start, while the volume holds no targets, with the same filters as the Helm chart: prompt events dropped and secrets redacted. After that, change it in the UI. The token is stored as the reference ${env:WILLOW_OTLP_TOKEN}, never as its value.

For settings these variables don't cover, mount a config file at /etc/otlp-router/config.yaml, starting from config.example.yaml. Values in the file take precedence over the variables.

Run one container per volume. The volume's store is not shared between containers; for more than one replica, use the Helm chart.

Add more destinations

Add targets on the router's Targets page. Each target has its own filters and its own retry queue, so a failing destination never holds up the others. Use Preview to see exactly what a target will receive from the latest real payload.

Don't type credentials into a target. Put them in a Secret, expose them to the router and reference them as ${env:NAME}:

kubectl -n observability create secret generic otlp-router-target-credentials \
  --from-literal=DD_API_KEY="<key>"

envFrom:
  - secretRef:
      name: otlp-router-target-credentials

With Docker, add DD_API_KEY=<key> to the env file and recreate the container instead.

Then set the target's header, for example dd-api-key: ${env:DD_API_KEY}. The router resolves the reference at send time, so the stored target list holds only the reference. Values typed into the UI are stored as entered (masked on screen) in the otlp-router-targets ConfigMap.

To manage targets only through Git, list them under targets in values and set readOnly: true. The UI then shows targets and tokens but cannot change them.

Air-gapped clusters

Mirror the chart and image into your registry, then install from there:

helm pull oci://ghcr.io/webrix-ai/charts/otlp-router --version 0.2.0
helm push otlp-router-0.2.0.tgz oci://registry.example.com/charts

# or skopeo, docker pull/tag/push
crane copy ghcr.io/webrix-ai/otlp-router:0.2.0 registry.example.com/otlp-router:0.2.0

image:
  repository: registry.example.com/otlp-router
  tag: "0.2.0"
imagePullSecrets:
  - name: registry-credentials

The router itself needs egress only to the destinations you configure. For on-prem Willow, that is your connect hostname.

Upgrade

helm upgrade otlp-router oci://ghcr.io/webrix-ai/charts/otlp-router \
  --version <new version> -n observability -f otlp-router-values.yaml

Targets and tokens are kept across upgrades. They live in the otlp-router-targets ConfigMap, which the chart only seeds on first install.

With Docker, pull the new image and recreate the container on the same volume:

docker pull ghcr.io/webrix-ai/otlp-router:<new version>
docker rm -f otlp-router
docker run -d --name otlp-router --restart unless-stopped \
  --env-file otlp-router.env \
  -p 4317:4317 -p 4318:4318 -p 127.0.0.1:8080:8080 \
  -v otlp-router:/data \
  ghcr.io/webrix-ai/otlp-router:<new version>

Targets and tokens are kept in the otlp-router volume.

Troubleshooting

Symptom	Check
Agents get 401	No ingest token exists, or the agent's token is not one of them. Create one on Tokens, or add it to ingest-tokens. otlp_router_ingest_unauthorized_total on /metrics counts rejections
Data reaches the router but not Willow	connectUrl is wrong or unreachable from the cluster, or Willow rejects the token. Check kubectl -n observability logs deploy/otlp-router
Willow rejects the router with 401	The Willow target's token must be ${env:WILLOW_OTLP_TOKEN}. An ${env:...} reference to a variable that isn't set resolves to an empty token. Confirm willow-token is in the Secret and restart the deployment after changing it
https://otel.example.com/ returns 404	Expected: only /v1/* is routed to the router
The Docker container exits with an admin password is required	Set OTLP_ROUTER_ADMIN_PASSWORD in the env file
The Docker container exits with WILLOW_OTLP_TOKEN is required	You set OTLP_ROUTER_WILLOW_CONNECT_URL or OTLP_ROUTER_WILLOW_ORG without the token. Add WILLOW_OTLP_TOKEN to the env file
Targets and tokens are gone after recreating the Docker container	The container was started without -v otlp-router:/data, so they were kept in a volume Docker created for that container only

Related

• OTLP proxy: point Willow-configured agents at a collector or the router
• Analytics data sources: where the org OTLP token is shown
• On-Prem Deployment: installing Willow itself
• Bring your own ClickHouse: where OTLP data is stored