Bring your own ClickHouse

On-prem Willow stores configuration, identities, and secrets in PostgreSQL. High-volume logs, OpenTelemetry, and analytics can optionally go to ClickHouse that you operate.

The Helm chart does not deploy ClickHouse. You bring an existing instance (self-managed or ClickHouse Cloud that your cluster can reach), give db-service HTTP access, and turn the log-store flags on. After that, Willow creates its own database and tables on startup and dual-writes new events.

Not the ClickHouse MCP connector

This page is about Willow's own log and analytics store. To let AI assistants query your business data in ClickHouse, use the ClickHouse connector instead.

Will it work automatically?

Mostly, once you point Willow at a reachable ClickHouse and enable writes. You do not create tables by hand.

What happens automatically	What does not happen automatically
db-service creates the Willow database (default otel) if it is missing	Willow does not ship an in-cluster ClickHouse
Schema migrations run on every db-service start (CREATE / ALTER … IF NOT EXISTS, safe to re-run)	Willow does not read or write your existing business tables
New logs, OTLP signals, and analytics events are written to ClickHouse after you enable it	Historical rows already in PostgreSQL are not backfilled
Later Willow upgrades apply new ClickHouse migrations on pod start	Reads stay on PostgreSQL until you flip LOGS_READ_STORE
Explore and Dashboards work after reads are served from ClickHouse	A replicated self-managed cluster is not configured for you (see Cluster topology)

PostgreSQL remains required. ClickHouse is an additional store for append-heavy log and telemetry data, not a replacement for the application database.

What you need

1. A ClickHouse instance the db-service pods can reach on the HTTP interface (native protocol port 9000 is not used).
   • Self-managed: typically http://clickhouse.your-namespace.svc.cluster.local:8123
   • TLS: https://clickhouse.your-company.example:8443
2. A dedicated database name for Willow (default otel). Use a database that is not shared with other applications.
3. A ClickHouse user with permission to create that database (if it does not already exist) and to create tables, alter them, insert, and select inside it.
4. Network path from the Willow namespace to ClickHouse (NetworkPolicy, private link, or in-cluster Service). No ingress to ClickHouse is required unless you expose it that way on purpose.
5. Helm values on db-service so it knows the URL, credentials, and log-store flags (below).

Recommended ClickHouse versions: 23.8 or newer (including ClickHouse Cloud). Cloud rewrites MergeTree engines to SharedMergeTree automatically.

1. Create a Willow user

Connect as an admin and create a user that owns only the Willow database:

CREATE USER willow IDENTIFIED BY 'a-strong-password';

-- Optional if you prefer to pre-create the database yourself:
CREATE DATABASE IF NOT EXISTS otel;

GRANT CREATE DATABASE ON *.* TO willow;
GRANT ALL ON otel.* TO willow;

Confirm HTTP access from a host that can reach ClickHouse the same way db-service will:

curl 'http://clickhouse.your-company.example:8123/ping'
curl 'http://clickhouse.your-company.example:8123/?query=SELECT%201' \
  --user willow:a-strong-password

Both should succeed (Ok. and 1).

2. Store the password in a Secret

Do not put the ClickHouse password in plaintext values.yaml.

kubectl create secret generic willow-clickhouse \
  --namespace <namespace> \
  --from-literal=CLICKHOUSE_PASSWORD='a-strong-password'

3. Point db-service at ClickHouse

Add the connection and flags to your Helm values. Keep PostgreSQL writes and reads on until you have validated ClickHouse.

deployments:
  db-service:
    secretName: "willow-clickhouse"
    env:
      LOGS_WRITE_POSTGRES: "true"
      LOGS_WRITE_CLICKHOUSE: "true"
      LOGS_READ_STORE: "postgres" # postgres | clickhouse
      CLICKHOUSE_URL: "http://clickhouse.your-namespace.svc.cluster.local:8123"
      CLICKHOUSE_USER: "willow"
      CLICKHOUSE_DATABASE: "otel"

Variable	Required	Purpose
CLICKHOUSE_URL	Yes, whenever ClickHouse is enabled	HTTP(S) base URL of the ClickHouse interface, including protocol and port
CLICKHOUSE_USER	Recommended	ClickHouse username
CLICKHOUSE_PASSWORD	Recommended	ClickHouse password (set via secretName, not plaintext values)
CLICKHOUSE_DATABASE	No	Willow database name. Defaults to otel
LOGS_WRITE_CLICKHOUSE	Yes to enable	true to dual-write (or write-only) logs and OTLP to ClickHouse
LOGS_WRITE_POSTGRES	Keep true at first	Leave on so Logs and Analytics keep working if ClickHouse is empty or down
LOGS_READ_STORE	No	postgres (default) or clickhouse. Must match a store you are writing to

If ClickHouse is enabled and CLICKHOUSE_URL is missing, db-service refuses to start. If both write flags are false, it also refuses to start (every log would be dropped).

Apply the change:

helm upgrade --install willow willow/webrix-helm \
  --namespace <namespace> \
  -f values.yaml \
  --wait

4. Confirm schema and writes

On start, db-service runs ClickHouse migrations, then boots. Watch the new pod:

kubectl logs -n <namespace> deployment/db-service --tail=200

You should see the migrate script create otel (or your CLICKHOUSE_DATABASE) and apply clickhouse/migrations/*.sql, then a line that the ClickHouse client is initialized.

In ClickHouse:

SHOW TABLES FROM otel;

Expect Willow tables such as otel_metrics, otel_events, otel_spans, audit_logs, and the other log mirrors. These are Willow's schema — not mappings onto tables you already had.

Generate activity (a tool call, or an OTLP export from an agent), then:

SELECT count() FROM otel.audit_logs;
SELECT count() FROM otel.otel_metrics;

Counts should start rising. The admin Logs and Analytics pages still read PostgreSQL at this stage.

5. Flip reads to ClickHouse

When new data is landing in ClickHouse and you have kept dual-write on long enough for the retention window you care about:

deployments:
  db-service:
    env:
      LOGS_WRITE_POSTGRES: "true"      # optional safety net; can be false later
      LOGS_WRITE_CLICKHOUSE: "true"
      LOGS_READ_STORE: "clickhouse"

Redeploy. From this point:

• Monitor → Logs and the default Analytics dashboard read ClickHouse.
• Explore and custom Dashboards become available (they have no PostgreSQL fallback).

No automatic backfill

Rows written to PostgreSQL before ClickHouse writes were enabled are not copied over. After you flip LOGS_READ_STORE, the UI only shows events that exist in ClickHouse. Keep dual-write for at least as long as the lookback you need, or plan a one-off backfill with Willow support.

Only after ClickHouse has been the read store for a full retention window should you set LOGS_WRITE_POSTGRES: "false". PostgreSQL is still required for everything that is not the log store.

TLS and private CAs

Use an https:// CLICKHOUSE_URL when ClickHouse expects TLS. If your network uses a private CA (or TLS inspection), trust it the same way as other on-prem outbound TLS:

global:
  caCertificate: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

See TLS / Custom CA.

Cluster topology

Willow's migrations create local MergeTree / ReplacingMergeTree tables (no ON CLUSTER, no ReplicatedMergeTree).

Topology	Support
Single-node ClickHouse	Works as documented
ClickHouse Cloud	Works; Cloud converts engines to SharedMergeTree
Replicated self-managed cluster (Keeper / ZooKeeper)	Not applied as replicated DDL. Use a dedicated single replica (or Cloud), or talk to Willow before pointing at a multi-replica cluster

Give Willow its own database. Do not point CLICKHOUSE_DATABASE at a database that already holds application data.

What ClickHouse is used for

Once writes are on, Willow stores the same logical tables it keeps in PostgreSQL for the log path, including:

• OTLP metrics, events, and spans (otel_*)
• Gateway / audit logs
• Admin, SCIM, git-sync, and delivery logs
• Extension and conversation streams used by analytics

Configuration, users, integrations, guards, and secrets stay in PostgreSQL.

Troubleshooting

Symptom	Likely cause	What to check
db-service crashloops mentioning CLICKHOUSE_URL	ClickHouse enabled without a URL	Set CLICKHOUSE_URL whenever LOGS_WRITE_CLICKHOUSE=true or LOGS_READ_STORE=clickhouse
[ch-migrate] ClickHouse not enabled — skipping.	Flags not set	LOGS_WRITE_CLICKHOUSE must be the string true
[ch-migrate] failed / permission error	User cannot create DB or tables	Grant the privileges in Create a Willow user; confirm CLICKHOUSE_USER / password
Connection timeout or ECONNREFUSED	Network path missing	From a debug pod in the Willow namespace, curl the same CLICKHOUSE_URL/ping
TLS handshake errors	Private CA or wrong scheme	Use https:// if required; add global.caCertificate
Logs UI empty after flipping reads	Flip happened before dual-write had data, or historical PG-only rows	Confirm SELECT count() in ClickHouse; remember there is no backfill
Explore / Dashboards still hidden	Reads still on PostgreSQL	Set LOGS_READ_STORE=clickhouse and wait for db-service to roll
Writes succeed in PostgreSQL but not ClickHouse	Secondary-store failure while reads are still PG	db-service logs [log-store] secondary store write failed; ClickHouse stays best-effort until it is the read store

kubectl logs -n <namespace> deployment/db-service --tail=200

Look for [clickhouse] client initialized for log store and [ch-migrate] done.

Related

• On-Prem Deployment
• Analytics
• Explore
• Logs
• ClickHouse connector (query your own data, not Willow's log store)