Skip to main content

LLM Gateway

The LLM Gateway is one endpoint that your organization's AI tools call instead of calling Anthropic or OpenAI directly. Each person signs in with their own Willow identity. The gateway checks which models they may use, applies your monthly budgets and runtime guards, and then calls the provider with your organization's API key. Every call is recorded with its tokens and cost, against the person who made it.

Open it from Control > LLM Gateway.

Use it when you want to:

  • Stop handing out provider API keys. Users never see the key; they authenticate with Willow.
  • Decide which models each group may call, and give them stable names (aliases) that do not change when you switch providers.
  • Cap monthly LLM spend for the whole organization, each group, or each user, and reroute to a cheaper model when a cap is reached.
  • Run the same runtime guards you use on MCP traffic on the prompts people send to models.
  • See spend and usage per user, model, provider, and day.

How a call is handled​

Clients call the gateway with the same APIs they already use: Anthropic Messages, OpenAI Chat Completions, or OpenAI Responses. For each call, the gateway:

  1. Identifies the caller from their Willow token. A missing, invalid, or expired token is refused with 401.
  2. Checks the gateway is on for the organization.
  3. Resolves the model the request names, by alias first and then by the provider's own model id, and checks the caller is in a group allowed to use it.
  4. Checks every budget that covers the call. A spent budget blocks the call, reroutes it to a fallback model, or lets it through with a warning.
  5. Runs your runtime guards on the newest user turn of the conversation. A guard that blocks stops the call before it reaches the provider.
  6. Calls the provider with your organization's key and streams the answer back unchanged.
  7. Records the call in the audit log with tokens, cost, model, provider, and the caller's groups.

The gateway does not store anything itself. Token lookups, settings, group memberships, and spend totals are cached for about a minute, so a revoked token, a settings change, or a crossed budget can take up to a minute to apply.

Before you start​

  • Beta. The LLM Gateway is in beta. If Control > LLM Gateway says it isn't enabled, ask Willow Support to turn it on for your organization.
  • Plan. The gateway is a premium feature. On a plan without it, the page shows an upgrade prompt and the gateway cannot be turned on. Turning it off is always allowed. See Billing & Plans.
  • Permissions. These come from the admin's role:
To do thisYou need
Open the page and read the settingsOrganization > View Settings
Turn the gateway on or off, or change providers, models, and budgetsOrganization > Edit Settings
See spend on the Overview, Budgets, and Usage tabsMonitoring & Shadow AI > View
Roll the gateway out to Claude Code from the Connect tabMonitoring & Shadow AI > Manage

Set it up​

The Overview tab tracks these steps in its Setup checklist. Each step links to the tab that completes it.

  1. Add a provider key on the Providers tab: an Anthropic, OpenAI, or OpenAI-compatible API key. See Providers and Models.
  2. Choose the models to offer on the Models tab, and which groups may use each one. See Providers and Models.
  3. Turn the gateway on with the switch at the top of the Overview tab.
  4. Set a monthly budget on the Budgets tab. Optional. See Budgets.
  5. Connect a client using the URL and snippets on the Connect tab. See Connect Clients.

The step is ticked once the gateway has served its first call.

The Overview tab​

The top card shows whether the gateway is serving requests, the gateway URL with a copy button, and the switch that turns the gateway on or off. The status reads The gateway is serving requests, On, but no models are offered yet, or The gateway is off.

Below it, four cards summarize the current month (UTC). Each one opens the tab with the detail:

CardShows
Spend this monthTotal cost of gateway calls, from the prices on your models
Requests this monthNumber of calls, and how many guards or budgets blocked
ModelsModels offered, and how many providers have a key and are enabled
Active budgetsBudgets that are switched on

Top models this month ranks the five models with the highest spend. When budgets are active, a Budgets card shows each one's spend against its limit: green below 80%, amber from 80%, and red once it is used up.

Where the gateway runs​

On Willow SaaS, the gateway is served next to your organization's MCP endpoint. The Connect tab shows the exact URL, which has the form https://<org>.withwillow.ai/llm.

On a hybrid deployment, the URL follows your default gateway. With an external run service it becomes https://willow.<your-domain>/llm, so the gateway has to run in your cluster next to run. See LLM gateway on hybrid.

Current limits​

  • No API translation. Anthropic models are only served on the Anthropic Messages API, and OpenAI and OpenAI-compatible models only on the OpenAI APIs. A Claude client cannot call a GPT model through the gateway.
  • Providers. Anthropic, OpenAI, and services that speak the OpenAI Chat Completions API with a bearer key. AWS Bedrock and Google Vertex are not supported.
  • One provider per model. There is no failover between providers for the same model.
  • No request rate limits. Budgets cap spend, not request volume. Rate Limits apply to MCP tool calls, not to gateway calls.
  • Guards check the prompt, not the answer. Guards run on the newest user turn going in. Streamed answers are not inspected.
  • Budgets can overshoot slightly. Spend is checked against a total that can be up to a minute old.
  • Not available with Write-Only KMS. Organizations that encrypt secrets with their own AWS KMS key cannot use the gateway, because Willow cannot decrypt the provider keys. See Write-Only KMS.

In this section​

On this page