Skip to main content

Budgets

A budget is a monthly spending limit in USD on calls through the LLM Gateway. When a budget is used up, the gateway blocks the call, reroutes it to a cheaper model, or lets it through and records a warning. Manage budgets from Control > LLM Gateway > Budgets.

Budgets reset on the 1st of each month (UTC). Spend is calculated from the prices on your models, so a model with no price never uses up a budget.

Add a budget​

  1. Go to Control > LLM Gateway > Budgets.

  2. Select Add Budget.

  3. Enter a Name, such as Engineering monthly, and a Monthly limit (USD).

  4. Choose what the budget Applies to:

    OptionWhat is counted against the limit
    The whole organization, sharedAll gateway spend in the organization, in one pool
    Each group, separatelyEach selected group's spend, in its own pool. Choose the Groups
    Each user, separatelyEach user's own spend. Optionally limit it with Only users in to users of certain groups; leave it on Everyone to cover every user

  5. Under Models, leave All models, or choose the models the budget covers.

  6. Choose what happens When spent:

    • Block the call refuses the call with a message naming the budget.
    • Allow and warn lets the call through. The call's log entry records the budget.

  7. Optionally choose a model to Reroute to. See Reroute to a cheaper model.

  8. Leave Enabled on, and select Save.

A per-group budget gives each group its own pool. A user in two of its groups is checked against both pools, and either one can stop the call. A per-user budget gives every covered user the full limit.

How budgets combine​

The gateway checks every enabled budget that covers a call: one whose scope includes the caller and whose models include the model called. The first budget that is used up and set to Block the call decides the outcome. Allow and warn budgets never stop a call.

For example, with an organization budget of $5,000 and a per-user budget of $200, a user who has spent $200 this month is blocked even while the organization is well under $5,000.

Reroute to a cheaper model​

A budget set to Block the call can reroute calls to a fallback model instead of refusing them. When the budget is used up, the gateway serves the call with the fallback, and the response carries an x-willow-fallback-from header naming the model that was asked for.

The fallback must be a model the budget does not cover. Otherwise the fallback would be blocked by the same budget. So to reroute, limit the budget to specific Models first. For example, cover only claude-opus and reroute to claude-sonnet.

The call is still refused if:

  • The fallback model is also over a budget.
  • The caller is not allowed to use the fallback model.
  • The fallback is served on a different API than the original model, for example an OpenAI model as a fallback for an Anthropic model. The gateway does not translate between APIs.

Spend on the budget cards​

Each budget card shows its limit, its action, what it covers, and its spend this month:

  • Organization budgets show one bar.
  • Per-group budgets show one bar per group.
  • Per-user budgets show the top three spenders and how many users are at the limit.

The bars are green below 80% of the limit, amber from 80%, and red once the budget is used up. Seeing spend needs the Monitoring & Shadow AI > View permission.

Spend lags by about a minute

The gateway checks spend against a total that is cached for up to a minute. A burst of calls can take a pool slightly past its limit before the budget closes.

Pause or remove a budget​

Turn off a budget's switch to pause it, or select the trash icon to delete it. Deleting takes effect immediately and asks for no confirmation.

On this page