Skip to main content

Shadow AI

AI tools, MCP servers, and accounts in use without approval, and how much of that traffic you can actually see.

Most KPIs in this category are measured from device scans. Fix Employees with no scanned device first. Every other KPI on this page can only measure devices that the scan agent covers, so this KPI limits how accurate all the others are.

KPITarget
Employees with no scanned device0
MCP servers outside the approved registry0
Unmanaged skills on devices0
Skills that load instructions from external sources0
Share of AI tool calls flowing through the gateway90%
Unreviewed skills shipping executable code0
New AI capabilities untriaged for over 7 days0
Unapproved GenAI apps with active traffic0
AI tools used from personal accounts0

Employees with no scanned device​

Employees in your directory with no device reporting to the scan agent. Target: 0.

Why it matters. Everything in this category is measured from device scans. If an employee has no scanned device, Willow cannot see what they do. The unapproved AI tools they use do not appear in any number on this page.

How to improve it.

  1. Deploy the scan agent through your MDM (mobile device management) system to reach the remaining devices.
  2. Link scanned devices to directory users so per-employee numbers are accurate.

See Deploy the scan agent.

Framework mapping: NIST AI RMF GOVERN 1.6 inventory · Practitioner

MCP servers outside the approved registry​

Partial coverage

MCP servers active on devices that are neither proxied through your gateway nor covered by an allow rule. Target: 0.

Why it matters. Nobody has reviewed the tools an unmanaged MCP server offers. The server sends its tool descriptions straight to the AI model. If an attacker changes one of those descriptions, the model can follow the attacker's instructions instead of yours.

How to improve it.

  1. For each server, decide: approve it with an Allow rule, replace it with a managed integration, or Block it.
  2. Set the MCP server enforcement default to Warn or Block. Willow then warns you or blocks a new server when it finds one, instead of allowing it without telling you.
  3. Publish the approved servers as a plugin so employees install the sanctioned version.

See Policy rules and Package and publish a plugin.

Framework mapping: MCP Spec Tool poisoning · MITRE ATLAS AML.T0010 AI Supply Chain Compromise · NIST AI RMF GOVERN 1.6

Unmanaged skills on devices​

Partial coverage

Skills active on devices that are not published by your organization and not covered by an allow rule. Target: 0.

Why it matters. A skill is a set of instructions that the agent runs and trusts. If nobody reviewed a skill, it can contain hidden instructions that redirect the agent or send your data to an outside server.

How to improve it.

  1. Run a guard check on the discovered skills to surface the risky ones first.
  2. Block the skills that fail review and publish approved replacements from the skills library.
  3. Turn on build-time skill review so new skills need approval before use.

See Skills settings and Build-time guards.

Framework mapping: OWASP LLM LLM01 Prompt Injection · OWASP AST10 AST09 No Governance · NIST AI RMF GOVERN 1.6

Skills that load instructions from external sources​

approximate Partial coverage

Skills on devices whose instructions tell the agent to download content from an outside server while the skill runs. That content can be documentation, schemas, or scripts. Target: 0.

Why it matters. The agent treats the downloaded content as instructions and runs it with the agent's full permissions. Nothing locks that content to the version you reviewed. The author can change it at any time after your review. If someone else takes over the server, they control what every skill that points to it downloads.

How to improve it.

  1. Copy the downloaded content directly into the skill, so that the version you reviewed is the version that runs.
  2. Where it genuinely has to stay current, replace the raw URL with a managed integration behind the gateway, so the fetch is guarded and logged.
  3. Save the list of external hosts that this KPI produces. If one of those hosts is compromised, the list tells you which skills to remove.

Framework mapping: OWASP AST10 AST05 Untrusted External Instructions · OWASP LLM LLM01 Prompt Injection · OWASP LLM LLM03 Supply Chain

Share of AI tool calls flowing through the gateway​

approximate

Gateway tool calls as a percentage of all tool calls seen in the last 30 days, including tool calls the scan agent sees on devices that do not go through the gateway. Target: 90%.

Why it matters. Coverage limits every other control on this page. A guard that only sees 40% of traffic can only stop 40% of the problem.

How to improve it.

  1. Replace the most-used unmanaged MCP servers with managed integrations behind the gateway.
  2. Distribute client configuration through a plugin so new installs point at the gateway by default.
  3. Use Forced Managed rules to rewrite direct server configurations so they point at the matching gateway integration instead.

See Policy rules and Analytics.

Framework mapping: Gartner TRiSM Layer 2 runtime enforcement · Practitioner

Unreviewed skills shipping executable code​

Partial coverage

Skills on devices that meet all three of these conditions:

  • your organization did not publish them
  • no allow rule covers them
  • they include scripts or compiled programs as well as instructions

Target: 0.

Why it matters. A reviewer normally reads only the skill's instructions. The bundled files are what actually run. Nobody can read a compiled program, so nobody can review it.

How to improve it.

  1. Review the bundled files, not just the skill's instructions. Start with any compiled program. A legitimate skill has no reason to include one.
  2. Block the ones that fail review and publish an approved replacement from the skills library.
  3. Turn on build-time skill review so bundles are scanned before they reach devices.

Framework mapping: OWASP AST10 AST01 Malicious Skills · OWASP AST10 AST02 Supply Chain Compromise · MITRE ATLAS AML.T0010 AI Supply Chain Compromise

New AI capabilities untriaged for over 7 days​

Partial coverage

MCP servers and skills that Willow first found more than 7 days ago, and that still have no risk assessment and no policy decision. Target: 0.

Why it matters. If you do not review new items quickly, the queue grows faster than you can work through it. An item you have not reviewed for a week is a risk your organization is carrying without ever having agreed to carry it.

How to improve it.

  1. Work through the list, oldest item first. Choose one of three outcomes for each item: allow it, block it, or replace it with a managed integration.
  2. Run AI risk assessment on the batch to rank what to review first.

See Risk assessment.

Framework mapping: OWASP AST10 AST08 Poor Scanning · Practitioner

Unapproved GenAI apps with active traffic​

Web AI applications that the browser extension saw in use in the last 30 days and that your web AI policy does not allow. Willow counts each application once. Target: 0.

This KPI reads unmeasured until the Willow Guard browser extension reports.

Why it matters. Nobody has reviewed how an unapproved app handles your data. That leaves three gaps:

  • You have no data processing agreement (DPA) with the vendor.
  • You have not told the vendor to keep your data out of model training.
  • You have no record of what employees sent to the app.

How to improve it.

  1. Triage the list: approve the apps you want and block the rest, both from the row itself.
  2. Set the web AI policy default action to Warn or Block. Willow then warns the user or blocks the app the first time someone uses it, instead of only after you review it.
  3. For approved apps, confirm the vendor contract disables training on your data.

See Deploy the Willow Guard browser extension.

Framework mapping: NIST AI RMF GOVERN 1.6 inventory · Gartner TRiSM Layer 1 AI catalog

AI tools used from personal accounts​

Email addresses that the browser extension saw signing in to AI tools in the last 30 days. Willow counts an address when its domain is not one of your company email domains. Each address counts once. Target: 0.

This KPI reads unmeasured until the Willow Guard browser extension reports.

Why it matters. Work done in a personal AI account happens outside your organization's account. You cannot control how long the data is kept, you have no audit trail, and the employee keeps the data when they leave.

How to improve it.

  1. Give employees corporate accounts on the AI tools they already use. Employees usually use a personal account because you have not given them a company one, not because they are ignoring the rules.
  2. Use your MDM to deploy a policy that forces the tool to sign in to your organization only. The tool then refuses a personal login when it starts, instead of only warning about it. For Claude Code, set forceLoginMethod to claudeai and forceLoginOrgUUID to your organization UUID. For GitHub Copilot, set ChatApprovedAccountOrganizations to your GitHub organization.
  3. Add a web AI policy rule that warns users on the personal sign-in pages of the tools your company already provides. This covers browser-only tools, because those tools have no settings an administrator can lock.

See Claude Code Policy.

Framework mapping: ISO 27001 A.5.15 Access control · Practitioner

On this page