AI Gateway (LLM Firewall)

The AI Gateway is a runtime enforcement point for the AI in your applications — the analog of the Package Firewall, but for prompts, responses, and tool calls instead of dependencies. It evaluates every LLM interaction against a set of guardrails and can observe, redact, or block, all without becoming a single point of failure for the protected app.

Rollout note. The gateway's decision core is built — the evaluation engine, the guardrail checks, the monitor-mode telemetry path, and the safe-fail behavior. Inline proxy enforcement (redact/block on live traffic) is rolling out. Monitor mode is available to evaluate traffic and emit guardrail events today; enforce mode is gated behind an explicit per-tenant policy plus the ff.aispm_gateway flag. Default behavior is fail-open with alert — the gateway never breaks your AI app.

Monitor vs enforce

The gateway runs as an inline proxy (or a thin SDK shim around your LLM client) between your app and the model provider, in one of two modes:

• Monitor — every prompt, response, and tool call is evaluated, a guardrail event is recorded, and traffic passes through unchanged. This is the safe way to baseline what enforcement would do before turning it on.
• Enforce — the gateway acts on its verdict: redact PII or secret egress, or block injection and jailbreak attempts, per your policy.

Enforce mode requires the ff.aispm_gateway feature flag and explicit per-tenant opt-in. Monitor mode is the default everywhere.

Guardrails

Each interaction is checked along the direction it flows:

Indirect injection matters specifically because retrieved content (RAG results, tool output) is untrusted — the gateway inspects it, not only the human's input.

Safe-fail by design

The gateway is built so that a failure in the gateway is never a failure of your application:

• Fail-open by default. On any gateway error or timeout, traffic is allowed through and an alert is raised. Fail-closed is available only if a tenant explicitly opts in.
• Resource-capped. Each request has an evaluation budget (default 150 ms). Over budget or under backpressure, the gateway passes traffic through with an alert rather than blocking the app.
• Bounded inspection. Bodies larger than the configured limit are truncated for scanning and flagged as truncated.

Configuration

The gateway is configured per tenant. A representative policy:

aigateway:
  mode: monitor                 # monitor | enforce
  fail_mode: open               # open (never break the app) | closed
  eval_budget_ms: 150           # per-request eval cap; over budget -> pass through + alert
  provider: { name: anthropic, upstream: "https://api.anthropic.com" }
  checks:
    injection: block
    indirect_injection: block
    pii_egress: redact
    secret_egress: redact
    jailbreak: block
    toxicity: monitor
    hallucination: monitor
  rate: { rpm_per_tenant: 600, burst: 60 }
  redact: { patterns: [email, ssn, api_key, jwt], replacement: "[REDACTED]" }

The default policy is monitor mode, fail-open, with injection/jailbreak set to block and PII/secret egress set to redact. Run it locally or in CI:

safeguard aigateway start --mode monitor --provider anthropic --port 8490

Findings and telemetry

Gateway decisions are high-volume runtime telemetry, so they are recorded as guardrail events rather than as standard findings — one event per decision, carrying the direction, verdict, category, model name, rule, mode, and latency overhead. Guardrail events are tenant-scoped and audited. Raw prompt and response text is never persisted — events carry redacted metadata only. Static AI issues (model scans, agent/MCP analysis) still flow into the unified findings model as aisec.

Related

• AI-SPM — model-artifact scanning for the weights your apps load.
• AI-BOM Discovery — inventory of the LLM apps and endpoints the gateway protects.
• Red Team — validating that your AI guardrails actually hold.
• Package Firewall — the same monitor/enforce pattern for dependencies.
• Unified Findings & Feature Flags — how the AI engines are toggled.