The Inference Gateway: the security boundary

The component that holds the keys and guards the egress: its pipeline, the threat model around it, what anonymization does to outbound prompts, how model routing stays inspectable, and where the inference-tier boundary sits.

Inference Gateway pipeline

The gateway is shown as a single service in the main diagram; its internal request flow matters enough to call out separately. Every inbound request follows this pipeline:

            ┌──────────────┐
   Inbound  │ Auth         │  (API key resolution; rejects unauthenticated)
   request──▶│              │
            └──────┬───────┘
                   ▼
            ┌──────────────┐
            │ Router       │  (provider/model selection; fallback chains)
            └──────┬───────┘
                   ▼
            ┌──────────────┐
            │ Rate Limit   │  (Redis token bucket; per-key, per-model)
            └──────┬───────┘
                   ▼
            ┌──────────────┐
            │ Tier         │  (annotates request with routed_inference_tier 1–5;
            │ Derivation   │   refuses if below skill or Project minimum)
            └──────┬───────┘
                   ▼
            ┌──────────────┐
            │ Anonymization│  (M2; pseudonymizes sensitive entities;
            │ — pre        │   stable mapping for the request lifetime)
            └──────┬───────┘
                   ▼
            ┌──────────────┐
            │ Provider     │  (HTTP/gRPC to Anthropic / OpenAI / Azure /
            │ Adapter      │   Ollama / vLLM; Vertex + Bedrock deferred — DE-034/035)
            └──────┬───────┘
                   │
                 (response)
                   │
                   ▼
            ┌──────────────┐
            │ Anonymization│  (M2; rehydrates pseudonyms in response and
            │ — post       │   inside cited chunks; mapping discarded)
            └──────┬───────┘
                   ▼
            ┌──────────────┐
            │ Cost Tracker │  (tokens × per-model rates; tagged for analytics)
            └──────┬───────┘
                   ▼
            ┌──────────────┐
   Outbound │ Telemetry    │  (OTel traces; Langfuse if configured)
   response │              │
   ◀────────│              │
            └──────────────┘

The pipeline is what makes the Inference Tier model operationally real. The Tier Derivation stage is the choke point: every request gets classified, every classification is logged in the audit trail, and every UI surface reflects the actual routed tier in real time. The user does not have to take the application's word for it — they can verify the tier badge against the operator's gateway configuration and against the audit log entries.

The Anonymization stages bracket the provider adapter; pseudonyms exist in the mapping table for the duration of the request (in process memory only — never persisted) and are rehydrated on the way back so the Citation Engine sees the original text for verification. Privilege-flagged Projects disable anonymization by default — for privileged content, the operator is better served by Tier 1 (local inference, no third-party touch) than by an anonymization layer that adds processing steps complicating a privilege analysis.

For the full gateway specification including the configuration YAML and the OpenAPI surface, see PRD §4.