Secure LLM Gateway Checklist: Keys, Tenants, Logs, and Policies

Separate caller identity from provider secrets

Applications should authenticate with scoped gateway identities, not shared provider credentials. Store provider keys encrypted, decrypt only where calls are made, and rotate them without changing every application.

A virtual key should map to one organization, environment, and approved policy scope. Avoid keys that can name arbitrary tenants in request headers.

Enforce tenant boundaries everywhere

Tenant scope must apply to logs, budgets, cache namespaces, vector searches, rate limits, analytics, and administration. A semantic cache with weak partitioning can leak content even when the database rows themselves are protected.

Test cross-tenant negative cases and include them in release gates.

Minimize prompt retention

Full prompts and outputs can contain personal data, secrets, customer content, or regulated information. Default to metadata and hashes, use explicit route policies for bodies, redact sensitive fields, and set retention periods.

Administrators who can change routing or read prompts should have separate permissions, strong authentication, and audit logs.

• Scoped keys and secret rotation
• Tenant-safe caches and queries
• TLS and restricted egress
• Prompt redaction and retention policy
• Admin RBAC and audit events
• Rate limits and abuse detection
• Dependency and provider incident plans

Model policy is a security control

Allowlists, data-region rules, tool restrictions, and provider contracts should be enforced before routing. A cost optimizer must never override a security constraint to select a cheaper path.