{"id":5879,"date":"2026-02-20T11:39:36","date_gmt":"2026-02-20T06:09:36","guid":{"rendered":"https:\/\/toolswift.com\/blog\/?p=5879"},"modified":"2026-02-20T11:39:36","modified_gmt":"2026-02-20T06:09:36","slug":"zero-trust-local-ai-hardening-vllm-docker-containers-behind-caddy-with-mtls-and-oidc-for-secure-production-inference","status":"publish","type":"post","link":"https:\/\/toolswift.com\/blog\/zero-trust-local-ai-hardening-vllm-docker-containers-behind-caddy-with-mtls-and-oidc-for-secure-production-inference\/","title":{"rendered":"Zero-Trust Local AI: Hardening vLLM Docker Containers Behind Caddy with mTLS and OIDC for Secure Production Inference"},"content":{"rendered":"<p>Running large language models locally is no longer a novelty; by 2026 it is a standard pattern for teams that need predictable latency, data residency, and cost control. vLLM has become a common serving layer because it exposes an OpenAI-compatible HTTP API and is designed for high-throughput inference.<\/p>\n<p>The security problem is that \u201clocal\u201d does not automatically mean \u201csafe.\u201d A production inference endpoint is still a high-value target:<\/p>\n<ul>\n<li>It can leak sensitive prompts and generated data.<\/li>\n<li>It can be abused for expensive GPU denial-of-service.<\/li>\n<li>It can become a pivot point into the rest of the network if container boundaries and credentials are weak.<\/li>\n<\/ul>\n<p>A zero-trust posture assumes the network is hostile\u2014even inside a homelab or \u201cinternal-only\u201d VLAN. The practical implication is that every hop must be authenticated, authorized, encrypted, observable, and least-privileged.<\/p>\n<p>This article presents an implementation-focused pattern for hardening vLLM in Docker behind Caddy, using:<\/p>\n<ul>\n<li>mTLS for device\/workload identity and encrypted transport.<\/li>\n<li>OIDC for user and service authentication with modern SSO.<\/li>\n<li>Network isolation + container hardening to reduce blast radius.<\/li>\n<li>Metrics\/logs to debug failure modes and detect abuse.<\/li>\n<\/ul>\n<p>The goal is secure production inference without turning the stack into a bespoke identity platform.<\/p>\n<h2>Architecture Overview<\/h2>\n<h3>Components<\/h3>\n<ul>\n<li><strong>vLLM<\/strong>: Runs the model and serves an OpenAI-compatible API over HTTP (default 8000).<\/li>\n<li><strong>Internal TLS sidecar (Nginx)<\/strong>: Terminates mTLS for upstream traffic to vLLM (because vLLM typically serves plain HTTP inside the container network).<\/li>\n<li><strong>Caddy<\/strong>: Edge reverse proxy, TLS termination, client certificate verification, and routing.<\/li>\n<li><strong>OIDC gateway (oauth2-proxy)<\/strong>: Handles OIDC login, session cookies, and identity headers for apps behind a reverse proxy (paired with Caddy forward_auth).<\/li>\n<li><strong>Prometheus\/Grafana (optional)<\/strong>: Scrapes vLLM \/metrics and visualizes key latency\/queue\/GPU cache signals. vLLM exposes Prometheus-compatible metrics at \/metrics.<\/li>\n<\/ul>\n<h3>Traffic flow<\/h3>\n<p><strong>Client \u2192 Caddy (mTLS + TLS)<\/strong><br \/>Client presents a certificate issued by an internal CA. Caddy requires and verifies it.<\/p>\n<p><strong>Caddy \u2192 oauth2-proxy (forward_auth)<\/strong><br \/>For protected routes, Caddy calls oauth2-proxy to verify an authenticated OIDC session.<\/p>\n<p><strong>Caddy \u2192 Nginx sidecar (mTLS)<\/strong><br \/>Even after the edge is authenticated, the internal hop is also authenticated with mTLS. This prevents any other container on the network from talking directly to inference.<\/p>\n<p><strong>Nginx \u2192 vLLM (loopback HTTP)<\/strong><br \/>Nginx forwards to 127.0.0.1:8000 inside the same pod\/compose \u201cservice\u201d namespace.<\/p>\n<h3>Why this composition<\/h3>\n<ul>\n<li>mTLS provides a strong, non-phishable identity factor at the transport layer.<\/li>\n<li>OIDC provides centralized user\/service identity and policy controls (MFA, device posture, conditional access) without hand-rolled auth.<\/li>\n<li>Defense in depth: Compromise of one layer does not automatically grant inference access.<\/li>\n<\/ul>\n<h2>Implementation Breakdown<\/h2>\n<h3>1) Certificates: internal CA and mTLS materials<\/h3>\n<p>Use an internal CA (Smallstep step-ca, Vault PKI, or an enterprise PKI). Smallstep is a common choice for mTLS automation in infrastructure environments.<\/p>\n<p>Create a root CA and issue:<\/p>\n<ul>\n<li>Server cert for ai.example.internal (Caddy edge)<\/li>\n<li>Client certs for operators\/services<\/li>\n<li>Server cert for the internal Nginx mTLS endpoint<\/li>\n<li>Client cert for Caddy \u2192 Nginx upstream authentication<\/li>\n<\/ul>\n<p>Example with step CLI (illustrative; adapt to your PKI):<\/p>\n<pre><code># Root CA (offline is preferable)\r\nstep certificate create \"Local AI Root CA\" root_ca.crt root_ca.key \\\r\n  --profile root-ca --no-password --insecure\r\n\r\n# Issue an edge server cert for Caddy\r\nstep certificate create ai.example.internal caddy.crt caddy.key \\\r\n  --profile leaf --ca root_ca.crt --ca-key root_ca.key \\\r\n  --san ai.example.internal --san ai --no-password --insecure\r\n\r\n# Issue a client cert for a human operator\/device\r\nstep certificate create \"DevOps Laptop\" client-devops.crt client-devops.key \\\r\n  --profile leaf --ca root_ca.crt --ca-key root_ca.key \\\r\n  --set \"subjectAltName=email:devops@example.com\" \\\r\n  --no-password --insecure<\/code><\/pre>\n<p><strong>Operational notes:<\/strong><\/p>\n<ul>\n<li>Store root CA private keys offline.<\/li>\n<li>Rotate leaf certs aggressively (days\/weeks), automate issuance, and revoke promptly.<\/li>\n<li>Separate Caddy\u2019s public TLS identity from mTLS trust roots when possible (don\u2019t reuse the same CA for everything).<\/li>\n<\/ul>\n<p>Caddy supports client certificate authentication configuration via its TLS settings, including client_auth with trusted_ca_cert_file.<\/p>\n<h3>2) Docker Compose: isolated network + hardened services<\/h3>\n<p>This Compose file:<\/p>\n<ul>\n<li>Puts only Caddy on the \u201cpublic\u201d network.<\/li>\n<li>Keeps vLLM isolated on an internal network.<\/li>\n<li>Forces all traffic to vLLM through the Nginx mTLS sidecar.<\/li>\n<li>Adds conservative container hardening defaults.<\/li>\n<\/ul>\n<pre><code>services:\r\n  caddy:\r\n    image: caddy:2.8\r\n    restart: unless-stopped\r\n    ports:\r\n      - \"443:443\"\r\n      - \"80:80\"\r\n    networks:\r\n      - public\r\n      - internal\r\n    volumes:\r\n      - .\/caddy\/Caddyfile:\/etc\/caddy\/Caddyfile:ro\r\n      - .\/pki\/root_ca.crt:\/etc\/caddy\/pki\/root_ca.crt:ro\r\n      - .\/pki\/caddy_upstream_client.crt:\/etc\/caddy\/pki\/up_client.crt:ro\r\n      - .\/pki\/caddy_upstream_client.key:\/etc\/caddy\/pki\/up_client.key:ro\r\n      - caddy_data:\/data\r\n      - caddy_config:\/config\r\n    read_only: true\r\n    security_opt:\r\n      - no-new-privileges:true\r\n    cap_drop: [\"ALL\"]\r\n\r\n  oauth2-proxy:\r\n    image: quay.io\/oauth2-proxy\/oauth2-proxy:v7.6.0\r\n    restart: unless-stopped\r\n    networks:\r\n      - internal\r\n    environment:\r\n      OAUTH2_PROXY_PROVIDER: oidc\r\n      OAUTH2_PROXY_OIDC_ISSUER_URL: \"https:\/\/idp.example.com\/realms\/prod\"\r\n      OAUTH2_PROXY_CLIENT_ID: \"local-ai\"\r\n      OAUTH2_PROXY_CLIENT_SECRET: \"${OAUTH2_PROXY_CLIENT_SECRET}\"\r\n      OAUTH2_PROXY_COOKIE_SECRET: \"${OAUTH2_PROXY_COOKIE_SECRET}\"\r\n      OAUTH2_PROXY_EMAIL_DOMAINS: \"*\"\r\n      OAUTH2_PROXY_HTTP_ADDRESS: \"0.0.0.0:4180\"\r\n      OAUTH2_PROXY_UPSTREAMS: \"static:\/\/200\"\r\n      OAUTH2_PROXY_REVERSE_PROXY: \"true\"\r\n      OAUTH2_PROXY_SET_XAUTHREQUEST: \"true\"\r\n      OAUTH2_PROXY_WHITELIST_DOMAINS: \".example.internal\"\r\n      OAUTH2_PROXY_COOKIE_SECURE: \"true\"\r\n      OAUTH2_PROXY_COOKIE_SAMESITE: \"lax\"\r\n    read_only: true\r\n    security_opt:\r\n      - no-new-privileges:true\r\n    cap_drop: [\"ALL\"]\r\n\r\n  vllm:\r\n    image: vllm\/vllm-openai:latest\r\n    restart: unless-stopped\r\n    networks:\r\n      - internal\r\n    # vLLM OpenAI-compatible server. API key enforcement is still useful for defense-in-depth.\r\n    command: &gt;\r\n      vllm serve NousResearch\/Meta-Llama-3-8B-Instruct\r\n      --dtype auto\r\n      --host 127.0.0.1\r\n      --port 8000\r\n      --api-key ${VLLM_API_KEY}\r\n      --disable-access-log-for-endpoints \"\/health,\/metrics\"\r\n    volumes:\r\n      - vllm_models:\/root\/.cache\/huggingface\r\n    deploy:\r\n      resources:\r\n        reservations:\r\n          devices:\r\n            - capabilities: [\"gpu\"]\r\n    # Optional: keep container writable if your model cache needs it; otherwise prefer read_only.\r\n    security_opt:\r\n      - no-new-privileges:true\r\n\r\n  vllm-mtls:\r\n    image: nginx:1.27-alpine\r\n    restart: unless-stopped\r\n    depends_on: [vllm]\r\n    networks:\r\n      - internal\r\n    ports: [] # no published ports\r\n    volumes:\r\n      - .\/nginx\/nginx.conf:\/etc\/nginx\/nginx.conf:ro\r\n      - .\/pki\/root_ca.crt:\/etc\/nginx\/pki\/root_ca.crt:ro\r\n      - .\/pki\/vllm_server.crt:\/etc\/nginx\/pki\/server.crt:ro\r\n      - .\/pki\/vllm_server.key:\/etc\/nginx\/pki\/server.key:ro\r\n    read_only: true\r\n    security_opt:\r\n      - no-new-privileges:true\r\n    cap_drop: [\"ALL\"]\r\n\r\nnetworks:\r\n  public:\r\n  internal:\r\n    internal: true\r\n\r\nvolumes:\r\n  caddy_data:\r\n  caddy_config:\r\n  vllm_models:<\/code><\/pre>\n<p><strong>Why these choices matter:<\/strong><\/p>\n<ul>\n<li><code>internal: true<\/code> prevents direct egress\/ingress from the internal network except through attached services.<\/li>\n<li>vLLM binds to <code>127.0.0.1<\/code> so only the sidecar can reach it.<\/li>\n<li>Removing access logs for <code>\/health<\/code> and <code>\/metrics<\/code> reduces noise; vLLM supports this via <code>--disable-access-log-for-endpoints<\/code>.<\/li>\n<\/ul>\n<h3>3) Nginx sidecar: enforce mTLS to the model API<\/h3>\n<p><code>nginx.conf<\/code> terminates TLS and requires client certificates for any request. It proxies to vLLM over loopback HTTP.<\/p>\n<pre><code>events {}\r\n\r\nhttp {\r\n  server {\r\n    listen 8443 ssl;\r\n\r\n    ssl_certificate     \/etc\/nginx\/pki\/server.crt;\r\n    ssl_certificate_key \/etc\/nginx\/pki\/server.key;\r\n\r\n    # mTLS requirement\r\n    ssl_client_certificate \/etc\/nginx\/pki\/root_ca.crt;\r\n    ssl_verify_client on;\r\n\r\n    # Basic hardening\r\n    ssl_session_cache shared:SSL:10m;\r\n    ssl_session_timeout 10m;\r\n\r\n    # Limit request sizes to reduce abuse\r\n    client_max_body_size 2m;\r\n\r\n    location \/ {\r\n      proxy_pass http:\/\/127.0.0.1:8000;\r\n      proxy_http_version 1.1;\r\n      proxy_set_header Host $host;\r\n      proxy_set_header X-Forwarded-Proto https;\r\n      proxy_set_header X-Forwarded-For $remote_addr;\r\n\r\n      # Optional timeouts (tune for streaming)\r\n      proxy_read_timeout 300s;\r\n      proxy_send_timeout 300s;\r\n    }\r\n  }\r\n}<\/code><\/pre>\n<p><strong>Key point:<\/strong> Even if an attacker lands on the internal Docker network, they still cannot talk to inference without presenting a valid client certificate trusted by the sidecar.<\/p>\n<h3>4) Caddy: edge TLS+mTLS, OIDC gate, and upstream mTLS<\/h3>\n<p>Caddy terminates public TLS, verifies client certificates, then uses forward_auth to oauth2-proxy for OIDC. Finally it proxies to the Nginx sidecar over mTLS, presenting its own upstream client certificate.<\/p>\n<p>Caddyfile:<\/p>\n<pre><code>{\r\n  # Reduce information leakage\r\n  servers {\r\n    protocols h1 h2\r\n  }\r\n}\r\n\r\nai.example.internal {\r\n  encode zstd gzip\r\n\r\n  # Edge TLS + require client certs (mTLS)\r\n  tls \/etc\/caddy\/certs\/caddy.crt \/etc\/caddy\/certs\/caddy.key {\r\n    client_auth {\r\n      mode require_and_verify\r\n      trusted_ca_cert_file \/etc\/caddy\/pki\/root_ca.crt\r\n    }\r\n  }\r\n\r\n  # Health endpoint for load balancers (still mTLS-protected)\r\n  respond \/health 200\r\n\r\n  # OIDC authentication via oauth2-proxy\r\n  # forward_auth is Caddy\u2019s opinionated auth_request-style integration.\r\n  forward_auth oauth2-proxy:4180 {\r\n    uri \/oauth2\/auth\r\n    copy_headers X-Auth-Request-User X-Auth-Request-Email Authorization Set-Cookie\r\n  }\r\n\r\n  # Protect vLLM OpenAI-compatible endpoints\r\n  @openai_api {\r\n    path \/v1\/* \/metrics\r\n  }\r\n\r\n  handle @openai_api {\r\n    reverse_proxy vllm-mtls:8443 {\r\n      transport http {\r\n        tls\r\n        tls_insecure_skip_verify false\r\n        tls_trusted_ca_certs \/etc\/caddy\/pki\/root_ca.crt\r\n\r\n        # Client cert for Caddy -&gt; Nginx sidecar mTLS\r\n        tls_client_certificate \/etc\/caddy\/pki\/up_client.crt\r\n        tls_client_certificate_key \/etc\/caddy\/pki\/up_client.key\r\n      }\r\n    }\r\n  }\r\n\r\n  # Default deny\r\n  respond 404\r\n}<\/code><\/pre>\n<p><strong>Why the layered controls matter:<\/strong><\/p>\n<ul>\n<li>Client mTLS at the edge blocks anonymous network scanning and forces device identity before anything else.<\/li>\n<li>OIDC via forward_auth adds user identity, MFA, group-based policy, and session management.<\/li>\n<li>Upstream mTLS prevents lateral movement inside the container network and avoids \u201cany container can call inference.\u201d<\/li>\n<\/ul>\n<h2>Observability and Debugging<\/h2>\n<h3>Metrics<\/h3>\n<p>vLLM exposes Prometheus-compatible metrics at <code>\/metrics<\/code> on the OpenAI-compatible server. Key signals to scrape and alert on:<\/p>\n<ul>\n<li>Request queue depth (waiting\/running)<\/li>\n<li>Time-to-first-token and per-token latency histograms<\/li>\n<li>GPU KV cache usage \/ cache hit behavior<\/li>\n<li>Error counters (5xx, request failures)<\/li>\n<\/ul>\n<p><strong>Operational guidance:<\/strong><\/p>\n<ul>\n<li>Scrape <code>\/metrics<\/code> from the internal network only.<\/li>\n<li>If <code>\/metrics<\/code> must be exposed, protect it with the same mTLS + OIDC policy as the inference endpoints.<\/li>\n<\/ul>\n<h3>Logs<\/h3>\n<p>Collect logs from:<\/p>\n<ul>\n<li>Caddy access logs: request metadata, auth outcomes, upstream errors.<\/li>\n<li>oauth2-proxy logs: OIDC handshake, cookie\/session validation, redirects.<\/li>\n<li>Nginx TLS logs: mTLS handshake failures (bad cert, unknown CA).<\/li>\n<li>vLLM logs: model load events, OOM errors, request failures.<\/li>\n<\/ul>\n<p>vLLM supports suppressing access logs for high-frequency endpoints to reduce log noise.<\/p>\n<h3>Failure modes and common misconfigurations<\/h3>\n<p><strong>mTLS handshake fails at the edge<\/strong><\/p>\n<p>Symptoms: browser shows \u201cbad certificate,\u201d curl fails with tlsv1 alert unknown ca.<\/p>\n<p>Causes: wrong trust root in Caddy, missing client cert, client cert not intended for clientAuth EKU.<\/p>\n<p><strong>OIDC loop \/ redirect storms<\/strong><\/p>\n<p>Symptoms: repeated redirects to <code>\/oauth2\/start<\/code>, never reaches API.<\/p>\n<p>Causes: cookie secret mismatch, wrong redirect URL, domain mismatch, missing <code>--reverse-proxy<\/code> (oauth2-proxy setting).<\/p>\n<p><strong>Caddy \u2192 upstream fails<\/strong><\/p>\n<p>Symptoms: 502\/504 from Caddy.<\/p>\n<p>Causes: Caddy not presenting upstream client cert, Nginx requires mTLS but Caddy configured without it, wrong trusted CA.<\/p>\n<p><strong>Inference latency spikes<\/strong><\/p>\n<p>Symptoms: increased time-to-first-token, queue depth grows.<\/p>\n<p>Causes: GPU saturation, KV cache pressure, too-large max_tokens, insufficient batching headroom, CPU throttling.<\/p>\n<h2>Security Considerations<\/h2>\n<h3>Attack surface<\/h3>\n<ul>\n<li>Edge proxy (TLS termination, auth routing)<\/li>\n<li>OIDC gateway (cookie\/session handling)<\/li>\n<li>Inference API (high-cost compute, sensitive payloads)<\/li>\n<li>Model artifacts and cache (supply chain and integrity)<\/li>\n<\/ul>\n<h3>Hardening strategies<\/h3>\n<p><strong>Least privilege containers<\/strong><\/p>\n<p>no-new-privileges, drop Linux caps, read-only filesystems where feasible.<\/p>\n<p><strong>Network isolation<\/strong><\/p>\n<ul>\n<li>Separate public and internal networks; publish only 443\/80 from Caddy.<\/li>\n<li>Bind vLLM to loopback and force sidecar mediation.<\/li>\n<\/ul>\n<p><strong>Policy layering<\/strong><\/p>\n<ul>\n<li>mTLS for device\/workload identity + OIDC for user identity.<\/li>\n<li>Keep vLLM\u2019s <code>--api-key<\/code> enabled as an additional check.<\/li>\n<\/ul>\n<p><strong>Secrets management<\/strong><\/p>\n<p>Store OIDC client secrets and cookie secrets outside Compose:<\/p>\n<ul>\n<li>Docker secrets, SOPS-encrypted env files, Vault Agent injection, or Kubernetes Secrets with sealed-secrets\/external-secrets.<\/li>\n<\/ul>\n<p>Keep TLS private keys on tmpfs or locked-down volumes; enforce file ownership and permissions.<\/p>\n<p><strong>Rotate:<\/strong><\/p>\n<ul>\n<li>OIDC client secret (as required by IdP policy)<\/li>\n<li>oauth2-proxy cookie secret (planned maintenance window)<\/li>\n<li>mTLS leaf certs (automated)<\/li>\n<\/ul>\n<h3>Plugin risk management<\/h3>\n<p>Caddy can do OIDC with plugins (for example, caddy-security provides OIDC and JWT-based controls). However, third-party auth plugins increase supply-chain and vulnerability management burden; prior security research has identified issues in at least one SSO-related Caddy plugin ecosystem.<\/p>\n<p><strong>Mitigations:<\/strong><\/p>\n<ul>\n<li>Prefer \u201cseparate process\u201d gateways (oauth2-proxy) when organizational policy requires conservative dependency surfaces.<\/li>\n<li>If using plugins, pin builds, track advisories, and implement compensating controls (mTLS, network isolation, WAF\/rate limits).<\/li>\n<\/ul>\n<h2>Performance Considerations<\/h2>\n<h3>Bottlenecks<\/h3>\n<ul>\n<li>GPU memory and KV cache: primary limiter for concurrency and context length.<\/li>\n<li>CPU scheduling: tokenization and request handling can become CPU-bound under load.<\/li>\n<li>Network and TLS overhead: usually minor relative to model compute, but mTLS adds handshake cost for short-lived clients.<\/li>\n<\/ul>\n<h3>Scaling options<\/h3>\n<p><strong>Vertical:<\/strong> more GPU memory, faster GPUs, tuned concurrency.<\/p>\n<p><strong>Horizontal:<\/strong><\/p>\n<ul>\n<li>Multiple vLLM replicas behind Caddy (or a L4\/L7 load balancer).<\/li>\n<li>Sticky routing for long streaming sessions if required.<\/li>\n<\/ul>\n<p><strong>Segmentation:<\/strong><\/p>\n<ul>\n<li>Separate deployments by model size or tenant to prevent noisy-neighbor effects.<\/li>\n<\/ul>\n<h3>Resource optimization<\/h3>\n<p>Use vLLM-native metrics to tune batching and concurrency. Consider:<\/p>\n<ul>\n<li>Quantized model variants where acceptable<\/li>\n<li>Request limits (max prompt size, max tokens)<\/li>\n<li>Rate limiting at the edge (per user\/client cert subject)<\/li>\n<\/ul>\n<h2>Tradeoffs and Design Decisions<\/h2>\n<h3>Why this approach vs alternatives<\/h3>\n<p><strong>Caddy + oauth2-proxy vs \u201cCaddy OIDC plugin everywhere\u201d<\/strong><br \/>forward_auth with oauth2-proxy keeps OIDC logic in a dedicated gateway, reduces custom Caddy builds, and aligns with common reverse-proxy auth_request patterns.<\/p>\n<p><strong>mTLS everywhere vs \u201cOIDC only\u201d<\/strong><br \/>OIDC alone does not authenticate machines at the transport layer. mTLS blocks unauthenticated network access even if OIDC endpoints are reachable and reduces exposure to token theft\/replay in internal hops.<\/p>\n<p><strong>Sidecar TLS vs \u201cTLS in the app\u201d<\/strong><br \/>Many inference servers (including typical vLLM deployments) run plain HTTP behind a proxy for simplicity. vLLM\u2019s standard Docker\/OpenAI server pattern emphasizes HTTP serving; a sidecar is a practical way to enforce mTLS without patching the app.<\/p>\n<h3>What to avoid<\/h3>\n<ul>\n<li>Exposing vLLM directly on a LAN without mTLS and authorization gates.<\/li>\n<li>Relying on \u201cinternal network = trusted network.\u201d<\/li>\n<li>Long-lived static certificates with no revocation\/rotation plan.<\/li>\n<li>Running auth plugins without pinned versions and a vulnerability response process.<\/li>\n<li>Leaving <code>\/metrics<\/code> unauthenticated; metrics often leak capacity and load patterns.<\/li>\n<\/ul>\n<h2>Conclusion<\/h2>\n<p>A zero-trust local AI inference stack in 2026 must treat the inference API as production-critical infrastructure: authenticated, authorized, encrypted, observable, and least-privileged by default.<\/p>\n<p>This architecture delivers that by combining:<\/p>\n<ul>\n<li>Caddy as a clean edge proxy with client certificate verification and forward_auth.<\/li>\n<li>OIDC via oauth2-proxy for SSO-grade identity and policy integration.<\/li>\n<li>End-to-end mTLS, including the internal hop to the inference service, enforced with an Nginx sidecar.<\/li>\n<li>First-class observability using vLLM\u2019s Prometheus metrics endpoint and structured proxy logs.<\/li>\n<\/ul>\n<p>The result is a practical, publishable pattern for secure production inference that works in enterprise environments and advanced homelabs alike\u2014without assuming a trusted network, and without turning model serving into an identity science project.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Running large language models locally is no longer a novelty; by 2026 it is a standard pattern for teams that need predictable latency, data&hellip;<\/p>\n","protected":false},"author":1,"featured_media":5880,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1018],"tags":[],"class_list":["post-5879","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-machine-learning"],"_links":{"self":[{"href":"https:\/\/toolswift.com\/blog\/wp-json\/wp\/v2\/posts\/5879","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/toolswift.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/toolswift.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/toolswift.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/toolswift.com\/blog\/wp-json\/wp\/v2\/comments?post=5879"}],"version-history":[{"count":1,"href":"https:\/\/toolswift.com\/blog\/wp-json\/wp\/v2\/posts\/5879\/revisions"}],"predecessor-version":[{"id":5881,"href":"https:\/\/toolswift.com\/blog\/wp-json\/wp\/v2\/posts\/5879\/revisions\/5881"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/toolswift.com\/blog\/wp-json\/wp\/v2\/media\/5880"}],"wp:attachment":[{"href":"https:\/\/toolswift.com\/blog\/wp-json\/wp\/v2\/media?parent=5879"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/toolswift.com\/blog\/wp-json\/wp\/v2\/categories?post=5879"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/toolswift.com\/blog\/wp-json\/wp\/v2\/tags?post=5879"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}