AI proxy vs LLM gateway: what is the difference?

Tell network forwarding, an API gateway, and model routing apart so you pick the layer that matches the job.

People say “AI proxy” for three different boxes: a network forwarder, a reverse API gateway, and a model router. This guide is for developers who need to pick one layer without mixing their jobs. It does not sell IP addresses, does not hand out keys, and does not claim a proxy changes account or geography eligibility.

01What people usually mean by “AI proxy”#

The phrase is not a standard. RFC 9110 (HTTP Semantics, June 2022, STD 97) already splits intermediaries into a proxy, a gateway, and a tunnel, and it warns that one process can switch roles per request. In product copy the same word is reused for something else again: a process that accepts an OpenAI-style JSON body and chooses a model.

Those three jobs look similar in a diagram because each sits between your client and something upstream. They fail in different places. A forward proxy can open a TCP tunnel and still leave you with a provider 401. An LLM gateway can return HTTP 200 with a JSON error about an unknown model while the network path was fine. If you debug the wrong layer, you will rotate the wrong secret.

A bounded local run on 2026-10-15 made that split concrete. A loopback HTTP CONNECT proxy forwarded TLS to a fixture origin. A separate loopback process routed fixture-alpha versus fixture-bravo after a local Bearer check. Nothing left 127.0.0.1. Nothing called a paid model API. The fixture JSON is labeled CONTROLLED_FIXTURE_NOT_A_PROVIDER on purpose.

When a vendor names a product “AI proxy”, read which of those jobs it actually implements. Do not assume retries, streaming, or fallback exist because the word gateway appears on the homepage.

02HTTP and SOCKS forward proxies, including CONNECT#

A forward proxy is chosen by the client. RFC 9110 §3.7 calls it a message-forwarding agent that receives requests for some absolute URIs and tries to satisfy them through HTTP. For ordinary http URLs the client sends the full URL to the proxy. For https the usual pattern is different.

RFC 9110 §9.3.6 defines CONNECT: the client asks the proxy to build a tunnel to host:port and, after any 2xx, to restrict itself to blind forwarding until the tunnel closes. Tunnels are commonly wrapped in TLS after that. A successful CONNECT is not an application success. It only means the tunnel exists. Any other response means the tunnel was not formed.

SOCKS5 is older and sits lower. RFC 1928 (March 1996) describes it as a shim between the application and transport, conventionally on TCP 1080. After method negotiation the client sends a command; CONNECT is X'01'. SOCKS does not parse HTTP methods. It relays TCP (and optionally UDP) to a destination the client names.

If your HTTP library speaks HTTPS through a proxy, it is usually doing CONNECT, then TLS to the origin. The proxy authenticates the tunnel, often with Proxy-Authorization. That header is not the Authorization: Bearer header the model API expects. Mixing them is a common source of “the proxy works in curl but the SDK still 401s”.

If the job is only “this process should leave through another IP”, start from the English lists of HTTP proxies, HTTPS proxies, and SOCKS5 proxies. Those pages compare exits. They are not model routers.

03Reverse proxies and ordinary API gateways#

A reverse proxy, which RFC 9110 also calls a gateway in §3.7, presents itself as the origin to the client and forwards inbound to another server. The client did not pick it as a proxy URL. The client just opened https://api.example.com.

The NGINX reverse-proxy guide (accessed 2026-10-15) describes the mechanical loop: send the request to a proxy_pass upstream, fetch the response, send it back. By default NGINX rewrites Host and Connection. That detail matters for APIs that key virtual hosts off Host. A reverse proxy can terminate TLS on the public edge and open a new connection inland. The client’s TLS session then ends at the reverse proxy, not at the model vendor.

This layer is the right tool for path routing (/v1/ versus /admin/), load balancing, header sanitizing, and exposing one hostname in front of several internal services. It may not know what a model name is. If it only forwards /v1/chat/completions to a single upstream, it is still an API gateway in the HTTP sense, not an LLM gateway.

Do not treat “we put NGINX in front” as proof of retries, token budgets, or model fallback. Those are extra features some products add. The NGINX page we read does not claim them.

04LLM gateways: keys, limits, routing, and logs#

An LLM gateway is an application service. Clients send a chat or embeddings body to one URL. The gateway checks a virtual key, may enforce budgets, may pick a provider and model, may log tokens, and only then talks to a vendor — or to several vendors.

LiteLLM’s AI Gateway page (HTTP 200 on 2026-10-15) describes a self-hosted server that offers one OpenAI-compatible endpoint. The documented jobs include virtual keys with per-key models and rate limits, spend tracking, load balancing, fallbacks, retries, guardrails, and logging. The same page says: if a client can use the OpenAI API, it can use the gateway without changing client code. That is a product claim about the chat completions request shape. It is not a proof that every OpenAI endpoint, header, and error maps 1:1. We did not install or run LiteLLM.

Cloudflare AI Gateway (page dateModified 2026-04-20, fetched 2026-10-15) lists analytics, logging, caching, rate limiting, request retries, and model fallback as features. The dynamic-routing page (updated 2026-10-02) is more precise about compatibility: dynamic routes accept the OpenAI chat completions request shape only; Anthropic Messages return HTTP 400. That sentence is why “OpenAI-compatible” must stay narrow. We did not call Cloudflare and we did not execute retries or fallback. If a gateway document lists those features, treat them as source-only until you run them against that product.

Locally we built a tiny router that is not those products. It accepted Authorization: Bearer local-demo-key-not-a-provider-secret, mapped fixture-alpha and fixture-bravo to two fixture payloads, returned 401 without a valid local token, and returned 400 for an unknown model. That is enough to show the control plane sits in HTTP JSON, not in CONNECT.

05Follow a request: TLS, Host, and who can read the body#

Walk one POST /v1/chat/completions through both designs.

Forward path used in the loopback run: the client opened TCP to 127.0.0.1:54318, sent CONNECT 127.0.0.1:54316 HTTP/1.1, received 200 Connection Established, then completed TLS to the origin certificate. Only after TLS did it send the JSON body and a Bearer header meant for the origin. The CONNECT process logged the target 127.0.0.1:54316. It did not see the fixture JSON or the Bearer token: the token list of HTTP body fragments observed on the tunnel was empty. TLS ended at the fixture origin. The origin answered HTTP 200 with the controlled fixture.

Gateway path: the client opened TCP to 127.0.0.1:54317 with no CONNECT. TLS was not used on that hop in the demo (plain HTTP on loopback). The gateway read the JSON model field and the local Bearer token before any origin existed for that request. Missing or wrong local auth never reached a model fixture as a successful route; both produced HTTP 401 with invalid_local_gateway_auth. An unknown model produced HTTP 400 with unsupported_model_or_route after auth succeeded. Transport was up. The application refused the job.

In production HTTPS, an LLM gateway almost always terminates TLS. It must, if it is going to parse JSON, apply a budget, or pick a model. A CONNECT forward proxy that is not a TLS middlebox does not terminate that origin TLS. If someone enables HTTPS inspection on a corporate proxy, that is a different deployment: the proxy becomes a TLS client to the origin and a TLS server to you. Do not assume inspection just because CONNECT succeeded.

OpenAI’s API overview (fetched 2026-10-15; the requested platform.openai.com/docs/api-reference redirected here) documents Bearer credentials on the origin request. It treats x-request-id as a request correlation identifier, separate from rate-limit headers such as x-ratelimit-limit-requests. Those fields belong to the provider conversation. A gateway may add its own request IDs. A forward proxy usually does not.

06Compatibility is a client shape, not the whole API#

SDKs often let you set base_url and keep using chat.completions.create. Vendors call that OpenAI-compatible. Cloudflare’s dynamic-routing note is the caution: even inside one vendor, a compatibility path may accept only one request shape and 400 another.

A compatible endpoint can still differ in authentication (virtual key versus provider key), model names (gpt-4.1 versus azure/gpt-4.1 versus a gateway alias), streaming event layouts, tool-call fields, file uploads, realtime sockets, and administration APIs. OpenAI’s own overview treats Responses, Realtime, and Administration as separate surfaces. Copying /v1/chat/completions does not copy that whole tree.

The loopback gateway returned JSON that looks like a chat completion (choices[0].message). That was a fixture. It does not make the demo “OpenAI compatible” in any product sense.

07Three different kinds of “it worked”#

Keep these three outcomes separate when you read logs.

On 2026-10-15 the local script recorded six cases and sixteen assertions, all passing, with Python 3.9.6 and LibreSSL 3.3.6:

Forwarding produced origin HTTP 200. Gateway routing produced two different fixture routes. Auth failures stayed at the gateway with 401. The unsupported model stayed at the gateway with 400. Retries, streaming, and fallbacks were not executed and are not claimed.

The What is my IP page reports the address that reached that page from your browser. The proxy checker is a server-side probe of an endpoint you supply. Neither one shows the route a Python SDK process will take, and neither one establishes LLM API eligibility. The checker does not prove a model vendor will accept an IP, that a key is valid, or that you are allowed to use a given account.

08Choose the layer that matches the job#

Use a forward HTTP CONNECT or SOCKS5 proxy when the problem is where the TCP session leaves — a corporate egress, a development laptop that cannot reach a vendor, or a published datacenter or mobile exit. Configure the SDK’s proxy settings (or HTTPS_PROXY) and keep the provider API key on the request that the origin will see. Do not expect the proxy to pick models.

Use a reverse proxy or generic API gateway when the problem is one hostname, many internal services, TLS at the edge, or header policy. Point DNS at that gateway. Do not expect it to know token spend unless you added that yourself.

Use an LLM gateway when the problem is many clients, many models, one policy. Issue virtual keys, set budgets, log prompts if your policy allows it, and map aliases onto provider models. Keep provider keys on the gateway host, not in every laptop. Read the vendor’s current compatibility notes; do not assume retries or fallbacks. If you only have one Python process and no shared policy, a gateway may be extra moving parts — LiteLLM’s own docs split “gateway versus SDK” on that axis.

Checklist before you buy or build:

1. Write down the failure you actually saw: TCP, TLS, HTTP status, or JSON error.
2. Name who terminates TLS.
3. Name which secret is for the tunnel, which is for the gateway, and which is for the provider.
4. Name the model string the client sends and the string the provider expects.
5. Confirm whether streaming, retries, and fallback are documented and tested on that product. If you have not run them, do not claim them.
6. Confirm the client request shape (chat completions versus another schema).
7. If you only needed a different exit IP, stop at the forward proxy and skip the model router.

The numbered table and WebP figure above come from a separate CONNECT+TLS loopback lab run the same day. That lab is actual client-selected forwarding: HTTP CONNECT, then TLS to the origin. It is not the copy-paste recipe below.

This public recipe is plain HTTP on 127.0.0.1 only. It does not open a CONNECT tunnel, does not use TLS, and does not call a model vendor. Its Forward hop always posts to a fixed origin_port, and the client uses an origin-form path (/v1/chat/completions). In the RFC 9110 sense that is a reverse-style, fixed-upstream pass-through, not a client-selected HTTP forward proxy. Compare that hop with the application model router in the same script. Do not read forward_http in the expected output as proof of a client-chosen forward proxy.

Setup: Python 3.9 or newer. Standard library only. No packages. No API keys.

1. Save the following script as forwarding_vs_routing.py in any directory you own.
2. Run: python3 forwarding_vs_routing.py
3. Your stdout should match the expected output after the script.

#!/usr/bin/env python3
"""Plain-HTTP loopback demo: forwarding vs model routing. Stdlib only."""
from __future__ import annotations

import json
import threading
from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

BIND = "127.0.0.1"
KEY = "***"
LABEL = "CONTROLLED_FIXTURE_NOT_A_PROVIDER"
TIMEOUT = 5


class Origin(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def log_message(self, *_args):
        return

    def do_POST(self):
        n = int(self.headers.get("Content-Length") or 0)
        self.rfile.read(n)
        body = json.dumps(
            {
                "label": LABEL,
                "origin": "loopback-http-origin",
                "choices": [
                    {
                        "index": 0,
                        "message": {
                            "role": "assistant",
                            "content": "FIXTURE: forwarded origin",
                        },
                    }
                ],
            }
        ).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.send_header("Connection", "close")
        self.end_headers()
        self.wfile.write(body)


class Gateway(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"
    ROUTES = {
        "fixture-alpha": {"label": LABEL, "route": "alpha"},
        "fixture-bravo": {"label": LABEL, "route": "bravo"},
    }

    def log_message(self, *_args):
        return

    def _send(self, code, obj):
        raw = json.dumps(obj).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(raw)))
        self.send_header("Connection", "close")
        self.end_headers()
        self.wfile.write(raw)

    def do_POST(self):
        n = int(self.headers.get("Content-Length") or 0)
        raw = self.rfile.read(n)
        auth = self.headers.get("Authorization") or ""
        try:
            model = json.loads(raw.decode() or "{}").get("model")
        except json.JSONDecodeError:
            model = None
        if auth != "Bearer " + KEY:
            self._send(401, {"error": "invalid_local_gateway_auth", "label": LABEL})
            return
        route = self.ROUTES.get(model)
        if route is None:
            self._send(
                400,
                {
                    "error": "unsupported_model_or_route",
                    "requested": model,
                    "label": LABEL,
                },
            )
            return
        self._send(200, route)


class Forward(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"
    origin_port = 0

    def log_message(self, *_args):
        return

    def do_POST(self):
        n = int(self.headers.get("Content-Length") or 0)
        body = self.rfile.read(n)
        upstream = HTTPConnection(BIND, self.origin_port, timeout=TIMEOUT)
        try:
            skip = {"host", "proxy-connection"}
            headers = {k: v for k, v in self.headers.items() if k.lower() not in skip}
            upstream.request("POST", self.path, body=body, headers=headers)
            resp = upstream.getresponse()
            data = resp.read()
            status = resp.status
            ctype = resp.getheader("Content-Type") or "application/json"
        finally:
            upstream.close()
        self.send_response(status)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        self.send_header("Connection", "close")
        self.end_headers()
        self.wfile.write(data)


def start(handler):
    httpd = ThreadingHTTPServer((BIND, 0), handler)
    httpd.daemon_threads = False
    httpd.timeout = 0.3
    thread = threading.Thread(
        target=httpd.serve_forever,
        name="srv-%s" % httpd.server_address[1],
        daemon=False,
    )
    thread.start()
    return httpd, thread


def post(port, path, payload, headers):
    conn = HTTPConnection(BIND, port, timeout=TIMEOUT)
    try:
        raw = json.dumps(payload).encode()
        hdrs = {"Content-Type": "application/json"}
        hdrs.update(headers)
        conn.request("POST", path, body=raw, headers=hdrs)
        resp = conn.getresponse()
        return resp.status, json.loads(resp.read().decode())
    finally:
        conn.close()


def main():
    origin_httpd = gw_httpd = fwd_httpd = None
    threads = []
    try:
        origin_httpd, origin_thread = start(Origin)
        threads.append(origin_thread)
        Forward.origin_port = origin_httpd.server_address[1]
        fwd_httpd, fwd_thread = start(Forward)
        threads.append(fwd_thread)
        gw_httpd, gw_thread = start(Gateway)
        threads.append(gw_thread)

        ping = {
            "model": "ignored-by-forward",
            "messages": [{"role": "user", "content": "ping"}],
        }
        rows = []
        status, body = post(
            fwd_httpd.server_address[1],
            "/v1/chat/completions",
            ping,
            {"Authorization": "***"},
        )
        rows.append(("forward_http", status, body.get("origin") or body.get("error")))
        assert status == 200
        assert body.get("label") == LABEL
        assert body.get("origin") == "loopback-http-origin"

        gw = gw_httpd.server_address[1]
        status, body = post(
            gw,
            "/v1/chat/completions",
            {"model": "fixture-alpha", "messages": []},
            {"Authorization": "Bearer " + KEY},
        )
        rows.append(("gateway_route_alpha", status, body.get("route") or body.get("error")))
        assert status == 200 and body.get("route") == "alpha"

        status, body = post(
            gw,
            "/v1/chat/completions",
            {"model": "fixture-bravo", "messages": []},
            {"Authorization": "Bearer " + KEY},
        )
        rows.append(("gateway_route_bravo", status, body.get("route") or body.get("error")))
        assert status == 200 and body.get("route") == "bravo"

        status, body = post(
            gw,
            "/v1/chat/completions",
            {"model": "fixture-alpha", "messages": []},
            {},
        )
        rows.append(("gateway_auth_missing", status, body.get("error")))
        assert status == 401

        status, body = post(
            gw,
            "/v1/chat/completions",
            {"model": "fixture-alpha", "messages": []},
            {"Authorization": "***"},
        )
        rows.append(("gateway_auth_invalid", status, body.get("error")))
        assert status == 401

        status, body = post(
            gw,
            "/v1/chat/completions",
            {"model": "no-such-model", "messages": []},
            {"Authorization": "Bearer " + KEY},
        )
        rows.append(("gateway_unsupported_model", status, body.get("error")))
        assert status == 400

        print("CONTROLLED_FIXTURE public recipe — not a provider call, not TLS")
        for name, status, extra in rows:
            print("%s HTTP %s %s" % (name, status, extra))
        print("assertions=6 ok")
        return 0
    finally:
        for httpd in (origin_httpd, gw_httpd, fwd_httpd):
            if httpd is not None:
                httpd.shutdown()
                httpd.server_close()
        for thread in threads:
            thread.join(timeout=TIMEOUT)


if __name__ == "__main__":
    raise SystemExit(main())

Expected output:

CONTROLLED_FIXTURE public recipe — not a provider call, not TLS
forward_http HTTP 200 loopback-http-origin
gateway_route_alpha HTTP 200 alpha
gateway_route_bravo HTTP 200 bravo
gateway_auth_missing HTTP 401 invalid_local_gateway_auth
gateway_auth_invalid HTTP 401 invalid_local_gateway_auth
gateway_unsupported_model HTTP 400 unsupported_model_or_route
assertions=6 ok

The forward_http line is that fixed-upstream pass-through. The CONNECT+TLS lab in the table and WebP remains the forwarding proof.