Local inference on a passthrough GPU, OpenAI-compatible

What this builds

One GPU instance serving an open-weights model over an HTTP API that speaks the OpenAI-compatible schema, so every client library you already have works against it with one changed base URL. Behind nginx, with TLS, an API key, and a systemd unit that survives reboots.

Written on a G-L40S: forty-eight gigabytes of VRAM on a single card, passed through to the instance over PCIe rather than time-sliced between tenants. That distinction is the difference between predictable latency and waiting behind somebody else’s training job. The card is yours until you cancel.

Before you start

• A G-L40S with Debian 13. The G-ADA at twenty gigabytes works for smaller models; the arithmetic in step four tells you which ones.
• Two terabytes of NVMe on this plan, which is the right place for weights. Downloading forty gigabytes twice because the cache was on the root filesystem is an annoying way to spend an hour.
• A hostname pointing at the instance, llm.example.com below.

1. Driver

sed -i "s/ main$/ main contrib non-free-firmware non-free/" /etc/apt/sources.list.d/debian.sources
apt update
apt install -y nvidia-driver firmware-misc-nonfree build-essential
reboot

After it comes back:

nvidia-smi

You want the card listed, the driver version printed, and memory usage near zero. A missing card at this point is almost always a stale initramfs; update-initramfs -u and reboot once more before opening a ticket.

nvidia-smi -q -d PERFORMANCE | grep -A3 "Clocks Event Reasons"
nvidia-smi -pm 1

Persistence mode keeps the driver loaded between processes, which removes several seconds of initialisation from every restart.

2. Runtime

apt install -y python3-venv python3-pip
install -d -o root -g root /srv/models /opt/vllm
python3 -m venv /opt/vllm/venv
/opt/vllm/venv/bin/pip install --upgrade pip
/opt/vllm/venv/bin/pip install vllm

The install pulls a large CUDA-linked wheel set and takes a while. Meanwhile, decide on a model.

3. Choosing what fits

Weights are the floor, not the budget. A model at sixteen-bit precision needs roughly two gigabytes of VRAM per billion parameters, plus the key-value cache for every concurrent request, plus a gigabyte or so of runtime overhead.

The cache column is what determines how many people can use the endpoint at once. Fill the card to the brim with weights and you have built a very fast single-user toy.

4. Serving it

Point the cache at the NVMe and start the server. Substitute the repository identifier of whichever open-weights model you chose; the runtime fetches it on first start.

cat > /etc/systemd/system/vllm.service <<EOF
[Unit]
Description=vLLM inference server
After=network-online.target

[Service]
Environment=HF_HOME=/srv/models
Environment=VLLM_API_KEY=<a long random string>
ExecStart=/opt/vllm/venv/bin/vllm serve <org>/<model> \\
  --host 127.0.0.1 --port 8000 \\
  --served-model-name local \\
  --max-model-len 16384 \\
  --gpu-memory-utilization 0.92 \\
  --max-num-seqs 32
Restart=on-failure
RestartSec=10
TimeoutStartSec=1800

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && systemctl enable --now vllm
journalctl -fu vllm

Binding to loopback is deliberate. The API key check in the runtime is a single shared secret with no rate limiting behind it, so nginx does the exposure and the runtime never touches the public interface.

The generous start timeout exists because the first launch downloads tens of gigabytes and then compiles kernels. Subsequent starts take under a minute.

5. Proxy and TLS

server {
  listen 443 ssl;
  listen [::]:443 ssl;
  http2 on;
  server_name llm.example.com;

  ssl_certificate     /etc/letsencrypt/live/llm.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/llm.example.com/privkey.pem;

  location /v1/ {
    proxy_pass http://127.0.0.1:8000;
    proxy_buffering off;
    proxy_read_timeout 600s;
    proxy_set_header Host $host;
  }
}

proxy_buffering off is the line that makes streaming responses stream. Leave it on and tokens arrive in polite batches whenever nginx feels a buffer is full, which looks exactly like a slow model and is not.

Verify it

Start with the card:

nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv

Used memory should be around ninety percent of total, because that is what --gpu-memory-utilization asked for. Then the endpoint:

curl -s https://llm.example.com/v1/models -H "Authorization: Bearer <the key>" | head -20

One model, named local. Now a real completion, and time it:

time curl -s https://llm.example.com/v1/chat/completions \
  -H "Authorization: Bearer <the key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"local","messages":[{"role":"user","content":"List three prime numbers."}],"max_tokens":64}'

Finally, measure it under load rather than one request at a time, which is the only number worth quoting:

/opt/vllm/venv/bin/vllm bench serve \
  --backend openai-chat \
  --base-url https://llm.example.com \
  --endpoint /v1/chat/completions \
  --model local --num-prompts 200 --request-rate 8

Read three numbers from the output: output tokens per second in aggregate, median time to first token, and the ninety-ninth percentile end to end. On a single L40S with a model in the ten-to-twenty-billion range, aggregate output throughput in the high hundreds of tokens per second and a time to first token well under a second is the expected shape. Time to first token that climbs with concurrency means your key-value cache is too small, so lower --max-model-len or --max-num-seqs and run it again.

Afterwards

Watch temperature and clock throttling for the first week with nvidia-smi dmon, since a card that thermally throttles under sustained load produces exactly the sort of intermittent slowness that gets blamed on the network. Bill-wise, remember that a GPU instance costs the same whether the card is busy or idle, so batch work rather than leaving it warm. Pricing is on the GPU page.