Running a large language model (LLM) on a GPU VPS: getting started with Ollama

Running large language models (LLMs) on your own server means your data doesn’t go to an outside service, you don’t pay per use and you can tune the model to your needs. A GPU speeds this up many times over a CPU. This guide covers the basic steps to run a local model on a server with a GPU using Ollama.

What is a GPU VPS good for?

• Inference: question answering, summarising and code help with a ready-made model.
• Privacy: documents and prompts never leave your server.
• Other GPU work: image generation, video processing, scientific computing and rendering.

Memory (VRAM) planning

What most determines a model’s speed, and whether it runs at all, is GPU memory (VRAM). As a rule of thumb it depends on the parameter count and quantisation: a small quantised 7–8 billion-parameter model runs in a few GB of VRAM, while much larger models want tens of GB. Memory needs grow with the model, the context length and the number of concurrent users. You can see the graphics card and VRAM in our GPU plans on the product page.

1. Check the driver

nvidia-smi

If it lists your GPU and the driver version, you are ready. If the command isn’t found, install the recommended driver on Ubuntu:

sudo ubuntu-drivers install
sudo reboot

2. Install Ollama

curl -fsSL https://ollama.com/install.sh | sh
systemctl status ollama

Security note: running a script downloaded from the internet directly is risky. You can first download it with curl -fsSL https://ollama.com/install.sh -o install.sh and read it.

3. Run your first model

ollama run llama3.2

The model is downloaded on first run and a chat opens. To list models use ollama list, and to watch GPU use run watch -n 1 nvidia-smi in another terminal; you will see VRAM use rise while the model runs.

4. Using the API

By default Ollama listens only on 127.0.0.1:11434:

curl http://127.0.0.1:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "What is a cloud server? Explain in two sentences.",
  "stream": false
}'

5. Safe access

Ollama’s API has no authentication; don’t expose the port to the internet. To use it from your own computer an SSH tunnel is enough:

ssh -L 11434:127.0.0.1:11434 user@SERVER_IP

Then use http://localhost:11434 as the address in your application. If you will share it with several people, run it behind WireGuard or add a reverse proxy that authenticates (see Caddy).

Running with Docker (optional)

If you use Docker, GPU access needs the NVIDIA Container Toolkit; after installing it:

docker run -d --gpus=all -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama/ollama

Performance and maintenance tips

• If the whole model doesn’t fit in VRAM, part of it spills to the CPU and slows down; choose a smaller or more heavily quantised model.
• Delete unused models with ollama rm MODEL; models take disk space.
• Harden the server with the security checklist.
• Take a snapshot before risky driver or system updates.

Start with Melinhost

Explore our GPU server plans and pick the memory and performance you need. For questions, our support team is with you.