Do You Need a GPU to Run an AI Agent? An Honest Answer

No, most AI agents do not need a GPU. If your agent calls a hosted model through an API, the heavy computation happens on the provider's hardware, and your server only needs enough CPU and RAM to run the agent code, its database and its tools. You need a GPU when you run the model yourself at a useful speed, fine-tune models, or generate images, video or audio at scale.

Why API-based agents run fine on CPU

Look at what an agent does on each request: receive a message, load the conversation history from a database, build a prompt, send it to a model API, wait, parse the JSON response, perhaps call a tool such as an HTTP request or a SQL query, then send the reply. Almost all of that is network I/O and light text processing. The token generation, the part that benefits from a GPU, runs in the model provider's data center.

In practice, an API-based agent's server spends most of its time waiting. CPU is used in short bursts, and RAM goes to the runtime (Node.js or Python), the database and any caches. A faster GPU on your side would not make the model API answer any sooner.

When you do need a GPU

• Self-hosting an LLM for production use. Running an open-weight model with acceptable response times, or serving many users at once, generally requires a GPU with enough memory to hold the model.

• Fine-tuning or training. Even parameter-efficient fine-tuning is impractically slow on CPU for anything beyond very small models.

• Large-scale embedding jobs. Embedding a few thousand documents on CPU is fine; re-embedding millions of chunks on a deadline is a GPU job.

• Image, video and voice generation. Diffusion and video models are GPU workloads. Speech-to-text runs on CPU for occasional short clips but becomes slow at volume.

• Rules that forbid sending text to an external API. If no data may leave your infrastructure, you have to run the model yourself, and a capable model at a usable speed means GPUs.

Small local models on CPU: realistic expectations

Tools such as Ollama and llama.cpp make it easy to run quantized open models on a CPU-only server. Quantization stores the weights in fewer bits, for example 4-bit instead of 16-bit, which cuts memory needs to a fraction at some cost in quality. As a rough guide, a 7 to 8 billion parameter model quantized to 4 bits needs somewhere around 4 to 6 GB of RAM including working space for the context, and larger models scale up from there.

What to expect:

• Speed: on CPU, generation is limited mainly by memory bandwidth. Replies are noticeably slower than a hosted API, and speed varies widely with the CPU, RAM speed, model size and quantization level.

• Concurrency: a CPU server handles very few simultaneous generations before every request starts queuing.

• Quality: small models are good at classification, extraction, short summaries and simple routing. They are weaker at multi-step reasoning and reliable tool calling, which is what agents depend on.

• Resource contention: inference will happily use every core, so it can slow down a database or web app on the same server. Limit its threads or give it its own server.

CPU inference is reasonable for offline batch tasks, internal tools with a few users, privacy-sensitive preprocessing and experiments. It is rarely the right engine for a customer-facing chat agent.

The hybrid pattern: API model plus local embeddings

A common and sensible setup uses a hosted API for the main reasoning model and a small embedding model running locally on CPU for retrieval:

1. Documents are split into chunks and embedded on your server with a compact embedding model.

2. The vectors are stored in PostgreSQL with pgvector, next to the rest of your data.

3. At query time the user's question is embedded locally, the closest chunks are retrieved, and only those chunks are sent to the API model with the question.

This keeps your document store under your control and avoids per-call embedding fees during bulk ingestion. Embedding models are small, so CPU throughput is usually acceptable for moderate document volumes. Two rules: use the same embedding model for ingestion and queries, and match the vector column's dimensions to that model. The pgvector RAG guide shows the schema and queries.

# A local embedding model through Ollama on a CPU-only server
ollama pull nomic-embed-text
curl http://localhost:11434/api/embed \
  -d '{"model": "nomic-embed-text", "input": "What is your refund policy?"}'

Keep the Ollama API bound to localhost, as it is by default, or behind authentication. It should not be reachable from the internet.

Sizing CPU and RAM for an API-based agent

Size for what runs around the model, not for the model itself. These are typical starting points, not hard rules:

• A single bot or simple agent (a Telegram, WhatsApp or Discord bot calling an API with light memory): 1 vCPU and 2 GB RAM is usually enough.

• Agent plus PostgreSQL and pgvector with a few thousand documents: around 2 vCPU and 4 GB gives the database room for caching and indexes.

• Several agents, workers and queues, or n8n running alongside your agents: 4 vCPU and 8 GB or more.

• Headless browsers (Playwright or Puppeteer for scraping or browsing agents): each browser instance can use hundreds of megabytes, so plan RAM per concurrent browser.

• Local embeddings or a small local model: add RAM for the model and expect it to keep several cores busy while it works.

Measure rather than guess. Run the agent under realistic load and watch htop, free -h and docker stats. If memory stays comfortably below the limit and the CPU is idle between requests, you are sized correctly. There is more detail on storage and networking in AI application server requirements.

Where this leaves your hosting decision

If your agent calls a model API, choose an always-on CPU server with enough RAM and fast NVMe storage, in a server location close to your users and the services you call. Our comparison of where to host an AI agent weighs VPS, serverless and managed options. If you genuinely need to self-host a large model, budget for GPU infrastructure from a GPU specialist, and keep the rest of the stack (agent logic, database, queues, webhooks) on regular CPU servers, which cost far less to run around the clock.

Frequently asked questions

Can I run an open-source LLM on a VPS without a GPU?

Yes. Small quantized models run on CPU through Ollama or llama.cpp. Expect slower answers and few simultaneous users, so it suits internal or batch work better than live customer chat.

Does an AI agent need a lot of RAM?

The agent process itself usually needs little. RAM grows with what runs beside it: a database, headless browsers, local embedding models or a local LLM.

Do embeddings need a GPU?

Not for moderate volumes. Compact embedding models run acceptably on CPU. A GPU helps when you embed very large collections in a short time.

Will a GPU make my API-based agent faster?

No. Response time is dominated by the model API and the network. To speed things up, use a faster model, shorter prompts and a shorter memory window, stream responses, and host the agent close to your users.

Running API-based agents on IM HOST

IM HOST offers CPU servers, not GPU servers, which suits agents that call model APIs. AI Agent Hosting plans range from 1 vCPU/2 GB to 12 vCPU/32 GB, with Docker, Node.js, Python and PostgreSQL with pgvector pre-installed and free daily backups. For CPU-heavy side jobs such as local embeddings, Linux VPS plans include AMD EPYC 7313P and AMD Ryzen 9 9950X options in Poland and Germany.