Français Login Register

How to Run a Local LLM: A Comparison of Ollama, llama.cpp, LM Studio, and vLLM

DaDesktop Team ·

Executing a large language model (LLM) locally can be achieved through various methodologies. Certain tools prioritize ease of onboarding, while others offer deeper configurability or are optimized for high-concurrency environments. The ideal solution hinges on your specific requirements, whether that involves a straightforward local chat interface, a highly configurable inference engine, or a robust production-grade API.

Ollama

Ollama stands out as one of the most accessible entry points for local model deployment. The process involves installation, model acquisition, and execution via the command line. Additionally, it offers a local API to facilitate integration with applications and third-party tools.

Advantages:

  • Streamlined installation and model administration
  • User-friendly command-line operations
  • OpenAI-compatible API interface
  • Compatibility with NVIDIA, AMD, Apple Silicon, and Vulkan-based GPU acceleration
  • Modelfiles enable customisation of model behaviour and parameters
  • Capability to handle concurrent requests, provided sufficient memory resources are available

Disadvantages:

  • Limited low-level control compared to llama.cpp
  • Model management is inherently tied to the Ollama ecosystem
  • Not the preferred option for scenarios demanding maximum serving throughput or distributed inference architectures

Complexity: Low. Ideal for users seeking to deploy a model rapidly without navigating complex inference configurations.

llama.cpp

llama.cpp is a lightweight C/C++ inference engine engineered for efficient model execution across diverse hardware platforms. Utilizing GGUF models, it provides granular control over model loading and execution dynamics.

Advantages:

  • Granular control over context, GPU offloading, batching, threading, quantization, and other inference parameters
  • Extensive hardware compatibility, covering CUDA, HIP, Metal, Vulkan, and SYCL
  • Support for a wide range of quantization levels, from low-bit formats up to 8-bit
  • Ability to distribute model layers across multiple GPUs
  • Capacity to utilize both CPU and GPU resources when models exceed available VRAM
  • Incorporates llama-server to deliver an OpenAI-compatible API

Disadvantages:

  • Requires more extensive configuration than Ollama or LM Studio
  • GGUF models typically need to be downloaded and managed externally
  • Optimizing performance often requires a deep understanding of inference parameters

Complexity: Medium. Well-suited for those who desire precise control over model execution or wish to experiment with performance metrics and quantization techniques.

LM Studio

LM Studio is a desktop application designed for downloading, configuring, and executing local LLMs. It features a graphical interface that simplifies model discovery and the management of settings such as GPU offloading and context size.

Advantages:

  • Intuitive graphical user interface
  • Integrated search and download functionality via Hugging Face
  • Pre-loading display of model and resource details
  • Built-in OpenAI-compatible API server
  • Support for headless operation via its llmster server
  • Compatibility with GGUF models through llama.cpp and MLX models on Apple Silicon

Disadvantages:

  • Limited low-level control relative to direct llama.cpp usage
  • Desktop application nature may be less suitable for certain server-side deployments
  • Not primarily engineered for large-scale, multi-user serving

Complexity: Low. A strong choice for experimenting with local models while minimizing time spent in the command line.

vLLM

vLLM is purpose-built for serving LLMs to applications and multiple users simultaneously. Its primary strength lies in efficient high-concurrency serving, leveraging techniques such as PagedAttention, continuous batching, prefix caching, and distributed inference.

Advantages:

  • High throughput capability for numerous concurrent requests
  • Efficient KV-cache management and continuous batching
  • OpenAI-compatible API server
  • Direct compatibility with a wide array of Hugging Face models
  • Support for diverse quantization methods, including FP8, INT4, GPTQ, AWQ, GGUF, and others
  • Support for tensor, pipeline, expert, and other forms of parallelism
  • Architected for production-grade inference and serving

Disadvantages:

  • More complex setup and configuration requirements
  • Primarily targeted at Linux environments
  • Generally excessive for single-user, interactive model execution
  • Hardware and model compatibility must be verified prior to deployment

Complexity: High. Best suited for professionals deploying inference services rather than those simply running models on personal computers.

Which one should you pick?

  • Seeking easy model execution: Ollama or LM Studio. Select Ollama for command-line simplicity and API access, or LM Studio for a graphical interface.
  • Desiring inference control: llama.cpp. It offers direct control over model loading, quantization, context, GPU offloading, and additional settings.
  • Requiring a local API: Ollama, llama.cpp, or LM Studio. All three options provide OpenAI-compatible APIs.
  • Serving multiple users: vLLM. Its continuous batching and distributed inference features are specifically designed for this use case.
  • Experimenting with quantizations: llama.cpp or LM Studio.

Run it on DaDesktop

If local GPU hardware is insufficient, you can execute these tools on a DaDesktop cloud desktop. Select a GPU with adequate VRAM for your chosen model, launch the desktop environment, and install your preferred inference software.

Ollama and LM Studio are advantageous when a simple local environment is desired. llama.cpp provides greater control over hardware and inference settings. vLLM is the preferred option when exposing a model as a high-throughput API is necessary.

View available GPUs to compare VRAM and other specifications.