zoom_in Click to enlarge

llama.cpp (CLI for Inference)

▲ ▼

language

description llama.cpp (CLI for Inference) Overview

This refers to the core, raw command-line interface of llama.cpp, used when maximum control over inference parameters is needed. It bypasses all GUI wrappers, giving the user direct access to the underlying C++ performance optimizations. While intimidating for casual users, it offers the absolute highest degree of control over quantization, context management, and hardware utilization for pure performance testing.

help llama.cpp (CLI for Inference) FAQ

What command actually starts a model in llama.cpp?

The interactive command-line program is named llama-cli in current builds. A basic invocation supplies a GGUF model with the -m option, followed by a prompt or conversation mode.

Which model format does llama.cpp use?

Modern llama.cpp builds use GGUF model files. Quantized GGUF variants reduce memory use, allowing models to run on hardware that could not hold the original full-precision weights.

Can llama.cpp use an NVIDIA or Apple GPU?

Yes, the project supports CUDA for NVIDIA hardware and Metal on Apple Silicon, along with backends such as Vulkan. GPU layers can be offloaded at runtime, but the available options depend on how the binary was compiled.

How is llama-cli different from llama-server?

llama-cli runs prompts directly in a terminal and exposes detailed sampling and context controls. llama-server instead provides an HTTP service that applications and interfaces can call.

swap_horiz

Looking for llama.cpp (CLI for Inference) alternatives? Compare top competitors ranked & scored

arrow_forward