llama.cpp-python Bindings

llama.cpp is the C/C++ inference engine created by Georgi Gerganov for running large language models efficiently on consumer hardware, while llama-cpp-python is a Python package that wraps that C++ core for programmatic use. The Python bindings allow developers to load models, generate text, and manage tokenization directly within Python scripts without subprocess overhead.

Yes, llama-cpp-python natively supports GGUF-format quantized models, which are the standard format produced by llama.cpp's quantization pipeline. Many popular models on Hugging Face have community-uploaded GGUF versions available, and you can load them by passing a file path or Hugging Face repo ID to the Llama constructor.

Yes, llama-cpp-python supports GPU offloading through CUDA (NVIDIA), Metal (Apple Silicon), and other backends supported by the underlying llama.cpp engine. You enable this at build time using environment variables like CMAKE_ARGS for CUDA, and control layer offloading at runtime via the n_gpu_layers parameter.

A basic CPU-only install is done via pip (pip install llama-cpp-python), but GPU-enabled builds require setting environment variables before installation, such as CMAKE_ARGS="-DGGML_CUDA=on" for NVIDIA GPUs. Pre-built wheels with GPU support are not always available for every platform and CUDA version, so compiling from source is often necessary.