Best LLM Runner
LM Studio is a desktop application designed to run large language models locally on your computer. It’s notable for its streamlined workflow allowing users to easily download, manage, and execute various LLMs without an internet connection. Primarily aimed at developers and technically-minded indivi...
LocalAI is a powerful and versatile local LLM runner built around the idea of seamless model management. It excels in its intuitive interface, offering granular control over model parameters like temperature and top_p. It boasts excellent support for various quantization methods (including GPTQ) an...
The Ollama Web UI offers an interactive web interface to run and experiment with local large language models. It’s notable for its ease of use and direct integration with the Ollama LLM runner. Developers and users interested in exploring locally hosted AI chatbots, particularly those seeking a Chat...
LlamaFile is a compact software package designed to run large language models locally. It presents a simple, interactive interface within a single executable file across various platforms. This makes it ideal for beginners and those seeking an easy way to experiment with LLMs without complex setup o...
ExLlamaV2 is a specialized machine learning engine designed to accelerate the processing of Large Language Models like LLaMA. It’s notable for its speed and efficiency, particularly when utilizing GPU hardware. The project emphasizes local, offline inference and supports quantization techniques. ExL...
Koboldcpp is a minimalist C++ application designed for interactive fiction and roleplaying experiences. It provides an offline LLM runner based on llama.cpp, offering a streamlined interface suitable for single-file projects. This tool is particularly useful for writers, game developers, and individ...
The Candle project offers a lightweight software solution built in Rust designed to execute Large Language Models (LLMs). It’s notable for its minimalist design and suitability for resource-constrained environments like embedded systems or edge computing. Candle provides an LLM runner optimized for...
TabbyAPI is an open-source Python application that provides a locally hosted API server compatible with OpenAI’s interface. It facilitates development using LLMs from LM Studio, enabling offline experimentation and integration with other applications. This tool is particularly useful for developers...
The Aphrodite Engine is a machine-learning tool designed for local, offline deep learning experimentation. It’s notable for its support of tensor parallelism and PagedAttention, enabling the execution of large language models on consumer GPUs. Researchers and developers working with advanced AI mode...
A desktop chat interface for local LLMs with a focus on simplicity. Can be used alongside LM Studio for a different UI experience.
The RWKV Runner is a software tool designed to efficiently run RWKV large language models offline. It facilitates inference using both CPU and GPU hardware, providing a cross-platform solution for users interested in experimenting with and deploying these models. This runner is particularly useful f...
