llama.cpp

Description of llama.cpp

llama.cpp is an open-source project for inference of language and vision-language models, written in plain C/C++ with no external dependencies. It is built on the ggml library and hosted under the ggml-org organization on GitHub. The licence is MIT, and there is no company pricing page: the software is free to download, build and use.

The project supports a wide range of hardware. On Apple silicon it uses Metal, Accelerate and ARM NEON; on x86 it uses AVX, AVX2, AVX512 and AMX; and RISC-V is supported. For graphics cards it lists CUDA for NVIDIA, HIP for AMD, MUSA for Moore Threads, Vulkan and SYCL for Intel, along with other backends such as BLAS, CANN for Ascend NPUs, Hexagon, OpenCL and WebGPU. Quantization from 1.5-bit to 8-bit integer formats lets larger models fit in less memory.

For everyday use, the repository shows a command that starts llama-server, an OpenAI-compatible API server with a built-in web interface, and links a REST API reference. Other desktop apps and runners are built on this engine, which is why it often appears underneath tools that look simpler.

It is aimed at technical users. You work from the command line, choose and download model files yourself, and tune settings for your hardware. A buyer who wants a ready chat window with model search and one-click downloads will usually find a packaged app easier. In return, llama.cpp offers direct control, runs fully offline once a model is downloaded, and gives you full access to the source code.

Features of llama.cpp

1. Runs language and vision models on CPUs and on NVIDIA, AMD, Intel, Apple and other accelerators through many backends.
2. Quantization from 1.5-bit to 8-bit integer formats helps large models fit in limited memory.
3. llama-server provides an OpenAI-compatible API and a built-in web UI for local use.

Frequently asked questions

Is llama.cpp free?

Yes. It is open source under the MIT licence and hosted by ggml-org on GitHub. There is no paid plan or account; your cost is the computer you run it on.

Who should use llama.cpp directly?

People comfortable with the command line who want control over model files and hardware settings. Others may prefer a packaged app that is built on the engine.

Does llama.cpp have an API?

Yes. The README shows a command that launches an OpenAI-compatible API server, llama-server, and links a REST API reference. It also includes a built-in web interface.

Website: https://github.com/ggml-org/llama.cpp

Company: ggml-org (open-source project)