LocalAI

Replacement-style API, with local/on-device inference for consumer-grade hardware, supporting multiple model families and backends compatible with standard formats like GGGUF. In short: A local OpenAI drop-in replacement REST API. You own your data. No GPU required for local/on-device inference (offline). Optional GPU/NPU acceleration available for Llama.cpp-compatible LLMs. Also see the "Build" section. Model inference pipeline/backend agnostic! (Install inference backends via Gallery WebUI or CLI) Supported task types: Text generation (with Llama.cpp, Transformers, vLLM, ExLlama2, GPT4All.cpp, and more) Text-to-audio: Sound/music generation (Transformers - MusicGen) Speech generation (Whisper, Bark, Piper, Bark.cpp) Speech-to-text (i.e., transcription, Whisper.cpp, etc.) Image generation using Diffusers/Stable-Diffusion.cpp (text-to-image, image-to-image, etc.) Text embeddings (with Sentence-Transformers, Transformers) Text reranking (Rerankers, SentenceTransformers) After the first load, models are cached in memory for faster inference. Distributed inference (federated and P2P modes) Additional notes: Performance/throughput may vary depending on the chosen inference pipeline. For faster inference and better performance, consider using C/C++-based pipelines like Llama.cpp. Refer to the LocalAI docs for the latest updates.

LocalAI is an open-source alternative compatible with the OpenAI API, enabling users to run large language models, image and audio generation, and other AI models on local hardware, achieving data privacy and cloud-independent AI inference.