llamafile

Llamafile allows you to distribute and run LLMs with a single file, offering an OpenAI-compatible API and the KoboldAI API. Our goal is to enable AI developers to "build anywhere, run anywhere." We achieve this by combining Llama.cpp with Cosmopolitan Libc into a framework that lets you build applications for LLMs as single-file artifacts, which run natively on most PCs and servers. First, your Llamafiles can run across multiple CPU microarchitectures. We’ve added runtime dispatching to Llama.cpp, allowing newer Intel systems to utilize modern CPU features without sacrificing compatibility with older computers. Second, your Llamafiles can operate on multiple CPU architectures. We accomplish this by concatenating builds for AMD64 and ARM64, launching the appropriate script. Our file format is compatible with Win32 and most Unix shells. When needed, it can also be easily converted (by you or your users) into a platform-native format. Third, your Llamafiles can run on six operating systems: macOS, Windows, Linux, FreeBSD, OpenBSD, and NetBSD. You only need to build your code once using a Linux-style toolchain. The GCC-based compiler we provide is itself a portable executable, so you can build software for all six OSes from the comfort of your preferred development environment. Finally, LLM weights can be embedded within your Llamafile. We’ve added PKZIP support to the GGML library, enabling uncompressed weights to be memory-mapped directly, similar to self-extracting archives. This allows quantized weights distributed online to be prefixed with a compatible version of the Llama.cpp software, ensuring their originally observed behavior can be replicated indefinitely.

Llamafile helps AI developers distribute and run large language models (LLMs) as single files, providing OpenAI and KoboldAI-compatible APIs to achieve the 'build once, run anywhere' goal for AI applications.