Mesh-LLM: Distributed LLM Inference System Using llama.cpp Across Multiple Machines

Summary

Mesh-LLM is a reference implementation that enables distributed inference of large language models across multiple machines by compiling llama.cpp for distributed execution. The system allows pooling spare GPU capacity to run LLMs at larger scale, automatically distributing models that don't fit on a single machine using pipeline parallelism for dense models and expert sharding for MoE models with zero cross-node inference traffic. It features a mesh network architecture where agents can gossip and share information without a central server, and includes installation instructions for macOS and Linux with requirements for building from source.