Optimizing LLM Inference by Combining NVIDIA DGX Spark and Apple Mac Studio Architectures

Summary

The article explores combining NVIDIA DGX Spark AI supercomputers with Apple Mac Studio systems to optimize large language model (LLM) inference performance. The author received early access to DGX Spark units (100 TFLOPs FP16, 128GB memory) and has been running LLMs on Mac Studio clusters (26 TFLOPs FP16, 512GB memory). The key insight is disaggregating the prefill and decode phases of LLM inference - using DGX Spark for compute-intensive prefill and Mac Studio for memory-bandwidth-intensive decode, achieving 4x faster inference with their EXO 1.0 system.