Researchers Serve 229B-Parameter MoE Model Across Five Consumer GPUs Over Public Internet
By
leyten
Summary
This article describes a technical demonstration where a 229-billion parameter Mixture-of-Experts AI model (MiniMax-M2.5) is served across five consumer-grade RTX 5090 GPUs located in five different European countries, connected only via the public internet. The system achieves interactive-speed inference (10-13 tokens/s for reasoning, 70-87 tokens/s for draftable text) and provides cryptographic receipts to verify each stage's computation. The authors derive a law for speculative decoding over high-latency links and note that on consumer hardware, performance is CPU-bound rather than GPU-bound.
Source
Twitter / XResearchers Serve 229B-Parameter MoE Model Across Five Consumer GPUs Over Public Internetdoi.orgKey quotes
· 4 pulledSingle-stream decoding reaches 10-13 tokens/s on interactive reasoning and 70-87 tokens/s on draftable text.
Every request returns cryptographic receipts proving each stage did its work, at a measured cost of 0.4% of stage compute.
We derive and validate a simple law for speculative decoding over high-latency links: pipelined speculation collapses below a per-token acceptance of about 0.8, which no drafter reaches on novel text.
We also show that stage time on consumer fleets is bounded by the host CPU, not the GPU.
You might also wanna read
Rotary GPU: Enabling Large Mixture-of-Experts Models on Consumer Laptop GPUs with Limited Memory
This paper presents Rotary GPU, an exploratory approach to running large Mixture-of-Experts (MoE) language models on consumer-grade hardware
arxiv.org·1mo ago
Building high-performance expert-parallel dispatch and combine kernels for MoE LLM inference
This article provides a deep technical deep-dive into the architecture and implementation of high-performance Expert Parallelism (EP) kernel
fergusfinn.com·27d ago
Breakthrough: 1.3-Second Cross-Machine Weight Transfer for Trillion-Parameter AI Models
Researchers have achieved ultra-fast 1.3-second cross-machine parameter updates for trillion-parameter AI models (Kimi-K2 with 1T parameters
research.perplexity.ai·5mo ago
Kog AI Launches Inference Engine Tech Preview: 3,000 Tokens/s on AMD MI300X GPUs
Kog AI launches a tech preview of the Kog Inference Engine (KIE), achieving 3,000 output tokens/s per request on 8× AMD MI300X GPUs and 2,10
blog.kog.ai·1mo agoGPEmu: A GPU Emulator for Rapid, Low-Cost Deep Learning Prototyping [pdf]
General Compute Launches ASIC-Based Inference Cloud for Faster AI Agent Performance
General Compute is an inference cloud built on ASICs (purpose-built alternatives to Nvidia GPUs) designed specifically for AI inference, not
Product Hunt·2mo ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.