Lucebox Hub: Hand-Tuned LLM Inference Optimization for Consumer Hardware

Summary

Lucebox Hub is an open-source optimization project focused on hand-tuning LLM inference for specific consumer hardware. The project rewrites LLM software from scratch for individual chips rather than waiting for better hardware. It includes two main projects: a megakernel for Qwen3.5 0.8B on RTX 3090 that achieves 1.87 tokens per joule, matching Apple's latest silicon performance on 2020 hardware, and a speculative decoding implementation for Llama 3.1 8B on RTX 4090 that achieves 2.5 tokens per joule. The approach involves custom kernels, speculative decoding, and quantization tailored per target hardware, with the philosophy that software optimization can extract maximum performance from existing consumer hardware.