ntransformer: C++/CUDA LLM Inference Engine Enables Running Llama 70B on RTX 3090

Summary

ntransformer is a high-efficiency C++/CUDA LLM inference engine that enables running large language models like Llama 70B on consumer-grade hardware like the RTX 3090 (24GB VRAM). The engine uses innovative memory management techniques including streaming model layers through GPU memory via PCIe and optional NVMe direct I/O that bypasses the CPU entirely. Performance results show the system can run Llama 3.1 8B Q8_0 models at 48.9 tokens/second with all layers resident in VRAM, and can handle much larger 70B models through tiered memory management that combines VRAM, RAM, and NVMe storage.