nanoeuler: A GPT-2-style LLM built from scratch in C/CUDA with hand-written backprop and FlashAttention
By
vforno
Summary
A developer has built a GPT-2-class language model entirely from scratch in C/CUDA without using any machine learning libraries like PyTorch or autograd. The project includes hand-written forward and backward passes, a byte-level BPE tokenizer, pretraining on books and web corpora, and supervised fine-tuning into a chat model. The implementation features a CPU version with OpenMP for small models and a full CUDA engine with cuBLAS matrix multiplications and hand-written FlashAttention, validated by full-model gradient checks against a CPU reference. The model trains approximately 116M parameters.
Source
Hacker Newsnanoeuler: A GPT-2-style LLM built from scratch in C/CUDA with hand-written backprop and FlashAttentiongithub.comKey quotes
· 3 pulledGPT-2-class language model built entirely from scratch in C/CUDA — no PyTorch, no autograd, no ML libraries.
The forward and backward passes are written and verified by hand, and the whole training pipeline lives in this repo.
A full from-scratch CUDA engine — cuBLAS matmuls, a hand-written FlashAttention, validated against a CPU reference by a full-model gradient check — trains a ~116M-p model.
You might also wanna read
Learning CUDA From Scratch: A 100-Problem Curriculum for GPU Programming and HPC Engineering
A structured GitHub repository offering 100 progressive problems to learn CUDA GPU programming, from basic "Hello World" kernels to producti
github.com·2d ago
NVIDIA Releases Nemotron-Labs-3-Puzzle-75B-A9B: A Compressed LLM for Efficient Inference
NVIDIA has released Nemotron-Labs-3-Puzzle-75B-A9B, a deployment-optimized large language model derived from Nemotron-3-Super-120B-A12B. The
A Practical Guide to Scaling Language Models: From Single Accelerators to Thousands
This article/book excerpt demystifies the science of scaling language models, explaining how TPUs and GPUs work, how they communicate, how L
jax-ml.github.io·12d agoA Practical Guide to Scaling Language Models: From Single Accelerators to Thousands
This article/book excerpt demystifies the science of scaling language models, explaining how TPUs and GPUs work, how they communicate, how L
jax-ml.github.io·12d ago
Optimizing LLM Inference: A C++ Backend for VRAM-Aware Sequence Packing
A technical deep-dive into optimizing LLM inference performance by eliminating wasteful padding in sequence batching. The article introduces
towardsdatascience.com·1mo ago
APEX4: Platform-Dependent W4A4 LLM Inference via Intra-SM Compute Rebalancing
This paper presents APEX4, a system for efficient W4A4 (4-bit weights, 4-bit activations) LLM inference that addresses the bottleneck of gro
arxiv.org·28d ago
Guide to Calculating GPU Memory for Self-Hosted LLM Inference
The article provides a guide on calculating GPU memory requirements and managing concurrent requests for self-hosted large language model (L
Product Hunt·11mo ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.