Nemotron-Labs-3-Puzzle-75B-A9B: A Compressed Hybrid MoE LLM for Efficient Interactive Deployment
By
[Submitted on 5 Jul 2026]
Summary
This paper presents Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of the Nemotron-3-Super hybrid Mixture-of-Experts (MoE) large language model, optimized for interactive deployment. The model achieves approximately 2x higher server throughput than the parent model on a single 8xB200 node under high user throughput constraints, and increases 1M-token concurrency from 1 to 8 requests on a single H100 GPU. The compression uses a multi-stage pipeline combining Iterative Puzzle compression, knowledge distillation, reinforcement learning, quantization, and a Multi-Token Prediction head, jointly optimizing heterogeneous MoE pruning, active parameter budget, and Mamba pruning. Despite substantial compression, the model retains strong downstream accuracy across reasoning, coding, multilingual, long-context, and agentic benchmarks.
Source
Twitter / XNemotron-Labs-3-Puzzle-75B-A9B: A Compressed Hybrid MoE LLM for Efficient Interactive Deploymentarxiv.orgKey quotes
· 5 pulledIn interactive serving workloads on a single 8xB200 node, Puzzle-75B-A9B achieves approximately 2x higher server throughput than Nemotron-3-Super at matched user throughput constraints.
In ultra-long-context deployment on a single H100 GPU, the compressed model increases 1M-token concurrency from 1 request to 8 requests.
Puzzle-75B-A9B is constructed using a multi-stage pipeline that combines the Iterative Puzzle compression framework with knowledge distillation, reinforcement learning, quantization, and a Multi-Token Prediction head.
The compression process jointly optimizes heterogeneous MoE pruning, active parameter budget, and Mamba pruning to improve inference efficiency while preserving model quality.
These results demonstrate that large hybrid MoE models can be substantially optimized for deployment efficiency while maintaining strong downstream capability.
You might also wanna read
NVIDIA Releases Nemotron-Labs-3-Puzzle-75B-A9B: A Compressed LLM for Efficient Inference
NVIDIA has released Nemotron-Labs-3-Puzzle-75B-A9B, a deployment-optimized large language model derived from Nemotron-3-Super-120B-A12B. The
Building high-performance expert-parallel dispatch and combine kernels for MoE LLM inference
This article provides a deep technical deep-dive into the architecture and implementation of high-performance Expert Parallelism (EP) kernel
fergusfinn.com·27d ago
Jet-Nemotron: Hybrid Language Model Architecture with PostNAS Achieves High Efficiency and Accuracy
Jet-Nemotron is a new family of hybrid-architecture language models that achieves comparable or superior accuracy to leading models like Qwe
arxiv.org·9mo ago
Rotary GPU: Enabling Large Mixture-of-Experts Models on Consumer Laptop GPUs with Limited Memory
This paper presents Rotary GPU, an exploratory approach to running large Mixture-of-Experts (MoE) language models on consumer-grade hardware
arxiv.org·1mo ago
Mesh-LLM: Distributed LLM Inference System Using llama.cpp Across Multiple Machines
Mesh-LLM is a reference implementation that enables distributed inference of large language models across multiple machines by compiling lla
github.com·3mo ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.