Krira Chunker: High-Performance Rust-Based RAG Chunking Engine for Processing Large Text Datasets
By
kriralabs
Slow-proofed and worth the wait. Worth its weight in flour.
Summary
Krira Chunker is a high-performance Rust-based chunking engine designed for RAG (Retrieval-Augmented Generation) pipelines. The beta software can process gigabytes of text data from various formats including CSV, PDF, JSON, DOCX, XLSX, and URLs in seconds with O(1) memory usage. It claims to be 40x faster than LangChain, with benchmark results showing processing of 42.4 million chunks in under 114 seconds. The tool is currently in active development with APIs subject to change, and the developers welcome bug reports and feedback.
Key quotes
· 4 pulledHigh-Performance Rust Chunking Engine for RAG Pipelines
Process gigabytes of text in seconds. 40x faster than LangChain with O(1) memory usage.
Processing 42.4 million chunks in 113.79 seconds (47.51 MB/s).
⚠️ Beta Software — Actively developed. APIs may change. We welcome bug reports and feedback.
You might also wanna read
Anthropic Releases Claude Opus 4.7 AI Model with 1M Context Window and Enhanced Coding Capabilities
Anthropic announces Claude Opus 4.7, their latest AI model featuring a hybrid reasoning architecture with a 1 million token context window.
anthropic.com·3d agoAnthropic Releases Claude Opus 4.7 AI Model with 1M Context Window and Enhanced Coding Capabilities
Anthropic announces Claude Opus 4.7, their latest AI model featuring a hybrid reasoning architecture with a 1 million token context window.
anthropic.com·3d agoIntegrating Type Systems into Neural Network Training for Reliable Code Generation
The article discusses the limitations of current neural network approaches to code generation, particularly how Large Language Models (LLMs)
Anthropic Releases Claude Opus 4.7 AI Model with Enhanced Coding and Creative Capabilities
Anthropic has released Claude Opus 4.7, its most powerful generally available AI model to date, which offers improvements over Opus 4.6 in a
The Verge·1mo ago
Anthropic Releases Claude Opus 4.7 AI Model for Complex Reasoning and Agentic Coding
Claude Opus 4.7 is Anthropic's most advanced generally available AI model, designed specifically for complex reasoning and agentic coding ta
Product Hunt·1mo ago
Anthropic Releases Claude Opus 4.7 with Enhanced Software Engineering and Vision Capabilities
Anthropic has released Claude Opus 4.7, a significant upgrade to their AI model that shows notable improvements in advanced software enginee
anthropic.com·1mo ago