hist-rs: High-Performance Command-Line Tool for Counting Unique Lines with 25x Speed Improvement
By
noamteyssier
Fresh out the oven, still warm. Top of the tray.
Summary
hist-rs is a high-performance command-line tool for counting unique lines in text files, offering significant speed improvements over traditional Unix pipelines like 'sort | uniq -c'. The tool provides equivalent functionality to the standard Unix command chain but with 25x faster performance, as demonstrated through benchmarks comparing various tools. It also supports deduplicating input streams by printing only unique lines. The article includes installation instructions, usage examples, and detailed performance benchmarks using hyperfine on a 1M line FASTQ file generated with nucgen.
Key quotes
· 5 pulledA high-throughput CLI to count unique lines.
This is a standalone tool with equivalent functionality to sort | uniq -c | sort -n.
There is also support for deduplicating an input stream (i.e. only printing unique lines).
I am measuring the performance of equivalent sort <file | uniq -c | sort -n functionality.
An efficient unique-line counter (25x over `sort | uniq -c`).
You might also wanna read
Optimizing .NET APIs for High Throughput: Techniques for 1M Requests Per Minute
Article discusses techniques for designing high-throughput .NET APIs capable of handling 1M requests per minute. It covers horizontal scalin
blog.elmah.io·18h ago
How micro-optimizations in Azure Service Bus SDK paved the way for a smarter redesign
The article discusses how micro-optimizations in the Azure Service Bus SDK led to meaningful design improvements. Rather than advocating for
How Kestra Improved Orchestrator Performance Across 14 Releases: A Year of Performance Engineering
Kestra's engineering team details their year-long performance engineering journey across releases 0.19 to 1.3, treating performance as an on
medium.com·2d ago
Optimizing Deep Learning Performance Through First-Principles Reasoning
The article discusses improving deep learning model performance by reasoning from first principles rather than relying on ad-hoc tricks and
auge: A Terminal-Based OCR and Vision Analysis Tool with On-Device Processing
auge is a command-line tool that provides Apple Vision-like OCR, classification, barcode detection, and face recognition capabilities direct
auge.franzai.com·1mo ago
Java Performance Optimization: Fixing 8 Common Anti-Patterns to Reduce Processing Time by 80%
The article presents a case study of Java performance optimization where fixing common anti-patterns dramatically improved application perfo
jvogel.me·2mo ago