Efficiently mounting tar.gz archives in WebAssembly using file index metadata
By
datajeroen
A five-star bake. Worth schmearing, sharing, saving.
Summary
This article describes a technique for mounting tar.gz archives directly into the Emscripten virtual filesystem without extracting them. Instead of downloading, decompressing, and extracting files from a tarball, the author proposes generating a small index file that lists the size and offset of each file in the tar archive. This metadata allows the tar blob to be mounted directly via Emscripten's WORKERFS, eliminating the need for copying files. The approach is particularly useful for WebAssembly applications that need to access data from tarballs efficiently.
Key quotes
· 3 pulledLots of data on the internet lives in tarballs, often distributed as gzipped .tar.gz files.
To get to this data, we have to download the entire .tar.gz file, decompress it, and then iterate through the blob from beginning to end to make copies of the files we need. This is expensive.
Instead of extracting a .tar.gz archive, we can generate a small index file which lists the size and offset of each file in the tar, and use this metadata to mount the tar blob directly via Emscripten's WORKERFS without any copying.
You might also wanna read
Running Gemma 4 on a 2016 Xeon Server with No GPU: A Technical Walkthrough
The article describes running Gemma 4 (a 25B-parameter Mixture-of-Experts model) on a severely outdated server with a 2016 Intel Xeon E5-262
point.free·1h ago
NVIDIA Announces "Hack for Impact" London Event for Autonomous AI Agent Development
NVIDIA is hosting a "Hack for Impact" event in London, challenging participants to build autonomous agentic applications using open-source m
luma.com·3h ago
Four practical steps to control Azure Foundry token costs for agentic AI workloads
This article provides practical guidance on controlling token costs in Microsoft Azure Foundry, particularly for agentic AI workloads where
purplefrogsystems.com·5h ago
MerLean-Prover: A Recursive Agent Harness for Lean 4 Theorem Proving Outperforms Baselines
MerLean-Prover is an end-to-end Lean4 theorem prover that replaces 'sorry' declarations with kernel-checkable proofs using three agent types
arxiv.org·5h ago
Why small pull request policies can backfire on software quality
The article critiques a common software engineering policy that limits pull requests (PRs) to small sizes (e.g., 500 lines, few files). Whil
apenwarr.ca·7h ago
How Anthropic contains Claude's expanding access across its products
Anthropic describes how it has evolved its approach to granting Claude, its AI assistant, increasingly broad access to internal systems over
anthropic.com·8h ago