How to shuffle a big dataset
At Jane Street, we often work with data that has a very low signal-to-noise ratio, but fortunately we also have a lot of data. Where practitioners in many fields might be accustomed to having tens or…
Read the full articleYou might also wanna read
End-to-End Basic Data Engineering Tutorial (Spark, Dremio, Superset)
> **Cross-posted.** This article's canonical home is [Iceberg Lakehouse](
Research Team Collects 10,000 Hours of Neuro-Language Data for Thought-to-Text Models
Over the last 6 months, we collected ~10k hours of data across thousands of unique individuals. As far as we know, this is the largest neuro
condu.it·7mo ago
Mining of Massive Datasets
Big-data is transforming the world. Here you will learn data mining and machine learning techniques to process large datasets and extract va
Exploring the Use of Randomly Generated Data for Model Pre-Training
We investigate the use of randomly generated data for the sake of pre-training a model. We justify this approach theoretically from the pers
arxiv.org·1y ago
What Zen Buddhism can teach us about data engineering
The data engineering profession is obsessed with answers. How do we improve data quality? Which platform should we choose? How do we govern
Technical Discussion: Distributed SQL Engine Requirements for Ultra-Wide Tables in ML and Multi-Omics Data
I ran into a practical limitation while working on ML feature engineering and multi-omics data.
news.ycombinator.com·5mo ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.