3FS & Smallpond

3FS (Fire-Flyer File System) is a distributed parallel file system designed for AI training clusters, combining raw NVMe-oF throughput with RDMA networking to deliver terabytes/sec of aggregate I/O bandwidth. Paired with Smallpond, a distributed DuckDB-based data processing framework for massive pretraining corpora.

https://github.com/deepseek-ai/3FS

• Streams training datasets across thousands of nodes at 6.6 TB/s peak aggregate throughput.
• Zero-copy direct RDMA transfer straight from distributed NVMe drives into host RAM.
• Smallpond processes multi-terabyte data filtering and tokenization using embedded DuckDB engines.

All libraries