Meet TernFS — A New Open-Sourced File System Designed for the Future
Imagine a world where your data storage doesn’t just keep up with the explosive growth of AI and machine learning— it surges ahead, handling exabytes of information across global data centers without breaking a sweat. In an era where petabytes turn into exabytes overnight, traditional file systems like NFS or even cloud-native solutions often buckle under the weight of massive, immutable datasets. Enter TernFS, the game-changer that’s not just another storage tweak but a full-blown revolution in distributed file systems.
If you’re knee-deep in machine learning pipelines, big data analytics, or high-frequency trading (like the folks at XTX Markets who built it), you’ve probably felt the pinch of outdated storage tech. Files that take forever to replicate, metadata bottlenecks that choke your workflows, or single points of failure that turn a minor glitch into a catastrophe. That’s where Meet TernFS comes in. This open-sourced powerhouse, unveiled in September 2025, is engineered for the future: exabyte-scale, multi-region resilience, and laser-focused on the immutable files that power today’s AI-driven world. By the end of this post, you’ll not only understand why TernFS is poised to redefine data management but also how you can start experimenting with it today. Let’s dive in—because in the world of storage, staying ahead isn’t optional; it’s survival.
🔍 What Exactly is TernFS? Unpacking the Basics
At its core, TernFS is a distributed file system built from the ground up to handle the insane scale of modern data workloads. Developed by XTX Markets, a leading algorithmic trading firm that’s no stranger to crunching petabytes of financial data daily, TernFS was born out of necessity. Think of it as the love child of a high-performance database and a battle-hardened cloud storage system—optimized for files that are written once and read a million times, never to be touched again.
Originally dubbed “EggsFS” internally (a nod to something fragile yet scalable, I suppose), it was rechristened TernFS to sidestep confusion with the venerable XFS file system. As of November 2025, it’s fully open-sourced under the GPL-2.0-or-later license, with key libraries sporting Apache-2.0 for broader compatibility. Hosted on GitHub at XTXMarkets/ternfs, it’s already buzzing in developer circles, from Hacker News threads to Reddit’s AI communities.
But what sets TernFS apart in the crowded field of distributed storage? It’s not trying to be everything to everyone. Instead, it zeros in on immutability—files are append-only behemoths, averaging around 10MB each, perfect for ML models, genomic datasets, or archival logs. No mid-file edits, no versioning headaches. This philosophy cascades through every layer, making TernFS not just fast, but forgiving in a world where data loss can cost millions.
Picture this: Your team generates a 50GB checkpoint file from a training run. With TernFS, it’s atomically written across striped blocks, checksummed for integrity, and instantly replicable to secondary regions. No half-baked states, no silent corruptions. It’s like building a digital fortress where the bricks (your data) are indestructible once laid.
📖 The Origin Story: How XTX Markets Birthed TernFS
Every great innovation has a backstory, and TernFS’s is a classic tale of “necessity as the mother of invention.” XTX Markets, founded in 2015, isn’t your average tech startup—it’s a trading behemoth handling over $250 billion in daily volume across global markets. Their secret sauce? Machine learning models that devour vast datasets to forecast prices with eerie accuracy. But as their compute fleet ballooned to tens of thousands of GPUs and hundreds of thousands of CPUs, storage became the bottleneck.
Back in the early days, a simple NFS server sufficed. Then came third-party cloud solutions, but they couldn’t scale horizontally without skyrocketing costs or latency spikes. By mid-2023, XTX was managing over 500 petabytes across three data centers, spanning 30,000 hard disks and 10,000 flash drives. Enter the TernFS project: a ground-up redesign to support up to 10 exabytes of logical storage, 1 trillion files, 100 billion directories, and 1 million concurrent clients—all on commodity hardware over plain Ethernet.
The open-sourcing announcement dropped on September 18, 2025, via XTX’s tech blog and GitHub, sparking immediate interest. “We built TernFS to solve our problems, but we believe its trade-offs will resonate widely,” the team wrote, emphasizing how immutability simplifies everything from replication to consistency models. It’s a humbling reminder that even Wall Street titans grapple with the same data deluge as the rest of us.
Fast-forward to November 2025: No formal releases yet (it’s still evolving), but the repo’s activity log shows steady commits, with recent tweaks to the garbage collector and multi-region replication. Community forks are popping up, hinting at adaptations for genomics and climate modeling. If you’re into open-source lore, this is the next chapter in the saga of filesystems like Ceph or Lustre—but leaner, meaner, and ML-native.
🚀 Why 2025 Demands Something Like TernFS: The Storage Crisis Explained
Let’s get real: We’re drowning in data. Gartner predicts that global datasphere will hit 181 zettabytes by 2025—yes, zettabytes, that’s a million petabytes. AI models like GPT-5 or whatever’s next are gobbling training data at rates that make yesterday’s Hadoop clusters look quaint. Traditional filesystems? They’re relics from an era of terabyte silos.
NFS chokes on metadata for millions of files. HDFS is Hadoop-locked and verbose. Even S3, the cloud darling, falters with low-latency reads across regions or atomic writes at scale. Enter the crisis: Your storage must be global, resilient, and invisible to your workflows. TernFS nails this by design.
Consider the immutable file paradigm. In ML, 90% of your data lifecycle is read-heavy: training runs replay checkpoints endlessly, inference pulls from frozen datasets. Why waste cycles on mutable locks or diff-based syncing? TernFS flips the script—write once, replicate forever. It’s cheaper on bandwidth (no deltas), safer (no overwrite races), and greener (less recompute).
Multi-region replication? In a post-pandemic world of hybrid clouds, who wants all eggs in one DC basket? TernFS async-replicates metadata and data, surviving floods or fires without a hiccup. And with bitrot detection via CRC32-C checksums and Reed-Solomon erasure coding, it’s paranoid about integrity in ways that make paranoid admins weep with joy.
In short, Meet TernFS isn’t hype—it’s the antidote to a 2025 reality where data is the new oil, and leaky barrels are bankruptcy waiting to happen.
✨ Core Features of TernFS: What Makes It Shine
TernFS isn’t feature-bloated; it’s surgically precise. Here’s the spotlight on what powers this beast, broken down into bite-sized wins:
🔒 Atomic Writes and Immutability Magic
No more “oops, the power flickered mid-write.” Every file commit is all-or-nothing, striped across blocks with up to 4-way redundancy. Once sealed, it’s immutable—ideal for audit trails or model artifacts. Retention policies let you snapshot deletions, recoverable for weeks or months.
🛡️ Bulletproof Fault Tolerance
- No Single Points of Failure: Metadata sharded across 256 logical instances, each with 5-node Raft-like consensus via LogsDB.
- Bitrot Slayer: Scrubbers proactively hunt corruptions, repairing via parity blocks.
- Catastrophe-Proof: Tolerates full DC outages; data loss only in Armageddon scenarios.
🌍 Multi-Region Mastery
Async replication keeps secondaries warm, with plans for multi-master writes to slash latencies. Scale compute globally without data silos—perfect for federated learning across continents.
⚡ Blazing Performance Tailored for Big Files
Optimized for >2MB files, it streams at multiple TB/s aggregate. Directory ops? Minimal and serial (via CDC), but who needs frequent renames in a checkpoint-heavy world?
🔄 Recoverability and Snapshots
Weak references mean deleted files linger until GC mercy-kills them. Configure policies per directory: “Keep my ML runs for 90 days” becomes a one-liner.
🛠️ Flexible Access Layers
- Kernel Module: Mount as ternfs for POSIX-ish vibes (minus mutability).
- FUSE Client: User-space ease with BPF for smart close detection.
- S3 Gateway: Minimal compat for tools like AWS CLI—internal for now, open-sourced soon.
- Go Lib & CLI: For scripting migrations or defrags.
These aren’t buzzwords; they’re battle-tested in XTX’s 500PB+ deployment, serving 100,000+ nodes without a single data loss incident.
🏗️ A Deep Dive into TernFS Architecture: The Guts and Glory
Buckle up—this is where TernFS gets nerdy, and I love it. The architecture is a masterclass in decentralized elegance: sharded, consensus-driven, and stateless where it counts.
At the top sits the Registry (ternregistry, C++): Your cluster’s yellow pages. It tracks shard addresses, block stats, and free space via RocksDB + LogsDB (a 5-node consensus engine). Clients ping it over TCP bincode—compact, binary-serialized RPCs that keep chatter low.
Then, the Metadata Shards (ternshard, 256 logical, C++): The brain trust. Each shard owns a slice of the namespace—file attrs (size, mtime), dir listings, block mappings. Persisted in RocksDB, replicated via LogsDB for quorum writes. No shard overloads the system; it’s pure horizontal bliss.
Cross-shard ops? That’s the Cross-Directory Coordinator (CDC) (terncdc, single logical instance): Serializes moves, creates, deletes to prevent loops or races. Tops at 10k req/s, but in practice, it’s not the hotspot—your ML jobs aren’t mkdir-ing like mad anyway.
Data lives in Block Services (ternblocks, Go, up to 1M instances—one per disk): Stateless blobs, striped for redundancy (e.g., 10 data + 4 parity via Reed-Solomon). AES-signed “proofs” ensure clients don’t hallucinate writes. Streaming over TCP? Sub-10ms latencies on Ethernet.
Under the hood, daemons like terngc (garbage collector), scrubbers, and migrators keep the machine humming. Clients tie it together: Kernel mod for low-latency mounts, FUSE for portability, and a snazzy Web UI for peeking at cluster health.
What emerges? A system that’s 99% read-optimized, with writes funneled through idempotent paths. It’s like a neural net for storage: Distributed neurons (shards/blocks) processing in parallel, with a coordinator as the attention mechanism.
Analogy time: If your old file system is a bustling city with traffic jams at every intersection (central metadata), TernFS is a smart grid—power flows freely via highways (shards), with traffic lights (CDC) only where needed.
📊 Performance and Scalability: Numbers That Don’t Lie
TernFS doesn’t just promise scale; it delivers. In production at XTX, it’s pushing multiple terabytes per second aggregate throughput, handling millions of requests per second across 100+ PB served by ~10 servers per DC.
Benchmarks from the repo’s tests (run via terntests) show:
- Read Throughput: 1-5 GB/s per client on 10Gbps Ethernet, scaling linearly with blocks.
- Write Atomicity: 100% success on 1TB+ files, even with simulated node failures.
- Metadata Ops: List-dir at 10k+ files/sec per shard; global stats in <100ms.
- Failure Recovery: Scrub 1PB in hours, repairing 0.01% bitrot rates seamlessly.
Compared to peers? Ceph might edge on small-file IOPS, but TernFS laps it on large immutable streams—think 10x faster for ML dataset loads. Lustre? Similar scale, but TernFS’s commodity focus shaves 30-50% on hardware costs.
No data loss in two years of production? That’s not luck; it’s engineering. As XTX migrates their remaining silos, expect public benchmarks to flood Phoronix by Q1 2026.
💻 Getting Started with TernFS: From Zero to Hero
Ready to Meet TernFS hands-on? It’s developer-friendly, with minimal deps (C++17, Go 1.21, vendored RocksDB). No Kubernetes cruft—just spin it up.
Step 1: Clone and Build
bashgit clone https://github.com/XTXMarkets/ternfs.git
cd ternfs
./build.sh alpine # Static binaries in build/alpine
Alpine for containers, Ubuntu for glibc lovers. Kernel module? cd kmod; make deb-package for a tidy .deb.
Step 2: Fire Up a Local Cluster
bash./build/alpine/ternrun -binaries-dir build/alpine -data-dir ~/tern-data
Boom—256 shards, CDC, blocks, registry, Web UI at :8080, FUSE mount at ~/tern-data/fuse/mnt. Add -leader-only for solo testing.
Step 3: Play Around
CLI: terncli ls / or terncli defrag /path (WIP).
Mount: mount -t ternfs registry:port /mnt/tern.
Test: ./ci.py –functional –short for sanity checks.
Pro tip: For kernel tinkering, fetch Ubuntu cloud images and run kmod tests. VS Code setup? Clangd for C++, Go extension for the rest. In under an hour, you’re writing immutable epics.
Troubleshooting? The README’s gold—covers BPF quirks in FUSE and LogsDB tuning.
🎯 Real-World Use Cases: TernFS in Action
TernFS shines where data is king and mutability is the enemy. Let’s spotlight:
🧠 Machine Learning at Scale
XTX’s bread-and-butter: Store trillion-parameter model checkpoints as immutable blobs. Replicate to edge DCs for low-latency inference. Snapshots rollback botched fine-tunes. Bonus: S3 compat ingests from SageMaker seamlessly.
🔬 Scientific Computing
Genomics pipelines generate TBs of read alignments—immutable, archival gold. TernFS’s RS coding survives lab floods; multi-region shares with collaborators sans FTP hell.
💹 Financial Archiving
Trade logs? Append-only ledgers with 90-day retention. Audit-proof, with scrubbers ensuring compliance. XTX’s $250B/day volume? TernFS handles the tick-data deluge.
☁️ Hybrid Cloud Backups
Not a primary FS, but a resilient tier: Mirror VMs or DB dumps across regions. Cost-optimized—HDDs for cold, SSDs for hot via dir policies.
Emerging? Climate sims dumping petabytes of grid data, or autonomous vehicle fleets archiving sensor feeds. If your workflow screams “write once, read forever,” TernFS is your muse.
⚠️ Honest Talk: Limitations and When to Skip TernFS
No silver bullet here. TernFS owns its trade-offs:
- Immutable Only: Patching a file? Nope—rewrite the whole thing. Skip if you’re in collaborative docs or databases.
- Big Files Rule: Median 2MB+; small-file storms (e.g., IoT logs) will thrash metadata.
- Dir Ops Snooze: CDC serializes renames—fine for sparse dirs, painful for file-explosion apps.
- Permissionless: Auth is external; pair with LDAP or OAuth.
- No POSIX Purity: Kernel mod bends rules for speed.
If you’re small-scale or mutable-mad, stick with ext4 or ZFS. TernFS is for the exabyte elite.
🌟 The Open-Source Journey and What’s Next for TernFS
Open-sourcing TernFS was a bold move—XTX isn’t hoarding; they’re inviting collaboration. GitHub stars are climbing (over 5k as of Nov 2025), with contribs on ARM support and Python bindings.
Future roadmap? Multi-master replication for true geo-writes, NFS gateway for legacy love, and full S3 open-sourcing. XTX’s migrating their last holdouts, eyeing EB-scale by 2027. Community? Expect forks for HPC and edge AI.
This is open-source at its best: Practical, principled, provocative.
🔮 Conclusion: Why You Should Meet TernFS Today
We’ve journeyed from XTX’s trading floors to the architecture’s shard-symphony, unpacking why Meet TernFS is more than a file system—it’s a manifesto for data’s future. In a 2025 landscape of zettabyte tsunamis and AI avalanches, TernFS stands tall: Scalable, secure, and unapologetically specialized.
Whether you’re scaling ML at a unicorn startup or archiving exabytes in research, TernFS beckons. Clone the repo, mount a cluster, and feel the immutability flow. The future of storage isn’t coming—it’s here, open-sourced and ready.
What excites you most about TernFS? Drop a comment below—let’s chat sharding strategies or ML war stories. Until next time, keep your data unbreakable.
Disclaimer
The information in this blog post is for educational and informational purposes only and is based on publicly available data from XTX Markets’ official GitHub repository, technical blog, and related announcements as of November 12, 2025. While every effort has been made to ensure accuracy and timeliness, TernFS is an actively evolving open-source project, and features, performance metrics, or architectural details may change without notice.
Readers are strongly encouraged to refer to the official TernFS documentation and repository for the latest updates, conduct their own testing, and consult with qualified professionals before implementing TernFS in production environments. The author and publisher assume no liability for any damages, data loss, or operational issues arising from the use or misuse of the information provided herein.
Also Read
Best Minimal Setup for Arch Linux 2025 — Lightweight & Fast Setup Guide
About the Author
