
Most home labbers and more than a few enterprise IT teams build storage systems under a quiet delusion: that every byte they own needs to be one millisecond away, forever. It's why people drop $3,000 on an 8-bay NAS, fill it with matched enterprise drives, and run RAID-Z2 around the clock — for a media library they touch twice a month and a backup folder they hope to never open again. The data lifecycle curve isn't subtle. The vast majority of files go cold within the first 30 to 90 days of being written, and after a year, the odds of any individual file getting opened again drop into the single digits. Keeping that cold tail on power-hungry NVMe or a constantly-spinning array isn't redundancy, it's a recurring charge. This is a piece about a different default: cheap and deep . Cold data gets cheap, dense, low-power media. Hot data keeps the SSD. The trick is making that split invisible to you and your applications — which is the part most homelab setups get wrong, and the part this article spends most of its time on. The Hidden Costs of Hot Storage RAID and Ceph are excellent at what they're built for: keeping production data available with zero tolerance for downtime. They're a bad fit for hoarding ISOs, finished video projects, and five-year-old backups, for two reasons. The setup fee. RAID and erasure coded pools want matched drives, parity overhead that quietly eats 20–30% of your raw capacity, and once you're past a handful of drives, controllers or RAM hungry nodes just to keep rebuild times sane. The operating expense. This is the one people underestimate. RAID stripes data across every disk in the pool, so reading a single 2GB file means every drive spins up — not just the one holding your data. Add a NAS scanner, a Samba ping, or Plex doing a routine library scan, and the array never gets the chance to sleep. Eight drives idling at 6–8W each is roughly 60W burned around the clock, which works out to something like 500 kWh a year spent keeping metal spinning for data nobody is reading. Add cooling the noise and the wear from drives that never get a break, and the bill climbs further. None of this is a knock on RAID — it's the right tool for a database or a production fileshare. It's the wrong tool for an archive. The Tape Threshold: When Does LTO Make Sense? LTO tape is the oldest trick in the cold-storage book, and the cost-per-gigabyte math still holds up. The catch has never been the media — it's the drive. A desktop LTO-9 unit runs $2,500–$4,000 , which means tape only starts paying for itself once you're archiving enough data to amortize that upfront hardware cost. Here's the 2026 picture, comparing raw media cost only (no enclosures, no RAID overhead, no electricity): | Storage Type | Unit | Cost / GB | 50 TB | 100 TB | 200 TB | |----|----|----|----|----|----| | NAS SSD | 4TB SATA SSD | $0.075 | $3,750 | $7,500 | $15,000 | | CMR HDD | 20TB Seagate IronWolf-class | $0.030 | $1,500 | $3,000 | $6,000 | | LTO-9 Tape | 18TB native cartridge | $0.005 | $300 | $580 | $1,160 | At roughly six times cheaper per gigabyte than CMR hard drives , and idle media that draws 0W on a shelf with a 30-year archival rating, tape wins decisively at scale. Factor in that 100TB of always-spinning HDDs pulls 60–80W continuously, and the electricity savings alone tend to cover the cost of the drive in around 18 months. So where's the threshold? Once you're comfortably past 50–100TB of genuinely cold data, the math tips firmly toward tape — even after paying for the drive. Below that, the upfront hardware cost is hard to justify, and a cheaper hardware tier does almost as well: SMR. SMR Drives: The Misunderstood Archival Hero If you've spent time on homelab forums, you've absorbed the cardinal rule: never buy SMR drives. A few manufacturers got caught quietly shipping Shingled Magnetic Recording drives in standard NAS lines a few years back, RAID arrays failed to rebuild, write speeds cratered, and the internet rightly revolted. The drives weren't the problem. The use case was. CMR (Conventional Magnetic Recording) writes data in concentric tracks with gaps between them — like lanes on a running track. The write head can update one lane without disturbing its neighbors, which is why CMR handles random writes gracefully. SMR overlaps those tracks like shingles on a roof to squeeze more density out of the same platter. That's great for capacity, but it means writing to one track can clip the edge of the next one, forcing the drive to read, erase, and rewrite an entire 256MB zone just to absorb a 4KB change. Hand an SMR drive to ZFS or ext4 — filesystems that constantly scatter journal writes, inode updates, and free-space bitmaps across the disk — and you get exactly the death spiral the forums warn about: the drive thrashes nonstop trying to satisfy random metadata writes it was never designed for. But SMR has exactly one superpower: sequential appends . Write to an empty zone start-to-finish without ever going backward, and the overlapping shingles fall into place perfectly — zero write amplification, sustained throughput of 150–200+ MB/s until the drive is full. That's not a hard drive behaving like an SSD. That's a hard drive behaving like a tape cartridge. Treat it that way ,raw block device, no filesystem, append only and it becomes one of the most cost effective storage tiers available, at roughly $14–$16/TB versus $20–$25/TB for standard CMR NAS drives, with manufacturers reporting meaningfully longer mechanical lifespans under sequential-only workloads. This is exactly the gap HuskHoard is built to fill. An Open Source Automated, Transparent Data-Tiering Engine HuskHoard is an open source (AGPL v3), Rust built archiving engine for Linux that does the cheap and deep split automatically. It watches your hot tier, and as files age past a policy you define, it compresses them and writes them off to cheap disk, physical LTO tape, or a cloud bucket, while every file stays fully visible in your filesystem the entire time. Nothing disappears from ls . Nothing breaks the application that expects the file to be there. # Install dependencies and build (Ubuntu 24.04 LTS) sudo apt install -y build-essential rclone libcap2-bin attr pkg-config libsqlite3-dev git curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh && source $HOME/.cargo/env git clone https://github.com/HuskHoard/HuskHoard.git && cd HuskHoard cargo build --release # Grant fanotify capabilities — no root daemon required sudo setcap cap_sys_admin,cap_dac_read_search+ep target/release/huskhoard # Format an archive volume (a disk image, an SMR drive, or /dev/nst0 for a tape drive) ./target/release/huskhoard format --tape-dev my_archive.img # Start the daemon ./target/release/huskhoard daemon # Stream a file straight off tape or disk, zero extraction needed ./target/release/huskhoard cat --file /media/movies/rushes.mp4 | mpv - No FUSE, no mount magic Most transparent archive tools lean on FUSE, which carries real overhead and a long history of edge-case weirdness. HuskHoard instead uses the Linux fanotify kernel API to watch for file access at the kernel level. When a stubbed file is opened, HuskHoard intercepts the request, pauses the calling process, recalls the data, and resumes — the application never knows a recall happened. From its point of view, the file was just slow to open once. Husks: lightweight stubs that keep your OS happy Once a file is archived, what's left behind on your hot tier is a husk, a sparse stub that reports the file's real logical size to ls , du , and your media scanner, while occupying close to zero actual disk space. Plex can still browse your whole library. Your backup tool can still see every path. The only thing that changes is what happens the moment something actually tries to read the bytes. Zstd Jump Frames: why partial reads don't cost full downloads This is the part that makes HuskHoard genuinely interesting from a systems-design angle, and it's worth a brief detour into why naive compressed archiving is so bad at random access. Compression algorithms build a rolling dictionary as they go. If you compress a 200GB file as one continuous Zstd stream and someone needs the bytes at the 100GB mark, you can't just jump there — the decompressor's state depends on everything that came before it. You'd have to stream and discard the first 100GB just to serve one query. HuskHoard breaks that chain on purpose. Every archived file is sliced into independent 16MB Jump Frames , with the Zstd dictionary deliberately flushed and reset at each boundary, at a cost of roughly 1–2% compression efficiency, in exchange for the ability to seek directly into the middle of a multi hundred gigabyte file. A custom TLV (Type-Length-Value) header prepended to each archived file maps logical byte offsets to physical frame locations, so a read request becomes: binary-search the index, issue one Range request or tape seek, decompress exactly one frame. A 5MB document buried inside a 150GB archive object costs 5MB of egress, not 150GB. Pre-compressed media gets an even simpler path: files like .mp4 and .mov skip the compression engine entirely and get written as raw blocks, so a video player's byte offset requests map 1-to-1 to physical offsets, a mathematically exact O(1) seek, whether the source is sitting on a spinning disk, an SMR archive volume, or LTO tape. | | $/GB | 100 TB | Idle Draw | |----|----|----|----| | NAS SSD | $0.075 | $7,500 | High | | CMR HDD | $0.030 | $3,000 | High | | LTO-9 Tape | $0.005 | $580 | 0W | One drive wakes up, not eight This is the direct answer to the RAID power problem described earlier. HuskHoard keeps its entire catalog — file paths, checksums, version history, and exact byte offsets in a SQLite database that lives on your NVMe hot tier, never on the archive media itself. That means routine filesystem chatter ( ls , a Plex scan, a backup tool checking what exists) never touches a spinning disk or a tape drive at all; it's all served from husks on SSD. Archive volumes are independent rather than striped, so reading one file wakes exactly one drive or loads one tape, while everything else stays in deep sleep at effectively 0W. There's no array-wide rebuild anxiety either: lose a drive, and you've lost the files on that drive, recoverable from whichever replica volume holds the copy, rather than risking an entire pool during a multi-day parity rebuild. Genuinely tiered: disk, cloud, and tape together It isn't a tape-only tool or a cloud-only tool, it treats all three cold tiers as interchangeable destinations for the same archive worker. N-way replication can mirror a file across a local SMR volume, a physical LTO cartridge, and a cloud bucket simultaneously. Cloud replication runs over rclone to 40+ providers, with files packed into the same 16MB Zstd frames used for local archiving which collapses thousands of small-file uploads into a handful of large sequential objects, cutting API transaction costs, while a targeted HTTP Range request still pulls back only the relevant frame on a read, regardless of how large the underlying archive object is. For tape specifically, HuskHoard includes a native SCSI driver spanning LTO-5 through LTO-9, handling block alignment and filemark navigation directly, and a StreamGate HTTP gateway that lets Plex, Jellyfin, or VLC seek through a file sitting on tape or in cloud storage via HTTP Range requests, scrubbing through a 4K video with zero SSD footprint and no full recall required. The exit clause that matters The detail that should matter most to anyone who has been burned by a proprietary backup format: HuskHoard's payload data is stored as plain Zstd streams, verified with BLAKE3 checksums. If the project disappeared tomorrow, your archive doesn't go with it, you can pull data back out using nothing more than dd and zstd . No license server, no proprietary container format, no asking permission to get your own files back. Putting It Together: A Genuinely Cheap, Genuinely Deep Build The cheap-and-deep philosophy scales down as well as it scales up. Picture a small, low-power Linux box, nothing fancier than a low end mini PC or single board computer with a handful of external SMR drives hanging off it for under a few hundred dollars in hardware. (Always check HuskHoard's current OS and architecture support before committing to unusual hardware; the project's documented baseline is a standard x86 Linux host on kernel 5.1+.) Format those drives as raw HuskHoard volumes, point the Janitor at your media library with an aging policy, and let it work: New files land on the NVMe hot tier and stay there while they're actually being used. Once a file crosses your configured age threshold, the Janitor queues it for archiving. The Archive Worker compresses it into Zstd Jump Frames and writes it sequentially to whichever volume SMR drive, tape, or cloud bucket your policy targets. The original location becomes a husk: same path, same apparent size, near-zero actual disk usage. Every other drive in the pool stays asleep until something specifically asks for the data on it. The result is a system that idles at close to 0W, generates almost no heat, puts essentially no mechanical wear on drives that spend most of their life parked, and costs pennies a year to keep running — while still behaving, from the outside, exactly like a single fast filesystem. Conclusion Stop treating archive data like production data. The file you haven't opened in eight months doesn't need to live on the same tier as your active project, and keeping it there is mostly just an invisible line item on your power bill. Tape wins decisively at real scale; SMR, used correctly, is the budget tier RAID convinced you to avoid for the wrong reasons; and cloud is fine as a third leg of replication as long as you're not paying to download a 150GB object to retrieve one document. HuskHoard's job is making that tiering automatic and reversible — files stay visible, recalls happen on demand via fanotify , and the underlying data is yours in a plain, documented format whether you ever touch the tool again or not. If you want to dig further: the project site covers the full architecture and economics, the GitHub repository has the source, quick start, and full command reference, and the blog has deep dives on Jump Frames, SMR economics, and the tape cost case if you want the math behind the claims in this article. \
View original source — Hacker Noon ↗


