Building 500T of Image Storage on FreeBSD + ZFS + Samba

We needed a big, cheap box to hold OS images, backups, and large files that rarely change for machines on our internal network. Commercial storage was out of budget, but we had a server stuffed with disks: 36 × 16T SATA, two NVMe drives, and two 25G ports with RDMA. So we built it ourselves on FreeBSD + ZFS + Samba, spending the money on spindles instead of on a storage head.

This is a record of the decisions we made, the tradeoffs we weighed, and the config we ended up running. The short version: for a capacity-first workload dominated by large sequential files, ZFS raidz2 with a SLOG and an L2ARC is almost unreasonably cost-effective, and you get end-to-end data integrity on top.

Why FreeBSD + ZFS

ZFS was written by Matthew Ahrens and Jeff Bonwick at Sun starting in 2001, and ported to FreeBSD in 2008. Because its CDDL license is incompatible with the GPL, it has never lived inside the Linux kernel, which is exactly why running it on FreeBSD is pleasant — it’s a first-class citizen, with none of the DKMS dance you do on Linux. After Oracle acquired Sun and closed OpenSolaris, most of the core developers left and founded OpenZFS in 2013; that’s the lineage FreeBSD ships today.

What actually matters for this box is a handful of ZFS features: a storage pool that abstracts all the disks into one namespace, copy-on-write so a power loss can’t corrupt data already on disk, end-to-end checksums that catch and repair silent corruption, RAID-Z for software redundancy, and transparent compression. Together they mean we don’t need a hardware RAID card, a dead disk is just a swap-and-resilver, and every byte we read back has been verified.

For engineers coming from Linux, the day-to-day commands map roughly like this:

The hardware

Here’s what’s in the box:

Network: 25G * 2 with rdma
Disks:
  36 * 16T sata        # data disks
  1  * 1T  nvme        # zfs log (SLOG, write path), really should be 2 drives mirrored
  1  * 3.8T nvme       # zfs cache (L2ARC, read cache)

Two tradeoffs are worth spelling out.

The SLOG (1T NVMe) is not a write cache in the usual sense. It only absorbs synchronous writes (the ZIL): requests that need an acknowledged commit land on the low-latency NVMe first and return immediately, then get flushed to raidz2 in the background. For a workload like Samba that issues sync writes, this noticeably cuts latency. The catch: if the SLOG dies at the exact moment of a power loss, you can lose the last few seconds of sync writes that hadn’t made it to the pool yet. So strictly it should be two mirrored NVMe drives — we only had one, which was a known compromise we wrote into the docs to remind ourselves to fix later.

The L2ARC (3.8T NVMe) is a second-level read cache that backstops hot data the in-memory ARC can’t hold. It only helps reads, and it costs some RAM to index, so bigger isn’t automatically better. For our read-heavy, write-light pattern — images pulled over and over — it pays off.

Building the zpool

First, look at the disks:

geom disk list

Then assemble all 36 into a single raidz2:

zpool create datasets raidz2 da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11 da12 da13 da14 da15 da16 da17 da18 da19 da20 da21 da22 da23 da24 da25 da26 da27 da28 da29 da30 da31 da32 da33 da34 da35

Why raidz2 and not raidz1? raidz2 gives you RAID6-level redundancy — two disks can fail at once. In a 36-disk pool, resilvering a single 16T drive takes a long time, and during that window every disk in the pool is reading hard, which is precisely when a second drive is most likely to follow the first into the grave. At this scale raidz1 is close to gambling with the data. raidz2 trades two disks of capacity for safety across the resilver window, which struck the balance we could live with between reliability, performance, and capacity.

An honest caveat: putting all 36 disks in one vdev is not textbook. The wider the vdev, the slower the resilver and the worse the random IOPS. The safer layout is several narrower raidz2 vdevs striped together. We chose a single vdev to max out usable capacity because this box is capacity-first and overwhelmingly sequential — if your workload is random small files, don’t copy this.

Add the SLOG (write log):

zpool add datasets log nvd0

If you have two NVMe drives, the SLOG should be mirrored (do this):

zpool add datasets log mirror xxx xxx

Add the L2ARC (read cache):

zpool add datasets cache nvd1

Check the result:

zpool list
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
datasets   524T  95.6T   428T        -         -     0%    18%  1.00x  ONLINE  -

zpool status datasets
  pool: datasets
 state: ONLINE
  scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	datasets    ONLINE       0     0     0
	  raidz2-0  ONLINE       0     0     0
	    da0     ONLINE       0     0     0
	    da1     ONLINE       0     0     0
	    da2     ONLINE       0     0     0
	    da3     ONLINE       0     0     0
	    da4     ONLINE       0     0     0
	    da5     ONLINE       0     0     0
	    da6     ONLINE       0     0     0
	    da7     ONLINE       0     0     0
	    da8     ONLINE       0     0     0
	    da9     ONLINE       0     0     0
	    da10    ONLINE       0     0     0
	    da11    ONLINE       0     0     0
	    da12    ONLINE       0     0     0
	    da13    ONLINE       0     0     0
	    da14    ONLINE       0     0     0
	    da15    ONLINE       0     0     0
	    da16    ONLINE       0     0     0
	    da17    ONLINE       0     0     0
	    da18    ONLINE       0     0     0
	    da19    ONLINE       0     0     0
	    da20    ONLINE       0     0     0
	    da21    ONLINE       0     0     0
	    da22    ONLINE       0     0     0
	    da23    ONLINE       0     0     0
	    da24    ONLINE       0     0     0
	    da25    ONLINE       0     0     0
	    da26    ONLINE       0     0     0
	    da27    ONLINE       0     0     0
	    da28    ONLINE       0     0     0
	    da29    ONLINE       0     0     0
	    da30    ONLINE       0     0     0
	    da31    ONLINE       0     0     0
	    da32    ONLINE       0     0     0
	    da33    ONLINE       0     0     0
	    da34    ONLINE       0     0     0
	    da35    ONLINE       0     0     0
	logs	
	  nvd0      ONLINE       0     0     0
	cache
	  nvd1      ONLINE       0     0     0

errors: No known data errors

524T raw. After raidz2’s two parity disks and ZFS’s own overhead, usable lands around 451T.

Mounting

zfs set mountpoint=/datasets datasets

zfs mount datasets

Confirm it’s mounted:

df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ada0p2    429G     11G    384G     3%    /
devfs          1.0K    1.0K      0B   100%    /dev
datasets       451T    9.5G    451T     0%    /datasets

Serving it over Samba

Most of the internal network is Windows, so SMB is the path of least resistance:

pkg install samba413-4.13.17_1

Enable it at boot — either form works (sysrc just edits /etc/rc.conf for you):

sysrc samba_server_enable=YES
# equivalent to adding this line to /etc/rc.conf:
# samba_server_enable="YES"

Then smb.conf. Several values here are deliberate, not defaults:

[global]
    workgroup = SAMBA
    security = user
    server string = ZFS SAMBA Version %v
    passdb backend = tdbsam
    hosts allow = 127. 10.20.193. 192.168.23. 192.168.1. 10.20.191. 10.88.88.
    load printers = no
    use sendfile = yes
    socket options = TCP_NODELAY SO_RCVBUF=16384 SO_SNDBUF=16384
    write raw = yes
    read raw = yes
    max xmit = 65535
    unix charset = UTF-8
    dos charset = GB2312
    server multi channel support = yes
    max protocol = SMB3
[zfs]
    comment = ZFS SAMBA Version %v Image Storage
    path = /datasets/
    browseable = yes
    writable = yes
    public = no
    guest ok = no
    create mask = 0755
    directory mode = 0755
    security = share

Why those lines matter: use sendfile = yes lets the kernel push files straight from ZFS to the socket without bouncing through userspace, which helps a lot with large transfers; server multi channel support = yes with SMB3 and two 25G ports lets a single client spread one transfer across multiple links; hosts allow restricts access to our own internal subnets as a first line of access control; and dos charset = GB2312 keeps Chinese filenames readable on older Windows clients.

Finally, create a user with smbpasswd -a, restart Samba, and clients can mount the share.

What we did next

Once this was running, a few things were on the list — some planned, some ongoing. Add a second NVMe to mirror the SLOG and close that known single point of failure. Schedule a periodic zpool scrub so ZFS proactively walks the pool and repairs silent corruption instead of waiting until we read a bad block. And monitor the L2ARC hit rate to confirm that 3.8T of read cache is actually earning its keep. If you’re also building storage out of a pile of cheap disks, this stack is worth copying — but mirror the SLOG from day one and don’t leave that gap open.