10 ZFS on Ubuntu/Linux Learnings

I thought I’d go through a few learnings from running the ZOL package on Ubuntu. Some are general observations on zfs, some are ZOL specific, and some are just issues to avoid. For those who aren’t familiar with ZFS, it’s a filesystem capable of storing incredible quantities of data. It’s able to detect and fix bit rot, it’s able to do a whole swathe of cool tricks such as live snapshots, deduplication, and compression.

These are in no particular order.

1. If you can, and you have data you want to treat differently, split them up as low as you can on the pool’s structure. It’s easy to configure one part of the pool to have deduplication or one part to have compression with a particular algorithm. It’s a bad idea to turn it one for a whole pool of heterogeneous data. Deduplication in general, is a performance killer. If you turn it on, do it for a small set of highly duplicated data, and it’s a valuable feature. If you turn it on for your whole pool, then everything from deleting snapshots, to writing large, unduplicated datasets becomes a huge chore. The same goes for compression.

2. Less is often more – choose the best feature, and use just that feature. Either compression or deduplication. Don’t go overboard with snapshots either, or disable them for data storage that has high turnover. Otherwise your snapshots will bloat out to many times the size of the base data at any point in time.

3. Deleting snapshots that are more recent, seems quicker than deleting older ones (with dedup turned on). Deleting snapshots with dedup on is a PITA.

4. If you are using SATA drives, instead of SAS drives then you have NCQ instead of TCQ. Too many acronyms? ZFS is configured by default to make great use of SAS drives, and to give SATA drives a headache. Set zfs_vdev_max_pending=1 and zfs_vdev_min_pending=1 using the following commands:

echo "1" > /sys/module/zfs/parameters/zfs_vdev_max_pending
echo "1" > /sys/module/zfs/parameters/zfs_vdev_min_pending

5. raidz doesn’t help you read speed. It’s good for writing and redundancy, but not for reading. It often caps at the speed of the slowest drive. So put mirrored groups inside your raidZ. With six drives, you could run a raidz1 of 3 x 2 mirror sets.

6. Don’t let it fill up!
This one should be in caps: DON’T LET YOUR RAIDZ FILL UP. Performance will drop off a cliff, on many systems it’s effectively an outage at 95%+ drive capacity used. Your disk I/O will fill up with sync processes.

7. L2Arc is great. Use it!

8. Don’t add drives to a raidz after initial creation – The extra capacity isn’t covered by the redundancy! If you have a raid array of 3TB, and you add an extra 1TB drive, you gain another TB of capacity – but this capacity is only located on one drive, and the loss of that drive means the loss of the data stored on it!!

9. zfs destroy -r tank will not only destroy tank, but everything related to it, including snapshots. If you need to make a copy of your data, and destroy the original, then copy the filesystem.

10. This one shouldn’t even need to be listed here. But monitor your ZPOOL status. You want to know if it’s degraded as soon as it’s degraded!


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s