10 ZFS on Ubuntu/Linux Learnings

I thought I’d go through a few learnings from running the ZOL package on Ubuntu. Some are general observations on zfs, some are ZOL specific, and some are just issues to avoid. For those who aren’t familiar with ZFS, it’s a filesystem capable of storing incredible quantities of data. It’s able to detect and fix bit rot, it’s able to do a whole swathe of cool tricks such as live snapshots, deduplication, and compression.

These are in no particular order.

1. If you can, and you have data you want to treat differently, split them up as low as you can on the pool’s structure. It’s easy to configure one part of the pool to have deduplication or one part to have compression with a particular algorithm. It’s a bad idea to turn it one for a whole pool of heterogeneous data. Deduplication in general, is a performance killer. If you turn it on, do it for a small set of highly duplicated data, and it’s a valuable feature. If you turn it on for your whole pool, then everything from deleting snapshots, to writing large, unduplicated datasets becomes a huge chore. The same goes for compression.

2. Less is often more – choose the best feature, and use just that feature. Either compression or deduplication. Don’t go overboard with snapshots either, or disable them for data storage that has high turnover. Otherwise your snapshots will bloat out to many times the size of the base data at any point in time.

3. Deleting snapshots that are more recent, seems quicker than deleting older ones (with dedup turned on). Deleting snapshots with dedup on is a PITA.

4. If you are using SATA drives, instead of SAS drives then you have NCQ instead of TCQ. Too many acronyms? ZFS is configured by default to make great use of SAS drives, and to give SATA drives a headache. Set zfs_vdev_max_pending=1 and zfs_vdev_min_pending=1 using the following commands:

echo "1" > /sys/module/zfs/parameters/zfs_vdev_max_pending
echo "1" > /sys/module/zfs/parameters/zfs_vdev_min_pending

5. raidz doesn’t help you read speed. It’s good for writing and redundancy, but not for reading. It often caps at the speed of the slowest drive. So put mirrored groups inside your raidZ. With six drives, you could run a raidz1 of 3 x 2 mirror sets.

6. Don’t let it fill up!
This one should be in caps: DON’T LET YOUR RAIDZ FILL UP. Performance will drop off a cliff, on many systems it’s effectively an outage at 95%+ drive capacity used. Your disk I/O will fill up with sync processes.

7. L2Arc is great. Use it!

8. Don’t add drives to a raidz after initial creation – The extra capacity isn’t covered by the redundancy! If you have a raid array of 3TB, and you add an extra 1TB drive, you gain another TB of capacity – but this capacity is only located on one drive, and the loss of that drive means the loss of the data stored on it!!

9. zfs destroy -r tank will not only destroy tank, but everything related to it, including snapshots. If you need to make a copy of your data, and destroy the original, then copy the filesystem.

10. This one shouldn’t even need to be listed here. But monitor your ZPOOL status. You want to know if it’s degraded as soon as it’s degraded!

Installing OpenSSH 6.6p1 on Ubuntu 13.10 “Saucy”

Ubuntu 14.04 includes a new version of OpenSSH. Version 6.2, which is present in Saucy has a vulnerability which potentially could lead to it being exploited. If you are doing PCI compliance, then fixing this is a must!

Luckily, replacing OpenSSH is pretty easy. This install assumes you are using 13.10, and that you already have OpenSSH installed and configured, and that you have already installed the Ubuntu 13.10 build dependencies. If you’ve ever compiled something from source, then you will have this. If you don’t, then useĀ  Continue reading

The following packages have been kept back

If you’ve ever seen the words “The following packages have been kept back” you’ll know it can be pretty frustrating. You’ve told it to update, why isn’t it updating?

This occurs because the package has had it’s dependencies changed. It’s either going to install more or uninstall software the new version doesn’t need. A lot of replies will tell you to do a dist-upgrade.

This is a very bad idea, unless you know what you are doing. This will cause a LOT of changes to your system, and it’s not massively unusual to see it prevent a system running until you sort out a raft of post dist-upgrade issues. Now, some people will argue that you should always dist-upgrade, and deal with issues as they crop up, and while there’s merit to this, you can’t do it on a production system, especially just because you need to upgrade a package.

What’s the solution?


apt-get update
apt-get dselect-upgrade

This will then follow up with the usual explanation of which packages will be added or removed. Type in “y” like you normally do, and it will install/uninstall/upgrade your packages. Done.

Ubuntu 12.04 Precise Pangolin – Complete VNC Server Setup

There are lots of guides for setting up VNC Server on 12.04, so why write another one?
I had my own requirements for setting up VNC:

1. It had to be configurable “headless” on a server or desktop ubuntu install via SSH.
2. I prefer to use vnc4server
3. You need to install gnome-session in 12.04 (in previous unity distros, it was pre-installed IIRC)

Without covering all the above points, a VNC install walkthrough is pretty pointless. There are plenty of prettier, easier, GUI based install guides. This assumes terminal, VNC and X11 knowledge. If you somehow break your config, you need to be able to work it out for yourself or with help from the ubuntu community.

First things first, make sure you have these packages, as they are essential to get this working. If you get a screen when you try and connect via VNC which only allows you to log out, it’s probably because you skipped these earlier steps.

sudo apt-get update
sudo apt-get install gnome-core gnome-session-fallback

Obviously this updates your package list to the latest 12.04 releases. You can optionally “sudo apt-get upgrade”, to keep your ubuntu up to date.
Now we get into install a VNC server and configuring it.

sudo apt-get install vnc4server
This installs the VNC server software we are going to use.

vncserver
vncserver -kill :1
cp .vnc/xstartup .vnc/xstartup.bak
nano .vnc/xstartup

The first line fires up VNC server with the default config. We do this because it will prompt you to enter a password. Bear in mind that passwords longer than 8 characters will be truncated to 8 characters :/ We’ll cover the security implications in a bit.
The second line kills the default VNC session.
The third line can be omitted, but it’s traditional IT paranoia.
The fourth line opens nano and allows you to edit the config for VNC server.

You need to edit the file to make it look like this:
#!/bin/sh

# Uncomment the following two lines for normal desktop:
unset SESSION_MANAGER
#exec /etc/X11/xinit/xinitrc
gnome-session –session=gnome-classic &

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
#x-terminal-emulator -geometry 80×24+10+10 -ls -title “$VNCDESKTOP Desktop” &
#x-window-manager &

The changes of note are removing the # in front of unset SESSION_MANAGER (making that line active) and adding the “gnome-session –session=gnome-classic &” line, before commenting out (with hash) the x-terminal-emulator and x-window-manager lines. If you don’t do this, your VNC session will start, but it will be a blank grey hatched background.

EDIT: Make sure your file looks like this. WordPress is turning my double-dash into a single longer dash by the look of it, and triple dash doesn’t seem to fix it:Screen Shot of VNC4 login configuration

Save your changes to the file, and exit (in nano, it’s ctrl-x).

Now, we killed the VNC server session that was running before. So recreate it using the screen size you want. I use 1024×768 on my iPhone to reduce bandwidth, but that might not suit what you want to do. You can add -depth 24 for more colours, or -depth 8 to force 256 colour mode which is required for some VNC clients especially on mobile phones:
vncserver -geometry 1680x1050

so a good phone config might be:
vncserver -geometry 1024x768 -depth 8

iSSH connecting

iSSH connecting

iSSH connected to Ubuntu 12.04 Precise Pangolin via VNC over SSH

iSSH connected to Ubuntu 12.04 Precise Pangolin via VNC over SSH

If you get a grey screen when you try and connect tunnelling over SSH, don’t panic. Try using port 5902, because you might not have killed the initial session properly, or restarted a default one by mistake. If you don’t know what I mean by tunnelling, you might find that you can’t connect to the VNC server. The problem is the built-in firewall, ufw.

You can use the following code to see if ufw is enabled, and what it’s rules are:
sudo ufw status

If it’s enabled we need to tell it that it’s safe to allow VNC connections to the machine. But before we do that, a shot digression on security:

Hopefully your computer is behind a firewalled NAT. Most are these days, but if you are connecting directly to a modem, with no other computers connected to it, this might not be safe. As the maximum password length is 8 characters, there is a very finite number of passwords that exist. In theory, a hacker could see that your VNC port is open, and start guessing your password using software that will keep trying combinations until it gets in. After we open a hole for VNC in the firewall, we will test whether it’s visible to the internet.

Okay, so to open a hole in the firewall we need to add a rule that allows connections on the correct port, which are initiated from outside your computer. Obviously the computer doesn’t dial out to your phone, and command it to connect via VNC, the phone or other external device is the instigator of the connection, and this is what a firewall is designed to prevent.

sudo ufw allow 5901
This tells ufw to allow connections for screen0 and screen1 over VNC to any outside computer.

Now we will test whether there is another firewall in between your computer and the internet.

Go to the ShieldsUp! site, and click “proceed” enter “5900-5902” into the blank box then “user specified custom port probe” button below. If it says “Stealth” in green for each port, then your system is secure.

ShieldsUp! Scan of VNC ports - Stealthed!

If you don’t get the green “Stealth” next to each port, you have opened a potential vulnerability to your system.
Fix this by reversing the firewall rule we made:
sudo ufw deny 5901

Assuming you have SSH access remotely for this machine, you can tunnel the SSH connection from an external machine with ssh -NfL 5901:127.0.0.1:5901 user@remote.host replacing the 5901s with the correct port for the desktop you want to access if necessary. You can now VNC to the server by telling the machine you run the above code on, to connect to itself on the same port. Alternatively, because of the ufw fix, you can connect directly while on the same subnet.

If you use an iOS device, i highly recommend iSSH as it allows you to easily set up an SSH tunnelled VNC connection.

We are nearly done!

Once you reboot, The VNC Server isn’t up any more.

sudo nano /etc/rc.local

Go to the line above “exit 0”.
paste in: “/usr/bin/vncserver -geometry 1024×768 -depth 8” or similar.

You are now done!

EDIT: If you get a grey screen, please read the comments. There is a problem with how wordpress renders double-dashes, apparently turning them into a single dash, which will misconfigure your vnc login data. The other most common problem is that VNC server doesn’t work on startup automatically. Check that you’ve edited your /etc/rc.local correctly, and that you are connecting to the correct port for screen 0. The way VNC server works, the first “screen” is 0, the next is 1, the next is 2. If you are trying to connect to screen 1, then it won’t appear to be running, because the rc.local version will be running on screen 0.