Backing up a Linux server using FreeNAS – harder than you would think

I recently finished building a new FreeNAS machine, my first actually. One of the goals for the machine was to use it to backup my servers. My requirements were the following:

• It needs to be able to backup an entire Linux server ( / ) but some paths needs to be excluded like /dev and /proc.
• It should not put too much strain on the server that is being backed up.
• The FreeNAS server needs to initiate the backup. It’s behind NAT and I don’t want to open up any ports.
• It would be very beneficial if the UID and GID of the files and folders are kept. So this is not strictly a requirement but I very much would like this feature.

RSync Tasks

My first thought was to use RSync since I know it can meet all of my requirements. FreeNAS has an option called RSync Tasks right in the menu so it seemed like the perfect choice.

But there are some issues. First issue is that RSync is so eager to backup the server it puts quite a lot of load on it. So much load it stops responding to web traffic for a couple of minutes. Which is not acceptable. You can solve this by using --rsync-path="ionice -c 3 nice -n 12 rsync". So that’s another thing you have to add to the Extra options field, manageable but annoying.

The real deal breaker for RSync Tasks is that since I backup everything on the server (or almost) I backup a lot of temporary files. A lot of these files are often deleted before RSync has had the time to transfer them which means it outputs this error message:

file vanished: /path/to/file

This is not something you can’t ignore with a RSync option. The official solution is to put RSync in a wrapper script and then catch the output and ignore it. I can’t really do that with RSync Tasks in FreeNAS. This causes a lot of false positive notifications which is not acceptable so onto the next candidate!

Cloud Sync

Next I turned my eyes towards Cloud Sync which sounds promising. Most of the options here are for specific third-party services like Amazon and Dropbox but there is SFTP support which could be used as a replacement for RSync.

Problem here is that I could not get excludes to work correctly. Since I backup / I want to exclude paths like /proc and /dev. It just ignores the excluded paths I enter into the exclude field. I was never able to find a solution for this, I did ask on the FreeNAS forum but never got a response.

Second issue is that SFTP won’t keep the GID and UID of the files and folders. I guess I could live with that but it would be really nice not to have to deal with that when doing a restore.

I didn’t want to do it but I guess it’s time to try an old fashioned cron job!

RSync cron job

Running RSync as a normal cron job would give me the ability to ignore the file has vanished error message so logically it should solve the deal breaker with RSync Tasks.

But right away it started to behave weird. It started the backup fine but it was taking forever to finish. Looking closer I found it was stuck backing up /dev/urandom despite /dev being excluded. Are you BLEEP kidding me!? At this point I was getting pretty tired of this.

After some more bleeps I found out that the exclude syntax, --exclude={"/path","/path"}, I was using was being ignored. Since I have quite a few directories to exclude I thought it would be easier than using a --exclude=/path for every directory.

If I use the --exclude={} syntax on the command line via SSH it works fine. Again I have no idea why and I did post a question in the FreeNAS forum but we never got to the bottom of it.

But at this point I have a solution that actually works the way I want it to. The RSync command might stretch the entire width of my three monitors but it does work. So I thought if I’m going this far why not just write a bash script that takes care of most of the options and exposes only what I need as command line options. So I did write one and here it is.

#!/bin/bash
set -u

IGNORE_OUTPUT='^(file has vanished: |rsync warning: some files vanished before they could be transferred)'
DEFAULT_EXCLUDES="*.sock,/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,/swapfile,/var/cache/*,/var/tmp/*"
CUSTOM_EXCLUDES=
FROM_IP=
RS_LOG_FILE=
VERBOSE=
SOURCE=
DESTINATION=
RS_OUTPUT=$(mktemp)
RSYNC_PATH=$(which rsync)
RS_COMMAND=()

die () {
  printf "%s\n" "${1-}"
  exit 1
}

exclude () {
  while read -r ROW; do
    RS_COMMAND+=(--exclude="$ROW")
  done < <(echo "$1" | tr -s ',' '\n')
}

# Get command lines option
while getopts "vl:i:e:s:d:" opt; do
  case "$opt" in
    l) RS_LOG_FILE="$OPTARG";;
    i) FROM_IP="$OPTARG";;
    e) CUSTOM_EXCLUDES="$OPTARG";;
    s) SOURCE="$OPTARG";;
    d) DESTINATION="$OPTARG";;
    v) VERBOSE=1;;
    :) die;;
    ?) die
  esac
done

# Build the RSync command
RS_COMMAND+=($RSYNC_PATH)
RS_COMMAND+=(--rsync-path="ionice -c 3 nice -n 12 rsync")
RS_COMMAND+=(-aS)
RS_COMMAND+=(--delete-excluded)
[ -z "$RS_LOG_FILE" ] || RS_COMMAND+=(--log-file="$RS_LOG_FILE")
[ -z "$VERBOSE" ] || RS_COMMAND+=(-v)
[ -z "$FROM_IP" ] || RS_COMMAND+=(--address="$FROM_IP")
[ -z "$CUSTOM_EXCLUDES" ] || exclude "$CUSTOM_EXCLUDES"
exclude "$DEFAULT_EXCLUDES"
RS_COMMAND+=($SOURCE)
RS_COMMAND+=($DESTINATION)

# Run the command
"${RS_COMMAND[@]}" &> "$RS_OUTPUT"

# Handle output
if [ "$?" != "0" ]; then
  if egrep -q "$IGNORE_OUTPUT" "$RS_OUTPUT"; then
    rm "$RS_OUTPUT"
    exit 0
  else
    cat "$RS_OUTPUT"
    die "Command: ${RS_COMMAND[*]}"
  fi
fi

Some of the features to highlight:

• A lot of paths are excluded as default so I don’t have to enter the same paths for every cron job.
• It uses nice and ionice so that it won’t overload the server.
• It handles the output from RSync so that I won’t get notified about “file has vanished”.
• If something does go wrong it will print some debug info to help you troubleshoot.

So instead of an extremely long cron command I most of the time just enter the source and destination.

/root/scripts/rsync.sh -s "source.server:/" -d "/path/to/local/destination/folder"

If I want to exclude a path that isn’t excluded as default I just add the -e option.

/root/scripts/rsync.sh -s "source.server:/" -d "/path/to/local/destination/folder" -e "/path/to/exclude/*,/second/path/to/exclude/*"

Keep in mind that you need to uncheck Hide Standard Output and Hide Standard Error in FreeNAS when you create the cron job, the script will handle the output.

Feel free to suggest changes to the script in the comments below. I will keep this blog post updated when I make changes to the script.

SSH keys

If you are going to use this script you of course need to setup SSH keys for each server you want to backup. I wanted to use the System -> SSH Keypairs function in FreeNAS but couldn’t find a way to access those keys from the command line (maybe someone knows and can leave a comment). So I had to setup my own keys and config in /root/.ssh. But I did write a script for that also which you can use to make the setup process easier.

#!/bin/bash
set -u

SERVER="$1"
KEY_PATH="/root/.ssh/$SERVER"

if [ -z "$SERVER" ]; then
  printf "You need to specify a server\n"
  exit 1
fi

ssh-keygen -q -N '' -f "$KEY_PATH" -t rsa -b 4096

cat << EOF >> /root/.ssh/config
Host $SERVER
  HostName $SERVER
  User root
  Port 22
  IdentityFile $KEY_PATH
EOF

cat "${KEY_PATH}.pub"

You use it like this.

/root/scripts/setup_ssh.sh server.to.backup

And then you just copy the public key and add it to the server you wish to backup.