Backups

Help with using the Webarchitects backup space.

Accessing the backups

You can ssh to the backup server and find the backups in .zfs/snapshot/ but due to shell restrictions it is easier to use SSHFS to mount the backups on another machine and then browse them.

First set up SSH, generate a key pair for root (this can also be done as a regular user on a local machine but when mounting the backups on to a server to find things it is more convenient to do it as root):

sudo -i
ssh-keygen -t rsa -b 4096

Create /root/ssh/.config containing:

Host *
  CompressionLevel 9
  CheckHostIP yes
  StrictHostKeyChecking ask

Host backup
  Hostname store1.webarch.net
  User XXX

When XXX is your username.

Test the connection:

ssh backup

Install SSHFS, for example:

aptitude install sshfs

Mount the filesystem:

mkdir -p /media/backups
chmod 700 /media/backups
echo "sshfs#backup: /media/backups fuse ro 0 0" >> /etc/fstab
mount -a

Backing up servers

We use a simple bash script and config file to backup files to the Webarchitects backup space.

To use this script first set up SSH as above, then create a /etc/agile/backup file containing a list of directories to backup, one per line, no leading slashes, for example:

/etc
/root
/home
/var/backups
/var/spool/cron/crontabs

Copy the following to /usr/local/bin/agile-backup and chomd 750 it.

agile-backup

#!/usr/bin/env bash

# List of local directories to be backed up
# is contained in this file, one per line 
# eg
# /etc
# /root
# /home
# /var/backups
# /var/spool/cron/crontabs
BACKUP_LIST="/etc/agile/backup"
# The hostname of this server, used to create 
# a directory on the remote server
HOSTNAME=$(hostname)
# The server to backup to, this is set in 
# /root/.ssh/config
SSH_SERVER="backup"
# This script uses lockfile which comes with procmail
LOCKFILE_BINARY="/usr/bin/lockfile"
LOCKFILE="/var/run/lock/$(basename $0).lock"
# Write the outcome to this log
LOGFILE="/var/log/$(basename $0)"

# Check that the script is being run by root
if [[ "$(id -u)" != "0" ]] ; then
  echo "You must run '$0' as root or via sudo" 
  exit 1
fi

# Test if the lockfile binary can be found
if [[ ! -e "$LOCKFILE_BINARY" ]]; then
  echo "$LOCKFILE_BINARY not found, please install the procmail package."
  exit 1
fi

# if the $LOCKFILE exists then exit
# the lockfile is read only
# the timeout is set to 2 hours (7200 secs)
# if the lockfile is older than this it will be 
# removed, this need to be improved so the log is updated
$LOCKFILE_BINARY -r 1 -l 7200 $LOCKFILE || exit 23

# Do the backup, first test that the file
# with the list of files to be backed up
# exists
if [[ -f ${BACKUP_LIST} ]]; then
  for dir in $(<${BACKUP_LIST}); do
    # Make the directory on the backup server
    ssh ${SSH_SERVER} mkdir -p ${HOSTNAME}${dir}
    # Copy the files to the backup server
    # Test for the scripts has been run with -v
    # for verbose output
    # bwlimit is set to 13 KBPS which is approx 0.1 megabit
    # so in the event of 10 backups running at the same time
    # on 10 servers the max bandwidth usage would be 1 megabit
    if [[ $1 == "-v" ]]; then
      rsync -av --delete --bwlimit=13 ${dir}/ ${SSH_SERVER}:${HOSTNAME}${dir}/
    else
      rsync -aq --delete --bwlimit=13 ${dir}/ ${SSH_SERVER}:${HOSTNAME}${dir}/ 2>&1
    fi
    # catch errors
    # http://idolinux.blogspot.co.uk/2008/08/bash-script-error-handling.html
    if [ $? != 0 ]; then
    {
      echo "Backup Error at $(date +%c)" >> $LOGFILE
      if [[ $1 == "-v" ]]; then
        rm -vf $LOCKFILE
      else
        rm -f $LOCKFILE
      fi
      exit 23
    } fi
  done
else
  echo "$BACKUP_LIST doesn't exist"
  echo "Backup Error at $(date +%c)" >> $LOGFILE
  if [[ $1 == "-v" ]]; then
    rm -vf $LOCKFILE
  else
    rm -f $LOCKFILE
  fi
  exit 23
fi

# Remove the lock file and update the log 
if [[ $1 == "-v" ]]; then
  rm -vf $LOCKFILE
else
  rm -f $LOCKFILE
fi
echo "Backup Success at $(date +%c)" >> $LOGFILE

The script can then be run to test it is working:

agile-backup -v

If is is then add a crontab via crontab -e:

45 00 * * * /usr/local/bin/agile-backup

Note that it if you want to backup MySQL databases you can use backupninja to dump the MySQL databases as plain text and then backup the directory the dumps are in.

If you use the script about then you can mount the backups onto the server like this:

mkdir -p /media/backups
chmod 700 /media/backups/
mkdir /media/backups/latest
mkdir /media/backups/archive
echo "sshfs#backup:$(hostname) /media/backups/latest fuse ro 0 0" >> /etc/fstab
echo "sshfs#backup:.zfs/snapshot /media/backups/archive fuse ro 0 0" >> /etc/fstab
mount -a