Backups for the hosted servers
Backup is done by using rdiff-backup to sync the contents of each machine from Phoenix2 (from the London Canonical datacenter and RDU2) to the gnome-backup01.phx2.fedoraproject.org machine kindly hosted by the Fedora Infrastructure Team.
What is backed up
We sync the contents of the machine from /, excluding files based on /etc/rsync.d/backup.exclude. The contents of this file are built from the puppet configuration. A typical node configuration looks like:
$backup_exclude = [ "/cvs", # nfs mount "/git", # bind mount "/mnt/git-data/git/preview", # scratch data "/mnt/git-data/svn", # rsync from svn.gnome.org ] include backup
Things that are typically excluded for a particular node might include:
- Bind bounds of data that is also present elsewhere on the system
- Large scratch data and other things that don't need backup
- Any files that rapidly change, like an outgoing mail queue; such files will cause backup errors
In addition, the standard configuration excludes special file systems such as /proc, /selinux and /sys, temporary directories such as /tmp, and the standard NFS mounts /home/admin and /home/users. Any other NFS mounts have to be added to the node configuration.
NOTE that adding the backup class to a node in puppet does not result it being backed up automatically; Make sure that the machine gets added into the backup script available on the Fedora Infrastructure's Ansible Repository.
For recovery of files, mail email@example.com which is an alias to both the GNOME and Fedora sysadmins with access to the backup machine. Our backups are currently happening both incrementally and historically, that allow us to recover a file that was modified or deleted an X number of days ago.
We've never had to recover an entire system from backup. The plan would be to do a fresh install onto the replacement machine, and then sync the backup into a subdirectory of that system (or onto a different machine) and compare and put stuff back into place from there. Restoring the entire system image onto an empty drive is likely going to be harder.
The rdiff-backup logs are mailed to firstname.lastname@example.org every night and then checked manually by the GNOME Sysadmin Team. The logs are being generated with the --print-statistics rdiff-backup's flag that provides several stats for each of the generated backups.