Backups for the hosted servers
Backup is done by using rdiff-backup to sync the contents of each machine from RAL3 to a virtual machine hosted within one of the Red Hat data centers.
What is backed up
We sync the contents of the machine from /, excluding files based on /etc/rsync.d/backup.exclude. The contents of this file are built from the puppet configuration. A typical node configuration looks like:
$backup_exclude = [
"/cvs", # nfs mount
"/git", # bind mount
"/mnt/git-data/git/preview", # scratch data
"/mnt/git-data/svn", # rsync from svn.gnome.org
]
include backupThings that are typically excluded for a particular node might include:
- Bind bounds of data that is also present elsewhere on the system
- Large scratch data and other things that don't need backup
- Any files that rapidly change, like an outgoing mail queue; such files will cause backup errors
In addition, the standard configuration excludes special file systems such as /proc, /selinux and /sys, temporary directories such as /tmp, and the standard NFS mounts /home/admin and /home/users. Any other NFS mounts have to be added to the node configuration.
Recovery
For recovery of files, please open an issue. Our backups are currently happening both incrementally and historically, that allow us to recover a file that was modified or deleted an X number of days ago.
We've never had to recover an entire system from backup. The plan would be to do a fresh install onto the replacement machine, and then sync the backup into a subdirectory of that system (or onto a different machine) and compare and put stuff back into place from there. Restoring the entire system image onto an empty drive is likely going to be harder.
Monitoring backups
The rdiff-backup logs are mailed to backups@gnome.org every night and then checked manually by the GNOME Infrastructure Team. The logs are being generated with the --print-statistics rdiff-backup's flag that provides several stats for each of the generated backups.