Backup server advice (linux)

I'm currently starting to redesign our backup server. The machine currently uses rsync and cp -al to keep snapshots of the live servers. It backs up about 10-20 other computers of varying FS structures. There are mail servers, database servers, webservers, even a windows machine or two (using SMB/CIFS to mount, then rsync locally). The current setup involves 4 400MB drives in a RAID 5 setup, with two ext3 partitions (smaller boot/config, larger backup volume) and swap. The backup script launches up to 3 backups at once. During the backup window, doing much of anything (even a simple ps aux) seems to react very slowly on the backup server, and I'm wondering if my problems aren't mostly filesystem related.

Looking at the redesign phase, we have some newer hardware (2TB Caviar drives, i7's, etc) to use. My thoughts so far include: Using RAID 1 on the 2TB drives to get rid of any potential performance issues with RAID 5. I was assuming that I would partition that 2TB drive into a smaller boot/config partition, and then a much larger data section. I'd like the backup process to happen as quickly as possible of course. It leads me to a few questions.

• Will I see better performance if I put the main system and swap partitions on a separate drive from /backup? I could use a 4 drive (2 RAID 1 arrays) setup with a smaller boot/config volume. Will this give me any major advantages?
• Should I use ext4, ReiserFS, XFS, or something else entirely for the backup data partition?
• Is there some other backup method that someone would suggest I research so long as I'm already re organizing my backup server?

From serverfault gnarf

• Will I see better performance if I put the boot and swap partitions on a separate drive?

  I doubt /boot being on the same drive hurts you. After the system is booted, that volume is pretty much never touched unless you are updating the kernel.

  It might be a good idea to have swap somewhere else, but ideally you will have enough RAM that you won't be using swap at all.

  Should I use ext4, ReiserFS, XFS, or something else entirely for the backup data partition?

  I believe ext2 is the fastest choice, if you are sure that the issue is the filesystem.

  I doubt the issue is related to the filesystem though. I suspect part of the problem is that you are trying to run three backups at once. You are probably thrashing the disks quite a bit as it is trying to write files all over the place. I suspect your are having to frequently seek and you aren't getting any sequential writes going on.

  You might be a lot better off if you run your backups sequentially. If you must run the backups at the same time I would suggest you consider having separate drives (not partitions) for each backup.

  gnarf : Sorry, my first question was worded badly, I was talking about putting everything but the backup data on its own physical drive. /var /usr /etc, not just /boot. And I doubt I'll touch swap either.

  Zoredache : Ah, well I still think the slow performance has more to do with the parallel backups then anything else.
  From Zoredache

• As for backup since you are using rsync already I suggest implimenting rsnapshot.

  http://rsnapshot.org/

  If you are using rhel/centos you can install via 'yum install rsnapshot'. A helpful tutorial is here:

  http://wiki.centos.org/HowTos/RsnapshotBackups

  From Bernie

• Use XFS, definitely. Nothing else even comes close to matching it's levels of performance (and reliability - xfs_repair on large filesystems works amazingly well when you need it to)

  If you only have 4 disks, then just RAID5 them with software RAID (you will not take a performance hit in this application, and the additional spindles will greatly help performance than if you were to split the disks)

  I highly recommend this method of doing backups. We have a 200TB XFS filesystem which we use to host our rsync backups on. We have a lot more disk behind it than in your setup, but you should still be able to run a couple of simultaneous rsyncs.

  From James