I’ve been increasingly worried about one of the media servers I backup, the backup time has been crawling it’s ways from 12 to about 14 hours as of today. It’s not that it’s so many TB’s, only 1.4 TB actually, the problem is that there is more than 2 000 000 small files, and rsync have to fstat every one of them. So even when there hasn’t been much change the backup takes longer and longer.
So yesterday I figured I should try out zfs. The media server is already on zfs, and I use zfs on the backup server + I have already got a zfs sync script which I modified some months ago to use mbuffer for maximum bandwith utilization .
And how did this turn out ? well, the backup time went from about 14 hours to 7 seconds … I just realised I could never go away from zfs for these kind of setups (backing up millions of files ) .