Large network file copy
-
As part of nightly backup job, we get a copy of the production database to one of the development testing machines. The database backup file size is little bit over 8GB, using RoboCopy it takes us close to 8 hours to copy. I am looking an alternate file copy utility that can copy large files fast. Googling for solution I came across Aaron Maxwell's suggestion.[^]. I am testing Roadkil's Unstoppable Copier [^] and TeraCopy[^]. Both tools claim that they can copy the file over the network in 4 hours, which is 50% faster. Do you copy large files over the network regularly? what tool(s) do you use? Any recommendations?
Yusuf May I help you?
That is super slow. We copy database backups totaling around 150-200GB to a backup server and restore the databases, then copy and restore the log files, some around 3-4GB, made since the last full backup all over 100MBps network and it's done around 12-15 hours later. Not sure why yours is so slow. I'm not a network guy, but maybe you have a bad NIC, wiring, running in half duplex mode? Someone more knowledgeable may know what to check. CT
-
As part of nightly backup job, we get a copy of the production database to one of the development testing machines. The database backup file size is little bit over 8GB, using RoboCopy it takes us close to 8 hours to copy. I am looking an alternate file copy utility that can copy large files fast. Googling for solution I came across Aaron Maxwell's suggestion.[^]. I am testing Roadkil's Unstoppable Copier [^] and TeraCopy[^]. Both tools claim that they can copy the file over the network in 4 hours, which is 50% faster. Do you copy large files over the network regularly? what tool(s) do you use? Any recommendations?
Yusuf May I help you?
Have you tried compressing the backup file before the copy?
-
That is super slow. We copy database backups totaling around 150-200GB to a backup server and restore the databases, then copy and restore the log files, some around 3-4GB, made since the last full backup all over 100MBps network and it's done around 12-15 hours later. Not sure why yours is so slow. I'm not a network guy, but maybe you have a bad NIC, wiring, running in half duplex mode? Someone more knowledgeable may know what to check. CT
CincDev wrote:
Not sure why yours is so slow. I'm not a network guy, but maybe you have a bad NIC, wiring, running in half duplex mode? Someone more knowledgeable may know what to check.
Good points, will check. Thanks
Yusuf May I help you?
-
As part of nightly backup job, we get a copy of the production database to one of the development testing machines. The database backup file size is little bit over 8GB, using RoboCopy it takes us close to 8 hours to copy. I am looking an alternate file copy utility that can copy large files fast. Googling for solution I came across Aaron Maxwell's suggestion.[^]. I am testing Roadkil's Unstoppable Copier [^] and TeraCopy[^]. Both tools claim that they can copy the file over the network in 4 hours, which is 50% faster. Do you copy large files over the network regularly? what tool(s) do you use? Any recommendations?
Yusuf May I help you?
-
Have you tried compressing the backup file before the copy?
No, but I am not sure how much difference it may make. It is database back up file, and I am assuming it may be somewhat compressed. :~
Yusuf May I help you?
-
No, but I am not sure how much difference it may make. It is database back up file, and I am assuming it may be somewhat compressed. :~
Yusuf May I help you?
Compress can make a difference even on database backups, if they don't contains binary data, however it's not worth it in my opinion. We used to compress our backups, copy them, decompress them and then restore them, which took longer than just copying the decompressed file. It wasn't worth the time or drive space since you have to have plenty of room for the compressed file and the temporary file during decompression. Some of our databases are 21GB or more, so that's 40GB+ to decompress the file, and the loser who built our server got the smallest possible drives he could find, so we were always running low on space. Thank God we are getting another new server with a disk array soon. :) Anyway, don't worry about compression unless you plan to leave them compressed and not restore them. CT
-
Compress can make a difference even on database backups, if they don't contains binary data, however it's not worth it in my opinion. We used to compress our backups, copy them, decompress them and then restore them, which took longer than just copying the decompressed file. It wasn't worth the time or drive space since you have to have plenty of room for the compressed file and the temporary file during decompression. Some of our databases are 21GB or more, so that's 40GB+ to decompress the file, and the loser who built our server got the smallest possible drives he could find, so we were always running low on space. Thank God we are getting another new server with a disk array soon. :) Anyway, don't worry about compression unless you plan to leave them compressed and not restore them. CT
:-D
Yusuf May I help you?
-
Thanks Rob, I'll check it out.
Yusuf May I help you?
-
No, but I am not sure how much difference it may make. It is database back up file, and I am assuming it may be somewhat compressed. :~
Yusuf May I help you?
We use Red-Gate backup on SQL 2005 and the shrinkage is impressive around 85%, most(all) of our data is text.
-
As part of nightly backup job, we get a copy of the production database to one of the development testing machines. The database backup file size is little bit over 8GB, using RoboCopy it takes us close to 8 hours to copy. I am looking an alternate file copy utility that can copy large files fast. Googling for solution I came across Aaron Maxwell's suggestion.[^]. I am testing Roadkil's Unstoppable Copier [^] and TeraCopy[^]. Both tools claim that they can copy the file over the network in 4 hours, which is 50% faster. Do you copy large files over the network regularly? what tool(s) do you use? Any recommendations?
Yusuf May I help you?
I use Rsync for Windows. Once the initial backup is complete Rsync backs up only those portions of the file that have changed - ideal for WAN / Internet based backups. Plus the software is free. The implementation I use is called DeltaCopy. I replicate about 50 gigs of data via a 512 kbps uplink every night...takes about 90 minutes. This includes two Exchange databases and a few small SQL databases. Again, only the changes to the files are moved. Rsync originated in the Unix world and as with many Unix utilities is a command line program with all kinds of parameters - but it's no more complicated than RoboCopy. DeltaCopy wraps Rsync in a simple GUI. http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp[^]
-
I use Rsync for Windows. Once the initial backup is complete Rsync backs up only those portions of the file that have changed - ideal for WAN / Internet based backups. Plus the software is free. The implementation I use is called DeltaCopy. I replicate about 50 gigs of data via a 512 kbps uplink every night...takes about 90 minutes. This includes two Exchange databases and a few small SQL databases. Again, only the changes to the files are moved. Rsync originated in the Unix world and as with many Unix utilities is a command line program with all kinds of parameters - but it's no more complicated than RoboCopy. DeltaCopy wraps Rsync in a simple GUI. http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp[^]
Gary, DeltaCopy looks intersting, I'll give it a test drive. Thanks
Yusuf May I help you?
-
As part of nightly backup job, we get a copy of the production database to one of the development testing machines. The database backup file size is little bit over 8GB, using RoboCopy it takes us close to 8 hours to copy. I am looking an alternate file copy utility that can copy large files fast. Googling for solution I came across Aaron Maxwell's suggestion.[^]. I am testing Roadkil's Unstoppable Copier [^] and TeraCopy[^]. Both tools claim that they can copy the file over the network in 4 hours, which is 50% faster. Do you copy large files over the network regularly? what tool(s) do you use? Any recommendations?
Yusuf May I help you?
have you tried automated ftp scripts using scheduled tasks? you might have to take the db down, copy to another file, bring db backup and do it. or in an sql task, run a backup, do an ftp of 2 files (the db backup and a seperate empty text file AFTER the db backup gets uploaded) then on the remote server have something running looking for that empty text file, which would then do a restore. Thats how did our backups, granted it was just to wait for windows backups before starting the tape backups.
Those who are too smart to engage in politics are punished by being governed by those who are dumber. - Aristotle