The year, 2006. The problem, we still can't copy files reliably
-
The task: Copy 2,300 files equaling 16gb from one laptop HD to an external HD. This should be easy, just copy the parent directory and paste in the destination directory. This I do. The dialog takes 3 minutes just to appear and then another 5 minutes to calculate how long it is going to copy all the files. It estimates it will take 89 minutes to copy which is fine by me. So I leave it for 30 minutes, come back and find the dialog is now saying 120 minutes. Hmmm. OK, maybe 2,300 files is too much for the copy app. So I decide to ZIP the files (without compression), copy the resulting 13gig file and then unzip on the destination directory. It takes 30 minutes to ZIP, 20 minutes to copy the file across and then I start to unzip. 30 minutes in it is about 15% through, WTF? OK, who cares, just leave it while I am off moving a boat. I come back 3 hours later and my girlfriend, who was told "not to touch the shiny laptop", is pointing at the external HD saying "It is making a clicking sound." Oh bugger. I lift the lid and see a screen full of "Delayed write failure" dialogs. 50 clicks later WinZIP tells me at 97% it failed. *sigh* We have computers more powerful than what launched man to the moon and yet we can't bleeding well reliably copy files from one HD to another.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
-
The task: Copy 2,300 files equaling 16gb from one laptop HD to an external HD. This should be easy, just copy the parent directory and paste in the destination directory. This I do. The dialog takes 3 minutes just to appear and then another 5 minutes to calculate how long it is going to copy all the files. It estimates it will take 89 minutes to copy which is fine by me. So I leave it for 30 minutes, come back and find the dialog is now saying 120 minutes. Hmmm. OK, maybe 2,300 files is too much for the copy app. So I decide to ZIP the files (without compression), copy the resulting 13gig file and then unzip on the destination directory. It takes 30 minutes to ZIP, 20 minutes to copy the file across and then I start to unzip. 30 minutes in it is about 15% through, WTF? OK, who cares, just leave it while I am off moving a boat. I come back 3 hours later and my girlfriend, who was told "not to touch the shiny laptop", is pointing at the external HD saying "It is making a clicking sound." Oh bugger. I lift the lid and see a screen full of "Delayed write failure" dialogs. 50 clicks later WinZIP tells me at 97% it failed. *sigh* We have computers more powerful than what launched man to the moon and yet we can't bleeding well reliably copy files from one HD to another.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
So the OS/X filecopy sucks as much as the Windows version? Or are you slumming... ;P Still, sounds like you're blaming a hardware failure on software. Oh, and since it has to be written, uncompressed, to the external harddrive, you're not really saving anything by zipping it. That only helps when you can move the thing compressed across the bottleneck and then uncompress on the other side.
---- Scripts i’ve known... CPhog 1.8.2 - make CP better. Forum Bookmark 0.2.5 - bookmark forum posts on Pensieve Print forum 0.1.2 - printer-friendly forums Expand all 1.0 - Expand all messages In-place Delete 1.0 - AJAX-style post delete Syntax 0.1 - Syntax highlighting for code blocks in the forums
-
The task: Copy 2,300 files equaling 16gb from one laptop HD to an external HD. This should be easy, just copy the parent directory and paste in the destination directory. This I do. The dialog takes 3 minutes just to appear and then another 5 minutes to calculate how long it is going to copy all the files. It estimates it will take 89 minutes to copy which is fine by me. So I leave it for 30 minutes, come back and find the dialog is now saying 120 minutes. Hmmm. OK, maybe 2,300 files is too much for the copy app. So I decide to ZIP the files (without compression), copy the resulting 13gig file and then unzip on the destination directory. It takes 30 minutes to ZIP, 20 minutes to copy the file across and then I start to unzip. 30 minutes in it is about 15% through, WTF? OK, who cares, just leave it while I am off moving a boat. I come back 3 hours later and my girlfriend, who was told "not to touch the shiny laptop", is pointing at the external HD saying "It is making a clicking sound." Oh bugger. I lift the lid and see a screen full of "Delayed write failure" dialogs. 50 clicks later WinZIP tells me at 97% it failed. *sigh* We have computers more powerful than what launched man to the moon and yet we can't bleeding well reliably copy files from one HD to another.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
Paul Watson wrote:
"Delayed write failure" dialogs
Paul, you may want to disable write caching on the external drive. While this will make the copy take longer, it'll hopefully make it more reliable. /ravi
This is your brain on Celcius Home | Music | Articles | Freeware | Trips ravib(at)ravib(dot)com
-
The task: Copy 2,300 files equaling 16gb from one laptop HD to an external HD. This should be easy, just copy the parent directory and paste in the destination directory. This I do. The dialog takes 3 minutes just to appear and then another 5 minutes to calculate how long it is going to copy all the files. It estimates it will take 89 minutes to copy which is fine by me. So I leave it for 30 minutes, come back and find the dialog is now saying 120 minutes. Hmmm. OK, maybe 2,300 files is too much for the copy app. So I decide to ZIP the files (without compression), copy the resulting 13gig file and then unzip on the destination directory. It takes 30 minutes to ZIP, 20 minutes to copy the file across and then I start to unzip. 30 minutes in it is about 15% through, WTF? OK, who cares, just leave it while I am off moving a boat. I come back 3 hours later and my girlfriend, who was told "not to touch the shiny laptop", is pointing at the external HD saying "It is making a clicking sound." Oh bugger. I lift the lid and see a screen full of "Delayed write failure" dialogs. 50 clicks later WinZIP tells me at 97% it failed. *sigh* We have computers more powerful than what launched man to the moon and yet we can't bleeding well reliably copy files from one HD to another.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
Um.... The year is 2007 :rolleyes:
-
Um.... The year is 2007 :rolleyes:
-
So the OS/X filecopy sucks as much as the Windows version? Or are you slumming... ;P Still, sounds like you're blaming a hardware failure on software. Oh, and since it has to be written, uncompressed, to the external harddrive, you're not really saving anything by zipping it. That only helps when you can move the thing compressed across the bottleneck and then uncompress on the other side.
---- Scripts i’ve known... CPhog 1.8.2 - make CP better. Forum Bookmark 0.2.5 - bookmark forum posts on Pensieve Print forum 0.1.2 - printer-friendly forums Expand all 1.0 - Expand all messages In-place Delete 1.0 - AJAX-style post delete Syntax 0.1 - Syntax highlighting for code blocks in the forums
This was Windows XP :) And in the past I have found Windows handles fewer larger files better than many thousands of smaller files. So ZIPing has often helped. The bottleneck seems to be in the handling routines not the actual copy routine. If I hadn't zipped it the dialog would probably be on 999999999999999 minutes left by now ;) And sure, the external HD did fail in the end but now instead of me being able to say "OK, continue from the failed point" I have to start all over again. Well done software. The external HD is fine now it seems. It probably just got bored and took its eye off the ball.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
-
The task: Copy 2,300 files equaling 16gb from one laptop HD to an external HD. This should be easy, just copy the parent directory and paste in the destination directory. This I do. The dialog takes 3 minutes just to appear and then another 5 minutes to calculate how long it is going to copy all the files. It estimates it will take 89 minutes to copy which is fine by me. So I leave it for 30 minutes, come back and find the dialog is now saying 120 minutes. Hmmm. OK, maybe 2,300 files is too much for the copy app. So I decide to ZIP the files (without compression), copy the resulting 13gig file and then unzip on the destination directory. It takes 30 minutes to ZIP, 20 minutes to copy the file across and then I start to unzip. 30 minutes in it is about 15% through, WTF? OK, who cares, just leave it while I am off moving a boat. I come back 3 hours later and my girlfriend, who was told "not to touch the shiny laptop", is pointing at the external HD saying "It is making a clicking sound." Oh bugger. I lift the lid and see a screen full of "Delayed write failure" dialogs. 50 clicks later WinZIP tells me at 97% it failed. *sigh* We have computers more powerful than what launched man to the moon and yet we can't bleeding well reliably copy files from one HD to another.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
Here's a link[^] I've used this software for copying large sets of files, and it is fantastic. Marc
People are just notoriously impossible. --DavidCrow
There's NO excuse for not commenting your code. -- John Simmons / outlaw programmer
People who say that they will refactor their code later to make it "good" don't understand refactoring, nor the art and craft of programming. -- Josh Smith -
Um.... The year is 2007 :rolleyes:
:laugh: Even worse then.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
-
Paul Watson wrote:
"Delayed write failure" dialogs
Paul, you may want to disable write caching on the external drive. While this will make the copy take longer, it'll hopefully make it more reliable. /ravi
This is your brain on Celcius Home | Music | Articles | Freeware | Trips ravib(at)ravib(dot)com
Thanks. I'm not going to bother trying that again though, it took it 3 hours to fail and I'm not waiting to see if disabling that feature will help. It should be easier and more reliable than this.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
-
Here's a link[^] I've used this software for copying large sets of files, and it is fantastic. Marc
People are just notoriously impossible. --DavidCrow
There's NO excuse for not commenting your code. -- John Simmons / outlaw programmer
People who say that they will refactor their code later to make it "good" don't understand refactoring, nor the art and craft of programming. -- Josh SmithThanks Marc, I'll give it a try. Would MS be sued if it improved its file copy routines?
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
-
Thanks Marc, I'll give it a try. Would MS be sued if it improved its file copy routines?
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
Paul Watson wrote:
Would MS be sued if it improved its file copy routines?
No M$ would be suing you! :~
Simply Elegant Designs JimmyRopes Designs
Think inside the box! ProActive Secure Systems
I'm on-line therefore I am. JimmyRopes -
This was Windows XP :) And in the past I have found Windows handles fewer larger files better than many thousands of smaller files. So ZIPing has often helped. The bottleneck seems to be in the handling routines not the actual copy routine. If I hadn't zipped it the dialog would probably be on 999999999999999 minutes left by now ;) And sure, the external HD did fail in the end but now instead of me being able to say "OK, continue from the failed point" I have to start all over again. Well done software. The external HD is fine now it seems. It probably just got bored and took its eye off the ball.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
Paul Watson wrote:
I have found Windows handles fewer larger files better than many thousands of smaller files. So ZIPing has often helped.
i've found the same thing. when migrating data to a new computer over a network, for example, i always ZIP, just to make the copying faster. there's an obvious, and huge, per-file overhead when copying.
image processing toolkits | batch image processing | blogging
-
Paul Watson wrote:
I have found Windows handles fewer larger files better than many thousands of smaller files. So ZIPing has often helped.
i've found the same thing. when migrating data to a new computer over a network, for example, i always ZIP, just to make the copying faster. there's an obvious, and huge, per-file overhead when copying.
image processing toolkits | batch image processing | blogging
Chris Losinger wrote:
there's an obvious, and huge, per-file overhead when copying.
Not just copying. Try burn one CD/DVD with 10,000 icon files...
Engaged in learning of English grammar ;)
For God so loved the world, that he gave his only begotten Son, that whosoever believeth in him should not perish, but have everlasting life.(John 3:16) :badger: -
The task: Copy 2,300 files equaling 16gb from one laptop HD to an external HD. This should be easy, just copy the parent directory and paste in the destination directory. This I do. The dialog takes 3 minutes just to appear and then another 5 minutes to calculate how long it is going to copy all the files. It estimates it will take 89 minutes to copy which is fine by me. So I leave it for 30 minutes, come back and find the dialog is now saying 120 minutes. Hmmm. OK, maybe 2,300 files is too much for the copy app. So I decide to ZIP the files (without compression), copy the resulting 13gig file and then unzip on the destination directory. It takes 30 minutes to ZIP, 20 minutes to copy the file across and then I start to unzip. 30 minutes in it is about 15% through, WTF? OK, who cares, just leave it while I am off moving a boat. I come back 3 hours later and my girlfriend, who was told "not to touch the shiny laptop", is pointing at the external HD saying "It is making a clicking sound." Oh bugger. I lift the lid and see a screen full of "Delayed write failure" dialogs. 50 clicks later WinZIP tells me at 97% it failed. *sigh* We have computers more powerful than what launched man to the moon and yet we can't bleeding well reliably copy files from one HD to another.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
-
Hmm, interesting. Just the other day I copied over 14,000 files totalling 33 GB from my hard drive to an external drive. It took about 45 minutes. No problems whatsoever. I simply dragged and dropped it to the new drive. XP Pro SP2. Cheers, Drew.
What external HD do you have?
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
-
This was Windows XP :) And in the past I have found Windows handles fewer larger files better than many thousands of smaller files. So ZIPing has often helped. The bottleneck seems to be in the handling routines not the actual copy routine. If I hadn't zipped it the dialog would probably be on 999999999999999 minutes left by now ;) And sure, the external HD did fail in the end but now instead of me being able to say "OK, continue from the failed point" I have to start all over again. Well done software. The external HD is fine now it seems. It probably just got bored and took its eye off the ball.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
Paul Watson wrote:
And in the past I have found Windows handles fewer larger files better than many thousands of smaller files.
True, that. Next time you might want to just use XCOPY or an equivalent utility to avoid all of the extra busywork the GUI copy goes through. XCOPY also has the advantage of providing an option to keep working when a single file fails:
> XCOPY /S /I /C /Y <sourcedir> <destdir>
(that'll copy a directory recursively, create the destination if it doesn't already exist, keep going if a single file fails, and not bug you about overwriting anything) And if you're copying to/from a network drive, the
/Zoption can be a huge frustration-reducer as well. :)---- Scripts i’ve known... CPhog 1.8.2 - make CP better. Forum Bookmark 0.2.5 - bookmark forum posts on Pensieve Print forum 0.1.2 - printer-friendly forums Expand all 1.0 - Expand all messages In-place Delete 1.0 - AJAX-style post delete Syntax 0.1 - Syntax highlighting for code blocks in the forums
-
What external HD do you have?
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
-
Paul Watson wrote:
And in the past I have found Windows handles fewer larger files better than many thousands of smaller files.
True, that. Next time you might want to just use XCOPY or an equivalent utility to avoid all of the extra busywork the GUI copy goes through. XCOPY also has the advantage of providing an option to keep working when a single file fails:
> XCOPY /S /I /C /Y <sourcedir> <destdir>
(that'll copy a directory recursively, create the destination if it doesn't already exist, keep going if a single file fails, and not bug you about overwriting anything) And if you're copying to/from a network drive, the
/Zoption can be a huge frustration-reducer as well. :)---- Scripts i’ve known... CPhog 1.8.2 - make CP better. Forum Bookmark 0.2.5 - bookmark forum posts on Pensieve Print forum 0.1.2 - printer-friendly forums Expand all 1.0 - Expand all messages In-place Delete 1.0 - AJAX-style post delete Syntax 0.1 - Syntax highlighting for code blocks in the forums
-
The task: Copy 2,300 files equaling 16gb from one laptop HD to an external HD. This should be easy, just copy the parent directory and paste in the destination directory. This I do. The dialog takes 3 minutes just to appear and then another 5 minutes to calculate how long it is going to copy all the files. It estimates it will take 89 minutes to copy which is fine by me. So I leave it for 30 minutes, come back and find the dialog is now saying 120 minutes. Hmmm. OK, maybe 2,300 files is too much for the copy app. So I decide to ZIP the files (without compression), copy the resulting 13gig file and then unzip on the destination directory. It takes 30 minutes to ZIP, 20 minutes to copy the file across and then I start to unzip. 30 minutes in it is about 15% through, WTF? OK, who cares, just leave it while I am off moving a boat. I come back 3 hours later and my girlfriend, who was told "not to touch the shiny laptop", is pointing at the external HD saying "It is making a clicking sound." Oh bugger. I lift the lid and see a screen full of "Delayed write failure" dialogs. 50 clicks later WinZIP tells me at 97% it failed. *sigh* We have computers more powerful than what launched man to the moon and yet we can't bleeding well reliably copy files from one HD to another.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
I never do massive copies with Explorer, because the GUI preparation steps take forever. I really like Robocopy, which has built-in support for retrying when errors happen. It comes with Vista, I think it's in a resource kit for XP.
--Mike-- Visual C++ MVP :cool: LINKS~! Ericahist | PimpFish | CP SearchBar v3.0 | C++ Forum FAQ Ford, what's this fish doing in my ear?
-
Paul Watson wrote:
And in the past I have found Windows handles fewer larger files better than many thousands of smaller files.
True, that. Next time you might want to just use XCOPY or an equivalent utility to avoid all of the extra busywork the GUI copy goes through. XCOPY also has the advantage of providing an option to keep working when a single file fails:
> XCOPY /S /I /C /Y <sourcedir> <destdir>
(that'll copy a directory recursively, create the destination if it doesn't already exist, keep going if a single file fails, and not bug you about overwriting anything) And if you're copying to/from a network drive, the
/Zoption can be a huge frustration-reducer as well. :)---- Scripts i’ve known... CPhog 1.8.2 - make CP better. Forum Bookmark 0.2.5 - bookmark forum posts on Pensieve Print forum 0.1.2 - printer-friendly forums Expand all 1.0 - Expand all messages In-place Delete 1.0 - AJAX-style post delete Syntax 0.1 - Syntax highlighting for code blocks in the forums
Shog9 wrote:
extra busywork the GUI copy goes through
Yes, indeed. I'm always reminded of a line from that immortal Mel Brooks epic Spaceballs[^]:
Why are we always preparing? Just go!
Software Zen:
delete this;