The year, 2006. The problem, we still can't copy files reliably
-
Paul Watson wrote:
"Delayed write failure" dialogs
Paul, you may want to disable write caching on the external drive. While this will make the copy take longer, it'll hopefully make it more reliable. /ravi
This is your brain on Celcius Home | Music | Articles | Freeware | Trips ravib(at)ravib(dot)com
Thanks. I'm not going to bother trying that again though, it took it 3 hours to fail and I'm not waiting to see if disabling that feature will help. It should be easier and more reliable than this.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
-
Here's a link[^] I've used this software for copying large sets of files, and it is fantastic. Marc
People are just notoriously impossible. --DavidCrow
There's NO excuse for not commenting your code. -- John Simmons / outlaw programmer
People who say that they will refactor their code later to make it "good" don't understand refactoring, nor the art and craft of programming. -- Josh SmithThanks Marc, I'll give it a try. Would MS be sued if it improved its file copy routines?
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
-
Thanks Marc, I'll give it a try. Would MS be sued if it improved its file copy routines?
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
Paul Watson wrote:
Would MS be sued if it improved its file copy routines?
No M$ would be suing you! :~
Simply Elegant Designs JimmyRopes Designs
Think inside the box! ProActive Secure Systems
I'm on-line therefore I am. JimmyRopes -
This was Windows XP :) And in the past I have found Windows handles fewer larger files better than many thousands of smaller files. So ZIPing has often helped. The bottleneck seems to be in the handling routines not the actual copy routine. If I hadn't zipped it the dialog would probably be on 999999999999999 minutes left by now ;) And sure, the external HD did fail in the end but now instead of me being able to say "OK, continue from the failed point" I have to start all over again. Well done software. The external HD is fine now it seems. It probably just got bored and took its eye off the ball.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
Paul Watson wrote:
I have found Windows handles fewer larger files better than many thousands of smaller files. So ZIPing has often helped.
i've found the same thing. when migrating data to a new computer over a network, for example, i always ZIP, just to make the copying faster. there's an obvious, and huge, per-file overhead when copying.
image processing toolkits | batch image processing | blogging
-
Paul Watson wrote:
I have found Windows handles fewer larger files better than many thousands of smaller files. So ZIPing has often helped.
i've found the same thing. when migrating data to a new computer over a network, for example, i always ZIP, just to make the copying faster. there's an obvious, and huge, per-file overhead when copying.
image processing toolkits | batch image processing | blogging
Chris Losinger wrote:
there's an obvious, and huge, per-file overhead when copying.
Not just copying. Try burn one CD/DVD with 10,000 icon files...
Engaged in learning of English grammar ;)
For God so loved the world, that he gave his only begotten Son, that whosoever believeth in him should not perish, but have everlasting life.(John 3:16) :badger: -
The task: Copy 2,300 files equaling 16gb from one laptop HD to an external HD. This should be easy, just copy the parent directory and paste in the destination directory. This I do. The dialog takes 3 minutes just to appear and then another 5 minutes to calculate how long it is going to copy all the files. It estimates it will take 89 minutes to copy which is fine by me. So I leave it for 30 minutes, come back and find the dialog is now saying 120 minutes. Hmmm. OK, maybe 2,300 files is too much for the copy app. So I decide to ZIP the files (without compression), copy the resulting 13gig file and then unzip on the destination directory. It takes 30 minutes to ZIP, 20 minutes to copy the file across and then I start to unzip. 30 minutes in it is about 15% through, WTF? OK, who cares, just leave it while I am off moving a boat. I come back 3 hours later and my girlfriend, who was told "not to touch the shiny laptop", is pointing at the external HD saying "It is making a clicking sound." Oh bugger. I lift the lid and see a screen full of "Delayed write failure" dialogs. 50 clicks later WinZIP tells me at 97% it failed. *sigh* We have computers more powerful than what launched man to the moon and yet we can't bleeding well reliably copy files from one HD to another.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
-
Hmm, interesting. Just the other day I copied over 14,000 files totalling 33 GB from my hard drive to an external drive. It took about 45 minutes. No problems whatsoever. I simply dragged and dropped it to the new drive. XP Pro SP2. Cheers, Drew.
What external HD do you have?
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
-
This was Windows XP :) And in the past I have found Windows handles fewer larger files better than many thousands of smaller files. So ZIPing has often helped. The bottleneck seems to be in the handling routines not the actual copy routine. If I hadn't zipped it the dialog would probably be on 999999999999999 minutes left by now ;) And sure, the external HD did fail in the end but now instead of me being able to say "OK, continue from the failed point" I have to start all over again. Well done software. The external HD is fine now it seems. It probably just got bored and took its eye off the ball.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
Paul Watson wrote:
And in the past I have found Windows handles fewer larger files better than many thousands of smaller files.
True, that. Next time you might want to just use XCOPY or an equivalent utility to avoid all of the extra busywork the GUI copy goes through. XCOPY also has the advantage of providing an option to keep working when a single file fails:
> XCOPY /S /I /C /Y <sourcedir> <destdir>
(that'll copy a directory recursively, create the destination if it doesn't already exist, keep going if a single file fails, and not bug you about overwriting anything) And if you're copying to/from a network drive, the
/Zoption can be a huge frustration-reducer as well. :)---- Scripts i’ve known... CPhog 1.8.2 - make CP better. Forum Bookmark 0.2.5 - bookmark forum posts on Pensieve Print forum 0.1.2 - printer-friendly forums Expand all 1.0 - Expand all messages In-place Delete 1.0 - AJAX-style post delete Syntax 0.1 - Syntax highlighting for code blocks in the forums
-
What external HD do you have?
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
-
Paul Watson wrote:
And in the past I have found Windows handles fewer larger files better than many thousands of smaller files.
True, that. Next time you might want to just use XCOPY or an equivalent utility to avoid all of the extra busywork the GUI copy goes through. XCOPY also has the advantage of providing an option to keep working when a single file fails:
> XCOPY /S /I /C /Y <sourcedir> <destdir>
(that'll copy a directory recursively, create the destination if it doesn't already exist, keep going if a single file fails, and not bug you about overwriting anything) And if you're copying to/from a network drive, the
/Zoption can be a huge frustration-reducer as well. :)---- Scripts i’ve known... CPhog 1.8.2 - make CP better. Forum Bookmark 0.2.5 - bookmark forum posts on Pensieve Print forum 0.1.2 - printer-friendly forums Expand all 1.0 - Expand all messages In-place Delete 1.0 - AJAX-style post delete Syntax 0.1 - Syntax highlighting for code blocks in the forums
-
The task: Copy 2,300 files equaling 16gb from one laptop HD to an external HD. This should be easy, just copy the parent directory and paste in the destination directory. This I do. The dialog takes 3 minutes just to appear and then another 5 minutes to calculate how long it is going to copy all the files. It estimates it will take 89 minutes to copy which is fine by me. So I leave it for 30 minutes, come back and find the dialog is now saying 120 minutes. Hmmm. OK, maybe 2,300 files is too much for the copy app. So I decide to ZIP the files (without compression), copy the resulting 13gig file and then unzip on the destination directory. It takes 30 minutes to ZIP, 20 minutes to copy the file across and then I start to unzip. 30 minutes in it is about 15% through, WTF? OK, who cares, just leave it while I am off moving a boat. I come back 3 hours later and my girlfriend, who was told "not to touch the shiny laptop", is pointing at the external HD saying "It is making a clicking sound." Oh bugger. I lift the lid and see a screen full of "Delayed write failure" dialogs. 50 clicks later WinZIP tells me at 97% it failed. *sigh* We have computers more powerful than what launched man to the moon and yet we can't bleeding well reliably copy files from one HD to another.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
I never do massive copies with Explorer, because the GUI preparation steps take forever. I really like Robocopy, which has built-in support for retrying when errors happen. It comes with Vista, I think it's in a resource kit for XP.
--Mike-- Visual C++ MVP :cool: LINKS~! Ericahist | PimpFish | CP SearchBar v3.0 | C++ Forum FAQ Ford, what's this fish doing in my ear?
-
Paul Watson wrote:
And in the past I have found Windows handles fewer larger files better than many thousands of smaller files.
True, that. Next time you might want to just use XCOPY or an equivalent utility to avoid all of the extra busywork the GUI copy goes through. XCOPY also has the advantage of providing an option to keep working when a single file fails:
> XCOPY /S /I /C /Y <sourcedir> <destdir>
(that'll copy a directory recursively, create the destination if it doesn't already exist, keep going if a single file fails, and not bug you about overwriting anything) And if you're copying to/from a network drive, the
/Zoption can be a huge frustration-reducer as well. :)---- Scripts i’ve known... CPhog 1.8.2 - make CP better. Forum Bookmark 0.2.5 - bookmark forum posts on Pensieve Print forum 0.1.2 - printer-friendly forums Expand all 1.0 - Expand all messages In-place Delete 1.0 - AJAX-style post delete Syntax 0.1 - Syntax highlighting for code blocks in the forums
Shog9 wrote:
extra busywork the GUI copy goes through
Yes, indeed. I'm always reminded of a line from that immortal Mel Brooks epic Spaceballs[^]:
Why are we always preparing? Just go!
Software Zen:
delete this; -
The task: Copy 2,300 files equaling 16gb from one laptop HD to an external HD. This should be easy, just copy the parent directory and paste in the destination directory. This I do. The dialog takes 3 minutes just to appear and then another 5 minutes to calculate how long it is going to copy all the files. It estimates it will take 89 minutes to copy which is fine by me. So I leave it for 30 minutes, come back and find the dialog is now saying 120 minutes. Hmmm. OK, maybe 2,300 files is too much for the copy app. So I decide to ZIP the files (without compression), copy the resulting 13gig file and then unzip on the destination directory. It takes 30 minutes to ZIP, 20 minutes to copy the file across and then I start to unzip. 30 minutes in it is about 15% through, WTF? OK, who cares, just leave it while I am off moving a boat. I come back 3 hours later and my girlfriend, who was told "not to touch the shiny laptop", is pointing at the external HD saying "It is making a clicking sound." Oh bugger. I lift the lid and see a screen full of "Delayed write failure" dialogs. 50 clicks later WinZIP tells me at 97% it failed. *sigh* We have computers more powerful than what launched man to the moon and yet we can't bleeding well reliably copy files from one HD to another.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
That is yet another feature I have enjoyed in Vista. They upgraded a few features in the file copy area that makes it much more usable. I copy large groups of files easily and with prompts in case of errors that allow me to continue. Much nicer :)
Rocky <>< Latest Code Blog Post: OpenID/CardSpace - Is it time? Latest Tech Blog Post: Scratch: fun for all ages for free!
-
That is yet another feature I have enjoyed in Vista. They upgraded a few features in the file copy area that makes it much more usable. I copy large groups of files easily and with prompts in case of errors that allow me to continue. Much nicer :)
Rocky <>< Latest Code Blog Post: OpenID/CardSpace - Is it time? Latest Tech Blog Post: Scratch: fun for all ages for free!
Cool though I think Chris was complaining about Vista's copy just the other week.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
-
Thanks Marc, I'll give it a try. Would MS be sued if it improved its file copy routines?
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
You can't sell a OS on "copying files just got better".
Developers, Developers, Developers, Developers, Developers, Developers, Velopers, Develprs, Developers!
We are a big screwed up dysfunctional psychotic happy family - some more screwed up, others more happy, but everybody's psychotic joint venture definition of CP
Linkify!|Fold With Us! -
So the OS/X filecopy sucks as much as the Windows version? Or are you slumming... ;P Still, sounds like you're blaming a hardware failure on software. Oh, and since it has to be written, uncompressed, to the external harddrive, you're not really saving anything by zipping it. That only helps when you can move the thing compressed across the bottleneck and then uncompress on the other side.
---- Scripts i’ve known... CPhog 1.8.2 - make CP better. Forum Bookmark 0.2.5 - bookmark forum posts on Pensieve Print forum 0.1.2 - printer-friendly forums Expand all 1.0 - Expand all messages In-place Delete 1.0 - AJAX-style post delete Syntax 0.1 - Syntax highlighting for code blocks in the forums
Heh - I was going to suggest to Paul that OS X would probably do a better job - I backed up my home directory on my iBook a couple of weeks ago (the motherboard/wi-fi is hosed, new motherboard being fitted) - just drag-dropped the directory onto a USB drive and it copied 40GB in 180,000+ files quite happily. It took a while (an hour or so? Maybe longer), but it just worked...
-
Paul Watson wrote:
And in the past I have found Windows handles fewer larger files better than many thousands of smaller files.
True, that. Next time you might want to just use XCOPY or an equivalent utility to avoid all of the extra busywork the GUI copy goes through. XCOPY also has the advantage of providing an option to keep working when a single file fails:
> XCOPY /S /I /C /Y <sourcedir> <destdir>
(that'll copy a directory recursively, create the destination if it doesn't already exist, keep going if a single file fails, and not bug you about overwriting anything) And if you're copying to/from a network drive, the
/Zoption can be a huge frustration-reducer as well. :)---- Scripts i’ve known... CPhog 1.8.2 - make CP better. Forum Bookmark 0.2.5 - bookmark forum posts on Pensieve Print forum 0.1.2 - printer-friendly forums Expand all 1.0 - Expand all messages In-place Delete 1.0 - AJAX-style post delete Syntax 0.1 - Syntax highlighting for code blocks in the forums
Thanks Shog9 from outer space. I have just kicked that command off for a copy of 80gb from an external HD to a networked drive in a Windows XP VM in Parallels on Mac OS X... if this works easily I'll eat your famous pickle.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
-
Paul Watson wrote:
And in the past I have found Windows handles fewer larger files better than many thousands of smaller files.
True, that. Next time you might want to just use XCOPY or an equivalent utility to avoid all of the extra busywork the GUI copy goes through. XCOPY also has the advantage of providing an option to keep working when a single file fails:
> XCOPY /S /I /C /Y <sourcedir> <destdir>
(that'll copy a directory recursively, create the destination if it doesn't already exist, keep going if a single file fails, and not bug you about overwriting anything) And if you're copying to/from a network drive, the
/Zoption can be a huge frustration-reducer as well. :)---- Scripts i’ve known... CPhog 1.8.2 - make CP better. Forum Bookmark 0.2.5 - bookmark forum posts on Pensieve Print forum 0.1.2 - printer-friendly forums Expand all 1.0 - Expand all messages In-place Delete 1.0 - AJAX-style post delete Syntax 0.1 - Syntax highlighting for code blocks in the forums
Yeah, I use XCOPY all the time for copying large numbers of files. The gui copy function is horrible dealing with slow connections or large numbers of files.
Using the GridView is like trying to explain to someone else how to move a third person's hands in order to tie your shoelaces for you. -Chris Maunder
-
The task: Copy 2,300 files equaling 16gb from one laptop HD to an external HD. This should be easy, just copy the parent directory and paste in the destination directory. This I do. The dialog takes 3 minutes just to appear and then another 5 minutes to calculate how long it is going to copy all the files. It estimates it will take 89 minutes to copy which is fine by me. So I leave it for 30 minutes, come back and find the dialog is now saying 120 minutes. Hmmm. OK, maybe 2,300 files is too much for the copy app. So I decide to ZIP the files (without compression), copy the resulting 13gig file and then unzip on the destination directory. It takes 30 minutes to ZIP, 20 minutes to copy the file across and then I start to unzip. 30 minutes in it is about 15% through, WTF? OK, who cares, just leave it while I am off moving a boat. I come back 3 hours later and my girlfriend, who was told "not to touch the shiny laptop", is pointing at the external HD saying "It is making a clicking sound." Oh bugger. I lift the lid and see a screen full of "Delayed write failure" dialogs. 50 clicks later WinZIP tells me at 97% it failed. *sigh* We have computers more powerful than what launched man to the moon and yet we can't bleeding well reliably copy files from one HD to another.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
Even worse is when it doesn't tell you that the copy has failed. I recently installed a new larger HDD in my computer, so I moved all my music over to it. I thought everything had gone fine. I looked in one of my folders and saw some files with weird names (like temp files). I figured that they had been artifacts of the encoding process so I deleted them. Then I looked closer and found that I had deleted actual songs, but that the copy procedure in Windows Explorer never got around to renaming them back to the original names!
-- Marcus Kwok
-
The task: Copy 2,300 files equaling 16gb from one laptop HD to an external HD. This should be easy, just copy the parent directory and paste in the destination directory. This I do. The dialog takes 3 minutes just to appear and then another 5 minutes to calculate how long it is going to copy all the files. It estimates it will take 89 minutes to copy which is fine by me. So I leave it for 30 minutes, come back and find the dialog is now saying 120 minutes. Hmmm. OK, maybe 2,300 files is too much for the copy app. So I decide to ZIP the files (without compression), copy the resulting 13gig file and then unzip on the destination directory. It takes 30 minutes to ZIP, 20 minutes to copy the file across and then I start to unzip. 30 minutes in it is about 15% through, WTF? OK, who cares, just leave it while I am off moving a boat. I come back 3 hours later and my girlfriend, who was told "not to touch the shiny laptop", is pointing at the external HD saying "It is making a clicking sound." Oh bugger. I lift the lid and see a screen full of "Delayed write failure" dialogs. 50 clicks later WinZIP tells me at 97% it failed. *sigh* We have computers more powerful than what launched man to the moon and yet we can't bleeding well reliably copy files from one HD to another.
regards, Paul Watson Ireland & South Africa
Shog9 wrote:
And with that, Paul closed his browser, sipped his herbal tea, fixed the flower in his hair, and smiled brightly at the multitude of cute, furry animals flocking around the grassy hillside where he sat coding Ruby on his Mac...
Turn off power management on your USB ports for starters. Then get syncbackse. I've been able to move 500 gigs of data 4 times in a row with it and never had a single problem. Something else to consider would be putting the external drive on another machine and mapping a drive. Don't ask me why but I just think this works better. I get some weird things happening when using USB at times... SyncBackSE would copy anything to anywhere and it just rules.