Hard drive write latency problem
-
I have a Windows Service running that does a number of file conversions docx->PDF->PostScript plus combining the files. I'm running into a problem of hard drive write latency. The application writes a file (converts to another format). When it finishes, it copies it elsewhere, but I am getting an error that the converted file is not accessible. I'm assuming that my server is configured to do buffered file writes like a storage server when what I really need is a file server that is done writing when it says it is done. I put in a time delay before the file copy, but it didn't seem to help. Is there a way to confirm the file write is complete or some way around this? Any suggestions or links to the problem would be appreciated. Thanks, M
-
I have a Windows Service running that does a number of file conversions docx->PDF->PostScript plus combining the files. I'm running into a problem of hard drive write latency. The application writes a file (converts to another format). When it finishes, it copies it elsewhere, but I am getting an error that the converted file is not accessible. I'm assuming that my server is configured to do buffered file writes like a storage server when what I really need is a file server that is done writing when it says it is done. I put in a time delay before the file copy, but it didn't seem to help. Is there a way to confirm the file write is complete or some way around this? Any suggestions or links to the problem would be appreciated. Thanks, M
This is really a programming question so you are in the wrong forum - you need to be in "quick answers", "Ask a question". However, I had the same issue and simply had my copying code loop (with a few milliseconds sleep) until the file was properly closed. Waiting an arbitrary time will be too flaky. Loop it!
- I would love to change the world, but they won’t give me the source code.
-
This is really a programming question so you are in the wrong forum - you need to be in "quick answers", "Ask a question". However, I had the same issue and simply had my copying code loop (with a few milliseconds sleep) until the file was properly closed. Waiting an arbitrary time will be too flaky. Loop it!
- I would love to change the world, but they won’t give me the source code.
Thanks. I'll move it there as well. Thanks for the answer too. I just don't really like a loop that doesn't have a fixed end. I do three tries now... ... and could you be trusted with the source code :)
-
Thanks. I'll move it there as well. Thanks for the answer too. I just don't really like a loop that doesn't have a fixed end. I do three tries now... ... and could you be trusted with the source code :)
Michael Breeden wrote:
could you be trusted with the source code
Nah. He'd turn round and flog it to the Chinese. He's done it before... ;)
Sent from my Amstrad PC 1640 Never throw anything away, Griff Bad command or file name. Bad, bad command! Sit! Stay! Staaaay... AntiTwitter: @DalekDave is now a follower!
-
Thanks. I'll move it there as well. Thanks for the answer too. I just don't really like a loop that doesn't have a fixed end. I do three tries now... ... and could you be trusted with the source code :)
I vaguely remember using a 100ms wait time and gave up with an error notification if I looped more than 300 times (30 seconds total) - computers don't get tired or mad about doing the same thing over and over so looping only 3 times because "that's probably enough" is not bringing enough firepower to bear!
- I would love to change the world, but they won’t give me the source code.
-
Michael Breeden wrote:
could you be trusted with the source code
Nah. He'd turn round and flog it to the Chinese. He's done it before... ;)
Sent from my Amstrad PC 1640 Never throw anything away, Griff Bad command or file name. Bad, bad command! Sit! Stay! Staaaay... AntiTwitter: @DalekDave is now a follower!
-
I have a Windows Service running that does a number of file conversions docx->PDF->PostScript plus combining the files. I'm running into a problem of hard drive write latency. The application writes a file (converts to another format). When it finishes, it copies it elsewhere, but I am getting an error that the converted file is not accessible. I'm assuming that my server is configured to do buffered file writes like a storage server when what I really need is a file server that is done writing when it says it is done. I put in a time delay before the file copy, but it didn't seem to help. Is there a way to confirm the file write is complete or some way around this? Any suggestions or links to the problem would be appreciated. Thanks, M
Hi,
Michael Breeden wrote:
problem of hard drive write latency
Liquid Nitrogen.
Michael Breeden wrote:
but I am getting an error that the converted file is not accessible
The scenario you are giving does not make a lot of sense to me. It sounds more like something has obtained an exclusive lock on the newly created file. Most likely Windows Defender or whatever third-party security product you are using. From your service when you create the file... use the [LockFile function](https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-lockfile) after calling [CreateFile](https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-createfilea) with the FILE_FLAG_NO_BUFFERING flag. Hold the lock for the entire time and finally after you copy the file to the final destination release it. Best Wishes, -David Delaune
-
I have a Windows Service running that does a number of file conversions docx->PDF->PostScript plus combining the files. I'm running into a problem of hard drive write latency. The application writes a file (converts to another format). When it finishes, it copies it elsewhere, but I am getting an error that the converted file is not accessible. I'm assuming that my server is configured to do buffered file writes like a storage server when what I really need is a file server that is done writing when it says it is done. I put in a time delay before the file copy, but it didn't seem to help. Is there a way to confirm the file write is complete or some way around this? Any suggestions or links to the problem would be appreciated. Thanks, M
You could try using a FileSystemWatcher (.Net) or creating a Folder Watcher. How reliable this would be I have no idea.
-
Hi,
Michael Breeden wrote:
problem of hard drive write latency
Liquid Nitrogen.
Michael Breeden wrote:
but I am getting an error that the converted file is not accessible
The scenario you are giving does not make a lot of sense to me. It sounds more like something has obtained an exclusive lock on the newly created file. Most likely Windows Defender or whatever third-party security product you are using. From your service when you create the file... use the [LockFile function](https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-lockfile) after calling [CreateFile](https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-createfilea) with the FILE_FLAG_NO_BUFFERING flag. Hold the lock for the entire time and finally after you copy the file to the final destination release it. Best Wishes, -David Delaune
Randor wrote:
Liquid Nitrogen.
I've used that. It just works...
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013 -
I have a Windows Service running that does a number of file conversions docx->PDF->PostScript plus combining the files. I'm running into a problem of hard drive write latency. The application writes a file (converts to another format). When it finishes, it copies it elsewhere, but I am getting an error that the converted file is not accessible. I'm assuming that my server is configured to do buffered file writes like a storage server when what I really need is a file server that is done writing when it says it is done. I put in a time delay before the file copy, but it didn't seem to help. Is there a way to confirm the file write is complete or some way around this? Any suggestions or links to the problem would be appreciated. Thanks, M
I ran into a similar problem years ago with DOS. I wrote the realtime code for a sucker rod inspection system. Basically the rods move through the detectors at 1 ft/second. I had to take the readings, do some filtering and spot flaws, then send a signal to a paint gun when the flaw passed by the paint gun. Still audir record was required, so I needed to save all of the measurements and filtered results to the hard disk, without missing any days passing by at 1 ft/sec. I made an end of rod detectotion routine and "vomited all the data to the hard disk in between rods. The trick was to tell when the hard disk write was completed to begin searching for the start of the next rod. In DOS it wasn't too hard, but I don't recall how I did it.
CQ de W5ALT
Walt Fair, Jr., P. E. Comport Computing Specializing in Technical Engineering Software
-
You could try using a FileSystemWatcher (.Net) or creating a Folder Watcher. How reliable this would be I have no idea.
-
This is really a programming question so you are in the wrong forum - you need to be in "quick answers", "Ask a question". However, I had the same issue and simply had my copying code loop (with a few milliseconds sleep) until the file was properly closed. Waiting an arbitrary time will be too flaky. Loop it!
- I would love to change the world, but they won’t give me the source code.
Forogar wrote:
I had the same issue and simply had my copying code loop (with a few milliseconds sleep) until the file was properly closed. Waiting an arbitrary time will be too flaky. Loop it!
I have a method in my static Globals class called
FileIsOpenthat does the same thing. It loops until the file is closed, or until a time limit is met (throws an exception).".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013 -
Not very. The trigger is fired when the file is created, not when it is closed, so he will still have the same problem.
- I would love to change the world, but they won’t give me the source code.
He can use a
FileSystemWatcherfor this.".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013 -
I vaguely remember using a 100ms wait time and gave up with an error notification if I looped more than 300 times (30 seconds total) - computers don't get tired or mad about doing the same thing over and over so looping only 3 times because "that's probably enough" is not bringing enough firepower to bear!
- I would love to change the world, but they won’t give me the source code.
Forogar wrote:
omputers don't get tired or mad about doing the same thing over and over
Bah my laptop will go on strike every time I try and loop database writes it considers too deep, cantankerous bitch that she is.
Never underestimate the power of human stupidity - RAH I'm old. I know stuff - JSOP
