Optimizing hard drive write performance
-
I'm working on an application that captures uncompressed video from 2 high resolution (1024x768) video cameras at 30 fps. In order to stream to the hard drive without dropping frames, I need to maintain an average transfer rate of ~47 megabytes per second. The drives I am using are rated to be able to handled a sustained write of 75 megabytes per second all the way to the end. However, simply writing data with fwrite and timing the performance, I am coming no where near the necessary level of performance. If anyone out there has any experience with high performance file writing, I'd love to get some tips/pointers on ways to increase write speed. Adam Kraver www.captivemotion.com
-
I'm working on an application that captures uncompressed video from 2 high resolution (1024x768) video cameras at 30 fps. In order to stream to the hard drive without dropping frames, I need to maintain an average transfer rate of ~47 megabytes per second. The drives I am using are rated to be able to handled a sustained write of 75 megabytes per second all the way to the end. However, simply writing data with fwrite and timing the performance, I am coming no where near the necessary level of performance. If anyone out there has any experience with high performance file writing, I'd love to get some tips/pointers on ways to increase write speed. Adam Kraver www.captivemotion.com
I believe the way to get the fastest file throughput is to use Overlapped I/O along with the
NO_BUFFERINGflag when the file is created/opened withCreateFile(...). There are some gotchas with buffer size and alignment, but if you are good enough to be streaming real-time video, it is likely nothing that you cannot handle! Also, just because your drive can go that fast does not mean that your system can. Other factors (the drive controller being used, any hardware caching, what other stuff your system is doing). Peace!-=- James
Please rate this message - let me know if I helped or not! * * *If you think it costs a lot to do it right, just wait until you find out how much it costs to do it wrong!
Avoid driving a vehicle taller than you and remember that Professional Driver on Closed Course does not mean your Dumb Ass on a Public Road!
See DeleteFXPFiles -
I believe the way to get the fastest file throughput is to use Overlapped I/O along with the
NO_BUFFERINGflag when the file is created/opened withCreateFile(...). There are some gotchas with buffer size and alignment, but if you are good enough to be streaming real-time video, it is likely nothing that you cannot handle! Also, just because your drive can go that fast does not mean that your system can. Other factors (the drive controller being used, any hardware caching, what other stuff your system is doing). Peace!-=- James
Please rate this message - let me know if I helped or not! * * *If you think it costs a lot to do it right, just wait until you find out how much it costs to do it wrong!
Avoid driving a vehicle taller than you and remember that Professional Driver on Closed Course does not mean your Dumb Ass on a Public Road!
See DeleteFXPFilesYes, James is correct.. If you are using a SATA disk, it can transfer up to 150MB of data/sec, and a SATA II disk will do up to 300MB of data/sec so you have to be careful with this.. if it is an ATA disk, then you're looking at up to 133MB of data/sec.. FireWire external HDDs can transfer up 800MB of data/sec.. Bearing these in mind also note that Windows schedules other tasks and so it uses the disk(s) for these aswell so writing to a "non-Windows" disk (i.e. a disk which doesnt have the running Windows on it) and a disk which is also not being accessed will improve data-transfer rate.. Also check your motherboard FSB.. this will also affect the speed of data transfer to and from the hard disks.. Hope this helps! --PerspX
"Nowadays, security guys break the Mac every single day. Every single day, they come out with a total exploit, your machine can be taken over totally. I dare anybody to do that once a month on the Windows machine." - Bill Gates
-
I believe the way to get the fastest file throughput is to use Overlapped I/O along with the
NO_BUFFERINGflag when the file is created/opened withCreateFile(...). There are some gotchas with buffer size and alignment, but if you are good enough to be streaming real-time video, it is likely nothing that you cannot handle! Also, just because your drive can go that fast does not mean that your system can. Other factors (the drive controller being used, any hardware caching, what other stuff your system is doing). Peace!-=- James
Please rate this message - let me know if I helped or not! * * *If you think it costs a lot to do it right, just wait until you find out how much it costs to do it wrong!
Avoid driving a vehicle taller than you and remember that Professional Driver on Closed Course does not mean your Dumb Ass on a Public Road!
See DeleteFXPFilesJames, thanks for the hint! So far, using CreateFile and WriteFile with NO_BUFFERING, I'm getting the performance I was expecting (and so far, the performance that it looks like I need). Of course, I'm testing this in a small test app, so I'm not sure what the final results will be in the capture app, but the results are quite promising.
-
I'm working on an application that captures uncompressed video from 2 high resolution (1024x768) video cameras at 30 fps. In order to stream to the hard drive without dropping frames, I need to maintain an average transfer rate of ~47 megabytes per second. The drives I am using are rated to be able to handled a sustained write of 75 megabytes per second all the way to the end. However, simply writing data with fwrite and timing the performance, I am coming no where near the necessary level of performance. If anyone out there has any experience with high performance file writing, I'd love to get some tips/pointers on ways to increase write speed. Adam Kraver www.captivemotion.com
Programatically you can acheive this by implementing multithreading. Put the code of writing to disk in separate thread. Regards, Paresh.
-
Programatically you can acheive this by implementing multithreading. Put the code of writing to disk in separate thread. Regards, Paresh.
Before using CreateFile and WriteFile, I had done that, but unfortunately, the performance of fwrite couldn't keep up with my transfer needs (it runs about 1/2 to 1/4 the speed of WriteFile when using unbuffered writes, even when there is no other real load on the CPU).