Question about Harddrive writes
-
Hi, I have about 3 Megs of data coming in at intervals of about 100 times a second. I am trying to write this into the harddrive so that I can do some post processing after all the data is written to a file. I am finding that the hard drive cant keep up and it actually causes data to be dropped. So what I did was implemented a queue where I can put into memory and then I wrote another thread to copy off the data to the harddrive but that is still not doing the trick. The memory writes are way too quick and my circular queue gets full before the hardrive writes are done, so then the thread needs to wait. I need to be able to record this data for a large amount of time, say about 1 to 2 hours without losing any data. I get loss of data within 10 seconds. I am going crazy here, if there is another approach to doing this I am all ears, even if it calls for a full overhaul of my application, or if there is something else I am doing wrong. All the code is written in C++. Thanks in advance
-
Hi, I have about 3 Megs of data coming in at intervals of about 100 times a second. I am trying to write this into the harddrive so that I can do some post processing after all the data is written to a file. I am finding that the hard drive cant keep up and it actually causes data to be dropped. So what I did was implemented a queue where I can put into memory and then I wrote another thread to copy off the data to the harddrive but that is still not doing the trick. The memory writes are way too quick and my circular queue gets full before the hardrive writes are done, so then the thread needs to wait. I need to be able to record this data for a large amount of time, say about 1 to 2 hours without losing any data. I get loss of data within 10 seconds. I am going crazy here, if there is another approach to doing this I am all ears, even if it calls for a full overhaul of my application, or if there is something else I am doing wrong. All the code is written in C++. Thanks in advance
-
Two questions: 1) Where is your data coming from? 2) Is your hard drive physically capable of that data rate?
1.) the data is coming from a DAQ Board 2.) I dont know the datarate of the hard drive. Is there a way to do this without using the harddrive? Thanks
-
Hi, I have about 3 Megs of data coming in at intervals of about 100 times a second. I am trying to write this into the harddrive so that I can do some post processing after all the data is written to a file. I am finding that the hard drive cant keep up and it actually causes data to be dropped. So what I did was implemented a queue where I can put into memory and then I wrote another thread to copy off the data to the harddrive but that is still not doing the trick. The memory writes are way too quick and my circular queue gets full before the hardrive writes are done, so then the thread needs to wait. I need to be able to record this data for a large amount of time, say about 1 to 2 hours without losing any data. I get loss of data within 10 seconds. I am going crazy here, if there is another approach to doing this I am all ears, even if it calls for a full overhaul of my application, or if there is something else I am doing wrong. All the code is written in C++. Thanks in advance
We too ran into a similar problem. We are streaming video over USB 2.0 from a cell phone to a PC as the device plays a video or pans with camera. We ran into a problem where the system could not keep up. So we tried using a memory mapped file instead of using the old standard CreateFile() and WriteFile(). That did speed things up to where we could capture data at 40 frames a second but we still have an issue at 80 fps.
-
1.) the data is coming from a DAQ Board 2.) I dont know the datarate of the hard drive. Is there a way to do this without using the harddrive? Thanks
godspeed123 wrote:
Is there a way to do this without using the harddrive?
Not and keep your "I am trying to write this into the harddrive so that I can do some post processing after all the data is written to a file." requirement.
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
-
godspeed123 wrote:
Is there a way to do this without using the harddrive?
Not and keep your "I am trying to write this into the harddrive so that I can do some post processing after all the data is written to a file." requirement.
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
Thanks for the response. So that means I am screwed, lol. Is there any other approach that I can use to get around this, even if it means rewriting all the code.
-
We too ran into a similar problem. We are streaming video over USB 2.0 from a cell phone to a PC as the device plays a video or pans with camera. We ran into a problem where the system could not keep up. So we tried using a memory mapped file instead of using the old standard CreateFile() and WriteFile(). That did speed things up to where we could capture data at 40 frames a second but we still have an issue at 80 fps.
Hmmm my stuff is coming in at about 100 times a second, I will give it a shot, but as you guys did I'm sure I will hit the same pitfalls. Is there anything else you tried that worked? Thanks again.
-
Hmmm my stuff is coming in at about 100 times a second, I will give it a shot, but as you guys did I'm sure I will hit the same pitfalls. Is there anything else you tried that worked? Thanks again.
-
Hi, I have about 3 Megs of data coming in at intervals of about 100 times a second. I am trying to write this into the harddrive so that I can do some post processing after all the data is written to a file. I am finding that the hard drive cant keep up and it actually causes data to be dropped. So what I did was implemented a queue where I can put into memory and then I wrote another thread to copy off the data to the harddrive but that is still not doing the trick. The memory writes are way too quick and my circular queue gets full before the hardrive writes are done, so then the thread needs to wait. I need to be able to record this data for a large amount of time, say about 1 to 2 hours without losing any data. I get loss of data within 10 seconds. I am going crazy here, if there is another approach to doing this I am all ears, even if it calls for a full overhaul of my application, or if there is something else I am doing wrong. All the code is written in C++. Thanks in advance
godspeed123 wrote:
I have about 3 Megs of data coming in at intervals of about 100 times a second.
You are receiving 300MB of data every second? I'm not sure of any HD that can write that fast.
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
-
Hi, I have about 3 Megs of data coming in at intervals of about 100 times a second. I am trying to write this into the harddrive so that I can do some post processing after all the data is written to a file. I am finding that the hard drive cant keep up and it actually causes data to be dropped. So what I did was implemented a queue where I can put into memory and then I wrote another thread to copy off the data to the harddrive but that is still not doing the trick. The memory writes are way too quick and my circular queue gets full before the hardrive writes are done, so then the thread needs to wait. I need to be able to record this data for a large amount of time, say about 1 to 2 hours without losing any data. I get loss of data within 10 seconds. I am going crazy here, if there is another approach to doing this I am all ears, even if it calls for a full overhaul of my application, or if there is something else I am doing wrong. All the code is written in C++. Thanks in advance
A standard hard drive maximum write speed is theoretically 12.5 MBs per second (Megabytes, not Megabits) if I remember correctly. Even if you had a 15,000 RPM SATA, SAS, or SCSI you probably won't see over 20-30 MB/s. You will need to have a RAID 5 or better of at least 10 high-performance drives to handle all this data, but even that would probably barely do it. The other option is to install GBs upon GBs of RAM and just Queue it all up like you've been doing. You're not going to get 1-2 hours of data, though, at all on your current setup. You'll have to read for 5 minutes, wait until the Queue is empty, and then collect more data. That's the only solution I can think of, and it's merely a money issue. Fast storage is expensive.
-
Thanks for the response. So that means I am screwed, lol. Is there any other approach that I can use to get around this, even if it means rewriting all the code.
Hello, Are you saying 3MB x 100 times/second = 300MegaByte/sec? If this is the case, you need some special hard drives, i.e. SAS or SCSI 15K rpm, in RAID 0 with 4 or 6 such drives. Then it may be possible to capture data without lose. Conventional single 7.2K rpm desktop drives only can maintain ~50MegaByte/sec and you need to pay attention on that. Hope it helps. :)
-
godspeed123 wrote:
I have about 3 Megs of data coming in at intervals of about 100 times a second.
You are receiving 300MB of data every second? I'm not sure of any HD that can write that fast.
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
I think by 3 Megs, he means 3 MB.
Shog on learning VB6: Ah, that would have been VB6. Kicked my ass anyway.
So easy to learn, just like falling down a flight of stairs... -
I think by 3 Megs, he means 3 MB.
Shog on learning VB6: Ah, that would have been VB6. Kicked my ass anyway.
So easy to learn, just like falling down a flight of stairs...brahmma wrote:
I think by 3 Megs, he means 3 MB.
No kidding. :rolleyes: I've never known "megs" to mean anything other than MB.
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
-
brahmma wrote:
I think by 3 Megs, he means 3 MB.
No kidding. :rolleyes: I've never known "megs" to mean anything other than MB.
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
DavidCrow wrote:
You are receiving 300MB of data every second? I'm not sure of any HD that can write that fast.
The above quote is from your previous post. I did not reply to tell megs is MB, but I said the OP meant 3MB and not 300 MB.
Nobody can give you wiser advice than yourself. - Cicero
-
DavidCrow wrote:
You are receiving 300MB of data every second? I'm not sure of any HD that can write that fast.
The above quote is from your previous post. I did not reply to tell megs is MB, but I said the OP meant 3MB and not 300 MB.
Nobody can give you wiser advice than yourself. - Cicero
brahmma wrote:
...the OP meant 3MB and not 300 MB.
Last time I checked, 3MB 100 times per second was equal to 300MB per second.
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
-
brahmma wrote:
...the OP meant 3MB and not 300 MB.
Last time I checked, 3MB 100 times per second was equal to 300MB per second.
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
What I understood from the original post was, the total amount of data received is 3MB every second, and is not received at a shot, but is received in 100 intervals. :~
Nobody can give you wiser advice than yourself. - Cicero