Malloc & Page File
-
Hi. I wish to create a 'circular buffer' that one thread can write to, and another read from. The data is going into the buffer very fast, and there is a lot of it. I will probably use malloc() to create a buffer, however I wish to use a large buffer, but due to the speed the data is going in and out, anything that will slow the whole process will result in an incomplete stream being read out of the buffer. I am very worried about using a too large buffer, and the memory being put into a page file. Could someone please inform me how likely this is, when it will happen, how I can prevent it, the largest memory allocation I can create etc? Thank you all!!
-
Hi. I wish to create a 'circular buffer' that one thread can write to, and another read from. The data is going into the buffer very fast, and there is a lot of it. I will probably use malloc() to create a buffer, however I wish to use a large buffer, but due to the speed the data is going in and out, anything that will slow the whole process will result in an incomplete stream being read out of the buffer. I am very worried about using a too large buffer, and the memory being put into a page file. Could someone please inform me how likely this is, when it will happen, how I can prevent it, the largest memory allocation I can create etc? Thank you all!!
-
For the record I am using VC++ (native). Also it looks as though I may have posted in the wrong category, sorry.
Hi, please elaborate on the context: what is large ? what is fast ? and what kind of data is it (byte oriented, record oriented, fixed record size...)? :)
Luc Pattyn
try { [Search CP Articles] [Search CP Forums] [Forum Guidelines] [My Articles] } catch { [Google] }
-
Hi. I wish to create a 'circular buffer' that one thread can write to, and another read from. The data is going into the buffer very fast, and there is a lot of it. I will probably use malloc() to create a buffer, however I wish to use a large buffer, but due to the speed the data is going in and out, anything that will slow the whole process will result in an incomplete stream being read out of the buffer. I am very worried about using a too large buffer, and the memory being put into a page file. Could someone please inform me how likely this is, when it will happen, how I can prevent it, the largest memory allocation I can create etc? Thank you all!!
Perhaps you don't have to make one (then again, perhaps you want to): http://opensource.adobe.com/classadobe_1_1circular__queue.html[^] http://lists.boost.org/Archives/boost/2004/03/62310.php[^]
Steve
-
Hi, please elaborate on the context: what is large ? what is fast ? and what kind of data is it (byte oriented, record oriented, fixed record size...)? :)
Luc Pattyn
try { [Search CP Articles] [Search CP Forums] [Forum Guidelines] [My Articles] } catch { [Google] }
The actual reading & writing is easy. The data is of fixed length, so no problems coding the buffer itself, I just really really cant afford any lost time due to paging or hard drive accesses. The data will be stored in 13 byte blocks, with data being added at about 53mb per second. This data will simultaneously be read & written to hard disk, the aim is to go as long as possible before the buffer fills up & data/stream integrity is lost. Thank you.
-
The actual reading & writing is easy. The data is of fixed length, so no problems coding the buffer itself, I just really really cant afford any lost time due to paging or hard drive accesses. The data will be stored in 13 byte blocks, with data being added at about 53mb per second. This data will simultaneously be read & written to hard disk, the aim is to go as long as possible before the buffer fills up & data/stream integrity is lost. Thank you.
Hi, there seems no need to have just one (large) buffer. this is how I would go about it: - pre-allocate a number N of memory blocks of 52KB each (yes exactly 52*1024 bytes); - have two queues, one with empty buffers, one with full buffers; - initialize with all the available buffers in the "empty queue"; - let the producer take an empty buffer, and fill it up completely, before putting it in the full queue; - let the consumer take a full buffer, empty it, and return it to the empty queue; take care of performance by: - not using function calls in producer/consumer other than the ones absolutely necessary, including one queue_get and one queue_put. - using a single disk I/O function that transfers an entire buffer at the time (that's why it best is a multiple of 4KB); and use the lowest-level I/O function available, in C that might be fread/fwrite or even read/write. Not sure. - trying to keep the buffers in cache (although they probably would have to be copied to/from memory to perform the disk's DMA); that's why I do not insist on having large buffers nor many buffers; I would try not to exceed 1 MB in total. - possibly: not using a locking queue; a queue that serves one producer and one consumer can de devised to lock only on extreme conditions (that is full and empty), but not while putting/getting an item (use a circular buffer, a get and a put index; only one party is allowed to write to the get index, the other to the put index) - possibly and somewhat tricky: play around with producer/consumer priorities: if there are more empty buffers, raise the producer; if there are more full buffers, lower the producer. Or something similar. - experimenting with N: this is your one degree of freedom, you can add memory to keep it alive longer (maybe with less cache efficiency), that is to overcome longer disturbances due to external causes. If you do this right, I would not be surprised you could make it run "forever", since 53MB/s is well below today's disk bandwidth. The thing that probably matters most is what else the PC is doing (networks, optical disks, etc; turn them of as much as possible!). I would not be concerned about the paging mechanism inside the PC; it knows how to handle bigger jobs than this. You only would be using around 1 MB of memory, and all your use is rather static and in multiples of 4KB, so dont worry. One final remark: I would first try a subset of the above, to get things working, so I could observe how well
-
Hi, there seems no need to have just one (large) buffer. this is how I would go about it: - pre-allocate a number N of memory blocks of 52KB each (yes exactly 52*1024 bytes); - have two queues, one with empty buffers, one with full buffers; - initialize with all the available buffers in the "empty queue"; - let the producer take an empty buffer, and fill it up completely, before putting it in the full queue; - let the consumer take a full buffer, empty it, and return it to the empty queue; take care of performance by: - not using function calls in producer/consumer other than the ones absolutely necessary, including one queue_get and one queue_put. - using a single disk I/O function that transfers an entire buffer at the time (that's why it best is a multiple of 4KB); and use the lowest-level I/O function available, in C that might be fread/fwrite or even read/write. Not sure. - trying to keep the buffers in cache (although they probably would have to be copied to/from memory to perform the disk's DMA); that's why I do not insist on having large buffers nor many buffers; I would try not to exceed 1 MB in total. - possibly: not using a locking queue; a queue that serves one producer and one consumer can de devised to lock only on extreme conditions (that is full and empty), but not while putting/getting an item (use a circular buffer, a get and a put index; only one party is allowed to write to the get index, the other to the put index) - possibly and somewhat tricky: play around with producer/consumer priorities: if there are more empty buffers, raise the producer; if there are more full buffers, lower the producer. Or something similar. - experimenting with N: this is your one degree of freedom, you can add memory to keep it alive longer (maybe with less cache efficiency), that is to overcome longer disturbances due to external causes. If you do this right, I would not be surprised you could make it run "forever", since 53MB/s is well below today's disk bandwidth. The thing that probably matters most is what else the PC is doing (networks, optical disks, etc; turn them of as much as possible!). I would not be concerned about the paging mechanism inside the PC; it knows how to handle bigger jobs than this. You only would be using around 1 MB of memory, and all your use is rather static and in multiples of 4KB, so dont worry. One final remark: I would first try a subset of the above, to get things working, so I could observe how well
Thank you, this gave me quite a few ideas to think about. A previous attempt by someone else 2 years ago managed a run time of about 20 seconds with half as much data, I need to increase by at least 200% and double the datarate. The system will be the same to start with but I am going to try to get it upgraded. Thank you once again.
-
Thank you, this gave me quite a few ideas to think about. A previous attempt by someone else 2 years ago managed a run time of about 20 seconds with half as much data, I need to increase by at least 200% and double the datarate. The system will be the same to start with but I am going to try to get it upgraded. Thank you once again.
I hope you're running it on a system with high speed hardrives or a RAID array. A 7200RPM drive only has a sustained transfer rate of 66mbps. You'd either need a raid array striped over 2+ disks (with a real hardware controller) or a 15k RPM drive to hit 106mpbs sustained. If you're doing reads and writes concurrently seek time will lower the maximum throughput farther.
-- You have to explain to them [VB coders] what you mean by "typed". their first response is likely to be something like, "Of course my code is typed. Do you think i magically project it onto the screen with the power of my mind?" --- John Simmons / outlaw programmer
-
I hope you're running it on a system with high speed hardrives or a RAID array. A 7200RPM drive only has a sustained transfer rate of 66mbps. You'd either need a raid array striped over 2+ disks (with a real hardware controller) or a 15k RPM drive to hit 106mpbs sustained. If you're doing reads and writes concurrently seek time will lower the maximum throughput farther.
-- You have to explain to them [VB coders] what you mean by "typed". their first response is likely to be something like, "Of course my code is typed. Do you think i magically project it onto the screen with the power of my mind?" --- John Simmons / outlaw programmer
Hi. The system is currently using RAID stiping across 2 IDE disks. The idea is for a batch system so data is only going to be written to start with, reading will come after the job has finished (initially, depending on performance obtained). I will be trying to get faster hard drives for the project, I'm not sure how fund are at the moment. I am having a hard time obtaining sustained drive datarates, where should I be looking? Thank you.
-
Hi. The system is currently using RAID stiping across 2 IDE disks. The idea is for a batch system so data is only going to be written to start with, reading will come after the job has finished (initially, depending on performance obtained). I will be trying to get faster hard drives for the project, I'm not sure how fund are at the moment. I am having a hard time obtaining sustained drive datarates, where should I be looking? Thank you.
Mr Simple wrote:
I am having a hard time obtaining sustained drive datarates, where should I be looking?
Dunno, IIRC the 66mpbs number is a benchmarking average. I'd assume performance of single drives would scale linearly.
-- You have to explain to them [VB coders] what you mean by "typed". their first response is likely to be something like, "Of course my code is typed. Do you think i magically project it onto the screen with the power of my mind?" --- John Simmons / outlaw programmer
-
Hi. I wish to create a 'circular buffer' that one thread can write to, and another read from. The data is going into the buffer very fast, and there is a lot of it. I will probably use malloc() to create a buffer, however I wish to use a large buffer, but due to the speed the data is going in and out, anything that will slow the whole process will result in an incomplete stream being read out of the buffer. I am very worried about using a too large buffer, and the memory being put into a page file. Could someone please inform me how likely this is, when it will happen, how I can prevent it, the largest memory allocation I can create etc? Thank you all!!
-
Have you considered using shared memory? There are implementations of circular buffers using shared memory and they are really fast. If speed is what you need I thing you should consider this solution.
Hi, I agree shared memory is fast, but I fail to see how a two-process approach with shared memory would be preferable over a two-thread single-process approach using local memory ? With two processes, you need some interprocess communication (you might use part of the shared memory for that), but you also have to take care of proper startup sequencing, and the possibility one process exiting for whatever reason without the other process knowing it... :)
Luc Pattyn
try { [Search CP Articles] [Search CP Forums] [Forum Guidelines] [My Articles] } catch { [Google] }
-
Hi, I agree shared memory is fast, but I fail to see how a two-process approach with shared memory would be preferable over a two-thread single-process approach using local memory ? With two processes, you need some interprocess communication (you might use part of the shared memory for that), but you also have to take care of proper startup sequencing, and the possibility one process exiting for whatever reason without the other process knowing it... :)
Luc Pattyn
try { [Search CP Articles] [Search CP Forums] [Forum Guidelines] [My Articles] } catch { [Google] }
Yes I agree with you on the interprocess communication but sm was only a suggestion. Maybe the solution requires the existence of two separate processes for some reason, then shared memory maybe a good choice. I am not arguing it will be the best though :-)