Memory allocation problem??
-
Hi, I'm having a general C program where I allocate two variables with predefined amount of memory depending upon the # of seconds user enters (this is a data acquisition program). It looks as follows where sizeof(float64) = 8 bytes- float64 *dataAI; dataAI = malloc(acq_time*sizeof(float64)*10000000); using above I can allocate upto 14sec (acq_time = 14) without any problem, but whenever I try to allocate more than that, it simply does nothing (it allocates nothing) and goes to next statement. Is there any memory allocation limitation in C?? I tried allocating same amount of memory using two variables rather than one, but even then it allocates only to one variable and does nothing for the second one. As follows (where acq_time = 16 or 18)- float64 *dataAI1, *dataAI2; dataAI1 = malloc((acq_time/2)*sizeof(float64)*10000000); //allocates for this dataAI2 = malloc((acq_time/2)*sizeof(float64)*10000000); // does nothing for this How can I get around this problem :confused::confused:. Any suggestions?? thanks, -Pavan
-
Hi, I'm having a general C program where I allocate two variables with predefined amount of memory depending upon the # of seconds user enters (this is a data acquisition program). It looks as follows where sizeof(float64) = 8 bytes- float64 *dataAI; dataAI = malloc(acq_time*sizeof(float64)*10000000); using above I can allocate upto 14sec (acq_time = 14) without any problem, but whenever I try to allocate more than that, it simply does nothing (it allocates nothing) and goes to next statement. Is there any memory allocation limitation in C?? I tried allocating same amount of memory using two variables rather than one, but even then it allocates only to one variable and does nothing for the second one. As follows (where acq_time = 16 or 18)- float64 *dataAI1, *dataAI2; dataAI1 = malloc((acq_time/2)*sizeof(float64)*10000000); //allocates for this dataAI2 = malloc((acq_time/2)*sizeof(float64)*10000000); // does nothing for this How can I get around this problem :confused::confused:. Any suggestions?? thanks, -Pavan
pavanbabut wrote:
dataAI = malloc(acq_time*sizeof(float64)*10000000);
That basically equates (roughly) to allocating 80 MB * some value of time (so for 14, it is 1.1 GB of data). It is VERY bad to try and allocate that much space. Generally, the heap won't let you allocate more than 1 GB. If you are allocating chucks close to 100 MB at a time, you really should start thinking of another way to solve your problem since you are putting way too much into memory at one time.
If you decide to become a software engineer, you are signing up to have a 1/2" piece of silicon tell you exactly how stupid you really are for 8 hours a day, 5 days a week Zac
-
pavanbabut wrote:
dataAI = malloc(acq_time*sizeof(float64)*10000000);
That basically equates (roughly) to allocating 80 MB * some value of time (so for 14, it is 1.1 GB of data). It is VERY bad to try and allocate that much space. Generally, the heap won't let you allocate more than 1 GB. If you are allocating chucks close to 100 MB at a time, you really should start thinking of another way to solve your problem since you are putting way too much into memory at one time.
If you decide to become a software engineer, you are signing up to have a 1/2" piece of silicon tell you exactly how stupid you really are for 8 hours a day, 5 days a week Zac
The porblem is, I acquire data continuously at 10MSamples/sec (where each sample is of 8bytes) per channels with 3 channels at a time. Where one channel is of type float64 and another two are of type uint8 (1byte). So, when I start the NI board to acquire data, it continuously records data for the specified time and stores the data (onto ram, at present) simultaneously. Coming to my hard disk, its speed is around 13MB/s (write speed) even though its a SCSI drive and so, I can not write data onto hard disk at the same speed or atleast near to that speed. It would really appreciate if you have any other idea on implementing this, with out getting any errors. thanks, -Pavan
-
The porblem is, I acquire data continuously at 10MSamples/sec (where each sample is of 8bytes) per channels with 3 channels at a time. Where one channel is of type float64 and another two are of type uint8 (1byte). So, when I start the NI board to acquire data, it continuously records data for the specified time and stores the data (onto ram, at present) simultaneously. Coming to my hard disk, its speed is around 13MB/s (write speed) even though its a SCSI drive and so, I can not write data onto hard disk at the same speed or atleast near to that speed. It would really appreciate if you have any other idea on implementing this, with out getting any errors. thanks, -Pavan
I don't know what your requirements for the project are, but here are some suggestions: 1) If you can, try to take a sampling of the data. Instead of pulling in all of it, pull in 3 or 4 sets per second. 2) Buffer the data into a smaller section. Pull in some small set of data (say 13 MB) and write it to a file when it is full. During the write process, block all incoming data until you are done 3) Similar to #2, if you can spawn multiple threads, create a buffer, fill it up, and then pass it to a thread to write it out to a file. While it is writing, you create a new buffer (or keep a pool of buffers around for this -- which would be more efficient) to write to. 4) If none of those will do, and you absolutely must allocate all the necessary memory up front (which, I would question those requirements if that is the case), you will need to create your own heap management (meaning, you will need to override the new and delete operators to suit your needs). This is NOT recommended and again, if you really think this is what you need, then I would question your requirements. Where are you receiving these samples from? That is, are they coming over a serial port? ethernet port? USB/firewire? hard drive? Keep in mind that when you declare large blocks of memory (such as 1.1 GB), you will be leaving RAM and using virtual memory (meaning that you will be reading/writing to the hard drive continuously for this). From your description of the problem, you will not be able to make use of that since the hard drive write speed can't keep up with your sampling rate.
If you decide to become a software engineer, you are signing up to have a 1/2" piece of silicon tell you exactly how stupid you really are for 8 hours a day, 5 days a week Zac
-
I don't know what your requirements for the project are, but here are some suggestions: 1) If you can, try to take a sampling of the data. Instead of pulling in all of it, pull in 3 or 4 sets per second. 2) Buffer the data into a smaller section. Pull in some small set of data (say 13 MB) and write it to a file when it is full. During the write process, block all incoming data until you are done 3) Similar to #2, if you can spawn multiple threads, create a buffer, fill it up, and then pass it to a thread to write it out to a file. While it is writing, you create a new buffer (or keep a pool of buffers around for this -- which would be more efficient) to write to. 4) If none of those will do, and you absolutely must allocate all the necessary memory up front (which, I would question those requirements if that is the case), you will need to create your own heap management (meaning, you will need to override the new and delete operators to suit your needs). This is NOT recommended and again, if you really think this is what you need, then I would question your requirements. Where are you receiving these samples from? That is, are they coming over a serial port? ethernet port? USB/firewire? hard drive? Keep in mind that when you declare large blocks of memory (such as 1.1 GB), you will be leaving RAM and using virtual memory (meaning that you will be reading/writing to the hard drive continuously for this). From your description of the problem, you will not be able to make use of that since the hard drive write speed can't keep up with your sampling rate.
If you decide to become a software engineer, you are signing up to have a 1/2" piece of silicon tell you exactly how stupid you really are for 8 hours a day, 5 days a week Zac
From a long time I am thinking on the same lines as your 3rd suggestion. But, I doubt that at somepoint the buffer will over run or we may skip data (as the HDD speeds are very very low compared to acquiring speeds). Actually the data of the 3 channels comes from a National Instruments Data Acquisition PCI board, the 3 channels (VIdeo signal, Hsync and Vsync)are connected to the board. I have seen that SCSI drives can write at around 150MB/s speeds. If I can get my SCSI drive to drive at that speeds, I think my problem will be solved for the most part. Maybe now I am thinking more onto hardware side, do you think I can improve my drive speeds. I dont understand why it has such slow speeds while it is an U320 SCSI interface (ofcourse its not in any RAID array configuration, its the only drive the system is having) and being a workstation system (dell PWS 650). wil keep trying your other suggestions too. thanks, -Pavan
-
From a long time I am thinking on the same lines as your 3rd suggestion. But, I doubt that at somepoint the buffer will over run or we may skip data (as the HDD speeds are very very low compared to acquiring speeds). Actually the data of the 3 channels comes from a National Instruments Data Acquisition PCI board, the 3 channels (VIdeo signal, Hsync and Vsync)are connected to the board. I have seen that SCSI drives can write at around 150MB/s speeds. If I can get my SCSI drive to drive at that speeds, I think my problem will be solved for the most part. Maybe now I am thinking more onto hardware side, do you think I can improve my drive speeds. I dont understand why it has such slow speeds while it is an U320 SCSI interface (ofcourse its not in any RAID array configuration, its the only drive the system is having) and being a workstation system (dell PWS 650). wil keep trying your other suggestions too. thanks, -Pavan
So this information is coming from an PCI board? I'm surprised you get that high of a data rate using PCI ... but anyway. Try this: 1) Create a pool of 10-20 byte buffers to recieve your data. 2) Pull data from your device and load it into the first buffer. 3) When the buffer is full, pass it off to a thread to write out the data 4.1) Your main thread will select the next available buffer and continue to place the received data in it. 4.2) Your write theads will write the data to a file (presumably locking the file so you don't get sync errors) and then release the buffer back to the available pool. 5) Repeate 2-4 until complete. Your buffers will need to be small enough that you can easily allocate the memory for them, but large enough so that your write operations will be optimal (that is, if your drives write at 13 MB/sec, 13 MB might be a good size for you). If you want to get really daring, you might try to create a set of thread pools as well (just to gain some efficiency by not creating and closing threads constantly).
If you decide to become a software engineer, you are signing up to have a 1/2" piece of silicon tell you exactly how stupid you really are for 8 hours a day, 5 days a week Zac
-
So this information is coming from an PCI board? I'm surprised you get that high of a data rate using PCI ... but anyway. Try this: 1) Create a pool of 10-20 byte buffers to recieve your data. 2) Pull data from your device and load it into the first buffer. 3) When the buffer is full, pass it off to a thread to write out the data 4.1) Your main thread will select the next available buffer and continue to place the received data in it. 4.2) Your write theads will write the data to a file (presumably locking the file so you don't get sync errors) and then release the buffer back to the available pool. 5) Repeate 2-4 until complete. Your buffers will need to be small enough that you can easily allocate the memory for them, but large enough so that your write operations will be optimal (that is, if your drives write at 13 MB/sec, 13 MB might be a good size for you). If you want to get really daring, you might try to create a set of thread pools as well (just to gain some efficiency by not creating and closing threads constantly).
If you decide to become a software engineer, you are signing up to have a 1/2" piece of silicon tell you exactly how stupid you really are for 8 hours a day, 5 days a week Zac
Hi, I tried both ways, but still either I am not able to keep up with the read speed or the porgram hangs by taking up 100% of processor. Here are the 2 ways I tried for- Method1- 1) Created 5 different same size buffers for each of the 3 channels (so 3x5 = 15 different buffers) where the buffers for channel one are of size 100000*6bytes each and that of channels 2&3 are of 100000*1byte each. 2) Created 15 threads to process each of those 15 buffers whenever they are ready (initiated by the main acquisition thread as soon as a buffer becomes ready). 3) Created different binary files to write the corresponding buffer data (to overcome the file access issue). -In this case the program hangs as soon as I run it and locks the processor at 100% (I think its due to the pool of threads ruuning at the same time). Method2- 1) Created only 3 threads (for each channel data) to write any of the 5 buffers (allocated in the same fashion as before) depending upon their corresponding buffer flags (again initiated by the main acquisition thread, same as before). 2) Created only 3 files to write the data into (since at any given point of time, only 1 buffer is being accessed by the corresponding threads). -In this case too, my porcessor is being locked up for a while and at some point the data acquisition board gives out buffer overrun error. Here is a sample of how I am writing my thread- DWORD WINAPI StartAThread0(LPVOID param) { wwrite_ASignal0(); //for buffer[0] of channel1 return 0; } void write_ASignal0() { int i, dataA, dataA1 = -8191; do { if (m_bUpdateA[0]) { for (i=0; i<100000; i++) { if (dataAI[0][i] < 0) fwrite(&dataA1, sizeof(int), 1, streamA[0] ; else { dataA = (int)((dataAI[i][0]-1.25)*6552.8); fwrite(&dataA, sizeof(int), 1, streamA[0] ); } } m_bUpdateA[0] = false; } }while(m_bInitialize); } Where m_bInitialize keeps the thread running waiting for m_bUpdate[0] to activate. As soon as it does it writes the buffer to the corresponding file and deactivates the flag. In the sam fashion I am having for all the other 14 threads. Here is my application to be exact- The board is a Nation Instruments Data Acquisition board which can acquire data at a max of 10MHz. The data is acquired using their own native commands in C. The minimum buffer size I can use to acquire data is 10000 and then 100000. Any number other than these will give out buffer overrun error. So if I use 100000 samples, it means that the board acquires data at 10000000 sam
-
Hi, I tried both ways, but still either I am not able to keep up with the read speed or the porgram hangs by taking up 100% of processor. Here are the 2 ways I tried for- Method1- 1) Created 5 different same size buffers for each of the 3 channels (so 3x5 = 15 different buffers) where the buffers for channel one are of size 100000*6bytes each and that of channels 2&3 are of 100000*1byte each. 2) Created 15 threads to process each of those 15 buffers whenever they are ready (initiated by the main acquisition thread as soon as a buffer becomes ready). 3) Created different binary files to write the corresponding buffer data (to overcome the file access issue). -In this case the program hangs as soon as I run it and locks the processor at 100% (I think its due to the pool of threads ruuning at the same time). Method2- 1) Created only 3 threads (for each channel data) to write any of the 5 buffers (allocated in the same fashion as before) depending upon their corresponding buffer flags (again initiated by the main acquisition thread, same as before). 2) Created only 3 files to write the data into (since at any given point of time, only 1 buffer is being accessed by the corresponding threads). -In this case too, my porcessor is being locked up for a while and at some point the data acquisition board gives out buffer overrun error. Here is a sample of how I am writing my thread- DWORD WINAPI StartAThread0(LPVOID param) { wwrite_ASignal0(); //for buffer[0] of channel1 return 0; } void write_ASignal0() { int i, dataA, dataA1 = -8191; do { if (m_bUpdateA[0]) { for (i=0; i<100000; i++) { if (dataAI[0][i] < 0) fwrite(&dataA1, sizeof(int), 1, streamA[0] ; else { dataA = (int)((dataAI[i][0]-1.25)*6552.8); fwrite(&dataA, sizeof(int), 1, streamA[0] ); } } m_bUpdateA[0] = false; } }while(m_bInitialize); } Where m_bInitialize keeps the thread running waiting for m_bUpdate[0] to activate. As soon as it does it writes the buffer to the corresponding file and deactivates the flag. In the sam fashion I am having for all the other 14 threads. Here is my application to be exact- The board is a Nation Instruments Data Acquisition board which can acquire data at a max of 10MHz. The data is acquired using their own native commands in C. The minimum buffer size I can use to acquire data is 10000 and then 100000. Any number other than these will give out buffer overrun error. So if I use 100000 samples, it means that the board acquires data at 10000000 sam
Well, here's your problem:
fwrite(&dataA, sizeof(int), 1, streamA[0] );You need to reorganize this into a single write - have your buffers as a coninuous block of bytes and do the single operation in a single WriteFile call. Also, try using CreateFile, Readfile, and WriteFile and avoid the 'C' functions, which have to get decoded into one of the other three calls anyways. I usually have two layers in a case like this: Aquisition -> Buffer Writes Buffer Read -> Disk Writes You just need to synchronize the access t your 'buffer pointers' between the threads. Only two threads required, one doing IO and one doing disk writes. Also, defragment hte hard drive frequently and defragment the paging file. Turn off the Virus Scnaner if possible, at least on the folder doing the disk IO.Any sufficiently gross incompetence is nearly indistinguishable from malice.
-
Well, here's your problem:
fwrite(&dataA, sizeof(int), 1, streamA[0] );You need to reorganize this into a single write - have your buffers as a coninuous block of bytes and do the single operation in a single WriteFile call. Also, try using CreateFile, Readfile, and WriteFile and avoid the 'C' functions, which have to get decoded into one of the other three calls anyways. I usually have two layers in a case like this: Aquisition -> Buffer Writes Buffer Read -> Disk Writes You just need to synchronize the access t your 'buffer pointers' between the threads. Only two threads required, one doing IO and one doing disk writes. Also, defragment hte hard drive frequently and defragment the paging file. Turn off the Virus Scnaner if possible, at least on the folder doing the disk IO.Any sufficiently gross incompetence is nearly indistinguishable from malice.
Hi, thanks for your reply. I tried to use WriteFile, but how can I write non-text data using this command. It writes text data fine, but not any other datatypes. Also where can I set the mode to binary while creating the file using CreateFile (I have seen lot of flags, but none relating to binary mode). One more thing I would like to point out is, my porgram does I/O operations in a single function (it acquires the 2 digital channels data at the same time it acquires analog channel data). So I need to do the disk write operations for the 3 channels data at the same time within the time delay of next I/O or atleast within a span of 5 buffers. So, atlast my read-in rate is (100000x6bytes + 2x100000x1byte) data every 10msec. So I need to write that much data to 3 different files at the same time. I am using fwrite(&dataA, sizeof(int), 1, streamA[0] );, instead of writing the buffer in a single function, as I need to convert the datatype from 6bytes to 2bytes (using some scaling function) and from 1byte to 2bytes data. But I can do that even after I do the acquisition, if it can resolve my problem. thanks, -Pavan.
-
Hi, thanks for your reply. I tried to use WriteFile, but how can I write non-text data using this command. It writes text data fine, but not any other datatypes. Also where can I set the mode to binary while creating the file using CreateFile (I have seen lot of flags, but none relating to binary mode). One more thing I would like to point out is, my porgram does I/O operations in a single function (it acquires the 2 digital channels data at the same time it acquires analog channel data). So I need to do the disk write operations for the 3 channels data at the same time within the time delay of next I/O or atleast within a span of 5 buffers. So, atlast my read-in rate is (100000x6bytes + 2x100000x1byte) data every 10msec. So I need to write that much data to 3 different files at the same time. I am using fwrite(&dataA, sizeof(int), 1, streamA[0] );, instead of writing the buffer in a single function, as I need to convert the datatype from 6bytes to 2bytes (using some scaling function) and from 1byte to 2bytes data. But I can do that even after I do the acquisition, if it can resolve my problem. thanks, -Pavan.
The low level File IO does not 'care' about text versus binary. Createfile always makes a binary data file. You only make it a 'text' file by choosing to add CRLF or LF or LFCR or whatever to the 'end' of a line. Youc an write out an entire block of memory with a single functionc all to Writefile. You are killing your performance with the thousands of individual fwrite calls.
Any sufficiently gross incompetence is nearly indistinguishable from malice.
-
The low level File IO does not 'care' about text versus binary. Createfile always makes a binary data file. You only make it a 'text' file by choosing to add CRLF or LF or LFCR or whatever to the 'end' of a line. Youc an write out an entire block of memory with a single functionc all to Writefile. You are killing your performance with the thousands of individual fwrite calls.
Any sufficiently gross incompetence is nearly indistinguishable from malice.
I did time comparison between the two functions fwrite and writefile for writing 800000 bytes of data and they both take same amount of time (among 20 trails). I tried using 1 thread for writing all the 3 channels data using 10 buffers for each channel (once the IO records data into 1 buffer, it updates the corresponding buffer flag and as soon as it updates the write thread writes the corresponding 3 channels buffers to file and updates the correspoding flags to false. I tried this for 10 times for acquiring data about 5sec, in those 3 times it gave out files of same size and 3 times it gave out different size files (so I think it is missing some data in between while writing). Will check and try different methods of implementation and see if I can acheive it. thanks, -Pavan