Memory leak
-
People, I found that CFileDialog class causes serious problem. I had a large free memory block in the address space of my prog: 1200 MB Then I create CFileDialog and delete it: CFileDialog *pdlg=new CFileDialog(TRUE); pdlg->DoModal(); delete pdlg; I check address space again and found only 700 MB: The reason was very simple: I found some dlls in the address space of my process like urlmon.dll, netapi.dll, modemInst.dll and so on. These dlls was loaded into my address space by CFileDialog class and was not unloaded by its destructor. As a result, I couldn't allocate memory for a large file- address space had been fragmented. Any idea how to solve this situation? I think to create CFileDialog in a separate process. May be, there are better variants?
Read here, especially the last part.
"One man's wage rise is another man's price increase." - Harold Wilson
"Fireproof doesn't mean the fire will never come. It means when the fire comes that you will be able to withstand it." - Michael Simmons
"Show me a community that obeys the Ten Commandments and I'll show you a less crowded prison system." - Anonymous
-
If you're sure the memory is being taken up by those dlls, why not try unloading them yourself with
FreeLibrary()
? How are you trying to allocate the memory? Have you tried the VirtualAlloc() function?
The difficult we do right away... ...the impossible takes slightly longer.
[quote]If you're sure the memory is being taken up by those dlls, why not try unloading them yourself with FreeLibrary()? How are you trying to allocate the memory? Have you tried the VirtualAlloc() function?[/quote] Yeah, I'm sure because I saw list of modules, downloaded in the address space before creation of CFileDialog and after it had been destructed. I tried to use FreeLibrary for such modules. After it (not immediately) program failed, giving me some inaudible error messages. Yeah, I used VirtualAlloc(). It is the best if you need to allocate large memory block. Currently I created a new process, which works with CFileDialog, leaving my address space untouched. It works, but I think that it is not very elegant approach. And one mysterious dll still presents- modemInst.dll (it is not connected with CFileDialog).
-
If you're sure the memory is being taken up by those dlls, why not try unloading them yourself with
FreeLibrary()
? How are you trying to allocate the memory? Have you tried the VirtualAlloc() function?
The difficult we do right away... ...the impossible takes slightly longer.
-
People, I found that CFileDialog class causes serious problem. I had a large free memory block in the address space of my prog: 1200 MB Then I create CFileDialog and delete it: CFileDialog *pdlg=new CFileDialog(TRUE); pdlg->DoModal(); delete pdlg; I check address space again and found only 700 MB: The reason was very simple: I found some dlls in the address space of my process like urlmon.dll, netapi.dll, modemInst.dll and so on. These dlls was loaded into my address space by CFileDialog class and was not unloaded by its destructor. As a result, I couldn't allocate memory for a large file- address space had been fragmented. Any idea how to solve this situation? I think to create CFileDialog in a separate process. May be, there are better variants?
A program using DLLs may delay loading them unto the point when they're first needed. This appears to have happened here. The problem with that is that the DLLs will then stay until you exit your program, or manually unload them (as suggested above). Why CFileDialog loads all these DLLs isn't clear to me. My guess would be file preview settings - but then I would have thought the DLLs used for previewing would be loaded to system memory, not application memory. Anyway, this is not in fact a memory leak, "only" a shortage of memory. Moreover, you should normally not even be able to notice memory fragmentation, except for the fact that the operators
newanddeletenotably slow down when you're low on memory. The system memory manager hides the true physical memory addresses from you, and translates them to a virtual application memory address space. So even if you get addresses from allocations that indicate a state of fragmentation, this may be misleading. That said, the memory manager will easily allocate huge blocks of memory for you even if fragmentation does split up your memory into unusably small chunks: it has the ability to pretend a contiguous block of memory, even if the physical memory is spread out all over the entire available physical memory space. Your problem is a different one however: you cannot load a large file into memory as a whole. that means your application is using more memory than available. Note that even with 16 GB RAM, the memory space of an application is limited to no more than 4 GBm or in Windows in fact only about 2-3 GB, depending ont the OS version. The solution is to split it up. It isn't a good idea to load such big files into memory anyway - exactly because of the problem you encountered. Depending on what you wish to do with that file, you don't need the full file at any point in time anyway. Instead use a buffer of at most a few MB in size, load a chunk, process the data, store the intermediary results, then repeat until you're done. -
Freeing libraries you didn't load is dangerous, it's like releasing a reference you don't hold on a COM object.
Steve
Thank you, I didn't realize that.
The difficult we do right away... ...the impossible takes slightly longer.
-
A program using DLLs may delay loading them unto the point when they're first needed. This appears to have happened here. The problem with that is that the DLLs will then stay until you exit your program, or manually unload them (as suggested above). Why CFileDialog loads all these DLLs isn't clear to me. My guess would be file preview settings - but then I would have thought the DLLs used for previewing would be loaded to system memory, not application memory. Anyway, this is not in fact a memory leak, "only" a shortage of memory. Moreover, you should normally not even be able to notice memory fragmentation, except for the fact that the operators
newanddeletenotably slow down when you're low on memory. The system memory manager hides the true physical memory addresses from you, and translates them to a virtual application memory address space. So even if you get addresses from allocations that indicate a state of fragmentation, this may be misleading. That said, the memory manager will easily allocate huge blocks of memory for you even if fragmentation does split up your memory into unusably small chunks: it has the ability to pretend a contiguous block of memory, even if the physical memory is spread out all over the entire available physical memory space. Your problem is a different one however: you cannot load a large file into memory as a whole. that means your application is using more memory than available. Note that even with 16 GB RAM, the memory space of an application is limited to no more than 4 GBm or in Windows in fact only about 2-3 GB, depending ont the OS version. The solution is to split it up. It isn't a good idea to load such big files into memory anyway - exactly because of the problem you encountered. Depending on what you wish to do with that file, you don't need the full file at any point in time anyway. Instead use a buffer of at most a few MB in size, load a chunk, process the data, store the intermediary results, then repeat until you're done.To unload libraries which you did not load is unsafe op- just as Stephen wrote. try delete twice the same object and you'll see what will happen. To perform this op, one has to know how they were loaded and unload them exact in opposite order. Moreover, he must know, that nobody else will try to unload these libraries too. I tried, and program failed. CFileDialog loads all these libraries for the very simple reason. It offers such service: one may open file from remote disk for example (or you may type an URL in address bar). Dear Stefan, you are not right about memory manager. I do not see physical addresses, but only virtual ones. In my virtual address space I have to have a continuous block of free memory in virtual address space to allocate large file (and no matter, what I use- VirtualAlloc, new or MapViewOfFile). I can clearly see fragmentation of my address space, when I run my prog under debugger. As I wrote, I had 1.2 GB continuous block before creation of CFileDialog and only 0.7 GB after it. If I can to download large file, I will do it. To split file is the last chance. Any idea, what headache will I have, if for example I need to rotate large file, which cannot be downloaded in memory?
-
To unload libraries which you did not load is unsafe op- just as Stephen wrote. try delete twice the same object and you'll see what will happen. To perform this op, one has to know how they were loaded and unload them exact in opposite order. Moreover, he must know, that nobody else will try to unload these libraries too. I tried, and program failed. CFileDialog loads all these libraries for the very simple reason. It offers such service: one may open file from remote disk for example (or you may type an URL in address bar). Dear Stefan, you are not right about memory manager. I do not see physical addresses, but only virtual ones. In my virtual address space I have to have a continuous block of free memory in virtual address space to allocate large file (and no matter, what I use- VirtualAlloc, new or MapViewOfFile). I can clearly see fragmentation of my address space, when I run my prog under debugger. As I wrote, I had 1.2 GB continuous block before creation of CFileDialog and only 0.7 GB after it. If I can to download large file, I will do it. To split file is the last chance. Any idea, what headache will I have, if for example I need to rotate large file, which cannot be downloaded in memory?
Yes, unloading is a risk. I've read Stephens comment and therefore omitted that warning. Maybe I should have pointed it out nonetheless. Regarding fragmentation, maybe I haven't been clear. Indeed, all you see is virtual addresses. That's what I meant when I said the MM hides the true addresses from you. As a consequence, you won't know if the addresses you see point to two blocks of memory that lie next to each other, or if they are sperated by a 5GB gap. In fact, you don't even know if one of those blocks is in fact contiguous - part of it may have been moved to the page file! In any case, as pointed out before, the state of fragmentation is irrelevant to your problem. Your problem is you're running out of memory. You only have a few options: a) reduce memory usage elsewhere; apparently that may not be easy or even work b) make your application a 64 bit application, enabling it to use a bigger address space; of course this only helps if your total memory is big enough, but you can increase available memory by increasing the page file on your system. c) split up your file and adapt your program to work on it bit by bit.
-
Yes, unloading is a risk. I've read Stephens comment and therefore omitted that warning. Maybe I should have pointed it out nonetheless. Regarding fragmentation, maybe I haven't been clear. Indeed, all you see is virtual addresses. That's what I meant when I said the MM hides the true addresses from you. As a consequence, you won't know if the addresses you see point to two blocks of memory that lie next to each other, or if they are sperated by a 5GB gap. In fact, you don't even know if one of those blocks is in fact contiguous - part of it may have been moved to the page file! In any case, as pointed out before, the state of fragmentation is irrelevant to your problem. Your problem is you're running out of memory. You only have a few options: a) reduce memory usage elsewhere; apparently that may not be easy or even work b) make your application a 64 bit application, enabling it to use a bigger address space; of course this only helps if your total memory is big enough, but you can increase available memory by increasing the page file on your system. c) split up your file and adapt your program to work on it bit by bit.
Have you ever read what I wrote? My problem is NOT that I'm running out of memory. I had enough of it before I created CFileDialog. I wanted open a file of size 880 MB, having 1.2 GB of free memory memory. But I had not a continuous block of memory after CFileDialog was destructed. One very simple but not elegant way to resolve this situation was to allocate 1.2 GB using VirtualAlloc, then create CFileDialog. It downloads its modules in memory anyway, but large block will remain intact. Memory would not be as fragmented as it was if I previously created CFileDialog. Another way, to which I came up was to create a separate process, create CFileDialog within it and return what I needed (path to file) to the parent process using one of appropriate methods (I used memory mapped file). It is a programming- there are usually many ways to do same thing. I thought about which method is the best. May be, there is a way to get CFileDialog to work more correctly than it really does. I did not dig in its source code. I do have too much problems with algorithms to waste time for such thing. When you are short of resources (few memory, weak processor and so on) it definitely makes you more disciplined. And I have no intention to buy another comp and 64 bit OS - let's my boss do it.
-
Have you ever read what I wrote? My problem is NOT that I'm running out of memory. I had enough of it before I created CFileDialog. I wanted open a file of size 880 MB, having 1.2 GB of free memory memory. But I had not a continuous block of memory after CFileDialog was destructed. One very simple but not elegant way to resolve this situation was to allocate 1.2 GB using VirtualAlloc, then create CFileDialog. It downloads its modules in memory anyway, but large block will remain intact. Memory would not be as fragmented as it was if I previously created CFileDialog. Another way, to which I came up was to create a separate process, create CFileDialog within it and return what I needed (path to file) to the parent process using one of appropriate methods (I used memory mapped file). It is a programming- there are usually many ways to do same thing. I thought about which method is the best. May be, there is a way to get CFileDialog to work more correctly than it really does. I did not dig in its source code. I do have too much problems with algorithms to waste time for such thing. When you are short of resources (few memory, weak processor and so on) it definitely makes you more disciplined. And I have no intention to buy another comp and 64 bit OS - let's my boss do it.
Sorry, apparently I've skipped part of your message in reading. So what you're seeing is a fragmentation of the virtual address space of your process, and that prevents the allocation of a block that doesn't fit into that space. My next suggestion then would be to allocate a big block of memory before calling CFileDialog, like you did. You don't even need VirtualAlloc for that, you could just call new, and then create your buffer using in-place construction. This of course requires that you know in advance the size of your file. But then, if the size would vary, there would be no point in even trying to load it into memory as a whole: if you don't know the size, you have to assume that you will eventually encounter files that don't fit into memory, no matter what! That said, even if the file is always the same size, where is it that this aplication will run? Is it on your PC? If not, a client running the program may have less memory available than you do! One last idea is to split the file buffer into multiple parts, allowing you to use up most of the available memory in spite of fragmentation. That way you could keep all of it in memory, just not within one contiguous block. You could even write a wrapper class that does behave like a contiguous block of memory, but internally translates an address into a location within one of those buffers. P.S.: here's some code to illustrate my suggestion:
template
class DistributedBuffer {
unsigned char* blocks[1+(total_size-1)/chunk_size];
public:
DistributedBuffer() {
for (unsigned int i = 0; i < 1+(total_size-1)/chunk_size; ++i)
blocks[i] = new unsigned char[chunk_size];
}
// read from file address
unsigned char operator[](unsigned int offset) const {
return blocks[offset/chunk_size][offset%chunk_size];
}
// write to file address
unsigned char& operator[](unsigned int offset) {
return blocks[offset/chunk_size][offset%chunk_size];
}
};
int foo() {
// create 1 GB buffer by allocating 1024 blocks of 1 MB each
DistributedBuffer<1024*1024*1024, 1024*1024> myFileBuffer;
// now do some work:
myFileBuffer[12345678] = 17; // calling unsigned char& operator[](unsigned int offset)
unsigned char c = myFileBuffer[12345678]; // calling unsigned char operator[](unsigned int offset) const
return (int)c; // returns 17
} -
Sorry, apparently I've skipped part of your message in reading. So what you're seeing is a fragmentation of the virtual address space of your process, and that prevents the allocation of a block that doesn't fit into that space. My next suggestion then would be to allocate a big block of memory before calling CFileDialog, like you did. You don't even need VirtualAlloc for that, you could just call new, and then create your buffer using in-place construction. This of course requires that you know in advance the size of your file. But then, if the size would vary, there would be no point in even trying to load it into memory as a whole: if you don't know the size, you have to assume that you will eventually encounter files that don't fit into memory, no matter what! That said, even if the file is always the same size, where is it that this aplication will run? Is it on your PC? If not, a client running the program may have less memory available than you do! One last idea is to split the file buffer into multiple parts, allowing you to use up most of the available memory in spite of fragmentation. That way you could keep all of it in memory, just not within one contiguous block. You could even write a wrapper class that does behave like a contiguous block of memory, but internally translates an address into a location within one of those buffers. P.S.: here's some code to illustrate my suggestion:
template
class DistributedBuffer {
unsigned char* blocks[1+(total_size-1)/chunk_size];
public:
DistributedBuffer() {
for (unsigned int i = 0; i < 1+(total_size-1)/chunk_size; ++i)
blocks[i] = new unsigned char[chunk_size];
}
// read from file address
unsigned char operator[](unsigned int offset) const {
return blocks[offset/chunk_size][offset%chunk_size];
}
// write to file address
unsigned char& operator[](unsigned int offset) {
return blocks[offset/chunk_size][offset%chunk_size];
}
};
int foo() {
// create 1 GB buffer by allocating 1024 blocks of 1 MB each
DistributedBuffer<1024*1024*1024, 1024*1024> myFileBuffer;
// now do some work:
myFileBuffer[12345678] = 17; // calling unsigned char& operator[](unsigned int offset)
unsigned char c = myFileBuffer[12345678]; // calling unsigned char operator[](unsigned int offset) const
return (int)c; // returns 17
}Thanks for tips. I think, that such class will slow my prog. As I wrote, I tried to allocate large memory block and only after it create CFileDialog. But even if I can preserve large memory block for my own use, I'll loose some volume of memory anyway- all shit that CFileDialog loads in my address space will stay there even after CFileDialog will have been destructed. I prefer to keep my address space clean. One way is for example to change base addresses for my libraries like FreeImage.dll, jpeglib.dll and so on- there is such useful utility named Rebase.exe. On the other hand, I'm really short of memory and this is why I prefer do not load all unnecessary libraries into address space. What I did- another separated process- gave me what I wanted. I cannot change base addresses of system libraries- it may help only if my prog runs at my own comp. There are many useful things, which can be implemented, but now the main problem for me is effectiveness of algorithms which I use.
