How to avoid thrid party lib(no source codes) to allocate memory from physical memory?
-
Suppose this scenario: I refer to a third party lib in my C++ app, but I don't want the third party lib to use my physical memory at all. Instead, I want it to only allocate memory from hard disk. I don't know source codes of third party lib, however as it run in the Windows platform, so I think it's possible to control the memory management with Win32 API. My problem is how to avoid thrid party lib to allocate memory from physical memory. Am I going in the wrong direction? Anybody can help me? PS: I'm using Visual C++ 2010.
-
Suppose this scenario: I refer to a third party lib in my C++ app, but I don't want the third party lib to use my physical memory at all. Instead, I want it to only allocate memory from hard disk. I don't know source codes of third party lib, however as it run in the Windows platform, so I think it's possible to control the memory management with Win32 API. My problem is how to avoid thrid party lib to allocate memory from physical memory. Am I going in the wrong direction? Anybody can help me? PS: I'm using Visual C++ 2010.
As a library presumably it is using the standard windows libraries and it links to those dynamically. That means it is using an external calls to manage memory. So it should be possible to re-link and/or wrap the library to use a different memory management calls. I doubt that it is easy however. You might also want to carefully examine what problem you think this will solve. It certainly won't give you more memory. It is going to make it much slower, and if you are not careful about how your allocator works it could make it vastly slower.
-
You can't. The library will do what its code is designed to do, and without the source code you cannot alter that.
Use the best guess
Richard MacCutchan wrote:
The library will do what its code is designed to do,
Actually it's not just the library, but the OS as well. And as jschell wrote, if the library allocates memory through a DLL, that DLL could in theory be replaced by one that does what the OP intends. That said, I doubt that it can be easily achieved. More importantly I doubt that is actually what the OP needs: his requirement to have the library (or any part of an application) "allocate memory from hard disk" simply doesn't make sense. Data needs to be in memory in order to be processed, so you have to load them to memory eventually. More to the point: the allocation of memory always happens in memory (big surprise!). Depending on the reasons for this requirement, there may be another solution altogether - forcedly moving library allocations to hard disk is definitely not the solution, even if it were feasible with a reasonable amount of effort.
-
Richard MacCutchan wrote:
The library will do what its code is designed to do,
Actually it's not just the library, but the OS as well. And as jschell wrote, if the library allocates memory through a DLL, that DLL could in theory be replaced by one that does what the OP intends. That said, I doubt that it can be easily achieved. More importantly I doubt that is actually what the OP needs: his requirement to have the library (or any part of an application) "allocate memory from hard disk" simply doesn't make sense. Data needs to be in memory in order to be processed, so you have to load them to memory eventually. More to the point: the allocation of memory always happens in memory (big surprise!). Depending on the reasons for this requirement, there may be another solution altogether - forcedly moving library allocations to hard disk is definitely not the solution, even if it were feasible with a reasonable amount of effort.
-
Suppose this scenario: I refer to a third party lib in my C++ app, but I don't want the third party lib to use my physical memory at all. Instead, I want it to only allocate memory from hard disk. I don't know source codes of third party lib, however as it run in the Windows platform, so I think it's possible to control the memory management with Win32 API. My problem is how to avoid thrid party lib to allocate memory from physical memory. Am I going in the wrong direction? Anybody can help me? PS: I'm using Visual C++ 2010.
Falconapollo wrote:
My problem is how to avoid thrid party lib to allocate memory from physical memory.
No, that is not your problem - that is a suggestion on how to solve your (as of yet unknown) problem. And it won't work for several reasons: 1. The operating system will do all memory allocations; most importantly even the most basic function you can call to allocate memory will allocate that within physical memory. There's no way to avoid that short of writing your own OS. 2. The OS will also move memory from and to hard disk as needed in order to keep the entire system running smoothly. Repeatedly. At runtime! Again, you cannot influence that short of rewriting the OS. 3. Technically, no application or library actually allocates or processes data in physical memory; instead they work on virtual memory. Under the hood, the OS will put the actual data anywhere into physical memory or on hard disk, but neither your application nor that library would even be able to tell where their data are currently residing, because from their point of view all data always resides in virtual memory. There is no way (and no sensible reason) to change that. Well, short of rewriting the OS ;) So, please start again, and start by describing your problem, not by suggesting a solution. Are you running out of memory? Are you experiencing performance issues? Are you suffering from memory fragmentation? Any of these could be a problem, and there may be reasonable solutions. But either solution will not solve all of these problems, so knowing the problem would help.
-
Falconapollo wrote:
My problem is how to avoid thrid party lib to allocate memory from physical memory.
No, that is not your problem - that is a suggestion on how to solve your (as of yet unknown) problem. And it won't work for several reasons: 1. The operating system will do all memory allocations; most importantly even the most basic function you can call to allocate memory will allocate that within physical memory. There's no way to avoid that short of writing your own OS. 2. The OS will also move memory from and to hard disk as needed in order to keep the entire system running smoothly. Repeatedly. At runtime! Again, you cannot influence that short of rewriting the OS. 3. Technically, no application or library actually allocates or processes data in physical memory; instead they work on virtual memory. Under the hood, the OS will put the actual data anywhere into physical memory or on hard disk, but neither your application nor that library would even be able to tell where their data are currently residing, because from their point of view all data always resides in virtual memory. There is no way (and no sensible reason) to change that. Well, short of rewriting the OS ;) So, please start again, and start by describing your problem, not by suggesting a solution. Are you running out of memory? Are you experiencing performance issues? Are you suffering from memory fragmentation? Any of these could be a problem, and there may be reasonable solutions. But either solution will not solve all of these problems, so knowing the problem would help.
my real problem is "running out of memory", this make me crazy, i don't know how to solve the issue. because it's the no-source lib who eat up my memory. but, i must use the lib. so, i'm stuck here. please give my your suggestion?? any help will be appreciated!
-
Richard MacCutchan wrote:
The library will do what its code is designed to do,
Actually it's not just the library, but the OS as well. And as jschell wrote, if the library allocates memory through a DLL, that DLL could in theory be replaced by one that does what the OP intends. That said, I doubt that it can be easily achieved. More importantly I doubt that is actually what the OP needs: his requirement to have the library (or any part of an application) "allocate memory from hard disk" simply doesn't make sense. Data needs to be in memory in order to be processed, so you have to load them to memory eventually. More to the point: the allocation of memory always happens in memory (big surprise!). Depending on the reasons for this requirement, there may be another solution altogether - forcedly moving library allocations to hard disk is definitely not the solution, even if it were feasible with a reasonable amount of effort.
thank you for the suggestion. yes, i'm going in the woring direction. i have to find another solution.
-
Falconapollo wrote:
My problem is how to avoid thrid party lib to allocate memory from physical memory.
No, that is not your problem - that is a suggestion on how to solve your (as of yet unknown) problem. And it won't work for several reasons: 1. The operating system will do all memory allocations; most importantly even the most basic function you can call to allocate memory will allocate that within physical memory. There's no way to avoid that short of writing your own OS. 2. The OS will also move memory from and to hard disk as needed in order to keep the entire system running smoothly. Repeatedly. At runtime! Again, you cannot influence that short of rewriting the OS. 3. Technically, no application or library actually allocates or processes data in physical memory; instead they work on virtual memory. Under the hood, the OS will put the actual data anywhere into physical memory or on hard disk, but neither your application nor that library would even be able to tell where their data are currently residing, because from their point of view all data always resides in virtual memory. There is no way (and no sensible reason) to change that. Well, short of rewriting the OS ;) So, please start again, and start by describing your problem, not by suggesting a solution. Are you running out of memory? Are you experiencing performance issues? Are you suffering from memory fragmentation? Any of these could be a problem, and there may be reasonable solutions. But either solution will not solve all of these problems, so knowing the problem would help.
Although posted in another forum, I believe it is related to this: http://www.codeproject.com/Messages/4583101/How-to-force-Cplusplus-allocate-memory-from-disk.aspx[^]. Soren Madsen
"When you don't know what you're doing it's best to do it quickly" - Jase #DuckDynasty
-
my real problem is "running out of memory", this make me crazy, i don't know how to solve the issue. because it's the no-source lib who eat up my memory. but, i must use the lib. so, i'm stuck here. please give my your suggestion?? any help will be appreciated!
Ok, there are probably several ways to solve this, but first let me clear up a misconception: The amount of physical memory available to your computer does not limit the amount of memory available to your application. Therefore, forcing a library to not work on physical memory - if that were possible - wouldn't change a thing! It is the OS that decides where allocated objects reside, and, yes, that does include the hard disk. The OS already moves objects from physical memory to the hard disk if it is running low on physical memory. And it does that at runtime. Your actual problem is that you are running out of virtual memory. Virtual memory is an abstract address space that the OS grants every application. Depending on your Windows version and settings, that address space may encompass anything between 2 and 3 GB at most, even if your machine has 16 GB of physical RAM. There are two probable causes for your problem: 1. The amount of data you have to deal with is simply more than this library can handle. If this is the problem, then you should consider ways how to partition your data into smaller chunks, and adapt your operations to process that data bit by bit. Loading a 4 GB file into memory simply does not work! 2. You may not be using the library as intended: many library functions create objects and return them as results. The intention is that you call appropriate destruction functions once you are finished with those objects. If you don't do that, the amount of memory you need will continually increase over time until you run out of memory. (and you may think it's the library that is the problem because the library does the allocation You should check the library documentation to see which of the functions create objects and expect you to release those objecs again, or call corresponding functions that clean them up properly. Also check your algorithms to see whether they 'lock' these objects unnecessarily, or for too long.
-
Ok, there are probably several ways to solve this, but first let me clear up a misconception: The amount of physical memory available to your computer does not limit the amount of memory available to your application. Therefore, forcing a library to not work on physical memory - if that were possible - wouldn't change a thing! It is the OS that decides where allocated objects reside, and, yes, that does include the hard disk. The OS already moves objects from physical memory to the hard disk if it is running low on physical memory. And it does that at runtime. Your actual problem is that you are running out of virtual memory. Virtual memory is an abstract address space that the OS grants every application. Depending on your Windows version and settings, that address space may encompass anything between 2 and 3 GB at most, even if your machine has 16 GB of physical RAM. There are two probable causes for your problem: 1. The amount of data you have to deal with is simply more than this library can handle. If this is the problem, then you should consider ways how to partition your data into smaller chunks, and adapt your operations to process that data bit by bit. Loading a 4 GB file into memory simply does not work! 2. You may not be using the library as intended: many library functions create objects and return them as results. The intention is that you call appropriate destruction functions once you are finished with those objects. If you don't do that, the amount of memory you need will continually increase over time until you run out of memory. (and you may think it's the library that is the problem because the library does the allocation You should check the library documentation to see which of the functions create objects and expect you to release those objecs again, or call corresponding functions that clean them up properly. Also check your algorithms to see whether they 'lock' these objects unnecessarily, or for too long.
Thank you for your advice. Yes, maybe I can do some cleanup work.
-
thank you for the suggestion. yes, i'm going in the woring direction. i have to find another solution.
What is the problem that you are trying to solve?
-
I'm not sure what this has to do with OP's question. He asked how to modify a DLL without the source code.
Use the best guess
If you're dynamically creating the objects in your app, you can overload the 'new' operator. But that's not really going to do the trick. As others have mentioned data has to be in memory to be used.
-
Thank you for your advice. Yes, maybe I can do some cleanup work.
That's the best place to start. Make sure you have no memory leaks. If there is some reason that you need gigabytes of data in memory to do some heavy complex processing (perhaps DNA analysis, simulations etc...,) then upgrade your hardware and go 64 bit.
-
If you're dynamically creating the objects in your app, you can overload the 'new' operator. But that's not really going to do the trick. As others have mentioned data has to be in memory to be used.
-
I don't think you meant to post this to me, or (maybe) even to this thread.
Use the best guess
I meant it for the thread anyway...
-
Falconapollo wrote:
My problem is how to avoid thrid party lib to allocate memory from physical memory.
No, that is not your problem - that is a suggestion on how to solve your (as of yet unknown) problem. And it won't work for several reasons: 1. The operating system will do all memory allocations; most importantly even the most basic function you can call to allocate memory will allocate that within physical memory. There's no way to avoid that short of writing your own OS. 2. The OS will also move memory from and to hard disk as needed in order to keep the entire system running smoothly. Repeatedly. At runtime! Again, you cannot influence that short of rewriting the OS. 3. Technically, no application or library actually allocates or processes data in physical memory; instead they work on virtual memory. Under the hood, the OS will put the actual data anywhere into physical memory or on hard disk, but neither your application nor that library would even be able to tell where their data are currently residing, because from their point of view all data always resides in virtual memory. There is no way (and no sensible reason) to change that. Well, short of rewriting the OS ;) So, please start again, and start by describing your problem, not by suggesting a solution. Are you running out of memory? Are you experiencing performance issues? Are you suffering from memory fragmentation? Any of these could be a problem, and there may be reasonable solutions. But either solution will not solve all of these problems, so knowing the problem would help.
Stefan_Lang wrote:
2. The OS will also move memory from and to hard disk as needed in order to keep the entire system running smoothly. Repeatedly. At runtime! Again, you cannot influence that short of rewriting the OS
I am rather certain that all desktop OSes provide access to some very raw memory management APIs. Without that there are whole classifications of applications that could not be written. As an example of one API for windows, I presume there are others, the following insures that memory is not written to the hard drive. http://msdn.microsoft.com/en-us/library/aa366895%28VS.85%29.aspx[^]
-
Suppose this scenario: I refer to a third party lib in my C++ app, but I don't want the third party lib to use my physical memory at all. Instead, I want it to only allocate memory from hard disk. I don't know source codes of third party lib, however as it run in the Windows platform, so I think it's possible to control the memory management with Win32 API. My problem is how to avoid thrid party lib to allocate memory from physical memory. Am I going in the wrong direction? Anybody can help me? PS: I'm using Visual C++ 2010.
If your problem is that your application fails because over time this library keeps consuming memory then the solution is to remove the library from your application. You can do that as follows. 1. Create an executable that wraps the library. 2. Provide a communication idiom: files, stdio or sockets. That provides the necessary API for your business needs (basically a proxy for what you are already doing.) 3. In your actual application use WinProcess (or whatever the method is called) to execute 1 and manage it. 4. Use the communication from 2 from your application in place of the straight to library calls you are currently using. Other variations on the above depend on your needs - Create multiple instance - Stop each instance after X (1 or more) uses - Have min/max instances and use a locked queue to send requests for processing - Detect failures, restart and retry a request. - Detect excessive memory usage, terminate and restart.
-
Stefan_Lang wrote:
2. The OS will also move memory from and to hard disk as needed in order to keep the entire system running smoothly. Repeatedly. At runtime! Again, you cannot influence that short of rewriting the OS
I am rather certain that all desktop OSes provide access to some very raw memory management APIs. Without that there are whole classifications of applications that could not be written. As an example of one API for windows, I presume there are others, the following insures that memory is not written to the hard drive. http://msdn.microsoft.com/en-us/library/aa366895%28VS.85%29.aspx[^]
Quote:
I am rather certain that all desktop OSes provide access to some very raw memory management APIs.
yes, i can't agree more. But I don't find what i need.
-
If your problem is that your application fails because over time this library keeps consuming memory then the solution is to remove the library from your application. You can do that as follows. 1. Create an executable that wraps the library. 2. Provide a communication idiom: files, stdio or sockets. That provides the necessary API for your business needs (basically a proxy for what you are already doing.) 3. In your actual application use WinProcess (or whatever the method is called) to execute 1 and manage it. 4. Use the communication from 2 from your application in place of the straight to library calls you are currently using. Other variations on the above depend on your needs - Create multiple instance - Stop each instance after X (1 or more) uses - Have min/max instances and use a locked queue to send requests for processing - Detect failures, restart and retry a request. - Detect excessive memory usage, terminate and restart.
Thank you for the specific suggestions. i will have a try.
-
What is the problem that you are trying to solve?
i'm trying to solve "out of memory" problem, it's the no-source third party lib who eat up my memory. but i'm rather my usage is right, and i have to do this way. so, i'm trying to find a way to make the third patry lib occupy less memory.