How to force C++ allocate memory from disk?
-
I have thousands lines of C++ codes which work well on small text files, but crashes on huge text files (such as 2 GB size). Crash reason: app eats up memory. Is it possible to allocate memory from disk? Because in most case, hard disk space is much bigger than physical memory. If I can borrow some space from hard disk for my app and return them back after use, then my app has little chance to crash. Here are my design thoughts: 1.Create a temporary file for file mapping(CreateFileMapping, OpenFileMapping) 2.Force my app to allocate memory from the temporary file 3.Do some clean up work: CloseFileMapping and delete the temporary file Because I have so much existing code, if my design thought is reasonable, I don't want to redesign my project. I'm not sure if the design thought is possible to implement. Anybody can help me? PS: I'm using Visual C++ 2010.
-
I have thousands lines of C++ codes which work well on small text files, but crashes on huge text files (such as 2 GB size). Crash reason: app eats up memory. Is it possible to allocate memory from disk? Because in most case, hard disk space is much bigger than physical memory. If I can borrow some space from hard disk for my app and return them back after use, then my app has little chance to crash. Here are my design thoughts: 1.Create a temporary file for file mapping(CreateFileMapping, OpenFileMapping) 2.Force my app to allocate memory from the temporary file 3.Do some clean up work: CloseFileMapping and delete the temporary file Because I have so much existing code, if my design thought is reasonable, I don't want to redesign my project. I'm not sure if the design thought is possible to implement. Anybody can help me? PS: I'm using Visual C++ 2010.
The most elegant approach is to use MapViewOfFile to only map a smaller piece of the file at any one time, one that does fit into your address space, and then shift the view by Unmap/Map-ing. In conjunction with this, a way to slightly extend the limit of what can fit in your address space is to set the /LARGEADDRESSAWARE compiler switch, which can be found under the properties. This tells the compiler that you are aware of that 32 bits pointer can use the full range of 0 - 2^32-1, rather than only half that range 2^31-1. With this, you might be able to reach up to ~3GB, depending on system configuration and your application. However, this only slightly shifts the available memory. Using the above approach, you map the piece of file that you want to use into your address space, and as soon as you are done with the current piece you map the next, etc.
-
The most elegant approach is to use MapViewOfFile to only map a smaller piece of the file at any one time, one that does fit into your address space, and then shift the view by Unmap/Map-ing. In conjunction with this, a way to slightly extend the limit of what can fit in your address space is to set the /LARGEADDRESSAWARE compiler switch, which can be found under the properties. This tells the compiler that you are aware of that 32 bits pointer can use the full range of 0 - 2^32-1, rather than only half that range 2^31-1. With this, you might be able to reach up to ~3GB, depending on system configuration and your application. However, this only slightly shifts the available memory. Using the above approach, you map the piece of file that you want to use into your address space, and as soon as you are done with the current piece you map the next, etc.
thank you for your advice, i will have a try. :)