Server Memory Management - large database.

[Mike Doner]

Hi all, Does anybody have any experience with Application Memory Management using a proprietary database? We have hit a bit of a snag using the 2gig process address space with our product for customers who have large amounts of data. Our server manages several databases, each with 200+ tables, and up to 65,000 records in each table. Currently, when the server fires up, we load in all the databases into Virtual Memory regardless if they're needed or not. As part of the load process, we "reserve" the maximum number of possible memory for each table so we can easily add records to the end that table. We only commit to the number of records that are active. Problem is, Windows will only allow us to "reserve" up to the 2gig mark. We've considered a few options: - including only loading in the required databases - but it IS possible that at any given time, all databases will be required (Client/Server application). - Writing a mechanism that would leave not recently used tables on disk until required. - Use a process for each database - could get messy with mutex's etc. A few questions: - Does anybody have any experience here? What kind of mechanism does your software use? - Is there a way to elegantly go beyond the 2gig limit? - How does SQL/MySql handle large databases - ie: When a new record is added to the table - does it append the new record to the end of the allocated memory block? Unfortunately for our market, the evil proprietary database is the way to go. We're not in a position to change to migrate to anything new at the moment. Any information/suggestions you might have would be greatly appreciated. Thanks. Mike.

[Mark Salsbery]

Have you considered using File Mapping[^]? Mark

Mark Salsbery Microsoft MVP - Visual C++ :java:

[upredsun]

you can use mmap function to store the data or you can use timesten product. Best, Kevin Jo http://www.upredsun.com **Easily and automatically build tcp-based or udp-based network protocol source code**

[Mike Doner]

Hi Mark, Thanks for your reply. We are using CreateFileMapping(), MapViewofFile() etc. We use this basically to give names to blocks of memory. We load each table off disk into blocks of memory managed by CreateFileMapping/VirtualAlloc. I guess the biggest problem is the addition of new objects to a table and the memory we reserve. Each table has a maximum number of objects... We reserve up to the maximum size required for that table and commit up to the number of active objects. When we reserve memory, its counted towards the 2g process limit. We reserve the bytes so that when a new object is added, we just need to commit "objectsize" at the end of the last committed bytes, we're not moving memory around or doing any further juggling. Is there something I'm overlooking with File mapping that might help us out? Thanks again for your reply.