File caching; cache manager; (cache interceptation)
-
hi! I've got a problem: I need to write a cache driver to cache some explicitly selected files with HUGE caches. The real situation is, that I should cache 3-4 files (each can have more GB in size, and can be placed on local workstation, local network or internet) with 1-30 MB caches (in memory!) per file. I tried to find some infos in the WinNT 4.0 DDK about the cache manager, but I didn't find any (now I'm searching some MSDN CDs) One basic idea of mine is, that actually I should intercept any file reads (and writes) like anti virus resident guards (shields) do, but this is only the idea... no code, no docs, so can anybody help me? The cache sould be completly transparent to other programs, but there are some limitations: 1) for the beginning it's sufficient to cache file reads, and 2) the file reads are generally sequencial (means that is enough to have the cacheing efficient for sequiencial file reads from the start to the end) alex ps. tips, websites, docs, etc. would also good, sample code would be the best;
-
hi! I've got a problem: I need to write a cache driver to cache some explicitly selected files with HUGE caches. The real situation is, that I should cache 3-4 files (each can have more GB in size, and can be placed on local workstation, local network or internet) with 1-30 MB caches (in memory!) per file. I tried to find some infos in the WinNT 4.0 DDK about the cache manager, but I didn't find any (now I'm searching some MSDN CDs) One basic idea of mine is, that actually I should intercept any file reads (and writes) like anti virus resident guards (shields) do, but this is only the idea... no code, no docs, so can anybody help me? The cache sould be completly transparent to other programs, but there are some limitations: 1) for the beginning it's sufficient to cache file reads, and 2) the file reads are generally sequencial (means that is enough to have the cacheing efficient for sequiencial file reads from the start to the end) alex ps. tips, websites, docs, etc. would also good, sample code would be the best;
Since we're talking NT here, there isn't a reasonably priced solution I'm aware of. I'd start by having a look at the www.ntfsd.org and www.osr.com, especially the formers mailing list archives. The next solution I can come to think of is the file system filter driver DDK (IIRC it's "only" $5000 without any support, but that might have been the full FSDK) from Microsoft. You'd probably have intercept those CreateFile calls and set the "don't cache" flag and then do the caching yourself for those handles. But I'm a little at loss here what you really want to do. You mention you want to cache files even over "internet". Are you using SMB over TCP over the 'net? Tunneling? This is about the only way I can see how CreateFile would accept this kind of file access (since you mention "cache file reads" it seems to be a HANDLE from CreateFile used). Could you please elaborate? Caching 30MB for a multi-GB file isn't that useful upon first glance. Btw, 30MB cache isn't really huge. Using W2k WS (oh, sorry, "Pro") my system often have five times that size for cached (local) files.
-
Since we're talking NT here, there isn't a reasonably priced solution I'm aware of. I'd start by having a look at the www.ntfsd.org and www.osr.com, especially the formers mailing list archives. The next solution I can come to think of is the file system filter driver DDK (IIRC it's "only" $5000 without any support, but that might have been the full FSDK) from Microsoft. You'd probably have intercept those CreateFile calls and set the "don't cache" flag and then do the caching yourself for those handles. But I'm a little at loss here what you really want to do. You mention you want to cache files even over "internet". Are you using SMB over TCP over the 'net? Tunneling? This is about the only way I can see how CreateFile would accept this kind of file access (since you mention "cache file reads" it seems to be a HANDLE from CreateFile used). Could you please elaborate? Caching 30MB for a multi-GB file isn't that useful upon first glance. Btw, 30MB cache isn't really huge. Using W2k WS (oh, sorry, "Pro") my system often have five times that size for cached (local) files.
1. actually it would be sufficient for now to handle local files and those one local network; 2. once again: the files would be readed sequentially, so I think that a 30MB cache per file is quite enough; on exceptions (random reads) doesn't maters the caching performance 3. if you have some time, PLEASE send me an example of how could I do a file caching for an intercepted handle; (this example would be really valuable, as I have no deep knowledge about NT api / Visual C++; I'm actually a Delphi / Database / Assembly programmer; ... and I'm only 19 years old :)) 4. where can I find some links to that Microsoft "file system filter driver DDK"? thanks, Sandor