I/O skipping the buffer???
-
Hi guys I've been consulting with this community regarding the handling of large data files and searching stuff in them. The search process using a multimap works wonderfully. HOWEVER, the actual reading from the disk is painfully long. At some point, somebody said something about not using cout because it abuses the buffer... or something like that. I know you don't use cout to read from a file... but since a stream object from the class ifstream created to read a file is equivalent to cin... and streams work through the buffer, it still made sense to me that using ifstream may not be the fastest way to read data from disk. SO... I wonder, is there a faster way to read data from a file that is not using the ifstream? I know that I could use the netCDF system... but I don't have time right now to learn how to use it. So I'm looking for something more direct and simple that is already embedded within C++. Please, in your response, can you include a simple code to exemplify the use of the alternative? Thanks a million... you guys rock!!! :suss:
-
Hi guys I've been consulting with this community regarding the handling of large data files and searching stuff in them. The search process using a multimap works wonderfully. HOWEVER, the actual reading from the disk is painfully long. At some point, somebody said something about not using cout because it abuses the buffer... or something like that. I know you don't use cout to read from a file... but since a stream object from the class ifstream created to read a file is equivalent to cin... and streams work through the buffer, it still made sense to me that using ifstream may not be the fastest way to read data from disk. SO... I wonder, is there a faster way to read data from a file that is not using the ifstream? I know that I could use the netCDF system... but I don't have time right now to learn how to use it. So I'm looking for something more direct and simple that is already embedded within C++. Please, in your response, can you include a simple code to exemplify the use of the alternative? Thanks a million... you guys rock!!! :suss:
ifstream may well be slow if you use it to read information in a typed manner. If you just want to read a big blob of info into memory, then you may want to look into memory mapped files ( basically you become able to reference a file as if it was a chunk of memory, the Richter book on Windows C++ development covers this, but I forget the details ). Or you may want to use a C style file object to read the file into memory as one big chunk. I recommend trying all possible approaches, and timing them to see if performance is improved, if not, I'd stick with iostream. Christian Graus - Microsoft MVP - C++
-
ifstream may well be slow if you use it to read information in a typed manner. If you just want to read a big blob of info into memory, then you may want to look into memory mapped files ( basically you become able to reference a file as if it was a chunk of memory, the Richter book on Windows C++ development covers this, but I forget the details ). Or you may want to use a C style file object to read the file into memory as one big chunk. I recommend trying all possible approaches, and timing them to see if performance is improved, if not, I'd stick with iostream. Christian Graus - Microsoft MVP - C++
-
Hi Sorry, I learned C++ by myself and never went through C. What do you mean "C style file object to read the file into memory as one big chunk."? :confused: Thanks a million!
In C ( which C++ is built on ), you use a function called ( I think ) OpenFile, and get a FILE object back. That's what I meant. Christian Graus - Microsoft MVP - C++