Getting speed in file reading
-
I'm looking to maximize my speed while reading a binary file - the file contains a mix of floats & ints(4bytes each) and occasional strings. Floats and ints are converted to the correct endianess post read using bit shifts and masks. I've fine tuned everything I can think of for reading the file, but have not previously considered changing the interface to the file. I am currently using FILE * with the associated fread, seek etc. functions for no better reason than this being what my original C code used (this entire app is now ported to VC++ with MFC). Is there a faster file reader I should consider? Any speed increase would help, as I am reading 100's of megs of data. Thanks for any opinions.
-
I'm looking to maximize my speed while reading a binary file - the file contains a mix of floats & ints(4bytes each) and occasional strings. Floats and ints are converted to the correct endianess post read using bit shifts and masks. I've fine tuned everything I can think of for reading the file, but have not previously considered changing the interface to the file. I am currently using FILE * with the associated fread, seek etc. functions for no better reason than this being what my original C code used (this entire app is now ported to VC++ with MFC). Is there a faster file reader I should consider? Any speed increase would help, as I am reading 100's of megs of data. Thanks for any opinions.
try CreateFile() API Please chk http://msdn2.microsoft.com/en-US/library/aa363858.aspx
AJay
-
I'm looking to maximize my speed while reading a binary file - the file contains a mix of floats & ints(4bytes each) and occasional strings. Floats and ints are converted to the correct endianess post read using bit shifts and masks. I've fine tuned everything I can think of for reading the file, but have not previously considered changing the interface to the file. I am currently using FILE * with the associated fread, seek etc. functions for no better reason than this being what my original C code used (this entire app is now ported to VC++ with MFC). Is there a faster file reader I should consider? Any speed increase would help, as I am reading 100's of megs of data. Thanks for any opinions.
As Ajaywinds mentioned, you can get rid of a bit of overhead by using the Windows file APIs directly, instead of through the ANSI CRT functions. If you must remain ANSI C compliant this isn't an option. I would think anything you can do in RAM will help much more. Buffer as much as you can into RAM before parsing, especially if everything you read will be parsed. Seeking is costly. If you have to jump all over the place in the file to parse it then performance will suffer. Design file layout for serial access if possible. If you can use file I/O APIs, and the format of the files allows it, consider using overlapped I/O. You can be parsing data in RAM while data is being read from disk. Just my 2 cents, Mark
Mark Salsbery Microsoft MVP - Visual C++ :java:
-
I'm looking to maximize my speed while reading a binary file - the file contains a mix of floats & ints(4bytes each) and occasional strings. Floats and ints are converted to the correct endianess post read using bit shifts and masks. I've fine tuned everything I can think of for reading the file, but have not previously considered changing the interface to the file. I am currently using FILE * with the associated fread, seek etc. functions for no better reason than this being what my original C code used (this entire app is now ported to VC++ with MFC). Is there a faster file reader I should consider? Any speed increase would help, as I am reading 100's of megs of data. Thanks for any opinions.
-
I'm looking to maximize my speed while reading a binary file - the file contains a mix of floats & ints(4bytes each) and occasional strings. Floats and ints are converted to the correct endianess post read using bit shifts and masks. I've fine tuned everything I can think of for reading the file, but have not previously considered changing the interface to the file. I am currently using FILE * with the associated fread, seek etc. functions for no better reason than this being what my original C code used (this entire app is now ported to VC++ with MFC). Is there a faster file reader I should consider? Any speed increase would help, as I am reading 100's of megs of data. Thanks for any opinions.
Try mapping the file to memory. Also if you know the exact layout of the file you could define a struct and cast the pointer directly.
-