_findfirst and fopen very slow
-
I have about 1800 text files in a dir and their number is slowly growing (several files in a day). The size of a file is about 40 kB. I need to enumerate all the file with the ".states" extension, open the file, check to see if there is a line that starts with a particular sequence of 4 chars and if the line exists, I save the line in a std::vector. I use the following code, but it takes very long time at the first run (the subsequent runs are very fast):
struct \_finddata\_t fd; long hFile; if((hFile=\_findfirst("\*.states", &fd))== -1L) return; // File not found do { FILE \*f= fopen(fd.name, "rS"); while(fgets(buf, sizeof(buf), f)) { if(0 == \_strnicmp(buf, "ABCD", 4)) { Save buf in a std::vector break; } } fclose(f); } while(\_findnext(hFile, &fd) == 0); \_findclose(hFile);Is there any way to speedup the code? If I merge all the files in a single file, I solve the problem, but I prefer to keep all the original files.
The main culprit is "fgets". Once you call that, the fopen series of calls immediately loads, I believe, 32k of data. On top of that fgets is relatively slow. For speed, you may be better off using fread, reading in 4k (or the page size) at a time and parsing the block yourself (by simply looking for ABCD. This could be sped up faster by doing a Boyer-Moore search, though since the string is short, simply scanning first for A and then checking for the rest may be faster. That said, I believe some new implementations of the standard library now include a Boyer-Moore algorithm.) Do also note that caching plays a big part here. Just recursing folders will take significantly longer the first pass than the second. This can be deceptive, however, since in actual operation those caches may be flushed between program runs.
-
That's a good approach, but as I understand it the main task is not to find files that contain the string, but find all lines within these files containing it. A specific kind of filename wouldn't be enough. Your suggestion to include the writing into the problem solution is a good idea. However, if we do that, we might as well write all the data to a database. Retrieving the correct lines would then only require a simple SQL query.
GOTOs are a bit like wire coat hangers: they tend to breed in the darkness, such that where there once were few, eventually there are many, and the program's architecture collapses beneath them. (Fran Poretto)
Stefan_Lang wrote:
as I understand it the main task is not to find files that contain the string, but find all lines within these files containing it.
Each file usually contains 190 to 220 lines and a file may or may not contain the wanted line (but almost all the files contain the wanted line). If the wanted line is in the file, there is only one line.
-
Maybe read the whole file at once as opposed to many fgets() calls, then do your string search in RAM?
"the debugger doesn't tell me anything because this code compiles just fine" - random QA comment "Facebook is where you tell lies to your friends. Twitter is where you tell the truth to strangers." - chriselst "I don't drink any more... then again, I don't drink any less." - Mike Mullikins uncle
-
The more obvious answer is get whatever is saving the files to put the ones which contain your string ABCD out under a special name string. Then you don't have to search inside the file at all to find the files you want. Another obvious choice is have the files on a ramdisk as there isn't much data. The whole process seems a bit backward to me you are working on the reading code not the writing code.
In vino veritas
leon de boer wrote:
The more obvious answer is get whatever is saving the files to put the ones which contain your string ABCD out under a special name string. Then you don't have to search inside the file at all to find the files you want.
What happens if I need to find "BCDE" or "S1 " or "01FA" or ...?
Another obvious choice is have the files on a ramdisk as there isn't much data.
If I put the folder on the SSD, the process is much faster: 60.9 s for the HD and 2.9 s for the SSD. The SSD is an "unusual" location for that folder because all the other files are on the HD, but it's the easiest solution. Thank you
-
leon de boer wrote:
The more obvious answer is get whatever is saving the files to put the ones which contain your string ABCD out under a special name string. Then you don't have to search inside the file at all to find the files you want.
What happens if I need to find "BCDE" or "S1 " or "01FA" or ...?
Another obvious choice is have the files on a ramdisk as there isn't much data.
If I put the folder on the SSD, the process is much faster: 60.9 s for the HD and 2.9 s for the SSD. The SSD is an "unusual" location for that folder because all the other files are on the HD, but it's the easiest solution. Thank you
Quote:
What happens if I need to find "BCDE" or "S1 " or "01FA" or ...?
Label them differently with a special name obviously, all you are doing is coming up with a filenaming convention :-) Hell use the file extension you already have (*.states) and mask the bits of it for what special strings are in it *.states = file with no special tags *.states1 = file with special tag 1 in it *.states2 = file with special tag 2 in it *.states3 = file with special tag 1 & 2 in it *.states4 = file with special tag 3 in it *.states5 = file with special tag 1 & 3 in it *.states6 = file with special tag 1 & 2 in it *.states7 = file with special tag 1, 2 & 3 in it You can know what tags are in the file without ever opening it all you need to know is the filename. This is also obviously a windows program why aren't you using the Windows API for the file open and reading?
HANDLE Handle = CreateFile (fd.name, GENERIC_READ, FILE_SHARE_READ,
0, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0); // Open the file
if (Handle != INVALID_HANDLE_VALUE)
{
DWORD Actual;
ReadFile(Handle, &buf[0], sizeof(buf)-1, &Actual, 0); // Read a buffer-1 of data (1 byte for #0 at end)
if (Actual > 0)
{
buf[Actual] = 0; // Make sure asciiz terminated for next string op
if(0 == _strnicmp(buf, "ABCD", 4)) {
Save buf in a std::vector
}
}
CloseHandle(Handle);
}In vino veritas
-
Quote:
What happens if I need to find "BCDE" or "S1 " or "01FA" or ...?
Label them differently with a special name obviously, all you are doing is coming up with a filenaming convention :-) Hell use the file extension you already have (*.states) and mask the bits of it for what special strings are in it *.states = file with no special tags *.states1 = file with special tag 1 in it *.states2 = file with special tag 2 in it *.states3 = file with special tag 1 & 2 in it *.states4 = file with special tag 3 in it *.states5 = file with special tag 1 & 3 in it *.states6 = file with special tag 1 & 2 in it *.states7 = file with special tag 1, 2 & 3 in it You can know what tags are in the file without ever opening it all you need to know is the filename. This is also obviously a windows program why aren't you using the Windows API for the file open and reading?
HANDLE Handle = CreateFile (fd.name, GENERIC_READ, FILE_SHARE_READ,
0, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0); // Open the file
if (Handle != INVALID_HANDLE_VALUE)
{
DWORD Actual;
ReadFile(Handle, &buf[0], sizeof(buf)-1, &Actual, 0); // Read a buffer-1 of data (1 byte for #0 at end)
if (Actual > 0)
{
buf[Actual] = 0; // Make sure asciiz terminated for next string op
if(0 == _strnicmp(buf, "ABCD", 4)) {
Save buf in a std::vector
}
}
CloseHandle(Handle);
}In vino veritas
Since almost all the files contain the wanted string, I'll need to open almost all the files, so the speed up would be negligible. I don't use Win Api because, afaik, there is no fgets() equivalent and there is no speed up if I read the whole file at once.
-
Since almost all the files contain the wanted string, I'll need to open almost all the files, so the speed up would be negligible. I don't use Win Api because, afaik, there is no fgets() equivalent and there is no speed up if I read the whole file at once.
I gave you the fgets equivalent above (its only a couple of lines of code) .. I am not convinced it isn't faster because you will be using the standard console file handler for opening and reading thru the standard library. Anyhow I will leave you to it
In vino veritas
-
I have about 1800 text files in a dir and their number is slowly growing (several files in a day). The size of a file is about 40 kB. I need to enumerate all the file with the ".states" extension, open the file, check to see if there is a line that starts with a particular sequence of 4 chars and if the line exists, I save the line in a std::vector. I use the following code, but it takes very long time at the first run (the subsequent runs are very fast):
struct \_finddata\_t fd; long hFile; if((hFile=\_findfirst("\*.states", &fd))== -1L) return; // File not found do { FILE \*f= fopen(fd.name, "rS"); while(fgets(buf, sizeof(buf), f)) { if(0 == \_strnicmp(buf, "ABCD", 4)) { Save buf in a std::vector break; } } fclose(f); } while(\_findnext(hFile, &fd) == 0); \_findclose(hFile);Is there any way to speedup the code? If I merge all the files in a single file, I solve the problem, but I prefer to keep all the original files.