How to speed up copy of .txt files into arrays?

[David Crow]

Arris7 wrote:

CFile::Seek() doesn't help because actually I do not know at which line to start and finish. I just know a starting and ending date and time. So I need to read the String and to compare it with a variable.

Fair enough, but you can still do it via a simple calculation, rather than using strtok() to find each line. strtok() is slowing you down as it has to examine each character to find the one you want. Since each line of the file is 18-19 characters in length, just compare the first 13 of those. Something like:

void main( void )
{
char *szBuffer = "20000103\t1658\t351\n"
"20000103\t1659\t352\n"
"20000103\t1700\t350\n"
"20000103\t1701\t352\n"
"20000103\t1702\t355\n"
"20000104\t0900\t354\n"
"20000104\t0901\t352\n"
"20000104\t0902\t350\n";
char *p = szBuffer;

while (p != NULL && \*p != '\\0')
{
    if (strncmp(p, "20000104\\t0900", 13) == 0)
    {
        printf("Found it!\\n");
        break;;
    }

    // advance to the next 'line'
    p += 18;
}   

}

---

"Approved Workmen Are Not Ashamed" - 2 Timothy 2:15

"Judge not by the eye but by the heart." - Native American Proverb

[Arris74]

Sorry I was not clear in my first post. Actually I just know the date and the time where to start. for example the starting date and time can be located at the line 250000 and I have to find the line.

[Arris74]

many thanks It sounds great. I gonna try it and let you know the results.

[PJ Arends]

For very large files (millions of lines) this will still take a while. Seeing how the file is sorted by date and time I would use your original idea of using CFile::Seek to do a binary search of the file. A binary search will be slower if the required data is right at the start of the file, but a heck of a lot faster if the data is anywhere else.

---

You may be right
I may be crazy
-- Billy Joel --

Within you lies the power for good, use it!!!

[David Crow]

PJ Arends wrote:

Seeing how the file is sorted by date and time...

Was that a guarantee? If so, then a binary search via Seek() is the way to go.

---

"Approved Workmen Are Not Ashamed" - 2 Timothy 2:15

"Judge not by the eye but by the heart." - Native American Proverb

[led mike]

If it is not sorted a sorted solution would likely be optimal for 4 million records. Either sorting the orginal file or creating and index file or memory based index, perhaps a Database should be considered.

led mike

[Arris74]

Many thanks for your solutions. I think that your idea of using strncmp(p, "20000104\t0900", 13) == 0) in a binary search is a good answer. I'll try it right now. thnks again

[Arris74]

Thnks for this solution. yes the file is sorted by date and time so Binary search is a good answer to my question. thnks again

[malaugh]

Are the dates in order? If the file starts with the earliest date and finishes with the latest date, then use the following method: 1) Fseek to the middle of the file 2) If the data is larger fseek to one quter of the way through the file, if smaller, fseek to 3 quarters of the way though the file. 3) Repeat. Like guessing a number, if you ask someone select a number between 0 and 15, the quickest way to find the number is to ask, Is it less than 8 If yes then ask is it less than 4 If no than ask is it less than 6 I'm sure you get the idea. Its a common technique.

[Arris74]

thanks, Yes dates are in order. the method you suggest is a binary search. I think it is the best solution for my problem.