Application that handles 200 000+ Lines?

Steve The Plant

Hi, I need to build an application that loads a 200, 000 line (or more) text file and lets the user interactively search through it in memory, generating sub-lists of those 200, 000 lines with various search criteria. I'm just wondering if it's wishful thinking to hope that a standard list box can handle that many lines, and be fast enough to traverse. If not, I wonder what other solutions or suggestions might there be. Has anyone seen any applications that sound similar in functionality to this? All the string searching going on makes me think to use Perl, but I don't know Perl at all. Also, I'm probably wrong, but it's a command line only tool, isn't it? Thanks for your time, Shawn

J Dunlap

You're best off using a virtual list box.

"Blessed are the peacemakers, for they shall be called sons of God." - Jesus
"You must be the change you wish to see in the world." - Mahatma Gandhi

Steve The Plant

Could you explain the technique a little bit? Shawn

Steve The Plant

Here's some more information about what I want to do: Typically, the data (per line in the file) is going to look like: ... so, in the file, it'll be very much something like 00:00:00 FireCommandOne 1 2 3 00:00:15 FireCommandTwo 0 0 00:00:30 FireCommandOne 1 2 3 ... and so on. I just have hundreds of thousands of them :) The app needs to be able to mainly search by the command name, and then have it possible to refine the search by specifying things like "oh, I want only the FireEventOne commands whose parameter one is 2 and over"

David Crow

Read these two MSDN articles: Q97371 Q197690

basementman

Sounds like an excellent candidate for using a DATABASE. Like SQL Server. That's what these programs do for a living.... If you really need to do this yourself, you should probably create a series of memory structures to house the data and create indexes on the various fields. The MFC/OO approach would be to create objects similar to the following:

class CommandObj
{
public:
CString m_cCommandTime;

CString m_cCommandName;

WORD m_wParam1;
WORD m_wParam2;
WORD m_wParam3;
};

class CommandDB
{
BOOL CheckMatch(CommandObj *pCmdObj, LPCSTR cpCommand, WORD *wpParam1, WORD *wpParam2, WORD *wpParam3);

public:

CObList m_oCommands; // list of Command objects

CMapStringToOb m_oCmdNameIndex; // index of cmd name to a CObList of Command objects that have this cmd name

CMapWordToOb m_oParam1Index; // index of param1 values to a CObList of Command objects that have this value

virtual ~CommandDB(); // clean up our lists

void AddCommandObj(CommandObj *pCmdObj);

BOOL SearchCommands(CObList *pOutputResults, LPCSTR cpCommand, WORD *wpParam1, WORD *wpParam2, WORD *wpParam3);
};

void CommandDB::AddCommandObj(CommandObj *pCmdObj)
{
CObList *pIdxList = NULL;

// update our indexes...
if (!m_oCmdNameIndex.Lookup(pCmdObj->m_cCommandName,pIdxList))
{
pIdxList = new CObList;
m_oCmdNameIndex.SetAt(pCmdObj->m_cCommandName,pIdxList);
}

pIdxList->AddTail(pCmdObj);

if (!m_oParam1Index.Lookup(pCmdObj->m_wParam1,pIdxList))
{
pIdxList = new CObList;
m_oParam1Index.SetAt(pCmdObj->m_wParam1,pIdxList);
}

pIdxList->AddTail(pCmdObj);

// and add to "table"
m_oCommands.AddTail(pCmdObj);
}

BOOL CommandDB::SearchCommands(CObList *pOutputResults, LPCSTR cpCommand, WORD *wpParam1, WORD *wpParam2, WORD *wpParam3)
{
BOOL bFound = TRUE;
CommandObj *pCmdObj;
CObList *pKeyList = NULL;

if (cpCommand != NULL && *cpCommand)
m_oCmdNameIndex.Lookup(cpCommand,pKeyList);
else if (wpParam1 != NULL)
m_oParam1Index.Lookup(*wpParam1,pKeyList);
else
pKeyList = &m_oCommands; // table scan time...

// scan keyset for matches
POSITION pPos = pKeyList->GetHeadPosition();
while (pPos)
{
pCmdObj = (CommandObj *)pKeyList->GetNext(pPos);
if (CheckMatch(pCmdObj,cpCommand,wpParam1,wpParam2,wpParam3))
{
bFound = TRUE;
pOutputResults->AddTail(pCmdObj);
}
}

return bFound;
}

BOOL CommandDB::Check