Speed Search Text File
-
I need to retrieve some information from a text file that basically consists of two columns of data in a tab delimited format. The first column consists of a unique identifier to lookup the main piece of information contained in the second column. This text file will consist of approximately 60,000 lines of data and I need to search through the text file in response to a particular event to retrieve the main piece of information from the text file using the unique identifier to search the file. This search process needs to occur rapidly (t < 1 second). Is there a way to search a text file that rapidly? Alternatively, I could pull this information into an MS Access database (the database format used by my application) each time the user logs in to my application and perform a basic SQL query to retrieve the relevant information. Any advice regarding the best method for searching this information (searching text file or importing text file data into database or some other method) would be appreciated. Thank you.
-
I need to retrieve some information from a text file that basically consists of two columns of data in a tab delimited format. The first column consists of a unique identifier to lookup the main piece of information contained in the second column. This text file will consist of approximately 60,000 lines of data and I need to search through the text file in response to a particular event to retrieve the main piece of information from the text file using the unique identifier to search the file. This search process needs to occur rapidly (t < 1 second). Is there a way to search a text file that rapidly? Alternatively, I could pull this information into an MS Access database (the database format used by my application) each time the user logs in to my application and perform a basic SQL query to retrieve the relevant information. Any advice regarding the best method for searching this information (searching text file or importing text file data into database or some other method) would be appreciated. Thank you.
Well, databases are optimized... :)
If the Lord God Almighty had consulted me before embarking upon the Creation, I would have recommended something simpler. -- Alfonso the Wise, 13th Century King of Castile.
This is going on my arrogant assumptions. You may have a superb reason why I'm completely wrong. -- Iain Clarke
[My articles] -
Well, databases are optimized... :)
If the Lord God Almighty had consulted me before embarking upon the Creation, I would have recommended something simpler. -- Alfonso the Wise, 13th Century King of Castile.
This is going on my arrogant assumptions. You may have a superb reason why I'm completely wrong. -- Iain Clarke
[My articles]The database option would be fine with me. I think that the text file that my application will be supplied with will be updated only once a day and my application is used throughout the day with only 1 or 2 log in events per 8 hour shift, so if I retrieve the information from the database only on log in, I should be OK. Another question I have, though, is how to optimize retrieving the information from the text file. I would need some sort of BULK INSERT utility with MS Access. I know SQL Server has a BULK INSERT command, but unfortunately I do not have the luxury of using SQL Server at present. Is there a way to get the same functionality from Access? Thanks.
-
The database option would be fine with me. I think that the text file that my application will be supplied with will be updated only once a day and my application is used throughout the day with only 1 or 2 log in events per 8 hour shift, so if I retrieve the information from the database only on log in, I should be OK. Another question I have, though, is how to optimize retrieving the information from the text file. I would need some sort of BULK INSERT utility with MS Access. I know SQL Server has a BULK INSERT command, but unfortunately I do not have the luxury of using SQL Server at present. Is there a way to get the same functionality from Access? Thanks.
Hi, there is a Microsoft SQL Server 2005 Express Edition which is free, and I trust a better performer than Access is. :)
Luc Pattyn [Forum Guidelines] [My Articles]
Voting for dummies? No thanks. X|
-
Hi, there is a Microsoft SQL Server 2005 Express Edition which is free, and I trust a better performer than Access is. :)
Luc Pattyn [Forum Guidelines] [My Articles]
Voting for dummies? No thanks. X|
Thanks, Luc. Yes, I understand that, but "my application" is not truly my application. I work for a company and they decide what database gets used. So, for now I'm stuck making this work with MS Access and need a solution using MS Access.
-
I need to retrieve some information from a text file that basically consists of two columns of data in a tab delimited format. The first column consists of a unique identifier to lookup the main piece of information contained in the second column. This text file will consist of approximately 60,000 lines of data and I need to search through the text file in response to a particular event to retrieve the main piece of information from the text file using the unique identifier to search the file. This search process needs to occur rapidly (t < 1 second). Is there a way to search a text file that rapidly? Alternatively, I could pull this information into an MS Access database (the database format used by my application) each time the user logs in to my application and perform a basic SQL query to retrieve the relevant information. Any advice regarding the best method for searching this information (searching text file or importing text file data into database or some other method) would be appreciated. Thank you.
I also think that you should try sql server 2005. Store your text files in the database and build some good search store procedures. With C#,Trigger the SP on an event and display it in a datagridview object. Good Luck :rolleyes:
-
I need to retrieve some information from a text file that basically consists of two columns of data in a tab delimited format. The first column consists of a unique identifier to lookup the main piece of information contained in the second column. This text file will consist of approximately 60,000 lines of data and I need to search through the text file in response to a particular event to retrieve the main piece of information from the text file using the unique identifier to search the file. This search process needs to occur rapidly (t < 1 second). Is there a way to search a text file that rapidly? Alternatively, I could pull this information into an MS Access database (the database format used by my application) each time the user logs in to my application and perform a basic SQL query to retrieve the relevant information. Any advice regarding the best method for searching this information (searching text file or importing text file data into database or some other method) would be appreciated. Thank you.
IMHO using a database is overkill for this application. A simple hash table will do the job easier and faster than a database. 1. Create a hash table with size around 120,000 (prime numbers work best). 2. Read (key, value) pairs from the text file and insert them into the hash table. 3. While you probably can't read and process a text file that size in under a second, you can retrieve an entry from the hash table in less than a hundredth of a second.
-
IMHO using a database is overkill for this application. A simple hash table will do the job easier and faster than a database. 1. Create a hash table with size around 120,000 (prime numbers work best). 2. Read (key, value) pairs from the text file and insert them into the hash table. 3. While you probably can't read and process a text file that size in under a second, you can retrieve an entry from the hash table in less than a hundredth of a second.
Thank you. That was the answer I needed. I have implemented the hash table into the application and it does the trick. The application I'm working with is well established and this particular feature, which happens to be a search for a serial number based on a bar code value, is a specific request by one customer. I can't redo the application simply to fit one customer need, and so I wanted something that works with the existing application architecture, which you have provided. Thanks.
