How to search specific record faster
-
Hi I have a text file which has thousands of records, I am opening it and populating it in dataset and then binding to a binding source. Then I am supposed to find specific item in specific column. In my scenario, I have to search thousands of strings within thousands of records in one routine. Currently I am using:
BindingSource1.Find("ColName","String")
Basically it is not taking much time but when it comes to thousands of records then it is taking approx 1 hour for one complete routine. (Some inside processing is also there but if records are lesser, I mean around 4,000 then it doesn't take much time but when it comes to 20,000 records then it just take me to the HELL and the PC is also at its highest, nothing else can be done however I am using multi-threading.) What I basically want to know is that is there any way to search specific strings in a more powerful way to allow me better processing. Any help will be highly appreciated. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP
-
Hi I have a text file which has thousands of records, I am opening it and populating it in dataset and then binding to a binding source. Then I am supposed to find specific item in specific column. In my scenario, I have to search thousands of strings within thousands of records in one routine. Currently I am using:
BindingSource1.Find("ColName","String")
Basically it is not taking much time but when it comes to thousands of records then it is taking approx 1 hour for one complete routine. (Some inside processing is also there but if records are lesser, I mean around 4,000 then it doesn't take much time but when it comes to 20,000 records then it just take me to the HELL and the PC is also at its highest, nothing else can be done however I am using multi-threading.) What I basically want to know is that is there any way to search specific strings in a more powerful way to allow me better processing. Any help will be highly appreciated. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP
So first you loop through all the strings without looking for the string, then you do it again just to find the string? Doesn't that sound like a waste of time? Skip the DataSet and the BindingSource, just read the file and look for the string.
Despite everything, the person most likely to be fooling you next is yourself.
-
Hi I have a text file which has thousands of records, I am opening it and populating it in dataset and then binding to a binding source. Then I am supposed to find specific item in specific column. In my scenario, I have to search thousands of strings within thousands of records in one routine. Currently I am using:
BindingSource1.Find("ColName","String")
Basically it is not taking much time but when it comes to thousands of records then it is taking approx 1 hour for one complete routine. (Some inside processing is also there but if records are lesser, I mean around 4,000 then it doesn't take much time but when it comes to 20,000 records then it just take me to the HELL and the PC is also at its highest, nothing else can be done however I am using multi-threading.) What I basically want to know is that is there any way to search specific strings in a more powerful way to allow me better processing. Any help will be highly appreciated. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP
If your bindingsource is a datatable then use a dataview and dynamically filter the view. I find this works OK but I would NEVER try and display 20k records. Why are you trying to locate 1 record in 20k in the UI. Isn't that what databases are for?
Never underestimate the power of human stupidity RAH
-
If your bindingsource is a datatable then use a dataview and dynamically filter the view. I find this works OK but I would NEVER try and display 20k records. Why are you trying to locate 1 record in 20k in the UI. Isn't that what databases are for?
Never underestimate the power of human stupidity RAH
Thanks for the reply. I will try using dataview though I have no idea about it. BTW all process is in Background routine which doesn't have any UI, I have to search 20K records individually within another 20K records and apply some functions where data found. So, it needs to be much faster than currently is. Hope someone might have more dynamic idea. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP
-
So first you loop through all the strings without looking for the string, then you do it again just to find the string? Doesn't that sound like a waste of time? Skip the DataSet and the BindingSource, just read the file and look for the string.
Despite everything, the person most likely to be fooling you next is yourself.
Well, thanks for the reply. But what I have to do is find a 20 char max string in a 20 char max field and then replace another field's data if matched and you are suggesting me to search 20 char max string in a 2000 char or more per record. Does that sound waste of time or mine? PS please first look at my original question, I think I was clear enough to give you the idea that why I need to search 20K records for one string although I have to search 20K strings within those 20K records continuously. BTW I hope you got the idea on which I have to work so hopefully you can help me to solve the problem. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP
-
Well, thanks for the reply. But what I have to do is find a 20 char max string in a 20 char max field and then replace another field's data if matched and you are suggesting me to search 20 char max string in a 2000 char or more per record. Does that sound waste of time or mine? PS please first look at my original question, I think I was clear enough to give you the idea that why I need to search 20K records for one string although I have to search 20K strings within those 20K records continuously. BTW I hope you got the idea on which I have to work so hopefully you can help me to solve the problem. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP
I got the impression that you only did search once, that's why I said that you should search the data directly. If you want to search the data several times, you should use a dictionary. Finding the key in a dictionary is an operation that is close to O(1), i.e. it takes about the same time regardless if you have 100 items or 100000 items. If you need to keep the data in the DataTable, you can set up a dictionary to use as index, i.e. use the column value as key, and the row reference or row index as value. If you need to search different columns, you would need one dictionary per column. Example creating an index dictionary for a table column:
Dim idx as New Dictionary(Of string, DataRow)(dataTable.Rows.Count);
For Each row As DataRow in dataTable
idx.Add(row("SomeColumn").ToString(), row)
NextOnce the index is set up, you can find a row instantly:
Dim r As DataRow = idx.Item("Joshua")
Despite everything, the person most likely to be fooling you next is yourself.
-
I got the impression that you only did search once, that's why I said that you should search the data directly. If you want to search the data several times, you should use a dictionary. Finding the key in a dictionary is an operation that is close to O(1), i.e. it takes about the same time regardless if you have 100 items or 100000 items. If you need to keep the data in the DataTable, you can set up a dictionary to use as index, i.e. use the column value as key, and the row reference or row index as value. If you need to search different columns, you would need one dictionary per column. Example creating an index dictionary for a table column:
Dim idx as New Dictionary(Of string, DataRow)(dataTable.Rows.Count);
For Each row As DataRow in dataTable
idx.Add(row("SomeColumn").ToString(), row)
NextOnce the index is set up, you can find a row instantly:
Dim r As DataRow = idx.Item("Joshua")
Despite everything, the person most likely to be fooling you next is yourself.
-
I got the impression that you only did search once, that's why I said that you should search the data directly. If you want to search the data several times, you should use a dictionary. Finding the key in a dictionary is an operation that is close to O(1), i.e. it takes about the same time regardless if you have 100 items or 100000 items. If you need to keep the data in the DataTable, you can set up a dictionary to use as index, i.e. use the column value as key, and the row reference or row index as value. If you need to search different columns, you would need one dictionary per column. Example creating an index dictionary for a table column:
Dim idx as New Dictionary(Of string, DataRow)(dataTable.Rows.Count);
For Each row As DataRow in dataTable
idx.Add(row("SomeColumn").ToString(), row)
NextOnce the index is set up, you can find a row instantly:
Dim r As DataRow = idx.Item("Joshua")
Despite everything, the person most likely to be fooling you next is yourself.
-
Thanks for the reply. I will try using dataview though I have no idea about it. BTW all process is in Background routine which doesn't have any UI, I have to search 20K records individually within another 20K records and apply some functions where data found. So, it needs to be much faster than currently is. Hope someone might have more dynamic idea. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP
If your data is stored in a database then that is where you need to do the processing. You should not be manipulating 20k records outside a database. If you are not using a database you have a design issue!
Never underestimate the power of human stupidity RAH
-
If your data is stored in a database then that is where you need to do the processing. You should not be manipulating 20k records outside a database. If you are not using a database you have a design issue!
Never underestimate the power of human stupidity RAH
Well dear, it is not a design issue but it is the customer demand, I am supposed to process various files which are generated by several other programs (of course I didn't develop them) so in processing those files, I have to keep the format intact without any changes. Personally I also prefer database or XML files, it is so easy to manage XML files if data is not that huge. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP
-
Well dear, it is not a design issue but it is the customer demand, I am supposed to process various files which are generated by several other programs (of course I didn't develop them) so in processing those files, I have to keep the format intact without any changes. Personally I also prefer database or XML files, it is so easy to manage XML files if data is not that huge. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP
Then I would look at some other options. Just because you have to retain the format does not mean you can't deal with it smarter in your app, you should be able to return the end result to the format required, does not limit your choices of processing. It would of course depend on your requirements but I would of thought using a bindingsource as your processing vehicle to be one of the slowest options! Can't you read the file into a decent container (table or xml X|) do your processing and then write it out. The customer is NOT always right, and if they insist then they must be aware of the cost, then they can make the decision to ware the (time) cost or allow you to suggest a better option.
Never underestimate the power of human stupidity RAH
-
Then I would look at some other options. Just because you have to retain the format does not mean you can't deal with it smarter in your app, you should be able to return the end result to the format required, does not limit your choices of processing. It would of course depend on your requirements but I would of thought using a bindingsource as your processing vehicle to be one of the slowest options! Can't you read the file into a decent container (table or xml X|) do your processing and then write it out. The customer is NOT always right, and if they insist then they must be aware of the cost, then they can make the decision to ware the (time) cost or allow you to suggest a better option.
Never underestimate the power of human stupidity RAH
Well, I would suggest you to plz review my first post where I have mentioned already that I m using BindingSource through populating the text data into table/dataset. As per my thinking I found the binding source object more useful than working through the usual datasets. But to populate xml file from text file and then process the records and then again transfer all the records to the format seems ugly to me. Well, no personal offense plz, but I personally like the BindingSource component, in some ways it makes the life easier however everyone has there own viewpoint. May be later on I think the same about BindingSource as you think or it might be vice-versa. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP
-
Well, I would suggest you to plz review my first post where I have mentioned already that I m using BindingSource through populating the text data into table/dataset. As per my thinking I found the binding source object more useful than working through the usual datasets. But to populate xml file from text file and then process the records and then again transfer all the records to the format seems ugly to me. Well, no personal offense plz, but I personally like the BindingSource component, in some ways it makes the life easier however everyone has there own viewpoint. May be later on I think the same about BindingSource as you think or it might be vice-versa. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP
No no no, you missed the point, your bindingsource is taking 1 hr+ to complete the process when there are a large number of records. My original suggestion was to use a dataview. I have never used the bindingsource so have no opinion of it as a tool. The only issue I have is that it is another layer of abstraction over your data and you have a speed issue.
MAP Tiger wrote:
seems ugly to me
Got to agree with you there, I would have thought a datatable/view would be faster than XML. Have you use a Dataview with rowfilter and what is the performance like. As for personal, how can a technical discussion be personal (except for preferences), the goal is to get the best solution, I don't give a rats who suggests it and I may not be right but I am happy to supply them (hopefully useful).
Never underestimate the power of human stupidity RAH
-
No no no, you missed the point, your bindingsource is taking 1 hr+ to complete the process when there are a large number of records. My original suggestion was to use a dataview. I have never used the bindingsource so have no opinion of it as a tool. The only issue I have is that it is another layer of abstraction over your data and you have a speed issue.
MAP Tiger wrote:
seems ugly to me
Got to agree with you there, I would have thought a datatable/view would be faster than XML. Have you use a Dataview with rowfilter and what is the performance like. As for personal, how can a technical discussion be personal (except for preferences), the goal is to get the best solution, I don't give a rats who suggests it and I may not be right but I am happy to supply them (hopefully useful).
Never underestimate the power of human stupidity RAH
Well, actually in the current scenario (in which I am currently working) and my formal experience with BindingSource component were matching but of course before the current project I have never come across with this heavy searching so I never knew the slowness of BindingSource.Find method. Although now I am using dictionary object for it and believe me I am so happy, the same work is now processed in just 6 to 8 mins instead 1.30 hrs. So I am much happy now :) As far as DataView component is concerned, I never came across it. I would be checking it though to let it be considered in future projects. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP