How to search specific record faster
-
So first you loop through all the strings without looking for the string, then you do it again just to find the string? Doesn't that sound like a waste of time? Skip the DataSet and the BindingSource, just read the file and look for the string.
Despite everything, the person most likely to be fooling you next is yourself.
Well, thanks for the reply. But what I have to do is find a 20 char max string in a 20 char max field and then replace another field's data if matched and you are suggesting me to search 20 char max string in a 2000 char or more per record. Does that sound waste of time or mine? PS please first look at my original question, I think I was clear enough to give you the idea that why I need to search 20K records for one string although I have to search 20K strings within those 20K records continuously. BTW I hope you got the idea on which I have to work so hopefully you can help me to solve the problem. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP
-
Well, thanks for the reply. But what I have to do is find a 20 char max string in a 20 char max field and then replace another field's data if matched and you are suggesting me to search 20 char max string in a 2000 char or more per record. Does that sound waste of time or mine? PS please first look at my original question, I think I was clear enough to give you the idea that why I need to search 20K records for one string although I have to search 20K strings within those 20K records continuously. BTW I hope you got the idea on which I have to work so hopefully you can help me to solve the problem. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP
I got the impression that you only did search once, that's why I said that you should search the data directly. If you want to search the data several times, you should use a dictionary. Finding the key in a dictionary is an operation that is close to O(1), i.e. it takes about the same time regardless if you have 100 items or 100000 items. If you need to keep the data in the DataTable, you can set up a dictionary to use as index, i.e. use the column value as key, and the row reference or row index as value. If you need to search different columns, you would need one dictionary per column. Example creating an index dictionary for a table column:
Dim idx as New Dictionary(Of string, DataRow)(dataTable.Rows.Count);
For Each row As DataRow in dataTable
idx.Add(row("SomeColumn").ToString(), row)
NextOnce the index is set up, you can find a row instantly:
Dim r As DataRow = idx.Item("Joshua")
Despite everything, the person most likely to be fooling you next is yourself.
-
I got the impression that you only did search once, that's why I said that you should search the data directly. If you want to search the data several times, you should use a dictionary. Finding the key in a dictionary is an operation that is close to O(1), i.e. it takes about the same time regardless if you have 100 items or 100000 items. If you need to keep the data in the DataTable, you can set up a dictionary to use as index, i.e. use the column value as key, and the row reference or row index as value. If you need to search different columns, you would need one dictionary per column. Example creating an index dictionary for a table column:
Dim idx as New Dictionary(Of string, DataRow)(dataTable.Rows.Count);
For Each row As DataRow in dataTable
idx.Add(row("SomeColumn").ToString(), row)
NextOnce the index is set up, you can find a row instantly:
Dim r As DataRow = idx.Item("Joshua")
Despite everything, the person most likely to be fooling you next is yourself.
-
I got the impression that you only did search once, that's why I said that you should search the data directly. If you want to search the data several times, you should use a dictionary. Finding the key in a dictionary is an operation that is close to O(1), i.e. it takes about the same time regardless if you have 100 items or 100000 items. If you need to keep the data in the DataTable, you can set up a dictionary to use as index, i.e. use the column value as key, and the row reference or row index as value. If you need to search different columns, you would need one dictionary per column. Example creating an index dictionary for a table column:
Dim idx as New Dictionary(Of string, DataRow)(dataTable.Rows.Count);
For Each row As DataRow in dataTable
idx.Add(row("SomeColumn").ToString(), row)
NextOnce the index is set up, you can find a row instantly:
Dim r As DataRow = idx.Item("Joshua")
Despite everything, the person most likely to be fooling you next is yourself.
-
Thanks for the reply. I will try using dataview though I have no idea about it. BTW all process is in Background routine which doesn't have any UI, I have to search 20K records individually within another 20K records and apply some functions where data found. So, it needs to be much faster than currently is. Hope someone might have more dynamic idea. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP
If your data is stored in a database then that is where you need to do the processing. You should not be manipulating 20k records outside a database. If you are not using a database you have a design issue!
Never underestimate the power of human stupidity RAH
-
If your data is stored in a database then that is where you need to do the processing. You should not be manipulating 20k records outside a database. If you are not using a database you have a design issue!
Never underestimate the power of human stupidity RAH
Well dear, it is not a design issue but it is the customer demand, I am supposed to process various files which are generated by several other programs (of course I didn't develop them) so in processing those files, I have to keep the format intact without any changes. Personally I also prefer database or XML files, it is so easy to manage XML files if data is not that huge. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP
-
Well dear, it is not a design issue but it is the customer demand, I am supposed to process various files which are generated by several other programs (of course I didn't develop them) so in processing those files, I have to keep the format intact without any changes. Personally I also prefer database or XML files, it is so easy to manage XML files if data is not that huge. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP
Then I would look at some other options. Just because you have to retain the format does not mean you can't deal with it smarter in your app, you should be able to return the end result to the format required, does not limit your choices of processing. It would of course depend on your requirements but I would of thought using a bindingsource as your processing vehicle to be one of the slowest options! Can't you read the file into a decent container (table or xml X|) do your processing and then write it out. The customer is NOT always right, and if they insist then they must be aware of the cost, then they can make the decision to ware the (time) cost or allow you to suggest a better option.
Never underestimate the power of human stupidity RAH
-
Then I would look at some other options. Just because you have to retain the format does not mean you can't deal with it smarter in your app, you should be able to return the end result to the format required, does not limit your choices of processing. It would of course depend on your requirements but I would of thought using a bindingsource as your processing vehicle to be one of the slowest options! Can't you read the file into a decent container (table or xml X|) do your processing and then write it out. The customer is NOT always right, and if they insist then they must be aware of the cost, then they can make the decision to ware the (time) cost or allow you to suggest a better option.
Never underestimate the power of human stupidity RAH
Well, I would suggest you to plz review my first post where I have mentioned already that I m using BindingSource through populating the text data into table/dataset. As per my thinking I found the binding source object more useful than working through the usual datasets. But to populate xml file from text file and then process the records and then again transfer all the records to the format seems ugly to me. Well, no personal offense plz, but I personally like the BindingSource component, in some ways it makes the life easier however everyone has there own viewpoint. May be later on I think the same about BindingSource as you think or it might be vice-versa. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP
-
Well, I would suggest you to plz review my first post where I have mentioned already that I m using BindingSource through populating the text data into table/dataset. As per my thinking I found the binding source object more useful than working through the usual datasets. But to populate xml file from text file and then process the records and then again transfer all the records to the format seems ugly to me. Well, no personal offense plz, but I personally like the BindingSource component, in some ways it makes the life easier however everyone has there own viewpoint. May be later on I think the same about BindingSource as you think or it might be vice-versa. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP
No no no, you missed the point, your bindingsource is taking 1 hr+ to complete the process when there are a large number of records. My original suggestion was to use a dataview. I have never used the bindingsource so have no opinion of it as a tool. The only issue I have is that it is another layer of abstraction over your data and you have a speed issue.
MAP Tiger wrote:
seems ugly to me
Got to agree with you there, I would have thought a datatable/view would be faster than XML. Have you use a Dataview with rowfilter and what is the performance like. As for personal, how can a technical discussion be personal (except for preferences), the goal is to get the best solution, I don't give a rats who suggests it and I may not be right but I am happy to supply them (hopefully useful).
Never underestimate the power of human stupidity RAH
-
No no no, you missed the point, your bindingsource is taking 1 hr+ to complete the process when there are a large number of records. My original suggestion was to use a dataview. I have never used the bindingsource so have no opinion of it as a tool. The only issue I have is that it is another layer of abstraction over your data and you have a speed issue.
MAP Tiger wrote:
seems ugly to me
Got to agree with you there, I would have thought a datatable/view would be faster than XML. Have you use a Dataview with rowfilter and what is the performance like. As for personal, how can a technical discussion be personal (except for preferences), the goal is to get the best solution, I don't give a rats who suggests it and I may not be right but I am happy to supply them (hopefully useful).
Never underestimate the power of human stupidity RAH
Well, actually in the current scenario (in which I am currently working) and my formal experience with BindingSource component were matching but of course before the current project I have never come across with this heavy searching so I never knew the slowness of BindingSource.Find method. Although now I am using dictionary object for it and believe me I am so happy, the same work is now processed in just 6 to 8 mins instead 1.30 hrs. So I am much happy now :) As far as DataView component is concerned, I never came across it. I would be checking it though to let it be considered in future projects. Regards,
MAP Tiger Tiger Softwares Software Designer and Developer VB.NET, ASP.NET, VFP