Filtering bulk data
-
Hi all, I want to know the better way of filtering Bulk data, pls guide me... I have two flat files with lakhs of rows having 16 digit numbers. i want to filter the numbers from first file with second file , resulting in a unique identified numbers. I am doing it like this. I am inserting these data into two seperate tables using bulk insert in Mysql. And then from my web interface iam inputing two flat files, whenever user clicks on filter button iam querying tables with a select statement using NOT IN sql statement, but it is taking lot of time and even resuling in a connection timeout. So pls suggest me the best way of doing it. Thanks in advance.....
-
Hi all, I want to know the better way of filtering Bulk data, pls guide me... I have two flat files with lakhs of rows having 16 digit numbers. i want to filter the numbers from first file with second file , resulting in a unique identified numbers. I am doing it like this. I am inserting these data into two seperate tables using bulk insert in Mysql. And then from my web interface iam inputing two flat files, whenever user clicks on filter button iam querying tables with a select statement using NOT IN sql statement, but it is taking lot of time and even resuling in a connection timeout. So pls suggest me the best way of doing it. Thanks in advance.....
First of all you should this is not appropriate place to post your query.. :doh: And please explain in detail...
Be an Eagle, Sky is Yours.
-
First of all you should this is not appropriate place to post your query.. :doh: And please explain in detail...
Be an Eagle, Sky is Yours.
Hi suresh, This is related to Asp.net web interface only.. In my post i clearly explained the way i am trying to do. Anyways i am doing it for u again. On my web interface i have option to browse two files and then a filter button. whenever user clicks on filer button i need to filter the first file with second file so i am inserting these files into mysql tables and querying with NOT IN select statement. I am getting connection timeout so pls suggest me the best way of filtering my files. Thanks
-
Hi suresh, This is related to Asp.net web interface only.. In my post i clearly explained the way i am trying to do. Anyways i am doing it for u again. On my web interface i have option to browse two files and then a filter button. whenever user clicks on filer button i need to filter the first file with second file so i am inserting these files into mysql tables and querying with NOT IN select statement. I am getting connection timeout so pls suggest me the best way of filtering my files. Thanks
333soori wrote:
This is related to Asp.net web interface only..
Then why you have asked the same question on C# Forum also ?
333soori wrote:
I am getting connection timeout so pls suggest me the best way of filtering my files.
Did you configured the Timeout Settings on connection string ? Or you can do one thing, Debug your application, and try to check where its taking the time. Thanks !
Abhijit Jana | Codeproject MVP Web Site : abhijitjana.net Don't forget to click "Good Answer" on the post(s) that helped you. View My Latest Article
-
Hi suresh, This is related to Asp.net web interface only.. In my post i clearly explained the way i am trying to do. Anyways i am doing it for u again. On my web interface i have option to browse two files and then a filter button. whenever user clicks on filer button i need to filter the first file with second file so i am inserting these files into mysql tables and querying with NOT IN select statement. I am getting connection timeout so pls suggest me the best way of filtering my files. Thanks
Index your tables with appropriate key(s). Avoid to use Not In. You can use something like this
select a.* from table1 a, table2 b
where a.column1<>b.column1Configure your connection string's timeout parameter... :cool:
Be an Eagle, Sky is Yours.