filtering Bulk data
-
Hi all, I want to know the better way of filtering Bulk data, pls guide me... I have two flat files with lakhs of rows having 16 digit numbers. i want to filter the numbers from first file with second file , resulting in a unique identified numbers. I am doing it like this. I am inserting these data into two seperate tables using bulk insert in Mysql. And then from my web interface iam inputing two flat files, whenever user clicks on filter button iam querying tables with a select statement using NOT IN sql statement, but it is taking lot of time and even resuling in a connection timeout. So pls suggest me the best way of doing it. Thanks in advance.....
-
Hi all, I want to know the better way of filtering Bulk data, pls guide me... I have two flat files with lakhs of rows having 16 digit numbers. i want to filter the numbers from first file with second file , resulting in a unique identified numbers. I am doing it like this. I am inserting these data into two seperate tables using bulk insert in Mysql. And then from my web interface iam inputing two flat files, whenever user clicks on filter button iam querying tables with a select statement using NOT IN sql statement, but it is taking lot of time and even resuling in a connection timeout. So pls suggest me the best way of doing it. Thanks in advance.....
In many cases, "NOT IN" takes a lot of time. If the table has key column, I suggest following steps. 1. Insert key value, to be filtered, into temporary table. 2. Query with "NOT EXISTS" like "select .. from table where not exists ( select ... temporary_table where table.key = temporary_table.key )"
ToshiyaTSURU turutosiya@gmail.com http://www.google.com/profiles/turutosiya