Data Imports and Exports - My Demon
-
I've spend a large part of my programming career doing these, and either I make them complicated (generic, with mapping files and a designer), or something else does. Today, after testing on two DB's at home, and conquering some model validation using EF, I deploy to the remote test site, and lo, I get a duplicate key error on AccountNum for employee xxxx, while importing from CSV. There is only one AccountNum like xxxx's in the entire CSV file, and no AccountNum like xxxx's in the already in the DB. About 100 other duplicate AccountNum issues are caught and reported by my code before going to the DB. One entry is inserted/updated per transaction, so where the frakking hell is this duplicate being detected. I quoted an hour for this job because I'm using the wonderful FileHelpers library, and everyone was happy, all was working - all I had to to was revert to CSV from fixed len. :mad: :mad: :mad:
-
I've spend a large part of my programming career doing these, and either I make them complicated (generic, with mapping files and a designer), or something else does. Today, after testing on two DB's at home, and conquering some model validation using EF, I deploy to the remote test site, and lo, I get a duplicate key error on AccountNum for employee xxxx, while importing from CSV. There is only one AccountNum like xxxx's in the entire CSV file, and no AccountNum like xxxx's in the already in the DB. About 100 other duplicate AccountNum issues are caught and reported by my code before going to the DB. One entry is inserted/updated per transaction, so where the frakking hell is this duplicate being detected. I quoted an hour for this job because I'm using the wonderful FileHelpers library, and everyone was happy, all was working - all I had to to was revert to CSV from fixed len. :mad: :mad: :mad:
It is most likely truncating to one character thus the matches :laugh: Importing to SQL 2008 is currently driving me deeper into insanity as what worked in SQL 2000 does not in 2008. Thus I feel your pain. Good Luck
-
It is most likely truncating to one character thus the matches :laugh: Importing to SQL 2008 is currently driving me deeper into insanity as what worked in SQL 2000 does not in 2008. Thus I feel your pain. Good Luck
I've just taken a 2km walk to buy supper, and get some air, and within the first 500m it struck me. I'm using EF, and if I add a new entity to an entity set, then detect it has a duplicate field or whatever, and don't remove it from the entity set, the the next call to SaveChanges includes two updates, not just the one the error is reported on, but the previous one that actually causes the error.
-
I've spend a large part of my programming career doing these, and either I make them complicated (generic, with mapping files and a designer), or something else does. Today, after testing on two DB's at home, and conquering some model validation using EF, I deploy to the remote test site, and lo, I get a duplicate key error on AccountNum for employee xxxx, while importing from CSV. There is only one AccountNum like xxxx's in the entire CSV file, and no AccountNum like xxxx's in the already in the DB. About 100 other duplicate AccountNum issues are caught and reported by my code before going to the DB. One entry is inserted/updated per transaction, so where the frakking hell is this duplicate being detected. I quoted an hour for this job because I'm using the wonderful FileHelpers library, and everyone was happy, all was working - all I had to to was revert to CSV from fixed len. :mad: :mad: :mad:
Ah stop yer bitching, I have spent the last 3 weeks getting 400gb of text files into a semi reasonable data structure by trial and error. I have 1 file with 79 million rows that I have loaded twice :sigh: Then the transaction log blows out to 39gb and crashes 12 hours into the processing AAahhh. 6 hours to apply and index, reduced a process from 22hrs to 4, SQL Server is just not up to this volume of data. We keep blowing the drive or log files.
Never underestimate the power of human stupidity RAH
-
I've spend a large part of my programming career doing these, and either I make them complicated (generic, with mapping files and a designer), or something else does. Today, after testing on two DB's at home, and conquering some model validation using EF, I deploy to the remote test site, and lo, I get a duplicate key error on AccountNum for employee xxxx, while importing from CSV. There is only one AccountNum like xxxx's in the entire CSV file, and no AccountNum like xxxx's in the already in the DB. About 100 other duplicate AccountNum issues are caught and reported by my code before going to the DB. One entry is inserted/updated per transaction, so where the frakking hell is this duplicate being detected. I quoted an hour for this job because I'm using the wonderful FileHelpers library, and everyone was happy, all was working - all I had to to was revert to CSV from fixed len. :mad: :mad: :mad:
Hey dude, did you perhaps present a session at Devs4Devs a few weeks back?
Harvey Saayman - South Africa Software Developer .Net, C#, SQL
you.suck = (you.Occupation == jobTitles.Programmer && you.Passion != Programming)1000100 1101111 1100101 1110011 100000 1110100 1101000 1101001 1110011 100000 1101101 1100101 1100001 1101110 100000 1101001 1101101 100000 1100001 100000 1100111 1100101 1100101 1101011 111111 -
Hey dude, did you perhaps present a session at Devs4Devs a few weeks back?
Harvey Saayman - South Africa Software Developer .Net, C#, SQL
you.suck = (you.Occupation == jobTitles.Programmer && you.Passion != Programming)1000100 1101111 1100101 1110011 100000 1110100 1101000 1101001 1110011 100000 1101101 1100101 1100001 1101110 100000 1101001 1101101 100000 1100001 100000 1100111 1100101 1100101 1101011 111111I was supposed to but pranged my scooter that morning and didn't make it.
-
I was supposed to but pranged my scooter that morning and didn't make it.
Ah that sux! I was looking out for you to say hi. I was there :)
Harvey Saayman - South Africa Software Developer .Net, C#, SQL
you.suck = (you.Occupation == jobTitles.Programmer && you.Passion != Programming)1000100 1101111 1100101 1110011 100000 1110100 1101000 1101001 1110011 100000 1101101 1100101 1100001 1101110 100000 1101001 1101101 100000 1100001 100000 1100111 1100101 1100101 1101011 111111