Hey Gerry...thanks for the reply! No exceptions yet. I stopped the code after about 1600 text files, and it didn't generate any errors. 1600 text files was around 130,000 records. I commented out the insert statement, and I was flying through all the text files. 24 per second. So, I'm guessing that means my insert command is the problem. I believe they prohibit bulk inserts here, I'll have to try again and see if I'm able to do that. I'm also thinking of maybe inserting multiple records with 1 insert statement? If I could get it to about 10 per second, I'd be happy with that I think. Thanks again! Dave
davers
Posts
-
Parse 17,000 text files and insert into database -
Parse 17,000 text files and insert into databaseHey Gang...I've got a folder with about 17,000 text files in it. I need to parse those text files and insert my parsed results into a Sql database. I've gotten it to where I'm doing about 4 - 5 files per second, but I need it to be faster that that. The text files aren't straight forward either. An example of the data in the text file is: orld Clock Location Entries=INXX0102|Pune|India Weather Location ID=48226|Detroit| MI (48226) English/Metric Units=0 CLNAME001= CLNUMBER001=xxx-xxx-xxxx CLTYPE001=4 CLDATE001=11/09/16 CLTIME001=18:07 CLDURATION001= CLBRIDGEDFLAG001=0 CLMISSEDCNTR001=1 CLBCALBL001= CLNAME002= CLNUMBER002=xxx-xxx-xxxx CLTYPE002=4 CLDATE002=11/09/16 CLTIME002=17:59 CLDURATION002= CLBRIDGEDFLAG002=0 CLMISSEDCNTR002=1 CLBCALBL002= CLNAME003= CLNUMBER003=xxxxxxxxxxxx CLTYPE003=3 CLDATE003=11/09/16 CLTIME003=16:57 CLDURATION003= 1:54 CLBRIDGEDFLAG003=0 CLMISSEDCNTR003=0 CLBCALBL003= etc...... This is a backup text file of an AVAYA 96xx phone. What you see above is 3 calls from the phones call history. There's more in that text file than just calls though, so to get just call info, I grab all the lines that start with "CL". Here's a blurb of my code:
while ((line = file.ReadLine()) != null)
{
if (line.Substring(0, 2) == "CL")
{
try
{
string[] strArray = line.Split("=".ToCharArray());
string key = strArray[0];
string str = strArray[1];One call is made up of 9 elements, so: CLNAME, CLNUMBER, CLTYPE, CLDATE, CLTIME, CLDURATION, CLBRIDGEDFLAG, CLMISSEDCNTR, CLBCALBL all make up one call. My question to you is, how would you go about parsing this out and inserting into a database? Am I going about it the right way? Below is my complete code:
public static void letsdoit(SqlConnection con)
{
string[] files;
string line;
int counter = 0;using (UNCAccessWithCredentials unc = new UNCAccessWithCredentials()) { if (unc.NetUseWithCredentials(@"\\\\ql1telutil1\\c$\\inetpub\\wwwroot", "xxxxxxxxxx", "xx", "xxxxxx")) { files = Directory.GetFiles(@"\\\\ql1telutil1\\c$\\inetpub\\wwwroot\\backup96XX"); foreach (string f in files) { sqlString = null; int myCounter = 0; List<Int32> myCountList = new List<Int32>(); List<Int32> UniqueC