looking for good ideas
-
hy everyone! i am looking for good ideas. what i want to do is to realize a file/directorywatcher, which handles xml files and when it is done it places a "finished" file into this directory to indicate the file was handled. example: i do have 5 directories (a, b, c, d and e) which contain different xml files. only one in each folder. when my program has handled a file in one folder, it places a "finished" file in there. sounds easy :) well my problem is, i found out if it takes to long to handle a file and the other proggy (which creates the xml files) finds a "finished" file, then it starts to consume these files and folders. this means when one "finished" file is found then it deletes all folders one by one. and if my program isn't quick enough, files are lost. that's why i tried to figure out a different solution to this problem. i have to add the "finished" files to the directories after having handled tue last xml file, which was found. well therefore i have to remember the directories, i parsed before. i thought about using a string similar to csv e.g. "dir1;dir2;...", split it afterwards and add the file. or to create an array adding one element after each other. but both solutions could become very timeconsuming. the bigger the string will get, the longer it will take to rewrite it (no idea if there is a function like stringbuilder in c#). and rewriting an array to add an element isn't nice neither. does anyone of you have a good idea how to handle it? i also thought about creating a temp "finished" file and rename it afterwards, but i will have to remeber the directories as well to do so. so to summarize the situation is like this: 1) read xml files from different directories and handle them (one by one - read one dir, handle xml file, next dir etc.) 2) when all xml files are handled which where found in the directories, then write a "finished" file in each directory to indicate the operation was finished 3) start at 1) again whit the next files it's allowed to have dirA finished dirB finished .... dirZ finished but it's not allowed to have dirA finished dirB finished ..... dirX file.xml dirZ file.xml my old solutions was quite easy: reading the file and write a "finished" file to this directory, then continue with the next one etc. with this solution no storage of directories was needed, but this didn't work properly enough :^) thanks for any good ideas! stephan.
-
hy everyone! i am looking for good ideas. what i want to do is to realize a file/directorywatcher, which handles xml files and when it is done it places a "finished" file into this directory to indicate the file was handled. example: i do have 5 directories (a, b, c, d and e) which contain different xml files. only one in each folder. when my program has handled a file in one folder, it places a "finished" file in there. sounds easy :) well my problem is, i found out if it takes to long to handle a file and the other proggy (which creates the xml files) finds a "finished" file, then it starts to consume these files and folders. this means when one "finished" file is found then it deletes all folders one by one. and if my program isn't quick enough, files are lost. that's why i tried to figure out a different solution to this problem. i have to add the "finished" files to the directories after having handled tue last xml file, which was found. well therefore i have to remember the directories, i parsed before. i thought about using a string similar to csv e.g. "dir1;dir2;...", split it afterwards and add the file. or to create an array adding one element after each other. but both solutions could become very timeconsuming. the bigger the string will get, the longer it will take to rewrite it (no idea if there is a function like stringbuilder in c#). and rewriting an array to add an element isn't nice neither. does anyone of you have a good idea how to handle it? i also thought about creating a temp "finished" file and rename it afterwards, but i will have to remeber the directories as well to do so. so to summarize the situation is like this: 1) read xml files from different directories and handle them (one by one - read one dir, handle xml file, next dir etc.) 2) when all xml files are handled which where found in the directories, then write a "finished" file in each directory to indicate the operation was finished 3) start at 1) again whit the next files it's allowed to have dirA finished dirB finished .... dirZ finished but it's not allowed to have dirA finished dirB finished ..... dirX file.xml dirZ file.xml my old solutions was quite easy: reading the file and write a "finished" file to this directory, then continue with the next one etc. with this solution no storage of directories was needed, but this didn't work properly enough :^) thanks for any good ideas! stephan.
We had a similar problem using a "hot" folder, the enterprise ETL tool would write the files and out local tool (custom written ETL) would get a locked exception when trying to read the file while it is still being written. Simply implemented a delay loop into our ETL. First time it sees the file during a processing period (18 hours NOT based on date) it sets a time flag. ETL will then ignore the file until the delay period has expired. As the delay is a setting we can vary it if there are still clashes, it is currently set on 120 seconds. I think your issues is a management problem, are the next set of files (5) the same name as the first set (very nasty) or are there discreet sets of 5 files all the same name going into the different folders 9you only have to manage the file names and a count of folders processed per file name). Create a table with filename and either a folder count or a time for each folder if more detail is required. Stick the filename in there the first time the file arrives with the folder details if required, keep going till filled (count = 5 or each folder col has a time) Write the file to a log. Flush the table every period with incomplete loads.
Never underestimate the power of human stupidity RAH
-
hy everyone! i am looking for good ideas. what i want to do is to realize a file/directorywatcher, which handles xml files and when it is done it places a "finished" file into this directory to indicate the file was handled. example: i do have 5 directories (a, b, c, d and e) which contain different xml files. only one in each folder. when my program has handled a file in one folder, it places a "finished" file in there. sounds easy :) well my problem is, i found out if it takes to long to handle a file and the other proggy (which creates the xml files) finds a "finished" file, then it starts to consume these files and folders. this means when one "finished" file is found then it deletes all folders one by one. and if my program isn't quick enough, files are lost. that's why i tried to figure out a different solution to this problem. i have to add the "finished" files to the directories after having handled tue last xml file, which was found. well therefore i have to remember the directories, i parsed before. i thought about using a string similar to csv e.g. "dir1;dir2;...", split it afterwards and add the file. or to create an array adding one element after each other. but both solutions could become very timeconsuming. the bigger the string will get, the longer it will take to rewrite it (no idea if there is a function like stringbuilder in c#). and rewriting an array to add an element isn't nice neither. does anyone of you have a good idea how to handle it? i also thought about creating a temp "finished" file and rename it afterwards, but i will have to remeber the directories as well to do so. so to summarize the situation is like this: 1) read xml files from different directories and handle them (one by one - read one dir, handle xml file, next dir etc.) 2) when all xml files are handled which where found in the directories, then write a "finished" file in each directory to indicate the operation was finished 3) start at 1) again whit the next files it's allowed to have dirA finished dirB finished .... dirZ finished but it's not allowed to have dirA finished dirB finished ..... dirX file.xml dirZ file.xml my old solutions was quite easy: reading the file and write a "finished" file to this directory, then continue with the next one etc. with this solution no storage of directories was needed, but this didn't work properly enough :^) thanks for any good ideas! stephan.