best pratice
-
Hi all, I need some advice on the best method to impelemt some functionality. I have a reader obj that reads all the files in a folder, running in a thread. After each file has been read I call a processor obj to manipulte the data and fill a database. I'd like to process one file while reading the next, but limit the processor to only one instance (to prevent duplication of data in the DB). If the reader finishes reading before the processor it should wait until the processor is done its current data load, but if the porcessor has finished then the reader should make a new processor and then continue to read the next file. So whats the best method of controling the reader depending on the state of the processor? I should point out that I'm fairly new to C# (six months) Thanks
-
Hi all, I need some advice on the best method to impelemt some functionality. I have a reader obj that reads all the files in a folder, running in a thread. After each file has been read I call a processor obj to manipulte the data and fill a database. I'd like to process one file while reading the next, but limit the processor to only one instance (to prevent duplication of data in the DB). If the reader finishes reading before the processor it should wait until the processor is done its current data load, but if the porcessor has finished then the reader should make a new processor and then continue to read the next file. So whats the best method of controling the reader depending on the state of the processor? I should point out that I'm fairly new to C# (six months) Thanks
If your only going to be using a single instance processor, why read ahead of it? You're not gaining any real performance benefit because of the single threaded nature of your design. One thread is constantly waiting for another to complete it's work. The FileStream classes already do read/write caching on their own, so why duplicate its effort? If you could use a multiple instance processor, instead of a single, THEN you'd see the performance gain. Also, your reader object shouldn't be reading the files itself. Your design should have an engine that hunts down files, then passes the filename to a threaded instance of the processor. That way, the processor thread is doing the reading and processing of the file while your engine goes on it's merry way to launch another processor on the next file. The thread pool could be used for such a task. That way, you wouldn't have to include very much code to do thread management. I only say this because I did something similar to this a few years ago. But in my case, the database made sure I wasn't duplicating data, not my processor objects. RageInTheMachine9532 "...a pungent, gastly, stinky piece of cheese!" -- The Roaming Gnome
-
If your only going to be using a single instance processor, why read ahead of it? You're not gaining any real performance benefit because of the single threaded nature of your design. One thread is constantly waiting for another to complete it's work. The FileStream classes already do read/write caching on their own, so why duplicate its effort? If you could use a multiple instance processor, instead of a single, THEN you'd see the performance gain. Also, your reader object shouldn't be reading the files itself. Your design should have an engine that hunts down files, then passes the filename to a threaded instance of the processor. That way, the processor thread is doing the reading and processing of the file while your engine goes on it's merry way to launch another processor on the next file. The thread pool could be used for such a task. That way, you wouldn't have to include very much code to do thread management. I only say this because I did something similar to this a few years ago. But in my case, the database made sure I wasn't duplicating data, not my processor objects. RageInTheMachine9532 "...a pungent, gastly, stinky piece of cheese!" -- The Roaming Gnome
Thanks for your comments. The majority of the time the app will be run on a single instance processor machine, but there may be a case where the volume of data is so large a multi machine could be used. I wish the database would control data duplication, but the data is so poor and badly supplied that I have to do all the checking that I can. For example the data is in XML format and one 33mb file had over 10,000 violations against the schema, but it was well formed! In the interests of simplicity I leave it as single threaded.