Ideas on how to design a file listener
-
I'm working on a project that uses the System.IO.FileSystemWatcher to monitor a folder. What I'm wanting to do is copy select types of files from the monitored folder when they are created. This works pretty slick, but the problem I run into is that the FileSystemWatcher event oftentimes fires before the file finishes copying, so copying the file results in an error. I resolved this by creating a timer that's started on the FileSystemWatcher.created event. It waits a reasonable amount of time and then copies the file. No issues. My problem is that this isn't a very good solution, particularly when hundreds or thousands of files are being copied onto the watched drive, as this can result in thousands of threads. I wanted to resolve this by having the FileSystemWatcher.created thread add the files to be copied to an arrayList, then have a separate single thread monitor the array... wait for the file to be available to be copied, then start copying files. However this has more problems since the arrayList collection is changed on a separate thread which introduces a lot of locking issues, which I'd rather avoid. Can anyone offer an idea of what the best practice for this situation would be? I appreciate any input. :laugh: Thanks! Dana
-
I'm working on a project that uses the System.IO.FileSystemWatcher to monitor a folder. What I'm wanting to do is copy select types of files from the monitored folder when they are created. This works pretty slick, but the problem I run into is that the FileSystemWatcher event oftentimes fires before the file finishes copying, so copying the file results in an error. I resolved this by creating a timer that's started on the FileSystemWatcher.created event. It waits a reasonable amount of time and then copies the file. No issues. My problem is that this isn't a very good solution, particularly when hundreds or thousands of files are being copied onto the watched drive, as this can result in thousands of threads. I wanted to resolve this by having the FileSystemWatcher.created thread add the files to be copied to an arrayList, then have a separate single thread monitor the array... wait for the file to be available to be copied, then start copying files. However this has more problems since the arrayList collection is changed on a separate thread which introduces a lot of locking issues, which I'd rather avoid. Can anyone offer an idea of what the best practice for this situation would be? I appreciate any input. :laugh: Thanks! Dana
Hi, 1. yes, FileSystemWatcher is rather good at signaling the start of an action, it does not offer a Done event, and the best I ever could find was a loop, trying and waiting some, till success. 2. hundreds of threads doesn't make sense, they all take a lot of resources. Anyway having multiple threads each copying one or a few files will not help, as soon the disk bandwidth will be the bottleneck. So don't take the multithreading hassle. 3. the FSW events occur asynchronously, so their handlers aren't running on your main thread, instead they use a ThreadPoolThread. If you copy the file right away, you would be too soon, and also hog the thread. So you will end up using at least two threads: one for the FSW event (probably filling a queue), and one for copying the files, maybe a BackgroundWorker. 4. Hence my proposal is: use a queue to store the incoming file paths; use a BGW that reads all entries from the queue and copies the files. If the copy fails (it will initially unless the file is really small), let it enqueue the file path again. When all entries have been processed (ignoring the ones requeued), then wait say one second, and repeat. You do need a lock for those queue operations, both at the producer side (FSW event and requeueing) and at the consumer side (BGW dequeue); make sure to keep the lock short, don't do File.Copy within the lock! :)
Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles]
I only read code that is properly formatted, adding PRE tags is the easiest way to obtain that.
[The QA section does it automatically now, I hope we soon get it on regular forums as well]
-
I'm working on a project that uses the System.IO.FileSystemWatcher to monitor a folder. What I'm wanting to do is copy select types of files from the monitored folder when they are created. This works pretty slick, but the problem I run into is that the FileSystemWatcher event oftentimes fires before the file finishes copying, so copying the file results in an error. I resolved this by creating a timer that's started on the FileSystemWatcher.created event. It waits a reasonable amount of time and then copies the file. No issues. My problem is that this isn't a very good solution, particularly when hundreds or thousands of files are being copied onto the watched drive, as this can result in thousands of threads. I wanted to resolve this by having the FileSystemWatcher.created thread add the files to be copied to an arrayList, then have a separate single thread monitor the array... wait for the file to be available to be copied, then start copying files. However this has more problems since the arrayList collection is changed on a separate thread which introduces a lot of locking issues, which I'd rather avoid. Can anyone offer an idea of what the best practice for this situation would be? I appreciate any input. :laugh: Thanks! Dana
Switch this round, and stop thinking of reacting and using the FileSystemWatcher - as it does have problems; which you've discovered now. So, if you can't copy the files like this, how can you do it? Well, a method I've had a lot of success with is to use Microsoft's Synchronisation Framework[^] which provides a file synchronisation provider, so you can transfer files based on periodical synchronisations of the folders.
"WPF has many lovers. It's a veritable porn star!" - Josh Smith
As Braveheart once said, "You can take our freedom but you'll never take our Hobnobs!" - Martin Hughes.
-
I'm working on a project that uses the System.IO.FileSystemWatcher to monitor a folder. What I'm wanting to do is copy select types of files from the monitored folder when they are created. This works pretty slick, but the problem I run into is that the FileSystemWatcher event oftentimes fires before the file finishes copying, so copying the file results in an error. I resolved this by creating a timer that's started on the FileSystemWatcher.created event. It waits a reasonable amount of time and then copies the file. No issues. My problem is that this isn't a very good solution, particularly when hundreds or thousands of files are being copied onto the watched drive, as this can result in thousands of threads. I wanted to resolve this by having the FileSystemWatcher.created thread add the files to be copied to an arrayList, then have a separate single thread monitor the array... wait for the file to be available to be copied, then start copying files. However this has more problems since the arrayList collection is changed on a separate thread which introduces a lot of locking issues, which I'd rather avoid. Can anyone offer an idea of what the best practice for this situation would be? I appreciate any input. :laugh: Thanks! Dana
As it was said before FSW isn't perfect, e.g. it doesn't report about each new file. My work around was to scan the whole dir every time FSW.created raised. Regarding other questions Luc Pattyn suggested a viable approach. PS: once you are doing such stuff you should familiarize yourself with synchronization methods.