C# Windows Service
-
Hi, Have written a small Windows service which monitors a single folder waiting for simple text files to arrive in said folder. Once a file or files arrive, the service converts them to XML, deletes the file(s) from the 'in' folder, copies it to an 'out' folder where another process deals with them. It's a simple, clean service which runs fine 99% of the time. However, the other 1% of the time, a file or files arrives in the 'in' folder and nothing happens. The service doesn't seem to have noticed that anything has arrived in the folder it's watching. Does anyone know of any reason why the service would stall every so often. It's a very small problem but one that I'd like to iron out if I can.... Thanks in advance Scott
-
Hi, Have written a small Windows service which monitors a single folder waiting for simple text files to arrive in said folder. Once a file or files arrive, the service converts them to XML, deletes the file(s) from the 'in' folder, copies it to an 'out' folder where another process deals with them. It's a simple, clean service which runs fine 99% of the time. However, the other 1% of the time, a file or files arrives in the 'in' folder and nothing happens. The service doesn't seem to have noticed that anything has arrived in the folder it's watching. Does anyone know of any reason why the service would stall every so often. It's a very small problem but one that I'd like to iron out if I can.... Thanks in advance Scott
Are you using the FileSystemWatcher to catch the fact that a file has been created? If you are, you should be aware that it's not 100% effective. It has some gaping deficiencies.
Forgive your enemies - it messes with their heads
My blog | My articles | MoXAML PowerToys | Mole 2010 - debugging made easier - my favourite utility
-
Are you using the FileSystemWatcher to catch the fact that a file has been created? If you are, you should be aware that it's not 100% effective. It has some gaping deficiencies.
Forgive your enemies - it messes with their heads
My blog | My articles | MoXAML PowerToys | Mole 2010 - debugging made easier - my favourite utility
-
Hi, Have written a small Windows service which monitors a single folder waiting for simple text files to arrive in said folder. Once a file or files arrive, the service converts them to XML, deletes the file(s) from the 'in' folder, copies it to an 'out' folder where another process deals with them. It's a simple, clean service which runs fine 99% of the time. However, the other 1% of the time, a file or files arrives in the 'in' folder and nothing happens. The service doesn't seem to have noticed that anything has arrived in the folder it's watching. Does anyone know of any reason why the service would stall every so often. It's a very small problem but one that I'd like to iron out if I can.... Thanks in advance Scott
Hi. Sometimes it happens that your thread which is responsible for creating XML files tries to read the arrived text file when it's not completely free (some service is still using it). Try to wait for some milliseconds before attempting to open the text file like this:
Thread.Sleep(100);
I hope this will work.
-
I am indeed using FileSystemWatcher to monitor the folder but I admit I was unaware of any deficiencies in it. Can you explain a little what they are please?
It will sometimes send a notification that a file was created before the file has finished being written. That is probably a problem more often with larger files. Like a person said below, the solution is to wait a few milliseconds if the file is not yet available. One way to do this would be to use a try/catch to check if you were able to gain a write lock on the file. If that fails, wait a few milliseconds and try again.
-
Hi, Have written a small Windows service which monitors a single folder waiting for simple text files to arrive in said folder. Once a file or files arrive, the service converts them to XML, deletes the file(s) from the 'in' folder, copies it to an 'out' folder where another process deals with them. It's a simple, clean service which runs fine 99% of the time. However, the other 1% of the time, a file or files arrives in the 'in' folder and nothing happens. The service doesn't seem to have noticed that anything has arrived in the folder it's watching. Does anyone know of any reason why the service would stall every so often. It's a very small problem but one that I'd like to iron out if I can.... Thanks in advance Scott
In addition to the previous answers, I would like to suggest you on increasing the InternalBufferSize property of your file system watcher object to a bigger value. It might help when a large number of files are being created at the same time. The lesser the value of the InternalBufferSize, the more are the chances of file changes / creations going unnoticed by the watcher. Hope this helps.
ramzg
-
In addition to the previous answers, I would like to suggest you on increasing the InternalBufferSize property of your file system watcher object to a bigger value. It might help when a large number of files are being created at the same time. The lesser the value of the InternalBufferSize, the more are the chances of file changes / creations going unnoticed by the watcher. Hope this helps.
ramzg
-
Hi, Have written a small Windows service which monitors a single folder waiting for simple text files to arrive in said folder. Once a file or files arrive, the service converts them to XML, deletes the file(s) from the 'in' folder, copies it to an 'out' folder where another process deals with them. It's a simple, clean service which runs fine 99% of the time. However, the other 1% of the time, a file or files arrives in the 'in' folder and nothing happens. The service doesn't seem to have noticed that anything has arrived in the folder it's watching. Does anyone know of any reason why the service would stall every so often. It's a very small problem but one that I'd like to iron out if I can.... Thanks in advance Scott
Hi, as others already explained, FSW has a limited buffer, so for a very active folder you might miss some events, however that seldom is the problem; the actual problem typically is FSW gets triggered by the beginning of some action (such as file creation), whereas your app typically is interested in the end of the action (file created and available). My solution (applies only if the producer can be slightly changed) is by using a sentinel file: have the producer create and write the file of interest first, then emit a small file that signals the former has been created and written; and now make the FSW watch out for that sentinel file only; it being small causes start and end creation to coincide, so there is no need to wait or to retry. And you could store the data file's name inside the sentinel file, to remove all uncertainty. :)
Luc Pattyn [My Articles] Nil Volentibus Arduum
The quality and detail of your question reflects on the effectiveness of the help you are likely to get.
Please use <PRE> tags for code snippets, they improve readability.
CP Vanity has been updated to V2.4 -
It will sometimes send a notification that a file was created before the file has finished being written. That is probably a problem more often with larger files. Like a person said below, the solution is to wait a few milliseconds if the file is not yet available. One way to do this would be to use a try/catch to check if you were able to gain a write lock on the file. If that fails, wait a few milliseconds and try again.
I'd just like to add that you should acquire a full lock, meaning you don't want to share the file for reading nor for writing (nor for deletion). That way, you'll be sure that no other handles to the file are open before acquiring the lock.
-
Hi, as others already explained, FSW has a limited buffer, so for a very active folder you might miss some events, however that seldom is the problem; the actual problem typically is FSW gets triggered by the beginning of some action (such as file creation), whereas your app typically is interested in the end of the action (file created and available). My solution (applies only if the producer can be slightly changed) is by using a sentinel file: have the producer create and write the file of interest first, then emit a small file that signals the former has been created and written; and now make the FSW watch out for that sentinel file only; it being small causes start and end creation to coincide, so there is no need to wait or to retry. And you could store the data file's name inside the sentinel file, to remove all uncertainty. :)
Luc Pattyn [My Articles] Nil Volentibus Arduum
The quality and detail of your question reflects on the effectiveness of the help you are likely to get.
Please use <PRE> tags for code snippets, they improve readability.
CP Vanity has been updated to V2.4 -
Sorry Luc for being lazy but would this still be an issue when streaming data in a separate thread?
"You get that on the big jobs."
Hi Rob, I have no idea what you are asking here. FWIW: the FSW events get handled in a ThreadPool thread. They fire upon filesystem events such as create file, write to file, delete file, which all indicate the start of an action. There is no event for flushing or closing a file. :)
Luc Pattyn [My Articles] Nil Volentibus Arduum
The quality and detail of your question reflects on the effectiveness of the help you are likely to get.
Please use <PRE> tags for code snippets, they improve readability.
CP Vanity has been updated to V2.4modified on Friday, June 10, 2011 8:38 PM
-
Hi, as others already explained, FSW has a limited buffer, so for a very active folder you might miss some events, however that seldom is the problem; the actual problem typically is FSW gets triggered by the beginning of some action (such as file creation), whereas your app typically is interested in the end of the action (file created and available). My solution (applies only if the producer can be slightly changed) is by using a sentinel file: have the producer create and write the file of interest first, then emit a small file that signals the former has been created and written; and now make the FSW watch out for that sentinel file only; it being small causes start and end creation to coincide, so there is no need to wait or to retry. And you could store the data file's name inside the sentinel file, to remove all uncertainty. :)
Luc Pattyn [My Articles] Nil Volentibus Arduum
The quality and detail of your question reflects on the effectiveness of the help you are likely to get.
Please use <PRE> tags for code snippets, they improve readability.
CP Vanity has been updated to V2.4Good idea. I suppose whatever is writing the file could also write it to a different folder and then perform a move operation to move it to the monitored folder once the file is fully written.
-
Good idea. I suppose whatever is writing the file could also write it to a different folder and then perform a move operation to move it to the monitored folder once the file is fully written.
That would work too, however I'm not fond of moving or renaming files (unless that is exactly what is wanted) as there are too many side-effects. One example is MS Word editing a doc file on the desktop; it renames the old file resulting in a weird filename on the desktop, and creates a new one, in an "arbitrary" location, also on the desktop of course; when done, the old is removed, the new one renamed to its original name, not its original location. And then there is the single responsibility idea, see SOLID. One file holding content, one file signaling new information! :)
Luc Pattyn [My Articles] Nil Volentibus Arduum
The quality and detail of your question reflects on the effectiveness of the help you are likely to get.
Please use <PRE> tags for code snippets, they improve readability.
CP Vanity has been updated to V2.4 -
That would work too, however I'm not fond of moving or renaming files (unless that is exactly what is wanted) as there are too many side-effects. One example is MS Word editing a doc file on the desktop; it renames the old file resulting in a weird filename on the desktop, and creates a new one, in an "arbitrary" location, also on the desktop of course; when done, the old is removed, the new one renamed to its original name, not its original location. And then there is the single responsibility idea, see SOLID. One file holding content, one file signaling new information! :)
Luc Pattyn [My Articles] Nil Volentibus Arduum
The quality and detail of your question reflects on the effectiveness of the help you are likely to get.
Please use <PRE> tags for code snippets, they improve readability.
CP Vanity has been updated to V2.4Well, I think the SRP refers more to the functionality than the data being operated on (the files, in this case). You could have one bit of functionality that creates the file, then another to move the file (or rename it). Now that I think about it, I'd probably go with the rename option (from a practical perspective). Less clutter IMO (no need to keep track of a second folder, and no extra file to clean up).
-
Well, I think the SRP refers more to the functionality than the data being operated on (the files, in this case). You could have one bit of functionality that creates the file, then another to move the file (or rename it). Now that I think about it, I'd probably go with the rename option (from a practical perspective). Less clutter IMO (no need to keep track of a second folder, and no extra file to clean up).
With renaming, you're potentially in for a surprise. You have to close the file before you can rename it, and in that short period of time the file could be opened by any of those utterly helpful little programs (indexers, virus scanners, etc) trying to inspect or process the new information, and thus preventing you from renaming the file. So you end up creating the same wait-and-retry loop you need to get reasonable chance of deleting a newly created file; and this logic needs to be added to the producer, who wouldn't really care much. That is why I prefer the sentinel file scheme. :)
Luc Pattyn [My Articles] Nil Volentibus Arduum
The quality and detail of your question reflects on the effectiveness of the help you are likely to get.
Please use <PRE> tags for code snippets, they improve readability.
CP Vanity has been updated to V2.4 -
Hi Rob, I have no idea what you are asking here. FWIW: the FSW events get handled in a ThreadPool thread. They fire upon filesystem events such as create file, write to file, delete file, which all indicate the start of an action. There is no event for flushing or closing a file. :)
Luc Pattyn [My Articles] Nil Volentibus Arduum
The quality and detail of your question reflects on the effectiveness of the help you are likely to get.
Please use <PRE> tags for code snippets, they improve readability.
CP Vanity has been updated to V2.4modified on Friday, June 10, 2011 8:38 PM
OK, I was just thinking one of the main reasons the FSW InternalBuffer would overflow is because the code block within the subscribing method is taking time (ie reading file) and so other events are backing up in the buffer. And maybe a better way would be to thread the reading of the file into a separate thread to help avoid the InternalBuffer overflowing. I guess it still isn't a solution to the problem anyway, just a way to help minimise it.
"You get that on the big jobs."
-
OK, I was just thinking one of the main reasons the FSW InternalBuffer would overflow is because the code block within the subscribing method is taking time (ie reading file) and so other events are backing up in the buffer. And maybe a better way would be to thread the reading of the file into a separate thread to help avoid the InternalBuffer overflowing. I guess it still isn't a solution to the problem anyway, just a way to help minimise it.
"You get that on the big jobs."
Oh, I see, yes when a consumer temporarily can not keep up with a producer, you would need more buffering, or split the consumption effort in a real-time task and a delayed task (a queue could be helpful here). Under the same conditions, and assuming new files (OnCreate) are of interest, it might be wise not to use a FSW at all, and to scan the folder yourself (e.g.with
Directory.GetFilesor the newerDirectory.EnumerateFiles) especially if the consumer is to also delete the files once processed. :)Luc Pattyn [My Articles] Nil Volentibus Arduum
The quality and detail of your question reflects on the effectiveness of the help you are likely to get.
Please use <PRE> tags for code snippets, they improve readability.
CP Vanity has been updated to V2.4