FileStream object and performance concern
-
Hello everyone, I'd like to describe thoroughly my current scenario but it'd get a little bit verbose and perhaps misleading. Let's say the main thread is queuing items to load and another thread should be loading asynchronously these items. My main question is about the performance of creating and disposing of several FileStream objects, used to load files asynchronously several times in a row. If you were to create a producer/consumer queue bound to async load files from hard-disk, how would you go? Would the disposal and creation of FileStream objects be of concern for you? Thanks in advance! Rob
-
Hello everyone, I'd like to describe thoroughly my current scenario but it'd get a little bit verbose and perhaps misleading. Let's say the main thread is queuing items to load and another thread should be loading asynchronously these items. My main question is about the performance of creating and disposing of several FileStream objects, used to load files asynchronously several times in a row. If you were to create a producer/consumer queue bound to async load files from hard-disk, how would you go? Would the disposal and creation of FileStream objects be of concern for you? Thanks in advance! Rob
You will be bound by the disk I/O at that point, so it will depend on how much work your doing with that stream. It might make more sense to use a memory stream and when your done, quickly dump that to disk. I'd probabyl use MSMQ for my process list and have an external service pick up those messages to do the processing. Using MSMQ allows you to seperate out the functions of accepting work and doing work. The messages will also be persisted during failures by MSMQ, so it also gives you a safe guard. I haven't looked at it much atall, but I think WCF can use MSMQ for it's message delivery medium. If you decide to go the route of seperating the processes, take a look at WCF as the communication mechanism.
-
Hello everyone, I'd like to describe thoroughly my current scenario but it'd get a little bit verbose and perhaps misleading. Let's say the main thread is queuing items to load and another thread should be loading asynchronously these items. My main question is about the performance of creating and disposing of several FileStream objects, used to load files asynchronously several times in a row. If you were to create a producer/consumer queue bound to async load files from hard-disk, how would you go? Would the disposal and creation of FileStream objects be of concern for you? Thanks in advance! Rob
Hi, It has already been said that disk I/O will most likely be your performance bottleneck anyway, making object allocation / garbage collection a non-issue from a performance point of view. However, disposing of file streams is in my experience often critical - as the physical file will stay open (and possibly locked) until the stream is disposed (or Close()d, which is equivalent if I remember correctly - either of them kills the Win32 file handle). Unless you're the only one touching the files, this can cause problems. I routinely wrap any file I/O in
usingblocks when possible, in order to minimize file lock durations. In similar scenarios I usually just enqueue what needs to be done in a Queue<my_job_type> and dequeue it from a worker thread (locking the Queue when enqueuing or dequeueing). Usually I use a Queue<my_result_type> for the results from the worker thread. Note: The normal restriction on only locking private stuff applies, of course. Your queues should not be visible outside of your class - if people in other classes (as would be expected) enqueue things to do and dequeue results, you wrap those operations (and queue lengths or HasWorkToDo and HasResults properties or whatever feels appropriate) in methods and properties of your own class. User code calls your en-/dequeueing methods (which uselockblocks), not the ones of the actual Queues. I usually raise an event as well whenever I have enqueued stuff to the results queue, but beware that naïve users of your class may run into problems when they touch UI stuff from the event handler if they forget to Invoke() them properly (as the event will be raised from another thread than the UI thread). You should document that if your class isn't strictly internal. Later,-- Peter
-
Hello everyone, I'd like to describe thoroughly my current scenario but it'd get a little bit verbose and perhaps misleading. Let's say the main thread is queuing items to load and another thread should be loading asynchronously these items. My main question is about the performance of creating and disposing of several FileStream objects, used to load files asynchronously several times in a row. If you were to create a producer/consumer queue bound to async load files from hard-disk, how would you go? Would the disposal and creation of FileStream objects be of concern for you? Thanks in advance! Rob
First, thanks for your feedback. I realize I've been a bit too vague in my initial post; however your suggestions reflect what I initially suspected. Let me clarify a bit. I'm currently working on a 3D game and I've created a texture loader using the producer / consumer queue pattern. Depending on the view distance and the player position, I load and unload landscape textures from hard disk. I assumed (wrongly) that loading textures from another thread would be non-blocking for the main thread; this was not the case, since upon testing I was experiencing hiccups. In another forum I read that perhaps an asynchronous loading would generally perform better (i.e. no hiccups). Seeing as I'm bound to load many textures per loader usage, I was curious to know about the performance of FileStream objects. I know they have to be correctly disposed; I'm not sure how the loader would behave when there are many concurrent asynchronous locks - if there's such a thing. I know some testing would pretty much answer, but I thought that having some feedback, some first-hand experience on the matter would save me some time (which is never enough!). Thanks again for any further advice, Rob PS: For any of you interested, some game screenshots can be found in the Gallery section of www.darkmana.com[^]