can't write all text to txt file
-
hi all i am working on a multi threaded app now in app, 5 or more threads have to write thousands of lines in a text file simultanously i know this kills the cpu but no prob there i have the lines to be written in an static array (I think static makes the process faster.is it true?) my code: For co% = 0 To (strarray.length - 1) writer.WriteLine(strarray(co)) Next where writer is StreamWriter i do the flush close etc my prob is that all files are not being written if array has 25000 elements then sometimes only 10000 or sometimes 15000 are written should i try a timer instead of loop? if not kindly suggest someother means.
TheMrProgrammer
TheMrProgrammer wrote:
i know this kills the cpu but no prob there
No, it kills the file if multiple threads try to write to it.
TheMrProgrammer wrote:
(I think static makes the process faster.is it true?)
No. Seems like you have no idea what you're doing.
TheMrProgrammer wrote:
sometimes only 10000 or sometimes 15000 are written
Because your different threads are going to clash. Try using one thread, given that your array is static, the data is always the same, so having 5 threads trying to write the same file, is kind of retarded, right ?
Christian Graus Driven to the arms of OSX by Vista. "! i don't exactly like or do programming and it only gives me a headache." - spotted in VB forums. I can do things with my brain that I can't even google. I can flex the front part of my brain instantly anytime I want. It can be exhausting and it even causes me vision problems for some reason. - CaptainSeeSharp
-
TheMrProgrammer wrote:
i know this kills the cpu but no prob there
No, it kills the file if multiple threads try to write to it.
TheMrProgrammer wrote:
(I think static makes the process faster.is it true?)
No. Seems like you have no idea what you're doing.
TheMrProgrammer wrote:
sometimes only 10000 or sometimes 15000 are written
Because your different threads are going to clash. Try using one thread, given that your array is static, the data is always the same, so having 5 threads trying to write the same file, is kind of retarded, right ?
Christian Graus Driven to the arms of OSX by Vista. "! i don't exactly like or do programming and it only gives me a headache." - spotted in VB forums. I can do things with my brain that I can't even google. I can flex the front part of my brain instantly anytime I want. It can be exhausting and it even causes me vision problems for some reason. - CaptainSeeSharp
-
TheMrProgrammer wrote:
i know this kills the cpu but no prob there
No, it kills the file if multiple threads try to write to it.
TheMrProgrammer wrote:
(I think static makes the process faster.is it true?)
No. Seems like you have no idea what you're doing.
TheMrProgrammer wrote:
sometimes only 10000 or sometimes 15000 are written
Because your different threads are going to clash. Try using one thread, given that your array is static, the data is always the same, so having 5 threads trying to write the same file, is kind of retarded, right ?
Christian Graus Driven to the arms of OSX by Vista. "! i don't exactly like or do programming and it only gives me a headache." - spotted in VB forums. I can do things with my brain that I can't even google. I can flex the front part of my brain instantly anytime I want. It can be exhausting and it even causes me vision problems for some reason. - CaptainSeeSharp
sorry i forgot to mention that all the threads write to separate files. now suggest some means
TheMrProgrammer
-
sorry i forgot to mention that all the threads write to separate files. now suggest some means
TheMrProgrammer
Your data source is static. I presume your problem is still that you have threads writing files as the data is being filled. Overall, it sounds insane to me. If you're saying that some of the data in the middle does not get written, then I assume this means that your problem is still one of concurrency. Why do you have 5 threads all writing at once ? do they write to the same HDD ?
Christian Graus Driven to the arms of OSX by Vista. "! i don't exactly like or do programming and it only gives me a headache." - spotted in VB forums. I can do things with my brain that I can't even google. I can flex the front part of my brain instantly anytime I want. It can be exhausting and it even causes me vision problems for some reason. - CaptainSeeSharp
-
hi all i am working on a multi threaded app now in app, 5 or more threads have to write thousands of lines in a text file simultanously i know this kills the cpu but no prob there i have the lines to be written in an static array (I think static makes the process faster.is it true?) my code: For co% = 0 To (strarray.length - 1) writer.WriteLine(strarray(co)) Next where writer is StreamWriter i do the flush close etc my prob is that all files are not being written if array has 25000 elements then sometimes only 10000 or sometimes 15000 are written should i try a timer instead of loop? if not kindly suggest someother means.
TheMrProgrammer
Hi, you failed to clearly tell us what you are doing. Writing data, any data, to different files, using one thread per file, can be done correctly, which does not mean it always makes sense. If you want detailed help, show us some code (in PRE tags!) for two of those threads and files. :)
Luc Pattyn [Forum Guidelines] [My Articles]
DISCLAIMER: this message may have been modified by others; it may no longer reflect what I intended, and may contain bad advice; use at your own risk and with extreme care.
-
Your data source is static. I presume your problem is still that you have threads writing files as the data is being filled. Overall, it sounds insane to me. If you're saying that some of the data in the middle does not get written, then I assume this means that your problem is still one of concurrency. Why do you have 5 threads all writing at once ? do they write to the same HDD ?
Christian Graus Driven to the arms of OSX by Vista. "! i don't exactly like or do programming and it only gives me a headache." - spotted in VB forums. I can do things with my brain that I can't even google. I can flex the front part of my brain instantly anytime I want. It can be exhausting and it even causes me vision problems for some reason. - CaptainSeeSharp
hi christian my program is a file searching utility i have made a class SEARCHER which goes to all directories and gets the files that match the search string. i wanted to index all files present in HDD so i made an array of SEARCHER with size = all drives and call it using *.* as search string. all members are called in a separate thread so as to do all drives at once i have added a list box which displays the found files. then i iterate through all items of listbox writing them one by one I have dropped the static arrays but the prob persists. if total items are 25000 then only top 15000 or 10000 are being written
Christian Graus wrote:
you have threads writing files as the data is being filled
no when the list box is filled then only the writing starts
Christian Graus wrote:
do they write to the same HDD ?
yes they write to the same HDD
Christian Graus wrote:
Why do you have 5 threads all writing at once ?
to write more in less time.The no of threads depend on the number of drives that are ready. i also tried running a single thread. in it also, only some files of top are being written.
TheMrProgrammer
-
Hi, you failed to clearly tell us what you are doing. Writing data, any data, to different files, using one thread per file, can be done correctly, which does not mean it always makes sense. If you want detailed help, show us some code (in PRE tags!) for two of those threads and files. :)
Luc Pattyn [Forum Guidelines] [My Articles]
DISCLAIMER: this message may have been modified by others; it may no longer reflect what I intended, and may contain bad advice; use at your own risk and with extreme care.
hi Luc please read my reply to Christian
TheMrProgrammer
-
hi Luc please read my reply to Christian
TheMrProgrammer
Hi, you didn't show any code, at least you explained a bit. "so i made an array of SEARCHER with size = all drives and call it using *.* as search string" is ambiguous; at best you have a double risk: 1. the number of files returned by GetFiles/GetDirectories could be huge, you may run out of memory 2. you may encounter "access denied" problems on special folders such as "System Volume Information". Both of these would show up if you were using try-catrch blocks showing all available exception information, i.e. using Exception.ToString(); qnd everything would vanish silently if you are just swallowing exceptions. :)
Luc Pattyn [Forum Guidelines] [My Articles]
DISCLAIMER: this message may have been modified by others; it may no longer reflect what I intended, and may contain bad advice; use at your own risk and with extreme care.
-
Hi, you didn't show any code, at least you explained a bit. "so i made an array of SEARCHER with size = all drives and call it using *.* as search string" is ambiguous; at best you have a double risk: 1. the number of files returned by GetFiles/GetDirectories could be huge, you may run out of memory 2. you may encounter "access denied" problems on special folders such as "System Volume Information". Both of these would show up if you were using try-catrch blocks showing all available exception information, i.e. using Exception.ToString(); qnd everything would vanish silently if you are just swallowing exceptions. :)
Luc Pattyn [Forum Guidelines] [My Articles]
DISCLAIMER: this message may have been modified by others; it may no longer reflect what I intended, and may contain bad advice; use at your own risk and with extreme care.
thamks for your suggestions Luc but my problem is that out of 25000 files in the list box only top 15000 or 10000 are being written i tried putting the contents of the list box in a string variable but without success For x = 0 to listbox1.Items.Count-1 strVar &= listbox1.Items(x).ToString Next my.comp.filesystem.writealltext(filepath,strVar) but instead "#ERROR 488# is written
TheMrProgrammer
-
thamks for your suggestions Luc but my problem is that out of 25000 files in the list box only top 15000 or 10000 are being written i tried putting the contents of the list box in a string variable but without success For x = 0 to listbox1.Items.Count-1 strVar &= listbox1.Items(x).ToString Next my.comp.filesystem.writealltext(filepath,strVar) but instead "#ERROR 488# is written
TheMrProgrammer
TheMrProgrammer wrote:
For x = 0 to listbox1.Items.Count-1 strVar &= listbox1.Items(x).ToString Next
for counts of many tenthousands, this is non-seense: the final result would probably be a string with a length exceeding one million characters, however building it also creates tenthousands of intermediate strings of growing length, so your code is bound to be extremely slow, and may well run out of memory. furthermore, there was no need whatsoever to concatenate everything, since File class also has a WriteAllLines() method, which takes an array of strings. finally, if you do not absolutely need to hold lots of data in memory, then don't. Just write the text lines as you get them. I know you are failing here so far, due to reasons unknown to all of us as you still haven't shown us real code. Lacking that, I will not continue this thread. :)
Luc Pattyn [Forum Guidelines] [My Articles]
DISCLAIMER: this message may have been modified by others; it may no longer reflect what I intended, and may contain bad advice; use at your own risk and with extreme care.