How to write log file directly without annoying virtual memory
-
Dear all, I developed one application which collect the list of files in a directory after a comparison. At the beginning I add the result into a datagridview but it seems I take too much virtual memory, for around 5 millions of file comparison. I tried to write the list inside a text file but still the same problem. So what I want to have is that how/what shoud I do to write the list of files after comparison directly into a text file (or list view) without taking too much memory. Thanks for your reply some piece of code would be appreciated.
-
Dear all, I developed one application which collect the list of files in a directory after a comparison. At the beginning I add the result into a datagridview but it seems I take too much virtual memory, for around 5 millions of file comparison. I tried to write the list inside a text file but still the same problem. So what I want to have is that how/what shoud I do to write the list of files after comparison directly into a text file (or list view) without taking too much memory. Thanks for your reply some piece of code would be appreciated.
It would appears that you're adding each filename to a list of some kind. If you don't want to use all that memory, you can't do that. Just write the information to the file as you find it, then don't add it to a collection of any kind.
A guide to posting questions on CodeProject[^]
Dave Kreskowiak Microsoft MVP Visual Developer - Visual Basic
2006, 2007, 2008 -
Dear all, I developed one application which collect the list of files in a directory after a comparison. At the beginning I add the result into a datagridview but it seems I take too much virtual memory, for around 5 millions of file comparison. I tried to write the list inside a text file but still the same problem. So what I want to have is that how/what shoud I do to write the list of files after comparison directly into a text file (or list view) without taking too much memory. Thanks for your reply some piece of code would be appreciated.
Hi, when you store all the file names somewhere (DataGridView, Array, List, ...) is does not scale well, i.e. it works fine for small amounts, but starts to behave badly for large amounts; and disks could hold millions of files. So try and avoid the need to have them all at once. Example: say you want to compare the contents of two disks; there is NO need to first collect a list of all the files, simply come up with a way to enumerate the files one by one in a predictable order, now apply this same order to both disks. When they contain A+B+C+D+E+F+G and C+A+K+B+L you can't compare them sequentially unless you enumerate in an ordered fashion, i.e. A+B+C+D+E+F+G and A+B+C+K+L, these are easily comparable. It you must have them all at once, try to keep the data to a minimum; for files and folders that means don't hold the full path for each, just keep the name and a pointer to its parent. This may save a factor of 2 to 5, which might be sufficient for your current needs, but again it does not scale well, sooner or later you will run into trouble again. :)
Luc Pattyn [Forum Guidelines] [My Articles]
Love, happiness and fewer bugs for 2009!
-
Hi, when you store all the file names somewhere (DataGridView, Array, List, ...) is does not scale well, i.e. it works fine for small amounts, but starts to behave badly for large amounts; and disks could hold millions of files. So try and avoid the need to have them all at once. Example: say you want to compare the contents of two disks; there is NO need to first collect a list of all the files, simply come up with a way to enumerate the files one by one in a predictable order, now apply this same order to both disks. When they contain A+B+C+D+E+F+G and C+A+K+B+L you can't compare them sequentially unless you enumerate in an ordered fashion, i.e. A+B+C+D+E+F+G and A+B+C+K+L, these are easily comparable. It you must have them all at once, try to keep the data to a minimum; for files and folders that means don't hold the full path for each, just keep the name and a pointer to its parent. This may save a factor of 2 to 5, which might be sufficient for your current needs, but again it does not scale well, sooner or later you will run into trouble again. :)
Luc Pattyn [Forum Guidelines] [My Articles]
Love, happiness and fewer bugs for 2009!
Firstly, thanks for your prompt reply. [Dave Kreskowiak] Yeah this is what I want I think write the information directly as I find it, but I don't know how to make it. [Luc Pattyn] So you mean that I'll always get into trouble everytime I make my comparison ? Let me explain to you first what my program do. I have two servers with disk capacity around 7 To each. The first one change everytime, 'cause it is like a transaction server, and second one is just for backing up the first one. Let's say transaction server = 1A and backup server = 1B Once someone made a transaction into 1A, it should be backed up to 1B but no need to delete files from 1B directly. Need to wait about two weeks and after that I should make a purge. So my application is about purging files. Means that it compares first all files of the two servers and then lists those files before delete. It's at this level I'm into a trouble, making the comparison and then list all files. So which solution should I adopt correctly ? Writing files directly inside a text file (by splitting if up to a specific size) or using a dictionnary or list... something like that. Or if have other solution I would appreciate it. for information: my app is made with C# .NET 3.5 Thank you so much
-
Firstly, thanks for your prompt reply. [Dave Kreskowiak] Yeah this is what I want I think write the information directly as I find it, but I don't know how to make it. [Luc Pattyn] So you mean that I'll always get into trouble everytime I make my comparison ? Let me explain to you first what my program do. I have two servers with disk capacity around 7 To each. The first one change everytime, 'cause it is like a transaction server, and second one is just for backing up the first one. Let's say transaction server = 1A and backup server = 1B Once someone made a transaction into 1A, it should be backed up to 1B but no need to delete files from 1B directly. Need to wait about two weeks and after that I should make a purge. So my application is about purging files. Means that it compares first all files of the two servers and then lists those files before delete. It's at this level I'm into a trouble, making the comparison and then list all files. So which solution should I adopt correctly ? Writing files directly inside a text file (by splitting if up to a specific size) or using a dictionnary or list... something like that. Or if have other solution I would appreciate it. for information: my app is made with C# .NET 3.5 Thank you so much
AFAIK I already gave the complete answer. The key word is SORT. One more detail, which should be obvious by now: use GetFiles/GetDirectories without implicit recursion, do the recursion yourself! :)
Luc Pattyn [Forum Guidelines] [My Articles]
Love, happiness and fewer bugs for 2009!
-
AFAIK I already gave the complete answer. The key word is SORT. One more detail, which should be obvious by now: use GetFiles/GetDirectories without implicit recursion, do the recursion yourself! :)
Luc Pattyn [Forum Guidelines] [My Articles]
Love, happiness and fewer bugs for 2009!