Memory optimisation
-
I have a routine which reads a header record from a file, that header record tells how many subsequent records to read, each record having another header which says how long that record is in bytes. Is it more efficient to allocate an array of (say) 10,000 bytes at the beginning of the loop which process the records then use:
BinaryReader.Read(buffer,0,reclength);or is it better to just allocate the bytes required on the fly:byte[] buffer = BinaryReader.ReadBytes(recLength);For this example the recLength will never exceed 10,000 but could be anything between 1 and 10,000 - the loop can require between 40 and 200 iterations. -
I have a routine which reads a header record from a file, that header record tells how many subsequent records to read, each record having another header which says how long that record is in bytes. Is it more efficient to allocate an array of (say) 10,000 bytes at the beginning of the loop which process the records then use:
BinaryReader.Read(buffer,0,reclength);or is it better to just allocate the bytes required on the fly:byte[] buffer = BinaryReader.ReadBytes(recLength);For this example the recLength will never exceed 10,000 but could be anything between 1 and 10,000 - the loop can require between 40 and 200 iterations.Especially since this is C#, I would code whichever method is easiest for you and then only worry about it if there is an actual performance problem. When you get to that point, creating one large buffer will typically go on the large object heap and requires more memory at one time, which might be faster on a single user system with few memory constraints. Your second approach will use more memory during processing, however it will give the garbage collector a chance to collect already processed buffers. A third approach is to use the stream directly, one pre-allocated buffer and read the data from the stream into the buffer without any allocations in your loop. This is more difficult especially if the records are not fixed size, but will typically perform the best of all the options since there is no allocation in the loop. However, your loop needs to be driving an efficient processing operation for optimizing reading to make sense. If you are just creating an array of record objects in your loop for example, these changes won't make much of a difference.
I can imagine the sinking feeling one would have after ordering my book, only to find a laughably ridiculous theory with demented logic once the book arrives - Mark McCutcheon
-
I have a routine which reads a header record from a file, that header record tells how many subsequent records to read, each record having another header which says how long that record is in bytes. Is it more efficient to allocate an array of (say) 10,000 bytes at the beginning of the loop which process the records then use:
BinaryReader.Read(buffer,0,reclength);or is it better to just allocate the bytes required on the fly:byte[] buffer = BinaryReader.ReadBytes(recLength);For this example the recLength will never exceed 10,000 but could be anything between 1 and 10,000 - the loop can require between 40 and 200 iterations.