Memory utilization of MemoryStream

elix545

Why MemoryStream object uses double the memory? For example, whether the capacity of MemoryStream is 1GB in Task Manager I see that is consuming 2GB.

Module Module1

Sub Main()
    Dim ObjMemoryStream As System.IO.MemoryStream
    ObjMemoryStream = New System.IO.MemoryStream()
    System.Console.WriteLine("Capacity: {0}", ObjMemoryStream.Capacity.ToString())
    System.Console.WriteLine("Length: {0}", ObjMemoryStream.Length.ToString())

    Dim j As Integer

    For i As Integer = 0 To 1073741824 Step 1
        If (i Mod 256) = 0 Then j = 0 Else j = i Mod 256
        ObjMemoryStream.WriteByte(j)
    Next i

    System.Console.WriteLine("Length: {0}", ObjMemoryStream.Length.ToString())

    Dim ObjFileStream As System.IO.FileStream
    ObjFileStream = System.IO.File.Create("C:\\TEST.TXT", 1024)

    Dim ObjBinaryWriter As System.IO.BinaryWriter
    ObjBinaryWriter = New System.IO.BinaryWriter(ObjFileStream, System.Text.Encoding.ASCII)

    ObjMemoryStream.Position = 0

    ObjBinaryWriter.Write(ObjMemoryStream.ToArray())

    ObjBinaryWriter.Flush()
    ObjBinaryWriter.Close()

    System.Console.ReadKey()

End Sub

End Module

Pete OHanlon

First of all, you don't see the MemoryStream in isolation as far as memory consumption goes. There's the rest of the program, including all the assemblies that get loaded in - they all take up memory. Secondly, don't rely on Task Manager to determine memory profile. Task manager is a notoriously poor way of judging memory usage. Part of the reason for this is that .NET grabs more memory than it needs - this is to give the application room "to grow into" as it runs. Memory allocation is an expensive operation, so having your application grab just the bytes it needs while it's running would be a waste of time.

Dave Kreskowiak

That's how much memory the MemoryStream has set aside for use. Since you didn't specify a capacity, it starts off with 256 bytes by default. As the stream is used and the current capacity is exceeded, it reallocates it's internal buffer and sets a new size larger than the current capacity. That new size is always twice the last size it used. Arrays in .NET are immutable. Once created, you cannot change the size of an array. That limitation applies to everything in the .NET Framework, including the buffer used by MemoryStream. In order to increase the size, a new array of some size must be created and all the elements of the previous array copied to it, then the previous array is destroyed.

A guide to posting questions on CodeProject[^]
Dave Kreskowiak

elix545

I think that object are storing the the data using two memory byte, please check the next example:

Module Module1

Sub Main()         
    Dim ObjMemoryStream As System.IO.MemoryStream
    ObjMemoryStream = New System.IO.MemoryStream(1073741825)
    System.Console.WriteLine("Capacity: {0}", ObjMemoryStream.Capacity.ToString())
    System.Console.WriteLine("Length: {0}", ObjMemoryStream.Length.ToString())

    Dim j As Integer

    For i As Integer = 0 To 1073741824 Step 1
        If (i Mod 256) = 0 Then j = 0 Else j = i Mod 256
        ObjMemoryStream.WriteByte(j)
    Next i

    System.Console.WriteLine("Capacity: {0}", ObjMemoryStream.Capacity.ToString())
    System.Console.WriteLine("Length: {0}", ObjMemoryStream.Length.ToString())

    Dim ObjFileStream As System.IO.FileStream
    ObjFileStream = System.IO.File.Create("C:\\TEST.txt", 1024)

    Dim ObjBinaryWriter As System.IO.BinaryWriter
    ObjBinaryWriter = New System.IO.BinaryWriter(ObjFileStream, System.Text.Encoding.ASCII)

    ObjMemoryStream.Position = 0

    ObjBinaryWriter.Write(ObjMemoryStream.ToArray())

    ObjBinaryWriter.Flush()
    ObjBinaryWriter.Close()

    System.Console.ReadKey()
End Sub

End Module

Dave Kreskowiak

No, it's not. What you're storing is irrelevant. If you keep exceeding the capacity of the MemoryStream, it will keep allocating new arrays, twice as large as the last, until it can't grow any more (2GB). These are the sizes the MemoryStream went through after you created it and keep adding more and more data to it:

256
512
1024
2048
4096
8192
16384
32768
65536
131072
262144
524288
1048576
2097152
4194304
8388608
16777216
33554432
67108864
134217728
268435456
536870912
1073741824
2147483648

Oh, look at that. It managed to hit 2GB in size right after you added 1073741824 bytes of information to it. It's allocating bigger and bigger blocks of memory because it has no way of knowing how big to make the array to that it fits your data perfectly. If you want to keep it from getting to 2GB of memory, allocate the MemoryStream with the expected size as a parameter, say 1.5GB:

Dim ms As New MemoryStream(1610612736)

Now, if you exceed that, the MemoryStream will try to reallocate itself using twice the size you initially specified and that will lead to an OutOfMemory exception. You can NOT look in Task Manager to see how much memory your app is using. It's telling you how much memory the .NET CLR as RESERVED for your app. This will always be larger than what your app is actually using.

A guide to posting questions on CodeProject[^]
Dave Kreskowiak

elix545

Thank you very mush.:thumbsup:

JonB

I assume you're looking at the process size when you hit the ReadKey(). Your ObjMemoryStream will be 1GB + 1 byte big (you start from 0). Then you evaluate ObjMemoryStream.ToArray(). That will create an array (Help: "This method returns a copy of the contents of the MemoryStream as a byte array"), also of size 1GB-odd. (You don't know when .NET will garbage-collect that, and process sizes often don't shrink anyway.) Don't convert your memory stream to an array! Try something like ObjMemoryStream.CopyTo() instead.