Read Large Files Into Memory
-
I am trying to decompress a file. To do this, with my current code which you can see below, I am reading the entire file into a byte[]. But if my file is big, I believe bigger then 64 KB, then I can't declare a byte[] big enough to hold the file. How do I fix this, work around this? ---------Code--------- public void GZipDeCompressStream(Stream InStream, Stream OutStream) { //Decompresser GZipStream gzDecompressed = new GZipStream(InStream, CompressionMode.Decompress, true); //Retrieve the size of the decompressed file from the compressed footer byte[] bufferWrite = new byte[4]; InStream.Position = (int)InStream.Length - 4; InStream.Read(bufferWrite, 0, 4); InStream.Position = 0; //Convert to int for using in declaring our Byte[] size int bufferLength = BitConverter.ToInt32(bufferWrite, 0); //Create our Buffer: size + 100 ------------//This is where my issue is. Buffer Lenght is WAY bigger then the 64 KB limit. byte[] buffer = new byte[bufferLength + 100]; int readOffset = 0; int totalBytes = 0; // Loop through the compressed stream and put it into the buffer while (true) { int bytesRead = gzDecompressed.Read(buffer, readOffset, 100); // If we reached the end of the data if (bytesRead == 0) break; readOffset += bytesRead; totalBytes += bytesRead; } // Write the content of the buffer to the destination stream OutStream.Write(buffer, 0, totalBytes); // Close the streams InStream.Close(); gzDecompressed.Close(); OutStream.Close(); }
-
I am trying to decompress a file. To do this, with my current code which you can see below, I am reading the entire file into a byte[]. But if my file is big, I believe bigger then 64 KB, then I can't declare a byte[] big enough to hold the file. How do I fix this, work around this? ---------Code--------- public void GZipDeCompressStream(Stream InStream, Stream OutStream) { //Decompresser GZipStream gzDecompressed = new GZipStream(InStream, CompressionMode.Decompress, true); //Retrieve the size of the decompressed file from the compressed footer byte[] bufferWrite = new byte[4]; InStream.Position = (int)InStream.Length - 4; InStream.Read(bufferWrite, 0, 4); InStream.Position = 0; //Convert to int for using in declaring our Byte[] size int bufferLength = BitConverter.ToInt32(bufferWrite, 0); //Create our Buffer: size + 100 ------------//This is where my issue is. Buffer Lenght is WAY bigger then the 64 KB limit. byte[] buffer = new byte[bufferLength + 100]; int readOffset = 0; int totalBytes = 0; // Loop through the compressed stream and put it into the buffer while (true) { int bytesRead = gzDecompressed.Read(buffer, readOffset, 100); // If we reached the end of the data if (bytesRead == 0) break; readOffset += bytesRead; totalBytes += bytesRead; } // Write the content of the buffer to the destination stream OutStream.Write(buffer, 0, totalBytes); // Close the streams InStream.Close(); gzDecompressed.Close(); OutStream.Close(); }
is long bigger than int in C# ? I thought it was.
Christian Graus - Microsoft MVP - C++ Metal Musings - Rex and my new metal blog "I am working on a project that will convert a FORTRAN code to corresponding C++ code.I am not aware of FORTRAN syntax" ( spotted in the C++/CLI forum )
-
is long bigger than int in C# ? I thought it was.
Christian Graus - Microsoft MVP - C++ Metal Musings - Rex and my new metal blog "I am working on a project that will convert a FORTRAN code to corresponding C++ code.I am not aware of FORTRAN syntax" ( spotted in the C++/CLI forum )
-
I am trying to decompress a file. To do this, with my current code which you can see below, I am reading the entire file into a byte[]. But if my file is big, I believe bigger then 64 KB, then I can't declare a byte[] big enough to hold the file. How do I fix this, work around this? ---------Code--------- public void GZipDeCompressStream(Stream InStream, Stream OutStream) { //Decompresser GZipStream gzDecompressed = new GZipStream(InStream, CompressionMode.Decompress, true); //Retrieve the size of the decompressed file from the compressed footer byte[] bufferWrite = new byte[4]; InStream.Position = (int)InStream.Length - 4; InStream.Read(bufferWrite, 0, 4); InStream.Position = 0; //Convert to int for using in declaring our Byte[] size int bufferLength = BitConverter.ToInt32(bufferWrite, 0); //Create our Buffer: size + 100 ------------//This is where my issue is. Buffer Lenght is WAY bigger then the 64 KB limit. byte[] buffer = new byte[bufferLength + 100]; int readOffset = 0; int totalBytes = 0; // Loop through the compressed stream and put it into the buffer while (true) { int bytesRead = gzDecompressed.Read(buffer, readOffset, 100); // If we reached the end of the data if (bytesRead == 0) break; readOffset += bytesRead; totalBytes += bytesRead; } // Write the content of the buffer to the destination stream OutStream.Write(buffer, 0, totalBytes); // Close the streams InStream.Close(); gzDecompressed.Close(); OutStream.Close(); }
There is no 64KB limit on arrays. If you get close to 2GB you might have problems on 32 bit systems. But anyway, the solution to your problem is quite simple - no need to keep all the data in memory, just save it as you go: byte[] buffer = new byte[1024*1024]; // 1MB buffer while (true) { int bytesRead = gzDecompressed.Read(buffer, 0, buffer.Length); // If we reached the end of the data if (bytesRead == 0) break; outStream.Write(buffer, 0, bytesRead) }
-
This post does not make sense in the context of my question, or you didn't explain yourself well enough.
You're storing the size in an int. A long is bigger.
Christian Graus - Microsoft MVP - C++ Metal Musings - Rex and my new metal blog "I am working on a project that will convert a FORTRAN code to corresponding C++ code.I am not aware of FORTRAN syntax" ( spotted in the C++/CLI forum )
-
You're storing the size in an int. A long is bigger.
Christian Graus - Microsoft MVP - C++ Metal Musings - Rex and my new metal blog "I am working on a project that will convert a FORTRAN code to corresponding C++ code.I am not aware of FORTRAN syntax" ( spotted in the C++/CLI forum )
-
There is no 64KB limit on arrays. If you get close to 2GB you might have problems on 32 bit systems. But anyway, the solution to your problem is quite simple - no need to keep all the data in memory, just save it as you go: byte[] buffer = new byte[1024*1024]; // 1MB buffer while (true) { int bytesRead = gzDecompressed.Read(buffer, 0, buffer.Length); // If we reached the end of the data if (bytesRead == 0) break; outStream.Write(buffer, 0, bytesRead) }
-
is long bigger than int in C# ? I thought it was.
Christian Graus - Microsoft MVP - C++ Metal Musings - Rex and my new metal blog "I am working on a project that will convert a FORTRAN code to corresponding C++ code.I am not aware of FORTRAN syntax" ( spotted in the C++/CLI forum )
Yeah, long is System.Int64, and int is System.Int32;
"Throughout human history, we have been dependent on machines to survive. Fate, it seems, is not without a sense of irony. " - Morpheus "Real men use mspaint for writing code and notepad for designing graphics." - Anna-Jayne Metcalfe