Reading large quantity of data from a file
-
I have to read a large quantity of bianry data from a file and then store this data in an XML tag. There is approximately 8 MB of data. This can be slow just to read the 8MB into memory using StreamReader. I tried reading sayy 500k at a time. But i still have the issue of storing 8MB into memory. If i would have just read 500k and then write 500k, i would be ok. But i am using XMLTextWriter that does not allow me to write tag data piece by piece. I have to have all the data and then write it to tag. Any suggestions on how to handle this effectively will be appreciated. Thanks, cw
-
I have to read a large quantity of bianry data from a file and then store this data in an XML tag. There is approximately 8 MB of data. This can be slow just to read the 8MB into memory using StreamReader. I tried reading sayy 500k at a time. But i still have the issue of storing 8MB into memory. If i would have just read 500k and then write 500k, i would be ok. But i am using XMLTextWriter that does not allow me to write tag data piece by piece. I have to have all the data and then write it to tag. Any suggestions on how to handle this effectively will be appreciated. Thanks, cw
if you are just saving binary to XML you'll double the size of your data. If you encode it to base64 it'll only be about 4/3 the orginal size, but with conversion overhead. Why do you need 8MB of binary data in XML anyway? To me it sounds like a design flaw not a memory issue.
-Spacix All your skynet questions[^] belong to solved
I dislike the black-and-white voting system on questions/answers. X|
-
if you are just saving binary to XML you'll double the size of your data. If you encode it to base64 it'll only be about 4/3 the orginal size, but with conversion overhead. Why do you need 8MB of binary data in XML anyway? To me it sounds like a design flaw not a memory issue.
-Spacix All your skynet questions[^] belong to solved
I dislike the black-and-white voting system on questions/answers. X|
I actually do encode to base64. It does not even get that far as it is stuck reading all the data from the file. The reason i need to do it this way is because another application creates this data(video) and dumps it to a folder. My application has to read it in, and put in an XML tag along with other data pertaining to it. Any ideas to make this process better from a bandwidth perspective?
-
I actually do encode to base64. It does not even get that far as it is stuck reading all the data from the file. The reason i need to do it this way is because another application creates this data(video) and dumps it to a folder. My application has to read it in, and put in an XML tag along with other data pertaining to it. Any ideas to make this process better from a bandwidth perspective?
Use two files the raw data file the XML info file Then you can use something like CabLib (http://www.codeproject.com/KB/files/CABCompressExtract.aspx)[^] to extract/compress the two files together.
-Spacix All your skynet questions[^] belong to solved
I dislike the black-and-white voting system on questions/answers. X|