OutOfMemory loading massive XML file

[Martin23]

Hi, I am writing a small program that opens a very large (almost 4GB) XML file, then looks through the data for various text matches etc. (I don't need to display or write to the XML file). The problem I have is using the following code to open the XML file; StreamReader sr = new StreamReader(Filename); XmlTextReader xr = new XmlTextReader(sr); XmlDocument XMLdoc = new XmlDocument(); XMLdoc.Load(xr); This loads the entire file, which causes an OutOfMemoryException error when it reaches the 2GB limit. It works ok for small XML files that I have been testing on, but I dont know how to deal with this massive file. I assume I am meant to open the file in chunks or something, but I haven't been able to find any info on how acheive this (other than upgrading to 64bit, but that isnt really practical). Can anybody point me in the right direction? thanks!

[Guffa]

You could parse the nodes yourself if the file has a basically simple structure, e.g. something similar to: <root> <node ... >...</node> <node ... >...</node> <node ... >...</node> <node ... >...</node> <node ... >...</node> ... lots'a nodes <node ... >...</node> </root> By reading the file in small parts, you could extract the complete nodes you find in that part of the file, put them in a separate xml document in a string and load it into a XmlDocument object. Pseudo code: buffer = "" loop { buffer += stream.Read(lotsabytes) find first "<node>" in buffer find last "</node>" in buffer nodes = get what's between buffer = what's after xmldoc.LoadXml("<root>" + nodes+ "</root>") ... do whatever you want with the nodes } --- b { font-weight: normal; }<

[Martin23]

I see what you're getting at, thanks!

[Christian Graus]

I believe the XMLDataReader provides a solution that reads the file as it parses it. If not, there must be some control that works as SAX instead of DOM, and doesn't hold the whole file in memory. Christian Graus - Microsoft MVP - C++

[Matt Gerrans]

Yes, take the SAX approach -- open it as a Stream and use XmlTextReader. Not only will it help in not running out of memory, but it would be faster than the DOM approach even if you did have enough memory to load the whole thing.

Stream stream = new FileStream(fileName, FileMode.Open);
XmlTextReader reader = new XmlTextReader(stream);
reader.WhitespaceHandling = WhitespaceHandling.None;
while (reader.Read())
{
...
}

This is also a lot better idea than parsing the file with regular expressions, or worse, junk like text.IndexOf(blah), especially when you consider the fact that there is no guaranteed that a whole xml file isn't a single line of text (then you are back to loading the whole thing into memory, probably). Matt Gerrans

[Martin23]

Thanks guys, I should mange from here!.

[Matt Gerrans]

> Thanks guys, I should mange from here! Hmm... I think soap and water will help with that. If not you should probably see a doctor. ;P Matt Gerrans