Working with 1GB XML file
-
I have a client server application and i am writing messages from the client to the XML file. sometimes the files reaches 1GB. Operating such a large XML file creates lot of problems such as memory, time. there may be loss of data too. please suggest me a method to operate my file with less memory????? Thanks in advance...
Praveen Raghuvanshi Software Engineer
-
I have a client server application and i am writing messages from the client to the XML file. sometimes the files reaches 1GB. Operating such a large XML file creates lot of problems such as memory, time. there may be loss of data too. please suggest me a method to operate my file with less memory????? Thanks in advance...
Praveen Raghuvanshi Software Engineer
XML is a bad choice handling large amounts of data. Use XmlTextReader and XmlTextWriter for reading/writing XML files. Do not use XmlDocument (it stores the whole document in RAM). Use XML only for import/export of data, store it inside a normal binary database for normal usage.
-
I have a client server application and i am writing messages from the client to the XML file. sometimes the files reaches 1GB. Operating such a large XML file creates lot of problems such as memory, time. there may be loss of data too. please suggest me a method to operate my file with less memory????? Thanks in advance...
Praveen Raghuvanshi Software Engineer
If your XML file is getting that big, then there seems to be an architectural problem here. Have you considered using a database instead (much better suited to things like this). Alternatively, you may want to consider SAX (Simple API for XML) which is an event based rather than tree based API. See http://saxdotnet.sourceforge.net/index.html[^]
the last thing I want to see is some pasty-faced geek with skin so pale that it's almost translucent trying to bump parts with a partner - John Simmons / outlaw programmer
Deja View - the feeling that you've seen this post before. -
I have a client server application and i am writing messages from the client to the XML file. sometimes the files reaches 1GB. Operating such a large XML file creates lot of problems such as memory, time. there may be loss of data too. please suggest me a method to operate my file with less memory????? Thanks in advance...
Praveen Raghuvanshi Software Engineer
I think you should consider to use the SAX approach. XmlDocument uses DOM and the whole document is loaded into the memory (as mentioned above). The SAX approach takes parts of the document only, which may be your solution... SAX is stream based and the stream is bi-directional. Which means you can write and read. It uses events. Unfortunately I can't help you about the .NET implementation of SAX. I don't know if there is any ... You may search for the MSXML (Microsoft XML Core Services). I think this is available as COM object. You also may want to see the following page : http://sourceforge.net/projects/saxdotnet[^] Hope it helps. P.S. if you find a solution to this problem - please drop us a line ...