Searching HUGE XML file
-
Hi, I have a very huge XML file ( around 30 MB in size).. The problem is i need to search nodes( many of them, basically BFS) XML file contains data in linear fashion.. now searhing around 30,000 nodes takes long time... can any one suggest which class could i use here to obtain faster results.. i am confused with all the xml classes .NET provides.. as to which class's object would solve my purpose... probably some code snippets supporting your answer would be of great help:) Thanks techno_adi -- modified at 11:33 Thursday 20th April, 2006
-
Hi, I have a very huge XML file ( around 30 MB in size).. The problem is i need to search nodes( many of them, basically BFS) XML file contains data in linear fashion.. now searhing around 30,000 nodes takes long time... can any one suggest which class could i use here to obtain faster results.. i am confused with all the xml classes .NET provides.. as to which class's object would solve my purpose... probably some code snippets supporting your answer would be of great help:) Thanks techno_adi -- modified at 11:33 Thursday 20th April, 2006
:omg: That's a large amount of XML data! Since its such a large file, I would advise against using XmlDocument or DataSet which will load the entire data into memory. Instead you need to use an XmlReader, which only provides one way access to maximise performance. I'll quote from MSDN: "Represents a reader that provides fast, non-cached, forward-only access to XML data.".
-
Hi, I have a very huge XML file ( around 30 MB in size).. The problem is i need to search nodes( many of them, basically BFS) XML file contains data in linear fashion.. now searhing around 30,000 nodes takes long time... can any one suggest which class could i use here to obtain faster results.. i am confused with all the xml classes .NET provides.. as to which class's object would solve my purpose... probably some code snippets supporting your answer would be of great help:) Thanks techno_adi -- modified at 11:33 Thursday 20th April, 2006
Speaking from personal experience, I would suggest moving the data to a database instead of using XML files that are that large. I did a project that started out by having a pure XML backend but as the files grew I ran into the same problem you are having. We ended up using Access for simplicity and later moved to MySQL because it was a free database our clients could run. Just my 2 cents. -Richard
-
Hi, I have a very huge XML file ( around 30 MB in size).. The problem is i need to search nodes( many of them, basically BFS) XML file contains data in linear fashion.. now searhing around 30,000 nodes takes long time... can any one suggest which class could i use here to obtain faster results.. i am confused with all the xml classes .NET provides.. as to which class's object would solve my purpose... probably some code snippets supporting your answer would be of great help:) Thanks techno_adi -- modified at 11:33 Thursday 20th April, 2006
I think you might want to look at vtd-xml it is ideally designed to do what you described..