How to de-batch very large XMLDocument into bit size chunks

[ONeil Tomlinson]

I have a very large XMLDocument with 56,000 child records each having 8 child field elements. The application that i have written performs a complicated process on the XML. It can process 10,000 records in 5 minutes and 20,000 in 25 minutes (increases exponentially) but takes maybe 10 hours to process the 56,000 record (which is unacceptable). Whats the best way to de-batch this XML Document into bite size chunks (eg 10,000 and after 5 iteration will process 50,000. which will execute in 25-30 minutes processing time) Thanks.

[daveyerwin]

How many megabytes is your xml file? Does the "complicated process" operate on only one of the 56,000 children independantly, without refferencing the other children ? Show me a childNode and tell me about this "complicated process". How much free memory does your computer have available ?

[ONeil Tomlinson]

There is a sample of the child record. And there are 56,000 of them in the xml document. I can realy give you the full details of what the complicated process is doing but part of it is using the Muenchian Method grouping on 'cost_centre' and 'expense_code'. The app is runningon a dev box 2.50GH, 2GB. It has 1/2GB free when app is not running

<Transaction>
<Entry>
<payroll_id>payroll_id_0</payroll_id>
<payroll_nm>payroll_nm_0</payroll_nm>
<payRunDate>payRunDate_0</payRunDate>
<payRunID>payRunID_0</payRunID>
<empno>10</empno>
<surname>surname_0</surname>
<forename>forename_0</forename>
<pos_ref>pos_ref_0</pos_ref>
<pos_name>pos_name_0</pos_name>
<element_id>element_id_0</element_id>
<element_name>element_name_0</element_name>
<cost_centre>c11</cost_centre>
<expense_code>e11</expense_code>
<debit_credit>debit_credit_0</debit_credit>
<amount>10</amount>
<signed_amount>10</signed_amount>
</Entry>
<Entry>
....
</Entry>
</Transaction>