Serialization of a datatable using FileMode.Append
-
Hi: I am trying to serialize/deserialize a datatable in chunks (say every 10,000 records of a 250,000 record table) using the filemode.append functionality of the binary formatter stream object as follows: dt.RemotingFormat = SerializationFormat.Binary; BinaryFormatter formatter = new BinaryFormatter(); Stream output = new FileStream(path, FileMode.Append, FileAccess.Write, FileShare.Write)) formatter.Serialize(output, dt); This serializes the file to disk just fine. I can watch the size of the resulting disk file increase with each chunk added. When I attempt to deserialize the file, I only get the first chunk (1st 10,000 records) of data from the datatable and the rest is apparently ignored. Looking at the file in a hex editor, I can see header records for each appended set of records. I believe that the multiple headers are the problem. I know I have to be overlooking something simple. If you have seen this or solved this, please tell me what I need to do. Thanks for your time. John
-
Hi: I am trying to serialize/deserialize a datatable in chunks (say every 10,000 records of a 250,000 record table) using the filemode.append functionality of the binary formatter stream object as follows: dt.RemotingFormat = SerializationFormat.Binary; BinaryFormatter formatter = new BinaryFormatter(); Stream output = new FileStream(path, FileMode.Append, FileAccess.Write, FileShare.Write)) formatter.Serialize(output, dt); This serializes the file to disk just fine. I can watch the size of the resulting disk file increase with each chunk added. When I attempt to deserialize the file, I only get the first chunk (1st 10,000 records) of data from the datatable and the rest is apparently ignored. Looking at the file in a hex editor, I can see header records for each appended set of records. I believe that the multiple headers are the problem. I know I have to be overlooking something simple. If you have seen this or solved this, please tell me what I need to do. Thanks for your time. John
Hi, I have also experienced this. The problem lies in the deserialization of the DataTable. Somehow when deserializing a DataTable it somehow reads some more bytes from the next object in the stream thus destroying the next deserialization step. This not only occurs when using several DataTables but also when you just append some other information after a DataTable. One workaround would be to save your tables in different files. The other possibility would be to add some more custom information to the stream: Saving: 1. Serialize the table into a MemoryStream. 2. Append the length of the MemoryStream (a simple integer should be alright) to your FileStream. 3. Append the contents of the MemoryStream to the FileStream. Repeat 1-3 for every DataTable. The resulting file should logically look somehow likes this: ... Loading: 1. Open the FileStream. 2. Deserialize an integer giving you the length of the next DataTable. 3. Deserialize the DataTable. 4. Adjust the Position property of the FileStream to match the length you read at point 2. Repeat 2-4 for every DataTable (or just until the FileStream is empty). Robert
-
Hi, I have also experienced this. The problem lies in the deserialization of the DataTable. Somehow when deserializing a DataTable it somehow reads some more bytes from the next object in the stream thus destroying the next deserialization step. This not only occurs when using several DataTables but also when you just append some other information after a DataTable. One workaround would be to save your tables in different files. The other possibility would be to add some more custom information to the stream: Saving: 1. Serialize the table into a MemoryStream. 2. Append the length of the MemoryStream (a simple integer should be alright) to your FileStream. 3. Append the contents of the MemoryStream to the FileStream. Repeat 1-3 for every DataTable. The resulting file should logically look somehow likes this: ... Loading: 1. Open the FileStream. 2. Deserialize an integer giving you the length of the next DataTable. 3. Deserialize the DataTable. 4. Adjust the Position property of the FileStream to match the length you read at point 2. Repeat 2-4 for every DataTable (or just until the FileStream is empty). Robert
Thanks for the help. Your suggestion did the trick. Didn't need to mess with the position as it was where it needed to be for the next deserialization. Thanks again.