Fast serialization
-
Hi, I need very fast serialization to disk of a large object (64 MB/128 MB etc). Serializing members one by one is not an option (Boost serialization for example is very slow), and the object does need to be shared across systems so I don't need to worry about endianness. Here's what I came up with:
void saveVoxels(Voxels * voxelData, const char * fileName) throw(...)
{
using namespace std;
ofstream file (fileName, ios::out | ios::binary);
if (file.is_open())
{
file.write((char *)voxelData, sizeof(Voxels));
unsigned int size = voxelData->m_sliceWidth * voxelData->m_sliceHeight * voxelData->m_numSlices * sizeof(unsigned short);
file.write((char *)voxelData->m_pTexture, size);
file.close();
}
}
Voxels* loadVoxels(const char * fileName) throw(...)
{
using namespace std;
ifstream file(fileName, ios::in | ios::binary | ios::ate);
if (file.is_open())
{
unsigned long size = file.tellg();
char * memblock = new char [size];
file.seekg (0, ios::beg);
file.read (memblock, size);
file.close();//initialize Voxels \* voxelData = (Voxels \*) memblock; voxelData->m\_pTexture = (unsigned short \*) (memblock + sizeof(Voxels)); return voxelData; }}
Which as far as I can tell works. However when deleting the object I get _BLOCK_TYPE_IS_VALID(pHead->nBlockUse) assertion error. I guess I didn't really allocate the m_pTexture pointer which is what gets deleted...Any solutions?
-
Hi, I need very fast serialization to disk of a large object (64 MB/128 MB etc). Serializing members one by one is not an option (Boost serialization for example is very slow), and the object does need to be shared across systems so I don't need to worry about endianness. Here's what I came up with:
void saveVoxels(Voxels * voxelData, const char * fileName) throw(...)
{
using namespace std;
ofstream file (fileName, ios::out | ios::binary);
if (file.is_open())
{
file.write((char *)voxelData, sizeof(Voxels));
unsigned int size = voxelData->m_sliceWidth * voxelData->m_sliceHeight * voxelData->m_numSlices * sizeof(unsigned short);
file.write((char *)voxelData->m_pTexture, size);
file.close();
}
}
Voxels* loadVoxels(const char * fileName) throw(...)
{
using namespace std;
ifstream file(fileName, ios::in | ios::binary | ios::ate);
if (file.is_open())
{
unsigned long size = file.tellg();
char * memblock = new char [size];
file.seekg (0, ios::beg);
file.read (memblock, size);
file.close();//initialize Voxels \* voxelData = (Voxels \*) memblock; voxelData->m\_pTexture = (unsigned short \*) (memblock + sizeof(Voxels)); return voxelData; }}
Which as far as I can tell works. However when deleting the object I get _BLOCK_TYPE_IS_VALID(pHead->nBlockUse) assertion error. I guess I didn't really allocate the m_pTexture pointer which is what gets deleted...Any solutions?
-
Hi, I need very fast serialization to disk of a large object (64 MB/128 MB etc). Serializing members one by one is not an option (Boost serialization for example is very slow), and the object does need to be shared across systems so I don't need to worry about endianness. Here's what I came up with:
void saveVoxels(Voxels * voxelData, const char * fileName) throw(...)
{
using namespace std;
ofstream file (fileName, ios::out | ios::binary);
if (file.is_open())
{
file.write((char *)voxelData, sizeof(Voxels));
unsigned int size = voxelData->m_sliceWidth * voxelData->m_sliceHeight * voxelData->m_numSlices * sizeof(unsigned short);
file.write((char *)voxelData->m_pTexture, size);
file.close();
}
}
Voxels* loadVoxels(const char * fileName) throw(...)
{
using namespace std;
ifstream file(fileName, ios::in | ios::binary | ios::ate);
if (file.is_open())
{
unsigned long size = file.tellg();
char * memblock = new char [size];
file.seekg (0, ios::beg);
file.read (memblock, size);
file.close();//initialize Voxels \* voxelData = (Voxels \*) memblock; voxelData->m\_pTexture = (unsigned short \*) (memblock + sizeof(Voxels)); return voxelData; }}
Which as far as I can tell works. However when deleting the object I get _BLOCK_TYPE_IS_VALID(pHead->nBlockUse) assertion error. I guess I didn't really allocate the m_pTexture pointer which is what gets deleted...Any solutions?
Budric B. wrote:
However when deleting the object I get _BLOCK_TYPE_IS_VALID(pHead->nBlockUse) assertion error.
Does the pointer ever get moved? In other words, if you did something like:
char *p = new char[5];
p++;
delete [] p;the result would not be what you'd expect because a different address is being freed.
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
-
Budric B. wrote:
However when deleting the object I get _BLOCK_TYPE_IS_VALID(pHead->nBlockUse) assertion error.
Does the pointer ever get moved? In other words, if you did something like:
char *p = new char[5];
p++;
delete [] p;the result would not be what you'd expect because a different address is being freed.
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
-
No, the pointers does not get moved. Voxels is a class and it gets returned from the loadVoxels() function. The problem is when I call "delete Voxels;" the destructor is deleting the member array and that's when the assertion fails.
Budric B. wrote:
The problem is when I call "delete Voxels;"...
Which should be:
delete [] Voxels;
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
-
Budric B. wrote:
The problem is when I call "delete Voxels;"...
Which should be:
delete [] Voxels;
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
-
Hi, I need very fast serialization to disk of a large object (64 MB/128 MB etc). Serializing members one by one is not an option (Boost serialization for example is very slow), and the object does need to be shared across systems so I don't need to worry about endianness. Here's what I came up with:
void saveVoxels(Voxels * voxelData, const char * fileName) throw(...)
{
using namespace std;
ofstream file (fileName, ios::out | ios::binary);
if (file.is_open())
{
file.write((char *)voxelData, sizeof(Voxels));
unsigned int size = voxelData->m_sliceWidth * voxelData->m_sliceHeight * voxelData->m_numSlices * sizeof(unsigned short);
file.write((char *)voxelData->m_pTexture, size);
file.close();
}
}
Voxels* loadVoxels(const char * fileName) throw(...)
{
using namespace std;
ifstream file(fileName, ios::in | ios::binary | ios::ate);
if (file.is_open())
{
unsigned long size = file.tellg();
char * memblock = new char [size];
file.seekg (0, ios::beg);
file.read (memblock, size);
file.close();//initialize Voxels \* voxelData = (Voxels \*) memblock; voxelData->m\_pTexture = (unsigned short \*) (memblock + sizeof(Voxels)); return voxelData; }}
Which as far as I can tell works. However when deleting the object I get _BLOCK_TYPE_IS_VALID(pHead->nBlockUse) assertion error. I guess I didn't really allocate the m_pTexture pointer which is what gets deleted...Any solutions?
I'm probably wrong, but at first glabce something seems fishy here.... When you write you do this
file.write((char *)voxelData->m_pTexture, size);
When you read you do this Voxels * voxelData = (Voxels *) memblock;
voxelData->m_pTexture = (unsigned short *) (memblock + sizeof(Voxels));
Is there a discrepancy there? MarkMark Salsbery Microsoft MVP - Visual C++ :java:
-
Hi, I need very fast serialization to disk of a large object (64 MB/128 MB etc). Serializing members one by one is not an option (Boost serialization for example is very slow), and the object does need to be shared across systems so I don't need to worry about endianness. Here's what I came up with:
void saveVoxels(Voxels * voxelData, const char * fileName) throw(...)
{
using namespace std;
ofstream file (fileName, ios::out | ios::binary);
if (file.is_open())
{
file.write((char *)voxelData, sizeof(Voxels));
unsigned int size = voxelData->m_sliceWidth * voxelData->m_sliceHeight * voxelData->m_numSlices * sizeof(unsigned short);
file.write((char *)voxelData->m_pTexture, size);
file.close();
}
}
Voxels* loadVoxels(const char * fileName) throw(...)
{
using namespace std;
ifstream file(fileName, ios::in | ios::binary | ios::ate);
if (file.is_open())
{
unsigned long size = file.tellg();
char * memblock = new char [size];
file.seekg (0, ios::beg);
file.read (memblock, size);
file.close();//initialize Voxels \* voxelData = (Voxels \*) memblock; voxelData->m\_pTexture = (unsigned short \*) (memblock + sizeof(Voxels)); return voxelData; }}
Which as far as I can tell works. However when deleting the object I get _BLOCK_TYPE_IS_VALID(pHead->nBlockUse) assertion error. I guess I didn't really allocate the m_pTexture pointer which is what gets deleted...Any solutions?
-
Hi, I need very fast serialization to disk of a large object (64 MB/128 MB etc). Serializing members one by one is not an option (Boost serialization for example is very slow), and the object does need to be shared across systems so I don't need to worry about endianness. Here's what I came up with:
void saveVoxels(Voxels * voxelData, const char * fileName) throw(...)
{
using namespace std;
ofstream file (fileName, ios::out | ios::binary);
if (file.is_open())
{
file.write((char *)voxelData, sizeof(Voxels));
unsigned int size = voxelData->m_sliceWidth * voxelData->m_sliceHeight * voxelData->m_numSlices * sizeof(unsigned short);
file.write((char *)voxelData->m_pTexture, size);
file.close();
}
}
Voxels* loadVoxels(const char * fileName) throw(...)
{
using namespace std;
ifstream file(fileName, ios::in | ios::binary | ios::ate);
if (file.is_open())
{
unsigned long size = file.tellg();
char * memblock = new char [size];
file.seekg (0, ios::beg);
file.read (memblock, size);
file.close();//initialize Voxels \* voxelData = (Voxels \*) memblock; voxelData->m\_pTexture = (unsigned short \*) (memblock + sizeof(Voxels)); return voxelData; }}
Which as far as I can tell works. However when deleting the object I get _BLOCK_TYPE_IS_VALID(pHead->nBlockUse) assertion error. I guess I didn't really allocate the m_pTexture pointer which is what gets deleted...Any solutions?
I get the impresion that Voxels is an object with actual behavior, and therefore shouldn't be written to a file like that. Things like virtual function table pointers can't be saved and restored that way. Also, it looks like Voxels is a relatively small object with a pointer to a huge array, so you wouldn't lose much performance by saving the simple members of Voxels individually. I would save the simple members of Voxels first and then save the array with one file.write call. To restore it, I'd create a Voxels object, read the simple members, and then allocate and read the array. Also, I noticed you are using unsigned int for the size to write to the file. If you use size_t, the type will be adjusted properly if you migrate to a 64 bit system. Nathan
-
I see...even though it's an object? I know I'm casting an array to an object, but I would have thought things would behave as they would for any object created with new.
Well, it wouldn't hurt to try. If it doesn't work, you're no worse off.
"A good athlete is the result of a good and worthy opponent." - David Crow
"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne
-
I'm probably wrong, but at first glabce something seems fishy here.... When you write you do this
file.write((char *)voxelData->m_pTexture, size);
When you read you do this Voxels * voxelData = (Voxels *) memblock;
voxelData->m_pTexture = (unsigned short *) (memblock + sizeof(Voxels));
Is there a discrepancy there? MarkMark Salsbery Microsoft MVP - Visual C++ :java:
No, I first write the object, which contains the data members and a pointer at the end. The size of the object is sizeof(Voxels). I then write the array of stuff. When I read, I read everything. I know that the first sizeof(Voxels) bytes is the object including a pointer with a value that's meaningless. I then assign the pointer to point to the data, which just happens to be right after the object (because that's where I read it into).
-
I get the impresion that Voxels is an object with actual behavior, and therefore shouldn't be written to a file like that. Things like virtual function table pointers can't be saved and restored that way. Also, it looks like Voxels is a relatively small object with a pointer to a huge array, so you wouldn't lose much performance by saving the simple members of Voxels individually. I would save the simple members of Voxels first and then save the array with one file.write call. To restore it, I'd create a Voxels object, read the simple members, and then allocate and read the array. Also, I noticed you are using unsigned int for the size to write to the file. If you use size_t, the type will be adjusted properly if you migrate to a 64 bit system. Nathan
It doesn't have that much behavior other than a constructor and a destructor to free the memory. The reason I was doing it this way was because I would be adding more data members to describe the data later. But you're probably right that it wouldn't add that much performance loss to write each data member. I'll have to time it and see. I've made a workaround where I allocate 2 buffers, 1 for the object and one for the data, the rest of the code stays the same. That doesn't trigger the assertion. However I don't know if the object's memory gets freed or not since I'm still calling delete Voxels and not delete [] voxels;
-
It doesn't have that much behavior other than a constructor and a destructor to free the memory. The reason I was doing it this way was because I would be adding more data members to describe the data later. But you're probably right that it wouldn't add that much performance loss to write each data member. I'll have to time it and see. I've made a workaround where I allocate 2 buffers, 1 for the object and one for the data, the rest of the code stays the same. That doesn't trigger the assertion. However I don't know if the object's memory gets freed or not since I'm still calling delete Voxels and not delete [] voxels;
Budric B. wrote:
It doesn't have that much behavior other than a constructor and a destructor to free the memory. The reason I was doing it this way was because I would be adding more data members to describe the data later. But you're probably right that it wouldn't add that much performance loss to write each data member. I'll have to time it and see.
Watch out for the copy constructor and assignment operator, since the compiler will generate invalid forms and call them freely.
Budric B. wrote:
I've made a workaround where I allocate 2 buffers, 1 for the object and one for the data, the rest of the code stays the same. That doesn't trigger the assertion. However I don't know if the object's memory gets freed or not since I'm still calling delete Voxels and not delete [] voxels;
I think that should work. The main difference between delete and delete[] is the way destructors are called, which shouldn't be a problem if it's just data. Nathan