Need fast method to read/convert floats from disk
-
I'm reading a bunch of raw numerical data (floats representing verticies) from disk in one endian format, and need to convert it to another (flip the bytes around). Currently I am reading the numbers one by one and bit-shifting them to the correct format, then placing them into their respective places in an array of verticies (float[3]'s). I am trying to speed up this whole operation, and may be able to do so with a better method/algorithm. The following also may help: - Each float is 4 bytes and needs a pure reversal of bytes - not bits (bytes 1234 become 4321) - the floats are contiguous on disk - I know ahead of time how many floats I will be loading into a given array - The array will be pre-allocated to hold the right number of verticies - I must keep (or end up with) the array two-dimensional (array of float[3]'s) for later OpenGL usage not sure if reading them all in at once, and applying a macro-bit-shift would be faster? Thanks for any help
-
I'm reading a bunch of raw numerical data (floats representing verticies) from disk in one endian format, and need to convert it to another (flip the bytes around). Currently I am reading the numbers one by one and bit-shifting them to the correct format, then placing them into their respective places in an array of verticies (float[3]'s). I am trying to speed up this whole operation, and may be able to do so with a better method/algorithm. The following also may help: - Each float is 4 bytes and needs a pure reversal of bytes - not bits (bytes 1234 become 4321) - the floats are contiguous on disk - I know ahead of time how many floats I will be loading into a given array - The array will be pre-allocated to hold the right number of verticies - I must keep (or end up with) the array two-dimensional (array of float[3]'s) for later OpenGL usage not sure if reading them all in at once, and applying a macro-bit-shift would be faster? Thanks for any help
nadiric wrote:
not sure if reading them all in at once, and applying a macro-bit-shift would be faster?
Yes. Reading them in one at a time increases disk access (which is expensive). Reading them in as a block decreases the number of reads required to get your data. Something along the lines of the following should work:
// for converting float float endian_shift(const float& f) { float ret = 0.0; ret |= (f << 24); ret |= ((f >> 8) << 24) >> 8; ret |= ((f >> 16) << 24) >> 16; ret |= (f >> 24); return ret; } // for reading data ifstream fin; fin.open("myfile.dat", ios::binary); vector<float> myData(SIZE); copy(istream_iterator<float>(fin), istream_iterator<float>(), back_inserter(myData)); fin.close(); transform(myData.begin(), myData.end(), myData.begin(), endian_shift);If you decide to become a software engineer, you are signing up to have a 1/2" piece of silicon tell you exactly how stupid you really are for 8 hours a day, 5 days a week Zac
-
nadiric wrote:
not sure if reading them all in at once, and applying a macro-bit-shift would be faster?
Yes. Reading them in one at a time increases disk access (which is expensive). Reading them in as a block decreases the number of reads required to get your data. Something along the lines of the following should work:
// for converting float float endian_shift(const float& f) { float ret = 0.0; ret |= (f << 24); ret |= ((f >> 8) << 24) >> 8; ret |= ((f >> 16) << 24) >> 16; ret |= (f >> 24); return ret; } // for reading data ifstream fin; fin.open("myfile.dat", ios::binary); vector<float> myData(SIZE); copy(istream_iterator<float>(fin), istream_iterator<float>(), back_inserter(myData)); fin.close(); transform(myData.begin(), myData.end(), myData.begin(), endian_shift);If you decide to become a software engineer, you are signing up to have a 1/2" piece of silicon tell you exactly how stupid you really are for 8 hours a day, 5 days a week Zac
Thanks for the reply, Just tossing a few other thoughts around... Given the following: - vertex is of type float[3] - verticies is an array of SIZE vertex (verticies = new vertex[SIZE];) - vertsfromfile is an array of (SIZE*3) floats are the floats in "verticies" aligned one after the other in this fashion: verticies[0][0]verticies[0][1]verticies[0][2]verticies[1][0]verticies[1][1]...and so on to allow for something like memcpy(verticies,vertsfromfile,(sizeof(vertex)*SIZE)) ? or is this just crazy talk? Thanks
-
Thanks for the reply, Just tossing a few other thoughts around... Given the following: - vertex is of type float[3] - verticies is an array of SIZE vertex (verticies = new vertex[SIZE];) - vertsfromfile is an array of (SIZE*3) floats are the floats in "verticies" aligned one after the other in this fashion: verticies[0][0]verticies[0][1]verticies[0][2]verticies[1][0]verticies[1][1]...and so on to allow for something like memcpy(verticies,vertsfromfile,(sizeof(vertex)*SIZE)) ? or is this just crazy talk? Thanks
If the file is written out such that each vertex is written as float1float2float3, then writing out a series of vertices would just mean you would have vertex1vertex2vertex3... where each vertex matched the pervious pattern. Thus, you could read them in in such a fashion (though, I would still stick with using the copy algorithm).
If you decide to become a software engineer, you are signing up to have a 1/2" piece of silicon tell you exactly how stupid you really are for 8 hours a day, 5 days a week Zac