Pragma pack, pointer, 32 vs 64 bits build
-
Hello folks, I'm encountering a problem. Doing message-passing, I'm sending/receiving buffers from various/mixed architectured machines (32 and 64 bits). I first tried to use this as a buffer skeleton :
typedef struct{ // size, offset 32 bits, offset 64 bits
int nCount; // 4, 0, 0
short nMode; // 2, 4, 4 <- ???
sHue sColor; // 8, 8, 8
char* pTest; // 4, 16, 16
char aData[10]; // 10, 20, 24
}sStack,*psStack; // 28 (real) -> 32 (sizeof 32 bits), 40 (sizeof 64 bits)On a 32 bits platform, if aData is grown to 12, it fits the whole struct size (12 bytes from offset 20 = 32), thus growing it to 13 push the struct's end to 36 (4 bytes alignment). However I *NEED* the struct to be exactly the same size, whenever this code runs on a 32 bits machine or, you might have guessed it, a 64 bits one. So I tried :
#pragma pack(8)
typedef struct{
[...]
}sStack,*psStack;
#pragma pack()Yet it just doesn't works like expected on a 32 bits build. The struct's size is still 32 bytes, whereas I expected at least 36 (pTest pointer using 8 bytes, thus aData lying at offset 24, hence growing the struct to 34 bytes, aligned to 36 bytes). stdcall or cdecl calls should remains set to 32 or 64 bits accordingly to the native platform, that means a ((psStack)buff)->pTest instruction shouldn't need to be casted for suiting 32 or 64 bits build. I just need the pointers in struct to be casted to the size of an __int64 (with the needed 4 padding bytes) ! Any luck someone to have a hint, a trick, or just the full answer ? Thanks anyway... Kochise
In Code we trust !
-
Hello folks, I'm encountering a problem. Doing message-passing, I'm sending/receiving buffers from various/mixed architectured machines (32 and 64 bits). I first tried to use this as a buffer skeleton :
typedef struct{ // size, offset 32 bits, offset 64 bits
int nCount; // 4, 0, 0
short nMode; // 2, 4, 4 <- ???
sHue sColor; // 8, 8, 8
char* pTest; // 4, 16, 16
char aData[10]; // 10, 20, 24
}sStack,*psStack; // 28 (real) -> 32 (sizeof 32 bits), 40 (sizeof 64 bits)On a 32 bits platform, if aData is grown to 12, it fits the whole struct size (12 bytes from offset 20 = 32), thus growing it to 13 push the struct's end to 36 (4 bytes alignment). However I *NEED* the struct to be exactly the same size, whenever this code runs on a 32 bits machine or, you might have guessed it, a 64 bits one. So I tried :
#pragma pack(8)
typedef struct{
[...]
}sStack,*psStack;
#pragma pack()Yet it just doesn't works like expected on a 32 bits build. The struct's size is still 32 bytes, whereas I expected at least 36 (pTest pointer using 8 bytes, thus aData lying at offset 24, hence growing the struct to 34 bytes, aligned to 36 bytes). stdcall or cdecl calls should remains set to 32 or 64 bits accordingly to the native platform, that means a ((psStack)buff)->pTest instruction shouldn't need to be casted for suiting 32 or 64 bits build. I just need the pointers in struct to be casted to the size of an __int64 (with the needed 4 padding bytes) ! Any luck someone to have a hint, a trick, or just the full answer ? Thanks anyway... Kochise
In Code we trust !
Reading elsewhere, seems there is a 'dirty tricky trick' that might do the job (unless verified more thoroughfully) :
typedef struct{ // size, offset 32 bits, offset 64 bits
int nCount; // 4, 0, 0
short nMode; // 2, 4, 4 <- ???
sHue sColor; // 8, 8, 8
char* pTest; // 4, 16, 16
__declspec( align(8))
char aData[10]; // 10, 24, 24
}sStack,*psStack; // 28 (real) -> 40 (sizeof 32 bits), 40 (sizeof 64 bits)Do you understand that ? However that doesn't works in every compiler, I need something more portable, such like a WORKING #pragma pack option... Kochise
In Code we trust !
modified on Friday, July 17, 2009 7:32 AM
-
Hello folks, I'm encountering a problem. Doing message-passing, I'm sending/receiving buffers from various/mixed architectured machines (32 and 64 bits). I first tried to use this as a buffer skeleton :
typedef struct{ // size, offset 32 bits, offset 64 bits
int nCount; // 4, 0, 0
short nMode; // 2, 4, 4 <- ???
sHue sColor; // 8, 8, 8
char* pTest; // 4, 16, 16
char aData[10]; // 10, 20, 24
}sStack,*psStack; // 28 (real) -> 32 (sizeof 32 bits), 40 (sizeof 64 bits)On a 32 bits platform, if aData is grown to 12, it fits the whole struct size (12 bytes from offset 20 = 32), thus growing it to 13 push the struct's end to 36 (4 bytes alignment). However I *NEED* the struct to be exactly the same size, whenever this code runs on a 32 bits machine or, you might have guessed it, a 64 bits one. So I tried :
#pragma pack(8)
typedef struct{
[...]
}sStack,*psStack;
#pragma pack()Yet it just doesn't works like expected on a 32 bits build. The struct's size is still 32 bytes, whereas I expected at least 36 (pTest pointer using 8 bytes, thus aData lying at offset 24, hence growing the struct to 34 bytes, aligned to 36 bytes). stdcall or cdecl calls should remains set to 32 or 64 bits accordingly to the native platform, that means a ((psStack)buff)->pTest instruction shouldn't need to be casted for suiting 32 or 64 bits build. I just need the pointers in struct to be casted to the size of an __int64 (with the needed 4 padding bytes) ! Any luck someone to have a hint, a trick, or just the full answer ? Thanks anyway... Kochise
In Code we trust !
This is the sort of case where I really wish C++ had Ada record representation clauses[^]. Anyway - I suspect #pragma pack(8) doesn't do what you think. Specifically, the docs[^] say:
Specifies the value, in bytes, to be used for packing. The default value for n is 8. Valid values are 1, 2, 4, 8, and 16. The alignment of a member will be on a boundary that is either a multiple of n or a multiple of the size of the member, whichever is smaller.
As a monitor (as opposed to a solution :-(), I'd suggest using a static assert (either Microsoft[^] or Boost[^] or it's easy to build your own[^]) to verify that you and the compiler agree what the size of the structure is...
Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p
-
Reading elsewhere, seems there is a 'dirty tricky trick' that might do the job (unless verified more thoroughfully) :
typedef struct{ // size, offset 32 bits, offset 64 bits
int nCount; // 4, 0, 0
short nMode; // 2, 4, 4 <- ???
sHue sColor; // 8, 8, 8
char* pTest; // 4, 16, 16
__declspec( align(8))
char aData[10]; // 10, 24, 24
}sStack,*psStack; // 28 (real) -> 40 (sizeof 32 bits), 40 (sizeof 64 bits)Do you understand that ? However that doesn't works in every compiler, I need something more portable, such like a WORKING #pragma pack option... Kochise
In Code we trust !
modified on Friday, July 17, 2009 7:32 AM
Hi Kochise, The 32 bit pointer is 4 bytes and the 64 bit pointer is 8 bytes. Why not store the pTest pointer in a 64 bit integer?
typedef struct { int nCount; short nMode; sHue sColor; __int64 pTest; char aData[10]; } sStack,*psStack;
This will be 40 bytes on both 32 and 64 bit platforms. Best Wishes, -David Delaune -
This is the sort of case where I really wish C++ had Ada record representation clauses[^]. Anyway - I suspect #pragma pack(8) doesn't do what you think. Specifically, the docs[^] say:
Specifies the value, in bytes, to be used for packing. The default value for n is 8. Valid values are 1, 2, 4, 8, and 16. The alignment of a member will be on a boundary that is either a multiple of n or a multiple of the size of the member, whichever is smaller.
As a monitor (as opposed to a solution :-(), I'd suggest using a static assert (either Microsoft[^] or Boost[^] or it's easy to build your own[^]) to verify that you and the compiler agree what the size of the structure is...
Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p
Yeah, the root of the problem is that I'm currently writing a generic linked-in driver for Erlang that should adapt to any kind of platform, be it 32 or 64 bits. Imagine the client side to be 64 and the server side to be 32 bits, of course the pointers provided in the structs can only be used locally, yet if the server needs a data chunk from the client side, it just have to request it using its handle (the pointer). That's why both side MUST have the same struct layout, so that the 32 bits server could catch up the valid 64 bits pointer from the raw piece of data extracted from the up-coming message, then send it back in a 'hands-up and gimme your memory' kinda request (handshake). I imagine it would be kind to pack the whole struct in a #pragma pack(1) fashion for message-passing, yet rendering it useless for run-time usage due to alignment fault that could arise. And packing/unpacking structs while they are sent/read is NOT the solution, the best would be to get them correctly shaped from the very beginning, the extra (wasteful) padding bytes included :/ Problems arises as the size but also the alignment of data change between a 32 and 64 bits architechture, what makes things a bit over complicated to my taste : http://docs.hp.com/en/5966-9844/ch03s02.html I would like to avoid requiring the use of an extra library, because one might lead to require another one, leading to a complete mess. I'm a macro freak, and if some well defined macros do the job, I'm pretty for that kind of solution. Kochise
In Code we trust !
modified on Friday, July 17, 2009 11:50 AM
-
Hello folks, I'm encountering a problem. Doing message-passing, I'm sending/receiving buffers from various/mixed architectured machines (32 and 64 bits). I first tried to use this as a buffer skeleton :
typedef struct{ // size, offset 32 bits, offset 64 bits
int nCount; // 4, 0, 0
short nMode; // 2, 4, 4 <- ???
sHue sColor; // 8, 8, 8
char* pTest; // 4, 16, 16
char aData[10]; // 10, 20, 24
}sStack,*psStack; // 28 (real) -> 32 (sizeof 32 bits), 40 (sizeof 64 bits)On a 32 bits platform, if aData is grown to 12, it fits the whole struct size (12 bytes from offset 20 = 32), thus growing it to 13 push the struct's end to 36 (4 bytes alignment). However I *NEED* the struct to be exactly the same size, whenever this code runs on a 32 bits machine or, you might have guessed it, a 64 bits one. So I tried :
#pragma pack(8)
typedef struct{
[...]
}sStack,*psStack;
#pragma pack()Yet it just doesn't works like expected on a 32 bits build. The struct's size is still 32 bytes, whereas I expected at least 36 (pTest pointer using 8 bytes, thus aData lying at offset 24, hence growing the struct to 34 bytes, aligned to 36 bytes). stdcall or cdecl calls should remains set to 32 or 64 bits accordingly to the native platform, that means a ((psStack)buff)->pTest instruction shouldn't need to be casted for suiting 32 or 64 bits build. I just need the pointers in struct to be casted to the size of an __int64 (with the needed 4 padding bytes) ! Any luck someone to have a hint, a trick, or just the full answer ? Thanks anyway... Kochise
In Code we trust !
When sending data between systems you will have two concerns: 1. endianness 2. structure packing We'll assume endianness isn't an issue, or is already handled. The only use for passing a pointer to a system is for the receiver to use it as an opaque handle to pass back to the sender at a later time. As such, Randor gave the correct solution. I would just add that you can make it more code friendly by using an anonymous union:
#pragma pack(push, 8)
typedef struct{
int nCount;
short nMode;
sHue sColor;
union {
char* pTest; // used by sender
__int64 hTest; // used by receiver
};
char aData[10];
} sStack,*psStack;
#pragma pack(pop)Also, always specify the packing for these types of structures.
...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack
-
When sending data between systems you will have two concerns: 1. endianness 2. structure packing We'll assume endianness isn't an issue, or is already handled. The only use for passing a pointer to a system is for the receiver to use it as an opaque handle to pass back to the sender at a later time. As such, Randor gave the correct solution. I would just add that you can make it more code friendly by using an anonymous union:
#pragma pack(push, 8)
typedef struct{
int nCount;
short nMode;
sHue sColor;
union {
char* pTest; // used by sender
__int64 hTest; // used by receiver
};
char aData[10];
} sStack,*psStack;
#pragma pack(pop)Also, always specify the packing for these types of structures.
...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack
Yeah, I'm slownly working toward a solution to the problem, yet some stuff should still find an issue : 1- #pragma pack(8) align on a 8-byte boundary members of size >= 8, it doesn't add padding byte (what is in fact not a problem) 2- #pragma pack(1) align on a 1-byte boundary every members, nice for sending/receiving the struct as a frame. However that show the problem of software alignment fault on FPU members (float, double) and members whose size is greater than the current architechture word 3- #pragma pack(2) might lift up the problem upon most cases, but FPU members still get troubles 4- anonymous union is a nice stuff, yet I not always own the headers. Understand I work with compiled third party libraries and their header file. I work with struct coming out from these header files, and while I need the native format to deal with the library, I'd like to use the struct definition as a skeleton for my message's declaration as well. I 'just' need to enforce that the format of my message will be both 32 and 64 bit compatible (best is packing on 2 with 64 bits pointer).
#include <>
/* ======== foreign_lib.h ========
typedef struct foreign_struct
{ char nMode;
float fValue;
int* pArray;
};foreign_struct* get_struct(void);
=== I HAVE NO CONTROL OVER THIS === */// int send_frame(void* pBuffer, size_t nSize);
int test_struct(foreign_struct* frame)
{
#pragma pack(2)
struct foreign_struct sFrame;
#pragma pack()// foreign_struct : native struct format
// sFrame : my packed struct frameif(send_frame(frame, sizeof(foreign_struct))
== send_frame(&sFrame, sizeof(sFrame)))
return (int) TRUE;
else
return (int) FALSE;
}int main(int argc, char *argv[])
{
return test_struct(get_struct());
}But thanks for you offer, good stuff to know ! Kochise
In Code we trust !
-
Yeah, I'm slownly working toward a solution to the problem, yet some stuff should still find an issue : 1- #pragma pack(8) align on a 8-byte boundary members of size >= 8, it doesn't add padding byte (what is in fact not a problem) 2- #pragma pack(1) align on a 1-byte boundary every members, nice for sending/receiving the struct as a frame. However that show the problem of software alignment fault on FPU members (float, double) and members whose size is greater than the current architechture word 3- #pragma pack(2) might lift up the problem upon most cases, but FPU members still get troubles 4- anonymous union is a nice stuff, yet I not always own the headers. Understand I work with compiled third party libraries and their header file. I work with struct coming out from these header files, and while I need the native format to deal with the library, I'd like to use the struct definition as a skeleton for my message's declaration as well. I 'just' need to enforce that the format of my message will be both 32 and 64 bit compatible (best is packing on 2 with 64 bits pointer).
#include <>
/* ======== foreign_lib.h ========
typedef struct foreign_struct
{ char nMode;
float fValue;
int* pArray;
};foreign_struct* get_struct(void);
=== I HAVE NO CONTROL OVER THIS === */// int send_frame(void* pBuffer, size_t nSize);
int test_struct(foreign_struct* frame)
{
#pragma pack(2)
struct foreign_struct sFrame;
#pragma pack()// foreign_struct : native struct format
// sFrame : my packed struct frameif(send_frame(frame, sizeof(foreign_struct))
== send_frame(&sFrame, sizeof(sFrame)))
return (int) TRUE;
else
return (int) FALSE;
}int main(int argc, char *argv[])
{
return test_struct(get_struct());
}But thanks for you offer, good stuff to know ! Kochise
In Code we trust !
For your case, you will have to create your own copy of the struct that uses a format similar to that Randor or I suggested. There is no way you can fix this by packing. The struct contains a pointer; a 32bit system will see this as 4 bytes, a 64bit 8 bytes. This is the crux of your problem, you can't do anything about it. Also, you may want to read http://en.wikipedia.org/wiki/Data_structure_alignment[^] ... in particular the bottom part on Default packing and #pragma pack. [EDIT] Doesn't have to do with your problem, but usefull to know. [/EDIT] Good luck.
...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack
-
Yeah, the root of the problem is that I'm currently writing a generic linked-in driver for Erlang that should adapt to any kind of platform, be it 32 or 64 bits. Imagine the client side to be 64 and the server side to be 32 bits, of course the pointers provided in the structs can only be used locally, yet if the server needs a data chunk from the client side, it just have to request it using its handle (the pointer). That's why both side MUST have the same struct layout, so that the 32 bits server could catch up the valid 64 bits pointer from the raw piece of data extracted from the up-coming message, then send it back in a 'hands-up and gimme your memory' kinda request (handshake). I imagine it would be kind to pack the whole struct in a #pragma pack(1) fashion for message-passing, yet rendering it useless for run-time usage due to alignment fault that could arise. And packing/unpacking structs while they are sent/read is NOT the solution, the best would be to get them correctly shaped from the very beginning, the extra (wasteful) padding bytes included :/ Problems arises as the size but also the alignment of data change between a 32 and 64 bits architechture, what makes things a bit over complicated to my taste : http://docs.hp.com/en/5966-9844/ch03s02.html I would like to avoid requiring the use of an extra library, because one might lead to require another one, leading to a complete mess. I'm a macro freak, and if some well defined macros do the job, I'm pretty for that kind of solution. Kochise
In Code we trust !
modified on Friday, July 17, 2009 11:50 AM
Kochise wrote:
I imagine it would be kind to pack the whole struct in a #pragma pack(1) fashion for message-passing, yet rendering it useless for run-time usage due to alignment fault that could arise
IIRC, x86 doesn't actually care about alignment - it's more for performance issues. Certainly this little program runs on x86 and x64, using non-word alignment:
int main()
{
char x[] = {1,2,3,4,5,6,7,8};
int* pp = (int*)(x+1);
std::cout << std::hex << (*pp) << std::endl;
}
Alternatively - here's another thought - bit-fields:
typedef struct{ // size, offset 32 bits, offset 64 bits
int nCount : 32; // 4, 0, 0
int nMode : 16; // 2, 4, 4 <- ???
int sColor : 8; // 8, 8, 8
int pTest : 32; // 4, 16, 16
int aData0_3 : 32; // 10, 20, 24
int aData4_7 : 32; // 10, 20, 24
int aData8_9 : 16; // 10, 20, 24}sStack,*psStack;
The array's been reconfigured slightly, but you could work around that.
Another option - serialize the struct to a byte-stream when sending, deserialize on the way back.
sStack Deserialize(BYTE* pBuffer, size_t nBytes)
{
_ASSERTE(nBytes >= 4 + 2 + 1 + 8 + 4 + 10);
sStack s;
s.nCount = *(int*)pBuffer;
// etc etc
return s;
}Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p