C++ Structure layouts - how to determine padding and what not - VS2008 but it's a general question
-
Maybe there is a setting for this or a utility or what not. I've been tracking down a buffer overrun and loss of data for a few days, and I have discovered that a data structure pretty clearly defined is blowing past it's size. I think it's due to silly datatype selection and not understanding padding. I'm working in VS2008 (no choice), and I've yet to find something to help me understand the exact byte layout of a structure. I really need to see the padding being added. The fact that I have a mix of unsigned ints, bools and bytebools can be attributed to ignorance and just poor programming. This seems to me something any IDE should be able to show me, but if you recall, Microsoft just broke help in VS2008. I was thinking the map file, but I'm not seeing it. What I would like to be able to do is to add a watch to the data structure. In the watch window, I want to see the addresses of each item in the structure by expanding it. But I think I'm going to have to do a lot of typing. Appreciate any pointers. Long weekend ahead.
Charlie Gilley <italic>Stuck in a dysfunctional matrix from which I must escape... "Where liberty dwells, there is my country." B. Franklin, 1783 “They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
-
Maybe there is a setting for this or a utility or what not. I've been tracking down a buffer overrun and loss of data for a few days, and I have discovered that a data structure pretty clearly defined is blowing past it's size. I think it's due to silly datatype selection and not understanding padding. I'm working in VS2008 (no choice), and I've yet to find something to help me understand the exact byte layout of a structure. I really need to see the padding being added. The fact that I have a mix of unsigned ints, bools and bytebools can be attributed to ignorance and just poor programming. This seems to me something any IDE should be able to show me, but if you recall, Microsoft just broke help in VS2008. I was thinking the map file, but I'm not seeing it. What I would like to be able to do is to add a watch to the data structure. In the watch window, I want to see the addresses of each item in the structure by expanding it. But I think I'm going to have to do a lot of typing. Appreciate any pointers. Long weekend ahead.
Charlie Gilley <italic>Stuck in a dysfunctional matrix from which I must escape... "Where liberty dwells, there is my country." B. Franklin, 1783 “They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
I'm not sure why you'd be having data overruns, assuming you're sizing any I/O or copying correctly. Even if you have an "anonymous" buffer (e.g. char *), that you're copying your struct from, things should still work correctly with the appropriate sizeof operator in use. I don't know of any tool within VS that will give you the information you need, but you might get some mileage out of [pack pragma | Microsoft Docs](https://docs.microsoft.com/en-us/cpp/preprocessor/pack?view=msvc-160), and or [offsetof Macro | Microsoft Docs](https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/offsetof-macro?view=msvc-160). Either or both won't save you from some typing though. I think you need to be careful with #pragma pack. X86-64 doesn't seem to have a problem accessing unaligned data, but I assume its slower to do so than correctly aligned data. Other architectures though are not as accommodating. For example, the following code works fine on X86-64 in both 64 bit and 32 bit modes (at least using linux), but gets a bus error when running under arm-32
#include #include int main()
{
void *ptr = malloc(1024);
char *tmp = ptr;for(size\_t i = 0; i < 24; i++, ++tmp) { long double \*d = (long double \*)tmp; \*d = i \* 2.0L; printf("tmp = %p, d = %p \*d = %Lf\\n", tmp, d, \*d); } free(ptr); return 0;}
Keep Calm and Carry On
-
I'm not sure why you'd be having data overruns, assuming you're sizing any I/O or copying correctly. Even if you have an "anonymous" buffer (e.g. char *), that you're copying your struct from, things should still work correctly with the appropriate sizeof operator in use. I don't know of any tool within VS that will give you the information you need, but you might get some mileage out of [pack pragma | Microsoft Docs](https://docs.microsoft.com/en-us/cpp/preprocessor/pack?view=msvc-160), and or [offsetof Macro | Microsoft Docs](https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/offsetof-macro?view=msvc-160). Either or both won't save you from some typing though. I think you need to be careful with #pragma pack. X86-64 doesn't seem to have a problem accessing unaligned data, but I assume its slower to do so than correctly aligned data. Other architectures though are not as accommodating. For example, the following code works fine on X86-64 in both 64 bit and 32 bit modes (at least using linux), but gets a bus error when running under arm-32
#include #include int main()
{
void *ptr = malloc(1024);
char *tmp = ptr;for(size\_t i = 0; i < 24; i++, ++tmp) { long double \*d = (long double \*)tmp; \*d = i \* 2.0L; printf("tmp = %p, d = %p \*d = %Lf\\n", tmp, d, \*d); } free(ptr); return 0;}
Keep Calm and Carry On
k5054 wrote:
I think you need to be careful with #pragma pack. X86-64 doesn't seem to have a problem accessing unaligned data, but I assume its slower to do so than correctly aligned data.
A bit, on average, but not due to mere misalignment itself. There is a small penalty for crossing cache line boundaries and a different one (~85 cycles on Haswell, even more on Core2, not so much on AMD processors or Skylake and newer) for crossing 4K boundaries. That makes random unaligned loads and stores a bit slower on average, since some of them will cross those boundaries, but it doesn't make all of them slower and the impact is averaged out. Usually the addresses wouldn't really be random of course, so you could run into some situation where it keeps hitting a bad case, which would amplify the problem. There are various other caveats.. Of course, avoiding unaligned operations avoids the whole issue.
-
Maybe there is a setting for this or a utility or what not. I've been tracking down a buffer overrun and loss of data for a few days, and I have discovered that a data structure pretty clearly defined is blowing past it's size. I think it's due to silly datatype selection and not understanding padding. I'm working in VS2008 (no choice), and I've yet to find something to help me understand the exact byte layout of a structure. I really need to see the padding being added. The fact that I have a mix of unsigned ints, bools and bytebools can be attributed to ignorance and just poor programming. This seems to me something any IDE should be able to show me, but if you recall, Microsoft just broke help in VS2008. I was thinking the map file, but I'm not seeing it. What I would like to be able to do is to add a watch to the data structure. In the watch window, I want to see the addresses of each item in the structure by expanding it. But I think I'm going to have to do a lot of typing. Appreciate any pointers. Long weekend ahead.
Charlie Gilley <italic>Stuck in a dysfunctional matrix from which I must escape... "Where liberty dwells, there is my country." B. Franklin, 1783 “They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
In general all struct items are aligned to word boundaries, so you should be able to draw the layout just by reading the source. However, if you use the #pragma pack directive anywhere then it tends to move things closer together. But without seeing any of your code it's impossible to say more.
-
Maybe there is a setting for this or a utility or what not. I've been tracking down a buffer overrun and loss of data for a few days, and I have discovered that a data structure pretty clearly defined is blowing past it's size. I think it's due to silly datatype selection and not understanding padding. I'm working in VS2008 (no choice), and I've yet to find something to help me understand the exact byte layout of a structure. I really need to see the padding being added. The fact that I have a mix of unsigned ints, bools and bytebools can be attributed to ignorance and just poor programming. This seems to me something any IDE should be able to show me, but if you recall, Microsoft just broke help in VS2008. I was thinking the map file, but I'm not seeing it. What I would like to be able to do is to add a watch to the data structure. In the watch window, I want to see the addresses of each item in the structure by expanding it. But I think I'm going to have to do a lot of typing. Appreciate any pointers. Long weekend ahead.
Charlie Gilley <italic>Stuck in a dysfunctional matrix from which I must escape... "Where liberty dwells, there is my country." B. Franklin, 1783 “They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
good comments all. I've been around the block, so I'm certain that due to the compiler's desire to arrange things in a preferential way. I've just started digging into this code for another issue and blundered into this one. The structure is stored as a blob in the system, so I must be very, very careful in how I touch this structure. Changing any of the alignment options will surely frag stored data. This leads me to realize that all one needs to do is change the packing on the compiler line, and the data will be trashed. Oh, but wait, it gets worse. This code base is built with two different IDEs. It's vulnerable there as well. :doh: It's 8am on a Sunday, and I have a long day ahead of me.
Charlie Gilley <italic>Stuck in a dysfunctional matrix from which I must escape... "Where liberty dwells, there is my country." B. Franklin, 1783 “They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
-
I'm not sure why you'd be having data overruns, assuming you're sizing any I/O or copying correctly. Even if you have an "anonymous" buffer (e.g. char *), that you're copying your struct from, things should still work correctly with the appropriate sizeof operator in use. I don't know of any tool within VS that will give you the information you need, but you might get some mileage out of [pack pragma | Microsoft Docs](https://docs.microsoft.com/en-us/cpp/preprocessor/pack?view=msvc-160), and or [offsetof Macro | Microsoft Docs](https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/offsetof-macro?view=msvc-160). Either or both won't save you from some typing though. I think you need to be careful with #pragma pack. X86-64 doesn't seem to have a problem accessing unaligned data, but I assume its slower to do so than correctly aligned data. Other architectures though are not as accommodating. For example, the following code works fine on X86-64 in both 64 bit and 32 bit modes (at least using linux), but gets a bus error when running under arm-32
#include #include int main()
{
void *ptr = malloc(1024);
char *tmp = ptr;for(size\_t i = 0; i < 24; i++, ++tmp) { long double \*d = (long double \*)tmp; \*d = i \* 2.0L; printf("tmp = %p, d = %p \*d = %Lf\\n", tmp, d, \*d); } free(ptr); return 0;}
Keep Calm and Carry On
Sorry, I got too specific in my comments and did not provide enough detail. The structure in question is unioned with a communications buffer. The code assumes the structure will fit in the buffer, but there are no checks. It just so happens that the structure extends beyond the communications buffer (from the alignment issue I think - hence my question. In any event, the result is that the structure's data is truncated with values lost at the end. Luckily, the last 200 bytes or so are spare, so no harm or foul but lousy coding all the same.
Charlie Gilley <italic>Stuck in a dysfunctional matrix from which I must escape... "Where liberty dwells, there is my country." B. Franklin, 1783 “They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
-
Sorry, I got too specific in my comments and did not provide enough detail. The structure in question is unioned with a communications buffer. The code assumes the structure will fit in the buffer, but there are no checks. It just so happens that the structure extends beyond the communications buffer (from the alignment issue I think - hence my question. In any event, the result is that the structure's data is truncated with values lost at the end. Luckily, the last 200 bytes or so are spare, so no harm or foul but lousy coding all the same.
Charlie Gilley <italic>Stuck in a dysfunctional matrix from which I must escape... "Where liberty dwells, there is my country." B. Franklin, 1783 “They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
If you have a fast data source, eg. a file or RAM, from which you know that the data element will always be available in a single operation, you should probably read the struct directly e.g.
struct S {
int i;
double d;
char str[25];
};S s;
std:ifstream data("data.bin", std::ios:binary);
data.read(reinterpret_cast(&s), sizeof s);If you have a slow data source, from which multiple reads may be needed to read a single data item, you should probably read and update a buffer and then copy that to the struct when you have enough bytes. Using a union for type punning is not really the best solution. There's a good explanation of that on Stack Overflow: c++ - Accessing inactive union member and undefined behavior? - Stack Overflow[^]
Keep Calm and Carry On
-
I'm not sure why you'd be having data overruns, assuming you're sizing any I/O or copying correctly. Even if you have an "anonymous" buffer (e.g. char *), that you're copying your struct from, things should still work correctly with the appropriate sizeof operator in use. I don't know of any tool within VS that will give you the information you need, but you might get some mileage out of [pack pragma | Microsoft Docs](https://docs.microsoft.com/en-us/cpp/preprocessor/pack?view=msvc-160), and or [offsetof Macro | Microsoft Docs](https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/offsetof-macro?view=msvc-160). Either or both won't save you from some typing though. I think you need to be careful with #pragma pack. X86-64 doesn't seem to have a problem accessing unaligned data, but I assume its slower to do so than correctly aligned data. Other architectures though are not as accommodating. For example, the following code works fine on X86-64 in both 64 bit and 32 bit modes (at least using linux), but gets a bus error when running under arm-32
#include #include int main()
{
void *ptr = malloc(1024);
char *tmp = ptr;for(size\_t i = 0; i < 24; i++, ++tmp) { long double \*d = (long double \*)tmp; \*d = i \* 2.0L; printf("tmp = %p, d = %p \*d = %Lf\\n", tmp, d, \*d); } free(ptr); return 0;}
Keep Calm and Carry On
k, see my other comments. it's not an overrun in the traditional sense. Just two unioned structures that are assumed to be size compatible, so the aligned structure extends past the other. sorry for the confusion. thanks
Charlie Gilley <italic>Stuck in a dysfunctional matrix from which I must escape... "Where liberty dwells, there is my country." B. Franklin, 1783 “They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
-
k, see my other comments. it's not an overrun in the traditional sense. Just two unioned structures that are assumed to be size compatible, so the aligned structure extends past the other. sorry for the confusion. thanks
Charlie Gilley <italic>Stuck in a dysfunctional matrix from which I must escape... "Where liberty dwells, there is my country." B. Franklin, 1783 “They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
-
A union of structs will take as much space as is required by the largest member of the union. So the mere fact that it is a union should not cause problems.
:) truth. I know how these things work. In my case, the data was longer than the communications buffer. Since the code one uses the communications buffer to talk to the hardware, the last 30 bytes or so were being dropped. thanks
Charlie Gilley <italic>Stuck in a dysfunctional matrix from which I must escape... "Where liberty dwells, there is my country." B. Franklin, 1783 “They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
-
I'm not sure why you'd be having data overruns, assuming you're sizing any I/O or copying correctly. Even if you have an "anonymous" buffer (e.g. char *), that you're copying your struct from, things should still work correctly with the appropriate sizeof operator in use. I don't know of any tool within VS that will give you the information you need, but you might get some mileage out of [pack pragma | Microsoft Docs](https://docs.microsoft.com/en-us/cpp/preprocessor/pack?view=msvc-160), and or [offsetof Macro | Microsoft Docs](https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/offsetof-macro?view=msvc-160). Either or both won't save you from some typing though. I think you need to be careful with #pragma pack. X86-64 doesn't seem to have a problem accessing unaligned data, but I assume its slower to do so than correctly aligned data. Other architectures though are not as accommodating. For example, the following code works fine on X86-64 in both 64 bit and 32 bit modes (at least using linux), but gets a bus error when running under arm-32
#include #include int main()
{
void *ptr = malloc(1024);
char *tmp = ptr;for(size\_t i = 0; i < 24; i++, ++tmp) { long double \*d = (long double \*)tmp; \*d = i \* 2.0L; printf("tmp = %p, d = %p \*d = %Lf\\n", tmp, d, \*d); } free(ptr); return 0;}
Keep Calm and Carry On
Well, the Packing pragma is very old. A Pragma is an adidional instruction to the compiler,stated in te file to be compiled. In the Packing case, it is about how to align data on Preferred Data Boundaries. A 64 bit CPU is most efficient if all items are stored at addresses, a multiple of 64. Failing to do so, the code will not break down. it will just take (very minutely) longer to execute, but will never break your code. Now, you can Easily enter "Packing Hell"! That occurs when different packing pragma's are used in different components. Unless if something is critical, the best idea is to leave all packing pragments out, unless you have reasons to use them, e.g. Interfacing a new 64 structure with say old 16 or 32 bit data.
Bram van Kampen