A story about code-frog, shog, Surfulater and DEP
-
Juts out of curiousity: how can you have access to the whole 4 gigs? I thought it was a requirement that the OS keep aside 1 Gig (or 2 gig in the more traditional layout) just for itself? Is this not the case? If it's not, how do you prevent userland memory from completely clobbering kernel mem? ¡El diablo está en mis pantalones! ¡Mire, mire! Real Mentats use only 100% pure, unfooled around with Sapho Juice(tm)! SELECT * FROM User WHERE Clue > 0 0 rows returned Save an Orange - Use the VCF!
PAE (Physical Address Extension) allows the CPU (and hence the OS) to access memory above the 4 Gig limit (imposed by the 32 address pins in the processor). Processors that support PAE have 36 address pins and therefore can access 2^36=64 Gigs of RAM. PAE has no effect on the Virtual Address space that Windows allocates to each process, it still remains 4 GB, with 2 GB taken up by the OS, unless you turn on /3GB in boot.ini, in which case the OS share reduces to ~1.2 GB. Regards Senthil _____________________________ My Blog | My Articles | WinMacro
-
PAE (Physical Address Extension) allows the CPU (and hence the OS) to access memory above the 4 Gig limit (imposed by the 32 address pins in the processor). Processors that support PAE have 36 address pins and therefore can access 2^36=64 Gigs of RAM. PAE has no effect on the Virtual Address space that Windows allocates to each process, it still remains 4 GB, with 2 GB taken up by the OS, unless you turn on /3GB in boot.ini, in which case the OS share reduces to ~1.2 GB. Regards Senthil _____________________________ My Blog | My Articles | WinMacro
Reducing the OS down to around 1GB can be hazardous. I had a customer try it on a system with a huge amount of stuff running as services in kernel mode and that thing was paging so hard they couldn't even open notepad to fix their boot.ini file and restart. They had to copy the boot.ini file to the machine to undo the /3GB switch. The reason they tried this in the first place is that they process some enormous data files and our app occasionally wants to use over 2GB and vomits when it tries.
-
Reducing the OS down to around 1GB can be hazardous. I had a customer try it on a system with a huge amount of stuff running as services in kernel mode and that thing was paging so hard they couldn't even open notepad to fix their boot.ini file and restart. They had to copy the boot.ini file to the machine to undo the /3GB switch. The reason they tried this in the first place is that they process some enormous data files and our app occasionally wants to use over 2GB and vomits when it tries.
Unless your app was linked with /LARGEADDRESSAWARE, it wouldn't have helped anyway - your application would still be limited to a 2GB address space, the space between 2GB and 3GB going unused. Stability. What an interesting concept. -- Chris Maunder
-
Long story short. Had a serious problem that code-frog (Rex) reported earlier this month where Surfulater would crash. Follows on from a report of the same problem back in Dec 2005 that I never resolved. After fantastic help from Rex he worked out that DEP (Data Execution Prevention) was causing the crash. After I enabled Hardware DEP I was finally able to reproduce the crash and track it down to some code that shog wrote back in Dec 2001 (see: http://www.codeproject.com/miscctrl/balloonhelp.as[^]) I use a sub-set of this code. Fortunately others had already found and fixed the problem and I'm most grateful to KHDev4u for his solution. But the real hero of the day is Rex, who I can't thank enough.:rose: And of course shog for his great code as well.:rose: I've written a long version of this here http://blog.surfulater.com/2006/05/22/data-execution-protection-rex-winn/[^] And the moral of this story is make sure you enable Hardware and Software DEP and test your code thoroughly in this environment. Neville Franks, Author of Surfulater www.surfulater.com "Save what you Surf" and ED for Windows www.getsoft.com
Another difference between XP SP2 and Server 2003 SP1 is that, IIRC, XP defaults to DEP 'Opt In' while Server 2003 defaults to 'Opt Out'. This setting is in the System control panel applet, Advanced tab, click Performance, the Data Execution Prevention tab. The options are "Turn on DEP for essential Windows programs and services only" (i.e. "Opt In") or "Turn on DEP for all programs and services except those I specify" (Opt Out), with a box to specify programs to exclude. This may be why you couldn't replicate it even after turning Hardware DEP on. It's still possible to execute dynamically-generated code, but you must tell the OS which regions you plan to do this from. For memory allocated with
VirtualAlloc, either supplyPAGE_EXECUTE_READWRITEat the point of allocation, or useVirtualProtectto applyPAGE_EXECUTEsubsequently. For heap memory, specify theHEAP_CREATE_ENABLE_EXECUTEflag when callingHeapCreate. It's not possible to specify this after creation, and the process default heap does not allow it IIRC. Stability. What an interesting concept. -- Chris Maunder -
Another difference between XP SP2 and Server 2003 SP1 is that, IIRC, XP defaults to DEP 'Opt In' while Server 2003 defaults to 'Opt Out'. This setting is in the System control panel applet, Advanced tab, click Performance, the Data Execution Prevention tab. The options are "Turn on DEP for essential Windows programs and services only" (i.e. "Opt In") or "Turn on DEP for all programs and services except those I specify" (Opt Out), with a box to specify programs to exclude. This may be why you couldn't replicate it even after turning Hardware DEP on. It's still possible to execute dynamically-generated code, but you must tell the OS which regions you plan to do this from. For memory allocated with
VirtualAlloc, either supplyPAGE_EXECUTE_READWRITEat the point of allocation, or useVirtualProtectto applyPAGE_EXECUTEsubsequently. For heap memory, specify theHEAP_CREATE_ENABLE_EXECUTEflag when callingHeapCreate. It's not possible to specify this after creation, and the process default heap does not allow it IIRC. Stability. What an interesting concept. -- Chris MaunderWhen Rex and I were going back and forth on this he got to a point where turning off Hardware DEP made no difference. ie. Surfulater still wouldn't start. There is a post in shog's article suggesting a solution using VirtualProtect() or similar, but other solutions including the one I adopted replaced the dynamically generated code (thunk), all together, which seems to me the best approach. It will be interesting to watch programs stop working as over time DEP becomes more widely used. Neville Franks, Author of Surfulater www.surfulater.com "Save what you Surf" and ED for Windows www.getsoft.com
-
When Rex and I were going back and forth on this he got to a point where turning off Hardware DEP made no difference. ie. Surfulater still wouldn't start. There is a post in shog's article suggesting a solution using VirtualProtect() or similar, but other solutions including the one I adopted replaced the dynamically generated code (thunk), all together, which seems to me the best approach. It will be interesting to watch programs stop working as over time DEP becomes more widely used. Neville Franks, Author of Surfulater www.surfulater.com "Save what you Surf" and ED for Windows www.getsoft.com
Programs using ATL windowing may well not work, because ATL uses dynamically-generated thunks to map from the static WindowProc implementation to the class's own WndProc (basically to work out what the
thispointer should be). In ATL 3.0 (which came with VS 6.0) this is a member of theCWindowImplRootclass template, so you'd have to allocate your window objects on an appropriately-configured heap to avoid DEP problems. ATL 7.1 (VS.NET 2003) has new code which allocates these thunks from the process-default heap, regardless of where the window object is allocated. ATL 8.0 has really wacky code used to work out if NX is enabled and VirtualAllocs special space for the thunks if it is. Stability. What an interesting concept. -- Chris Maunder -
Programs using ATL windowing may well not work, because ATL uses dynamically-generated thunks to map from the static WindowProc implementation to the class's own WndProc (basically to work out what the
thispointer should be). In ATL 3.0 (which came with VS 6.0) this is a member of theCWindowImplRootclass template, so you'd have to allocate your window objects on an appropriately-configured heap to avoid DEP problems. ATL 7.1 (VS.NET 2003) has new code which allocates these thunks from the process-default heap, regardless of where the window object is allocated. ATL 8.0 has really wacky code used to work out if NX is enabled and VirtualAllocs special space for the thunks if it is. Stability. What an interesting concept. -- Chris MaunderThanks for that Mike. Makes me glad I'm not using ATL.:) Neville Franks, Author of Surfulater www.surfulater.com "Save what you Surf" and ED for Windows www.getsoft.com
-
Maybe it is. ;) I've an update to the WTL version incorporating the DEP patch and theme support if you want it...:cool: Anna :rose: Currently working mostly on: Visual Lint :cool: Anna's Place | Tears and Laughter "Be yourself - not what others think you should be" - Marcia Graesch "Anna's just a sexy-looking lesbian tart" - A friend, trying to wind me up. It didn't work.
-
Maybe it is. ;) I've an update to the WTL version incorporating the DEP patch and theme support if you want it...:cool: Anna :rose: Currently working mostly on: Visual Lint :cool: Anna's Place | Tears and Laughter "Be yourself - not what others think you should be" - Marcia Graesch "Anna's just a sexy-looking lesbian tart" - A friend, trying to wind me up. It didn't work.
Sure, send it on. :)
---- Scripts i’ve known... CPhog 1.0.0.0 - make CP better. Forum Bookmark 0.2.5 - bookmark forum posts on Pensieve Print forum 0.1.2 - printer-friendly forums Expand all 1.0 - Expand all messages In-place Delete 1.0 - AJAX-style post delete Syntax 0.1 - Syntax highlighting for code blocks in the forums
-
Sure, send it on. :)
---- Scripts i’ve known... CPhog 1.0.0.0 - make CP better. Forum Bookmark 0.2.5 - bookmark forum posts on Pensieve Print forum 0.1.2 - printer-friendly forums Expand all 1.0 - Expand all messages In-place Delete 1.0 - AJAX-style post delete Syntax 0.1 - Syntax highlighting for code blocks in the forums
You should have it later today then. :) Anna :rose: Currently working mostly on: Visual Lint :cool: Anna's Place | Tears and Laughter "Be yourself - not what others think you should be" - Marcia Graesch "Anna's just a sexy-looking lesbian tart" - A friend, trying to wind me up. It didn't work.