How to set IMAGE_DLLCHARACTERISTICS_NX_COMPAT bit to make DLL DEP compatible with VC 6.0 compiler

dolly

Hello Friends, I am having a c++ ATL DLL built in VC 6.0 And Vista's DEP blocks it, As I found that we can set the NXCOMPAT bit (IMAGE_DLLCHARACTERISTICS_NX_COMPAT) to mark it DEP compatible so that Vista DEP will allow it to run. by following command: (as post build step) editbin.exe /NXCOMPAT but I want to do the same in VC 6.0, and we dont have editbin command or NXCOMPAT bit in VC 6. It's in Visual Studio 2005 and later versions. Kindly tell me how to do the same as there must be some way out other than this command so that DEP will not block my DLL built in older vc++ 6 compiler. Please Help!!!

dolly, N,IN

Stuart Dootson

Before you think about that - are you using CWindow objects? ATL 3.0 (&7.1, IIRC) relied on constructing code thunks in data memory to perform dispatching of window messages to C++ objects. This is precisely the sort of code that DEP is intended to protect against. If you believe your DLL is OK - what's to stop you using the editbin.exe from VS2005? To make it easier, copy it into your solution's root directory and use it from there.

dolly

Stuart Dootson wrote:

Before you think about that - are you using CWindow objects? ATL 3.0 (&7.1, IIRC) relied on constructing code thunks in data memory to perform dispatching of window messages to C++ objects. This is precisely the sort of code that DEP is intended to protect against. If you believe your DLL is OK - what's to stop you using the editbin.exe from VS2005? To make it easier, copy it into your solution's root directory and use it from there.

------------------------------------------------------------------ Hi and thanks for replying! I am using a simple ATL 3.0 c++ DLL project, and compiling it through Visual C++ 6.0 editbin.exe is not working even if i explicitely copy it in my root folder along with the required DLLs and exes it needs to execute. It says unrecognized option /NXCOMPAT and is ignored! I read some where that VS 2005 onwards every compiled page has IMAGE_DLLCHARACTERISTICS_NX_COMPAT bit set, which makes it DEP compatible, but we dont have such bit setting option in earlier VS versions. That is the main issue with me! But I am sure there is some way of running the old built DLLs so that DEP will not stop them. I am using ATL for the first time, so trying hard to find a way out! Please help if u can! Thanks a lot!

dolly, N,IN

Stephen Hewitt

Here's a program I knocked together which will set the bit for you. It's been tested so I know it works.

// SetNXCompat.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include <windows.h>
#include <tchar.h>
#include <Imagehlp.h>
#include <shlwapi.h>
#include <malloc.h>
#include <iostream>
#include <string>
#pragma comment(lib, "Imagehlp.lib")
#pragma comment(lib, "shlwapi.lib")
using namespace std;

// In case you're using an old SDK that doesn't include the flag.
#ifndef IMAGE_DLLCHARACTERISTICS_NX_COMPAT

define IMAGE_DLLCHARACTERISTICS_NX_COMPAT 0x0100

#endif

// We'll be nice and support both ANSI and UNICODE.
#ifdef UNICODE

define tcout wcout

define tcerr wcerr

typedef wstring tstring;

#else

define tcout cout

define tcerr cerr

typedef string tstring;

#endif

void PrintUsage()
{
tcout << _T("Usage:") << endl;
tcout << _T("\tSetNXCompat <file>") << endl;
}

bool Worker(LPCTSTR pDir, LPCTSTR pFile)
{
// Load the image.
LOADED_IMAGE li;
BOOL ok = MapAndLoad(
const_cast<LPTSTR>(pFile), // PSTR ImageName
const_cast<LPTSTR>(pDir), // PSTR DllPath
&li, // PLOADED_IMAGE LoadedImage
FALSE, // BOOL DotDll
FALSE // BOOL ReadOnly
);
if (!ok)
{
return false;
}

// Set the flag.
IMAGE_NT_HEADERS *pHeaders = li.FileHeader;
pHeaders->OptionalHeader.DllCharacteristics |= IMAGE_DLLCHARACTERISTICS_NX_COMPAT;

// We'll go the extra mile and set the checksum.
DWORD org_sum;
DWORD new_sum;
IMAGE_NT_HEADERS *pNT = CheckSumMappedFile(
li.MappedAddress, // PVOID BaseAddress
li.SizeOfImage, // DWORD FileLength
&org_sum, // PDWORD HeaderSum
&new_sum // PDWORD CheckSum
);
if (pNT != NULL)
{
pNT->OptionalHeader.CheckSum = new_sum;
}
else
{
tcerr << "WARNING: Failed to update checksum." << endl;
}

// Were done!
ok = UnMapAndLoad(&li);

return (ok) ? true : false;
}

int main(int argc, TCHAR* argv[])
{
if (argc != 2)
{
PrintUsage();
return -1;
}

// Get the directory.
tstring dir;
if (PathIsRelative(argv[1]))
{
// Get the current directory.
DWORD len = GetCurrentDirectory(0, NULL);
LPTSTR pBuffer = static_cast<LPTSTR>(_alloca((len+1)*sizeof(TCHAR)));
GetCurrentDirectory(len+1, pBuffer);
dir = pBuffer;
}
else
{

Stuart Dootson

5 for you - answers don't get much more helpful than that :-)

Stephen Hewitt

Thanks.

Steve

dolly

Hi Steve, Thanks so much for your reply! I tried running the code you posted, and it seems to work well too, but still my plugin dll is blocked by Vista's DEP. OR I am not clear with how and when exactly to execute this code of yours!! Actually I build my add-in DLL and then run a "Lanucher.exe" which registers the dll in windows registry! I made a "NXSet.exe" which invokes your "worker()" and I called NXSet.exe from my ATL add-in project's Post-Build Step as below: NXSet.exe "$(TargetPath)" and then my addin DLL gets registered in Windows Registry by custom build step! I also done both the things in custom build step...together...and when I checked it actually was successful in setting the IMAGE_DLLCHARACTERISTICS_NX_COMPAT bit in the DLL. but still DEP block my addin DLL. Is there anything else I need to do, so that Vista DEP does not block my DLL? I am giving the absolute add-in DLL path in the command line argument for executing your "worker()" thru my sample exe. I am doing something wrong or misplaced?? Please tell, as when i run your code, it runs well without any error. Thanks Steve!

dolly, N,IN

modified on Monday, July 21, 2008 12:03 AM

Elmue · modified on Monday, July 21, 2008 12:03 AM

The problem is that you did not read the documentation. Setting the IMAGE_DLLCHARACTERISTICS_NX_COMPAT does NOT "make your DLL DEP compatible". This is your invention only. Already your question is wrong. This flags tells Windows that your DLL IS already DEP compatible. If it is not, setting this flag is wrong. The correct solution is to use SetProcessDEPPolicy() to tell Windows NOT to use DEP on your process. Or compile with linker option /NXCOMPAT:NO

Richard Deeming

It's extremely unlikely that "dolly" is still looking for an answer twelve years later. :doh:

"These people looked deep within my soul and assigned me a number based on the order in which I joined." - Homer

Elmue

I did not write my answer for "dolly". This question has the highest ranking on Google. It appears in the first place! Anybody searching for IMAGE_DLLCHARACTERISTICS_NX_COMPAT on Google will come here.