IO Completion Ports and windows sockets

csrss

Hi! I am completely stuck with it and don't know what to do. The actual code is pretty huge so i'll skip it and try to explain things. So. I am trying to write an echo server using completion ports and AcceptEx winsock extension. So far i got a server which can accept only one client and it works. But. The problem is: it can accept one and the only one client. This is how it goes. Basically i am doing almost same things like in tutorials on the net, extending wsaoverlapped, passing my struct as a parameter (keydata) to createiocompletionport where i am specifying current operation (read, write, accept, etc), and on GetQueuedCompletionStatus i am getting my struct by casting it to returned overlapped struct - if you are familiar with this subject you know what i am talking about. I am creating a server (in case of server app) thread where i am calling AcceptEx which completion then ends up in worker thread and then i am working only in this worker thread, while initial server thread just sleeps waiting for termination. So, when AcceptEx succeeds, i am creating a new my own defined struct, which extends wsaoverlapped and calling AcceptEx again, then processing other operations, like read, write etc. And there is a problem, when new client connects to a server, this new AcceptEx accepts new client connection, but then previous connection gets terminated and this new connection gets terminated too. I have just started learning IOCP and there is a question: in my project there is one server thread, which first do: 1. creating main listen socket 1a. launching worker thread with GetQueuedCompletionStatus 2. creating this fancy, so called, OverlappedPlus struct which holds some info 3. creating new socket for AcceptEx call 4. binds on port and starts listen 4a. Setting current operation to IOAccept 5. call AcceptEx (with all required params including pointer to wsaoverlapped struct which is first in my OverlappedPlus struct) and goes to sleep forever. ----[ at this point controls jumps to worker thread ]---- 1. GetQueuedCompletionStatus and then get OverlappedPlus. 2. Check which operation do we have? We got IOAccept (i have written separate functions for processing these "events"), ok, jump to OnIOAccept. 3. Create new OverlappedPlus struct, new socket for AcceptEx call, update completion port with this new socket and call AcceptEx which then have to wait for new client. 4. Process then any other IO operations. And all this works fin

jschell

In general the usual cause for that is that the listen/accept only occurs once. Some reasons for that - The processing is not threaded so the thread that is doing the accept is used to process or otherwise blocks. - The accept must be restarted in some way (such as the windows async methods which must use Begin and End type methos appropriately.)

csrss

Well, in my case i got one server thread and one worker thread. Server thread first do all preparations, call listen and then call AcceptEx, and then just sleep. Every next call for AcceptEx occurs in worker thread, which processes everything. I am just wondering, how one worker thread can process everything? And more then one client? I am familiar with regular sockets, where you create a new thread for each accepted socket, but in this case i should just update my IO port with this newly created socket and worker thread will operate on every socket and every client connection?

011011010110000101100011011010000110100101101110 0110010101110011

Mark Salsbery

I may not be following all your steps, but it appears all your threads end up waiting for socket accepts so everything is deadlocked. Here's what I do... Create a listener thread. Its job is to accept incoming connections and queue up the first completion job on the I/O completion port. I personally don't use AcceptEx anymore. I use a regular listener socket (bind()) and use an event (::WSAEventSelect(hListenSocket, hNetEvent, FD_ACCEPT);) to wake my listener thread when a connection request arrives. The listener thread proc just loops waiting on the event (and also a termination event for shutdown time). It looks like this:

DWORD CClientListener::ThreadProc()
{
while (1)
{
#define EVENT_NET 0
#define EVENT_TERMINATE 1
HANDLE Events[] =
{
hNetEvent,
hTerminateEvent
};
DWORD dwEventIndex = ::WaitForMultipleObjects(sizeof(Events) / sizeof(HANDLE),
Events, FALSE, INFINITE);
if (dwEventIndex == WAIT_FAILED)
{
break;
}
dwEventIndex -= WAIT_OBJECT_0;

//\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*

    // Check if thread termination requested

    if (dwEventIndex == EVENT\_TERMINATE)
    {
        break;
    }

//\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*

    // Check if FD\_ACCEPT socket notification

    if (dwEventIndex == EVENT\_NET)
    {
        WSANETWORKEVENTS WsaNetworkEvents;
        if (0 == ::WSAEnumNetworkEvents(hListenSocket, hNetEvent, &WsaNetworkEvents)) //(resets hNetEvent)
        {
            if (WsaNetworkEvents.lNetworkEvents & FD\_ACCEPT)
            {
                OnAccept();
            }
        }
    }

//\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*


}  //while (1)

return 0U;

}

OnAccept() 1) creates a socket for the connection using ::WSAAccept() 2) creates an object of a class representing a client connection and adds this object to a collection 3) associates the new socket with the completion port using ::CreateIoCompletionPort() 4) creates/initializes a custom overlapped structure 5) Queues up a receive operation on the completion port using WSARecv() That's it. I don't go to another thread to

csrss

Hey Mark, thanks for your reply - i was in fact doing almost same things :P. Well, but, thats it, and thank you so much!! I got rid of AcceptEx call, and, implemented accept event (like you do) and now it works!! Finally! :)

011011010110000101100011011010000110100101101110 0110010101110011

Mark Salsbery

csrss wrote:

I got rid of AcceptEx call, and, implemented accept event (like you do) and now it works!

Cool! It was so long ago I forget why I don't use AcceptEx()...but I'm sure I couldn't get something to work the way I needed to :) I do really like I/O completion ports. Not just for I/O... For native C++ apps they make a nice built-in thread pool for queuing operations on worker threads. Very efficient and Microsoft did most of the work :)

Mark Salsbery Microsoft MVP - Visual C++ :java:

csrss

I am working with IOCP just for a 3 days now (current project requirement) and starting to love it. Completion ports are a way better then creating tons of threads for connections and tons of events as well - you got everything in one place, its like, kind of message loop, isnt it? :) having one struct, jumping from reading to writing and queuing stuff, defying operations, and performance for sure. Totally agree - IOCP is outstanding option - one port to rule them all :D Ok, back to coding

011011010110000101100011011010000110100101101110 0110010101110011