IOCP, Completion Key and Overlapped
-
I am simply trying to figure this out. So, we have this API,
GetQueuedCompletionStatus, which among others params, returning completion key which is "per device" and overlapped struct, which is "per call". Or something like this. In every example on the net and in platform SKD's samples, The Overlapped structure gets extended like this:struct MyOverlapped
{
OVERLAPPED Overlp;
// my data goes below
};And then it is used with
GetQueuedCompletionStatus
and other calls (WSAREcv, WSASend, etc). So the question is: why is that so? Why not use only completion key? From my observations, (debugging), data, which arrives with the completion key and data which gets extracted from casting MyOverlapped to received overlapped, is completely the same. For example, somewhere we do: MyOverlapped * myOver = do_something_insert_something();
PostQueuedCompletionStatus(..., (ULONG_PTR)myOver, ...);
and inside worker thread, we have:
MyOverlapped * myOver = NULL;
GetQueuedCompletionStatus(..., (PULONG_PTR)&myOver, ...);This code above returns the same what this code below: MyOverlapped * myOver = do_something_insert_something();
PostQueuedCompletionStatus(..., (ULONG_PTR)0, &myOver->Overlp);
and then in worker thread:
MyOverlapped * myOver = NULL;
ULONG_PTR Data;
LPWSAOVERLAPPED Overlapped = NULL;
GetQueuedCompletionStatus(..., (PULONG_PTR)&Data, (LPOVERLAPPED *)&Overlapped);
myOver = (MyOverlapped *)Overlapped;So why to extend Overlapped structure and whats the deal with these: "per device" and "per call"? Thanks
011011010110000101100011011010000110100101101110 0110010101110011
-
I am simply trying to figure this out. So, we have this API,
GetQueuedCompletionStatus, which among others params, returning completion key which is "per device" and overlapped struct, which is "per call". Or something like this. In every example on the net and in platform SKD's samples, The Overlapped structure gets extended like this:struct MyOverlapped
{
OVERLAPPED Overlp;
// my data goes below
};And then it is used with
GetQueuedCompletionStatus
and other calls (WSAREcv, WSASend, etc). So the question is: why is that so? Why not use only completion key? From my observations, (debugging), data, which arrives with the completion key and data which gets extracted from casting MyOverlapped to received overlapped, is completely the same. For example, somewhere we do: MyOverlapped * myOver = do_something_insert_something();
PostQueuedCompletionStatus(..., (ULONG_PTR)myOver, ...);
and inside worker thread, we have:
MyOverlapped * myOver = NULL;
GetQueuedCompletionStatus(..., (PULONG_PTR)&myOver, ...);This code above returns the same what this code below: MyOverlapped * myOver = do_something_insert_something();
PostQueuedCompletionStatus(..., (ULONG_PTR)0, &myOver->Overlp);
and then in worker thread:
MyOverlapped * myOver = NULL;
ULONG_PTR Data;
LPWSAOVERLAPPED Overlapped = NULL;
GetQueuedCompletionStatus(..., (PULONG_PTR)&Data, (LPOVERLAPPED *)&Overlapped);
myOver = (MyOverlapped *)Overlapped;So why to extend Overlapped structure and whats the deal with these: "per device" and "per call"? Thanks
011011010110000101100011011010000110100101101110 0110010101110011
The key is per-handle, not per device. Many Win32 APIs have a pointer-sized parameter that applications can use and the system simply passes through. This is one of those. Depending on implementation, one may need a place to store some data associated with the lifetime of the handle - a pointer to a C++ object, an index into an array, an ID number, etc. That may not be something wanted in the overlapped structure which technically is only used by the system for each overlapped operation. As I've mentioned to you, I personally reuse the same custom OVERLAPPED structure object for multiple I/O calls, so for my implementations I can keep per-handle data there as well. So, yes the key is not relevant to me. It's there if you need it, ignore it if you don't need it (the system ignores it) :)
Mark Salsbery Microsoft MVP - Visual C++ :java:
-
I am simply trying to figure this out. So, we have this API,
GetQueuedCompletionStatus, which among others params, returning completion key which is "per device" and overlapped struct, which is "per call". Or something like this. In every example on the net and in platform SKD's samples, The Overlapped structure gets extended like this:struct MyOverlapped
{
OVERLAPPED Overlp;
// my data goes below
};And then it is used with
GetQueuedCompletionStatus
and other calls (WSAREcv, WSASend, etc). So the question is: why is that so? Why not use only completion key? From my observations, (debugging), data, which arrives with the completion key and data which gets extracted from casting MyOverlapped to received overlapped, is completely the same. For example, somewhere we do: MyOverlapped * myOver = do_something_insert_something();
PostQueuedCompletionStatus(..., (ULONG_PTR)myOver, ...);
and inside worker thread, we have:
MyOverlapped * myOver = NULL;
GetQueuedCompletionStatus(..., (PULONG_PTR)&myOver, ...);This code above returns the same what this code below: MyOverlapped * myOver = do_something_insert_something();
PostQueuedCompletionStatus(..., (ULONG_PTR)0, &myOver->Overlp);
and then in worker thread:
MyOverlapped * myOver = NULL;
ULONG_PTR Data;
LPWSAOVERLAPPED Overlapped = NULL;
GetQueuedCompletionStatus(..., (PULONG_PTR)&Data, (LPOVERLAPPED *)&Overlapped);
myOver = (MyOverlapped *)Overlapped;So why to extend Overlapped structure and whats the deal with these: "per device" and "per call"? Thanks
011011010110000101100011011010000110100101101110 0110010101110011
One framework I created that used ICOP had the following classes:
IoObj - Base class for I/O
IoHnd - IFS handle based
File - OS file access
NetSocket - Network socket
NetIpClnt/NetIpSvc, NetTCP*, NetUdp*, NetIrda*, NetBth*, ...
NetFPT*, NetNNTP*, ...WSAOVERLAPPED
IoOL - Base overlapped; adds flags, timestamps, type, control, buffer
NetIoOLAddr - specialied for sendto/recfrom, contains remote address
NetIoOLAccept - specialized for handling AcceptExIoCP - I/O completion port
The IoCP had the main posting functions and GetQueuedCompletionStatus loop. To post a msg on an IoCP you'd pass an IoObj* and IoOL*. The IoObj* would be the key, the IoOL* the overlapped (per call). The IoCP GetQueuedCompletionStatus loop would call (something like) IoObj->Msg(IoOL*). With the above you can use one or more IoCP instances to handle any combination of files, sockets, or other IoObj based objects (e.g. jobs/tasks). I found both having key and OL convenient. Without an explicit key param you'd have to add it as state to your extended OL ... which you may want anyways.
...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack
-
The key is per-handle, not per device. Many Win32 APIs have a pointer-sized parameter that applications can use and the system simply passes through. This is one of those. Depending on implementation, one may need a place to store some data associated with the lifetime of the handle - a pointer to a C++ object, an index into an array, an ID number, etc. That may not be something wanted in the overlapped structure which technically is only used by the system for each overlapped operation. As I've mentioned to you, I personally reuse the same custom OVERLAPPED structure object for multiple I/O calls, so for my implementations I can keep per-handle data there as well. So, yes the key is not relevant to me. It's there if you need it, ignore it if you don't need it (the system ignores it) :)
Mark Salsbery Microsoft MVP - Visual C++ :java:
Mark Salsbery wrote:
The key is per-handle, not per device.
Thanks! This explains couple of things. Ok, so (from that what i can see) if this will be a good practice (currently i am reusing custom Overlapped struct too): 1. Key data structure, which holds, lets say, a socket - accepted or connected or listening, some other things related to a socket, which are valid as long as our socket is valid, for example local and remote ip addr and port. - so all this lives as long as our socket lives 2. Extended Overlapped structure, which holds, for example, such info like current I/O operation and WSABUF and probably nothing else. Or maybe all this is just playing around and doesn't matter at all and it's ok to just use extended Overlapped all the time? And, (probably) last (probably lame :P) question - it just came to my mind, for now my server prototype is running on localhost and not event close to be loaded so every call to wsasend / wsarecv completes right away without waiting and i am manually posting completion packets with
PostQueuedCompletionStatusto simulate expected behavior, where key param is my struct of course, and IO operation is updated before packet is actually posted. So on the other end, i am parsing IO operations correctly. But, i am thinking now, if wsarecv will block and completion packet will arrive some time later - so how on the other end i can correctly parse current IO operation? And maybe exactly in such case - i need this extended overlapped and only, because on the other end (in worker thread in fact) - when packet will arrive, - i'll get this exact Overlapped and will be able to grab current IO? Am i crazy, wrong or maybe this is how it is? :D Thanks in advance011011010110000101100011011010000110100101101110 0110010101110011
-
One framework I created that used ICOP had the following classes:
IoObj - Base class for I/O
IoHnd - IFS handle based
File - OS file access
NetSocket - Network socket
NetIpClnt/NetIpSvc, NetTCP*, NetUdp*, NetIrda*, NetBth*, ...
NetFPT*, NetNNTP*, ...WSAOVERLAPPED
IoOL - Base overlapped; adds flags, timestamps, type, control, buffer
NetIoOLAddr - specialied for sendto/recfrom, contains remote address
NetIoOLAccept - specialized for handling AcceptExIoCP - I/O completion port
The IoCP had the main posting functions and GetQueuedCompletionStatus loop. To post a msg on an IoCP you'd pass an IoObj* and IoOL*. The IoObj* would be the key, the IoOL* the overlapped (per call). The IoCP GetQueuedCompletionStatus loop would call (something like) IoObj->Msg(IoOL*). With the above you can use one or more IoCP instances to handle any combination of files, sockets, or other IoObj based objects (e.g. jobs/tasks). I found both having key and OL convenient. Without an explicit key param you'd have to add it as state to your extended OL ... which you may want anyways.
...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack
Thanks for the reply. Currently i am trying to dig into IOCP and get some descent understanding what's going on behind scenes. Could you check my latest reply to Mark? There is some important question i am trying to figure out though. Thanks
011011010110000101100011011010000110100101101110 0110010101110011
-
Mark Salsbery wrote:
The key is per-handle, not per device.
Thanks! This explains couple of things. Ok, so (from that what i can see) if this will be a good practice (currently i am reusing custom Overlapped struct too): 1. Key data structure, which holds, lets say, a socket - accepted or connected or listening, some other things related to a socket, which are valid as long as our socket is valid, for example local and remote ip addr and port. - so all this lives as long as our socket lives 2. Extended Overlapped structure, which holds, for example, such info like current I/O operation and WSABUF and probably nothing else. Or maybe all this is just playing around and doesn't matter at all and it's ok to just use extended Overlapped all the time? And, (probably) last (probably lame :P) question - it just came to my mind, for now my server prototype is running on localhost and not event close to be loaded so every call to wsasend / wsarecv completes right away without waiting and i am manually posting completion packets with
PostQueuedCompletionStatusto simulate expected behavior, where key param is my struct of course, and IO operation is updated before packet is actually posted. So on the other end, i am parsing IO operations correctly. But, i am thinking now, if wsarecv will block and completion packet will arrive some time later - so how on the other end i can correctly parse current IO operation? And maybe exactly in such case - i need this extended overlapped and only, because on the other end (in worker thread in fact) - when packet will arrive, - i'll get this exact Overlapped and will be able to grab current IO? Am i crazy, wrong or maybe this is how it is? :D Thanks in advance011011010110000101100011011010000110100101101110 0110010101110011
1. Agree. 2. Agree. Or maybe ... So on other end ... Which is how you can use an IOCP to be a job/task manager like you could also create using the Win32^ Job Object / Thread Pool functions. But I am thinking now ... wsarecv(OL) will not block your posting thread. When GetQueuedCompletionStatus returns the results of the wsarecv, the specified buffer will be filled with the received data ... of course this means the buffer specified in OL must persist from time wsarecv(OL) is called until you are done with it after GetQueuedCompletionStatus. If you did not get all data (i.e. only posted recv to get a header) then either post another async recv, or (depending on how you handle things) you may be able to call a sync recv to get rest of data (e.g. payload).
...cmk The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying. - John Carmack
-
Mark Salsbery wrote:
The key is per-handle, not per device.
Thanks! This explains couple of things. Ok, so (from that what i can see) if this will be a good practice (currently i am reusing custom Overlapped struct too): 1. Key data structure, which holds, lets say, a socket - accepted or connected or listening, some other things related to a socket, which are valid as long as our socket is valid, for example local and remote ip addr and port. - so all this lives as long as our socket lives 2. Extended Overlapped structure, which holds, for example, such info like current I/O operation and WSABUF and probably nothing else. Or maybe all this is just playing around and doesn't matter at all and it's ok to just use extended Overlapped all the time? And, (probably) last (probably lame :P) question - it just came to my mind, for now my server prototype is running on localhost and not event close to be loaded so every call to wsasend / wsarecv completes right away without waiting and i am manually posting completion packets with
PostQueuedCompletionStatusto simulate expected behavior, where key param is my struct of course, and IO operation is updated before packet is actually posted. So on the other end, i am parsing IO operations correctly. But, i am thinking now, if wsarecv will block and completion packet will arrive some time later - so how on the other end i can correctly parse current IO operation? And maybe exactly in such case - i need this extended overlapped and only, because on the other end (in worker thread in fact) - when packet will arrive, - i'll get this exact Overlapped and will be able to grab current IO? Am i crazy, wrong or maybe this is how it is? :D Thanks in advance011011010110000101100011011010000110100101101110 0110010101110011
csrss wrote:
...my server prototype is running on localhost and not event close to be loaded so every call to wsasend / wsarecv completes right away without waiting and i am manually posting completion packets with
PostQueuedCompletionStatusto simulate expected behavior, where key param is my struct of course, and IO operation is updated before packet is actually posted. So on the other end, i am parsing IO operations correctly. But, i am thinking now, if wsarecv will block and completion packet will arrive some time laterMaybe overthinking it? First, even if wsasend/wsarecv complete immediately, you still get completion packet from IOCP on an IOCP thread, so no need to handle special case of error on wsasend/wsarecv if there's no error or error code is "pending". Second, WSARecv won't block on an overlapped operation, so I'm not sure what you're thinking there.
Mark Salsbery Microsoft MVP - Visual C++ :java:
-
csrss wrote:
...my server prototype is running on localhost and not event close to be loaded so every call to wsasend / wsarecv completes right away without waiting and i am manually posting completion packets with
PostQueuedCompletionStatusto simulate expected behavior, where key param is my struct of course, and IO operation is updated before packet is actually posted. So on the other end, i am parsing IO operations correctly. But, i am thinking now, if wsarecv will block and completion packet will arrive some time laterMaybe overthinking it? First, even if wsasend/wsarecv complete immediately, you still get completion packet from IOCP on an IOCP thread, so no need to handle special case of error on wsasend/wsarecv if there's no error or error code is "pending". Second, WSARecv won't block on an overlapped operation, so I'm not sure what you're thinking there.
Mark Salsbery Microsoft MVP - Visual C++ :java:
Alright, thanks guys, going to experiment a bit longer with this. By "blocking" i meant a situation where you "post" wsarecv / wsasend and it returns error_io_pending and then awaits for completion packet in worker thread :P
011011010110000101100011011010000110100101101110 0110010101110011
-
Alright, thanks guys, going to experiment a bit longer with this. By "blocking" i meant a situation where you "post" wsarecv / wsasend and it returns error_io_pending and then awaits for completion packet in worker thread :P
011011010110000101100011011010000110100101101110 0110010101110011
csrss wrote:
By "blocking" i meant a situation where you "post" wsarecv / wsasend and it returns error_io_pending and then awaits for completion packet in worker thread :P
Gotcha. But technically that is not blocking...that's asynchronous I/O :)
Mark Salsbery Microsoft MVP - Visual C++ :java: