IOCP And WSANOBUFS Error
-
Hi, I'm writing a server application using UDP and the I/O Completion port. I red some articles and i know the things about it. My application scans a list of servers, requests some information, unpacks the information and stores the information, this is a process in loop. However, after a while i get the 10055 Error or better said WSANOBUFS. Now i red that that means that there cant be made any more locks ( source[^] ) I cant quit seem to find where the error takes place, there might be a sort of loop in locking where the the unlock never hits, but i dont know where that would be then. Is there any way to detect a possible deadlock? or do you know something else that might be wrong? Thanks already -Koen
-
Hi, I'm writing a server application using UDP and the I/O Completion port. I red some articles and i know the things about it. My application scans a list of servers, requests some information, unpacks the information and stores the information, this is a process in loop. However, after a while i get the 10055 Error or better said WSANOBUFS. Now i red that that means that there cant be made any more locks ( source[^] ) I cant quit seem to find where the error takes place, there might be a sort of loop in locking where the the unlock never hits, but i dont know where that would be then. Is there any way to detect a possible deadlock? or do you know something else that might be wrong? Thanks already -Koen
Where are you getting the WSAENOBUFS error? On a send, receive, connect? If on a send, have you disabled send buffering for the socket? Are you using IOCP for sends, recvs, connects? All overlapped operations?
-
Where are you getting the WSAENOBUFS error? On a send, receive, connect? If on a send, have you disabled send buffering for the socket? Are you using IOCP for sends, recvs, connects? All overlapped operations?
int res = WSARecvFrom(m_sSocket, &m_wsaBuf, 1, reinterpret_cast(&buflen), &temp, reinterpret_cast(&m_objAddress), &sockaddr_len, &m_objOverlapped, NULL);There i get the error 10055, Sending goes just fine. Since im using UDP im not connecting. Next to that, i dont use overlapped operations everywhere, i got a mysql class wich just sends a status update every 10 seconds, and a remote manager class wich uses TCP. both the mysql and the rmanager are using normal winsock so without overlapped operations.m_sSocket = socket(AF_INET, SOCK_DGRAM, 0); m_objAddress.sin_family = AF_INET; m_objLocalAddress.sin_family = AF_INET; m_objLocalAddress.sin_addr.s_addr = htonl(INADDR_ANY); m_objLocalAddress.sin_port = 0; m_hIOCP = hIOCP; if (bind(m_sSocket, reinterpret_cast(&m_objLocalAddress), sizeof(m_objLocalAddress))==SOCKET_ERROR){ TRACE("Error binding localaddress with the socket, Error: %i\n", WSAGetLastError()); } if (CreateIoCompletionPort(reinterpret_cast(m_sSocket), hIOCP, reinterpret_cast(const_cast(this)), 0)==NULL){ //Error binding? TRACE("Error binding Session to the main object, Error: %i\n", WSAGetLastError()); return; }Here i initialize is UDP session, there are like 400 udp virtual sessions continiously sending and recieving data. the first minut it goes all well, but after that, every sessions returns the error 10055 on recieving -
int res = WSARecvFrom(m_sSocket, &m_wsaBuf, 1, reinterpret_cast(&buflen), &temp, reinterpret_cast(&m_objAddress), &sockaddr_len, &m_objOverlapped, NULL);There i get the error 10055, Sending goes just fine. Since im using UDP im not connecting. Next to that, i dont use overlapped operations everywhere, i got a mysql class wich just sends a status update every 10 seconds, and a remote manager class wich uses TCP. both the mysql and the rmanager are using normal winsock so without overlapped operations.m_sSocket = socket(AF_INET, SOCK_DGRAM, 0); m_objAddress.sin_family = AF_INET; m_objLocalAddress.sin_family = AF_INET; m_objLocalAddress.sin_addr.s_addr = htonl(INADDR_ANY); m_objLocalAddress.sin_port = 0; m_hIOCP = hIOCP; if (bind(m_sSocket, reinterpret_cast(&m_objLocalAddress), sizeof(m_objLocalAddress))==SOCKET_ERROR){ TRACE("Error binding localaddress with the socket, Error: %i\n", WSAGetLastError()); } if (CreateIoCompletionPort(reinterpret_cast(m_sSocket), hIOCP, reinterpret_cast(const_cast(this)), 0)==NULL){ //Error binding? TRACE("Error binding Session to the main object, Error: %i\n", WSAGetLastError()); return; }Here i initialize is UDP session, there are like 400 udp virtual sessions continiously sending and recieving data. the first minut it goes all well, but after that, every sessions returns the error 10055 on recievingThis may or may not be the problem, but based on what I've seen it's the first thing that comes to mind. I hope this helps a bit... Unless you always keep enough receive buffers queued in your IOCP then the socket can receive datagrams faster than you are pulling them out of its buffer. If the socket is not "connected" (for UDP that just means all datagrams will be ignored except from the "connected" address) then the socket's recv buffer gets filled with every UDP datagram it sees on the network. You need to pull them out fast enough to keep enough buffer space for receives. One option is to use a connected socket to receive UDP datagrams on. The drawback is you'd need 400 sockets, or one for every session as you described. Another option is to always have a pool of overlapped WSARecvFrom() calls pushed on the IOCP stack so there will always be one available when a datagram is received. You could use the FD_READ notification to help notify a thread that data has been received on the socket and make sure there's a pending overlapped receive operation ready to accept the data. Also make sure your IOCP is properly "tuned" with the right amount of threads in the pool to most efficiently handle all the received datagrams. These articles may be helpful too (you may have seen them already - I keep them bookmarked for any IOCP troubles I encounter :))... Windows Sockets 2.0: Write Scalable Winsock Apps Using Completion Ports[^] INFO: Design Issues When Using IOCP in a Winsock Server[^]
-
This may or may not be the problem, but based on what I've seen it's the first thing that comes to mind. I hope this helps a bit... Unless you always keep enough receive buffers queued in your IOCP then the socket can receive datagrams faster than you are pulling them out of its buffer. If the socket is not "connected" (for UDP that just means all datagrams will be ignored except from the "connected" address) then the socket's recv buffer gets filled with every UDP datagram it sees on the network. You need to pull them out fast enough to keep enough buffer space for receives. One option is to use a connected socket to receive UDP datagrams on. The drawback is you'd need 400 sockets, or one for every session as you described. Another option is to always have a pool of overlapped WSARecvFrom() calls pushed on the IOCP stack so there will always be one available when a datagram is received. You could use the FD_READ notification to help notify a thread that data has been received on the socket and make sure there's a pending overlapped receive operation ready to accept the data. Also make sure your IOCP is properly "tuned" with the right amount of threads in the pool to most efficiently handle all the received datagrams. These articles may be helpful too (you may have seen them already - I keep them bookmarked for any IOCP troubles I encounter :))... Windows Sockets 2.0: Write Scalable Winsock Apps Using Completion Ports[^] INFO: Design Issues When Using IOCP in a Winsock Server[^]
-
This may or may not be the problem, but based on what I've seen it's the first thing that comes to mind. I hope this helps a bit... Unless you always keep enough receive buffers queued in your IOCP then the socket can receive datagrams faster than you are pulling them out of its buffer. If the socket is not "connected" (for UDP that just means all datagrams will be ignored except from the "connected" address) then the socket's recv buffer gets filled with every UDP datagram it sees on the network. You need to pull them out fast enough to keep enough buffer space for receives. One option is to use a connected socket to receive UDP datagrams on. The drawback is you'd need 400 sockets, or one for every session as you described. Another option is to always have a pool of overlapped WSARecvFrom() calls pushed on the IOCP stack so there will always be one available when a datagram is received. You could use the FD_READ notification to help notify a thread that data has been received on the socket and make sure there's a pending overlapped receive operation ready to accept the data. Also make sure your IOCP is properly "tuned" with the right amount of threads in the pool to most efficiently handle all the received datagrams. These articles may be helpful too (you may have seen them already - I keep them bookmarked for any IOCP troubles I encounter :))... Windows Sockets 2.0: Write Scalable Winsock Apps Using Completion Ports[^] INFO: Design Issues When Using IOCP in a Winsock Server[^]
If the socket is not "connected" (for UDP that just means all datagrams will be ignored except from the "connected" address) then the socket's recv buffer gets filled with every UDP datagram it sees on the network. You need to pull them out fast enough to keep enough buffer space for receives. I tried some solutions but i cant seem to get them working. Its probably my lack of experiance but well, The fundamental code:
while (1){ BOOL bIORet = GetQueuedCompletionStatus ( // Get a completed IO request pThis->m_hIOCompiltionPort, &dwIoSize, reinterpret_cast(pThis), &ptrOverlapped, INFINITE); if (!bIORet){ TRACE("Winsock error code: %i\n", GetLastError()); Gameye::Library::Thread::CMPThread::Sleep(0); } else{ ptrSession = static_cast(ptrOverlapped->Pointer); CAbstractServerClass* ptrGameClass = (*pThis->m_ptrGameManager)[ptrSession->getNDServer()->GameID]; if (ptrGameClass == NULL){ TRACE("Game is unknown, probably removed\n"); } else{ objToolset.Session = ptrSession; switch(ptrSession->getTask()){ case RequestSend: //Request to send a packet if (WSASendTo(sSocket, ptrSession->getWSABuffer(), 1, reinterpret_cast(&ptrSession->getWSABuffer()->len), 0,reinterpret_cast(ptrSession->getAddress()), sizeof(sockaddr_in), ptrSession->getOverlapped(), NULL) == -1) { int iErrorCode = WSAGetLastError(); if (iErrorCode != 997){ //997 = Data is already recieved TRACE("Sending failed because: %i\n", WSAGetLastError()); break; } } ptrSession->setStatus(AfterSend); break; case RequestRecieve: //Request to recieve a packet if (WSARecvFrom(sSocket, ptrSession->getWSABuffer(), 1, reinterpret_cast(&ptrSession->getWSABuffer()->len), &iTemp,reinterpret_cast(ptrSession->getAddress()), &iSockAddrSize, ptrSession->getOverlapped(), NULL) == -1) { int iErrorCode = WSAGetLastError(); if (iErrorCode != 997){ //997 = Data is already recieved TRACE("Recieving failed because: %i\n", WSAGetLastError()); ptrSession->setStatus(AfterRecieve); } } ptrSession->setStatus(AfterRecieve); -
If the socket is not "connected" (for UDP that just means all datagrams will be ignored except from the "connected" address) then the socket's recv buffer gets filled with every UDP datagram it sees on the network. You need to pull them out fast enough to keep enough buffer space for receives. I tried some solutions but i cant seem to get them working. Its probably my lack of experiance but well, The fundamental code:
while (1){ BOOL bIORet = GetQueuedCompletionStatus ( // Get a completed IO request pThis->m_hIOCompiltionPort, &dwIoSize, reinterpret_cast(pThis), &ptrOverlapped, INFINITE); if (!bIORet){ TRACE("Winsock error code: %i\n", GetLastError()); Gameye::Library::Thread::CMPThread::Sleep(0); } else{ ptrSession = static_cast(ptrOverlapped->Pointer); CAbstractServerClass* ptrGameClass = (*pThis->m_ptrGameManager)[ptrSession->getNDServer()->GameID]; if (ptrGameClass == NULL){ TRACE("Game is unknown, probably removed\n"); } else{ objToolset.Session = ptrSession; switch(ptrSession->getTask()){ case RequestSend: //Request to send a packet if (WSASendTo(sSocket, ptrSession->getWSABuffer(), 1, reinterpret_cast(&ptrSession->getWSABuffer()->len), 0,reinterpret_cast(ptrSession->getAddress()), sizeof(sockaddr_in), ptrSession->getOverlapped(), NULL) == -1) { int iErrorCode = WSAGetLastError(); if (iErrorCode != 997){ //997 = Data is already recieved TRACE("Sending failed because: %i\n", WSAGetLastError()); break; } } ptrSession->setStatus(AfterSend); break; case RequestRecieve: //Request to recieve a packet if (WSARecvFrom(sSocket, ptrSession->getWSABuffer(), 1, reinterpret_cast(&ptrSession->getWSABuffer()->len), &iTemp,reinterpret_cast(ptrSession->getAddress()), &iSockAddrSize, ptrSession->getOverlapped(), NULL) == -1) { int iErrorCode = WSAGetLastError(); if (iErrorCode != 997){ //997 = Data is already recieved TRACE("Recieving failed because: %i\n", WSAGetLastError()); ptrSession->setStatus(AfterRecieve); } } ptrSession->setStatus(AfterRecieve);You get a completion packet indicating "RequestRecieve" and in response you call WSARecvFrom() before calling ptrSession->setStatus(AfterRecieve); Chances are pretty good that another thread has already processed the completion of the WSARecvFrom() call before you can even check the error code! :) So, you'll get ANOTHER RequestRecieve before you have a chance to set the session status to AfterRecieve. Make sense? In other words, you probably want to call ptrSession->setStatus(AfterRecieve) BEFORE you call WSARecvFrom(). Also, use WSA_IO_PENDING instead of 997 so I can read your code haha :) I don't have all the WSAE codes memorized yet ;P
-
You get a completion packet indicating "RequestRecieve" and in response you call WSARecvFrom() before calling ptrSession->setStatus(AfterRecieve); Chances are pretty good that another thread has already processed the completion of the WSARecvFrom() call before you can even check the error code! :) So, you'll get ANOTHER RequestRecieve before you have a chance to set the session status to AfterRecieve. Make sense? In other words, you probably want to call ptrSession->setStatus(AfterRecieve) BEFORE you call WSARecvFrom(). Also, use WSA_IO_PENDING instead of 997 so I can read your code haha :) I don't have all the WSAE codes memorized yet ;P