How to send huge data via sockets in continuous intervals

manoharbalu

Please refer my posting dated 29-Apr-16 3:52. I am still finding problem with that. Even If I try to send the data as small chunks as per the suggestions here, it gets struck or hanged. Sever Side Code for Sending data: void CMainFrame::OnSendactiveInsdata() { // TODO: Add your command handler code here if(!InstClosedFlag) { TransData.Init(); long nSize = sizeof(struct TagModalDB); memset(EsimData, 0, nSize); memcpy(EsimData, Esim, nSize); long nApproxSize = 900000; int nEnd = 0; long j = 0; int nOffset = nApproxSize/100; for (int i=0; i<100; i++) { TransData.Init(); memset(TestData, 0, 9000); memcpy(TestData, EsimData + nEnd, nOffset); for(j = 0; j < nOffset; j++) { TransData.m_sData.Insert(j, TestData[j]); } SendData(m_pInsSocket, &TransData); nEnd += nOffset; if ((nEnd + nOffset) > nSize) nOffset = nSize - nEnd; } Sleep(500); } } Client Side Code for Receiving data: void CMainFrame::ProcessPendingReadIns(void) { long nApproxSize = 900000; int nEnd = 0; long j = 0; int nOffset = nApproxSize/100; long nSize = sizeof(struct TagModalDB); memset(EsimData, 0, nApproxSize); for (int i=0; i<100; i++) { TransData.Init(); ReceiveData(&TransData); memcpy(EsimData + nEnd, (const char*)TransData.m_sData, nOffset); nEnd += nOffset; if ((nEnd + nOffset) > nSize) nOffset = nSize - nEnd; } memcpy(Esim, EsimData, nSize); Sleep(500); } Please provide me suggestions and step by step approach to send/receive huge data continously in cycles. Previous Posting: How to send huge data via sockets in continuous intervals Our MFC Server application send huge real time data of size 900000bytes using CSocket (TCP/IP Sockets) via serialization continuously every 2 second to the MFC Client. But After a few minutes, the application gets slowed down. Is there any specific ways or methods of send/receive huge data continuosly via Csocket. How to send the big data every interval in a better way.

Lost User

Your code could be made more efficient just by a bit of tidying up. The following two lines will add considerably to processing time:

memset(EsimData, 0, nSize);
memcpy(EsimData, Esim, nSize);

Why clear the buffer to zeroes and then immediately overwrite it with data? The memset serves no purpose. Later you copy partial buffers into your large buffer a block at a time, again using memset for no good reason. Why not just receive the data direct into the large buffer?

leon de boer

Of coarse it slows down you send the client and server to sleep for 1/2 a second everytime it sends or recieves. You do get that sleeps the entire MFC framework don't you? It appears to be a some sort of half arsed timing mechanism ... there are things called TIMERS for that sort of thing use them for what they are intended.

In vino veritas

leon de boer

Of coarse it slows down you send the client and server to sleep for 1/2 a second everytime it sends or recieves. You do get that sleeps the entire MFC framework don't you? It appears to be a some sort of half arsed timing mechanism that will inevitably fail ... there are things called TIMERS for timing functions, use them. If you want it blunt get rid of these and things will work a whole lot quicker =====> Sleep(500); If you want me to be super blunt there is no good reason to sleep an MFC or any Win32 program ever, forget that call and never use it in proper code it's a hack we sometimes use when trying to proto stuff. There are a multitude of proper ways to load balance and wait for things in nice Windows proper ways.

In vino veritas

manoharbalu

I removed the Sleep, but still the client is hanging.

manoharbalu

I removed the memset statement. Server is Ok now. But the client is still hanging

Lost User

manoharbalu wrote:

But the client is still hanging

Then you need to do some debugging. There is no way anyone here can guess why.

David Crow

manoharbalu wrote:

Is there any specific ways or methods of send/receive huge data continuosly via Csocket. How to send the big data every interval in a better way.

See here.

"One man's wage rise is another man's price increase." - Harold Wilson

"Fireproof doesn't mean the fire will never come. It means when the fire comes that you will be able to withstand it." - Michael Simmons

"You can easily judge the character of a man by how he treats those who can do nothing for him." - James D. Miles

leon de boer

This needs to be done properly but lets see if we can get this at least going, so let me guess at the next problem what is Transdata? You loop in some sort of weird poll loop 100 times and call Transdata.Init ... if that is a constructor you just made 100 transdata and you never dispose of them. Check if it is your responsibility to dispose of TransData. However you can pull up windows resource manager and see if your client program memory growing and growing and growing because that will be the source and rather quickly. You don't dispose of transdata and if what I fear is true your program will leak memory like a sponge. I actually can't get why you even need transdata, can't you just send the receive data to your buffer EsimData Wouldn't something like ReceiveData(&Esimdata[nEnd]); put the read data directly where you need it without the the need for transdata. I don't know what EsimData is so the format may not be right but explain why you can't directly put the data in in Esimdata. I sort of need to know what Transdata and Esimdata are to help any further. Finally if you really want to do the receiving by polling, which is fine then do it properly setup a timer and check for received data from time to time don't sit in a loop and spin your wheels for 100 times.

In vino veritas

manoharbalu

The loop is not a poll loop. Just segmenting the data 900000bytes/100 and sending it 100 times, the same way I am receiving it at the client. And the Init function just initialises the CString var. I send/receive the data through Serialization and CTrans is derived from CObject for that purpose. And thats why I am using the EsimData as intermediate. Its declared globally as... char Esimdata[900000]; void CTrans::Init() { m_bClose = FALSE; m_sData = _T(""); } Please correct me if I am wrong.

leon de boer

I am still concerned you are bleeding memory did you check the program memory usage isn't continually growing? Can you use m_sData.Empty(); rather that m_sData = _T(""); I rarely use MFC strings but that line is ringing alarm bells with me. To smooth the recption down you really need to build a parser on the receive data rather than just sit looping around waiting for the full packet. Can I get a look at the code for ReceiveData .. as in the call ... ReceiveData(&TransData); Finally can I suggest you format the data into 3 sections, which was actually suggested to you back in April to make parsing easier and robust. The packet transmission will be a (i) A fixed signature (ii) a data body size (iii) the body data of the size of (ii) The parser would be 4 stage .... read signature, read data size, read data, data available (do what you want with it). enum ReadParseStatus { readSigState = 0, readDataSizeState, readDataBodyState, readDataAvailState }; The fix signature needs to be something unlikely to be in data along lines of const char signature[] = "PACKETSIG%$#"; Your server needs to output matching data packets. Same signature, the size of the data body and then the data body. If you provide the ReceiveData code I can help with the parser setup, which is a lot better than looping around waiting for data and it's pretty trivial.

In vino veritas

manoharbalu

void CMainFrame::ReceiveData(CTrans* pData) { /*if(CAsyncSocket::FromHandle(m_pSocket->m_hSocket) == NULL) { m_pSocket->Attach(m_pSocket->m_hSocket); }*/ TRY { if( m_pArchiveIn ) pData->Serialize(*m_pArchiveIn); else { pData->m_bClose = TRUE; m_pArchiveOut->Abort(); } } CATCH(CFileException, e) { pData->m_bClose = TRUE; m_pArchiveOut->Abort(); } END_CATCH if (pData->m_bClose) { delete m_pArchiveIn; delete m_pArchiveOut; delete m_pFile; delete m_pSocket; m_pArchiveIn = NULL; m_pArchiveOut = NULL; m_pFile = NULL; m_pSocket = NULL; } }

Joe Woodbury

I rarely suggest a completely different approach, but have you considered ASIO[^](the non-Boost version if you aren't using Boost) and making both the send and receive asynchronous?

leon de boer

Okay 1 polled 4 state parser code .. I commented it so you can follow what its doing. The parse buffer is 4K so you need to poll min rate of 20-30 times a second. You can increase or decrease the parse buffer size and change the poll rate requirements. Just remember your 90K data/ parse buffer size = min poll rate you need. You can use a timer or hook into the MFC message system to fire of a poll request ... personally I would use a timer. If its just writing the data to a file or something you could just loop the poll call in a thread. The data body is available in the dataReadyState and you haven't indicated what you do with it so its blank. The parser just cycles an maintains and cleans its own buffers.

enum ReadParseStatus {
readSigState = 0,
readDataSizeState,
readDataBodyState,
readDataAvailState
};

// This is some unique signature make it what you like
// It should be something unlikely to occur in the data itself
// Make sure the server sends an identical signature and the data size before each packet body
const char signature[] = "PACKETSIG%$#";

// Parse status always starts as read signature
enum ReadParseStatus parseState = readSigState;

// Parse buffer
char parseBuffer[4096]; // The parse buffer itself ... you may adjust this up or down depending on poll rate
int parsePos = 0; // Position of parse buffer
long bodyDataSize = 0; // Body data size for current packet
unsigned char* bodyData = NULL; // Body data storage
long bodyDataPos = 0; // Body data position

// Parses socket receive data
BOOL Poll_Receive_Data (CAsyncSocket sock){

// Read whatever data is available into the parse buffer up to parse buffer full
// sock would be m\_pSocket in your code
int iResult = sock.Receive(&parseBuffer\[parsePos\], sizeof(parseBuffer) - parsePos);

// If return is zero or less there is either no data of something is wrong
// So return a false so caller knows and can do something
if (iResult <= 0) return (FALSE);

// Adjust the parse position so space in buffer left can be checked
parsePos += iResult;

switch (parseState){
	case readSigState: {	
			// Temp buffer .. its 32 bytes here which must be bigger than signature length
			char tempbuf\[32\];
			// Hold signature length we will use it a bit in this phase
			int siglen = strlen(signature);
			// Preset sigFound to false
			BOOL sigFound = FALSE;
			// Loop while signature not found and enough data to run signature test
			while ((sigFound == FALSE) &a