A problem while transferring data over the NetworkStream
-
Hi, I am trying to send a big file over a TCP connection using NetworkStream and TCPClient. Both the client and the server is written in C#. There is no problem while transferring small files around the size of 300KB. My buffer size is 2KB and here is how the protocol works. -Server requests the file. -Client tells the server the size of the file, and then sends the first 2KB of it. -The server asks for another chunk of the file -like, send 2KB data, starting from 42649- -Client sends the chunk -Server asks for more, and client sends it... -When the end of file is reached, the client acknowledges the server that transfer is complete. Without looking at the code, since it is too long to be posted here, can you see any possible problems that may be causing this problem? I even tried sleeping the thread while receiving and processing the incoming chunks of data. Regards
Some possibilities - Ignored exceptions on server - Buffer doesn't match size (read/write). Either end. - Something is pausing the server. Seems that the problem is deterministic so you can add logging and print exactly what the sizes are that you read and write each time.
-
Hi, I am trying to send a big file over a TCP connection using NetworkStream and TCPClient. Both the client and the server is written in C#. There is no problem while transferring small files around the size of 300KB. My buffer size is 2KB and here is how the protocol works. -Server requests the file. -Client tells the server the size of the file, and then sends the first 2KB of it. -The server asks for another chunk of the file -like, send 2KB data, starting from 42649- -Client sends the chunk -Server asks for more, and client sends it... -When the end of file is reached, the client acknowledges the server that transfer is complete. Without looking at the code, since it is too long to be posted here, can you see any possible problems that may be causing this problem? I even tried sleeping the thread while receiving and processing the incoming chunks of data. Regards
what you should do IMO is this: 1. improve observability: provide logging on both sides, with actual times, and relevant data, such as offset and length. 2. create a synthetic test file, so the receiver can predict and hence verify what should and is received. Here is one useful algorithm: for each block, start with the offset (yes, I mean stuff the offset in the data, I know you already send it as metadata), then add incrementing data modulo some number that isn't a power of 2 (for bytes, 255 works just fine). The net result is (1) each block is different (the offset!), and (2) yet very predictable. You'll see immediately if and when what gets received is completely wacked. 3. As you have packets already in your yet extremely simple protocol, add a checksum to the protocol, so the receiver can verify things are correct. And send an ACK (when OK) or a RESEND-REQUEST (when checksum fails or something goes wrong). 4. Refinement: don't wait for the ACK before you send the next packet; a strictly sequential process would slow things down needlessly (sender calculates checksum, sends data, waits for ACK, receiver gets data, checks checksum, sends ACK/RESEND, resulting in two periods the line isn't active at all). Instead, allow for at least 1, better a few, packets in progress, i.e. only wait for ACK for packet 1 after having send packet 1+N where N is 1 upto a few. :)
Luc Pattyn [My Articles] Nil Volentibus Arduum
The quality and detail of your question reflects on the effectiveness of the help you are likely to get.
Please use <PRE> tags for code snippets, they improve readability.
CP Vanity has been updated to V2.3 -
Hi, I am trying to send a big file over a TCP connection using NetworkStream and TCPClient. Both the client and the server is written in C#. There is no problem while transferring small files around the size of 300KB. My buffer size is 2KB and here is how the protocol works. -Server requests the file. -Client tells the server the size of the file, and then sends the first 2KB of it. -The server asks for another chunk of the file -like, send 2KB data, starting from 42649- -Client sends the chunk -Server asks for more, and client sends it... -When the end of file is reached, the client acknowledges the server that transfer is complete. Without looking at the code, since it is too long to be posted here, can you see any possible problems that may be causing this problem? I even tried sleeping the thread while receiving and processing the incoming chunks of data. Regards
Since it works for small files, and appears to be deterministic (does it always fail at the same point? or is it just more likely that larger files will fail?), it must be to do with filling an array, running out of space in a TCP buffer or something similar. If it is not so simple (i.e. it fails more on larger files, but it is not deterministic) then it is probably to do with the handshake to request more data. It is impossible to go into any more detail than that with the information available. I can only echo the advice to sprinkle your code liberally with logging statements to find out what is actually happening. Why are you doing that, anyway? TCP guarantees the ordering and transmission so you can just send the whole file on the sender end. What you are doing adds latency (2 way ping) each chunk.
-
You have no mechanism in your protocol to force either the client or server to force another request for the same data. If, after a timeout expires, the client doesn't get an ACK from the server, it should be able to try and reestablish communication, telling the server it's going to resend a block of data.
A guide to posting questions on CodeProject[^]
Dave Kreskowiak -
TCP guarantees that the sent packets will arrive, unless the connection drops entirely (which the OP would have noticed), so I doubt that this is the problem.
The connection is still intact, it doesn't drop. The only problem is that the server stops requesting chunks.
-
Since it works for small files, and appears to be deterministic (does it always fail at the same point? or is it just more likely that larger files will fail?), it must be to do with filling an array, running out of space in a TCP buffer or something similar. If it is not so simple (i.e. it fails more on larger files, but it is not deterministic) then it is probably to do with the handshake to request more data. It is impossible to go into any more detail than that with the information available. I can only echo the advice to sprinkle your code liberally with logging statements to find out what is actually happening. Why are you doing that, anyway? TCP guarantees the ordering and transmission so you can just send the whole file on the sender end. What you are doing adds latency (2 way ping) each chunk.
I can't say that it is deterministic since if I pause the program at certain intervals, it can complete the transmission. If I don't pause the program at all, it stops after transferring 200-300KB. I am aware that what I am doing is just slowing down the transfer rate, but I am new to this are of programming and currently trying to develop this project to increase my experience. I will now add a timeout system to the server and request the rest of the data if nothing is incoming.
-
TCP guarantees that the sent packets will arrive, unless the connection drops entirely (which the OP would have noticed), so I doubt that this is the problem.
I know. It works great too, on paper. In actual practice, it's not 100% accurate.
A guide to posting questions on CodeProject[^]
Dave Kreskowiak -
I can't say that it is deterministic since if I pause the program at certain intervals, it can complete the transmission. If I don't pause the program at all, it stops after transferring 200-300KB. I am aware that what I am doing is just slowing down the transfer rate, but I am new to this are of programming and currently trying to develop this project to increase my experience. I will now add a timeout system to the server and request the rest of the data if nothing is incoming.
SimpleData wrote:
I can't say that it is deterministic since if I pause the program at certain intervals, it can complete the transmission. If I don't pause the program at all, it stops after transferring 200-300KB.
As another thought - a firewall. It is throttling your throughput. When you pause it you are throttling yourself so it doesn't get involved. The same behavior could be due to some oddity in your server code though. Such as attempting to be clever by using threads for reading and writing.
-
SimpleData wrote:
I can't say that it is deterministic since if I pause the program at certain intervals, it can complete the transmission. If I don't pause the program at all, it stops after transferring 200-300KB.
As another thought - a firewall. It is throttling your throughput. When you pause it you are throttling yourself so it doesn't get involved. The same behavior could be due to some oddity in your server code though. Such as attempting to be clever by using threads for reading and writing.
Can you elaborate on "attempting to be clever by using threads for reading and writing"? Because it sounds like something I might have done. :)
-
Can you elaborate on "attempting to be clever by using threads for reading and writing"? Because it sounds like something I might have done. :)
-
To use two threads something needs to control access to the shared data. If you mess that control up then you can end up blocking or taking a large amount of time.
Wouldn't using the "lock" directive be enough?
-
Wouldn't using the "lock" directive be enough?