Sending file though socket as byte[]
-
I have a socket though which I'm sending files. If the file is under the buffer size (8192 in my case) then it is fine, but if the file is over that size, then when it writes to file, it writes the first chunk of 8192 bites every time. I'm using ObjectStream because beside the file, I'm sending other objects as well in the actual app, but this is the part of sending the file. Client it sends the file data (testing):
System.out.println("Starting client"); Socket socket = new Socket("localhost", port); System.out.println("Client sending data"); ObjectOutputStream dataOut = new ObjectOutputStream(socket.getOutputStream()); BufferedInputStream is = new BufferedInputStream(new FileInputStream(inFile)); byte\[\] buffer = new byte\[8192\]; int sizeRead = 0; while ((sizeRead = is.read(buffer, 0, 8192)) >= 0) { printByte(buffer, "in"); //Function that writes to a file as strings the byte\[\], and after 8192 you can see it is the first byte\[\] each time, also it writes at the end of file dataOut.writeObject(true); dataOut.writeObject(buffer); dataOut.writeObject(sizeRead); dataOut.flush(); } is.close(); System.out.println("Done sending file"); dataOut.writeObject(false); socket.close(); System.out.println("Client closed");Server that creates the file:
System.out.println("Starting server"); ServerSocket server = new ServerSocket(port); System.out.println("Server waiting"); Socket socket = server.accept(); System.out.println("Client connected"); ObjectInputStream dataIn = new ObjectInputStream(socket.getInputStream()); BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(outFile)); Object auxKeepReading = dataIn.readObject(); if (auxKeepReading instanceof Boolean) { while ((boolean) auxKeepReading) { Object auxBuffer = dataIn.readObject(); if (!(auxBuffer instanceof byte\[\])) { System.out.println("Broke byte"); break; } Object auxSize = dataIn.readObject(); if (!(auxSize instanceof Integer)) { System.out.println("Broke int"); break; } printByte((byte\[\]) auxBuffer, "out"); //Same as on client but different file, it writes at the end of file each time bos.write((byte\[\]) auxBuffer, 0, (int) auxSize); auxKeepReading = dataIn.readObject(); if (!(auxKeepReading instanceof Boole -
I have a socket though which I'm sending files. If the file is under the buffer size (8192 in my case) then it is fine, but if the file is over that size, then when it writes to file, it writes the first chunk of 8192 bites every time. I'm using ObjectStream because beside the file, I'm sending other objects as well in the actual app, but this is the part of sending the file. Client it sends the file data (testing):
System.out.println("Starting client"); Socket socket = new Socket("localhost", port); System.out.println("Client sending data"); ObjectOutputStream dataOut = new ObjectOutputStream(socket.getOutputStream()); BufferedInputStream is = new BufferedInputStream(new FileInputStream(inFile)); byte\[\] buffer = new byte\[8192\]; int sizeRead = 0; while ((sizeRead = is.read(buffer, 0, 8192)) >= 0) { printByte(buffer, "in"); //Function that writes to a file as strings the byte\[\], and after 8192 you can see it is the first byte\[\] each time, also it writes at the end of file dataOut.writeObject(true); dataOut.writeObject(buffer); dataOut.writeObject(sizeRead); dataOut.flush(); } is.close(); System.out.println("Done sending file"); dataOut.writeObject(false); socket.close(); System.out.println("Client closed");Server that creates the file:
System.out.println("Starting server"); ServerSocket server = new ServerSocket(port); System.out.println("Server waiting"); Socket socket = server.accept(); System.out.println("Client connected"); ObjectInputStream dataIn = new ObjectInputStream(socket.getInputStream()); BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(outFile)); Object auxKeepReading = dataIn.readObject(); if (auxKeepReading instanceof Boolean) { while ((boolean) auxKeepReading) { Object auxBuffer = dataIn.readObject(); if (!(auxBuffer instanceof byte\[\])) { System.out.println("Broke byte"); break; } Object auxSize = dataIn.readObject(); if (!(auxSize instanceof Integer)) { System.out.println("Broke int"); break; } printByte((byte\[\]) auxBuffer, "out"); //Same as on client but different file, it writes at the end of file each time bos.write((byte\[\]) auxBuffer, 0, (int) auxSize); auxKeepReading = dataIn.readObject(); if (!(auxKeepReading instanceof BooleOn client side after sending the size that it read, I added
buffer = new byte[8192];and now I don't have the problem and it is sending new data but there are 2 problems: 1. I don't know if this is what you should do; 2. The speed is limited to 3 MB/s when using IP instead of localhost, on an internet connection of 1.000 Mbps (~125 MB/s) upload/download (they are the same). I tried to increase the buffer size but still the same speed. -
On client side after sending the size that it read, I added
buffer = new byte[8192];and now I don't have the problem and it is sending new data but there are 2 problems: 1. I don't know if this is what you should do; 2. The speed is limited to 3 MB/s when using IP instead of localhost, on an internet connection of 1.000 Mbps (~125 MB/s) upload/download (they are the same). I tried to increase the buffer size but still the same speed.I was looking at this code earlier and wondered why you sent the buffer size after the data. It makes much more sense to send the size first. Also there is no point in sending 8192 bytes if you have only read one byte from the input file. So only send the amount of data that has been read. Tuning TCP performance is not an exact science unfortunately, but you can find plenty of discussions of the subject on the internet.
-
I was looking at this code earlier and wondered why you sent the buffer size after the data. It makes much more sense to send the size first. Also there is no point in sending 8192 bytes if you have only read one byte from the input file. So only send the amount of data that has been read. Tuning TCP performance is not an exact science unfortunately, but you can find plenty of discussions of the subject on the internet.
Richard MacCutchan wrote:
I was looking at this code earlier and wondered why you sent the buffer size after the data. It makes much more sense to send the size first.
Because when they are used it doesn't matter the order they were sent, I didn't even think about that.
Richard MacCutchan wrote:
Also there is no point in sending 8192 bytes if you have only read one byte from the input file. So only send the amount of data that has been read.
I changed that now on client:
dataOut.writeObject(Arrays.copyOf(buffer, sizeRead));What aboutbuffer = new byte[8192];I added, is that how it should be done to send now data each time?dataOut.writeObject(true); dataOut.writeObject(sizeRead); dataOut.writeObject(Arrays.copyOf(buffer, sizeRead)); dataOut.flush(); buffer = new byte\[8192\];Richard MacCutchan wrote:
Tuning TCP performance is not an exact science unfortunately, but you can find plenty of discussions of the subject on the internet.
From what I've seen they suggest two things most of the time: 1. Setting
socket.setTcpNoDelay(true);. I tried that on both client and server bust still no use; 2. Using BufferedOutputStream/BufferedInputStream inside ObjectOutputStream/ObjectInputStream, but still the same 3 MB/s. Even after combining 1. and 2. -
Richard MacCutchan wrote:
I was looking at this code earlier and wondered why you sent the buffer size after the data. It makes much more sense to send the size first.
Because when they are used it doesn't matter the order they were sent, I didn't even think about that.
Richard MacCutchan wrote:
Also there is no point in sending 8192 bytes if you have only read one byte from the input file. So only send the amount of data that has been read.
I changed that now on client:
dataOut.writeObject(Arrays.copyOf(buffer, sizeRead));What aboutbuffer = new byte[8192];I added, is that how it should be done to send now data each time?dataOut.writeObject(true); dataOut.writeObject(sizeRead); dataOut.writeObject(Arrays.copyOf(buffer, sizeRead)); dataOut.flush(); buffer = new byte\[8192\];Richard MacCutchan wrote:
Tuning TCP performance is not an exact science unfortunately, but you can find plenty of discussions of the subject on the internet.
From what I've seen they suggest two things most of the time: 1. Setting
socket.setTcpNoDelay(true);. I tried that on both client and server bust still no use; 2. Using BufferedOutputStream/BufferedInputStream inside ObjectOutputStream/ObjectInputStream, but still the same 3 MB/s. Even after combining 1. and 2.JohnCodding wrote:
What about buffer = new byte[8192]; I added, is that how it should be done to send now data each time?
Yes, that is fine, as you allocate the buffer before you start the read/write loop. So each read from the input file will refill the same buffer. But I would be tempted to use ObjectOutputStream write(byte[](Java Platform SE 7 )[^], rather than
writeObject, to send the byte data. I have never had to do any TCP tuning so can't offer any useful suggestions, sorry. -
JohnCodding wrote:
What about buffer = new byte[8192]; I added, is that how it should be done to send now data each time?
Yes, that is fine, as you allocate the buffer before you start the read/write loop. So each read from the input file will refill the same buffer. But I would be tempted to use ObjectOutputStream write(byte[](Java Platform SE 7 )[^], rather than
writeObject, to send the byte data. I have never had to do any TCP tuning so can't offer any useful suggestions, sorry.Richard MacCutchan wrote:
But I would be tempted to use ObjectOutputStream write(byte[](Java Platform SE 7 )[^], rather than writeObject, to send the byte data.
In the past when I used socket but not for files, I had some problems when I was using combination of ObjectOutputStream writeBoolean(), writeInt(), writeUTF(), writeObject(), the order was correct when writing/reading but I was still getting some exceptions, and when I switch to only writeObject() I didn't had any problems. And with objects I have more control over what the server gets from the client when the data is wrong, and shadow ban the IP if it keeps sending the wrong data based on some checks. As java class files can easily be decompiled, they can recreate the code and send part of the data good then wrong one just to be jerks or for phishing data. I have some checks on the server to know when someone clearly wasn't supposed to send data, and ignore it, and it did help.
Richard MacCutchan wrote:
I have never had to do any TCP tuning so can't offer any useful suggestions, sorry.
Yeah, until now I didn't had problems, but now with files, also some that can be big, I do care about speed a lot. I'll keep looking and maybe I find a way to remove the limit. As there is clearly a limit because the speed stays around 3 MB/s and I did see someone complaining about the same exact speed, but there wasn't a solution on that thread.
-
On client side after sending the size that it read, I added
buffer = new byte[8192];and now I don't have the problem and it is sending new data but there are 2 problems: 1. I don't know if this is what you should do; 2. The speed is limited to 3 MB/s when using IP instead of localhost, on an internet connection of 1.000 Mbps (~125 MB/s) upload/download (they are the same). I tried to increase the buffer size but still the same speed.JohnCodding wrote:
The speed is limited to 3 MB/s when using IP instead of localhost, on an internet connection of 1.000 Mbps
Not quite sure what you are saying there but... Yes localhost and the actual network are different. So a speed difference is expected. Matter of fact if it was the same I would expect (most likely) a problem with the actual measurement or that there was a hardware/software throttle happening in the client.
JohnCodding wrote:
internet connection ... upload/download (they are the same).
Noting also that I would not expect that. Even for a business. But I do want to emphasize that there is a difference between 'network' and 'internet'.
-
JohnCodding wrote:
The speed is limited to 3 MB/s when using IP instead of localhost, on an internet connection of 1.000 Mbps
Not quite sure what you are saying there but... Yes localhost and the actual network are different. So a speed difference is expected. Matter of fact if it was the same I would expect (most likely) a problem with the actual measurement or that there was a hardware/software throttle happening in the client.
JohnCodding wrote:
internet connection ... upload/download (they are the same).
Noting also that I would not expect that. Even for a business. But I do want to emphasize that there is a difference between 'network' and 'internet'.
jschell wrote:
Noting also that I would not expect that. Even for a business.
jschell wrote:
Not quite sure what you are saying there but...
When transferring files over internet using the code I posted for sending/receiving, the transfer speed is always limited to 3 MB/s (bounces around 3 MB/s, it goes over or under with 100-200 KB/s) instead of the actual capabilities of internet speed and hardware (SSD). If localhost is used, then the transfer is not limited to 3 MB/s.
-
jschell wrote:
Noting also that I would not expect that. Even for a business.
jschell wrote:
Not quite sure what you are saying there but...
When transferring files over internet using the code I posted for sending/receiving, the transfer speed is always limited to 3 MB/s (bounces around 3 MB/s, it goes over or under with 100-200 KB/s) instead of the actual capabilities of internet speed and hardware (SSD). If localhost is used, then the transfer is not limited to 3 MB/s.
JohnCodding wrote:
If localhost is used, then the transfer is not limited to 3 MB/s.
I meant that 'localhost' and 'network' should not have the same speed. A connection, any connection, goes through actual hardware. So your speedtest is connecting to a computer somewhere. You client test app is connecting to a different computer somewhere. So something is throttling it. Best way is to test your app with another app that you control. You need two boxes. They can be local but you could also spin on a cloud box. Probably ideal on the cloud (internet connected.) If that test shows a speed limitation then it is your apps problem (with I doubt.) If not then it indicates that something at the other end is probably throttling it. Also possible that your service provider is throttling it. They are 'faking' the results with speedtest by explicitly looking for it. Or you might be hitting a data limit with your app that the document (or don't.)
