Can not deserialize [modified]

[Jacob D Dixon]

Well let me first try to correct the way I am getting the data. I'm going to have to lookup what you are talking about though since I'm not sure how exactly I need to do that since I am just sending and receiving Serialized data and not just text or whatever. I did leave my laptop at work so I will have to try this tomorrow morning :-(

[Luc Pattyn]

two ways: have the length field go through the MemoryStream (may be difficult as you don't know the length yet), or write it to the socket separately. So replace s.Send(ms.ToArray()); by

byte[] bytes=ms.ToArray();
int len=bytes.Length;
s.Send(BitConverter.GetBytes(len));
s.Send(bytes);

plus some similar changes at the receiver of course. :)

Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles] Nil Volentibus Arduum

Please use <PRE> tags for code snippets, they preserve indentation, and improve readability.

[Jacob D Dixon]

Luc, Thank you for the replies. So I do understand how to send the bytes that contain the length of how many bytes are in my custom object. What I am trying to understand is how I will work this into my program. How will I know to try to deserialize or just convert the bytes to integer? On the server the first receive would always be the integer. Once I received all that data I would then call another method to perform another receive but this time deserializing it? Example:

private void FirstReadCallback(IAsyncResult iar)
{
StateObject so = iar as StateObject;
Socket handler = so.workSocket;

        int bytesRead = handler.EndReceive(iar);
        if (bytesRead > 0)
        {
              int myLength = BitConverter.ToInt32(state.buffer);
              state.buffer = new byte\[myLength\];
              
              // Now receive the serialized data
              handler.BeginReceive(state.buffer, 0, state.buffer.Length, 0, new AsyncCallback(SecondReadCallback), state);
        }

}

private void SecondReadCallback(IAsyncResult iar)
{
StateObject state = iar.AsyncState as StateObject;
Socket handler = state.worker;

    int bytesRead = handler.EndReceive(iar);
    if (bytesRead > 0)
    {
         state.ms.Write(state.buffer, 0, bytesRead);
         state.ms.Seek(0, SeekOrigin.Begin);

         IFormatter formatter = new BinaryFormatter();
         object receivedObject = null;

         receivedObject = formatter.Deserialize(state.ms);

         // and so on
    }

}

[Luc Pattyn]

1. you would need a Length data member in your class, initially zero. 2. I guess you have a BeginReceive() somewhere outside the callback; if so, just make it ask for 4 bytes. That is the easiest way. 3. You can do all EndReceives in a single callback; the first time (when Length is still zero), you need an EndReceive that matches step #2, hope for 4 bytes, and decode them to an integer, to be stored in Length. Use BitConverter.ToInt32(). Later on (when Length isn't zero) do an EndReceive for a full load of data (the maximum of Length and your buffer size), and decrement Length. Keep doing till Length is zero, or timeout or other error occurs. :)

Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles] Nil Volentibus Arduum

Please use <PRE> tags for code snippets, they preserve indentation, and improve readability.

[Jacob D Dixon]

Luc, Sorry for sounding dumb.. but what is with the 4 bytes? I'm searching google some more for example code of this but what I find is people doing things differently than what I'm trying to do. I see most of the code using Encoding to get the string until a EOF was passed or something. I guess I could do it that way. Pass the bytes as a string with a and cut off the EOF and convert to integer. I do want to try to understand your method, but right now I'm not fully understanding it. Would you mind providing a short example? [Edit] I'm guessing the 4 bytes would be the BitConverter.GetBytes()..... but how certain is 4 bytes would contain the entire length? I did some test and even with doing like a: byte[] b = new byte[4958454] the array length is still only 4. So let me do some test tomorrow and I'll post what I come up with. Thanks for your help

modified on Wednesday, December 1, 2010 11:29 PM

[Luc Pattyn]

If the length fits an int at the client (i.e. when it is less than 2 GB), all it takes to communicate it is the four bytes that hold the binary representation of that length value. So it will always be exactly four bytes. One could pass that as a string (with variable length) but why do so? it only complicates matters, as then one does not know how many bytes/characters to receive, and furthermore the payload is binary data anyway, as it is using binary serialization. I probably would pass length as a string if all the data were text, but absolutely not for binary data. Here is the receiver in pseudo-code (that looks like C# but isn't):

int length;
byte[] buffer;
int bufsize;
int lengthRequested;

startReceiver() {
length=0;
bufsize=1024;
buffer=new byte[bufsize];
installReceiveHandler(receiveHandler);
startRead(buffer, 0, 4); // go read 4 bytes holding length
}

receiveHandler() {
if (length==0) {
int len=endRead(); // we hope len==4
length=BitConverter.ToInt32(buffer);
} else {
int len=endRead();
length-=len;
// there now are len data bytes in buffer, process them
...
}
if (length<=0) {
// we're all done, must signal caller somehow
...
} else {
// we need more data, still missing length bytes
lengthRequested=length;
if (lengthRequested>bufsize) lengthRequested=bufsize;
startRead(buffer, 0, lengthRequested);
}
}

Warning: not included are error handling, timeouts, etc. :)

Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles] Nil Volentibus Arduum

Please use <PRE> tags for code snippets, they preserve indentation, and improve readability.

[Jacob D Dixon]

Ok I fixed it but still have a little problem: Agent Log:

Sent notification to server that we were about to send 20230 bytes
Transferred the following actions to server: SERVICES,

Server Log:

[12/2/2010 9:48:56 AM]: -- 192.168.1.163:21064 has connected to the server
[12/2/2010 9:48:56 AM]: 192.168.1.163:21064 is about to send 20230 bytes
[12/2/2010 9:48:56 AM]: Finished receiving 8760 bytes from 192.168.1.163:21064
[12/2/2010 9:48:56 AM]: Agent 192.168.1.163:21064 sent back null
[12/2/2010 9:48:56 AM]: Finished sending 118 bytes to 192.168.1.163:21064

So as you can see the agent is sending all the data, but the server is only getting 8760 bytes? Why is the server not reading all of the data? State Object:

    public Socket worker = null;
    public int BufferSize = 4;
    public byte\[\] buffer;
    public MemoryStream ms = new MemoryStream();

Server:

int bytesRead = handler.EndReceive(iar);
if (bytesRead > 0)
{
if (state.BufferSize == 4)
{
state.BufferSize = BitConverter.ToInt32(state.buffer, 0);
state.buffer = new byte[state.BufferSize];

                    Logging.Log(handler.RemoteEndPoint.ToString() + " is about to send " + state.BufferSize.ToString() + " bytes", true);
                    handler.BeginReceive(state.buffer, 0, state.BufferSize, 0,
                        new AsyncCallback(ReadCallback), state);
                }
                else
                {
                    state.ms.Write(state.buffer, 0, bytesRead);

                    Logging.Log("Finished receiving " + state.ms.Length.ToString() + " bytes from " +
                        handler.RemoteEndPoint.ToString(), true);

                    // All data has been received from client
                    state.ms.Seek(0, SeekOrigin.Begin);
                    IFormatter formatter = new BinaryFormatter();
                    object receivedObject = null;

                    try
                    {
                        receivedObject = formatter.Deserialize(state.ms);
                    }
                    catch (SerializationException se)
                    {
                        Logging.Log("Object passed was not capable of being deserialized. Error: " + se.ToString(), false);

[Luc Pattyn]

your client code seems OK; I do have a few comments on the server side: 1. you never have shown the code that launches the first read, i.e. the BeginReceive that sits outside any callback. 2. I suggested you first receive the length (4 bytes), then one or more large amounts of data; my pseudo-code was using the same buffer for all of these, so it had to set length to 4 initially, and to the actual buffer size later on. I don't see your code do that. 3. you could always (temporarily) add more logging to better grasp what is going on. 4. I suggest your first test is with a very small data set, so it fits in one receive buffer. Only when that works you should test larger amounts of data. :)

Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles] Nil Volentibus Arduum

Please use <PRE> tags for code snippets, they preserve indentation, improve readability, and make me actually look at them.

[Jacob D Dixon]

Ahhhhh ok I see what you are doing now.. but if I'm calling BeginReceive again within the Callback then I don't think that will work. Because the first thing you are doing is checking if Length == 0. There are two times this will happen... for the first BeginReceive and also when we read all the bytes (because we know we are done reading when Length == 0). I get the idea.. unless I'm missing smoething.. changing it now

[Luc Pattyn]

you kick off the receiving actions outside the callback once, then continue the process by issuing another receive inside the callback until you had enough. And length holds the amount still to be received, so once you got it all, you don't issue a new beginreceive, so the callback isn't called any more, the only time it is called with length zero is at the start of a conversation. :)

Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles] Nil Volentibus Arduum

Please use <PRE> tags for code snippets, they preserve indentation, improve readability, and make me actually look at them.

[Jacob D Dixon]

Ah yeah I got it now. Luc I just want to thank you for your help. Here is what I have now and it seems to be working well:

int bytesRead = handler.EndReceive(iar);
if (bytesRead > 0)
{
if (state.Length == 0)
{
state.Length = BitConverter.ToInt32(state.buffer, 0);

                    Logging.Log(handler.RemoteEndPoint.ToString() + " is about to send " + state.Length.ToString() + " bytes", true);
                    handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
                        new AsyncCallback(ReadCallback), state);
                }
                else
                {
                    state.Length -= bytesRead;
                    state.ms.Write(state.buffer, 0, bytesRead);

                    if (state.Length > 0)
                        handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
                            new AsyncCallback(ReadCallback), state);
                    else
                    {
                        Logging.Log("Finished receiving " + state.ms.Length.ToString() + " bytes from " +
                                        handler.RemoteEndPoint.ToString(), true);

                        state.ms.Seek(0, SeekOrigin.Begin);
                        IFormatter formatter = new BinaryFormatter();
                        object receivedObject = null;

                        try
                        {
                            receivedObject = formatter.Deserialize(state.ms);

[Luc Pattyn]

Hi Jacob, you're welcome. Glad you got it working reliably. BTW: You could still reduce your code by sharing the read-more stuff, look again at my pseudo-code[^]. :)

Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles] Nil Volentibus Arduum

Please use <PRE> tags for code snippets, they preserve indentation, improve readability, and make me actually look at them.