Can not deserialize [modified]

Jacob D Dixon

Error:

[12/1/2010 4:28:54 PM]: Object passed was not capable of being deserialized. Error: System.Runtime.Serialization.SerializationException: End of Stream encountered before parsing was completed.
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream)
at JDMonitoring.Listen.Listener.ReadCallback(IAsyncResult iar) in C:\Users\Jacob\documents\visual studio 2010\Projects\JDMonitoring\JDMonitoring\Listen\Listener.cs:line 99

Ok this is for a Client/Server application. I marked my custom object as serializable. The problem is when I changed it from using an array to a generic list. But before I send the data it is being sent as an Array and not a generic list. CLIENT:

    private static List<Commons> pendingCommons;

    public static Commons\[\] PendingTasks
    {
        get
        {
            if (Reg.Registered == 0)
                SetRegisterDefaults();

            if (pendingCommons == null || pendingCommons.Count < 1)
                SetCheckinDefaults();

            return pendingCommons.ToArray(); // As you can see here before returning it i'm turning it into a Commons\[\] object
        }
        set
        {
            pendingCommons = new List<Commons>(value);
        }
    }

Here is how I'm sending:

Commons[] commons = Tasks.PendingTasks;

// SOCKET CODE
// blah blah

MemoryStream ms = new MemoryStream();
IFormatter formatter = new BinaryFormatter();
formatter.Serialize(ms, commons);

                // Send to server
                s.Send(ms.ToArray());
                Logging.Log("Transferred " + ms.Length.ToString() + " bytes to server containg these actions: " + Tasks.GetActions(), true);
                ms.Dispose();

Ok so I see in my log that it tranferred that to the server so it was able to serialize it. The Tasks.GetActi

jschell

What is the relationship between state.buffer where you put the data and state.ms where you attempt to extract it for deserialization?

Luc Pattyn

Hi Jacob, I don't see anything wrong with your code, however I doubt your observations are correct. Your client code is turning the List into an array right away, so everything that relates to serializing/deserializing and sending/receiving is unchanged, therefore it should continue to work or continue not to work when switching between arrays and lists. You could check the number of bytes transferred; they should be identical for arrays and lists; and of course also at the sending and at the receiving end of the cable. BTW: I don't fully understand the server's if (bytesRead == StateObject.BufferSize) statement; I'm not saying it is wrong, however it looks strange and potentially dangerous as the amount of data depends on the previous receive request, not the current buffer size. :)

Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles] Nil Volentibus Arduum

Please use <PRE> tags for code snippets, they preserve indentation, and improve readability.

Jacob D Dixon

Luc I think you are right about the StateObject.BufferSize. It really was the only way I could think to do it. So I'm receiving the data asynchronously. So as you can tell it is doing a loop passing the StateObject until all data is received. I am reading only 1024 bytes. So lets say I send the server 1048 bytes. The server will read 1024 bytes then call BeginReceive again. Now the server should only receive 24 bytes this next go around due to 1024 + 24 = 1048. So this time it is seeing that it received LESS than 1024 which tells me that we are done receiving all the data. So lets say we send 2048 (1024 + 1024).. then this code will not work because after the first time it will call begin receive and get 1024 bytes, then it will call it again and receive 1024 bytes, then it will call it a third time and not get any bytes (which won't execute the code and the data is lost. Hope I explained / understand that correctly. I'm thinking this could be my issue?

Luc Pattyn

When the data length can vary, there are a couple of ways to cope: 1. prefix the data with the total length; i.e. first insert an (or a long!) value that specifies total payload length. At the receiver, read the length first, then loop your payload read operations. 2. alternatively, implement some more elaborate protocol (maybe data packets of a fixed size with a there-is-more flag. This is slightly harder but pays off when you don't know the length in advance. Your receiver accidentally failing because the payload length was a multiple of 1024 could be very likely up to very unlikely depending on the size of a Common instance; if it is 1024 (or a multiple) then you have the problem for sure; if it is 512B, then you have 50% chance; etc. But all this does not explain your different list/array behavior... :)

Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles] Nil Volentibus Arduum

Please use <PRE> tags for code snippets, they preserve indentation, and improve readability.

modified on Wednesday, December 1, 2010 8:20 PM

Jacob D Dixon · modified on Wednesday, December 1, 2010 8:20 PM

Well let me first try to correct the way I am getting the data. I'm going to have to lookup what you are talking about though since I'm not sure how exactly I need to do that since I am just sending and receiving Serialized data and not just text or whatever. I did leave my laptop at work so I will have to try this tomorrow morning :-(

Luc Pattyn

two ways: have the length field go through the MemoryStream (may be difficult as you don't know the length yet), or write it to the socket separately. So replace s.Send(ms.ToArray()); by

byte[] bytes=ms.ToArray();
int len=bytes.Length;
s.Send(BitConverter.GetBytes(len));
s.Send(bytes);

plus some similar changes at the receiver of course. :)

Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles] Nil Volentibus Arduum

Please use <PRE> tags for code snippets, they preserve indentation, and improve readability.

Jacob D Dixon

Luc, Thank you for the replies. So I do understand how to send the bytes that contain the length of how many bytes are in my custom object. What I am trying to understand is how I will work this into my program. How will I know to try to deserialize or just convert the bytes to integer? On the server the first receive would always be the integer. Once I received all that data I would then call another method to perform another receive but this time deserializing it? Example:

private void FirstReadCallback(IAsyncResult iar)
{
StateObject so = iar as StateObject;
Socket handler = so.workSocket;

        int bytesRead = handler.EndReceive(iar);
        if (bytesRead > 0)
        {
              int myLength = BitConverter.ToInt32(state.buffer);
              state.buffer = new byte\[myLength\];
              
              // Now receive the serialized data
              handler.BeginReceive(state.buffer, 0, state.buffer.Length, 0, new AsyncCallback(SecondReadCallback), state);
        }

}

private void SecondReadCallback(IAsyncResult iar)
{
StateObject state = iar.AsyncState as StateObject;
Socket handler = state.worker;

    int bytesRead = handler.EndReceive(iar);
    if (bytesRead > 0)
    {
         state.ms.Write(state.buffer, 0, bytesRead);
         state.ms.Seek(0, SeekOrigin.Begin);

         IFormatter formatter = new BinaryFormatter();
         object receivedObject = null;

         receivedObject = formatter.Deserialize(state.ms);

         // and so on
    }

}

Luc Pattyn

1. you would need a Length data member in your class, initially zero. 2. I guess you have a BeginReceive() somewhere outside the callback; if so, just make it ask for 4 bytes. That is the easiest way. 3. You can do all EndReceives in a single callback; the first time (when Length is still zero), you need an EndReceive that matches step #2, hope for 4 bytes, and decode them to an integer, to be stored in Length. Use BitConverter.ToInt32(). Later on (when Length isn't zero) do an EndReceive for a full load of data (the maximum of Length and your buffer size), and decrement Length. Keep doing till Length is zero, or timeout or other error occurs. :)

Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles] Nil Volentibus Arduum

Please use <PRE> tags for code snippets, they preserve indentation, and improve readability.

Jacob D Dixon

Luc, Sorry for sounding dumb.. but what is with the 4 bytes? I'm searching google some more for example code of this but what I find is people doing things differently than what I'm trying to do. I see most of the code using Encoding to get the string until a EOF was passed or something. I guess I could do it that way. Pass the bytes as a string with a and cut off the EOF and convert to integer. I do want to try to understand your method, but right now I'm not fully understanding it. Would you mind providing a short example? [Edit] I'm guessing the 4 bytes would be the BitConverter.GetBytes()..... but how certain is 4 bytes would contain the entire length? I did some test and even with doing like a: byte[] b = new byte[4958454] the array length is still only 4. So let me do some test tomorrow and I'll post what I come up with. Thanks for your help

modified on Wednesday, December 1, 2010 11:29 PM

Luc Pattyn · modified on Wednesday, December 1, 2010 11:29 PM

If the length fits an int at the client (i.e. when it is less than 2 GB), all it takes to communicate it is the four bytes that hold the binary representation of that length value. So it will always be exactly four bytes. One could pass that as a string (with variable length) but why do so? it only complicates matters, as then one does not know how many bytes/characters to receive, and furthermore the payload is binary data anyway, as it is using binary serialization. I probably would pass length as a string if all the data were text, but absolutely not for binary data. Here is the receiver in pseudo-code (that looks like C# but isn't):

int length;
byte[] buffer;
int bufsize;
int lengthRequested;

startReceiver() {
length=0;
bufsize=1024;
buffer=new byte[bufsize];
installReceiveHandler(receiveHandler);
startRead(buffer, 0, 4); // go read 4 bytes holding length
}

receiveHandler() {
if (length==0) {
int len=endRead(); // we hope len==4
length=BitConverter.ToInt32(buffer);
} else {
int len=endRead();
length-=len;
// there now are len data bytes in buffer, process them
...
}
if (length<=0) {
// we're all done, must signal caller somehow
...
} else {
// we need more data, still missing length bytes
lengthRequested=length;
if (lengthRequested>bufsize) lengthRequested=bufsize;
startRead(buffer, 0, lengthRequested);
}
}

Warning: not included are error handling, timeouts, etc. :)

Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles] Nil Volentibus Arduum

Please use <PRE> tags for code snippets, they preserve indentation, and improve readability.

Jacob D Dixon

Ok I fixed it but still have a little problem: Agent Log:

Sent notification to server that we were about to send 20230 bytes
Transferred the following actions to server: SERVICES,

Server Log:

[12/2/2010 9:48:56 AM]: -- 192.168.1.163:21064 has connected to the server
[12/2/2010 9:48:56 AM]: 192.168.1.163:21064 is about to send 20230 bytes
[12/2/2010 9:48:56 AM]: Finished receiving 8760 bytes from 192.168.1.163:21064
[12/2/2010 9:48:56 AM]: Agent 192.168.1.163:21064 sent back null
[12/2/2010 9:48:56 AM]: Finished sending 118 bytes to 192.168.1.163:21064

So as you can see the agent is sending all the data, but the server is only getting 8760 bytes? Why is the server not reading all of the data? State Object:

    public Socket worker = null;
    public int BufferSize = 4;
    public byte\[\] buffer;
    public MemoryStream ms = new MemoryStream();

Server:

int bytesRead = handler.EndReceive(iar);
if (bytesRead > 0)
{
if (state.BufferSize == 4)
{
state.BufferSize = BitConverter.ToInt32(state.buffer, 0);
state.buffer = new byte[state.BufferSize];

                    Logging.Log(handler.RemoteEndPoint.ToString() + " is about to send " + state.BufferSize.ToString() + " bytes", true);
                    handler.BeginReceive(state.buffer, 0, state.BufferSize, 0,
                        new AsyncCallback(ReadCallback), state);
                }
                else
                {
                    state.ms.Write(state.buffer, 0, bytesRead);

                    Logging.Log("Finished receiving " + state.ms.Length.ToString() + " bytes from " +
                        handler.RemoteEndPoint.ToString(), true);

                    // All data has been received from client
                    state.ms.Seek(0, SeekOrigin.Begin);
                    IFormatter formatter = new BinaryFormatter();
                    object receivedObject = null;

                    try
                    {
                        receivedObject = formatter.Deserialize(state.ms);
                    }
                    catch (SerializationException se)
                    {
                        Logging.Log("Object passed was not capable of being deserialized. Error: " + se.ToString(), false);

Luc Pattyn

your client code seems OK; I do have a few comments on the server side: 1. you never have shown the code that launches the first read, i.e. the BeginReceive that sits outside any callback. 2. I suggested you first receive the length (4 bytes), then one or more large amounts of data; my pseudo-code was using the same buffer for all of these, so it had to set length to 4 initially, and to the actual buffer size later on. I don't see your code do that. 3. you could always (temporarily) add more logging to better grasp what is going on. 4. I suggest your first test is with a very small data set, so it fits in one receive buffer. Only when that works you should test larger amounts of data. :)

Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles] Nil Volentibus Arduum

Please use <PRE> tags for code snippets, they preserve indentation, improve readability, and make me actually look at them.

Jacob D Dixon

Ahhhhh ok I see what you are doing now.. but if I'm calling BeginReceive again within the Callback then I don't think that will work. Because the first thing you are doing is checking if Length == 0. There are two times this will happen... for the first BeginReceive and also when we read all the bytes (because we know we are done reading when Length == 0). I get the idea.. unless I'm missing smoething.. changing it now

Luc Pattyn

you kick off the receiving actions outside the callback once, then continue the process by issuing another receive inside the callback until you had enough. And length holds the amount still to be received, so once you got it all, you don't issue a new beginreceive, so the callback isn't called any more, the only time it is called with length zero is at the start of a conversation. :)

Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles] Nil Volentibus Arduum

Please use <PRE> tags for code snippets, they preserve indentation, improve readability, and make me actually look at them.

Jacob D Dixon

Ah yeah I got it now. Luc I just want to thank you for your help. Here is what I have now and it seems to be working well:

int bytesRead = handler.EndReceive(iar);
if (bytesRead > 0)
{
if (state.Length == 0)
{
state.Length = BitConverter.ToInt32(state.buffer, 0);

                    Logging.Log(handler.RemoteEndPoint.ToString() + " is about to send " + state.Length.ToString() + " bytes", true);
                    handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
                        new AsyncCallback(ReadCallback), state);
                }
                else
                {
                    state.Length -= bytesRead;
                    state.ms.Write(state.buffer, 0, bytesRead);

                    if (state.Length > 0)
                        handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
                            new AsyncCallback(ReadCallback), state);
                    else
                    {
                        Logging.Log("Finished receiving " + state.ms.Length.ToString() + " bytes from " +
                                        handler.RemoteEndPoint.ToString(), true);

                        state.ms.Seek(0, SeekOrigin.Begin);
                        IFormatter formatter = new BinaryFormatter();
                        object receivedObject = null;

                        try
                        {
                            receivedObject = formatter.Deserialize(state.ms);

Luc Pattyn

Hi Jacob, you're welcome. Glad you got it working reliably. BTW: You could still reduce your code by sharing the read-more stuff, look again at my pseudo-code[^]. :)

Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles] Nil Volentibus Arduum

Please use <PRE> tags for code snippets, they preserve indentation, improve readability, and make me actually look at them.