Reversing time, without a flux capacitor
-
My group builds controller applications for our commercial ink-jet printing systems[^]. The apps are structured as a front-end UI application and one or more Windows services. The services can end up running on other machines in the system. Using TCP/IP sockets makes it easy to move services around for hardware or performance reasons. We also use TCP/IP to communicate with a lot of the hardware, so having a common protocol throughout simplifies things.
Software Zen:
delete this; -
Our C# application uses TCP/IP sockets for interprocess communication. I use the
Socketclass asynchronous methods to process received messages, something like this:// start initial receive operation
s.BeginReceive(...,OnReceive);
...
void OnReceive(...)
{
lock(...)
{
// fetch incoming message
...
// start next receive operation
s.BeginReceive(...,OnReceive)
}
// process received message outside of lock to avoid holding it for a long time
...
}I discovered that the
OnReceivewas being called recursively under heavy traffic conditions, resulting in messages being processed in reverse order for brief periods :doh: :sigh:. The solution went something like this:// start initial receive operation
s.BeginReceive(...,OnReceive);
...
void OnReceive(...)
{
lock(...)
{
// fetch incoming message
...
}
// process received message outside of lock to avoid holding it for a long time
...
// start next receive operation
s.BeginReceive(...,OnReceive)
}The time-reversing version has been in the product since 2008, and I only discovered it a week ago. One petard-wedgie, coming up.
Software Zen:
delete this;Ouch!
Bob Dole
The internet is a great way to get on the net.
:doh: 2.0.82.7292 SP6a
-
Out of curiosity, why did you use TCP/IP for the IPC? I have seen a lot of programs use this, but would something like http://www.techrepublic.com/article/use-microsoft-message-queuing-in-c-for-inter-process-communication/6170794[^] not be simpler to use? I have never done IPC so I am asking purely out of curiosity.
=====
\ | /
\|/
|
|-----|
| |
|_ |
_) | /
_) __/_
_) ____
| /|
| / |
| |
|-----|
|===
TCP is a great technology to use when you want to avoid all the pitfalls with same-machine process communication or networking technology. Back when I worked in a financial company our software was a 'hub and worker client' type setup (10 or so client processes), each in a different process and potentially on a different machine, sometimes Windows and sometimes Unix/Linux. Using TCP sockets means you don't really have to worry where the clients are.
-
Our C# application uses TCP/IP sockets for interprocess communication. I use the
Socketclass asynchronous methods to process received messages, something like this:// start initial receive operation
s.BeginReceive(...,OnReceive);
...
void OnReceive(...)
{
lock(...)
{
// fetch incoming message
...
// start next receive operation
s.BeginReceive(...,OnReceive)
}
// process received message outside of lock to avoid holding it for a long time
...
}I discovered that the
OnReceivewas being called recursively under heavy traffic conditions, resulting in messages being processed in reverse order for brief periods :doh: :sigh:. The solution went something like this:// start initial receive operation
s.BeginReceive(...,OnReceive);
...
void OnReceive(...)
{
lock(...)
{
// fetch incoming message
...
}
// process received message outside of lock to avoid holding it for a long time
...
// start next receive operation
s.BeginReceive(...,OnReceive)
}The time-reversing version has been in the product since 2008, and I only discovered it a week ago. One petard-wedgie, coming up.
Software Zen:
delete this;I think I had exactly this bug in my sockets library at one point and it was really confusing trying to catch it. Particularly as I don't have a load-saturation test rig so it was pretty rare for me (generally it would happen when I was streaming a file or something like that).
-
Our C# application uses TCP/IP sockets for interprocess communication. I use the
Socketclass asynchronous methods to process received messages, something like this:// start initial receive operation
s.BeginReceive(...,OnReceive);
...
void OnReceive(...)
{
lock(...)
{
// fetch incoming message
...
// start next receive operation
s.BeginReceive(...,OnReceive)
}
// process received message outside of lock to avoid holding it for a long time
...
}I discovered that the
OnReceivewas being called recursively under heavy traffic conditions, resulting in messages being processed in reverse order for brief periods :doh: :sigh:. The solution went something like this:// start initial receive operation
s.BeginReceive(...,OnReceive);
...
void OnReceive(...)
{
lock(...)
{
// fetch incoming message
...
}
// process received message outside of lock to avoid holding it for a long time
...
// start next receive operation
s.BeginReceive(...,OnReceive)
}The time-reversing version has been in the product since 2008, and I only discovered it a week ago. One petard-wedgie, coming up.
Software Zen:
delete this;It seems that the sample code given should just put the result into a work queue so that the network IO can be overlapping the actual work without getting out of order... Kind of reminds me of a mystery when teaching an Intro to C class. Lab: Open a file, write 5 lines into the file, close the file. Result: The lines were coming out backwards! Cause: The fopen() was being called inside the loop. Line 1 was written (buffered). open the file again Line 2 was written (buffered) etc. On exit, the handles must have been closed in reverse order so Line 5 flushed first, followed by Line 4, ... Line1. Buffered IO strikes again.
-
Our C# application uses TCP/IP sockets for interprocess communication. I use the
Socketclass asynchronous methods to process received messages, something like this:// start initial receive operation
s.BeginReceive(...,OnReceive);
...
void OnReceive(...)
{
lock(...)
{
// fetch incoming message
...
// start next receive operation
s.BeginReceive(...,OnReceive)
}
// process received message outside of lock to avoid holding it for a long time
...
}I discovered that the
OnReceivewas being called recursively under heavy traffic conditions, resulting in messages being processed in reverse order for brief periods :doh: :sigh:. The solution went something like this:// start initial receive operation
s.BeginReceive(...,OnReceive);
...
void OnReceive(...)
{
lock(...)
{
// fetch incoming message
...
}
// process received message outside of lock to avoid holding it for a long time
...
// start next receive operation
s.BeginReceive(...,OnReceive)
}The time-reversing version has been in the product since 2008, and I only discovered it a week ago. One petard-wedgie, coming up.
Software Zen:
delete this;This only works if you have only two calls to OnReceive() happening at once, no? If you have two threads waiting on the lock, what guarantee do you have that the "correct" thread will be the next one to acquire the lock?
Fred the Ranger
-
Our C# application uses TCP/IP sockets for interprocess communication. I use the
Socketclass asynchronous methods to process received messages, something like this:// start initial receive operation
s.BeginReceive(...,OnReceive);
...
void OnReceive(...)
{
lock(...)
{
// fetch incoming message
...
// start next receive operation
s.BeginReceive(...,OnReceive)
}
// process received message outside of lock to avoid holding it for a long time
...
}I discovered that the
OnReceivewas being called recursively under heavy traffic conditions, resulting in messages being processed in reverse order for brief periods :doh: :sigh:. The solution went something like this:// start initial receive operation
s.BeginReceive(...,OnReceive);
...
void OnReceive(...)
{
lock(...)
{
// fetch incoming message
...
}
// process received message outside of lock to avoid holding it for a long time
...
// start next receive operation
s.BeginReceive(...,OnReceive)
}The time-reversing version has been in the product since 2008, and I only discovered it a week ago. One petard-wedgie, coming up.
Software Zen:
delete this;Sorry, but this code violates causality... it won't compile. :)
CEO at: - Rafaga Systems - Para Facturas - Modern Components for the moment...
-
Our C# application uses TCP/IP sockets for interprocess communication. I use the
Socketclass asynchronous methods to process received messages, something like this:// start initial receive operation
s.BeginReceive(...,OnReceive);
...
void OnReceive(...)
{
lock(...)
{
// fetch incoming message
...
// start next receive operation
s.BeginReceive(...,OnReceive)
}
// process received message outside of lock to avoid holding it for a long time
...
}I discovered that the
OnReceivewas being called recursively under heavy traffic conditions, resulting in messages being processed in reverse order for brief periods :doh: :sigh:. The solution went something like this:// start initial receive operation
s.BeginReceive(...,OnReceive);
...
void OnReceive(...)
{
lock(...)
{
// fetch incoming message
...
}
// process received message outside of lock to avoid holding it for a long time
...
// start next receive operation
s.BeginReceive(...,OnReceive)
}The time-reversing version has been in the product since 2008, and I only discovered it a week ago. One petard-wedgie, coming up.
Software Zen:
delete this;That kind of error is funny. For many years a C++ application worked. It used a non initialized pointer but, for many time, it was always null (calling the right initialization). One day it simple decided to get garbage.
-
It seems that the sample code given should just put the result into a work queue so that the network IO can be overlapping the actual work without getting out of order... Kind of reminds me of a mystery when teaching an Intro to C class. Lab: Open a file, write 5 lines into the file, close the file. Result: The lines were coming out backwards! Cause: The fopen() was being called inside the loop. Line 1 was written (buffered). open the file again Line 2 was written (buffered) etc. On exit, the handles must have been closed in reverse order so Line 5 flushed first, followed by Line 4, ... Line1. Buffered IO strikes again.
englebart wrote:
It seems that the sample code given should just put the result into a work queue so that the network IO can be overlapping the actual work without getting out of order...
The sample I provided is a substantial simplification of the actual code, which pretty much does what you suggest.
Software Zen:
delete this; -
This only works if you have only two calls to OnReceive() happening at once, no? If you have two threads waiting on the lock, what guarantee do you have that the "correct" thread will be the next one to acquire the lock?
Fred the Ranger
Member 7931978 wrote:
This only works if you have only two calls to OnReceive() happening at once, no
The key here is the
BeginReceivecall, which tells the .NET socket support to call theOnReceivedelegate asynchronously on a thread pool thread when a message is received. In my initial code that has the bug, I was issuing the secondBeginReceivecall to start receipt of the next message before I'd finished processing the current message. Under high load conditions, this could cause theOnReceivehandler to be called recursively. The actual message handling would then be processed in reverse order as the stack unwound.Member 7931978 wrote:
what guarantee do you have that the "correct" thread will be the next one to acquire the lock?
The lock in this case was protecting the socket connection, so there is no 'correct' thread per se. The .NET socket support can call the asynchronous delegate on any thread it has available. The lock is essentially a critical section.
Software Zen:
delete this; -
Our C# application uses TCP/IP sockets for interprocess communication. I use the
Socketclass asynchronous methods to process received messages, something like this:// start initial receive operation
s.BeginReceive(...,OnReceive);
...
void OnReceive(...)
{
lock(...)
{
// fetch incoming message
...
// start next receive operation
s.BeginReceive(...,OnReceive)
}
// process received message outside of lock to avoid holding it for a long time
...
}I discovered that the
OnReceivewas being called recursively under heavy traffic conditions, resulting in messages being processed in reverse order for brief periods :doh: :sigh:. The solution went something like this:// start initial receive operation
s.BeginReceive(...,OnReceive);
...
void OnReceive(...)
{
lock(...)
{
// fetch incoming message
...
}
// process received message outside of lock to avoid holding it for a long time
...
// start next receive operation
s.BeginReceive(...,OnReceive)
}The time-reversing version has been in the product since 2008, and I only discovered it a week ago. One petard-wedgie, coming up.
Software Zen:
delete this;I assume this had something to do with
-
Our C# application uses TCP/IP sockets for interprocess communication. I use the
Socketclass asynchronous methods to process received messages, something like this:// start initial receive operation
s.BeginReceive(...,OnReceive);
...
void OnReceive(...)
{
lock(...)
{
// fetch incoming message
...
// start next receive operation
s.BeginReceive(...,OnReceive)
}
// process received message outside of lock to avoid holding it for a long time
...
}I discovered that the
OnReceivewas being called recursively under heavy traffic conditions, resulting in messages being processed in reverse order for brief periods :doh: :sigh:. The solution went something like this:// start initial receive operation
s.BeginReceive(...,OnReceive);
...
void OnReceive(...)
{
lock(...)
{
// fetch incoming message
...
}
// process received message outside of lock to avoid holding it for a long time
...
// start next receive operation
s.BeginReceive(...,OnReceive)
}The time-reversing version has been in the product since 2008, and I only discovered it a week ago. One petard-wedgie, coming up.
Software Zen:
delete this;I assume this had something to do with
IAsyncResult.CompletedSynchronously[^] (possibly derived fromSocket.Available[^])? Interesting because under extreme and consistent load this means that the async loop could cause aStackOverflowException(even with your fixed code); I wonder if there is a way to turn off this 'optimization' without having to resort to:private void StartAsyncReadLoop()
{
ReadAsyncLoop(null);
}private void ReadAsyncLoop(IAsyncResult state)
{
try
{
if (state != null)
{
var length = _socket.EndReceive(state);
// ...
}// Ensure that only one level of recursion happens. if (state == null || !state.CompletedSynchronously) \_socket.BeginReceive(..., ReadAsyncLoop, null); else // Maybe it's best to just always do this. Although considering how sockets use the thread pool // it's very likely overkill. ThreadPool.QueueUserWorkItem(\_ => \_socket.BeginReceive(..., ReadAsyncLoop, null)); } catch (SocketException ex) { // ... }}
I wonder if
Socket.UseOnlyOverlappedIO[^] would have an effect on this, considering Overlapped IO is my 'asyncy' than IOCP.He who asks a question is a fool for five minutes. He who does not ask a question remains a fool forever. [Chinese Proverb] Jonathan C Dickinson (C# Software Engineer)