Story of a single threaded server
-
About 5 years ago I transferred to the internet department in my company and started working on asp.net applications. In one web service we had to make calls to various legacy code/services. I reused a COM dll I wrote for other purposes and made .net code call this dll. Everything seemed to be fine at the time. The load on the server increased gradually. Now each day we get over 800 mega bytes of log files. The server still works fine. However, when we tried to introduce a new feature recently, the server became extremely slow, a lot of timeouts and thread aborts occurred. We brought in outside help (consultants) and failed to find a solution. In the end, we had to rollback one major release and recode another to get around the problem. Here is the new feature: Suppose a user is trying to get two or three pieces of information from the backend system, instead of retrieving the data whenever it is needed, the new code will get all data on the first request and cache the data in memory. This feature increased (doubled or tripled) the time for each call to the com dll but reduced the total number of calls. I finally remembered that I read an article on "COM Interoperability With .NET" which basically said .NET will make apartment-threaded com component single-threaded. In this case, user requests were handled by multiple .NET threads, but they would have to go through a bottle-neck, the singled-threaded com dll. Note that the com dll itself is not single-threaded, .NET made it so. Based on this knowledge, I suggested reconfiguring the com dll to be run within a com+ server application, there is no code change for this approach. My suggestion was not taken too seriously until the next big release when the same problem occurred (they were trying to do similar things without finding a solution first). My solution was put into production directly without much testing, and it worked like a charm. I felt responsible for these failures because I did not prevent the problem or find a solution right away. Ironically, I was considered "hero of the day" in the office. Life is not always bad. :-D
-
About 5 years ago I transferred to the internet department in my company and started working on asp.net applications. In one web service we had to make calls to various legacy code/services. I reused a COM dll I wrote for other purposes and made .net code call this dll. Everything seemed to be fine at the time. The load on the server increased gradually. Now each day we get over 800 mega bytes of log files. The server still works fine. However, when we tried to introduce a new feature recently, the server became extremely slow, a lot of timeouts and thread aborts occurred. We brought in outside help (consultants) and failed to find a solution. In the end, we had to rollback one major release and recode another to get around the problem. Here is the new feature: Suppose a user is trying to get two or three pieces of information from the backend system, instead of retrieving the data whenever it is needed, the new code will get all data on the first request and cache the data in memory. This feature increased (doubled or tripled) the time for each call to the com dll but reduced the total number of calls. I finally remembered that I read an article on "COM Interoperability With .NET" which basically said .NET will make apartment-threaded com component single-threaded. In this case, user requests were handled by multiple .NET threads, but they would have to go through a bottle-neck, the singled-threaded com dll. Note that the com dll itself is not single-threaded, .NET made it so. Based on this knowledge, I suggested reconfiguring the com dll to be run within a com+ server application, there is no code change for this approach. My suggestion was not taken too seriously until the next big release when the same problem occurred (they were trying to do similar things without finding a solution first). My solution was put into production directly without much testing, and it worked like a charm. I felt responsible for these failures because I did not prevent the problem or find a solution right away. Ironically, I was considered "hero of the day" in the office. Life is not always bad. :-D
Ack! Never, ever change a component's threading model. What's actually happening is COM rules: a component marked ThreadingModel='Apartment' will always run on the thread on which it was created [update: if the thread is an apartment-threaded thread] (NOTE: Incorrect if the creating thread is marked multithreaded, please see message below[^]). The COM runtime will ensure (as long as you behave and always use marshalling functions to pass interface pointers between threads) that the calls end up on the correct thread. Components marked ThreadingModel='Free' always end up in the multithreaded apartment. Over here, threads can call into the component concurrently; it is the component developer's responsibility to manage state so that results are consistent (this typically means placing access to shared resources in a critical section). Components marked ThreadingModel='Both' will end up in either the creating thread's Single-Threaded Apartment or in the Multi-Threaded Apartment depending on the threading model of the thread that created it (STA for a thread that called CoInitializeEx with COINIT_APARTMENTTHREADED or MTA for a thread that called it with COINIT_MULTITHREADED). That means that the component has to work properly in both environments: allowing concurrent callers, and pumping window messages rather than blocking. As a client of a component, you only get to choose how your thread is marked. .NET defaults to marking
COINIT_MULTITHREADEDunless you specify theSTAThreadattribute, or useThread.SetApartmentState. This affects whether marshalling is used. Windows Forms apps normally setSTAThreadin case they need to interop with OLE (e.g. for drag-and-drop). So what was happening was that all your calls to the object were going into the single thread that created the object. The simplest fix would have been to use a different object for each client request, created on demand, rather than going through the single shared object (as COM marshalled the calls back to the thread that created the object). That ensures that you have at least one object per thread and the call doesn't have to cross threads (which is costly in any case). [Update: this did not apply as the original poster's code does use multiple objects, -
Ack! Never, ever change a component's threading model. What's actually happening is COM rules: a component marked ThreadingModel='Apartment' will always run on the thread on which it was created [update: if the thread is an apartment-threaded thread] (NOTE: Incorrect if the creating thread is marked multithreaded, please see message below[^]). The COM runtime will ensure (as long as you behave and always use marshalling functions to pass interface pointers between threads) that the calls end up on the correct thread. Components marked ThreadingModel='Free' always end up in the multithreaded apartment. Over here, threads can call into the component concurrently; it is the component developer's responsibility to manage state so that results are consistent (this typically means placing access to shared resources in a critical section). Components marked ThreadingModel='Both' will end up in either the creating thread's Single-Threaded Apartment or in the Multi-Threaded Apartment depending on the threading model of the thread that created it (STA for a thread that called CoInitializeEx with COINIT_APARTMENTTHREADED or MTA for a thread that called it with COINIT_MULTITHREADED). That means that the component has to work properly in both environments: allowing concurrent callers, and pumping window messages rather than blocking. As a client of a component, you only get to choose how your thread is marked. .NET defaults to marking
COINIT_MULTITHREADEDunless you specify theSTAThreadattribute, or useThread.SetApartmentState. This affects whether marshalling is used. Windows Forms apps normally setSTAThreadin case they need to interop with OLE (e.g. for drag-and-drop). So what was happening was that all your calls to the object were going into the single thread that created the object. The simplest fix would have been to use a different object for each client request, created on demand, rather than going through the single shared object (as COM marshalled the calls back to the thread that created the object). That ensures that you have at least one object per thread and the call doesn't have to cross threads (which is costly in any case). [Update: this did not apply as the original poster's code does use multiple objects,I don't think we changed the threading model of our com component.
Mike Dimmick wrote:
So what was happening was that all your calls to the object were going into the single thread that created the object. The simplest fix would have been to use a different object for each client request, created on demand, rather than going through the single shared object (as COM marshalled the calls back to the thread that created the object). That ensures that you have at least one object per thread and the call doesn't have to cross threads (which is costly in any case).
The .NET code does create a different object for each thread (no sharing of the same object by multiple threads). However, a debugging tool from Microsoft shows all .NET threads on the server are waiting for a signle thread (I am not clear what this tool is, the admins have access to it).
Mike Dimmick wrote:
Moving the object to a COM+ server application shouldn't have worked.
No, that is what made the app work. It is the ONLY change we made. -- modified at 23:16 Saturday 16th June, 2007
-
Ack! Never, ever change a component's threading model. What's actually happening is COM rules: a component marked ThreadingModel='Apartment' will always run on the thread on which it was created [update: if the thread is an apartment-threaded thread] (NOTE: Incorrect if the creating thread is marked multithreaded, please see message below[^]). The COM runtime will ensure (as long as you behave and always use marshalling functions to pass interface pointers between threads) that the calls end up on the correct thread. Components marked ThreadingModel='Free' always end up in the multithreaded apartment. Over here, threads can call into the component concurrently; it is the component developer's responsibility to manage state so that results are consistent (this typically means placing access to shared resources in a critical section). Components marked ThreadingModel='Both' will end up in either the creating thread's Single-Threaded Apartment or in the Multi-Threaded Apartment depending on the threading model of the thread that created it (STA for a thread that called CoInitializeEx with COINIT_APARTMENTTHREADED or MTA for a thread that called it with COINIT_MULTITHREADED). That means that the component has to work properly in both environments: allowing concurrent callers, and pumping window messages rather than blocking. As a client of a component, you only get to choose how your thread is marked. .NET defaults to marking
COINIT_MULTITHREADEDunless you specify theSTAThreadattribute, or useThread.SetApartmentState. This affects whether marshalling is used. Windows Forms apps normally setSTAThreadin case they need to interop with OLE (e.g. for drag-and-drop). So what was happening was that all your calls to the object were going into the single thread that created the object. The simplest fix would have been to use a different object for each client request, created on demand, rather than going through the single shared object (as COM marshalled the calls back to the thread that created the object). That ensures that you have at least one object per thread and the call doesn't have to cross threads (which is costly in any case). [Update: this did not apply as the original poster's code does use multiple objects,I can no longer find the article I read. Here is what I found on msdn.com which may explain the problem we had and why my com+ fix worked: XML Web Services and Apartment-Threaded COM Components XML Web services can call apartment-threaded COM components using the COM interoperability layer, but only if the component is registered as a COM+ library application or if the component is registered in a COM+ server application that resides on a computer that is running Windows 2000 Service Pack 2 or later. ... When you call apartment objects from XML Web services, remember the following points: ... If you want to run your Web services in multi-threaded apartments (MTAs), calls to your apartment object are made by a single-threaded apartment thread. COM+ creates the proxy and stub and also handles the marshalling from the MTA in which the Web service resides to the single-threaded apartment (STA) in which the apartment object resides.
-
I don't think we changed the threading model of our com component.
Mike Dimmick wrote:
So what was happening was that all your calls to the object were going into the single thread that created the object. The simplest fix would have been to use a different object for each client request, created on demand, rather than going through the single shared object (as COM marshalled the calls back to the thread that created the object). That ensures that you have at least one object per thread and the call doesn't have to cross threads (which is costly in any case).
The .NET code does create a different object for each thread (no sharing of the same object by multiple threads). However, a debugging tool from Microsoft shows all .NET threads on the server are waiting for a signle thread (I am not clear what this tool is, the admins have access to it).
Mike Dimmick wrote:
Moving the object to a COM+ server application shouldn't have worked.
No, that is what made the app work. It is the ONLY change we made. -- modified at 23:16 Saturday 16th June, 2007
I had a bit of an experiment and you're right. It appears to be a consequence of the thread creating an object being an MTA thread. A C++ client application does the same. The SDK says: "When a free-threaded apartment (multithreaded apartment model) in a client creates an apartment-threaded in-process server, COM spins up a single-threaded apartment model "host" thread in the client. This host thread will create the object, and the interface pointer will be marshaled back to the client's free-threaded apartment." (source[^]) That documentation does not say that all the MTA threads share the same host thread! Clearly, however, they do. Creating the COM+ server application eliminated the problem as the host process will spin up multiple threads with the correct threading model, one per concurrent client, I think (depending on the attributes of the server application). You can fix it and keep the component loaded as a library in the .NET process by marking the .NET threads as STA. For an application where you create the threads, use Thread.ApartmentState or Thread.SetApartmentState (.NET 2.0). If you're using an ASP.NET web page, set
AspCompat=truein the@Pagedirective. For a web service, you have to set theapartmentThreadingattribute in thehttpRuntimeelement inweb.config. I think.Stability. What an interesting concept. -- Chris Maunder
-
I had a bit of an experiment and you're right. It appears to be a consequence of the thread creating an object being an MTA thread. A C++ client application does the same. The SDK says: "When a free-threaded apartment (multithreaded apartment model) in a client creates an apartment-threaded in-process server, COM spins up a single-threaded apartment model "host" thread in the client. This host thread will create the object, and the interface pointer will be marshaled back to the client's free-threaded apartment." (source[^]) That documentation does not say that all the MTA threads share the same host thread! Clearly, however, they do. Creating the COM+ server application eliminated the problem as the host process will spin up multiple threads with the correct threading model, one per concurrent client, I think (depending on the attributes of the server application). You can fix it and keep the component loaded as a library in the .NET process by marking the .NET threads as STA. For an application where you create the threads, use Thread.ApartmentState or Thread.SetApartmentState (.NET 2.0). If you're using an ASP.NET web page, set
AspCompat=truein the@Pagedirective. For a web service, you have to set theapartmentThreadingattribute in thehttpRuntimeelement inweb.config. I think.Stability. What an interesting concept. -- Chris Maunder
Thanks for your research, now I feel more asured. :-O
-
I can no longer find the article I read. Here is what I found on msdn.com which may explain the problem we had and why my com+ fix worked: XML Web Services and Apartment-Threaded COM Components XML Web services can call apartment-threaded COM components using the COM interoperability layer, but only if the component is registered as a COM+ library application or if the component is registered in a COM+ server application that resides on a computer that is running Windows 2000 Service Pack 2 or later. ... When you call apartment objects from XML Web services, remember the following points: ... If you want to run your Web services in multi-threaded apartments (MTAs), calls to your apartment object are made by a single-threaded apartment thread. COM+ creates the proxy and stub and also handles the marshalling from the MTA in which the Web service resides to the single-threaded apartment (STA) in which the apartment object resides.
Xiangyang Liu wrote:
I can no longer find the article I read...
This might be the article you are thinking of: "Wicked Code: Running ASMX Web Services on STA Threads" by Jeff Prosise, October 2006 MSDN Magazine, at http://msdn.microsoft.com/msdnmag/issues/06/10/WickedCode/[^] Mike
-
Xiangyang Liu wrote:
I can no longer find the article I read...
This might be the article you are thinking of: "Wicked Code: Running ASMX Web Services on STA Threads" by Jeff Prosise, October 2006 MSDN Magazine, at http://msdn.microsoft.com/msdnmag/issues/06/10/WickedCode/[^] Mike
It is not the same article, but it definitely describes the same problem. The solution provided in the article is not that good (not as simple as configuring the exiting com component to be run in COM+). Thanks.
-
It is not the same article, but it definitely describes the same problem. The solution provided in the article is not that good (not as simple as configuring the exiting com component to be run in COM+). Thanks.
It may not be as simple, but it does not have the extra overhead of the interprocess communication required when running the component out-of-process as a COM+ Server Application.
Stability. What an interesting concept. -- Chris Maunder
-
It may not be as simple, but it does not have the extra overhead of the interprocess communication required when running the component out-of-process as a COM+ Server Application.
Stability. What an interesting concept. -- Chris Maunder
Yes, interprocess communication does have some overhead. But in this case the overhead is so insignificant in the overall picture that users do not notice any difference. The admins cannot see any difference in terms of CPU usage, etc.