Unexpected performance
-
By the way, just curious - what's your test platform (CPU(s)/speed) and compiler version?
Mark Salsbery Microsoft MVP - Visual C++ :java:
Vista B. x64 SP 1 Intel Quad 2.4, 2 GB RAM, VS 2008
Greetings - Gajatko Portable.NET is part of DotGNU, a project to build a complete Free Software replacement for .NET - a system that truly belongs to the developers.
-
I'm not sure what you're expecting here. First, if you were running tests in the debugger, forgetaboutit. Second, comparing times for two sets of loops isn't going to work. Try putting the loop with the new/delete combined below the other two loops and you'll see what I mean. There's alot going on with the heap there, making the results...well, there's no comparison. Third, comparing native to CLR build...the MSIL is better optimized than the debug native code (assuming that's what you were comparing to). Again, no comparison. I would guess, if you figure out the right combination of optimization switches, you MAY be able to make the native version a TINY bit faster than the managed version...maybe... Lastly, don't believe people that say managed code is slower :)
Mark Salsbery Microsoft MVP - Visual C++ :java:
Mark Salsbery wrote:
I'm not sure what you're expecting here.
Not sure what to expect, but definately not this (sounds good).
Mark Salsbery wrote:
First, if you were running tests in the debugger, forgetaboutit.
Hey, I'm not an urgent-codes-plz-gimme-guy and I do not do tests in a debugger!!
Mark Salsbery wrote:
Second, comparing times for two sets of loops isn't going to work.
That was secondary tests. In fact, the first one was important to me (ok, not important. All this talk is neither interesting nor important, but I must figure the thing out).
Mark Salsbery wrote:
the MSIL is better optimized
What could be optimized in such a simple code?
Mark Salsbery wrote:
Lastly, don't believe people that say managed code is slower
Lucky and happy you. The problem is that if you believe something (God, Holy Grail, etc.) you likely believe people as well. Oh sorry I'm getting too soap-boxy.
Greetings - Gajatko Portable.NET is part of DotGNU, a project to build a complete Free Software replacement for .NET - a system that truly belongs to the developers.
-
Mark Salsbery wrote:
I'm not sure what you're expecting here.
Not sure what to expect, but definately not this (sounds good).
Mark Salsbery wrote:
First, if you were running tests in the debugger, forgetaboutit.
Hey, I'm not an urgent-codes-plz-gimme-guy and I do not do tests in a debugger!!
Mark Salsbery wrote:
Second, comparing times for two sets of loops isn't going to work.
That was secondary tests. In fact, the first one was important to me (ok, not important. All this talk is neither interesting nor important, but I must figure the thing out).
Mark Salsbery wrote:
the MSIL is better optimized
What could be optimized in such a simple code?
Mark Salsbery wrote:
Lastly, don't believe people that say managed code is slower
Lucky and happy you. The problem is that if you believe something (God, Holy Grail, etc.) you likely believe people as well. Oh sorry I'm getting too soap-boxy.
Greetings - Gajatko Portable.NET is part of DotGNU, a project to build a complete Free Software replacement for .NET - a system that truly belongs to the developers.
gajatko wrote:
Not sure what to expect, but definately not this
Definitely didn't expect what?
gajatko wrote:
Lucky and happy you.
I have no idea what you mean by that. I didn't state anything about me.
gajatko wrote:
The problem is that if you believe something (God, Holy Grail, etc.) you likely believe people as well.
Not sure what that means either. Just because I believe I'll have another beer doesn't mean I believe people.
Mark Salsbery Microsoft MVP - Visual C++ :java:
-
gajatko wrote:
Not sure what to expect, but definately not this
Definitely didn't expect what?
gajatko wrote:
Lucky and happy you.
I have no idea what you mean by that. I didn't state anything about me.
gajatko wrote:
The problem is that if you believe something (God, Holy Grail, etc.) you likely believe people as well.
Not sure what that means either. Just because I believe I'll have another beer doesn't mean I believe people.
Mark Salsbery Microsoft MVP - Visual C++ :java:
It looks like I reached a maximum humour abstraction level last night. Do not take that serious please. :-O :rolleyes:
Greetings - Gajatko Portable.NET is part of DotGNU, a project to build a complete Free Software replacement for .NET - a system that truly belongs to the developers.
-
It looks like I reached a maximum humour abstraction level last night. Do not take that serious please. :-O :rolleyes:
Greetings - Gajatko Portable.NET is part of DotGNU, a project to build a complete Free Software replacement for .NET - a system that truly belongs to the developers.
No problem :) Cheers :beer:
Mark Salsbery Microsoft MVP - Visual C++ :java:
-
I'm not sure what you're expecting here. First, if you were running tests in the debugger, forgetaboutit. Second, comparing times for two sets of loops isn't going to work. Try putting the loop with the new/delete combined below the other two loops and you'll see what I mean. There's alot going on with the heap there, making the results...well, there's no comparison. Third, comparing native to CLR build...the MSIL is better optimized than the debug native code (assuming that's what you were comparing to). Again, no comparison. I would guess, if you figure out the right combination of optimization switches, you MAY be able to make the native version a TINY bit faster than the managed version...maybe... Lastly, don't believe people that say managed code is slower :)
Mark Salsbery Microsoft MVP - Visual C++ :java:
Mark Salsbery wrote:
Lastly, don't believe people that say managed code is slower
Actually...this is exactly what we're finding. Real world problem: - MFC application, 1 EXE and 5 DLL's. The DLL's do all the computations and are highly optimized C++ code which have been validated for correctness and operation many times in the past decade. And we have spent many late nights finding performance gains - the code is good. The front-end is GUI. We have MFC, STL, ATL technologies incorporated, plus a lot of hand-written code - because STL was too slow. The DLL's implement linear algebra, various geometry calculations, and 1st-order PDE's. There are also simple array/table look-ups. The project is 300,000+ lines code. - we also use ZLIB, GDAL/OGR, CGAL but they aren't used in these tests Test results: - baseline is current release build, test case taks 190seconds. - simply turn on the /clr switch on as many files as possible (following the bouncing ball, trying to do it right) and performance drops to 345 seconds (not all could run managed because of MFC) - retest native release build, it stays the same - concerned we are doing too many native/managed swaps to translate 1 DLL to entirely a C++/CLI classlib, performance drops to 355 seconds. Confused so try it again, 361 seconds - try NGEN on the classlib, performance becomes 349 seconds - try NGEN on the entire project, performance becomes 476 seconds! - back out of changes, start over with the native release build, back around 188 seconds - incorporate the classlib and do MINIMUM required to get things to link back together (to use native as much as possible) and performance is around 244 seconds All other test cases yield similar results. We WANT to move to .Net (to clean up the code, make it easier to maintain and read, etc.), but we need a migratory path where we don't have to rewrite the entire front end out of MFC right away. But these performance #'s just suck. We can't ask the client to live with such poor performance during migration in hopes it will improve when it's all done and running strictly in managed. We have experience with C# in another project but are new in the C++/CLI world. I'd happily discuss this online or offline but 'til then I can't agree with your comment at all Mark! --Rob
-
Mark Salsbery wrote:
Lastly, don't believe people that say managed code is slower
Actually...this is exactly what we're finding. Real world problem: - MFC application, 1 EXE and 5 DLL's. The DLL's do all the computations and are highly optimized C++ code which have been validated for correctness and operation many times in the past decade. And we have spent many late nights finding performance gains - the code is good. The front-end is GUI. We have MFC, STL, ATL technologies incorporated, plus a lot of hand-written code - because STL was too slow. The DLL's implement linear algebra, various geometry calculations, and 1st-order PDE's. There are also simple array/table look-ups. The project is 300,000+ lines code. - we also use ZLIB, GDAL/OGR, CGAL but they aren't used in these tests Test results: - baseline is current release build, test case taks 190seconds. - simply turn on the /clr switch on as many files as possible (following the bouncing ball, trying to do it right) and performance drops to 345 seconds (not all could run managed because of MFC) - retest native release build, it stays the same - concerned we are doing too many native/managed swaps to translate 1 DLL to entirely a C++/CLI classlib, performance drops to 355 seconds. Confused so try it again, 361 seconds - try NGEN on the classlib, performance becomes 349 seconds - try NGEN on the entire project, performance becomes 476 seconds! - back out of changes, start over with the native release build, back around 188 seconds - incorporate the classlib and do MINIMUM required to get things to link back together (to use native as much as possible) and performance is around 244 seconds All other test cases yield similar results. We WANT to move to .Net (to clean up the code, make it easier to maintain and read, etc.), but we need a migratory path where we don't have to rewrite the entire front end out of MFC right away. But these performance #'s just suck. We can't ask the client to live with such poor performance during migration in hopes it will improve when it's all done and running strictly in managed. We have experience with C# in another project but are new in the C++/CLI world. I'd happily discuss this online or offline but 'til then I can't agree with your comment at all Mark! --Rob
Hi Rob, My comment was in the context of the OP. I challenged the OP to optimize his sample to be as fast or faster than the managed version. I also implied he couldn't do it :) Of course highly optimized C++ code is going to be faster - faster than ANY high level language. That's its purpose.
Rob Bryce wrote:
simply turn on the /clr switch on as many files as possible
I certainly wouldn't do that. C++/CLI is meant to be a bridge from native C++ to the managed .NET world. It should be used as that. Use the #pragma managed/#pragma unmanaged macros to control what code gets compiled managed and what stays native. Then you can keep your optimized or performance critical code compiled native. This can be done at the function level! :) Cheers, Mark
Mark Salsbery Microsoft MVP - Visual C++ :java:
-
Mark Salsbery wrote:
Lastly, don't believe people that say managed code is slower
Actually...this is exactly what we're finding. Real world problem: - MFC application, 1 EXE and 5 DLL's. The DLL's do all the computations and are highly optimized C++ code which have been validated for correctness and operation many times in the past decade. And we have spent many late nights finding performance gains - the code is good. The front-end is GUI. We have MFC, STL, ATL technologies incorporated, plus a lot of hand-written code - because STL was too slow. The DLL's implement linear algebra, various geometry calculations, and 1st-order PDE's. There are also simple array/table look-ups. The project is 300,000+ lines code. - we also use ZLIB, GDAL/OGR, CGAL but they aren't used in these tests Test results: - baseline is current release build, test case taks 190seconds. - simply turn on the /clr switch on as many files as possible (following the bouncing ball, trying to do it right) and performance drops to 345 seconds (not all could run managed because of MFC) - retest native release build, it stays the same - concerned we are doing too many native/managed swaps to translate 1 DLL to entirely a C++/CLI classlib, performance drops to 355 seconds. Confused so try it again, 361 seconds - try NGEN on the classlib, performance becomes 349 seconds - try NGEN on the entire project, performance becomes 476 seconds! - back out of changes, start over with the native release build, back around 188 seconds - incorporate the classlib and do MINIMUM required to get things to link back together (to use native as much as possible) and performance is around 244 seconds All other test cases yield similar results. We WANT to move to .Net (to clean up the code, make it easier to maintain and read, etc.), but we need a migratory path where we don't have to rewrite the entire front end out of MFC right away. But these performance #'s just suck. We can't ask the client to live with such poor performance during migration in hopes it will improve when it's all done and running strictly in managed. We have experience with C# in another project but are new in the C++/CLI world. I'd happily discuss this online or offline but 'til then I can't agree with your comment at all Mark! --Rob
Rob Bryce wrote:
We WANT to move to .Net (to clean up the code, make it easier to maintain and read, etc.)
Obviously I don't know your project but that statement certainly raises my eyebrow in general. I am not convinced that the C++ language produces less readable or maintainable code. Furthermore, with the additions of the VS2008 C++ Feature pack including TR1 items, it seems even less likely that the language is a barrier to readable, maintainable and flexible design. In fact I believe the exact opposite is true. Features available only in C++ are the reason e.g., like being able to utilize stack rather than always heap, in my mind simplifies readability. PInvoke is another that comes to mind. X| In any event, I would certainly not agree to migrate to any managed platform based solely on the belief that the resulting code will be intrinsically more readable, maintainable.
led mike
-
Rob Bryce wrote:
We WANT to move to .Net (to clean up the code, make it easier to maintain and read, etc.)
Obviously I don't know your project but that statement certainly raises my eyebrow in general. I am not convinced that the C++ language produces less readable or maintainable code. Furthermore, with the additions of the VS2008 C++ Feature pack including TR1 items, it seems even less likely that the language is a barrier to readable, maintainable and flexible design. In fact I believe the exact opposite is true. Features available only in C++ are the reason e.g., like being able to utilize stack rather than always heap, in my mind simplifies readability. PInvoke is another that comes to mind. X| In any event, I would certainly not agree to migrate to any managed platform based solely on the belief that the resulting code will be intrinsically more readable, maintainable.
led mike
Hi led mike, I believe that my comment needs more clarification. The motivations (maintainable, readable) are 2-fold and can be better defined as: 1.) I (personally) have never found that the IDL/ATL approach to defining/implementing COM object interfaces is particularly "clean". Nor has the ability to inherit from a COM object been "nice". I should have been more clear that the motivation is to use .Net technology instead of COM technology as a motivation to make it easier to deal with object interfaces intended for re-use outside this project. 2.) I'm (personally) probably one of the few that still enjoy some aspects of MFC for UI programming. But even I have noticed that it's more cost-effective to use more modern languages/features (such as C#) than MFC. And that means entering the managed land, at least in part. With respect to generic C/C++, these are still my personal language of preference for general day-to-day programming. I agree with you in that the choice of language doesn't necesssarily make code more/less easier to read. However, that language's technologies that we are (forced to?) use (e.g. ATL, MFC) do impact that legibility. As an aside, I'm doing more management and less programming, and our new staff is being trained (by local universities) to work in managed languages, with little or no exposure to C++. This is a reality that I'm not happy with but have to live with. This may be a problem in the future. --Rob
-
Hi led mike, I believe that my comment needs more clarification. The motivations (maintainable, readable) are 2-fold and can be better defined as: 1.) I (personally) have never found that the IDL/ATL approach to defining/implementing COM object interfaces is particularly "clean". Nor has the ability to inherit from a COM object been "nice". I should have been more clear that the motivation is to use .Net technology instead of COM technology as a motivation to make it easier to deal with object interfaces intended for re-use outside this project. 2.) I'm (personally) probably one of the few that still enjoy some aspects of MFC for UI programming. But even I have noticed that it's more cost-effective to use more modern languages/features (such as C#) than MFC. And that means entering the managed land, at least in part. With respect to generic C/C++, these are still my personal language of preference for general day-to-day programming. I agree with you in that the choice of language doesn't necesssarily make code more/less easier to read. However, that language's technologies that we are (forced to?) use (e.g. ATL, MFC) do impact that legibility. As an aside, I'm doing more management and less programming, and our new staff is being trained (by local universities) to work in managed languages, with little or no exposure to C++. This is a reality that I'm not happy with but have to live with. This may be a problem in the future. --Rob
Thanks Rob. Yes your additional information changes the scenario greatly.
Rob Bryce wrote:
as a motivation to make it easier to deal with object interfaces intended for re-use outside this project.
I agree that COM is inherently difficult. ATL does improve working with COM greatly but can only go so far. However COM is only necessary to support use of components developed in a different language. This is a heavy restriction and not a requirement for every project. Conversely we can choose the opposite restriction, that being, supporting native C++ components only.
led mike
-
Thanks Rob. Yes your additional information changes the scenario greatly.
Rob Bryce wrote:
as a motivation to make it easier to deal with object interfaces intended for re-use outside this project.
I agree that COM is inherently difficult. ATL does improve working with COM greatly but can only go so far. However COM is only necessary to support use of components developed in a different language. This is a heavy restriction and not a requirement for every project. Conversely we can choose the opposite restriction, that being, supporting native C++ components only.
led mike
led mike wrote:
However COM is only necessary to support use of components developed in a different language. This is a heavy restriction and not a requirement for every project.
Yeah, unfortunately, defining the language of choice for projects outside my control, but using the object interfaces that I define, isn't an option - which mean COM or now .Net. Thanks, --Rob
-
Hi Rob, My comment was in the context of the OP. I challenged the OP to optimize his sample to be as fast or faster than the managed version. I also implied he couldn't do it :) Of course highly optimized C++ code is going to be faster - faster than ANY high level language. That's its purpose.
Rob Bryce wrote:
simply turn on the /clr switch on as many files as possible
I certainly wouldn't do that. C++/CLI is meant to be a bridge from native C++ to the managed .NET world. It should be used as that. Use the #pragma managed/#pragma unmanaged macros to control what code gets compiled managed and what stays native. Then you can keep your optimized or performance critical code compiled native. This can be done at the function level! :) Cheers, Mark
Mark Salsbery Microsoft MVP - Visual C++ :java:
Mark Salsbery wrote:
I certainly wouldn't do that. C++/CLI is meant to be a bridge from native C++ to the managed .NET world. It should be used as that. Use the #pragma managed/#pragma unmanaged macros to control what code gets compiled managed and what stays native. Then you can keep your optimized or performance critical code compiled native. This can be done at the function level!
Hi Mark, I took your advice and created a new configuration in the project. Here, everything possible is native C++, and we're only using C++/CLI to provide the interface to one (previously COM) DLL. It was the easiest to use because it was the simplest. Plus it's used through-out the project quite a bit. Calculations are simple and quick. Another way to state this is we did nothing but change the interface technology from COM to .Net/managed on 1 DLL. We observed a performance impact of about 35% by just doing this change! :wtf: So we then put that entire DLL into CLR and the performance did improve - impact was around 23%. So in this instance the cost of switching from native->managed->native out-weighed the cost of executing the DLL code all as managed. But it's still a big hit from native C++. These #'s were surprising, to say the least...but are consistent with some tests we did about a year ago. One advantage of doing this exercise, though, was finidng some compiler options that needed to be changed, which improved the baseline performance (strictly C++). The above performance impact %-ages do not consider those gains, though. :omg: I'm not posting this to beat a dead horse, but am opening up this discussion to look for things we may have missed during our testing.:confused: --Rob
-
Mark Salsbery wrote:
I certainly wouldn't do that. C++/CLI is meant to be a bridge from native C++ to the managed .NET world. It should be used as that. Use the #pragma managed/#pragma unmanaged macros to control what code gets compiled managed and what stays native. Then you can keep your optimized or performance critical code compiled native. This can be done at the function level!
Hi Mark, I took your advice and created a new configuration in the project. Here, everything possible is native C++, and we're only using C++/CLI to provide the interface to one (previously COM) DLL. It was the easiest to use because it was the simplest. Plus it's used through-out the project quite a bit. Calculations are simple and quick. Another way to state this is we did nothing but change the interface technology from COM to .Net/managed on 1 DLL. We observed a performance impact of about 35% by just doing this change! :wtf: So we then put that entire DLL into CLR and the performance did improve - impact was around 23%. So in this instance the cost of switching from native->managed->native out-weighed the cost of executing the DLL code all as managed. But it's still a big hit from native C++. These #'s were surprising, to say the least...but are consistent with some tests we did about a year ago. One advantage of doing this exercise, though, was finidng some compiler options that needed to be changed, which improved the baseline performance (strictly C++). The above performance impact %-ages do not consider those gains, though. :omg: I'm not posting this to beat a dead horse, but am opening up this discussion to look for things we may have missed during our testing.:confused: --Rob
Rob Bryce wrote:
native->managed->native
Managed/native interop is certainly the biggest performance killer. Those percentages still seem high though (you're not testing in the debugger, right? ;)), although I don't use COM (besides interfaces to things like DirectX, DirectShow, etc) so I don't know what that adds to the mix. To get numbers that high, I figure you must be doing some extremely performance critical stuff across native/managed boundaries. If that's the case, eliminating that interop is a necessity. IMO, all managed code gives you is access to the .NET framework. After all, no self-respecting C++ programmer needs a garbage collector :) Given that and what I've read about your situation so far, I would keep everything compiled native and sprinkle in managed where you need something in the .NET framework. When you need to cross the native/managed boundary, keep parameter passing to a minimum - it's that marshaling that's going to hurt performance the most. Mark
Mark Salsbery Microsoft MVP - Visual C++ :java:
