What do you do?
-
I'm only looking for general advice with this question, that's why I posted it in the lounge. Please accept my groveling apologies if this is not appropriate for the lounge. When your app is having mysterious crashes on the client's machine, and you cannot replicate them on your development machine, what on Earth is there to do about it? Assuming it's not an option to install Visual Studio onto the client machine, and the fact that the crashes are random and infrequent enough to make a remote debug session impractical, what strategy can be employed?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
Have a productive discussion with the user(s). This depends heavily on the communication skills of everyone in the discussion. It also depends on the observational skills of the users (whether they see key symptoms that help diagnose the problem. I have some hidden logging hooks. So I can tell the user how to turn them on and send me the logs. This has actually not been nearly as useful as I hoped. The most efficient method is perhaps unique to my major app. It is a data collection app that runs mostly stand-alone, and usually stores the data in the same directory as the executable. So I can ask a user to zip up the contents of that directory and send it to me. The problem is often a problem in their data that I don't handle gracefully. And sometimes it's a legitimate case I overlooked or misunderstood. Travel to the user with as many debugging aids as I can muster.
-
Software problems: Burn incense. Sacrifice chickens. Hardware problems: Burn incense. Sacrifice goats. Management problems: Burn incense. Sacrifice managers.
Author of The Career Programmer and Unite the Tribes www.PracticalStrategyConsulting.com
The problem with the last part of that advice is that the practice of "sacrificing" denotes the giving up of something of value :~
-
The problem with the last part of that advice is that the practice of "sacrificing" denotes the giving up of something of value :~
Saaaaay... Aren't you in Management now? Boys, fire up the grille! :-D
Author of The Career Programmer and Unite the Tribes www.PracticalStrategyConsulting.com
-
I'm only looking for general advice with this question, that's why I posted it in the lounge. Please accept my groveling apologies if this is not appropriate for the lounge. When your app is having mysterious crashes on the client's machine, and you cannot replicate them on your development machine, what on Earth is there to do about it? Assuming it's not an option to install Visual Studio onto the client machine, and the fact that the crashes are random and infrequent enough to make a remote debug session impractical, what strategy can be employed?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
The incense & sacrifice post is about spot-on for this :) However, I have had marginal luck with two techniques. Borrow the customer's machine and remote debug it locally. Or, use the log file. Borrowing practically never works, even when it's the sales guy's laptop that is having the problems. It is, however, the best way to find and fix the exact bug. For a log file, we built debug logging into our product. Create the magic directory, get a debug log written to a new file in it when run the program. Delete the directory, no log file. This way, we can turn logging on and off for release builds at individual customers by having them create/delete that magic directory. The logging will theoretically change the timing of multithreaded app, but after doing MT programming for about 20 years now, I've found that all such problems are really just latent bugs anyway, so might as well get them fixed before they suprise you with strange failures on some customer's system that you can't easily debug. What do we put in the log? Everything we can think of, but mostly we developers use it as one of our tools for debugging the apps, so there's all sorts of useful debugging stuff like deeper explanation of failures before the app takes a dump, mention of detected events that are not supposed to cause failures but are unexpected, etc. The logs never have quite what we want in them, but if you have an idea of what might have happened based on your mental model of how the code works and the failure report, then they often have just enough clues to confirm or deny your theory... or you put more in for the next release. And when that doesn't work, well, there's always incense and sacrifice.
patbob
-
I'm only looking for general advice with this question, that's why I posted it in the lounge. Please accept my groveling apologies if this is not appropriate for the lounge. When your app is having mysterious crashes on the client's machine, and you cannot replicate them on your development machine, what on Earth is there to do about it? Assuming it's not an option to install Visual Studio onto the client machine, and the fact that the crashes are random and infrequent enough to make a remote debug session impractical, what strategy can be employed?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
I'm not vouching for it (haven't tried it) but there's a "black box" product out there at http://www.identify.com/products/index.php[^]. I have no idea if it will help but a product like this may help. Depending on the product, it's sometimes a good idea for the product to support tracing. Then in cases like this you ask the customer to enable it and send you the log. If you don't already support tracing, you could send the customer a hacked-up version that traces to a log.
-
I'm not vouching for it (haven't tried it) but there's a "black box" product out there at http://www.identify.com/products/index.php[^]. I have no idea if it will help but a product like this may help. Depending on the product, it's sometimes a good idea for the product to support tracing. Then in cases like this you ask the customer to enable it and send you the log. If you don't already support tracing, you could send the customer a hacked-up version that traces to a log.
Oh BTW I just hacked this tidbit together for tracing. I'll just throw it out there in case someone finds it useful. Obviously I was completely hacking... #include class FT { public: FT(const char* pcFunc) : mpcFunc(pcFunc) { printf("Enter %s\n", mpcFunc); } ~FT() { printf("Exit %s\n", mpcFunc); } const char* mpcFunc; }; #define __TR FT oFT(__FUNCSIG__); void function() { __TR } That will spit out enter/exit messages for each function that has the __TR macro.
-
Crash dumps and Remote debugging are two options you have.
Regards, Nish
Nish’s thoughts on MFC, C++/CLI and .NET (my blog)
C++/CLI in ActionFly on your way like an eagle Fly as high as the sun On your wings like an eagle Fly and touch the sun
I will add to that, Logging, Logging, more Logging As well as that, try to identify differences between the development environment and the customers, hardware being number 1. For example I saw one where the developer had tested on a single processor machine, but it would crash on a multi-processor (yes it was a threading issue, but only one where a single move eax is not atomic).
-
Oh BTW I just hacked this tidbit together for tracing. I'll just throw it out there in case someone finds it useful. Obviously I was completely hacking... #include class FT { public: FT(const char* pcFunc) : mpcFunc(pcFunc) { printf("Enter %s\n", mpcFunc); } ~FT() { printf("Exit %s\n", mpcFunc); } const char* mpcFunc; }; #define __TR FT oFT(__FUNCSIG__); void function() { __TR } That will spit out enter/exit messages for each function that has the __TR macro.
Thank you, that is brilliant! I never would have thought of using an automatic variable to automatically record the entrance and exit of a function. The beauty of it is that I don't have to code an exit message for every place in the function that it has a "return" statement. Come to think about it, I'll bet this technique would also work with CRITICAL_SECTION's, so that I don't have to manually code every LeaveCriticalSection() call everywhere the function returns.
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
-
Thank you, that is brilliant! I never would have thought of using an automatic variable to automatically record the entrance and exit of a function. The beauty of it is that I don't have to code an exit message for every place in the function that it has a "return" statement. Come to think about it, I'll bet this technique would also work with CRITICAL_SECTION's, so that I don't have to manually code every LeaveCriticalSection() call everywhere the function returns.
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
Why thank you, and while I don't deny any claims of being brilliant :-) it's a fairly common technique. The CWaitCursor MFC class uses it to turn on and off the wait cursor automatically. It's also handy to free up resources (memory, files, whatever). In an unmanaged language like C++ this is a very important technique if you're using exceptions, since an exception can cause a resource leak otherwise.
-
The problem with the last part of that advice is that the practice of "sacrificing" denotes the giving up of something of value :~
-
I'm only looking for general advice with this question, that's why I posted it in the lounge. Please accept my groveling apologies if this is not appropriate for the lounge. When your app is having mysterious crashes on the client's machine, and you cannot replicate them on your development machine, what on Earth is there to do about it? Assuming it's not an option to install Visual Studio onto the client machine, and the fact that the crashes are random and infrequent enough to make a remote debug session impractical, what strategy can be employed?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
* Limit network connections (try one socket only) * Limit process threads (try with one, not with forty) * Limit multiprocessor execution (set affinity for cpu 0) * Run with your own DLLs, not customer's (already suggested) * Verify DLLS using Sysinternals ProcessExplorer * OutputDebugString printf's even from services * Use those minidumps * Assert with wild abandon * Divide and conquer - disable parts run the rest * Code review with trusted peer * Follow Robbins' advice in Debugging MS .NET 2.0 Applications That's what I do. :)
