What do you do?
-
I'm only looking for general advice with this question, that's why I posted it in the lounge. Please accept my groveling apologies if this is not appropriate for the lounge. When your app is having mysterious crashes on the client's machine, and you cannot replicate them on your development machine, what on Earth is there to do about it? Assuming it's not an option to install Visual Studio onto the client machine, and the fact that the crashes are random and infrequent enough to make a remote debug session impractical, what strategy can be employed?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
Logging, logging, and more logging. In my main product (see .sig) I put in lots of logging just in case some critical part blows up. And since the app deals with IE and network communication - two areas that are susceptible to errors - it's really come in handy many times. We can ask the customer to send us a log (the log is encrypted, but they can click a button and it's zipped up into a CAB file on their desktop that they mail us) and many times that gives us enough info to provide a fix, or at least an explanation and a workaround. It does take extra work to do the logging, but it's like comments - if you write the logging at the same time as the "real" code, it ain't bad.
--Mike-- Visual C++ MVP :cool: LINKS~! Ericahist | PimpFish | CP SearchBar v3.0 | C++ Forum FAQ Ford, what's this fish doing in my ear?
-
I'm only looking for general advice with this question, that's why I posted it in the lounge. Please accept my groveling apologies if this is not appropriate for the lounge. When your app is having mysterious crashes on the client's machine, and you cannot replicate them on your development machine, what on Earth is there to do about it? Assuming it's not an option to install Visual Studio onto the client machine, and the fact that the crashes are random and infrequent enough to make a remote debug session impractical, what strategy can be employed?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
Yeah, as Michael said it's pretty much something you have to plan for in advance. Our app has a debug mode that can be set by the end user in the config file and it will log everything that goes on of any consequence. I remember going through the phase your going through many years ago, once you have a very large user base these kinds of things are much easier to diagnose and after a few years inevitably they are something oddball on the computer that is failing. In a very serious case where it's important and you really need to get it going I would not at all hesitate to arrange with the end user to have a courier pick up the computer and bring it back to your shop, that's not out of the question in some situations. If you don't have serious logging in place I would not trust at all what the client is telling you, not because they are malicious necessarily, they just don't have an eye for detail like a programmer does and often miss out on critical things when relating to you what's happening. This is where you should consider something like CoPilot[^] It's dead easy to set up a session, nothing special is needed and you can walk through it with them and see exactly what's happening as well you can examine the machine in detail for yourself. We use that product all the time for sticky situations where it's difficult to communicate with the end user or they require extra help to set something up.
"110%" - it's the new 70%
-
I'm only looking for general advice with this question, that's why I posted it in the lounge. Please accept my groveling apologies if this is not appropriate for the lounge. When your app is having mysterious crashes on the client's machine, and you cannot replicate them on your development machine, what on Earth is there to do about it? Assuming it's not an option to install Visual Studio onto the client machine, and the fact that the crashes are random and infrequent enough to make a remote debug session impractical, what strategy can be employed?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
The old standby: LUser Error.
¡El diablo está en mis pantalones! ¡Mire, mire! Real Mentats use only 100% pure, unfooled around with Sapho Juice(tm)! SELECT * FROM User WHERE Clue > 0 0 rows returned Save an Orange - Use the VCF! VCF Blog
-
I'm only looking for general advice with this question, that's why I posted it in the lounge. Please accept my groveling apologies if this is not appropriate for the lounge. When your app is having mysterious crashes on the client's machine, and you cannot replicate them on your development machine, what on Earth is there to do about it? Assuming it's not an option to install Visual Studio onto the client machine, and the fact that the crashes are random and infrequent enough to make a remote debug session impractical, what strategy can be employed?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
You must rely on mini-dumps. They have worked great for me for last 6 years. I hope that you build the release module with symbols and that you store the symbols for each release. Once you get the mini-dump back you can analyze it on the development machine. You can also rely on logging, but I have not found that as effective as mini-dumps. In fact all my applications generate mini-dumps when they crash and I have a special setting to generate mini-dump with full memory. I think XP and Vista automatically generate a mini-dump file. All you need to do is to have symbols ready. But I need to verify this. I am planning on an extensive article that relates to mini-dumps, symbol servers and remote debugging. Your post motivates me to think that I need to start working on it. In the mean time look at this excellent article: http://www.codeproject.com/debug/postmortemdebug_standalone1.asp[^]
-
Software problems: Burn incense. Sacrifice chickens. Hardware problems: Burn incense. Sacrifice goats. Management problems: Burn incense. Sacrifice managers.
Author of The Career Programmer and Unite the Tribes www.PracticalStrategyConsulting.com
Ok, that get's a 5...
-
You must rely on mini-dumps. They have worked great for me for last 6 years. I hope that you build the release module with symbols and that you store the symbols for each release. Once you get the mini-dump back you can analyze it on the development machine. You can also rely on logging, but I have not found that as effective as mini-dumps. In fact all my applications generate mini-dumps when they crash and I have a special setting to generate mini-dump with full memory. I think XP and Vista automatically generate a mini-dump file. All you need to do is to have symbols ready. But I need to verify this. I am planning on an extensive article that relates to mini-dumps, symbol servers and remote debugging. Your post motivates me to think that I need to start working on it. In the mean time look at this excellent article: http://www.codeproject.com/debug/postmortemdebug_standalone1.asp[^]
Thank you very much for this information! I think this is the course I will follow. Does the binary that I ship to the client need to have symbols for the dump to work correctly, or do I just need the symbols on my machine?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
-
Logging, logging, and more logging. In my main product (see .sig) I put in lots of logging just in case some critical part blows up. And since the app deals with IE and network communication - two areas that are susceptible to errors - it's really come in handy many times. We can ask the customer to send us a log (the log is encrypted, but they can click a button and it's zipped up into a CAB file on their desktop that they mail us) and many times that gives us enough info to provide a fix, or at least an explanation and a workaround. It does take extra work to do the logging, but it's like comments - if you write the logging at the same time as the "real" code, it ain't bad.
--Mike-- Visual C++ MVP :cool: LINKS~! Ericahist | PimpFish | CP SearchBar v3.0 | C++ Forum FAQ Ford, what's this fish doing in my ear?
Thanks for your advice. I will see about implementing some logging.
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
-
Thank you very much for this information! I think this is the course I will follow. Does the binary that I ship to the client need to have symbols for the dump to work correctly, or do I just need the symbols on my machine?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
You just need to have symbols on your machine. You don't need to ship the symbols. Using symbols people can reverse engineer your application.:)
-
Yeah, as Michael said it's pretty much something you have to plan for in advance. Our app has a debug mode that can be set by the end user in the config file and it will log everything that goes on of any consequence. I remember going through the phase your going through many years ago, once you have a very large user base these kinds of things are much easier to diagnose and after a few years inevitably they are something oddball on the computer that is failing. In a very serious case where it's important and you really need to get it going I would not at all hesitate to arrange with the end user to have a courier pick up the computer and bring it back to your shop, that's not out of the question in some situations. If you don't have serious logging in place I would not trust at all what the client is telling you, not because they are malicious necessarily, they just don't have an eye for detail like a programmer does and often miss out on critical things when relating to you what's happening. This is where you should consider something like CoPilot[^] It's dead easy to set up a session, nothing special is needed and you can walk through it with them and see exactly what's happening as well you can examine the machine in detail for yourself. We use that product all the time for sticky situations where it's difficult to communicate with the end user or they require extra help to set something up.
"110%" - it's the new 70%
Thanks for your excellent post. The client and I do make extensive use of Remote Assistance, but the crashes are intermittent, and we can't be connected all day long.
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
-
I'm only looking for general advice with this question, that's why I posted it in the lounge. Please accept my groveling apologies if this is not appropriate for the lounge. When your app is having mysterious crashes on the client's machine, and you cannot replicate them on your development machine, what on Earth is there to do about it? Assuming it's not an option to install Visual Studio onto the client machine, and the fact that the crashes are random and infrequent enough to make a remote debug session impractical, what strategy can be employed?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
Simple. You need to utilize your inner psychic debugging skills. ;)
Tech, life, family, faith: Give me a visit. I'm currently blogging about: Virginia Tech Shootings, Guns, and Politics The apostle Paul, modernly speaking: Epistles of Paul Judah Himango
-
I'm only looking for general advice with this question, that's why I posted it in the lounge. Please accept my groveling apologies if this is not appropriate for the lounge. When your app is having mysterious crashes on the client's machine, and you cannot replicate them on your development machine, what on Earth is there to do about it? Assuming it's not an option to install Visual Studio onto the client machine, and the fact that the crashes are random and infrequent enough to make a remote debug session impractical, what strategy can be employed?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
I've seen sometimes that the users are not telling the truth, in our programs we log almost everything, and even doing that sometimes the users can catch us by not telling all the truth or by missing some important things because they don't know that they were important. You can: - Remote debug through internet (this will imply modifying the internal VPN's of the customer so this can become a problem if you must do it yourself (I'd never do that by myself)). - Remote connect to that computer in order to be able to see what has happened exactly once the app has crashed or even having an eye on it working in order to see what is the customer doing. In order to do that I've started using the "logmein" software and it works like charm because it goes through the 80 port and in this way you must not modify anything in the routers... - Log everything, it is very important when you deploy an application to be able to control what is happening. - Buy a tent and start living in front of that computer... ;) Hope this helps, and tons of luck finding the cause of your problems! :rose:
-
I'm only looking for general advice with this question, that's why I posted it in the lounge. Please accept my groveling apologies if this is not appropriate for the lounge. When your app is having mysterious crashes on the client's machine, and you cannot replicate them on your development machine, what on Earth is there to do about it? Assuming it's not an option to install Visual Studio onto the client machine, and the fact that the crashes are random and infrequent enough to make a remote debug session impractical, what strategy can be employed?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
-
I've seen sometimes that the users are not telling the truth, in our programs we log almost everything, and even doing that sometimes the users can catch us by not telling all the truth or by missing some important things because they don't know that they were important. You can: - Remote debug through internet (this will imply modifying the internal VPN's of the customer so this can become a problem if you must do it yourself (I'd never do that by myself)). - Remote connect to that computer in order to be able to see what has happened exactly once the app has crashed or even having an eye on it working in order to see what is the customer doing. In order to do that I've started using the "logmein" software and it works like charm because it goes through the 80 port and in this way you must not modify anything in the routers... - Log everything, it is very important when you deploy an application to be able to control what is happening. - Buy a tent and start living in front of that computer... ;) Hope this helps, and tons of luck finding the cause of your problems! :rose:
Thanks, Joan. My customer does not have a VPN, he's just a little guy, so that pretty much rules out remote debugging. Thanks for your knowledge and encouragement!
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
-
I'm only looking for general advice with this question, that's why I posted it in the lounge. Please accept my groveling apologies if this is not appropriate for the lounge. When your app is having mysterious crashes on the client's machine, and you cannot replicate them on your development machine, what on Earth is there to do about it? Assuming it's not an option to install Visual Studio onto the client machine, and the fact that the crashes are random and infrequent enough to make a remote debug session impractical, what strategy can be employed?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
Richie308 wrote:
what on Earth is there to do about it?
In my application is a multithreaded debugging logger as well as an execution startup log. The startup log does not affect real-time only because it is there only during startup. It logs steps in the process of starting the application such that I know where something crashes. But that is only for startup, as I said. I have a multi-threaded UDP transmission thread that when enabled monitors application activity. Think of it as a debugger/profiler internal to the system. When enabled I know what is running, what was last operated on, and what is taking longer than it should. The client has a copy of my debugging monitoring application, though they usually don't know it. When something goes wrong, I walk them through enabling the debug-thread, and starting up the client to monitor it. When it dies I ask for the last few lines. This on top of what others have said.
_________________________ Asu no koto o ieba, tenjo de nezumi ga warau. Talk about things of tomorrow and the mice in the ceiling laugh. (Japanese Proverb)
-
I'm only looking for general advice with this question, that's why I posted it in the lounge. Please accept my groveling apologies if this is not appropriate for the lounge. When your app is having mysterious crashes on the client's machine, and you cannot replicate them on your development machine, what on Earth is there to do about it? Assuming it's not an option to install Visual Studio onto the client machine, and the fact that the crashes are random and infrequent enough to make a remote debug session impractical, what strategy can be employed?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
Any app that I write will deploy with code to write an error log that gives me a stack track and error information if the app crashes. I do sometimes write code that adds to this log file, for deployment to machines where a bug is hard to track down, but, usually, I look and see what the problem is. Writing small functions is generally a good thing, but it really helps here - if there's a way to log the line number of a crash in C#, I don't know it. So, a 100 line function would be hard to debug, a 10 line function, less so :-)
Christian Graus - Microsoft MVP - C++ Metal Musings - Rex and my new metal blog "I am working on a project that will convert a FORTRAN code to corresponding C++ code.I am not aware of FORTRAN syntax" ( spotted in the C++/CLI forum )
-
I'm only looking for general advice with this question, that's why I posted it in the lounge. Please accept my groveling apologies if this is not appropriate for the lounge. When your app is having mysterious crashes on the client's machine, and you cannot replicate them on your development machine, what on Earth is there to do about it? Assuming it's not an option to install Visual Studio onto the client machine, and the fact that the crashes are random and infrequent enough to make a remote debug session impractical, what strategy can be employed?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
I'm a big fan of crash dumps. Here's a good article to get started: http://www.codeproject.com/debug/postmortemdebug_standalone1.asp[^]
-
I'm a big fan of crash dumps. Here's a good article to get started: http://www.codeproject.com/debug/postmortemdebug_standalone1.asp[^]
-
Richie308 wrote:
what on Earth is there to do about it?
In my application is a multithreaded debugging logger as well as an execution startup log. The startup log does not affect real-time only because it is there only during startup. It logs steps in the process of starting the application such that I know where something crashes. But that is only for startup, as I said. I have a multi-threaded UDP transmission thread that when enabled monitors application activity. Think of it as a debugger/profiler internal to the system. When enabled I know what is running, what was last operated on, and what is taking longer than it should. The client has a copy of my debugging monitoring application, though they usually don't know it. When something goes wrong, I walk them through enabling the debug-thread, and starting up the client to monitor it. When it dies I ask for the last few lines. This on top of what others have said.
_________________________ Asu no koto o ieba, tenjo de nezumi ga warau. Talk about things of tomorrow and the mice in the ceiling laugh. (Japanese Proverb)
Did you roll your own debugging monitor, or is it a commercial variety?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
-
Thank you very much for this information! I think this is the course I will follow. Does the binary that I ship to the client need to have symbols for the dump to work correctly, or do I just need the symbols on my machine?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
To add to what Rama's said: do the work to build a symbol repository, and modify your build process to automatically add the symbols and executables for each released build to the repository. Being able to rely on your debugger pulling the correct .exes, .dlls, and .pdbs for whatever dump you're opening is... just really nice. '
----
It appears that everybody is under the impression that I approve of the documentation. You probably also blame Ken Burns for supporting slavery.
--Raymond Chen on MSDN
-
Did you roll your own debugging monitor, or is it a commercial variety?
-------------------------------- "All that is necessary for the forces of evil to win in the world is for enough good men to do nothing" -- Edmund Burke
Richie308 wrote:
Did you roll your own debugging monitor, or is it a commercial variety?
I rolled my own but there are similar ones commercially available. If work doesn't pay for it, and I think I need it, I write it. :)
_________________________ Asu no koto o ieba, tenjo de nezumi ga warau. Talk about things of tomorrow and the mice in the ceiling laugh. (Japanese Proverb)
