I've always thought about debugging in terms of tree pruning. Start at the top of the tree and eliminate branches that do not get you to the root of the problem. In order to do that, you must have a mental picture of what the "tree" looks like. I think that is part of the problem with debugging today, the systems are so complex that no single person can put together that mental picture. Think any big software system, Windows, Linux, AEGIS, Space Station, etc. Many moons ago on a super minicomputer I had a hardware problem that occurred once every 2-3 weeks, on a system doing millions of instructions per second. The problem was easily to see (I was working in assembly language); the machine had word & half word instructions. If a half word instruction was followed by a full word instruction the assembler automatically generated a right hand half word NOOP so that the word instruction fell on a word boundary. If everything is working properly, the RH half word NOOP was skipped during instruction executions by the machine. Except that every 2-3 weeks it didn't skip the RH NOOP (PSW was screwed up and pointed to the NOOP). I had a pretty good mental picture of what was going on, wrote a program to exercise I/O devices based on front panel switches and display each time the RH NOOP was executed in the front panel display (this was back in 1973-1974). After fiddling to find the right combination of switches that would cause the counter to increment, the tech came in I ran the program & he fixed the issue in less time that it took to show him the problem. Turned out to be a tap on a delay line that had to be moved to the next tap (5 nsecs).
