Maybe one of the worst bugs I've ever had
-
I wrote a video display driver for an RA8875 display controller. I need it because I use it in one of my commercial projects. Well, it works "fine" (a bit slow, but the controller is slow) except in one demo. I am trying to draw a series of rounded rectangles of random size, random coordinates and random colors. The code is relatively simple and should be innocuous. It also works when it's used as part of a larger routine. But when I simplify the code, remove the big routine, and just run a lines demo followed by the rounded rectangles the whole MCU freezes solid during the 3rd iteration. (Always in the same place but that doesn't matter because even randomization yields the same result boot to boot. There's no real RTC or any sort of non-determinism so freezing in the same spot every time isn't much of a clue in this case). I need to fix this, or at the very least verify it will never happen in my commercial project but I am totally stumped. What a morning.
To err is human. Fortune favors the monsters.
I know sfa about this. But could it be a timing thing, like the simplified code running too fast and overloading the MCU?
Robust Services Core | Software Techniques for Lemmings | Articles
The fox knows many things, but the hedgehog knows one big thing. -
I know sfa about this. But could it be a timing thing, like the simplified code running too fast and overloading the MCU?
Robust Services Core | Software Techniques for Lemmings | Articles
The fox knows many things, but the hedgehog knows one big thing.It doesn't happen when it's part of a larger routine, and basically it's just spinning a loop for 90 iterations drawing rounded rectangles - using
draw::filled_rounded_rectangle<>(). The 3rd time it begins that loop (which itself is part of a larger loop) it freezes about 5 or so iterations in. Not when it's part of the larger subroutine though.To err is human. Fortune favors the monsters.
-
If your "main" can't call your new extracted routine, then you still have dependencies. That's what I use "statics" for (real or simulated).
"Before entering on an understanding, I have meditated for a long time, and have foreseen what might happen. It is not genius which reveals to me suddenly, secretly, what I have to say or to do in a circumstance unexpected by other people; it is reflection, it is meditation." - Napoleon I
it can call it.
To err is human. Fortune favors the monsters.
-
It doesn't happen when it's part of a larger routine, and basically it's just spinning a loop for 90 iterations drawing rounded rectangles - using
draw::filled_rounded_rectangle<>(). The 3rd time it begins that loop (which itself is part of a larger loop) it freezes about 5 or so iterations in. Not when it's part of the larger subroutine though.To err is human. Fortune favors the monsters.
-
I wrote a video display driver for an RA8875 display controller. I need it because I use it in one of my commercial projects. Well, it works "fine" (a bit slow, but the controller is slow) except in one demo. I am trying to draw a series of rounded rectangles of random size, random coordinates and random colors. The code is relatively simple and should be innocuous. It also works when it's used as part of a larger routine. But when I simplify the code, remove the big routine, and just run a lines demo followed by the rounded rectangles the whole MCU freezes solid during the 3rd iteration. (Always in the same place but that doesn't matter because even randomization yields the same result boot to boot. There's no real RTC or any sort of non-determinism so freezing in the same spot every time isn't much of a clue in this case). I need to fix this, or at the very least verify it will never happen in my commercial project but I am totally stumped. What a morning.
To err is human. Fortune favors the monsters.
A long time ago I got assigned to fix an issue that sounds similar. C code, back in the DOS days. The previous person that worked on it couldn't solve it and jacked up the stack size. This allowed it to run longer, but didn't fix the underlying problem. Turns out it was using calling convention (Pascal? It has been a long time...) so that the called routine would clear the stack and not the caller. And the function signature did not exactly match between the two. As a result, two bytes of the stack were not released. After X-iterations through that routine, it ran out of stack space and crashed. After correcting the function signature, app ran. Could also trim back the stack to a tenth its size... Not sure if any of that helps. But maybe it will trigger some out of the box thinking....
-
A long time ago I got assigned to fix an issue that sounds similar. C code, back in the DOS days. The previous person that worked on it couldn't solve it and jacked up the stack size. This allowed it to run longer, but didn't fix the underlying problem. Turns out it was using calling convention (Pascal? It has been a long time...) so that the called routine would clear the stack and not the caller. And the function signature did not exactly match between the two. As a result, two bytes of the stack were not released. After X-iterations through that routine, it ran out of stack space and crashed. After correcting the function signature, app ran. Could also trim back the stack to a tenth its size... Not sure if any of that helps. But maybe it will trigger some out of the box thinking....
I wish it was a stack leak issue. It's not too hard to track those down once you identify them. But yeah, thanks. The ESP32 will notify you if it reboots due to stack overflow vs. something else, so that's helpful. It doesn't freeze though. Or at least, it shouldn't. Arduino disables the watchdog timer I think, but if it didn't, this would be causing a reboot.
To err is human. Fortune favors the monsters.
-
I wish it was a stack leak issue. It's not too hard to track those down once you identify them. But yeah, thanks. The ESP32 will notify you if it reboots due to stack overflow vs. something else, so that's helpful. It doesn't freeze though. Or at least, it shouldn't. Arduino disables the watchdog timer I think, but if it didn't, this would be causing a reboot.
To err is human. Fortune favors the monsters.
Back in the day when I did deal with this was the early 90's, and the diagnostics weren't as good. The only way I found this issue was to trace through the generated assembler for that code. I actually enjoy a challenging bug hunt, it really gets the creative juices going. And solving a hard one is quite satisfying. I don't do much embedded controller work anymore, or I would offer to look at this with you...
-
I wrote a video display driver for an RA8875 display controller. I need it because I use it in one of my commercial projects. Well, it works "fine" (a bit slow, but the controller is slow) except in one demo. I am trying to draw a series of rounded rectangles of random size, random coordinates and random colors. The code is relatively simple and should be innocuous. It also works when it's used as part of a larger routine. But when I simplify the code, remove the big routine, and just run a lines demo followed by the rounded rectangles the whole MCU freezes solid during the 3rd iteration. (Always in the same place but that doesn't matter because even randomization yields the same result boot to boot. There's no real RTC or any sort of non-determinism so freezing in the same spot every time isn't much of a clue in this case). I need to fix this, or at the very least verify it will never happen in my commercial project but I am totally stumped. What a morning.
To err is human. Fortune favors the monsters.
Does it happen with straight rectangles? At the same spot? What happens if you add a fixed offset to each rounded or straight rect? What happens if you change the seed for the pseudo random generator? How is the corner rounding calculated? Try to round only one corner of each rect. Which corner blows? See previous. This might be useful as a primitive. Roundedrectwithcornerfkags
-
Does it happen with straight rectangles? At the same spot? What happens if you add a fixed offset to each rounded or straight rect? What happens if you change the seed for the pseudo random generator? How is the corner rounding calculated? Try to round only one corner of each rect. Which corner blows? See previous. This might be useful as a primitive. Roundedrectwithcornerfkags
It doesn't stall with any other driver. The rounded rect code has nothing to do with the driver, which is completely decoupled from it. I will try changing the seed, but i'm not sure what it will tell me if the problem changes with it.
To err is human. Fortune favors the monsters.
-
It doesn't stall with any other driver. The rounded rect code has nothing to do with the driver, which is completely decoupled from it. I will try changing the seed, but i'm not sure what it will tell me if the problem changes with it.
To err is human. Fortune favors the monsters.
I am thinking there is an issue/edge in underlying trigonometry code that is calculating the rounded corners. Changing the seed might work without issue or blow up on a different iteration. It would help point to an edge case. Also, is it possible that it is blowing on the first pixel of the next figure? Have you tried displaying rounded rects starting at 1 pixel and increasing 1 pixel per iteration? Did random kick out a negative due to sign issues?
