Strange Embedded Problem
-
I write embedded code for devices that don’t look like computers. A LITTLE BACKGROUND My client is in the Los Angeles, California area. I live in the Silicon Valley area just south of San Francisco Bay. It takes 6 hours driving to go between the two locations. (More if LA traffic is its usual horrible mess.) If all goes well I will never meet my client face to face. I will, however, see his signature on a check. I am replacing the software that controls a pellet stove. Once the stove is fired up and running I use a timer to be forgiving about accidental pressing of stop or other non run button. If the stove is out of run mode for >60 seconds the stove will restart when a run mode button is pressed. Fewer than 60 seconds and I simply return to the already up and running mode. Part of restarting - if I need to restart - is saving the current temperature + 15C as the target temperature. Another part is turning on the 350-Watt heater used to ignite pellets. A STRANGE PROBLEM With the controller code all but done a client employee started a stove using my code. It worked well until the relay used to control the heater started going crazy. He tried the existing code that I am replacing. No chatter. Back to my code. No chatter. I spent the weekend staring at my code trying to find some kind of stupid bug. On Monday the owner of the company fired up a stove with my code and finally got the problem to reproduce. The heater relay chattered randomly about 10 clicks in 20 seconds. With sometimes 5 clicks in 2 seconds. WTF! There is simply NO WAY my code can do that! A DESIGN ISSUE There were two problems. One was a bit of a software design issue the other was hardware. But any fix, quite naturally, has to be software. The design issue is related to 95C and up is over temperature. When that point is reached I stop the pellet (fuel) feed motor and put the fan into a medium speed per directions from the owner. When I stop feeding pellets I stop resetting the 60-second timer that forgives accidental pressings of the stop button. Thus 60 seconds after over temperature is reached the stove is willing to startup again as soon as we drop to 94C. Ignore the fact that 94C + 15 degrees is well into over temperature territory. HARDWARE Consider what causes the heater relay to chatter. My guess (and I’ve been doing this a long time) is that the 350 Watt heater changes the analog to digital reference voltage or the supply voltage to the LM35 temperature sensors. That hardw
-
I write embedded code for devices that don’t look like computers. A LITTLE BACKGROUND My client is in the Los Angeles, California area. I live in the Silicon Valley area just south of San Francisco Bay. It takes 6 hours driving to go between the two locations. (More if LA traffic is its usual horrible mess.) If all goes well I will never meet my client face to face. I will, however, see his signature on a check. I am replacing the software that controls a pellet stove. Once the stove is fired up and running I use a timer to be forgiving about accidental pressing of stop or other non run button. If the stove is out of run mode for >60 seconds the stove will restart when a run mode button is pressed. Fewer than 60 seconds and I simply return to the already up and running mode. Part of restarting - if I need to restart - is saving the current temperature + 15C as the target temperature. Another part is turning on the 350-Watt heater used to ignite pellets. A STRANGE PROBLEM With the controller code all but done a client employee started a stove using my code. It worked well until the relay used to control the heater started going crazy. He tried the existing code that I am replacing. No chatter. Back to my code. No chatter. I spent the weekend staring at my code trying to find some kind of stupid bug. On Monday the owner of the company fired up a stove with my code and finally got the problem to reproduce. The heater relay chattered randomly about 10 clicks in 20 seconds. With sometimes 5 clicks in 2 seconds. WTF! There is simply NO WAY my code can do that! A DESIGN ISSUE There were two problems. One was a bit of a software design issue the other was hardware. But any fix, quite naturally, has to be software. The design issue is related to 95C and up is over temperature. When that point is reached I stop the pellet (fuel) feed motor and put the fan into a medium speed per directions from the owner. When I stop feeding pellets I stop resetting the 60-second timer that forgives accidental pressings of the stop button. Thus 60 seconds after over temperature is reached the stove is willing to startup again as soon as we drop to 94C. Ignore the fact that 94C + 15 degrees is well into over temperature territory. HARDWARE Consider what causes the heater relay to chatter. My guess (and I’ve been doing this a long time) is that the 350 Watt heater changes the analog to digital reference voltage or the supply voltage to the LM35 temperature sensors. That hardw
So you see, software is not all about zeroes and ones, on and off. As soon as it interfaces to the real world you have to work with 'almost on', and 'nearly one'. Using sensitive analogue circuits like the LM35 near large electrical devices like motors and heaters is always tricky. Actually I think it would be better to use a digital output temperature sensor (I think Dallas makes them) That would free you from long line interference. And of course the best would be to use real industrial type temperature sensors with a 4-20mA (or Hart) interface.
Cees Meijer Software / Hardware Engineer Qmetrix BV C-Scope Blog C-Scope Website
-
So you see, software is not all about zeroes and ones, on and off. As soon as it interfaces to the real world you have to work with 'almost on', and 'nearly one'. Using sensitive analogue circuits like the LM35 near large electrical devices like motors and heaters is always tricky. Actually I think it would be better to use a digital output temperature sensor (I think Dallas makes them) That would free you from long line interference. And of course the best would be to use real industrial type temperature sensors with a 4-20mA (or Hart) interface.
Cees Meijer Software / Hardware Engineer Qmetrix BV C-Scope Blog C-Scope Website
I agree with Cees Meijer 100%. The world is analog. Disk drive recording / read channels are analog. I have a Silicon Valley hardware engineer friend who is VERY fussy about analog circuits. Unless reminded of cost and actual requirements he fails to consider the concept of “good enough.” Had he designed the PCB and specified the sensors, this totally unintended feedback would never have happened. But the only thing I could change in my client’s existing product was the controller code. Indeed, I would rather use a microcontroller with interrupts and more than a 2 level stack. When I agreed to do this code I knew what the PIC16C57 (OTP version) could and could not do. I bought PIC16F57 (flash version) parts and added wires to the existing PCB so I could flash in place. Better sensors would cost more and every dollar the product does not cost to make is important. A redesign of the 15-year-old PCB is certainly called for. Example: their PCB uses two serial load parallel output parts with R2R ladders with individual resistors to set analog voltages for motor control. Modern components would go a long way to pull cost out of the product. And it would be easier to write the controller code. I asked for a complete description of what the microcontroller would see and what it was supposed to do about what it saw. The company owner did not know that the only time the LEDs next to the user mode selection buttons would flash is when the pellet heater was turned on. It has been a slow process.
-
I write embedded code for devices that don’t look like computers. A LITTLE BACKGROUND My client is in the Los Angeles, California area. I live in the Silicon Valley area just south of San Francisco Bay. It takes 6 hours driving to go between the two locations. (More if LA traffic is its usual horrible mess.) If all goes well I will never meet my client face to face. I will, however, see his signature on a check. I am replacing the software that controls a pellet stove. Once the stove is fired up and running I use a timer to be forgiving about accidental pressing of stop or other non run button. If the stove is out of run mode for >60 seconds the stove will restart when a run mode button is pressed. Fewer than 60 seconds and I simply return to the already up and running mode. Part of restarting - if I need to restart - is saving the current temperature + 15C as the target temperature. Another part is turning on the 350-Watt heater used to ignite pellets. A STRANGE PROBLEM With the controller code all but done a client employee started a stove using my code. It worked well until the relay used to control the heater started going crazy. He tried the existing code that I am replacing. No chatter. Back to my code. No chatter. I spent the weekend staring at my code trying to find some kind of stupid bug. On Monday the owner of the company fired up a stove with my code and finally got the problem to reproduce. The heater relay chattered randomly about 10 clicks in 20 seconds. With sometimes 5 clicks in 2 seconds. WTF! There is simply NO WAY my code can do that! A DESIGN ISSUE There were two problems. One was a bit of a software design issue the other was hardware. But any fix, quite naturally, has to be software. The design issue is related to 95C and up is over temperature. When that point is reached I stop the pellet (fuel) feed motor and put the fan into a medium speed per directions from the owner. When I stop feeding pellets I stop resetting the 60-second timer that forgives accidental pressings of the stop button. Thus 60 seconds after over temperature is reached the stove is willing to startup again as soon as we drop to 94C. Ignore the fact that 94C + 15 degrees is well into over temperature territory. HARDWARE Consider what causes the heater relay to chatter. My guess (and I’ve been doing this a long time) is that the 350 Watt heater changes the analog to digital reference voltage or the supply voltage to the LM35 temperature sensors. That hardw
PICguy wrote:
There were two problems. One was a bit of a software design issue the other was hardware. But any fix, quite naturally, has to be software. The design issue is related to 95C and up is over temperature. When that point is reached I stop the pellet (fuel) feed motor and put the fan into a medium speed per directions from the owner. When I stop feeding pellets I stop resetting the 60-second timer that forgives accidental pressings of the stop button. Thus 60 seconds after over temperature is reached the stove is willing to startup again as soon as we drop to 94C.
PICguy wrote:
Consider what causes the heater relay to chatter. My guess (and I’ve been doing this a long time) is that the 350 Watt heater changes the analog to digital reference voltage or the supply voltage to the LM35 temperature sensors. That hardware issue makes an apparent temperature rise from 94C to 95C. The heater relay opens and the temperature appears to drop to 94C and launch starts all over.
Could you implement the same solution as in better ADCs and simply use a larger 'indeterminate area' to avoid flopping. Ie turns off at 95, won't turn back on again until 93.
-- CleaKO The sad part about this instance is that none of the users ever said anything [about the problem]. Pete O`Hanlon Doesn't that just tell you everything you need to know about users?
-
PICguy wrote:
There were two problems. One was a bit of a software design issue the other was hardware. But any fix, quite naturally, has to be software. The design issue is related to 95C and up is over temperature. When that point is reached I stop the pellet (fuel) feed motor and put the fan into a medium speed per directions from the owner. When I stop feeding pellets I stop resetting the 60-second timer that forgives accidental pressings of the stop button. Thus 60 seconds after over temperature is reached the stove is willing to startup again as soon as we drop to 94C.
PICguy wrote:
Consider what causes the heater relay to chatter. My guess (and I’ve been doing this a long time) is that the 350 Watt heater changes the analog to digital reference voltage or the supply voltage to the LM35 temperature sensors. That hardware issue makes an apparent temperature rise from 94C to 95C. The heater relay opens and the temperature appears to drop to 94C and launch starts all over.
Could you implement the same solution as in better ADCs and simply use a larger 'indeterminate area' to avoid flopping. Ie turns off at 95, won't turn back on again until 93.
-- CleaKO The sad part about this instance is that none of the users ever said anything [about the problem]. Pete O`Hanlon Doesn't that just tell you everything you need to know about users?
This hysteresis would prevent rapid chattering. But restart would begin from 93C. This would feed pellets at a high rate into an already hot burn pot. The auto-start heater would turn on but given that the fire is still burning the heater is a nop. Once the stove reached 95C again the pellet feed would stop and restart would be aborted. Restart would start again at 93C. The stove would be running hot with an improper fuel / air mixture. Not good. My fix (being tested as I write this) is to pause the “we don’t need to restart timer” when the stove is in an overheat situation. Thus when the temperature drops below the official definition of overheat the stove resumes normal operation.
-
This hysteresis would prevent rapid chattering. But restart would begin from 93C. This would feed pellets at a high rate into an already hot burn pot. The auto-start heater would turn on but given that the fire is still burning the heater is a nop. Once the stove reached 95C again the pellet feed would stop and restart would be aborted. Restart would start again at 93C. The stove would be running hot with an improper fuel / air mixture. Not good. My fix (being tested as I write this) is to pause the “we don’t need to restart timer” when the stove is in an overheat situation. Thus when the temperature drops below the official definition of overheat the stove resumes normal operation.
I don't know dude... I'd bet your solution works, but it sounds like you're just acheiving the same thing as a larger deadband using a timer instead of your sensor.
Your relay chatter sounds like a classical deadband problem, where you're using a binary-valued actuator to control a continuous-valued process. There's a direct tradeoff between tolerable following error in your process variable and the duty-cycle of your actuator.
But you've been doing this a long time, so I'm probably not telling you anything you don't already know.