Maker Pro
Maker Pro

weird reboots of embedded pc

Q

Quack

Jan 1, 1970
0
Hi,

This is a verry strange one :)

The system;
A large cabinet (casino slot machine) - made of wood.
-1 x standard pc motherboard (p4 1.5ghz)
-1 x standard ide disk
-1 x 17" flat panel with resistive touchscreen/usb controlled
-1 x key operated switch, connected to the CDS line of the serial port

The problem;
If someone with a slight static charge touches the key to the keyswitch - it
sometimes hard boots. As if the power was cycled.

We tried changing power supplies.
We tried grounding everything, including the keyswitch.
We tried with everything except power disconnected from the motherboard,
still happens.
(thats right, everything disconnected, including the keyswitch!)

It still happens, even when the keyswitch is completely disconnected - the
keyswitch is also mounted in wood so makes absolutely no connection
anywhere - but for some reason, touching it can reboot the machine.. wow.

Also, there are 10 machines, all worked fine for 3 months - this problem
just 'started' one day - on all 10 machines. So we assumed an external
influence caused it.

So, we replace all motherboards with a different brand - the problem
disappeared.
We added a regulator to feed a UPS for _each_ machine.

This ran for a few weeks - and now the problem has returned. Wow - how
weird!.

any ideas ? this is killing me.

Alex.
 
B

Bob Stephens

Jan 1, 1970
0
Hi,

This is a verry strange one :)

The system;
A large cabinet (casino slot machine) - made of wood.
-1 x standard pc motherboard (p4 1.5ghz)
-1 x standard ide disk
-1 x 17" flat panel with resistive touchscreen/usb controlled
-1 x key operated switch, connected to the CDS line of the serial port

The problem;
If someone with a slight static charge touches the key to the keyswitch - it
sometimes hard boots. As if the power was cycled.

We tried changing power supplies.
We tried grounding everything, including the keyswitch.
We tried with everything except power disconnected from the motherboard,
still happens.
(thats right, everything disconnected, including the keyswitch!)

It still happens, even when the keyswitch is completely disconnected - the
keyswitch is also mounted in wood so makes absolutely no connection
anywhere - but for some reason, touching it can reboot the machine.. wow.

Also, there are 10 machines, all worked fine for 3 months - this problem
just 'started' one day - on all 10 machines. So we assumed an external
influence caused it.

So, we replace all motherboards with a different brand - the problem
disappeared.
We added a regulator to feed a UPS for _each_ machine.

This ran for a few weeks - and now the problem has returned. Wow - how
weird!.

any ideas ? this is killing me.

Alex.

I had something similar happen several years ago. I replaced the core
memory on an old mainframe with SRAM. It would run fine for days and then
all of a sudden glitch for a few seconds, then return to normal ops. This
was in a DC8 flight simulator, so whenever the memory got corrupted things
got very dramatic in the cockpit. We tried everything to no avail. Finally
on one of the few times we actually observed it flaking, we had the memory
unit pulled out of the rack and for some test or other we removed the DC
supply from the chassis and by chance set it on top of a manila folder. We
left it like that over the weekend, and on Monday morning the pilots log
showed no problems in 48+ hours. By trial and error we were finally able to
duplicate the problem. When a certain combination of switches, relays and
rotary solenoids were engaged in the cockpit, lots of high frequency noise
was getting interjected onto the chassis ground of the entire system. This
was getting capacitively coupled through the mains input of the DC supply
onto the DC rails. We ended up solving it by a combination of input power
conditioning and insulating the DC supply's chassis from the rack.

If these things are in a casino, I'd assume that the AC mains is pretty
noisy so you might be seeing something similar.

Just a thought...


Bob
 
Q

Quack

Jan 1, 1970
0
Hi Bob,

Thankyou very much for sharing that experience.
Atleast i know im not alone or imagining all this :)
If these things are in a casino, I'd assume that the AC mains is pretty
noisy so you might be seeing something similar.

Yes definantly.
But i installed a voltage regulator to feed a ups for _each_ machine along
with the new motherboards.
Surely this must clean up the power ? :(

Its also not just displaying this problem at random times - once it has
started, it wont go away until the motherboard is replaced with a new one.
So something is cooking :(. slowly. argh.

Alex.
 
R

rein wiehler

Jan 1, 1970
0
Quack said:
Hi,

This is a verry strange one :)

The system;
A large cabinet (casino slot machine) - made of wood.
-1 x standard pc motherboard (p4 1.5ghz)
-1 x standard ide disk
-1 x 17" flat panel with resistive touchscreen/usb controlled
-1 x key operated switch, connected to the CDS line of the serial port

The problem;
If someone with a slight static charge touches the key to the
keyswitch - it sometimes hard boots. As if the power was cycled.

We tried changing power supplies.
We tried grounding everything, including the keyswitch.
We tried with everything except power disconnected from the
motherboard, still happens.
(thats right, everything disconnected, including the keyswitch!)

It still happens, even when the keyswitch is completely disconnected
- the keyswitch is also mounted in wood so makes absolutely no
connection anywhere - but for some reason, touching it can reboot the
machine.. wow.

Also, there are 10 machines, all worked fine for 3 months - this
problem just 'started' one day - on all 10 machines. So we assumed an
external influence caused it.

So, we replace all motherboards with a different brand - the problem
disappeared.
We added a regulator to feed a UPS for each machine.

This ran for a few weeks - and now the problem has returned. Wow -
how weird!.

any ideas ? this is killing me.

Alex.

install a humidifier in the room
 
J

John Woodgate

Jan 1, 1970
0
I read in sci.electronics.design that Quack <[email protected]>
Its also not just displaying this problem at random times - once it has
started, it wont go away until the motherboard is replaced with a new
one. So something is cooking :(. slowly. argh.

Did you try swapping used motherboards between machines? Could the BIOS
be corrupted?

Your UPS trial won't have worked if the problem is noise in the
grounding.

In these cases, brainstorming can sometimes work. Almost no test is 'too
crazy to bother with'.
 
M

Mark Jones

Jan 1, 1970
0
Quack said:
Hi,

This is a verry strange one...


Yikes! This sounds like something from a Steven King novel... :)

Hmm, short of calling in a witch doctor, have you tried shielding
with a faraday cage?

And another idea, you could try putting a clamping zener (or MOV,
neon lamp?) across the lockswitch to prevent any static discharges
into the serial port. Even better, clamp the switch and buffer it with
a schmitt buffer, powered from a heavily bypassed 78L05 or something.
Pulling any fast communication wiring outside of the box sounds like
trouble.

I'd seen very unusual things happen in RS-485 networks because of
ground loop inductance. If you're using shielded cable, make sure only
one end of it is grounded.

As others have said, the solution has got to be a process of
elimination. At least with nothing connected, you can still duplicate
the problem... perhaps touching the switch is adding capacitance to
the system, allowing enough ambient noise to influence operation?

As for ground noise, could a high-power LC tank in the DC ground
supply suppress the HF noise? Never tried it, just speculating.

Lastly... find the reboot trace on the PC, and add a 100k pullup
resistor? :)

Good luck.

-M
 
R

Rich Grise

Jan 1, 1970
0
Hi Bob,

Thankyou very much for sharing that experience.
Atleast i know im not alone or imagining all this :)


Yes definantly.
But i installed a voltage regulator to feed a ups for _each_ machine along
with the new motherboards.
Surely this must clean up the power ? :(

Its also not just displaying this problem at random times - once it has
started, it wont go away until the motherboard is replaced with a new one.
So something is cooking :(. slowly. argh.

You might try a web search on "ESD protection." From what you've
described, it's nothing but static damage. You haven't said what kind of
protection the motherboards have, but since you use the term
"motherboard," it sounds like it's a PC-on-a-board. I think (but this
would have to be verified) that they don't do a lot of ESD protection on a
typical motherboard, since they're expected to mounted inside of a metal
box.

I've heard that static-sensitive chips can degrade a little at a time, so
they won't fail on the first zap.

Rein Wiehler said, "put a humidifier in the room." This would help.
Another thing is, you could use fabric softener (diluted) in a spray
bottle, and spray the carpets. You could use that aerosol antistatic, but
it's ridiculously expensive. (or like this one guy said, "Oh, any ol'
anionic surfactant will work.")

Good Luck!
Rich
 
J

John Woodgate

Jan 1, 1970
0
At least with nothing connected, you can still duplicate
the problem... perhaps touching the switch is adding capacitance to the
system, allowing enough ambient noise to influence operation?

Or pushing volts of r.f. from the local radio stations on to the wire
that was still connected to the switch. Been there!

'The switch doesn't work with only one wire connected, so why do I have
to disconnect both of them?'
 
J

john jardine

Jan 1, 1970
0
Quack said:

Had similar as a long, ongoing, problem 15 years ago. Operator touched a
keyboard in the control room and the computers 2 floors further down,
crashed. Purpose built computers in a positive pressure, air conditioned
room, with exquisitely clean power supplies and magnificent screening and
earthing. Realtime control of a natural gas network, so people stressed out
a little, when both the running and hot standby computers fell over.

Problem cured completely by spraying the control room carpeting with
antistatic solution. Rein's solution would no doubt have the same effect.

Earthing the keyswitch should have been the first target but this didn't
help . Possibly the series inductance was too high?.
I'd think that touching the machine wooden casing and its laminate covering,
impulsively transfers charge to it (the keyswitch act as a solidly connected
charge path), as if to one plate of a capacitor. The other plate is *all* of
the electronic components, casings, PCB tracking and earthed metalwork.
A spike of capacitor charging current thus passes through the air dielectric
via the myriad electronic circuit paths back to true earth. Some
particularly sensitive point is noticing the transient current.

Try lining the wooden casing with aluminium cooking foil and connect to
earth at a number of points (ie make a large area, good quality, 2nd
capacitor plate) Such that any transient current has an easy path with
little distance to travel, before it hits earth.
regards
john
 
N

Nicholas O. Lindan

Jan 1, 1970
0
Quack said:
The system;
A large cabinet (casino slot machine) - made of wood.
-1 x standard pc motherboard (p4 1.5ghz)
-1 x standard ide disk
-1 x 17" flat panel with resistive touchscreen/usb controlled
-1 x key operated switch, connected to the CDS line of the serial port

The problem;
If someone with a slight static charge touches the key to the
keyswitch - it sometimes hard boots. ... It still happens, even when
the keyswitch is completely disconnected
- the keyswitch is also mounted in wood so makes absolutely no
connection anywhere - but for some reason, touching it can reboot the
machine

Sounds pretty common. These problems are -great- revenue generators
for consultants....

Advice:

1) You need to be able to cause the problem at will with 100% reliability.
For this you need a static test 'gun', a standard rental item, though
every ee development organization should have one readily available.

2) High-frequency/high-voltage electricity travels over surfaces, in this
context wood is a pretty good conductor, as your experience demonstrates.

3) Your grounding effort wasn't done well - by definition - as it did not
fix the problem.

4) Try a wide copper tape from the keyswitch mounting nut (or whatever
hardware is on the inside of the box) to the AC entry panel ground.
Do not connect this to anything else. Route it far away from anything
electronic or conductive.

5) Spray conductive paint all over the inside of the box. The static
discharge will flow on the outside surface and should (in theory) not
effect anything on the inside.

5) Keep all AC power wiring far away from anything else.

6) Consider using a 1/2" wide copper braid from the machine ground to
the building structural steel. Think lightning conductors ...

7) Isolate the keyswitch: use a mechanical flag to interrupt an optical
sensor; a long plastic rod to activate a micro-switch; a relay;
etc. etc. ...

Question: do you have problems with static discharge to the touch
panel/lcd.

A 1.5GHz P4 to run a slot machine? -- back in my day, we did it with a
Fairchild F8. Still used real reels and a chrome arm sticking out the
side, though: nobody would trust the thing if they knew it was electronic.
Kinda silly, though, not trusting something that is designed to rob you.
 
G

Guy Macon

Jan 1, 1970
0
General advice on this topic:

Never use a standard motherboard in an embedded system.
The quality of the layout and the noise margins are too
much of an unknown. I would use a board from one of the
well-known embedded vendors.

Never use any version of Microsoft Windows in an embedded
system. There are too many thousands of bugs that MS refuses
to address. Linux is good, BSD is better, QNX is the most
stable, crashproof and bug-freeOS on earth. (A few other
RTOSs come close).

Never run a program in an embedded system that was written
by someone who knows general-purpose programming but is not
an expert in real-time embedded systems programming.
 
N

Nicholas O. Lindan

Jan 1, 1970
0
Guy Macon said:
General advice on this topic:

Never use a standard motherboard in an embedded system.
The quality of the layout and the noise margins are too
much of an unknown. I would use a board from one of the
well-known embedded vendors.

Never use any version of Microsoft Windows in an embedded
system. There are too many thousands of bugs that MS refuses
to address. Linux is good, BSD is better, QNX is the most
stable, crashproof and bug-freeOS on earth. (A few other
RTOSs come close).

Never run a program in an embedded system that was written
by someone who knows general-purpose programming but is not
an expert in real-time embedded systems programming.

Advice worth repeating. So I have.

BTW: QNX is the only system to use with embedded PC
architecture, don't bother with anything else, sorry
Linus.
 
J

john jardine

Jan 1, 1970
0
Nicholas O. Lindan said:
Advice worth repeating. So I have.

BTW: QNX is the only system to use with embedded PC
architecture, don't bother with anything else, sorry
Linus.

--
Nicholas O. Lindan, Cleveland, Ohio
Consulting Engineer: Electronics; Informatics; Photonics.
Remove spaces etc. to reply: n o lindan at net com dot com
psst.. want to buy an f-stop timer? nolindan.com/da/fstop/

QNX ...
I read a couple dozen pages but it seemed all marketing.
My best vision of an embedded PC would be to buy a PC and load say QNX
instead of microsoft stuff. Yet still run DOS programmes that would run on a
PC Is it this the case? or does everything have to be specially written for
QNX?.
regards
john
 
R

Rich Grise

Jan 1, 1970
0
QNX ...
I read a couple dozen pages but it seemed all marketing.
My best vision of an embedded PC would be to buy a PC and load say QNX
instead of microsoft stuff. Yet still run DOS programmes that would run on a
PC Is it this the case? or does everything have to be specially written for
QNX?.

I think what Mr. Macon is trying to tell you, and I concur, is that a
motherboard that's intended to go into a desktop PC is the worst possible
choice for an embedded application. You _can_ get boards that are designed
for embedded applications - you'd only need something as high-level as QNX
(which, AFAIK, is just another unix) if you're doing some massive
multitasking or devilish signal processing or something.

Check out some of these links:
http://www.google.com/search?q=embedded+sbc&btnG=Google+Search
(SBC == 'single board computer')

Good Luck!
Rich
 
Q

Quack

Jan 1, 1970
0
Thanks for the advice :)

I totally agree, i love _real_ embedded boards and have never had such a
problem with them.

The above problem is not my system, i am helping someone out. (really :) )

But i do like linux, i have never used QNX... i suppose i should try that
out then.

Thanks.

Alex.
 
Q

Quack

Jan 1, 1970
0
Nicholas,

Thankyou very much for that! I will go through that list and try
_everything_ :).

Someone else mentioned 'faraday cage', i thaught of that too - and have
attacked it with cooking foil, but on the old 'fried' motherboards.. I have
not re-attempted that with the new boards.
(perhaps the old ones were already too far gone).

Interesting that wood conducts static.. i had no idea... okay, off to build
an 'easy path' for the static :).

thanks again

Alex.
 
Q

Quack

Jan 1, 1970
0
(whoops)
Question: do you have problems with static discharge to the touch
panel/lcd.

Yes, the lcd/touch panel is surounded by an aluminium frame, and that can
also (but less often) cause a reboot.

Alex.
 
R

Rolavine

Jan 1, 1970
0
Subject: weird reboots of embedded pc
From: "Quack" [email protected]
Date: 12/10/04 2:32 AM Pacific Standard Time

This is a verry strange one :)

The system;
A large cabinet (casino slot machine) - made of wood.
-1 x standard pc motherboard (p4 1.5ghz)
-1 x standard ide disk
-1 x 17" flat panel with resistive touchscreen/usb controlled
-1 x key operated switch, connected to the CDS line of the serial port

The problem;
If someone with a slight static charge touches the key to the keyswitch - it
sometimes hard boots. As if the power was cycled.

We tried changing power supplies.
We tried grounding everything, including the keyswitch.
We tried with everything except power disconnected from the motherboard,
still happens.
(thats right, everything disconnected, including the keyswitch!)

It still happens, even when the keyswitch is completely disconnected - the
keyswitch is also mounted in wood so makes absolutely no connection
anywhere - but for some reason, touching it can reboot the machine.. wow.

Also, there are 10 machines, all worked fine for 3 months - this problem
just 'started' one day - on all 10 machines. So we assumed an external
influence caused it.

So, we replace all motherboards with a different brand - the problem
disappeared.
We added a regulator to feed a UPS for _each_ machine.

This ran for a few weeks - and now the problem has returned. Wow - how
weird!.

any ideas ? this is killing me.
Maybe you got a customer that comes around every few weeks and tries to get a
payoff by tazering the beasts?

Rocky
 
J

john jardine

Jan 1, 1970
0
Rich Grise said:
Nicholas O. Lindan said:
"Guy Macon" <_see.web.page_@_www.guymacon.com_> wrote
[...]
QNX ...
I read a couple dozen pages but it seemed all marketing.
My best vision of an embedded PC would be to buy a PC and load say QNX
instead of microsoft stuff. Yet still run DOS programmes that would run on a
PC Is it this the case? or does everything have to be specially written for
QNX?.

I think what Mr. Macon is trying to tell you, and I concur, is that a
motherboard that's intended to go into a desktop PC is the worst possible
choice for an embedded application. You _can_ get boards that are designed
for embedded applications - you'd only need something as high-level as QNX
(which, AFAIK, is just another unix) if you're doing some massive
multitasking or devilish signal processing or something.

Check out some of these links:
http://www.google.com/search?q=embedded+sbc&btnG=Google+Search
(SBC == 'single board computer')

Good Luck!
Rich
Unix, ta, makes more sense now.
I'm not sure what "embedded" refers to nowadays but occasionally a query
will come in for some kind of industrial machine controller that is easily
and cheaply satisfied by use of a standard PC running just DOS.
Said PC would as a matter of course have a fast processor, display,
keyboard, large hard disc, floppy disc, I/O.
A recent request was for a turbine blade checking unit. Essentially a
bed-o-nails type setup using 100's of LVDTs. A PC is ideal as the LPT port
can interface to the real world black box stuff. Run a DOS PowerBasic
control prog from power up. Calculate and display numerous visual items and
store and recall data for setups, auditing ect.
If anything fails, the idea is that the PC is a cheap item and can be
easily replaced using a spare from the office down the corridor.
One similar system I did 5 years ago using a standard PC has been running
24/7/365. Only special item was a CVT for the power feed. I've no worries
with hardware reliability of the commmercial PCs.
This path has recently been nipped off, as PCs no longer have LPTs and ISA
sockets. I must look to the 'industrial PCs' on offer. Eg PC104 and upwards
and what their software setups and special requirements are.
Problem is, they are soooo damned expensive and sooooo specialist and not
particularly standard. The customer would be locked into a particular
industrial PC supplier from day one.
A bit like the risks involved in basing a companys product range around one
specialist IC.
The future is here. We will all do as we are told :)
regards
john
 
G

Guy Macon

Jan 1, 1970
0
john said:
I'm not sure what "embedded" refers to nowadays but occasionally
a query will come in for some kind of industrial machine controller
that is easily and cheaply satisfied by use of a standard PC running
just DOS.

While I agree 100% about DOS, I strongly disagree with your conclusion
that a standard PC is OK for an embedded system. It simply will not
meet any reasonable requirements for reliability or durability.
A recent request was for a turbine blade checking unit. Essentially a
bed-o-nails type setup using 100's of LVDTs. A PC is ideal as the LPT port
can interface to the real world black box stuff. Run a DOS PowerBasic
control prog from power up. Calculate and display numerous visual items and
store and recall data for setups, auditing ect.
If anything fails, the idea is that the PC is a cheap item and can be
easily replaced using a spare from the office down the corridor.

In the system described, the "real world black box stuff" is the
embedded system. The PC is filling the role of a desktop PC, which
is what it is good at.
One similar system I did 5 years ago using a standard PC has been
running 24/7/365.

Either you got lucky, or your "standard PC" was by a company such
as Compaq which was making PCs to the high quality levels needed
for embedded work back then (today's Compaq / HP hybrid is another
matter). Try that with a modern asian motherboard and you are
unlikely to see it survive to its 5th birthday.
I've no worries with hardware reliability of the commmercial PCs.

You should have worries.
This path has recently been nipped off, as PCs no longer have
LPTs and ISA sockets. I must look to the 'industrial PCs' on
offer. Eg PC104 and upwards

That is the wise choice. Have you used a scope to look at the
noise and timing margins on the motherboards that are shipping
nowdays?
and what their software setups and special requirements are.

If your PC104 has any special requirements, then something
is wrong. It should act just like any other PC.
Problem is, they are soooo damned expensive

$300 will get you a fine 386SX PC104 stack. That's three
hours of my time at my standard consulting rate.
and sooooo specialist and not particularly standard.

If your PC104 isn't standard, then something is wrong.
It should act just like any other PC.
The customer would be locked into a particular
industrial PC supplier from day one.

If your PC104 locks you in, then something is wrong.
You should be able to replace it with any PC104 board.
A bit like the risks involved in basing a companys product
range around one specialist IC.

I have no idea what you are getting at here. PC104s use standard
parts such as Intel 386SX or 386EX - parts that Intel will be
cranking out long after they stop making the latest Pentium X.
 
Top