Troubleshooting MH-Z14 CO2 Sensor Output Connected To Microcontroller - Sometimes readings go crazy

[Mahonroy]

Hey guys,
I have some MH-Z14 CO2 sensors connected up to a Particle Photon microcontroller. Its a relatively simple circuit consisting of a 12vdc wall wort as power input, a 5v voltage regulator (RECOM R-78E5.0-1.0), a couple sensors, and a 12VDC fan.

I have a bunch of these devices running in different environments (offices, greenhouses). Occasionally the CO2 sensor enters a state where its output begins to oscillate up and down. Sometime the oscillation frequency is quick (8 minutes peak to peak), and other times its less (couple hours from peak to peak). When it begins to oscillate, you can tell that its oscillating around the real value - this almost looks like noise doesn't it?

The MH-Z14 has multiple output options available (0 to 2.5v analog out, PWM out, UART). I am currently using the PWM option. I took some scope readings and I can see the PWM signal is pretty clean... meaning the MH-Z14 sensor is deliberately outputting these crazy values.

Can you guys offer any advice on what might be going on, or how I should go about troubleshooting this? Here are some of the things that come to mind, but I am not sure if it is legitimate?
* Are the MH-Z14 sensors simply defective?
* Do the MH-Z14 sensors have a design flaw, and either my power supply design, signal interface, or environment conditions are exploiting the sensors flaw causing it to behave like this?
* Did I integrate this sensor into my board incorrectly?
* Is there an environmental condition (such as interference) that is doing this?

Here are some images and video to show you what I am talking about.

I put 2 identical devices in the same room. The one on the left is working currently, and the one on the right is experiencing this problem:


Here is another example.. you can see that periodically this is getting good readings, then the problem starts and stops:


Sometimes the sensor goes for days and works perfectly before the oscillation begins. The oscillation can last several hours to several days, and then it can go away... seems pretty random.

Here is one more example of how the sensors output will be bad, turns clean for the majority of the day, then switches to bad again:

I cannot directly correlate this behavior to anything else happening in the room, it seems to be really random. I had to wait for days for it to start happening so I could take some oscilloscope readings.

Here is a 13 minute long video showing how the PWM signal is clean and deliberately oscillating between 14% and 28% PWM (roughly 700ppm to 1400ppm CO2 concentration). Take note of the "+Duty" value, this is indicating the PWM output.
https://goo.gl/photos/JeZRbcnQMQWYXqyG7

Here is the schematic of the power, how the CO2 sensor is wired up, and how the CO2 sensor is placed on the board:


Thanks for taking the time to check this out! If you couldn't tell, this problem has been plaguing me for a while and I can't figure it out. Hoping to get some advice and info from you guys, thanks again!

 

[Sunnysky]

1st get rid of the "grass" on your PWM by any means, improved probing with very short grounds, improved shielded wires,,CM choke on sensor cable. It should look like a textbook waveform everywhere you probe. If noise is on the PWM output, there must be too much CM noise on the input, ( internal PD) used for gas ratio transmission loss.

I suspect the 8 minute cycle is a beat frequency of two harmonically related frequencies crystal controlled. Check the ripple out and DC levels. What is the current fluctuation for 5V and 3V?

The aliasing could be with the ADC sampling and the sin-wave interference... just a low probability guess

 

[Sunnysky]

It seems you are using audio connectors for power, which is not a great idea. IT is always better to connect ground pin before power when you hot swap peripheral powered sensors using "dual,supplies". Often power sequencing matters as CMOs devices can latch up with active data connections before ground and then power is supplied. just FYI.

 

[Mahonroy]

It seems you are using audio connectors for power, which is not a great idea. IT is always better to connect ground pin before power when you hot swap peripheral powered sensors using "dual,supplies". Often power sequencing matters as CMOs devices can latch up with active data connections before ground and then power is supplied. just FYI.

Its not an audio connector, Its a standard barrel connector that is on most dc power supplier - like this:

 

[Mahonroy]

1st get rid of the "grass" on your PWM by any means, improved probing with very short grounds, improved shielded wires,,CM choke on sensor cable. It should look like a textbook waveform everywhere you probe. If noise is on the PWM output, there must be too much CM noise on the input, ( internal PD) used for gas ratio transmission loss.

I suspect the 8 minute cycle is a beat frequency of two harmonically related frequencies crystal controlled. Check the ripple out and DC levels. What is the current fluctuation for 5V and 3V?

The aliasing could be with the ADC sampling and the sin-wave interference... just a low probability guess

Thanks for taking a look.

So you think there may be too much noise on the 5V power input going to the sensor and I should attempt to filter it more? I already have bypassing capacitors on the power line right next to sensor input (1uF and 0.1uF). How would you recommend filtering the power better? And does this filter need to go in front of every sensor on board or can I do this in 1 spot instead? Thanks again for the advice!

I have several of these boards here with me, so I can butcher a couple to try some things out on them.

 

[Mahonroy]

Here is a video of the 5V rail right up next to the MH-Z14 sensor (using the cheap chinese 12vdc power adaptors we have been using):
https://goo.gl/photos/KEVMEQLgyUst7jeBA

I tried a different 12vdc power adaptor and here is what the 5V rail looks like:
https://goo.gl/photos/MBckLNoGTdXnzw3K8

 

[Sunnysky]

If those two supplies don't change the results then it is not conducted noise. They should.
Yes caps on each sensor near pins. 1uF//0.1uF

How long are the wires for DC?
Have you tried to monitor current consumption for correlation?

Calibrate ADC for long term stability.

I read the manual... nothing obvious. http://eleparts.co.kr/data/design/product_file/SENSOR/gas/MH-Z14_CO2 Manual V2.pdf

Or contact the sensor OEM tech support. with your test report?
[email protected]

I don't have any other ideas.

 

[Mahonroy]

If those two supplies don't change the results then it is not conducted noise. They should.
Yes caps on each sensor near pins. 1uF//0.1uF

How long are the wires for DC?
Have you tried to monitor current consumption for correlation?

Calibrate ADC for long term stability.

I read the manual... nothing obvious. http://eleparts.co.kr/data/design/product_file/SENSOR/gas/MH-Z14_CO2 Manual V2.pdf

Or contact the sensor OEM tech support. with your test report?
[email protected]

I don't have any other ideas.

Thanks again for the input.

Yeah I was certain that the new clean power supply would fix the issue, then I would know it was due to noise... but it didn't help. I also disconnected the 12vdc fan to further reduce noise and rule our... still did not help.

The cheap 12vdc power supplies wire length is about 3 feet. The new clean power supply wire length is 6 feet from wall to adapter, then another 6 feet from adapter to the device.

I have not tried monitoring current consumption. The CO2 sensor pulls current every second during normal operation. The device typically pulls about 200mA, though I have not tried to correlate this with the malfunction stretches.

Yeah I have been going back and forth with sales@winsensor,com for the past 2 weeks or so. They are thinking it might have to do with noise, or one of the other sensors on the board. I could try building another board with nothing but the Photon and CO2, but some of the fully assembled devices work and some experience this, so I would not know what the case is. Luckily I currently have a board present that malfunctions constantly and thats what I have been doing experiments on.

I am at a loss as well, thanks again for taking the time to check it out.

 

[Sunnysky]

is the sensor physically isolated from any other sensor, equipment except scope ,

affected by ceiling lights? stable in a sealed box?

In other words can you alter the PWM effects ,,worse or better in any environmental way besides breathing on it. I am curious about the stray capacitance in the "keep-out zone" effects on internal PD noise.

 

[Mahonroy]

is the sensor physically isolated from any other sensor, equipment except scope ,

affected by ceiling lights? stable in a sealed box?

In other words can you alter the PWM effects ,,worse or better in any environmental way besides breathing on it. I am curious about the stray capacitance in the "keep-out zone" effects on internal PD noise.

At one point someone thought that an external device (AC compressor) was triggering the event to occure. We were never able to reproduce anything. Lights turning off and on right next to it, fans on/off, nothing seems to correlate to the behavior starting and stopping.

What do you mean by the stray capacitance in the "keep-out zone"?

Thanks again.

 

[Sunnysky]

the black area above U$5 in the layout,,which I thought was the sensor , but now I see it is U$3 not on board. So where is it located? I assume the 0-2.5V analog out matches the PWM.

 

[Mahonroy]

the black area above U$5 in the layout,,which I thought was the sensor , but now I see it is U$3 not on board. So where is it located? I assume the 0-2.5V analog out matches the PWM.

Hey Sunny, sorry for the late response I missed the notification on the topic.

To answer your question. U$3 is that large yellow box outline, its the CO2 sensor mounted to the PCB in the right/middle area.

And yes, the 0-2.5V analog out matches the PWM.