Some days are better than others. We sometimes run into failure when working to create a new product. With a little investigation we suspect the components are not working as expected.
We’ll call the vendor and ask for an explanation. If this is normal production and variability of performance, our product will suffer an higher than expected failure rate. The vendor will assure us with:
- It must have been damaged during assembly or use by us.
- It was a very rare manufacturing mistake and won’t happen again.
- We haven’t seen this before and you’re the first to report such an issue.
- We know about this issue and it’s been resolved in our process.
- Must have been an ESD (electrostatic discharge) event.
Not helpful.
My recommendation when you experience a component failure is to first determine the cause of the failure, as the component that failed may just be the victim or hero and not the cause.
Victim is a component that takes the brunt of a failure somewhere in the system. For example, a diode failure may expose the failed component to excessive current.
The hero is a component that saves the rest of the system by failing. Think fuze.
If the component wasn’t a victim or hero, and should have operated yet failed, then you should do a bit more failure analysis. In short, you should learn as much about the failure as possible before approaching the vendor.
Here are three examples, where supply chain errors led to product failures in the field and detailed failure analysis helped find a resolution quickly.
Missing Electrolytic Solution
Electrolytic capacitors are cans with a metal foil curled up within. The foil has a oxide layer that forms the dielectric barrier creating the capacitance structure. The electrolytic solution helps form and maintain the dialectic barrier.
Hydrogen gas is one of the by products of a functioning capacitor. One of the components of the dielectric solution is a hydrogen scavenging compound which effective traps (or removes) the hydrogen gas. If the gas builds too much pressure the capacitor will ‘vent’, meaning explode and fail.
A vendor created new less expensive capacitors that unknown to those buying the capacitors did not contain the hydrogen scavenging compound. The capacitors initially function just as expected and given the lower cost quickly became part of many circuit boards. Checking the internal construction of a part is not a common practice, and the common testing generally only lasted a few hours.
After a few months we heard reports of widespread venting within our products. Gathering a few failed items we quickly identified the vendor and part numbers of the failing parts. Failure analysis confirmed there was a missing element within the parts.
The cause of the failure was a company chemist either forgetting or ignoring the need for a hydrogen scavenger.
When buying capacitors or any component, we have to balance the risk of basic design errors and the impact on reliability. Yet, we often do not as it’s rare.
Small Change in Red Phosphorous Size
Red phosphorous is a flame retardant added to epoxy over-molded components. When buying an integrated circuit (IC) component, we may or may not investigate the ability of the flame retardant, other than to simple check that it is listed not he data sheet.
It’s there, it’s standard, it’s not a big deal.
The teams that create IC’s focus on the functions that device provides, the active elements of the circuit. The team that packages the IC adds the over-molding epoxy and focuses on creating a consistent seal across the lead frame and not damaging the enclosed IC.
The team the creates the epoxy focuses on consistency of the formulation which includes the addition of red phosphorous material.
The team that makes the red phosphorous also focuses on consistency of there process, which includes a specific particle size. They use a series of meshes to filter the particles to a specific size, which optimizes the flame retardant properties.
At some point, one of the screens was replace with a larger mesh allowing larger particles to join the finished batch.
The red phosphorous, the epoxy, the component packaging, and the IC team didn’t notice the change in particle size as each process and various process checks all didn’t detect the impact of the larger particles.
The larger particles when exposed with a modest amount of vibration tend to separate from the epoxy. Red phosphorous is corrosive and would damage the circuit board relatively quickly causing failures. Thousands of disk drives failed across the industry as they created the perfect conditions for the larger particles to fall out of the epoxy and cause problems.
When I buy a disk drive I assume the supply chain is working well and something as simple of a change in particle size of one material will cause my device to fail. It can happen though.
Insufficient Crystal Attach
Crystals provide timing for a digital circuit. There is at least one on every circuit board, it seems. When a crystal fails the circuit may behave oddly or not function at all.
Crystals are robust parts and rarely fail. So, during an exploration of the operating limits (temperature and vibration stresses) we were surprised to find that the crystal had failed.
The outside of the package and the solder joints all appeared fine, so we took the failed parts to failure analysis and cut open the component. We found the inside of the component to be very simple. Two metal stands that supported the crystal (looks like a rectangular chuck of glass). The crystal is attached to the metal stands with a silver epoxy (I think).
The first failed one we opened we found the crystal had cracked. The same with the second one, that had failed. Then the technician noticed that the crack was closer to the end where the attaching epoxy was only partially traversing the length of the metal and crystal edge. It was about 50% across the joining edge.
On the phone with the vendor, we discovered that they had recently changed the dispensing equipment and it was not as consistent as the older equipment. They only revealed this information after seeing the images. Their first response included a couple of those listed at the start of this article.
We caught this issue during development and took steps to insure the consistency of crystal attach going forward. Again a supply chain process change impacting product reliability.
Summary
How well do you know your supply chain and the critical processes that will impact your system’s reliability performance?
Remember, customers will not call you capacitor, flame retardant, or crystal supplier, they will call you when a product fails. Every failure has a source and cause, and often it’s in the form of an element from your supply chain.
BIO
I am the reliability expert at FMS Reliability, a reliability engineering and management consulting firm I founded in 2004. I left Hewlett Packard (HP)’s Reliability Team, where I helped create a culture of reliability across the corporation, to assist other organizations.