Sample Size Determination

When planning for quality or reliability testing, an often posed question is: How many samples do you need? The trite answer is: Just as many samples as you need and not one more. A better answer is: Enough samples to make the right decision. The realistic expectation is : You will not have enough samples.

In some cases, sample may take months to create and are expensive. In other cases, you will have to deal with mock ups or partially functional samples. In all cases, the samples you need for reliability testing will have to provide value. This upfront investment in samples should provide a return in knowledge. You need to convert those sparse groups of units into valuable information, revealing what will fail or when failures will occur.

So, how do you estimate how many samples you need for a specific experiment? To determine the necessary sample size you need three pieces of information: the risk, the variance, and the precision. Let’s discuss each of these in a little more detail.

The Risk

There are two kinds of risk here: risk of the unknown and statistical risk.

The first type concerns not doing the experiment and making a decision without the data. This risk also includes unknown errors in the samples or test process. The testing itself may produce faulty (often hidden) results. If the experiment is not carefully designed, the results have a chance of being bogus. This risk has little to do with the sample size.

The second risk, the statistical risk, involves the sample size directly. There is a statistical chance that the randomly selected set of units for the sample are generally better than the population or vice versa. The basis for statistical testing and using a random sample is that the sample will represent the population. Alas, there is a chance it will not.

We use the term statistical confidence, or type I and type II error, or power, to describe the ability of the sample to properly represent the population. For example, a 95% confidence means that the associated sample has a 5% chance of misrepresenting the population, resulting in the analysis producing a false conclusion. The amount of statistical risk that is acceptable is a policy decision.

The Variance

Variation happens. Variance is the statistical measure of variability in a population of data. It is what it is and the only way to reduce variability is by changing the product design or production process.

A large variance will require a higher sample size to detect a shift in results of an experiment. We often do not know the population variance and have to estimate the variance elements using the sample data.

The Precision

Precision is related to what we are trying to detect. If the new design has more strength than the old design, then how much stronger does the new design have to be to be worth noticing? If we want to detect a one-gram increase in average load-bearing capability, we would require many more samples than if we only wanted to detect a kilogram difference. The larger the difference that we desire to detect, the fewer samples we will require.

This sample size changes because it is more difficult to conclusively detect a small change. We may measure one sample from the old and new designs and see a difference, maybe even a very small difference. Yet does this represent the actual shift and amount of shift in the overall population? Most likely not.

However, if we measure a new sample and it is 1,000 times stronger, statistically speaking we can be pretty sure the new design is stronger. This is rarely the case and we need to detect smaller changes and thus we need more samples.

The Bottom Line

Most sample size formulas contain these three elements. Two parts, risk and precision, are business or technical decisions. The variance is part of the underlying data and not an option to increase or decrease easily.

The sample size formula is useful for a discussion around risk and decision points prior to conducting the experiment. It is one way to design and conduct measurements that provide value. If our results are to be useful then considering all three elements that make up a sample size formula becomes important.

Bio:

Fred Schenkelberg is an experienced reliability engineering and management consultant with his firm FMS Reliability. His passion is working with teams to create cost-effective reliability programs that solve problems, create durable and reliable products, increase customer satisfaction, and reduce warranty costs. If you enjoyed this articles consider subscribing to the ongoing series at Accendo Reliability.