#275 – RISKS INTRODUCED BY RISK ASSESSMENT – DUKE OKES

Posted on by

While the past 20 years has seen risk management spread from primarily the financial industry to now being utilized in nearly all sectors, it’s become more evident that much of these efforts have failed to prevent some very serious failures.  Part of the reason is the increased complexity of systems and interconnections, but some of the failures can also be attributed to the risk management process itself. 

The purpose of this article is to focus solely on the risk assessment process, since it is key to the subsequent decision making about how to treat and monitor identified risks.

Many others have pointed out some of the risks associated with how risk assessments are performed (a few references are provided at the end of this article).  This author has identified seven common errors seen over his years of studying, instructing and reviewing risk efforts.

  1. The use of math (Likelihood X Impact) to determine level of risk
  2. Using averages (vs. actual) when evaluating potential impact
  3. Using single point estimates (vs. a range) to describe a risk
  4. Use of qualitative (vs. quantitative) risk ranking scales
  5. Using probability (vs. frequency) to describe risk opportunities
  6. Using linear risk ranking scales
  7. Lack of aggregation of risks

#1 – It’s common to define the level of risk as Likelihood X Impact, and to have a scale (e.g., 1 to 5, with 1 being low and 5 being high) for each of the two factors.  For each risk a number for Likelihood and Impact are selected based a predefined rubric and the total risk calculated as the multiple of these numbers.  But let’s consider that the multiple is 4.  Such a result could come from a Likelihood of 1 and an Impact of 4, a Likelihood of 4 and an Impact of 1, or a Likelihood of 2 and an Impact of 2.  Are those three risks actually of equal significance to the organization?  Doubtful!  Imagine if the scales used were 1-10 … the number of ways this could occur is even greater, as has been recognized by the auto industry where instead of using the multiple to score risks identified and evaluated by an FMEA, the risk is determined by looking up the combination of numbers in a table.

#2 – Suppose instead of an ordinal scale of 1-5 we use actual numbers (e.g., probability for likelihood and dollars for impact).  Now when we multiply the two numbers it has a bit more meaning, right?  Well watch out, because that means if we say there’s a probability of 1% that we’d lose $1M if it happened that the risk is $10k.  But we said if it actually happened that the impact would be be $1M!  So while the multiplication may be of value over the long term or when combining many risks (and the distribution isn’t highly skewed), when evaluating a single risk (such as within project) it can be very misleading.

#3 – It’s amazing how often risk is defined as a single number (e.g., the $10k in the previous example, or even the $1M).  The reason we need risk assessment/management is because of the uncertainties we have to deal with.  Then we state a single number with no uncertainty.  Instead, any risk quantification should at a minimum be stated as a range, with confidence limits.  In our example this might mean we state the likelihood as between .1% and 2%, and impact between $.5M and $2M, with an 80% confidence band.

#4 – While qualitative scales may be appropriate for simple applications, the differences in how individuals may interpret the language will inherently do little to reduce uncertainty.  For example, one study found that the term “usually” may be interpreted as anywhere between 45 and 100%, while “unlikely” ranged from 0 to 50% (figures estimated from a visual by Mauboussin and Mauboussin, 2018).  While quantitative scales don’t immediately solve this problem, if used along with data it is more likely that accurate levels of likelihood and impact will be identified.

#5 – Likelihood is often stated as probability, but not only are humans not very good at estimating probabilities, they are also not time-bounded.  Frequency is a better way to express likelihood (e.g., number of occurrences/year) since while probability might stay constant, frequency can change based on the number of opportunities.  That is, if you do one activity at a high frequency and another at a lot frequency, given the same probability of failure the first activity is more likely to fail within a given time period.

#6 – Linear scales often don’t accurately express the level of differences.  For example, a death or loss of $1m (scored as a 5 on scale of 1-5) is more than 5 times worse than no injury or loss of a loss of $1k (a 1 on the scale).  Logarithmic scales (a change in magnitude at each level) do a much better job at differentiation.

#7 – Risks are most often rated/ranked and evaluated individually, with a decision of whether it is too high and needs treatment.  However, if one reviews a list of risks one can often find that the same potential impact, same causes, etc. applies to many.  So, for example while the probability of each single risk may be 1%, if the same impact exists ten times then the probability of the impact is actually 10%, assuming they are independent.

So if you are involved in risk assessment, which of these factors might be important to your application?  How important is accuracy?  Are you trying to just come up with a risk stack, or better understand overall exposure?  Do a quick risk assessment of your risk assessment process to determine whether or not change (risk treatment) would be useful.

References/Resources

Author:

Check out his Risk Based, Quality Auditing webinar.

Duke Okes has been in private practice for 35 years as a trainer, consultant, writer and speaker on quality management topics.  Two of his books, “Root Cause Analysis: The Core of Problem Solving and Corrective Action (2nd ed.)” and “Performance Metrics: The Levers for Process Management” have been cited as references all quality professionals should read.  He is an ASQ Fellow and holds certifications as a CMQ/OE, CQE and CQA.

https://drive.google.com/file/d/1OfF613SCt1HHIfSeBo1_rH-U5uq8E4hz/view

Leave a Reply

Your email address will not be published. Required fields are marked *