Are Qualitative Risk Assessments Fatally Flawed?

Early last month, I published an article explaining why I don’t discuss quantitative assessment methods like modeling, Monte Carlo simulation, and others very much and what companies must have in place before they can use methods like this to guide their decision making and strategic planning.

In short, without establishing corporate governance policies, a strategic plan, or clearly defined roles and responsibilities, any sort of advanced ERM methods will fail to be a useful tool for ensuring a company is taking the right amount of risks in pursuit of strategic objectives.

In this article on enterprise risk assessment from two years ago, I explain how qualitative methods use descriptive elements to rank a particular risk. Risk managers can then take this information and develop risk scores based on likelihood, impact and other parameters. In some cases, these scores can be plotted on a heat map, which as COSO explains, is “…a graphical representation of likelihood and impact of one or more risks.”

However, thought leaders at the forefront of ERM are extremely skeptical about the value qualitative risk assessments and heat maps can provide an organization. For example, in response to my article from last month, Hans Læssøe comments:

Relying on qualitative risk assessment is infected by human biases and potentially dangerous…it will not help you run [or] perform better, so waste as little effort on this as possible.

If a company’s goal with ERM is to make better decisions and improve performance, Hans explains:

…there is no avoidance of quantitative approaches – including Monte Carlo simulation. This will require skilled people and effort – but also add the value of enhancing performance.

Other thought leaders are even more critical of qualitative methods, including David Vose, who explains in this article:

Qualitative risk analysis may be appealingly simple, and lots of companies use it, but it’s almost valueless for making risk-based decisions.

David goes onto explain in his article and in a presentation at Risk Awareness Week 2019 how qualitative scores are “…too imprecise and inflexible.” 

If you browse around my site and others like RIMS, NC State’s ERM Initiative, and others on the topic of risk assessments, you will find a lot of information on qualitative methods, but…

Based on comments from Hans, David, and others, should we assume qualitative risk assessments are fatally flawed?

I certainly agree with Hans and David that qualitative methods, especially how they’re currently done by many companies, are fatally flawed due to human bias. 

While I agree with their opinions, the actual move from qualitative-based risk assessments and heat maps to quantitative modeling is an entirely different matter.

For many companies, qualitative risk assessment is all they know…asking them to go from this to more quantitative methods like modeling and Monte Carlo simulation represents a huge shift. If the culture of the organization will not support such a change, it is doomed to fail.

As an example, heat maps are quite popular with not only the general risk management community, but executives and boards as well. This popularity is likely due to their simple color scheme (red/yellow/green) and quick glance comprehension – finding or developing a quantitative reporting style that meets these needs will be challenging.

In the meantime, companies can use different techniques to overcome bias and quantify risks that are normally qualitatively assessed…

Despite the urgency for companies to take informed risks in pursuit of strategic objectives, getting to the ideal situation promoted by Hans and others will take a long time. Certain companies like those in finance may already use quantitative methods, but for many, this is simply out of reach at this time.

However, this doesn’t mean companies shouldn’t be on the lookout for ways to address the shortcomings of qualitative risk assessments.

One way to overcome bias is to ask executives what history tells them. They may consider a risk to be of grave importance, but after careful examination, you may learn that when the risk materialized in the past, it was not that significant after all.

There are numerous books and resources out there for helping reduce bias. 

Many people think a quantitative risk assessment is providing a scale of 1 to 5 with criteria associated with each number. No. But taking a risk that is normally thought of as qualitative or difficult to put a number to it…that is the hard part. 

Let’s walk through an example using a frequently discussed risk: the ability to recruit and retain quality talent. 

Rather than say that the risk is a 4 on a scale of 1 to 5, there needs to be some hard numbers put to it. If your organization cannot recruit and retain quality talent, what will the organization not be able to achieve? Will certain revenue numbers not be met? Will products be designed and manufactured on time? Putting scope and boundaries on a risk will help you identify those numbers. And when all else fails, you can always check out the book “How to Measure Anything” by Douglas Hubbard.

While Hans, David Vose, and others are correct about the shortcomings of qualitative risk assessments and are visionary in how risk management must change in order to help companies create a strategic advantage, getting there is going to be a long road for many. 

But as Hans also explains, the world is changing at such an unprecedented speed that companies may not have a choice if they wish to remain competitive.

Do you think qualitative risk assessments are fatally flawed? 

What methods have you used to overcome bias in your organization when it comes to risk management and decision-making?

Share your thoughts by leaving a comment below or join the conversation on LinkedIn.

And if you are struggling to develop risk assessment methods for your company, please don’t hesitate to send me an email to discuss your specific situation today.

Featured image courtesy of Kevin Ku via Unsplash.com

, , , ,

Related Posts

8 Comments. Leave new

  • Hans Læssøe
    February 5, 2020 3:42 am

    True – I am passionately opposing heat maps and qualitative assessments. They are not only dangerous – but sometimes directly dangerous.
    Take the parallel “industry” of finance. You would never attempt to run a company without looking at the financial data – so, why would you wish to make decisions in a volatile world (which only gets worse, there is no slow-down in sight) without measuring and knowing the risks and opportunities, and being able to prioritize these validly.

    Learn and apply quantitative fact based insights – or you will not survive the next century.

    The fact that heat maps and qualitative measures are popular is a result of people without risk management insight and understanding (eg. auditors) promoting something they do understand.

    Reply
    • Moving to a more quantitative risk analysis is a must. However, my only fear is some companies will move too fast into a territory they’re not familiar with and do more harm than good. And I certainly concur that the reason many avoid it is because they want to stick with what they’re familiar with…this of course could be fatal too, so it’s a careful balancing act.

      Reply
  • I do agree qualitative assessment might be biased. However, there are situtations where quantitative assessment are not as easy as percieved. To come up with financail risk, quantitiave assessment is a good tool but what about non-financial risk, where usually organization do not have large history of loss/ incident database, there you need a qualitative judgement.

    Heat map is good resource to give a fair idea of the risks, however, this is not an end rather a picture to work on the risk.

    Reply
    • Good points Abdul – without the right data, any quantitative analysis can be just as “dangerous,” or at best, not helpful for making risk-informed decisions. Heat maps have a very limited role for communicating about risks. While easy to “read,” they can be very misleading to the users based on any caveats or explanations – they shouldn’t be an end, but rather a means for prompting further analysis and discussion.

      Reply
  • I agree with Abdul, somewhat. Quantitative analysis can be great in fields where things can be quantified. But I look at the carnage in the finance industry after the rating agencies tweaked their quantitative models to show that there were no problems in the housing industry or in the CLO world. There are biases in everything – to pretend that quantifying something with a hard number is somehow more diligent and robust than a heat map is the height of absurdity.

    Reply
    • Thank you for your comment, Paul! Interesting contrast to Hans’ comment – I can see your point though that just because it’s quantitative doesn’t mean it’s unbiased. Taking steps to account for or address bias is something risk professionals have to address. BTW – thank you for prompting a new topic idea!!

      Reply
  • As a PhD student I have had as a goal implementing risk management process across four communities. I had utilized qualitative analysis in order to identify, analyze and evaluate risks. So I think using qualitative analysis can be used for laying the foundations for risk management policy. Afterwards, when the process get mature, quantitative data can be used.

    Reply
    • Absolutely Hanane…an organization must have its governance and other pieces in place before it’s ready for quantitative analysis. But others are correct that not adapting quick enough could mean a loss of competitive advantage. Consider that just a few decades ago, it usually took a company many years to reach 100 million customers – now that can happen in just a few short weeks (…like the new Disney streaming service). Check out my recent interview with Hans (02/19/20) where we discuss how companies can best use qualitative vs quantitative assessments.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Fill out this field
Fill out this field
Please enter a valid email address.

Menu