Risks associated with measuring impact and likelihood

Posted on Categories Stream of Consciousness

Subjectivity is all around us

Any evaluation, however objective you want it to be, is necessarily subjective. Just read some of Nassim Nicolas Taleb’s books which provide ample illustration of how easily we start to act based on subjective assessments.
Now, contrast this to new risk management methodologies and applications which frequently tout new and improved ways and means of measuring impact of a risk on objectives and likelihood of occurrence of that risk as part of their process.

Impact and likelihood are subjective

We need to raise the question: can subjectively assessed impact and likelihood be considered that relevant? Can we ensure that the evaluation of these two criteria is done in an as objective as possible manner?
* The negative: performing this assessment in an entirely objective and therefore relevant manner will be very difficult.
* The positive: these criteria do not necessarily need to be evaluated to perform good risk management.

We frequently over-evaluate the likelihood of recent occurrences

When assessing likelihood of occurrence of a risk, participants tend to over-evaluate risks which occurred recently or at all. If there is a reference point, people charged with evaluating will often attribute a higher likelihood to these recent events, even if the probability of occurrence has in effect been reduced by the (over)reaction to the event.
An example: Remember 9/11? People were more scared of terrorist events after the attacks on New York and Washington than before, whereas the actual likelihood of occurrence had diminished because of reactive measures taken.
The conclusion If it has happened before, we think it more likely to happen again. Turning this around, we also tend to under-evaluate those risks we know little or nothing about. Often these risks won’t even show up in an assessment until they occur … after which they are over-evaluated in terms of likelihood of occurrence.

Abstract risk description leads to under-evaluating the impact of a risk

If we cannot imagine a risk occurring, we cannot assess the potential impact of it and we tend to underestimate its impact. On the contrary, the more informed we are, and the more concrete a risk is formulated, the better we are at assessing its impact. Now, this does not only make the case of a significant investment in a risk (identification) model which aims at translating a risk in as concrete as possible terms, but it also warns for risks in skewing assessments if risks are not appropriately described.

What this means for assessed versus “real” inherent risk

Assessed inherent risk, as a function of impact and likelihood of occurrence will likely not be a correct representation of the actual inherent risk. Assessments are skewed as the evaluations are done by people, are always subjective and are very difficult to correct for as we have no insight in the motivation to vote one way or another.

Trusted collaborators skew our perception of current control level

The problems, however, do not end there. Often, a third dimension is measured: the current control level or the current risk management level. In this assessment, the presence of known and trusted collaborators charged with working on internal controls will skew management’s assessment of the current level of internal control or current level of risk management, which they will tend to overrate. The better the measures functioned in the past, the more concrete the measures are to the manager evaluating them, the more likely he or she will actually overestimate their effectiveness.

First conclusions: is risk management doomed?

Not necessarily. There are however a couple of elements to keep in mind. The traditional risk matrix, representing impact and likelihood on two separate axis will more than probably misrepresent the objective truth. When using a standard risk matrix, do so with caution. The risk control matrix can be used as a good tool subject to certain preconditions:

  • Do not merely and blindly use impact and likelihood as this will create a false sense of security. Evaluate level of (inherent) risk as one evaluation instead. Inform participants in the assessment level of (inherent) risk is a function of their perception of impact and likelihood, but ask them to perform their own ‘integration’ of the two factors. Level of (inherent) risk remains an intuitive assessment.

  • Instead of assessing level of current control, reverse the question and ask participants to assess ‘exposure’ or ‘vulnerability’. Again, this is an intuitive assessment. I refer to this post for some more ideas on that.

  • Develop the risk control matrix by combining level of (inherent) risk with exposure or vulnerability in a two dimensional representation.

  • Remain very aware the assessment is a subjective assessment at all times. The map is NOT the territory.

  • Correct quadrant III of the risk control matrix (which I will detail in a further post) for under-evaluation of level of (inherent) risk due to the factors discussed above. Internal audit, in executing its assurance function, needs to focus on both quadrant II and III of the risk control matrix.