The baseline
As CAE of a small audit shop in a complex environment, I have to comply with the IIA standards like any other CAE. The performance standard for planning purposes is of course “2010 – Planning”, which states that “The chief audit executive must establish risk-based plans to determine the priorities of the internal audit activity, consistent with the organization’s goals.”
The context
Now, our internal audit department of two people is responsible for an audit universe consisting of an main office, country offices in 18 different countries and about 200 active projects per year, with give or take about 250 million euro in spend in these projects on a yearly basis. These projects are very wide ranging, from building roads to assisting foreign governments in developing strategic plans in certain sectors. As we work mainly in fragile states, the risk profile of our projects is often quite high. This is a signficant challenge for planning the slightly more than 400 mandays per year I have available to me.
So we had to come up with an efficient as well as effective way of complying with the IIA standard 2010 and ensuring our assessment was as relevant as possible as well, to make sure our focus is where it should be.
This is what we did …
Phase I – Open online questionnaire
While my initial intention was to ask both project managers, country responsibles (equivalent to middle management) and headquarters based middle management for their opinion, we quickly determined this was not feasible from a practical point of view. Why? I decided to work with open questions, allowing all participants to voice their opinion on top five risks ahead of them in the coming two years. If we had to integrate all these open questions for more than 200 participants, it would have been too time consuming. In the end, we queried about 50 people in total, using the forms function of Google Docs as our system.
For each of the five most important risks, we asked the participants to evaluate the following three elements:
- the likelihood of the risk occurring in the next two years
- the impact the risk would have on the area under their responsibility if it were to occur
- the current level of risk mitigation based on existing procedures and controls with respect to the risk
We provided only limited guidance on the quantification of these evaluations, but the evaluation was done on a qualitative, not a purely quantitative basis, but using statements such as ‘very high’, ‘very likely’, rather than numerical values.
As was to be expected, the results were quite varied. Some respondents looked at risks from a very high level, with a significant focus on external threats, while others approached it from a very detailed position.
Lessons learned from phase I
We learned the following two important lessons from phase I:
- Although difficult to reconcile, this exercise brought us a lot of different points of view which were highly complementary. This information has become an important input in customizing the risk model which we will be using next year for the risk based audit planning.
- We shied away from using a comprehensive risk model as the basis for questioning in our initial open online questionnaire. However, in order to involve more people in this initial assessment, we will be using a structured, closed questionnaire next year.
Phase II – Team meetings per department
After processing the information gathered in phase I, we followed up with meetings in which all middle managers were invited to participate. Some of them declined because they had already shared their considerations in the open online questionnaire. Others felt they wanted to further detail their considerations.
In the meetings, we steered the discussion towards the following three elements:
- We started discussing risks related to processes in their area of responsibility;
- we then moved to discussing the risks related to people;
- and finally we discussed systems at their disposal and risks related to these.
Based on these meetings, which we conducted with each of the departments of our organization, we arrived at an enhanced list of ‘risks’ related to each of the departments.
Lessons learned from phase II
We learned the following important lessons from phase II:
- We again used an open format. While this is valuable in the context of such meetings, providing the participants with some information on the structure we intended to follow may have focused the discussion more.
- It remains a trade-off to be made between focusing the group and perhaps losing essential information on less clearly perceived risks and getting the group to be as broad in its scope and discussion as possible and perhaps losing focus on some key challenges to them.
Phase III – Delphi analysis within the internal audit department
Based on phase I and II, we now had quite some information on risk exposures of our departements. Now we needed to translate this to a comprehensive, internal audit owned risk analysis.
We developed a spreadsheet in which, for each of the departments and functions in our audit universe, we were to assess independently, as internal audit experts, based on the information gathered, the impact and likelihood of the risk and the perceived current risk management level. With respect to impact, we defined different types of impact, i.e. impact on finances, impact on reputation …
We compared the results of our independent assessments, focusing mainly on those assessments that were significantly different, looking at the underlying scores we each attributed to the different departments and functions. In a very open exchange, we agreed on a final score for each of the departments and functions.
Lessons learned from phase III
The results were quite nuanced. The independent internal audit risk assessment is but one input in the overall planning, which we will detail in a later post. We will continue to own the final assessment ourselves, as this is required if we want to remain entirely independent and objective.
Overall conclusions
In short, we will be using a more structured approach for the first two phases, in order to both involve more people in the exercise in phase I and provide better guidance for the discussions in phase II. However, the two step approach will remain in force.
Using the informatiion gathered to develop an independent audit centric risk analysis, in which we use a Delphi technique, has proven to work very well. It aligned with the risk profile the external auditors estimated for our organization, which was an additional validation for us.
Exploring The Black Box 