by Matthew Leitch

During 2004 I've been carrying out a series of online research studies, announced through the AuditNet newsletter and discussion list, and with the kind participation of Jim Kaplan and members of the AuditNet community. Previous research reported has included an analysis of the value of different types of evidence for Sarbanes-Oxley 404 compliance and views on what "embedded" risk management should be like.

Here are the highlights from two more studies in the series.

1. Refreshing audit recommendations

Over the last decade internal controls thinking has collided with risk management thinking to form a broader, more complex field offering many ways to manage risk and uncertainty. The study strongly suggests that there are opportunities to make valuable recommendations for control improvements based on risk management techniques not traditionally associated with auditors.

In the study, participants were presented with eight imaginary audit reviews, each of which was followed by five potential recommendations for improved controls. For each recommendation participants were asked if (1) in their experience of organisations the control would probably be in place already, (2) if they thought the recommendation was probably a good one, and (3) if they thought their organisation expected them to make such recommendations.

The recommendations were based on a variety of control mechanisms but, broadly, fell into two groups. One group consisted of traditional favourites for auditors, like sign offs, documentation, reconciliations, and segregation of duties. The other group consisted of other risk management mechanisms, such as evolutionary project management, training managers in risk management skills, performing risk analyses, quantitive modelling of uncertainty, and product portfolio management.

The main results were that respondents thought (1) the traditional favourites were almost twice as likely to be in place already compared to other risk management mechanisms suggested, (2) new recommendations were almost as likely to be good as traditional ones, but (3) other people tended to expect traditional recommendations.

On this final point, there was a marked difference between public and private sector auditors. Public sector auditors felt they were expected to make more wide ranging recommendations on managing risk than their private sector peers.

My interpretation is that participants were able to recognise good methods of managing risk even when they were outside the traditional auditing repertoire and that there will often be opportunities to make those recommendations because the controls are unlikely to be in place already. Auditors in the private sector who make such new recommendations are more likely to face surprise than auditors in the public sector.

This is an opportunity to refresh our toolkit of recommendations, but it does involve some new learning. Full details of the results are available in "Results of a survey on internal control and risk management recommendations".

2. Asking for "areas of uncertainty" as an alternative to "risks"

Most of us who have been involved in promoting internal controls and risk management for a number of years realise that other people need persuading. It's not just that they are ignorant or careless when it comes to internal control. They genuinely don't see the need at times and even when a control is in place a common reason for it failing is that people don't think it is worthwhile.

This is caused in part by human psychology. We tend to look at the future with blinkers on. We can't easily imagine something happening so we conclude that it won't. But since when was our lack of imagination a guide to the future? The system could be showing the wrong numbers? No way. The CEO could be fiddling her expenses? Impossible. Our biggest customer could be far less successful than it appears to be? Ridiculous. The great results of our Singapore division are a massive fraud? Preposterous!

You can probably think of plenty of examples from your own experience.

When we try to get people, line managers for example, to suggest risks as part of a self assessment or controls design exercise one of the things we must try to do is open their minds to what might happen in future. There are various methods of doing this.

The research set out to test one such method by comparing the "risks" and "actions" thought up by participants in response to differently worded instructions.

When participants loaded the experiment page they were offered a choice of four scenarios. This was to ensure people would have some knowledge of what they were analysing. (Planning an extension to your house was the most popular, while planning a wedding was the least popular.)

The instructions asked participants to list one of the following: (1) "risks", (2) "sets of risks", (3) "risk factors", or (4) "areas of uncertainty". They also had to think of actions that might be taken to manage what they had thought of.

The main conclusions were that:

• Overall, the most common type of response concerned bad things that might happen.

• However, in the "areas of uncertainty" condition there was a strong tendency to suggest items that were "open", meaning that they could be either good or bad. For example, "Overall cost" is an area of uncertainty and not specifically a good or bad outcome or set of outcomes.

• Open responses were also common in the scenario about buying a company.

• There was a greater tendency to suggest actions that involved finding out more or doing more analysis in the "areas of uncertainty" condition.

• No participant listed risk factors when asked to do so, and in fact the responses to all instructions mentioning risk were roughly the same: a list of bad things that might happen.

My impression from reading all the responses and analysing them is that "areas of uncertainty" provokes a different kind of thinking that is more open to future possibilities (both good and bad) and more likely to lead to actions that involve finding out more. This suggests greater awareness of uncertainty.

These findings are also relevant to people who are trying to integrate risk and opportunity management.

Full details of the results are available in "Results of an experiment in risk and uncertainty management".

Matthew Leitch
www.internalcontrolsdesign.co.uk
matthew@internalcontrolsdesign.co.uk