AuditNet®
Give Your Stat Sample the Boot!
by Mike Blakely
Part 2
In last month’s article we discussed the situation where a random sample has been taken, but almost all the sampled items are correct, i.e. very few differences between the examined value and the audited value. Using the traditional sample assessment approach, which is based on the Central Limit Theorem, the resulting precision interval is often very wide, meaning that the auditor may not be satisfied with the results obtained.
However, there is an alternative procedure, which may be very well suited to this situation, and it is called “boot strapping”. The term arises from the technique itself, which is to pull yourself up by your boot straps, in this case by relying simply on the sample that has already been selected, and applying some computer resources to arrive at a more precise sample assessment.
If computer resources are to be used, then special computer software is required. There are two primary approaches – build or buy. If you already have a copy of a statistical software package such as SAS, Stata, SPSS etc. you may be able to run the procedure directly using that software. But if you don’t, then probably the most feasible alternative is to build your own application. To illustrate the process, there is an Excel workbook which has all the code needed to run the boot strap process. Excel was selected mainly because almost every auditor has it. However, the code (which is written in the VBA language) could very easily be converted over to other computer languages as PHP, Java, C++ etc. All this code is made freely available for any use.
What the workbook named “QS.xls” contains:
Besides the code, the workbook contains several worksheets as follows:
- Samples – The sampled items and the audited sample values
- Results – Sheet created by the program to report the results
- Data – The replacement sample results (intermediate values, shown only as documentation)
The sheet named “Samples” consists of four columns. The first column is the sample number, in this example the samples are numbered from 1 to 100 for a sample of 100 items. The next column is the examined value, i.e. the value that is being audited, typically a “book value” amount. The next column is the audited value. For this example almost all the audited values are the same as the examined values. The final column is a computed value, the difference between the examined and audited values. In most cases, this difference is zero.
How the code works
The code takes the following steps:
- Stores all the audit sample difference values in a work area named “A” for quick access
- Repeats
the following task 10,000 times (number can be set) which consists
of the following sub-steps:
- Selects a random sample from the sample, with replacement. The number of samples is the same as the sample size. For example if the sample consists of 30 items, then 30 samples would be selected. With replacement means that any sample item can be selected zero or more times.
- Calculates the total amount of the sampled items
- Stores this total amount in a work area named “S” for later reference
- Sort the items in the work area “S” in ascending order
- Determine the lower limit by selecting the appropriate percentile amount, depending upon the confidence interval desired. For example, if a 90% confidence interval is desired, the 5th and 95th percentile values would be chosen. If an 80% confidence interval were desired, the 10th and 90th percentile values would be selected.
- Reports the sampling criteria on the sheet named “Results”
- Re-computes the upper and lower limits using the traditional method (based on the Central Limit Theorem) and reports these results on the sheet “Results”.
Operating the system
To operate the system, enter the sample results on the sheet named “Samples” and click the button labeled “Run Simulation” on the sheet named “Samples”. The default number of simulation runs is 10,000. This vale can be changed by changing the statement in the code “Const SIMSIZE = 10000”. The number of items in the population is specified by changing the value in the code “Const POPSIZE = 125000”. The default value for the confidence interval is 90% which is specified by 1 – 90% = 10% “Const CONF = 0.1”.
Example results obtained
Various “made up” samples were run to obtain and compare the results using each method. In all instances, an assumed sample size of 100 was used, a population of 125,000 and a confidence interval of 90%. The number of non-zero values ranged from 2 to 5, with all remaining items being zero. Obviously these are highly skewed populations, which makes sample assessment using traditional methods very conservative indeed, and in some cases simply inaccurate.
Example results
|
Sample number |
Non zero items |
Boot strap lower limit |
Lower limit using traditional method (Central Limit Theorem) |
||||
|
1 |
|
102,350.00
|
-33,352
|
||||
|
2 |
|
102,350.00
|
-4,213
|
||||
|
3 |
|
102,350.00
|
-34,967
|
||||
|
4 |
|
100,000.00
|
26,130
|
||||
|
5 |
|
61,512.50
|
28,990
|
Summary results
In every case, the Boot strap lower limit amount is higher than that obtained using the traditional method. Because the lower limit is typically used by government auditors pursuing underpayments, the traditional method is more conservative and results in smaller dollar recoveries in audits based upon the Central Limit Theorem. However, this means that taxpayers obtain fewer recovered dollars.
Certainly samples assessed using upon the Central Limit Theorem are valid, yet conservative. Recall that the Central Limit Theorem is valid for “large” samples (what is large?). Audit lore has it that “30” is large. But in actuality the size required depends to some extent in just how skewed the population is. In the examples we have presented here, the samples are highly skewed.
In each of the cases above, because a confidence interval of 90% was used, the audit can conclude, at a 95% confidence level, that the population values are at least as much as the lower limit. Stated a little differently, the auditor could state that if the test were repeated 100 times, on average, 95% of those tests would have lower limit estimates which are lower than the actual population value.
In conclusion, there are several reasons why auditors should consider “boot strapping” in instances where their samples are highly skewed:
- Boot strapping makes no assumptions about the population – i.e. it works on both skewed and non-skewed populations
- It provides a “sanity check” on sample assessment results
- It can be more precise than traditional methods (based on CLT)
- It lessens the need for larger sample sizes (or obtaining additional samples)
About the author
Mike Blakley is a CPE instructor for various audit topics and can be reached at Mike.Blakley@ezrstats.com.
Article citation
Kvanli, A.H. and Schauer, R., A
The Bootstrap: What the Government Auditor Should Know,@ Journal of
Government Financial Management, Vol. 51, No. 3, 2002.
Web link to Article
The opinions, beliefs and viewpoints expressed by
the various authors and forum participants on this web site do not
necessarily reflect the opinions, beliefs and viewpoints of AuditNet®