March 19, 2007

 

Companies spend millions upon millions of dollars developing software with specific applications for their business, and it is not unusual for these development efforts to languish, run far over budget, be scrapped, or simply fail. In cases where the projects are completed, many companies are not measuring whether their efforts were successful.

 

Statistics that flesh out this picture are staggering. A ComputerWorld survey found that 68 percent of respondents said their companies do not measure the results of information technology (IT) projects. In addition, a 2004 Standish Group CHAOS report predicted that the software-project success rate would be 35 percent in 2006, based on increases of a mere 1.7 percent each year, according to data gathered since 1994.

 

Then there are news reports of big-ticket projects gone awry. In 2004, the Ford Motor Company abandoned a web purchasing system it had just deployed with an estimated cost of around $400 million. A more recent incident concerned the nation’s top law enforcement and anti-terrorism agency. The headline on the August 18, 2006 Washington Post account put it succinctly: “The FBI's Upgrade That Wasn't: $170 Million Bought an Unusable Computer System.”

 

Whether the goal is staying competitive in the automobile market or keeping digital data relevant to national security, the stakes for an organization could not be higher. For businesses, monitoring and measuring application development requires a heightened commitment and awareness, and skill sets that may be completely unfamiliar to personnel outside the IT department. However, there is positive news: With the recent publication of Val IT, companies now have a conceptual framework for aligning IT projects with business objectives, as well as for monitoring progress and measuring results.

Not measuring up

Before a new conceptual framework can take root at a company, it is good to clear away some of the conceptual underbrush that often clutters the path to IT-project success.

To begin with, how does the company define success? Is it to bring the project in on-time, on-budget? Is it to reduce costs or reduce head count? Is it to get off a legacy system and onto a new system? In too many cases, companies have not defined or set-up measurements for a successful implementation, and that results from simply not having a business case.

 

Project requirements that are ambiguous, unclear, incomplete or contradictory will create an unstable foundation for the effort itself. No matter how good “production tools” are in developing software, they cannot overcome a misdirected specification. The failure rate for projects with errors in requirements is estimated to be around 50 percent.

When we find that the project objectives, benefits and requirements are fuzzy, it is not surprising that, in many cases, the right stakeholders are not involved in the effort.

For instance, there may be a lack of CEO support and oversight, with someone at the VP-level signing off on the project’s details. Many times, when a project is delayed, changed or canceled, it is because a CEO did not realize that, say, $30 million was going to be spent for what was initially understood simply as “changes” to an IT system. Corporate culture and egos also play a role in this trend. IT departments may look askance at participation from the business side since IT personnel are the only ones who know how to implement the changes. For their part, business personnel unfamiliar with current technology may be all too willing to let IT take the reins.

An ideal project team would include a blend of IT, business unit and risk management personnel and possibly also external stakeholders, such as shareholders and external customers. Customers are stakeholders that are often forgotten.

 

From my experience of working with companies, I have found that risk management and Internal Audit (IA) organizations often struggle to bring together the right skill sets for evaluating complex IT projects. IA has the opportunity to be involved, for example, with reviewing the project budget, making sure the requirements are being designed appropriately, evaluating test results and reviewing the final conversion plans for the new system.

Using the Val IT framework for benefits/ROI analysis

Whether or not they have the skill sets, IA and risk management organizations have not always had a good framework to measure business cases and return on investment (ROI) for these projects. It is a very difficult thing to do, even for IT departments.

 

This need sets the stage for the Val IT framework developed and published in April 2006 by the IT Governance Institute, a research think tank headquartered in Rolling Meadows, IL.

 

Val IT complements the CobiT framework and is not meant to replace it. CobiT aims to implement and measure controls to assist with the delivery of IT services; Val IT measures business benefits and aims to align projects with business objectives. The gist of Val IT is: “What we should be doing,” while the gist of CobiT is: “How we should do it.”

 

The Val IT framework is organized into three key processes and related management practices:

 

• Value Governance. The key processes for a company here are providing a strategic direction (How do they manage each project and its business case?); and establishing portfolio parameters (What parameters are used to make sure that the proper budget is allocated to deliver benefits for a project?).

• Portfolio Management. These activities center on allocating and monitoring the investments that are needed: hardware, software, additional resources, consultants, and so on. As part of the business case, this process ensures that portfolio activities are properly aligned and are going to be funded as budgeted.

Monitoring and reporting on portfolio performance is a key activity. For example, if the project goal is to shut down a legacy system and save money, you need to monitor this goal to make sure that the savings obtained from shutting it down align with the original business case.

 

• Investment Management. Activities, in this area, relate to putting together the business case, executing the project and “retiring” the project once it has delivered the value. There is a lot of work to be done within this category: identifying the business requirements, defining the project and analyzing alternatives, assigning accountability and documenting the business case. Making that case rests on answering four key questions: Are we doing the right things? Are we doing them the right way? Are we getting them done well? Are we getting the benefits?

 

By carefully identifying these processes, Val IT provides a clear understanding of the project and transparency of the associated costs, risks and benefits. In addition, it is more likely the project team will select those investments with the highest potential return. With a better understanding of what needs to be done, the project team can reduce costs by not performing unnecessary activities, taking early corrective action when something goes wrong, and terminating investments that are not delivering the expected benefits. Ultimately, Val IT reduces the risk of failure and the number of “surprises,” such as unexpected costs, that may emerge.

 

Measurement options

Val IT covers the entire lifecycle of the project. It is not a one-time analysis. For more of a milestone-type analysis, there are several measurement options that can be used at the beginning and the end of a project.

 

Payback Period measures how long it will take a company to recoup its investment. It is a very simple ROI measurement that does not account for inflation and only measures performance until the end of the time measurement period.

 

A somewhat more challenging ROI measurement, though still relatively simple, is the Net Present Value (NPV), which is one of the most commonly used measures of cash flow, particularly for multi-year efforts. A program such as Microsoft Excel takes much of the difficulty out of NPV, which allows consideration of factors such as cost of capital, interest rates and investment opportunity costs.

 

Still another technique is Project Risk Management, which refers to the processes concerned with identifying, analyzing and responding to uncertainty elements. Most IA/risk management organizations are measuring for some risk, but they are not taking a comprehensive look at the picture. The risks can be measured during the entire lifecycle of the project, beginning when the project is being estimated and resources assigned, ending with a post-conversion analysis of the project implementation efforts.

 

A final measurement that can be used is a concept called Stage Containment. This measurement takes more time than the first two because you have to be sure you are gathering all the failure and defect issues that emerge during the course of the project before asking, “How did this come about? Where was it found?” The analysis of the identified issues may lead to a root cause failure in the project development lifecycle.

 

Knowing the warning signs

Monitoring and assessing the project during its lifecycle may bring to light some indicators that success may not be achieved. Some of these indicators were discussed earlier but they bear repeating.

 

Lack of executive support. CEOs and other executives may know the IT project is being undertaken but they may not understand the cost or benefit of the effort. Many times, an economic downturn at a company will result in IT projects getting “the ax” because management never truly understood that the project had any strategic value and would deliver some great benefits upon completion.

 

Lack of project scope and supervision. Here is an example of what should not happen: Employee A insists that he or she must be able to perform a particular action in the new system because it is done in the current system. Strong supervision means being able to say, “No, you have to learn the new system because we cannot change the inherent best practices in the application for that one single requirement.”

 

To take another example: If the company is going to use off-the-shelf software like SAP, one challenge is to avoid customizing the package too heavily. When you customize, you inherently introduce risk. The next time an upgrade or a fix is rolled-out, you have to invest a tremendous amount of effort to ensure those customizations are not lost.

 

No detailed and workable project plan with a “reserve buffer.” Companies often have a “high-level plan” showing what is happening from month-to-month. We recommend developing a deep, detailed work plan showing activities by day and by task, with assigned resources and interdependencies between tasks noted so you can start to understand the project’s critical path.

Having a reserve buffer means having enough contingency built into the schedule and budget to address unforeseen circumstances that may delay the project or drive up the cost. Typically, a reserve of 10 - 15% of the overall budget may be needed.

 

Lack of project schedule and resource management. This is a key concern. Many times, personnel are assigned to a project part-time, so that they have to spend, say, half of their time on regular workday responsibilities. Inevitably, this split does not work well for the IT project, as employees are drawn back into their day-to-day work. So, resources and the project schedule should be monitored at least weekly, if not daily. Every time an employee is doing other work, or out sick or on vacation, that has implications for the project, i.e. how do you plan to catch up?

Still more warning signs:
 

• The project team is not a cross-section of stakeholder groups;
• No detailed business case with specific benefits has been made;
• Poor planning of resources such as training, support and post-IT implementation costs and planning;
• No formal plan exists for communicating project news and results;
• There are no ongoing “lessons learned” assessments with all parties involved.

 

Internal Audit adds value

IA and risk management organizations may not have historically been involved with IT projects for a variety of reasons, such as a lack of system-development experience and perception that such departments are not able to bring value to the table. By utilizing such tools as Val IT and measurement techniques, these departments can now begin to take the proverbial “baby steps” toward fulfilling their role as key stakeholders.

 

An IA department, just beginning its involvement with IT projects, should start by not taking on too many tasks at once. Instead, they should pick one aspect of a project, measure it with a finance, quality or risk metric, and in effect, make it a pilot project from which they can learn. Ideally, they would look at the project schedule, determine where the most risk is located, identify the key milestones and pick a spot to use one of its frameworks. They can then report back to management with what they found “at this point in time,” instead of being intertwined in a complex, multi-year project in their first time out.

 

It is also a good idea for IA to get involved right at the beginning of a project rather than later on. By participating at the start and understanding the business case, resource allocation, project schedule and so on, IA can help head-off many potential issues.

 

Clearly, it is a challenge for business departments without IT experience to become part of an IT process and start poking around. As key stakeholders, they are likely to recognize the costs of not being involved. Whether it is an embarrassing high-profile systems failure or project that implodes after spending millions of dollars, the stakes are too high to stand on the sidelines.