Internal auditors play a valuable role in ensuring that IT investments are well-managed and have a positive impact on an organization. Their assurance role supports senior management, the audit committee, the board of directors, and other stakeholders. Internal auditors need to take a risk-based approach in planning their many activities on IT project audits. With limited audit resources, auditors must focus on the highest-risk project areas, while adding value to the organization. Audit best practices suggest internal auditors should be involved throughout a project’s life cycle — not just in post-implementation assessments.

Converting from old to new computer systems is an important, but often underestimated, aspect of IT projects. Implementing a new system usually requires a variety of system changes, production data conversion and migration, and new operational policies and procedures. Each of these areas poses significant risk to the organization during the actual system conversion. Consequently, conversion audit efforts typically focus on reviewing plans and results for:

1. The overall implementation of the IT solution.
2. The systems that are required to implement the IT solution.
3. The operational changes required within the organization as part of the implementation.

When planning system conversion audits, auditors need a good understanding of the business requirements for the system, the project’s risks, and how the proposed system will work. To understand the solution, they must identify the various operational and system changes that will be implemented. Also, because project information usually becomes available over time, some audit planning will have to be completed on an iterative basis.

AUDIT OBJECTIVES
Internal auditors’ involvement with a system conversion typically involves reviewing and assessing the overall project plan and project management. Auditors should also assess the accuracy and completeness of the systems and data requirements for the IT solution, including:

• Evaluating and monitoring management’s project plans for the various system changes that are required.
• Assessing the completeness and appropriateness of management’s systems and database design, including the security and privacy aspects.
• Reviewing the user-acceptance and parallel test planning and results to demonstrate successful end-to-end system operations and preparedness for implementation. In a parallel test, the project team simultaneously tests both the old and new systems using the same data and compares their results in a comprehensive manner.
• Reviewing the startup of production systems and associated client data to ensure data integrity is maintained and back-out plans in the event of a problem are effective.

In addition, auditors should assess the accuracy and completeness of the start-up of operational responsibilities within the organization by:

• Evaluating and monitoring management’s project plans for the various operational requirements.
• Assessing the completeness and appropriateness of the operational policies and procedures that are developed.
• Reviewing the training materials and training strategies to help staff learn the new systems and practices being implemented.

Finally, auditors should assess the risk-management processes applied by project management to ensure that they are followed throughout the project. Identifying and assessing the risk involved during the various stages of the project will reduce the risk to the organization by ensuring that implementation of the IT solution is controlled and accurate. The audit objectives need to be finalized during detailed audit project planning to ensure all bases are covered appropriately.

EIGHT-STEP AUDIT FRAMEWORK/PROCESS
The following audit framework/process describes the steps to be completed by internal auditors for a typical system conversion. These straightforward and flexible steps include the IT and non-IT aspects of the conversion. The actual level of audit involvement and audit approach should be determined by internal auditors during a detailed audit project planning phase when the first step of the process is finalized. (See sidebar, “Level of Audit Involvement in System Conversions.”)

Depending on a variety of factors — such as the size, complexity, and time frame of the systems project — this generic audit process should be refined and finalized during the detailed audit project planning phase. Reporting of audit results can also be flexible. For example, if the auditors feel it is important to communicate concerns, they can do so throughout the project; otherwise, they should normally produce written pre-implementation and post-implementation audit reports.

By being involved early in the IT project, auditors can informally raise questions and suggestions that influence a project without actually making formal audit recommendations. Auditors should voice concerns during the project if they believe management is not taking the more informal recommendations seriously and they feel their concerns need to be escalated to senior management or the audit committee.

Auditors typically report to the audit committee twice a year about IT projects, communicating audit efforts that are in progress but have not been reported to senior management.

1. Understand the IT initiative and project plan.
Internal auditors should review with both the project’s management team and the business units the scope of the initiative, supporting project plans, high-level implementation plans and schedules, and proposed implementation programs and project controls. A good understanding of the overall solution is needed to provide the big-picture context in support of the audit risk assessment and detailed audit-planning effort. It is important to understand how the system benefits the organization and what the system’s role will be in delivering business objectives. It is also important that the project team develop well-documented objectives and have them approved by management.

During this step, auditors should also finalize the system conversion audit plan by modifying the generic audit process to reflect the actual IT project’s objectives, issues, and assurance needs. The high-level risk assessment would further guide the system-conversion audit-planning process.

2.  Assess the project and implementation plans.
Internal auditors should complete an assessment of the appropriateness of the project plans, implementation and testing plans, and data-conversion program specifications that are being developed. Auditors should provide feedback about their assessment of the implementation approach to the project’s management team.

The project team and management benefit from receiving an independent assessment of the project efforts and plans, because the intense pressure to show early progress can cause some important aspects of planning to be overlooked. Having internal auditors review the early project efforts further assures stakeholders that all important aspects of the project have been considered in planning. Auditors’ views on the new system’s internal controls can benefit the implementation of the overall solution, which may not be as robust as mature legacy systems that have had years of controls hardening.

3. Assess the data mapping efforts and the operational changes planned.
Internal auditors should assess the adequacy and completeness of the planned data conversion control reporting, including a review of the system data mapping for the implementation of changes to production data, if any. Data mapping sets the necessary foundation for maintaining the integrity of the production data during implementation. In addition, the historical requirements for data can be extensive. For example, new financial systems for U.S. companies must be able to store and provide access to historical data for seven years to meet Internal Revenue Service requirements.

During this step, internal auditors need to assess the adequacy and completeness of data mapping for implementing production data by reviewing the systems data mapping itself and the controls and reporting with regard to this activity. Special attention should be given to the impact the data conversion has on other systems that are not involved in the conversion. This is a high-risk area in all conversions, because downstream systems may be affected in unexpected ways.

4. Assess the adequacy of testing plans.
At this stage, internal auditors should assess the adequacy of the testing plans, the data-conversion control reports to be used, and test scripts involved with the implementation. Auditors should ensure appropriate project resources have been assigned to this activity, so that the level of testing efforts reflects the risks involved. Auditors should review the assignment of responsibilities (i.e., Who does what?) and related service level agreements for completeness and appropriateness.

Auditors should also review the project’s outputs regarding business and IT processes envisioned for implementation. Finally, review and feedback on the implementation plans should be provided to the project management team, where appropriate.

5. Complete a risk assessment of the operational changes planned.
Internal auditors should develop a detailed understanding of the operational changes that are being planned and facilitate an independent risk assessment of those changes. Without an ongoing risk assessment process, surprises will occur frequently, with negative consequences to schedule, cost, and quality. Ideally, the results of the audit analysis should be factored into the project team’s plans and focus. The audit risk assessment should be documented in a one- or two-page high-level summary called an internal audit readiness analysis.

6. Complete an implementation readiness assessment.
Internal auditors should complete a high-level readiness assessment of the organization’s ability to implement the required operational and system changes. This analysis will identify whether or not additional management efforts are required to complete the preparation for the actual conversion. Auditors should review the documentation being developed to support the solution’s implementation and operation — user training materials, new and refined policies and procedures, internal and external communication, and other key project outputs — as determined in the previous steps.

The audit conversion readiness assessment should be documented in an internal audit worksheet — usually a one- or two-page high-level summary. An audit report should be provided to management before the solution’s implementation. Auditors should also attend key project management checkpoint meetings and provide their views verbally as part of the audit feedback to project management.

7. Develop and execute the audit verification plan.
Internal auditors should complete the detailed audit program to review production implementation. At implementation, management and the board will look to internal auditors to provide additional assurance that the implementation is executed prudently and successfully. The audit tests will usually focus on reviewing the results of the project team’s and business unit’s test results. This includes reviewing the results of the start-up at the first site and monitoring the rollout at the other sites if the conversion is a multi-site implementation. A review of the testing of the initial production operations using the new solution should also be completed.

8. Complete the overall audit analysis of the implementation.
Internal auditors should complete the audit analysis of the results of the implementation and the management actions that have been completed to address any issues raised during the actual implementation. For major conversions with hundreds of implementation activities, some operational issues may need to be addressed during implementation that were not considered ahead of time. For example, an obscure data conversion problem may occur for a minor data element, requiring manual corrections to be instituted to allow the implementation to go forward. Management review of this manual effort following implementation might be recommended to ensure nothing significant was missed.

Because decommissioning the old system can be another challenge facing management, ensuring appropriate efforts are completed in this area can be productive. A post-implementation audit report on the adequacy and accuracy of the implementation should be developed and presented to management. Finally, auditors should complete the working paper file.

AUDITS ADD VALUE
Auditors can bring considerable value to an organization by evaluating both the IT and organizational aspects of system conversion projects. Auditing these conversions provides assurance for management and the board that all that can be done is being done. Because a conversion to a new system is one of the highest risks that an organization can face, internal auditors’ involvement and independent assessment of the issues and project plans can provide value far in excess of the audit’s costs.

Conversion audits should be done in lockstep with the project planning and frequent and timely reporting of the findings. Being involved early in the project initiative is vital to developing a good understanding of the project’s issues, challenges, and risks, as well as what management is doing to address them.

Finally, another excellent audit practice is to complete a short “lessons learned” review of the audit efforts performed throughout the conversion audit and apply them to the generic conversion audit process that the organization has adopted. This review can be used in future conversion audits. If done consistently, this last step can result in a more effective conversion audit process within just three to five audits.

Sidebar: Level of Audit Involvement in System Conversions
Internal auditors’ involvement in an organization’s system conversion initiatives can range from minimal involvement (level 1) to extensive audit efforts throughout every phase of the project (level 10). 

• Level 1: Audit risk assessment during the project initiation phase.
• Level 2–3: Audit review of documentation and project deliverables.
• Level 4–5: Attend project meetings, conduct some interviews, and produce verbal audit reports.
• Level 6–7: Increased audit efforts, conduct more interviews, and produce formal audit reports.  
• Level 8–9: Review all milestones, perform extensive audit tests, and produce formal and comprehensive audit reports.
• Level 10: All of the above efforts.

Internal auditors should determine their level of involvement and approach during the project’s initiation phase. The audit involvement decision should be based on the project risk assessment, as well as factors such as the project team’s project management experience, level of management involvement, size and complexity of the initiative, and impact on the organization if the initiative is delayed or unsuccessfully implemented. The most appropriate audit approach needs to be defined during the audit project planning phase.

Following step 1 of the generic eight-step audit process will complete the definition of audit’s level of involvement. Further adjustment of audit involvement may be required depending on the results of the project’s efforts and auditors’ assessment throughout the project’s life cycle. Another important consideration to discuss with management and the project team is the internal auditors’ roles and responsibilities in attending project team meetings throughout the conversion audit.

BIO/NOTE – This article is based on the author’s experience in completing more than 75 different conversion audits over the years. These ranged from 50-to-80-hour audit projects to completing a 1,600-hour review of a 100,000-person-hour project over a 16-month period. For this latter audit, extensive audit planning was completed for each step of the conversion audit process that is presented here. To make suggestions about Auditing System Conversions, please contact the author at dswanson_2008@yahoo.ca.

1. Improving Mission Performance Through Strategic Information Management and Technology http://archive.gao.gov/t2pbat3/151707.pdf
2. IT Audit Checklist: IT Governance and Strategy http://www.itcinstitute.com/display.aspx?id=2499
3. 20 Questions Directors Should Ask About IT http://www.cica.ca/1/0/0/0/index1.shtml
4. 20 Questions Directors Should Ask about IT Projects http://www.cica.ca/index.cfm/ci_id/39232/la_id/1.htm
5. Maximizing the Success of Chief Information Officers: Learning From Leading Organizations http://www.gao.gov/new.items/d01376g.pdfAcknowledgement – the above article was originally published in “New Perspectives on Healthcare Risk Management, Control and Governance” by the Association of Healthcare Internal Auditors.