Auditing Software Development Projects

Software development project audits are performed to provide an independent assessment of whether a project follows established development methodologies and procedures, meets organizational needs, and includes adequate security and management controls. This article discusses the various steps of a software project audit, including planning, methodology assessment, review of the various development stages, and reporting of results.

Planning the Audit

Determining which development project needs to be audited should be based on the annual audit plan, the criticality and risk of the application, and any compliance requirements. For instance, if the audit is part of a regulatory compliance effort such as SOX, a sample of financially significant projects will need to be selected according to established sampling criteria. Once a project is selected, the audit should be planned by identifying the following:

The kind of review to be performed (i.e., pre-implementation, parallel or concurrent, or post-implementation review).
All of the legal, regulatory, and policy aspects the company must comply with, if any.
The scope of the review.
The start and end dates of the review.
The process for reporting observations and recommendations, as well as for following up on agreed actions.

As a first step, auditors need to contact the project manager and business owner to inform them of their involvement in the project. Doing so will enable management to keep auditors updated on any project meetings and planning sessions and help them obtain the documentation they need to perform the audit. It will also establish senior management support to allow the audit to be successful and the audit recommendations to be implemented.

As a general rule, it is important to become involved in the project as close as possible to the start date to ensure that issues are addressed early and that adequate controls are built into the application. This is because it is much more expensive to retrofit fixes and controls into existing applications than to ensure they are there in the first place.

Organizational Policy

The first step in the audit is to determine whether the organization has a formal Software Development Life Cycle (SDLC). This is the structured methodology that guides the development of application systems. It typically begins with the project initiation and extends through the feasibility study, business requirements, functional specifications, development, testing, implementation, and post-implementation. For each phase, auditors need to ensure the following:

Relevant tasks and activities are clearly described.
A measurable end product or deliverable is defined.
The individual(s) or role(s) responsible for the execution of the phase is identified along with the required approvals and signoffs before proceeding to the next phase.

Finally, auditors need to review any documents or examples that are available as part of the SDLC. Two important documents include checklists that cover each phase of the process and templates that are associated with project deliverables.

If an SDLC methodology is not in place, the audit can be conducted against the organization’s current best practices with a recommendation that an SDLC be developed and implemented to guide future software projects.

Project Initiation

A software development project is usually initiated through a formalized project request by management. Auditors need to ensure that the project request is present and includes the following:

An overview of the business problem being addressed.
The desired system features.
The expected users of the system.
A high-level cost and benefit estimate.
The criticality of the proposed system.
The time frame for completion.
The approval of the senior executive who is authorized to initiate requests.

As a general best practice, IT departments should have a formalized system to receive, organize, and prioritize user requests. The auditors should determine if the timeframe within which the request was approved was adequate and whether the users were notified of any decision in a timely manner.

Feasibility Study

The feasibility study is a preliminary assessment that determines a project’s viability, whether to proceed with system development, and any alternative approaches. This is generally required for major development or enhancement projects to ensure the solution chosen is economically or technologically appropriate. When reviewing a feasibility study, auditors need to determine whether it addresses the following areas:

The software’s or system’s business objectives.
The project’s priority and impact.
Any alternative courses of action and their evaluation based on technical, financial, and operational factors, such as time and cost estimates, resource requirements, a cost/benefit analysis, and risks.
The proposed solution, which can include a recommendation that the project is not developed.
Start date and completion timeframes.
Reviews and signoffs by appropriate management.

Business Requirements

Business requirements describe current and future business needs to ensure they are understood and addressed before developing the system. They detail the business functions the system is required to support, usually expressed in terms of broad outcomes rather than specific functions the system must perform. Auditors need to identify if they:

Clearly document current and future business needs.
Identify security, control, and privacy issues.
Include feedback from all potential user areas.
Have user and IT management approvals.

Functional Specifications

The functional specifications document describes in detail a product’s intended functionality from a technical point of view. In this document, business requirements are analyzed and converted to produce a preliminary design of the proposed system. As part of the audit process, auditors need to determine whether the functional specifications include the following elements:

Expected data input, processing, and output.
A mockup and design of the user interface screens.
Reports, including audit and logging reports for monitoring and technical support.
Interfaces to other systems.
Internal control and security requirements.
Transaction volumes.
New hardware and software requirements.
Infrastructure and support considerations.
IT management reviews and approvals.

Systems Design and Development

Before actual programming can begin, a system design document is created that provides a detailed technical description of the proposed software and gives the software development team overall guidance on the actual software architecture. This document also describes the desired software features in detail and serves as inputs for one or more software programs. As part of their work, auditors need to review the system design document and check that:

Inputs are defined in detail, including screen designs.
Outputs and reports are defined in detail.
The functional logic of program modules is documented.
Data entities, relationships, and flows are documented.
Controls are in place to prevent unauthorized access and to maintain audit trails of user activity.
Responsible departments signed off on the design.

Note that any changes to functionality during the system design phase should require the functional specifications document to be updated and reviewed by the user. This should be addressed through the established change management process. As far as the actual programming effort is concerned, auditors need to make sure that development takes place in an environment that is separate from the production environment and that the development staff do not have access to production. Finally, if the programming effort is partially or completely outsourced to a third party, auditors need to assess the adequacy of and compliance with outsourcing controls.

Testing

A testing strategy must be developed and followed for new or updated applications. At a minimum, testing should consist of the following:

Unit testing (testing that validates whether individual programs or modules are working properly).
System testing (testing conducted on a complete system to evaluate its compliance with specified requirements).
UAT (user acceptance testing performed by end users to determine whether the software meets the business requirements).

Throughout the testing phase, auditors need to identify whether written test plans and scripts are available and adequate. For instance, test plans should include the test set-up and scope, testing procedures and data, expected results, and sign-off from the appropriate staff. In addition, auditors need to determine whether all testing is performed in a test environment that is separate from the production environment, whether test results are logged, and whether tests having unexpected results are adequately re-tested. Keep in mind that nothing should be installed in the production environment until it has been successfully tested in a test environment and formally approved by the business user(s).

Data Conversion and Migration

If the system being developed requires data to be converted from an existing system, auditors need to obtain and review the conversion plan. At a minimum, auditors need to determine if:

Source data is identified, verified, and correctly mapped to the new system.
The data conversion and migration is tested to confirm it is complete, accurate, and valid.
Any data conversion and migration mismatches have been logged and corrected.

Implementation

In this phase, the new system is installed and made operational in a production environment. The phase is initiated after successful user acceptance testing and sign off. Documentation should have been developed and users trained before being given access to the system. Auditors need to check the following:

A detailed implementation plan was developed and followed.
A rollback contingency plan is available.
The approval of business, development, and operations management has been obtained.
User and operation documentation is available.
Training has been conducted for users to acquire the knowledge necessary to use the system prior to the system’s implementation.
Support personnel are available and accessible to users.
Backup procedures are in place.

Post-Implementation Review

A post implementation review is performed to ensure that the system adequately meets the requirements of the business. After a few months of live operation, auditors should be able to determine if the expected benefits of the new system were realized and users are satisfied with the new system. At this stage, auditors need to review the problems that occurred throughout the project and whether the subsequent actions taken were adequate. If differences remain between expectations and actual results, auditors need to determine whether they are due to:

Any misunderstanding or incomplete user requirements.
Inadequate design, programming, or testing.
The need for additional user training.
Other reasons.

Each of these items can help auditors evaluate the current situation and offer guidelines for future projects.

Reporting Audit Results

While conducting an audit, auditors can submit interim reports as issues are identified or in a final document that lists all the issues raised during the review. In either case, the audit plan should include timely notification of any control weaknesses so that they can be resolved prior to system implementation.

Audit reports should address risk and issue causes, effects, and suggested mitigation actions; IT control strengths and weaknesses; and lessons learned that can be applied to future projects. In addition, auditors need to discuss their findings with management and obtain their commitment to ensure that corrective actions are implemented and deadlines for remedying significant deficiencies are followed. To this end, auditors should follow up after an appropriate period of time to make sure that issues were resolved satisfactorily.

Moving Forward

This article outlined the steps and activities to successfully perform a software project audit. By following a systematic approach, auditors can help provide an accurate and objective assessment and increase the likelihood of the project’s success. In addition, audit observations and recommendations will help the organization avoid repeating earlier mistakes in future software development projects as well as produce useful best practice guides. By providing an independent assessment of a software project, auditors can make a significant contribution to IT project success.