Common Problems and Lessons Learned from

94 International Journal of Social Ecology and Sustainable Development, 2(3), 94-111, July-September 2011

Common Problems and Lessons Learned from Managing Large-Scale US Government IS/IT Projects Peerasit Patanakul, Stevens Institute of Technology, USA Saif Syed Omar, Stevens Institute of Technology, USA

ABSTRACT To promote sustainable development, many government agencies initiated large-scale government projects. However, managing such government projects to their success is often a challenge to many project managers. Many projects fail because of their large scope and high degree of complexity. The failure of these government projects has significant impact on sustainable development, both at the organizational level and the public at large. To enhance the success of government projects, this study investigates the management of selected government IS/IT projects in the US; identifies common problems; discusses some lessons learned; and provides propositions for future research. The results of this study will provide significant contributions to the literature and implications to practitioners. Keywords:

IS/IT Projects, Large-Scale Projects, Lessons Learned, Problems, Project Performance, Sustainability

1. INTRODUCTION Among many important issues, sustainable development is the issue that concerns many government and international agencies. Many projects were initiated by these agencies to promote economic, social, and environmental sustainability (World Bank, 2010). Besides the concern at the government and global levels, sustainable development is also a major con-

DOI: 10.4018/jsesd.2011070106

cern at the corporate or organizational level. Referring to the societal notion of sustainable development, the corporate sustainable development is achieved through the combination of: (1) environmental integrity through corporate environmental management, (2) social equity through corporate social responsibility, and (3) economic prosperity through value creation (Gladwin et al., 1995; van Marrewijk & Werre, 2003; Bansal, 2005). To achieve corporate sustainability, firms have to maintain and grow their economic, social, and environmental capital base while actively contributing to

Copyright © 2011, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.

International Journal of Social Ecology and Sustainable Development, 2(3), 94-111, July-September 2011 95

sustainability in the political domain (Dyllick & Hockert, 2002). To promote sustainability, many projects were initiated at international, government, and corporate levels. These projects, especially at the government and international levels, are typically large in scope and scale (World Bank, 2010). Some of them can be categorized as “megaprojects, ” which often refers to an extremely large-scale project that its cost is usually more than one billion dollars (Flyvbjerg et al., 2003; Koppenjan, 2005). With its large scope, uncertainty, complexity, politicallysensitivity, and involvement of multiple partners, successfully managing these projects are very challenging. The projects typically attract public attention because of their substantial impacts on communities, the environment, and its budget (Flyvbjerg et al., 2003). With the above challenges, the large-scale government projects often ended up with poor performance (Flyvbjerg et al., 2003). Sometimes, the project fails to deliver in the terms that justify the needs of the customers. Since these projects were initiated to promote sustainability, their performance directly impact the accomplishment of sustainability set forth by government and international agencies. To help promote the success of the government projects, the objective of this study is to identify common problems in managing largescale government projects. At this stage, our primary focus is on the IS/IT projects of the US government. Five projects were randomly selected from different US government agencies. Besides their contribution to the literature, the research findings may provide practitioners with a better understanding of common problems in managing large-scale IS/IT projects so that they can take appropriate measures to prevent such problems. The success of the projects may, in turn, contribute to the accomplishment of sustainable development. To better understand the issues in managing large-scale government projects, the literature on large-scale project management is reviewed in the next section, followed by the sections on research method, descriptions of projects in this study, find-

ings and discussion, managerial implications, and conclusion.

2. BACKGROUND Many researchers have studied the management of large-scale projects and indicated that a majority of them have ended up with poor project performance. Studies have also been conducted to investigate the causes of poor performance. The results show that the main causes of cost overruns of megaprojects stemmed from a lack of realism in initial cost estimates, low level of contingencies, insufficient consideration on changes in project specifications, designs, and exchange rates, and undervaluation of price changes, expropriation cost, and safety and environmental demands (Flyvbjerg et al., 2003). The major causes of schedule delay of a major public transportation project came from lack of owner’s abilities and strategies to manage hi-tech megaprojects, lack of appropriate scheduling tools, underestimating the technical requirements, and public resistance due to environmental concerns (Han et al., 2009). Project complexity is also one of the main causes of poor performance (Jolivet & Navarre, 1996). Besides possible technical difficulties, the complexity of large-scale projects stems from a range of ambiguous and uncertain external and internal forces. Researchers have found that often times, the project teams managed projects to the best of their abilities in the context of very complex operations, paradoxes, uncertainties, influences, and ambiguities that surrounded these projects; the different project outcomes were, however, realized through different approaches that the project team pursued to manage complexity (van Marrewijk et al., 2008). To successfully manage large-scale projects, researchers have proposed different approaches to address project complexity. A project management approach, referred to as a self-organization and meta-rules, is recommend for the management of a large-scale project that involves a large element of novelty or must be developed at an accelerated pace Jolivet and



Navarre (1996). Miller and Lessard (2001) propose managerial strategies and processes for risk management in large engineering projects. With multiple partners involved in megaprojects, several researchers suggest the creation of public-private partnerships (PPPs) (Koppenjan, 2005; van Marrewijk et al., 2008). Others propose the creation of appropriate project cultures. In particular, Clegg et al. (2002) suggest that the project team should design an alliance culture of inter-organizational collaboration. van Marrewijk (2007) observed functional and dysfunctional project cultures throughout the life cycle of one megaproject and suggested that a project team may need to create a distinct culture for each phase of a megaproject.

3. RESEARCH METHOD The main objective of this research is to identify common problems in managing large-scale government projects, in particular, the IS/IT projects in the US. Sample: We randomly selected five projects from the US as our samples. The projects had an estimated duration between 2-9 years and the original budgets ranged from $60 million to $2.5 billion. The execution dates of these projects ranges from as early as the 1980’s to the as recently as the year 2001. The description of each case is discussed in the next section. Data sources: For our analysis, we gathered project information from audit reports generated by the Government Accountability Office. Other independent reviews and reports were also used depending on the availability and the need on a case by case basis. At least 20 reports and related articles were reviewed. All of these reports are available to the public. Content analysis: Using reports from the government auditing agencies of the respective countries, we conducted content analysis (Bauer, 2000) to identify the problems and challenges in managing large-scale

IS/IT projects. For each project, multiple reports were reviewed. To extract the most information from each report, we performed content coding. Based on such coding, we were able to create groups of problems and challenges for each project. We then developed a diagram to illustrate the impact of those problems and challenges on the performance of the project. After analysis within each case, we performed a cross-case analysis to identify common problems and challenges and their impact on project performance across cases. Since we performed content analysis from the audit reports where the problems and some of the causes, including their impact on project performance were already identified by the auditors, our content analysis process involved little subjectivity. To ensure the validity of our findings, we also compared and contrasted our findings with the literature.

4. PROJECT DESCRIPTION Customer Account Data Engine Release 1 (CADE): The project was initiated by the internal revenue service with the goal to replace the IRS’s 30 year old outdated and inefficient data management system that relied on outdated storage technology (magnetic tapes) with a single, authoritative corporate data source for enabling future customer service and financial management applications (GAO, 2001, 2004, 2006; Anonymous, 2004). The new system should help promote sustainability for the IRS by helping the agency better utilize its resources, provide better service to the community, and achieve its long-term goals. In reality, the project lasted between 1999 and 2004 and cost the Internal Revenue Service $60 million though it was supposed to be complete after 3 years from the start at the cost of $98 million. FBI Trilogy: The project was initiated by the Federal Bureau of Investigation (FBI) to



replace FBI’s existing investigative application - Automated Case Support (ACS) system with a Virtual Case File (VCF) system. The VCF system was expect to help enhance the ability of the FBI agents to organize, access, and analyze information. The scope of the project also included the modernization of the IT infrastructure by providing a high-speed network linking the offices of the FBI and updating the workstations and software within each office for every FBI employee (OIG, 2002, 2003, 2005; Anonymous, 2005; GAO 2005, 2006). Upon accomplishment, the new system should help the FBI become sustainable, all in economic, social, and environmental aspects. The project was initiated in 2001 with an initial budget of $379.8 million and was supposed to complete within 2 years. Because of the delays and cost overruns, the project was terminated in 2005 with about $200 million over the initial estimate and without meeting the project’s initially defined scope. Logistics Systems Modernization (LSM): This program was kicked off by the US Air Force with an aim to modernize the Air Force’s Logistics Systems (GAO 1989, 1992). Three out of ten projects were investigated: (1) Requirements Data Bank (RDB): To modernize the Air Force’s automated and manual requirements functions. (2) Contracting Data Management System (CDMS): To provide a database management system to manage contracting information. (3) Depot Maintenance Management Information System (DMMIS): To improve the Air Force’s depot management and maintenance functions. The program was initiated in early 1980 and was expected to be complete in 1989. Because of the inconsistencies and shortcomings, the program delayed until 1994 with about $200 million of extra expenses ($359.4 M to $564 M).

ERP Pilot Projects for the US Navy: These pilot projects were parts of the Navy’s program for operation, maintenance, and modernization of its business systems and related infrastructure. This project should promote sustainable development of the US Navy. The pilot projects consisted of the CABRILLO (Financial Management), SMART (Supply Management), NEMAIS (Regional Management), and SIGMA (Program management) (GAO, 2005). Although these projects had a planned duration of 2 years, the projects were still not complete after six years of execution. The project teams spent $1.5 billion more than the initial estimate of $1 billion. Weather Systems Modernization (WSM) was a program to modernize systems to improve the accuracy, timeliness, and efficiency of US National Weather Service - NWS’s weather forecasts and warnings (Anonymous, 1991; GAO 1994, 2000). WSM consisted of four major systems: Next Generation Weather Radar (NEXRAD), the Next Generation Geostationary Operational Environmental Satellite (GOES-Next), the Automated Surface Observing System (ASOS), and the Advanced Weather Interactive Processing System (AWIPS). Similar to most other major projects, these projects experienced delays (NEXRAD: 9yrs to16yrs, GOES-Next: 3yrs to 8yrs, ASOS: 4yrs to 9yrs, AWIPS: 9yrs to 12yrs) as well as budget overruns (NEXRAD: $340 M/$1.5 B, GOES-Next: $640 M/$2 B, ASOS: $72 M/$120 M, AWIPS: $350 M/$465 M).

5. FINDINGS AND DISCUSSION Based on the data analysis, we identified four major categories of problems across the cases. They are systems management, governance, project management, and contract management, see Table 1. We found all five projects we studied had problems related to system management and governance. In the system management



Table 1. Common problems in managing large-scale IS/IT government projects in the US

System management

Governance

CADE

Trilogy

LSM

ERP

WSM

Total

Requirement management: Lack of clear definition, stakeholder involvement, documentation, and traceability

X

X

X

X

X

5

System architecture, integration: Lack of understanding of complexity, architecture, and integration

X

X

X

X

4

Management oversight: Lack of senior responsible owner and process

X

X

X

X

X

5

X

X

X

3

X

X

2

Internal verification & validation: Lack of process and function for IV&V Project management

Risk management: Lack of proper process and response plan Change management: Lack of proper process

X

Monitoring and control: Lack of proper process and methodologies, project manager, and competency Contract management

Process: Lack of competitive bidding and proper evaluation Managerial issues: Poor perception, relationships, and oversight

category, the problems related to requirement management are the most common. They are the lack of clear requirement definition, lack of stakeholder involvement, lack of requirement documentation, and lack of requirement traceability. Lack of system architecture and integration is found in four projects we studied. The common problems in the governance category are related to the management oversight. We found the lack of project senior responsibility owner and the lack of governance process are the major common problems in five cases. Lack of process and function for internal verification and validation is found in three cases. In project management category, lack of proper change management process is found in four cases. We also found problems related

X

X

X

X

X

X

4

2

X

X

4

X

X

2

to risk management and project monitoring and control in two out of five cases we studied. In contract management, the problems related to the proper bidding and evaluation process were found in four cases. Two out of five cases had some managerial problems such as relationship management and contractor oversight.

A. Systems Management Most of the projects we studied involved the development of systems to support the functioning of related government agencies. The development process of most projects was plagued by technical issues ranging from requirements management to system development and integration. The development of systems architecture at



the systems level and enterprise architecture at an organizational level was also one of the major problems in the projects we studied.

Requirement Management Regardless of the scope of a project, requirement management is among the major challenges in managing projects. It is typically referred to as a process of eliciting, documenting, analyzing, prioritizing and agreeing on requirements and then controlling change and communicating it to relevant stakeholders. It is a continuous process throughout a project. Effective requirement management is critical to the success of projects and lack of requirement management is often seen as one of the reasons for projects failing to meet their objectives. In the cases we studied, requirement management was an obvious challenge. Within requirement management process, requirement identification was a common challenge in all the cases. Being among the earliest steps in project management, requirement identification is crucial to project success. Poorly defined requirements have a cascading effect on the project performance throughout the project life cycle. In fact, identifying the user requirements for a typical IT project is a challenge in itself, let alone when one considers the size and the complexity of a project. In the cases we studied, we found glaring examples where lack of clear definition of requirements negatively impacted the long-term project success. There was clear evidence in the IRS’s Customer Data Account Engine (CADE) project that poorly defined initial requirements led to cost over-runs and schedule delays (Anonymous, 2004). In some other cases, poorly defined initial requirements led to the termination of the projects as was seen in the case of the FBI Trilogy project (OIG, 2002, 2003, 2005; Anonymous, 2004, 2005), and the Navy ERP pilots (GAO, 2005). In these cases, poor definition of the requirements either led to infeasible technical solutions or to the development of systems that were not serviceable. Thus:

Why were requirements not clearly defined? We found several reasons in the cases we studied. First, the complexity of the system was not fully understood or underestimated such that requirements were not developed fully e.g., CADE project (GAO, 2001) and WSM (GAO, 2001). The second reason was the challenge in the requirement elicitation process. We found in some cases (the LSM project GAO, 1989, for example) that the process that the project teams used to gather requirements was ineffective and inconsistent. In the CADE project, the contentious relationship between the owner and the prime contractor hindered communication, which made requirement elicitation difficult (GAO, 2004). Lack of user involvement was found in the National Probation Services Information System (NPSIS) case. Third, although efforts were made to elicit requirements, the elicitation data was not always refined into high-quality requirements— some areas were left open, others were over specified. Therefore: Besides requirement identification, requirement documentation is another common problem. Sound documentation of the requirements also serves as a repository to which the project team can refer to when required. Lack of documenting of requirements has a direct effect on requirements traceability, which is referred to as the ability to describe and follow the life of a requirement, in both the forward and backward direction. That is, from its origins, through its development and specification, to its subsequent deployment and use, and through all periods of on-going refinement and iteration in any of these phases. Requirement documentation and traceability facilitate the understanding of the product under development and helps manage changes. Traceability is also critical to understanding interconnections and dependencies among the individual requirements. In the CADE project, there was a lack of traceability of requirements which was a drawback for the project as the impact of changes to one system on the other systems could not be estimated (GAO, 2001). In collaboration with the literature, problems in requirements management have been



pointed out in literature as a major factor in IS project failure by a number of researchers (Lemon et al., 2002; Yeo 2002; Pan et al., 2008). Some researchers such as Thomas and Fernandez (2008) and Lemon et al. (2002) have said that a clear statement of requirements and selection of the right technology architecture are very vital for project success. Most of the problems in requirements management revolve around poorly defined requirements, ambiguous business needs, poor user input and interaction, poor understanding of system complexity and lack of system architecture. Studying IS projects in a public hospital, Gauld (2007) identified the lack of proper definition of IS needs and objectives – which in turn defines requirements; as well as lack of user involvement in defining such needs and objectives as one of the major failure factors of the projects. Our findings support the literature. However, what make our findings unique is that they were drawn from the large-scale IS/IT projects. By contrasting with the literature, we may be able to conclude that requirement management is a common problem in the management of IS/IT projects regardless of the project size. Based on the research evidence, we suggest the following propositions for future research: Proposition 1: In managing large-scale government IS/IT projects, requirement management is a common problem that negatively impacts project performance. Proposition 1.1: The problem in requirement identification will have a cascading effect on the project performance throughout the project life cycle. Proposition 1.2: Project complexity, lack of requirement elicitation process, and lack of user involvement impact the success in requirement identification. Proposition 1.3: The problem in requirement documentation impacts requirement traceability and in turn, impact project performance.

Lack of System Architecture and Challenges in System Integration Many problems in development arise due to the lack of understanding of system complexity and deficiencies in the system design. Every case we studied had a problem of underestimating the complexity of the system, which was the result of a lack of clearly defined requirement as discussed earlier. Such issues were evident in cases such as CADE, WSM, and LSM. Without a good understanding of system complexity, the estimation of project cost and schedule could be inaccurate. Consequently, the projects may face problems of cost overrun and schedule delay. A system architecture or systems architecture is the conceptual design that defines the structure and/or behavior of a system. It is the fundamental organization of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution. Lack of system architecture can have an adverse effect on system development as was seen in many cases such as CADE, WSM, and ERP (GAO, 1994, 2001, 2005). In the WSM case, for example, the team did not develop system architecture. Modernization subsystems were developed independently. As a result, the subsystems were incompatible and required additional resources to interconnect them. Similar issues were found in the ERP project (GAO, 2005). Development issues related to system integration are generally due to the multiple platforms on which systems within the same organization operate. Integration issues often result in degraded performance and cost overruns as seen in CADE, ERP, and WSM cases (GAO, 2001, 2004, 2005). The issue of lack of system architecture leading to problems of integration and interoperability was obvious in the ERP case (GAO, 2005). In this case, four separate Navy organizations began their ERP Pilot programs independent of each other, at different times, and with separate funding. Even though each Pilot implemented the same



ERP software and each Pilot was small in scale relative to the entire program, software configuration was done differently. Based on the research evidence, we suggest the following propositions for future research: Proposition 2: In managing large-scale government IS/IT projects, lack of system architecture is a common problem that impacts project performance. Proposition 2.1: Lack of system architecture has an adverse effect on system development. Proposition 2.2: Lack of system architecture causes problems in system integration, which in turn, impacts project performance.

B. Governance In managing megaprojects, it is sensible to anticipate project governance as a major challenge. In general, governance can be defined as the decision accountability structure to leverage the behaviors and empower teams to drive value. Governance relates to consistent management, cohesive policies, processes and decision-rights for a given area of responsibility. Governance acts as an important factor in determining project outcomes as timely and effective management decisions are critical for project success. From the cases we studied, twelve out of fourteen cases faced a major challenge in project governance. The issue existed both at the project level and the organizational level, which includes internal verification and validation, internal review and management oversight. Many of the cases we studied showed that problems at the management level as well as lack of oversight and reviews resulted in cost overruns, schedule delays and missed opportunities to identify problems early.

Lack of Management Oversight Lack of senior management oversight was a major issue in twelve of the projects we studied. We found in many cases that the lack of

management oversight is rooted in the lack of a senior individual who has total responsibility for the project, referred to as the Senior Responsible Owner (SRO). When multiple partners/agencies are involved in a project, the lack of an SRO could lead to conflicts between partners and, in some cases, lack of involvement from any of the partners. This could, in turn, result in a poor governance structure affecting a variety of other project activities along with costs and schedule. In the WSM case (GAO, 1994), there was no individual who had a total responsibility for the modernization effort. Management viewed the modernization project as a collection of autonomous subsystem, causing the diffusion in management. Individual subsystem changes were managed through change control groups that bring together representatives from multiple subsystems. However, this process, which focuses on individual subsystem changes, was not an effective alternative to having system architecture, a chief engineer or a single manager, thus limiting the ability to enforce architecture. Lack of coordinated management oversight was also found in the ERP project (GAO, 2005). The project was funded by different major commands within the Navy. This allowed the project to be developed independently but it caused stove-piped systems that could not operate with each other. A lack of coordinated management oversight was clearly lacking in the WSM case. It was found that different departments carried out projects independently without any coordination and without a single project manager. This also resulted in a fractured system architecture which ultimately led to integration issues (GAO, 2001). We also saw in the CADE project that the lack of business unit ownership led to poor oversight and project scope creep. These findings support the literature. Many studies have indicated that the absence of an influential champion and change agent, corporate management and culture related issues as context driven issues could result in the poor performance of IS/IT projects (Yeo, 2002). Since having project accountability is necessary for project success, it



is important for management to assign a person or a group of persons responsibility for an IS project provides a single point of contact for deliverables, requests and requirements (Lemon et al., 2002). Based on the research evidence, we suggest the following propositions for future research:

Proposition 4: In managing large-scale government IS/IT projects, inappropriate internal verification and validation and internal review is a common problem that impacts project governance.

Proposition 3: In managing large-scale government IS/IT projects, lack of senior management oversight is a common problem that impacts project performance. Proposition 3.1: Conflicts among partners often arise when the project lack of Senior Responsibility Owner. Proposition 3.2: Lack of Senior Responsibility Owner can cause problems related to system architecture and integration.

Besides problems in system management and governance, we found some major problems related to project management in most cases as well. Those problems were associated with the lack of standard project management processes such as risk management, change management, and project monitoring and control.

Inappropriate Internal Verification and Validation, and Internal Reviews Another problem that could lead to a challenge in project governance is inappropriate internal verification and validation (IV&V) and internal reviews. In many cases we studied, such reviews were not completed. Thus any shortcomings in the project were not escalated to upper management for immediate attention. In the ERP case (GAO, 2005), there was no established independent verification and validation function to provide an assessment of the ERP project to the Department of Defense and the management of the Navy. In addition to the lack of IV&V, there was a lack of reporting structure to capture quantitative data that could be used to assess the effectiveness of the overall effort. Similar patterns were found in the LSM, EDGE and PMKeyS projects (GAO, 1989). On the other hand, too many audits and reviews during the course of the projects hampered the progress of the project and the performance of the project personnel, as was found in the Trilogy and CADE cases (GAO, 2001; Anonymous, 2004). Based on the research evidence, we suggest the following proposition for future research:

C. Project Management

Issues Related to Risk Management Another area of real concern is risk management in megaprojects. The common problems that we saw across these cases were the lack of an effective risk management process and the effective execution of the process. The underlying cause of these problems may be the complexity of the system. Lack of effective risk management was evident in two of the five cases we studied, for example ERP and LSM (GAO 1989, 2005). In the ERP project (GAO, 2005), risks were not effectively managed in the sense that the disciplined processes were not implemented, increasing the susceptibility of the system to risks, which directly or indirectly affected the schedule and cost objectives of the project. The LSM case (GAO, 1989a) was also plagued by performance issues due to poor risk analysis in first place. Based on the research evidence, we suggest the following propositions for future research: Proposition 5: In managing large-scale government IS/IT projects, project risk management is a common problem that impacts project performance. Proposition 5.1: Lack of an effective risk management process and the effective execution of the process impacts project performance.



Proposition 5.2: System complexity negatively impacts project risk management.

Ineffective Change Management Processes Without clearly defined and documented requirements, there is a high possibility of new requirements being added to the project and existing requirements being discarded. In the Trilogy case, some of the requirements were left to a trial and error process. As the software evolved in a spiral approach, there were numerous requirement additions and changes, as well as places where the requirements were left open for future specification or for developer interpretation. To effectively manage requirements, a well-established change management process must be put in place. Such a process must require approvals and impact assessments to be completed when there are changes to requirements. In fact, four out of five cases we studied experienced changes in requirements and some of them did not have a good change management process. We found in the cases such as LSM (GAO, 1992), Trilogy (OIG, 2003), and WSM (Anonymous, 1991) projects that additional and evolving requirements led to schedule delays which further resulted in increased costs. We also found that a pattern of deferring project requirements and missed release deployment dates continued from the beginning of CADE project (GAO, 2001; Anonymous, 2004) because the IRS and its prime contractor had agreed on an unrealistic scope of work. Based on the research evidence, we suggest the following proposition for future research: Proposition 6: In managing large-scale government IS/IT projects, Effective change management process is particularly needed when requirements are not clearly defined. Ineffective change management process is a common problem that impacts project performance.

Insufficient Project Monitoring and Control One of the underlined problems of project management was insufficient project monitoring and control. In many cases, lack of project monitoring and control led to missed opportunities in identifying problems early. One of the reasons that project monitoring and control was not effectively done was the lack of standard processes, procedures, and metrics. We found in some cases that projects were set up without significant milestones, checkpoints, incentives, or other ways of assessing and rewarding project progress. In nine out of fourteen cases we studied, lack of experienced project managers and clear roles and responsibilities of project managers caused monitoring and control problems. Without experienced project managers and their clear roles and responsibilities, the project suffered from lack of oversight of project activities leading to development of deficient systems, cost and schedule escalations, etc. In the Trilogy case (Anonymous, 2005), inexperienced IT program managers and contract managers made it difficult for the upper management to deal aggressively or effectively with its contractors. Inexperienced managers often lack the ability to assume proactive management roles and are often held hostage to the perspectives of the contractor. In the WSM case (GAO, 1994), the lack of a single/central project manager for individual systems led to a lack of coordination and integration. With multiple parties involved in the projects, defining clear roles and responsibilities of project managers can be difficult, especially when communication between parties is ineffective. Our finding is sensible. Though project management is at the core of every project from the perspective of our study, issues arising out of projects are inevitable due to the scale of the projects, their complexity as well as the variety of stakeholders involved. Similar to our findings, several IS/IT researchers identify several project management issue impacting IS/



IT project performance. They are project risk management (Lemon et al., 2002; Yeo, 2002) and experience of project manager (Pan et al., 2008). In the study conducted by Gauld (2007), several of the issues indicated by our research have been highlighted. However, the research was not done on the large-scale projects and was restricted to one organization. Based on the research evidence, we suggest the following propositions for future research: Proposition 7: In managing large-scale government IS/IT projects, insufficient project monitoring and control is a common problem that impacts project performance. Proposition 7.1: Lack of standard process, procedure and metric leads to ineffective project monitoring and control. Proposition 7.2: Lack of experienced project managers leads to problems in project oversight, and in turn impact project performance.

D. Contract Management Effectively managed contracts can deliver significant benefits to an organization. Often, the effective management of contracts separates successful projects from the unsuccessful ones. Therefore, to ensure that a project’s implementation is successful, it is absolutely necessary to manage contracts effectively from the bidding process to the operational support after a project’s completion. In the cases we studied, contract management issues surfaced in 3 out of 5 cases making them one of the more prominent issues faced by organizations.

Inefficient Bidding Process, Tender Evaluation The bidding process is critical to a project as its outcome decides the contractor(s) who will be responsible for the deliverables of the project. It is very important for an organization to have a competitive bidding process for the projects with clearly defined and detailed requirements. In the case of CADE we found that the contracting

process was highly inefficient leading to extra overhead for both IRS as well as the PRIME. Another important factor to be considered while inviting bids for a contract is the decision on the type of contract. Organizations should choose the type of contract that best suits the need of the organization. Factors to be considered are the financial standing of the organization, the risk taking capacity of the organization and management controls and oversight of the organization among others. For example, a cost plus award fee contract needs tighter management controls and oversight than a fixed price contract. Similarly the management bears all the risk in a Time and Materials. Both of these projects, however, suffered the adverse effects of the Private Finance Initiative as the contractors were unable to mitigate the risks of contracts of this type. An inefficient tender evaluation process leads to a contract that lacks detail and leads to issues and inconsistencies during the implementation of the project. It is therefore very important to thoroughly verify the capabilities of the contractor in terms of technical expertise, experience, skilled workforce as well as appropriate documentation, financial standing of the contractor and the financial model proposed by the contractor to implement the project. Other practices to be considered in the contract are benchmarking against industry standards, financial penalties for missing timelines, detailed organizational policies and procedures, etc. We see that in some cases the lack of technical expertise of the contractor and a lack of experience in managing similar projects has damning effects not only on the cost and schedule of the project but also on the relationship between the contractor and the owner . In the WSM project (GAO, 2000), the contractor lacked experience or technical expertise in developing weather systems that were as technologically advanced as needed by the National Weather Service. In the case of LSM inefficient contract management during the initial stages led to the change of contract type from cost plus award fee to fixed price contract. Although this saved a few million dollars for the Air force, it was



damage limitation to a large extent. Based on the research evidence, we suggest the following proposition for future research: Proposition 8: In managing large-scale government IS/IT projects, insufficient bidding process and tender evaluation are common problems that impact project performance.

Managerial Issues in Contract Management One of the biggest problems leading to contractual disputes is the lack of communication between the stakeholders, i.e., the owner and the contractor in most cases. When multiple parties are involved, communication management becomes crucial in deciding an approach for successful implementation of a project. Poor performance by the contractor, resulting in missed milestones, delayed payments and contract modifications, often leads to contentious relationships between the contractor and the owner. This invariably results in numerous contract modifications and deteriorating relationships affecting future performance. Strong interpersonal relationship is required between the contractor and the owner for effective implementation of a project. In the case of CADE project (GAO, 2001; Anonymous, 2004), there was a lack of communication between the stakeholders and the contractor. This could be attributed to a certain extent to lack of technical and management leadership at the top level which had a telling effect on requirements management as well as the schedule of the project. Another major obstacle to effective contract management is the lack of contractor oversight by the owner. Contractor oversight is essential not only to ensure that the contractor’s performance is on par with expectations but also to make sure that the contractor is adhering to the terms of the contract. However poor contractor oversight may easily lead to communication gaps between the owner and the contractor, over optimistic expectations of the owner and contractor losing direction of the project. In

the Trilogy case (GAO, 2006), the FBI lacked an effective review and approval process for contractor invoices, strong internal or physical controls over procurement activities, a written policy to govern its tracking and oversight activities and a contractor monitoring plan. This left the FBI vulnerable to questionable contractor costs and missing equipment. Though there has been a lot written on contract management in general, the literature with respect to IS/IT projects could be seen as somewhat limited when compared to the other sections of this study. Yeo (2002) states that poor internal communication as well as communication with contractors as a context driven issue that may result in project failure. Contractor problems arising from lack of trusting relationship with the contractors could lead to low morale and contractual disputes (Pan et al., 2008). Researchers also point out that ad-hoc reliance on contractors for advice did not serve well for the project and suggested a greater management role in such decisions (Gauld, 2007). Drawn from the large-scale public projects, our findings support what stated in the literature. Based on the research evidence, we suggest the following propositions for future research: Proposition 9: In managing large-scale government IS/IT projects, management of contract is a common problem that impacts project performance. Proposition 9.1: Lack of communication between the owner and the contractors leads to contract disputes. Proposition 9.2: Relationship management and contractor oversight by the owner are necessary for managing large-scale IS/IT projects.

6. MANAGERIAL IMPLICATIONS In this section, we present some managerial implications in the forms of the lessons that we learned from reviewing these five cases. They are classified as lessons learned in system management, governance, project management, and contract management.



A. Systems Management The information from our cases consistently suggests that without clear requirements, the system being developed will have limited benefits. While marginal improvement may be realized, it is typically difficult to ascertain the sustained, long-term benefits. Since a mega IS/ IT project is very complex in nature, the project team must follow disciplined processes to identify and manage requirements. Requirement identification must be done with the involvement of project stakeholders. A process of documenting the requirements must be put in place and requirement traceability must be practiced. It is possible that requirement identification is an on-going continuous process. In such cases, the management must be certain that a change management process is established. It is also important that the formulation of system architecture must be done and received the highest priority in the IT effort (Philbin, 2008). Well-established system architecture will lessen the challenges in system integration in the later phases. To formulate the system architecture, management must be personally involved in the process and be part of making key decisions. It is possible that a contractor might assist the system owner in developing the system architecture. The contractor should not be solely responsible for this effort since they do not fully understand the operational issues that must be reflected in the architecture. The validity of system architecture is another issue that management must pay attention to. Management may utilize external expert to independently and regularly review the architecture. For a large-scale and complex IS/IT project, it is rather a common practice that the project is broken up into manageable pieces that can be delivered incrementally. This, however, must be done with well-established system architecture.

B. Governance Based on the evidence in our research, the lack of management control and oversight resulted in the poor performance of the project. It is

obvious that appropriate management controls and processes must be in place to have reasonable assurance that an enterprise effort will be successful. As it is a high-level organizational activity, an organization embarking on largescale projects must insure that it has established procedures and policies specifically for project governance (Williams et al., 2010). It is important that the owner’s management assign an SRO for the project. Management must also make sure that the authority of the SRO is commensurate with his or her roles and responsibilities. The SRO must be capable and committed to the project since the capabilities of the SRO have an important impact on the dynamics of a project and its performance (Miller & Hobbs, 2005; Sutterfield et al., 2006). The SRO should have an integrative business perspective, ability to evaluate complex systems from multiple perspectives, relational and coalition-building competencies, and political and negotiation skills (Miller & Hobbs, 2005). The SRO may establish a project office to provide support to the project teams and become a formal channel to relay project information to the SRO and vice versa. In addition, management can establish an enterprise project management office to oversee all high priority projects. The office can set up standards and ensure that management of the large-scale IS/IT projects follow the standards. These standards may range from financial controls to governance and senior management oversight. In addition, management must establish an independent verification and validation (IV&V) function. Formal procedures and policies for IV&V and internal reviews must be established.

C. Project Management The management and oversight of projects by experienced project managers is essential for ensuring project success. The successful implementation of IS/IT systems also calls for sound project management processes and methodologies. As already indicated, processes for project risk management, change management, and monitoring and control are necessary.



Although it is not indicated explicitly, project management processes related to project planning are also important. The prior discussions on requirement management and system architecture already addressed some of these project planning processes. It is important that a formal and standard risk management process should be established (Miller & Lessard, 2001; Patanakul, 2008). The process should include risk identification, risk assessment (qualification and quantification), risk handling (response planning), risk surveillance (monitoring and control), and risk closure. Since a large-scale project consists of many programs and projects, risk management should be conducted on multiple levels. For example, the implementation of a team-level board encourages the management of risks at the lowest possible level. Risks that require management attention may be escalated to a higher-level board, like a program level board. Creating public private partnerships can also help manage risk in project delivery (Abednego & Ogunlana, 2006; Shen et al., 2006). Because of the complexity of a mega IS/IT project, it is likely that many changes will occur as the project progresses. As we already learned from the cases, requirement changes can have some negative impacts on project performance and changes occurring in the later phases of the project can be detrimental to the project objectives. Change decisions must be implemented with careful consideration. A formal change management process, tied to the change management board, can facilitate the deliberation to implement changes and may help assure management that all the changes implemented in the project are beneficial to the overall project objectives. The success of projects is also associated with the project management competency possessed by an organization. Depending on how project work is allocated, the majority of the work could be done by the contractors. However, the project owner must assign key personnel from its staff to the project and these personnel should work hand-in-hand with the contractors. A project manager is among the key personnel that represent the owner. It is

important that the project manager assigned to the project must be experienced and work closely with the contractor’s project manager. The project manager must be proactive and be on top of the project issues such that s/he can make project decisions or escalate the issues when needed.

D. Contract Management Contract management is a significant issue in managing megaprojects since the majority of the project work is contracted out. A rigorous selection process must ensure that the awarded contractor is capable of delivering the expected outcomes. Being engaged in the project and keeping open communication lines with the contractor may be a better approach for the owner to facilitate a good relationship with the contractor. For contract selection, it is very important that the owner organization has a competitive bidding process. The project owner must have a set of criteria for contract selection. It is important that the owner chooses the type of contract that best suits the needs of the organization (Turner & Simister, 2001). The financial standing, the risk taking capacity, and the management controls and oversight ability are some of the factors to be considered when deciding on the types of contracts. Forming partnerships with the contractors seems to be an effective approach for contract management. Public-private partnership is considered to be a form of structured cooperation between public and private parties in which they share or reallocate risks, costs, benefits, resources, and responsibilities (Koppenjan, 2005). The partnership creates a sense of involvement and ownership both for the project owner and the contractors. The partnership should also facilitate the oversight of the project by the owner. Without the partnership, negative tensions and strong polarization could occur. Such tensions and polarization could jeopardize the project performance as we have learned from some of the cases. Creating an alliance culture is also an approach to promote



inter-organizational collaborations (Clegg et al., 2002). To do so, management must build a collaborative commitment, create transparency and seek to constitute each stakeholder in such a way that they have something to gain from greater collaboration within the project. The alliance contracts seem to work best where there is uncertainty in the product and process, the owner has some knowledge of the situation and looks for a breakthrough performance, and the alliance partners have the competence and financial backing to share the risk (Turner & Simister, 2001).

7. CONCLUSION Since many of the government projects are initiated to promote sustainable development, the success of those projects is important to the government. In this study, we investigated large-scale IS/IT projects in the US, identified common problems, compared and contrasted them with existing literature and discussed some lessons learned. What we found in this study should provide practitioners with some awareness in managing large-scale IS/IT projects. Certain practices, as suggested in the Managerial Implication section, could be implemented to prevent such problems, and in turn could lead to better project performance, and contribute in part to accomplishing sustainability. The research findings can also be used as a basis for future studies. A future large-sample study can be conducted, using our research propositions as a foundation for the development of research hypotheses and questionnaires. Potential limitations of this study center on its research methodology. First, our sample size is small since we studied only five projects. Generalization of the research findings must be done with caution. Second, we conducted content analysis based on the audit reports that were available. Instead of analyzing raw information as in the case study research methodology, the information that we analyzed had already been processed. We, however, analyzed information

from the audit reports generated by government agencies. Our sources of information should be considered reliable since these audit reports only provide factual information. Third, although the information from the reports is factual, there is a possibility that emphasis had been set forth on certain areas depending on the focus of the audit agency at the time of auditing. This may lead to the appearance or disappearance of certain issues in the report. For a future study, researchers may consider implementing a case study methodology in order to study each case in-depth. However, such research would require a tremendous amount of time and resources.

REFERENCES Abednego, M. P., & Ogunlana, S. O. (2006). Good project governance for proper risk allocation in public–private partnerships in Indonesia. International Journal of Project Management, 24(7), 622–634. doi:10.1016/j.ijproman.2006.07.010 Anonymous. (1991). Weather forecasting: Cost growth and delays in billion-dollar weather service modernization. Washington, DC: General Accounting Office, US Congress. Anonymous. (2004). A review of the FBI’s trilogy information technology program. Washington, DC: National Research Council of the National Academies. Anonymous. (2004). To ensure the customer account data engine’s success, prescribed management practices need to be followed. Washington, DC: Treasury Inspector General for Tax Administration. Anonymous. (2005). A report to the committee on appropriations. Washington, DC: U.S. Congress, House of Representatives, House Surveys and Investigations. Bansal, P. (2005). Evolving sustainability: A longitudinal study of corporate sustainable development. Strategic Management Journal, 26, 197–218. doi:10.1002/smj.441 Bauer, M. W. (2000). Classical content analysis: A review. In Bauer, M. W., & Gaskell, G. (Eds.), Qualitative researching with text, image, and sound: A practical handbook (pp. 131–151). London, UK: Sage.



Clegg, S. R., Pitsis, T. S., Rura-Polley, T., & Marosszeky, M. (2002). Governmentality matters: Designing an alliance culture of inter-organizational collaboration for managing projects. Organization Studies, 23(3), 317–337. doi:10.1177/0170840602233001 Dyllick, T., & Hockert, K. (2002). Beyond the business case for corporate sustainability. Business Strategy and the Environment, 11, 130–141. doi:10.1002/bse.323 Flyvbjerg, B., Bruzelius, N., & Rothengatter, W. (2003). Megaprojects and risk: An anatomy of ambition. Cambridge, MA: Cambridge University Press. GAO. (1989). ADP acquisition: Air Force logistics system modernization projects: Report to the Honorable John Conyers, Jr., chairman, Legislation and National Security Subcommittee, Committee on Government Operations, House of Representatives. Washington, DC: United States Government Accountability Office. GAO. (1989). Air Force ADP: Evaluations needed to substantiate modernization program benefits. Washington, DC: United States Government Accountability Office. GAO. (1989). Automated information systems: Schedule delays and cost overruns plague DOD systems, report to the Chairman, Legislation and National Security Subcommittee, Committee on Government Operations, House of Representatives. Washington, DC: United States Government Accountability Office. GAO. (1992). Air Force ADP: Status of logistics modernization projects and CIM impacts. Washington, DC: United States Government Accountability Office. GAO. (1994). Weather forecasting- Systems architecture needed for national weather service modernization - National weather service modernization. Washington, DC: United States Government Accountability Office. GAO. (2000). Report to the Chairman, Committee on Science, Space, and Technology, House of Representatives, Weather satellites: Action required to resolve status of U.S. geostationary satellite program. Washington, DC: United States Government Accountability Office.

GAO. (2000). Testimony before the Subcommittee on Energy and Environment, Committee on Science, House of Representatives, National Oceanic and Atmospheric Administration, National Weather Service Modernization and Weather Satellite Program, Statement of Joel C. Willemssen Director, Civil Agencies Information Services Accounting and Information Management Division. Washington, DC: United States Government Accountability Office. GAO. (2001). Business systems modernization: Results of review of IRS’ customer account data engine project. Washington, DC: United States Government Accountability Office. GAO. (2004). Business systems modernization: Internal revenue service needs to further strengthen program management. Washington, DC: United States Government Accountability Office. GAO. (2005). DOD business systems modernization: Navy ERP adherence to best business practices critical to avoid past failures. Washington, DC: United States Government Accountability Office. GAO. (2005). Post-hearing questions on FBI management capabilities. Washington, DC: United States Government Accountability Office. GAO. (2006). Business systems modernization: IRS needs to complete recent efforts to develop policies and procedures to guide requirements development and management. Washington, DC: United States Government Accountability Office. GAO. (2006). Federal Bureau of Investigation: Weak controls over trilogy project led to payment of questionable contractor costs and missing assets. Washington, DC: United States Government Accountability Office. Gauld, R. (2007). Public sector information system project failures: Lessons from a New Zealand hospital organization. Government Information Quarterly, 24, 102–114. doi:10.1016/j.giq.2006.02.010 Gladwin, T. N., Kennelly, J. J., & Krause, T.-S. (1995). Shifting paradigms for sustainable development: Implications for management theory and research. Academy of Management Review, 20(4).



Han, S. H., Yun, S., Kim, H., Kwak, Y. H., Park, H. K., & Lee, S. H. (2009). Analyzing schedule delay of mega project: Lessons learned from Korea train express. IEEE Transactions on Engineering Management, 56(2), 243–256. doi:10.1109/ TEM.2009.2016042 Jolivet, F., & Navarre, C. (1996). Large-scale projects, self organizing and meta-rule: Towards new forms of management. International Journal of Project Management, 14(5), 265–271. doi:10.1016/02637863(96)84509-1 Koppenjan, J. F. M. (2005). The formation of publicprovate partnerships: Lessons from nine transport infrastrcuture projects in The Netherlands. Public Administration, 83, 135–157. doi:10.1111/j.00333298.2005.00441.x Lemon, W. F., Liebowitz, J., Burn, J., & Hackney, R. (2002). Information systems project failure: A comparative study of two countries. Journal of Global Information Management, 10(2). doi:10.4018/ jgim.2002040103 Miller, R., & Hobbs, B. (2005). Governance regimes for large complex projects. Project Management Journal, 36(3), 42–50. Miller, R., & Lessard, D. (2001). Understanding and managing risks in large engineering projects. International Journal of Project Management, 19, 437–443. doi:10.1016/S0263-7863(01)00045-X OIG. (2002). Federal Bureau of Investigation’s management of information technology investments. Washington, DC: Office of the Inspector General. OIG. (2003). Federal Bureau of Investigation’s implementation of information technology recommendations. Washington, DC: Office of the Inspector General. OIG. (2005). The Federal Bureau of Investigation’s management of the trilogy information technology modernization project. Washington, DC: Office of the Inspector General. Pan, G., Hackney, R., & Pan, S. (2008). Information systems implementation failure: Insights from prism. International Journal of Information Management, 28, 259–269. doi:10.1016/j.ijinfomgt.2007.07.001 Patanakul, P. (2008). Program risk management: How it is done in major defense programs. Paper presented at the Project Management Research Conference, Warsaw, Poland.

Philbin, S. P. (2008). Managing complex technology projects. Research Technology Management, 51(2), 32–40. Shen, L.-Y., Platten, A., & Deng, X. P. (2006). Role of public private partnerships to manage risks in public sector projects in Hong Kong. International Journal of Project Management, 24(7), 587–594. doi:10.1016/j.ijproman.2006.07.006 Sutterfield, J. S., Friday-Stroud, S. S., & ShiversBlackwell, S. L. (2006). A case study of project and stakeholder management failures: Lessons learned. Project Management Journal, 37(5), 26–35. Thomas, G., & Fernandez, W. (2008). Success in IT projects: A matter of definition? International Journal of Project Management, 26, 733–742. doi:10.1016/j. ijproman.2008.06.003 Turner, R. J., & Simister, S. J. (2001). Project contract management and a theory of organization. International Journal of Project Management, 19(8), 457–464. doi:10.1016/S0263-7863(01)00051-5 van Marrewijk, A. (2007). Managing project culture: The case of Environ Megaproject. International Journal of Project Management, 25(3), 290–299. doi:10.1016/j.ijproman.2006.11.004 van Marrewijk, A., Clegg, S. R., Pitsis, T., & Veenswijk, M. (2008). Managing public-private megaprojects: Paradoxes, complexity, and project design. International Journal of Project Management, 26, 591–600. doi:10.1016/j.ijproman.2007.09.007 van Marrewijk, M., & Werre, M. (2003). Multiple levels of corporate sustainability. Journal of Business Ethics, 44(2-3), 107–119. doi:10.1023/A:1023383229086 Williams, T., Klakegg, O. J., Magnussen, O. M., & Glasspool, H. (2010). An investigation of governance frameworks for public projects in Norway and the UK. International Journal of Project Management, 28(1), 40–50. doi:10.1016/j.ijproman.2009.04.001 World Bank. (2010). Status of projects in execution FY10: Operation policy and country services. Washington, DC: World Bank. Yeo, K. T. (2002). Critical failure factors in information system projects. International Journal of Project Management, 20, 241–246. doi:10.1016/ S0263-7863(01)00075-8



Peerasit Patanakul is an Assistant Professor at Stevens Institute of Technology, NJ USA. His research encompasses project portfolio management, multiple project management, strategic and value-focused project management, and large-scale government projects and programs. His work was published in e.g. IEEE Transactions on Engineering Management, International Journal of Project Management, Project Management Journal, and Systems Engineering. Dr. Patanakul received a BE in Chemical Engineering from Chulalongkorn University, Thailand, an M.S. in Engineering Management, and a PhD in Systems Science/Engineering Management from Portland State University, Oregon USA. Saif Syed Omar recently graduated from the Stevens Institute of Technology, with a Master’s degree in Information systems. Currently, he works for Credit Suisse in IT Project and Program Mangement group. Mr. Omar has been actively involved in project management research especially in large scale public sector IS/IT initiatives. He received a bachelor’s degree in Electronics and Communication from RNS Institute of Technology, India and worked for Perot systems TSI (now a part of Dell Services) as a part of the Enterprise Resource Planning (ERP) division.
