Contract Management
Service Contract Approach to Aircraft Simulator Training Has Room for Improvement Gao ID: GAO-06-830 September 22, 2006The Air Force has turned to service contracts for the F-15C, F-16, Airborne Warning and Control System, and F-15E, and the Army has done the same for helicopter simulator training at its Flight School XXI. The contractors own, operate, and maintain the simulator hardware and software. The military services rely on industry to capitalize the required up-front investment, with the understanding that the contractors will amortize this investment by selling training services by the hour. GAO was asked to address (1) the factors that led the Air Force and Army to acquire simulator training as a service and whether the decision to use this approach was adequately supported; (2) whether implementation of the approach has resulted in the planned number of simulator training sites being activated; and (3) whether the Air Force and Army are effectively tracking the return on their expenditure of taxpayer dollars. GAO makes recommendations to the Secretary of Defense intended to improve management and oversight of these service contracts to help ensure that the best approach is used to provide the war-fighter with needed training. In written comments on a draft of this report, DOD concurred with all but one of the recommendations, only partially concurring with one pertaining to the Army's simulator utilization rates. GAO continues to believe that the Army needs to track the extent to which it is using simulator availability.
The Air Force and Army turned to service contracts for simulator training primarily because efforts to modernize existing simulator hardware and software had lost out in the competition for procurement funds. As a result, the simulators were becoming increasingly obsolete. Buying training as a service meant that operation and maintenance (O&M) funds could be used instead of procurement funds. Shifting the responsibility for simulator ownership, operation, and maintenance from the government to the contractor was thought to more quickly enable simulator upgrades to match the changing configurations of aircraft. However, the decision to take a service contract approach was not supported by a thorough analysis of the costs and benefits as compared to other alternatives, despite a Department of Defense directive that provided for such an analysis. While Air Force and Army officials told GAO the new simulators are significant improvements over the previous ones, the expected number of Air Force training sites have not been activated. For the Air Force, O&M funds have not been allocated at the anticipated levels, leading to schedule slippages. The F-16 simulator contractor cited the funding problems and subsequent schedule slippages as the basis for notifying the Air Force that its situation under the contract was no longer financially viable. The Air Force is in the process of re-competing the F-16 training contract, which will likely result in a training gap for pilots--possibly over 2 years--and additional costs to the Air Force. The start date of the Army's flight simulator training was rebaselined twice, but Army officials told us that adequate training was in place for the flight school participants. The return on expenditure of taxpayer dollars is not being effectively tracked in three key ways: Air Force utilization of simulator training frequently falls well below the hours for which the government is paying. The Army is not collecting data on utilization rates at all. The government has little insight into what it is paying for during the development period before training is activated, which can take more than a year. While invoices for preparatory efforts reflect only discrete tasks such as training capabilities assessments, the wide range of invoice amounts and GAO's discussions with contractor representatives suggest that the government is actually making milestone payments to the contractors for a portion of their up-front costs to acquire and develop the simulators. Most of the contracts contain award-term provisions, where the contractors can earn an extension of the contract period for good performance. GAO found that the award-term evaluation factors do not always measure key acquisition outcomes such as simulator availability and concurrency with aircraft upgrades.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-06-830, Contract Management: Service Contract Approach to Aircraft Simulator Training Has Room for Improvement
This is the accessible text file for GAO report number GAO-06-830
entitled 'Contract Management: Service Contract Approach to Aircraft Simulator Training Has Room for Improvement' which was released on September 22, 2006.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
GAO Highlights:
Highlights of GAO-06-830, a report to the Subcommittee on Readiness, Committee on Armed Services, House of Representatives.
Why GAO Did This Study:
contracts for the F-15C, F-16, Airborne Warning and Control System, and F-15E, and the Army has done the same for helicopter simulator training at its Flight School XXI. The contractors own, operate, and maintain the simulator hardware and software. The military services rely on industry to capitalize the required up-front investment, with the understanding that the contractors will amortize this investment by selling training services by the hour.
GAO was asked to address (1) the factors that led the Air Force and Army to acquire simulator training as a service and whether the decision to use this approach was adequately supported; (2) whether implementation of the approach has resulted in the planned number of simulator training sites being activated; and (3) whether the Air Force and Army are effectively tracking the return on their expenditure of taxpayer dollars.
GAO makes recommendations to the Secretary of Defense intended to improve management and oversight of these service contracts to help ensure that the best approach is used to provide the war-fighter with needed training. In written comments on a draft of this report, DOD concurred with all but one of the recommendations, only partially concurring with one pertaining to the Army‘s simulator utilization rates. GAO continues to believe that the Army needs to track the extent to which it is using simulator availability.
What GAO Found:
The Air Force and Army turned to service contracts for simulator training primarily because efforts to modernize existing simulator hardware and software had lost out in the competition for procurement funds. As a result, the simulators were becoming increasingly obsolete. Buying training as a service meant that operation and maintenance (O&M) funds could be used instead of procurement funds. Shifting the responsibility for simulator ownership, operation, and maintenance from the government to the contractor was thought to more quickly enable simulator upgrades to match the changing configurations of aircraft. However, the decision to take a service contract approach was not supported by a thorough analysis of the costs and benefits as compared to other alternatives, despite a Department of Defense directive that provided for such an analysis.
While Air Force and Army officials told GAO the new simulators are significant improvements over the previous ones, the expected number of Air Force training sites have not been activated. For the Air Force, O&M funds have not been allocated at the anticipated levels, leading to schedule slippages. The F-16 simulator contractor cited the funding problems and subsequent schedule slippages as the basis for notifying the Air Force that its situation under the contract was no longer financially viable. The Air Force is in the process of re-competing the F-16 training contract, which will likely result in a training gap for pilots”possibly over 2 years”and additional costs to the Air Force. The start date of the Army‘s flight simulator training was rebaselined twice, but Army officials told us that adequate training was in place for the flight school participants.
The return on expenditure of taxpayer dollars is not being effectively tracked in three key ways:
* Air Force utilization of simulator training frequently falls well below the hours for which the government is paying. The Army is not collecting data on utilization rates at all.
* The government has little insight into what it is paying for during the development period before training is activated, which can take more than a year. While invoices for preparatory efforts reflect only discrete tasks such as training capabilities assessments, the wide range of invoice amounts and GAO‘s discussions with contractor representatives suggest that the government is actually making milestone payments to the contractors for a portion of their up-front costs to acquire and develop the simulators.
* Most of the contracts contain award-term provisions, where the contractors can earn an extension of the contract period for good performance. GAO found that the award-term evaluation factors do not always measure key acquisition outcomes such as simulator availability and concurrency with aircraft upgrades.
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-830].
To view the full product, including the scope and methodology, click on the link above. For more information, contact Lisa Shames at (202) 512-4841 or shamesl@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
Service Contract Approach Had Perceived Benefits but Was Not Adequately Supported:
While Training Has Improved, the Expected Number of Simulator Sites Has Not Been Activated:
Return on the Expenditure of Taxpayer Dollars Is Not Effectively Tracked:
Conclusion:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Comments from the Department of Defense:
Appendix III: Comparison of Contractor Requirements under Commercial and Non-commercial Acquisition Procedures:
Appendix IV: Potential Duration of Air Force and Army Simulation Contracts:
Appendix V: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Air Force and Army Simulator Training Contracts:
Table 2: Operational Simulator Training Locations and Start Dates as of July 2006:
Table 3: Comparison of DMO Estimated Funding Requirement with the Fiscal Year 2002 Program Objective Memorandum Funding Levels:
Table 4: Key Events That Have Affected Funding for the DMO Program:
Table 5: Comparison of Planned and Actual Site Activations for AWACS:
Table 6: Comparison of Planned and Actual Site Activations for F-15C:
Table 7: Comparison of Planned and Actual Site Activations for F-16:
Table 8: Comparison of Monthly Utilization Rate Calculation Methods for DMO System Simulators[a]:
Table 9: Comparison of Site Survey Costs at Selected DMO System Sites:
Table 10: Comparison of Training Capability Requirements Assessments and Training Capability Assessments for DMO Systems:
Table 11: Comparison of Award-term Areas of Evaluation and the Weight Given to Each Area by System:
Figures:
Figure 1: Flight School XXI Schedule Rebaselines:
Figure 2: Comparison of Monthly Utilization Rates for AWACS:
Figure 3: Original Rate Structure for F-15C Simulator Service Contracts:
Abbreviations:
AAVS: Advanced Aircraft Virtual Simulators:
AB: air base:
AFB: Air Force base:
AWACS: Airborne Warning and Control System:
DMO: Distributed Mission Operations:
DOD: Department of Defense:
FAR: Federal Acquisition Regulation:
O&M: operation and maintenance:
OMB: Office of Management and Budget:
TH: training helicopter:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
September 22, 2006:
The Honorable Joel Hefley:
Chairman:
The Honorable Solomon P. Ortiz:
Ranking Minority Member:
Subcommittee on Readiness:
Committee on Armed Services:
House of Representatives:
In the late 1990s, the Air Force embarked upon an innovative strategy for acquiring flight simulator training for several of its aircraft platforms. Faced with increasingly obsolete simulators and growing competition for procurement dollars, the Air Force decided to buy simulator services from contractors”using operation and maintenance (O&M) funds”rather than continue the practice of having the government own and operate the simulators. This strategy was the beginning of an approach to war-fighting, known as Distributed Mission Operations (DMO), in which high-fidelity simulators would be used for training across dispersed locations. The initial platforms included in the DMO program were the F-15C, F-16, Airborne Warning and Control System (AWACS), and F-15E. In 2003, the Army followed the Air Force lead and awarded a service contract to obtain simulator training for the helicopters in its Flight School XXI program. The estimated value of the Air Force and Army contracts is nearly $2 billion. A key aspect of the service contract approach is the increased reliance on contractors to keep the simulators concurrent with aircraft upgrades and ensure they are available for use.
At your request, we addressed (1) the factors that led the Air Force and Army to acquire simulator training as a service and whether the decision to use this approach was adequately supported, (2) whether implementation of this approach has resulted in the planned number of simulator training sites being activated, and (3) whether the Air Force and Army are effectively tracking the return on their expenditure of taxpayer dollars.
To address these objectives, we interviewed Air Force and Army officials, as well as contractor representatives; performed a detailed analysis of the Air Force and Army service contracts; reviewed contractor performance measurements, annual evaluations, and payment invoices; and analyzed historical documents such as acquisition plans, briefings, and decision memorandums. We visited Langley Air Force Base (AFB), Virginia, to view F-15C simulator training; Shaw AFB, South Carolina, to view F-16 training; and the U. S. Army Aviation Center, Fort Rucker, Alabama to view the use of Flight School XXI training helicopters (TH), the TH-67 Virtual Simulators and the Blackhawk and Chinook simulators. At each of these installations, we met with military officials and the contractor officials responsible for the training simulators. See appendix I for additional details on our scope and methodology. We performed our review from September 2005 to July 2006 in accordance with generally accepted government auditing standards.
Results in Brief:
The Air Force and Army turned to service contracts for simulator training primarily because efforts to modernize existing simulator hardware and software had lost out in the competition for available procurement funds. Buying simulator training as a service meant that O&M funds could be used to fund the requirement. Shifting the responsibility for simulator ownership, operation, and maintenance from the government to the contractor was thought to provide quicker state-of-the-art pilot and aircrew training capabilities and allow for automatic simulator upgrades to match the changing configurations of aircraft. However, the decision to use service contracts was not supported by a thorough analysis of the costs and benefits of this approach versus alternative approaches, despite a Department of Defense (DOD) directive on training that provides for an evaluation of the benefits and trade-offs of potential alternative training solutions. While the Air Force and Army provided us with briefings that showed the decision-making process, the information does not provide adequate rationale for why certain alternatives were eliminated from consideration. A more in-depth analysis could have proactively addressed and possibly mitigated the risks associated with this approach, where if a contractor fails to perform, the government is left without simulator hardware, software, or the anticipated level of training services.
While Air Force and Army officials told us the new simulators are significant improvements over what they had previously, the expected number of Air Force simulator training sites has not been activated. For the Air Force, O&M funds have not been allocated to the contracts at the anticipated levels, leading to schedule slippages and fewer simulator sites in place than planned. Efforts to add funds, for example by shifting flying hour funds into the DMO program in 2003, have not been sufficient to regain the schedule and activate the number of sites anticipated. Further, the F-16 simulator training contractor, citing the funding problems and subsequent schedule slips, notified the Air Force in March 2004 that its situation was no longer financially viable under the contract and that it could not continue to provide simulator training as originally agreed. The Air Force will allow the contract to expire in June 2007. It is re-competing the F-16 training contract, which will likely result in a training gap for pilots”possibly over 2 years”and additional costs to the Air Force. Two aspects of the original contract, awarding it as a commercial acquisition and including an award-term provision, [Footnote 1] will not be included in the new contract.
The return on expenditure of taxpayer dollars is not being effectively tracked in three key ways:
* Air Force utilization of simulator training frequently falls well below the hours for which the government is paying. In addition, utilization rates at different sites are reported based on very different criteria, leading to decisions to activate new training sites based on noncomparable information. The Army is not collecting data on utilization rates at all.
* The government has little insight into what it is paying for during the development period before training is activated, which can take more than a year. Our analysis of invoices and discussions with contractor officials suggest that the government is, in effect, funding the private sector for a portion of its up-front costs to acquire and develop the simulators, although the invoices merely reflect discrete tasks such as site surveys.
* Award-term evaluations are being used to measure performance, but the evaluation factors do not always measure key acquisition outcomes such as simulator availability and concurrency. In addition, the ability to ’roll over“ positive points on Air Force contracts can result in the contractor being awarded an additional year for only satisfactory performance”a scenario that is highly unlikely under the Army‘s award-term plan. Air Force, Army, and some contractor officials indicated that award-term provisions may not be an effective incentive for improved performance.
We make recommendations in this report to DOD on actions that can be taken to help ensure that the best approach is used to provide the war-fighter with needed training, that the incentives motivate contractor performance toward achieving desired training outcomes, and that available simulator training is used in the most effective and efficient manner. In written comments on a draft of this report, DOD concurred with all but one of our recommendations. DOD partially concurred with our recommendation that the Army track and record monthly utilization rates on simulators at Flight School XXI. Nevertheless, DOD stated that the contractor is required to submit utilization data and that the data are available. Our recommendation was intended to encourage DOD to fully understand its student training requirements and to collect the information to decide whether it needs to adjust those requirements or contract provisions regarding simulator availability. Whether the utilization rates pertain to individual simulators or the student training load as a whole, we believe that the Army needs to know the extent to which it is using the simulator availability it is buying. DOD also offered two corrections to information in our draft report; we made changes as appropriate. DOD‘s comments are included in their entirety in appendix II.
Background:
Air Force and Army cite the increasingly complex training requirements needed to prepare for the ever more lethal battlefield environment as a factor that has led to greater reliance on flight simulators. A flight simulator is a system that tries to realistically replicate, or simulate, the experience of flying an aircraft. Flight simulators range from video games to full-sized cockpit replicas mounted on hydraulic (or electromechanical) actuators and controlled by state-of-the-art computer technology. According to Air Force and Army officials, aircraft simulators are a cost-effective way of helping to develop and refine operational flight skills. Simulators can facilitate training that might be impractical or unsafe if done with actual systems and allow for concentrated pilot practice in selected normal and emergency actions. Simulators also can train operators and maintainers to diagnose and address possible equipment faults, and enhance proficiency despite shortages of equipment, space, ranges, or time.
In the late 1990s, the Air Force and Army were faced with increasingly obsolete simulators and the need to quickly acquire up-to-date pilot and aircrew training. In 1997, the then-Commander of the Air Force‘s Air Combat Command proposed an innovative approach of buying training as a service, under which the contractors would own, operate, and maintain the simulator hardware and software. The simulator service contracts are one component of a much broader effort, now known as the DMO program. The DMO goal is to provide state-of-the-art simulator training on-demand at the location of the trainee, with the ultimate vision of networking different sites together to create more realistic flying scenarios. Plans call for each fighter unit eventually to be equipped with high-fidelity simulators. As of the fiscal year 2002 budget, the DMO program was formalized in the Air Force budget with the assignment of a program element line item that combined previous program elements for the various simulator systems. For fiscal year 2006, over $200 million was budgeted for the program.
In the early 2000s, Army use of rotary-wing aviation simulation training was limited because the simulators being used were grossly obsolete and based on late 1970s‘ technology. To revamp its helicopter training, the Army in late 2001 began the Flight School XXI program. Following the Air Force‘s lead, the Army decided to acquire up-to-date simulator training using a service contract.
Congress and the Office of Management and Budget (OMB) have recently addressed the growing level of procurement of services. For example, Congress included provisions in Section 801 of the National Defense Authorization Act for Fiscal Year 2002 [Footnote 2] designed to improve management and oversight of procurement of services. To ensure that DOD acquires services by means that are in its best interest and managed in compliance with applicable statutes, regulations, directives, and other requirements, the Act required DOD to establish a service acquisition management structure, comparable to the management structure that applies to the procurement of products. [Footnote 3] In September 2003, we reported that DOD and the military services had a management structure in place for reviewing individual services acquisitions valued at $500 million or more, but that approach did not provide a departmentwide assessment of how spending for services could be more effective. [Footnote 4] Also, OMB Circular A-11‘s Appendix B, ’Budgetary Treatment of Lease-Purchases and Leases of Capital Assets,“ was amended in 2005 to require agencies to submit to OMB for review any service contracts that require the contractor to acquire or construct assets valued over $50 million. While these provisions do not apply to the previously-awarded simulator training contracts, future replacement contracts will be covered.
Contract Information:
All of the Air Force and Army simulator service contracts are funded with O&M funds. O&M funds are typically used for such things as military force operations, training and education, and depot maintenance. The contracts are requirements contracts, meaning that the government, within available funds, shall order from the contractor all the training services specified for each of the aircraft platforms that are required during the effective performance periods. Additionally, each contract contains language limiting the government‘s liability in the event the contract is terminated. For example, the Air Force F-15C contract states that the government reserves the right to terminate the contract for its sole convenience and that such termination prior to the issuance of a funded task order shall result in no payment to the contractor of any amount for any work performed or costs incurred. Table 1 provides additional descriptive information for each contract.
Table 1: Air Force and Army Simulator Training Contracts (dollars in millions):
System: AWACS;
Contract Award Date: Feb. 1999;
Prime Contractor: Plexsys Interface Products, Inc.;
Contract Value[a]: 101.0.
System: F-15C;
Contract Award Date: Nov. 1997;
Prime Contractor: Boeing Company[b];
Contract Value[a]: 277.0.
System: F-15E[c];
Contract Award Date: Aug. 2003;
Prime Contractor: Boeing Company;
Contract Value[a]: 287.0.
System: F-16;
Contract Award Date: June 1999;
Prime Contractor: Lockheed Martin Integrated Systems, Inc.;
Contract Value[a]: 178.0.
System: Flight School XXI;
Contract Award Date: Sept. 2003;
Prime Contractor: Computer Sciences Corporation - Defense;
Contract Value[a]: 1108.0.
Source: Air Force and Army contracts.
[a] Estimated costs for contract base years and all potential award-term or option years.
[b] The F-15C and F-15E contracts were originally awarded to McDonnell Douglas Corporation, which became a wholly-owned subsidiary of the Boeing Company.
[c] Training services for the F-15E are ordered under a much larger umbrella contract, Training Systems Acquisition II.
[End of table]
The military services are relying on industry to capitalize the required up-front investment needed to acquire simulator hardware and software, with the understanding that the contractors will amortize this investment by selling training services by the hour. Each contract establishes operating hours and the hourly payment rates for the life of the contracts, with rates structured to provide the contractor with higher income in the initial years of service. In calendar year 2004, for example, if the F-16 contractor provided Shaw AFB with simulator availability that met 95 percent of the required system elements, [Footnote 5] the hourly rate would be $5,225, whereas in calendar year 2006, it would drop to $709 per hour.
We have previously identified the need to examine the appropriate role for contractors to be among the challenges in meeting the nation‘s defense needs in the 21st century. [Footnote 6] We recently reported that the government‘s increasing reliance on contractors for missions previously performed by government employees highlights the need for sound planning and contract execution. [Footnote 7]
Acquisition Reforms Shaped Contract Structures:
The structure of the simulator service contracts was heavily influenced by mid-1990s‘ acquisition reform initiatives such as the Federal Acquisition Streamlining Act of 1994 [Footnote 8] and the Clinger-Cohen Act of 1996. [Footnote 9] These Acts encouraged agencies to use commercial acquisition procedures as a way to streamline the acquisition process. Differences under commercial versus non-commercial procedures pertain, for example, to the contracting officer‘s determination of price reasonableness, the government‘s right to inspect and test, and government rights to acquire technical data. Appendix III outlines these and other key differences. The Air Force contracts for simulator training are structured as commercial acquisitions, but the Army‘s is not. Army officials told us they could not justify calling the requirement ’commercial“ because the simulators would be configured to reflect combat helicopters, which do not exist in the commercial market. In August 2005, a DOD Inspector General review of the procurement procedures for the F-16 contract concluded that the simulator service did not meet the definition or intent of a commercial service and recommended that the Air Force not use commercial procedures for the re-competed F-16 contract. The Air Force is using non-commercial procedures for the new contract.
To allow for contractor recoupment of up-front investment, the strategy to acquire simulator services envisioned longer duration contracts. This coincided with practices in commercial industry, where long-term relationships between buyer and seller were becoming common. The Air Force and Army adopted this approach by including award-term incentives in the contracts. This incentive can best be described as a variant of an award-fee incentive, where the contractor is rewarded for excellent performance with an extension of the contract period instead of additional fee. Under the award-term concept, an assessment of the contractor‘s performance is presented to the term determining official, who unilaterally determines whether to award an extension or a reduction to the contract ordering period. The potential total years of contract performance under the simulator contracts range from 13 to 19.5 years. Appendix IV contains the details of each contract. Award-term incentives are relatively new in government contracting and are not addressed in the Federal Acquisition Regulation (FAR).
Key Players:
Several key players are involved with acquisition and use of simulator training.
For the Air Force:
* Air Combat Command: The requiring entity”the user of simulator training services”is located at Langley AFB, Virginia. The command trains, equips and maintains combat-ready forces for rapid deployment and employment.
* Aeronautical Systems Center: This organization is the acquisition agency for the simulator contracts. Located at Wright-Patterson AFB, Ohio, it manages development, acquisition, modification, and in some cases, sustainment for a wide variety of aircraft and related equipment programs. The center develops attack, bomber, cargo, fighter, trainer, and reconnaissance aircraft for the Air Force.
* Air Force fighter units: These are the users of the simulator training, which currently is taking place in 10 fighter units.
For the Army:
* Fort Rucker, Alabama: Fort Rucker is the requiring entity for the Army‘s helicopter flight simulator services. It is the home of all Army aviation flight training and the location of the initial training for new aviators, known as Flight School XXI. The types of helicopters used in Flight School XXI training are the TH-67 basic training helicopter, Chinook, Blackhawk, Apache, and the Attack Reconnaissance Helicopter. Unlike the Air Force‘s multiple sites, the helicopter simulators provided under the service contract are located at only this one training site, not at each operational unit.
* Army Program Executive Office for Simulation, Training and Instrumentation: This office‘s mission is to provide training, testing, and simulation solutions for soldier readiness. The office is co-located in Orlando, Florida, with the Naval Air Systems Command, which awarded the contract on behalf of the Army.
Service Contract Approach Had Perceived Benefits but Was Not Adequately Supported:
Both the Air Force and Army were faced with obsolete simulators due to decisions to not devote sufficient procurement funds to upgrade existing simulator hardware and software. The decision to buy simulator training as a service allowed use of O&M funds, which would alleviate the need to compete for procurement funds. Further, it was envisioned that service contracts would allow for automatic simulator upgrades to match the changing aircraft configurations, because industry would be responsible for acquiring, operating, and maintaining the simulators and keeping them concurrent. However, the decision to embark on a services approach was not supported by a thorough analysis of the costs and benefits, despite a DOD directive providing that the acquisition of simulators is to be based on an evaluation of the benefits and trade-offs of potential alternative training solutions.
Simulators Had Lost Out in Competition for Procurement Funds:
The difficulty associated with competition for limited procurement dollars was a key factor in the decision to turn to service contracts for war-fighting training. Frequently, simulators have lost out in this competition and ended up under-funded. In 1997, the Air Force identified simulators for four aircraft”the F-15C, F-16, F-15E, and AWACS”as ’obsolete or grossly non-concurrent“ due to age, technological obsolescence, and lack of concurrency with operational aircraft. By early 2002, the Army was also faced with non-concurrent helicopter simulators, and field unit commanders were reporting decreased unit readiness. For example, while the goal of the training at Fort Rucker is to produce aviators trained at a proficiency level of two (with level one being the highest), Army officials reported that most of the aviators were leaving school with only a proficiency level of three.
These degraded situations existed despite a DOD directive that provides for the military services to ensure that all development, procurement, operation, and support costs for the acquisition of training simulators were programmed and funded. Recognizing the need to keep simulators current with aircraft configurations, particularly as the use of simulators to substitute for live flying hours was rising, the Air Force issued specific guidance on training devices. For example, Air Force Instruction 36-2248, Operation and Maintenance of Aircrew Training Devices, provides that funding be established for simulator modifications concurrently with modifications to the weapon system. Also, Air Force Instruction 36-2251, Management of Air Force Training Systems, provides that the training system receive the same precedence rating as the prime mission system it supports and the same visibility, funding, and documentation. Nevertheless, Air Force funding decisions had not kept flight training simulators for the four aircraft systems concurrent with aircraft configurations.
Also in the late 1990s, the Air Combat Command had unexpended O&M flying hour funds available due to flight crew deployments and obstacles in scheduling training. Use of these funds for service contracts would alleviate the need to compete for procurement funds in an increasingly tight arena. The competition for procurement dollars was also a factor for the Army, which noted that the funds necessary to maintain and upgrade its helicopter training simulators had ’not competed effectively against other Army operational and logistics requirements.“
Service Contracts Intended to Yield More Concurrent Simulators:
Air Force acquisition officials conducted market research to determine how civilian airlines acquired flight training. They found ’turnkey“ training services contracts [Footnote 10] in place in the commercial airline industry. These officials envisioned that services contracts would provide quicker state-of-the-art pilot and aircrew training and keep up with the rapid pace of technology development by shifting the responsibilities for simulator ownership, operation, and maintenance from the government to the contractor. Further, with the contractor responsible for any development, production, and testing necessary to ready the simulators for use, the Air Force saw that it would be relieved of these multiple acquisition efforts, an important factor given the recently downsized acquisition offices. In addition, a stated benefit of the service contracting approach for simulator training as initially implemented was the streamlining or reduction in government oversight. Since commercial acquisition procedures were used to buy these services, fewer government system reviews were required.
When it decided to take a new approach to solve its helicopter simulator concurrency problems, the Army conducted its own market research, solicited business solutions from industry, and conferred with Air Force DMO officials.
Decision to Use Service Approach Not Supported with Thorough Analysis of Costs and Benefits of Alternatives:
Neither the Air Force nor the Army thoroughly analyzed the costs and benefits of alternative approaches before pursuing this new approach. DOD‘s August 1986, Directive 1430.13, Training Simulators and Devices, provides that the acquisition of simulators be based on an analysis of the training need, the potential use of existing devices to satisfy that need, and an evaluation of the benefits and trade-offs of potential alternative training solutions. [Footnote 11] A 1999 report to the Air Force on the DMO program also noted the importance of identifying key business factors before embarking on a major acquisition. As a result of the failure to conduct a thorough review of the various alternatives to solving the problem of non-concurrent simulators, decision makers lacked information on the potential cost and benefit estimates that would be encountered should facts, circumstances, and assumptions change.
The historical documents we reviewed demonstrate that within the Air Force there was uncertainty about the cost-effectiveness of the service contract approach to simulator training. Although the potential for reduced costs through outsourcing certain responsibilities and eliminating government logistics support were cited in some decision documents, other documents indicated that the service contract approach would not cost significantly more or less than the traditional ownership strategy. Air Force officials told us that a comprehensive study of various options for providing simulator training had been commissioned. However, they have been unable to locate it.
In preparing to re-compete the F-16 simulator contract, the DMO program office completed a formal business case analysis in November 2005, in response to a July 2005 congressional request. Air Force officials acknowledged that, if not for the request, the formal business case analysis would not have been completed.
The Army completed two business case analyses prior to contracting for simulator services under the Flight School XXI program, but the analyses lacked sufficient detail to provide a thorough examination of the pros and cons of the new approach. The scope of the analyses was limited to determining (1) what length of service contract would be appropriate to justify the large up-front investment required of the contractor and (2) whether projected funding was sufficient to meet program costs in the event the Army was required to follow the traditional acquisition approach. The Army provided us with decision briefings that set forth various options for simulator training, but the documents ruled out all but the service contract approach without providing supporting analyses of the costs and benefits associated with each alternative. Further, the traditional method, where the government bought the simulators, was eliminated as an option due its perceived inability to meet the Flight School XXI 15-month start-up time frame. This schedule eventually slipped more than 10 months with, according to Army officials, no detrimental effect on student training schedules. The briefings do not address the possibility that the 15-month time frame was flexible.
While Training Has Improved, the Expected Number of Simulator Sites Has Not Been Activated:
Air Force and Army officials told us the new simulators are big improvements over what they had previously. However, the Air Force has faced funding uncertainties using O&M funds for the contracts, and subsequent schedule slippages have resulted in fewer simulator sites activated than planned. In particular, the F-16 simulator training contractor, citing the reduced activations, notified the Air Force as early as May 2001, that it was unable to provide simulator services as originally agreed and wished to restructure the contract. Later, the company cited Air Force funding problems and schedule slips as the basis for claims against the Air Force and notified the Air Force that its financial situation under the contract was no longer viable. The Air Force will let the current F-16 simulator training contract expire in June 2007 and is in the process of re-competing the contract, which will likely result in a training gap for pilots and additional costs to the Air Force.
Some Air Force and Army Sites Are Operating and Officials Cite Improved Training Results:
At the locations we visited, officials told us they were pleased with the quality of the simulator training, particularly when compared with the level of training they had in the past. Pilots are routinely surveyed about the training they receive, and officials told us that, generally, the results have been very positive. For example, the Director of Operations for the F-16 mission training center at Shaw AFB told us that the simulation hardware and software are outstanding and that the training received by young pilots is great. Initial training began under the Army‘s Flight School XXI contract in November 2005. While all planned simulators have not yet been activated, according to Flight School XXI officials the school is now meeting its training goal and producing aviators with a proficiency level of two, an improvement over the old regime.
As of July 2006, the Air Force had 16 training simulator sites operational, as shown in table 2.
Table 2: Operational Simulator Training Locations and Start Dates as of July 2006:
System: F15-C;
Location: Eglin AFB, Florida;
Start date: May 1999.
System: F15-C;
Location: Langley AFB, Virginia;
Start date: May 2000.
System: F15-C;
Location: Elmendorf AFB, Alaska;
Start date: July 2003.
System: F15-C;
Location: Kadena air base (AB), Okinawa;
Start date: April 2005.
System: F15-C;
Location: Lakenheath AB, United Kingdom;
Start date: January 2006.
System: AWACS;
Location: Tinker AFB, Oklahoma[a];
Start date: December 2001.
System: AWACS;
Location: Elmendorf AFB, Alaska;
Start date: September 2003.
System: AWACS;
Location: Tinker AFB, Oklahoma;
Start date: December 2003.
System: AWACS;
Location: Kadena (AB), Okinawa;
Start date: May 2005.
System: AWACS;
Location: Tinker AFB, Oklahoma;
Start date: September 2005.
System: F-16;
Location: Shaw AFB, South Carolina;
Start date: May 2002.
System: F-16;
Location: Shaw AFB, South Carolina;
Start date: January 2003.
System: F-16;
Location: Mountain Home AFB, Idaho;
Start date: September 2002.
System: F-16;
Location: Mountain Home AFB, Idaho;
Start date: February 2004.
System: F-16;
Location: Spangdahlem AB, Germany;
Start date: April 2004.
System: F-16;
Location: Misawa AB, Japan;
Start date: June 2005.
Source: GAO analysis of Air Force data.
[a] Some locations have multiple mission training centers.
[End of table]
Air Force Funding Uncertainties Have Resulted in Schedule Slippages:
The use of O&M funds under the service contract approach was intended to overcome the situation the military services had faced in the past, when internal decisions on funding priorities had resulted in inadequate procurement funds being made available for simulators. However, almost from the start of the DMO program, funding has been less than projected. As a result, schedule slippages have occurred for many sites compared with original Air Force requirements set forth in acquisition plans. Army officials told us that, to date, O&M funding for the Flight School XXI program has not been reduced. Army officials committed at the outset to fully fund the contract in accordance with the originally projected funding profile and, to date, the funding level has remained stable.
As early as the 2002 budget planning process, Air Force budget requests did not fully fund planned activations, with a total difference between estimated requirements and funding of $524 million over the future year defense plan, as shown in table 3.
Table 3: Comparison of DMO Estimated Funding Requirement with the Fiscal Year 2002 Program Objective Memorandum Funding Levels (dollars in millions):
DMO estimated funding requirement:
FY 2002: $97;
FY 2003: $169;
FY 2004: $148;
FY 2005: $157;
FY 2006: $133;
FY 2007: $86;
Total: $790.
Fiscal year 2002 funding approved in the program objective memorandum:
FY 2002: $46;
FY 2003: $49;
FY 2004: $43;
FY 2005: $42;
FY 2006: $43;
FY 2007: $43;
Total: $266.
Funding difference:
FY 2002: -$51;
FY 2003: -$120;
FY 2004: -$105;
FY 2005: -$115;
FY 2006: -$90;
FY 2007: -$43;
Total: -$524.
Source: GAO analysis of Air Force documents.
[End of table]
An October 2000 Air Force ’roadmap“ report stated that this funding scenario would ’severely impact the executability of the current contracted efforts, as well as the entire [DMO] vision.“ Further, other Air Force decisions, in reaction to fiscal constraints and programs viewed as higher priority, have led to additional funding differences. The Air Combat Command sought to mitigate the impact of these funding differences by shifting flying hour funds into the DMO program in 2003. Table 4 depicts some key events pertaining to the program‘s funding impacts and the command‘s attempts to secure additional O&M funds.
Table 4: Key Events That Have Affected Funding for the DMO Program:
Event: 1999: Unexpended flying hour funds centralized at Air Force headquarters;
Impact: Air Combat Command no longer had flexibility to use its unexpended flying hour funds for DMO; it now had to request, justify, and compete for funding.
Event: 2001: Budget decision results for fiscal years 2002–2007; Impact: Decisions made by the Air Force to fund other, higher-priority programs resulted in a difference of $524 million over the 6-year period, compared to the DMO plan.
Event: 2004: Decision to accelerate F/A-22 DMO compliance;
Impact: Office of the Secretary of Defense mandated that the schedule for linking the F/A-22 into the DMO network be accelerated. The $57 million required to do this was unexpectedly taken from the DMO budget for fiscal years 2006-08, instead of from the F/A-22 program‘s budget.
Event: 2004: Budget decision results for fiscal year 2006;
Impact: Decision was made by Air Force not to fund $250 million to facilitate access to the DMO network by other aircraft (B-1, B-2, B-52, A-10), although flying hour funds had already been shifted out of these programs to support the DMO program.
Event: 2004: Budget decision results for fiscal years 2006-2011; Impact: Differences ranging from over $225 million in fiscal year 2006 to over $50 million in fiscal year 2011 as compared to the DMO plan could not be alleviated, even with a 5-percent transfer of flying hour funds from other programs.
Event: 2005: Decision to end F-16 simulator training contract;
Impact: The Air Force decision to end the F-16 simulator contract in 2007 will result in additional costs for re-competition efforts and upgrades of old training devices.
Source: GAO analysis of Air Force documents.
[End of table]
Largely as a result of these funding uncertainties, many Air Force mission training centers have been activated significantly behind the planned schedule contained in acquisition management plans. These schedule slippages for AWACs, F-15C, and F-16 are shown in tables 5, 6, and 7, respectively.
Table 5: Comparison of Planned and Actual Site Activations for AWACS:
Location: Tinker AFB (#1);
Scheduled activation date: 1QFY01;
Actual activation date: 1QFY02;
Quarters behind scheduled activation date: 4.
Location: Tinker AFB (#2);
Scheduled activation date: 1QFY03;
Actual activation date: 2QFY03;
Quarters behind scheduled activation date: 1.
Location: Tinker AFB (#3);
Scheduled activation date: No date set;
Actual activation date: 4QFY05
Quarters behind scheduled activation date: Cannot be determined.
Location: Elmendorf AFB;
Scheduled activation date: 4QFY02;
Actual activation date: 4QFY03;
Quarters behind scheduled activation date: 4.
Location: Kadena AB;
Scheduled activation date: 4QFY02;
Actual activation date: 3QFY05;
Quarters behind scheduled activation date: 11.
Source: GAO analysis of Air Force data.
[End of table]
Table 6: Comparison of Planned and Actual Site Activations for F-15C:
Location: Eglin AFB;
Scheduled activation date: 2QFY99;
Actual activation date: 2QFY99;
Quarters behind scheduled activation date: 0.
Location: Langley AFB;
Scheduled activation date: 3QFY99;
Actual activation date: 3QFY99;
Quarters behind scheduled activation date: 0.
Location: Kadena AB;
Scheduled activation date: 1QFT02;
Actual activation date: 3QFY05
Quarters behind scheduled activation date: 14.
Location: Lakenheath AFB;
Scheduled activation date: 1QFT02;
Actual activation date: 2QFY06;
Quarters behind scheduled activation date: 17.
Location: Elmendorf AB;
Scheduled activation date: 1QFY03;
Actual activation date: 4QFY03;
Quarters behind scheduled activation date: 3.
Source: GAO analysis of Air Force data.
[End of table]
Table 7: Comparison of Planned and Actual Site Activations for F-16:
Location: Shaw AFB (#1);
Scheduled activation date: 1QFY02;
Actual activation date: 3QFY02;
Quarters behind scheduled activation date: 2.
Location: Shaw AFB (#2);
Scheduled activation date: 3QFY02;
Actual activation date: 2QFY03;
Quarters behind scheduled activation date: 3.
Location: Mt. Home AFB (#1);
Scheduled activation date: 3QFT02;
Actual activation date: 4QFY02
Quarters behind scheduled activation date: 1.
Location: Mt. Home AFB (#2);
Scheduled activation date: 4QFT03;
Actual activation date: 2QFY04;
Quarters behind scheduled activation date: 2.
Location: Misawa AB;
Scheduled activation date: 2QFY03;
Actual activation date: 3QFY05;
Quarters behind scheduled activation date: 9.
Location: Spangdahlem AB;
Scheduled activation date: 3QFY03;
Actual activation date: 3QFY04;
Quarters behind scheduled activation date: 4.
Source: GAO analysis of Air Force data.
[End of table]
Air Force officials told us that since most of the original dates were ’notional,“ meaning that they were not firm requirements, but rather were intended to provide contractors with information about potential mission training center sites, the timely achievement of the schedules was not required. However, contractor representatives told us that their proposals relied upon the planned site activation schedules contained in the contracts, and delays could directly affect their profitability.
Army‘s Simulator Activation Schedule Was Also Delayed:
The Army has twice rebaselined the activation schedules for the Flight School XXI simulators”the TH-67 and the advanced aircraft virtual simulators (AAVS)”as shown in figure 1.
Figure 1: Flight School XXI Schedule Rebaselines:
[See PDF for image]
This figure is a line graph with two lines: AAVS and TH-67. The vertical axis of the graph represents months after award from 0 to 40. The horizontal axis of the graph represents three scenarios: months to activate after contract award; months to activate after contract award rebaseline 1; and months to activate after contract award rebaseline 2.
Source: GAO analysis of Army data.
[End of figure]
In the original contract, the TH-67 basic training helicopter simulators were scheduled to begin operation in December 2004, 15 months after contract award. The Flight School XXI project manager could not provide documentation to support this time frame and, in fact, told us that the flight school could not have been ready for students at that time. The Army subsequently rebaselined the schedule to allow for an 8-month delay. Similarly, the Army revised the AAVS activation schedule”originally set at 18 months after contract award”to allow a 7-month delay. According to the project manager, these delays resulted from a protest of the contract award by a competitor and the contractor‘s renegotiation with its subcontractors. The schedule was rebaselined a second time, as shown above, because the contractor was not able to meet the adjusted schedule. The Army agreed to the further slippages in exchange for the contractor‘s providing two extra terrain databases as consideration. Despite these schedule changes, the necessary simulators and facilities were ready for the first flight school class in November 2005, in accordance with the final revisions to the contract schedule.
F-16 Simulator Training Gap Anticipated:
The risk the government faces if a contractor fails to perform as expected under the service contracts is heightened because the government does not own anything”the hardware, software, and data rights are owned by the contractor. In the traditional approach, the government would own the hardware and any software or data it had acquired rights to. While there would be no guarantees as to the condition of these items if the contractor had failed to perform, the government would at least be able to provide them to the replacement contractor, who could potentially make use of them under a new contract. The situation the Air Force has faced with the F-16 simulator contract is illustrative of the potential for not only a degradation in training, but also increased costs to the government when contract performance does not occur as planned.
From the outset, the Air Force believed that the F-16 simulator contractor‘s cost estimate was low, as it was about $70 million less than the government‘s estimate. According to Air Force and contractor officials, the reason for the low cost estimate was that the contractor amortized its development costs over all the sites that were planned to be activated rather than the minimum number that were contractually required. When schedule delays occurred and the expected sites were not activated, the contractor reported that it lacked the financial viability to continue work under the contract. In April 2003, the contractor stopped work toward making the simulators concurrent with the aircraft, stating that it considered the tasks beyond the contract scope. Subsequently, it told the Air Force it was not in its best interest to activate additional training sites. [Footnote 12]
The Air Force will allow the F-16 simulator training contract to expire in June 2007 because, according to DOD, the contractor failed to earn enough award-term points to extend the period of performance. The Air Force plans to re-compete the contract. Two aspects of the original contract, awarding it as a commercial acquisition and including an award-term provision, will not be included in the new contract.
Because of the time needed to re-compete the contract and for the winning contractor to provide initial training capabilities, the Air Force faces a potential training gap of over 2 years, during which even the current degraded level of F-16 simulator training services will not be available to pilots. In an effort to ensure some level of continued training during that period, the Air Force plans to award a contract for interim service capability at three air bases. This interim capability will be available for block 50 aircraft only. [Footnote 13] For the block 40 aircraft, the Air Combat Command plans to spend approximately $20 million to refurbish old F-16 unit training devices. These devices are limited in training potential compared to the current level of simulation.
Return on the Expenditure of Taxpayer Dollars Is Not Effectively Tracked:
The Air Force and the Army are not effectively tracking the return on their expenditure of taxpayer dollars to acquire simulator training services. The extent to which the simulators are being used is either not measured or is measured inconsistently. The government is paying for activities conducted during the simulator development period but lacks insight into what it is actually paying for. Finally, award-term evaluations that were established to encourage excellent contractor performance do not always measure key acquisition outcomes such as simulator availability and concurrency, and can result in additional contract years being awarded for only ’satisfactory“ performance.
Information on Utilization Rates Is Lacking:
The utilization rate is the percentage of available hours the simulators are actually used. The Army is not tracking the extent to which aviators are using the contracted service for Flight School XXI simulators, even though for simulators the Army owns, utilization rates are tracked. Program officials told us that, because the Army is contracting for simulator training to be available, there was no need to track the extent to which the government is using this availability. Without data on utilization rates, the Army has no basis for determining the extent to which it is using the services it is buying.
We found that Air Force installations are collecting information on monthly utilization rates, as provided for in a May 1998 Air Force instruction. [Footnote 14] However, rates at the locations we examined were often far less than the hours the government purchased. For the three AWACS mission training centers at Tinker AFB, for example, we found that, during the 2-year period ending December 2005, monthly utilization rates were frequently reported at less than 50 percent, as shown in figure 2.
Figure 2: Comparison of Monthly Utilization Rates for AWACS:
[See PDF for image]
This figure is a line graph with four lines: Tinker 1; Tinker 2; Tinker 3; and Full capacity. The vertical axis of the graph represents percent from 0 to 120, with the full capacity line running horizontally at 100. The horizontal axis of the graph represents months from October 2003 to December 2005.
Source: GAO analysis of Air Force data.
[End of figure]
The Air Force Audit Agency has reported that installations had acquired excess simulator capacity and unnecessarily consumed O&M funds that could have been applied to other mission requirements. At Shaw AFB, for example, the agency found that the Air Force had paid to use the simulator 10 hours a day, but only used it about 6 hours per day over a 4-month period. [Footnote 15] The underutilization was attributed to missions being either not scheduled or canceled. Deployment requirements and range training were identified as contributing factors. At Spangdahlem AB, the audit agency reported that the Air Force had contracted for excess hours of simulator availability to provide the maximum flexibility for pilot schedules. [Footnote 16] As a result, the Air Force paid for enough simulator availability to hold 3,952 training events in fiscal year 2005, even though it needed only 1,982 training events to meet training requirements.
Our analysis also found that monthly utilization rate calculations are inconsistent among DMO system sites, even though an Air Force instruction provides guidance on how to calculate and report utilization data. We asked six installation quality assurance representatives how they calculated utilization rates. Four of the six representatives were unaware of the instruction, telling us that they had not received any guidance for calculating simulator use. Several different calculation methods are being used, as described in table 8.
Table 8: Comparison of Monthly Utilization Rate Calculation Methods for DMO System Simulators[a]:
Simulator: AWACS;
Calculation method: Number of hours where some training was given divided by the number of available training hours.
Simulator: F15-C;
Calculation method: Number of training periods in which one or more simulators was in use divided by the number of training periods available.
Simulator: F-16 (Shaw AFB);
Calculation method: Number of system elements used that were effective divided by the number of possible system elements.
Simulator: F-16 (Mountain Home AFB);
Calculation method: Number of simulators used divided by the number of available simulators.
Source: Air Force quality assurance representatives.
[a] F-15E service not yet available.
[End of table]
Air Combat Command officials told us the reported utilization rates are used to determine whether or when to activate another training center at a site. They also said they are using utilization rate information to determine how many additional ’live“ flying hours can be moved to the simulators, in particular to alleviate the burden of high fuel costs for aircraft. Because of the very different methods being used to calculate the rates, however, decisions are being made based on non-comparable information.
In addition, we found that the Air Force‘s instruction for calculating monthly simulator utilization rates could result in overstating the rates, thus overstating the return on the expenditures made. The instruction directs that utilization be reported when any or all devices at a given location are used. Thus, the Air Force can pay to have four simulators available at a site, use only one of the four during a training period, and still report that simulator utilization was 100 percent as opposed to 25 percent of the paid availability.
Payments for Tasks during Development Period Are Not Transparent:
Under the services approach, contractors commit to major investment at the front end, with the return on their investment to come from hourly fees received for providing simulator service. As an additional way to help the contractor recoup its costs earlier, the government added ’preparatory“ tasks during the development period prior to the start of service. These tasks are defined in the contracts as discrete events, such as site surveys and training capability assessments, that are ordered and paid for prior to the start of service. Payments for these tasks provide the contractor cash flow between contract award and the planned service start dates and give the government a contractual avenue for contract oversight prior to receiving services. We found that the Air Force and Army have little insight into what they are paying for under the preparatory tasks. Although the invoices reflect only the discrete tasks, such as training capabilities assessments, the wide range of invoice amounts”from $91,000 to more than $6.5 million for similar tasks”and our discussions with contractor officials suggest that the government is actually making milestone payments to the contractors for a portion of their up-front costs to acquire and develop the simulators.
The original service contract concept for the F-15C, the first simulator contract awarded, had no provision for the contractor to recoup any costs during the development period, which usually lasts more than a year. Figure 3 shows the development period before the start of simulator services and the original hourly rate structure under the F-15C contract.
Figure 3: Original Rate Structure for F-15C Simulator Service Contracts:
This is a line graph with a line depicting the original rate structure for F-15C Simulator Service Contracts. The vertical axis of the graph represents Rate from 0.0 to 7.0. The horizontal axis of the graph represents fiscal years from 2000 to 2015. Also depicted are the contract award date (early in 2000) and the planned service start (2002). From 2007 through 2015, a notation on the initial rate line indicates steady-state rate includes inflation growth.
Source: Air Force data.
[End of figure]
This original approach, according to Air Force and contractor officials, contributed to schedule and certification delays with the F-15C. Air Force officials told us that they had no contractual avenue to obtain insight into the contractor‘s performance during the development period and thus were not aware that the contractor had encountered delays in obtaining information from other programs and in determining the complexity of some simulation elements. As a result, full service was not implemented on schedule and certification of simulation service was delayed until after the start of initial service. Further, according to the contractor, it suffered an unrecoverable loss of income during the high-rate, initial service period. Subsequently, based on feedback received from industry, the Air Force changed its approach and incorporated preparatory services into the F-15C contract and all subsequent DMO system contracts to obtain more visibility into contractor activities during the development period. The Army also paid for preparatory services during the development period of the Flight School XXI contract.
Our analysis of the Air Force‘s payments for preparatory services found significantly disparate costs for site surveys and training assessments, as reflected in tables 9 and 10, respectively.
Table 9: Comparison of Site Survey Costs at Selected DMO System Sites (dollars in thousands):
Sites: Tinker AFB (#1);
System: AWACS: $130;
System: F-15C: [Empty];
System: F-15E: [Empty];
System: F-16: [Empty].
Sites: Tinker AFB (#2):
System: AWACS: $126;
System: F-15C: [Empty];
System: F-15E: [Empty];
System: F-16: [Empty].
Sites: Elmendorf AFB;
System: AWACS: $22;
System: F-15C: $55;
System: F-15E: $90;
System: F-16: [Empty].
Sites: Kadena AB;
System: AWACS: $330;
System: F-15C: $67;
System: F-15E: [Empty];
System: F-16: [Empty].
Sites: Lakenheath AB;
System: AWACS: [Empty];
System: F-15C: $56;
System: F-15E: $100;
System: F-16: [Empty].
Sites: Shaw AFB (#1);
System: AWACS: [Empty];
System: F-15C: [Empty];
System: F-15E: [Empty];
System: F-16: $625.
Sites: Shaw AFB (#2);
System: AWACS: [Empty];
System: F-15C: [Empty];
System: F-15E: [Empty];
System: F-16: $1,000.
Sites: Shaw AFB (#3);
System: AWACS: [Empty];
System: F-15C: [Empty];
System: F-15E: [Empty];
System: F-16: $1,100.
Source: GAO analysis of contractor invoices.
[End of table]
Table 10: Comparison of Training Capability Requirements Assessments and Training Capability Assessments for DMO Systems (dollars in thousands):
Task: Training capability requirements assessments[a]: Number 1;
System: AWACS: $122;
System: F-15C[c]: [Empty];
System: F-15E: $2,000;
System: F-16: $2,850.
Task: Training capability requirements assessments[a]; Number 2;
System: AWACS: [Empty];
System: F-15C[c]: [Empty];
System: F-15E: $2,600;
System: F-16: $4,850.
Task: Training capability requirements assessments[a]; Number 3;
System: AWACS: [Empty];
System: F-15C[c]: [Empty];
System: F-15E: $200;
System: F-16: $1,800.
Task: Training capability requirements assessments[a]; Number 4;
System: AWACS: [Empty];
System: F-15C[c]: [Empty];
System: F-15E: $100;
System: F-16: [Empty].
Task: Training capability assessments[b]; Number 1;
System: AWACS: $356;
System: F-15C[c]: $91;
System: F-15E: [Empty];
System: F-16: $1,644.
Task: Training capability assessments[b]; Number 2;
System: AWACS: $454;
System: F-15C[c]: $722;
System: F-15E: [Empty];
System: F-16: $6,575.
Task: Training capability assessments[b]; Number 3;
System: AWACS: $261;
System: F-15C[c]: $565;
System: F-15E: [Empty];
System: F-16: $3,161.
Task: Training capability assessments[b]; Number 4;
System: AWACS: $289;
System: F-15C[c]: [Empty];
System: F-15E: [Empty];
System: F-16: $1,000.
Task: Training capability assessments[b]; Number 5;
System: AWACS: [Empty];
System: F-15C[c]: [Empty];
System: F-15E: [Empty];
System: F-16: $115.
Task: Training capability assessments[b]; Number 6;
System: AWACS: [Empty];
System: F-15C[c]: [Empty];
System: F-15E: [Empty];
System: F-16: $150.
Source: GAO analysis of contractor invoices.
[a] Training capability requirements assessments trace the capabilities of the simulator to the training task list, which provides descriptions of training for various mission profiles.
[b] Training capability assessments include reviews of system performance evaluations and verification and validation tests to authorize shipment of simulator service to designated destinations.
[c] The F-15C contractor was not paid for training capability requirements assessments as preparatory services.
[End of table]
We asked Air Force and Army officials what was specifically included in these preparatory services and how they determined what they received in return for payments made. They told us that the contractors determine what is included and needed for each service at each site.
Three of the four contractors we spoke with agreed that funding for preparatory tasks helped defray their development costs. They said that, in effect, they bill for these tasks as milestone payments rather than for the discrete tasks themselves. Thus, they are able to begin defraying hardware and software development costs before the start of services. Officials from the fourth contractor stated that site survey tasks are standard but that there is some leeway in what is to be done for training capability assessments and training capability requirements assessments.
With the upcoming re-competition of the F-16 simulator training contract, the Air Force may pay again for the preparatory service tasks in the new contract‘s development period, having already spent nearly $42 million on these tasks in the initial contract. Air Force officials told us they cannot assume that potential offerors would make use of the preparatory work the original contractor has performed.
Award-term Evaluations Do Not Measure Key Acquisition Outcomes:
In an effort to measure performance and encourage the contractors to perform in an efficient and effective manner, both the Air Force and Army employ award-term incentives. However, while the award-term evaluation areas include pilot and crew satisfaction, they do not always measure the key acquisition outcomes of system availability and concurrency with aircraft upgrades.
While the Air Force does include system availability as an evaluation area, it is assigned only 25 to 30 percent of the total score. Concurrency is not included as a separate evaluation area. The Army‘s evaluation areas, on the other hand, include concurrency but not system availability. While the Army requires the tabulation and submission of such data as operational availability and training service completion rate, these data are not included in the award-term evaluations. [Footnote 17] In addition, several of the evaluation areas include assessments of such things as responsiveness to government requests for cost and pricing data for proposed work not in the initial contract. We recently recommended that DOD move toward more outcome-based award-fee criteria that would promote accountability for acquisition outcomes, rather than include criteria such as responsiveness to government customers or the quality of proposals submitted. [Footnote 18]
Table 11 compares the award-term evaluation areas and the weight given to each area.
Table 11: Comparison of Award-term Areas of Evaluation and the Weight Given to Each Area by System:
Evaluation area: Pilot/crew satisfaction;
Weight of evaluation area (percent of total), AWACS: 50;
Weight of evaluation area (percent of total), F-15C: 50;
Weight of evaluation area (percent of total), F-16: 50;
Weight of evaluation area (percent of total), Flight School XXI: 40.
Evaluation area: System availability;
Weight of evaluation area (percent of total), AWACS: 30;
Weight of evaluation area (percent of total), F-15C: 25;
Weight of evaluation area (percent of total), F-16: 20;
Weight of evaluation area (percent of total), Flight School XXI: [Empty].
Evaluation area: Technical insertion;
Weight of evaluation area (percent of total), AWACS: 10;
Weight of evaluation area (percent of total), F-15C: 15;
Weight of evaluation area (percent of total), F-16: 10;
Weight of evaluation area (percent of total), Flight School XXI: 10.
Evaluation area: Proposal quality (also called affordability);
Weight of evaluation area (percent of total), AWACS: 10;
Weight of evaluation area (percent of total), F-15C: 5;
Weight of evaluation area (percent of total), F-16: 15;
Weight of evaluation area (percent of total), Flight School XXI: 5.
Evaluation area: Small business participation;
Weight of evaluation area (percent of total), AWACS: [Empty];
Weight of evaluation area (percent of total), F-15C: 5;
Weight of evaluation area (percent of total), F-16: 5;
Weight of evaluation area (percent of total), Flight School XXI: [Empty].
Evaluation area: Concurrency;
Weight of evaluation area (percent of total), AWACS: [Empty];
Weight of evaluation area (percent of total), F-15C: [Empty];
Weight of evaluation area (percent of total), F-16: [Empty];
Weight of evaluation area (percent of total), Flight School XXI: 20.
Evaluation area: Management (includes small business participation);
Weight of evaluation area (percent of total), AWACS: [Empty];
Weight of evaluation area (percent of total), F-15C: [Empty];
Weight of evaluation area (percent of total), F-16: [Empty];
Weight of evaluation area (percent of total), Flight School XXI: 25.
Evaluation area: Total;
Weight of evaluation area (percent of total), AWACS: 100;
Weight of evaluation area (percent of total), F-15C: 100;
Weight of evaluation area (percent of total), F-16: 100;
Weight of evaluation area (percent of total), Flight School XXI: 100.
Source: Air Force and Army award-term plans.
Note: F-15E does not include an award-term incentive.
{End of table]
Source: Air Force and Army award-term plans. Note: F-15E does not include an award-term incentive.
The Air Force and Army both assign the largest weight to ’pilot/crew satisfaction.“ However, this measure has limitations, particularly when it is heavily relied on to inform award-term decisions. Air Force officials told us that it is in the pilots‘ best interests to assign a high rating to this factor; otherwise, they could be viewed as not having received adequate training and could be asked to retake it. Additionally, pilots are frequently hurried in completing their surveys and dash off check marks without much consideration. Also, the distinction between the levels of satisfaction can be blurry. For the Army, for example, if training and support are adversely impacted for an ’extended period,“ user satisfaction is to be rated as unsatisfactory. However, if the adverse impact occurs ’infrequently or temporarily,“ it is considered marginal. Because the terms are not defined, the Army cannot be certain that pilots are providing consistent ratings.
We also found that, under the Air Force‘s award-term plan, contractors can earn an additional award-term year for only satisfactory performance because awarded points are rolled over to the next evaluation period. A contractor with only satisfactory performance in each of five rating areas can receive up to 51 points each year; thus, within 2 years, it can accumulate the 100 points needed for a 1-year contract extension. The F-16 simulator training contractor, for example, which recently notified the government that it could not continue to perform under the contract, received overall award-term evaluations of ’very good“ for the first two rating periods (May 2002 through July 2003) and ’satisfactory“ in the third and fourth periods (July 2003 through January 2005) and earned one contract year extension. The Army has taken a different approach; under its award-term plan it is very unlikely that the contractor can be awarded contract extensions for ’satisfactory“ performance because rollover is allowed only when more than 100 points are earned. Thus, the contractor with only satisfactory performance cannot accumulate enough points for an additional contract year.
While service is not yet available, the F-15E simulator training contract, awarded in August 2003, does not include an award-term incentive because, according to the contracting officer, ’it doesn‘t work.“ Contractor officials told us that the subjective nature of the criteria and the manner in which they are applied negate the award term as a performance incentive. Both the Air Force and Army indicated that they are moving away from using award-term incentives on future contracts. The Air Force will not include such an incentive in its re-competition for the F-16 simulator training contract because, according to the DMO director, it has not been found to be a significant motivator to the contractor; experience has shown that withholding payment for poor service is a much more effective tool to induce improved performance. In addition, since a recent statutory provision limits future total contract periods of performance to 10 years, an award-term provision can no longer be used to implement long-term arrangements such as those in place for the existing simulator training contracts. [Footnote 19]
We recently reported that DOD has little evidence to support its belief that award fees improve contractor performance and acquisition outcomes and, in fact, frequently pays out most of the available award fee to contractors regardless of their performance outcomes. [Footnote 20] We also found that DOD contracts frequently included rollover provisions, where unearned award fee from one evaluation period was shifted to a subsequent evaluation period or periods, thus providing the contractor an additional opportunity to earn previously unearned fee. We recommended that DOD issue guidance on when rollover of award fee is appropriate. A March 2006 guidance on award fee contracts states, among other things, that use of rollover provisions should be the exception rather than the rule and that the decision to use rollover provisions should be addressed in the acquisition strategy, including a rationale as to why a rollover provision is appropriate.
Conclusion:
Because simulator training had lost out in the internal competition for procurement funds, the Air Force and Army turned to service contracts, expecting that O&M funds would be made available to meet requirements. In the case of the Air Force, this expectation has not materialized and planned site activations have been slowed. In addition, although the Air Force and Army plan to continue with the service contract approach for simulator training, neither supported the decision with a thorough analysis of the costs and benefits of alternative approaches to delivering the training. Finally, the heightened risks associated with increased reliance on contractors to deliver simulator training calls for careful attention to contract management and oversight. Effective and well-managed incentives for motivating performance are especially important. Better government visibility into the contractors‘ activities, such as preparatory tasks, during the development period is critical so that the government can understand the basis for what are essentially milestone payments during that phase. In addition, unless utilization rates are tracked in a consistent manner, the government will not know whether it is making the best use of what it is buying.
Recommendations for Executive Action:
To help ensure that the best approach is used to provide the war-fighter with needed training, we recommend that the Secretary of Defense direct the Secretaries of the Air Force and Army to conduct a thorough analysis of the costs and benefits of using service contracts for simulator training to determine if it is indeed the best approach. The analysis should proactively address potential risks associated with the service contract approach and identify the level of simulator training needed to meet requirements.
To help ensure that the required training is provided to pilots, we recommend that the Secretary of the Air Force reconcile the funding level needed for simulator training with the requirements identified in the evaluation of costs and benefits of the service contract approach and take steps to allocate funds accordingly.
To help ensure that the incentives motivate contractor performance toward achieving desired training outcomes, we recommend that the Secretary of Defense direct the Secretaries of the Air Force and Army to take the following two actions:
* Determine whether it is in the government‘s best interest to retain the award-term incentive under these service contracts.
* If the award-term incentive is retained, take appropriate steps to improve the approach by reassessing the areas to be rated and the definitions of performance levels for the various grade categories. For the Air Force, improvements to the approach should include a determination as to whether to continue allowing rollover of award-term points.
To help ensure greater transparency into what the government is paying for preparatory tasks during the development phase, we recommend that the Secretary of Defense direct the Secretaries of the Air Force and Army to take the following two actions:
* Reassess the pricing of any up-front payments made to the contractors during the development period on future replacement or restructured contracts.
* If retained, take appropriate measures to (1) create an appropriate and transparent contract payment mechanism, separate from the preparatory tasks, if development costs are to be reimbursed; and (2) increase visibility into the percentage of upfront development costs contractors are recouping from these preparatory tasks and development payments.
To help ensure that available simulator training for the warfighter is used in the most effective and efficient manner, we recommend that the Secretary of Defense take the following four actions:
* Direct the Secretaries of the Air Force and Army to determine whether and how simulator utilization can be increased in order to maximize use of taxpayer dollars.
* Direct the Secretary of the Army to track and record monthly utilization rates on Flight School XXI contracted simulator training in order to have the data necessary to adjust training requirements and contract provisions, as necessary.
* Direct the Secretary of the Air Force to revise Air Force Instruction 36-2248, Operation and Management of Aircrew Training Devices, to ensure that, for the purposes of reporting utilization rates, the usage of individual training simulators is calculated.
* Direct the Secretary of the Air Force to ensure that all sites consistently track and report simulator utilization.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, DOD concurred with all but one of our recommendations. DOD partially concurred with our recommendation that the Army track and record monthly utilization rates on simulators at Flight School XXI. DOD stated that the service contract approach requires only that the vendor meet the programmed student training load. Nevertheless, DOD stated that the contractor is required to submit utilization data and that the data are available for use in future adjustments to the contracting strategy, requirements, or provisions. Our recommendation was intended to encourage DOD to fully understand its student training requirements and to collect the information to decide whether it needs to adjust requirements or contract provisions regarding simulator availability. Whether the utilization rates pertain to individual simulators or the student training load as a whole, we believe that the Army needs to know the extent to which it is actually using the simulator availability it is buying. DOD also offered two corrections to information in the draft, and we made changes as appropriate. DOD‘s comments are included in their entirety in appendix II.
We will send copies of this report to the Secretaries of Defense, the Air Force, and the Army; appropriate congressional committees; and other interested parties. We make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at [hyperlink, http://www.gao.gov].
If you or your staff has questions concerning this report, please contact me at (202) 512-4841 or by e-mail at shamesl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report.
Signed by:
Lisa Shames:
Acting Director:
Acquisition and Sourcing Management:
[End of section]
Appendix I: Scope and Methodology:
To determine which factors led the Air Force and Army to acquire simulator training as a service contract using operation and maintenance funds, we analyzed historical documents such as acquisition plans, briefings, and decision memorandums. For the Air Force, we interviewed Air Force management, including officials at the Office of the Assistant Secretary of the Air Force, Acquisition; Aeronautical Systems Center (responsible for contracting the simulator training services) and the Air Combat Command (funds and uses the simulator training). For the Army, we interviewed officials at the Office of the Assistant Secretary of the Army for Acquisition, Logistics, and Technology; and Army officials responsible for managing the Army‘s Flight School XXI initiative, including officials of the Program Executive Office for Simulation, Training and Instrumentation. We visited Langley Air Force Base, Virginia, to observe F-15C simulator training; Shaw Air Force Base, South Carolina, to observe F-16 simulator training; and Fort Rucker, Alabama, to observe the Flight School XXI helicopter simulator training. Additionally, to evaluate whether the military services adequately justified the new service contract approach, we reviewed the Office of Management and Budget‘s Circular A-11, Appendix B, ’Budgetary Treatment of Lease-Purchases and Leases of Capital Assets,“ and Air Force and Army regulations and guidance regarding business case analyses. We also drew from our prior reviews of Department of Defense systems, in particular our recent review of the Army‘s Future Combat System. [Footnote 21]
To assess whether the new approach has resulted in the planned number of simulator training sites being activated, we evaluated contract documents and information provided by the Air Combat Command and Aeronautical Systems Center to compare planned to actual schedule activations. We gathered and analyzed budget data related to program schedules and interviewed program officials. We analyzed contract documents and other program documents from Flight School XXI and discussed the schedule rebaselining with Army officials. We analyzed the Air Force‘s request for proposals for the F-16 simulator training contract re-competition to determine whether key differences in the acquisition approach were incorporated. To determine if the Air Force and Army are effectively tracking the return on their expenditure of taxpayer dollars, we analyzed simulator utilization data and military service guidance on utilization rates; analyzed contractor performance measurements, annual evaluations, and award-term plans for the simulator training contracts; and compared preparatory service costs charged to the government under each of the four Air Force contracts and the Army contract. We also interviewed contractor representatives and government officials.
[End of section]
Appendix II: Comments from the Department of Defense:
Office Of The Under Secretary Of Defense:
Acquisition, Technology And Logistics:
3000 Defense Pentagon:
Washington, DC 20301-3000:
September 8, 2006:
Ms. Lisa Shames:
Acting Director:
Acquisition and Sourcing Management:
U.S. General Accountability Office:
441 G Street, NW:
Washington, D.C. 20548:
Dear Ms. Shames:
This is the Department of Defense (DoD) response to the GAO draft report, GAO-06-830, "Contract Management: Service Contract Approach to Aircraft Simulator Training Has Room for Improvement," dated August 9, 2006 (GAO Code 120492). We welcome your assistance in improving the management and acquisition of requirements for aircraft training. Our detailed comments regarding the recommendations on pages 29 and 30 of your draft report are enclosed.
For the record, I offer the following corrections:
a. The current F-16 Mission Training Center (MTC) simulation service contract is ending in June 2007 because the contractor failed to earn enough award term points to extend the period of performance; it is not due to financial viability of the contract (pp. 2-3).
b. The reduction in government system reviews are a result of contracting for this requirement as a commercial service, not due to the use of O&M funds (p. 10). The report accurately states that follow-on contracts will not use a commercial service strategy
We appreciate the opportunity to review and comment on your findings.
Signed by:
Shay D. Assad:
Director:
Defense Procurement and Acquisition Policy:
Enclosure:
As stated:
[End of letter]
Enclosure:
GAO Draft Report Dated August 9, 2006:
GAO-06-830 (GAO CODES 120492):
"Contract Management: Service Contract Approach To Aircraft Simulator Training Has Room For Improvement"
Department Of Defense Comments To The GAO Recommendation:
RECOMMENDATION 1: The GAO recommended that the Secretary of Defense direct the Secretaries of the Air Force and Army to conduct a thorough analysis of the costs and benefits of using service contracts for simulator training to determine if it is indeed the best approach to provide the war-fighter with needed training. (p. 29/GAO Draft Report)
DOD Response: Concur. The Air Force is already implementing this recommendation. It recently conducted a thorough analysis of the costs and benefits of using a service contract for the next F-16 Mission Training Center simulation contract. This type of analysis will be conducted for all future simulator training.
The Army's Flight School XXI (FSXXI) service contract requirements are based on an analysis of the training need and an evaluation of potential alternative training solutions.
Recommendation 2: The GAO recommended that the Secretary of the Air Force reconcile the funding level needed for simulator training with the requirements identified in the evaluation of costs and benefits of the service contract approach and take steps to allocate funds accordingly. (p. 29/GAO Draft Report)
DOD Response: Concur.
Recommendation 3: The GAO recommended that the Secretary of Defense direct the Secretaries of the Air Force and Army to determine whether it is in the government's best interest to retain the award-term incentive under these service contracts and take appropriate steps to improve the approach by reassessing the areas to be rated and the definitions of performance levels for the various grade categories. (p. 29/GAO Draft Report)
DOD Response: Concur.
Recommendation 4: The GAO recommended that the Secretary of Defense direct the Secretaries of the Air Force and Army to reassess the pricing of any up-front payments made to the contractors during the development period on future replacement or restructured contracts. (p. 29/GAO Draft Report)
DOD Response: Concur.
Recommendation 5: The GAO recommended that the Secretary of Defense direct the Secretaries of the Air Force and Army to (1) create an appropriate and transparent contract payment mechanism, separate from the preparatory tasks; and (2) increase visibility into the percentage of up-front development costs contractors are recouping from these preparatory tasks and development payments. (p. 30/GAO Draft Report)
DOD Response: Concur.
Recommendation 6: The GAO recommended that the Secretary of Defense direct the Secretaries of the Air Force and Army to determine whether and how simulator utilization can be increased to maximize use of taxpayer dollars. (p. 30/GAO Draft Report)
DOD Response: Concur. The Air Force will determine optimum simulator utilization to maximize value of taxpayer investment. The Army has a process in place to refine requirements and will continue to address those requirements routinely. See below, however, the response to GAO Recommendation 7 regarding the applicability of simulator utilization as an appropriate metric under the service contracting strategy.
Recommendation 7: The GAO recommended that the Secretary of Defense direct the Secretary of the Army to track and record monthly utilization rates on flight school XXI contracted simulators in order to have the data necessary to adjust training requirements and contract provisions, as necessary. (p. 30/GAO Draft Report)
DOD Response: Partially Concur. The GAO recommendation is erroneously based on the premise that the Army contracts for a particular number of simulators, whereas the FSXXI service contract approach requires only that the vendor meet the programmed student training load. The government tracks simulator availability on a monthly basis and the data is used to calculate the schedule of deductions that applies when simulator availability to meet specified training requirements drops below specified percentages. Nevertheless, the FSXXI contract requires that the contractor submit utilization data, as required by the Contract Data Requirements List, so this data is available should it become relevant to future adjustments in contracting strategy, requirements, or provisions.
Recommendation 8: The GAO recommended that the Secretary of Defense direct the Secretary of the Air Force to revise the Air Force Instruction 36-2248, Operation and Management of Aircrew Training Devices, to ensure that, for the purposes of reporting utilization rates, the usage of Individual training simulators is calculated. (p. 30/GAO Draft Report)
DOD Response: Concur. The Air Force will revise AFI 36-2248 to prescribe utilization rates using a single standard and in greater detail to accurately determine the training utilization of individual simulator systems.
Recommendation 9: The GAO recommended that the Secretary of Defense direct the Secretary of the Air Force to ensure that all sites consistently track and report simulator utilization. (p. 30/GAO Draft Report)
DOD Response: Concur. The revised AFI 36-2248 will also prescribe procedures to ensure consistent tracking and reporting of simulator utilization. It will standardize reporting across contracts for similar types of trainers.
[End of enclosure]
[End of section]
Appendix III: Comparison of Contractor Requirements under Commercial and Non-commercial Acquisition Procedures:
The following table shows differences, as set forth in the Federal Acquisition Regulation (FAR), for contractor requirements under commercial versus non-commercial acquisition procedures.
Requirement: Type of contract;
Under noncommercial procedures: A wide selection of contract types is available in order to provide flexibility. (FAR 16.101(a)).
Under commercial procedures: Limited contract types are authorized. Agencies shall use firm-fixed-price contracts or fixed-price contracts with economic price adjustment. These contract types may be used in conjunction with an award fee and performance or delivery incentives when the award fee or incentive is based solely on factors other than cost. (FAR 12.207) To implement the Services Acquisition Reform Act of 2003 (contained in Section 1432 of the National Defense Authorization Act for Fiscal Year 2004, Pub. L. No. 108-136 (2003)), a proposed amendment to FAR would expressly authorize the use of time-and-materials and labor-hour contracts for certain categories of commercial services under specified conditions. (FAR Case 2003-027, 70 Federal Register 56318, Sept. 26, 2005.)
Requirement: Inspection and testing;
Under noncommercial procedures: Government has right to inspect and test. (FAR 46.102 & 46.202-3);
Under commercial procedures: Contracts for commercial items shall rely on contractors‘ existing quality assurance systems as a substitute for Government inspection and testing before tender for acceptance unless customary market practices for the commercial item being acquired include in-process inspection. Any in-process inspection by the Government shall be conducted in a manner consistent with commercial practice. (FAR 12.208).
Requirement: Determination of price reasonableness;
Under noncommercial procedures: Price must be determined fair and reasonable through various proposal analysis techniques. (FAR 15.404-1);
Under commercial procedures: While price reasonableness must be established, the contracting officer should be aware of customary commercial terms and conditions when pricing commercial items. Commercial item prices are affected by factors that include, but are not limited to, speed of delivery, length and extent of warranty, limitations of seller‘s liability, quantities ordered, length of the performance period, and specific performance requirements. (FAR 12.209).
Requirement: Cost or pricing data (Requirements based on Truth in Negotiations Act, 10 U.S.C. 2306a; 41 USC 254b);
Under noncommercial procedures: Required for contract award and modifications unless applicable exception, such as adequate competition or prices agreed upon are based on prices set by law or regulation. Threshold for application is $550,000. (FAR 15.403-1 and -4);
Under commercial procedures: Commercial items are exempt (FAR 15.403-1(b)(3) and (c)(3)).
Requirement: Contract financing;
Under noncommercial procedures: The contracting officer must consider the following order of preference when a contractor requests contract financing., (a) Private financing without Government guarantee.(b) Customary contract financing. (c) Loan guarantees. (d) Unusual contract financing. (e) advance payments. (FAR 32.106). However, government financing provided only to extent actually needed for prompt and efficient performance, considering availability of private financing and probable impact on working capital of predelivery expenditures and productions lead-times. (FAR 32.104).
Under commercial procedures: For purchases of commercial items, financing of the contract is normally the contractor‘s responsibility. (32.202-1) However, customary market practice for some commercial items may include buyer contract financing. In these circumstances, the contracting officer may offer Government financing in accordance with the policies and procedures in Part 32. (FAR 12.210). Government financing of commercial purchases is expected to be different from that used for non-commercial purchases. While the contracting officer may adapt non-commercial techniques and procedures for use in implementing commercial contract financing arrangements, the contracting officer must have a full understanding of effects of the differing contract environments and of what is needed to protect the interests of the Government in commercial contract financing. (FAR 32.202-1(c)) Types of payments for commercial item purchases. (FAR 32.202-2):
1. Commercial advance payment: payment made before any performance of work (not to exceed 15 percent of contract price);
2. Commercial interim payment: payment made after some, but not all, work has been performed;
3. Delivery payment: payment made for accepted supplies or services, including partial deliveries (FAR 32.001).
Requirement: Technical data;
Under noncommercial procedures: The Government may acquire technical data and rights in technical data for multiple purposes. Agencies shall strike a balance between the government‘s need and the contractor‘s legitimate proprietary interest. (FAR 27.4);
Under commercial procedures: Generally, the Government shall acquire only the technical data and the rights in that data customarily provided to the public with a commercial item or process. The contracting officer shall presume that data delivered under a contract for commercial items was developed exclusively at private expense. When a contract for commercial items requires delivery of technical data, the contracting officer shall include appropriate provisions and clauses delineating the rights in the technical data in the contract. (FAR 12.211).
Requirement: Computer software/documentation;
Under noncommercial procedures: The Government may acquire computer software/documentation for multiple purposes. Agencies shall strike a balance between the government‘s need and the contractor‘s legitimate proprietary interest. (FAR 27.402);
Under commercial procedures: Commercial computer software or commercial computer software documentation shall be acquired under licenses customarily provided to the public to the extent such licenses are consistent with federal law and otherwise satisfy the government‘s needs. Generally, offerors and contractors shall not be required to:
1. Furnish technical information related to commercial computer software or commercial computer software documentation that is not customarily provided to the public; or;
2. Relinquish to, or otherwise provide, the Government rights to use, modify, reproduce, release, perform, display, or disclose commercial computer software or commercial computer software documentation except as mutually agreed to by the parties. (FAR 12.212(a)).
Requirement: Cost accounting standards, 41 U.S.C. 422;
Under noncommercial procedures: Compliance generally required for contractors in connection with negotiated contracts in excess of $500,000. Contractors must disclose and consistently follow their cost accounting practices. (FAR 30.101);
Under commercial procedures: Cost Accounting Standards do not apply to contracts for acquisition of commercial items when they are firm-fixed-price or fixed-price with economic price adjustment. (FAR 12.214).
Requirement: Preaward Survey;
Under noncommercial procedures: In determining whether a potential awardee is a responsible contractor, per criteria in FAR 9.104-1, contracting officers may require a preaward survey when the information on hand or readily available is not sufficient to make such a determination. (FAR 9.106-1);
Under commercial procedures: If the contemplated contract will involve the acquisition of commercial items, the contracting officer should not request a preaward survey unless circumstances justify its cost. (FAR 9.106-1(a)).
Requirement: Audit Rights;
Under noncommercial procedures: When contracting by negotiation, the contracting officer shall insert the clause at FAR 52.215-2, Audit and Records”Negotiation in solicitations and contracts which allows contracting officer examination of costs when cost or pricing data is required or for cost-reimbursement, incentive, time-and-materials, labor-hour, or price redeterminable contracts. (FAR 15.209(b) and 52.215-2);
Under commercial procedures: Commercial item contracts exempted. (FAR 15.209(b)(1)(iii)).
Requirement: Applicability of certain laws: Walsh-Healy Public Contracts Act;
Under noncommercial procedures: On contracts for supplies over $10,000, contractors must adhere to provisions pertaining to minimum wages, maximum hours, child labor, convict labor, safe/sanitary working conditions. (FAR 22.602);
Under commercial procedures: Not applicable. (FAR 12.503(a)).
Requirement: Contingent fees;
Under noncommercial procedures: Contractor must warrant that it has not employed or retained anyone, on a contingent fee basis, to obtain this contract. (FAR 3.404, 52.203-5);
Under commercial procedures: Not applicable. (FAR 12.503(a)).
Requirement: Drug-Free Workplace Act of 1988;
Under noncommercial procedures: Contractor must agree that it will provide a drug-free workplace (FAR 23.504(a));
Under commercial procedures: Not applicable. (FAR 12.503(a)).
Requirement: Vietnam Era Veterans‘ Readjustment Assistance Act of 1972;
Under noncommercial procedures: Contractor must report on its affirmative actions to employ and advance covered veterans (FAR 22.1302(a));
Under commercial procedures: Law‘s limitation on use of appropriated funds for contracts with entities not meeting veterans employment reporting requirements is not applicable. (FAR 12.503(a)).
Requirement: Trafficking Victims Protection Reauthorization Act of 2003;
Under noncommercial procedures: Contracts for services must prohibit contractor activities regarding, and require contractor policies to combat, severe forms of trafficking in persons, the procurement of commercial sex acts, and use of forced labor. (FAR 22.1705);
Under commercial procedures: Not applicable. (FAR 12.503(a)).
Requirement: Contract Work Hours and Safety Standards Act;
Under noncommercial procedures: Contract clause required providing that contractors employing laborers or mechanics are required to compensate them for overtime. (FAR 52.222-4);
Under commercial procedures: Requirements for a certificate and contract clause related to the Act are not applicable. (FAR 12.503(b)).
Requirement: Anti-Kickback Act of 1986;
Under noncommercial procedures: Contract clause requires prime contractors to (1) have in place and follow reasonable procedures designed to prevent and detect violations of the Act; and (2) cooperate fully with any Federal agency investigating a possible violation of the Act. (FAR 3.502-2(i));
Under commercial procedures: Requirements for a clause and certain other requirements related to the Act are not applicable. (FAR 12.503(b)).
Requirement: International Air Transportation Fair Competitive Practices Act of 1974 (Fly America Act);
Under noncommercial procedures: Contracts must include clause requiring use of U.S-Flag Air Carriers by government contractors when available (FAR 47.405);
Under commercial procedures: Requirement for a clause related to the Act is not applicable. (FAR 12.503(b)).
Requirement: Prohibition on Limiting Subcontractor Direct Sales to the United States;
Under noncommercial procedures: Contracts must include clause precluding contractors from restricting direct subcontractor sales to the Government. (FAR 3.503-2 and 52.203-6(a));
Under commercial procedures: Contractors may restrict subcontractors‘ sales to the Government, as long as the Government is treated no differently than other prospective purchaser. (FAR 52.203-6, Alternate I).
Requirement: Selected contract administration provisions: Changes;
Under noncommercial procedures: Generally, contracting officer permitted to make unilateral changes within the scope of the contract and to require continued contractor performance of the contract as changed. (FAR 43.201);
Under commercial procedures: Changes may be made only by written agreement of the parties (bilateral). (FAR 12.301(b)(3); 52.212-4(c)).
Requirement: Termination for the convenience of the government;
Under noncommercial procedures: Generally, termination costs for fixed-price contracts limited to total contract price less payments made or to be made under contract plus reasonable costs incurred in performance of work terminated, to include fair and reasonable profit, and reasonable settlement costs. Cost principles and procedures of FAR Part 31 apply to costs. (FAR 49.502(b); 52.249-2);
Under commercial procedures: Termination costs limited to percentage of contract price reflecting percentage of work performed prior to termination plus reasonable charges resulting from termination. For payments thereunder, contractor not required to comply with cost accounting standards or contract cost principles in FAR Part 31. (FAR 12.301(b)(3); 52.212-4(l)).
Source: GAO analysis of Federal Acquisition Regulation requirements.
[End of table]
[End of section]
Appendix IV: Potential Duration of Air Force and Army Simulation Contracts:
System: Air Force, F-15C;
Base performance period: Nov. 1997 to Nov. 2005 (8 years);
Additional years possible via award term: 7 years;
Potential total years of contract performance: 15 years.
System: Air Force, AWACS;
Base performance period: Feb.1999 to Feb. 2006 (7 years)
Additional years possible via award term: 8 years;
Potential total years of contract performance: 15 years.
System: Air Force, F-16;
Base performance period: June 1999 to June 2006 (7 years);
Additional years possible via award term: 8 years;
Potential total years of contract performance: 15 years.
System: Air Force, F-15E;
Base performance period: Aug. 2003 to July 2016 (13 years);
Additional years possible via award term: not applicable;
Potential total years of contract performance: 13 years.
System: Army, Flight School XXI;
Base performance period: Sept. 2003 to Mar. 2015 (11.5 years);
Additional years possible via award term: 8 years;
Potential total years of contract performance: 19.5 years.
Source: GAO analysis of contract data.
[End of table]
[End of section]
Appendix V: GAO Contact and Staff Acknowledgments:
GAO contact:
Lisa Shames (202) 512-4841 or shamesl@gao.gov:
Acknowledgments:
In addition to the individual named above, Michele Mackin, Assistant Director; Marie Ahearn; Christine Bonham; Gary Delaney; Carlos Diz; Benjamin Federlein; Victoria Klepacz; and Sanford Reigle made key contributions to this report.
[End of section]
Footnotes:
[1] Award-term incentives are similar to award-fee incentives, but the contractor is rewarded for excellent performance with an extension of the contract period instead of additional fee.
[2] Pub. L. No. 107-107 (2001) codified at 10 U.S.C. 2330.
[3] Implemented by Defense Federal Acquisition Regulation Supplement 237.170.
[4] GAO, Contract Management: High-Level Attention Needed to Transform DOD Services Acquisition, GAO-03-935 (Washington, D.C.: Sept. 10, 2003).
[5] Required system elements include instructor stations, pilot cockpits, manned combat stations, local and long haul networks, and observation rooms.
[6] GAO, 21st Century Challenges: Reexamining the Base of the Federal Government, GAO-05-325SP (Washington, D.C.: February 2005).
[7] GAO, Contract Security Guards: Army‘s Guard Program Requires Greater Oversight and Reassessment of Acquisition Approach, GAO-06-284 (Washington, D.C.: Apr. 3, 2006); and Hurricane Katrina: Improving Federal Contracting Practices in Disaster Recovery Operations, GAO-06-714T (Washington, D.C.: May 4, 2006).
[8] Pub. L. No. 103-355, Title VIII (1994).
[9] Pub. L. No. 104-106, Div. D (1996), formerly Federal Acquisition Reform Act of 1996 and renamed in Treasury, Postal Service and General Government Appropriations Act, 1997, Pub. L. No. 104-208, Sec. 808 (1996) (contained in Omnibus Consolidation Appropriations Act, 1997, Pub. L. No. 104-208, (1996)).
[10] Turnkey contracts are those under which the contractor provides all needed supplies, services, equipment, facilities, etc., to produce the desired product or service, so that the customer has only to ’turn the key“ to avail himself of the result.
[11] This directive was canceled in 2005.
[12] The contractor claimed that it had incurred $73.3 million in non-recurring costs and that it had only recovered $31.7 million of these costs through payment for preparatory services. According to a March 2006 DOD Inspector General report, an improper contract restructure directed by the Air Force Principal Deputy Assistant Secretary for Acquisition and Management committed the Air Force to pay the remaining $41.6 million in claimed non-recurring costs even though the government received minimal value.
[13] The primary mission of F-16 block 50 aircraft is the destruction and suppression of enemy air defenses. Block 40 aircraft have the primary mission of filling the air-to surface attack role.
[14] Air Force Instruction 36-2248, Operation and Management of Aircrew Training Devices, provides that training units calculate the utilization rate.
[15] Air Force Audit Agency, Flight Simulator Utilization, 20th Fighter Wing, Shaw AFB, SC; F2004-0051-FDM0000 (Mid-Atlantic Area Audit Office, May 5, 2004).
[16] Air Force Audit Agency, Distributed Mission Operations, 52d Fighter Wing, Spangdahlem AB, Germany; F2006-0018-FDE000 (European Area Audit Office, Jan. 25, 2006).
[17] Operational availability is the total time of effective training service divided by the total time of scheduled simulation services. The training service completion rate is the number of completed training periods divided by the number of required training periods.
[18] GAO, Defense Acquisitions: DOD Has Paid Billions in Award and Incentive Fees Regardless of Acquisition Outcomes, GAO-06-66 (Washington, D.C.: Dec. 19, 2005).
[19] Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005, Pub. L. No.108-375, Sec. 813 (2004) limits the total contract period for multiyear task and delivery order contracts, as extended pursuant to options, to 10 years unless the head of the agency determines in writing that exceptional circumstances necessitate a longer contract period.
[20] GAO-06-66.
[21] GAO, Defense Acquisitions: Improved Business Case Is Needed for Future Combat System‘s Successful Outcome, GAO-06-367 (Washington, D.C.: Mar. 14, 2006).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday,
GAO posts newly released reports, testimony, and correspondence on its Web site.
To have GAO e-mail you a list of newly posted products every afternoon,
go to [hyperlink, http://www.gao.gov] and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Paul Anderson, Managing Director, AndersonP1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
