Defense Infrastructure
Further Actions Needed to Support Air Force Electronic Warfare Evaluation Simulator Relocation Plans Gao ID: GAO-11-123R January 26, 2011According to Department of Defense (DOD) officials, the Air Force Electronic Warfare Evaluation Simulator (the Simulator), located at Air Force Plant in Fort Worth, Texas, is an important asset for helping to protect U.S. and allied pilots and aircraft against the missile threats posed by adversaries. Most missiles use one of two electronic warfare technologies in order to pursue aircraft in flight and deliver an explosive warhead with the intent to inflict maximum damage. Small shoulder-launched missiles generally use infrared seekers that search for heat sources on an aircraft, while more sophisticated air-to-air and larger surface-to-air missiles can use radio waves and infrared seekers to determine an aircraft's location in flight. DOD continually develops and tests countermeasures to protect U.S. and allied aircraft from both types of missile threats. The Air Force Electronic Warfare Evaluation Simulator at Plant 4 is one of only two Air Force facilities of its kind that test countermeasures against heat-seeking missiles, and it is the only Air Force facility that currently houses the equipment necessary to test countermeasures against more sophisticated radio frequency surface-to-air missiles. The Simulator uses an array of computer hardware and software and other equipment to simulate the firing of a missile under various conditions and scenarios, precluding the need to actually fire and destroy a missile in the process. Conducting such tests provides DOD, the Department of Homeland Security, and allied governments with the necessary data to develop various countermeasures for use by military and commercial aircraft. The Test Resource Management Center within the Office of the Secretary of Defense (OSD) is tasked with reviewing Air Force and other services' proposals to change the test and evaluation infrastructure in accordance with OSD guidance and congressional direction.5 In a July 8, 2009, report, the center provisionally approved the Air Force's relocation proposal and submitted the report to congressional defense committees in response to congressional direction. Subsequently, on July 24, 2009, the House Appropriations Committee expressed concern about DOD's proposed relocation of the Air Force Electronic Warfare Evaluation Simulator. House Report 111-230 directed that funds shall not be obligated or expended to relocate the Simulator until a comprehensive cost-benefit analysis, reviewed by GAO, is provided to the congressional defense committees. Furthermore, the House report, noting that the Simulator's specialized test capabilities are a vital element of our national defense posture, directed that the study's findings should demonstrate the technical merits of any proposed relocation. In August 2009, the Test Resource Management Center submitted OSD's July 2009 report to us in response to the congressional direction in House Report 111-230 and, pending our review, has not submitted that report to the congressional defense committees. Our objectives for this review were to determine (1) to what extent OSD's report on the proposed relocation of the Air Force Electronic Warfare Evaluation Simulator includes a comprehensive cost-benefit analysis and (2) to what extent OSD has addressed the technical issues involved in the proposed relocation.
In its July 2009 report, OSD provided some limited cost and benefit information but did not include all expected costs and benefits associated with the proposal to relocate the Air Force Electronic Warfare Evaluation Simulator, and therefore the report does not constitute a comprehensive cost-benefit analysis. Since the report was issued, the Air Force provided us with additional estimated cost and benefit information intended to better support its relocation rationale. OSD's report identified the annual cost to operate, maintain, and modernize the Air Force Electronic Warfare Evaluation Simulator, but did not specifically identify any estimated onetime transition costs or other costs that may be associated with the relocation. Air Force officials subsequently provided us with an estimated total onetime transition cost of approximately $7 million that was not included in OSD's report. Additionally, during the course of our review, the Air Force identified other potential costs not included in this $7 million transition cost estimate or OSD's report, such as an estimated $3 million cost to temporarily retain Lockheed Martin personnel and an additional $200,000 to $300,000 cost to transfer its infrared test equipment to Eglin. OSD's report also identified the expected benefit of an annual recurring savings ranging from $2.8 million to $4.4 million, but did not include a detailed methodology supporting this estimate. The Air Force subsequently provided us with supporting information for this estimate as well as descriptions of the additional benefits expected. Nevertheless, the Air Force did not follow relevant guidance or best practices for completing a cost-benefit analysis. Air Force economic analysis guidance includes detailed procedures and a checklist for conducting such an analysis and obtaining a certification to ensure the reliability of cost estimates. OMB guidance also identifies several elements that should be included to promote independent analysis, and GAO-identified best practices for developing cost-benefit analyses include steps such as obtaining an independent cost estimate. Air Force officials told us they felt that the information they submitted was sufficient and indicated that conducting the kind of costbenefit analysis described in Air Force guidance would have been cost prohibitive. While Air Force guidance permits officials to secure a waiver under those and other circumstances, the Air Force office developing the cost-benefit analysis did not request such a waiver or provide us with documentation supporting its rationale that conducting such an analysis would have been cost prohibitive. Ultimately, by not following relevant guidance for conducting a comprehensive cost-benefit analysis, the Air Force may lack reasonable assurance that its proposal includes all the potential costs, benefits, and impacts associated with its relocation proposal and the proposal may also not sufficiently satisfy congressional direction to provide a comprehensive cost-benefit analysis. We are making two recommendations to improve DOD's proposal to relocate the Air Force Electronic Warfare Evaluation Simulator. First, we recommend that OSD, in consultation with the Air Force, revise the July 2009 cost-benefit analysis to adhere to internal Air Force guidance and identify all costs and benefits associated with the relocation proposal and submit it to the congressional defense committees. Second, we recommend that the Air Force finalize a transition plan that includes steps for staffing and training personnel to operate and maintain the relocated Air Force Electronic Warfare Evaluation Simulator capabilities and submit that plan to the congressional defense committees as well.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Brian J. Lepore Team: Government Accountability Office: Defense Capabilities and Management Phone: (202) 512-4523GAO-11-123R, Defense Infrastructure: Further Actions Needed to Support Air Force Electronic Warfare Evaluation Simulator Relocation Plans
This is the accessible text file for GAO report number GAO-11-123R
entitled 'Defense Infrastructure: Further Actions Needed to Support
Air Force Electronic Warfare Evaluation Simulator Relocation Plans'
which was released on January 26, 2011.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
GAO-11-123R:
United States Government Accountability Office:
Washington, DC 20548:
January 26, 2011:
The Honorable Thad Cochran:
The Honorable Daniel K. Inouye:
The Honorable Carl Levin:
The Honorable John McCain:
United States Senate:
The Honorable C.W. Bill Young:
Chairman:
The Honorable Norman D. Dicks:
Ranking Member:
Committee on Appropriations:
House of Representatives:
The Honorable Howard P. McKeon:
Chairman:
The Honorable Adam Smith:
Ranking Member:
Committee on Armed Services:
House of Representatives:
Subject: Defense Infrastructure: Further Actions Needed to Support Air
Force Electronic Warfare Evaluation Simulator Relocation Plans:
According to Department of Defense (DOD) officials, the Air Force
Electronic Warfare Evaluation Simulator (the Simulator), located at
Air Force Plant 4 in Fort Worth, Texas, is an important asset for
helping to protect U.S. and allied pilots and aircraft against the
missile threats posed by adversaries. Most missiles use one of two
electronic warfare technologies in order to pursue aircraft in flight
and deliver an explosive warhead with the intent to inflict maximum
damage. Small shoulder-launched missiles generally use infrared
seekers that search for heat sources on an aircraft,[Footnote 1] while
more sophisticated air-to-air and larger surface-to-air missiles can
use radio waves and infrared seekers to determine an aircraft's
location in flight.[Footnote 2] DOD continually develops and tests
countermeasures to protect U.S. and allied aircraft from both types of
missile threats. The Air Force Electronic Warfare Evaluation Simulator
at Plant 4 is one of only two Air Force facilities of its kind that
test countermeasures against heat-seeking missiles, and it is the only
Air Force facility that currently houses the equipment necessary to
test countermeasures against more sophisticated radio frequency
surface-to-air missiles. The Simulator uses an array of computer
hardware and software and other equipment to simulate the firing of a
missile under various conditions and scenarios, precluding the need to
actually fire and destroy a missile in the process. Conducting such
tests provides DOD, the Department of Homeland Security, and allied
governments with the necessary data to develop various countermeasures
for use by military and commercial aircraft.
Although the Air Force Electronic Warfare Evaluation Simulator
equipment and facility are government owned and managed, they have
been operated by Lockheed Martin since 1958. However, as part of a
larger Air Force initiative to consolidate and streamline test and
evaluation activities, the Air Force is planning to terminate the
Simulator contractor operations in Texas; relocate its six radio
frequency threat simulators to Wright-Patterson Air Force Base (Wright-
Patterson), Ohio; and redirect infrared test work to Eglin Air Force
Base (Eglin), Florida.[Footnote 3] Both proposed receiving locations
currently have other government-operated electronic warfare test
facilities. The Air Force considers the infrared capabilities at Plant
4 in Texas and at the Eglin facility redundant, and Eglin's facility
has been utilized at a higher rate than the Simulator at Plant 4 over
the last several years. In light of this lower utilization at Plant 4,
the Air Force has placed the Simulator's infrared test equipment in on-
site storage. It does not consider moving the Simulator's infrared
test configuration essential to current mission needs, but would
prefer to do so to potentially reuse this test equipment at Eglin.
According to Air Force officials, this relocation proposal would
reduce costs; result in critical technical advantages, such as higher-
fidelity testing; and provide them with more operational control over
these testing and evaluation assets. Air Force officials added that a
temporary lull in the Simulator's infrared and radio frequency
workloads over the last several years and expected lower-than-average
customer demand over the next 2 years as new weapons systems and
countermeasures are developed make this an opportune time for
relocation.
The Test Resource Management Center within the Office of the Secretary
of Defense (OSD) is tasked with reviewing Air Force and other
services' proposals to change the test and evaluation infrastructure
in accordance with OSD guidance[Footnote 4] and congressional
direction.[Footnote 5] In a July 8, 2009, report, the center
provisionally approved the Air Force's relocation proposal and
submitted the report to congressional defense committees in response
to congressional direction. Subsequently, on July 24, 2009, the House
Appropriations Committee expressed concern about DOD's proposed
relocation of the Air Force Electronic Warfare Evaluation Simulator.
House Report 111-230 directed that funds shall not be obligated or
expended to relocate the Simulator until a comprehensive cost-benefit
analysis, reviewed by GAO, is provided to the congressional defense
committees. Furthermore, the House report, noting that the Simulator's
specialized test capabilities are a vital element of our national
defense posture, directed that the study's findings should demonstrate
the technical merits of any proposed relocation. In August 2009, the
Test Resource Management Center submitted OSD's July 2009 report to us
in response to the congressional direction in House Report 111-230
and, pending our review, has not submitted that report to the
congressional defense committees. Our objectives for this review were
to determine (1) to what extent OSD's report on the proposed
relocation of the Air Force Electronic Warfare Evaluation Simulator
includes a comprehensive cost-benefit analysis and (2) to what extent
OSD has addressed the technical issues involved in the proposed
relocation.
Scope and Methodology:
To determine the extent to which OSD's report on the proposed
relocation of the Air Force Electronic Warfare Evaluation Simulator
includes a comprehensive cost-benefit analysis, we reviewed the report
and met with officials from Air Force Headquarters and OSD's Test
Resource Management Center to discuss the rationale for the proposed
relocation. We analyzed cost and benefit data originally included in
OSD's report and analyzed additional cost and benefit data
subsequently provided to us by the Air Force that were not included in
OSD's report. In addition, because the Air Force provided all of the
cost and benefit data used to support the rationale to relocate the
Simulator; we reviewed Air Force cost-benefit and cost-estimating
guidance to determine whether the Air Force followed its own
procedures in developing its analysis. In addition, we compared the
Air Force's analysis to Office of Management and Budget (OMB) cost-
benefit analysis guidance[Footnote 6] and best practices identified by
GAO. Furthermore, we interviewed Lockheed Martin personnel responsible
for operating the Simulator to obtain their views on estimated costs
and benefits associated with the Air Force's relocation proposal.
Since House Report 111-230 referenced an earlier proposal to relocate
the Simulator that was part of the 1995 Base Realignment and Closure
(BRAC) deliberations, we reviewed BRAC documents to determine if
similar issues regarding cost and savings estimates identified in
those documents were applicable to the current relocation proposal.
To determine the extent to which OSD's report addressed technical
issues associated with the relocation proposal, we met with Test
Resource Management Center and Air Force officials to discuss the
primary technical issues identified in the written report. As part of
our analysis we also conducted fieldwork at Wright-Patterson in Ohio,
Eglin in Florida, and Plant 4 in Fort Worth, Texas, to determine what
technical limitations or facility constraints, if any, currently exist
at each location. We met with employees and test customers at each
location to obtain an overview of the capabilities of each facility
and observed testing demonstrations. Additionally, we met with
Lockheed Martin personnel to obtain their perspective on the potential
technical issues that they felt may affect the successful relocation
of the Simulator. We also obtained and analyzed information regarding
the experience levels and technical core competencies of engineer
personnel expected to operate and maintain the Simulator.
We conducted this performance audit from April 2010 to January 2011 in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.
Summary:
In its July 2009 report, OSD provided some limited cost and benefit
information but did not include all expected costs and benefits
associated with the proposal to relocate the Air Force Electronic
Warfare Evaluation Simulator, and therefore the report does not
constitute a comprehensive cost-benefit analysis. Since the report was
issued, the Air Force provided us with additional estimated cost and
benefit information intended to better support its relocation
rationale. OSD's report identified the annual cost to operate,
maintain, and modernize the Air Force Electronic Warfare Evaluation
Simulator, but did not specifically identify any estimated onetime
transition costs or other costs that may be associated with the
relocation. Air Force officials subsequently provided us with an
estimated total onetime transition cost of approximately $7 million
that was not included in OSD's report. Additionally, during the course
of our review, the Air Force identified other potential costs not
included in this $7 million transition cost estimate or OSD's report,
such as an estimated $3 million cost to temporarily retain Lockheed
Martin personnel and an additional $200,000 to $300,000 cost to
transfer its infrared test equipment to Eglin. OSD's report also
identified the expected benefit of an annual recurring savings ranging
from $2.8 million to $4.4 million, but did not include a detailed
methodology supporting this estimate. The Air Force subsequently
provided us with supporting information for this estimate as well as
descriptions of the additional benefits expected. Nevertheless, the
Air Force did not follow relevant guidance or best practices for
completing a cost-benefit analysis. For example, the Air Force did not
have its analysis independently reviewed and certified by installation-
or major command-level comptroller offices as specified by its
guidance. Air Force economic analysis guidance includes detailed
procedures and a checklist for conducting such an analysis and
obtaining a certification to ensure the reliability of cost estimates.
OMB guidance also identifies several elements that should be included
to promote independent analysis, and GAO-identified best practices for
developing cost-benefit analyses include steps such as obtaining an
independent cost estimate. Air Force officials told us they felt that
the information they submitted was sufficient and indicated that
conducting the kind of cost-benefit analysis described in Air Force
guidance would have been cost prohibitive. While Air Force guidance
permits officials to secure a waiver under those and other
circumstances, the Air Force office developing the cost-benefit
analysis did not request such a waiver or provide us with
documentation supporting its rationale that conducting such an
analysis would have been cost prohibitive. Additionally, securing such
a waiver would not have been responsive to congressional direction to
conduct a comprehensive cost-benefit analysis. Ultimately, by not
following relevant guidance for conducting a comprehensive cost-
benefit analysis, the Air Force may lack reasonable assurance that its
proposal includes all the potential costs, benefits, and impacts
associated with its relocation proposal and the proposal may also not
sufficiently satisfy congressional direction to provide a
comprehensive cost-benefit analysis.
The Air Force has addressed one of the two technical considerations
identified in OSD's relocation report, but some issues still exist
regarding plans to train personnel at Wright-Patterson to ensure that
they can operate and maintain the Air Force Electronic Warfare
Evaluation Simulator's unique surface-to-air missile simulations. In
its report that conditionally approved the relocation, OSD conducted a
technical comparison between the capabilities of the current location
of the Simulator and the proposed receiving locations and identified
two primary technical issues that could have an impact on a successful
relocation. First, OSD recommended that the Air Force demonstrate that
Eglin's Guided Weapons Evaluation Facility provides high-intensity
infrared engine and flare sources comparable to the current Air Force
Electronic Warfare Evaluation Simulator capability, which Eglin has
since demonstrated. Second, OSD noted that the relocation of the radio
frequency capability posed a moderate risk because no resident surface-
to-air threat expertise existed at Wright-Patterson at the time of
OSD's review. OSD recommended that the Air Force address this issue
before moving forward with the relocation, but we found that the Air
Force's plan for training and maintaining personnel with the needed
expertise was still in development as of November 2010, and no Air
Force personnel had begun receiving hands-on training to operate the
Simulator.
Consistent with OSD's recommendation, our prior work on sound
transition planning states that agencies should ensure that personnel
with the right skills are in place to support a transition effort.
Although Air Force officials have expressed confidence that sufficient
technical expertise currently exists at Wright-Patterson to operate
the Simulator's radio frequency capability, the Air Force has not
trained or finalized its plans to train Air Force personnel to
specifically operate and maintain these one-of-a-kind surface-to-air
missile simulations. According to both Lockheed Martin and Air Force
officials, it would be ideal for Lockheed Martin personnel to assist
with the proposed transition and help ensure that Air Force personnel
are trained to operate and maintain the Simulator. However, no
transition plan or agreement with Lockheed Martin has been finalized
to document how the Air Force plans to ensure that there are
adequately trained personnel located at Wright-Patterson if the
Simulator is relocated. Until the Air Force finalizes a plan that
ensures continuity of operations, including training or maintaining
personnel with the specific hands-on experience of operating the
Simulator's unique surface-to-air radio frequency capability, DOD may
continue to face risks that the Air Force Electronic Warfare
Evaluation Simulator's capabilities designed to protect U.S. and
allied aircraft may not be fully operational within the planned
transition time frame.
We are making two recommendations to improve DOD's proposal to
relocate the Air Force Electronic Warfare Evaluation Simulator. First,
we recommend that OSD, in consultation with the Air Force, revise the
July 2009 cost-benefit analysis to adhere to internal Air Force
guidance and identify all costs and benefits associated with the
relocation proposal and submit it to the congressional defense
committees. Second, we recommend that the Air Force finalize a
transition plan that includes steps for staffing and training
personnel to operate and maintain the relocated Air Force Electronic
Warfare Evaluation Simulator capabilities and submit that plan to the
congressional defense committees as well. In written comments on a
draft of this report, DOD stated that it concurred with our
recommendations and plans to revise its cost-benefit analysis and
finalize a transition plan that includes steps for staffing and
training personnel to operate and maintain relocated Simulator
capabilities at Wright-Patterson. DOD also stated that it plans to
submit its revised cost-benefit analysis and its finalized transition
plan to the congressional defense committees within 90 days after
publication of our report.
Background:
Established and operated by Lockheed Martin since 1958, the Air Force
Electronic Warfare Evaluation Simulator is located in a secure, 42,000-
square-foot government-owned and contractor-operated test and
evaluation facility in Fort Worth, Texas. The Air Force uses the
Simulator to develop and operate high-fidelity simulations of infrared
and radio frequency missiles to evaluate the effectiveness of DOD and
allied airborne electronic warfare systems. Test facilities of this
kind use computer modeling and flight motion tables to simulate the
firing of a missile under variable conditions and scenarios,
precluding the need to actually fire a missile and destroy it in the
process. These simulations allow the Air Force to test and evaluate
various countermeasures designed to protect its aircraft from enemy
missiles. DOD recommended relocating the Simulator during the 1995
BRAC round, but the BRAC Commission rejected the recommendation
because of the estimated costs and technical risks associated with
that relocation. Figure 1 provides a timeline of key events related to
the Simulator from 1995 through 2010.
Figure 1: Timeline of Air Force Electronic Warfare Evaluation
Simulator Key Events:
[Refer to PDF for image: timeline]
1995:
BRAC Commission rejects DOD recommendation to relocate the Simulator
to Edwards Air Force Base, citing that it was not cost effective and
posed a technical risk.
1997:
GAO reports that the cost estimates DOD provided to the BRAC
Commission were understated and there was no evidence of cost savings
in the Simulator proposal.
1998:
DOD realigns Simulator capabilities, leaving much of the facility and
equipment unused.
2000:
Congress encourages Air Force to submit a comprehensive cost-benefit
analysis before relocating the Simulator, but Air Force subsequently
determines that there is no overriding reason to relocate and does not
prepare one.
January 2008:
Test Resource Management Center must approve any changes to test
facilities.
December 2008:
Electronic Warfare Test & Evaluation Consortium is formed to reduce
duplication and increase efficiencies throughout test facilities.
May 2009: Electronic Warfare Test & Evaluation Consortium compares
facilities and recommends that Air Force not renew contract to Operate
the Simulator and instead redirect work and relocate capabilities to
Eglin and Wright-Patterson Air Force bases.
May - July 2009:
Test Resource Management Center reviews Air Force proposal and
conditionally approves the relocation of the Simulator.
July 8, 2009:
Test Resource Management Center submits its report to Congress in
response to Senate Report 110-77 requiring it to review proposed
changes to test and evaluation infrastructure.
July 24, 2009: The mandate directing GAO to review DOD's comprehensive
cost-benefit analysis is included in House Report 111-230.
December 31, 2010: The Simulator contract extension expires.
December 31, 2009:
The Simulator contract was scheduled to expire on this date, but was
extended.
December 31, 2010:
The Simulator contract extension expires.
Source: GAO analysis of legislative report, DOD, and BRAC data.
[End of figure]
As part of the 1995 BRAC deliberations, DOD recommended in its report
to the BRAC Commission the disestablishment of the Air Force
Electronic Warfare Evaluation Simulator and relocation of its
essential capabilities to Edwards Air Force Base, California. In its
rationale for this recommended action, the Air Force cited the
Simulator's low rate of utilization. However, the BRAC Commission
found that the Simulator's disestablishment was not cost effective.
The Air Force estimated a cost to transition the Simulator of $8.9
million and a return on investment in 13 years, whereas the commission
estimated a $34.9 million transition cost, which would result in a
payback in excess of 100 years. The BRAC Commission also found that
relocating electronic combat testing capabilities posed a major
technical risk, so it recommended that the facility remain open.
Subsequently, we reported that the cost estimates DOD presented to the
BRAC Commission were understated and that there was no evidence of
savings in the proposal to relocate the Simulator.[Footnote 7]
Conditions have changed significantly since all of the 20 simulations
at Plant 4 that were considered by the BRAC 1995 Commission were
decommissioned as a result of technical obsolescence. The Air Force
decommissioned these simulations in 1998 as a result of technological
advancements and regular test and evaluation facility reviews. Air
Force officials told us that these simulations were proven to be
unreliable, and as a result the Air Force Electronic Warfare
Evaluation Simulator's mission was amended to exclusively conduct
higher-fidelity testing with six radio frequency simulators and one
infrared test configuration. Decommissioning of the 20 simulations
considered in the BRAC 1995 round has resulted in unused space and
equipment at Plant 4. According to the Air Force, approximately 50
percent of the square footage of the Fort Worth facility is not used
to support current operations and 60 percent of the hardware is not
needed to meet current mission requirements. In addition, BRAC 1995
considered establishing a new facility at Edwards Air Force Base,
while the current approach is to relocate the radio frequency
simulations to an existing, operational facility at Wright-Patterson.
The current Air Force proposal to relocate the Simulator was initiated
by the Air Force Materiel Command Electronic Warfare Test & Evaluation
Consortium (the Consortium), a body formed in December 2008 to
coordinate electronic warfare test and evaluation activities. The
Consortium is charged with reducing duplication and increasing
efficiencies and effectiveness throughout the Air Force Materiel
Command's test and evaluation infrastructure. As a result of regular
periodic reviews of test facilities, the Consortium found that similar
infrared test capabilities existed at the Air Force Electronic Warfare
Evaluation Simulator location in Texas and the Guided Weapons
Evaluation Facility at Eglin, and that utilization of the Simulator in
Texas lagged behind utilization of the facility at Eglin. After
completing a study that compared the capabilities of each facility,
the Consortium recommended in May 2009 that the Air Force not renew
the Simulator contract with Lockheed Martin and instead redirect
infrared testing work to Eglin and relocate radio frequency
simulations to the Hangar 4F test facility at Wright-Patterson. The
pressures of a decreasing test and evaluation budget were also a
factor in the Consortium's recommendation to relocate the Simulator.
The Air Force approached the Test Resource Management Center for
approval to relocate the Simulator's test capabilities to facilities
at Eglin and Wright-Patterson in accordance with guidance from OSD and
congressional direction in Senate Report 110-77.[Footnote 8] In
response to this guidance and direction, the Test Resource Management
Center reviewed the Air Force relocation proposal and developed a
consolidated summary report of its findings.[Footnote 9] As a result
of these findings, the Test Resource Management Center gave
provisional approval to the Air Force relocation request. Senate
Report 110-77 also directed the Test Resource Management Center to
transmit its report to the congressional defense committees; this was
done on July 8, 2009. The Test Resource Management Center report was
developed in response to this OSD guidance and the congressional
direction in Senate Report 110-77, but OSD is using this same report
to address the congressional direction in House Report 111-230, which
is the basis for our review. OSD has not resubmitted the report to the
congressional defense committees pending GAO review.
OSD Has Identified Some but Not All Costs and Benefits Associated with
the Proposal to Relocate the Simulator, and Did Not Fully Follow Cost-
Benefit Analysis Guidance:
In its July 2009 report, OSD provided some limited costs and benefits
associated with the planned relocation, but the report does not
constitute a comprehensive cost-benefit analysis because it does not
include major costs, such as the $7 million estimated transition cost.
Furthermore, the process by which the Air Force developed its cost-
benefit information was not performed in accordance with relevant
guidance or best practices. Although Air Force officials provided us
with additional cost and benefit information intended to better
support its relocation rationale, we found that while compiling its
cost-benefit data, the Air Force did not have its cost-benefit
analysis reviewed and certified by Air Force Comptroller personnel, as
described in its internal guidance, to ensure that the analysis was
conducted properly and that assumptions included in the analysis were
reasonable.
OSD Identified Some but Not All Costs Associated with the Proposed
Relocation:
OSD's report identified the annual cost to operate, maintain, and
modernize the Air Force Electronic Warfare Evaluation Simulator, but
the report does not include the expected onetime total transition
costs to relocate the Simulator. For example, the report does not
include the estimated transition costs of equipment disassembly,
documentation, shipping, and reassembly. Additionally, the OSD report
does not include other estimated transition costs, such as facility
construction costs or expected costs of hiring or training personnel
at the receiving locations. When the Air Force submitted its
relocation proposal to the Test Resource Management Center for
approval, it did not provide these expected onetime transition costs
to the center or include a breakdown of these costs. A Test Resource
Management Center official responsible for the report noted that one
of the primary goals when reviewing the Air Force's proposal was to
assess whether the proposal would result in an unacceptable loss of
test and evaluation capability, and the report was not solely based on
whether the proposal was necessarily cost effective.
Subsequent to OSD issuing its report to Congress in response to Senate
Report 110-77, the Air Force provided us with additional documentation
that may help support its relocation rationale. For example, the Air
Force provided us with an estimated onetime total transition cost of
just over $7 million to relocate the Simulator that was not included
the report. This onetime transition cost estimate includes required
facility renovations needed at Wright-Patterson and the cost of
transferring equipment to this location, but does not include the
estimated cost of decommissioning the Simulator's infrared test
equipment or the cost of moving it to Eglin, which Air Force officials
have estimated to be approximately $200,000 to $300,000.[Footnote 10]
The Air Force completed an initial design study to derive cost
estimates for the required building renovations at Wright-Patterson.
As a result of this study, Air Force officials estimated that this
onetime cost to house the Simulator's radio frequency assets could
range from $2.7 million to as much as $3.9 million, depending on which
of the three renovation cost estimates developed as part of the design
study is selected.[Footnote 11]
However, Lockheed Martin representatives told us they believed that
the Air Force had considerably underestimated the relocation costs and
said the Air Force may not be including various costs elements in its
estimate. For example, Lockheed Martin estimates that developing new
hardware and software documentation for the Simulator would cost
approximately $14 million. However, Air Force officials consider
existing documentation to be sufficient and do not intend to purchase
additional documentation. Additionally, the potential overlap of
bringing Wright-Patterson personnel to Plant 4 for training while
sustaining Lockheed Martin staff, which was proposed by the Air Force,
was not included in the Air Force's transition cost estimate.
According to Air Force officials, retaining a limited Lockheed Martin
presence for an 18-month transition period may cost up to $3 million
in addition to the $7 million onetime transition cost.
Our analysis of the supporting documentation and discussions with Air
Force and Lockheed Martin personnel showed that the Air Force and
Lockheed Martin used different assumptions and cost data when
developing their relocation cost estimates, and that Lockheed Martin
may not be fully aware of the Air Force's relocation plans nor are its
personnel familiar with the test facilities at Eglin or Wright-
Patterson. For example, Lockheed Martin's estimate includes the costs
to disassemble, document, ship, reassemble, and train personnel for
the Simulator's infrared capability, costs the Air Force did not
include in its transition cost estimate since it intends to redirect
infrared testing work to Eglin without transferring the Simulator's
infrared equipment. Air Force officials told us that Eglin has
sufficient infrared test capacity to handle the current and future
workload, but they would consider moving the infrared equipment from
Plant 4 to Eglin if there are sufficient funds in the Air Force's
electronic warfare testing budget. The Air Force would use this
surplus equipment to assist with any potential surges in infrared
testing demand or for replacement parts. The $3 million potential cost
of temporarily retaining Lockheed Martin staff and the $200,000 to
$300,000 estimated cost of moving Plant 4's infrared equipment to
Eglin were not included in the Air Force's $7 million transition cost
estimate or OSD's report and underscore the importance of the Air
Force following its cost estimate guidance to reduce the risk that
other potential relocation costs are not omitted from its analysis.
OSD Identified Some but Not All Benefits Associated with the Proposed
Relocation:
OSD's report on the relocation proposal included some expected
financial benefits but excluded others that could have been used to
support the Air Force's relocation rationale. For example, OSD
reported that the relocation proposal would result in a return on
investment after the transition period and achieve an annual recurring
savings of about $2.8 million to $4.4 million. Air Force officials
told us that these savings would likely be realized in the third year
after the 1-to 2-year relocation is complete. The report shows the
basis of these savings to be a reduction in personnel, but does not
specifically detail the savings that would accrue from hiring
government personnel as opposed to using contractors. The Air Force
expects annual recurring cost savings to largely accrue from reducing
the Air Force Electronic Warfare Evaluation Simulator's support staff
from 30 to 18 and filling these 18 positions with newly hired
government employees instead of more costly contractor personnel,
which provides the basis of the Air Force's savings estimate. Air
Force officials have since provided us with a detailed labor rate
comparison between government and contractor operations that shows a
more expensive contractor rate, which is the basis for the Air Force's
cost comparison.
The Air Force also subsequently identified and provided us with
multiple expected benefits of the proposal to move the Air Force
Electronic Warfare Evaluation Simulator that were omitted from OSD's
report. According to Air Force officials, one of the primary benefits
that would result from the relocation would be building government
capacity and expertise over time by reducing the dependency on
contractors. Air Force officials told us they believe that by
eliminating contractors and hiring government employees to operate the
Simulator, the Air Force would effectively insource the expertise
currently held by contractors and build government capacity in this
highly technical and sensitive test environment. Additionally, Air
Force officials noted that contractor expertise is perishable, leaving
the Air Force at risk of losing integral skills, knowledge, and
experience since it has limited control over the retention of
contractor experts in this highly technical field. For example, since
the Air Force developed its relocation plan, approximately 12 of the
30 Lockheed Martin employees involved in the Simulator's operations
have left their positions.
The Air Force also cited technical benefits that would result from the
relocation of radio frequency assets to Wright-Patterson. For example,
according to Air Force officials, the relocation of radio frequency
assets is necessary to support emerging electronic protection methods
that cannot be supported at Plant 4. They maintain that the more
advanced infrastructure and assets at Wright-Patterson will allow for
the growth of simulation capability and provide for higher-fidelity
testing. The Air Force cited the synergies that are expected from both
the relocation of the Simulator's radio frequency capability to Wright-
Patterson, where other radio frequency capabilities exist, and from
the consolidation of infrared testing at Eglin.
The Air Force Did Not Fully Follow Standards and Guidance for
Conducting Its Cost-Benefit Analysis:
Although the information that Air Force officials provided to us
during the course of our review may provide more support for the Air
Force's proposal by identifying some of the associated costs and
potential benefits of relocating the Air Force Electronic Warfare
Evaluation Simulator that were not included in OSD's report, the
process by which the Air Force developed this information was not
performed in accordance with relevant guidance or best practices. Air
Force economic analysis guidance includes detailed procedures and a
checklist that explains the types of cost and benefit data that should
be included in a comprehensive analysis with steps on how to ensure
the reliability of estimates, such as having the results certified by
appropriate officials outside of the program office. The Air Force
employed a contractor to develop its preliminary transition plan, but
the cost estimates therein were not certified by Air Force Comptroller
staff as specified by Air Force guidance. Furthermore, OMB Circular A-
94, Guidelines and Discount Rates for Benefit-Cost Analysis of Federal
Programs, also states that key data and results, such as year-by-year
estimates of benefits and costs, should be reported to promote
independent analysis and review. In addition, GAO has identified best
practices for developing cost-benefit analyses, such as evaluating
alternatives and obtaining an independent cost estimate developed by
an entity outside the program office.
Air Force officials told us they believe that they provided a
sufficient amount of information to support their rationale for
relocating the Simulator. According to Air Force officials, completing
a comprehensive cost-benefit analysis would have been prohibitively
expensive, particularly given the limited scope and size of this
program. While Air Force guidance permits them to secure a waiver
under these and other circumstances, the Air Force office developing
the cost-benefit analysis did not request such a waiver, or provide us
with documentation to support its rationale that conducting such an
analysis would have been cost prohibitive. However, securing such a
waiver would not have been responsive to congressional direction to
conduct a comprehensive cost-benefit analysis. Moreover, the 1995 DOD
recommendation was rejected by the BRAC Commission primarily for
having underestimated relocation costs, which emphasizes the
importance of following relevant cost-benefit guidance to reduce the
risk of costs being underestimated in this relocation proposal.
Air Force Instruction 65-501[Footnote 12] and Air Force Manual 65-506
[Footnote 13] specify the steps and procedures Air Force program
offices should follow when conducting a cost-benefit analysis to help
make rational decisions among alternatives. This guidance specifies
the process for conducting a cost-benefit analysis, and Air Force
Comptroller personnel are responsible for issuing a certificate of
satisfaction for any such analysis. Table 1 identifies the key steps
in the Air Force guidance for conducting a comprehensive cost-benefit
analysis and our assessment of the extent to which OSD's July 2009
report and additional supplemental information provided to us by the
Air Force during our review conformed to the Air Force guidance for
conducting a cost-benefit analysis.
Table 1: GAO's Assessment of the Cost-Benefit Analysis to Support the
Proposal to Relocate the Air Force Electronic Warfare Evaluation
Simulator:
Key economic analysis step[A]: Provide a problem statement;
Cost-benefit information provided to support the proposal to relocate
the Air Force Electronic Warfare Evaluation Simulator: OSD's report:
The report includes some problem statements that support the rationale
to relocate the Simulator, but the report does not include all of the
problem statements later identified by the Air Force;
Supplemental information provided by the Air Force:
* Air Force has little control over the retention of critical
expertise while the Simulator is under contractor operation;
* Emerging technological advancements in testing cannot be supported
by Plant 4 but can be supported at Wright-Patterson;
Information that could help make the cost-benefit analysis report more
comprehensive: No additional information is needed.
Key economic analysis step[A]: Identify and explain all assumptions;
Cost-benefit information provided to support the proposal to relocate
the Air Force Electronic Warfare Evaluation Simulator: OSD's report:
The report includes assumptions that the relocation will be completed
within a 1-to 2-year time frame; customer demand is expected to remain
low for the next 2 years; and infrared customer demand can be
sufficiently handled by Eglin;
Supplemental information provided by the Air Force: None;
Information that could help make the cost-benefit analysis report more
comprehensive: No project plan was provided that supports the
rationale that the relocation will take 1 to 2 years to complete and
that this time frame is realistic and feasible. No documentation of
customer testing schedules was provided that supports the assumption
that customer demand is expected to remain low during the planned
transition period.
Key economic analysis step[A]: Include all feasible alternatives;
Cost-benefit information provided to support the proposal to relocate
the Air Force Electronic Warfare Evaluation Simulator: OSD's report:
The report includes the proposal to relocate the Simulator's radio
frequency capability to Wright-Patterson and redirect infrared work to
Eglin;
Supplemental information provided by the Air Force: None;
Information that could help make the cost-benefit analysis report more
comprehensive: Although other test locations were identified by the
Air Force, no alternatives other than the relocation proposal and the
status quo were provided during the course of our review.
Key economic analysis step[A]: Include a cost analysis;
Cost-benefit information provided to support the proposal to relocate
the Air Force Electronic Warfare Evaluation Simulator: OSD's report:
The report considered nonfinancial costs, such as the risk of
capability loss and the potential impact to test customers;
Supplemental information provided by the Air Force: The information
included an estimated $7.15 million onetime cost to transition the
radio frequency capability to Wright-Patterson. This includes:
* $3.9 million for needed building renovations;
* $2.9 million for disassembling, packing, shipping, and reassembling
equipment; and;
* $150,000 for closeout costs at Plant 4;
Information that could help make the cost-benefit analysis report more
comprehensive: OSD's report did not include any estimated transition
costs. The following additional expected costs were not included in
the Air Force's transition cost estimate:
* Recruitment or hiring costs for 18 new government personnel;
* Training costs;
* The cost of retaining Lockheed Martin staff during the transition;
* The cost to decommission or relocate Plant 4's infrared test
equipment to Eglin;
* The total life cycle cost of each alternative.
Key economic analysis step[A]: Include a benefit analysis;
Cost-benefit information provided to support the proposal to relocate
the Air Force Electronic Warfare Evaluation Simulator: OSD's report:
The report includes the expected annual institutional savings ($2.8
million to $4.4 million) subsequent to the transition;
Supplemental information provided by the Air Force:
* Test customers are expected to realize savings because of lower fees
at Eglin and Wright-Patterson;
* Relocation of six radio frequency simulators to a more technically
advanced architecture at Wright-Patterson is expected to result in
better utilization of these assets and higher-fidelity testing;
* Insourcing is expected to build critical government capacity to
operate highly technical test assets;
Information that could help make the cost-benefit analysis report more
comprehensive: The Air Force did not quantify all the benefits it
identified, including the following expected benefits common to both
the infrared and radio frequency capabilities:
* Reduced Air Force cost for operation and maintenance oversight;
* Reduced Air Force cost for improvement and modernization oversight;
* Reduced cost for supporting duplicate sets of support equipment;
* Reduced contract management/negotiation hours;
* Reduced security oversight requirements;
* Total life cycle benefits.
Key economic analysis step[A]: Conduct a comparison selection
evaluation;
Cost-benefit information provided to support the proposal to relocate
the Air Force Electronic Warfare Evaluation Simulator: OSD's report: A
comparison selection evaluation was not included;
Supplemental information provided by the Air Force: None;
Information that could help make the cost-benefit analysis report more
comprehensive: OSD and the Air Force did not provide an evaluation
that compared the merits of alternatives nor did they quantify the
differences between them.
Key economic analysis step[A]: Conduct a sensitivity analysis;
Cost-benefit information provided to support the proposal to relocate
the Air Force Electronic Warfare Evaluation Simulator: OSD's report:
No sensitivity analysis was included;
Supplemental information provided by the Air Force: None;
Information that could help make the cost-benefit analysis report more
comprehensive: OSD and the Air Force did not include an evaluation of
the relocation project schedule to determine the operational and cost
impacts, or other possible impediments that may affect the Air Force
or test customers, if slippages or advancements in the relocation
schedule should occur.
Key economic analysis step[A]: Executive summary;
Cost-benefit information provided to support the proposal to relocate
the Air Force Electronic Warfare Evaluation Simulator: OSD's report:
The report includes an executive summary that contains a very basic
rationale for the relocation;
Supplemental information provided by the Air Force: None;
Information that could help make the cost-benefit analysis report more
comprehensive: OSD and the Air Force did not provide an executive
summary that includes key elements of the information subsequently
provided by the Air Force, such as:
* a problem statement that clearly identifies the objectives;
* criteria for the recommendation;
* details describing any feasible alternatives, and;
* assumptions and constraints.
Key economic analysis step[A]: Certify conclusions;
Cost-benefit information provided to support the proposal to relocate
the Air Force Electronic Warfare Evaluation Simulator: OSD's report:
No certifications accompanied the report;
Supplemental information provided by the Air Force: None;
Information that could help make the cost-benefit analysis report more
comprehensive: The Air Force's information submissions were not
certified by the appropriate comptroller offices.
Source: GAO analysis of Air Force Manual 65-506, the July 2009 OSD
report, and cost-benefit information provided by the Air Force.
[A] These steps are included in Air Force Manual 65-506.
[End of table]
The cost-benefit information included in the OSD report and the
additional information subsequently provided to us by the Air Force
help support the rationale for relocating the Air Force Electronic
Warfare Evaluation Simulator, but this information was not
consolidated in the Air Force's submission or certified by the
appropriate Air Force Comptroller offices, which would help ensure
that the information was reliable and complete. House Report 111-230
directs DOD to provide a comprehensive cost-benefit analysis, and by
not adhering to relevant guidance and best practices, the Air Force
increases the risk that decisions regarding the relocation of the
Simulator will be made without a clear understanding of all the
potential costs, benefits, and impacts associated with the relocation.
Furthermore, OSD and the Air Force may be unable to reassure Congress
that the proposal to relocate the Simulator is based on a
methodologically sound rationale.
The Air Force Has Addressed One of Two Technical Issues Associated
with the Relocation Proposal but Has Not Finalized a Transition Plan:
In its July 2009 report that conditionally approved the relocation
proposal, OSD conducted a technical comparison between the Air Force
Electronic Warfare Evaluation Simulator's current location and the
test facilities at the proposed receiving locations and identified two
primary technical issues that could have an impact on the successful
relocation. Those issues were the lack of an engine and flare
simulator capability at Eglin that now exists at Fort Worth and the
lack of personnel experience and technical depth at Wright-Patterson
required to understand and operate the Simulator's unique surface-to-
air simulations. The Air Force has addressed the first of these
concerns by equipping Eglin with an additional high-intensity infrared
engine and flare source, and also expressed confidence that sufficient
expertise currently exists at Wright-Patterson to learn to operate the
Simulator's capabilities. However, as of November 2010, none of the
Wright-Patterson personnel had undergone any hands-on training to
operate and maintain the Air Force Electronic Warfare Evaluation
Simulator's one-of-a-kind surface-to-air missile testing capabilities
and the Air Force had not finalized a transition plan to document how
the engineers at Wright-Patterson will be adequately trained prior to
the proposed relocation.
The Air Force Has Addressed the Technical Consideration Identified at
Eglin:
In its July 2009 report, OSD noted that the Eglin test facility lacked
the necessary engine and flare simulator capability needed to support
future customer testing needs and recommended that Eglin demonstrate a
high-intensity infrared engine and flare source comparable to the
current Air Force Electronic Warfare Evaluation Simulator's capability
within 1 year. Since OSD's report was issued, Eglin personnel have
demonstrated this capability. During our fieldwork at Eglin, we met
with Air Force officials responsible for oversight of Eglin's Guided
Weapons Evaluation Facility and toured the facility. During our tour,
Air Force officials showed us that the hardware and software necessary
to provide a high-intensity infrared engine and flare source
comparable to the Simulator had been procured and installed using
lasers and mirrors. This newly implemented infrared engine and flare
source was funded in part by the Department of the Navy, which plans
to use this system during upcoming countermeasure testing scheduled
for the summer of 2011. According to Air Force officials, Eglin has
demonstrated this capability, which addresses OSD's recommendation,
and is confident that it will meet its customers' needs. Based on our
fieldwork and discussions with various customers and Eglin engineers,
it appears that the Air Force will likely be in position to achieve
its testing goal. Further, an official with DOD's Test Resource
Management Center visited Eglin in July 2010 to confirm that Eglin had
acquired the newly installed engine and flare capability, and told us
that the center will formally review that capability once Eglin
completes its pending operational test report.
Although, during our review, Lockheed Martin employees we interviewed
expressed concerns that Eglin's approach for infrared testing may have
some technical limitations that could affect the testing results of
its future customers, the Air Force and the test customers we
interviewed did not share those concerns. According to Lockheed Martin
employees, the infrared testing simulations conducted at Plant 4 in
Texas provide a more flexible and capable solution than the
simulations currently used at Eglin. Nevertheless, Air Force officials
consider the infrared capabilities at the Plant 4 and Eglin facilities
redundant and intend to redirect all infrared test work to Eglin.
Although Air Force officials acknowledged that the Simulator's
facility in Texas and the facility at Eglin have differing technical
approaches for testing, they reiterated that customer demand at Eglin
is greater and the infrared testing capability at the Plant 4 facility
in Texas is duplicative, so the consolidation makes good business
sense for the Air Force. In addition, test customers, such as
government officials from the Joint Strike Fighter Program and
officials from the private sector, told us that they were confident
and satisfied with the infrared testing and evaluation capabilities
provided at Eglin.
The Air Force Has Not Finalized Its Transition Plan to Include Steps
for Training and Maintaining Staff to Operate the Simulator's Unique
Surface-to-Air Capability at Wright-Patterson:
OSD noted in its July 2009 report that the relocation of the Air Force
Electronic Warfare Evaluation Simulator's radio frequency capability
to Wright-Patterson poses a moderate risk because, although some
Wright-Patterson personnel have the technical core competencies to
pick up the Simulator's workload, the unique experience and technical
depth required to operate the Simulator does not exist there. As of
November 2010, the Air Force had not begun training Wright-Patterson
staff to operate the Simulator's unique radio frequency capability or
finalized a transition plan to document how it planned to do so.
As part of the proposed transition effort, OSD's report recommended
that the Air Force retain Lockheed Martin contract employees
permanently or temporarily to ensure that this capability can be
successfully transitioned and brought up to operational status within
the 1-to 2-year transition time frame estimated by the Air Force.
However, Air Force officials assert that they are in the final stages
of reviewing the technical manuals and available documentation
necessary for Wright-Patterson engineers to operate the Simulator's
unique surface-to-air radio frequency capability. According to Air
Force officials responsible for the program, they had not yet assessed
the sufficiency of the documentation during the time of OSD's review
in 2009, leading OSD to conclude that the technical depth required to
operate and understand the Simulator's capability did not exist at
Wright-Patterson. As of November 2010, Air Force officials had not
finalized their review of the existing documentation, but had
preliminarily determined that the documentation was sufficient to
enable engineers at Wright-Patterson to operate the radio frequency
capabilities. Air Force officials told us that Wright-Patterson
engineers have as much, if not more, years of experience and technical
knowledge related to electronic warfare countermeasure testing than
Lockheed Martin engineers and provided documentation detailing their
years of experience and areas of expertise. Although the Wright-
Patterson contract and government engineers with whom we spoke
attested that they have all of the technical expertise and core
competencies necessary to undertake the role of operating the
Simulator's capabilities, they said that they would prefer some
assistance from the current Lockheed Martin operators to decrease the
transition time and risks associated with relocating this equipment.
Air Force engineers told us that a transition plan, which includes a
proposed training strategy, is currently under review, but the Air
Force has not finalized the plan. Our prior work on sound transition
planning states that agencies should ensure that personnel with the
right skills are in place to support a transition effort, including
identifying and requiring training for those carrying out the
transition or operating and maintaining newly transitioned equipment.
Although Air Force officials have expressed confidence that sufficient
expertise and documentation exists at Wright-Patterson to operate the
Simulator's radio frequency capability, OSD and congressional decision
makers lack assurance that the Air Force will be able to maintain
continuity of operations after the proposed relocation because the Air
Force has not specifically trained any of its personnel or finalized
how it plans to provide hands-on training for personnel expected to
operate this capability. Further, although Air Force officials told us
that they have conducted preliminary discussions with Lockheed Martin
representatives and are willing to temporarily retain some of their
employees to transition and operate this system long enough to train
Air Force personnel, no agreement between Lockheed Martin and the Air
Force had been finalized as of November 2010.
If the Air Force does not make arrangements to secure personnel with
the needed hands-on experience at Wright-Patterson during the planned
relocation, or provide training for Air Force personnel, the Simulator
may encounter delays in providing needed test capabilities to its
customers. Lockheed Martin representatives told us they believe that
if the Air Force moves forward with the relocation without Lockheed
Martin's assistance, it could take the Air Force up to 5 years to
transfer, set up, and operate this unique testing capability. Air
Force officials acknowledge that if Lockheed Martin employees are not
utilized during the planned 1-to 2-year transition period, the
transition period could be prolonged. Air Force officials showed us a
draft transition plan that identifies actions to help ensure a smooth
transition should the Air Force be allowed to proceed with the
proposed relocation. However, until the Air Force adheres to sound
transition planning practices and finalizes a plan that ensures
continuity of operations--including identifying how it plans to train
or maintain personnel with needed experience to operate the
Simulator's unique surface-to-air testing capabilities--DOD may
continue to face the risk that the Simulator's capabilities designed
to protect U.S. and allied aircraft may not be fully operational
within the planned transition time frame.
Conclusions:
If the Air Force expects the relocation of the Air Force Electronic
Warfare Evaluation Simulator to improve its management of electronic
warfare infrastructure, reduce costs, and help DOD achieve its goal of
decreasing reliance on contractors and developing in-house talent,
then it is important that the Air Force respond to the congressional
direction included in House Report 111-230 by following relevant
guidance for conducting a comprehensive cost-benefit analysis. A
comprehensive cost-benefit analysis that has been reviewed and vetted
by an independent party outside the immediate organization, and that
includes all expected costs and benefits associated with the proposal,
would assist both DOD and congressional decision makers in making an
informed decision regarding the proposed relocation of the Simulator's
capabilities. Conforming to its own policies, as well as best
practices in formulating cost-benefit analyses, would help to ensure
the accuracy and credibility of the analysis performed and mitigate
the risk that underestimated costs or decreased savings would
compromise program budgets. Until the Air Force submits additional
information that identifies all expected costs and benefits associated
with its proposal, in consultation with the appropriate Air Force
Comptroller and Financial Management offices, it may not be able to
provide reasonable assurance that its proposal to relocate the
Simulator is cost effective and in the best interest of the Air Force.
Moreover, because the Simulator comprises many complex and technical
parts, it is vital that the Air Force demonstrate that personnel with
the appropriate skills and experience can fully operate and maintain
the Simulator's capabilities if a relocation is to occur. Until OSD
and the Air Force finalize a comprehensive transition plan to ensure
continuity of the Simulator's operations--particularly its unique
capability to test countermeasures used against radio frequency
surface-to-air missiles--the personnel at Wright-Patterson may
experience a learning curve because of the complexity of the Simulator
and their unfamiliarity with its unique capabilities. Overcoming this
learning curve may pose the risk that important test capabilities will
not be fully available to customers when tests are requested.
Finalizing a transition plan that includes requisite actions for
staffing and training personnel prior to operating and maintaining the
Simulator's equipment will help the Air Force mitigate the risk of a
potential countermeasure test capability gap, reduce the potential
negative impact on test customers, and increase the likelihood of a
successful transition over the next several years.
Recommendations for Executive Action:
To satisfy congressional direction included in House Report 111-230 to
provide a comprehensive cost-benefit analysis of the Air Force's
proposal to relocate the Air Force Electronic Warfare Evaluation
Simulator, we recommend that the Secretary of Defense direct the Under
Secretary of Defense (Acquisition, Technology and Logistics), in
consultation with the Secretary of the Air Force, to take the
following two actions:
* revise the previously prepared cost-benefit analysis, in line with
internal guidance and in consultation with the appropriate Air Force
Comptroller and Financial Management offices, and identify all
expected costs and benefits associated with the proposed relocation to
determine whether the proposed relocation is cost effective and in the
best interests of national security and:
* submit this revised analysis to the congressional defense committees.
To ensure an effective phased transition of the Air Force Electronic
Warfare Evaluation Simulator's radio frequency capabilities from its
current location to Wright-Patterson and to minimize the potential
impact of a delayed transition on test customers, we recommend that
the Secretary of Defense direct the Under Secretary of Defense
(Acquisition, Technology and Logistics), in consultation with the
Secretary of the Air Force, to take the following two actions:
* finalize a transition plan that includes steps for staffing and
training personnel to operate and maintain relocated Simulator
capabilities at Wright-Patterson and:
* submit this plan to the congressional defense committees.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, DOD stated that it
concurred with all of our recommendations. In response to our
recommendation to revise its previously prepared cost-benefit analysis
in line with internal guidance and in consultation with the
appropriate Air Force Comptroller and Financial Management offices,
and identify all expected costs and benefits associated with the
proposed relocation, DOD stated that it plans to revise its cost-
benefit analysis based on the current Simulator situation, and plans
to submit its revised cost-benefit analysis to the congressional
defense committees within 90 days after publication of our report. In
response to our recommendation to finalize a transition plan that
includes steps for staffing and training personnel to operate and
maintain relocated Simulator capabilities at Wright-Patterson, DOD
stated that it will finalize the current transition plan to operate,
maintain, and relocate Simulator capabilities to Wright-Patterson, and
plans to submit its revised transition plan to the congressional
defense committees within 90 days after publication of our report.
DOD's comments are reprinted in their entirety in enclosure I. DOD
also provided a number of technical and clarifying comments, which we
have incorporated where appropriate.
We are sending copies of this report to the Secretary of Defense; the
Director, Office of Management and Budget; and interested
congressional committees. This report is also available at no charge
on the GAO Web site at [hyperlink, http://www.gao.gov].
If you or your staff have any questions about this report, please
contact me at (202) 512-4523 or leporeb@gao.gov. Contact points for
our Offices of Congressional Relations and Public Affairs may be found
on the last page of this report. GAO staff who made key contributions
to this report are listed in enclosure II.
Signed by:
Brian J. Lepore, Director:
Defense Capabilities and Management:
Enclosures - 2:
[End of section]
Enclosure I: Comments from the Department of Defense:
Test Resource Management Center:
3000 Defense Pentagon:
Washington, DC 20301-3000:
Mr. Brian Lepore:
Director, Defense Capabilities and Management:
U.S. Government Accountability Office:
441 G Street, N.W.
Washington, DC 20548:
Dear Mr. Lepore:
This is the Department of Defense (DoD) response to the GAO Draft
Report GAO-11123, "Defense Infrastructure: Further Actions Needed to
Support Air Force Electronic Warfare Evaluation Simulator Relocations
Plans," dated December 6, 2010 (GAO Code 351495). Detailed comments on
the report recommendations are enclosed.
The Department concurs with the recommendations of the GAO. Eighteen
months have transpired since AT&L conducted its original review of the
AFEWES relocation. Therefore, AT&L in consultation with the Secretary
of the Air Force will update the cost benefit analysis and transition
plan to reflect the present situation and submit to the Congressional
Defense Committees in accordance with GAO recommendations.
Sincerely,
Signed by:
Dr. John B. Foulkes:
Director:
Enclosures: As stated:
[End of letter]
Draft Report Dated December 6, 2010, GAO-11-123, Code 351495:
"Defense Infrastructure: Further Actions Needed To Support Air Force
Electronic Warfare Evaluation Simulator Relocations Plans"
Department Of Defense Comments To The GAO Recommendations:
Recommendation 1: The GAO recommends that the Secretary of Defense
direct the Under Secretary of Defense (Acquisition, Technology and
Logistics), in consultation with the Secretary of the Air Force, to
revise its previously prepared cost benefit analysis, in-line with
internal guidance and in consultation with the appropriate Air Force
Comptroller and Financial Management offices, and identify all
expected costs and benefits associated with the proposed relocation to
determine whether the proposed relocation is cost effective and in the
best interests of national security. (See page 25/GAO Draft Report.)
DOD Response: Concur: The previously prepared cost benefit analysis is
no longer applicable given the current AFEWES situation and amount of
time that has passed since original submission. AT&L in consultation
with the Secretary of the Air Force will revise the cost benefit
analysis based on the present situation and address the GAO
recommendations outlined in Table 1 (See page 17-18/GAO Draft Report).
Recommendation 2: The GAO recommends that the Secretary of Defense
direct the Under Secretary of Defense (Acquisition, Technology and
Logistics), in consultation with the Secretary of the Air Force, to
submit the revised cost benefit analysis to the congressional defense
committees. (See page 25/GAO Draft Report.)
DOD Response: Concur. Within 90 days after the publication of the GAO
final report, the Department will submit the revised cost benefit
analysis to the congressional defense committees.
Recommendation 3: The GAO recommends that the Secretary of Defense
direct the Under Secretary of Defense (Acquisition, Technology and
Logistics), in consultation with the Secretary of the Air Force, to
finalize a transition plan that includes steps for staffing and
training personnel to operate and maintain relocated Simulator
capabilities at Wright-Patterson Air Force Base. (See page 25/GAO
Draft Report.)
DOD Response: Concur: The Secretary of the Air Force will finalize the
current transition plan to operate, maintain, and relocate Simulator
capabilities to Wright-Patterson Air Force Base. Upon completion and
USD AT&L approval of the transition plan, the AFEWES transition and
relocation will move forward.
Recommendation 4: The GAO recommends that the Secretary of Defense
direct the Under Secretary of Defense (Acquisition, Technology and
Logistics), in consultation with the Secretary of the Air Force, to
submit this transition plan to the congressional defense committees.
(See page 25/GAO Draft. Report.)
DOD Response: Concur. Within 90 days after the publication of the GAO
final report, the Department will forward the transition plan to the
congressional defense committees.
[End of section]
Enclosure II: GAO Contact and Staff Acknowledgments:
GAO Contact:
Brian J. Lepore, (202) 512-4523 or leporeb@gao.gov:
Staff Acknowledgments:
In addition to the contact named above, James Reifsnyder, Assistant
Director; Laura Durland, Assistant Director; Steven Banovac; Tisha
Derricotte; Susan Ditto; Jason Jackson; Charles Perdue; Amie Steele;
and Michael Willems made key contributions to this report.
[End of section]
Footnotes:
[1] Infrared missile systems' seekers use engine exhaust and other
heat sources on an aircraft to follow it in flight. Infrared
countermeasures attempt to create alternate heat sources as decoys to
redirect the heat-seeking missile away from the aircraft.
[2] In radio frequency missile systems, radio waves are transmitted
either by an autonomously guided missile or from a ground-based
tracking and command guidance system. These waves bounce off of the
aircraft and back to the radio wave seeker on a missile or a ground-
based receiver, which then analyzes these bounced-back waves to
identify the range, altitude, direction, and speed of the aircraft in
flight. Radio frequency countermeasures attempt to interrupt or
deceive the enemy's radio waves and debilitate the missile, causing it
to miss the aircraft.
[3] The Air Force Electronic Warfare Evaluation Simulator at Plant 4
consists of six radio frequency simulators and an infrared simulator.
These simulators can conduct multiple simulations and are operated
with different sets of equipment and computers.
[4] Office of the Under Secretary of Defense (Acquisition, Technology
and Logistics) Memorandum, Changes to the Composition of the Major
Range and Test Facility Base (Jan. 18, 2008).
[5] Senate Report 110-77, to accompany a proposed bill for the Fiscal
Year 2008 Defense Authorization Act (S. 1547).
[6] OMB Circular A-94, Guidelines and Discount Rates for Benefit-Cost
Analysis of Federal Programs (Oct. 29, 1992).
[7] GAO, Electronic Combat: Consolidation Master Plan Does Not Appear
to Be Cost-Effective, [hyperlink,
http://www.gao.gov/products/GAO/NSIAD-97-10] (Washington, D.C.: July
10, 1997).
[8] The OSD guidance was included in a January 18, 2008, memorandum
from the Office of the Under Secretary of Defense (Acquisition,
Technology and Logistics), entitled Changes to Composition of the
Major Range and Test Facility Base. The Test Resource Management
Center is tasked with approving Air Force proposals that would change
any major range and test facility base, which includes the Simulator.
Congressional direction was included in Senate Report 110-77 to
accompany a proposed bill for the Fiscal Year 2008 Defense
Authorization Act (S. 1547).
[9] Test Resource Management Center, Impact Report on Air Force
Materiel Command Electronic Warfare T&E Enterprise (Realignment of Air
Force Electronic Warfare Evaluation Simulator, Fort Worth, TX)
(Washington D.C., July 2009).
[10] Decommissioning the infrared test equipment would include
dismantling and removing it from Plant 4.
[11] The $3.9 million renovation cost estimate is included within the
$7 million onetime transition cost estimate.
[12] Air Force Instruction 65-501, Economic Analysis (Nov. 10, 2004).
[13] Air Force Manual 65-506, Economic Analysis (Nov. 10, 2004),
implementing Air Force Instruction 65-501.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
