Defense Transportation
Study Limitations Raise Questions about the Adequacy and Completeness of the Mobility Capabilities Study and Report Gao ID: GAO-06-938 September 20, 2006The Department of Defense (DOD) issued the Mobility Capabilities Study (MCS), which was intended to identify and quantify the mobility capabilities needed to support U.S. strategic objectives into the next decade. The MCS found that projected capabilities are adequate to achieve U.S. objectives with an acceptable level of risk--that is, current U.S. inventory of aircraft, ships, prepositioned assets, and other capabilities are sufficient, in conjunction with host nation support, and assuming planned investments take place. The Senate report accompanying the bill for the fiscal year 2005 Defense Authorization Act required GAO to report on the adequacy and completeness of the MCS. GAO assessed the extent to which the MCS met generally accepted research standards that this type of study would be expected to meet to be considered sound and complete.
DOD used an innovative approach in conducting the study and acknowledged methodological limitations in its report; however, it did not fully disclose how these limitations could affect the MCS conclusions and recommendations. Therefore, it is not transparent how the analyses done for the study support DOD's conclusions. Measured against relevant generally accepted research standards, GAO has identified limitations in the MCS and its report that raise questions about their adequacy and completeness. GAO suggests that Congress and other decision makers exercise caution in using the MCS to make investment decisions. Among GAO's findings: Aspects of modeling and data were inadequate in some areas because data were lacking and the models used could not simulate all relevant aspects of the missions. The report did not explain how these limitations could affect the study results or what the impact on projected mobility capabilities might be. Generally accepted research standards require that models used are adequate for the intended purpose, represent a complete range of conditions, and that data used are properly generated and complete. For example, the MCS modeled hypothetical homeland defense missions rather than homeland defense demands derived from a well defined and approved concept of operations for homeland defense, because the specific details of the missions were still being determined and the data used may be incomplete. The MCS also was unable to model the flexible deterrent options/deployment order process to move units and equipment into theater because of lack of data, but the study assumed a robust use of this process. In addition, the MCS report contains over 80 references to the need for improved modeling or data. While the MCS concluded that combined U.S. and host nation transportation assets were adequate, in describing the use of warfighting metrics in its analyses, the report does not provide a clear understanding of the direct relationship of warfighting objectives to transportation capabilities. Additionally, the report stated that further analysis is required to understand the operational impact of increased or decreased strategic lift on achieving warfighting objectives. Relevant generally accepted research standards require that conclusions be supported by analyses. The use of both warfighting and mobility metrics would allow decision makers to know whether combat tasks were achieved and how much strategic transportation is needed to accomplish those tasks. In some cases, the MCS results were incomplete, unclear, or contingent on further study, making it difficult to identify findings and evaluate evidence. Relevant research standards require results to be presented in a complete, accurate, and relevant manner. For example, the report contains recommendations for further studies and assessments, five of which are under way. However, DOD has no plans to report the impact of these studies on the MCS results after the studies are complete. In addition, the report contains qualified information that is not presented clearly, such as varying assessments of intra-theater assets in three different places.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-06-938, Defense Transportation: Study Limitations Raise Questions about the Adequacy and Completeness of the Mobility Capabilities Study and Report
This is the accessible text file for GAO report number GAO-06-938
entitled 'Defense Transportation: Study Limitations Raise Questions
about the Adequacy and Completeness of the Mobility Capabilities Study
and Report' which was released on September 20, 2006.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
September 2006:
Defense Transportation:
Study Limitations Raise Questions about the Adequacy and Completeness
of the Mobility Capabilities Study and Report:
Defense Transportation:
GAO-06-938:
GAO Highlights:
Highlights of GAO-06-938, a report to congressional committees
Why GAO Did This Study:
The Department of Defense (DOD) issued the Mobility Capabilities Study
(MCS), which was intended to identify and quantify the mobility
capabilities needed to support U.S. strategic objectives into the next
decade. The MCS found that projected capabilities are adequate to
achieve U.S. objectives with an acceptable level of risk”that is,
current U.S. inventory of aircraft, ships, prepositioned assets, and
other capabilities are sufficient, in conjunction with host nation
support, and assuming planned investments take place.
The Senate report accompanying the bill for the fiscal year 2005
Defense Authorization Act required GAO to report on the adequacy and
completeness of the MCS. GAO assessed the extent to which the MCS met
generally accepted research standards that this type of study would be
expected to meet to be considered sound and complete.
What GAO Found:
DOD used an innovative approach in conducting the study and
acknowledged methodological limitations in its report; however, it did
not fully disclose how these limitations could affect the MCS
conclusions and recommendations. Therefore, it is not transparent how
the analyses done for the study support DOD‘s conclusions. Measured
against relevant generally accepted research standards, GAO has
identified limitations in the MCS and its report that raise questions
about their adequacy and completeness. GAO suggests that Congress and
other decision makers exercise caution in using the MCS to make
investment decisions. Among GAO‘s findings:
* Aspects of modeling and data were inadequate in some areas because
data were lacking and the models used could not simulate all relevant
aspects of the missions. The report did not explain how these
limitations could affect the study results or what the impact on
projected mobility capabilities might be. Generally accepted research
standards require that models used are adequate for the intended
purpose, represent a complete range of conditions, and that data used
are properly generated and complete. For example, the MCS modeled
hypothetical homeland defense missions rather than homeland defense
demands derived from a well defined and approved concept of operations
for homeland defense, because the specific details of the missions were
still being determined and the data used may be incomplete. The MCS
also was unable to model the flexible deterrent options/deployment
order process to move units and equipment into theater because of lack
of data, but the study assumed a robust use of this process. In
addition, the MCS report contains over 80 references to the need for
improved modeling or data.
* While the MCS concluded that combined U.S. and host nation
transportation assets were adequate, in describing the use of
warfighting metrics in its analyses, the report does not provide a
clear understanding of the direct relationship of warfighting
objectives to transportation capabilities. Additionally, the report
stated that further analysis is required to understand the operational
impact of increased or decreased strategic lift on achieving
warfighting objectives. Relevant generally accepted research standards
require that conclusions be supported by analyses. The use of both
warfighting and mobility metrics would allow decision makers to know
whether combat tasks were achieved and how much strategic
transportation is needed to accomplish those tasks.
* In some cases, the MCS results were incomplete, unclear, or
contingent on further study, making it difficult to identify findings
and evaluate evidence. Relevant research standards require results to
be presented in a complete, accurate, and relevant manner. For example,
the report contains recommendations for further studies and
assessments, five of which are under way. However, DOD has no plans to
report the impact of these studies on the MCS results after the studies
are complete. In addition, the report contains qualified information
that is not presented clearly, such as varying assessments of intra-
theater assets in three different places.
What GAO Recommends:
GAO recommends that the Secretary of Defense, in future mobility s
capabilities studies beginning with any study currently under way,
develop models and data for critical missions and processes; include in
study reports an explanation of how stated limitations might impact
results; and, incorporate both mobility and warfighting metrics to
determine capabilities. In comments, DOD concurred with two of the
recommendations and claimed they did not understand the third, which
GAO clarified.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-938].
To view the full product, including the scope and methodology, click on
the link above. For more information, contact William M. Solis at
(202)512-8365 or solisw@gao.gov.
[End of Section]
Contents:
Letter:
Results in Brief:
Background:
Limitations in the MCS Study and Report Raise Questions about Adequacy
and Completeness:
Conclusions:
Recommendation for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Comments from the Department of Defense:
Appendix III: GAO Contact and Staff Acknowledgments:
Table:
Table 1: Generally Accepted Research Standards Relevant to MCS
Requirements:
Figure:
Figure 1: MCS Hypothetical Ongoing Contingencies during 7-year Baseline
Security Posture Time Frame:
United States Government Accountability Office:
Washington, DC 20548:
September 20, 2006:
Congressional Committees:
The National Security Strategy of the United States requires global
mobility through rapid, effective, and efficient projections of power
at home and abroad to deploy and sustain America's armed forces. To
improve its mobility capabilities, the Department of Defense (DOD)
plans to spend more than $50 billion from fiscal years 2006 through
2011 for aircraft, ships, ground transportation, prepositioned assets,
and other mobility assets. DOD has conducted several studies to
determine mobility requirements and recently completed a study of its
mobility capabilities and issued a report in December 2005. The intent
of the Mobility Capabilities Study (MCS) was to identify and quantify
the mobility capabilities needed to support U.S. strategic objectives
into the next decade. The MCS determined that the projected mobility
capabilities are adequate to achieve U.S. objectives with an acceptable
level of risk during the period from fiscal years 2007 through 2013;
that is, the current U.S. inventory of aircraft, ships, prepositioned
assets, and other capabilities are sufficient, in conjunction with host
nation support. The MCS emphasized that continued investment in the
mobility system, in line with current departmental priorities and
planned spending, is required to maintain these capabilities in the
future. This includes, for example, fully funding Army prepositioned
assets as planned and completing a planned reengineering of the C-5
aircraft. The MCS report also made recommendations to conduct further
studies, develop plans and strategies, and improve data collection and
mobility models. In fact, DOD officials told us that a Mobility
Capabilities Study--2006 is underway.
In the Senate report accompanying the bill for the fiscal year 2005
Defense Authorization Act, you asked us to monitor the process used to
conduct the MCS and report on the adequacy and completeness of the
study.[Footnote 1] Specifically, our objective was to determine whether
the MCS was adequate and complete. On March 1, 2006, we briefed your
staff on our preliminary observations. This report expands on that
briefing and makes recommendations to the Secretary of Defense.
To conduct our review of the MCS, we analyzed the final MCS report, the
MCS Terms of Reference and MCS Study Plan, as well as other DOD
policies and guidance concerning how DOD would conduct the MCS and the
databases and models used in the study. We identified generally
accepted research standards that define a sound and complete quality
study that were relevant to the MCS, and assessed the extent to which
the MCS report met these standards. We interviewed study officials,
study participants, and subject matter experts from several DOD
entities, including the combatant commands and the military services.
As we monitored the development of the MCS, we requested that DOD
provide documentation supporting and verifying key analytical and
decision-making processes.[Footnote 2] DOD officials could not produce
this documentation during the development of the MCS or following
issuance of the report. Consequently, we were unable to fully determine
whether the analytical and decision-making processes that we believe
are significant to the credibility of the study supported the MCS
effort and its conclusions. Our scope and methodology are discussed in
more detail in appendix I. We conducted our work from July 2004 through
July 2006 in accordance with generally accepted government auditing
standards.
Results in Brief:
DOD used an innovative approach in conducting the study and
acknowledged some methodological limitations in its report, as any
sound study should. However, it did not fully disclose how these
limitations could affect the MCS conclusions and recommendations.
Therefore, it is not transparent how the analyses done for the study
support DOD's conclusions. As measured against relevant generally
accepted research standards, we identified limitations in the MCS study
and report that raise questions about their adequacy and completeness.
Among our findings:
* Aspects of modeling and data were inadequate in some areas because
data were lacking and some of the models used could not simulate all
relevant aspects of the missions. The report did not explain how these
limitations could affect the study results or what the effect on the
projected mobility capabilities might be. Relevant research standards
require that models used are adequate for the intended purpose,
represent a complete range of conditions, and that data used are
properly generated and complete. For example, the MCS modeled
hypothetical homeland defense missions rather than homeland defense
demands derived from a well defined and approved concept of operations
for homeland defense, because the specific details of the missions were
still being determined, and DOD acknowledged that the data used may be
incomplete. The MCS also was unable to model the flexible deterrent
options/deployment order process to move units and equipment into
theater due to lack of data, but the study assumed a robust use of this
process, which in one scenario accounted for approximately 60 percent
of the airlift prior to beginning combat operations.[Footnote 3] In
addition, the MCS report contains more than 80 references to the need
for improved modeling, and 12 of these references call for additional
data or other refinements. Additionally, the MCS modeled the year 2012
to determine the transportation capabilities needed for the years 2007
through 2013. The year 2012 did not place as much demand for mobility
assets in support of smaller military operations, such as peacekeeping,
as other years. However, DOD officials considered 2012--the year
modeled--as "most likely" to occur and stated that statistically it was
not different from other years in the 2007 to 2013 period even though
the number of smaller military operations is the least of any of the
years reviewed.
* While the MCS concluded that combined U.S. and host nation
transportation assets were adequate to meet U.S. objectives with
acceptable risk, the report, in describing the use of warfighting
metrics in its analyses, does not provide a clear understanding of the
direct relationship of warfighting objectives to transportation
capabilities. Acknowledging this point, the report stated that further
analysis is required to understand the operational impact of increased
or decreased strategic lift on achieving warfighting objectives.
Relevant generally accepted research standards require that conclusions
be supported by analyses. The use of warfighting metrics is a measure
to determine whether combat tasks, such as achieving air superiority,
are achieved. However, they do not measure whether appropriate
personnel, supplies, and equipment arrived in accordance with
timelines. As a result, we could not determine how the study concluded
that planned transportation assets were adequate because the study did
not contain a transparent analysis to support its conclusion or a clear
roadmap in the report to help decision makers understand what that
conclusion meant in terms of type and number of mobility assets needed.
Previous DOD mobility studies primarily used mobility metrics, which
measured success in terms of tons of equipment and personnel moved per
day to accomplish military objectives. The use of both warfighting and
mobility metrics to measure success would allow decision makers to know
whether combat tasks were achieved and how much strategic
transportation is needed to accomplish those tasks.
* In some cases, the MCS results were incomplete, unclear, or
contingent on further study, making it difficult to identify findings
and evaluate evidence. Relevant research standards require results to
be presented in a complete, accurate, and relevant manner. For example,
the report contains several recommendations for further studies and
assessments, five of which are under way. However, DOD has no plans to
report the effect of these studies on the MCS results after the studies
are complete. In addition, the report contains qualified information
that is not presented clearly, such as varying assessments of
intratheater assets in three different places in the report. The lack
of clarity and conciseness of the reported results can limit the
study's usefulness to decision makers and stakeholders.
* Verification, validation, and accreditation (VV&A) of models and data
used to conduct the study was not complete because it was not done in
accordance with DOD policy or relevant research standards. Moreover,
relevant research standards state that a study report should include a
VV&A accreditation report that is signed by the study director and
addresses the models and data certification. DOD officials acknowledged
that they did not comply with DOD VV&A policy when using legacy models
in the MCS because they contended that long-term use of models and data
constitutes an equivalent VV&A process. Other than a description of the
process contained in the MCS report, DOD officials could provide no
additional documentation to verify and validate this equivalent process
to provide the assurance that models and data used in the MCS reduced
the risk inherent in modeling and simulation and added to the
credibility of the results. Moreover, officials could not provide
documentation to support key analytical and decision-making processes
used by senior DOD leadership, thus undermining the credibility of the
reported study results.
These limitations to the study's methodology raise questions concerning
the accuracy of the study's finding that projected capabilities are
adequate to achieve U.S. objectives with an acceptable level of risk.
Until DOD conducts an adequate and complete future MCS and clearly
discloses all limitations and their effects on the study results,
decision makers may be unable to clearly understand the operational
implications of the study results and make fully informed programmatic
investment decisions concerning mobility capabilities. We are
recommending that the Secretary of Defense, when conducting future
mobility capabilities studies, beginning with any study currently under
way, develop and use models and data for critical missions and
processes that are verified, validated, and accredited as required;
include in study reports an explanation of how stated limitations might
impact the study results and, at a minimum, describe how recommended
future studies might be conducted to enhance the results of the
original study; and incorporate both mobility and warfighting metrics
in determining capabilities.
In commenting on a draft of this report, DOD concurred with the first
and third recommendations and claimed that they did not understand the
second. We have clarified that recommendation to the Secretary of
Defense to include in study reports an explanation of how stated
limitations might impact the study results and, at a minimum, describe
how recommended future studies might be conducted to enhance the
results of the original study. In its comments, DOD also stated that
the report contained misleading information and factual errors. We
disagree with DOD's assertion. We did modify our report to respond to a
DOD technical comment related to homeland defense missions. DOD's
comments and our evaluation of them are discussed in the agency
comments section of this report.
Background:
The MCS was the first assessment of DOD's mobility system since 2000.
The study was designed to identify changes in DOD's transportation
force structure due to changes in threats and national security and
military strategies. The MCS is the fourth in a series of major
mobility studies that DOD has conducted since the end of the Cold War.
The first study, the Mobility Requirements Study, conducted in 1992,
was undertaken because of concern about the DOD's strategic mobility
capabilities in the wake of Operation Desert Shield and Operation
Desert Storm. That study established mobility requirements for the post-
Cold War era; defined baseline requirements for intertheater, or
strategic, mobility; and proposed a long-range investment plan to meet
these requirements. The Mobility Requirements Study Bottom-Up Review
Update, conducted in 1994, reaffirmed the need for increases in key
mobility components and validated the prior study's recommendation for
the procurement of additional ships for afloat prepositioning and for
surge deployments of forces based in the continental United States. The
Mobility Requirements Study--2005, issued in 2001, projected future
mobility requirements based on two nearly simultaneous major regional
contingencies. It included a broader range of factors, including host
nation support and enemy use of weapons of mass destruction, than the
previous studies.
The current MCS, which began in May 2004, reassessed DOD's mobility
capabilities against the backdrop of a revised National Military
Strategy that included the ongoing war against violent extremism, an
evolving global defense posture, a new force-sizing construct, revised
campaign scenarios, and ongoing departmentwide transformation efforts.
The study results were intended to support decisions on future
strategic airlift, aerial refueling aircraft, and sealift procurements
needed to meet varying military requirements. The study used an
innovative "capabilities-based" approach, measuring existing and
currently projected mobility capabilities against warfighting demands
that could be expected in fiscal year 2012 while also considering
mobility demands during the 7-year period from fiscal year 2007 through
fiscal year 2013. According to DOD officials, the Secretary of Defense
believed this approach would give him greater flexibility in deciding
which capabilities to fund in a constrained budget environment. In
considering each aspect of the National Military Strategy,[Footnote 4]
the MCS modeled warfighting scenarios in the year 2012 using different
premises with varying assumptions to develop and evaluate mobility
capability mix alternatives. The models were used to evaluate
transportation alternatives, including variations in alternative
transportation modes (air, land, sea) and sources (military, civilian,
foreign), as well as factors that affect transportation mode and source
decisions. The scope of the MCS described the study as an assessment of
the full range of transportation needs required to support (1) combat
operations; (2) smaller military efforts, such as peacekeeping or
overseas evacuation of American citizens; (3) homeland defense/civil
support, such as disaster relief and antiterrorism response; and (4)
other strategic missions, such as strategic nuclear and global strike
missions. The study was coauthored by the Office of the Secretary of
Defense, Office of the Director, Program Analysis and Evaluation
(PA&E), and the Chairman, Joint Chiefs of Staff, Office of the Director
of Logistics. Other DOD components involved in the study included the
U.S. Transportation Command and its subordinate commands, the Office of
the Under Secretary of Defense (Acquisition, Technology and Logistics),
the combatant commanders, the military services, and others. The final
report was signed on December 19, 2005, by the Deputy Secretary of
Defense.
Limitations in the MCS Study and Report Raise Questions about Adequacy
and Completeness:
As measured against relevant generally accepted research standards,
limitations in the MCS study and report raise questions about their
adequacy and completeness. For example, aspects of modeling and data
were inadequate in some areas because data were lacking and some of the
models used could not simulate all relevant aspects of the missions.
Furthermore, the exclusive use of warfighting metrics in the MCS
analyses limited the usefulness of the report. Moreover, in some cases
the MCS results were incomplete, unclear, or contingent on further
study, making it difficult to identify findings and evaluate evidence.
Finally, verification, validation, and accreditation of models and data
used to conduct the study were incomplete because they were not done in
accordance with DOD policy or relevant research standards, and
supporting documentation for key processes could not be provided.
Aspects of Modeling and Data Were Inadequate:
Aspects of modeling and data were inadequate in some areas because data
were lacking and some of the models used could not simulate all
relevant aspects of the missions. Relevant research standards require
that models used are adequate for the intended purpose, represent a
complete range of conditions, and that data used are properly generated
and complete. As DOD acknowledged in the MCS report as a study
limitation, some modeling tools were not available to analyze key
missions. The MCS cited deficiencies in several existing mobility
models and the need for follow-on MCS analyses. The MCS report also
acknowledged that the identified deficiencies in data hindered analysis
of future requirements and transportation system performance. However,
the report did not explain how these limitations could affect the study
results or what the effect on the projected mobility capabilities might
be.
For example, the MCS modeled hypothetical homeland defense missions
rather than homeland defense demands derived from a well defined and
approved concept of operations for homeland defense, because the
specific details of the missions were still being determined, and DOD
acknowledged that the data used are incomplete. The MCS report
recommended further analysis of mobility capabilities after homeland
defense needs are refined. However, the report did not identify the
potential effect that using these hypothetical scenarios might have on
the MCS results. The MCS also was unable to model the flexible
deterrent options/deployment order process to move units and equipment
into theater because of lack of data on how deployment orders have been
issued in the past for major combat operations. However, the MCS
assumed a robust use of the flexible deterrent option/deployment order
process, which in one scenario accounted for approximately 60 percent
of the early airlift movement prior to the beginning of combat
operations. Instead, the MCS modeled the flow of forces and equipment
contained in the time-phased force deployment data process.[Footnote 5]
Based on the scenarios provided for the MCS analyses, we could not
determine how the deployment order process would affect the mobility
assets required for major combat operations. The MCS report noted that
additional analysis is required to determine the implications of the
deployment order process and to provide sufficient information for
decision makers concerning the amount of future mobility assets
actually needed.
In addition to these modeling and data issues, the MCS report contains
more than 80 references to the need for improved modeling, and 12 of
these references call for additional data or other refinements, such as
follow-on analyses, further assessments, future evaluations, additional
study, and investigation of alternatives in a wide range of areas, such
as antiterrorism response, infrastructure availability, intratheater
airlift, strategic sealift, air refueling, and availability of civil
reserve aircraft. Some of these studies are currently underway, as
discussed later in this report.
Moreover, our analysis of the MCS report showed that the year modeled
(2012) to determine the DOD transportation capabilities needed for the
years 2007 through 2013 did not place as much demand for mobility
assets in support of smaller military operations, such as peacekeeping,
as other years. To establish transportation demands for mission
requirements, DOD developed and used a baseline security
posture[Footnote 6] that covered a 7-year period. This baseline was
developed, in part, using a historical analysis of DOD's movement of
personnel, equipment, supplies, and other items. According to DOD
officials, Office of the Under Secretary of Defense for Policy, which
developed the baseline security posture, selected the year modeled in
the MCS because it was deemed the "most likely" to occur in terms of
transportation demands and because it was not statistically different
from other years in the 7-year period. However, our analysis showed
that 2012 involved the least demand for transportation assets in
support of smaller military efforts than any year in the 7-year period
and did not fully stress the defense transportation system. Figure 1
depicts the number of hypothetical ongoing contingencies for each year
in the baseline as shown in the MCS.
Figure 1: MCS Hypothetical Ongoing Contingencies during 7-year Baseline
Security Posture Time Frame:
[See PDF for image]
Source: GAO's analysis of MCS notional lesser contingency schedule.
Note: A particular contingency may be ongoing in more than 1 year. Each
contingency has unique cargo and passenger requirements. For example, a
contingency that may be ongoing over a 2-or 3-year time frame may
require more or less mobility capability than a 6-month contingency.
[End of figure]
Although not transparent in the study, DOD officials said scenarios in
the year modeled were not intended to fully stress the defense
transportation system. DOD officials provided no further explanation
for the year selected to develop the DOD transportation capabilities
other than it was directed by Office of the Under Secretary of Defense
for Policy and agreed to by the study leadership. We believe that
selection of a different year that placed an increased demand on
transportation assets for smaller military efforts may have revealed
gaps in mobility requirements. Therefore, we found that the selection
of 2012 as the modeling year was a limitation in the MCS with respect
to smaller military efforts.
Because of these modeling and data limitations, the MCS may have
incorrectly estimated the future mobility requirements needed to
support homeland defense missions, major combat operations, and smaller
contingencies. Until DOD improves aspects of the modeling and data used
to conduct the MCS--to include defining its homeland defense mission
needs, developing models for the deployment order process, and
explaining how identified modeling and data limitations could affect
the study results--decision makers may not have adequate and complete
information about DOD's mobility capabilities.
Exclusive Use of Warfighting Metrics in MCS Analyses Limited Usefulness
of Report:
While the MCS concluded that combined U.S. and host nation
transportation assets were adequate to meet U.S. objectives with
acceptable risk, the report, in describing the use of warfighting
metrics in its analyses, does not provide a clear understanding of the
direct relationship of warfighting objectives to transportation
capabilities. The report acknowledged that further analysis is required
to understand the operational impact of increased or decreased
strategic lift on achieving warfighting objectives. Relevant generally
accepted research standards require that conclusions be supported by
analyses. The use of warfighting metrics is a measure to determine
whether combat tasks, such as establishing air superiority, are
achieved. However, they do not measure success in terms of whether
appropriate personnel, supplies, and equipment arrived in accordance
with timelines. As a result, we could not determine how the study
concluded that planned transportation assets were adequate because the
study did not contain a transparent analysis to support its conclusion.
In our opinion, it is important for decision makers to have an
understanding of both combat tasks that must be achieved and the amount
of transportation assets needed to achieve those tasks with some level
of success. This understanding would allow creation of a clear roadmap
for investment decisions. However, we could not determine how the study
calculated the specific numbers of transportation assets needed or
whether there are specific gaps, overlaps, or excesses in
transportation assets, a key objective of the study. Previous DOD
mobility studies, including the Mobility Requirements Study--2005,
primarily used mobility metrics, which measured success in terms of
tons of equipment and personnel moved per day to accomplish military
objectives. Million-ton-miles per day is a commonly accepted measure of
airlift performance and reflects how much cargo can be delivered over a
given distance in a given period of time based on the capability of
each type of aircraft. A similar mobility metric--short tons--is used
to measure ship capability. However, these studies did not fully
integrate combat tasks as a metric. The use of both warfighting and
mobility metrics to measure success would allow decision makers to know
whether there is sufficient capability to achieve warfighting
objectives, as well as to understand the number, type, and mix of
mobility assets that are actually needed.
Results Are Not Always Complete or Presented Clearly and Are Qualified
or Contingent on Further Study or Analysis:
In some cases, the MCS results were incomplete, unclear, or contingent
on further study, making it difficult to identify findings and evaluate
evidence. Relevant research standards require results to be presented
in a complete, accurate, and relevant manner; conclusions to be sound
and complete; and recommendations to be supported by analyses. Our
analysis of the MCS report found that it contains several
recommendations for further studies and assessments, five of which are
under way. The five studies address intratheater lift capabilities;
sealift petroleum, oil, and lubricants; logistics contingency
operations capabilities; aerial refueling; and integrated global
presence and basing. However, the report does not explain the potential
effect of these ongoing studies on the MCS results after the studies
are complete, nor does DOD have plans to report the effect of these
studies on the MCS results.
In addition, the report contains qualified information that is not
presented clearly in the report, such as varying assessments of
intratheater assets in three different places. For example, the report
states in the assessment section of the executive summary that
projected transportation assets are sufficient to address intratheater
demands in the fiscal years 2007 through 2013 time frame. However, in
the recommendations section of the executive summary, the report states
that DOD should take action to determine the proper mix of intratheater
assets needed to meet requirements. Then, in the part of the report
that discusses intratheater airlift, the report states that a detailed
analysis of intratheater airlift needs would require improved modeling
tools to accurately capture interactions among land, sealift, and
airlift capabilities and that data sets must be developed that
accurately describe the requirement in light of emerging concepts.
VV&A of Models and Data Was Not Complete:
VV&A of models and data used to conduct the study was not complete
because it was not done in accordance with DOD policy or relevant
research standards. DOD policy issued by the Under Secretary of Defense
for Acquisition, Technology and Logistics requires that DOD models and
data go through a VV&A process. Moreover, relevant research standards
state that a study report should include a VV&A accreditation report
that is signed by the study director and addresses the models and data
certification. DOD officials acknowledged that they did not comply with
the VV&A policy when using legacy models in the MCS because they
believed such an approach was not warranted for legacy models that have
been used for many years and have proved reliable. Moreover, these
officials believe that such long-term use constitutes a VV&A process
equivalent to that required in the policy. However, the DOD policy does
not specify that the actual use of a model constitutes an equivalent
VV&A process. VV&A of models and data reduces the risk inherent in the
use of models and simulations by improving the credibility of modeling
and simulation results. We previously reported our concerns that DOD
did not follow its policy in executing the MCS and had little
documentation to support the VV&A process used.[Footnote 7] We found
that the final MCS report contained a description of the equivalent
VV&A process, but DOD officials could provide no further documentation
to verify and validate this equivalent process other than the
description included in the report. We also found no documentation in
the study report to support DOD claims that the models have proven
reliable.
Furthermore, DOD officials were unable to provide documentation to
support and verify key analytical and decision-making processes used by
senior DOD leadership throughout the study. Relevant research standards
support documenting the study's analytical and decision-making
processes to ensure the accuracy, completeness, and credibility of
study results. DOD officials told us that the study's key analytical
and decision-making processes were validated and approved by study
participants during working group meetings and by senior leadership
during General Officer Steering Committee meetings and Executive
Committee meetings. PA&E officials could not produce documentation of
these meetings because they said documentation did not exist. Nor could
they produce other documents we requested during the development of the
MCS or following issuance of the report. Consequently, we were unable
to determine the adequacy and completeness of the analytical and
decision-making processes that supported the MCS effort and that we
believe are significant to the credibility of the study and its
conclusions.
Conclusions:
The methodological limitations in the MCS that we identified--some of
which were acknowledged by DOD in the MCS report--raise questions about
the adequacy and completeness of the study and its report. Until DOD
improves aspects of the modeling and data used to conduct the MCS--such
as defining its homeland defense mission needs and developing models
for the deployment order process--decision makers may not have adequate
and complete information about DOD's mobility capabilities to enable
them to make fully informed investment decisions. Furthermore, in the
absence of an explanation of how identified modeling and data
limitations could affect the study results or how such limitations
could affect projected mobility capability requirements, the accuracy
of the study's finding that projected capabilities are adequate to
achieve U.S. objectives with an acceptable level of risk during the
period from fiscal years 2007 through 2013 is unclear. Moreover,
without a transparent comparison between existing mobility assets and
projected needed assets, decision makers will be unable to use study
results to identify and quantify the specific types and mix of mobility
assets needed to address mobility gaps, overlaps, and excesses. Until
DOD conducts an adequate and complete future MCS and clearly discloses
all limitations and their effects on the study results, decision makers
likely will not have full information concerning DOD's mobility
capabilities. As a result, we suggest that Congress and other decision
makers exercise caution in using the MCS to make programmatic
investment decisions.
Recommendation for Executive Action:
To provide decision makers with adequate and complete information
concerning mobility capabilities so they are able to clearly understand
the operational implications of the study and make fully informed
programmatic investment decisions, and to improve the usefulness of
future mobility capabilities studies, we recommend that the Secretary
of Defense take the following three actions, when conducting future
mobility capabilities studies beginning with any study currently
underway:
* develop models and data for all critical missions, such as homeland
defense, and processes, such as the flexible deterrent options/
deployment order process;
* include in study reports an explanation of how stated limitations
might impact the study results and, at a minimum, describe how
recommended future studies might be conducted to enhance the results of
the original study; and:
* incorporate both mobility and warfighting metrics in determining
capabilities.
Agency Comments and Our Evaluation:
In commenting on a draft of this report, DOD concurred with our first
and third recommendations. DOD stated it did not understand our second
recommendation that the Secretary of Defense, when conducting future
mobility studies, beginning with any study currently underway, include
in study reports an explanation of how ongoing and follow-on studies
and modeling and data limitations that are referenced in the report
could affect the reported results. DOD also noted that it plans to
continue its ongoing efforts to enhance the models and data collection
processes used to assess mobility capabilities across the full range of
strategic missions and supports the notion that continual improvements
are needed. As we noted throughout our report, the MCS report contains
numerous and repeated references to the need for improved modeling and
additional data or other refinements, such as follow-on analyses,
further assessments, future evaluations, additional study, and
investigation of alternatives in a wide range of areas. DOD further
commented that while a completed study can recommend that follow-on
studies be conducted, it cannot explain how future studies might affect
the results of the current study. We acknowledge that DOD cannot
quantitatively predict the outcome of an ongoing or future study.
However, we believe DOD should be able to explain what ongoing follow-
on studies or evaluations seek to determine, what changes are being
made to the data inputs and modeling tools that are being used to
conduct the studies, and how DOD expects the results may differ from
current study results. While the explanation may be hypothetical, as
are many operations research study hypotheses, it can provide decision
makers with a better understanding of the current study's limitations
and results and how an ongoing or future study's results may differ.
Therefore, we refined our recommendation to recommend that the
Secretary of Defense, when conducting future mobility studies,
beginning with any study currently under way, include in study reports
an explanation of how stated limitations might impact the study results
and, at a minimum, describe how recommended future studies might be
conducted to enhance the results of the original study. For example, if
modeling and data are limitations to a study, the report should discuss
the ways in which the results might change with better models and data.
DOD also commented that our report contained misleading information and
factual errors and that it stands by the adequacy and completeness of
the MCS. The department provided examples in its technical comments
where it believed our report contained misleading information and
factual errors. We disagree with the department's comments regarding
the facts in our report and have addressed each of the department's
comments in appendix II.
Lastly, DOD stated that the MCS and its conclusions are well accepted
by the civilian and military leadership of the department, and pointed
out that in March 2006 testimony before the House Armed Services
Committee, the Commander, U.S. Transportation Command, stated that the
planned strategic airlift fleet determined by the MCS is "about the
right capacity". However we note that in the same hearing, the
Commander also stated that he thought DOD needed "somewhere in the
neighborhood of" 20 C-17 cargo aircraft beyond what is planned. We also
note that in the Commander's April 2006 testimony before the Senate
Armed Services Committee, Subcommittee on Sealift, he stated that, in
an internal Focused Mobility Analysis to study strategic mobility from
a Transportation Command perspective, the MCS will be the baseline,
"but we will explore how changes in key assumptions may impact the
analytical outcome."
We are sending copies of this report to the Secretary of Defense; the
Director of PA&E; and the Office of the Chairman, Joint Chiefs of
Staff. We will also make copies available to others upon request. In
addition, the report is available at no charge on the GAO Web site at
[Hyperlink, http://www.gao.gov]. If you or your staff have any
questions regarding the briefing or this report, please contact me at
(202) 512-8365 or solisw@gao.gov. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last
page of this report. GAO staff who made key contributions to this
report are listed in appendix III.
Signed by:
William M. Solis:
Director, Defense Capabilities and Management:
List of Congressional Committees:
The Honorable John Warner:
Chairman:
The Honorable Carl Levin:
Ranking Minority Member:
Committee on Armed Services:
United States Senate:
The Honorable Ted Stevens:
Chairman:
The Honorable Daniel K. Inouye:
Ranking Minority Member:
Subcommittee on Defense:
Committee on Appropriations:
United States Senate:
The Honorable Duncan L. Hunter:
Chairman:
The Honorable Ike Skelton:
Ranking Minority Member:
Committee on Armed Services:
House of Representatives:
The Honorable C.W. Bill Young:
Chairman:
The Honorable John P. Murtha:
Ranking Minority Member:
Subcommittee on Defense:
Committee on Appropriations:
House of Representatives:
[End of section]
Appendix I: Scope and Methodology:
To conduct our review of the Mobility Capabilities Study (MCS), we
reviewed and analyzed the final MCS report; the MCS Terms of Reference;
the MCS Study Plan; applicable Department of Defense (DOD) strategic
planning guidance; as well as other DOD guidance, directives,
instructions, and memos that describe how DOD would conduct its MCS. We
also reviewed the National Security Strategy of the United States of
America and the National Military Strategy of the United States of
America; DOD guidance concerning data collection, development, and
management in support of strategic analysis; DOD modeling and
simulation instruction; Defense Modeling and Simulation Office
guidance; descriptions of models used to conduct the study; and the
databases used in the models. We interviewed study officials from the
Office of the Secretary of Defense, Program Analysis and Evaluation
(PA&E), and the office of the Chairman, Joint Chiefs of Staff,
Logistics, as well as study participants and subject matter experts
from the U.S. Transportation Command, Air Mobility Command, Surface
Deployment and Distribution Command, the combatant commands, and the
military services concerning the extent of their input to the study. We
also interviewed officials from the Office of the Secretary of Defense,
Acquisition, Technology and Logistics, and the Modeling and Simulation
Technical Director at the Defense Modeling and Simulation Office.
Additionally, we reviewed research literature and DOD guidance and
identified frequently occurring, generally accepted research standards
that are relevant for defense studies such as the MCS that define a
quality or sound and complete study. The following were our sources for
these standards:
* GAO, Government Auditing Standards: 2003 Revision, GAO-03-673G
(Washington, D.C.: June 2003);
* GAO, Designing Evaluations, GAO/PEMD-10.1.4 (Washington, D.C.: March
1991);
* GAO, Dimensions of Quality, GAO/QTM-94-1 (Washington, D.C.: February
2004);
* RAND Corporation, RAND Standards for High-Quality Research and
Analysis (Santa Monica, Calif.: June 2004);
* Air Force Office of Aerospace Studies, Analysts Handbook: On
Understanding the Nature of Analysis (January 2000);
* Air Force, Office of Aerospace Studies, Air Force Analysis Handbook,
A Guide for Performing Analysis Studies: For Analysis of Alternatives
or Functional Solution Analysis (July 2004);
* Department of Defense, DOD Modeling and Simulation (M&S)
Verification, Validation, Accreditation (VV&A), Instruction 5000.61
(Washington, D.C.: May 2003);
* Department of Defense, Data Collection, Development, and Management
in Support of Strategic Analysis, Directive 8260.1 (Washington, D.C.:
December 2, 2003); and:
* Department of Defense, Implementation of Data Collection,
Development, and Management for Strategic Analyses, Instruction 8260.2
(Washington, D.C.: January 21, 2003).
During the process of identifying generally accepted research standards
we noted that not all studies are conducted the same way. For example,
while all studies use data, not all use baseline data. Likewise, all
studies require analyses, but not all use models or simulation to
conduct analyses. We tailored the research standards we identified as
relevant to the MCS, as shown in table 1.
Table 1: Generally Accepted Research Standards Relevant to MCS
Requirements:
Design: The Study is well designed:
I; Design: The Study is well designed: Study plan, scope, and
objectives follow Defense Planning Guidance.
I.a; Design: The Study is well designed: (Do the study scope and
objectives fully address the charter presented in the 2004 Defense
Planning Guidance?).
I.a.1; Design: The Study is well designed: Does the study plan address
specified guidance?.
I.b; Design: The Study is well designed: Was the study plan followed?.
I.c; Design: The Study is well designed: Were deviations from the study
plan explained and documented?.
I.d; Design: The Study is well designed: Was the study plan updatedover
the course of the study and the updates explicitly identified in the
study and updated study plan?.
II; Design: The Study is well designed: Assumptions and constraints are
reasonable and consistent:
II.a; Design: The Study is well designed: Are assumptions and
constraints explicitly identified?.
II.a.1; Design: The Study is well designed: (Are the study assumptions
necessary and reasonable?).
II.b; Design: The Study is well designed: Do the study assumptions
support a sound analysis?.
II.c; Design: The Study is well designed: Are the assumptions used in
analyses common throughout the study and models?.
II.d; Design: The Study is well designed: Do the assumptions contribute
to an objective and balanced research effort?.
III; Design: The Study is well designed: Scenarios and threats are
reasonable:
III.a; Design: The Study is well designed: Are scenarios traceable back
to formal guidance?.
III.b; Design: The Study is well designed: Were the threat scenarios
validated and Defense Intelligence Agency approved and documented?.
III.c; Design: The Study is well designed: Do scenarios represent a
reasonably complete range of conditions?.
III.d; Design: The Study is well designed: (Were the threats varied to
allow for the conduct of sensitivity analysis?).
Execution: The study is well executed.
IV; Design: The Study is well designed: Methodology is successfully
executed:
IV.a; Design: The Study is well designed: Was the study methodology
executed consistent with the (MCS) study plan and schedule?.
IV.b; Design: The Study is well designed: (Does the methodology support
accomplishing the objectives presented in the study plan?).
IV.c; Design: The Study is well designed: Were the models used to
support the analyses adequate for their intended purpose?.
IV.d; Design: The Study is well designed: Were the model input data
properly generated to support the methodology?.
V; Design: The Study is well designed: (Analytical ) Baseline data and
other data used to support study and analyses validated, verified, and
approved:
V.a; Design: The Study is well designed: Is the (analytical) baseline
fully and completely identified and used consistently throughout the
study for the various analyses?.
V.b; Design: The Study is well designed: Were data limitations
identified (and the impact of the limitations fully explained?).
V.c; Design: The Study is well designed: Were the (baseline security
posture) data verified and validated?.
V.d; Design: The Study is well designed: Was the data verification and
validation process documented?.
VI; Design: The Study is well designed: Models, simulations, and
verification, validation, and accreditation are reasonable:
VI.a; Design: The Study is well designed: Was a VV&A accreditation
report that addresses the models and data certification signed by the
study director and included in the report?.
VI.b; Design: The Study is well designed: Were modeling and simulation
limitations identified and explained?.
VI.c; Design: The Study is well designed: Has each model in the study
been described?.
VI.d; Design: The Study is well designed: Are the model processes
clearly explained, documented and understood?.
VII; Design: The Study is well designed: Measures of effectiveness
(MOEs) and essential elements of analysis (EEAs) are addressed:
VII.a; Design: The Study is well designed: (Do MOEs adhere to the
guidance in the study terms of reference?).
VII.b; Design: The Study is well designed: (Are the MOEs fully
addressed in the study?).
VII.c; Design: The Study is well designed: (Are the EEAs addressed in
the study?).
Presentation of results: Timely, complete, accurate, concise, and
relevant to the client and stakeholders;
VIII; Design: The Study is well designed: Presentation of results
support findings:
VIII.a; Design: The Study is well designed: Does the report address the
objectives?.
VIII.b; Design: The Study is well designed: Does the report present an
assessment that is well documented and conclusions that are supported
by the analyses?.
VIII.c; Design: The Study is well designed: Are conclusions sound and
complete?.
VIII.d; Design: The Study is well designed: Are recommendations
supported by analyses?.
VIII.e; Design: The Study is well designed: Is a realistic range of
options provided?.
VIII.f; Design: The Study is well designed: Are the study results
presented in the report in a clear manner?.
VIII.g; Design: The Study is well designed: Are study participants/
stakeholders (i.e., services and Combatant Commands) informed of the
study results and recommendations?.
Source: GAO analysis of industry and DOD study and research standards.
[End of table]
We used these relevant standards as our criteria to assess the reported
MCS results. All eight key areas of the study process were considered
to have equal importance relative to the soundness and completeness of
the study; that is, a sufficiently serious concern in any category
could raise questions concerning the adequacy and completeness of the
report. The analysts independently reviewed evidence relevant to each
subquestion, including the study itself, the study Terms of Reference,
and its strategic planning guidance. For each of the subquestions in
the key study process areas, the analysts determined whether (1) the
evidence had no limitations or raised no concerns, (2) the evidence had
some limitations or raised some concerns, (3) the evidence had
significant limitations or raised significant concerns, or (4) we could
not determine the extent of limitations or concerns because there was
not sufficient information. The analysts then met, compared, and
discussed their individual assessments, and reached an overall
assessment for each subquestion. Areas of the study where we identified
either "some" or "significant" limitations or concerns were considered
to affect the adequacy or completeness of the study. Additionally,
areas of the study that could not be assessed because of the lack of
supporting documentation were considered to affect the credibility of
the study.
Throughout our review PA&E officials told us that the documentation
needed to support and verify the key analytical and decision-making
processes used to conduct the MCS, documentation that was vetted and
approved by DOD leadership and all of the study participants, would not
be completed and available for our review until the study report was
issued. However, after the report was issued, we were told that the
report provides all of the supporting documentation needed and that the
other documentation we requested could not be provided. As a result, we
were unable to determine the adequacy and completeness of the
analytical and decision-making processes that supported the MCS effort
to evaluate the credibility of the study. We believe these processes
are significant to the credibility of the study and its results.
We conducted our review from July 2004 through July 2006 in accordance
with generally accepted government auditing standards.
[End of section]
Appendix II: Comments from the Department of Defense:
Deputy Secretary Of Defense:
1010 Defense Pentagon:
Washington, DC 20301-1010:
Aug 30 2006:
The Honorable David M. Walker:
Comptroller General of the United States:
U.S. Government Accountability Office:
441 G Street, N.W.
Washington, DC 20548:
Dear Mr. Walker:
This is the Department of Defense (DoD) response to the GAO draft
report, GAO-06-938, "Defense Transportation: Study Limitations Raise
Questions about the Adequacy and Completeness of the Mobility
Capabilities Study (MCS) and Report," dated August 21, 2006 (GAO Code
350558). DoD's responses to the report's recommendations are enclosed.
The GAO draft report contains misleading information and factual
errors. The Department stands by the adequacy and completeness of the
MCS, which is one of the most comprehensive capability studies ever
conducted by the Department of Defense. The data and models used by the
study are sound, and the results of the study are valid. Furthermore,
in its groundbreaking assessment of homeland defense mission needs, the
MCS accurately reflects the Department's most current understanding of
this developing national mission. Lastly, MCS insights - drawn from the
study's broad array of mobility metrics and detailed operational
simulations - significantly enhanced senior leaders' deliberations
during the 2005 Quadrennial Defense Review.
The MCS was a collaborative effort led by the Office of the Secretary
of Defense and the Joint Staff, and its conclusions are well accepted
by the civilian and military leadership of the Department. In his March
2006 Congressional testimony, the Commander, U.S. Transportation
Command acknowledged the study's finding by stating that the planned
strategic airlift fleet of 292 aircraft "is about the right capacity."
Likewise, during a press conference in November 2005, the Vice Chairman
of the Joint Chiefs of Staff expressed his support for the study,
citing the MCS finding that the Department has "a very capable and
adequate airlift fleet."
Signed:
Enclosure:
As stated:
Unclassifed:
GAO Draft Report - Dated August 21, 2006 GAO CODE 350558/GAO-06-938:
"Defense Transportation: Study Limitations Raise Questions about the
Adequacy and Completeness of the Mobility Capabilities Study and
Report"
Department Of Defense Comments To The Recommendations:
Recommendation 1: The GAO recommended that the Secretary of Defense,
when conducting future mobility capabilities studies, beginning with
any study currently underway, develop models and data for all critical
missions, such as homeland defense, and processes, such as the flexible
deterrent options/deployment order process.
DOD Response: DoD concurs. The Department plans to continue its ongoing
efforts to enhance the models and data collection processes used to
assess mobility capabilities across the full range of strategic
missions. The models used in the MCS are sound and have consistently
produced valid results in the four mobility studies conducted by DoD
since the end of the Cold War. Furthermore, as recommended in the MCS
report, DoD supports the notion that continual improvements are needed
to provide enhanced analytic tools for the Department as it strives to
address complex real-world processes.
Recommendation 2: The GAO recommended that the Secretary of Defense,
when conducting future mobility studies, beginning with any study
currently underway, include in study reports an explanation of how
ongoing and follow-on studies and modeling and data limitations that
are referenced in the report could affect the reported results.
DOD Response: Dod does not understand this recommendation. While a
completed study can recommend that follow-on studies be conducted, it
cannot explain how future studies might affect the results of the
current study.
Recommendation 3: The GAO recommended that the Secretary of Defense,
when conducting future mobility capabilities studies, beginning with
any study currently underway, incorporate both mobility and warfighting
metrics in determining capabilities.
DOD Response: DoD concurs. In fact, the MCS employed a rigorous process
by which the Services, Combatant Commands, and the Joint Staff
collaboratively developed warfighting metrics. These metrics were used
along with a comprehensive set of mobility metrics for airlift,
sealift, surface transport, and prepositioned equipment to determine
the adequacy of the full spectrum of mobility capabilities.
GAO Draft Report --Dated August 21, 2006 GAO CODE 350558/GAO-06-938:
"Defense Transportation: Study Limitations Raise Questions about the
Adequacy and Completeness of the Mobility Capabilities Study and
Report"
Department Of Defense Technical Comments:
The following are three examples of factual errors and misleading
information contained in the draft GAO report:
1. Page 3: "The [modeled] year of 2012 did not place as much demand for
mobility assets in support of smaller military operations, such as
peacekeeping, as other years."
Response: False. The seven-year demand (2007-2013) developed as part of
the Baseline Security Posture, and used by the MCS, does not have
significant variance from year to year. 2012 demand is not
significantly less than 2009, and is larger than 2013.
It is important to note the fact that the MCS modeled a surge demand on
the mobility system in 2012 that far exceeds anything this nation has
experienced since World War II, In doing so, the study correctly
applied the Department's strategic planning framework (1-4-2-1) and its
guidance with respect to the conduct of concurrent lesser contingencies
during overlapping war fights. GAO has incorrectly focused on the
number of operations, not the level of effort.
2. Page 4: "The use of war fighting metrics is a measure to determine
whether combat tasks, such as achieving air superiority, are achieved.
However, they do not measure whether appropriate personnel, supplies,
and equipment arrived in accordance with timelines."
Response: False. The war fighting metrics developed by the MCS do
measure whether appropriate personnel, supplies, and equipment arrived
in accordance with timelines.
In fact, the MCS employed a rigorous process to develop war fighting
metrics that were used along with a comprehensive set of mobility
metrics for airlift, sealift, and surface transport. These metrics were
used to determine if the modeled war fights accomplished the
commander's objectives within the right timelines. As we explained to
GAO, being able to achieve a desired task within the desired timeline
requires the appropriate personnel, supplies and equipment to be in
place on time, indicating that the transportation capabilities are
adequate.
3_ Page 3: "The MCS also was unable to model the flexible deterrent
options/deployment order process . .
Response: False. The MCS DID analyze flexible deterrent option (FDO)
movements to the theater. As part of the MCS analysis, the study
modeled the flow of forces in response to rising tensions in various
regions of the world, Forces were flowed to the region in anticipation
of operations before war plans were executed. These flexible deterrent
options were included in the MCS analysis. What the study did not do
was model the deployment order process used in OIF. Rather, the study
used the time-phased force deployment data (TPFDD) process as the
accepted methodology for flowing follow-on forces.
4. Page 3: "The MCS modeled hypothetical homeland defense missions
rather than actual homeland defense demands because the specific
details of the missions were still being determined, and DoD
acknowledged that the data used may be incomplete."
Response: The statement is misleading. We are not sure what the report
means by "actual homeland defense demands." Actual demands are
encountered in response to actual events. The MCS was intended to
inform the DoD leadership concerning the impact of potential demand on
the mobility system, and to assess the risks associated with different
potential demand levels. The study used the latest approved homeland
defense scenarios developed by NORTHCOM and PACOM, as well as the
Department's current planning guidance, to determine the range of
mobility assets needed to support a range of missions. Given the many
unknowns associated with homeland defense, the study assessed low,
moderate, and high levels of DoD support for these missions.
5. Page 3: "Aspects of modeling and data were inadequate."
Response: This statement is misleading. These are the same models that
the Department has relied on to complete three previous mobility
studies since the end of the Cold War. Furthermore, the Department has
dedicated significant resources to make sure that these models and the
data collection processes are adequate. The fact that the study
recommends future improvements was meant to focus future enhancements
as the Department strives to address increasingly complex real world
issues. The MCS models and data were adequate to assess relevant
aspects of the missions required to support the National Military
Strategy.
GAO's Responses to DOD's Technical Comments:
1. DOD disagreed with our assessment that the modeled year--2012--did
not place as much demand for mobility assets in support of smaller
military operations, such as peacekeeping, as other years. DOD also
stated that we incorrectly focused on the number of operations, not the
level of effort. We disagree. The MCS report (Annex A to Appendix F)
made no distinction between the number of lesser contingencies and the
level of effort. Specifically, the Vignettes for Baseline Security
Posture Analysis did not report the level of effort by year and instead
aggregated the data, in many instances across several modeled years.
Consequently, we compared the number of operations conducted in the
model year. Throughout our review, PA&E officials consistently told us
that the completed MCS report would contain all the documentation
needed to support its analyses. Furthermore, although demand in the
modeled year may exceed previous efforts, the MCS was chartered to
assess the ability of the mobility system to support the National
Military Strategy into the next decade. The size of the selected model
year in relation to efforts conducted between 1941 and 2006 is not at
issue. As our report makes clear, our concern is that modeling what
appears to be the least demanding year does not address whether the
United States has sufficient capability to support national objectives
during a peak demand period and may underestimate and underreport
demands to senior decision makers.
2. DOD disagreed with our observation that the MCS report does not
provide a clear understanding of the direct relationship of warfighting
objectives to transportation capabilities. We disagree. We understand
that achieving a combat task requires delivering the right commodity to
the right place at the right time. However, the specific combat tasks
(e.g., attaining air superiority) necessary to satisfy the commander's
campaign objectives are not a direct measure of mobility capability.
For example, the problems in using a single metric are reflected in the
MCS Appendix H, where the MCS report states that "the study itself
still had difficulty in evaluating the operational impact of the
delivery of theater support elements," adding that "we [DOD] were
unable to develop a satisfactory mechanism to capture the linkage of
the closely related, but delayed, follow-on support needed." Finally,
the MCS concludes that "there was no way to model a decrease in [Air
Force] squadron effectiveness if this support was late. Additional
effort is required to develop a methodology for evaluation the
operational impact of support equipment availability." We continue to
believe, and DOD agreed with our recommendation, that warfighting
metrics, in conjunction with mobility metrics, can give decision makers
a full picture of the capabilities needed to meet a specific
warfighting goal.
3. DOD disagreed with our statement that the MCS was unable to model
the flexible deterrent options/deployment order process and that the
study in fact analyzed flexible deterrent option (FDO) movements to
theater. We do not dispute that DOD analyzed FDO movements as part of
the MCS analysis. However, the degree to which the MCS analyses
successfully modeled FDOs is in question. The MCS report, Appendix H,
stated that an individual FDO is essentially the same as a deployment
order. It also states in that section that "Deployment orders [DEPORDS]
are issued to deploy specific capabilities as commitment decisions are
made. This was not modeled due to lack of data on how DEPORDS would
have been issued for an MCO [major combat operation] deployment." In
the same paragraph, the MCS concludes that "the impact on the mobility
system of the DEPORD process should be assessed in follow-on MCS
analyses," adding that "there is a data deficit on how to model and
execute a DEPORD process." Furthermore, the MCS report states that
"additional analysis is required to investigate the implications of the
DEPORD process decisions and provide data for future decision-makers to
develop a DEPORD execution process."
4. DOD believes our statement concerning homeland defense missions is
misleading and is not sure what the report means by "actual homeland
defense demands." We removed the word "actual" and clarified our report
to discuss "demands derived from a well defined and approved concept of
operations for homeland defense", which were not available for the
study according to the MCS report. Furthermore, in chapter 4, the MCS
report states that "maintaining a dedicated capability to support
multiple, nearly simultaneous homeland defense/civil support events
concurrent with the peak demand period of two overlapping warfights,
greatly exceeds programmed lift capabilities". This raises questions
about the conclusions of the MCS that there are adequate mobility
capabilities to meet national security objectives. Also, in Chapter 3
of the MCS report, it states that the DOD homeland defense concept of
operations required refinement and was one of nine issue areas cited
within the homeland defense portion of the study that "need to be
addressed and investigated in more detail." All of these nine areas
potentially impact mobility support for homeland defense operations.
Moreover, the MCS Executive Summary notes that reassessment of these
missions is required as DOD's role in homeland defense evolves. The MCS
report, chapter 4, concludes by calling for further refinement of
mission requirements, continuing risk assessments, and an effort to
determine corresponding mobility solutions. We continue to believe that
the MCS conclusion that adequate mobility capability exists is
conditional given the results of the homeland defense portion of the
study and that the accuracy and completeness of the data, modeling, and
results for this portion of the MCS remain in question.
5. We disagree with DOD's characterization that our information was
"misleading" regarding the adequacy of some aspects of the MCS'
modeling and data. Furthermore, we continue to disagree with DOD's
statement that the models and data used by the study were sound and
adequate to assess relevant aspects of missions required to support the
National Military Strategy, and that the results of the study are
valid. In this report, as in our September 14, 2005 report,[Footnote 8]
we reaffirm our concern that the data and models used by the study may
not be sound and the results may not be valid since the verification,
validation, and accreditation (VV&A) of the models and data used to
conduct the study was not done in accordance with DOD policy or
relevant research standards. VV&A of models and data reduces the risk
inherent in the use of models and simulations by improving the
credibility of modeling and simulation results. We do not dispute DOD's
assertion that it has relied upon the same models to produce mobility
studies done "since the end of the Cold War". However, as we discuss in
our report, the MCS report fails to explain or qualify the impact that
identified data or modeling limitations might have on its results. For
example, in the MCS chapter 4, entitled Operational Data, the MCS
states that "data deficiencies negatively affected MCS's ability to use
current execution data to project future requirements and assess system
performance." Unclear is the extent to which these deficiencies
impacted the MCS ability to meet the objective of identifying mobility
capability gaps, overlaps, or excesses and provide associated risk
assessments, an MCS objective. Similarly, in the section of chapter 4
entitled Analysis Tools, the report states that "MCS analysis revealed
several deficiencies in existing mobility models." The section
concludes with five recommended tool enhancements but it does not
explain the impact that the absence of these enhanced tools may have.
We continue to believe that because of these modeling and data
limitations, the MCS may have incorrectly estimated the future mobility
requirements needed to support homeland defense missions, smaller
contingencies, and major combat operations.
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
William M. Solis, (202) 512-8365 or solisw@gao.gov:
Acknowledgments:
Ann Borseth, Assistant Director; Brian Lepore, Assistant Director;
Nabajyoti Barkakati; Renee Brown; Claudia Dickey; Ron La Due Lake;
Oscar Mardis; Deborah Owolabi; Kenneth Patton; and Stephen Woods made
significant contributions to this report.
FOOTNOTES
[1] S. Rep. 108-260, at 126 (2004).
[2] Examples of the documentation we requested to support and verify
key analytical and decision-making processes used by DOD to conduct the
MCS included (1) the accreditation report and supporting documentation
or evidence of the verification, validation, and accreditation process
for the models and data used in the MCS; (2) copies of MCS working
group meeting minutes that verify and validate the analytical processes
the various MCS study teams and study participants used to vet and
agree upon data, scenarios, assumptions, models, and associated risk;
and (3) copies of MCS General Officer Steering Committee and Executive
Committee meeting minutes that verify and validate the analytical and
decision-making processes the DOD senior leadership used to vet and
agree upon the key data, scenarios, assumptions, models, and associated
risk used to conduct the MCS, as well as agreement with the study
results.
[3] Deployment orders are issued to deploy specific capabilities as
commitment decisions are made, rather than a deploying unit's full set
of equipment or capabilities. Flexible Deterrent Options (FDOs) provide
escalation options during the initial stages of a conflict. FDOs are
employed under certain conditions to deter adversarial actions contrary
to U.S. interests.
[4] The 2004 National Military Strategy of the United States calls for
a force sized to defend the homeland, proactively dissuade adversaries
in and from four global regions, and conduct two overlapping "swift
defeat" campaigns. Even when committed to a limited number of lesser
contingencies, the force must be able to "win decisively" in one of the
two campaigns. This "1-4-2-1" force-sizing construct places a premium
on increasingly innovative and efficient methods to achieve objectives.
[5] In joint military planning, time-phased force deployment data are
defined as a computer database that contains detailed personnel and
cargo planning data; it usually includes priority and sequencing of
deploying forces.
[6] The baseline security posture projects the position from which
combatant commanders will perform future missions, including how they
will address the global war on terrorism, ongoing operations, and other
day-to-day activities to which U.S. forces remain committed and from
which they are not likely to disengage entirely.
[7] See GAO, Defense Transportation: Opportunities Exist to Enhance the
Credibility of the Current and Future Mobility Capabilities Studies,
GAO-05-659R (Washington, D.C.: Sept. 14, 2005), for a more detailed
discussion.
[8] See GAO, Defense Transportation: Opportunities Exist to Enhance the
Credibility of the Current and Future Mobility Capabilities Studies,
GAO-05-659R (Washington, D.C., September 14, 2005), for a more detailed
discussion.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
