Defense Transportation
Opportunities Exist to Enhance the Credibility of the Current and Future Mobility Capabilities Studies Gao ID: GAO-05-659R September 14, 2005We are reviewing the processes the Department of Defense (DOD) is using to conduct its Mobility Capabilities Study (MCS). The MCS is to address changes in DOD's transportation force structure and mobility requirements due to changes in threats and certain national security and military strategies. The study results may underpin decisions on future strategic airlift, aerial refueling aircraft, and sealift procurements. The study relies on the use of various models and data inputs to develop and evaluate transportation alternatives, including variations in alternative transportation modes (air, land, sea) and sources (military, civilian, foreign), as well as factors that affect transportation mode and source decisions. The Senate Armed Services Committee directed us to monitor the conduct of the MCS and report on the adequacy and completeness of the report no later than 30 days after DOD completes the study. DOD plans to issue the MCS report during 2005. This letter is intended to bring to the Secretary of Defense's attention preliminary observations on certain aspects of the MCS methodology to permit you to ensure the credibility of this and future studies. In our letter, we address the adequacy of the department's verification, validation, and accreditation (VV&A) of the models and simulations being used to conduct the MCS--that is, the process the MCS team is using to identify the models' capabilities, limitations, and performance relative to the real world events they simulate. We will continue to monitor the MCS and will report on the adequacy and completeness of the methodology after DOD issues its report.
We are unable to assess the adequacy of the process DOD used to verify, validate, and accredit the models used to conduct the MCS. Although officials in the Office of Program Analysis and Evaluation stated that they have performed an equivalent VV&A process for the models used in the MCS, there is little documentation available to describe the equivalent process that was used. An adequate evaluation of this self-described equivalent process cannot be conducted due to this absence of documentation, which is compounded because DOD currently does not plan to disclose how it conducted its equivalent VV&A process in its MCS report. This could negatively impact the credibility of the MCS report. DOD guidance requires that models and data go through a VV&A process, but officials in the Office of Program Analysis and Evaluation believe that this guidance is not relevant for models that have been used for many years, called legacy models, because, in their view, the models and data have already undergone an equivalent VV&A process consisting of actual use, although the guidance does not identify actual use as an appropriate equivalent process. Moreover, DOD was conducting VV&A on one legacy model being used in the MCS, raising questions about the need for such actions given the department's statements that it is unnecessary.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-05-659R, Defense Transportation: Opportunities Exist to Enhance the Credibility of the Current and Future Mobility Capabilities Studies
This is the accessible text file for GAO report number GAO-05-659R
entitled 'Defense Transportation: Opportunities Exist to Enhance the
Credibility of the Current and Future Mobility Capabilities Studies'
which was released on September 14, 2005.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
September 14, 2005:
The Honorable Donald H. Rumsfeld:
Secretary of Defense:
Subject: Defense Transportation: Opportunities Exist to Enhance the
Credibility of the Current and Future Mobility Capabilities Studies:
Dear Mr. Secretary:
We are reviewing the processes the Department of Defense (DOD) is using
to conduct its Mobility Capabilities Study (MCS). The MCS is to address
changes in DOD's transportation force structure and mobility
requirements due to changes in threats and certain national security
and military strategies. The study results may underpin decisions on
future strategic airlift, aerial refueling aircraft, and sealift
procurements. The study relies on the use of various models and data
inputs to develop and evaluate transportation alternatives, including
variations in alternative transportation modes (air, land, sea) and
sources (military, civilian, foreign), as well as factors that affect
transportation mode and source decisions.
The Senate Armed Services Committee directed us to monitor the conduct
of the MCS and report on the adequacy and completeness of the report no
later than 30 days after DOD completes the study.[Footnote 1] As you
may be aware, DOD plans to issue the MCS report during 2005. This
letter is intended to bring to your attention preliminary observations
on certain aspects of the MCS methodology to permit you to ensure the
credibility of this and future studies. In our letter, we address the
adequacy of the department's verification, validation, and
accreditation (VV&A) of the models and simulations being used to
conduct the MCS--that is, the process the MCS team[Footnote 2] is using
to identify the models' capabilities, limitations, and performance
relative to the real world events they simulate. We will continue to
monitor the MCS and will report on the adequacy and completeness of the
methodology after DOD issues its report.
To do our work, we reviewed applicable DOD guidance, directives,
instructions, and memos that describe how DOD would conduct its
mobility capabilities assessments to include the National Security and
Military Strategies; DOD Strategic Planning Guidance; DOD data
collection, development, and management in support of strategic
analysis directives; DOD modeling and simulation instruction; Defense
Modeling and Simulation Office guidance; MCS Study Plan and Terms of
Reference; descriptions of models used to conduct the study; and the
databases used in the models. Additionally, we reviewed and analyzed
previously published DOD mobility studies and past GAO reports related
to the studies. We interviewed study officials from the Office of the
Secretary of Defense, Program Analysis and Evaluation and the office of
the Joint Chiefs of Staff, as well as study participants and subject
matter experts from the U.S. Transportation Command, Air Mobility
Command, Surface Deployment and Distribution Command, the combatant
commands, and military services concerning the extent of their input to
the study. We interviewed a modeling and simulation subject matter
expert at the Defense Modeling and Simulation Office. We interviewed
DOD officials to try and identify the process used to ensure the
validation and verification of the models and the reliability of the
data used in the study models, and conducted a comparative analysis of
this process with applicable DOD guidance. We did not evaluate the
relevancy of the DOD guidance because it was outside the scope of our
work. We conducted this portion of our review from July 2004 through
July 2005 in accordance with generally accepted government auditing
standards.
Results in Brief:
We are unable to assess the adequacy of the process DOD used to verify,
validate, and accredit the models used to conduct the MCS. Although
officials in the Office of Program Analysis and Evaluation stated that
they have performed an equivalent VV&A[Footnote 3] process for the
models used in the MCS, there is little documentation available to
describe the equivalent process that was used. An adequate evaluation
of this self-described equivalent process cannot be conducted due to
this absence of documentation, which is compounded because DOD
currently does not plan to disclose how it conducted its equivalent
VV&A process in its MCS report. This could negatively impact the
credibility of the MCS report. DOD guidance requires that models and
data go through a VV&A process, but officials in the Office of Program
Analysis and Evaluation believe that this guidance is not relevant for
models that have been used for many years, called legacy models,
because, in their view, the models and data have already undergone an
equivalent VV&A process consisting of actual use, although the guidance
does not identify actual use as an appropriate equivalent process.
Moreover, DOD was conducting VV&A on one legacy model being used in the
MCS, raising questions about the need for such actions given the
department's statements that it is unnecessary.
We are making recommendations to improve DOD's documentation of any
equivalent VV&A process the department may have and to establish the
relevancy of VV&A guidance for use with legacy models. In commenting on
a draft of this report, DOD concurred or partially concurred with all
of our recommendations. DOD's comments are reprinted in their entirety
in enclosure I.
Background:
The Office of the Secretary of Defense directed its Office of Program
Analysis and Evaluation to conduct the MCS. DOD was using an array of
models and baseline data to develop transportation alternatives and
evaluate their impact on the department's capability to support
military strategy. As with past mobility requirements studies,[Footnote
4] the MCS uses a variety of models and data analyses to achieve the
overall study objectives[Footnote 5] and determine the effect of the
study variables on the defense transportation system and its resultant
effect on the capabilities required to meet the mobility needs for all
aspects of the National Military Strategy. The baseline data used in
the mobility models are the foundations for the strategic analyses and
contain such data as a specific warfight scenario, concept of
operations for the scenario, needed forces and equipment, battlefield
terrain and weather, and time frames. According to DOD officials, the
models have become increasingly complex over the past 15 years and are
used to analyze large volumes of data to define mobility requirements,
assess risk based on the forces' ability to achieve war-fighting
objectives, identify mobility gaps, and determine alternative methods
to achieve desired capabilities. For example, the analysis would
identify tons of equipment or number of passengers to be moved, the
number of aircraft and ships needed to move equipment and forces, and
the number of aircraft to be aerially refueled.
Modeling and simulation are assuming a larger role in military
assessments, driven in part by an appreciation for the cost, logistics,
and acquisition implications associated with DOD programs. DOD models
and simulations are to be developed in accordance with DOD policies,
plans, and guidance.[Footnote 6] Generally, overall VV&A policy is
established in DOD modeling and simulation master plans. DOD and
service instructions clarify the policy and guidance indicates how to
implement the policy. VV&A constitutes processes that gather and
evaluate evidence to determine, based on the simulation's intended use,
the simulation's capabilities, limitations, and performance relative to
the real objects or events it simulates.
The VV&A process entails the review, analysis, evaluation, and testing
of models and simulations, incrementally over time as the models are
being developed, by an independent agent or authority to improve the
credibility of the process. Furthermore, VV&A provides enhanced user
confidence, improved performance and reliability for the subject model
and simulation results, more predictable and accurate
modeling/simulation behavior, and reduced risk of inaccurate model
outputs. Verification is the process of determining that a model
implementation and its associated data accurately represent the
developer's conceptual design. Validation is generally understood as an
independently administered process where multiple parties that have no
vested interest in the outcomes participate in developing (1) an
appropriate set of standard protocols for a simulation and (2) protocol
reviews across several occasions and settings. Generally, before formal
validation is applied, the goals of the simulation's performance are
thoroughly developed and specified. The validation process establishes
the credibility of a simulation by evaluating its capability and
accuracy relative to its intended use. Successfully completing
validation enhances the credibility of the simulation by offering
assurances that it can be relied on for reproducible results
appropriate for its objectives. Additionally, data validation is to be
put in the context of its suitability for use in models. For this
reason, the data cannot be validated independently of the models for
which they are intended. The results of the verification and validation
phase are used to support the accreditation decision, which is the
user's official certification that a model, simulation, or federation
of models and simulations and the associated data are acceptable for
use for a specific purpose.
Documentation of VV&A Process Is Lacking:
Officials in the Office of Program Analysis and Evaluation stated that
they have performed an equivalent VV&A process for the legacy models
used in the MCS, but there is little documentation available to
describe the equivalent process that was used. As a result, we are not
able to assess the adequacy of DOD's self-described equivalent process.
Office of Program Analysis and Evaluation and Joint Data Support
officials told us that most of the documentation does not yet exist and
will not be completed until after the MCS is completed. Moreover, at
the time of our review, DOD had not planned to describe the equivalent
VV&A process in its MCS report. The absence of VV&A documentation for
the models and data used to conduct the MCS and the lack of disclosure
in the published MCS report could limit the study's credibility.
DOD guidance, issued by the Under Secretary of Defense for Acquisition,
Technology and Logistics, requires that DOD models and data go through
a VV&A process. The Office of Program Analysis and Evaluation
acknowledged that it did not comply with the guidance because it
believes such an approach is not warranted for legacy models that have
been used for many years. Moreover, these officials believe that such
long-term use constitutes a VV&A process equivalent to that required in
the DOD guidance. However, the DOD guidance does not identify the
actual use of a model as constituting an equivalent VV&A process.
DOD is using the following nine mobility models to conduct the MCS:
* Aerial Port of Debarkation:
* Air Mobility Operations Simulation:
* Combined Mating and Ranging Planning System:
* CONUS (Continental U.S.) Enhanced Logistics Intra-theater Support
Tool:
* Model for Inter-theater Deployment by Air and Sea:
* Integrated Computerized Development System:
* Joint Throughput Modeling Tool:
* TRANSPORT:
- Airlift/Sealift Throughput Tool:
- Airlift/Sealift Rapid Analysis Tool:
- Airlift Simulation Tool & Seaport Simulation Tool, and:
* Analysis of Mobility Platform Federation.
According to DOD officials, eight of the nine models did not go through
the VV&A process specified in the DOD guidance.[Footnote 7] Office of
Program Analysis and Evaluation, U.S. Transportation Command, Air
Mobility Command, and Surface Deployment and Distribution Command
officials told us that (1) most of the models have been used within DOD
for many years and have proved reliable, and (2) many subject matter
experts work with the models and the output daily to assure ongoing
error detection and swift corrections when needed. These officials
maintain that actual use of the models for a long period of time
constitutes an equivalent VV&A process.
Nonetheless, DOD officials were conducting VV&A on one of the legacy
models being used in the MCS while MCS officials were simultaneously
questioning the relevancy of DOD's guidance for the legacy models. For
example, Surface Deployment and Distribution Command officials told us
that the CONUS Enhanced Logistics Intra-theater Support Tool model was
undergoing VV&A at the time of our review. It is unclear why DOD is
conducting VV&A on this model given Office of Program Analysis and
Evaluation and other officials' belief that it is unnecessary.
Furthermore, the extent to which DOD's guidance may be irrelevant as
asserted by these officials is unknown, because the department has not
evaluated the current VV&A guidance to determine its relevancy for use
with legacy models.
Conclusions:
Models and simulations approximate the real world. The approximations
must be justified to assure modeling and simulation users that their
predictions are credible within the bounds of specific situations,
environments, and circumstances. When modeling and simulation are
credible, decision makers have greater assurance that they are well
informed and thus can make well-founded decisions. VV&A reduces the
risk inherent in the use of models and simulations by improving the
credibility of modeling and simulation results. VV&A also enhances
credibility by applying a process of incremental review, analysis,
evaluation, and testing by an independent agent. In light of the fact
that DOD did not follow its guidance, the absence of model and baseline
data VV&A documentation for the models and data used to conduct the
MCS, and the planned lack of disclosure in the soon to be published MCS
report, could limit the study's credibility. Moreover, MCS officials
maintain that DOD guidance regarding VV&A is not relevant to legacy
models and data. However, until the department evaluates the guidance
to determine its relevancy for use with legacy models or incorporates
guidance showing how actual model usage is to be applied as an
equivalent VV&A process, the validity of DOD's assertion is uncertain.
When conducted as intended, VV&A provides greater assurance that the
MCS outputs are accurate. Ultimately, if the MCS inaccurately
identifies mobility requirements, DOD officials may be less well
informed and may therefore inadvertently obtain insufficient mobility
assets or more than needed and thus waste resources.
Recommendations for Executive Action:
We recommend that you direct the Director, Office of Program Analysis
and Evaluation, to take the following three actions:
* develop documentation that describes the equivalent VV&A process used
to verify and validate the mobility models and baseline data used to
conduct the MCS prior to publishing any portion of the study results,
* disclose in the published MCS report the equivalent VV&A process used
on the models and baseline data, and:
* work with the Office of the Under Secretary of Defense for
Acquisition, Technology and Logistics to evaluate the current DOD VV&A
guidance to determine its relevance for use with legacy models and to
change the guidance if appropriate.
Agency Comments and Our Evaluation:
In commenting on a draft of this report, the DOD concurred with two of
our recommendations and partially concurred with the third.
In its comments, DOD concurred with our first and second
recommendations that the Office of Program Analysis and Evaluation
develop documentation that describes the equivalent VV&A process used
to verify and validate the mobility models and baseline data used to
conduct the MCS prior to publishing any portion of the study results,
and disclose in the published MCS report the equivalent VV&A process
used on the models and baseline data. In its comments, DOD stated that
the mobility modeling community has amassed substantial expertise
during the past 25 years and that a "vigorous, collaborative VV&A
process that is fully consistent with and in many respects exceeds the
intent of DOD VV&A guidance has been put into place." DOD noted the MCS
report will provide information on the VV&A process, and will exceed
the level of documentation provided in past reports on DOD mobility
studies.
DOD partially concurred with our recommendation to evaluate the current
DOD VV&A guidance to determine its relevance for use with legacy models
and change the guidance if deemed appropriate. In its response, DOD
essentially agreed with our recommendation, but pointed out that the
Office of the Under Secretary of Defense for Acquisition, Technology
and Logistics is responsible for VV&A guidance. Therefore, we refined
our recommendation to recommend that the Director of the Office of
Program Analysis and Evaluation work with the Office of the Under
Secretary of Defense for Acquisition, Technology and Logistics to
evaluate the current DOD VV&A guidance to determine its relevance for
use with legacy models and to change the guidance if appropriate.
Additionally, DOD expressed concern that it was premature to reach a
conclusion as to the adequacy of the department's VV&A process because
some of the documentation requested cannot be provided until the final
report is written and that our draft report should be amended. We agree
that we cannot assess the adequacy until the documentation is available
and have so stated in our report. We also stated in our report that at
the time of our review, DOD had not planned to describe the VV&A
process in its MCS report nor had it planned to perform VV&A because
the legacy models being used were reliable. As we noted in our report,
at least one of the models was already undergoing a VV&A.
Moreover, DOD questioned our conclusion that if an adequate VV&A
process could not be documented, then the credibility of the MCS could
be limited. Specifically, DOD stated that convincing empirical evidence
indicates that the MCS report's credibility will not be limited by the
VV&A documentation associated with the legacy models, because DOD has
used the models for two decades with no credibility limitations noted.
As we noted in our report, DOD guidance indicates that a well-
documented VV&A process for the models used to conduct the study will
add to the MCS report's credibility. Also as we noted in our report,
DOD guidance states that the VV&A process provides enhanced user
confidence, improved performance and reliability for the subject model
and simulation results, reduced risk of inaccurate model outputs, and
offers assurances that a particular model or simulation can be relied
on for reproducible results appropriate for its objectives. While we
support the use of empirical evidence, such evidence is normally
verifiable.
Finally, DOD expressed concern that we linked the VV&A of mobility
models used to conduct the MCS with the prospect of inaccurate
identification of mobility requirements. We disagree. DOD guidance
states that VV&A reduces the risk inherent in the use of models and
simulations by improving the credibility of their results and provides
greater assurance that the study outputs are accurate. If the models
supporting the MCS do not effectively simulate the real world and DOD
uses the results to complete the MCS and subsequently base acquisition
decisions, then we maintain our caution--DOD could inadvertently obtain
insufficient mobility assets or could acquire more than needed and thus
waste resources.
Enclosure I contains the full text of DOD's comments.
We are sending copies of this report to the Chairman and Ranking
Members of the Senate Armed Services Committee as well as to the
Chairmen and Ranking Members of the Senate Appropriations Committee,
Subcommittee on Defense, the House Armed Services Committee, and the
House Appropriations Committee, Subcommittee on Defense. This letter is
also available at no charge on the GAO's Web site at http:www.gao.gov.
If you or your staff have any questions on the matters discussed in
this letter, please contact me at (202) 512-8365 or solisw@gao.gov. Key
contributors to this report are listed in enclosure II.
Sincerely yours,
Signed by:
William M. Solis, Director:
Defense Capabilities and Management:
Enclosures:
Enclosure I: Comments from the Department of Defense:
OFFICE OF THE SECRETARY OF DEFENSE:
PROGRAM ANALYSIS AND EVALUATION:
1800 DEFENSE PENTAGON:
WASHINGTON, DC 20301-1800:
August 23, 2005:
Mr. William M. Solis:
Director, Defense Capabilities and Management:
U.S. Government Accountability Office:
441 G Street, N.W.:
Washington, DC 20548:
Dear Mr. Solis:
This is the Department of Defense's (DoD) response to the GAO Draft
Report, GAO-05-659R, "DEFENSE TRANSPORTATION: Opportunities Exist to
Enhance the Credibility of the Current and Future Mobility Capabilities
Studies," dated August 10, 2005.
The DoD concurs with two of the GAO's recommendations and partially
concurs with the remaining recommendation. Specific comments on each
recommendation are attached. Additional concerns with the report are
also forwarded for your consideration.
Kathleen Conley, Director of PA&E's Projection Forces Division, is my
point of contact for this issue. Please contact her at (703) 697-0802,
or at Kathleen.Conley@osd.mil, if you have any questions.
We appreciate the opportunity to comment on the draft report.
Signed by:
Stanley R. Szemborski:
VADM, USN:
Principal Deputy Director:
Attachment As stated:
GAO DRAFT REPORT - DATED AUGUST 10, 2005 GAO CODE 350709/GAO-05-659R:
"DEFENSE TRANSPORTATION: Opportunities Exist to Enhance the Credibility
of the Current and Future Mobility Capabilities Studies"
DEPARTMENT OF DEFENSE COMMENTS ON THE RECOMMENDATIONS:
RECOMMENDATION 1: The GAO recommended that the Director, Office of
Program Analysis and Evaluation (PA&E):
* Develop documentation that describes the equivalent verification,
validation, and accreditation (VV&A) process used to verify and
validate the mobility models and baseline data used to conduct the
mobility capabilities study (MCS) prior to publishing any portion of
the study results. (p. 6):
DoD RESPONSE: Concur. As DoD representatives have stated in discussions
with GAO officials, the MCS report will provide information on the VV&A
process, and will exceed the level of documentation provided in past
reports on DoD mobility studies. The transparent, collaborative
approach taken by the Department to develop realistic scenarios and
data and to accurately model complex mobility processes will be
documented for the benefit of readers who may not be familiar with the
Department's analytic best practices. Specifically, the scenarios,
baseline data, and mobility modeling associated with the MCS reflect
the Department's considerable experience in conducting mobility studies
over the past twenty-five years. The mobility modeling community has
amassed substantial expertise during that period, and a vigorous,
collaborative V V&A process has been put into place. That process is
fully consistent with-and in many respects exceeds-the intent of DoD
Instruction 5000.61. As a result of their continuous involvement in the
study process, senior DoD leaders have developed a high degree of
confidence in the MCS methodology. Because the process has provided an
effective forum for resolving questions about the underlying data,
models, and assumptions, the insights gleaned from the MCS can be used
to frame discussions about capabilities needed to support the defense
strategy.
RECOMMENDATION 2: The GAO recommended that the Director, Office of
Program Analysis and Evaluation:
* Disclose in the published MCS report the equivalent VV&A process used
on the models and baseline data. (p. 6):
DoD RESPONSE: Concur. As stated above and in discussions with GAO
officials, the MCS report will provide information on the V V&A
process, and will exceed the level of documentation provided in past
reports on DoD mobility studies. The transparent, collaborative
approach taken by the Department to develop realistic scenarios and
data and to accurately model complex mobility processes will be
documented for the benefit of readers who may not be familiar with the
Department's analytic best practices.
RECOMMENDATION 3: The GAO recommended that the Director, Office of
Program Analysis and Evaluation:
* Evaluate the current DoD V V&A guidance to determine its relevance
for use with legacy models and change the guidance if deemed
appropriate. (p. 6):
DoD RESPONSE: Partially concur. The Department will continuously assess
the applicability of its VV&A guidance to the legacy models used in the
MCS and other studies. The office of the Under Secretary of Defense for
Acquisition, Technology, and Logistics (AT&L) is responsible for
managing VV&A guidance. The Department is considering revisions to this
guidance.
ADDITIONAL ISSUES: The GAO draft report notes that GAO was unable to
assess the adequacy of DoD's V V &A process. Further, the report states
that DoD "does not plan to disclose how it conducted its equivalent W&A
process in its MCS report." Consequently, GAO expressed concern that a
failure to fully document the study's V V&A process could limit the
credibility of the MCS. (pp. 2, 4, 6):
DoD COMMENT: The VV&A process will be fully described in the final
report-a point that was emphasized by DoD officials in discussions with
GAO. Furthermore, DoD officials have notified GAO that some of the
documentation requested cannot be provided until the final report is
written. Therefore, it is premature to reach a conclusion as to the
adequacy of the V V&A process. For this reason, either the assessment
of adequacy should be omitted from the section of the GAO draft report
providing preliminary observations, or the discussion should be amended
to inform the reader that a full assessment is not currently possible,
and that one will be completed upon receipt of the final report.
Pages 2, 4, and 6 of the GAO draft report raise the possibility that
failure to fully document the VV&A process used in the MCS could limit
the study's credibility. The documentation of W&A processes in the MCS
report-well beyond that presented in previous studies-should benefit
those readers who may not be familiar with the study's methodology.
Moreover, convincing empirical evidence indicates that the report's
credibility will not be limited by the VV&A documentation associated
with legacy models: two decades of mobility studies have been conducted
using many of the legacy models employed by MCS participants with no
such effect noted.
On page 6, the GAO draft report attempts to link the current VV&A
processes for legacy mobility models with the prospect of inaccurate
identification of mobility requirements. Again, the evidence does not
support this contention. The mobility community's long-standing VV&A
processes have not resulted in inaccurate assessments of mobility
requirements, or of the investments needed to achieve capability
objectives.
[End of section]
GAO Contact and Staff Acknowledgments:
GAO Contact: William M. Solis, (202) 512-8365:
Acknowledgments: Key contributors to this report include Brian J.
Lepore, Claudia Dickey, Ron La Due Lake, Oscar Mardis, Deborah Owolabi,
Kenneth Patton, and R.K. Wild.
(350708):
FOOTNOTES
[1] S. Rep. No. 108-260, at 126 (2004).
[2] The MCS study team includes officials from the Office of the
Secretary of Defense, Program Analysis and Evaluation and the office of
the Director of Logistics, Joint Chiefs of Staff, identified as co-
leads and study management, as well as study participants to include
representatives from the military services, combatant commands, and
contractors employed by any of the aforementioned DOD organizations to
provide input to or services in support of the MCS.
[3] DOD, DOD Modeling and Simulation (M&S) Verification, Validation,
Accreditation (VVA), Instruction 5000.61 (Washington, D.C.: May 2003).
[4] Past mobility requirements studies conducted since the early 1990s
include: Mobility Requirements Study (1992); Mobility Requirements
Study-Bottom Up Review Update (1995); and Mobility Requirements Study
for Fiscal Year 2005 (2001).
[5] The overall study objectives are to identify and/or quantify (1)
how variations in mobility capabilities support the defense strategy
from point of origin to point of use and return in the 2012 time frame;
(2) mobility capability gaps, overlaps, or excesses and associated risk
assessments with regard to conducting operations; (3) mobility
capability alternatives that mitigate operational logistic impacts
caused by challenges; (4) combinations of mobility, engineering, and
infrastructure capabilities required to support deployments and
distributions required by the defense strategy; (5) new metrics for
assessing mobility capabilities; (6) potential impact of evolving
service force transformation and research and development efforts that
integrate mobility concepts for the 2024 time frame; and (7)
transformational accelerants to enable the defense transportation
system to operate in a net-centric environment.
[6] DOD, DOD modeling and Simulation Verification, Validation and
Accreditation, DOD Instruction 5000.61 (Washington, D.C.: May 2003);
and DOD Modeling and Simulation Office, DOD Verification, Validation
and Accreditation Recommended Practices Guide (Washington, D.C.: August
2004).
[7] DOD conducted VV&A on the Combined Mating and Ranging Planning
System model about 15 years ago in compliance with departmental
guidance but could not locate documentation to demonstrate how VV&A was
done. As a result, we could not evaluate the adequacy of the VV&A
process.
