DOD Personnel
Documentation of the Army's Civilian Workforce-Planning Model Needed to Enhance Credibility Gao ID: GAO-03-1046 August 22, 2003Between fiscal years 1989 and 2002, the Department of Defense (DOD) reduced its civilian workforce by about 38 percent, with little attention to shaping or specifically sizing this workforce for the future. As a result, the civilian workforce is imbalanced in terms of the shape, skills, and experience needed by the department. DOD is taking steps to transform its civilian workforce. To assist with this transformation, the department is considering adopting an Army workforce-planning model, known as the Civilian Forecasting System (CIVFORS), which the Army uses to forecast its civilian workforce needs. Other federal agencies are also considering adopting this model. GAO was asked to review the adequacy of the steps the Army has taken to ensure the credibility of the model.
The Army has taken adequate steps to ensure that the historical personnel data used in the model are sufficiently reliable and that the information technology structure adequately and appropriately supports the model. For example, the Army has established adequate control measures (e.g., edit checks, expert review, etc.) to ensure that the historical data that goes into the model are sufficiently reliable. Moreover, it has taken adequate steps to ensure that the information technology support structure (i.e., the software and hardware used to interface with and house the model) would enable continuity of operations, functionality, and system modification and operations. However, the Army has not demonstrated that it has taken adequate steps to ensure that the model's forecasting capability provides the basis for making accurate forecasts of the Army's civilian workforce. The Army's original certification of CIVFORS in 1987 was based on a formal documented verification and validation of the model structure that has not been formally updated since that time even though the Army has undertaken several model improvements. According to the Army's CIVFORS program manager, the Army has taken several steps, to include an independent review, peer reviews, and a comparison of forecasted data to actual data. However, documentation of these steps is incomplete and, therefore, does not provide adequate evidence to demonstrate the credibility of the forecast results. Without adequate documentation, the Army cannot show that it has taken sufficient steps to ensure the model's credibility in terms of its forecasting capability; consequently, there exists a risk that the forecasts it produces may be inaccurate or misleading. Furthermore, without documentation of CIVFORS's forecasting capability, it may be difficult for DOD and other federal organizations to accurately determine its suitability for their use.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-03-1046, DOD Personnel: Documentation of the Army's Civilian Workforce-Planning Model Needed to Enhance Credibility
This is the accessible text file for GAO report number GAO-03-1046
entitled 'DOD Personnel: Documentation of the Army's Civilian
Workforce-Planning Model Needed to Enhance Credibility' which was
released on August 22, 2003.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
On January 5, 2004, this document was revised to add various
footnote references missing in the text of the body of the document.
Report to the Ranking Minority Member, Subcommittee on Readiness,
Committee on Armed Services, House of Representatives:
United States General Accounting Office:
GAO:
August 2003:
DOD Personnel:
Documentation of the Army's Civilian Workforce-Planning Model Needed to
Enhance Credibility:
GAO-03-1046:
GAO Highlights:
Highlights of GAO-03-1046, a report to the Ranking Minority Member,
Subcommittee on Readiness, Committee on Armed Services, House of
Representatives
Why GAO Did This Study:
Between fiscal years 1989 and 2002, the Department of Defense (DOD)
reduced its civilian workforce by about 38 percent, with little
attention to shaping or specifically sizing this workforce for the
future. As a result, the civilian workforce is imbalanced in terms of
the shape, skills, and experience needed by the department. DOD is
taking steps to transform its civilian workforce. To assist with this
transformation, the department is considering adopting an Army
workforce-planning model, known as the Civilian Forecasting System
(CIVFORS), which the Army uses to forecast its civilian workforce
needs. Other federal agencies are also considering adopting this
model. GAO was asked to review the adequacy of the steps the Army has
taken to ensure the credibility of the model.
What GAO Found:
The Army has taken adequate steps to ensure that the historical
personnel data used in the model are sufficiently reliable and that
the information technology structure adequately and appropriately
supports the model. For example, the Army has established adequate
control measures (e.g., edit checks, expert review, etc.) to ensure
that the historical data that goes into the model are sufficiently
reliable. Moreover, it has taken adequate steps to ensure that the
information technology support structure (i.e., the software and
hardware used to interface with and house the model) would enable
continuity of operations, functionality, and system modification and
operations.
However, the Army has not demonstrated that it has taken adequate
steps to ensure that the model‘s forecasting capability provides the
basis for making accurate forecasts of the Army‘s civilian workforce.
The Army‘s original certification of CIVFORS in 1987 was based on a
formal documented verification and validation of the model structure
that has not been formally updated since that time even though the
Army has undertaken several model improvements. According to the
Army‘s CIVFORS program manager, the Army has taken several steps, to
include an independent review, peer reviews, and a comparison of
forecasted data to actual data. However, documentation of these steps
is incomplete and, therefore, does not provide adequate evidence to
demonstrate the credibility of the forecast results. Without adequate
documentation, the Army cannot show that it has taken sufficient steps
to ensure the model‘s credibility in terms of its forecasting
capability; consequently, there exists a risk that the forecasts it
produces may be inaccurate or misleading. Furthermore, without
documentation of CIVFORS‘s forecasting capability, it may be difficult
for DOD and other federal organizations to accurately determine its
suitability for their use.
What GAO Recommends:
To assure the reliability of Army civilian workforce projections and
the appropriateness of the model for use DOD-wide and by other federal
agencies, we recommend that the Secretary of Defense direct the
Secretary of the Army to appropriately document the forecasting
capability of the model.
Although DOD stated, in written comments on a draft of this report, it
did not concur with GAO‘s recommendation, the Army is taking actions
that, in effect, implement it.
www.gao.gov/cgi-bin/getrpt?GAO-03-1046.
To view the full product, including the scope and methodology, click
on the link above. For more information, contact Derek Stewart, 202-
512-5559, stewartd@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
Civilian Workforce-Planning Model's Data Reliability and Information
Technology Structure Are Adequate, but Forecasting Ability Not Fully
Established:
Conclusions:
Recommendation for Executive Action:
Agency Comments and Our Evaluation:
Scope and Methodology:
Appendix I: Comments from the Department of Defense:
Appendix II: GAO Contact and Staff Acknowledgments:
Abbreviations:
CIVFORS: Civilian Forecasting System:
DOD: Department of Defense:
WASS: Workforce Analysis Support System:
United States General Accounting Office:
Washington, DC 20548:
August 22, 2003:
The Honorable Solomon P. Ortiz
Ranking Minority Member
Subcommittee on Readiness
Committee on Armed Services
House of Representatives:
Dear Mr. Ortiz:
Between fiscal years 1989 and 2002, the Department of Defense (DOD)
reduced its civilian workforce from 1,075,437 to 670,166--about a 38
percent reduction--with little attention to shaping or specifically
sizing this workforce for the future. As a result, the civilian
workforce is imbalanced in terms of the shape, skills, and experience
needed by the department. DOD plans to downsize its civilian workforce
by an additional 55,000 through fiscal year 2007. In addition, in April
2003, DOD submitted a proposal to Congress that would authorize DOD to
establish a National Security Personnel System to transform its current
civilian personnel system.[Footnote 1] DOD is also exploring the
feasibility of placing hundreds of thousands of civilians into
essentially nonmilitary jobs that are currently performed by uniformed
personnel. To assist in determining its future workforce, DOD will need
reliable workforce planning tools, such as workforce projection models.
According to DOD guidance, a model used to provide data for decision
making should be accredited--that is, the model should be officially
certified as acceptable for use for a specific purpose.
In a February 2003 testimony, the Chief of Staff of the Army stated
that the Army has begun to transform its civilian personnel system. To
assist with this transformation, the Army is using its workforce-
planning model, known as the Civilian Forecasting System (CIVFORS),
which forecasts future civilian workforce needs.[Footnote 2] The Army
is working closely with the Office of the Secretary of Defense and the
Office of Personnel Management to demonstrate the applicability of the
model for use DOD-wide and in other federal agencies. According to Army
guidance (Army Regulation 5-11 and Department of the Army Pamphlet 5-
11), verification is one of the steps needed to ensure a model's
suitability to perform its intended purpose. The verification process
evaluates the extent to which a model has been developed using sound
and established software engineering techniques, and it establishes
whether the computer code correctly performs the intended functions.
Army guidance also states that assessment of the correctness and
forecasting capability is required.
In this report, we reviewed the adequacy of the steps the Army has
taken to ensure the credibility of the model. In March 2003, we briefed
your staff on our preliminary findings. To determine the Army's efforts
to ensure the credibility of its model, we interviewed and obtained
pertinent documentation from the Army's CIVFORS program manager. We
also reviewed DOD and Army guidance relevant to the management of Army
models and interviewed DOD officials to discuss their plans to adopt
CIVFORS. We conducted our review from September 2002 to June 2003. More
detailed information on our scope and methodology appears at the end of
this report.
Results in Brief:
The Army's steps were adequate to ensure that the historical personnel
data used in the model are sufficiently reliable and that the
information technology support structure[Footnote 3] adequately and
appropriately supports the model, but the Army has not documented its
steps to ensure the credibility of the model's forecasting capability.
The Army has established adequate control measures (e.g., edit checks,
expert review, etc.) to ensure that the historical data that goes into
the model are sufficiently reliable. Moreover, it has taken adequate
steps to ensure that the information technology support structure would
enable continuity of operations, functionality, and system modification
and operations. However, the Army has not documented that it has taken
adequate steps to ensure that the model's structure (including its
forecasting capability and the appropriateness of its assumptions)
provides the basis for making accurate forecasts of the Army's civilian
workforce. The Army's original certification of CIVFORS in 1987 was
based on a formal documented verification and validation of the model
structure that has not been formally updated since that time, even
though the Army has undertaken several model improvements. According to
the Army's CIVFORS program manager, the Army has taken several steps,
to include an independent review, peer reviews, and a comparison of
forecasted data to actual data. However, documentation of these steps
is incomplete and, therefore, does not provide adequate evidence to
demonstrate the credibility of the forecast results. Without adequate
documentation, the Army cannot show that it has taken sufficient steps
to ensure the credibility of the model's forecasting capability;
consequently, there exists a risk that the forecasts it produces may be
inaccurate or misleading. Furthermore, without documentation of
CIVFORS's forecasting capability, it may be difficult for DOD and other
federal organizations to accurately determine its suitability for their
use.
We are recommending that the Secretary of Defense direct the Secretary
of the Army to appropriately document the Army's forecasting capability
of the civilian workforce-planning model. Although DOD stated, in
written comments on a draft of this report, it did not concur with our
recommendation, the Army is taking actions that, in effect, implement
it.
Background:
According to an Army Human Resource official, the Army uses the
workforce-planning model--CIVFORS--for human resources management.
CIVFORS is a collection of software programs that anticipate future
impacts on the workforce so that management can plan for changes
instead of reacting to them. The model is used to evaluate a number of
critical areas in civilian workforce planning, including projected
recruitment of personnel, impact of organizational realignments, and
changes in workforce trends (such as aging, retention, and projected
personnel shortfalls). It is a life-cycle modeling and projection tool
that models the most significant events that describe the life-cycle
path of personnel, which includes accessions, promotions,
reassignments, retirements, and voluntary and involuntary separations
over a 7-year period.
Verification and validation of models are important steps to building
credible models because they provide the foundation for the
accreditation process to ensure the suitability of the models for their
intended purposes, as stated in Army guidance, Management of Army
Models and Simulations.[Footnote 4] The verification process evaluates
the extent to which a model has been developed using sound and
established software engineering techniques, and it establishes whether
the model's computer code correctly performs the intended functions.
Model verification includes data verification, model documentation, and
testing of the information technology structure that supports the
model; model verification is contained in such documents as the
programmer's manual, installation's manual, user's guide, analyst's
manual, and trainer's manual. According to Army guidance, assessment of
the correctness and forecasting capability of the model is also
required, and it should be performed by a subject matter expert
independent from the model developer; however, the developer is
expected to conduct in-house verification and testing to assist in the
overall model development process. Validation is the process of
determining the extent to which the model adequately represents the
real world.
Civilian Workforce-Planning Model's Data Reliability and Information
Technology Structure Are Adequate, but Forecasting Ability Not Fully
Established:
The Army has taken steps to ensure the reliability of the historical
personnel data used by the model and the adequacy of its information
technology structure used to support the model, but it has not provided
documentation that it has sufficiently tested and reviewed the most
critical aspect of the model--its forecasting capability and the
appropriateness of its assumptions. As a result, the forecasting
credibility of the current version of the model is not sufficiently
validated or documented. Without proper documentation of the abilities
of the model, there is a risk that the forecasts it produces may be
inaccurate or misleading and the suitability for use by other
organizations may be difficult to determine.
Historical Personnel Data Reliability Is Adequate:
The Army's review of the historical personnel data used to provide
information for workforce planning was adequate to show that the data
are sufficiently reliable for use in the workforce model. Data
regarding personnel (such as date hired, education, age, grade level,
and occupational series) are taken from the Army's Workforce Analysis
Support System (WASS).[Footnote 5] CIVFORS uses the most recent 5 years
of historical data to forecast the civilian workforce planning needs
during the next 7 years.
According to Army guidance, to ensure that data are sufficiently
reliable for use in the Army model, support documents should contain
information about the overall characteristics of the database.
Furthermore, the documents should show the intended range of
appropriate uses for the model as well as constraints on its use. They
should also include concise statements of the condition of the database
for the purpose of indicating its stability. The Army provided most,
but not all, of the documents referred to in Army guidance; we believe
that the documents provided are key ones and are adequate to show that
WASS data are sufficiently reliable for use in CIVFORS. In addition,
the Army program manager for the CIVFORS workforce-planning model
stated that the workforce data are checked by reviewing the arithmetic
in the numerical algorithms to verify that there is no unexplained
change in the size of the civilian personnel workforce contained in the
database. Further, edit checks include matching social security numbers
for personnel from one time period to another to account for actual
personnel and personnel transactions processed. In addition, CIVFORS
has automated checks for inappropriate numbers or characters. Such
steps help to assure that the data contained in WASS accurately and
completely reflect critical personnel aspects and transactions.
Information Technology Structure Is Adequate:
The Army's procedures for validating the information technology support
structure (the software and hardware used to interface with and house
the model) were also sufficient. For example, the Army (1) adequately
documented the information technology structure to allow for continuity
of operations, (2) tested its functionality, and (3) provided expertise
for system modification and operation. Procedures used by the Army
include documenting the model's system description and hardware and
software requirements, providing system and user manuals, planning for
configuration management,[Footnote 6] and conducting functionality
tests to help ensure the system's usability and operability over time
and to demonstrate the adequacy of the information technology structure
to support use of the workforce model.
Model's Forecasting Ability Is Not Fully Established:
The Army's documentation cannot show that the forecasting ability of
CIVFORS has been adequately evaluated and, therefore, we cannot fully
assess the credibility of the model. According to Army guidance,
validation is the process of determining the extent to which a model
adequately represents the real world. According to the Army program
manager, over a 7-year period, CIVFORS forecasts the anticipated
impacts on the workforce based on the most significant events in the
life-cycle path of personnel (to include accessions, promotions,
reassignments, retirements, voluntary separations, and involuntary
separations). Army guidance states that an independent, peer, and
subject matter expert review of the model should be conducted. The Army
guidance also suggests generally accepted methods, such as conducting a
careful line-by-line examination of the model design and computer code
and algorithms. The Army's program manager said this had been done for
the original certification of CIVFORS in 1987. However, no formal
document of the reviews has been prepared in the years since, even
though the Army has undertaken several model improvements, such as (1)
an expanded scope to include more dimensions in the modeling process;
(2) a more integrated, streamlined process that involves fewer steps;
and (3) greater flexibility, achieved by generalizing the formulas and
parameters.
In addition, there is insufficient documentation regarding tests
performed, since 1987, in which CIVFORS's forecasts for prior years are
compared against equivalent historical data (called an "out of sample"
test) to measure the model's forecasting capability. Such testing,
which is one method to validate a model's forecasting capability, would
involve using the first 5 of the last 7 years of historical data to
forecast the 2 subsequent years. The forecasts for the last 2 years
could then be compared to the actual historical data. The Army,
however, performed tests comparing patterns of forecasts against
historical data (called "in sample" tests), showing that forecasts
reflect the same patterns as the historical data used to develop them
for a sample of three Army major commands. However, the draft document
that was provided to us was inadequate to fully assess the sampling
used by the Army and the value of the tests.
Finally, the Army could not provide adequate documentation of an
independent or peer review of the model. The Army's CIVFORS program
manager stated that the major commands served as peer reviewers by
conducting a comparison of their workforce data to WASS and CIVFORS
workforce data. We believe that such assessments by users provide
important information but do not constitute a peer review as defined in
Army guidance. Also, the results of these assessments were not
available for us to review. The program manager also stated that an
independent subject matter expert reviewed the functional design and
the code in 1999, but a formal report of the activities performed and
the specific changes or modifications implemented during the review
were not produced.
Documentation has often not been a priority for several reasons.
According to the Army's CIVFORS program manager, lack of documentation
is primarily due to limited funding, which was spent on implementing
changes to CIVFORS and WASS rather than on the production of formal
documents. Further, a shortage of staff (only one staff person--the
program manager) and loss of documents during the attack on the
Pentagon on September 11, 2001, also affected the amount of
documentation the Army could provide us. The program manager also
stated that some documentation was not needed because CIVFORS's design
is predicated on proven methods in other Army active-duty, military
manpower forecasting models. In addition, the program manager stated
that the Army and contractors have primarily been adapting technology
(upgrading from mainframe to personal computer to Web-based) to improve
model functionality rather than creating new technology. However,
without proper documentation of the abilities of the model, there
exists a risk that the forecasts it produces may be inaccurate or
misleading. Consequently, decisions about future workforce
requirements may be questionable, and planning for the size, shape, and
experience level of the future workforce may not adequately meet the
Army's needs.
These issues may extend beyond the Army. In April 2002, DOD published a
strategic plan for civilian personnel, which includes a goal to obtain
management systems to support workforce planning. According to a DOD
official responsible for civilian workforce planning tools, components
within DOD have been requesting a modeling tool to assist them with
civilian workforce planning. As a result, DOD has decided to test the
Army's civilian forecasting model. In October 2002, DOD purchased
hardware, installed modified software, and provided training to a small
number of personnel. Recently, DOD obtained a historical database of
civilian personnel data from the Defense Management Data Center and
provided the database to the contractor to load into the model. Two
agencies have volunteered to test the model: the Defense Logistics
Agency and the Washington Headquarters Service. DOD is working to
develop a test for these organizations using their own civilian
personnel data to test the model. At the end of the testing period, DOD
will assess the model to obtain a better understanding of its logic and
determine whether or not it should be implemented departmentwide.
Conclusions:
As DOD continues to transform and downsize its civilian workforce, it
is imperative that the department properly shape and size the
workforce. One tool that could assist in this effort is CIVFORS--the
Army's workforce planning model. However, proper documentation of the
verification and validation of CIVFORS is needed before expanding its
use. The Army has taken adequate steps to ensure that the historical
personnel data used in the model are sufficiently reliable and the
information technology structure appropriately supports the model;
however, it has not fully documented that it has taken adequate steps
to demonstrate the credibility of the model's forecasting capability.
Further, a model should be fully scrutinized before each new
application because a change in purpose, passage of time, or input data
may invalidate some aspects of the existing model. Without sufficient
documentation to demonstrate that adequate steps have been taken to
ensure the credibility of the model's forecasting capabilities,
decisions about the Army's future civilian workforce may be based on
questionable data and other potential users cannot determine with
certainty the model's suitability for their use.
Recommendation for Executive Action:
To assure the reliability of Army civilian workforce projections, as
well as the appropriateness of the model for use DOD-wide and by other
federal agencies, we recommend that the Secretary of Defense direct the
Secretary of the Army to appropriately document the Army's forecasting
capability of the model.
Agency Comments and Our Evaluation:
Although DOD stated, in written comments on a draft of this report,
that it did not concur with our recommendation, the Army is taking
actions that, in effect, implement it. DOD's written comments are
contained in appendix I.
Regarding our recommendation that the Secretary of Defense direct the
Secretary of the Army to appropriately document the Army's forecasting
capability of the model, DOD stated that the Army recognizes the need
to fully document its verification and validation efforts. Further, DOD
stated the staff of the Assistant Secretary of the Army, Manpower and
Reserve Affairs, has begun developing a verification and validation
plan to enable outside parties to assess the suitability and
adaptability of the model for their organizational use. This
verification and validation process is scheduled for completion in
September 2003. However, during our review, DOD did not provide
information about the full scope of this verification and validation
effort. We believe that as the Army undertakes its verification and
validation effort, it should clearly document, as we recommended, its
assumptions, procedures, and the results so that future users can
replicate the tests to appropriately establish the model's validity for
their purposes.
DOD also did not concur with our finding that the forecasting ability
of the model has not been fully established. DOD stated that the
ultimate test of a system is performance and that CIVFORS has been
consistently generating Army projections with high standards of
accuracy. We did not independently evaluate the model's accuracy. As
our report makes clear, our basic point is that the model's forecasting
ability has not been documented in accordance with Army guidance. We
continue to believe that without adequate documentation, the Army
cannot show that it has taken sufficient steps to ensure the model's
credibility in terms of its forecasting capability. DOD also provided
technical comments, which we incorporated where appropriate.
Scope and Methodology:
We did not independently evaluate the model or the application of the
steps; rather, we reviewed the adequacy of the steps that the Army
program manager stated were taken to ensure the credibility of the
model. To determine the adequacy of the steps the Army has taken to
ensure the credibility of its civilian workforce-forecasting model, we
discussed CIVFORS with the Army's CIVFORS program manager in the Army
G-1 office, Civilian Personnel Policy Directorate, who has overall
responsibility for the workforce analysis and the forecasting system.
In addition, Army contractor officials who are responsible for
providing technical, analytic, and management support to operate,
maintain, and enhance the planning tool and model participated in
several of our discussions with the program manager. We reviewed the
following CIVFORS's documents regarding the information technology
support structure: the Configuration Management Manual, the System's
Specifications, the Design/Subsystem Documentation, the Operator's
Manual, and the User's Manual. In addition, we reviewed the 1987 and
draft 2002 test analysis report on the Civilian Forecasting System and
other documentation provided by the Army to obtain information on how
the model operates according to model assumptions. We also reviewed the
DOD Defense Modeling and Simulation Office guidance on verification and
validation of models, the Army regulation and pamphlet pertaining to
the management of Army models and simulations, and other literature
regarding model credibility. We also interviewed DOD officials in the
Civilian Personnel Management Service responsible for developing plans
to adopt the Army's workforce forecasting model to discuss the status
of their efforts.
We conducted our review from September 2002 to June 2003 in accordance
with generally accepted government auditing standards.
We are sending copies of this report to the appropriate congressional
committees, the Secretary of Defense, the Under Secretary of Defense
for Personnel and Readiness, and the Secretary of the Army. We will
also make copies available to others upon request. In addition, the
report will be available at no charge on the GAO Web site at http://
www.gao.gov.
If you or your staff have any questions about this report, please
contact me at (202) 512-5559. Key contributors to this report are
listed in appendix II.
Sincerely yours,
Derek B. Stewart
Director, Defense Capabilities and Management:
Signed by Derek B. Stewart:
[End of section]
Appendix I: Comments from the Department of Defense:
OFFICE OF THE UNDER SECRETARY OF DEFENSE:
4000 DEFENSE PENTAGON WASHINGTON, D.C. 20301-4000:
AUG 8 2003:
PERSONNEL AND READINESS:
Mr. Derek B. Stewart:
Director, Defense Capabilities and Management Issues U.S. General
Accounting Office:
441 G Street, N.W. Washington, DC 20548:
Dear Mr. Stewart:
This is the Department of Defense (DoD) response to the General
Accounting Office (GAO) draft report, "DOD PERSONNEL: Greater Review
and Documentation of the Army's Civilian Workforce-Planning Model
Needed to Enhance Credibility," dated July 23, 2003, (GAO Code 350385
Prior Code 350198/GAO-03-1046). It is important to note that the
genesis of this report was the GAO notice of the intent to review
"DOD's Civilian Human Capital Strategic and Workforce Planning" on May
16, 2002; that GAO chose not to include DOD's reply in its March 2003
report, "DoD Actions Needed to Strengthen Civilian Human Capital
Strategic Planning and Integration with Military Personnel and Sourcing
Decisions;" and that GAO began another "Review of DOD's Strategic
Workforce Planning for Civilians" on June 17, 2003. Specifically listed
as an objective in this study was for GAO to determine "What actions
have DoD and the military services taken to ensure the credibility of
the workforce data and the analyses used in developing the civilian
strategic workforce plans?" While the second study is ongoing,
recommendations concerning the Civilian Forecasting System (CIVFORS)
must now be considered premature.
We nonconcur with the recommendation that the Secretary of Defense
direct the Secretary of the Army to appropriately document the Army's
forecasting capability of the model. The Department of the Army
recognizes the need to fully document Verification and Validation (V&V)
for its CIVFORS model. The staff of the Assistant Secretary of the
Army, Manpower and Reserve Affairs, has begun this process, scheduled
for completion in September 2003.
We also nonconcur with the major finding of the report that the model's
forecasting ability is not fully established. The ultimate test of the
system is performance. Using Army analysts to produce specific Army
products, the CIVFORS system has been consistently generating Army
projections with high standards of accuracy. The Army offers the
following comments to substantiate the system's abilities:
CIVFORS has been projecting Army civilian personnel strengths, gains
and losses since Fiscal Year (FY) 1990. Data studies provided to GAO
documented deviation of total projected (CIVFORS forecast) Army end
strength within one percent of actual on-board strength totals for FY90
through FY01. In FY02 the model predicted projected end strength of
222,600; the actual Army end strength was 223,449. The difference
between the projected and actual was $49; this is a deviation of a
little less than four-tenths of one percent (0.4 percent). Established
standards of accuracy by most government and private sector
organizations for forecasting models is 2.5 to 5 percent deviation. The
Army model has consistently exceeded this standard during its entire
operational life.
Documented evidence was provided to GAO to support the credibility of
the model forecasts in the analysis/evaluation of employee transaction
patterns. A specific example of behavior pattern forecasting was
illustrated by the Civil Service Retirement System (CSRS) retirement
bubble analysis. CIVFORS data predicted that the anticipated mass
exodus of employees covered by the CSRS retirement system would not
materialize. This conclusion, despite being in opposition to the
established theory of the personnel community, validated CIVFORS
capability in the area of employee gains and losses forecasting. In
projections developed for FY02 retirements for Army Materiel Command,
the model forecasted 166; actual retirements for FY02 were 1694, a
deviation of 29 for a projection accuracy of 99.3 percent.
For a complex mathematical procedure to consistently perform at a 9970
accuracy rate validates. by most conventional standards, the
credibility of the results. Consistently accurate results produced over
a long period of lime can only be produced by u model whose conceptual
design is founded in mathematically sound theory and is executed by
correctly programmed automalion code. 11 is statistically impossible
for any system to generate 99 percent accurate forecasts for over 13
years operating with incorrect mathematical algorithms and/or erroneous
programming code.
We suggest that either this report be delayed until after September 30,
2003, to allow the Army to complete its V&V process or that the
information concerning Army CIVFORS be included in the current review
of "DOD's Strategic Workforce Planning for Civilians.":
Sincerely,
Charles S. Abel
Principal Deputy:
Signed by Charles S. Abel:
[End of section]
Appendix II: GAO Contact and Staff Acknowledgments:
GAO Contact:
Christine Fossett (202) 512-2956:
Acknowledgments:
In addition to the name above, David Dornisch, Barbara Johnson, Barbara
Joyce, John Smale, Dale Wineholt, and Susan Woodward made significant
contributions to this report.
FOOTNOTES
[1] The system is proposed in the Transformation for the 21st Century
Act of 2003. The act also proposes other wide-ranging changes,
affecting civilian personnel pay and performance management, collective
bargaining, rightsizing, and other human capital areas.
[2] The Civilian Forecasting System was adapted from an Army military
forecasting model for civilian use in 1987.
[3] The software and hardware used to interface with and house the
model.
[4] Headquarters, Department of the Army, Management of Army Models and
Simulations, Army Regulation 5-11 (Washington, D.C., July 10, 1997).
This regulation prescribes policy and guidance and assigns
responsibilities for the management of Army models and simulations,
including development and maintenance.
[5] WASS enables analysis of data on Army civilians from 1974 to the
present. It has analysis capabilities that range from frequency
distribution to trend analysis.
[6] Configuration management is the control and documentation of
changes made to system hardware, software, and documentation throughout
the development and operational life of the system.
GAO's Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548:
