Defense Acquisition Workforce
DOD's Training Program Demonstrates Many Attributes of Effectiveness, but Improvement Is Needed Gao ID: GAO-11-22 October 28, 2010The President has announced his intention to improve the acquisition process, particularly given the half a trillion dollars the federal government spent in fiscal year 2009 on acquiring goods and services. The Department of Defense (DOD) spent $384 billion in fiscal year 2009 on goods and services--double what it spent in 2001. A high-quality workforce with the right competencies and skill sets will be critical to improving DOD acquisitions. GAO was mandated to determine the efficacy of DOD's certification training for its acquisition workforce. GAO assessed (1) DOD's capability to provide certification training, (2) the extent that such training reaches members of the workforce, and (3) the extent that previous training recommendations have been implemented. To conduct this work, GAO compared DOD's certification training to GAO guidance for effective training programs and analyzed policies, data, and previous reports on acquisition training.
DOD's certification training program--provided by the Defense Acquisition University (DAU)--generally demonstrates the capability to provide effective training, though some attributes of an effective training program are lacking. DAU ensures that strategic and tactical changes are promptly incorporated into training; uses centralized and decentralized training approaches in design and development; collects data during implementation to ensure feedback on its training programs; and analyzes its training during evaluation. However, DOD lacks complete information on the skill sets of the current acquisition workforce and does not have outcome-based metrics to assess results achieved in enhancing workforce proficiency and capability through training efforts. In 2009, GAO recommended that the Secretary of Defense identify and update on an ongoing basis the number and skill sets of the total acquisition workforce--including civilian, military, and contractor personnel--that the department needs to fulfill its mission. DOD agreed and to date has completed about one-fifth of its workforce competency assessments. At the end of fiscal year 2009, 90 percent of DOD's acquisition workforce personnel had completed required certification training or were within required time frames to do so, according to DAU data. However, DAU reports that it cannot provide for all training requested for the entire acquisition workforce. DAU has offered more courses in recent years, and high-priority personnel--those needing to complete classes for certification in their current position--constitute the majority in DAU classes. DAU plans the number and location of its classes based on data that DOD officials noted are generally incomplete when submitted, and DAU must adapt during the year to support new requirements as they are identified. DAU has identified the need for a new, integrated student information system that will provide better insight into the workforce it supports and is in the early stages of its procurement. DOD reports that most of the training-related recommendations from previous reviews--the Gansler Commission, the Panel on Contracting Integrity, and a prior GAO report--have been fully implemented and some actions are still under way. DOD has either fully or partially implemented 15 of the 19 recommendations GAO reviewed. Both the Army and the Office of the Secretary of Defense have taken steps to respond to the Gansler Commission recommendations. Most of the recommendations made by the Panel on Contracting Integrity have been implemented, with the exception of two recommendations related to assessing guidance and reviewing a specific training topic. GAO made four recommendations pertaining to the Defense Contract Audit Agency's government auditing standards training and expertise, of which one has been partially implemented and three have not been implemented, but some actions have been taken. GAO recommends DOD establish milestones for developing metrics to measure how certification training improves acquisition workforce capability and a time frame for acquiring and implementing an integrated information system. DOD concurred with the second but not the first recommendation. GAO continues to believe DOD needs to develop additional metrics.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: John K. Needham Team: Government Accountability Office: Acquisition and Sourcing Management Phone: (202) 512-5274GAO-11-22, Defense Acquisition Workforce: DOD's Training Program Demonstrates Many Attributes of Effectiveness, but Improvement Is Needed
This is the accessible text file for GAO report number GAO-11-22
entitled 'Defense Acquisition Workforce: DOD's Training Program
Demonstrates Many Attributes of Effectiveness, but Improvement Is
Needed' which was released on October 28, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
October 2010:
Defense Acquisition Workforce:
DOD's Training Program Demonstrates Many Attributes of Effectiveness,
but Improvement Is Needed Defense Acquisition Workforce:
GAO-11-22:
GAO Highlights:
Highlights of GAO-11-22, a report to the Congressional Committees.
Why GAO Did This Study:
The President has announced his intention to improve the acquisition
process, and current fiscal challenges highlight the need to scrutinize
half a trillion dollars the federal government spent in fiscal year
2009 on acquiring goods and services. The Department of Defense (DOD)
spent $384 billion in fiscal year 2009 on goods and services--double
what it spent in 2001. A high quality workforce with the right
competencies and skill sets will be critical to improving DOD
acquisitions. GAO was mandated to determine the efficacy of DOD‘s
certification training for its acquisition workforce. GAO assessed (1)
DOD‘s capability to provide certification training, (2) the extent that
such training reaches members of the workforce, and (3) the extent that
previous training recommendations have been implemented. To conduct
this work, GAO compared DOD‘s certification training to GAO guidance
for effective training programs and analyzed policies, data, and
previous reports on acquisition training.
What GAO Found:
DOD‘s certification training program”provided by the Defense
Acquisition University (DAU)”generally demonstrates the capacity to
provide effective training, though some attributes of an effective
training program are lacking. DAU ensures that strategic and tactical
changes are promptly incorporated into training; uses centralized and
decentralized training approaches in design and development; collects
data during implementation to ensure feedback on its training programs;
and analyzes its training during evaluation. However, DOD lacks
complete information on the skill sets of the current acquisition
workforce and does not have metrics to assess results achieved in
enhancing workforce proficiency and capability through training
efforts. In 2009, GAO recommended that the Secretary of Defense
identify and update on an ongoing basis the number and skill sets of
the total acquisition workforce”including civilian, military, and
contractor personnel”that the department needs to fulfill its mission.
DOD agreed and to date has completed about one fifth of its workforce
competency assessments.
At the end of fiscal year 2009, 90 percent of acquisition workforce
personnel had completed required certification training or were within
required timeframes to do so, according to DAU data. However, DAU
reports that it cannot provide for all training requested for the
entire acquisition workforce. DAU has offered more courses in recent
years, and high-priority personnel”those needing to complete classes
for certification in their current position”constitute the majority in
DAU classes. DAU plans the number and location of its classes based on
data that DOD officials noted is generally incomplete when submitted,
and DAU must adapt during the year to support new requirements as they
are identified. DAU has identified the need for a new, integrated
student information system that will provide better insight into the
workforce it supports and is in the early stages of its procurement.
DOD reports that most of the training-related recommendations from
previous reviews”the Gansler Commission, the Panel on Contracting
Integrity, and a prior GAO report”have been fully implemented and some
actions are still underway. DOD has either fully or partially
implemented 15 of the 19 recommendations GAO reviewed. Both the Army
and the Office of the Secretary of Defense have taken steps to respond
to the Gansler Commission recommendations. Most of the recommendations
made by the Panel on Contracting Integrity have been implemented, with
the exception of two recommendations related to assessing guidance and
reviewing a specific training topic. GAO made four recommendations
pertaining to Defense Contract Audit Agency‘s government auditing
standards training and expertise, which have not been implemented, but
some actions have been taken on two recommendations related to
developing agencywide training on government audit standards.
What GAO Recommends:
GAO recommends DOD establish milestones for developing metrics to
measure how certification training improves acquisition workforce
capability and a timeframe for acquiring and implementing an integrated
information system.
View [hyperlink, http://www.gao.gov/products/GAO-11-22] or key
components. For more information, contact John Needham at (202) 512-
4841 or NeedhamJK1@gao.gov
[End of section]
Contents:
Letter:
Background:
DOD's Certification Training Program Has the Capability to Provide
Training, Although It Lacks Some Attributes of Effective Training:
Despite Incomplete Data and High Demand, Most Workforce Members Receive
Training Required for Certification:
DOD Has Addressed Most Previous Training Recommendations, However, Some
Remain to Be Implemented:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Attributes of Effective Training and Development Programs:
Appendix III: Comments from the Department of Defense:
Appendix IV: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Defense Acquisition Workforce Personnel by Career Field in
Fiscal Year 2009:
Table 2: SPRDE--Systems Engineering Certification Requirements by
Career Level:
Table 3: Implementation Status of Gansler Commission Training
Recommendations:
Table 4: Panel 2008 and 2009 Recommended Training Actions' Status:
Table 5: GAO Training Recommendations' (for DCAA) Status:
Figures:
Figure 1: Components of the Training and Development Process:
Figure 2: DOD's Multifaceted Training Intended to Prepare Acquisition
Personnel to Perform the Job:
Figure 3: DAU Graduates, Fiscal Year 2005-2009:
Abbreviations:
DAU: Defense Acquisition University:
DAWIA: Defense Acquisition Workforce Improvement Act:
DACM: Director of Acquisition Career Management:
DCAA: Defense Contract Audit Agency:
DCAI: Defense Contract Audit Institute:
DOD: Department of Defense:
OSD: Office of the Secretary of Defense:
USD(AT&L): Office of the Under Secretary of Defense for Acquisition,
Technology and Logistics:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
October 28, 2010:
Congressional Committees:
The President has announced the administration's intention to achieve
improvements in the acquisition system. Current fiscal challenges
highlight the need to scrutinize the processes used by the federal
government to spend half a trillion dollars in fiscal year 2009
acquiring goods and services. The Department of Defense's (DOD)
spending on goods and services has more than doubled since 2001 to
nearly $384 billion;[Footnote 1] however, the number of acquisition
personnel has remained relatively stable. According to the Office of
the Under Secretary of Defense for Acquisition, Technology and
Logistics (USD(AT&L)), to successfully accomplish the acquisition
mission and ensure the best value for the expenditure of public
resources, DOD will place greater emphasis on having a high-quality
workforce with the right competencies and skill sets.[Footnote 2] In
addition to building the capability of the existing workforce, the
Secretary of Defense plans to augment the capacity of the defense
acquisition workforce by increasing its numbers by 20,000 employees
over the next 5 years.[Footnote 3]
In 1990 the Defense Acquisition Workforce Improvement Act (DAWIA)
established career paths for the improvement of training for the
acquisition workforce.[Footnote 4] Given the enduring critical role of
the acquisition workforce in achieving the administration's goal of
improving federal acquisition, it is important they have the skills and
support needed to do their jobs. Accordingly, in the National Defense
Authorization Act for Fiscal Year 2010,[Footnote 5] Congress mandated
that GAO study the efficacy of DOD's training for its acquisition and
audit workforce. As agreed with your respective offices, we assessed
(1) DOD's capability to provide defense acquisition workforce
certification training, (2) the extent that such training reaches
members of DOD's acquisition workforce, and (3) the extent that
training recommendations from previous reviews, including the Gansler
Commission, have been implemented.[Footnote 6]
To determine DOD's capability to provide acquisition certification
training, we focused on training for DOD personnel covered under
DAWIA.[Footnote 7] We compared DOD's certification training programs
and processes with the attributes of effective training and development
programs identified in our previous work on strategic training and
development efforts in the federal government.[Footnote 8] We
interviewed officials at the Defense Acquisition University (DAU) and
Defense Contract Audit Institute (DCAI) to obtain an understanding of
their training programs and processes, and we obtained documents--such
as briefings, guidance, strategic plans, and course catalogs--
describing the training programs and processes. In addition, we
selected a nongeneralizable sample of one command from each military
service near a DAU regional office--based on level of procurement
dollars spent and number of DAU courses completed in fiscal years 2008
through 2009--and program offices within those commands that would
likely have a large cross-section of acquisition workforce personnel to
obtain customer perspectives on the effectiveness and usefulness of the
DAU training and to determine the use of supplementary training. We
also interviewed Defense Contract Management Agency officials to obtain
their views on DAU training. Finally, we visited a nongeneralizable
sample of two Defense Contract Audit Agency (DCAA) locations to obtain
customer perspectives on DCAI training. We did not examine the
appropriateness of the certification training itself nor the content of
courses required for certification.
To assess the extent to which acquisition training reaches acquisition
personnel, we reviewed DAU and DCAI policies on determining training
requirements, allocating resources, and scheduling classes. We
collected and analyzed defense acquisition workforce and training data
maintained in the AT&L Data Mart system[Footnote 9] used by DAU to
determine course demand and certification status of acquisition
workforce members. This provided an understanding of the number of
class requests received, class seats scheduled, and students who
registered and completed these courses in past fiscal years. We
assessed the reliability of these data by reviewing data query
information and interviewing knowledgeable officials who collect and
use these data. We determined that the data were sufficiently reliable
for the purposes of this report. In addition, we interviewed DAU
officials and obtained budget documents to determine DOD's use of the
Defense Acquisition Workforce Development Fund for training.
To determine the extent to which training recommendations from previous
reviews have been implemented, we identified previous reviews and
interviewed and obtained documentation from agency officials on the
status of DOD's implementation of the training recommendations.
We conducted this performance audit from December 2009 to September
2010 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our audit objectives. See
appendix I for additional details on our objectives, scope, and
methodology.
Background:
Congress passed DAWIA in 1990 to ensure effective and uniform
education, training, and career development of members of the
acquisition workforce.[Footnote 10] Accordingly, the act established
DAU to provide training for the DOD acquisition workforce and charged
DOD officials with designating acquisition positions, setting
qualification requirements, and establishing policies and procedures
for training the acquisition workforce. DOD, as part of implementing
DAWIA, established career fields, such as program management (See table
1).
Table 1: Defense Acquisition Workforce Personnel by Career Field in
Fiscal Year 2009:
Career field: Auditing[A];
Army: 0;
Navy/Marine Corps: 0;
Air Force: 0;
Defense agencies: 3,777;
Total: 3,777.
Career field: Business--cost estimating and financial management[B];
Army: 2,771;
Navy/Marine Corps: 2,286;
Air Force: 1,845;
Defense agencies: 360;
Total: 7,262.
Career field: Contracting;
Army: 8.391;
Navy/Marine Corps: 5,516;
Air Force: 7,443;
Defense agencies: 6,305;
Total: 27,655.
Career field: Facilities engineering;
Army: 719;
Navy/Marine Corps: 4,683;
Air Force: 6;
Defense agencies: 12;
Total: 5,420.
Career field: Industrial/contract property management;
Army: 92;
Navy/ Marine Corps: 73;
Air Force: 29;
Defense agencies: 281;
Total: 475.
Career field: Information technology;
Army: 1,843;
Navy/Marine Corps: 1,240;
Air Force: 966;
Defense agencies: 309;
Total: 4,358.
Career field: Life-cycle logistics;
Army: 7,952;
Navy/Marine Corps: 4,784;
Air Force: 1,989;
Defense agencies: 127;
Total: 14,852.
Career field: Production, quality, and manufacturing;
Army: 1,930;
Navy/Marine Corps: 2,064;
Air Force: 389;
Defense agencies: 4,640;
Total: 9,023.
Career field: Program management;
Army: 3,452;
Navy/Marine Corps: 4,598;
Air Force: 4,461;
Defense agencies: 911;
Total: 13,422.
Career field: Purchasing;
Army: 330;
Navy/Marine Corps: 567;
Air Force: 146;
Defense agencies: 195;
Total: 1,238.
Career field: System planning, research, development, and engineering
(SPRDE)--science and technology manager[C];
Army: 204;
Navy/Marine Corps: 243;
Air Force: 51;
Defense agencies: 125;
Total: 623.
Career field: SPRDE--systems engineering;
Army: 10,175;
Navy/Marine Corps: 18,003;
Air Force: 7,113;
Defense agencies: 1,179;
Total: 36,470.
Career field: SPRDE--program system engineer;
Army: 33;
Navy/Marine Corps: 82;
Air Force: 84;
Defense agencies: 35;
Total: 234.
Career field: Test and Evaluation;
Army: 2,235;
Navy/Marine Corps: 2,833;
Air Force: 2,630;
Defense agencies: 194;
Total: 7,892.
Career field: Unknown;
Army: 229; Navy/Marine Corps: 0;
Air Force: 22;
Defense agencies: 151;
Total: 402[D].
Career field: Total;
Army: 40,356;
Navy/Marine Corps: 46,972;
Air Force: 27,174;
Defense agencies: 18,601;
Total: 133,103.
Source: AT&L Workforce Data Mart.
[A] DAU oversees the delivery of certification training for all of the
DAWIA career fields except auditing. The Director, Defense Contract
Audit Agency, under the Under Secretary of Defense (Comptroller),
provides a program of education, training, career development, and
workforce management at the Defense Contract Audit Institute for the
auditor career field, in coordination with DAU, as appropriate.
[B] Effective October 1, 2009, the Business--cost estimating and
financial management career field was split into two career fields:
Business--cost estimating and Business--financial management, bringing
the number of career fields to 15.
[C] System planning, research, development, and engineering (SPRDE) has
three distinct career paths: (1) SPRDE--science and technology manager,
(2) SPRDE--systems engineering, and (3) SPRDE--program system engineer.
[D] 402 records submitted with no career field data.
[End of table]
The act also required DOD to establish career paths, referred to by DOD
as certification requirements, for the acquisition workforce.[Footnote
11] DOD military services and defense agencies must track that
acquisition workforce members meet mandatory standards established for
level I (basic or entry), level II (intermediate or journeyman), or
level III (advanced or senior) in a career field, such as contracting,
life-cycle logistics, and program management. DAU is responsible for
certification training and for designing, maintaining, and overseeing
the delivery of certification training courses at each level, among
other things. For each career field and level, there are requirements
in three areas--education, experience, and training. Certification
requirements are the same for civilian and military acquisition
workforce members. Table 2 shows the nature of certification training
for one of the DAWIA career fields--system planning, research,
development, and engineering (SPRDE)--systems engineering, as well as
shows the education and experience requirements for each level in the
career field.
Table 2: SPRDE--Systems Engineering Certification Requirements by
Career Level:
Career level: I;
Training (class titles and codes): Fundamentals of Systems Acquisition
Management (ACQ 101); Fundamentals of Systems Planning, Research,
Development, and Engineering (SYS 101);
Education: Baccalaureate or graduate degree in a technical or
scientific field such as engineering, physics, chemistry, biology,
mathematics, operations research, engineering management, or computer
science;
Experience: One year of technical experience in an acquisition position
from among the following career fields: SPRDE--systems engineering
(SE); SPRDE--science and technology manager (S&TM); information
technology (IT); test and evaluation (T&E); production, quality, and
manufacturing (PQM); facilities engineering (FE); program management
(PM); or life-cycle logistics (LCL); Similar experience gained from
other government positions or industry is acceptable as long as it
meets the above standards.
Career level: II;
Training (class titles and codes): Intermediate Systems Acquisition,
Part A (ACQ 201A); Intermediate Systems Acquisition, Part B (ACQ 201B);
Intermediate Systems Planning, Research, Development, and Engineering,
Part I (SYS 202); Intermediate Systems Planning, Research, Development,
and Engineering, Part II (SYS 203); Technical Reviews (CLE 003);
Education: Same as level I;
Experience: Through September 30, 2010: Two years of technical
experience in an acquisition position, including government or industry
equivalent, from among the career fields listed for level I; Effective
October 1, 2010: Two years of technical experience in an acquisition
position. Of that, at least 1 year in a SPRDE-SE, SPRDE-program system
engineer (PSE), or SPRDE-S&TM position; remainder may come from IT,
T&E, PQM, PM, or LCL; Similar experience gained from other government
positions or industry is acceptable as long as it meets the above
standards.
Career level: III;
Training (class titles and codes): Technical Leadership in Systems
Engineering (SYS 302); Designing for Supportability in DOD Systems (CLL
008);
Education: Same as level I;
Experience: Through September 30, 2010: Four years of technical
experience in an acquisition position, to include government or
industry equivalent, from among the career fields listed for level I;
Effective October 1, 2010: Four years of technical experience in an
acquisition position. Of that, at least 3 years in a SPRDE-SE, SPRDE-
PSE, or SPRDE-S&TM position; remainder may come from IT, T&E, PQM, PM,
or LCL; Similar experience gained from other government positions or
industry is acceptable as long as it meets the above standards.
Source: GAO (presentation); DAU Web site (data).
[End of table]
Besides the certification training it offers, DAU approves alternative
certification training providers based on a review by an independent
organization--the American Council on Education--of the capability of a
potential provider to offer acquisition training and whether the
provider's course content addresses the DAU course's learning
outcomes.[Footnote 12] An equivalent course provider must certify
annually that its course is current with the DAU plan of instruction
for the course. Similarly, DCAI provides both required certification
training and supplemental training for the auditor career field.
In addition to certification training, DAU offers supplemental training
for each career field and for particular types of assignments. For
example, for level II contracting in contingency or combat operations,
DAU provides courses such as a contingency contracting simulation, a
contingency contracting officer refresher, and a joint contingency
contracting course. DAU also provides continuous learning modules
online to provide acquisition workforce members with a quick reference
for material already introduced and courses to help them maintain
currency in their career field by achieving the required 80 continuous
learning points biennially. Additionally, DAU provides consulting
support to program offices, rapid-deployment training on new
initiatives, and training targeted to the needs of acquisition field
organizations. DAU also engages in knowledge-sharing initiatives,
including hosting a number of acquisition communities of practice and
providing Web-based acquisition policy and reference materials.
In March 2004, we issued a guide for assessing federal training
programs that breaks the training and development process into four
broad, interrelated components--(1) planning and front-end analysis,
(2) design and development, (3) implementation, and (4)
evaluation.[Footnote 13] The guide discusses attributes of effective
training and development programs that should be present in each of the
components and identifies practices that indicate the presence of the
attribute.[Footnote 14] For example, under the design and development
component, to determine whether an organization possesses the attribute
of incorporating measures of effectiveness into courses it designs, the
guide suggests looking for practices, such as (1) clear linkages
between specific learning objectives and organizational results and (2)
well-written learning objectives that are unambiguous, achievable, and
measurable. For a complete list of the attributes of effective training
and development programs, see appendix II.
Figure 1 depicts the training and development process along with the
general relationships between the four components that help to produce
a strategic approach to federal agencies' training and development
programs. These components are not mutually exclusive and encompass
subcomponents that may blend with one another. Evaluation, for example,
should occur throughout the process.
Figure 1: Components of the Training and Development Process:
[Refer to PDF for image: Illustration]
The following chart has three folders on the left side of the
illustration moving downward. There is a fourth folder on the side of
the chart with arrows going from the other three folders and then back
to the fourth folder.
1. Planning/Front-end Analysis:
* Develop a strategic approach that establishes priorities and
leverages investments in training and development to achieve agency
results.
2. Design/Development:
* Identify specific training and development initiatives that, in
conjunction with other strategies, improve individual and agency
performance.
3. Implementation:
* Ensure effective and efficient delivery of training and development
opportunities in an environment that supports learning and change.
4. Evaluation:
* Demonstrate how training and development efforts contribute to
improved performance and results.
Source: GAO.
[End of figure]
DOD's Certification Training Program Has the Capability to Provide
Training, Although It Lacks Some Attributes of Effective Training:
DOD's acquisition workforce certification training---centrally
administered by DAU--has many attributes of effective training programs
that demonstrate the capability to deliver training. DAU's
certification training program has a formal process in planning and
front-end analysis to ensure that strategic and tactical changes are
promptly incorporated into training; use of centralized and
decentralized training approaches in design and development; data
collection during implementation to ensure feedback on its training
programs; and appropriate analytical approaches to assess its training
during evaluation. However, DOD lacks complete information on the skill
sets of the current acquisition workforce for planning and front-end
analysis and does not have metrics to assess results achieved in
enhancing workforce proficiency and capability through training efforts
during evaluation. Complete data on acquisition skill sets are needed
to accurately identify workforce gaps, and appropriate metrics are
necessary to increase the likelihood that desired changes will occur in
the acquisition workforce's skills, knowledge, abilities, attitudes, or
behaviors.
DOD's Certification Training Program Has Many Attributes of Effective
Training Programs:
DOD's certification training program possesses attributes of effective
training programs in each of the four components of the training and
development process. Following are examples of the attributes we
observed in DOD training categorized by the components of effective
training programs.
Planning and front-end analysis: Planning and front-end analysis can
help ensure that training efforts are not initiated in an ad hoc,
uncoordinated manner, but rather are strategically focused on improving
performance toward the agency's goals. DAU had processes to ensure that
training efforts were coordinated and focused on improving agency
goals. Through a formal process that ensures that strategic and
tactical changes are promptly incorporated into training, DAU and other
DOD stakeholders plan for and evaluate the effectiveness of DAU's
training efforts. Each career field has a functional leader, a senior
subject-matter expert in the career field who is responsible for
annually certifying that course content for certification is current,
technically accurate, and consistent with DOD acquisition
policies.[Footnote 15] Functional leaders are supported by a functional
integrated process team for each career field, which consists of
subject-matter experts, acquisition career management representatives
from the military services and other DOD agencies, and DAU
representatives. The functional integrated process team analyzes and
reviews data, including end-of-course evaluations, number of students
completing a class, wait lists, and certification rates, as well as DOD
policy changes and recommendations from reviews, such as the Gansler
Commission to support functional leaders. DAU designs courses in
accordance with the functional leader and functional integrated process
team decisions. Using this process, strategic and tactical changes were
promptly incorporated into training. For example, DAU developed and
fielded a new contracting course on federal acquisition regulation
fundamentals within a year of direction by the functional leader's
organization to create it.
Design and development: In design and development, it is important that
agencies consciously consider the advantages and disadvantages of using
centralized and decentralized approaches. Centralizing design can
enhance consistency of training content and offer potential cost
savings. DAU evaluates and uses centralized and decentralized
approaches for training after considering the advantages and
disadvantages. DAU's curriculum development and technologies
organizations located at Fort Belvoir, Virginia, provide centralized,
integrated design and development of certification courses. These
courses are then delivered to the acquisition workforce by five
regionally-oriented campuses and the Defense Systems Management College
School of Program Managers. DAU also compares training delivery
mechanisms to determine the appropriate use of different delivery
mechanisms (such as classroom or computer-based training) and to ensure
efficient and cost-effective delivery. In addition, supplementary
training is offered at the Army, Navy, and Air Force commands and
program offices we visited, as well as at the Defense Contract
Management Agency. While DAU provides a foundation for acquisition and
career field knowledge in its certification training, various
decentralized sources provide supplementary training more targeted to
specific jobs, such as training on service-specific processes or
databases and technical topics. Acquisition workforce members at the
commands we visited provided the following examples of supplementary
training.
* The contracting offices at the Army Aviation and Missile Command
(AMCOM), Alabama, and the Air Force Aeronautical Systems Center (ASC),
Ohio, provided unique training in the contracting area. AMCOM's
Contracting Center University teaches employees how to do day-to-day
tasks associated with their job, such as price analysis, price
negotiation, and how to use the Army Materiel Command-unique system for
preparing contract documents. ASC's "jump start" program teaches,
reinforces, and supplements DAU certification training in the
contracting career field with illustrative examples not provided in the
computer-based contracting courses as well as offers an opportunity to
interact with instructors and other students.
* The Naval Air Systems Command (NAVAIR), Maryland, provides
supplementary training for DAWIA career fields. For example, in the
program management career field, NAVAIR offers courses in configuration
management and on NAVAIR's technical directives system.
* Other acquisition workforce members provided examples of training
from other federal agencies or commercial vendors, such as financial
training from the Graduate School, United States Department of
Agriculture, and Management Concepts, while others said they had brown
bag lunches on various topics.
Figure 2 below identifies DOD's multifaceted training approach, both
centralized and decentralized. The objective of the multifaceted
training, in conjunction with the other two certification components--
education and experience--is acquisition personnel with the training,
education, and experience to perform the acquisition job.
Figure 2: DOD's Multifaceted Training Intended to Prepare Acquisition
Personnel to Perform the Job:
[Refer to PDF for image: Chart]
[Bottom] Objective: Acquisition personnel with the training, education,
and experience to perform the job;
Lines drawn up from Objective leading to:
On-the-job training;
Programs;
Outside vendors;
[Highlighted] Education;
[Highlighted] DAU;
[Highlighted] Experience;
Universities;
Commands;
Services.
Source: GAO analysis od DOD information on acquisition training.
[End of figure]
Implementation/Evaluation: As with other programs and services that
agencies deliver, it is important that agencies collect program
performance data during implementation and select an analytical
approach that best measures the program's effect to evaluate their
training and development efforts. DAU collects customer feedback data
during implementation and, during evaluation, uses the four-level
Kirkpatrick model[Footnote 16] as an analytical approach for measuring
training effectiveness. As a part of evaluating training, DAU conducts
student end-of-course surveys (Level 1-Reaction) and, to a lesser
degree, follow-up surveys of students and their managers 60 and 120
days, respectively, after course completion (Level 3-Behavior). DAU
tracks the scores from the various surveys by survey section, such as
job impact, and uses red-yellow-green stoplight indicators to identify
areas of concern overall and by specific courses. DAU also administers
pre-and post-training tests to measure learning (Level 2-Learning). To
measure organizational impact (Level 4-Business Results), DAU employs
measures of efficiency in evaluating and analyzing multiyear data, such
as number of students completing courses, cost efficiency, and customer
satisfaction trends. Level 4 assessments are resource intensive and
have not been extensively used by DAU.
Some Attributes of DOD's Certification Training Programs are
Inadequate:
DOD is deficient in two attributes of an effective training program--
determining the skills and competencies of its workforce for planning
and front-end analysis and using performance data to assess the results
achieved through training efforts during evaluation. In March 2009, we
reported that USD(AT&L) lacks complete information on the skill sets of
the current acquisition workforce and whether these skill sets are
sufficient to accomplish DOD's missions.[Footnote 17] We recommended
and DOD agreed to identify and update on an ongoing basis the number
and skill sets of the total acquisition workforce--including civilian,
military, and contractor personnel--that the department needs to
fulfill its mission. Complete data on skill sets are needed to
accurately identify its workforce gaps. Not having these data limits
DOD's ability to make informed workforce allocation decisions. We
reported that USD(AT&L) was conducting a competency assessment to
identify the skill sets of its current acquisition workforce, but also
found that the lack of key data on the in-house acquisition workforce
identified in the prior report still exists, though progress has been
made. Since we released that report, DOD issued its Strategic Human
Capital Plan Update in April 2010. According to DOD, progress was made
in completing over 22,000 assessments involving 3 of the 15 career
fields--program management, life-cycle logistics, and contracting
career fields. The assessments completed to date represent
approximately one-fifth of the personnel and career fields.
Although DAU uses performance data--including customer feedback, number
of students completing classes, and cost--to assess the results
achieved through training efforts during evaluation, USD(AT&L) has only
partially established metrics required in 2005 by its own guidance to
provide senior leaders with appropriate oversight and accountability
for management and career development of the acquisition
workforce.[Footnote 18] The purpose of these metrics is to help DOD
ensure a sufficient pool of highly qualified individuals for
acquisition positions and, therefore, relates to the knowledge, skills,
abilities, and size of the acquisition workforce, while the DAU
performance data measure the performance of DAU against its goals. By
incorporating these metrics into the training and development programs
they offer, DOD can better ensure that they adequately address training
objectives and thereby increase the likelihood that desired changes
will occur in the acquisition workforce's skills, knowledge, abilities,
attitudes, or behaviors. AT&L programs lacking appropriate outcome
metrics will be unable to demonstrate how the certification training
contributes to organizational performance results. According to
USD(AT&L)'s Deputy Director for Human Capital Initiatives, DOD has
established some metrics to measure the size of the acquisition
workforce that partially satisfy the requirements identified in DOD
Instruction 5000.66. For example, DOD measures the cumulative number of
civilian and military acquisition positions added as a result of in-
sourcing acquisition functions performed by contractors.[Footnote 19]
However, for metrics related to acquisition workforce proficiency and
capability, there are no discernable targets, except improvement over
the previous year. In addition, DOD's April 2010 Strategic Human
Capital Plan Update identified an initiative to establish certification
goals as a management tool for improving workforce quality by June 10,
2010. According to the Deputy Director, certification goals are being
discussed but they had not been established at the time of this report.
Despite Incomplete Data and High Demand, Most Workforce Members Receive
Training Required for Certification:
Although DAU is unable to provide all training requested for
acquisition workforce personnel and receives incomplete data for
planning its training schedule, most personnel who need required DAWIA
certification training receive it within required time frames. DAU
plans the number and location of its classes based on data submitted by
the Directors of Acquisition Career Management (DACM). However, DOD
acquisition and training officials noted that data are generally
incomplete when submitted and additional steps must be taken during the
year to meet new requirements as they are identified. DAU has
identified the need for an integrated student information system to
improve the quality of the data and to provide greater insight into the
workforce it supports. Additionally, though the number of DAU course
graduates has grown over the past 5 years,[Footnote 20] DAU has not
been able to provide enough class seats to meet the training
requirements reported by military departments and defense components.
DAU Receives Incomplete Information to Develop an Annual Teaching
Schedule:
DAU receives annual DACM data submissions for the course scheduling
process, but the submissions do not provide the exact information
needed to determine training demand for the acquisition workforce. DAU
receives class requirements data annually from the DACM offices that it
uses when developing course schedules to identify the number and
location of DAU courses. DACM offices compile this information for all
offices to establish the overall demand for each military department
and the defense agencies for each DAU course. DAU and DACM offices work
together throughout the process to improve the accuracy of this
information when possible. According to DAU and DACM officials,
however, data that are transmitted for schedule development do not
fully reflect all demand for the upcoming year as new requirements
arise once the schedule is developed. As a result additional planning
and coordination between DAU and DACM offices is necessary to meet the
training requirements of the acquisition workforce. For example, in
fiscal year 2009, DAU received requests for 142 additional classes
outside of the normal scheduling process. DAU was able to support 45 of
these requested classes in such areas as program management,
contracting, business management, and logistics. According to DAU
officials, resources for additional classes are made available when
other classes are cancelled. Also, DAU may reallocate allotted
classroom seats among departments and agencies to fill additional
training needs. DAU officials stated that data on selected acquisition
support services that are currently performed by contractors who may
transition to in-house DOD personnel are not adequate for planning
specific training requirements. Though DOD has established goals for
the number of contracted personnel to be converted, DAU officials noted
that the exact time or training backgrounds of the personnel are not
known in advance.
DAU also uses acquisition workforce data provided quarterly by the DACM
offices that include information such as the number of personnel in
each acquisition career field as well as the career level, job titles,
and status of progress against certification requirements of each
workforce member to inform course demand management. According to DAU
officials, these data provide a snapshot of the acquisition workforce
and certification status, and they use this information to estimate the
number, location, and type of classes needed by the acquisition
workforce for certification. The data are compiled to create a demand
management tool that provides DAU with an imprecise estimate of course
requirements and are used to supplement and inform the estimates
developed during the scheduling process. However, this demand
management tool alone cannot be used by DAU to determine the exact
number of classroom seats required each fiscal year. According to DAU
officials, the workforce data collected may overstate training
requirements because it does not account for training that has already
been completed when individuals held a previous acquisition position,
nor does it discern between multiple classes that may fulfill the same
training requirement.
Citing incomplete data for scheduling, as well as other deficiencies,
DAU has taken steps to procure a student information system that will
improve insight into and enhanced management of the defense acquisition
workforce's training needs. DAU began its market research for an
integrated student information system in December 2007, viewed vendor
presentations and demonstrations throughout 2008, and issued a request
for proposal in August 2010. In the request for proposal, DAU
identified the need for an integrated system for registration, student
services, career management, schedule management, catalog requirements,
recording transcripts, and reporting intended to improve its management
of training needs and schedules. Without an integrated system, DAU
states that it will remain reliant on a web of decentralized
information that makes reporting and trend analysis difficult and time-
consuming. A primary goal of the new system is to provide a
comprehensive approach that improves, among other things, tracking of
certification status and ensures training reaches the right people at
the right time. DAU plans call for the contractor to complete
implementation of this new student information system 24 months after
the date of contract award, which had not been made as of September
2010.
DCAI develops its training schedule based on the requirements expressed
in the individual development plans and availability of DCAI resources.
Registration for DCAI courses is prescribed and based largely on the
individual developmental plans submitted by DCAA's approximately 3,700-
member auditing workforce in fiscal year 2009. Each year DCAA employees
develop an individual development plan that lists DCAI courses as well
as outside training deemed necessary with the input and approval of
their supervisors. This information is input into a system that tracks
course requirements and individuals' status against training
requirements. Individuals are automatically enrolled into the scheduled
DCAI courses.
Personnel Receive Required Training within Required Time frames
According to DAU Data, However, DAU Cannot Support All Training
Requirements for the Defense Acquisition Workforce:
Most of the acquisition workforce receives training within required
certification time frames. At the end of fiscal year 2009,
approximately 90 percent of the 133,103 members of the defense
acquisition workforce had met certification requirements associated
with their position or were within allowed time frames to do so.
Acquisition workforce members we met with from all three military
departments and the Defense Contract Management Agency (DCMA) noted
challenges for receiving training at the time and location they
desired, noting that local DAU locations would fill up quickly and that
they would often have to register for courses multiple times prior to
enrollment. However, acquisition staff and supervisors told us that
this had little effect on being certified within the required time
frames for their current positions. Nearly all of the remaining
uncertified personnel required training to become certified. While
additional training was needed, these individuals may also have been
deficient in meeting education or experience requirements also needed
for DAWIA certification. Furthermore, DACM officials noted that there
could be a number of reasons these individuals had not received
required training and stated that while some individuals may not have
adequately planned for their training needs, other factors, such as
deployment of military personnel abroad, may have limited their access
to training.
DCAA auditors do not face the same issues with DAWIA certification as
the rest of the acquisition workforce. According to DCAI officials,
this is largely because they do not have to coordinate demand for
courses across several different agencies. All new hires are
automatically enrolled in the courses required for level I and level II
DAWIA certification.[Footnote 21] Additionally, DOD reported that
approximately 99 percent of the auditing workforce had met
certification requirements or were within allowed time frames to do so.
By completing the mandatory learning track taught through DCAI classes,
DCAA auditors complete certification training within required time
frames.
Even though 90 percent of the acquisition personnel who required
certification training for their current position received training on
time or were within allowed time frames to do so, DAU acknowledges that
requests for acquisition workforce training as a whole submitted by the
DOD components and military departments exceed what DAU can provide.
DAU has incorporated expansion of training into its strategic plans. In
its Strategic Plan for 2010-2015, DAU notes that it will play a key
role in the USD(AT&L) acquisition workforce growth strategy. For
example, USD(AT&L) efforts to grow, train, and develop the defense
acquisition workforce will affect DAU's strategic planning over the
next several years. DAU notes that workforce growth goals put forth by
the Secretary of Defense in April 2009 will increase the demand for DAU
training and therefore affect how DAU plans for development of
acquisition personnel, requiring careful consideration of resource
allocation. The strategic plan also points out a number of other
factors that will drive the demand for acquisition workforce training
in the coming years, including annual workforce turnover, turnover
related to Base Realignment and Closure, and support for new
acquisition development needs. As part of its strategy, DAU has also
established short-term goals to expand training capacity in its fiscal
year 2010 Organizational Performance Plan, including expanding
classroom training by 10,000 seats over fiscal year 2009 levels. DAU
officials stated that they plan to increase capacity further to provide
54,000 classroom seats in fiscal year 2011. In addition, DAU
established and has fulfilled a strategic goal of graduating 150,000
students from its Web-based courses annually.
DAU has increased the total number of course graduates and classes in
recent years to address demand for acquisition training. DAU has
supported more classes than in the past, seeing an increase from 1,279
classroom courses in fiscal year 2005 to 1,505 in fiscal year 2009. In
addition, from fiscal year 2005 through 2009, the number of individual
graduates from DAU classroom and Web-based courses rose by
approximately 77 percent (see fig. 3).
Figure 3: DAU Graduates, Fiscal Year 2005-2009:
[Refer to PDF for image: bar graph]
Fiscal year: 2005;
Classroom: 34,587;
Web-based: 75,079.
Fiscal year: 2006;
Classroom: 35,697;
Web-based: 77,582.
Fiscal year: 2007;
Classroom: 33,191;
Web-based: 90,600.
Fiscal year: 2008;
Classroom: 35,861;
Web-based: 118,391.
Fiscal year: 2009;
Classroom: 39,568;
Web-based: 15,4399.
Source: AT&L Data Mart.
[End of figure]
To support increases in certification training demand due to workforce
growth through new hiring and in-sourcing, DAU uses funding from the
Defense Acquisition Workforce Development Fund to provide additional
facilities and courses.[Footnote 22] Though the majority of funding is
intended to support the hiring of new staff, DAU, military departments,
and defense agencies received more than $225 million dollars to support
new training and additional seats in fiscal years 2008 and 2009. Funds
have been used by the military departments to support Army and Navy
acquisition boot camps,[Footnote 23] the Air Force's mission-ready
contracting course, and other acquisition training developed by
specific military commands. For example, funding was used to develop
and implement the "jump start" program at the Air Force's Aeronautical
Systems Center that combines material taught through DAU's Level I
contracting courses with Air Force-specific information. The Defense
Acquisition Workforce Development Fund has also been used by DAU to
expand its teaching facilities, hire additional instructors, and
schedule additional classes needed for DAWIA certification. DAU
received nearly $165 million in fiscal years 2008 and 2009 to expand
training. In fiscal year 2009, this funding permitted DAU to offer
nearly 7,000 additional classroom seats in 31 different courses. DAU
has also used these funds to develop new training--such as a 4-week
course focusing on the Federal Acquisition Regulation that senior DOD
contracting officials said was needed to provide a foundation for
acquisition fundamentals--and to support acquisition professionals in
the field through Service Acquisition Workshops and expanded
contingency acquisition training.
Despite these increased class offerings that have accommodated more
graduates, DAU has not been able to provide the total number of
classroom seats that are requested by the defense acquisition workforce
through the DACMs. Classroom seats requested and class seats scheduled
both increased from fiscal year 2007 through 2009. For example, in
fiscal year 2009, DOD components requested 52,998 seats for the
acquisition workforce across 66 different DAU classroom courses; DAU
was able to allocate resources to meet 71 percent of this demand based
on its annual budget. However, DAU made use of the Defense Acquisition
Workforce Development Fund to provide additional classroom seats to
meet the demand for training, allowing them to meet 87 percent of the
workforce's requirement in fiscal year 2009. Further, DAU data
demonstrate that workforce personnel who require certification training
for their current or future position within their career field
constitute a large majority of classroom students graduating from DAU
courses.
DOD Has Addressed Most Previous Training Recommendations, However, Some
Remain to Be Implemented:
DOD reports that most of the training-related recommendations from
previous reviews--the Gansler Commission, the Panel on Contracting
Integrity, and our prior report--have been fully implemented. We
reviewed 19 recommendations addressing some aspect of acquisition
training and found that 11 have been fully implemented, 4 have been
partially implemented, and 4 have not been implemented but action has
been taken.
* Two of the four Gansler Commission Report recommendations have been
implemented; however, the Army and Office of the Secretary of Defense
(OSD) need to take additional steps to ensure the Army "trains as it
fights" and that DAU has the resources it needs to train the
acquisition workforce.
* Nine of the 11 Panel on Contracting Integrity recommendations have
been fully implemented. DOD has taken actions to address performance-
based acquisitions training; however, DOD has not conducted a formal
assessment of its guidance or the training. Also, on the basis of
information from DOD, we could not determine whether it conducted a
review of its Fraud Indicator Training and the Continuity
Book/Contracting Office Transition Plan.
* One of the training-related recommendations we made to DCAA has been
partially implemented, and three have not been implemented but action
was taken. DCAA needs to take further steps to develop appropriate
training for its auditors and it should seek outside expertise in doing
so.
Army and OSD Have Taken Steps to Respond to Gansler Commission Training
Recommended Changes, but Additional Steps Need to Be Taken:
In response to the Gansler Commission report, the Army and OSD have
taken steps to improve training and implement the report's
recommendations. In 2007 the Gansler Commission made 4 overarching
recommendations and, within those 4, the Commission described 35 more
in-depth recommendations on Army acquisition and program management in
expeditionary operations. Four of those in-depth recommendations
pertain to training the DAWIA workforce.[Footnote 24] As shown in table
3, 2 of the commission's training recommendations have been fully
implemented, while the remaining 2 training recommendations require
additional action.
Table 3: Implementation Status of Gansler Commission Training
Recommendations:
Recommendations: "Train as we fight:" exercise for rapid acquisitions,
logistics, and contracting in expeditionary operations.[B];
DOD status[A]: Completed;
Our assessment: Partially implemented.
Recommendations: Develop and field the contract tools needed for the
expeditionary forces;
DOD status[A]: Completed;
Our assessment: Fully implemented.
Recommendations: Focus DAU to train and educate the civilian and
military acquisition, logistics, and contracting workforce for
expeditionary operations;
DOD status[A]: Completed;
Our assessment: Fully implemented.
Recommendations: Provide DAU the necessary resources to accommodate the
Army's emphasis on Level I certification;
DOD status[A]: Completed;
Our assessment: Partially implemented.
Source: GAO analysis and Gansler Commission Report.
[A] The agency status reported in this column is taken from the OSD and
Army reports to Congress, which used different language to report the
status of implementation.
[B] The recommendations' language has been edited; the full language
can be found in Report of the Commission on Army and Program Management
in Expeditionary Operations -Urgent Reform Required ("Gansler
Commission Report") (Washington, D.C.: Oct. 31, 2007).
[End of table]
While DOD has taken action, additional steps are needed to fully
implement the Gansler Commission training recommendations. The
following is our rationale for ongoing efforts needed to continue for
the Army and OSD to fully address the Gansler Commission training
recommendations.
* "Train as we fight:" DOD officials stated that training exercises
include contracting and logistics, incorporate lessons learned, and may
include training for commanders,[Footnote 25] but we could not
determine the extent to which they are included due to lack of
documentation. The Army has mechanisms to capture lessons learned, but
it is unclear how they are incorporated into training exercises. For
example, the Expeditionary Contracting Command informally receives
lessons learned from other Army commands and brigades, but we could not
determine whether and how they are incorporated into training exercises
because they are not tracked or formally documented.
* Provide DAU with needed resources to certify Army individuals
requiring level I certification: DAU and the Army do not have the
needed resources to emphasize level I DAWIA certification, according to
DOD officials. DAU is not adequately funded to meet the acquisition
training demand DOD-wide. For example, according to OSD officials, DAU
is not fully funded to meet the fiscal year 2011 services and defense-
wide agency demand for contracting level I courses. DAU projects
meeting 60 percent of the fiscal year 2011 requested seats for these
level I courses. The Army depends not only on DAU, but also on the Army
Logistics University and the Air Force Mission Ready Airman Course to
provide the contracting training needed to its active component
personnel.
DOD Has Implemented Most Training Recommendations from the Panel on
Contracting Integrity:
DOD has not implemented all recommended actions related to defense
acquisition workforce training included in the Panel on Contracting
Integrity's 2008 and 2009 reports to Congress. The Panel recommended a
total of 49 actions to improve acquisition outcomes. Of these
recommended actions, 11 specifically addressed acquisition training.
See table 4 for a complete list of the recommended actions related to
training included in the Panel's reports to Congress in 2008 and 2009.
Table 4: Panel 2008 and 2009 Recommended Training Actions' Status:
2009 Recommended actions: Use case studies in contracting integrity to
promote discussion and communicate standards in areas of ambiguity;
Panel reported status: Completed;
Our assessment: Fully implemented.
2009 Recommended actions: Have Defense Procurement and Acquisition
Policy and senior contracting leaders develop short-term gap closure
strategies for recruiting, hiring, development, and retention
initiatives for consideration of the Defense Acquisition Workforce
Development Fund;
Panel reported status: Completed;
Our assessment: Fully implemented.
2009 Recommended actions: Assess effectiveness of departmental guidance
and training for executing performance-based acquisition and perform
gap analysis in conjunction with DAU;
Panel reported status: Completed;
Our assessment: Not implemented-action taken.
2009 Recommended actions: Provide updated guidance and training on
competition initiatives and continue emphasis on enhancing competition
for contracts and orders placed under multiple-award contracts;
Panel reported status: Completed;
Our assessment: Fully implemented.
2009 Recommended actions: Update and Web-enable the Joint Contingency
Contracting Handbook; build upon previous efforts and revise
contingency contracting training curriculum as required;
Panel reported status: Completed;
Our assessment: Fully implemented.
2009 Recommended actions: Communicate with contracting officers,
auditors, and DCMA representatives regarding an advanced course on
procurement fraud indicators and determine feasibility of development
during 2009;
Panel reported status: Completed;
Our assessment: Fully implemented.
2008 Recommended actions: Assess need for revised/additional training
on competition requirements and differing pricing alternatives;
Panel reported status: Completed;
Our assessment: Fully implemented.
2008 Recommended actions: Improve training by leveraging Marine Corps
and Air Force training capabilities;
Panel reported status: Completed;
Our assessment: Fully implemented.
2008 Recommended actions: Improve training on how to run a contracting
office in a combat/contingent environment;
Panel reported status: Completed;
Our assessment: Fully implemented.
2008 Recommended actions: Subgroups review Fraud Indicator Training and
Continuity Book/Contracting Office Transition Plan;
Panel reported status: Completed;
Our assessment: Partially implemented.
2008 Recommended actions: Create DAU Training Module on Procurement
Fraud Indicators and Risk Mitigation;
Panel reported status: Completed;
Our assessment: Fully implemented.
Source: GAO analysis and Panel on Contracting Integrity 2008 and 2009
reports.
[End of table]
While the Panel reported that all of the recommended actions had been
completed, we determined that two of the recommended actions pertaining
to training had not been fully implemented; we determined that one was
not implemented, but action was taken, and one has been partially
implemented.
* Assess effectiveness of DOD guidance and training for executing
performance-based acquisition and perform gap analysis in conjunction
with DAU: The report did not indicate if DOD conducted a formal
assessment of departmental guidance or a gap analysis of training. The
Panel's Appropriate Contracting Approaches & Techniques Subcommittee
worked with DAU to determine if training needed to be updated and
collected examples of complex and high-dollar acquisitions and posted
them to an Acquisition Community Connection Web site. The report also
noted that DAU would select the best examples from this group for
inclusion in its web-based integrated training tool.
* Review Fraud Indicator Training and Continuity Book/Contracting
Office Transition Plan: The Panel report did not specifically address
whether a formal review determined specific gaps in training, as
recommended. In 2008, the Panel's Contracting Integrity in a
Combat/Contingent Environment Subcommittee reported that DOD
incorporated transition planning and fraud indicator training into the
Joint Contingency Contracting Handbook and updated DAU's Joint
Contingency Contracting Course.
In addition to the recommendations above that are specific to training,
the Panel recommended other actions that also affected training, one of
which was not fully implemented. The Contractor Employee Conflicts of
Interest Subcommittee reviewed and recommended that the Secretary of
Defense issue guidance to clarify the circumstances in which contracts
risk becoming improper personal services contracts.[Footnote 26] DOD
formed an ad hoc team to respond to the recommendation which focused on
establishing a Defense Federal Acquisition Regulation Supplemental
case, DAU course updates, and a DOD instruction update. While the DOD
instruction was published, the Panel's report did not mention the
status of the Defense Federal Acquisition Regulation Supplemental case
or the DAU course updates.
DCAA Has Not Fully Implemented GAO's Recommendations on Auditor
Training:
In 2009, we made four recommendations regarding DCAA auditor training,
which have not been fully implemented (see table 5 for our full
recommendations). Three of the recommendations have not been
implemented but action was taken, and one has been partially
implemented. As stated in our September 2009 report, DCAA faces many
challenges and fundamental structural and cultural changes related to
developing a strong management environment and human capital strategic
plan. First, we recommended that once DCAA establishes a risk-based
audit approach it will need to develop a staffing plan that identifies
auditor resource requirements including training needs. Second, we
recommended that DCAA establish a position for an expert or consult
with an outside expert on auditing standards to shape audit policy,
provide guidance, and develop training. While DCAA has taken steps to
improve its audit training, such as implementing an initiative to
identify the knowledge, skills, and competencies required for DCAA
auditors and develop training, according to a DCAA official, it has not
yet hired or consulted with an outside expert on auditing to shape its
policies and provide guidance. Third, we recommended that DCAA develop
agencywide training on government audit standards. Agency officials
stated that as of July 2010, DCAA had developed a new online,
introductory course on Generally Accepted Government Auditing Standards
(GAGAS) all DCAA auditors are supposed to complete by September 30,
2010. We are reviewing the new course content and continue to work with
DCAA on planned improvements to address the fundamental structural and
cultural changes previously identified. Fourth, as DCAA's audit quality
assurance program identifies actions needed to address serious
deficiencies and GAGAS noncompliance, we recommended that DCAA provide
training and follow-up to ensure that appropriate corrective actions
have been taken. DCAA has issued audit alerts and provided some
guidance through periodic regional office and field office conferences,
but has not yet incorporated this guidance into the body of its DCAI
audit courses.
Table 5: GAO Training Recommendations' (for DCAA) Status:
Recommendation: Based on a risk-based audit approach, develop a
staffing plan that identifies auditor resource requirements as well as
auditor skill levels and training needs;
DCAA status: NA[A];
Our assessment: Not implemented-action taken.
Recommendation: Establish a position for an expert on auditing
standards or consult with an outside expert on auditing standards to
assist in revising contract audit policy, providing guidance on
sampling and testing, and developing training on professional auditing
standards;
DCAA status: NA;
Our assessment: Not implemented-action taken.
Recommendation: Develop agencywide training on government audit
standards. This training should emphasize the level of assurance
intended by the various types of engagements and provide detailed
guidance on auditor independence, planning, fraud risk, level of
testing, supervision, auditor judgment, audit documentation, and
reporting;
DCAA status: NA;
Our assessment: Not implemented-action taken.
Recommendation: Make appropriate recommendations to address annual
quality assurance review findings of serious deficiencies and GAGAS
noncompliance, provide training, and followup to ensure that
appropriate corrective actions have been taken;
DCAA status: NA;
Our assessment: Partially implemented.
Source: GAO analysis and GAO-09-468.
[A] GAO-09-468 was published in September 2009. Officials from DCAA
stated that they are currently reviewing policy to implement these
recommendations.
[End of table]
Conclusions:
DOD's acquisition workforce training program demonstrates many
attributes of effective training and development programs; however,
there is room for further improvement. DOD recognizes the need to
continue its efforts to assess competencies for its acquisition
workforce. Importantly, if this effort is not completed, DOD will be
limited in its ability to identify gaps in the skill sets of
acquisition personnel, ultimately hampering its ability to effectively
acquire the goods and services it needs to accomplish its mission.
Notably, opportunities exist to improve the measurement of training's
impact on overall organizational performance. If DOD is to fully assess
performance improvements, it needs to go beyond measuring the size of
the workforce. To provide appropriate oversight of the proficiency and
capability of its acquisition workforce, DOD will need metrics to
measure skills, knowledge, and abilities, and how certification
training contributes to organizational performance results.
Furthermore, DAU faces challenges with the management and forecasting
of training demand data for specific training courses, which hinders
its ability to accurately plan the course schedule for the upcoming
year in a manner that will facilitate getting the required training to
acquisition workforce members in an efficient and cost-effective
manner. Accurate and timely forecasting of acquisition workforce
training requirements and the development of metrics for the
proficiency of the workforce are imperative to support DOD's
initiatives to improve and grow the acquisition workforce.
Recommendations for Executive Action:
We recommend that the Secretary of Defense direct USD(AT&L) to take the
following two actions to improve the development, implementation, and
evaluation of acquisition workforce training.
* In order to demonstrate and track how training efforts contribute to
improved acquisition workforce performance, establish milestones for
the development of metrics to measure how acquisition certification
training improves the proficiency and capability of the acquisition
workforce.
* In order to improve DOD's ability to identify specific acquisition
training needs for planning and front-end analysis, establish a time
frame for completion and ensure resources are available for
implementing an enterprisewide, integrated student information system.
Agency Comments and Our Evaluation:
We provided a draft of this report to DOD for comment. In written
comments, DOD did not agree with our first recommendation and did agree
with our second recommendation. DOD's comments are discussed below and
are reprinted in appendix III.
DOD did not concur with our recommendation that it should develop
milestones for the development of metrics to demonstrate and track how
acquisition certification training improves the acquisition workforce
performance. While DOD agreed that metrics should be used to measure
the capability of the acquisition workforce, it believes developing
milestones for such metrics is unnecessary because existing metrics can
be used to this end. DOD states that workforce capability is a function
of having the correct number of people working in the right areas with
the proper level of education, training, and experience. Specifically,
DOD notes five metrics used to measure size and composition of the
workforce as well as the education, training, and experience levels of
the individuals that comprise it.
We recognize that metrics for measuring these elements are valuable for
gaining insight into the degree to which required workforce personnel
are being certified and filling needed positions. However, as we note
in this report and in GAO's guidance for assessing strategic training
and development programs, training effectiveness must be measured
against organizational performance. DOD's existing metrics measure the
outputs for certification training, not the outcome in terms of
proficiency or capability of the acquisition workforce. Without outcome
metrics, DOD cannot demonstrate how certification training contributes
to improving organizational performance results. Given the scale and
value of DOD acquisitions, we maintain that metrics that link training
to acquisition performance outcomes should be developed by the
department.
We are sending copies of this report to the Secretary of Defense, the
DOD Inspector General, and other interested parties. The report is also
available at no charge on the GAO Web site at [hyperlink,
http://www.gao.gov].
If you or your staff have any questions about this report, please
contact me at (202) 512-4841 or needhamjk1@gao.gov. Contact points for
our Offices of Congressional Relations and Public Affairs may be found
on the last page of this report. GAO staff who made major contributions
to this report are listed in appendix IV.
Signed by:
John K. Needham:
Director, Acquisition and Sourcing Management:
List of Congressional Committees:
The Honorable Carl Levin:
Chairman:
The Honorable John McCain:
Ranking Member:
Committee on Armed Services:
United States Senate:
The Honorable Daniel Inouye:
Chairman:
The Honorable Thad Cochran:
Ranking Member:
Subcommittee on Defense:
Committee on Appropriations:
United States Senate:
The Honorable Ike Skelton:
Chairman:
The Honorable Howard P. McKeon:
Ranking Member:
Committee on Armed Services:
House of Representatives:
The Honorable Norman D. Dicks:
Chairman:
The Honorable C.W. Bill Young:
Ranking Member:
Subcommittee on Defense:
Committee on Appropriations:
House of Representatives:
[End of section]
Appendix I: Scope and Methodology:
Congress included a provision in the National Defense Authorization Act
for Fiscal Year 2010 requiring us to report on the efficacy of the
Department of Defense's (DOD) acquisition and audit workforce
training.[Footnote 27] To determine the efficacy of DOD's acquisition
and audit workforce training, we assessed (1) DOD's capability to
provide defense acquisition workforce certification training, (2) the
extent that such training reaches members of DOD's acquisition
workforce, and (3) the extent that training recommendations from
previous reviews, including the Gansler Commission, have been
implemented. We were not able to report on the efficacy of training for
the Defense Contract Audit Agency's (DCAA) auditing career field
because DCAA lacks a strategic plan. A strategic plan is a key document
for assessing training programs using the strategic training efforts
attributes.
For this engagement, we focused on training for DOD personnel covered
under the Defense Acquisition Workforce Improvement Act
(DAWIA).[Footnote 28] To assess DOD's capability to provide defense
acquisition workforce certification training, we compared DOD's
certification training programs and processes with the attributes of
effective training and development programs identified in GAO's 2004
guide for assessing strategic training and development efforts in the
federal government, which we identified as the most comprehensive
source for attributes of effective training programs for our
purpose.[Footnote 29] We interviewed officials at the Defense
Acquisition University (DAU) and Defense Contract Audit Institute
(DCAI) to obtain an understanding of their training programs and
processes, and we obtained documents--such as briefings, guidance,
strategic plans, and course catalogs--describing the training programs
and processes. We interviewed the Directors of Acquisition Career
Management (DACM) for the military services and defense agencies to
obtain an understanding of their role in DOD training, to obtain their
views on the effectiveness and usefulness of DAU training, and to find
out whether supplementary training is provided by the military
services. We interviewed the leaders of the functional integrated
process teams that support the functional leaders of the 15 DAWIA
career fields to obtain an understanding of their role in Acquisition,
Technology, and Logistics' (AT&L) process and criteria for reviewing
and approving acquisition workforce training. In addition, we visited
selected military commands and program offices within those commands to
obtain customer perspectives on the effectiveness and usefulness of DAU
training and to determine the use of supplementary training. For this
purpose, we selected a nongeneralizable sample of one command from each
military service based on the following criteria: (1) high level of
procurement dollars spent in fiscal years 2008 and 2009 relative to
other commands in their military service, based on data from the
Federal Procurement Data System-Next Generation; (2) large number of
DAU courses completed in fiscal years 2008-2009; and (3) proximity to a
DAU regional office with an on-site dean. The commands we visited were
the Army Aviation and Missile Command (AMCOM) in Huntsville,
Alabama;[Footnote 30] the Air Force Materiel Command (AFMC) in Dayton,
Ohio; and the Naval Air Systems Command (NAVAIR) at Patuxent River,
Maryland. In selecting program offices to visit, we reviewed our
assessment of selected weapon programs[Footnote 31] and consulted with
the GAO team responsible for our assessment to determine which program
offices would likely have a large cross-section of acquisition
workforce personnel with whom to discuss training. We visited the
following program offices: Joint Attack Munition Systems and Apache at
AMCOM; Broad Area Maritime Surveillance Unmanned Aircraft System and E-
2D Advance Hawkeye at NAVAIR; and Global Hawk Unmanned Aircraft System
at AFMC. At AFMC, we also visited the Aeronautical Systems Center's
Contracting Directorate, and, at AMCOM, we visited the Contracting
Center. We also visited Defense Contract Management Agency personnel to
obtain their perspectives on DAU training and to find out about their
use of supplementary training. Finally, we visited a nongeneralizable
sample of two DCAA locations--the Alabama Branch Office in Huntsville,
Alabama, and the Boston Branch Office in Boston, Massachusetts--to
obtain the customers' perspectives on DCAI training and determine the
use of supplementary training. We did not examine the appropriateness
of the certification training itself nor the content of courses
required for certification. We did not assess the efficacy of training
provided by supplementary training sources.
To assess the extent to which acquisition training reaches appropriate
acquisition personnel, we reviewed DAU and DCAI policies, and we
received briefings from DAU and DCAI personnel concerning the
determination of training requirements, resource allocation, and
scheduling of classes. We reviewed and analyzed the training
requirements for all defense acquisition career fields. We collected
and analyzed defense acquisition workforce and training data maintained
in the AT&L Data Mart system used by DAU for determining course demand
and certification status of acquisition workforce members. This
provided an understanding of the number of class requests received,
class seats scheduled, and students who registered and completed these
courses in past fiscal years. We also used these data to analyze the
number and reasons for uncertified acquisition workforce personnel. We
assessed the reliability of these data by reviewing data query
information for specific data requests and interviewed knowledgeable
officials who collect and use these data. We intended to focus all
analysis of data for fiscal years 2005 through 2009; however, due to
data reliability concerns, we limited portions of our analysis to data
available for fiscal years 2007 through 2009. We determined that data
were sufficiently reliable for the purposes of this report. We
conducted interviews with DAU, DCAI, military department, and defense
agency representatives who have a role in communicating or analyzing
training requirements demand and training resource allocation to gain a
fuller understanding of the processes and challenges faced when
providing training for the defense acquisition workforce. In addition,
we conducted interviews with acquisition workforce members and
supervisors to understand the degree to which they are able to enroll
in needed acquisition training and challenges they may face in
completing this training. We interviewed DAU officials and obtained
budget documents to determine DOD's use of the Defense Acquisition
Workforce Development Fund (Section 852 of the National Defense
Authorization Act for Fiscal Year 2008) for training and for helping to
meet training demand.
To determine the extent to which training recommendations from previous
reviews, including the Gansler Commission, have been implemented, we
identified previous reviews with training recommendations, and we
interviewed and obtained documentation from agency officials on the
status of DOD's implementation of the recommendations. Specifically,
for Gansler Commission recommendations, we interviewed Defense
Procurement and Acquisition Policy (DPAP) officials to determine the
applicability of the training recommendations to the acquisition
workforce, and we obtained the Office of the Secretary of Defense's
(OSD) and the Army's status in implementing the recommendations and
supporting documents, including reports detailing the recommendations
and action items. We analyzed the supporting documents to assess the
status, and, based on our review, we assigned one of the following six
status assessments to each of the recommendations.
(1) Fully Implemented. The entire wording of the action item has been
fulfilled.
(2) Partially Implemented. Only a portion of the action has been
implemented. When the wording of the action item had multiple parts, if
one part or a portion of a part had been implemented (but not all
parts), we categorized the action item as "partially implemented."
(3) Not Implemented-Action Taken. No part of the action item has been
implemented, but steps have been taken toward the completion of the
action item. For example, if legislation had been introduced to address
the action but had not been enacted into law, we categorized the action
item as "not implemented-action taken."
(4) Not Implemented-No Action. No part of the action item has been
completed, and no action has been taken to address the action item. For
example, if the action item called for changes in legislation but no
legislation has even been proposed, we categorized the action item as
"not implemented-no action."
(5) Insufficient Information. Insufficient or conflicting information
prevented us from determining the status of the action item.
(6) Other. Implementation has occurred or action has been taken that,
while not responsive to the letter of the action item, generally was
consistent with its purpose. For example, if the action item states
that a particular position should be created to coordinate an effort
but the coordination is achieved without the creation of the position,
we categorized the action item as "other."
We compared our assessment with OSD's and the Army's assessment, and,
in making our final determination on implementation status, we provided
OSD and Army officials the results of our initial determinations. The
officials reviewed these results and provided us with additional,
clarifying information that we considered and, when we believed
appropriate, used in making our final determination. For the Panel on
Contracting Integrity reports, we examined whether DOD had implemented
the Panel's recommendations in 2007 and 2008 by reviewing the 2007,
2008, and 2009 reports. Specifically, we compared the recommended
actions from the 2007 report with the reported action in the 2008
report. The same comparative analysis was conducted using the
recommended actions from 2008 and the 2009 report. We differentiated
between recommendations that specifically mention training from those
that did not, as well as recommendations in which training was involved
in the implementation of the recommendation. We compared our assessment
with the Panel's assessment. We provided our analysis to DPAP officials
to review and provide additional information that we considered in
making our final determination. To determine whether DCAA has
implemented GAO's recommendations from a prior report, we interviewed
officials at DCAA to understand what actions had been initiated in
response to our recommendations.[Footnote 32]
We conducted this performance audit from December 2009 to September
2010 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our audit objectives.
[End of section]
Appendix II: Attributes of Effective Training and Development Programs:
Table:
Planning/front-end analysis;
(a) Training goals are consistent with its overall mission, goals, and
culture;
(b) Has strategic and annual performance planning processes that
incorporate human capital professionals;
(c) Determines the skills and competencies of its workforce;
(d) Identifies the appropriate level of investment to provide for
training;
(e) Has measures to assess the contributions that training efforts make
toward individual mastery of learning;
(f) Incorporates employees' developmental goals in its planning
processes;
(g) Integrates the need for continuous and lifelong learning into its
planning processes;
(h) Considers governmentwide reforms and other targeted initiatives to
improve management and performance when planning its training programs;
(i) Has a formal process to ensure that strategic and tactical changes
are promptly incorporated into training.
Design/development;
(a) Ensures that training is connected to improving individual and
agency performance in achieving specific results;
(b) The design of the training program is integrated with other
strategies to improve performance and meet emerging demands;
(c) Uses the most appropriate mix of centralized and decentralized
approaches for its training;
(d) Uses criteria in determining whether to design training programs in-
house or obtain from a contractor or other external sources;
(e) Compares the merits of different delivery mechanisms (such as
classroom or computer-based training) and determines what mix to use to
ensure efficient and cost-effective delivery;
(f) Determines a targeted level of improved performance in order to
ensure that the cost of a training program is appropriate to achieve
the anticipated benefit;
(g) Incorporates measures of effectiveness into courses it designs.
Implementation;
(a) Agency leaders communicate the importance of training and
developing employees, and their expectations for training programs;
(b) Has a training and performance organization that is held
accountable, along with the line executives, for the maximum
performance of the workforce;
(c) Agency managers are responsible for reinforcing new behaviors,
providing useful tools, and identifying and removing barriers to help
employees implement learned behaviors on the job;
(d) Selects employees (or provides the opportunity for employees to
self select) to participate in training and development efforts;
(e) The agency considers options in paying for employee training and
development and adjusting employee work schedules so that employees can
participate in these developmental activities;
(f) Takes actions to foster an environment conducive to effective
training;
(g) Takes steps to encourage employees to buy into the goals of
training efforts;
(h) Collects data during implementation to ensure feedback on its
training programs.
Evaluation;
(a) Systematically plans for and evaluates the effectiveness of its
training efforts;
(b) Uses the appropriate analytical approaches to assess its training;
(c) Uses performance data (including qualitative and quantitative
measures) to assess the results achieved through training efforts;
(d) Incorporates evaluation feedback into the planning, design, and
implementation of its training efforts;
(e) Incorporates different perspectives (including those of line
managers and staff, customers, and experts in areas such as financial,
information, and human capital management) in assessing the impact of
training on performance;
(f) Tracks the cost and delivery of its training programs;
(g) Assesses the benefits achieved through training programs;
(h) Compares its training investment, methods, or outcomes with those
of other organizations to identify innovative approaches or lessons
learned.
Source: GAO, Human Capital: A Guide for Assessing Strategic Training
and Development Efforts in the Federal Government, GAO-04-546G
(Washington, D.C.: Mar. 2004).
[End of table]
[End of section]
Appendix III: Comments from the Department of Defense:
Department Of Defense:
Defense Acquisition University:
Office Of The President:
9820 Belvoir Road:
Fort Belvoir, Va:
22060-5565:
October 27, 2010:
Mr. John K. Needham:
Director, Acquisition and Sourcing Management:
U.S. Government Accountability Office:
441 G Street, N.W.:
Washington, DC 20548:
Dear Mr. Needham:
This is the Department of Defense (DoD) response to the GAO draft
report, GAO-11-22, "Defense Acquisition Workforce: Training for Defense
Acquisition Workforce Demonstrates Many Attributes of Effective
Training but Can Be Improved," dated September 24, 2010 (GAO Code
120870).
The Department does not concur with the report's first recommendation
and concurs with the report's second recommendation. Detailed comments
on both recommendations are enclosed.
Sincerely,
Signed by:
James S. McMichael:
Acting President:
Enclosure:
As stated:
GAO Draft Report Dated September 24, 2010:
GAO-11-22 (GAO CODE 120870):
"Defense Acquisition Workforce: Training For Defense Acquisition
Workforce Demonstrates Many Attributes Of Effective Training But Can Be
Improved":
Department Of Defense Comments To The Gao Recommendations:
Recommendation 1: The GAO recommends that the Secretary of Defense
direct the USD(AT&L) to establish milestones for the development of
metrics to measure how acquisition certification training improves the
proficiency and capability of the acquisition workforce. (See page
29/GAO Draft Report.)
DoD Response: Non concur. DoD agrees that there should be metrics to
measure the capability of the Defense Acquisition Workforce. However,
the establishment of milestones for the development of metrics is not
necessary because existing metrics can be used for this purpose. Taken
together, the five metrics shown below measure proficiency and
capability of the acquisition workforce.
Workforce capability is a function of having the right number of people
in the right functional areas with the right education, training, and
experience. The chosen metrics address these elements of capability in
terms of size and composition and in terms of individual education,
training, and experience.
Workforce Size and Composition:
In April 2009, the Secretary of Defense announced a time-phased growth
strategy for the Defense Acquisition Workforce”grow by 20,000 people by
2015. Responding to significantly increased acquisition workload and a
loss of critical skill sets, this initiative was aimed at strengthening
organic core capability, particularly in areas of systems engineering,
program management, logistics, contracting, pricing, and cost
estimating. DoD has used authority granted in Section 852 of the 2008
National Defense Authorization Act to hire people with the right skills
sets to meet the secretary's growth targets. The DoD Strategic
Management Plan contains two metrics related to workforce growth.
DoD Strategic Management Plan Metrics:
* Cumulative increase in the number of DoD civilian and military end
strengths performing acquisition functions.
* Cumulative number of DoD civilian and/or military authorizations
added as a result of in-sourcing acquisition functions.
Workforce Quality”Education, Training, and Experience:
In 1990 Congress passed the Defense Acquisition Workforce Improvement
Act, which required that DoD improve the quality of the acquisition
workforce by establishing education, experience, and training
requirements for each acquisition position. DoD responded to this
requirement by developing a certification construct that incorporated
these three components. Responsible for setting certification standards
and for oversight, DoD-appointed functional leaders and functional
integrated product teams for each acquisition functional area ensure
their respective career fields maintain quality and relevance.
The DoD Strategic Management Plan contains two metrics relating to
certification.
DoD Strategic Management Plan Metrics
* Annual improvement in percent of acquisition positions filled with
personnel meeting or exceeding Level II certification requirements.
* Annual improvement in percent of acquisition positions filled with
personnel meeting or exceeding Level III certification position
requirements.
Education and Experience: Functional leaders review the certification
education and experience requirements on a continuous basis and change
requirements as necessary to enhance the effectiveness and capability
of the acquisition workforce. For example, the experience requirement
for the Systems Planning, Research, Development, and Engineering ”
Program Systems Engineer career path was expanded from four to eight
years. This places greater emphasis on experience as a critical element
in improving workforce quality and capability. Another example is the
creation of a new Cost Estimating career path, which now requires a
bachelor degree and seven years of experience to achieve Level III
certification.
Training: DoD has an extensive workforce training requirements process
in place which involves functional leaders and joint functional teams.
The joint functional teams develop/revise course requirements to ensure
alignment of training with emerging DoD acquisition policy and
initiatives. These teams also provide a rigorous, ongoing quality
assessment of Defense Acquisition University (DAU) course offerings.
This expert-based workforce training requirements process is
strengthened by a data-driven competency-based approach. Once training
requirements are determined, the curricula developers at DAU create new
courses or update existing courses to meet the newly established
requirements. DAU uses a state-of-the-art, end-of-course survey
program, Metrics that Matter (MTM), to assess the quality of its
courses and other learning assets. MTM is a web-based learning
evaluation system with an extensive database of performance benchmarks
gathered from the private and public sectors. In addition to an
extensive end-of-course survey system, DAU also follows up with
students and their supervisors to assess the impact of training on job
performance. MTM collects survey data from all students at the end of
all courses. Surveys are sent to students of selected courses 60 days
after course completion and also to their supervisors 120 days after
course completion for their assessment of the impact of DAU's training.
DAU Performance Dashboard Training Quality Metrics:
* Meet or exceed the government/industry benchmark averages for
classroom and online course delivery and for student and manager
training effectiveness follow-up surveys.
While certification training is critical, it must be reinforced with
career-long learning assets to support continued proficiency and
capability of the workforce. The award winning AT&L Performance
Learning Model equips the workforce for success through a core training
foundation combined with online continuous learning, mission assistance
to acquisition organization and teams, and online knowledge sharing
resources and communities. The Performance Learning Model provides the
student with "reach back" to DAU and gives them access to professors,
acquisition resources, and other acquisition professionals. The five
metrics listed above address the proficiency and capability issues
raised in the report.
Recommendation 2: The GAO recommends that the Secretary of Defense
direct the USD(AT&L) to establish a timeframe for completion and ensure
resources are available for implementing an enterprise-wide, integrated
student information system. (See page 30/GAO Draft Report.)
DoD Response: Concur. DoD has already established a timeline for
implementation of an enterprise-wide student information system and
funding is in place for this effort.
[End of section]
Appendix IV: GAO Contact and Staff Acknowledgments:
GAO Contact:
John K. Needham, (202) 512-4841, needhamJK1@gao.gov:
Acknowledgments:
In addition to the individual named above, key contributors to the
report were Penny Berrier Augustine, Assistant Director; Johana Ayers;
Alezandra Brady; Helena Brink; John Krump; Morgan Delaney Ramaker; Erin
Schoening; Angela Thomas; Desiree Thorp; and Tom Twambly.
[End of section]
Footnotes:
[1] According to the Federal Procurement Data System-Next Generation,
DOD's total obligations in fiscal year 2009 were about $370 billion.
However, this figure reflects an approximately $13.9 billion downward
adjustment made by DOD to correct an administrative error made in
fiscal year 2008. As this adjustment significantly affected DOD's
reported obligations in fiscal year 2009, the $384 billion figure we
report reflects what DOD would have reported had the error not
occurred.
[2] According to 10 U.S.C.§ 1702, USD(AT&L) shall carry out all powers,
functions, and duties of the Secretary of Defense with respect to the
DOD AT&L workforce. DOD Instruction 5000.57--Defense Acquisition
University--lists establishing a program of education and training
standards, requirements, and performance learning assets for the
civilian and military AT&L workforce as one of the specific duties of
the USD(AT&L).
[3] DOD defines its acquisition workforce to include 15 career fields,
based on the Defense Acquisition Workforce Improvement Act (Pub. L. No.
101-510, § 1202(a) (1990)). These career fields are program management;
contracting; industrial/contract property management; purchasing;
facilities engineering; production, quality, and manufacturing;
business--cost estimating; business--financial management; lifecycle
logistics; information technology; systems planning, research,
development, and engineering--systems engineering; systems planning,
research, development and engineering--program systems engineer;
systems planning, research, development and engineering--science and
technology manager; test and evaluation; and auditing.
[4] For the purposes of this report, the personnel covered under the
Defense Acquisition Workforce Improvement Act (DAWIA) will be referred
to as the acquisition workforce. Under DAWIA, career fields, such as
program management, and certification of personnel in those career
fields, were established to provide the necessary skills. To be
certified, acquisition workforce members must meet mandatory standards
for the career field level (I, II, or III) required for their position.
For DAWIA certification in each career field, there are requirements in
three areas--education, experience, and training. We compared DOD's
certification training program for 14 of the 15 career fields with the
attributes of effective training and development programs identified in
GAO's guide (GAO, Human Capital: A Guide for Assessing Strategic
Training and Development Efforts in the Federal Government, GAO-04-546G
(Washington, D.C.: Mar. 2004)) and listed in appendix II. We were not
able to apply these criteria to the auditing career field because the
Defense Contract Audit Agency (DCAA) lacks a strategic plan (GAO, DCAA
Audits: Widespread Problems with Audit Quality Require Significant
Reform, [hyperlink, http://www.gao.gov/products/GAO-09-468](Washington, D.C.: Sept. 23, 2009)). A strategic plan is a key document for assessing training programs using the strategic training efforts attributes.
[5] Pub. L. No. 111-84, § 1108 (b) (2) (2009).
[6] The Commission on Army Acquisition and Program Management in
Expeditionary Operations--commonly referred to as the Gansler
Commission--assessed aspects of recent expeditionary operations,
including training, to ensure that the Army is properly equipped for
future operations. The commission's October 2007 report made four
overarching recommendations, including providing training and tools for
overall contracting activities in expeditionary operations.
[7] This report addresses the training component of acquisition
workforce certification, not the education and experience components.
Further, we limited our review of the auditing career field to DCAA
auditors because they are the only DOD auditors covered by DAWIA.
Additionally, we have ongoing work to review training for DOD personnel
with acquisition responsibilities in contingency and non-contingency
environments but who are not covered by DAWIA.
[8] GAO, Human Capital: A Guide for Assessing Strategic Training and
Development Efforts in the Federal Government, [hyperlink,
http://www.gao.gov/products/GAO-04-546G] (Washington, D.C.: Mar. 2004).
[9] The Acquisition Technology and Logistics Workforce Data Mart was
designed to be the single source of acquisition data for career and
workforce management for both civilian and military personnel.
Information contained within AT&L Data Mart comes from multiple,
external military and civilian data sources.
[10] Pub. L. No. 101-510, § 1202(a) (1990).
[11] It is the responsibility of the DOD functional leaders to
establish, oversee, and maintain the education, training, and
experience requirements including competencies and certification
standards, position category description(s), and content of the DAU
courses as current, technically accurate, and consistent with DOD
acquisition policies. Para. 5.5.3, DOD Instruction 5000.66, paragraph
5.5.3. (Dec. 2005).
[12] DOD Instruction 5000.57 requires DAU to establish a course
equivalency program that identifies alternatives for the workforce to
attain required training. With over 30 years of experience in reviewing
learning programs offered by business, government, and military
organizations for equivalent college credit, the American Council on
Education assesses all potential equivalency providers and their
courses, and, based on its recommendations, DAU approves providers--
including universities, DOD schools, and commercial vendors--to offer
courses approved as equivalent to students attempting to meet DAU
course requirements.
[13] GAO developed the guide through consultations with government
officials and experts in the private sector, academia, and nonprofit
organizations; examinations of laws and regulations related to training
and development in the federal government; and reviewing the sizeable
body of literature on training and development issues, including
previous GAO products on a range of human capital topics. GAO, Human
Capital: A Guide for Assessing Strategic Training and Development
Efforts in the Federal Government, [hyperlink,
http://www.gao.gov/products/GAO-04-546G] (Washington, D.C.: Mar. 2004).
[14] According to the guide, these practices should serve as a guide
for assessment and do not comprise a complete or mandatory "set" of
elements needed in response to each question; their relevance will vary
depending on each agency's specific circumstances.
[15] In some cases, functional leaders are responsible for more than
one career field. For example, the same functional leader is
responsible for contracting, purchasing, and industrial/contract
property management.
[16] The Kirkpatrick model is a balanced, multilevel approach to
evaluate an organization's training and development efforts. In this
model, the first level measures the training participants' reaction to,
and satisfaction with, the training program. The second level measures
the extent to which learning has occurred because of the training
effort. The third level measures the application of this learning to
the work environment through changes in behavior that trainees exhibit
on the job because of training. The fourth level measures the impact of
the training program on the agency's program or organizational results.
[17] GAO, Department of Defense: Additional Actions and Data Are Needed
to Effectively Manage and Oversee DOD's Acquisition Workforce,
[hyperlink,
http://www.gao.gov/products/GAO-09-342] (Washington, D.C.: Mar. 25,
2009).
[18] DOD's instruction for operation of the defense acquisition,
technology, and logistics workforce education, training, and career
development program requires that appropriate metrics be developed for
AT&L's senior leadership to have appropriate oversight and
accountability for management and career development of the acquisition
workforce. DOD Instruction 5000.66, paragraph E2.5.1. (Dec. 2005).
[19] In-sourcing refers to converting functions performed by
contractors to DOD personnel.
[20] The term graduate refers to a student who successfully completes a
DAU class.
[21] In fiscal year 2009, there were 2,764 level I and level II
auditors, making up approximately 73 percent of the total DAWIA
auditing workforce.
[22] The National Defense Authorization Act for fiscal year 2008
established the Defense Acquisition Workforce Development Fund to
provide additional funds for the recruitment, training, and retention
of acquisition personnel of the Department of Defense. Pub. L. No. 110-
181 § 852; codified in 10 U.S.C. § 1705.
[23] Boot camps provide an overview of acquisition and contracting
issues developed by commands.
[24] The report made five additional recommendations within the
overarching training recommendation. The fifth recommendation pertains
to training for Contracting Officer's Representatives, which are not
included in the DAWIA workforce and, therefore, the recommendation was
not included in this report. Training recommendations related to DOD
personnel in a contingency environment with acquisition
responsibilities but who are not covered by DAWIA will be reviewed in a
future GAO report.
[25] The Army also developed Operational Contract Support pre-
deployment training to implement this recommendation. However, it is
not required for operational commanders and Army officials could not
tell us how many opted to complete the 1-hour training. This training
will also be reviewed in a future GAO report.
[26] GAO, Defense Contracting: Army Case Study Delineates Concerns with
Use of Contractors as Contract Specialists, [hyperlink,
http://www.gao.gov/products/GAO-08-360] (Washington, D.C.: Mar. 26,
2008).
[27] Pub. L. No. 111-84, § 1108(b)(2) (2009).
[28] Pub. L. No. 101-510, § 1202(a) (1990). For the purposes of this
report, the personnel covered under DAWIA will be referred to as the
acquisition workforce. GAO will review training for DOD personnel in
contingency and non-contingency environments who have acquisition
responsibilities but are not covered by DAWIA in two forthcoming
reports.
[29] GAO, Human Capital: A Guide for Assessing Strategic Training and
Development Efforts in the Federal Government, [hyperlink,
http://www.gao.gov/products/GAO-04-546G] (Washington, D.C.: Mar. 2004).
We compared DOD's certification training program for 14 of the 15
career fields with the attributes of effective training and development
programs identified in GAO's guide for assessing strategic training.
However, we were not able to apply these criteria for the fifteenth
career field--auditing--because DCAA does not have a strategic plan in
place (GAO, DCAA Audits: Widespread Problems with Audit Quality Require
Significant Reform, [hyperlink,
http://www.gao.gov/products/GAO-09-468] (Washington, D.C.: Sept.
2009)). Because a strategic plan is a key document for assessing
training programs using the strategic training efforts criteria, we did
not assess the attributes of training for the auditing career field.
[30] AMCOM is a major subordinate command of the Army Materiel Command.
[31] GAO, Defense Acquisitions: Assessments of Selected Weapon
Programs, [hyperlink,
http://www.gao.gov/products/GAO-10-388SP] (Washington, D.C.: Mar. 30,
2010).
[32] GAO, DCAA Audits: Widespread Problems with Audit Quality Require
Significant Reform, [hyperlink,
http://www.gao.gov/products/GAO-09-468] (Washington, D.C.: Sept. 23,
2009).
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
