Military Personnel
Evaluation Methods Linked to Anticipated Outcomes Needed to Inform Decisions on Army Recruitment Incentives Gao ID: GAO-08-1037R September 18, 2008Since the September 11, 2001 attacks on the World Trade Center and the Pentagon, the United States has launched several military operations that have dramatically increased the operations tempo of the military services and required the large-scale mobilization of reservists. These factors have particularly affected the active Army, Army Reserve, and Army National Guard, which have shouldered the bulk of the personnel burden associated with ongoing operations in Iraq. A 2007 Congressional Research Service report notes that many observers have expressed concern that these factors might lead to lower recruiting and retention rates, thereby jeopardizing the vitality of today's all-volunteer military. Additionally, in 2004 the Army began its modular force transformation to restructure itself from a division-based force to a more agile and responsive modular brigade-based force--an undertaking it considers to be the most extensive reorganization of its force since World War II. Both ongoing military operations and transformation have prompted the Army to increase its recruitment efforts. To encourage military service, Congress, through Section 681 of the National Defense Authorization Act for Fiscal Year 2006, temporarily authorized the Army to provide not more than four new recruitment incentives and directed the Secretary of the Army to submit to Congress a plan for each recruitment incentive it develops under the authority provided. Section 681 states that each plan should include (1) a description of the incentive, including its purpose and the potential recruits to be addressed by the incentive; (2) a description of the provisions of the U.S. Code relevant to the military that would need to be waived in order for the Army to provide the incentive and an explanation of why these provisions would need to be waived; (3) a statement of the anticipated outcomes as a result of providing the incentive; and (4) the method to be used to evaluate the effectiveness of the incentive. The Army is also required to submit an annual report to Congress on each of the recruitment incentives developed under this authority; this report is to include a description of the incentives and an assessment of their impact on recruitment during the previous fiscal year. The Army began providing recruitment incentives under this authority in June 2006 and is currently using it to pilot three recruitment incentives. Under Section 681, the Army's authority to provide these new recruitment incentives expires on December 31, 2009. The Army may modify, expand, or take steps to make permanent some or all of these recruitment incentives, based on the data it collects during this pilot phase. For this report, we (1) identified and described the recruitment incentives the Army has developed under Section 681 of the National Defense Authorization Act for Fiscal Year 2006 and (2) assessed the extent to which the plans for each incentive included anticipated outcomes and a methodology for evaluating these outcomes.
The Secretary of the Army has notified Congress--as the law requires--of the three recruitment incentives that the Army has developed under Section 681 of the National Defense Authorization Act for Fiscal Year 2006. As further required by Section 681, the Army has developed a plan for each of these incentives. The Recruiter Incentive Pay Pilot Program, which the Army began offering in June 2006, provides monthly, quarterly, and annual bonus pay to Army, Army Reserve, and Army National Guard recruiters-- of both officers and enlisted soldiers--who exceed their recruitment goals. The Army Advantage Fund Pilot Program, which the Army began offering in February 2008, provides payment to enlisted active Army and Army Reserve recruits with no prior service, following their completion of service, to be used for home ownership or small business development. The Officer Accession Pilot Program, which the Army began offering in August 2008, offers three options--two that target Reserve Officer Training Corps (ROTC) cadets and one that targets certain qualified medical professionals and chaplains between the ages of 43 and 60. One of the ROTC options provides a monetary bonus to cadets who attend the Leader Training Course, a 4-week program that ROTC students take during the summer between their sophomore and junior years of college. The second ROTC option provides a monthly stipend to cadets who participate satisfactorily in approved language immersion programs, programs of study abroad, or academic courses involving instruction in a foreign language of strategic importance to DOD. Both of these options require cadets to sign a contract to serve as commissioned officers after they graduate. The third option reduces the mandatory length of the military service obligation from 8 years to 2 years for certain qualified medical professionals and chaplains. Army officials with whom we spoke stated that the Army does not currently expect to develop a fourth recruitment incentive. However, the Army noted in technical comments on our draft report that it reserves the option to develop one more recruitment incentive pursuant to Section 681. Two of the Army's three recruitment incentive plans do not include anticipated outcomes and none describes a methodology to be used to evaluate its effectiveness. Section 681 requires that each plan the Secretary of the Army submits to Congress include, among other things, (1) a statement of the outcomes the Army anticipates will result from the incentive offered, and (2) a description of the method to be used to evaluate the incentive's effectiveness. Although the Army did not include this information in the plans it sent to Congress for the first two incentives, the Army did include information about anticipated outcomes in accompanying notification letters. In addition, the Army assigned responsibility for developing evaluation methodologies and assessing the Recruiter Incentive Pay Pilot Program to the U.S. Army's Accession Command's Center for Accession Research and for the Army Advantage Fund Pilot Program to RAND. For the Officer Accession Pilot Program, the Army did include anticipated outcomes in the plan, and it assigned the evaluation responsibilities to the U.S. Army Accessions Command, the Army Surgeon General, and the Army Chief of Chaplains. However, the information on anticipated outcomes is limited for all three incentives in that it does not include the assumptions behind the numbers, and, therefore, the link between the desired outcomes and the methodology to be used to assess progress is not clear.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-08-1037R, Military Personnel: Evaluation Methods Linked to Anticipated Outcomes Needed to Inform Decisions on Army Retirement Incentives
This is the accessible text file for GAO report number GAO-08-1037R
entitled 'Military Personnel: Evaluation Methods Linked to Anticipated
Outcomes Needed to Inform Decisions on Army Recruitment Incentives'
which was released on September 18, 2008.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
GAO-08-1037R:
United States Government Accountability Office:
Washington, DC 20548:
September 18, 2008:
Congressional Committees:
Subject: Military Personnel: Evaluation Methods Linked to Anticipated
Outcomes Needed to Inform Decisions on Army Recruitment Incentives:
Since the September 11, 2001 attacks on the World Trade Center and the
Pentagon, the United States has launched several military operations
that have dramatically increased the operations tempo of the military
services and required the large-scale mobilization of reservists. These
factors have particularly affected the active Army, Army Reserve, and
Army National Guard, which have shouldered the bulk of the personnel
burden associated with ongoing operations in Iraq. A 2007 Congressional
Research Service report notes that many observers have expressed
concern that these factors might lead to lower recruiting and retention
rates, thereby jeopardizing the vitality of today's all-volunteer
military.[Footnote 1] Additionally, in 2004 the Army began its modular
force transformation to restructure itself from a division-based force
to a more agile and responsive modular brigade-based force--an
undertaking it considers to be the most extensive reorganization of its
force since World War II. Both ongoing military operations and
transformation have prompted the Army to increase its recruitment
efforts.
To encourage military service, Congress, through Section 681 of the
National Defense Authorization Act for Fiscal Year 2006,[Footnote 2]
temporarily authorized the Army to provide not more than four new
recruitment incentives and directed the Secretary of the Army to submit
to Congress a plan for each recruitment incentive it develops under the
authority provided. Section 681 states that each plan should include
(1) a description of the incentive, including its purpose and the
potential recruits to be addressed by the incentive; (2) a description
of the provisions of the U.S. Code relevant to the military[Footnote 3]
that would need to be waived in order for the Army to provide the
incentive and an explanation of why these provisions would need to be
waived; (3) a statement of the anticipated outcomes as a result of
providing the incentive; and (4) the method to be used to evaluate the
effectiveness of the incentive. The Army is also required to submit an
annual report to Congress on each of the recruitment incentives
developed under this authority; this report is to include a description
of the incentives and an assessment of their impact on recruitment
during the previous fiscal year. The Army began providing recruitment
incentives under this authority in June 2006 and is currently using it
to pilot three recruitment incentives. Under Section 681, the Army's
authority to provide these new recruitment incentives expires on
December 31, 2009. The Army may modify, expand, or take steps to make
permanent some or all of these recruitment incentives, based on the
data it collects during this pilot phase.
Section 681 also directs the Secretary of the Army to submit each
recruitment incentive plan to the Comptroller General and requires GAO
to report to Congress on the expected outcomes of each recruitment
incentive in terms of cost effectiveness and mission achievement as
soon as practicable after receipt of each plan. For this report, we (1)
identified and described the recruitment incentives the Army has
developed under Section 681 of the National Defense Authorization Act
for Fiscal Year 2006 and (2) assessed the extent to which the plans for
each incentive included anticipated outcomes and a methodology for
evaluating these outcomes.
To address our first objective, we obtained information about the
recruitment incentives that the Army has developed under Section 681
through the two guiding documents for each incentive--the notification
letter and the accompanying plan provided to Congress.[Footnote 4] For
our second objective, we analyzed the extent to which the plans that
the Army developed for each incentive included anticipated outcomes and
a methodology for evaluating these outcomes. For the one incentive
developed pursuant to Section 681 that the Army has offered for more
than 1 year, we reviewed the letter to Congress which comprises the
Army's required annual report; further, we reviewed the two assessments
that provided input to this letter. We also obtained information from
interviews and correspondence with officials from the Army's Office of
the Deputy Chief of Staff for Personnel; the Army National Guard; Army
General Counsel; U.S. Army Accessions Command; U.S. Army Recruiting
Command; the Office of the Assistant Secretary of the Army for Manpower
and Reserve Affairs; and the Office of the Undersecretary of Defense
for Personnel and Readiness.
We conducted this performance audit from April 2008 through September
2008 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our audit objectives.
Results in Brief:
The Secretary of the Army has notified Congress--as the law requires--
of the three recruitment incentives that the Army has developed under
Section 681 of the National Defense Authorization Act for Fiscal Year
2006. As further required by Section 681, the Army has developed a plan
for each of these incentives.
* The Recruiter Incentive Pay Pilot Program, which the Army began
offering in June 2006, provides monthly, quarterly, and annual bonus
pay to Army, Army Reserve, and Army National Guard recruiters--of both
officers and enlisted soldiers--who exceed their recruitment goals.
* The Army Advantage Fund Pilot Program, which the Army began offering
in February 2008, provides payment to enlisted active Army and Army
Reserve recruits with no prior service, following their completion of
service, to be used for home ownership or small business development.
* The Officer Accession Pilot Program, which the Army began offering in
August 2008, offers three options--two that target Reserve Officer
Training Corps (ROTC) cadets and one that targets certain qualified
medical professionals and chaplains between the ages of 43 and 60. One
of the ROTC options provides a monetary bonus to cadets who attend the
Leader Training Course, a 4-week program that ROTC students take during
the summer between their sophomore and junior years of college. The
second ROTC option provides a monthly stipend to cadets who participate
satisfactorily in approved language immersion programs, programs of
study abroad, or academic courses involving instruction in a foreign
language of strategic importance to DOD. Both of these options require
cadets to sign a contract to serve as commissioned officers after they
graduate. The third option reduces the mandatory length of the military
service obligation from 8 years to 2 years for certain qualified
medical professionals and chaplains.
Army officials with whom we spoke stated that the Army does not
currently expect to develop a fourth recruitment incentive. However,
the Army noted in technical comments on our draft report that it
reserves the option to develop one more recruitment incentive pursuant
to Section 681.
Two of the Army's three recruitment incentive plans do not include
anticipated outcomes and none describes a methodology to be used to
evaluate its effectiveness. Section 681 requires that each plan the
Secretary of the Army submits to Congress include, among other things,
(1) a statement of the outcomes the Army anticipates will result from
the incentive offered, and (2) a description of the method to be used
to evaluate the incentive's effectiveness. Although the Army did not
include this information in the plans it sent to Congress for the first
two incentives, the Army did include information about anticipated
outcomes in accompanying notification letters. In addition, the Army
assigned responsibility for developing evaluation methodologies and
assessing the Recruiter Incentive Pay Pilot Program to the U.S. Army's
Accession Command's Center for Accession Research and for the Army
Advantage Fund Pilot Program to RAND. For the Officer Accession Pilot
Program, the Army did include anticipated outcomes in the plan, and it
assigned the evaluation responsibilities to the U.S. Army Accessions
Command, the Army Surgeon General, and the Army Chief of Chaplains.
However, the information on anticipated outcomes is limited for all
three incentives in that it does not include the assumptions behind the
numbers, and, therefore, the link between the desired outcomes and the
methodology to be used to assess progress is not clear. For example,
while the U.S. Army Accessions Command's Center for Accessions Research
recommended that the Army stop offering the Recruiter Incentive Pay
Pilot Program based on its assessment of the incentive on the active
Army and Army Reserve, the National Guard Bureau recommended that the
Army continue to offer it based on the Bureau's assessment of the
incentive's effect on the Army National Guard. Because information on
this incentive's anticipated outcomes is limited, Army officials
continue to lack the information they need to make informed, objective
decisions regarding the extent to which--and the reasons why--the
program did or did not meet its intended goals. In this case, for the
Recruiter Incentive Pay Pilot Program, the Secretary of the Army
determined that further study was required and modified the incentive.
The results of an initial status report on the Army Advantage Fund
Pilot Program were similarly inconclusive because the anticipated
outcomes that the Army included in the notification letter to Congress
were not applicable to the incentive as implemented. After this initial
status report, the Army determined it would need more time before it
could provide information about the effectiveness of this incentive.
Because Army officials responsible for managing the recruitment
incentives did not incorporate all of the necessary elements into their
recruitment incentive plans, the plans do not present a clear and
convincing business case for the incentives, and it is difficult for
those exercising oversight to make an independent assessment of the
extent to which the incentives have been a success.
To improve management of the incentives and ensure that the required
annual assessments will be of use to Army decision makers, we are
recommending that--for each recruitment incentive developed pursuant to
Section 681--the Secretary of the Army issue guidance to clearly
specify anticipated outcomes, describe the assumptions behind these
anticipated outcomes, identify the evaluation method to be used to
assess progress toward these outcomes, and link anticipated outcomes to
the stated evaluation method. We are also recommending that the
Secretary of the Army ensure that this information is specifically
addressed in the required annual assessments for each recruitment
incentive. Additionally, should the Army decide--based on the results
of the annual assessments of each incentive--to seek to make any of
these incentives permanent, we further recommend that the Secretary of
the Army, for any proposals the Army may wish to develop, include in
its business case clearly specified anticipated outcomes for each
incentive, a description of the assumptions behind these anticipated
outcomes, an identification of the evaluation method to be used to
assess progress toward these outcomes, and an explanation that links
anticipated outcomes to the stated evaluation method. In commenting on
a draft of our report, DOD concurred with each of our recommendations.
The department's comments are reprinted in enclosure I.
Background:
To meet its human capital needs, the Army must annually recruit and
retain more than twice the number of uniformed personnel needed by any
other military service. In fiscal year 2008, the Army's recruiting
mission was over 167,000. Like the other services, the Army employs
three primary types of tools: (1) military recruiters, (2) advertising,
and (3) incentives such as enlistment bonuses, educational benefits,
and military pay. The Army has about 14,050 recruiters, most located in
the United States, who recruit from the local population. To address
its recruiting needs, the Army may raise or lower the size of the
recruiter force and may use other methods to enhance recruiter
effectiveness and efficiency. In terms of advertising, the Army manages
its own program and works closely with its own contracted advertising
agency. In fiscal year 2007, the Army spent $476 million for
advertising to support recruiting efforts. Finally, the Army offers
enlistment bonuses to attract high-quality recruits; it also uses
educational benefits such as the Montgomery GI Bill, the Army College
Fund, tuition assistance, and college loan repayment. In fiscal year
2007, the Army spent approximately $473 million on enlistment bonuses
for active Army recruits and for Army Reserve and Army National Guard
recruits who were mobilized to support ongoing operations.
Generally, before the Army can offer a new recruitment incentive, DOD
policy requires that it develop and submit a legislative proposal
through DOD's Unified Legislation and Budgeting Process. DOD's Office
of the Undersecretary of Defense for Personnel and Readiness
established this process in 1993 to ensure the quality of personnel-
related proposals developed by the services and to align these
proposals with DOD's budgeting process. According to DOD policy,
[Footnote 5] this process provides a joint forum to develop, review,
and coordinate legislative initiatives that affect personnel and
readiness.[Footnote 6] As part of this process, the services are
expected to develop a description of the personnel initiative,
including a business case and an estimate of the approximate cost per
year. Only after legislation is passed by Congress and signed into law
by the president can the military services offer the incentive or other
personnel initiative. According to DOD officials, because Section 681
directly authorized the Secretary of the Army to develop new
recruitment incentives, the Army's new incentives were not reviewed
through DOD's Unified Legislation and Budgeting process. On prior
occasions, the Office of the Under Secretary of Defense for Personnel
and Readiness has proposed to Congress that it be granted similar pilot
program authority, but Congress has not, to date, included these
proposals in legislation. According to Personnel and Readiness
officials, this type of authority would enhance the Office of the
Secretary of Defense's ability to assess the effectiveness of personnel
programs and hence allow for better informed investment decisions
throughout the services.
The Army Developed Three New Recruitment Incentive Programs under
Section 681:
The Secretary of the Army has notified Congress of the three new
recruitment incentives it has developed pursuant to Section 681. These
are (1) the Recruiter Incentive Pay Pilot Program,
(2) the Army Advantage Fund Pilot Program, and (3) the Officer
Accession Pilot Program. As required by Section 681, the Army has
developed a plan for each of these incentives. Each plan cites the
authority for the incentive and describes the incentive offered, the
purpose of the incentive, and the types of individuals eligible for the
incentive. Table 1 highlights key elements of the three incentives. The
Army has the authority to develop no more than four recruitment
incentives under Section 681 and has already developed three.[Footnote
7] While Army officials with whom we spoke stated that the Army does
not currently expect to develop any additional recruitment incentives
under Section 681, the Army noted in technical comments on our draft
report that it reserves the option to develop one more recruitment
incentive pursuant to Section 681.
Table 1: Description of Recruitment Incentives.
Program: Recruiter Incentive; Pay Pilot Program;
Description, availability, and applicability:
* Provides bonus pay to active Army, Army Reserve, and Army National
Guard recruiters nationwide;
* Assumes additional pay will motivate recruiters to exceed recruitment
goals;
* Provides bonuses that range from $100 to $8,600 per year, depending
on the extent to which recruiters exceed set benchmarks.
Program: Army Advantage Fund Pilot Program;
Description, availability, and applicability:
* Provides recruits payment, following their completion of service,
intended for home ownership or small business development;
* Provides up to $40,000 for active Army recruits and up to $20,000 for
Army Reserve recruits who enlist at selected recruiting battalions.
Program: Officer Accession Pilot Program;
Description, availability, and applicability:
* Option A provides a $5,000 bonus to students who complete the ROTC
Leader Training Course and sign a contract to serve in the Army as
commissioned officers after they graduate;
* Option B provides between $100 and $250 per month, up to a limit of
$3,000 per academic year, to senior ROTC students who are enrolled in
an approved language immersion program, program of study abroad, or
academic course involving instruction in a foreign language of
strategic importance to DOD and who sign a contract to serve in the
Army as commissioned officers after they graduate;
* Option C waives the required 8-year military service obligation for
qualified medical and religious professionals between the ages of 43
and 60 and authorizes a 2-year military service obligation.
Source: GAO analysis of Army data.
[End of table]
The Army began offering bonuses under the Recruiter Incentive Pay Pilot
Program in June 2006. This incentive provides bonus pay to active Army,
Army Reserve, and Army National Guard recruiters--of both officers and
enlisted soldiers--nationwide who exceed their recruitment goals. The
incentive's purpose is to increase the number and quality of new Army
recruits. In general, DOD and the Army measure enlisted recruit quality
based on two criteria: graduation from high school and Armed Forces
Qualification Test score. Of the new recruits enlisting in the active
Army, Navy, Air Force, and Marine Corps each year, DOD's goal is that
at least 90 percent meet the graduation criteria and at least 60
percent score above average on the Armed Forces Qualification Test. To
help ensure quality, the plan for the Recruiter Incentive Pay Pilot
Program requires that at least 50 percent of the new recruits signed by
a recruiter meet the Army's quality benchmarks. Originally, the Army
designed the incentive to provide bonuses to recruiters based solely on
individual performance. Subsequently, in May 2008, the Army modified
the incentive in response to a Secretary of the Army decision related
to the Army's assessment of the program's effect during its first year
of implementation. The modified incentive continues to provide bonuses
to most recruiters based solely on individual performance, although for
recruiters in one of the Army's six recruiting brigades, the recruiting
station as a whole must meet its goal in order for individual
recruiters to receive bonus pay for exceeding their recruitment goals.
The Army began offering enlistment bonuses under the Army Advantage
Fund Pilot Program in February 2008. Like the Recruiter Incentive Pay
Pilot Program, the Army Advantage Fund Pilot Program provides a
monetary incentive; however, rather than providing a monetary incentive
to recruiters, the Army Advantage Fund Pilot Program provides incentive
pay to high-quality recruits who enlist for specified time frames.
Specifically, it provides payment to enlisted active Army and Army
Reserve recruits with no prior service, after they have completed their
period of service, to be used for home ownership or small business
development.[Footnote 8] The incentive also has an advertising
component. The Army is currently offering the incentive through 5 of 45
Army recruiting battalions nationwide, so that it can assess the
incentive's effect before considering further expansion. The Army is
initially offering the Army Advantage Fund at the following locations:
Albany, New York; Cleveland, Ohio; Montgomery, Alabama; San Antonio,
Texas; and Seattle, Washington. The Army selected Albany, Cleveland,
and Seattle because the demographics of these recruiting battalions
reflect nationwide demographics, better allowing it to project the
results of the incentive nationwide. According to Army officials, the
Army selected Montgomery and San Antonio in order to obtain information
about the incentive's effect in areas with larger populations of
African Americans and Hispanics. The incentive's purpose is to give the
Army a competitive advantage over the other services in attracting
individuals to military service.
The Army began offering incentives under the Officer Accession Pilot
Program in August 2008, targeting individuals qualified to accept
officer commissions. As we have previously reported, the Army faces
unique officer accession challenges in the future.[Footnote 9] These
challenges are the result of the Army's extensive role in ongoing
military operations, as well as the expansion of the officer corps as
part of the congressionally authorized 30,000-soldier increase to the
Army's end strength and the Army's need for more officers as part of
its ongoing transformation effort.[Footnote 10] The Officer Accession
Pilot Program provides both monetary and nonmonetary incentives through
three options. Option A pays a bonus to ROTC cadets who complete the
Leader Training Course and sign a contract to serve in the Army as
commissioned officers after they graduate. The Leader Training Course
is a 4-week, field-oriented program that ROTC students take during the
summer between their sophomore and junior years of college. Option B
provides a stipend to senior ROTC students who participate
satisfactorily in approved language immersion programs, programs of
study abroad, or an academic course involving instruction in a foreign
language of strategic importance to DOD and who sign a contract to
serve in the Army as commissioned officers after they graduate. The
list of approved languages will be reviewed annually. Currently,
approved languages are Arabic, Persian-Farsi, Persian-Dari, Chinese-
Mandarin, Pashto, Urdu, Indonesian, Swahili, Hausa, and Korean. Option
C reduces the military service obligation for certain qualified medical
and religious professionals between the ages of 43 and 60 from 8 years
to 2 years. The 8-year military service obligation is regarded as a
barrier to recruiting older, experienced professionals for initial
appointment and service in a critical career field for which they are
otherwise well qualified. An individual may qualify under only one of
these options.
Section 681 specifies that the Army may not continue to provide any of
these incentives after December 31, 2009. According to officials from
the Office of the Under Secretary of Defense for Personnel and
Readiness, to make any of the incentives piloted under section 681
permanent, the Army would need to submit a legislative proposal to
Congress, as it does for other recruitment incentives. The Army would
need to use DOD's Unified Legislation and Budgeting Process to do so.
It generally takes about 2 years for a proposal for a personnel
initiative to go through this process and only after legislation is
passed by Congress and signed into law by the president can the
military services offer the incentive or other personnel initiative. As
a result, the potential may exist for a gap between the December 31,
2009 expiration of the authority provided in Section 681 and the date
on which the Army can again offer any incentives initially developed
under Section 681 that it seeks to make permanent.
Two Plans for Recruitment Incentives Lack Outcomes and None Includes
Evaluation Methodologies:
Two of the Army's three recruitment incentive plans do not include
anticipated outcomes and none describes a methodology to be used to
evaluate its effectiveness. Section 681 requires the Army to include in
the plan it submits to Congress before implementing each incentive (1)
a statement of the outcomes the Army anticipates will result from the
incentive offered and (2) a description of the method to be used to
evaluate the incentive's effectiveness. In addition, Section 681
requires the Army to develop an annual report to Congress on each
incentive, to include both a description of the recruitment incentives
provided under Section 681 during the previous fiscal year and an
assessment of each incentive's impact on the recruitment of individuals
as officers or enlisted members of the Army. Although the Army did not
include this information in the plans it developed for the first two
incentives, the Army did include information about anticipated outcomes
in its notification letters to Congress. In addition, the Army assigned
responsibility for developing evaluation methodologies and assessing
the Recruiter Incentive Pay Pilot Program to the U.S. Army's Accessions
Command's Center for Accession Research and for the Army Advantage Fund
Pilot Program to RAND. For the Officer Accession Pilot Program, the
Army did include anticipated outcomes in the plan, and it assigned the
assessment responsibilities to the U.S. Army Accessions Command, the
Army Surgeon General, and the Army Chief of Chaplains. All of these
organizations have conducted, are currently conducting, or plan to
conduct assessments of each of the recruitment incentives. The
Secretary of the Army and Army officials responsible for the incentives
intend to use these assessments, along with other data collected as
part of the assessment process, to inform their decisions about each
incentive's future--for example, whether to expand the incentive
nationwide and whether to develop a legislative proposal to make the
incentive permanent.
Prior GAO work shows that one of the benefits of piloting programs is
that they can enable organizations to make choices based on rigorous
evaluation,[Footnote 11] and Army Regulation 5-1 states that data-
driven management decision making is one of the core values of the
Army's performance improvement criteria.[Footnote 12] But if officials
do not clearly establish at the outset the metrics on which they intend
to base their decisions about the pilot programs, they may not be in a
position to ensure that the data collected and the analysis produced
will enable them to make informed, objective decisions regarding the
extent to which such programs are meeting their goals or to judge their
relative effectiveness and cost, both compared to each other and to
existing programs. Moreover, since the methodologies to evaluate the
effectiveness of recruitment incentives developed pursuant to Section
681 are--like all evaluation designs--constrained by a number of
factors, it is important to recognize these factors candidly and
describe their effect. For example, these methodologies should disclose
the extent to which factors other than the Section 681 incentives
themselves are controlled for, and the extent to which these factors
could independently influence any changes in recruitment numbers.
Because the Army did not incorporate all of the necessary elements into
its recruitment incentive plans, the plans do not present a clear and
convincing business case for the incentives, making it difficult for
those exercising oversight to independently assess the extent to which
the incentives have been a success.
Recruiter Incentive Pay Pilot Program:
The March 2006 plan for the Recruiter Incentive Pay Pilot Program does
not include a statement of anticipated outcomes or a methodology to be
used to evaluate its effectiveness. However, in an April 2006
notification letter to Congress, the Secretary of the Army estimated
that the incentive would result in an additional 6,500 active Army
accessions, an additional 2,000 Army Reserve accessions, and an
additional 5,100 Army National Guard accessions per year.[Footnote 13]
But the letter does not include information on the process--or
assumptions--that the Army used to develop these anticipated outcomes.
Because the Army's plan does not include the outcomes, it is also
unclear whether the Army intended to assess the incentive's
effectiveness against the estimated accessions stated in the letter.
Because it is the only one of the three incentives that has been
offered for more than 1 year, the Recruiter Incentive Pay Pilot Program
is the only one for which the Army has completed its required annual
assessment, the results of which it has provided to Congress. Based on
the results of this assessment, the Secretary of the Army concluded
that further study of the incentive's effect was required. However,
neither the plan for the incentive nor the April 2006 notification
letter to Congress identified the assumptions that were used to develop
the program's anticipated outcomes. Similarly, these documents did not
identify key factors--such as the ways in which the Army National
Guard's recruiting environment differs from that of the active Army and
the Army Reserve. As a result, Army officials lack the information they
need to make informed, objective decisions regarding the extent to
which--and the reasons why--the program did or did not meet its
intended goals. Specifically, U.S. Army Accessions Command's Center for
Accessions Research assessed the incentive's effect on the active Army
and Army Reserve and concluded that the Recruiter Incentive Pay Pilot
Program (1) did not increase either the quality or the number of new
recruits, (2) encouraged recruiters to focus on nonquality recruits,
and (3) did not appear to be cost-effective, at a cost of approximately
$3.7 million for the first year. Accordingly, the U.S. Army Accessions
Command's Center for Accessions Research recommended that the Army stop
offering the incentive as of December 2007. In contrast, based on the
results of its assessment of the incentive's effect on the Army
National Guard, the National Guard Bureau recommended that the Army
continue to offer the incentive. A March 2008 letter, which comprises
the Army's annual report to Congress on the incentive, combines the
results of the two. In the letter, the Army compared the number of
individuals signing contracts in the year after it began to offer
bonuses under the Recruiter Incentive Pay Pilot Program to the number
of individuals who had joined during the year before. This comparison
revealed that during the first year of the incentive, the active Army
had 2,899 fewer accessions than the previous year and the Army Reserve
1,362 fewer, while the Army National Guard had 6,711 more. To assist in
further assessing the incentive, the Army modified the incentive--as
discussed earlier--to include a team concept in selected areas.
However, because this modification does not apply to the Army National
Guard, it will not provide Army officials additional insight into what
factors may have contributed to positive assessment of the program for
the Army National Guard and the negative assessment for the active Army
and the Army Reserve.
Army Advantage Fund Pilot Program:
The plan developed for the Army Advantage Fund Pilot Program does not
include a statement of anticipated outcomes or a methodology to be used
to evaluate its effectiveness. In its notification letter to Congress,
the Army estimated the incentive would result in an additional 1,500
active Army accessions, an additional 900 Army Reserve accessions, and
an additional 350 Army National Guard accessions during fiscal year
2008.[Footnote 14] Because the letter does not discuss how the Army
developed these anticipated outcomes, the extent to which they are
appropriate to measuring the incentive's effectiveness is unclear. For
example, an official from the organization conducting the assessment
for the Army Advantage Fund Pilot Program said that these numbers
likely assume that the incentive will be implemented nationwide, but to
date the incentive has only been implemented in five locations for
purposes of the pilot, as discussed earlier. Moreover, because the
assumptions underlying this estimate are not made clear in either of
the two guiding documents for the incentive--the notification letter
and the plan--the extent to which the assumptions that were used to
develop these estimated outcomes are also being used in ongoing
assessments of the incentive is unclear. In the plan for the Army
Advantage Fund Pilot Program, the Army assigned RAND the responsibility
for assessing the incentive's effectiveness, and the Army obtained
information on RAND's 3-month assessment in June 2008. RAND is
continuing to collect information about the program. The plan states
that Army leaders will determine if the Army Advantage Fund Pilot
Program should be modified, depending on the success of the incentive,
and officials responsible for managing the incentive indicated that
they intend to use assessment data to make decisions about the
incentive's future--including whether and when they will offer the
incentive nationwide.
Officer Accession Pilot Program:
The plan developed for the Officer Accession Pilot Program does not
include a methodology to be used to evaluate its effectiveness. It does
state anticipated outcomes for each of the incentive's three options in
terms of numbers of officer accessions and assigns responsibility to
other organizations for assessing each option. The Army anticipates
that Option A, which provides a bonus to ROTC cadets who attend the
Leader Training Course--a requirement for the ROTC Advanced Course--
will result in the contracting of between 250 and 290 additional cadets
and other ROTC Advanced Course participants per year. The Army expects
Option B, which provides a monthly stipend to students enrolled in the
study of languages of strategic importance to DOD--and determined by
the Secretary of the Army to also be of importance to the Army--to
result in the contracting and commitment to military service of between
200 and 300 additional students per year. Option B is expected to
motivate students with an interest in or pre-existing knowledge of
strategic foreign languages to participate in Senior ROTC and commit to
a period of military service. Finally, the Army anticipates that Option
C, which reduces the military service obligation for qualified midlife
medical and religious professionals, will increase the number of
appointments by an additional 20 to 30 officers per year. For none of
the three options does the plan include an explanation about how the
anticipated outcomes were developed, although conversations with Army
officials indicate that the estimates are based on relevant data, such
as the number of additional students that the Army's Leader Training
Course, as currently structured, could accommodate. Nonetheless, based
on the plan alone, it is unclear what assumptions the Army used to
develop the outcomes, and what data the Army intends to collect and
analyze in order to assess each option's effectiveness. The incentive's
plan assigns responsibility to U.S. Army Accessions Command for
assessing Options A and B, and the Army Chief of Chaplains and the
Office of the Army Surgeon General responsibility for assessing the
effectiveness of Option C. The incentive's plan states that the Army's
Deputy Chief of Staff for Personnel, in determining the overall effect
of the Officer Accession Pilot Program, will analyze reports on the
effect of each option of the incentive and that the Secretary of the
Army or his designee will consider these data and other information in
deciding whether or not to continue the incentive. The Army began
offering the program in August 2008 and so has not yet conducted any
assessments.
Conclusions:
The Army's role in--among other things--simultaneously supporting
multiple military operations means that it must recruit substantially
greater numbers of individuals than the other military services. To
address this challenge, Section 681 of the National Defense
Authorization Act for Fiscal Year 2006 authorizes the Army to develop
not more than four new recruitment incentives. The Army has established
three such incentives. However, because the Army began offering these
incentives before it presented a sound business case that clearly
defined desired outcomes with transparent assumptions and established a
methodology for evaluating performance linked to those desired
outcomes, its basis for proposing the three incentives remains somewhat
unclear. Additionally, because the plans do not provide the necessary
information, by relying on the plans alone it is not possible to assess
either each recruitment incentive's cost effectiveness or the extent to
which it has achieved its intended mission. Unless changes are made,
such challenges will continue to hamper the Army's ability to assess
the effectiveness of any recruitment incentive developed under Section
681 that it decides to seek to make permanent. Furthermore, because
Army officials have yet to develop an approach to systematically
evaluate the data that the Army collects on the incentives, there is
currently no assurance that the Army's decisions about any incentives
they decide to seek to make permanent will be informed by a sound
business case.
Recommendations for Executive Action:
To improve management of the incentives and ensure that the required
annual assessments will be of use to Army decision makers, we recommend
that--for each recruitment incentive developed pursuant to Section 681-
-the Secretary of the Army take the following two actions:
Issue guidance to clearly specify anticipated outcomes for each
incentive, describe the assumptions behind these anticipated outcomes,
identify the evaluation method to be used to assess progress toward
these outcomes, and link anticipated outcomes to the stated evaluation
method.
Ensure that this guidance is specifically addressed in the required
annual assessments for each recruitment incentive.
Additionally, should the Army decide--based on the results of the
annual assessments of each incentive--to seek to make any of these
incentives permanent, we further recommend that the Secretary of the
Army, for any proposals the Army may wish to develop, include in its
business case clearly specified anticipated outcomes for each
incentive, a description of the assumptions behind these anticipated
outcomes, an identification of the evaluation method to be used to
assess progress toward these outcomes, and an explanation that links
anticipated outcomes to the stated evaluation method.
Agency Comments and Our Evaluation:
In commenting on a draft of our report, DOD concurred with our three
recommendations and specifically stated that it would take actions
intended to improve management of the incentives and help ensure that
the required annual assessments will be of use to Army decision makers.
We have reprinted DOD's official comments as enclosure I. DOD also
provided technical comments, which we have incorporated in the report
as appropriate.
We are sending copies of this report to the Secretary of Defense and
the Secretaries of the Army, Navy, and Air Force. We will also make
copies available to others on request. In addition, the report will be
available at no charge on the GAO Web site at [hyperlink,
http://www.gao.gov].
Should you or your staff have any questions on the matters discussed in
this report, please contact me at (202) 512-3604 or farrellb@gao.gov.
Contact points for our Offices of Congressional Relations and Public
Affairs may be found on the last page of this letter. GAO staff who
made key contributions to this report are listed in enclosure II.
Signed by:
Brenda S. Farrell:
Director, Defense Capabilities and Management:
Enclosures - 2:
List of Congressional Committees:
The Honorable Carl Levin:
Chairman:
The Honorable John McCain:
Ranking Member:
Committee on Armed Services:
United States Senate:
The Honorable Ike Skelton:
Chairman:
The Honorable Duncan L. Hunter:
Ranking Member:
Committee on Armed Services:
House of Representatives:
Enclosure I: Comments from the Department of Defense:
Department Of The Army:
Office Of The Deputy Chief Of Staff, G-1:
Directorate Military Personnel Management:
300 Army Pentagon:
Washington, DC 20310-0300:
September 9, 2008:
Brenda S. Farrell:
Director, Defense Capabilities and Management:
U.S. Government Accountability Office:
441 G Street, N.W.,
Washington, DC 20548:
Dear Ms. Farrell:
This is the Department of Defense (DoD) response to the GAO draft
report, GAO-08-1037, "Military Personnel: Evaluation Methods Linked to
Anticipated Outcomes Needed to Inform Decisions on Army Recruitment
Incentives," dated August 25, 2008 (GAO Code 351200).
DoD appreciates the opportunity to review and comment on the draft
report. Detailed comments on the GAO recommendations and technical
comments are enclosed. For further questions concerning this report
please contact my action officer, Lieutenant Colonel Denise Corley,
(703) 695-6673.
Signed by:
Gina S. Farrisee:
Brigadier General, U.S. Army:
Director of Military Personnel Management:
Enclosures:
GAO Draft Report - Dated August 25, 2008:
GAO Code 351200/GAO-08-1037:
"Military Personnel: Evaluation Methods Linked to Anticipated Outcomes
Needed to Inform Decisions on Army Recruitment Incentives"
Department Of Defense Comments To The Recommendations:
Recommendation 1: The GAO recommends that for each recruitment
incentive developed pursuant to Section 681 of the National Defense
Authorization Act for FY 2006, the Secretary of the Army issue guidance
to:
* clearly specify anticipated outcomes for each incentive;
* describe the assumptions behind these anticipated outcomes;
* identify the evaluation method to be used to assess progress toward
these outcomes; and;
* link anticipated outcomes to the stated evaluation method.
DOD Response: Concur. The Secretary of the Army will ensure that for
each of the recruitment incentives developed pursuant to Section 681 of
the National Defense Authorization Act for FY 2006, that the
anticipated outcomes, underlying assumptions, evaluation methods and
linkage between the anticipated outcome and evaluation method are
clearly articulated in the annual report to Congress on each recruiting
incentive.
Recommendation 2: The GAO recommends that the Secretary of the Army
ensure that the guidance referred to in Recommendation 1 is
specifically addressed in the required annual assessments for each
recruitment incentive.
DOD Response: Concur. The Secretary of the Army will ensure that the
anticipated outcomes, underlying assumptions, evaluation methods and
linkage between the anticipated outcome and evaluation method are
specifically addressed in the annual report to Congress on each
recruiting incentive and serve as the basis for continuing, modifying,
or discontinuing any or all of the incentive programs.
Recommendation 3: Should the Army decide - based on the results of the
annual assessments of each incentive - to seek to make any of these
incentives permanent, the GAO recommends that the Secretary of the
Army, for any proposals the Army may wish to develop, include in its
business case:
* clearly specified anticipated outcomes for each incentive;
* a description of the assumptions behind the anticipated outcomes;
* an identification of the evaluation method to be used to assess
progress toward these outcomes; and;
* an explanation that links anticipated outcomes to the stated
evaluation method.
DOD Response: Concur. Should the Secretary of the Army decide to seek
to make any of the recruitment incentives permanent, the Army will
develop a business case that includes clearly specified anticipated
outcomes, a description of the assumptions behind the anticipated
outcomes, an identification of the evaluation method to be used to
assess progress toward these outcomes, an explanation that links
anticipated outcomes to the stated evaluation method.
[End of enclosure]
Enclosure II: GAO Contacts and Staff Acknowledgements:
GAO Contact:
Brenda S. Farrell, (202) 512-3604 or farrellb@gao.gov:
Acknowledgements:
In addition to the individual named above, David Moser (Assistant
Director), Catherine H. Brown, Charles Perdue, Joanne Landesman, Julia
Matta, Terry Richardson, and Barbara J. Williams made key contributions
to this report.
[End of enclosure]
Footnotes:
[1] Congressional Research Service, The Library of Congress, Recruiting
and Retention: An Overview of FY2005 and FY2006 Results for Active and
Reserve Component Enlisted Personnel (Washington, D.C.: Updated Jan.
26, 2007).
[2] Pub. L. No. 109-163 (2006).
[3] See titles 10 and 37, U.S. Code.
[4] Section 681 requires the Army to notify Congress by submitting a
plan for each recruitment incentive. The Army must then wait 45 days
before providing the new recruitment incentive.
[5] DOD Directive 5500.20, Unified Legislation and Budgeting Process
(Mar. 28, 2005).
[6] The Army and other services may also submit proposals directly to
DOD's Office of General Counsel.
[7] The Officer Accession Pilot Program provides both monetary and
nonmonetary incentives through three options. According to the General
Counsel of the Department of the Army, although this incentive is
comprised of three options, it counts as a single incentive for several
reasons; among the five reasons cited by the General Counsel of the
Department of the Army is that the options within the Officer Accession
Pilot Program are mutually exclusive, meaning a recruit cannot select
more than one.
[8] The incentive amount varies by length of service and by component.
An individual enlisting in the active Army for 3 years receives
$25,000; an individual enlisting for 4 years receives $35,000; and an
individual enlisting for 5 years receives $40,000. An individual
enlisting in the Army Reserve for 3, 4, or 5 years receives $10,000,
$15,000, or $20,000, respectively. Currently, recruits who accept the
Army Advantage Fund are not required to provide evidence that they
intend to use the bonus for home ownership or small business
development. If the incentive is made permanent, the Army expects to
both make such evidence a requirement and develop an associated funding
mechanism.
[9] GAO, Military Personnel: Strategic Plan Needed to Address Army's
Emerging Officer Accession and Retention Challenges, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO-07-224] (Washington, D.C.: Jan.
19, 2007).
[10] As part of its modular force transformation effort, the Army
redesigned its 10 active duty division force into a force of 42 or 43
brigade combat teams. The redesign has resulted in entirely new
organizational structures and increased officer requirements by 4,131
spaces from 2001 to July 2006.
[11] GAO, Comptroller General's Forum, High-Performing Organizations:
Metrics, Means, and Mechanisms for Achieving High Performance in the
21st Century Management Environment, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-04-343SP] (Washington, D.C.: Feb. 13, 2004).
[12] Army Regulation 5-1, Total Army Quality Management (Mar. 15,
2002).
[13] For the active Army, "accessions" are individuals who have begun
their military service, as distinguished from those who have signed a
contract to serve but who have not yet begun their service. For the
Army Reserve and Army National Guard, the term has a broader meaning;
accession can occur shortly after an individual signs a contract, when
he or she is "shipped" to basic training, or when a service member
transfers from the active Army to the Army Reserves or Army National
Guard.
[14] Recruits who enlist in the Army National Guard are not currently
eligible for the Army Advantage Fund Pilot Program.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
