Military Recruiting
DOD Needs to Establish Objectives and Measures to Better Evaluate Advertising's Effectiveness Gao ID: GAO-03-1005 September 19, 2003The Department of Defense (DOD) must convince more than 200,000 people each year to join the military. To assist in recruiting, the military services advertise on television, on radio, and in print and participate in other promotional activities. In the late 1990s, some of the services missed their overall recruiting goals. In response, DOD added recruiting resources by increasing its advertising, number of recruiters, and financial incentives. By fiscal year 2003, DOD's total recruiting budget was approaching $4 billion annually. At the request of Congress, GAO determined the changes in DOD's advertising programs and funding trends since the late 1990s and assessed the adequacy of measures used by DOD to evaluate the effectiveness of its advertising. GAO recommends that DOD set clear, measurable advertising
Since the late 1990s, DOD has revamped its recruiting advertising programs and nearly doubled the funding for recruiting advertising. The military services have revised many of their advertising campaigns and focused on complementing traditional advertising, such as by increasing the use of the Internet, and participating in more promotional activities, such as sports car racing events. DOD's total advertising funding increased 98 percent in constant dollars from fiscal year 1998 through fiscal year 2003--from $299 million to $592 million. The advertising cost per enlisted recruit has nearly tripled and is now almost $1,900. The military services agree that the revised strategies and increased investments have energized their advertising campaigns and better positioned them to recruit in an increasingly competitive marketplace. Today, almost all of the active and reserve components are meeting their overall recruiting goals in terms of the quality and quantity of new recruits. DOD does not have clear program objectives and adequate outcome measures to evaluate the effectiveness of its advertising as part of its overall recruiting effort. Thus, DOD cannot show that its increased advertising efforts have been a key reason for its overall recruiting success. Isolating the impact of advertising on recruiting efforts is inherently difficult because joining the military is a profound life decision. Moreover, DOD has not consistently tracked key information, such as public awareness of military recruiting advertising and the willingness of young adults to join the military. Such data could be used to help evaluate the effectiveness of advertising. Without sufficient information on advertising's effectiveness, DOD cannot determine the return on its advertising funding or make fact-based choices on how its overall recruiting investments should be allocated.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-03-1005, Military Recruiting: DOD Needs to Establish Objectives and Measures to Better Evaluate Advertising's Effectiveness
This is the accessible text file for GAO report number GAO-03-1005
entitled 'Military Recruiting: DOD Needs to Establish Objectives and
Measures to Better Evaluate Advertising's Effectiveness' which was
released on September 19, 2003.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Senate and House Committees on Armed Services:
United States General Accounting Office:
GAO:
September 2003:
Military Recruiting:
DOD Needs to Establish Objectives and Measures to Better Evaluate
Advertising's Effectiveness:
GAO-03-1005:
GAO Highlights:
Highlights of GAO-03-1005, a report to the Senate and House Committees
on Armed Services
Why GAO Did This Study:
The Department of Defense (DOD) must convince more than 200,000 people
each year to join the military. To assist in recruiting, the military
services advertise on television, on radio, and in print and
participate in other promotional activities. In the late 1990s, some
of the services missed their overall recruiting goals. In response,
DOD added recruiting resources by increasing its advertising, number
of recruiters, and financial incentives. By fiscal year 2003, DOD‘s
total recruiting budget was approaching $4 billion annually.
At the request of Congress, GAO determined the changes in DOD‘s
advertising programs and funding trends since the late 1990s and
assessed the adequacy of measures used by DOD to evaluate the
effectiveness of its advertising.
What GAO Found:
Since the late 1990s, DOD has revamped its recruiting advertising
programs and nearly doubled the funding for recruiting advertising.
The military services have revised many of their advertising campaigns
and focused on complementing traditional advertising, such as by
increasing the use of the Internet, and participating in more
promotional activities, such as sports car racing events. DOD‘s total
advertising funding increased 98 percent in constant dollars from
fiscal year 1998 through fiscal year 2003”from $299 million to $592
million. The advertising cost per enlisted recruit has nearly tripled
and is now almost $1,900. The military services agree that the revised
strategies and increased investments have energized their advertising
campaigns and better positioned them to recruit in an increasingly
competitive marketplace. Today, almost all of the active and reserve
components are meeting their overall recruiting goals in terms of the
quality and quantity of new recruits.
DOD does not have clear program objectives and adequate outcome
measures to evaluate the effectiveness of its advertising as part of
its overall recruiting effort. Thus, DOD cannot show that its
increased advertising efforts have been a key reason for its overall
recruiting success. Isolating the impact of advertising on recruiting
efforts is inherently difficult because joining the military is a
profound life decision. Moreover, DOD has not consistently tracked key
information, such as public awareness of military recruiting
advertising and the willingness of young adults to join the military.
Such data could be used to help evaluate the effectiveness of
advertising. Without sufficient information on advertising‘s
effectiveness, DOD cannot determine the return on its advertising
funding or make fact-based choices on how its overall recruiting
investments should be allocated.
What GAO Recommends:
GAO recommends that DOD set clear, measurable advertising objectives;
develop outcome measures to evaluate advertising programs‘
performance; and use these measures to monitor advertising‘s
performance and make choices on recruiting investments.
In its comments on this report, DOD concurred with the recommendations
and stated that it will develop a DOD advertising strategic framework
to provide overall direction.
www.gao.gov/cgi-bin/getrpt?GAO-03-1005.
To view the full product, including the scope and methodology, click
on the link above. For more information, contact Derek B. Stewart at
(202) 512-5559 or stewartd@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
DOD Has Revised Advertising Programs and Increased Funding:
DOD Does Not Adequately Measure Advertising's Effectiveness:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Comments from the Department of Defense:
Related GAO Products:
Tables:
Table 1: DOD's Advertising Campaign Slogans, Program Descriptions, and
Examples of Key Changes:
Table 2: Summary of DOD's Recruiting Advertising Funding:
Figures:
Figure 1: Army-Sponsored Sports Racing Car:
Figure 2: Total DOD Recruiting Advertising Funding for Fiscal Years
1998 to 2003:
Figure 3: Total Recruiting Investment per Enlisted Recruit for Fiscal
Years 1990 to 2003:
Figure 4: The Use of Advertising throughout the Recruiting Process:
Abbreviations:
DOD: Department of Defense:
GAO: General Accounting Office:
GPRA: Government Performance and Results Act:
United States General Accounting Office:
Washington, DC 20548:
September 19, 2003:
The Honorable John W. Warner
Chairman
The Honorable Carl Levin
Ranking Minority Member
Committee on Armed Services
United States Senate:
The Honorable Duncan L. Hunter
Chairman
The Honorable Ike Skelton
Ranking Minority Member
Committee on Armed Services
House of Representatives:
To meet its human capital needs, the Department of Defense (DOD) must
convince about 200,000 people each year--the majority of them recent
high school graduates--to join the military. To assist in this
recruiting effort, the military services advertise on television, on
radio, in print, and on the Internet; sponsor sports teams; and
participate in other promotional activities. Such advertising is
designed primarily to raise awareness of the military as a career
option and help recruiters meet their goals for new recruits. During
the exceptionally strong U.S. economy of the late 1990s, most of the
services missed their overall recruiting goals. In response, DOD put
additional resources into recruiting by increasing advertising, the
number of recruiters, and various incentives, such as enlistment
bonuses. By fiscal year 2003, DOD's total recruiting budget was
approaching $4 billion annually.
The Senate Committee on Armed Services directed that we examine DOD's
growing investments in military recruitment advertising.[Footnote 1] As
agreed with your committees, the objectives of this report were to (1)
determine the changes in DOD's advertising programs and funding trends
since the late 1990s and (2) assess the adequacy of the measures used
by DOD to evaluate the effectiveness of its advertising. In March 2003,
we provided your committees with an interim briefing that described the
trends in advertising funding requests since fiscal year 2000 and DOD's
justifications for those requests.
This report updates the information discussed in the interim briefing
and provides our analysis of the other issues in your request. To
determine the changes in DOD's advertising programs and funding trends
since the late 1990s, we reviewed the changes in the services'
advertising programs, DOD's and the services' congressional
justification books, and DOD funding data. To assess the adequacy of
DOD's outcome measures, we used established management guidance
provided in the Government Performance and Results Act[Footnote 2]
(GPRA) and in Office of Management and Budget guidance. Our scope
included DOD's active duty services, reserve components, and joint
advertising program. We conducted our review from October 2002 through
July 2003 in accordance with generally accepted government auditing
standards. (See appendix I for more detailed information on our scope
and methodology.):
Results in Brief:
Since the late 1990s, DOD has revamped its recruiting advertising
programs and nearly doubled the funding devoted to recruiting
advertising. The active duty military services, except for the Marine
Corps, substantially revised their advertising campaigns and selected
new advertising agencies as contractors. Long-time and well-recognized
advertising slogans such as the Army's "Be All You Can Be" were
abandoned in favor of campaigns designed to better appeal to today's
young adults. The military services agree that these revised strategies
and increased investments have energized their advertising campaigns
and better positioned them to recruit young adults in an increasingly
competitive marketplace. Today, almost all of the active and reserve
components report that they are meeting their overall recruiting goals,
both in terms of the quality and quantity of recruits. To better reach
today's young adults, the services have focused on complementing
traditional advertising by increasing funding for events marketing,
public relations, and the Internet. The expenditures for paid
television, which remains the single largest advertising cost, and
other national media have declined as a percentage of total advertising
funding. DOD's total advertising funding increased 98 percent from
fiscal year 1998 through fiscal year 2003--from $299 million to $592
million.[Footnote 3] Today, DOD is spending $1,900 in advertising per
enlisted recruit, which is almost three times as much as it did in
fiscal year 1990. The increases in funding have not been evenly
distributed across DOD's advertising programs. The size of each
service's advertising programs varies greatly. The Army has the largest
advertising programs; its active and reserve components account for
nearly half of the total advertising funding.
DOD does not have adequate outcome measures to evaluate the
effectiveness of its advertising as part of its overall recruiting
effort. Thus, DOD cannot show that its increased advertising efforts
have been a key reason for its overall recruiting success. Evaluating
advertising's effectiveness requires that DOD establish clear program
objectives and outcome measures. DOD has not established such
objectives and outcome measures for two reasons. First, isolating the
impact of advertising on recruiting efforts is inherently difficult
because joining the military is a profound life decision influenced by
many factors, including the opportunities available in college or in
the job market. Second, even though DOD has developed recruiting goals
to ensure that it meets its human capital needs, these goals do not
directly relate to or measure the effectiveness of advertising. Owing
to the absence of program objectives and outcome measures, DOD has not
consistently tracked key information, such as public awareness of
military recruiting advertising and changes in the willingness of young
adults to join the military. Such information could be used to help
evaluate the effectiveness of advertising. In our 2000 report, we
recommended that DOD and the services assess the relative success of
their recruiting strategies, including how the services can create the
most cost-effective mix of recruiters, enlistment bonuses, college
incentives, advertising, and other recruiting tools.[Footnote 4]
Although DOD acknowledges the need for such information, current DOD
guidance does not require the measurement of outcomes or reports on
advertising's effectiveness. Without sufficient information on
advertising's effectiveness, DOD cannot determine the return on its
advertising funding or make fact-based choices on how its overall
recruiting investments should be allocated.
We are making recommendations to DOD to improve its guidance to better
evaluate recruiting advertising's effectiveness. We are recommending
that DOD set clear, measurable advertising objectives for its
advertising programs and develop outcome measures to evaluate the
performance of its advertising programs. We are also recommending that
DOD use these outcome measures to monitor its advertising programs'
performance and make fact-based choices about advertising funding as
part of the overall recruiting investment.
In commenting on a draft of this report, DOD concurred with all of our
recommendations. DOD stated in its comments that it will implement the
recommendations by developing a DOD advertising strategic framework to
provide overall direction for its advertising programs and by
conducting research initiatives intended to advance the measurement of
the performance of recruiting and advertising.
Background:
Most of the military services' active and reserve components faced
recruiting difficulties during the strong economic climate of the late
1990s. As a result, the services stepped up their recruiting to ensure
that they would have enough recruits to fill their ranks. Recruiting
efforts focus on three initiatives. First, a "sales force" of more than
15,000 recruiters, who are mostly located in the United States, recruit
from the local population. Second, these recruiters have financial and
other incentives that they can use to convince young adults to consider
a military career. Such incentives include enlistment bonuses and
college benefits. Finally, the services use advertising to raise the
public's awareness of the military and help the sales force of
recruiters reach the target recruiting population and generate
potential leads for recruiters. This advertising can include television
and radio commercials, Internet and printed advertisements, and special
events.
DOD believes that advertising is increasingly critical to its
recruiting effort because convincing young adults to join the military
is becoming more difficult. In 2001, over 70 percent of polled young
adults said that they probably or definitely would not join the
military, compared with 57 percent in 1976.[Footnote 5] The number of
veterans is declining, which means that fewer young adults have
influencers--a relative, coach, or teacher--who have past military
experience. Compounding these difficulties, proportionally more high
school graduates are attending college. Finally, the perception that
service in the military is arduous--and possibly dangerous--can inhibit
recruiting efforts. DOD believes that these factors together make the
military an increasingly harder sell as a career choice and life-style
option for young adults.
The Office of the Secretary of Defense is responsible for establishing
policy and providing oversight for the military recruiting and
advertising programs of the active and reserve components. Within the
Office of the Secretary of Defense, the Under Secretary for Personnel
and Readiness is responsible for developing, reviewing, and analyzing
recruiting policy, plans, and resource levels. The office provides
policy oversight for advertising programs and coordinates them through
the Joint Marketing and Advertising Committee. DOD's strategic plan for
military personnel human resources emphasizes the need to recruit,
motivate, and retain adequate and diverse numbers of quality
recruits.[Footnote 6]
DOD's recruiting and advertising programs are not centrally managed.
All of the active components and some of the reserve components manage
their separate advertising programs and work closely with their own
contracted advertising agencies.[Footnote 7] DOD and the services
believe that this decentralized approach better differentiates between
the service "brands" (i.e., Army, Navy, Air Force, Marines). The Joint
Advertising, Market Research, and Studies program, which is funded
separately by DOD, exists to address common DOD requirements, such as
conducting market research and obtaining and distributing lists of
potential leads. The joint program has developed a DOD-wide advertising
campaign to target the adult influencers of potential recruits, but
this program had not been fully implemented at the time of our review.
DOD Has Revised Advertising Programs and Increased Funding:
After most of the services experienced recruiting shortfalls in the
late 1990s, DOD reviewed its advertising programs and identified
opportunities for improvement. The services, except the Marine Corps,
substantially revised their advertising campaigns and slogans and
contracted with new advertising agencies. The services told us that
their revised campaigns place them in a better position to recruit
today's young adults. Currently, almost all of the services and reserve
components are achieving their recruiting goals, and advertising
funding has almost doubled since fiscal year 1998. The increases in
funding have not been used to buy more national media, such as
television commercials. Rather, the funding increases are being
directed to other types of advertising, such as special events
marketing and the Internet, that are intended to better reach today's
young adults. Advertising funding for DOD increased from $299 million
in fiscal year 1998 to $592 million in fiscal year 2003, an increase of
98 percent.[Footnote 8]
Military Services Have Revised Their Advertising Campaigns to Better
Attract Today's Young Adults:
Recruiting shortfalls in the late 1990s led to an examination and
revision of DOD's advertising programs. The Army, Navy, and Air Force
missed their recruiting quantity goals, while some of the reserve
components fell short of both their quantity and quality
goals.[Footnote 9] Following these recruiting shortfalls, Congress
asked the Secretary of Defense to review DOD's advertising programs and
make recommendations for improvements.[Footnote 10] DOD has revamped
its advertising programs. The active-duty services, except for the
Marine Corps, substantially revised their advertising campaigns and
selected new advertising agencies as their contractors. They produced
new advertising strategies and campaigns, complete with new slogans and
revised television, print, and radio advertisements, along with new
brand images defined by distinct logos, colors, and music. The
services, in conjunction with their advertising agencies, conducted new
research on young adults--their primary target market. During this
period, the joint program developed an advertising campaign to target
influencers of prospective recruits, as recommended in DOD's review.
In addition to their overall campaigns, all of the services have
specialized campaigns to target diverse segments of the young adult
population. For instance, the Navy created a Web site, called El Navy,
which is designed to better communicate with the Hispanic market, and
the Army has specifically tailored radio advertisements to reach the
African American market. The services also incorporated a greater
variety of public relations and promotional activities, such as
participating in job fairs and sponsoring sports car racing teams, as
an integral part of their advertising programs. As shown in table 1,
there are essentially nine advertising programs that are managed
separately by the military services, reserve components, and the Office
of the Secretary of Defense.
Table 1: DOD's Advertising Campaign Slogans, Program Descriptions, and
Examples of Key Changes:
Components: Army, Army Reserve; Current campaign slogans: (year
established): An Army of One (2001); Program descriptions and examples
of key changes: * Army and Army Reserve combined programs; * Army
National Guard program independently managed; * Largest active and
reserve recruiting mission and advertising budget; * New advertising
campaigns and contractors; * Advertises in all national media venues;
* Engages in promotional events, such as sports car racing sponsorship,
high school sports, and video games; * Initiated on-line recruiting.
Components: Army National Guard; Current campaign slogans: (year
established): You Can (1997).
Components: Navy; Current campaign slogans: (year established):
Accelerate Your Life (2001); Program descriptions and examples of key
changes: * Separate advertising programs for Navy and Naval Reserve
(reorganized recruiting under one commander); * Second largest active
duty recruiting mission; * New advertising campaigns and contractors;
* Advertises in all major media.
Components: Naval Reserve; Current campaign slogans: (year
established): Stay Strong (2001).
Components: Air Force; Current campaign slogans: (year established):
Cross into the Blue (2001); Program descriptions and examples of key
changes: * Three independent advertising programs; * Third largest
active duty recruiting mission; * New advertising campaigns and
contractors; * Active Air Force emphasizing promotional activities and
events, such as traveling recruiting trucks and sports car racing; *
Initiated a national television campaign for the active Air Force; *
Increased use of Internet recruiting across the active and reserve
components.
Components: Air Force Reserve; Current campaign slogans: (year
established): Above and Beyond (1998).
Components: Air National Guard; Current campaign slogans: (year
established): Fuel Your Future (1999).
Components: Marine Corps, Marine Corps Reserve; Current campaign
slogans: (year established): Marines, The Few, The Proud (1986);
Program descriptions and examples of key changes: * Marine Corps and
Marine Corps reserve integrated programs; * Smallest recruiting
mission of the services; * Marine Corps brand image not changed for 30
years and same advertising contractor for 56 years; * Emphasis on
television, especially sports programming.
Components: Joint Program; Current campaign slogans: (year
established): Today's Military--See It for What It Really Is (2003);
Program descriptions and examples of key changes: * Developed
advertising campaign to target influencers of prospective recruits that
includes magazine advertisements, use of Web site, and television
public service announcements; * New advertising contractor; *
Conducts market research and studies for DOD's advertising programs; *
Provides other support for DOD's advertising programs.
Source: DOD.
[End of table]
The active services told us that they are pleased with their new
advertising campaigns and agencies, and they believe that the revised
and better-funded campaigns have placed them in a more competitive
position to recruit young adults. The sluggish U.S. economy has also
narrowed employment options and is considered to be an important factor
in easing the recruiting challenge. Today, all of the active services
are meeting or exceeding their overall recruiting goals. Most of the
reserve components are also achieving their recruiting goals. As of
June 2003, the Army National Guard was falling short of its recruiting
goals because of extensive overseas deployments and the implementation
of stop loss (restrictions on leaving the military). Army National
Guard officials stated that they expect to meet their goals by the end
of fiscal year 2003. Some reserve officers expressed concerns about the
negative impact of the recent high deployment rates on future
recruiting. The services, especially the reserve components, continue
to face challenges in recruiting individuals with some types of
specific training or skills, such as medical, legal, and construction,
and they have developed some specialized advertising campaigns targeted
to recruit them.
Since fiscal year 1998, the services have changed how they allocate
advertising funding, according to the figures provided by DOD. Grouped
into three broad categories, advertising funding includes: (1) events
marketing, public affairs and public relations, Internet, and other;
(2) national media; and (3) direct mail and miscellaneous recruiting
support. One of the categories--events marketing, public affairs and
public relations, Internet, and other--has shown the greatest increase
as a percentage of the total budget, nearly tripling from around 10
percent in fiscal year 1998 to 29 percent in fiscal year 2003. This
increase was used partly to create and produce new advertising
campaigns and strategies. Service officials told us that event
marketing and public relations activities provide recruiters with
greater opportunities to interact with potential recruits and
supplement their national media campaigns and other methods of
advertising. One example is the Army's sponsorship of a sports racing
car. (See fig. 1.) Internet and Web-site recruiting have also increased
significantly from fiscal year 1998 through fiscal year 2003. All of
the active military services have increased the amount of advertising
on the Internet and have used interactive Web sites to complement their
traditional recruiting and advertising methods.
Figure 1: Army-Sponsored Sports Racing Car:
[See PDF for image]
[End of figure]
The expenditures for the national media category, which includes paid
television, radio, and magazine advertisements, have remained
relatively constant. This means that this category's proportion of the
growing total advertising budgets has actually declined. Specifically,
expenditures for the national media in fiscal year 1998 were more than
half of the advertising budget; currently, it represents about 40
percent. Television advertising--which offers tremendous reach to
target audiences--dominates this category. Television advertising has
remained the single largest advertising expenditure: paid television is
still about a quarter of the total advertising budget for all of the
military components.
DOD Has Significantly Increased Funding for Advertising:
DOD's advertising funding has nearly doubled in the years since 1998
and most of these increases occurred in the earlier years. (See fig.
2.) Total advertising funding for all of the services increased 98
percent, from $299 million in fiscal year 1998 to $592 million in
fiscal year 2003.[Footnote 11] The total DOD advertising budget request
to Congress for fiscal year 2004 was $592.8 million.
Figure 2: Total DOD Recruiting Advertising Funding for Fiscal Years
1998 to 2003:
[See PDF for image]
Note: The funding amounts were taken from DOD's and the services'
congressional budget justification books (adjusted to account for
inflation).
[A] In-year estimate.
[End of figure]
Since fiscal year 1998, DOD's advertising funding, which is included in
DOD's operation and maintenance appropriations, has increased at a
significantly higher rate than the total of all of DOD's operation and
maintenance funding. DOD officials cite media inflation as one reason
for increased advertising funding. Inflation for some types of media,
especially for television commercials, has been higher than general
inflation. However, this is not the reason for all of the increases in
advertising funding during this period because not all of the
advertising funding is used for media advertising. For example, only
about a quarter of advertising funds are currently spent to buy time to
run television commercials.
Growing advertising costs are only part of a rapidly increasing total
investment in recruiting. The rising advertising and overall recruiting
costs can be seen in the investment per enlisted recruit--an important
bottom-line measure that shows the amount of money spent to enlist each
recruit. Today, the services are spending almost three times as much on
advertising per recruit than in fiscal year 1990. We examined data
collected by DOD from the services, and it showed that the total
advertising investment per enlisted recruit rose from approximately
$640 to $1,900 between fiscal year 1990 and fiscal year 2003. As a
proportion of the total recruiting investment, advertising has
increased from 8 percent in fiscal year 1990 to 14 percent in fiscal
year 2003. Bonuses and incentives to enlist have also increased
substantially during this same period. The total recruiting investment
per recruit increased almost 65 percent, from approximately $8,100 in
fiscal year 1990 to $13,300 in fiscal year 2003. Very steep growth
occurred between fiscal year 1998 and fiscal year 2002. This is shown
in figure 3.
Figure 3: Total Recruiting Investment per Enlisted Recruit for Fiscal
Years 1990 to 2003:
[See PDF for image]
[End of figure]
The increases are not evenly distributed across the services'
advertising programs. (See table 2.) The Army has the largest
advertising budget, and the Army active and reserve components account
for nearly half (about $295 million) of the total advertising funding.
The Marine Corps, at just under $50 million, has the smallest
advertising budget. The Air Force has experienced the most significant
increase in funding, in part owing to the creation of its first
national television campaign. The Navy's advertising funding has also
increased, but this is primarily due to the addition of costs related
to the Blue Angels[Footnote 12] and a program to test recruiting kiosks
at public locations.
DOD's Joint Advertising, Market Research, and Studies Program is
responsible for (1) providing market research and studies for
recruiting and (2) developing an advertising campaign to target adult
influencers, such as parents, coaches, and career counselors.
Currently, the joint program is conducting market research and studies
and providing other support for the services' advertising programs,
such as purchasing lists of high school students and recent graduates
for use in mailing advertisements. In addition, the program is
implementing a limited print advertising campaign targeting influencers
in fiscal year 2003.
The joint advertising campaign has not had consistent funding. Program
managers told us that the current funding level is insufficient to
fully implement the influencer advertising campaign they have
developed. In past years, DOD cut funding for the joint advertising
program because of concerns that the program office was not adequately
executing its advertising budget. For fiscal year 2003, Congress
provided the joint advertising program with less funding than DOD
requested, and DOD subsequently reallocated part of the remaining joint
advertising funding to a program that it considered a higher priority.
Table 2: Summary of DOD's Recruiting Advertising Funding:
Constant fiscal year 2003 dollars in millions.
Army; 1998: (actual): $113.7; 2003: (in-year estimate): $196.9;
Percentage: change: 73.
Army Reserve; 1998: (actual): 17.0; 2003: (in-year estimate): 50.2;
Percentage: change: 196.
Army National Guard[A]; 1998: (actual): 23.2;
2003: (in-year estimate): 48.2; Percentage: change: 108.
Navy; 1998: (actual): 75.7; 2003: (in-year
estimate): 107; Percentage: change: 41.
Naval Reserve; 1998: (actual): 2.4; 2003:
(in-year estimate): 7.4; Percentage: change: 208.
Air Force; 1998: (actual): 18.3; 2003:
(in-year estimate): 90.5; Percentage: change: 395.
Air Force Reserve; 1998: (actual): 4.6; 2003:
(in-year estimate): 13.5; Percentage: change: 193.
Air National Guard[A]; 1998: (actual): 4.4;
2003: (in-year estimate): 5.8; Percentage: change: 31.
Marine Corps; 1998: (actual): 29.8; 2003:
(in-year estimate): 46.5; Percentage: change: 56.
Marine Corps Reserve; 1998: (actual): 3.0;
2003: (in-year estimate): 2.9; Percentage: change: -3.
Joint Program; 1998: (actual): 6.8; 2003:
(in-year estimate): 22.9[B]; Percentage: change: 237.
Totals; 1998: (actual): $298.9; 2003: (in-year
estimate): $591.8; Percentage: change: 98.
Source: DOD.
Note: The funding amounts were taken from DOD's and the services'
congressional budget justification books (adjusted to account for
inflation).
[A] The advertising funding for the Army National Guard and the Air
National Guard is for both recruiting and retention. These figures do
not include the funding for recruiting and retention advertising done
by the states and territories.
[B] DOD subsequently reallocated part of this funding.
[End of table]
DOD Does Not Adequately Measure Advertising's Effectiveness:
DOD does not have adequate outcome measures to evaluate the
effectiveness of its advertising as part of its overall recruiting
effort. Effective program management requires the establishment of
clear objectives and outcome measures to evaluate the program, and DOD
has established neither. This has been a long-standing problem for DOD
primarily because measuring the impact of advertising is inherently
difficult, especially for a major life decision such as joining the
military. Owing to the absence of established advertising objectives
and outcome measures, DOD has not consistently collected and
disseminated key information that would allow it to better assess
advertising's contribution to achieving recruiting goals. This
information would include public awareness of military recruiting
advertising and the willingness of young adults to join the military.
Rather, the services report outcome measures that focus on achieving
overall recruiting goals instead of isolating the specific contribution
of advertising. Without adequate information and outcome measures, the
Office of the Secretary of Defense cannot satisfactorily review the
services' advertising budget justifications nor can it determine the
return on their advertising dollars as part of their overall recruiting
investment.
Effective Management Requires Clear Program Objectives and Outcome
Measures:
The Secretary of Defense is required by law to enhance the
effectiveness of DOD's recruitment programs through an aggressive
program of advertising and market research targeted at prospective
recruits and those who may influence them.[Footnote 13] DOD guidance
requires the services, by active and reserve components, to report
their resource inputs--how much they are spending on
advertising.[Footnote 14] DOD guidance also requires the services to
report on overall recruiting outcomes[Footnote 15]--their recruit
quantity and quality. However, the guidance does not require active and
reserve components to report information specifically about the
advertising programs' recruiting effectiveness.
Effective program management requires the establishment of clearly
defined objectives and outcome measures to evaluate programs. The
Government Performance and Results Act was intended to help federal
program managers enhance the effectiveness of their programs.[Footnote
16] It requires agencies to establish strategic plans for program
activities that include, among other things, a mission statement
covering major functions and operations, outcome-related goals and
objectives, and a description of how these goals and objectives are to
be achieved. GPRA shifted the focus of accountability for federal
programs from inputs, such as staffing and resource levels, to
outcomes. This requires agencies to measure the outcomes of their
programs and to summarize the findings of program evaluations in their
performance reports. The Office of Management and Budget's guidance
implementing GPRA requires agencies to establish meaningful program
objectives and identify outcome measures that compare actual program
results with established program objectives.[Footnote 17]
Lack of Adequate Outcome Measures Limits DOD's Ability to Effectively
Allocate Its Recruiting Investments:
DOD does not have adequate information to measure the effectiveness of
its advertising as part of the overall recruiting effort. Measuring
advertising's effectiveness has been a long-standing problem, partly
because it is inherently difficult to measure the impact that
advertising has on recruiting. DOD has not established advertising
program objectives and it lacks adequate outcome measures of the impact
that advertising programs have on recruiting. Outcome measures are used
to evaluate how closely a program's achievements are aligned with
program objectives, and to assess whether advertising is achieving its
intended outcome. DOD currently requires the services and reserve
components to report on inputs and outcomes related to overall
recruiting. These measures are important in assessing DOD's overall
recruiting success; however, they do not assess advertising's
contribution to the recruiting process.
Measuring Advertising's Effectiveness Is a Long-standing and Difficult
Problem:
In our 2000 report, we noted that the services do not know which of
their recruiting initiatives--advertising, recruiters, or bonuses--
work best.[Footnote 18] This prevented DOD from being able to
effectively allocate its recruiting investment among the multiple
recruiting resources. We recommended that DOD and the services assess
the relative success of their recruiting strategies, including how the
services can create the most cost-effective mix of recruiters,
enlistment bonuses, college incentives, advertising, and other
recruiting tools. In comments on that report, DOD stated that it
intended to develop a joint-service model that would allow trade-off
analyses to determine the relative cost-effectiveness of the various
recruiting resources. This has not been done, and the current DOD cost
performance trade-off model does not support analyses across the
services' budgets.
Similarly, a 2002 Office of Management and Budget assessment, known as
the Program Assessment Rating Tool, found that DOD's recruiting program
had met its goal of enlisting adequate numbers of recruits; however,
since there were no measures of program efficiency, the overall rating
for the recruiting program was only "moderately effective." In its
assessment, the Office of Management and Budget noted the inability of
the recruiting program to assess the impact of individual resources,
such as advertising and recruiters. The services continually adjust the
mix of funding between advertising and other recruiting resources to
accomplish their program goals. They have generally increased spending
on advertising, added recruiters, and increased or added bonuses at the
same time, making it impossible to determine the relative value of each
of these initiatives. Other studies have reached similar conclusions.
In 2000, a review of DOD's advertising programs resulted in a
recommendation that they be evaluated for program
effectiveness.[Footnote 19] More recently, the National Academy of
Sciences also cited the need to evaluate advertising's direct influence
on actual enlistments.[Footnote 20] The academy is now doing additional
work on evaluating DOD's advertising and recruiting.
The lack of adequate information is partly attributable to the inherent
difficulty in measuring advertising's affect on recruiting. Measuring
advertising's effectiveness is a challenge for all businesses,
according to advertising experts. Private-sector organizations cannot
attribute increases in sales directly to advertising because there are
many other factors influencing the sale of products, such as quality,
price, and the availability of similar products. Many factors impact
recruiting as well, such as employment and educational opportunities,
making it especially difficult to isolate and measure the effectiveness
of advertising. Enlisting in a military service is a profound life
decision. Typically, an enlistment is at least a 4-year commitment and
can be the start of a long military career.
Another complicating factor in measuring advertising's effectiveness is
that it consists of different types and is employed differently
throughout the recruiting process to attract and enlist potential
recruits. Figure 4 displays the recruiting process and demonstrates the
role of advertising while a young adult may be considering enlisting in
the military.
Figure 4: The Use of Advertising throughout the Recruiting Process:
[See PDF for image]
[End of figure]
As the figure shows, the use of multiple types of advertising at
various stages in the recruiting continuum makes it difficult to assess
the effectiveness of specific types of advertising. A single recruit
may be exposed to some or all of these advertising types. Traditional
advertising in the national media, such as television and magazines, is
intended to disseminate information designed to influence consumer
activity in the marketplace. The services typically use such national
media to make young people aware of a military service, the career
options available in a service, and other opportunities the services
have to offer them. Direct mail, special events, and the services' Web
sites are utilized to provide more detailed information about the
services and the opportunities available for persons who enlist. These
marketing resources give people the opportunity to let a recruiter know
they are possibly interested in enlisting in a service.
DOD Lacks Guidance Establishing Advertising Programs' Objectives and
Outcome Measures:
Another contributing factor to the absence of advertising objectives
and outcome measures is the lack of DOD-wide guidance. Officials from
the Office of the Secretary of Defense view their role as overseeing
the decentralized programs managed by the individual services and
reserve components. They scrutinize the quality and quantity of
recruits and gather data about the uses of advertising funds. However,
they told us they were reluctant to be more prescriptive because of a
concern about appearing to micromanage the successful recruiting
programs of the active and reserve components. On the basis of our
work, their sensitivity is warranted. The active and reserve components
tend to guard their independence, seeking to maintain their "brand" and
arguing that the current decentralized structure allows them to be more
responsive to their individual needs. The Office of the Secretary of
Defense seeks to coordinate the active and reserve components'
activities through joint committees and to centralize research that can
be utilized by all.
Defining exactly what to measure may be difficult, but it is not
impossible. DOD and the services, as well as their contracted
advertising agencies, generally agree that there are at least two key
advertising outcomes that should be measured: (1) the awareness of
recruiting advertising and (2) the willingness or "propensity" to
consider joining the military. However, this is not clearly stated in
any program guidance. Current DOD guidance requires only that the
services provide information on funding for advertising, the quality
and quantity of recruits, and the allocation of resources to the
various advertising categories.[Footnote 21] Although this information
is valuable--in fact, critical--it is not sufficient to evaluate and
isolate the effectiveness of the services' advertising programs.
DOD's efforts thus far to measure the awareness of recruiting
advertising and willingness to join the military have met with
problems. Inconsistent funding for the Joint Advertising, Market
Research, and Studies program has hampered consistent collection of
this information. DOD has sponsored an advertising tracking study
designed to monitor the awareness of individual service campaigns since
2001. However, officials from the Army, Navy, and Marine Corps told us
that they do not regularly use the research provided by this study.
According to program officials, there were numerous problems with the
advertising tracking study.[Footnote 22] DOD is implementing changes to
the study that are intended to improve its usefulness to all of the
active and reserve components. In the absence of reliable and timely
advertising tracking, the Army implemented its own tracking study, and
the Air Force is currently planning an experimental study to assess the
effectiveness of its national television advertising campaign,
according to program managers. To monitor the willingness to join the
military, DOD sponsors youth and adult polls, which are designed to
track changes in attitudes and young adults' aspirations. These polls
replaced the Youth Attitude Tracking Survey, which had been in place
for a number of years and provided long-term trend data about the
propensity of young adults to consider the military. The services
expressed concern that the current polls ask questions that are
significantly different from those asked in the prior survey, which
makes the analysis of trends difficult.
DOD officials also pointed to research indicating that advertising is a
cost-effective recruiting investment when compared with other
recruiting initiatives. For example, a report that was done for DOD
found that it was less expensive to enlist a recruit through increased
investments in advertising than through increased investments in
military pay for new recruits in the Army and Navy.[Footnote 23]
Similarly, a study for DOD analyzed the marginal cost of different
recruiting initiatives and concluded that, under certain conditions, it
was more cost-effective to invest additional funds in advertising than
in military pay for recruits or recruiters.[Footnote 24] DOD officials
told us that these reports, which used data from the 1980s and early
1990s, provide the best research available on the topic. However, the
situation has changed dramatically in recent years. DOD has altered its
advertising and recruiting strategies and is spending much more on
advertising. Advertising itself is also changing and is more fragmented
with an expanding array of television channels and other media.
Finally, media inflation, which has increased faster than general
inflation even in the sluggish economy, has lessened buying power.
Conclusions:
Funding devoted to advertising has increased considerably since fiscal
year 1998. Although the military services are now generally meeting
their overall recruiting goals, the question of whether the significant
increases in advertising budgets were a main contributor to the
services' recruiting successes remains open. During the same period,
DOD also greatly increased funding for bonuses and other incentives to
enlist recruits. At the same time, the U.S. economy slowed
dramatically, narrowing the other employment options available to young
people. These factors make it difficult to disentangle the effects of
the internal DOD investments made in recruiting from the changes in the
external recruiting environment. Even though the effect of advertising
is inherently difficult to measure, this issue needs to be addressed.
This is crucial because DOD is now spending nearly $592 million
annually on recruiting advertising, or about $1,900 per enlisted
recruit. In addition, the total funding for all of DOD's recruiting
efforts is now almost $4 billion.
DOD needs better advertising outcome measures to allow it to oversee
and manage the advertising investment as part of its overall recruiting
effort. DOD and the services have an understandable focus on the most
important program outcome--to ensure that the military has enough
quality recruits to fill its ranks. Judged by this short-term measure,
the recruiting programs are successful. But now that DOD is meeting its
recruiting goals, should it reduce advertising funding or continue at
its current funding levels? DOD believes that continued investments in
advertising are critical to keeping awareness up in the young adult
population and combating the declining propensity among today's young
adults to join the military. However, DOD has neither stated these
goals clearly in its guidance, nor consistently gathered information to
ensure that these objectives are being met. Now that it is meeting its
recruiting goals, DOD needs to turn its attention to program
effectiveness and efficiency to ensure that the active and reserve
components are getting the best return on their recruiting and
advertising investments.
Recommendations for Executive Action:
To improve DOD's ability to adequately measure the impact of its
advertising programs on its recruiting mission, we recommend that the
Secretary of Defense direct the Under Secretary of Defense for
Personnel and Readiness to issue guidance that would (1) set clear,
measurable objectives for DOD's advertising programs; (2) develop
outcome measures for each of DOD's advertising programs that clearly
link advertising program performance with these objectives; and (3) use
these outcome measures to monitor the advertising programs' performance
and make fact-based choices about advertising funding as part of the
overall recruiting investment in the future.
Agency Comments and Our Evaluation:
DOD concurred with all of our recommendations. In commenting on this
report, DOD stated that the Office of the Under Secretary of Defense
for Personnel and Readiness, in concert with the services, will develop
an advertising strategic framework to provide overall direction for
DOD's advertising programs. The framework, with associated outcome
measures, would allow the office to monitor advertising results
regularly and make fact-based decisions at a strategic level. It would
provide an overarching structure within which each service would
develop its own advertising program strategy, program objectives, and
outcome measures. The framework would also direct the activities of the
DOD joint program to ensure support to the services. DOD also commented
that current research has not advanced to the point where models exist
that adequately account for the many factors that affect recruiting as
well as for the differences in the services. DOD stated that it will
address this research gap through several initiatives intended to
advance the measurement of the performance of recruiting and
advertising. The National Academy of Sciences is currently developing
an evaluation framework for recruiting and advertising and expects to
publish a report in early 2004.
DOD's comments are provided in their entirety in appendix II. DOD
officials also provided technical comments that we have incorporated as
appropriate.
We are sending copies of this report to interested congressional
committees; the Secretaries of Defense, the Army, the Navy, and the Air
Force; and the Commandant of the Marine Corps. We will send copies to
other interested parties upon request. In addition, the report will be
available at no charge on the GAO Web site at http://www.gao.gov.
Please contact me at (202) 512-5559 if you or your staffs have any
questions regarding this report. Key contributors to this report were
John Pendleton, Lori Atkinson, Nancy Benco, Kurt Burgeson, Alan
Byroade, Chris Currie, LaTonya Gist, Jim McGaughey, Charles Perdue,
Barry Shillito, and John Smale.
Derek B. Stewart
Director, Defense Capabilities and Management:
Signed by Derek B. Stewart:
[End of section]
Appendix I: Scope and Methodology:
To describe the changes in the Department of Defense's (DOD)
advertising programs and advertising funding trends since the late
1990s, we reviewed advertising exhibits in the operation and
maintenance congressional justification books as well as budget
information provided by the Office of the Secretary of Defense. Since
our objective was to look at broad funding trends, we did not reconcile
these requested amounts with actual obligations or expenditures by the
active and reserve components. We interviewed active and reserve
component officials to understand program changes since the late 1990s.
We obtained recruiting mission goals and actual accessions back to
fiscal year 1990 from the Office of the Secretary of Defense and the
services. We obtained information on the quality of accessions of each
of the active and reserve components back to fiscal year 1990, as well
as the investment per active enlisted accession back to fiscal year
1990. We reviewed information from the Defense Human Resources Activity
and the Joint Marketing and Advertising Committee for discussions of
advertising programs. The services provided additional information
regarding the types of advertising media they use.
To assess the adequacy of the measures used by DOD to evaluate the
effectiveness of advertising, we reviewed information on outcome
measures used to evaluate the effectiveness of advertising provided by
each of the active and reserve components; the advertising agencies
that are their contractors; and the DOD Joint Advertising, Market
Research, and Studies program. We spoke with the advertising
contractors to learn what measures of effectiveness they are aware of
and use. We also reviewed the requirements for establishing program
objectives and outcome measures in the Government Performance and
Results Act and in Office of Management and Budget guidance.
We interviewed DOD and advertising officials from each of the active
and reserve components, as well as representatives from the services'
advertising agencies. We also reviewed their programs, procedures, and
oversight activities. These interviews were conducted with officials in
the Office of the Under Secretary of Defense for Personnel and
Readiness; Office of the Under Secretary of Defense (Comptroller/Chief
Financial Officer); Defense Human Resources Activity, Joint
Advertising, Market Research, and Studies Office; Army Accessions
Command, Fort Knox, Kentucky; Air Force Recruiting Service, Randolph
Air Force Base, Texas; Navy Recruiting Command, Millington, Tennessee;
Marine Corps Recruiting Command, Quantico Marine Corps Base, Virginia;
Army National Guard Recruiting and Retention Command, Arlington,
Virginia; Naval Reserve Command, New Orleans, Louisiana; Air Force
Reserve Command, Robins Air Force Base, Georgia; and the Air National
Guard Office of Recruiting and Retention, Arlington, Virginia. We also
interviewed officials at the contracted advertising agencies for the
joint program, the Army, the Navy, the Marine Corps, and the Air Force.
We reviewed reports on recruiting and advertising from DOD, the
Congressional Research Service, the private sector, and others. We
obtained recruiting advertising budget and funding data for types of
advertising from the Office of the Secretary of Defense. We reviewed,
but did not verify, the accuracy of the data provided by DOD.
We conducted our review from October 2002 through July 2003 in
accordance with generally accepted government auditing standards.
[End of section]
Appendix II: Comments from the Department of Defense:
Note: Page numbers in the draft report may differ from those in this
report.
PERSONNEL AND READINESS:
OFFICE OF THE UNDER SECRETARY OF DEFENSE:
4000 DEFENSE PENTAGON WASHINGTON, D.C. 20301-4000:
SEP 2 2003:
Mr. Derek B. Stewart:
Director, Defense Capabilities and Management U.S. General Accounting
Office:
441 G. Street, N.W. Washington, DC 20548:
Dear Mr. Stewart:
This is the Department of Defense (DoD) response to the GAO draft
report, "MILITARY RECRUITING: DoD Needs to Establish Objectives and
Measures to Better Evaluate Advertising's Effectiveness," dated August
1, 2003 (GAO Code 350274/GAO-03-1005). We appreciate the opportunity to
comment.
The Department concurs with the report's recommendations that the Under
Secretary of Defense for Personnel and Readiness set clear measurable
objectives, develop outcome measures, and use those measures to monitor
advertising
program performance and make fact-based choices about advertising
funding. Of course, at the Service (both Active and Reserve component)
level, objectives and outcome measures already exist.
The Office of the Under Secretary of Defense for Personnel and
Readiness (OUSD-PR), in concert with the Services, will develop a DoD
Advertising Strategic Framework. The framework, with associated outcome
measures, will allow OUSD-PR to monitor advertising results and make
fact-based decisions at a strategic level for the Department and the
Joint Advertising, Market Research, and Studies Program. This will
provide an overarching structure within which each Service will
continue to develop its own advertising program strategy, program
objectives, and outcome measures.
Outcome measures based simply upon advertising awareness and
propensity, as suggested by the report, must be considered as an
interim measure. There is danger in attempting to measure advertising
program performance out of context. However, current research has not
advanced to the point where models exist that adequately account for
the many factors that impact recruiting as well as for Service
differences simultaneously. OUSD-PR will address this research gap
through several initiatives in FY 2004, as well as with follow-on work
derived from those efforts.
The enclosure contains detailed Departmental comments on each of the
three recommendations identified by the GAO. The Department appreciates
the opportunity to comment on the draft report.
Sincerely,
Charles S. Abell
Principal Deputy:
SIGNED FOR: Charles S. Abell:
Enclosures: As stated:
GAO DRAFT REPORT - DATED AUGUST 1, 2003 GAO CODE 350274/GAO-03-1005:
"MILITARY RECRUITING: DoD Needs to Establish Objectives and Measures to
Better Evaluate Advertising Effectiveness":
DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS:
RECOMMENDATION 1: The GAO recommended that the Secretary of the Defense
direct the Under Secretary of Defense for Personnel and Readiness to
issue guidance that would set clear measurable objectives for the
advertising programs on its recruiting mission. (Page 22/GAO Draft
Report):
DOD RESPONSE: Concur.
DoD agrees, in principle, with the GAO's interest in evaluative
criteria and fact-based decisions on advertising funding.
The GAO notes in the body of the report that measuring advertising
effectiveness is a challenge for all businesses, let alone for DoD with
its unique set of characteristics (e.g., impact on recruiting of
employment and educational opportunities, as well as the fact that
entering military service is a profound life decision, not merely the
sale of a product). In addition to the special challenges that face DoD
mentioned in the report, there is considerable difficulty in attempting
to apply "one size fits all" evaluative standards to DoD advertising.
The number of new recruits required by each Service (both Active and
Reserve component) varies, as does the attractiveness of Service career
offerings, the absolute size of total Service annual recruiting
investments, and their allocations to recruiters, bonuses, educational
benefits, and advertising programs. DoD-level objectives must
necessarily be broad in nature in order to allow each Service to tailor
its strategic and tactical efforts to its particular needs and
strengths. Given that the report fails to make mention of the core
objectives set by each Service's advertising program which were
provided to the GAO, DoD assumes that the GAO's recommendation is a
call for broad, over-arching DoD-level objectives.
The Office of the Under Secretary of Defense for Personnel and
Readiness, in concert with the Services, will develop a DoD Advertising
Strategic Framework. This framework will be used by DoD to chart the
overall direction of departmental advertising efforts, to direct the
activities of the Joint Advertising, Market Research, and Studies
Program to ensure support to the Services, and to provide a framework
for the Services to use in constructing their strategy and tactical
efforts.
RECOMMENDATION 2: The GAO recommended that the Secretary of the Defense
direct the Under Secretary of Defense for Personnel and Readiness to
develop outcome measures for each of DoD's advertising programs that
clearly link advertising program performance with those objectives.
(Page 22/GAO Draft Report):
DOD RESPONSE: Concur.
DoD-level outcome measures will follow from the DoD Advertising
Strategic Framework. Of course, at the Service level, outcome measures
already exist. Some examples include advertising awareness, propensity-
related measures, advertising lead generation, and lead-to-contract
conversion.
Currently, several tools are available that support development of
outcome measures for DoD-level objectives. The Joint Advertising,
Market Research, and Studies Program has conducted an advertising
tracking study since 2001. As mentioned in the report, the tracking
study has recently been revised in an attempt to better meet Service
advertising program needs. In addition, and also mentioned in the
report, the Joint Advertising, Market Research, and Studies Program
conducts a Youth Survey that measures propensity. Prior to this quick
turn-around poll, the Department conducted an annual youth attitude
tracking study dating back to 1974. While funding issues in the Joint
Advertising, Market Research, and Studies Program have hampered
consistent collection of these data, the program is currently
conducting both studies.
Outcome measures based simply upon advertising awareness and propensity
must be considered as an interim measure. Given the interaction effect
that exists between the many factors that impact recruiting, of which
advertising is only one, there is a danger in attempting to measure
advertising program performance out of context. Current research has
not advanced to the point where models exist that adequately account
for these factors as well as for Service differences simultaneously.
The Office of the Under Secretary of Defense for Personnel and
Readiness will address this research gap through several initiatives.
The National Academy of Sciences Committee on the Youth Population and
Military Recruitment is currently developing an evaluation framework
for advertising and recruiting. Expected publication date of the report
is early 2004. In addition, research will be funded in FY 2004 to
identify data elements necessary to support empirical research on
advertising effectiveness, with follow-on work planned to conduct
quantitative analysis to develop advertising effectiveness models that
would support decision-making at the macro (Office of the Under
Secretary of Defense for Personnel and Readiness) and the micro
(Service) levels. Specifically, more research must be undertaken on the
determinants of propensity in general, and on advertising's
contribution to propensity - both DoD and Service-specific.
RECOMMENDATION 3: The GAO recommended that the Secretary of the Defense
direct the Under Secretary of Defense for Personnel and Readiness to
use the above:
outcome measures to monitor advertising program performance and make
fact-based choices about advertising funding as part of the overall
recruiting investment in the future. (Page 22/GAO Draft Report):
DOD RESPONSE: Concur.
The DoD Advertising Strategic Framework, with associated outcome
measures, will allow the Office of the Under Secretary of Defense for
Personnel and Readiness to monitor advertising results regularly and
make fact-based decisions at a strategic level for the Department and
the Joint Advertising, Market Research, and Studies Program. This will
provide an oversight framework within which each Service will continue
to develop its own advertising program strategy, program objectives,
and outcome measures. Each Service will continue to vary the mix of
support elements to improve effectiveness based on its particular
strategy, thus also continuing to be responsible for making those
decisions, and to be accountable for demonstrating program results--
effectiveness and efficiency. Meetings of the Joint Marketing and
Advertising Committee will continue to serve as a forum for information
sharing among the Service advertising programs.
[End of section]
Related GAO Products:
Program Evaluation: Strategies for Assessing How Information
Dissemination Contributes to Agency Goals. GAO-02-923. Washington,
D.C.: September 30, 2002.
Military Personnel: Services Need to Assess Efforts to Meet Recruiting
Goals and Cut Attrition. GAO/NSIAD-00-146. Washington, D.C.: June 23,
2000.
Military Personnel: First-Term Recruiting and Attrition Continue to
Require Focused Attention. GAO/T-NSIAD-00-102. Washington, D.C.:
February 24, 2000.
Military Recruiting: DOD Could Improve Its Recruiter Selection and
Incentive Systems. GAO/NSIAD-98-58. Washington, D.C.: January 30, 1998.
Military Personnel: High Aggregate Personnel Levels Maintained
Throughout Drawdown. GAO/NSIAD-95-97. Washington, D.C.: June 2, 1995.
Military Recruiting: More Innovative Approaches Needed. GAO/NSIAD-95-
22. Washington, D.C.: December 22, 1994.
Military Downsizing: Balancing Accessions and Losses Is Key to Shaping
the Future Force. GAO/NSIAD-93-241. Washington, D.C.: September 30,
1993.
FOOTNOTES
[1] S. Rep. 107-151, at 300 (2002).
[2] Pub. L. No. 103-62, Aug. 3, 1993.
[3] All dollars are in constant fiscal year 2003 dollars unless
otherwise indicated.
[4] U.S. General Accounting Office, Military Personnel: Services Need
to Assess Efforts to Meet Recruiting Goals and Cut Attrition, GAO/
NSIAD-00-146 (Washington, D.C.: June 23, 2000).
[5] J.G. Bachman, L.D. Johnston, and P.M. O'Malley, Monitoring the
Future: Questionnaire Responses from the Nation's High School Seniors
(Ann Arbor, Mich.: Institute for Social Research, 2001).
[6] Office of the Under Secretary of Defense for Personnel and
Readiness, Military Personnel Human Resources Strategic Plan
(Washington, D.C.: Aug. 2002).
[7] The Navy and Naval Reserve have separate advertising programs;
however, their recruiting programs were recently reorganized under one
commander.
[8] These amounts are in fiscal year 2003 constant dollars using DOD's
Operation and Maintenance funding deflators. In nominal dollars, DOD's
advertising funding increased from $271 million in fiscal year 1998 to
$592 million in fiscal year 2003--an increase of 118 percent.
[9] "Quantity" is the total number of recruits needed in a given fiscal
year. "Quality" is the achievement of high school diplomas and adequate
scores on the Armed Forces Qualification Test.
[10] Office of the Under Secretary of Defense (Force Management
Policy), A New Focus for Military Advertising and Market Research
(Washington, D.C.: March 2000) and Rand Corporation, A Report on the
Audit of the Armed Services Recruitment Advertising (Santa Monica,
Calif.: 2002).
[11] In nominal dollars, DOD's total recruiting advertising funding for
fiscal years 1998-2002, consecutively, was $271 million, $379 million,
$450 million, $501 million, and $595 million.
[12] The Blue Angels, the Navy's flight demonstration team, performs at
air shows and special events. The recruiting advertising budget funds
the operation and maintenance costs related to the team.
[13] 10 U.S.C. § 503.
[14] DOD Instruction 1304.8, Military Procurement Resources Report, May
28, 1991.
[15] DOD Instruction 7730.56, Monthly Report of Personnel Statistics,
September 15, 1975.
[16] Pub. L. No. 103-62, Aug. 3, 1993.
[17] Office of Management and Budget, Preparation and Submission of
Strategic Plans, Annual Performance Plans, and Annual Program
Performance Reports, Circular No. A-11, Part 6, June 2002.
[18] GAO/NSIAD-00-146.
[19] Office of the Assistant Secretary of Defense (Force Management
Policy), A New Focus for Military Advertising and Market Research
(March 2000).
[20] National Research Council, Attitudes, Aptitudes, and Aspirations
of American Youth: Implications for Military Recruitment (Washington,
D.C.: The National Academies Press, 2003).
[21] DOD Instruction 1304.8, Military Procurement Resources Report, May
28, 1991; and DOD Instruction 7730.56, Monthly Report of Personnel
Statistics, September 15, 1975.
[22] The advertising agency contracted by DOD's joint program also
identified several problems, including (1) having an extremely large
sample size, (2) voluminous data but poor analysis, and (3) poor
training for the services in using the data.
[23] John Warner, Curtis Simon, and Deborah Payne, Enlistment Supply in
the 1990s: A Study of the Navy College Fund and Other Enlistment
Incentive Programs, Defense Manpower Data Center, report No. 2000-015
(April 2001), p. 45.
[24] Jim Dertouzos and Steven Garber, Is Military Advertising
Effective? An Estimation Methodology and Applications to Recruiting in
the 1980s and 90s (Santa Monica, Calif.: Rand Corporation, 2003).
GAO's Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548:
