Information Security
Concerted Effort Needed to Improve Federal Performance Measures
Gao ID: GAO-10-159T October 29, 2009
Cyber security is a critical consideration for any organization that depends on information systems and computer networks to carry out its mission or business. Organizations are faced with a variety of information security threats, such as fraudulent activity from cyber criminals, unauthorized access by disgruntled or dishonest employees, and denial-of-service attacks and other disruptions. The recent dramatic increase in reports of security incidents, the wide availability of hacking tools, and steady advances in the sophistication and effectiveness of attack technology all contribute to the urgency of ensuring that adequate steps are taken to protect the federal government's information and the systems that contain and process it. The Federal Information Security Management Act (FISMA), which was enacted in 2002, sets forth a comprehensive framework for ensuring the effectiveness of security controls over information resources that support federal operations and assets. The act assigns specific responsibilities to federal agencies, the Office of Management and Budget (OMB), and the National Institute of Standards and Technology (NIST). It also requires agencies and OMB to annually report on the adequacy and effectiveness of agency information security programs and compliance with the provisions of the act. To help meet these requirements, OMB established a uniform set of information security measures that all federal agencies report on annually.
Leading organizations and experts have identified different types of measures that are useful in helping to achieve information security goals. While officials categorized these types using varying terminology, we concluded that they generally fell into three types: (1) compliance, (2) control effectiveness, and (3) program impact. These types are consistent with those laid out by NIST in its information security performance measurement guide. In addition, while information security measures can be grouped into these three major types, organizations and experts reported that all such measures generally have certain key characteristics, or attributes. These attributes include being (1) measurable, (2) meaningful, (3) repeatable and consistent, and (4) actionable. Further, these organizations and experts indicated that the successful development of information security measures depends on adherence to a number of key practices, including focusing on risks, involving stakeholders, assigning accountability, and linking to business goals. Additional practices are critical to ensuring that the measures are useful in effectively conveying information to operational managers, executives, and oversight officials. These include tailoring measures to the audience; correlating data; and capturing progress, trends, and weaknesses. We determined that federal agencies have not always followed key practices identified by leading organizations for developing information security performance measures. While agencies have developed measures that fall into each of the three major types (i.e. compliance, control effectiveness, and program impact), on balance they have relied primarily on compliance measures, which have a limited ability to gauge program effectiveness. Agencies stated that, for the most part, they predominantly collected measures of compliance because they were focused on measures associated with OMB's FISMA reporting requirements. In addition, while most agencies have developed some measures that include the four key attributes identified by leading organizations and experts, these attributes were not always present in all agency measures. Further, agencies have not always followed key practices in developing measures, such as focusing on risks.
GAO-10-159T, Information Security: Concerted Effort Needed to Improve Federal Performance Measures
This is the accessible text file for GAO report number GAO-10-159T
entitled 'Information Security: Concerted Effort Needed to Improve
Federal Performance Measures' which was released on October 29, 2009.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Mr. Chairman and Members of the Subcommittee:
Thank you for the opportunity to participate in today‘s hearing on how
agencies can establish cost-effective cyber defense. My statement today
is based on our report titled Information Security: Concerted Effort
Needed to Improve Federal Performance Measures, which is being released
at this hearing.[Footnote 1]
Cyber security is a critical consideration for any organization that
depends on information systems and computer networks to carry out its
mission or business. Organizations are faced with a variety of
information security threats, such as fraudulent activity from cyber
criminals, unauthorized access by disgruntled or dishonest employees,
and denial-of-service attacks and other disruptions. The recent
dramatic increase in reports of security incidents, the wide
availability of hacking tools, and steady advances in the
sophistication and effectiveness of attack technology all contribute to
the urgency of ensuring that adequate steps are taken to protect the
federal government‘s information and the systems that contain and
process it.
The Federal Information Security Management Act (FISMA), which was
enacted in 2002, sets forth a comprehensive framework for ensuring the
effectiveness of security controls over information resources that
support federal operations and assets. The act assigns specific
responsibilities to federal agencies, the Office of Management and
Budget (OMB), and the National Institute of Standards and Technology
(NIST). It also requires agencies and OMB to annually report on the
adequacy and effectiveness of agency information security programs and
compliance with the provisions of the act. To help meet these
requirements, OMB established a uniform set of information security
measures that all federal agencies report on annually.
Mr. Chairman, in light of questions about whether or not agencies are
measuring the right things in securing their systems, you requested
that GAO examine how organizations develop and use metrics to assess
the performance and effectiveness of information security activities.
In response to your request, our report and my statement provide (1) a
description of key types and attributes of performance measures; (2)
information about the practices of leading organizations for developing
and using measures to guide and monitor information security control
activities;[Footnote 2] (3) information on the measures used by federal
agencies to guide and monitor information security control activities
and how they are developed; and (4) an assessment of the effectiveness
of the measures-reporting practices that the federal government uses to
inform Congress on the effectiveness of information security programs.
In conducting this work, we collected and analyzed information from
leading organizations, security experts, NIST, 24 major federal
agencies, and OMB.[Footnote 3] Our work for this report was performed
in accordance with generally accepted government auditing standards.
In brief, Mr. Chairman, leading organizations and experts have
identified different types of measures that are useful in helping to
achieve information security goals. While officials categorized these
types using varying terminology, we concluded that they generally fell
into three types: (1) compliance, (2) control effectiveness, and (3)
program impact. These types are consistent with those laid out by NIST
in its information security performance measurement guide.[Footnote 4]
In addition, while information security measures can be grouped into
these three major types, organizations and experts reported that all
such measures generally have certain key characteristics, or
attributes. These attributes include being (1) measurable, (2)
meaningful, (3) repeatable and consistent, and (4) actionable.[Footnote
5]
Further, these organizations and experts indicated that the successful
development of information security measures depends on adherence to a
number of key practices, including focusing on risks, involving
stakeholders, assigning accountability, and linking to business goals.
Additional practices are critical to ensuring that the measures are
useful in effectively conveying information to operational managers,
executives, and oversight officials. These include tailoring measures
to the audience; correlating data; and capturing progress, trends, and
weaknesses. Figure 1 illustrates the interrelationship of these key
practices with the key characteristics.
Figure 1: Measures Development and Use Cycle:
[Refer to PDF for image: circle chart]
Development:
* Focus on risks;
* Involve stakeholders;
* Assign accountability;
* Link to business goals;
Measures characteristics:
Types:
Compliance;
Control effectiveness;
Program impact;
Attributes:
Meaningful;
Measurable;
Repeatable;
Actionable;
Effective reporting:
* Tailor to the audience;
* Correlate data;
* Capture progress, trends, and weaknesses;
Source: GAO.
[End of figure]
We determined that federal agencies have not always followed key
practices identified by leading organizations for developing
information security performance measures. While agencies have
developed measures that fall into each of the three major types (i.e.
compliance, control effectiveness, and program impact), on balance they
have relied primarily on compliance measures, which have a limited
ability to gauge program effectiveness. Agencies stated that, for the
most part, they predominantly collected measures of compliance because
they were focused on measures associated with OMB‘s FISMA reporting
requirements. In addition, while most agencies have developed some
measures that include the four key attributes identified by leading
organizations and experts, these attributes were not always present in
all agency measures. Further, agencies have not always followed key
practices in developing measures, such as focusing on risks.
Lastly, we determined that OMB‘s measures did not address the
effectiveness of several key areas of information security controls,
including, for example, agency security control testing and evaluation
processes. There is no measure of the quality of agencies‘ test and
evaluation processes or results that demonstrate the effectiveness of
the controls that were evaluated.[Footnote 6] In addition, OMB‘s report
to Congress does not fully employ key practices for reporting and thus
provides limited information about the effectiveness of agency
information security programs.
We made five recommendations to OMB to assist federal agencies in
developing and using measures that better address the effectiveness of
their information security programs:
* issue revised guidance to chief information officers for developing
measures;
* direct chief information officers to ensure that measures exhibit key
attributes;
* direct chief information officers to employ the key practices for
developing a measure as identified by leading organizations;
* revise annual FISMA reporting guidance to agencies; and:
* revise the annual FISMA report to Congress to provide better status
information on the security posture of the federal government.
Implementing these recommendations will help to focus attention on
activities that will enhance the effectiveness of agency information
security controls and improve the cyber defense of federal computer
systems and information. In providing oral comments on a draft of the
report, representatives of OMB's Office of E-Government and Information
Technology stated that they generally agreed with the contents and
recommendations of the report.
Mr. Chairman, this concludes my prepared statement. I would be pleased
to respond to any questions that you or other members of the
subcommittee may have.
For questions about this statement, please contact Gregory C. Wilshusen
at (202) 512-6244 or wilshuseng@gao.gov. Individuals making key
contributors to this testimony include Ashley Brooks, John de Ferrari,
Season Dietrich, Neil Doherty, Ronalynn Espedido, Min Hyun, Anjalique
Lawrence, Joshua Leiling, Lee McCracken, and David Plocher.
[End of section]
Footnotes:
[1] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-09-617]
(Washington, D.C.: Sept. 14, 2009).
[2] For the purposes of this report, ’leading organizations“ refers to
prominent, nationally known organizations, academic institutions, and
state agencies that have implemented comprehensive enterprisewide
information security programs.
[3] The 24 major federal agencies are the Agency for International
Development; the Departments of Agriculture, Commerce, Defense,
Education, Energy, Health and Human Services, Homeland Security,
Housing and Urban Development, the Interior, Justice, Labor, State,
Transportation, the Treasury, and Veterans Affairs; the Environmental
Protection Agency; the General Services Administration; the National
Aeronautics and Space Administration; the National Science Foundation;
the Nuclear Regulatory Commission; the Office of Personnel Management;
the Small Business Administration; and the Social Security
Administration.
[4] National Institute of Standards and Technology, Performance
Measurement Guide for Information Security, NIST Special Pub. 800-55
Revision 1 (Gaithersburg, Md.: July 2008).
[5] Although we focused on identifying attributes and practices for
measuring the performance of information security programs, our
findings conformed closely to our prior work on effective performance
measurement and reporting practices for the federal government in
general. See, for example, GAO, Managing for Results: Enhancing Agency
Use of Performance Information for Management Decision Making,
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05-927] (Washington,
D.C.: Sept. 9, 2005).
[6] OMB does require agency inspectors general to assess agencies‘
certification and accreditation process; however, the assessment may or
may not include an assessment of security control testing and
evaluation processes. Further, OMB does not provide a transparent
depiction of how an assessment of an agency‘s security control testing
and evaluation process contributes to the overall certification and
accreditation quality rating.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
