Information Assurance
National Partnership Offers Benefits, but Faces Considerable Challenges Gao ID: GAO-06-392 March 24, 2006In 1997, the National Security Agency and the National Institute of Standards and Technology formed the National Information Assurance Partnership (NIAP) to boost federal agencies' and consumers' confidence in information security products manufactured by vendors. To facilitate this goal, NIAP developed a national program that requires accredited laboratories to independently evaluate and validate the security of these products for use in national security systems. These systems are those under control of the U.S. government that contain classified information or involve intelligence activities. GAO was asked to identify (1) the governmentwide benefits and challenges of the NIAP evaluation process on national security systems, and (2) the potential benefits and challenges of expanding the requirement of NIAP to non-national security systems, including sensitive but unclassified systems.
While NIAP process participants--vendors, laboratories, and federal agencies--indicated that the process offers benefits for use in national security systems, its effectiveness has not been measured or documented, and considerable challenges to acquiring and using NIAP-evaluated products exist. Specific benefits included independent testing and evaluation of products and accreditation of the performing laboratories, the discovery and correction of product flaws, and improvements to vendor development processes. However, process participants also face several challenges, including difficulty in matching agencies' needs with the availability of NIAP-evaluated products, vendors' lack of awareness regarding the evaluation process, and a lack of performance measures and difficulty in documenting the effectiveness of the NIAP evaluation process. Collectively, these challenges hinder the effective use of the NIAP evaluation process by vendors and agencies. Expanding the requirement of the NIAP evaluation process to non-national security systems is likely to yield similar benefits and challenges as those experienced by current process participants. For example, a current benefit--independent testing and evaluation of IT products--gives agencies confidence that validated features of a product will perform as claimed by the vendor. However, federal policy already allows agencies with non-national security systems to consider acquiring NIAP-evaluated products for those systems, and requiring that they do so may further exacerbate current resource constraints related to the evaluation and validation of products. In the absence of such a requirement, agencies seeking information assurance (measures that defend and protect information and information systems by ensuring their confidentiality, integrity, authenticity, availability, and utility) for their non-national security systems have other federal guidance and standards available to them.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-06-392, Information Assurance: National Partnership Offers Benefits, but Faces Considerable Challenges
This is the accessible text file for GAO report number GAO-06-392
entitled 'Information Assurance: National Partnership Offers Benefits,
but Faces Considerable Challenges' which was released on March 27,
2006.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Honorable William Lacy Clay, House of Representatives:
March 2006:
Information Assurance:
National Partnership Offers Benefits, but Faces Considerable
Challenges:
GAO-06-392:
GAO Highlights:
Highlights of GAO-06-392, a report to the Honorable William Lacy Clay,
House of Representatives:
Why GAO Did This Study:
In 1997, the National Security Agency and the National Institute of
Standards and Technology formed the National Information Assurance
Partnership (NIAP) to boost federal agencies‘ and consumers‘ confidence
in information security products manufactured by vendors. To facilitate
this goal, NIAP developed a national program that requires accredited
laboratories to independently evaluate and validate the security of
these products for use in national security systems. These systems are
those under control of the U.S. government that contain classified
information or involve intelligence activities.
GAO was asked to identify (1) the governmentwide benefits and
challenges of the NIAP evaluation process on national security systems,
and (2) the potential benefits and challenges of expanding the
requirement of NIAP to non-national security systems, including
sensitive but unclassified systems.
What GAO Found:
While NIAP process participants”vendors, laboratories, and federal
agencies”indicated that the process (see figure below) offers benefits
for use in national security systems, its effectiveness has not been
measured or documented, and considerable challenges to acquiring and
using NIAP-evaluated products exist. Specific benefits included
independent testing and evaluation of products and accreditation of the
performing laboratories, the discovery and correction of product flaws,
and improvements to vendor development processes. However, process
participants also face several challenges, including difficulty in
matching agencies‘ needs with the availability of NIAP-evaluated
products, vendors‘ lack of awareness regarding the evaluation process,
and a lack of performance measures and difficulty in documenting the
effectiveness of the NIAP evaluation process. Collectively, these
challenges hinder the effective use of the NIAP evaluation process by
vendors and agencies.
Simplified Overview of NIAP Evaluation Process:
[See PDF for image]
[End of figure]
Expanding the requirement of the NIAP evaluation process to non
national security systems is likely to yield similar benefits and
challenges as those experienced by current process participants. For
example, a current benefit”independent testing and evaluation of IT
products”gives agencies confidence that validated features of a product
will perform as claimed by the vendor. However, federal policy already
allows agencies with non-national security systems to consider
acquiring NIAP-evaluated products for those systems, and requiring that
they do so may further exacerbate current resource constraints related
to the evaluation and validation of products. In the absence of such a
requirement, agencies seeking information assurance (measures that
defend and protect information and information systems by ensuring
their confidentiality, integrity, authenticity, availability, and
utility) for their non-national security systems have other federal
guidance and standards available to them.
What GAO Recommends:
GAO is making two recommendations to address challenges with the NIAP
evaluation process, including establishing and documenting performance
measures on process effectiveness. The Department of Defense concurred
with one of our recommendations and partially concurred with the other.
www.gao.gov/cgi-bin/getrpt?GAO-06-392. To view the full product,
including the scope and methodology, click on the link above. For more
information, contact Gregory C. Wilshusen at (202) 512-6244 or
wilshuseng@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
NIAP Offers Benefits for Use in National Security Systems, but Process
Faces Considerable Challenges:
Expanding NIAP Requirement to Non-national Security Systems May Yield
Many of the Same Benefits and Challenges and Could Exacerbate Resource
Constraints:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendixes:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Common Criteria Evaluation Assurance Levels:
Appendix III: Comments from the Department of Defense:
Appendix IV: GAO Contact and Staff Acknowledgments:
Table:
Table 1: Summary of the Common Criteria Evaluation Assurance Levels:
Figures:
Figure 1: The NIAP Evaluation Process:
Figure 2: Range of Sample Cost of NIAP Evaluations to Vendors by
Evaluation Assurance Level:
Figure 3: Laboratory Accreditation Process:
Figure 4: Range of Time Required for Completing Product Evaluations at
Various Evaluation Assurance Levels:
Abbreviations:
IT: information technology:
NIAP: National Information Assurance Partnership:
NIST: National Institute of Standards and Technology:
NVLAP: National Voluntary Laboratory Accreditation Program:
Letter March 24, 2006:
The Honorable William Lacy Clay:
House of Representatives:
Dear Mr. Clay:
The sophistication and effectiveness of cybersecurity attacks have
advanced steadily over the past few years and have drastically changed
the way we think about protecting our information and information
systems, including national security systems.[Footnote 1] Commercial-
off-the-shelf information assurance products and information assurance-
enabled products or technologies[Footnote 2] are readily available to
agencies as well as consumers providing needed security services.
Acquiring such products for use on national security systems that
perform as claimed by the vendors who manufacture these products is a
governmentwide challenge.
In 1997, the National Information Assurance Partnership (NIAP) was
formed by the National Security Agency and the National Institute of
Standards and Technology (NIST) to boost federal agencies' and
consumers' confidence in commercial-off-the-shelf products. To
facilitate this goal, NIAP developed a national program that uses
accredited laboratories to independently evaluate and validate the
security of vendor products using standardized processes. The NIAP
program allows the U.S. to meet federal agency needs by participating
in an international arrangement to validate security products using
standardized processes. In addition, the Committee on National Security
Systems established a federal policy which mandates, among other
things, the use of NIAP-evaluated products for national security
systems. Further, the policy allows but does not require users of non-
national security systems to acquire NIAP-evaluated products.
Our objectives were to identify (1) the governmentwide benefits and
challenges of the NIAP evaluation process for national security systems
and (2) the potential benefits and challenges of expanding the
requirement of using NIAP-evaluated products for non-national security
systems, including sensitive but unclassified systems. To address these
objectives, we obtained perspectives from selected industry groups and
various NIAP process participants, such as vendors, accredited
laboratories, and government officials. We also developed and submitted
a survey questionnaire to the 24 federal agencies--also process
participants--cited in the Chief Financial Officer's Act of 1990
(Public Law 101-576) to determine their use of, and perspectives on,
NIAP-evaluated products. In addition, we analyzed documentation related
to NIAP evaluation and validation policies and processes, test and
evaluation criteria, and laboratory accreditation processes. We
conducted our work in Washington, D.C., from May 2005 through February
2006, in accordance with generally accepted government auditing
standards. See appendix I for more details about our objectives, scope,
and methodology.
Results in Brief:
While the NIAP process offers benefits to national security systems,
its effectiveness has not been measured or documented, and considerable
challenges to acquiring and using NIAP-evaluated products exist.
Specific benefits include:
* independent testing and evaluation of products and accreditation of
the performing laboratories, which can increase agencies' confidence
that products will perform as claimed;
* the ability to participate in an international arrangement of
recognized products, which gives agencies broader product selection and
reduces vendor workload;
* the discovery and correction of product flaws, which help to give
agencies greater confidence that the product will perform as claimed;
and:
* improvements to vendor development processes, which can result in
quality improvements to current and future products.
However, the NIAP process also faces several challenges, including:
* difficulty in matching agencies' needs with the availability of NIAP-
evaluated products;
* vendors' lack of awareness regarding the evaluation process;
* a reduction in the number of validators to certify products; and:
* a lack of performance measures and difficulty in documenting the
effectiveness of the NIAP process.
Collectively, these challenges hinder the effective use of the NIAP
process by vendors and agencies.
Expanding the requirement of the NIAP evaluation process to non-
national security systems is likely to yield the same benefits and
challenges as those experienced by current process participants. While
federal policy allows agencies with non-national security systems to
consider using the NIAP process to acquire evaluated and validated
products, requiring that they do so may further exacerbate current
resource constraints related to the evaluation and validation of
products. In the absence of such a requirement, agencies seeking
information assurance for their non-national security systems have
other federal guidance and standards available to them.
We are making recommendations to assist NIAP officials in addressing
process challenges, including developing awareness training workshops
for program participants and establishing and documenting performance
measures on process effectiveness.
In providing written comments on a draft of this report, the Deputy
Assistant Secretary of Defense (Deputy Chief Information Officer)
partially agreed with one of our recommendations, agreed with the
other, and described ongoing and planned efforts to address them. The
Deputy Assistant Secretary's comments are reprinted in appendix III.
The Department of Defense and the Department of Homeland Security also
provided technical comments, which we considered and addressed in our
report, as appropriate.
Background:
The growing sophistication and effectiveness of cyber attacks, and the
increase of information assurance and information assurance-enabled
information technology (IT) products available for use on national
security systems, have heightened federal attention to the need for
information assurance.[Footnote 3] As a result of these trends,
acquiring commercial IT products that perform as vendors claim on
national security systems has become a governmentwide challenge. While
not a complete solution, an important way to increase confidence in
commercial IT products is through independent testing and evaluation of
their security features and functions during design and development.
Federal Partnership Formed to Promote the Use of Evaluated IT Products:
In 1997, NIST and the National Security Agency[Footnote 4] collaborated
to form the NIAP. The purpose of the partnership is to boost consumers'
and federal agencies' confidence in information security products and
enhance the ability of U.S. companies to gain international recognition
and acceptance for their products. The five main goals of NIAP are to:
* promote the development and use of evaluated IT products and systems;
* champion the development and use of national and international
standards for IT security;
* foster research and development in IT security requirements
definition, test methods, tools, techniques, and assurance metrics;
* support a framework for international recognition and acceptance of
IT security testing and evaluations; and:
* facilitate development and growth of a commercial security testing
industry within the U.S.
To facilitate achievement of these goals, NIAP developed a national
program called the Common Criteria Evaluation and Validation Scheme.
The program is based on an international standard of general concepts
and principles of IT security evaluations[Footnote 5] for the
international community. The program evaluates, through various
evaluation assurance levels (see app. II),[Footnote 6] commercial-off-
the-shelf information assurance and information assurance-enabled
products for the federal government. These products can be items of
hardware, software, or firmware.[Footnote 7] As part of the evaluation,
agencies can specify a degree of confidence desired in a product
through protection profiles.[Footnote 8] While a protection profile is
not required in order to have a product evaluated, a vendor is required
to develop a security target.[Footnote 9]
NIAP evaluations are performed by accredited Common Criteria testing
laboratories.[Footnote 10] While a product is undergoing evaluation,
the NIAP validation body--an activity currently managed by the National
Security Agency--approves participation of security testing
laboratories in accordance with accreditation policies and
procedures.[Footnote 11] It also reviews the results of the security
evaluations performed by the laboratories and issues a validation
report, which summarizes and provides independent validation of the
results. A product is considered NIAP-certified only after it is both
evaluated by an accredited laboratory and validated by the validation
body. Upon successful completion of these requirements, the validation
body issues a Common Criteria certificate for the evaluated product.
All evaluated products that receive a NIAP Common Criteria certificate
appear on a validated products list available on NIAP's Web site.
According to the Committee on National Security Systems[Footnote 12]--
a forum for the discussion of policy issues that sets federal policy
and promulgates direction, operational procedures, and guidance for the
security of national security systems--the fact that a product appears
on the validated products list does not by itself mean that it is
secure. A product's listing on any Common Criteria validated products
list means that the product was evaluated against its security claims
and that it has met those claims.[Footnote 13] Figure 1 outlines the
NIAP evaluation process.
Figure 1: The NIAP Evaluation Process:
[See PDF for image]
[End of figure]
In order to maintain the validity of an evaluation when a product
upgrades to its next version, a vendor can request either a re-
evaluation of the entire new product version or validation of only the
changes in the product. To request the latter, a vendor must
participate in the NIAP Assurance Maintenance Program. To participate
in this program, a vendor must submit a request that addresses how it
plans to maintain the product and a report of what will be maintained.
Vendors can select any one of the 10 accredited commercial testing
laboratories to perform product evaluations.[Footnote 14] The vendor
and testing laboratory negotiate evaluation costs, which can vary
according to the laboratory and the assurance level the product is
tested against[Footnote 15] (see fig. 2).
Figure 2: Range of Sample Cost of NIAP Evaluations to Vendors by
Evaluation Assurance Level:
[See PDF for image]
[End of figure]
Other factors that influence the overall cost of NIAP product
evaluations include:
* the scope of evaluation--the tendency of vendors to include elements
in their security target that agencies may not require introduces
additional costs; and:
* the design of the product--if a product is designed so that its
security functions are performed by a small number of modules, it may
be possible to limit the portion of the product that must be examined.
Federal Policy Requires Evaluated Products for National Security
Systems:
In January 2000, as revised in June 2003, a federal policy was
established that required the use of evaluated products for national
security systems. Specifically, the Committee on National Security
Systems established National Security Telecommunications and
Information Systems Security Policy Number 11.[Footnote 16] The policy
required, effective July 1, 2002, that all commercial-off-the-shelf
information assurance and information assurance-enabled IT products
acquired for use on national security systems be evaluated and
validated in accordance with one of the following criteria:
1. The International Common Criteria for Information Security
Technology Evaluation Recognition Arrangement,[Footnote 17]
2. The NIAP Common Criteria Evaluation and Validation Scheme,
3. The NIST Federal Information Processing Standards Cryptographic
Module Validation Program.[Footnote 18]
The objective of the policy is to ensure that these products, which are
acquired by the federal government, undergo a standardized evaluation
validating that a product either performs as its claims or meets the
user's security requirements. The policy requires that the evaluation
and validation of such products be conducted by accredited commercial
laboratories or by the National Security Agency for government off-the
shelf products. It does not require mandatory compliance for
information assurance products acquired prior to July 1, 2002, and
includes a provision for deferred compliance, on a case-by-case basis,
when information assurance-evaluated products do not cover the full
range of potential user application, or do not incorporate the most
current technology.
Moreover, while not a requirement, the federal policy includes
provisions for departments and agencies who may wish to consider using
the NIAP process for the acquisition and appropriate implementation of
evaluated and validated products for non-national security systems.
NIAP Evaluation Process Contributes to System Security, but Is Not a
Complete Solution:
The use of commercial products that have been independently tested and
evaluated is only a part of a security solution that contributes to the
overall information assurance of a product. Other complementary
controls are needed, including sound operating procedures, adequate
information security training, overall system certification and
accreditation,[Footnote 19] sound security policies, and well-designed
system architectures. According to the Committee on National Security
Systems, the protection of systems encompasses more than just acquiring
the right product. The committee notes that once acquired, these
products must be integrated properly and subjected to a system
accreditation process, as discussed above, which will help to ensure
the integrity of the information and systems to be protected.
For federal agencies, such an overall security solution is spelled out
by the Federal Information Security Management Act. The act requires
federal agencies to protect and maintain the confidentiality,
integrity, and availability of their information and information
systems. Among other things, the act requires each agency (including
agencies with national security systems) to develop, document, and
implement agencywide information security programs to provide
information security for the information and information systems that
support the operations and assets of the agency, including those
provided or managed by another agency, contractor, or other source.
More specifically, the Federal Information Security Management Act
stipulates that the head of each agency operating or exercising control
of a national security system is responsible for providing information
security protections commensurate with the risk and magnitude of harm
that could result should a security breach occur. The act also
stipulates that agency heads are responsible for implementing
information security policies and practices as required by standards
and guidelines for national security systems. The Department of Defense
and the Director of Central Intelligence have authority under the act
to develop policies, guidelines, and standards for national security
systems.
The Federal Information Security Management Act also requires NIST,
among other things, to provide technical assistance to agencies; to
evaluate private sector security policies and practices; to evaluate
commercially available IT, as well as practices developed for national
security systems; and to assess the potential application by agencies
to strengthen information security for non-national systems.
NIAP Offers Benefits for Use in National Security Systems, but Process
Faces Considerable Challenges:
While the NIAP evaluation process offers benefits to national security
systems, its effectiveness has not been measured or documented, and
considerable challenges to acquiring and using NIAP-evaluated products
exist.
NIAP Evaluation Process Offers Benefits:
NIAP process participants--vendors, laboratories, federal agencies, and
NIAP officials--identified benefits to using the process for use in
national security systems, including:
* independent testing and evaluation of IT products and accreditation
of the performing laboratories, which can give agencies confidence that
the products will perform as claimed;
* international recognition of evaluated products, which provides
agencies broader product selection and reduces vendor burden;
* discovery of software flaws in product security features and
functions, which can cause vendors to fix them; and:
* improvements to vendor development processes, which help to improve
the overall quality of current and future products.
Independent Testing and Evaluation of Products and Accreditation of
Laboratories Can Increase Product Assurance:
Independent testing and evaluation of commercial IT products and
accreditation of the laboratories that perform the test and evaluations
can give agencies increased assurance that the products will perform as
vendors claim. Independent testing is a best practice for assuring
conformance to functional, performance, reliability, and
interoperability specifications--especially for systems requiring
elevated levels of security or trust. As discussed previously, NIAP
requires vendors to obtain independent testing and evaluation of
specific security features and functions that are built into their
products. Agencies are able to use the results of validation reports to
distinguish between competing products and thus make better-informed IT
procurement decisions. Further, the Committee on National Security
Systems encourages agencies to review the security target of a product
and determine its appropriateness for the environment in which the
product will operate.
In our survey, 15 of 18 federal agencies[Footnote 20] reported that
they have derived benefits from acquiring and using products evaluated
by the NIAP process. Of these 15 agencies,
* 11 reported that the availability of evaluated products helped the
agency make IT procurement decisions;
* 9 reported that the process provided their agency with thorough and
accurate product documentation; and:
* 1 reported that evaluated products provided a common method of
performing a particular security service that is implemented in
different types of security or security-enabled devices, potentially
resulting in a greater degree of standardization of elements (such as
audit entries).
Moreover, the NIST-administered National Voluntary Laboratory
Accreditation Program (NVLAP) reviews laboratories annually to ensure
competence and compliance with standards. Accreditation is granted to
laboratories following their successful completion of a process that
includes an application submission and fee payment by the laboratory,
an on-site assessment, participation in proficiency testing, resolution
of any deficiencies identified during the process, and a technical
evaluation. The issuance of a certificate formally signifies that a
laboratory has demonstrated that it meets all NVLAP requirements and
operates in accordance with management and the technical requirements
of the relevant standards. However, the accreditation does not imply
any guarantee of laboratory performance or test and calibration data;
it is solely a finding of laboratory competence and compliance with
standards. Figure 3 shows the laboratory accreditation process.
Figure 3: Laboratory Accreditation Process:
[See PDF for image]
[End of figure]
NIAP Membership in International Recognition Arrangement Gives Agencies
Broader Product Selection and Reduces Vendor Burden:
Another benefit of the NIAP evaluation process is NIAP's membership in
the Arrangement on the Recognition of Common Criteria Certificates in
the Field of IT Security. As part of the goals of the arrangement,
members can increase the availability of evaluated IT products and
protection profiles for national use and eliminate duplicate
evaluations of IT products and protection profiles, thus giving
agencies a broader selection of evaluated products from which to
choose. Agencies have the ability to acquire products that have been
evaluated at evaluation assurance levels 1 through 4 from any of the
countries that have an evaluation scheme. As of February 2006, there
were 22 global signatories[Footnote 21] to the recognition arrangement,
and 247 evaluated products available.
The recognition arrangement also reduces the burden on vendors by
limiting the number of criteria to which their products must conform
and the number of evaluations that a vendor needs to complete in order
to sell a product internationally. Because NIAP evaluations (evaluation
assurance levels 1-4) are accepted by the arrangement, vendors that go
through the NIAP process can sell their evaluated products in any of
the 22 member countries. Vendors are able to save time and money since
they do not need to complete multiple evaluations to sell their product
in different countries.
Product Evaluations Can Uncover Flaws and Cause Vendors to Fix Them:
Another benefit of the NIAP process is that it uncovers flaws during
product evaluations and can cause vendors to fix them. NIAP, vendor,
and laboratory officials stated that the NIAP evaluation process has
uncovered flaws and vulnerabilities in evaluated products. According to
NIAP officials, software flaws are found in nearly all evaluated
products, with an evaluation resulting in an average of two to three
fixes. According to the four vendors included in our review, the NIAP
evaluation process discovered flaws or vulnerabilities in their
products or their product documentation. Also, officials from one of
the laboratories included in our review stated that out of the 90
products they have evaluated, all of them had documentation flaws.
Although vendors have the option of removing from the evaluation
security features or functions in which flaws have been identified, any
flaws in the remaining security features or functions must be fixed in
order to successfully complete the product evaluation. Nonetheless,
agencies procuring NIAP-evaluated products have a higher level of
assurance that the product's security features and functions will
perform as claimed in the validation report.
Product Evaluations Can Result in Improvements to Vendors' Development
Processes:
Product evaluations can influence vendors to make improvements to their
development processes that raise the overall quality of their current
and future products. To complete a successful evaluation, vendors
submit to laboratories their development documentation, which describes
various processes related to security, such as software configuration
controls. Officials at six of the seven vendors we visited stated that
product evaluations had a positive influence on their development
process. According to one of the six vendors, completed product
evaluations that result in improvements to their development process
would likely transfer to the development process of other products and
help improve the overall quality of their products. Laboratory
officials also stated that NIAP evaluations often result in vendors
improving their software development process because vendors adopt some
of the methodologies used to pass evaluation, such as test methods and
documentation, for their own quality assurance processes. Additionally,
we previously reported that vendors who are proactive and adopt
effective development processes and practices can drastically reduce
the number of flaws in their products.[Footnote 22]
NIAP Evaluation Process Faces Challenges:
NIAP process participants--NIAP officials and selected vendors,
laboratories, and federal agencies--identified challenges to acquiring
and using NIAP-evaluated products.
* NIAP-evaluated products do not always meet agencies' needs, which
limit agencies' acquisition and use of these products.
* A lack of vendor awareness of the NIAP evaluation process impacts the
timely completion of the evaluation and validation of products.
* A reduction in the number of validators available to certify products
could contribute to delays in validating products for agency use; and:
* A lack of performance measures and difficulty in documenting the
effectiveness of the NIAP process makes it difficult to demonstrate the
program's usefulness or improvements made to products' security
features and functions or improvements to vendors' development
processes.
Collectively, these challenges hinder the effective use of the NIAP
evaluation process by vendors and agencies.
NIAP-Evaluated Products Do Not Always Meet Agencies' Needs:
Meeting agency needs for NIAP-evaluated products for use in national
security systems can be a challenge. According to agency responses to
our survey, 10 of 18 agencies[Footnote 23] that purchased NIAP-
evaluated products reported experiencing challenges in acquiring those
products. Specifically,
* 10 agencies noted that products on the NIAP-evaluated product list
were not the most current versions; and:
* 7 agencies noted that products needed by their agency were not
included on the NIAP-evaluated product list.
Agencies also reported additional challenges for acquiring NIAP--
evaluated products. Specifically,
* choices for evaluated products are somewhat limited compared to the
general product marketplace; and:
* the length of time required for a product to complete the evaluation
process can delay availability of the most up-to-date technology.
However, opportunities exist to better match agency needs with the
availability of NIAP-evaluated products:
* Agencies can write protection profiles to define the exact security
parameter specifications that they need. For example, two of the
vendors we visited stated that they had their products evaluated
against the Controlled Access Protection Profile, which provides
agencies with a set of security functional and assurance requirements
for their IT products and also provides a level of protection against
threats of inadvertent or casual attempts to breach the system
security.
* Vendors can enter the evaluation process before their products are
publicly released, which can allow consumers to acquire the most up-to-
date technology. One vendor we visited had taken such a proactive
approach.
* Agencies can use the NIAP-validated products list to identify
products that meet their needs. Because the number of available NIAP-
evaluated products is increasing, agencies now have a variety of
products from which to choose. In January 2002, there were about 20
evaluated products. As of February 2006, there were 127 evaluated
products and 142 products in evaluation. These evaluated products span
across 26 categories of information assurance products and information
assurance-enabled products from which to choose, including operating
systems and firewalls. As products continue to enter evaluation,
agencies' needs may be better met.
* Vendors can, by participating in the NIAP Assurance Maintenance
Program, maintain the validity of an evaluation when a product upgrades
to its next version by either requesting a re-evaluation of the entire
new product version or validation of only the changes in the product.
Vendors' participation in this program may allow agencies to have the
most recent products available to them.
* Agencies can increase their selection of products through the Common
Criteria Recognition Arrangement--available on the Common Criteria
portal Web site--which currently has 247 evaluated products available.
The products listed on the Web site give agencies more choices of
products evaluated at evaluation assurance levels 4 and below.
Lack of Vendors Awareness of NIAP Evaluation Process Affects Efficiency
of Evaluations:
Another challenge faced by the NIAP process is the lack of vendor
awareness regarding the requirements of the evaluation process. For
example, vendors who are new to the evaluation process are not aware of
the extensive documentation requirements. Creating documentation to
meet evaluation requirements can be an expensive and time-consuming
process. According to laboratory officials, about six months is the
average time for vendors to complete the required documentation before
test and evaluation can begin. However, if vendors consistently
maintain their documentation, subsequent evaluations can be faster and
less expensive since the vendor has previously produced the
documentation and is already familiar with the process.
Also, some vendors are not as active as others in the evaluation
process, which can cause varying lengths of time for completing the
evaluation. Vendors who are actively involved in the process are
usually able to complete the process more quickly, including fixing
flaws, than those who are not actively involved. According to one
laboratory, the more active a vendor is in the evaluation process, the
faster and less expensive it will be for the vendor. As such, the
amount of involvement by the vendor during the process and the
timeliness with which it fixes discovered flaws affects the length of
time the product is in evaluation.
Furthermore, some vendors and laboratories do not have the same
perception of the length of time required to perform the evaluation.
According to laboratory officials, the length of time needed for
conducting product evaluations varies depending on the type of product
being evaluated and the evaluation assurance level (see fig. 4).
Vendors are often not aware of these requirements and tend to
underestimate the length of time required for evaluations.
Figure 4: Range of Time Required for Completing Product Evaluations at
Various Evaluation Assurance Levels:
[See PDF for image]
[End of figure]
Vendors and laboratories also perceive the length of evaluations
differently because they punctuate start and end dates differently.
Vendors measure the length of an evaluation from the day they decide to
go into evaluation to the day they receive their product certificate.
Their measurement includes selecting and negotiating with a laboratory,
preparing required documentation, and testing the security features and
functions. Laboratories, on the other hand, consider the length of an
evaluation to be from the day they sign a contract with the vendor to
the day they complete testing.
While Common Criteria user forums for program participants have been
held, which NIAP participated in, NIAP itself has not developed
education and training workshops that focus on educating participants
on specific requirements--such as the documentation requirements. These
workshops could help ensure that vendors and laboratories are aware of
the NIAP process and could contribute to the efficiency of product
evaluations. NIAP officials acknowledge that such educational offerings
could be beneficial.
Reduction in the Number of Validators May Affect Timeliness of
Certifications:
Over the last year, NIAP has seen a reduction in the number of
qualified validators. NIAP officials stated that one of the most
significant challenges the NIAP process faces is hiring and maintaining
qualified personnel to validate products. In fiscal year 2005, the NIAP
program lost approximately four government validators and six
contractor validators. According to the NIAP Director, maintaining
qualified personnel to perform validation tasks is difficult largely
because many validators are nearing retirement age and the job is not
an attractive position for recent college graduates. Validators have a
complex job with tasks that span the entire evaluation process; they
incrementally review the results of the various tests of functional and
assurance requirements as they are completed by the laboratory. As a
result, once validators are hired, it typically takes 12 to 24 months
to train new validators to become proficient in performing validation
tasks.
If the NIAP program continues to see a reduction in validators, there
could be an increased risk that a backlog of products needing to obtain
NIAP certifications will develop, which could also impact the already
lengthy evaluation process. The number of products entering evaluation
is steadily increasing (in fiscal year 2002 there were approximately 20
products in evaluation and as of February 2006, there were 142 products
in evaluation). Additionally, approximately five to seven products
enter into evaluation each month. To address the widening gap between
the number of products entering the process and the number of
validators available to review products, NIAP intends to pursue
legislation allowing it to recoup the costs of validations and hire
additional staff.
Lack of Performance Measures and Difficulty in Documenting the
Effectiveness of the NIAP Process:
A best practice in public and private organizations is the use of
performance measurements to gain insight into--and make adjustments to-
-the effectiveness and efficiency of programs, processes, and
people.[Footnote 24] Performance measurement is a process of assessing
progress toward achieving predetermined goals, and includes gathering
information on the efficiency with which resources are transformed into
goods and services, the quality of those outputs, and the effectiveness
of government operations in terms of their specific contributions to
program objectives. Establishing, updating, and collecting performance
metrics to measure and track progress can assist organizations in
determining whether they are fulfilling their vision and meeting their
customer-focused strategic goals.
The NIAP program lacks performance metrics to measure process
effectiveness and thus faces difficulty in documenting its
effectiveness. The program has not collected and analyzed data on the
findings, flaws, and fixes resulting from product tests and
evaluations. NIAP officials pointed out that nondisclosure agreements
between laboratories and vendors make it difficult to collect and
document such data. According to NIAP officials, there is existing
laboratory information on findings, flaws, and fixes, but it has not
been collected because of nondisclosure agreements. Nondisclosure
agreements are important for protecting vendors' proprietary data from
being released to the public and competitors. However, releasing
summary laboratory information on findings, flaws and fixes, while at
the same time considering the requirements of nondisclosure agreements,
could be beneficial to determining the effectiveness of the NIAP
program. Without this type of information, NIAP will have difficulty
demonstrating its effectiveness and will be challenged to know and to
demonstrate whether the process is meeting its goals.
Expanding NIAP Requirement to Non-national Security Systems May Yield
Many of the Same Benefits and Challenges and Could Exacerbate Resource
Constraints:
While the National Security Telecommunications and Information Systems
Security Policy Number 11 already allows agencies with non-national
security systems to acquire NIAP-evaluated products, expanding the
policy to mandate that such systems acquire NIAP-evaluated products may
yield many of the same benefits and challenges experienced by current
process participants, and could further exacerbate resources. For
example, one identified benefit for national security systems--
independent testing and evaluation of IT products--gives agencies
confidence that validated features of a product, whether acquired for
national or non-national security systems, will perform as claimed by
the vendor. Similarly, one challenge--a reduction in the number of
validators for certifying products--could contribute to delays in
validating products, whether for national or non-nation security
systems. Further, expanding the requirement to mandate the policy for
non-national security systems may further exacerbate current resource
constraints, related to hiring and maintaining qualified personnel to
validate products.
Nevertheless, agencies with non-national security systems have in fact
acquired NIAP-evaluated products. Specifically, ten of the federal
agencies we surveyed indicated that they have used the NIAP process to
acquire evaluated products for non-national security systems, even
though they are not required to do so. One agency is considering the
use of NIAP-evaluated products during its product reviews, and is also
considering including NIAP-evaluated products as part of its
procurement strategy.
Moreover, agencies seeking information assurance for their non-national
security systems, but who do not acquire NIAP-evaluated products, have
guidance and standards available to them. Specifically, as required by
the Federal Information Security Management Act, NIST has developed and
issued standards and guidelines, including minimum information security
requirements, for the acquisition and use of security-related IT
products for non-national security systems.[Footnote 25] These
standards and guidelines are to be complementary with those established
for the protection of national security systems and information
contained in such systems. Further, NIST issued additional guidance to
agencies for incorporating security into all phases of the system
development life cycle process[Footnote 26] as a framework for
selecting and acquiring cost-effective security controls. In August
2000, NIST also issued guidance on security assurance for non-national
security systems in NIST Special Publication 800-23: Guideline to
Federal Organizations on Security Assurance and Acquisition/Use of
Tested/Evaluated Products.
Conclusions:
While a range of controls are needed to protect national security
systems against increasingly sophisticated cyber attacks, establishing
effective policies and processes for acquiring products that have been
validated by an independent party is important to the federal
government's ability to procure and deploy the right technologies.
Acquiring NIAP-evaluated products can increase the federal government's
confidence that its IT products and systems will perform security
features and functions as claimed. Despite the benefits of acquiring
and using IT products that have gone through the rigorous tests and
evaluations of NIAP, the program faces considerable challenges that
hinder its effective use by vendors and agencies. These challenges
include the difficulty in matching agencies' needs with the
availability of NIAP-evaluated products, vendors' lack of awareness
regarding the evaluation process, a reduction in the number of
validators to certify products, and difficulty in measuring and
documenting the effectiveness of the NIAP process. Until these
challenges are addressed, they will continue to undermine the efficacy
of NIAP. Regarding expanding the NIAP requirement to non-national
security systems, pursing this approach may further exacerbate current
resource constraints.
Recommendations for Executive Action:
To assist the NIAP in documenting the effectiveness of the NIAP
evaluation process, we recommend that the Secretary of Defense direct
the Director of the National Security Agency, in coordination with NIST
under the provisions of the NIAP partnership, to take the following two
actions:
1. Coordinate with vendors, laboratories, and various industry
associations that have knowledge of the evaluation process to develop
awareness training workshops for program participants.
2. Consider collecting, analyzing, and reporting metrics on the
effectiveness of NIAP tests and evaluations. Such metrics could include
summary information on the number of findings, flaws, and associated
fixes.
Agency Comments and Our Evaluation:
In providing written comments on a draft of this report (reprinted in
app. III), the Deputy Assistant Secretary of Defense (Deputy Chief
Information Officer), partially agreed with one of our recommendations,
agreed with the other, and described ongoing and planned efforts to
address them. While the Deputy Assistant Secretary agreed with our
recommendation to develop awareness training workshops for NIAP program
participants, she stated that the NIAP must also live with the
realities of the challenges that we identified in our report. The
Deputy Assistant Secretary noted that, as our report highlights, the
NIAP program is facing considerable challenges with resources and
funding to sustain the current day-to-day running of the program and
that it is not feasible for the NIAP office to increase its current
efforts in developing and hosting the recommended training and
education. Nonetheless, she also noted that the Secretary of Defense
should direct the Director of the National Security Agency, in
coordination with the NIST under the provisions of the NIAP, to
coordinate with the vendors, laboratories, and various industry
associations that have knowledge of the evaluation process to develop
awareness training workshops for program participants within the
current constraints and to work with the commercial laboratories,
vendors, and others to identify ways that organizations outside of NIAP
can further this initiative. We agree that NIAP should continue its
efforts in awareness and education training, and endorse increasing
such efforts as resources permit.
The Deputy Assistant Secretary agreed with our recommendation to
collect, analyze, and report metrics on the effectiveness of NIAP tests
and evaluations, and stated that the NIAP has already started
researching ways to institute metrics to help determine the
effectiveness of the evaluation program. She noted that the goal of
collecting metrics is to demonstrate to the NIAP constituency that NIAP
evaluations do provide value by improving the security of the evaluated
products and by providing the end customer with assurance that these
products perform their security functions as intended even when faced
with adverse conditions.
The Department of Defense and the Department of Homeland Security also
provided technical comments, which we considered and addressed in our
report, as appropriate.
We are sending copies of this report to the Departments of Commerce
(National Institute of Standards and Technology), Defense, and Homeland
Security; the Office of Management and Budget; the General Services
Administration, and to other interested parties. In addition, the
report will be available at no charge on the GAO Web site at
http://www.gao.gov.
If you or your staff have any questions about this report, please
contact me at (202) 512-6244 or [Hyperlink, wilshuseng@gao.gov].
Contact points for our Offices of Congressional Relations and Public
Affairs may be found on the last page of this report. GAO staff who
made major contributions to this report are listed in appendix IV.
Sincerely yours,
Signed by:
Gregory C. Wilshusen:
Director, Information Security Issues:
[End of section]
Appendixes:
Appendix I: Objectives, Scope, and Methodology:
Our objectives were to identify (1) the governmentwide benefits and
challenges of the National Information Assurance Partnership (NIAP)
evaluation process; and (2) the potential benefits and challenges of
expanding the requirement of NIAP to non-national security systems,
including sensitive but unclassified systems.
To determine the benefits and challenges for both objectives, we
analyzed and reviewed a number of policy documents and reports from
both industry and government. We also reviewed relevant federal
policies relating to information security issues. To gain insight into
the NIAP evaluation process, we met with software vendors and
certification laboratories to discuss their experiences with NIAP,
their applicable processes, and reviewed their relevant documentation.
We selected vendors based on broad or distinguishing product
capabilities demonstrating a range of features, brand recognition based
on high ratings received in reviews conducted by information security
magazines, and vendors mentioned more frequently in various discussions
with industry experts and in information security literature. Vendors
selected represented different information technology (IT) market
sectors, are considered leaders in their field, and varied in size.
To determine the industrywide perspective on NIAP, we met with two IT
industry groups: The Information Technology Association of America and
Cyber Security Industry Alliance. We selected these industry groups
because they represent a cross-section of the IT industry as a whole.
To gain insight into the program's functions and usefulness to
agencies, we spoke with government officials from the Department of
Commerce (specifically the National Institute of Standards and
Technology), Department of Defense, Department of Homeland Security,
General Services Administration, and the Office of Management and
Budget. We also surveyed officials from the 24 federal agencies
designated under the Chief Financial Officers Act of 1990 to determine
their current use of NIAP-evaluated products, the perceived usefulness
of the program, and the benefits and challenges associated with
acquiring and using NIAP-evaluated products.
For each agency survey, we identified the office of the chief
information officer, notified them of our work, and distributed the
survey instrument to each via an e-mail attachment. In addition, we
discussed the purpose and content of the survey instrument with agency
officials when requested. All 24 agencies responded to our survey. We
did not verify the accuracy of the agencies' responses; however, we
reviewed supporting documentation that agencies provided to validate
their responses. We contacted agency officials when necessary for
follow-up information. We then analyzed the agencies' responses.
Although this was not a sample survey, and, therefore, there were no
sampling errors, conducting any survey may introduce other kinds of
errors. For example, difficulties in how a particular question is
interpreted, in the sources of information that are available to
respondents, or in how the data are entered into a database (or were
analyzed) can introduce unwanted variability into the survey results.
We took steps in the development of the survey instrument, the data
collection, and the data analysis to minimize these survey-related
errors. For example, we developed the questionnaire in two stages.
First, we had a survey specialist design the survey instrument in
collaboration with subject-matter experts. Then, we pretested the
instrument at two federal departments and internally at GAO to ensure
that questions were relevant, clearly stated, and easy to answer.
We conducted our work in Washington, D.C., from May 2005 through
February 2006, in accordance with generally accepted government
auditing standards.
[End of section]
Appendix II: Common Criteria Evaluation Assurance Levels:
Table 1: Summary of the Common Criteria Evaluation Assurance Levels:
Evaluation assurance level: 1;
What is tested: Functionality;
Description: Evaluation provides independent testing against a
specification and an examination of the guidance documentation. Used
when confidence in correct operation is required but the threats to
security are not viewed as serious.
Evaluation assurance level: 2;
What is tested: Structure;
Description: Evaluation provides a low to moderate level of
independently assured security as required by vendors or users.
Evaluation assurance level: 3;
What is tested: Methodology;
Description: Evaluation provides an analysis supported by testing,
selective independent confirmation of the vendor test results, and
evidence of a vendor search for obvious vulnerabilities.
Evaluation assurance level: 4;
What is tested: Methodology and Design;
Description: Evaluation provides a moderate to high level of
independently assured security in conventional commodity products.
Testing is supported by an independent search for obvious
vulnerabilities.
Evaluation assurance level: 5;
What is tested: Semiformal Design;
Description: Evaluation provides a high level of independently assured
security in a planned development, with a rigorous development
approach. The search for vulnerabilities must ensure resistance to
penetration attackers with a moderate attack potential.
Evaluation assurance level: 6;
What is tested: Semiformal Verified Design;
Description: Used for the development of specialized security products,
for application in high risk situations. The independent search for
vulnerabilities must ensure resistance to penetration attackers with a
high attack potential.
Evaluation assurance level: 7;
What is tested: Formal Design;
Description: Used in the development of security products for
application in extremely high risk situations. Evidence of vendor
testing and complete independent confirmation of vendor test results
are required.
Source: GAO analysis of Common Criteria data.
[End of table]
[End of section]
Appendix III: Comments from the Department of Defense:
DEPARTMENT OF DEFENSE:
CHIEF INFORMATION OFFICER:
6000 DEFENSE PENTAGON:
WASHINGTON, DC 20301-6000:
March 21, 2006:
Mr. Gregory C. Wilshusen:
Director, Information Security Issues:
U.S. Government Accountability Office:
441 G Street, N.W.
Washington, D.C. 20548:
Dear Mr. Wilshusen:
This is the Department of Defense (DoD) response to the GAO Draft
Report, GAO-06-392, "INFORMATION ASSURANCE: National Partnership Offers
Benefits, but Faces Considerable Challenges," dated March 10, 2006, GAO
Code 310551. The DoD has reviewed the report and concurs with the
findings, partially concurs with Recommendation 1, and concurs with
Recommendation 2.
Enclosure 1 contains the DoD response and rationale for the report's
two recommendations to the Secretary of Defense. Enclosure 2 contains
general administrative comments offered by the DoD reviewers of the
draft report.
Sincerely,
Signed for:
Priscilla E. Guthrie:
Deputy Assistant Secretary of Defense (Deputy CIO):
GAO DRAFT REPORT DATED MARCH 10, 2006 GAO-06-392 (GAO CODES 310551):
"INFORMATION ASSURANCE: NATIONAL PARTNERSHIP OFFERS BENEFITS, BUT FACES
CONSIDERABLE CHALLENGES"
DEPARTMENT OF DEFENSE COMMENTS TO THE GAO RECOMMENDATION:
RECOMMENDATION 1: The GAO recommended that the Secretary of Defense
direct the Director of the National Security Agency, in coordination
with the National Institute of Standards and Technology under the
provisions of the National Information Assurance Partnership (NIAP) to
coordinate with the vendors, laboratories, and various industry
associations that have knowledge of the evaluation process to develop
awareness training workshops for program participants. (p. 29/GAO Draft
Report):
DOD RESPONSE: Partial Concur:
While DoD agrees with the recommendation to promote awareness training
and understanding of the NIAP CCEVS, we also must live with the
realities of the challenges identified in the GAO report. NIAP has and
will continue to work with vendors, labs and industry associations to
provide awareness training and education. Over the past year, the NIAP
has provided briefings at a wide variety of for a, such as the Federal
Information Assurance Conference, the DoD Identity Management
Conference, the Cross Domain Solutions Workshop, and the Annual
Computer Security Applications Conference. The NIAP has also been
working closely with the Common Criteria User's Forum and the newly
established Common Criteria Vendor's Forum on education and awareness
efforts. Unfortunately, as the report highlights, the NIAP program is
facing considerable challenges with resources and funding to sustain
the current day-to-day running of the program. When coupled with the
significant loss of support for the NIAP from KIST due to higher
priorities within NIST, and increasing pressures on DoD resources to
support current military operations supporting the Global. War On
Terrorism, it is not feasible for the NIAP office to increase its
current efforts in developing and hosting the recommended awareness
training and education. The Secretary of Defense should direct the
Director of the National Security Agency, in coordination with the
NIST, under the provisions of the NIAP, to coordinate with the vendors,
laboratories, and various industry associations that have knowledge of
the evaluation process to develop awareness training workshops for
program participants within the current constraints and to work with
the commercial labs, vendors, and others to identify ways that
organizations outside of NIAP can further this initiative.
RECOMMENDATION 2: The GAO recommended that the Secretary of Defense
direct the Director of the National Security Agency, in coordination
with the National Institute of Standards and Technology under the
provisions of the NIAP to consider collecting, analyzing, and reporting
metrics on the effectiveness of NIAP tests and evaluations; such
metrics could include summary information on the number of findings,
flaws, and associated fixes. (p. 29/GAO Draft Report):
DOD RESPONSE: Concur.
NIAP CCEVS has already started researching ways to institute metrics to
help determine the effectiveness of the evaluation program. NIAP CCEVS
has already begun collecting metrics on the effectiveness of NIAP
testing. In CY 2004 and 2005, most of the metrics collected were based
on informal queries to the NIAP labs and were gathered to provide some
general statistics during NIAP presentations. Since that time, based on
requests from the Committee for National Security Systems, NIAP CCEVS
has begun collecting additional general metrics and anecdotes from
their commercial labs on how evaluations have improved vendor products.
NIAP CCEVS has also begun collecting specific detailed metrics through
their NIAP Monthly Status Reports, which are required for each
evaluation in progress. In addition, NIAP CCEVS is developing a
template for a comprehensive end-of-evaluation report which will detail
all changes or improvements made to the product or the vendor's
processes during the evaluation process. This will include fixes to
critical user documentation, improvements to vendor processes, and
changes to the product itself. The goal of collecting these metrics is
to demonstrate to the NIAP constituency that NIAP evaluations provide
value by improving the security of the evaluated products and by
providing the end customer with assurance that these products perform
their security functions as intended, even when faced with adverse
conditions.
[End of section]
Appendix IV: GAO Contact and Staff Acknowledgments:
GAO Contact:
Gregory C. Wilshusen, Director, Information Security Issues, (202) 512-
6244:
Staff Acknowledgments:
In addition to the individual named above, Jenniffer Wilson (Assistant
Director), Neil Doherty, Jennifer Franks, Joel Grossman, Matthew Grote,
Min Hyun, Anjalique Lawrence, J. Paul Nicholas, Karen Talley, and Amos
Tevelow were key contributors to this report.
(310551):
FOOTNOTES
[1] National security systems are telecommunications and information
systems under control of the United States government which contain
classified information or the function, operation, or use of which
involves intelligence activities, cryptologic activities related to
national security, command and control of military forces, equipment
that is an integral part of a weapon or weapon system; or is critical
to the direct fulfillment of military or intelligence missions.
[2] An information assurance product is an information technology (IT)
product or technology that primarily provides security services (such
as confidentiality and integrity), corrects known vulnerabilities, and
provides layered defense against various categories of non-authorized
and malicious penetration of information systems or networks. An
information assurance-enabled product is an IT product or technology
that provides security services as an associated feature of its
intended operating capabilities, rather than as its primary role.
[3] Information assurance refers to measures that defend and protect
information and information systems by ensuring their confidentiality,
integrity, authenticity, availability, and utility.
[4] The Director of the National Security Agency is the Department of
Defense focal point for National Information Assurance Partnership
(NIAP).
[5] Known as the Common Criteria, the international standard contains
IT security requirements, constructs for describing IT security
objectives, and a framework for writing high-level security
specifications for a product. It specifies functional security
requirements and seven predefined assurance packages, referred to as
evaluation assurance levels.
[6] Evaluation assurance levels provide a reference for the amount of
analysis and testing performed on a product.
[7] Computer programs that are stored in read-only memory are called
firmware.
[8] Protection profiles define a security problem for a given
collection of systems or products and delineate security requirements
to address that problem without specifying how these requirements will
be implemented. U.S. government protection profiles are developed into
one of three robustness levels--basic, medium, and high.
[9] A security target is a specifications document that describes the
security functionality of a product and the environment in which it
will operate. The security target details the desired evaluation
assurance levels that the vendor wants the product to be tested
against. Vendors can also claim conformance to a protection profile in
their security targets.
[10] The National Voluntary Laboratory Accreditation Program (NVLAP) is
administered by the National Institute of Standards and Technology
(NIST), and operates as an unbiased third-party to accredit testing and
calibration laboratories in many fields. NVLAP operates on a cost-
reimbursable basis from fees paid by participating laboratories.
[11] According to NIAP, as the initiative has evolved, the National
Security Agency has assumed all of the validation responsibilities;
NIST no longer participates in discharging these responsibilities.
[12] The Committee on National Security Systems consists of
representatives from 20 U.S. government departments and agencies who
are given voting privileges on all committee activities. National
Security Directive 42 specifies the membership of the committee. The
Assistant Secretary of Defense for Networks and Information
Integration/Department of Defense Chief Information Officer serves as
the chair of the committee.
[13] Federal agencies and consumers should review the security targets
that describe the threats, objectives, and requirements against which
the product has been tested to determine if the product is appropriate
for a particular application or system and that it provides adequate
information security protections for the intended operational
environment.
[14] Products whose evaluations have assurance components above
assurance level 4 must be tested by the National Security Agency for
that portion of the product's features and function that are above
level 4.
[15] NVLAP identifies NVLAP-accredited laboratories on its Web site.
Accreditation criteria are established in accordance with the U.S. Code
of Federal Regulations (CFR, Title 15, Part 285), NVLAP Procedures and
General Requirements, and encompass the requirements of ISO/IEC 17025
and the relevant requirements of ISO 9002.
[16] The Committee on National Security Systems was formerly known as
the National Security Telecommunications and Information Systems
Security Committee.
[17] In October 1998, the U.S., Canada, France, Germany, and the United
Kingdom signed an arrangement for Common-Criteria-based security
evaluations covering evaluated assurance levels 1-4 known as the
Arrangement on the Recognition of Common Criteria Certificates in the
Field of IT Security. The arrangement sought to create a situation in
which IT products and protection profiles that earn a Common Criteria
certificate in one arrangement member country can be procured or used
without the need for further evaluation in other arrangement member
countries.
[18] The policy requires that any commercial off-the-shelf information
assurance or information assurance-enabled product using cryptographic
technology be certified by the Cryptographic Module Validation Program,
which evaluates products for conformance to FIPS 140-2, Security
Requirements for Cryptographic Modules.
[19] Certification is a comprehensive evaluation of security controls
that provides the necessary information for a designated approving
authority to formally declare that a system is approved to operate at
an acceptable level of risk. Accreditation is the authorization of an
information system to process, store, or transmit information that
provides a form of quality control. The accreditation decision, which
is supported by the certification, provides the necessary information
for a designated approving authority to formally declare that a system
is approved to operate.
[20] Twenty-four agencies completed the survey: fifteen agencies
identified benefits; three did not; the remaining six agencies did not
purchase any NIAP-evaluated products.
[21] The 22 countries include the certificate-authorizing countries--
Australia, Canada, France, Germany, Japan, New Zealand, The
Netherlands, United Kingdom, and the United States; and certificate-
consuming countries--Austria, Czech Republic, Finland, Greece, Hungary,
India, Israel, Italy, Norway, Republic of Singapore, Spain, Sweden and
Turkey. Of the 22 signatories, 9 have evaluation schemes that authorize
them to certify products in accordance with the arrangement. The
remaining signatories do not have a scheme but have agreed to accept
the certificates authorized by countries with schemes.
[22] GAO, Information Security: Continued Action Needed to Improve
Software Patch Management, GAO-04-706 (Washington, D.C.: June 2004).
[23] Although 24 agencies completed the survey, 18 purchased NIAP-
evaluated products; the remaining 6 agencies did not.
[24] National Performance Review, Serving the American Public: Best
Practices in Performance Measurement, June 1997,
http://govinfo.library.unt.edu/npr/library/papers/benchmrk/nprbook.html.
[25] In February 2005, NIST issued "Recommended Security Controls for
Federal Information Systems" (Special Publication 800-53) to provide
guidelines for selecting and specifying security controls for
information systems categorized in accordance with FIPS Publication
199, "Standards for Security Categorization of Federal Information and
Information Systems," issued in February 2004.
[26] The phases of a system life cycle, as defined by NIST, are:
initiation, development/acquisition, implementation, operation and
maintenance, and disposition.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
