Warfighter Support
Independent Expert Assessment of Army Body Armor Test Results and Procedures Needed Before Fielding Gao ID: GAO-10-119 October 16, 2009The Army has issued soldiers in Iraq and Afghanistan personal body armor, comprising an outer protective vest and ceramic plate inserts. GAO observed Preliminary Design Model testing of new plate designs, which resulted in the Army's awarding contracts in September 2008 valued at a total of over $8 billion to vendors of the designs that passed that testing. Between November and December 2008, the Army conducted further testing, called First Article Testing, on these designs. GAO is reporting on the degree to which the Army followed its established testing protocols during these two tests. GAO did not provide an expert ballistics evaluation of the results of testing. GAO, using a structured, GAO-developed data collection instrument, observed both tests at the Army's Aberdeen Test Center, analyzed data, and interviewed agency and industry officials to evaluate observed deviations from testing protocols. However, independent ballistics testing expertise is needed to determine the full effect of these deviations.
During Preliminary Design Model testing the Army took significant steps to run a controlled test and maintain consistency throughout the process, but the Army did not always follow established testing protocols and, as a result, did not achieve its intended test objective of determining as a basis for awarding contracts which designs met performance requirements. In the most consequential of the Army's deviations from testing protocols, the Army testers incorrectly measured the amount of force absorbed by the plate designs by measuring back-face deformation in the clay backing at the point of aim rather than at the deepest point of depression. Army testers recognized the error after completing about a third of the test and then changed the test plan to call for measuring at the point of aim and likewise issued a modification to the contract solicitation. At least two of the eight designs that passed Preliminary Design Model testing and were awarded contracts would have failed if measurements had been made to the deepest point of depression. The deviations from the testing protocols were the result of Aberdeen Test Center's incorrectly interpreting the testing protocols. In all these cases of deviations from the testing protocols, the Aberdeen Test Center's implemented procedures were not reviewed or approved by the Army and Department of Defense officials responsible for approving the testing protocols. After concerns were raised regarding the Preliminary Design Model testing, the decision was made not to field any of the plate designs awarded contracts until after First Article Testing was conducted. During First Article Testing, the Army addressed some of the problems identified during Preliminary Design Model testing, but GAO observed instances in which Army testers did not follow the established testing protocols and did not maintain internal controls over the integrity and reliability of data, raising questions as to whether the Army met its First Article Test objective of determining whether each of the contracted designs met performance requirements. The following are examples of deviations from testing protocols and other issues that GAO observed: (1) The clay backing placed behind the plates during ballistics testing was not always calibrated in accordance with testing protocols and was exposed to rain on one day, potentially impacting test results. (2) Testers improperly rounded down back-face deformation measurements, which is not authorized in the established testing protocols and which resulted in two designs passing First Article Testing that otherwise would have failed. Army officials said rounding is a common practice; however, one private test facility that rounds told GAO that they round up, not down. (3) Testers used a new instrument to measure back-face deformation without adequately certifying that the instrument could function correctly and in conformance with established testing protocols. The impact of this issue on test results is uncertain, but it could call into question the reliability and accuracy of the measurements. (4) Testers deviated from the established testing protocols in one instance by improperly scoring a complete penetration as a partial penetration. As a result, one design passed First Article Testing that would have otherwise failed. With respect to internal control issues, the Army did not consistently maintain adequate internal controls to ensure the integrity and reliability of test data. In one example, during ballistic testing, data were lost, and testing had to be repeated because an official accidentally pressed the delete button and software controls were not in place to protect the integrity of test data. Army officials acknowledged that before GAO's review they were unaware of the specific internal control problems we identified.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-10-119, Warfighter Support: Independent Expert Assessment of Army Body Armor Test Results and Procedures Needed Before Fielding
This is the accessible text file for GAO report number GAO-10-119
entitled 'Warfighter Support: Independent Expert Assessment of Army
Body Armor Test Results and Procedures Needed Before Fielding' which
was released on October 16, 2009.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
October 2009:
Warfighter Support:
Independent Expert Assessment of Army Body Armor Test Results and
Procedures Needed Before Fielding:
GAO-10-119:
GAO Highlights:
Highlights of GAO-10-119, a report to congressional requesters.
Why GAO Did This Study:
The Army has issued soldiers in Iraq and Afghanistan personal body
armor, comprising an outer protective vest and ceramic plate inserts.
GAO observed Preliminary Design Model testing of new plate designs,
which resulted in the Army‘s awarding contracts in September 2008
valued at a total of over $8 billion to vendors of the designs that
passed that testing. Between November and December 2008, the Army
conducted further testing, called First Article Testing, on these
designs. GAO is reporting on the degree to which the Army followed its
established testing protocols during these two tests. GAO did not
provide an expert ballistics evaluation of the results of testing. GAO,
using a structured, GAO- developed data collection instrument, observed
both tests at the Army‘s Aberdeen Test Center, analyzed data, and
interviewed agency and industry officials to evaluate observed
deviations from testing protocols. However, independent ballistics
testing expertise is needed to determine the full effect of these
deviations.
What GAO Found:
During Preliminary Design Model testing the Army took significant steps
to run a controlled test and maintain consistency throughout the
process, but the Army did not always follow established testing
protocols and, as a result, did not achieve its intended test objective
of determining as a basis for awarding contracts which designs met
performance requirements. In the most consequential of the Army‘s
deviations from testing protocols, the Army testers incorrectly
measured the amount of force absorbed by the plate designs by measuring
back-face deformation in the clay backing at the point of aim rather
than at the deepest point of depression. The graphic below depicts the
difference between the point of aim and the deepest point.
[Refer to PDF for image: illustration]
The following are depicted on the illustration:
Air laser:
Shooting barrel:
Point of aim:
Armor plate:
Clay backing:
Guide plane:
Back-face deformation:
Point of aim depression:
Deepest point of depression:
Source: GAO analysis.
[End of figure]
Army testers recognized the error after completing about a third of the
test and then changed the test plan to call for measuring at the point
of aim and likewise issued a modification to the contract solicitation.
At least two of the eight designs that passed Preliminary Design Model
testing and were awarded contracts would have failed if measurements
had been made to the deepest point of depression. The deviations from
the testing protocols were the result of Aberdeen Test Center‘s
incorrectly interpreting the testing protocols. In all these cases of
deviations from the testing protocols, the Aberdeen Test Center‘s
implemented procedures were not reviewed or approved by the Army and
Department of Defense officials responsible for approving the testing
protocols. After concerns were raised regarding the Preliminary Design
Model testing, the decision was made not to field any of the plate
designs awarded contracts until after First Article Testing was
conducted.
During First Article Testing, the Army addressed some of the problems
identified during Preliminary Design Model testing, but GAO observed
instances in which Army testers did not follow the established testing
protocols and did not maintain internal controls over the integrity and
reliability of data, raising questions as to whether the Army met its
First Article Test objective of determining whether each of the
contracted designs met performance requirements. The following are
examples of deviations from testing protocols and other issues that GAO
observed:
* The clay backing placed behind the plates during ballistics testing
was not always calibrated in accordance with testing protocols and was
exposed to rain on one day, potentially impacting test results.
* Testers improperly rounded down back-face deformation measurements,
which is not authorized in the established testing protocols and which
resulted in two designs passing First Article Testing that otherwise
would have failed. Army officials said rounding is a common practice;
however, one private test facility that rounds told GAO that they round
up, not down.
* Testers used a new instrument to measure back-face deformation
without adequately certifying that the instrument could function
correctly and in conformance with established testing protocols. The
impact of this issue on test results is uncertain, but it could call
into question the reliability and accuracy of the measurements.
* Testers deviated from the established testing protocols in one
instance by improperly scoring a complete penetration as a partial
penetration. As a result, one design passed First Article Testing that
would have otherwise failed.
With respect to internal control issues, the Army did not consistently
maintain adequate internal controls to ensure the integrity and
reliability of test data. In one example, during ballistic testing,
data were lost, and testing had to be repeated because an official
accidentally pressed the delete button and software controls were not
in place to protect the integrity of test data. Army officials
acknowledged that before GAO‘s review they were unaware of the specific
internal control problems we identified.
As a result of the deviations from testing protocols that GAO observed,
four of the five designs that passed First Article Testing and were
certified by the Army as ready for full production would have instead
failed testing at some point during the process, either during the
Preliminary Design Model testing or the subsequent First Article Test.
Thus, the overall reliability and repeatability of the test results are
uncertain. Although designs passed testing that would not have if the
testing protocols were followed, independent ballistics experts have
not assessed the impact of the deviations from the testing protocols to
determine if the effect of the deviations is sufficient to call into
those designs to meet requirements. Vendors whose designs passed First
Article Testing have begun production of plates. The Army has ordered
2,500 sets of plates (at two plates per set) from these vendors to be
used for additional ballistics testing and 120,000 sets of plates to be
put into inventory to address future requirements. However, to date,
none of these designs have been fielded because, according to Army
officials, there are adequate numbers of armor plates produced under
prior contracts already in the inventory to meet current requirements.
GAO‘s Recommendations:
To determine what effect, if any, the problems GAO observed had on the
test data and on the outcomes of First Article Testing, the Army should
provide for an independent ballistics evaluation of the First Article
Testing results by ballistics and statistical experts external to the
Department of Defense before any armor is fielded to soldiers under
this contract solicitation. Because DOD did not concur with this
recommendation, GAO added a matter for congressional consideration to
this report suggesting that Congress direct DOD to either conduct such
an independent external review of these test results or repeat First
Article Testing.
To better align actual test practices with established testing
protocols during future body armor testing, the Army should assess the
need to change its test procedures based on the outcome of the
independent experts‘ review and document these and all other key
decisions made to clarify or change the testing protocols during future
body armor testing. Although DOD did not agree that an independent
expert review of test results was needed, DOD stated it will address
protocol discrepancies identified by GAO as it develops standardized
testing protocols. DOD also agreed to document all decisions made to
clarify or change testing protocols.
To improve internal controls over the integrity and reliability of test
data for future testing as well as provide for consistent test
conditions and comparable data among tests, the Army should provide for
an independent external peer review of Aberdeen Test Center‘s body
armor testing protocols, facilities, and instrumentation to ensure that
proper internal controls and sound management practices are in place.
DOD generally concurred with this recommendation, but stated that it
will also include DOD members on the review team.
What GAO Recommends:
GAO makes several recommendations, which are discussed on the next
page, including to provide for an independent assessment of First
Article Testing data, to assess the need to change Army‘s procedures
based on that assessment, documenting this and all other key decisions
made, and to provide for an external peer review of Aberdeen Test
Center‘s protocols, facilities, and instrumentation.
View GAO-10-119 or key components. For more information, contact
William M. Solis at (202) 512-8365 or solisw@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
Army Took Significant Steps during Preliminary Design Model Testing to
Run a Controlled Test and Maintain Consistency but Did Not Consistently
Follow Established Testing Protocols and, as a Result, Did Not Achieve
the Intended Test Objective:
During First Article Testing the Army Addressed Some of the Problems
Identified in Preliminary Design Model Testing, but Army Testers Did
Not Always Follow Established Testing Protocols and Did Not Maintain
Some Internal Controls:
Conclusions:
Recommendations for Executive Action:
Matter for Congressional Consideration:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Comments from the Department of Defense:
Appendix III: GAO Contact and Staff Acknowledgments:
Table:
Table 1: Organizations Contacted for Information about Body Armor
Testing:
Figures:
Figure 1: ESAPI Plates as Worn inside Outer Tactical Vest:
Figure 2: Timeline of Key Preliminary Design Model Testing and First
Article Testing Events:
Figure 3: Clay Being Calibrated with Pre-Test Drops:
Figure 4: Graphic Representation of the Difference between the Point of
Aim and the Deepest Point:
Figure 5: Photographic Representation of the Difference between the
Point of Aim and the Deepest Point:
Figure 6: Tears in Kevlar Backing Material after a Penetration of the
Plate:
Figure 7: Briefing Slide from DOD's Test Overview (Nov. 14, 2007):
Figure 8: Briefing Slide from DOD's Test Strategy and Schedule (Nov.
14, 2007):
Abbreviations:
ASTM: American Society for Testing and Materials:
DOD: Department of Defense:
DODIG: Department of Defense Inspector General:
ESAPI: Enhanced Small Arms Protective Insert:
FSAPV-E: Flexible Small Arms Protective Vest-Enhanced:
FSAPV-X: Flexible Small Arms Protective Vest-X level:
MDA: Milestone Decision Authority:
NIJ: National Institute of Justice:
PEO: Program Executive Office:
XSAPI: Small Arms Protective Insert-X level:
[End of section]
United States Government Accountability Office: Washington, DC 20548:
October 16, 2009:
Congressional Requesters:
Since combat operations began in Afghanistan after September 11, 2001,
and in Iraq in 2003, U.S. forces have been subjected to frequent and
deadly attacks from insurgents using improvised explosive devices,
mortars, rocket launchers, and increasingly lethal ballistic threats.
To protect the military and civilian personnel of the Department of
Defense (DOD) against these ballistic threats, since 2003 the U.S.
Central Command has required that DOD personnel in its area of
operations be issued the Interceptor Body Armor system, comprising
ceramic plates that are inserted into pockets of an outer protective
vest. Over the past several years, the media and Congress have raised
concerns about whether the Army has adequately evaluated and tested
this body armor solution and about the transparency of the Army's body
armor testing. Additionally, several audits have found problems with
the Army's body armor testing programs. For example, in 2009, both the
DOD Inspector General and Army Audit Agency reported that the Army had
not followed established test procedures during prior tests of body
armor plates.[Footnote 1]
In 2007, we reported to the House and Senate Armed Services Committees
and testified to the House Armed Services Committee about the Army's
and Marine Corps's individual body armor systems.[Footnote 2] In that
report we found that the Army relied on several controls to ensure that
body armor met performance requirements, including testing at National
Institute of Justice (NIJ)-certified testing facilities. Later, under
the Comptroller General's authority, we observed the testing of body
armor solutions submitted under a May 2007 Army contract solicitation
for four categories of body armor-- specifically, the Enhanced Small
Arms Protective Insert (ESAPI), the Small Arms Protective Insert-X
level (XSAPI), the Flexible Small Arms Protective Vest-Enhanced (FSAPV-
E), and the Flexible Small Arms Protective Vest-X level (FSAPV-X).
While present, we observed the test procedures utilized by Army
testers, spoke with Army officials, and compared our observations with
established testing protocols. The purchase descriptions accompanying
the contract solicitation announcement identified the test procedures
to be followed during the first round of testing--called Preliminary
Design Model testing. Traditionally, Army body armor testing had been
performed at an NIJ-certified facility. However, one manufacturer of
flexible small arms protective vests, which had failed previous testing
conducted for the Program Executive Office (PEO) Soldier at an NIJ-
certified facility, made allegations that the PEO Soldier and the
facility had wrongly failed its designs. As a result of these
allegations, the Army decided instead to conduct testing for this
current solicitation at the Army's Aberdeen Test Center, which had not
performed testing of Interceptor Body Armor for PEO Soldier since the
1990s. Additionally, PEO Soldier decided not to provide any on-site
testing oversight to avoid any appearance of bias against that
manufacturer.[Footnote 3]
Preliminary Design Model testing was conducted by the Army's Aberdeen
Test Center from February 2008 though June 2008. The objective of the
Preliminary Design Model testing was to determine whether candidate
designs submitted under the solicitation met required ballistics
performance specifications and would be awarded a production contract.
[Footnote 4] In October 2008, on the basis of the Preliminary Design
Model testing results, the Army awarded four 5-year indefinite
delivery/indefinite quantity[Footnote 5] contracts at a total of over
$8 billion for the production of the ESAPI and the XSAPI--two
categories of ceramic plates. No FSAPV-E or FSAPV-X solutions passed
the testing. The Army decided to repeat testing, through First Article
Testing, of all of the ESAPI and XSAPI plates that were awarded
production contracts to determine whether these plate designs indeed
met the required ballistics performance specifications before fielding
the plates. The Aberdeen Test Center conducted First Article Testing
between November 2008 and December 2008.
In connection with the Army's decision to conduct First Article Testing
on each of the designs that passed Preliminary Design Model testing and
that were awarded contracts, the House Armed Services Committee and its
Subcommittee on Air and Land Forces requested that we observe this
follow-on First Article Testing to assess the degree to which testing
was conducted according to the established testing protocols. After
completing our analysis of both the Preliminary Design Model testing
and First Article Testing of body armor solutions, we are reporting on
the degree to which the Army followed its established testing protocols
during (1) Preliminary Design Model testing of the ESAPI, XSAPI, FSAPV-
E and FSAPV-X and (2) First Article Testing of the ESAPI and XSAPI
models that were awarded contracts after the Preliminary Design Model
testing.[Footnote 6] We did not provide an expert ballistics evaluation
of the results of testing.
To conduct our review, we observed Preliminary Design Model testing and
First Article Testing at the Army's Aberdeen Test Center in Aberdeen,
Maryland. We observed testing from inside the video viewing room and
firing lanes and also from the conditioning, X-ray, and physical
characterization rooms. We interviewed and collected information from
officials from the Aberdeen Test Center, the U.S. Army Evaluation
Center, PEO Soldier, DOD's office of the Director of Operational Test
and Evaluation, and other Army components, as well as from body armor
manufacturers and private body armor testing facilities. We recorded
selected test data in a systematic and structured manner using a data
collection instrument we developed, analyzed selected test data, and
compared our observations of the way the Aberdeen Test Center conducted
Preliminary Design Model testing and First Article Testing with the
testing protocols that Army officials told us served as the testing
standards at the Aberdeen Test Center. Specifically, these testing
protocols were: (1) test procedures described in the contract
solicitation announcement's purchase descriptions and (2) the Army's
detailed test plans and test operations procedures that were to serve
as guidance to Aberdeen Test Center testers and that were developed by
the Army Test and Evaluation Command and approved by PEO Soldier, the
office of the Director of Operational Test and Evaluation, Army
Research Labs, and cognizant Army components. In this report, we refer
to these testing standards that were to be used at Aberdeen Test Center
as testing protocols. We also reviewed NIJ testing standards because
Aberdeen Test Center officials told us that, although Aberdeen Test
Center is not an NIJ- certified testing facility, they have made
adjustments to their procedures based on those standards and consider
them when evaluating Aberdeen Test Center testing practices. Complete
details on our scope and methodology appear in appendix I.
We conducted this performance audit from July 2007 through October 2009
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.
Results in Brief:
During Preliminary Design Model testing the Army took significant steps
to run a controlled test and maintain consistency throughout the
process but did not always follow established testing protocols and, as
a result, did not achieve the intended test objective of determining
which designs met performance requirements as a basis for awarding
contracts. The Army's significant steps to run a controlled test
included, for example, the consistent documentation of testing
procedures using audio, video, and other electronic means and extensive
efforts to maintain proper temperature and humidity in the test lanes.
However, we identified several instances in which the Aberdeen Test
Center deviated from testing protocols, including failing to test the
ease of insertion of the plates into both pockets of the outer
protective vest as required by the testing protocols; shooting several
plates at the wrong velocity or location on the plate; and repeating
failed clay calibration tests on the same block of clay--the latter
having the potential to significantly affect test results. In the most
consequential of the deviations from testing protocols, the Army
testers incorrectly measured the amount of force absorbed by the
designs tested by measuring back- face deformation at the point of aim
rather than at the deepest point of depression.[Footnote 7] Army
testers recognized the error after completing about a third of the test
and then changed the test plan to call for measuring at the point of
aim and likewise issued a modification to the contract solicitation. At
least two[Footnote 8] of the eight designs that passed Preliminary
Design Model testing and were awarded contracts would have failed if
measurements had been made to the deepest point of depression. The
deviations from the testing protocols were the result of Aberdeen Test
Center's incorrectly interpreting the testing protocols. Although
Aberdeen Test Center officials told us that any deviations from the
testing protocols required approval from PEO Soldier, the office of the
Director of Operational Test and Evaluation, and other activities, in
all these cases the Aberdeen Test Center procedures implemented were
not reviewed or approved by officials from PEO Soldier, the Director of
Operational Test and Evaluation, and other activities responsible for
approving the testing protocols. Furthermore, PEO Soldier
representatives were not present at Aberdeen Test Center during most of
the testing, an absence that may have contributed to the fact that
these deviations were not identified earlier during testing. PEO
Soldier officials told us that they were not present at testing in
order to ensure the independence of the testing facility, but they
later acknowledged that they should have been more involved in that
testing and would be more involved in future testing.[Footnote 9] After
concerns were raised regarding the testing conducted at Aberdeen Test
Center under the May 2007 solicitation, the decision was made to not
field any of the of the ESAPI and XSAPI plates awarded contracts as a
result of Preliminary Design Model testing until after First Article
Testing was conducted.
During First Article Testing, while the Army addressed some of the
problems identified during Preliminary Design Model testing, we
observed instances in which Army testers did not follow the established
testing protocols and did not maintain internal controls over the
integrity and reliability of test data, raising questions as to whether
the Army met its First Article Testing objective of determining whether
each of the contracted armor plate designs met performance
requirements. The Army resolved the problems with shot location and
velocity and with the ease-of-insertion test. Also, Army technical
experts from PEO Soldier who served on the Integrated Product Team were
charged with testing oversight and maintained an on-site presence in
the test lanes. However, Army testers continued to deviate from
established testing protocols with respect to clay calibration and
back- face deformation measurement as follows:
* For the clay calibration test, the Army testers followed an orally
agreed-upon set of procedures that deviated from the established
testing protocols. Specifically, Army testers used clay in testing that
had failed the initial clay calibration test. The use of clay that has
failed the calibration test could significantly impact test results.
This was especially significant on a day with high failure rates when
we observed clay being exposed to constant heavy, cold rain. The
established testing protocols require the use of a specific type of
non- hardening oil-based clay. Officials from the Army, private NIJ-
certified ballistics laboratories, and the clay manufacturer told us
that water exposure may contaminate the clay by changing its chemical
bonding characteristics as well as by causing rapid and uneven cooling,
which could affect test results. Although Army Test and Evaluation
Command officials said covering the clay was not required and its
exposure to water would not impact testing, these officials were unable
to provide any documentation to support their position, raising
concerns that exposure to rain may have impacted the testing results.
* Army testers rounded down back-face deformation measurements, which
is not authorized in established testing protocols or consistent with
their testing practice during Preliminary Design Model testing. Army
officials said that rounding is a common industry practice and that
they should have also rounded Preliminary Design Model testing results.
While we did not validate this assertion, officials we spoke with from
one private industry ballistics testing facility said that their
practice was to always round results up, not down, which has the same
effect as not rounding. As a result of the rounding, two designs passed
First Article Testing that would have failed if the measurements had
not been rounded.
* The Army used a laser scanner as a new method to measure back-face
deformation without adequately certifying that the scanner could
function: (1) in its operational environment, (2) at the required
accuracy, (3) in conjunction with its software package, or (4) without
overstating deformation measurements. Army officials told us they are
unable to estimate the accuracy of the laser scanner used in testing,
raising concerns regarding the reliability of back-face deformation
results. Aberdeen Test Center officials said they initially decided to
use the laser because they did not believe it was possible to measure
back-face deformations to the required level of accuracy using the
digital caliper. However, officials from PEO Soldier and private NIJ-
certified laboratories have told us that they believe the digital
caliper method is capable of making these measurements and that the
back-face deformation measurements in the testing protocols were
developed using a digital caliper.[Footnote 10] While it is uncertain
what impact this issue had on the test results, the reliability and
accuracy of the measurements may be called into question.
During First Article Testing, Army testers deviated from the
established testing protocols by improperly scoring a complete
penetration as a partial penetration. Army testers said they used a
method to evaluate the penetration results that was discussed
internally before First Article Testing but that was not described in
the testing protocols or otherwise documented. As a result of this
incident, one design passed First Article Testing that would have
otherwise failed.[Footnote 11] With respect to internal control issues,
the Army did not consistently maintain adequate internal controls to
ensure the integrity and reliability of test data. In one example,
during ballistic testing, data were lost and testing had to be repeated
because an official accidentally hit the delete button and software
controls were not in place to protect the integrity of test data.
Federal internal control standards require that federal agencies
maintain effective controls over information processing to help ensure
completeness, accuracy, authorization, and validity of all
transactions. Army officials acknowledged that before our review they
were unaware of the specific internal control problems we identified.
As a result of the deviations from testing protocols that we observed,
three of the five designs that passed First Article Testing would not
have passed under the existing testing protocols and one of the
remaining two designs that passed would have failed Preliminary Design
Model testing if those testing protocols had been fully followed. Thus,
four of the five designs that passed First Article Testing and were
certified by the Army as ready for full production would have instead
failed testing at some point during the process, either during the
initial Preliminary Design Model testing or the subsequent First
Article testing, if the established testing protocols had been fully
followed. As a result, the overall reliability and repeatability of the
test results are uncertain. Although designs passed testing that would
not have if the testing protocols had been followed, ballistics experts
have not assessed the impact of the deviations from the testing
protocols to determine if their effect is sufficient to call into
question the ability of those designs to meet mission requirements. The
Army has ordered 2,500 sets of plates (at two plates per set) from
those vendors whose designs passed First Article Testing to be used for
additional ballistics testing and 120,000 sets of plates to be put into
inventory to address future requirements. However, to date, none of
these designs have been fielded because, according to Army officials,
there are adequate quantities of armor plates produced under prior
contracts already in the inventory to meet current requirements.
To help ensure that test results are reliable, we are recommending that
before any body armor plates are fielded to soldiers under the May 2007
solicitation, an assessment of the First Article Testing test data be
performed by independent experts to determine whether the issues we
identified had a significant effect on the test results. We are also
making several recommendations intended to improve the transparency of
testing by fully documenting any revised test practices so that their
alignment with testing protocols is clear. Finally, we are making
several recommendations to address the specific inconsistencies in test
conditions we observed and to improve internal controls.
In written comments on a draft of this report, DOD generally concurred
with our finding that there were deviations from the established
testing protocols during Preliminary Design Model testing and First
Article Testing and with our recommendations to fully document revised
test practices in the testing protocols and to improve internal
controls over testing. However, DOD did not concur with our
recommendation that an independent expert assessment of First Article
Testing data be performed before any body armor plates are fielded to
soldiers under contracts awarded under this solicitation. In the
comments, DOD wrote that the deviations we identified have no
significant impact on the test results and the subsequent contracting
actions taken by the Army based on these test results. We disagree with
DOD's assertions in this regard and continue to state that such an
independent assessment is necessary to ensure that the body armor
plates meet all protection requirements. We were unable to determine
the full effects of deviations we observed as they relate to the
quality of the armor designs and believe that such a determination
should only be made based on a thorough assessment of the testing data
by independent ballistics-testing experts. In light of such uncertainty
and the critical need for confidence in the equipment by the soldiers,
the Army would be taking unacceptable risk if it were to field these
armor designs without taking additional steps to gain the needed
confidence that the armor will perform as required. Consequently, we
have added a matter for congressional consideration to our report
suggesting that Congress consider directing DOD to either require that
an independent external review of these body armor test results be
conducted or require that DOD officially amend its testing protocols to
reflect any revised test procedures and repeat First Article Testing to
ensure that designs are properly tested. DOD's written comments are
reprinted in appendix II.
Background:
Army Solicitation for Body Armor:
In May 2007, the Army issued a solicitation for body armor designs to
replenish stocks and to protect against future threats by developing
the next generation (X level) of protection. According to Army
officials, the solicitation would result in contracts that the Army
would use for sustainment of protective plate stocks for troops in Iraq
and Afghanistan. The indefinite delivery/indefinite quantity contracts
require the Army to purchase a minimum of 500 sets per design and allow
for a maximum purchase of 1.2 million sets over the 5-year period.
[Footnote 12]
The Army's solicitation, which closed in February 2008, called for
preliminary design models in four categories of body armor protective
plates:
* Enhanced Small Arms Protective Insert (ESAPI)--plates designed to
same protection specifications as those currently fielded and to fit
into currently fielded Outer Tactical Vests.
* Flexible Small Arms Protective Vest-Enhanced (FSAPV-E)--flexible
armor system designed to same protection specifications as armor
currently fielded.
* Small Arms Protective Insert-X level (XSAPI)--next-generation plates
designed to defeat higher level threat.
* Flexible Small Arms Protective Vest-X level (FSAPV-X)--flexible armor
system designed to defeat higher level threat.
In figure 1, we show the ESAPI plates inside the Outer Tactical Vest.
Figure 1: ESAPI Plates as Worn inside Outer Tactical Vest:
[Refer to PDF for image: illustration]
Source: Army.
[End of figure]
Between May of 2007 and February of 2008 the Army established testing
protocols, closed the solicitation, and provided separate live-fire
demonstrations of the testing process to vendors who submitted items
for testing and to government officials overseeing the testing.
Preliminary Design Model testing was conducted at Aberdeen Test Center
between February 2008 and June 2008[Footnote 13] at an estimated cost
of $3 million. Additionally, over $6 million was spent on
infrastructure and equipment improvements at Aberdeen Test Center to
support future light armor test range requirements, including body
armor testing.
First Article Testing was then conducted at Aberdeen Test Center from
November 10, 2008, to December 17, 2008,[Footnote 14] on the three
ESAPI and five XSAPI designs that had passed Preliminary Design Model
testing.[Footnote 15] First Article Testing is performed in accordance
with the Federal Acquisition Regulation to ensure that the contractor
can furnish a product that conforms to all contract requirements for
acceptance. First Article Testing determines whether the proposed
product design conforms to contract requirements before or in the
initial stage of production. During First Article Testing, the proposed
design is evaluated to determine the probability of consistently
demonstrating satisfactory performance and the ability to meet or
exceed evaluation criteria specified in the purchase description.
Successful First Article Testing certifies a specific design
configuration and the manufacturing process used to produce the test
articles. Failure of First Article Testing requires the contractor to
examine the specific design configuration to determine the improvements
needed to correct the performance of subsequent designs.
Testing of the body armor currently fielded by the Army was conducted
by private NIJ-certified testing facilities under the supervision of
PEO Soldier. According to Army officials, not a single death can be
attributed to this armor's failing to provide the required level of
protection for which it was designed. However, according to Army
officials, one of the body armor manufacturers that had failed body
armor testing in the past did not agree with the results of the testing
and alleged that the testers tested that armor to higher-than-required
standards. The manufacturer alleged a bias against its design and
argued that its design was superior to currently fielded armor. As a
result of these allegations and in response to congressional interest,
after the June 2007 House Armed Services Committee hearing, the Army
accelerated completion of the light armor ranges to rebuild small arms
ballistic testing capabilities at Aberdeen Test Center and to conduct
testing under the May 2007 body armor solicitation there, without
officials from PEO Soldier supervising the testing. Furthermore, the
decision was made to allow Aberdeen Test Center, which is not an NIJ-
certified facility, to be allowed to conduct the repeated First Article
Testing. In February 2009 the Army directed that all future body armor
testing be performed at Aberdeen Test Center. According to Army
officials, as of this date, none of the body armor procured under the
May 2007 solicitation had been fielded.
Given the significant congressional interest in the testing for this
solicitation and that these were the first small arms ballistic tests
conducted at Aberdeen Test Center in years, multiple defense
organizations were involved in the Preliminary Design Model testing.
These entities include the Aberdeen Test Center, which conducted the
testing; PEO Soldier, which provided the technical subject-matter
experts; and DOD's office of the Director of Operational Test and
Evaluation, which combined to form the Integrated Product The
Integrated Product Team was responsible for developing and approving
the test plans used for the Preliminary Design Model testing and First
Article Testing. Figure 2 shows a timeline of key Preliminary Design
Model testing and First Article Testing events.
Figure 2: Timeline of Key Preliminary Design Model Testing and First
Article Testing Events:
[Refer to PDF for image: timeline]
5/25/07: Solicitation issued;
6/6/07: House Armed Services Committee Hearing;
6/14/07: Purchase descriptions signed;
9/11/07: Detailed test plans signed;
2/7/08: Solicitation closed;
2/20-21/08: Live fire demonstration for vendors;
2/08: Start of Preliminary Design Model testing;
3/26/08: Testing halted due to back face deformation issue;
4/10/08: Testing resumed;
6/08: End of Preliminary Design Model testing;
8/08: Source selections made;
11/6/08: Start of First Article Testing;
11/14/08: First Article Testing halted;
11/19/08: First Article Testing resumed;
12/17/08: End of First Article Testing.
Source: GAO observation and Army data.
[End of figure]
Body Armor Test Procedures:
The test procedures to be followed for Preliminary Design Model testing
were established and identified in the purchase descriptions
accompanying the solicitation announcement and in the Army's detailed
test plans (for each of the four design categories), which served as
guidance to Army testers and were developed by the Army Test and
Evaluation Command and approved by PEO-Soldier, DOD's office of the
Director of Operational Test and Evaluation, and others. Originally,
PEO Soldier required that testing be conducted at an NIJ-certified
facility. Subsequently, the decision was made to conduct testing at
Aberdeen Test Center, which is not NIJ-certified.[Footnote 16]
The test procedures for both Preliminary Design Model testing and First
Article Testing included both (1) physical characterization steps
performed on each armor design to ensure they met required
specifications, which included measuring weight, thickness, curvature,
and size and (2) ballistic testing performed on each design. Ballistics
testing for this solicitation included the following subtests: (1)
ambient testing to determine whether the designs can defeat the
multiple threats assigned in the respective solicitation's purchase
descriptions 100 percent of the time; (2) environmental testing of the
designs to determine whether they can defeat each threat 100 percent of
the time after being exposed to nine different environmental
conditions; and (3) testing, called V50 testing, to determine whether
designs can defeat each threat at velocities significantly higher than
those present or expected in Iraq or Afghanistan at least 50 percent of
the time.
Ambient and environmental testing seek to determine whether designs can
defeat each threat 100 percent of the time by both prohibiting the
bullet from penetrating through the plate and by prohibiting the bullet
from causing too deep of an indentation in the clay backing behind the
plate. Preventing a penetration is important because it prevents a
bullet from entering the body of the soldier. Preventing a deep
indentation in the clay (called "back-face deformation") is important
because the depth of the indentation indicates the amount of blunt
force trauma to the soldier. Back-face deformation deeper than 43
millimeters puts the soldier at higher risk of internal injury and
death.
The major steps taken in conducting a ballistic subtest include:
1. For environmental subtests, the plate is exposed to the
environmental condition tested (e.g., impact test, fluid soaks,
temperature extremes, etc.).
2. The clay to be used to back the plate is formed into a mold and is
placed in a conditioning chamber for at least 3 hours.
3. The test plate is placed inside of a shoot pack.
4. The clay is taken out of the conditioning chamber. It is then tested
to determine if it is suitable for use[Footnote 17] and, if so, is
placed behind the test plate.
5. The armor and clay are then mounted to a platform and shot.
6. If the shot was fired within required specifications,[Footnote 18]
the plate is examined to determine if there is a complete or partial
penetration, and the back-face deformation is measured.
7. The penetration result and back-face deformation are scored as a
pass,[Footnote 19] a limited failure,[Footnote 20] or a catastrophic
failure.[Footnote 21] If the test is not conducted according to the
testing protocols, it is scored as a no- test.
Army Took Significant Steps during Preliminary Design Model Testing to
Run a Controlled Test and Maintain Consistency but Did Not Consistently
Follow Established Testing Protocols and, as a Result, Did Not Achieve
the Intended Test Objective:
Army Took Significant Steps to Run a Controlled Test and Maintain
Consistency:
Following are significant steps the Army took to run a controlled test
and maintain consistency throughout Preliminary Design Model testing:
* The Army developed testing protocols for the hard-plate (ESAPI and
XSAPI) and flexible-armor (FSAPV-E and FSAPV-X) preliminary design
model categories in 2007. These testing protocols were specified in
newly created purchase descriptions, detailed test plans, and other
documents. For each of the four preliminary design model categories,
the Army developed new purchase descriptions to cover both hard-plate
and flexible designs. These purchase descriptions listed the detailed
requirements for each category of body armor in the solicitation issued
by the Army. Based on these purchase descriptions, the Army developed
detailed test plans for each of the four categories of body armor.
These detailed test plans provided additional details on how to conduct
testing and provided Army testers with the requirements that each
design needed to pass. After these testing protocols were developed,
Army testers then conducted a pilot test in which they practiced test
activities in preparation for Preliminary Design Model testing, to help
them better learn and understand the testing protocols.
* The Army consistently documented many testing activities by using
audio, video, and other electronic means. The use of cameras and
microphones to provide 24-hour video and audio surveillance of all of
the major Preliminary Design Model testing activities provided
additional transparency into many testing methods used and allowed for
enhanced oversight by Army management, who are unable to directly
observe the lanes on a regular basis but who wished to view select
portions of the testing. The Army utilized an electronic database to
maintain a comprehensive set of documentation for all testing
activities. This electronic database included a series of data reports
and pictures for each design including: physical characterization
records, X-ray pictures, pre-and post-shot pictures, ballistics testing
results, and details on the condition of the clay backing used for the
testing of those plates.
The Army took a number of additional actions to promote a consistent
and unbiased test. For example, the Army disguised vendor identity for
each type of solution by identifying vendors with random numbers to
create a blind test. The Army further reduced potential testing
variance by shooting subtests in the same shooting lane. The Army also
made a good faith effort to use consistent and controlled procedures to
measure the weight, thickness, and curvature of the plates.
Additionally, the Army made extensive efforts to consistently measure
and maintain room temperature and humidity within desired ranges. We
also observed that projectile yaw[Footnote 22] was consistently
monitored and maintained. We also found no deviations in the monitoring
of velocities for each shot and the re- testing of plates in cases
where velocities were not within the required specifications. We
observed no instances of specific bias against any design, nor did we
observe any instances in which a particular vendor was singled out for
advantage or disadvantage.
Army Did Not Consistently Follow All Testing Protocols:
We identified several instances in which the Aberdeen Test Center did
not follow established testing protocols. For example, during V50
testing, testers failed to properly adjust shot velocities. V50 testing
is conducted to discern the velocity at which 50 percent of the shots
of a particular threat would penetrate each of the body armor designs.
The testing protocols require that after every shot that is defeated by
the body armor the velocity of the next shot be increased. Whenever a
shot penetrates the armor, the velocity should be decreased for the
next shot. This increasing and decreasing of the velocities is supposed
to be repeated until testers determine the velocity at which 50 percent
of the shots will penetrate. In cases in which the armor far exceeds
the V50 requirements and is able to defeat the threat for the first six
shots, the testing may be halted without discerning the V50 for the
plate, and the plate is ruled as passing the requirements. During
Preliminary Design Model testing, in cases in which plates defeated the
first three shots, Army testers failed to increase shot velocities, but
rather continued to shoot at approximately the same velocity or lower
for shots four, five, and six in order to obtain six partial
penetrations and conclude the test early. Army officials told us that
this deviation was implemented by Aberdeen Test Center to conserve
plates for other tests that needed repeating as a result of no-test
events, according to Aberdeen Test Center officials-- but was a
practice not described in the protocols. Army officials told us that
this practice had no effect on which designs passed or failed; however,
this practice made it impossible to discern the true V50s for these
designs and was a deviation from the testing protocols that require
testers to increase velocities for shots after the armor defeats the
threat.
In another example, Aberdeen Test Center testers did not consistently
follow testing protocols in the ease-of-insertion test. According to
the testing protocols, one barehanded person shall demonstrate
insertion and removal of the ESAPI/XSAPI plates in the Outer Tactical
Vest[Footnote 23] pockets without tools or special aids. Rather than
testing the insertion of both the front and the rear pockets as
required, testers only tested the ability to insert into the front
pocket. Testing officials told us that they did not test the ability to
insert the plates into the rear pocket because they were unable to
reach the rear pocket while wearing the Outer Tactical Vest. The cause
for this deviation is that the testers misinterpreted the testing
protocols, as there is no requirement in the established testing
protocols to wear the Outer Tactical Vest when testing the ability to
insert the plates in the rear pocket of the Outer Tactical Vest.
Officials from PEO Soldier told us that, had they been present to
observe this deviation during testing, they would have informed testers
that the insertion test does not require that the Outer Tactical Vest
be worn, which would have resulted in testers conducting the insertion
test as required. According to Aberdeen Test Center officials, this
violation of the testing protocols had no impact on test results. While
we did not independently verify this assertion, Aberdeen Test Center
officials told us that the precise physical characterization
measurements of the plate's width and dimensions are, alone, sufficient
to ensure the plate will fit.
In addition, testers deviated from the testing protocols by placing
shots at the wrong location on the plate. The testing protocols require
that the second shot for one of the environmental sub- tests, called
the impact test, be taken approximately 1.5 inches from the edge of the
armor. However, testers mistakenly aimed closer to the edge of the
armor for some of the designs tested. Army officials said that the
testing protocols were unclear for this test because they did not
prescribe a specific hit zone (e.g., 1.25 - 1.75 inches), but rather
relied upon testers' judgment to discern the meaning of the word
"approximately." One of the PEO Soldier technical advisors on the
Integrated Product Team told us he was contacted by the Test Director
after the plates had been shot and asked about the shot location. He
told us that he informed the Test Director that the plates had been
shot in the wrong location. The PEO Soldier Technical advisor told us
that, had he been asked about the shot location before the testing was
conducted, he could have instructed testers on the correct location at
which to shoot. For 17 of the 47 total designs that we observed and
measured,[Footnote 24] testers marked target zones that were less than
the required 1.5 inches from the plate's edge, ranging from .75 inches
to 1.25 inches from the edge. Because 1.5 inches was outside of the
marked aim area for these plates, we concluded that testers were not
aiming for 1.5 inches. For the remaining 30 designs tested that we
observed and measured, testers used a range that included 1.5 inches
from the edge (for example, aiming for 1 to 1.5 inches). It is not
clear what, if any, effect this deviation had on the overall test
results. While no design failed Preliminary Design Model testing due to
the results of this subtest, there is no way to determine if a passing
design would have instead failed if the testing protocol had been
correctly followed. However, all designs that passed this testing were
later subject to First Article Testing, where these tests were repeated
in full using the correct shot locations.[Footnote 25]
Of potentially greater consequence to the final test results is our
observation of deviations from testing protocols regarding the clay
calibration tests. According to testing protocols, the calibration of
the clay backing material was supposed to be accomplished through a
series of pre-test drops.[Footnote 26] The depths of the pre-test drops
should have been between 22 and 28 millimeters. Aberdeen Test Center
officials told us that during Preliminary Design Model testing they did
not follow a consistent system to determine if the clay was conditioned
correctly. According to Aberdeen Test Center officials, in cases in
which pre-test drops were outside the 22- to 28-millimeter range,
testers would sometimes repeat one or all of the drops until the
results were within range-- thus resulting in the use of clay backing
materials that should have been deemed unacceptable for use. These
inconsistencies occurred because Army testers in each test lane made
their own, sometimes incorrect, interpretation of the testing
protocols. Members of the Integrated Product Team expressed concerns
about these inconsistencies after they found out how calibrations were
being conducted. In our conversations with Army and private body armor
testing officials, consistent treatment and testing of clay was
identified as critical to ensure consistent, accurate testing.
According to those officials if the clay is not conditioned correctly
it will impact the test results. Given that clay was used during
Preliminary Design Model testing that failed the clay calibration
tests, it is possible that some shots may have been taken under test
conditions different than those stated in the testing protocols,
potentially impacting test results. Figure 3 shows an Army tester
calibrating the clay with pre- test drops.
Figure 3: Clay Being Calibrated with Pre-Test Drops:
[Refer to PDF for image: photograph]
Source: Army.
[End of figure]
The most consequential of the deviations from testing protocols we
observed involved the measurement of back-face deformation, which did
affect final test results. According to testing protocol, back-face
deformation is to be measured at the deepest point of the depression in
the clay backing. This measure indicates the most force that the armor
will allow to be exerted on an individual struck by a bullet. According
to Army officials, the deeper the back-face deformation measured in the
clay backing, the higher the risk of internal injury or death. During
approximately the first one-third of testing, however, Army testers
incorrectly measured deformation at the point of aim, rather than at
the deepest point of depression. This is significant because, in many
instances, measuring back-face deformation at the point of aim results
in measuring at a point upon which less ballistic force is exerted,
resulting in lower back-face deformation measurements and
overestimating the effectiveness of the armor. The Army's subject
matter experts on the Integrated Product Team were not on the test
lanes during testing and thus not made aware of the error until
approximately one-third of the testing had been completed.
When members of the Integrated Product Team overseeing the testing were
made aware of this error, the Integrated Product Team decided to begin
measuring at the deepest point of depression. When senior Army
leadership was made aware of this error, testing was halted for 2 weeks
while Army leadership considered the situation. Army leadership
developed many courses of action, including restarting the entire
Preliminary Design Model testing with new armor plate submissions, but
ultimately decided to continue measuring and scoring officially at the
point of aim, since this would not disadvantage any vendors. The Army
then changed the test plans and modified the contract solicitation to
call for measuring at the point of aim. The Army also decided to
collect deepest point of depression measurements for all shots from
that point forward, but only as a government reference. During the
second two-thirds of testing, we observed significant differences
between the measurements taken at the point of aim and those taken at
the deepest point, as much as a 10-millimeter difference between
measurements. As a result, at least two of the eight designs that
passed Preliminary Design Model testing and were awarded contracts
would have failed if the deepest point of depression measurement had
been used. Figures 4 and 5 illustrate the difference between the point
of aim and the deepest point.
Figure 4: Graphic Representation of the Difference between the Point of
Aim and the Deepest Point:
[Refer to PDF for image: illustration]
The following are depicted on the illustration:
Air laser:
Shooting barrel:
Point of aim:
Armor plate:
Clay backing:
Guide plane:
Back-face deformation:
Point of aim depression:
Deepest point of depression:
Source: GAO analysis.
[End of figure]
Figure 5: Photographic Representation of the Difference between the
Point of Aim and the Deepest Point:
[Refer to PDF for image: photograph]
Source: Army.
[End of figure]
Army Decided to Repeat Testing in First Article Testing in an Attempt
to Address Back-Face Deformation Measurement Problem Identified during
Preliminary Design Model Testing:
Before Preliminary Design Model testing began at Aberdeen Test Center,
officials told us that Preliminary Design Model testing was
specifically designed to meet all the requirements of First Article
Testing. However, Preliminary Design Model testing failed to meet its
goal of determining which designs met requirements, because of the
deviations from established testing protocols described earlier in this
report. Those deviations were not reviewed or approved by officials
from PEO Soldier, the office of the Director of Operational Test and
Evaluation, or by the Integrated Product Team charged with overseeing
the test. PEO Soldier officials told us that the reason for a lack of
PEO Soldier on-site presence during this testing was because of a
deliberate decision made by PEO Soldier management to be as removed
from the testing process as possible in order to maximize the
independence of the Aberdeen Test Center. PEO Soldier officials told us
that it was important to demonstrate the independence of the Aberdeen
Test Center to quash allegations of bias made by a vendor whose design
had failed prior testing and that this choice may have contributed to
some of the deviations not being identified by the Army earlier during
testing. After the conclusion of Preliminary Design Model testing, PEO
Soldier officials told us that they should have been more involved in
the testing and that they would be more involved in future testing.
After the completion of Preliminary Design Model testing, the
Commanding General of PEO Soldier said that, as the Milestone Decision
Authority[Footnote 27] for the program, he elected to repeat the
testing conducted during Preliminary Design Model testing through First
Article Testing before any body armor was fielded based on the
solicitation. According to PEO Soldier officials, at the beginning of
Preliminary Design Model testing, there was no intention or plan to
conduct First Article Testing following contract awards given that the
Preliminary Design Model testing was to follow the First Article
Testing protocol. However, because of the fact that back-face
deformation was not measured to the deepest point, PEO-Soldier and Army
Test and Evaluation and Command acknowledged that there was no longer
an option of forgoing First Article Testing. PEO Soldier also expressed
concerns that Aberdeen Test Center test facilities have not yet
demonstrated that they are able to test to the same level as NIJ-
certified facilities. However, officials from Army Test and Evaluation
Command and DOD's office of the Director of Operational Test and
Evaluation asserted that Aberdeen Test Center was just as capable as
NIJ-certified laboratories, and Army leadership eventually decided that
First Article Testing would be performed at Aberdeen.
During First Article Testing the Army Addressed Some of the Problems
Identified in Preliminary Design Model Testing, but Army Testers Did
Not Always Follow Established Testing Protocols and Did Not Maintain
Some Internal Controls:
During First Article Testing, the Army Addressed Some of the Problems
Identified during Preliminary Design Model Testing:
PEO Soldier maintained an on-site presence in the test lanes and the
Army technical experts on the Integrated Product Team charged with
testing oversight resolved the following problems during First Article
Testing:
* The Army adjusted its testing protocols to clarify the required shot
location for the impact test, and Army testers correctly placed these
shots as required by the protocols.
* After the first few days of First Article Testing, in accordance with
testing protocols, Army testers began to increase the velocity after
every shot defeated by the armor required during V50 testing.
* As required by the testing protocols, Army testers conducted the
ease- of-insertion tests for both the front and rear pockets of the
outer protective vest, ensuring that the protective plates would
properly fit in both pockets.
The Army began to address the problems identified during Preliminary
Design Model testing with the clay calibration tests and back-face
deformation measurements. Army testers said they developed an informal
set of procedures to determine when to repeat failed clay calibration
tests. The procedures, which were not documented, called for repeating
the entire series of clay calibration drops if one of the calibration
drops showed a failure.[Footnote 28] If the clay passes either the
first or second test, the clay is to be used in testing. If the clay
fails both the first and the second series of drops, the clay is to
then be placed back in conditioning and testers get a new block of
clay. With respect to back-face deformation measurements, Army testers
measured back-face deformation at the deepest point, rather than at the
point of aim.[Footnote 29]
Army Did Not Follow All Established Testing Protocols during First
Article Testing:
Although the Army began to address problems relating to the clay
calibration tests and back-face deformation measurements, Army testers
still did not follow all established testing protocols in these areas.
As a result, the Army may not have achieved the objective of First
Article Testing--to determine if the designs tested met the minimum
requirements for ballistic protection. First, the orally agreed-upon
procedures used by Army testers to conduct the clay calibration tests
were inconsistent with the established testing protocols. Second, with
respect to back-face deformation measurements, Army testers rounded
back-face deformation measurements to the nearest millimeter, a
practice that was neither articulated in the testing protocols nor
consistent with Preliminary Design Model testing. Third, also with
respect to back-face deformation measurements, Army testers introduced
a new, unproven measuring device.
Although Army testers told us that they had orally agreed upon an
informal set of procedures to determine when to repeat failed clay
calibration tests, those procedures are inconsistent with the
established testing protocols. The Army deviated from established
testing protocols by using clay that had failed the calibration test as
prescribed by the testing protocols. The testing protocols specify that
a series of three pre- test drops of a weight on the clay must be
within specified tolerances before the clay is used. However, in
several instances, the Army repeated the calibration test on the same
block of clay after it had initially failed until the results of a
subsequent series of three drops were within the required
specifications. Army officials told us that the testing protocols do
not specify what procedures should be performed when the clay does not
pass the first series of calibration drops, so Army officials stated
they developed the procedure they followed internally prior to First
Article Testing and provided oral guidance on those procedures to all
test operators to ensure a consistent process. Officials we spoke with
from the Army, private NIJ-certified laboratories, and industry had
mixed opinions regarding the practice of re-testing failed clay, with
some expressing concerns that performing a second series of calibration
drops on clay that had failed might introduce risk that the clay may
not be at the proper consistency for testing because as the clay rests
it cools unevenly, which could affect the
calibration.[Footnote 30] Aberdeen Test Center's Test Operating
Procedure states that clay should be conditioned so that the clay
passes the clay calibration test, and Army officials, body armor
testers from private laboratories, and body armor manufacturers we
spoke to agreed that when clay fails the calibration test, this
requires re-evaluation and sometimes adjustment of the clay calibration
procedures used. After several clay blocks failed the clay calibration
test on November 13, 2008, Army testers recognized that the clay
conditioning process used was yielding clay that was not ideal and, as
a result, Army testers adjusted their clay conditioning process by
lowering the temperature at which the clay was stored.
On that same day of testing, November 13, 2008, we observed heavy, cold
rain falling on the clay blocks that were being transported to test
lanes. These clay blocks had been conditioned that day in ovens located
outside of the test ranges at temperatures above 100 degrees Fahrenheit
to prepare them for testing, and then were transported outside
uncovered on a cold November day through heavy rain on the way to the
temperature-and humidity-controlled test lane. We observed an
abnormally high level of clay blocks failing the clay calibration test
and a significantly higher-than-normal level of failure rates for the
plates tested on that day. The only significant variation in the test
environment we observed that day was constant heavy rain throughout the
day. Our analysis of test data[Footnote 31] also showed that 44 percent
(4 of 9) of the first shots and 89 percent (8 of 9) of the second shots
taken on November 13, 2008, resulted in failure penalties.[Footnote 32]
On all of the other days of testing only 14 percent (10 of 74) of the
first shots and 42 percent (31 of 74) of the second shots resulted in
failure penalties. Both of these differences are statistically
significant, and we believe the differences in the results may be
attributable to the different test condition on that day. The
established testing protocols require the use of a specific type of non-
hardening oil-based clay.[Footnote 33] Body armor testers from NIJ-
certified private laboratories, Army officials experienced in the
testing of body armor, body armor manufacturers, and the clay
manufacturer we spoke with said that the clay used for testing is a
type of sculpting clay that naturally softens when heat is added and
that getting water on the clay backing material could cause a chemical
bonding change on the clay surface.[Footnote 34] Those we spoke with
further stated that the cold water could additionally cause the outside
of the clay to cool significantly more rapidly than the inside causing
the top layer of clay to be harder than the middle. They suggested that
clay be conditioned inside the test lanes and said that clay exposed to
water or extreme temperature changes should not be used. Army Test and
Evaluation Command officials we spoke with said that there is no
prohibition in the testing protocols on allowing rain to fall onto the
clay backing material and that its exposure to water would not impact
testing. However, these officials were unable to provide data to
validate their assertion that exposure to water would not affect the
clay used during testing or the testing results. Army test officials
also said that, since the conclusion of First Article Testing, Aberdeen
Test Center has procured ovens to allow clay to be stored inside test
lanes, rather than requiring that the clay be transported from another
room where it would be exposed to environmental conditions, such as
rain.
With respect to the issue of the rounding of back-face deformation
measurements, during First Article Testing Army testers did not award
penalty points for shots with back-face deformations between 43.0 and
43.5 millimeters. This was because the Army decided to round back-face
deformation measurements to the nearest millimeter--a practice that is
inconsistent with the Army's established testing protocols, which
require that back-face deformation measurements in the clay backing not
exceed 43 millimeters and that is inconsistent with procedures followed
during Preliminary Design Model testing. Army officials said that a
decision to round the measurements for First Article Testing was made
to reflect testing for past Army contract solicitations and common
industry practices of recording measurements to the nearest millimeter.
[Footnote 35] While we did not validate this assertion that rounding
was a common industry practice, one private industry ballistics testing
facility said that its practice was to always round results up, not
down, which has the same effect as not rounding at all. Army officials
further stated that they should have also rounded Preliminary Design
Model results but did not realize this until March 2008--several weeks
into Preliminary Design Model testing--and wanted to maintain
consistency throughout Preliminary Design Model testing. The Army's
decision to round measurement results had a significant outcome on
testing because two designs that passed First Article Testing would
have instead failed if the measurements had not been rounded.
With respect to the introduction of a new device to measure back-face
deformation, the Army began to use a laser scanner to measure back-face
deformation without adequately certifying that the scanner could
measure against the standard established when the digital caliper was
used as the measuring instrument. Although Army Test and Evaluation
Command certified[Footnote 36] the laser scanner as accurate for
measuring back- face deformation, we observed the following
certification issues:
* The laser was certified based on testing done in a controlled
laboratory environment that is not similar to the actual conditions on
the test lanes. For example, according to the manufacturer of the laser
scanner, the scanner is operable in areas of vibration provided the
area scanned and the scanning-arm are on the same plane or surface.
[Footnote 37] This was not the case during testing, and thus it is
possible the impact of the bullets fired may have thrown the scanner
out of alignment or calibration.
* The certification is to a lower level of accuracy than required by
the testing protocols. The certification study says that the laser is
accurate to 0.2 millimeters; however, the testing protocols require an
accuracy of 0.1 millimeters or better. Furthermore, the official letter
from the Army Test and Evaluation Command certifying the laser for use
incorrectly stated the laser meets an accuracy requirement of 1.0
millimeter rather than 0.1 millimeters as required by the protocols.
Officials confirmed that this was not a typographical error.
* The laser certification was conducted before at least three[Footnote
38] major software upgrades were made to the laser, which according to
Army officials may have significantly changed the accuracy of the
laser. Because of the incorporation of the software upgrades, Army
testers told us that they do not know the accuracy level of the laser
as it was actually used in First Article Testing.
* In evaluating the use of the laser scanner, the Army did not compare
the actual back-face deformation measurements taken by the laser with
those taken by digital caliper, previously used during Preliminary
Design Model testing and by NIJ- certified laboratories. According to
vendor officials and Army subject matter experts, the limited data they
had previously collected have shown that back-face deformation
measurements taken by laser have generally been deeper by about 2
millimeters than those taken by digital caliper. Given those
preliminary findings, there is a significant risk that measurements
taken by the laser may represent a significant change in test
requirements.
Although Army testing officials acknowledged that they were unable to
estimate the exact accuracy of the laser scanner as it was actually
used during testing, they believed that based on the results of the
certification study, it was suitable for measuring back-face
deformation. These test officials further stated that they initially
decided to use the laser because they did not believe it was possible
to measure back-face deformations to the required level of accuracy
using the digital caliper. However, officials from PEO Soldier and
private NIJ- certified laboratories have told us that they believe the
digital caliper method is capable of making these measurements with the
required level of accuracy[Footnote 39] and have been using this
technique successfully for several years. PEO Soldier officials also
noted that the back-face deformation measurements in the testing
protocols were developed using this digital caliper method. Army
testing officials noted that the laser certification study confirmed
their views that the laser method was more accurate than the digital
caliper. However, because of the problems with the study that we have
noted in this report, it is still unclear whether the laser is the most
appropriate and accurate technique for measuring back-face deformation.
Although we did not observe problems in the Army's determination of
penetration results during Preliminary Design Model testing, during
First Article Testing we observed that the Army did not consistently
follow its testing protocols in determining whether a shot was a
partial or a complete penetration. Army testing protocols require that
penalty points be awarded when any fragment of the armor material is
imbedded or passes into the soft under garment used behind the plate;
however, the Army did not score the penetration of small debris through
a plate as a complete penetration of the plate in at least one case
that we observed. In this instance, we observed small fragments from
the armor three layers deep inside the Kevlar backing behind the plate.
This shot should have resulted in the armor's receiving 1.5 penalty
points, which would have caused the design to fail First Article
Testing.[Footnote 40] Army officials said that testers counted the shot
as only a partial penetration of the plate because it was determined
that fibers of the Kevlar backing placed behind the plate were not
broken,[Footnote 41] which they stated was a requirement for the shot
to be counted as a complete penetration of the plate. This
determination was made with the agreement of an Army subject-matter
expert from PEO-Soldier present on the lane. However, the requirement
for broken fibers is inconsistent with the written testing protocols.
Army officials acknowledged that the requirement for broken fibers was
not described in the testing protocols or otherwise documented but said
that Army testers discussed this before First Article Testing began.
Figure 6 shows the tear in the fibers of the rear of the plate in
question.
Figure 6: Tears in Kevlar Backing Material after a Penetration of the
Plate:
[Refer to PDF for image: photograph]
Source: Army]
[End of figure]
Army Did Not Maintain Internal Controls over the Integrity and
Reliability of Test Data at All Times:
Federal internal control standards require that federal agencies
maintain effective controls over information processing to help ensure
completeness, accuracy, authorization, and validity of all
transactions.[Footnote 42] However, the Army did not consistently
maintain adequate internal controls to ensure the integrity and
reliability of its test data. For example, in one case bullet velocity
data were lost because the lane Test Director accidentally pressed the
delete button on the keyboard, requiring a test to be repeated.
Additionally, we noticed that the software being used with the laser
scanner to calculate back-face deformation measurements lacked
effective edit controls, which could potentially allow critical
variables to be inappropriately modified during testing. We further
observed a few cases in which testers attempted to memorize test data
for periods of time, rather than writing that data down immediately. In
at least one case, this practice resulted in the wrong data being
reported and entered into the test records.
Army Did Not Formally Document Significant Procedures That Deviated
from Established Testing Protocols or Assess the Impact of These
Deviations:
According to Army officials, decisions to implement those procedures
that deviated from testing protocols were reviewed and approved by
appropriate officials. However, these decisions were not formally
documented, the testing protocols were not modified to reflect the
changes, and vendors were not informed of the procedures. At the
beginning of testing, the Director of Testing said that any change to
the testing protocols has to be approved by several Army components;
however, the Army was unable to produce any written documentation
indicating approval of the deviations we observed by those components.
With respect to internal control issues, Army officials acknowledged
that before our review they were unaware of the specific internal
control problems we identified. We noted during our review that in
industry, as part of the NIJ certification process, an external peer
review process is used to evaluate testing processes and procedures of
ballistics testing facilities to ensure that effective internal
controls are in place. However, we found that the Aberdeen Test Center
has conducted no such reviews, a contributing factor to the Army's lack
of unawareness of the control problems we noted.
As a result of the deviations from testing protocols that we observed,
three of the five designs that passed First Article Testing would not
have passed under the existing testing protocols. Furthermore, one of
the remaining two designs that passed First Article Testing was a
design that would have failed Preliminary Design Model testing if back-
face deformation was measured in accordance with the established
protocols for that test. Thus, four of the five designs that passed
First Article Testing and were certified by the Army as ready for full
production would have instead failed testing at some point during the
process, either during the initial Preliminary Design Model testing or
the subsequent First Article testing, if all the established testing
protocols had been followed.[Footnote 43] As a result, the overall
reliability and repeatability of the test results are uncertain.
However, because ballistics experts from the Army or elsewhere have not
assessed the impact of the deviations from the testing protocols we
observed during First Article Testing, it is not certain whether the
effect of these deviations is sufficient to call into question the
ability of the armor to meet mission requirements. Although it is
certain that some armor passed testing that would not have if specific
testing protocols had been followed, it is unclear if there are
additional factors that would mean the armor still meets the required
performance specifications. For example, the fact that the laser
scanner used to measure back-face deformation may not be as accurate as
what the protocol requires may offset the effects of rounding down
back- face deformations. Likewise, it is possible that some of the
deviations that did not on their own have a visible effect on testing
results could, when taken together with other deviations, have a
combined effect that is greater. In our opinion, given the significant
deviations in the testing protocols, independent ballistics testing
expertise would be required to determine whether or not the body armor
designs procured under this solicitation provide the required level of
protection. The Army has ordered 2,500 sets of plates (at two plates
per set) from those vendors whose designs passed First Article Testing
to be used for additional ballistics testing and 120,000 sets of plates
to be put into inventory to address future requirements. However, to
date, none of these designs have been fielded because, according to
Army officials, there are adequate quantities of armor plates produced
under prior contracts already in the inventory to meet current
requirements.
Conclusions:
Body armor plays a critical role in protecting our troops, and the
testing inconsistencies we identified call into question the quality
and effectiveness of testing performed at Aberdeen Test Center. Because
we observed several instances in which actual test practices deviated
from the established testing protocols, it is questionable whether the
Army met its First Article Testing objectives of ensuring that armor
designs fully met Army's requirements before the armor is purchased and
used in the field. While it is possible that the testing protocol
deviations had no significant net effect or may have even resulted in
armor being tested to a more rigorous standard, it is also possible
that some deviations may have resulted in armor being evaluated against
a less stringent standard than required. We were unable to determine
the full effects of these deviations as they relate to the quality of
the armor designs and believe such a determination should only be made
based on a thorough assessment of the testing data by independent
ballistics testing experts. In light of such uncertainty and the
critical need for confidence in the equipment by the soldiers, the Army
would take an unacceptable risk if it were to field these designs
without taking additional steps to gain the needed confidence that the
armor will perform as required.
The Army is now moving forward with plans to conduct all future body
armor testing at Aberdeen Test Center. Therefore, it is essential that
the transparency and consistency of its program be improved by ensuring
that all test practices fully align with established testing protocols
and that any modifications in test procedures be fully reviewed and
approved by the appropriate officials, with supporting documentation,
and that the testing protocols be formally changed to reflect the
revised or actual procedures. Additionally, it is imperative that all
instrumentation, such as the laser scanner, used for testing be fully
evaluated and certified to ensure its accuracy and applicability to
body armor testing. Furthermore, it is essential that effective
internal controls over data and testing processes be in place. The body
armor industry has adopted the practice, through the NIJ certification
program, of using external peer reviews to evaluate and improve private
laboratories' test procedures and controls. This type of independent
peer review could be equally beneficial to the Aberdeen Test Center.
Without all of these steps, there will continue to be uncertainty with
regard to whether future testing data are repeatable and reliable and
can be used to accurately evaluate body armor designs. Until Aberdeen
Test Center has effectively honed its testing practices to eliminate
the types of inconsistencies we observed, concerns will remain
regarding the rigor of testing conducted at that facility.
Recommendations for Executive Action:
To determine what effect, if any, the problems we observed had on the
test data and on the outcomes of First Article Testing, we recommend
the Secretary of Defense direct the Secretary of the Army to provide
for an independent evaluation of the First Article Testing results by
ballistics and statistical experts external to DOD before any armor is
fielded to soldiers under this contract solicitation and that the Army
report the results of that assessment to the office of the Director of
Operational Test and Evaluation and the Congress. In performing this
evaluation, the independent experts should specifically evaluate the
effects of the following practices observed during First Article
Testing:
* the rounding of back-face deformation measurements;
* not scoring penetrations of material through the plate as a complete
penetration unless broken fibers are observed in the Kevlar backing
behind each plate;
* the use of the laser scanner to measure back-face deformations
without a full evaluation of its accuracy as it was actually used
during testing, to include the use of the software modifications and
operation under actual test conditions;
* the exposure of the clay backing material to rain and other outside
environmental conditions as well as the effect of high oven
temperatures during storage and conditioning; and:
* the use of an additional series of clay calibration drops when the
first series of clay calibration drops does not pass required
specifications.
To better align actual test practices with established testing
protocols during future body armor testing, we recommend that the
Secretary of the Defense direct the Secretary of the Army to document
all key decisions made to clarify or change the testing protocols. With
respect to the specific inconsistencies we identified between the test
practices and testing protocols, we recommend that the Secretary of the
Army, based on the results of the independent expert review of the
First Article Test results, take the following actions:
* Determine whether those practices that deviated from established
testing protocols during First Article Testing will be continued during
future testing and change the established testing protocols to reflect
those revised practices.
* Evaluate and re-certify the accuracy of the laser scanner to the
correct standard with all software modifications incorporated and
include in this analysis a side-by-side comparison of the laser
measurements of the actual back-face deformations with those taken by
digital caliper to determine whether laser measurements can meet the
standard of the testing protocols.
To improve internal controls over the integrity and reliability of test
data for future testing as well as provide for consistent test
conditions and comparable data between tests, we recommend that the
Secretary of Defense direct the Secretary of the Army to provide for an
independent peer review of Aberdeen Test Center's body armor testing
protocols, facilities, and instrumentation to ensure that proper
internal controls and sound management practices are in place. This
peer review should be performed by testing experts external to the Army
and DOD.
Matter for Congressional Consideration:
DOD did not concur with our recommendation for an independent
evaluation of First Article Testing results and accordingly plans to
take no action to provide such an assessment. DOD asserted that the
issues we identified do not alter the effects of testing. However,
based on our analysis and findings there is sufficient evidence to
raise questions as to whether the issues we identified had an impact on
testing results. As a result, we continue to believe it is necessary to
have an independent external expert review these test results and the
overall effect of the testing deviations we observed on those results
before any armor is fielded to military personnel. Without such an
independent review, the First Article Test results remain questionable,
undermining the confidence of the public and those who might rely on
the armor for protection. Consequently, Congress should consider
directing the Office of the Secretary of Defense to either require that
an independent external review of these body armor test results be
conducted or that DOD officially amend its testing protocols to reflect
any revised test procedures and repeat First Article Testing to ensure
that only properly tested designs are fielded.
Agency Comments and Our Evaluation:
In written comments on a draft of this report, DOD takes the position
that our findings had no significant impact on the test results and on
the subsequent contracting actions taken by the Army. DOD also does not
concur with what it perceives as our two overarching conclusions: (1)
that Preliminary Design Model testing did not achieve its intended
objective of determining, as a basis for contract awards, which designs
met performance requirements and (2) that First Article Testing may not
have met its objective of determining whether each of the contracted
plate designs met performance requirements. DOD commented that it
recognizes the importance of personal protection equipment such as body
armor and provided several examples of actions DOD and the Army have
taken to improve body armor testing. DOD generally concurred with our
findings that there were deviations from the testing protocols during
Preliminary Design Model testing and First Article Testing. We agree
that DOD has taken positive steps to improve its body armor testing
program and to address concerns raised by Congress and others. DOD also
concurred with our second recommendation to document all key decisions
made to clarify or change the testing protocols. DOD did not concur
with our first recommendation that an independent evaluation of First
Article Testing results be performed by independent ballistics and
statistical experts before any of the armor is fielded to soldiers
under contracts awarded under this solicitation. Similarly, DOD did not
agree with our conclusions that Preliminary Design Model testing did
not meet its intended objectives and that First Article Testing may not
have met its intended objectives. In supporting its position, DOD
cited, for example, that rounding back-face deformation measurements
during First Article Testing was an acceptable test practice because
rounding is a practice that has been used historically. It was the
intent of PEO Soldier to round back-face deformations for all testing
associated with this solicitation, and the Integrated Product Team
decided collectively to round back-face deformations during First
Article Testing. However, as stated in our report and acknowledged by
DOD, the rounding down of back-face deformations was not spelled out or
provided for by any of the testing protocol documents. Additionally, it
created an inconsistency between Preliminary Design Model testing,
where back-face deformations were not rounded down and in First Article
Testing, where back-face deformations were rounded down. Of greatest
consequence, rounding down back-face deformations lowered the
requirements that solutions had to meet to pass testing. Two solutions
passed First Article Testing because back-face deformations were
rounded down, meaning that the Army may be taking unacceptable risk if
plates are fielded without an additional, independent assessment by
experts.
DOD also did not agree with our finding that a penetration of a plate
was improperly scored. DOD did agree that figure 6, which shows the
tear in the Kevlar fibers of the rear of the plate in question, appears
to show evidence of a perforation and that an Aberdeen Test Center
ballistics subject matter expert found particles in the soft backing
material behind the plate. Nevertheless, DOD did not concur with our
finding because it asserted that no threads were broken on the first
layer of Kevlar. However, as we stated in the report, the protocols
define a complete penetration[Footnote 44] as having occurred when the
projectile, fragment of the projectile, or fragment of the armor
material is imbedded or passes into the soft under garment used behind
the protective inserts plates, not when threads of the Kevlar are
broken. The fragments found by the Aberdeen Test Center subject matter
expert, as well as the three frayed, tattered, and separated Kevlar
layers that we and Army testers observed, confirm our observations
during testing. DOD also stated that the first layer of soft armor
behind the plate under test serves as a witness plate during testing
and if that first layer of soft armor is not penetrated, as determined
by the breaking of threads on that first layer of soft armor, the test
shot is not scored as a complete penetration in accordance with the PEO
Soldier's scoring criteria. We disagree with DOD's position because the
protocols do not require the use of a "witness plate" during testing to
determine if a penetration occurred. If this shot would have been ruled
a complete penetration rather than a partial penetration, this design
would have accrued additional point deductions causing it to fail First
Article Testing.
DOD did not agree that the certification of the laser scanner was
inadequate and made several statements in defense of both the laser and
its certification. Among these is the fact that the laser removes the
human factor of subjectively trying to find the deepest point,
potentially pushing the caliper into the clay, and removing the need to
use correction factors, all of which we agree may be positive things.
However, we maintain that the certification of the laser was not
adequately performed. As indicated in the certification letter, the
laser was certified to a standard that did not meet the requirement of
the testing protocols. Additionally, DOD stated that software
modifications added to the laser after certification did not affect
measurements; however, Army testers told us on multiple occasions that
the modifications were designed to change the measurements reported by
the laser. DOD added that the scanner does not artificially overstate
back- face deformations and relies on the verified accuracy of the
scanner and the study involving the scanning of clay replicas to
support its claim. Based on our observations, the scanner was certified
to the wrong standard and the certification study was not performed in
the actual test environment using actual shots. DOD asserts that the
scanner does not overstate back-face deformations and that it does not
establish a new requirement. However, DOD cannot properly validate
these assertions without a side-by-side comparison of the laser scanner
and the digital caliper in their operational environment. Given the
numerous issues regarding the laser and its certification, we maintain
that its effect on First Article Testing should be examined by an
external ballistics expert.
DOD also stated that it did not agree with our finding that exposure of
the clay backing to heavy rain on one day may have affected test
results. DOD challenged our statistical analysis and offered its own
statistical analysis as evidence that it was the poor designs
themselves that caused unusual test results that day. We stand by our
analysis, in combination with statements made by DOD and non-DOD
officials with testing expertise and by the clay manufacturer, that
exposure of the clay to constant, heavy cold rain may have had an
effect on test results. Further, in analyzing the Army's statistical
analysis presented in DOD's comments, we did not find this information
to demonstrate that the designs were the factor in unusual test results
that day or that the rain exposure could not have had an effect on the
results. More detailed discussions of the Army's analysis and our
conclusions are provided in comments 13 and 24 of appendix II.
DOD partially disagreed that the use of an additional series of clay
calibration drops when the first series of drops were outside
specifications did not meet First Article Test requirements and added
that all clay used in testing passed the clay calibration in effect at
the time. However, we witnessed several clay calibration drops that
were not within specifications. These failed clay boxes were repaired,
re-dropped, and either used if they passed the subsequent drop
calibration series or discarded if they failed. The protocols only
allow for one series of drops per clay box, which is the methodology
that Army testers should have followed. DOD stated that NIJ standards
do permit the repeating of failed calibration drops. However, our
review of the NIJ standards[Footnote 45] reveals that there is no
provision that allows repeat calibration drops. DOD states in its
comments that NIJ standards are inappropriate for its test facilities,
stating that these standards are insufficient for the U.S. Army given
the expanded testing required to ensure body armor meets U.S. Army
requirements. NIJ standards were not the subject of our review, but
rather Aberdeen Test Center's application of the Army's current
solicitation's protocols during testing. Further, DOD acknowledged in
its comments that National Institute of Standards and Technology
officials recommended only one series of drops for clay calibration.
However, DOD stated that it will partner with the National Institute of
Standards and Technology to study procedures for clay calibration, to
include repeated calibration attempts, and document any appropriate
procedural changes, which we agree is a good step.
Based on our analyses as described in our report and in our above
responses to DOD's comments, we believe there is sufficient evidence to
raise questions as to whether the issues we identified had an impact on
testing results. As a result, we continue to believe that it is
necessary that DOD allow an independent external expert to review these
test results and the overall effect of DOD's deviations on those
results before any armor is fielded to military personnel. Without such
an independent review, it is our opinion that the First Article Testing
results will remain questionable. Consequently, we have added a matter
for congressional consideration to our report suggesting that Congress
consider either directing DOD to require that an independent external
review of these body armor test results be conducted or require that
DOD officially amend its testing protocols to reflect any revised test
procedures and repeat First Article Testing to ensure properly tested
designs.
DOD partially concurred with our third recommendation to determine
whether those procedures that deviated from established testing
protocols during First Article Testing should be continued during
future testing and to change the established testing protocols to
reflect those revised procedures. DOD recognized the need to update
testing protocols and added that when the office of the Director of
Operational Test and Evaluation promulgates standard testing protocols
across DOD, these standards will address issues that we identified. As
long as DOD specifically addresses all the inconsistencies and
deviations that we observed prior to any future body armor testing,
this would satisfy our recommendation.
DOD stated that it partially concurs with our fourth recommendation to
evaluate and recertify the accuracy of the laser scanner to the correct
standard with all software modifications incorporated, based on the
results of the independent expert review of the First Article Testing
results. We also recommended that this process include a side-by-side
comparison of the laser's measurement of back-face deformations and
those taken by digital caliper. DOD concurred with the concept of an
independent evaluation, but it did not concur that one is needed in
this situation because according to DOD its laser certification was
sufficient. We disagree that the laser certification was performed
correctly. As discussed in the body of our report and further in
appendix II, recertification of the laser is critical because (1) the
laser was certified to the wrong standard, (2) software modifications
were added after the certification of the laser, and (3) these
modifications did change the way the laser scanner measured back-face
deformations. DOD did not explicitly state whether it concurred with
our recommendation for a side-by-side comparison of the laser scanner
and the digital caliper in their operational environment. We assert
that such a study is important because without it the Army and DOD do
not know the effect the laser scanner may have on the back-face
deformation standard that has been used for many years and was
established with the intention of being measured with a digital
caliper. If the comparison reveals a significant difference between the
laser scanner and the digital caliper, DOD and the Army may need to
revisit the back-face deformation standard of its requirements with the
input of industry experts and the medical community.
DOD generally concurred with our fifth recommendation to conduct an
independent evaluation of the Aberdeen Test Center's testing protocols,
facilities, and instrumentation and stated that such an evaluation
would be performed by a team of subject matter experts that included
both DOD and non-DOD members. We agree that in principal this approach
meets the intent of our recommendation as long as the DOD[Footnote 46]
members of the evaluation team are independent and not made up of
personnel from those organizations involved in the body armor testing
such as office of the Director of Operational Test and Evaluation, the
Army Test and Evaluation Command, or PEO Soldier. DOD's comments and
our specific responses to them are provided in appendix II.
We are sending copies of this report to the appropriate congressional
committees, the Secretary of Defense, and the Secretary of the Army. In
addition, the report will be available at no charge on GAO's Web site
at [hyperlink, http://www.gao.gov]. If you or your staff has any
questions about this report, please contact me at (202) 512-8365 or
solisw@gao.gov. Contact points for our Offices of Congressional
Relations and Public Affairs may be found on the last page of this
report. GAO staff who made major contributions to this report are
listed in appendix III.
Signed by:
William M. Solis:
Director, Defense Capabilities and Management:
List of Requesters:
The Honorable Carl Levin:
Chairman:
The Honorable John McCain:
Ranking Member:
Committee on Armed Services:
United States Senate:
The Honorable Jim Webb:
United States Senate:
The Honorable Ike Skelton:
Chairman:
The Honorable Howard McKeon:
Ranking Member:
Committee on Armed Services:
United States:
House of Representatives:
The Honorable Neil Abercrombie:
Chairman:
The Honorable Roscoe Bartlett:
Ranking Member:
Subcommittee on Air and Land Forces: Committee on Armed Services:
United States House of Representatives:
The Honorable Joe Courtney:
United States House of Representatives:
[End of section]
Appendix I: Scope and Methodology:
Our review of body armor testing focused on testing conducted by the
Army in response to specific concerns raised by the House and Senate
Armed Services Committees and multiple members of Congress. During our
review, we were present during two rounds of testing of body armor
designs that were submitted in response to a May 2007-February 2008
Army contract solicitation. The first round of testing, called
Preliminary Design Model testing, was conducted from February 2008
through June 2008 with the objective of determining whether designs
submitted under the contract solicitation met the required ballistic
performance specifications and were eligible for contract award. The
second round of testing, called First Article Testing, was conducted
between November 2008 and December 2008 on the body armor designs that
passed the Preliminary Design Model testing. Both tests were conducted
at Aberdeen Proving Grounds in Aberdeen, Md., and were performed by
Aberdeen Test Center. During the course of our review, we observed how
the Army conducted its body armor testing and compared our observations
with the established body armor testing protocols. We did not verify
the accuracy of the Army's test data and did not provide an expert
evaluation of the results of testing. To understand the practices the
Army used and the established testing protocols we were comparing the
practices with, we met with and/or obtained data from officials from
the Department of Defense (DOD) organizations and the industry experts
listed in table 1:
Table 1: Organizations Contacted for Information about Body Armor
Testing:
DOD acquisition organization:
Program Executive Office Soldier.
DOD testing organization:
Army Test and Evaluation Command;
Developmental Test Command;
Aberdeen Test Center;
Army Research Laboratory;
DOD's office of the Director of Operational Test and Evaluation.
Industry expert:
U.S. Laboratories;
H.P. White Laboratories;
Various body armor manufacturers.
Source: GAO.
[End of table]
To determine the degree to which the Army followed established testing
protocols during the Preliminary Design Model testing of body armor
designs, we were present and made observations during the entire period
of testing, compared our observations with established testing
protocols, and interviewed numerous DOD and other experts about body
armor testing. We observed Army testers as they determined whether
designs met the physical and ballistics specifications described in the
contract solicitation, and as encouraged by Aberdeen Test Center
officials, we observed the ballistics testing from inside a viewing
room equipped with video and audio connections to the firing lanes. We
also were present and observed the physical characterization of the
test items and visited the environmental conditioning chambers, the
weathering chamber, and the X-ray facility. We were at Aberdeen Test
Center when the designs were delivered for testing on February 7, 2008,
and were on-site every day of physical characterization, which
comprises the steps performed to determine whether each design meets
the required weight and measurement specifications. We systematically
recorded our observations of physical characterization on a structured,
paper data-collection instrument that we developed after consulting
with technical experts from Program Executive Office (PEO) Soldier
before testing started. We were also present for every day except one
of the ballistics testing, observing and collecting data on
approximately 80 percent of the tests from a video viewing room that
was equipped with an audio connection to each of the three firing
lanes. To gather data from the day that we were not present to observe
ballistic testing, we viewed that day's testing on video playback. We
systematically recorded our observations of ballistics testing using a
structured, electronic data-collection instrument that we developed to
record relevant ballistic test data--such as the shot velocity,
penetration results, and the amount of force absorbed (called "back-
face deformation") by the design tested. Following testing, we
supplemented the information we recorded on our data collection
instrument with some of the Army's official test data and photos from
its Vision Digital Library System. We developed the data collection
instrument used to collect ballistics testing data by consulting with
technical experts from Program Executive Office Soldier and attending a
testing demonstration at Aberdeen Test Center before Preliminary Design
Model testing began. After capturing the Preliminary Design Model
testing data in our data collection instruments, we compared our
observations of the way the Aberdeen Test Center conducted testing with
the testing protocols that Army officials told us served as the testing
standards at the Aberdeen Test Center. According to these officials,
these testing protocols comprised the (1) test procedures described in
the contract solicitation announcement's purchase descriptions and (2)
Army's detailed test plans and Test Operating Procedure that serve as
guidance to the Aberdeen Test Center testers and that were developed by
the Army Test and Evaluation Command and approved by Program Executive
Office Soldier, the office of the Director of Operational Test and
Evaluation, the Army Research Labs, and cognizant Army components. We
also reviewed National Institute of Justice testing standards because
Aberdeen Test Center officials told us that, although Aberdeen Test
Center is not a National Institute of Justice-certified testing
facility, they have made adjustments to their procedures based on those
standards and consider them when evaluating Aberdeen Test Center's test
practices. Regarding the edge shot locations for the impact test
samples, we first measured the area of intended impact on an
undisturbed portion of the test item on all 56 test samples after the
samples had already been shot.[Footnote 47] The next day we had
Aberdeen Test Center testers measure the area of intended impact on a
random sample of the impact test samples to confirm our measurements.
Throughout testing we maintained a written observation log and compiled
all of our ballistic test data into a master spreadsheet. Before,
during, and after testing, we interviewed representatives from numerous
Army entities--including the Assistant Secretary of the Army for
Acquisition, Technology and Logistics; Aberdeen Test Center;
Developmental Test Command; Army Research Laboratories; and Program
Executive Office Soldier--and also attended Integrated Product Team
meetings.
To determine the degree to which the Army followed established testing
protocols during First Article Testing of the body armor designs that
passed Preliminary Design Model testing, we were present and made
observations during the entire period of testing, compared our
observations with established testing protocols, and interviewed
numerous DOD and industry experts about body armor testing. As during
Preliminary Design Model testing, we observed Army testers as they
determined whether designs met the physical and ballistics
specifications described in the contract solicitation. However,
different from our review of Preliminary Design Model testing, we had
access to the firing lanes during ballistic testing. We also still had
access to the video viewing room used during Preliminary Design Model
testing, so we used a bifurcated approach of observing testing from
both the firing lanes and the video viewing room. We were present for
every day except one of First Article Testing--from the first day of
ballistics testing on November 11, 2008, until the final shot was fired
on December 17, 2008.[Footnote 48] We noted the weights and measures of
plates during physical characterization on the same data collection
instrument that we used during Preliminary Design Model testing. For
the ballistics tests, we revised our Preliminary Design Model testing
data collection instrument so that we could capture data while in the
firing lane--data that we were unable to confirm first hand during
Preliminary Design Model testing. For example, we observed the pre-shot
measurements of shot locations on the plates and the Aberdeen Test
Center's method for recording data and tracking the chain of custody of
the plates; we also recorded the depth of the clay calibration drops
(the series of pre-test drops of a weight on clay that is to be placed
behind the plates during the shots), the temperature of the clay, the
temperature and humidity of the firing lane, the temperatures in the
fluid soak conditioning trailer, and the time it took to perform tests.
We continued to record all of the relevant data that we had recorded
during Preliminary Design Model testing, such as the plate number, type
of ballistic subtest, the charge weight of the shot, the shot velocity,
the penetration results, and the back-face deformation. Regarding the
new laser arm that Aberdeen Test Center acquired to measure back-face
deformation during First Article Testing, we attended a demonstration
of the arm's functionality performed by Aberdeen Test Center and also
acquired documents related to the laser arm's certification by Army
Test, Measurement, and Diagnostic Equipment activity. With a GAO senior
methodologist and a senior technologist, we made observations related
to Aberdeen Test Center's methods of handling and repairing clay,
calibrating the laser guide used to ensure accurate shots, and
measuring back-face deformation. Throughout testing we maintained a
written observation log and compiled all of our ballistic test data
into a master spreadsheet. Following testing, we supplemented the
information we recorded on our data collection instrument with some of
the Army's official test data and photos from its Vision Digital
Library System to complete our records of the testing. After capturing
the testing data in our data collection instruments, we compared our
observations of the way Aberdeen Test Center conducted testing with the
testing protocols that Army officials told us served as the testing
standards at the Aberdeen Test Center. In analyzing the potential
impact of independent variables on testing, such as the potential
impact of the November 13th rain on the clay, we conducted statistical
tests including chi- square and Fisher's Exact Test methods to
accommodate small sample sizes. Before, during, and after testing, we
interviewed representatives from numerous Army agencies, including
Aberdeen Test Center, Developmental Test Command, Army Research
Laboratories, and Program Executive Office Soldier. We also spoke with
vendor representatives who were present and observing the First Article
Testing, as well as with Army and industry subject matter experts.
We conducted this performance audit from July 2007 through October 2009
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.
[End of section]
Appendix II: Comments from the Department of Defense:
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
Office Of The Secretary Of Defense: Operational Test And Evaluation:
1700 Defense Pentagon:
Washington, DC 20301-1700:
August 29 2009:
Mr. William M. Solis:
Director:
Defense Capabilities and Management: U.S. Government Accountability
Office: 441 G Street, N.W.
Washington, DC 20548:
Dear Mr. Solis:
This is the Department of Defense (DoD) response to the Government
Accountability Office (GAO) draft report, GAO-09-827, "Warfighter
Support: Independent Expert Assessment of Army Body Armor Test Results
and Procedures Needed," dated July 31, 2009 (GAO Code 351282). This
response includes the DoD's overall position regarding the proposed GAO
report, the DoD position on each of the five GAO recommendations for
executive action, and a detailed response to specific issues and
assertions contained within the proposed GAO report.
The Office of the Director, Operational Test and Evaluation (DOT&E)
along with the Army leadership, particularly the Office of the
Assistant Secretary for Acquisition, Logistics, and Technology, Program
Executive Office (PEO) Soldier, and the Army Test and Evaluation
Command (ATEC), teamed in an open, collaborative, and unified manner to
prepare this response. This partnership of Office of Secretary of
Defense and Army leadership underscores the commitment of the DoD to
develop, test, and ultimately field only the very best personal
protection equipment possible. The DoD appreciates the opportunity to
comment on this important report.
Background:
In preparing this response, the DOT&E, as tasked by the DoD Inspector
General (IG), was the principal office of responsibility. Given that
the GAO report primarily focused on testing protocols and procedures
executed at the Army's Aberdeen Test Center (ATC), ATEC, parent
organization of ATC, provided input to this response. As the Army's
materiel developer and program manager for individual protection
equipment, PEO Soldier also provided input to this response.
Additionally, the U.S. Army Research, Development, and Engineering
Command's Contracting Agency participated in the review of the GAO
report and provided input particularly regarding protection of source
selection sensitive information in the report.
DoD recognizes the importance of personal protection equipment, the
last line of defense for combat troops. In October 2003, the Acting
Secretary of the Army and Army Chief of Staff directed that all
measures that provide protection to Soldiers would be a focused top
priority. Since that date, the Army has continually improved its
personal protection products, as well as its processes associated with
the procurement and testing of these items. In fact, personal
protection products receive the focused attention of all Army senior
leaders and over the last year its activities have been reviewed by the
Secretary of the Army on a weekly basis.
In 2003, only about 10 percent of the fighting force in Iraq had
ballistic inserts (hard armor plates) for their body armor. The DoD,
largely led by the Army and the U.S. Marine Corps, undertook an urgent
effort, working in partnership with the industrial base, to develop,
procure, and field body armor plates to defeat the most significant
threat in Iraq. By April 2004, about 440,000 sets of Small Arms
Protective Inserts (SAPI) were fielded, equipping 100 percent of the
Army and U.S. Marine Corps deployed fighting forces. Beginning about
January 2005, in response to a developing threat in theater, the DoD
embarked on another urgent effort to develop and field the Enhanced
Small Arms Protective Insert (ESAPI). In February 2005, PEO Soldier
shipped the first sets of ESAPI, and by March 2006, all Army and U.S.
Marine Corps deployed warfighters were equipped with the ESAPI plates.
Additionally, during this period the DoD acquired, tested, and fielded
other various body armor system components in answer to warfighter
needs, such as arm protection, side plates, and neck and groin
protection.
Undertakings of this magnitude, done urgently in time of war, are not
without flaw. Inherent in this process was consideration by the DoD to
incorporate into the contractual requirements, where appropriate,
factors of safety above the threshold operational requirement. What was
most important, however, was fielding body armor plates that defeated
the threat. The DoD accomplished that objective, not once, but twice.
In spite of flaws in procedures documented in recent DoD IG and Army
Audit Agency reports, plates that were fielded have consistently
defeated and continue to defeat the threat for which they were
designed. The DoD has full confidence in the performance of the
personal protection equipment its forces depend upon in combat
operations. [See comment 1]
In June 2007, the House Armed Services Committee convened a hearing to
discuss allegations by a body armor manufacturer that the Army did not
fairly test and evaluate its product.[Footnote 66] Subsequent to that
hearing and after publication of reports by the DoD IG and the GAO, the
DoD has undertaken efforts to improve procedures associated with body
armor testing. The DoD, using the Office of the DOT&E, has initiated
several efforts to respond to members of the Armed Services Committees
and to address issues raised by the DoD IG. Specifically, the DOT&E:
[See comment 2]
* Implemented the recommendations of the DoD IG to improve the testing
process via oversight and direct participation with the Services;
* Established a DoD-wide integrated project team to standardize test
protocols for personal protection equipment. Included on that team are
representatives from other government agencies (e.g., Federal Bureau of
Investigation, Central Intelligence Agency, National Institute of
Standards and Technology);
* Implemented an extensive test of body armor that will increase the
statistical confidence in body armor performance and provide key input
to the standardization of testing protocols;
* Implemented a process of oversight of the testing of key components
of personal protection equipment;
* Established a policy to conduct personal protection equipment First
Article Testing (FAT) that leads to design acceptance, at government
facilities; and,
* Advised the Army and the U.S. Marines that if personal protection
equipment testing is contracted to private laboratories, they should
maintain government oversight during the conduct of those tests.
Additionally, based on the directive of the Secretary of Defense,
[Footnote 67] DOT&E exercised oversight of Preliminary Design Model
(PDM) testing and FAT, both of which are addressed in the GAO report.
DOT&E exercised this oversight by determining the scope of testing
required,[Footnote 68] approving test plans, on-site monitoring of
testing, and leading the body armor Integrated Product Team (IPT). [See
comment 3]
Actions By The Army:
Since 2007, ATEC has instituted procedures and policies that improve
the testing of personal protection equipment. ATEC has made several
investments to improve its capability and capacity for testing body
armor and other personal protection equipment. The Army's investment in
ATC establishes a DoD center of excellence to test personal protection
equipment for all the Services. From June 2007 to the present,
continued improvements have been made and are identified below. [See
comment 4]
* Completed four (4) new state-of-the-art body armor test ranges with
plans to construct four (4) additional test ranges within the next 18
months;
* Procured and certified a state-of-the-art laser scanner measurement
device that provides accurate and repeatable measurements of Back Face
Deformations (BFD);
* Developed and published Army Test Operating Procedures for testing of
hard body armor that addresses Army specific requirements, which exceed
those of law enforcement standards published by the National Institute
of Justice (NU). These Army requirements address the harsh combat
environmental operating conditions that Army body armor systems must
endure without any degradation in performance;
* Completed installation of new clay conditioning chambers inside each
test range;
* Improved velocity measurement accuracy by conducting a study of the
effect of drag and creation of correction tables to more accurately
capture the striking velocity of test rounds; and,
* Implemented use of electronic data collection and processing for body
armor testing via the ATC Versatile Information Systems Integrated On-
Line Digital Library System. Data is collected in real time and, once
reviewed and authenticated, is available to authorized users over the
Internet through a secure U.S. Army website. This process typically
enables customers, PEO Soldier, and body armor vendors to view test
results within 24 hours of test completion.
PEO Soldier has also instituted a number of efforts to improve its
acquisition of personal protection equipment. These are listed below.
[See comment 5]
* Ensured that all prospective body armor manufacturers may compete for
new contracts;
* Transferred testing expertise and experience to support Army
Acquisition Executive direction in February 2009, that all first
article and lot acceptance testing would be conducted by ATEC, the
Army's test agency independent of the materiel developer;
* Initiated and organized a Task Force focused on Soldier Protection
that is now evolving into a new structure including a Senior Executive
Service (SES) civilian managed organization with focus on quality
control and procedures, decision management, process control, and
compliance;
* Developed a non-destructive test capability to accurately and quickly
assess ballistic plates for defects and established an in-theater,
post- fielding surveillance program to examine body armor plates for
cracks and other defects; and,
* Developed a comprehensive and holistic personal protection evaluation
process which includes pre-production, production, and post fielding
activities. With this process, the Army can evaluate the effectiveness
of its products at any stage of its life cycle. [See comment 6]
The DoD recognizes the need to continually improve its procedures for
the development, testing, and procurement of personal protection
equipment. Many of the actions by ATEC and PEO Soldier were initiated
and improved upon during the course of the GAO audit.
General Comments On The Report:
While the GAO Report GAO-09-827 points out some weaknesses in
procedures and discrepancies in testing recently conducted by ATC, it
is the DoD's position that these findings have no significant impact on
the test results and the subsequent contracting actions taken by the
Army based on these test results. The DoD does not concur with what it
perceives as two over-arching conclusions by the GAO: 1) That
Preliminary Design Model (PDM) testing did not achieve its intended
objective of determining, as a basis for contract awards, which designs
met performance requirements; and, 2) The FAT may not have met its
objective of determining whether each of the contracted plate designs
met performance requirements.
Preliminary Design Model Testing:
The DoD and the Army concluded that PDM testing achieved its objective
to identify those vendor designs that met the performance objectives
stated in PEO Soldier's Purchase Description. The GAO cites as the
Army's most consequential deviation from the test protocols described
in the Purchase Description, the practice of measuring back-face
deformation at the point of aim. Upon discovery of this deviation from
the test protocol described in the Purchase Description, the Army
stopped testing. The Army leadership, after a deliberative internal
process and in consultation with DOT&E, decided to use the point of aim
measurement technique as it was determined by proper authority to be an
accurate and repeatable process and that it did not bias the test
results against any vendor's design. The contract solicitation was
modified to reflect the decision to measure at point of aim. Therefore,
it is incorrect to state that the Army deviated from the test process
and it is incorrect to state that "at least two" of the preliminary
design models should have failed as they passed in accordance with the
modified solicitation. Additional technical details supporting the
rationale for this decision are found later in this letter. [See
comment 7]
First Article Testing:
The DoD and the Army concluded that the FAT achieved its objective of
verifying that contracted vendors could produce, in a full-rate
capacity, plates that passed PDM. In 2007, prior to initiation of PDM
testing by ATEC, DOT&E, Army leadership, and ATEC all agreed that FAT
would be conducted as part of the Army body armor testing effort that
is the subject of the GAO report. Though conduct of FAT became more
essential following PDM testing due to measuring BFD at point of aim as
opposed to the deepest deformation, as discussed in detail in the
responses to GAO's recommendations, it was never the DoD's intent to
waive FAT during this effort. As a system on DOT&E oversight, it was
DOT&E's responsibility to determine the required scope of testing. [See
comment 8]
The multi-phase concept that included PDM testing, FAT, and extended
ballistic testing to support development of an improved test standard
was briefed to congressional member and professional staff on November
14, 2007. The test plan was updated and briefed again to member and
professional staff on October 27, 2008. As indicated by the GAO, PEO
Soldier has in the past granted FAT waivers to vendors that submit
production representative material that subsequently passes PDM
testing. Though the GAO indicates that PEO Soldier may have initially
contemplated FAT waivers, waivers were not permitted under the amended
solicitation. Further, additional coordination by DOT&E and the Deputy
Director for Land Warfare and Munitions, Under Secretary of Defense for
Acquisition, Technology and Logistics on July 25, 2007, with the
Military Deputy to the Army Acquisition Executive (superior of the
PEO), confirmed that the Army would subject all vendors to all tests
and conditions and conduct all testing at ATC. As the GAO noted, the
Army refined procedures for FAT to address lessons- learned from PDM.
With regard to back-face deformation (BFD) measurement, ATEC acquired
and certified a laser scanning device to accurately measure and record
BFD. This device removes human interpretation from measuring a non-
uniform back-face signature in clay and greatly improves measurement
accuracy and repeatability. Introduction of new technology to improve
accuracy beyond the existing procedures reflects ATEC's effort to
improve body armor testing. This decision was approved by ATEC and
senior Army leadership. It is the DoD's position that ATEC properly
implemented the laser scanning instrumentation.
During FAT, ATEC and the PEO Soldier maintained open and continuous
dialogue. PEO Soldier ballistic experts, the authors of many of the
technical criteria in the Purchase Description, assisted in providing
technical interpretations associated with scoring and ballistic
phenomena.
With regard to rounding of the BFD measurement, ATEC, in keeping with
historical procedures of PEO Soldier, instituted this process for FAT.
This action was approved by the body armor IPT.
Though non-conformities did occur, FAT achieved its objective to
identify those vendors that could mass produce acceptable plates. It is
DoD's position that ATEC used proper scoring procedures during FAT.
Therefore, it is incorrect to say that FAT did not meet its objective
and it is incorrect to assert that three of five vendor designs should
have failed FAT.
Comments and insights made by the GAO, as well as the DoD IG, have
helped the Army refine and improve the procedures relating to body
armor testing. Additionally, the DoD will continue to engage with
external test and technology experts, such as the National Institute of
Standards and Technology (NIST), a principal author of National
Institute of Justice (NIJ) standards for law enforcement, as
appropriate, during development and refinement of test procedures.
While the DoD recognizes the role independent test laboratories
certified by NIJ can serve in helping meet its testing capacity needs,
the DoD does not believe those laboratories should be considered
external experts upon which to rely for critique of the DoD's current
policies and procedures, as indicated in the GAO report. The DoD will
continue to scrutinize its procedures and will pursue more open
collaboration between agencies responsible for the development,
acquisition, and testing of all personal protection equipment,
primarily PEO Soldier, ATEC, and DOT&E. Further, the DoD will continue
to accept critiques and criticisms from oversight agencies, and will
continue to improve its test procedures.
Responses To GAO Recommendations For Executive Action:
Recommendation 1: The GAO recommends that the Secretary of Defense
direct the Secretary of the Army to provide for an independent
evaluation of the First Article Testing (FAT) results by ballistics and
statistical experts external to DoD before any armor is fielded to
Soldiers under this contract solicitation and that the Army report the
results of that assessment to the Director of Operational Testing and
Evaluation and the Congress. In performing this evaluation, the
independent experts should specifically evaluate the effects of the
following procedures observed during first article testing:
* The rounding of back-face deformation measurements.
* Not scoring penetrations of material through the plate as a complete
penetration unless broken fibers are observed in the Kevlar backing
behind each plate.
* The use of the laser scanner to measure back-face deformations
without a full evaluation of its accuracy as it was actually used
during testing, to include the use of the software modifications and
operation under actual test conditions.
* The exposure of the clay backing material to rain and other outside
environmental conditions as well as the effect of high oven
temperatures during storage and conditioning. [See comment 9]
* The use of an additional series of clay calibration drops when the
first series of clay calibration drops does not pass required
specifications.
DOD Response: Non-Concur. The DoD does not concur with the GAO
recommendation for an independent evaluation of First Article Test
(FAT) results before any armor is fielded to Soldiers. The DoD's
position is that the objectives of FAT were achieved; the FAT verified
the vendors' ability to mass produce ballistic plates while maintaining
performance standards. Anomalies identified by the GAO do not alter the
results of FAT. The DoD is satisfied the FAT was properly scored
despite the process discrepancies and documentation issues noted by the
GAO. However, as noted below, the DoD will review test processes in
general, with external assistance, which will include the areas
identified below. Non-concurrence and partial concurrence pertaining to
the bulleted items above are noted below, accordingly.
Subject: Rounding of back-face deformation measurements. [See comment
10]
Response: Non-Concur. The DoD does not agree that how testers recorded
results by numerically rounding them should be reviewed by experts
external to DoD before any armor is fielded to Soldiers under this
contract solicitation. The procedure has been used historically by the
National Institute of Justice (NU) and its certified laboratories since
at least 1999. Since 1999, the Small Arms Protective Inserts (SAPI),
Enhanced Small Arms Protective Inserts (ESAPI), and next generation
SAPI (XSAPI) Purchase Descriptions have adopted NIJ's practice for
measuring back-face deformation (BFD). Program Executive Office (PEO)
Soldier established requirements of 43 mm and 48 mm for ESAPI and XSAPI
BFD, as documented in the Purchase Description with the intent that
testers round to the nearest whole number using ASTM E-29[Footnote 69]
as a guide. The DoD acknowledges that the details of the rounding
practice are not adequately described in the Purchase Description. HP
White, a certified NIJ lab, has historically used the same rounding
rule for SAPI, ESAPI, and enhanced side ballistic inserts testing for
past U.S. Army, U.S. Marine Corps, and Defense Supply Center
Philadelphia contracts.[Footnote 70] In November 2008, the body armor
Integrated Product Team (IPT), consisting of representatives of the
U.S. Army Test and Evaluation Command (ATEC), PEO Soldier, and the
Office of the Director, Operational Test and Evaluation (DOT&E), agreed
to use that same common rounding method and did use that technique
during FAT and is using that technique for all lot acceptance tests
(LATs). This practice is a policy decision that is not prohibited by
any DoD or NIJ standard. As a result of the significantly smaller
measurement error associated with the laser scanner, the DoD, PEO
Soldier, and ATEC are reviewing the rounding methodology associated
with the BFD scoring process and will make documented changes to the
procedure as appropriate.
Subject: Not scoring penetrations of material through the plate as a
complete penetration unless broken fibers are observed in the Kevlar
backing behind each plate. [See comment 11]
Response: Non-concur. The DoD does not agree that use of the first
layer of soft armor behind the plate as a witness plate should be
reviewed by experts external to DoD before any armor is fielded to
Soldiers under this contract solicitation. The first layer of soft
armor behind the plate under test serves as a witness plate during
testing. If that first layer of soft armor is not penetrated, as
determined by the breaking of threads on that first layer of soft
armor, the test shot is not scored as a complete penetration in
accordance with the Program Executive Office (PEO) Soldier's scoring
criteria. The breaking of a thread in the deeper layers of soft armor
does not constitute a penetration since the stretching of material due
to the force of impact could cause a thread in layers below the surface
to break even though a penetration did not occur. Aberdeen Test Center
(ATC) had a ballistic expert provide a preliminary score of test
results. Those scores were reviewed and either agreed to or amended by
three individual subject matter experts designated by the PEO Soldier
as the official scorers for the test. If there is any question as to
whether or not a thread is broken on the first layer, it is examined
under a microscope at the discretion of the official scorers. In this
case, three independent subject matter experts agreed with the ATC
subject matter expert and scored the test results in question as a
partial penetration. The GAO acknowledged there were no broken threads
on the first layer of the soft armor. GAO also reports that its test
observer discovered fragments three layers deep in the soft armor.
However, the ATC subject matter expert who examined the test specimen
following the test only found dust particles and some discoloration,
which is not indicative of a complete penetration.
Subject: The use of the laser scanner to measure back-face deformation
(BFD) without a full evaluation of its accuracy as it was actually used
during testing, to include the use of the software modifications and
operation under actual test conditions. [See comment 12]
Response: Non-concur. The DoD does not agree that use of a laser
scanner needs to be reviewed by experts external to DoD before any
armor is fielded to Soldiers under this contract solicitation. The
laser scanner measurement device provides a superior tool for providing
accurate, repeatable, defensible BFD measurements to the deepest point
of penetration in clay. It is significant to note that it also
eliminates human errors such as incorrectly selecting the location of
the deepest point (a subjective decision) or piercing the clay with the
sharp edge of the caliper and making the depression deeper. The laser
also alleviates the need to use correction factors[Footnote 71] to
account for the curvature of plates when making a BFD measurement that
is not aligned with the path of the projectile (discussed in detail
later in this letter). The technique of using correction factors
results in only an approximation of the BFD measurement.
The U.S. Army Test, Measurement and Diagnostic Equipment (TMDE)
Activity, Huntsville, Alabama, collaborated with Aberdeen Test Center
(ATC) to conduct a study that led to the certification of the laser
scanner measurement system. Certification testing was performed in both
a lab environment and on the actual ranges used for testing. ATC
conducted a comprehensive 3- month study of the laser scanner involving
1,920 measurements of replicas of clay forms to assess the laser's
ability to accurately replicate the clay deformation.[Footnote 72]
Based on this study and the contributions made by TMDE, in accordance
with Army Regulation 73-1, the Commanding General, Army Test and
Evaluation Command (ATEC) certified the laser measurement system as
approved for use for all future body armor testing. As the Army
Executive Agent for testing, the implementation of the laser scanner as
the most accurate tool available for measuring BFD in clay is within
ATEC's mission and authority. The testing to support laser
certification, the comprehensive study, and the certification by the
Commanding General of ATEC all supported a Senior Army Level Integrated
Product Team (IPT) decision to implement the laser as the Army-wide
means of measuring BFD on clay.
Regarding use of the scanner system in the testing environment, the
laser measurement system is protected in an armored enclosure to
prevent damage from spall fragments. Bullets fired during testing do
not impact or affect the scanner calibration. Additionally, shock
effects caused by bullet impact to items under test do not affect
alignment, because the alignment is independent of the testing
environment. Alignment is accomplished through fixed reference points
taken before each scan to ensure accuracy. The laser measurement system
is calibrated twice daily per the manufacturer's instructions and will
not operate until successfully calibrated after start-up.
Regarding software modifications, the software upgrades referred to in
the report did not affect the measurement system in the laser scanner.
The software changes made efficiencies in post- processing of the data
and made enhancements to the graphical user interface of the system.
These software changes had no effect on the physical measurement
process of BFD that was validated through the certification process.
Regarding the laser scanner "overstating" (artificially inflating) the
BFD measurement, it is DoD's position that the verified accuracy of the
scanner, coupled with the study involving the scanning of clay
replicas, documents that the laser accurately measures the true BFD.
Additionally, use of the laser scanner does not constitute the
establishment of a new requirement.
The same laser scanner measurement device is widely used throughout
commercial industry, to include the aeronautical industry, which has
far tighter measurement tolerances than those required for body armor
testing. It is also relevant to note that a similar laser arm from the
same manufacturer is also in use by the National Institute of Science
and Technology (NISI) for planar and non-planar measurements.
Subject: The exposure of the clay backing material to rain and other
outside environmental conditions as well as the effect of high oven
temperatures during storage and conditioning. [See comment 13]
Response: Non-concur. The GAO asserts that its statistical analyses
indicate that exposure of the clay used to test a vendor on November
13, 2008, to "constant heavy, cold rain" caused unusual test results on
that day. The DoD does not concur with that assessment. Based on the
GAO assertion, the Army Test and Evaluation Command (ATEC) conducted
its own statistical analyses on the test results for all designs tested
on November 13, 2008.
The analyses show that the poor performance of the design in question
was attributable to its marginal performance against the most
formidable threat round under test, not to a brief time (seconds) of
exposure of the clay to "constant heavy, cold rain." The design in
question had a 70 percent failure rate during testing against the most
formidable threat. Because all vendor designs were not tested on every
test day, statistical analyses by test day provides far less insight
than performing statistical analyses by individual design, as conducted
by ATEC. Additionally, the GAO statisticians included "No
Test"[Footnote 73] data in their analyses. In accordance with the
scoring protocols established by Program Executive Office (PEO) PEO
Soldier, "No Test" data are excluded from the pool of test results and
are not considered in any post-test analyses. Therefore, the statistics
contained in the GAO draft report (44 percent first shot/90 percent
second shot failures) are erroneous because they include invalid test
data. It is also relevant to note that the Army statisticians (one from
ATEC and one from the Army Research Laboratory) responsible for the
review of those data are both well experienced in applying general
statistical tools to body armor testing and one statistician was a
major contributor to a recently released NIJ standard.[Footnote 74]
Although the effects to the clay after a brief exposure to "constant
heavy, cold rain" had no impact on the test results, Aberdeen Test
Center (ATC) completed the planned installation of new clay
conditioning chambers inside the test ranges precluding any external
environmental conditioning interacting with the clay. This action
improved overall test efficiency and mitigated safety risks to those
handling heavy clay blocks.
Regarding high oven temperatures, the Purchase Description was modified
prior to Preliminary Design Model (PDM) testing, removing the
requirement for specific thermal conditioning of the clay blocks prior
to calibration and subsequent testing. The purpose of thermal
conditioning is to affect the clay in such a way as to promote
successful calibration per the Purchase Description (three drops of a
cylindrical steel mass shall each produce a deformation in the clay of
25 +/-3 mm). PEO Soldier removed the thermal conditioning requirement,
because regardless of any thermal conditioning used, the clay must pass
the calibration test before it can be used for testing.[Footnote 75]
Subject: The use of an additional series of clay calibration drops when
the first series of clay calibration drops does not pass required
specifications. [See comment 14]
Response: Partially concur. The DoD concurs with the establishment of a
written standard for conducting subsequent clay calibration drop tests,
but non-concurs with the GAO's assertion that failed clay blocks were
used during the conduct of ballistic testing at Aberdeen Test Center
(ATC). All clay used in testing passed the required clay calibration
standard in effect at that time. The National Institute of Justice
(NIJ) standard[Footnote 76] as verified by personnel at the National
Institute of Standards and Technology (NIST), does not address
specifically the issue of repeating clay calibration tests. Though NIST
officials would recommend only one series of drops for clay
calibration, that is not a requirement of the NIJ standard, and nothing
prohibits a test activity from repeating calibration attempts on a
block of clay. NIST also indicated they arc not aware of any scientific
studies or literature that describe how the clay properties might
change as a result of performing repeated validation attempts. The DoD
has agreed to partner with NIST to conduct experiments to improve the
testing community's understanding of clay performance in ballistic
testing. Upon completion of testing under the current Army
solicitation, in coordination with NIST, the Director of Operational
Test and Evaluation (DOT&E) and the Army will review the procedures for
clay calibration, to include repeated calibration attempts, and
document any appropriate procedural changes.
During the period of review by GAO, for reasons pertaining to the time
limit established to complete ballistic testing and the concerns cited
in the GAO report, ATC established and documented a revised procedure
stating that only one repeat of a calibration attempt can be made. If
the clay does not pass calibration upon the second attempt, it is
reconditioned for later use and a new block of clay is substituted for
calibration. During timed subtests, once a clay block is removed from
the conditioning environment, all body armor plate testing using that
clay block must be completed in 30 minutes or less or the clay block
must be reconditioned and the test repeated. All clay backing material
used during testing passed the calibration drop test prior to use.
Recommendation 2: The GAO recommends that the Secretary of Defense
direct the Secretary of the Army to document all key decisions made to
clarify or change the testing protocols.
DOD Response: Concur. The DoD recognizes the need for contemporaneous
documentation and proper approvals to support any significant change to
the testing protocols. The Director of Operational Test and Evaluation
(DOT&E) and the Army will ensure that all key decisions made to clarify
or change testing protocols be sufficiently documented. Additionally,
the DoD intends to publish a series of standard personal protection
equipment test protocols, beginning with soft and hard body armor. The
DoD expects to publish the first of these standards by the end of this
year. The standard will include detailed documentation requirements and
will remediate the process discrepancies noted by the GAO.
Recommendation 3: With respect to the specific inconsistencies that the
GAO identified between the test procedures and the testing protocols,
the GAO recommends that the Secretary of the Army, based on the results
of the independent expert review of the First Article Test results,
determine whether those procedures that deviated from established
testing protocols during First Article Testing will be continued during
future testing and change the established testing protocols to reflect
those revised procedures.
DOD Response: [See comment 15] Partially concur. It is the DoD's
position that identified inconsistencies in procedures used to
implement the test protocols in First Article Tests (FAT) did not alter
test results. The DoD recognizes the need to update test protocols as
necessitated by the adoption of new technologies and improved test
procedures. The Army Test and Evaluation Command (ATEC) routinely
updates Test Operating Procedures[Footnote 77] and participates in DoD
actions to update Military Standards to ensure the latest approved test
procedures are being followed. As noted above, the DoD will use the
Director, Operational Test and Evaluation (DOT&E) to promulgate
standard test protocols across the DoD. As reflected in the draft FY10
National Defense Authorization Act, DOT&E will ensure these standards
are staffed to appropriate external agencies. Those new standards will
address issues identified by the GAO.
Recommendation 4: With respect to the specific inconsistencies that the
GAO identified between the test procedures and the testing protocols,
the GAO recommends that the Secretary of the Army, based on the results
of the independent expert review of the First Article Test results,
evaluate and recertify the accuracy of the laser scanner to the correct
standard with all software modifications incorporated and include in
this analysis a side-by-side comparison of the laser measurements of
the actual back-face deformations with those taken by digital caliper
to determine whether laser measurements can meet the standard of the
testing protocols.
DOD Response: [See comment 16] Partially concur. While the DoD does not
concur with the GAO conclusion regarding inconsistencies and the need
to recertify the laser measurement system, the DoD does concur with the
concept of an independent certification of the laser measurement system
and process. That process was completed prior to implementation of the
laser scanner for back-face deformation (BFD) measurement. Per Army
Regulation 750-43, the U.S. Army Test, Measurement and Diagnostic
Equipment (TMDE) Agency, under the Assistant Secretary of the Army for
Acquisition, Logistics, and Technology, is responsible for calibration
of U.S. Army test instrumentation with traceability to National
Institute of Standards and Technology (NIST) requirements. Following
completion of this calibration, in accordance with Army Regulation 73-
1, the Commanding General, Army Test and Evaluation Command (ATEC)
certified the instrumentation for use during Army testing. Per
documentation already provided to GAO, ATEC adhered to proper
procedures and processes for certification of the laser measurement
system prior to its use during testing and is in compliance with all
applicable Army regulations. Software changes reported by the GAO did
not affect the measurement system in the laser scanner, as indicated
previously.
Recommendation 5: The GAO recommends that the Secretary of Defense
direct the Secretary of the Army to provide for an independent peer
review of Aberdeen Test Center's body armor test protocols, facilities,
and instrumentation to ensure that proper internal controls and sound
management procedures are in place. This peer review should be
performed by testing experts external to the Army and DoD.
DOD Response: Partially concur. The DoD will conduct an independent
evaluation of Aberdeen Test Center (ATC) test protocols, facilities,
and instrumentation by subject matter experts for the ballistic testing
of armor materiel for military applications. The DoD is in discussion
with the National Institute of Standards and Technology (NIST) to form
a team of subject matter experts to review the DoD's testing
procedures. This review will be broad and will include measurement
processes, clay conditioning, and other areas as appropriate. The DoD
will include experts from within the DoD as part of this team. [See
comment 17]
Detailed Comments Keyed To The GAO Report:
Part 1: GAO Observations and Conclusions on Preliminary Design Model
Testing:
Assertion 1 ” Army's Aberdeen Test Center (ATC) had never before
performed testing on body armor plates. (Page 2 of 48, paragraph 1;
Page 13 of 48, paragraph 1; and, Page 36 of 48, paragraph 2.) [See
comment 18]
Response: ATC did the initial testing on the Interceptor Body Armor
system in the 1990s and has been extensively involved in body armor
long before and since that time. While ATC did not perform any
additional testing on the Interceptor Body Armor system for Program
Manager Soldier Equipment since that initial testing, ATC has
consistently performed required body armor testing for the U.S. Special
Operations Command, as well as research and development testing on body
armor for organizations such as the Natick Soldier Safety Center and
other service Program Managers. ATC tested over a dozen different hard
armor plate designs between 1997 and 2007, to include both Small Arms
Protective Inserts (SAPI) and Enhanced Small Arms Protective Inserts
(ESAPI) plates.
Assertion 2 ” Based on Preliminary Design Model (PDM) Test results, the
Army awarded contracts totaling over $8 Billion for production of ESAPI
and XSAPI. (Page 2 of 48; paragraph 2.) [See comment 19]
Response: The Army awarded three Indefinite Delivery, Indefinite
Quantity (IDIQ) contracts for ESAPI and XSAPI ballistic plates. To
date, those contracts have obligated $119,703,145.49 for XSAPI and
$1,756,044.80 for ESAPI. The contractually guaranteed quantities on
each of the three contracts have been satisfied.
Assertion 3 ” Aberdeen Test Center shot several plates at the wrong
location on the plate. (Page 5 of 48, paragraph 1; and, Page 19 of 48,
paragraph 1.) [See comment 20]
Response: ATC followed the Purchase Description protocol for shot
location during PDM testing and FAT. Specifically, the second test shot
location for the impact subtest during PDM testing, as stated in the
Purchase Description, was approximately 1.5 inches from an edge of the
plate. There were no limits or range specified for this second test
shot location. Given the potential variances between the actual aim
point and impact point during testing, the tester interpreted 1.0 inch
to be an acceptable aim point location for this subtest. In this case,
shooting closer to the edge would have increased the risk of a failure
for this subtest, but no vendors failed testing as a result of the
tester's interpretation of the second test shot location. Therefore,
there was no impact on the outcome of the test.
Assertion 4 ” Aberdeen Test Center shot several plates at the wrong
velocity. (Page 5 of 48, paragraph 1; and, Page 17 of 48, paragraph 3.)
[See comment 21]
Response: During V50 testing, it is worthwhile to note that one of the
test threats, threat "c," is not robust enough to achieve a complete
penetration no matter how much the velocity is increased; therefore,
following the test procedure to achieve a complete penetration is an
impossible task for threat "c."[Footnote 78] However, in accordance
with the PDM test protocol, the threat "c" V50 testing was completed to
a degree that provided the required government reference data for
baseline comparison to data generated during testing of previous
generations of body armor. The V50 subtests for more robust threats,
during which complete penetrations were achievable, were executed to
the standard protocols.
Assertion 5 ” Army tester's incorrectly measured the amount of force
absorbed by the designs tested by measuring back-face deformation at
the point of aim rather than at the deepest point of depression. (Page
5 of 48, paragraph 1; and, Page 21- 23 of 48.) [See comment 22]
Response: Though ATC deviated from the test protocol in the Purchase
Description regarding measuring BFD at the lowest deformation, Army
leadership after a deliberative internal process and in consultation
with DOT&E, decided to use the BFD point-of-aim measurement as it was
determined to be an accurate and repeatable process that did not bias
any vendor's design. The following information highlights the rationale
for this decision. For clarity and reference, Figures 1 and 2 depict
the geometry associated with this issue. Further discussion follows the
figures.
Figure 1. Test setup before test shot:
[Refer to PDF for image: illustration]
Clay Backing Material:
Impact point on the surface of the plate; reference zero for BFD
measurement.
Fair shot must be between .75 and 1.25 in from edge of plate.
This pane establishes the reference line for the depth of deformation
measurement.
[End of figure]
Figure 2: Post shot measurement:
[Refer to PDF for image: illustration]
Clay Backing Material:
Impact point on the surface of the plate; reference zero for BFD
measurement.
Must add this difference to the measured.
Must subtract this difference from the measured.
Deepest deformation not along shot line.
This plane establishes the reference line for the depth of deformation
measurement.
[End of figure]
The Purchase Description, Paragraph 4.9.9.3 Back-Face Deformation
Measurement, states:
"Back-face deformations in the clay will not exceed 1.70-inches (43mm)
max (Paragraph 3.9.3) when measured from the original undisturbed
surface of the backing material to the lowest point of the depression.
All Back-Face Deformation measurements will be conducted at 0 degree
obliquity only. Indentation measurements will utilize measurement
devices (+/- 0.1 mm accuracy) incorporating a fixed reference "guide"
(See Figure 2)[Footnote 79] that can rest solidly upon two edges of the
fixture, establishing the reference plane across the diameter of the
indentation. The distance between the reference "guide" and original
undisturbed surface will be measured at the point of intended impact
prior to impact. The distance between reference "guide" and the lowest
point of depression will be measured after impact. Back-face
deformation will be the difference between the two."
Referencing the first sentence of the above quote, the only known
reference point prior to a test shot is the aim point ("original
undisturbed surface of the backing material"). Prior to the shot, the
test technician has no way of knowing the surface location
perpendicular to the lowest point of depression. The fixed reference
guide identified in the quote, shown in Figure 3, can only assist in
establishing the aim point as the reference point from which post-test
measurements can be made. Therefore, the ATC interpretation was to use
that point to measure BFD following the test.
Figure 3. BFD Measurement Reference Guide from Purchase Description:
[Refer to PDF for image: illustration]
Example Of Reference Guide:
Note: Example of Reference Guide (For Information Only) See Paragraph
4.9.93, Bask Face Defamation Measurement.
Figure 2:
Reference guide:
Backing Material (clay):
Backing Material box:
Measurement before impact:
Measurement after impact.
[End of figure]
There are two important considerations pertaining to the guidance in
the Purchase Description: First, the lowest point of depression must be
subjectively selected by a test technician, and second, measuring
perpendicular from the point selected as the lowest point of depression
to the reference guide provides the distance between that point and the
original undisturbed surface at the aim point, not the original
undisturbed surface perpendicular to the lowest point of depression,
which is what is required to obtain an accurate measurement. Using the
technique described, an accurate measurement can only be made if the
measurement is made on a flat surface with a uniform original
undisturbed reference surface. However, since the plates are curved and
the radius varies between vendor plates and even within vendors for
same plate sizes, as shown in Figure 4, the only point at which an
accurate measurement can be made with the descri measuring device is
the aim point.
Figure 4: Picture illustrates variance in curvature between same size
ballistic plates:
[Refer to PDF for image: photograph]
Same manufacturer, same size shows variability.
Different manufacturers, same size shows variability.
Differences in curvature as much as 1/2 inch (12.7 mm) at center of
plate.
[End of figure]
Although not contained in the Purchase Description, PEO Soldier had an
internally documented process to account for plate curvature when the
deepest point of deformation was laterally offset from the point of
aim. However, due to plate curvature variance, a correction factor only
approximates the deepest deformation measurement. That is one reason
ATC elected to measure BFD at point of aim at the beginning of testing,
but also became the rationale for a better procedure using the laser
scanner as explained elsewhere in this letter.
During the early stages of PDM testing, ATC analyzed the effect of
measuring BFD at the aim point in lieu of the lowest point of
depression with a caliper. The results of the analysis for those 222
test data points are summarized below.
* Mean delta between aim point and deepest point BFD was 0.45 mm.
* For the 222 test shots, the minimum delta was 0 mm and the maximum
was 5.95 mm.
* 170 of 222 data points showed no difference, i.e., aim point same as
lowest point of depression within the BFD.
* Approximately 24 percent of measurements (52) showed a difference.
i.e., aim point different than lowest point of depression.
When Army leadership and DOT&E learned of this issue of measuring BFD
at aim point versus deepest deformation, ATEC halted testing. Following
analysis of available data, the Army Acquisition Executive, the PEO
Soldier, the Commanding General of ATEC, and a senior representative
from DOT&E agreed that for the remainder of PDM testing, ATEC would
score BFD at the aim point, while recording for government reference
the BFD at the lowest point using the aforementioned curvature
correction standard. This decision is also referenced in the below
executive summary from the acting Army Acquisition Executive's office
following the Senior Army Leadership Executive IPT meeting:
"Body Armor (ESAPI and XSAPI) Testing (U). (SAAL-SMS) On 26 March, the
Army Test and Evaluation Command Commander suspended testing pending
discussion with Army leaders from the acquisition and test communities
on test procedures. The Army Acquisition Executive (AAE) convened a
meeting on 07 April 2008 to explore the solicitation language and test
procedures concluding that Aberdeen Test Center methods were applied
consistently to all vendors and remain defendable. The AAE directed
Program Executive Officer Soldier, in collaboration with the test
community, to amend the solicitation to clarify the description of this
procedure. The source selection and resultant contract awards will be
made in accordance with the terms and conditions of the solicitation.
As previously briefed to the staff of the HASC and SASC, a phase II,
First Article Test, will commence later this year. This test, conducted
by ATEC, will use government provided test articles purchased from the
contract awardees and will encompass both ballistic and operational
testing. The Research, Development, and Engineering Center Contracting
Agency subsequently modified the solicitation (Amendment 14, dated
April 17, 2008) to evident the Army's decision to measure BFD at point
of aim for PDM testing."
At PDM completion, ATC analyzed the 3,404 data points (1,702 shots; two
measurements per shot) of the two measurement methods. The results of
that analysis are below.
* For 1,091 of the 1,702 test shots (64 percent), the aim point was the
same as the deepest point.
* For 611 of the 1,702 test shots (36 percent), the deepest point did
not occur at the aim point.
* The mean difference (deepest minus aim point) of all 1,702 test shots
was 0.60 mm.
* The mean difference of only the 611 test shots with a non-zero
difference between aim point and deepest point was 1.67 mm.
* For the 1,702 test shots, the minimum delta was -0.28 mm and the
maximum was 10.66 mm.
These data show that while there is a difference in depth between the
BFD measured at aim point and at lowest point, the difference is small.
Nonetheless, the Army's adoption of the laser scanner measurement
technique resolves this issue completely.
Assertion 6 ” Deviations from the test protocols (e.g., measuring back-
face deformation at the aim point) were not reviewed or approved by
officials from PEO Soldier, Director of Operational Test and
Evaluation, and other activities responsible for approving testing
protocols. (Page 5-6 of 48.) [See comment 23]
Response: The DoD acknowledges these shortcomings.
* The DoD acknowledges that measuring BFD at the point of aim during
the early stages of PDM testing was not known by all members of the IPT
or senior Army or DOT&E leaders. However, once that issue became
evident, all members of the IPT and their leadership acted decisively
to arbitrate, resolve, and document the resolution.
* The issue of rounding BFD was discussed at an IPT meeting and was
agreed to by all members present.
* ATC developed internal procedures for clay calibration that at first
were not documented adequately. However, once the issue became evident,
ATC proceeded to adequately document and discuss its procedures with
the IPT.
These issues have been subsequently addressed. It is the DoD position
that none of these issues prevented the Army from achieving its PDM
objectives.
Part 2: GAO Observations and Conclusions on First Article Testing:
Assertion 7 ” Calibration test procedures deviated from established
test protocols. GAO observed clay being exposed to "constant heavy,
cold rain." (Pg 6 of 48 paragraph 3; Page 20 of 48, paragraph 1; Page
27 of 48, paragraph 2; and, Page 38 of 48, paragraph 1.) Response. This
discussion appends and adds detailed information to the discussion on
this same subject contained in an earlier section. Clay is largely
impervious to water penetration and the first step in preparing the
clay for the calibration drop test is to scrape off the top layer of
clay. This action cleared the clay block of any residual water.
Regardless of the exposure to environmental conditions, the test
standard is clear: If the clay backing material passes the calibration
drop test it is acceptable for use for ballistic testing. [See comment
24]
ATC analysis also shows that the poor performance of the vendor's
design in question was attributed to its marginal performance against
the most formidable threat round under test, not to a brief time
(seconds) of exposure to a "constant heavy, cold rain and low
temperatures" or the effects of rain on the clay surface. Statistical
analyses were performed on the test results for all designs including
those tested on November 13, 2008. The result of that analysis, as
shown in Figure 5, indicate Design K was the weakest design on days
with no rain as well as days with rain. Design K had a 70 percent
failure rate during all testing against the most formidable threat.
Even when excluding the rain day data, which was when that design was
subjected to the most stressing tests, the design still had a 57
percent failure rate.
Figure 5: Results of ATC analysis of test results for all vendor
designs against the most formidable threat:
[Refer to PDF for image: illustration]
[End of figure]
In the report, GAO statisticians stated that "No Test" data was
included in the statistical calculations reported on page 13 of the
draft GAO report. In accordance with the scoring protocols established
between the PM and ATEC and included in the Purchase Description, "No
Test" data is excluded from the database and does not constitute a
"failure" since it is an unfair test shot impact that requires a
retest. Therefore, the statistics contained in the GAO draft report (44
percent first shot/90 percent second shot failures) are erroneous
because they include invalid test data.
Assertion 8 - During First Article Testing Army testers improperly
scored a complete penetration as a partial penetration. (Page 8 of 48,
paragraph 1; and, Page 32 of 48, paragraph 2.) [See comment 25]
Response: This information appends that provided in the earlier
discussion of this issue. While Figure 6 of the GAO report appears to
show evidence of a perforation on the rear of the test plate in
question, by the definition in the Purchase Description, a complete
perforation is scored based upon the damage to the soft armor shoot
pack directly behind the ballistic test plate. The front (face
immediately behind the plate under test) of the shoot pack in question
is shown in Figure 6 of this response, located below. Though deformed,
the damage does not constitute a complete perforation of the plate
according to the definition in the Purchase Description.
Figure 6: Front face of shoot pack for the test in question:
[Refer to PDF for image: photograph]
[End of figure]
Assertion 9 ” The Army did not maintain internal controls over the
integrity and reliability of test data at all times. (Page 8 of 48,
paragraph 1; and, Page 33 of 48, paragraph 1.) [See comment 26]
Response. ATEC did maintain adequate internal controls to ensure the
integrity and reliability of test data. GAO reports the incident of a
Test Director accidentally pressing the delete key on a computer
keyboard, thereby resulting in loss of data. That did occur, but only
once in over 4,100 test events. Additionally, as it was obvious the
loss of data occurred, the test shot was immediately repeated. This had
no effect on the outcome of the test. ATC employs a multiple, redundant
system of human checks to ensure data is accurately recorded, whether
recorded initially by human or computer. All test shot data is
immediately uploaded to an archival system. At the end of each day, a
team of analysts scrutinize those data to ensure accuracy. Only then do
those data become authenticated. This is a rigorous process that helps
to ensure accuracy and integrity in data acquisition, recording, and
archiving. Regarding the laser scanner, only two persons are authorized
and able to modify the laser scanner software. Range personnel cannot
alter the scanner software or settings.
Assertion 10 ” The Army did not formally document significant procedure
changes that deviated from established testing protocols or assess the
impact of these deviations. (Page 34 of 48, paragraph 2.) [See comment
27]
Response. Acknowledged. As indicated previously, these shortcomings
have been identified, partly by the GAO, and have been remedied. Part
3: GAO Observations and Other Issues.
Assertion 11 ” GAO statement that the requirement to test at an NIJ
certified test lab was withdrawn because Aberdeen Test Center is not
NIJ certified. (Page 12 of 48; paragraph 2; and, Page 14 of 48;
paragraph 1.) [See comment 28]
Response: The DoD does not believe that NIJ certification is
appropriate for its test facilities. There are significant differences
between NIJ and U.S. Army body armor test requirements. The NIJ
certification process is intended to ensure domestic police forces and
Justice Department personnel are provided body armor that meets
appropriate standards for their job duties. However, these standards
are insufficient for the U.S. Army, given the expanded testing required
to ensure body armor meets U.S. Army requirements. Figure 7 depicts the
differences between NIJ and Army body armor test requirements.
Table: Comparison of the NIJ Standard and U.S. Army Performance Tests:
National Institute of Justice Standard 0101.04 versus Army ESAPI First
Article Test:
Body Armor Test: Resistance to Penetration Test (VO); NIJ Standard:
[Check];
Army Standard: [Check].
Body Armor Test: Ballistic Limit Test (V50); NIJ Standard: [Check];
Army Standard: [Check].
Body Armor Test: 3-Minute Water Spray Resistance to Penetration Test;
NIJ Standard: [Check];
Army Standard: [A].
Body Armor Test: Flammability Resistance to Penetration Test (250
degrees F);
NIJ Standard: Not required;
Army Standard: [Check].
Body Armor Test: Oil Immersion Resistance to Penetration Test; NIJ
Standard: Not required;
Army Standard: [Check].
Body Armor Test: Diesel Immersion Resistance to Penetration Test; NIJ
Standard: Not required;
Army Standard: [Check].
Body Armor Test: 2-Hour Salt Water Immersion Resistance to Penetration;
NIJ Standard: Not required;
Army Standard: [Check].
Body Armor Test: Impact Drop Resistance to Penetration Test; NIJ
Standard: Not required;
Army Standard: [Check].
Body Armor Test: Low Temperature Resistance to Penetration Test (-60
degrees F);
NIJ Standard: Not required;
Army Standard: [Check].
Body Armor Test: High Temperature Resistance to Penetration Test (160
degrees F);
NIJ Standard: Not required;
Army Standard: [Check].
Body Armor Test: Temperature Shock Resistance to Penetration Test (-25
to 120 degrees F);
NIJ Standard: Not required;
Army Standard: [Check].
Body Armor Test: Weatherometer/Accelerated Aging Resistance to
Penetration Test;
NIJ Standard: Not required;
Army Standard: [Check].
Body Armor Test: Altitude Resistance to Penetration Test;
NIJ Standard: Not required;
Army Standard: [Check].
[A] 2-Hour Salt Water Immersion Test conducted by Army is more
stringent than the NIJ 3-minute Water Spray Test.
Note: not required = not required by NIJ, though NIJ-certified
laboratories could execute or contract out these tests.
[End of table]
Assertion 12: After the June 2007 House Armed Services Committee
hearing, the Army decided to rebuild small arms ballistic testing
capabilities at Aberdeen Test Center. (Page 12 of 48; paragraph 2.)
[See comment 29]
Response: This assertion is incorrect. The contract to construct
additional test ranges at the Aberdeen Test Center Light Armor Range
was already awarded (September 2006) and construction was already
underway at the time of June 2007 House Armed Services Committee
hearing. This upgrade was not in response to any particular event, but
was undertaken to meet projected future Army ballistic test
requirements.
Signed by:
David W. Duma:
Acting Director:
GAO Comments:
1. The Department of Defense (DOD) stated that undertakings of this
magnitude are not without flaws and that what was most important was
fielding body armor plates that defeated the threat. While DOD may have
identified some flaws that may not be serious enough to call the
testing results into question, several of the deviations to the testing
protocols that we observed do call the testing results into question
for the reasons stated in our report. An independent expert has not
evaluated the impact of these deviations on the test results and, until
such a study is conducted, DOD cannot be assured that the plates that
passed testing can defeat the threat. DOD also noted several actions
DOD and the Army have taken to improve procedures associated with body
armor testing. Our responses to these actions are included in comments
2 through 6.
2. The office of the Director of Operational Test and Evaluation's
efforts to respond to members of the Armed Services Committees and to
address issues raised by the Department of Defense Inspector General
were outside the scope of our audit. Therefore, we did not validate the
implementation of the actions DOD cited or evaluate their effectiveness
in improving test procedures. With regard to the office of the Director
of Operational Test and Evaluation's establishing a policy to conduct
First Article Testing at government facilities, using a government
facility to conduct testing may not necessarily produce improved test
results.
3. Regarding the office of the Director of Operational Test and
Evaluation's oversight of testing, the office of the Director of
Operational Test and Evaluation led the Integrated Product Team and
approved the test plans. However, while we were present at the Aberdeen
Test Center during Preliminary Design Model testing and First Article
Testing, we did not observe on-site monitoring of the testing by the
office of the Director of Operational Test and Evaluation staff beyond
incidental visits during VIP events and other demonstrations.
4. Regarding the procedures and policies DOD stated were implemented by
the Army Test and Evaluation Command to improve testing:
* Only two of the test ranges were completed prior to Preliminary
Design Model testing. Two additional test ranges were completed after
Preliminary Design Model testing.
* Regarding the certification of the laser scanner measurement device,
as noted in our report, the Army had not adequately certified that it
was an appropriate tool for body armor testing (see our comment 12).
* The Army's Test Operating Procedure was not completed or implemented
until after Preliminary Design Model testing.[Footnote 49]
* New clay conditioning chambers inside each test range were not
constructed until after all testing was completed (see our comment 13).
* The improved velocity measurement accuracy study was not conducted
until after all testing was completed.
* Regarding the implementation of electronic data collection and
processing for body armor testing, as stated in our report, we observed
that not all data are electronically collected. Many types of data are
manually collected and are later converted to electronic data storage.
5. Regarding Program Executive Office (PEO) Soldier's efforts to
improve the acquisition of personal protection equipment:
* The contract solicitation allowed all prospective body armor
manufacturers to compete for new contracts.
* We observed that PEO Soldier did transfer expertise and experience to
support Army Acquisition Executive direction that all First Article
Testing and lot-acceptance testing be conducted by the Army Test and
Evaluation Command.
* The task force that focused on soldier protection was not initiated
until February 2009, after all Preliminary Design Model testing and
First Article Testing was completed.
* According to Army officials, PEO Soldier instituted a non-destructive
test capability that became operational after Preliminary Design Model
testing, but prior to First Article Testing.
* PEO Soldier's personal protection evaluation process was described in
our previous report--GAO-07-662R. Although we recognized the strength
of PEO Soldier's personal protection evaluation process in our earlier
report, not all the protections that were in place at that time remain
in place. For example, the requirement that testing be conducted at a
National Institute of Justice (NIJ)-certified facility was waived.
6. DOD stated that many of the actions by Army Test and Evaluation
Command and PEO Soldier were initiated and improved upon during the
course of our review. However, as discussed above, several of these
actions were initiated before and during testing, but many of them were
not completed until after testing was completed.
7. DOD and the Army stated that Preliminary Design Model testing had
achieved its objective to identify those vendor designs that met the
performance objectives stated in PEO Soldier's purchase description and
that "it is incorrect to state that 'at least two' of the preliminary
design models should have failed as they passed in accordance with the
modified solicitation." We disagree with these statements. As stated in
our report, the most consequential of the deviations from testing
protocols we observed involved the measurement of back-face
deformation, which did affect final test results. According to original
testing protocols, back-face deformation was to be measured at the
deepest point of the depression in the clay backing. This measure
indicates the most force that the armor will allow to be exerted on an
individual struck by a bullet. According to Army officials, the deeper
the back-face deformation measured in the clay backing, the higher the
risk of internal injury or death. DOD and the Army now claim that these
solutions passed in accordance with the modified solicitation, which
overlooks the fact that the reason the solicitation had to be modified
was that Army testers deviated from the testing protocols laid out in
the purchase descriptions and did not measure back- face deformation at
the deepest point. DOD and the Army also stated in their response that
they decided to use the point of aim because they determined it was an
accurate and repeatable process. Yet in DOD's detailed comments
regarding edge shot locations, DOD acknowledged that there were
"potential variances between the actual aim point and impact point
during testing." Army Research Laboratory[Footnote 50] and NIJ-
certified laboratories use the benchmark process of measuring back-face
deformation at the deepest point, not at the point of aim.
As set forth in our report, at least two solutions passed Preliminary
Design Model testing that would have failed if back- face deformation
had been measured to the deepest point. This statement came directly
from Aberdeen Test Center officials during a meeting in July 2008,
where they specifically told us which two solutions would have failed.
We said "at least" two because Army testers did not record deepest
point back-face deformation data for the first 30 percent of testing,
and therefore there could be more solutions that would have failed had
the deepest point been measured during this first portion of the test.
Because the Army did not measure back-face deformation to the deepest
point, it could not identify whether these two solutions in particular
and all the solutions in general met performance requirements. As a
result, Army could not waive First Article Testing for successful
candidates and was forced to repeat the test to ensure that all
solutions did indeed meet requirements. By repeating testing, the Army
incurred additional expense and further delays in fielding armor from
this solicitation to the soldiers. During the course of our audit, the
Army also acknowledged that the Preliminary Design Model testing did
not meet its objective because First Article Testing could not be
waived without incurring risk to the soldiers.
DOD and the Army stated that, upon discovery of the back-face
deformation deviation from the testing protocols described in the
purchase descriptions, the Army stopped testing. The Army's Contracting
Office was informed of this deviation through a series of questions
posed by a vendor who was present at the Vendor Demonstration Day on
February 20, 2008. This vendor sent questions to the Contracting Office
on February 27 asking whether testers were measuring at the aim point
or at the deepest point. This vendor also raised questions about how
damage to the soft pack would be recorded and about the location of
edge shots. Based on our observations, all of these questions involved
issues where Army testers deviated from testing protocols and are
discussed in our responses to subsequent comments. The Army did not
respond until March 19 and replied that its test procedures complied
with solicitation requirements. It was not until Army leadership
learned of the vendor's questions and of the deviation in measuring
back-face deformation that testing was finally halted on March 27, a
full month after the issue came to the Army Test and Evaluation
Command's attention.
8. DOD stated that in 2007, prior to the initiation of Preliminary
Design Model testing, the Army Test and Evaluation Command, the office
of the Director of Operational Test and Evaluation, and Army leadership
all[Footnote 51] agreed that First Article Testing would be conducted
as part of the Army's body armor testing. However, DOD did not provide
any documentation dated prior to April 2008--that is, prior to the
discovery of the back-face deformation deviation--that suggested that
DOD intended to conduct First Article Testing following Preliminary
Design Model testing. In July 2008, the Army Test and Evaluation
Command and PEO Soldier stated in official written responses to our
questions regarding Preliminary Design Model testing that the conduct
of First Article Testing became essential following Preliminary Design
Model testing because of the Army's measuring back-face deformation at
the point of aim as opposed to at the deepest point of deformation. In
fact, because of this deviation, DOD could not waive First Article
Testing as originally planned and was forced to conduct subsequent
tests to verify that the designs that had passed Preliminary Design
Model testing met testing requirements. DOD asserted that a multi-phase
concept including Preliminary Design Model testing, First Article
Testing, and extended ballistic testing to support the development of
an improved test standard was briefed to a congressional member and
professional staff on November 14, 2007. We were present at this
November 14 test overview and strategy/schedule briefing and noted that
it did not include plans for First Article Testing to be performed in
addition to Preliminary Design Model testing. Excerpts from the slides
briefed that day showed Preliminary Design Model (Phase 1) testing and
a subsequent ballistic and suitability testing (Phase 2). As indicated
in the slides (see figure 7 and figure 8) from that November 14
briefing, the Phase 2 test was designed to test the form, fit, and
function of those solutions that had passed Preliminary Design Model
testing as well as the ballistic statistical confidence tests.[Footnote
52] According to information we obtained, Phase 2 was never intended to
be First Article Testing and was to have no impact on whether or not a
solution received a contract.
Figure 7: Briefing Slide from DOD's Test Overview (Nov. 14, 2007):
[Refer to PDF for image: table]
Phase 1:
1. Physical inspection to ensure compliance with contract requirements
and to document material condition.
2. Ballistic testing and analysis to evaluate test article performance
pursuant to contract requirements.
3. Service test data and all supporting data provided to the Source
Selection Technical Factor Chief.
4. Contract awards(s) made based on best value to the government as
determined by the Source Selection Panel analyses of all available
data. Bid samples provided by vendors subject to Source Selection
Testing IAW First Article Test protocol. Material provided under those
awards also provides test articles for Phase 2.
Phase 2:
1. Ballistic testing to provide high statistical confidence in material
performance.
2. Suitability testing for form fit, and function. Testing conducted
using operarational Soldiers (Ft. Benning, GA to support).
3. Service T&E report.
Phase 1 testing described is per the Army test plan approved by CG ATEC
on September 11, 2007 and by DOT&E on September 19, 2007.
Source: Provided by Director of Operational Test and Evaluation at
House Armed Services Committee briefing on November 14, 2007.
[End of figure]
Figure 8: Briefing Slide from DOD's Test Strategy and Schedule (Nov.
14, 2007):
[Refer to PDF for image: illustration]
ESAPI”XSAPI”FSAPV-E”FSAPV-X:
System picture:
Body armor.
Description:
PM Soldier Equipment RFP for the following Body Armor items:E
SAPI: Enhanced Small Arms Protective Insert;
XSAPI: (X) Small Arms Protective Insert;
FSAPV-E: Flexible Small Arms Protective Vesg-Enhanced;
FSAPV-X: Flexible Small Arms Protective Vest-X;
RFP Closes December 12, 2007.
Test strategy/schedule:
Phase 1 (Proposal selection testing):
Begin: December 13, 2007:
PDM: Ballistic testing;
V50 Ballistic limit testing;
V0 Ambient testing;
V50 Environmental conditions (9 subtests).
Phase 2 (Testing to gain statistical confidence and H FE data).
Begin: TBD:
V50 Select environmental conditions HFE/Suitability testing.
Issues/status:
Phase 1: Ballistic test plans approved by DOTE.
Phase 2: Ballistic test plans and suitability test plans are written in
draft form.
Congressional interest: GAO and DOTE oversight.
Path forward:
Execute phase 1: Ballistic test.
Finalize phase 2: Ballistic and suitability test plans.
Source: Provided by Director of Operational Test and Evaluation at
House Armed Services Committee Briefing on November 14, 2007. Sensitive
ballistic information removed form the "description" box (lower, left-
hand-side quadrant).
[End of figure]
It was not until after the back-face deformation deviation was
discovered that briefing slides and other documentation on test plans
and schedules started describing First Article Testing as following
Preliminary Design Model testing. For example, as stated by DOD in its
comments, the October 2008 briefing to a congressional member and
professional staff clearly showed First Article Testing as following
Preliminary Design Model testing (Phase 1) and preceding Phase 2.
Therefore, it is not clear why DOD's test plan briefings would make no
mention of a First Article Testing prior to the back-face deformation
measurement deviation while including First Article Testing in
subsequent briefings if the plan had always been to conduct both
Preliminary Design Model testing and First Article Testing.
Furthermore, it is not clear why DOD would intentionally plan at the
start of testing to repeat Preliminary Design Model testing (which was
supposed to be performed in accordance with the First Article Testing
protocol) with an identical test (First Article Testing) given that it
has been the Army's practice to use such Preliminary Design Model
testing to meet First Article Testing requirements - a practice that
was also supported by the DOD Inspector General and the Army
Acquisition Executive after an audit of the Army's body armor testing
program.[Footnote 53] DOD also stated that First Article Testing
waivers were not permitted under the body armor solicitation. However,
the solicitation and its amendments are unclear as to whether waivers
of First Article Testing would be permitted. Nonetheless, in written
answers to questions we posed to the Army in July 2008, the Army Test
and Evaluation Command and PEO Soldier in a combined response stated
that due to the fact that back-face deformation was not measured to the
deepest point of penetration during Phase I tests, there would be no
waivers of First Article Testing after the contract award.
DOD also stated that it and the Army concluded that First Article
Testing had achieved its objective of verifying that contracted vendors
could produce, in a full-rate capacity, plates that had passed
Preliminary Design Model testing. DOD further stated that it is
incorrect to say that First Article Testing did not meet its objective
and it is incorrect to assert that three of five vendor designs should
have failed First Article Testing. However, our analysis showed that
two solutions that passed First Article Testing would have failed if
back- face deformations had not been rounded and had been scored as
they were during Preliminary Design Model testing.[Footnote 54] The
third solution that passed would have failed if Army testers had
correctly scored a shot result as a complete penetration in accordance
with the definition of a complete penetration in the purchase
description, rather than as a partial penetration. Because questions
surround these scoring methods and because DOD and the Army cannot
confidently identify whether these vendors can mass produce acceptable
plates, we restate that First Article Testing may not have achieved its
objective. See comments 12, 10, and 11 regarding DOD's statements about
the certification of the laser scanning equipment, the rounding of
back- face deformations, and the Aberdeen Test Center's scoring
procedures, respectively.
We agree with DOD that an open dialog with the DOD Inspector General,
external test and technology experts, and us will improve the current
body armor testing. However, we disagree with DOD's statement that NIJ-
certified laboratories lack expertise to provide reliable information
on body armor testing issues. Before the current solicitation, the Army
relied[Footnote 55] on these NIJ-certified laboratories for all body
armor source selection and lot acceptance tests. The Marine Corps also
conducts source selection tests at these facilities. As these
independent laboratories have performed numerous tests for the Army
conducted in accordance with First Article Testing protocol, we assert
that the credentials of these laboratories warrant consideration of
their opinions on body armor testing matters.
9. DOD did not concur with our recommendation for an independent
evaluation of First Article Testing results before any armor is fielded
to soldiers because the First Article Testing achieved its objectives.
We disagree with DOD's position that First Article Testing and
Preliminary Design Model testing achieved their objectives because we
found numerous deviations from testing protocols that allowed solutions
to pass testing that otherwise would have failed. Due to these
deviations, the majority of which seem to make the testing easier to
pass and favor the vendors, we continue to believe that it is necessary
to have an independent external expert review the results of First
Article Testing and the overall effect of DOD's deviations on those
results before the plates are fielded. An independent observer,
external to DOD, is best suited to determine the overall impact of
DOD's many deviations during the testing associated with this
solicitation. Consequently, we have added a matter for Congress to
consider directing DOD to either conduct this external review or direct
that DOD officially amend its testing protocols to reflect any revised
test procedures and repeat First Article Testing.
10. DOD did not concur with our recommendation that the practice of
rounding down back-face deformations should be reviewed by external
experts because the practice has been used historically by NIJ-
certified laboratories. Although DOD acknowledged that the practice of
rounding is not adequately described in the testing protocols, it
stated that rounding is permitted under American Society for Testing
and Materials (ASTM) E- 29. The purchase descriptions (attachments 01
and 02 of the solicitation) referenced five ASTM documents, but ASTM E-
29 is not referenced and therefore is not part of the protocol. The
detailed test plans state that solutions shall incur a penalty on
deformations greater than 43 millimeters, and the Army is correct that
neither the purchase description nor the detailed test plans provide
for rounding. During Preliminary Design Model testing, Army testers
measured back-face deformations to the hundredths place and did not
round. Any deformation between 43.00 and 43.50 received a penalty.
During First Article Testing, deformations in this range were rounded
down and did not incur a penalty, so the decision to round effectively
changed the standard in favor of the vendors. Two solutions passed
First Article Testing that would have failed if back-face deformations
had been scored without rounding as they were during Preliminary Design
Model testing. We recognize that there are other factors, such as the
fact that the new laser scanner may overstate back-face deformations
that might justify the decision to round down back-face deformations.
However, as a stand-alone event, rounding down deformations did change
the standard in the middle of the solicitation between Preliminary
Design Model testing and First Article Testing. That is why it is
important for an independent external expert to review the totality of
the test and the Army's deviations from testing protocols to determine
the actual effect of this and other deviations.
11. Regarding the incorrect scoring of a complete penetration as a
partial penetration, DOD stated that the first layer of soft armor
behind the plate serves as a witness plate during testing. If that
first layer of soft armor is not penetrated, as determined by the
breaking of threads on that first layer of soft armor, the test shot is
not scored as a complete penetration in accordance with the PEO
Soldier's scoring criteria. However, DOD's position is not consistent
with the established testing protocols as evidenced by the following:
(1) We did not observe the use of and the testing protocols do not
require the use of a witness plate during testing to determine if a
penetration occurred; and:
(2) The testing protocols do not state that "the breaking of threads"
is the criterion for determining a penetration.
The language of the testing protocols, not undocumented criteria,
should be used in scoring and determining penetration results. The
criteria for scoring a penetration are found in the current
solicitation's protocols. Paragraph 6.6, of each of the purchase
descriptions state, under "Definitions: Complete Penetration (CP) for
Acceptance Testing--Complete penetrations have occurred when the
projectile, fragment of the projectile, or fragment of the armor
material is imbedded or passes into the soft under garment used behind
the protective inserts plates" (ESAPIs or XSAPIs).
Our multiple observations and thorough inspection of the soft armor in
question revealed that black-grayish particles had penetrated at least
three Kevlar layers as evidenced by their frayed, fuzz-like and
separated appearance to the naked eye. The black-grayish particles were
stopped by the fourth Kevlar layer. DOD acknowledged that figure 6 of
our report appears to show evidence of a perforation on the rear of the
test plate in question and that the Aberdeen Test Center's subject
matter expert found dust particles. These particles are fragments of
the projectile or fragments of the armor material that were imbedded
and indeed passed into the soft undergarment used behind the protective
insert; therefore, the shot should have been ruled a complete
penetration according to the testing protocols, increasing the point
penalties and causing the design to fail First Article Testing.
DOD's comments stated that we acknowledged there were no broken threads
on the first layer of the soft armor. We made no such comment and this
consideration is not relevant as the requirement for broken fibers is
not consistent with the written testing protocols as we have stated. Of
consequence, DOD and Army officials acknowledged that the requirement
for broken fibers was not described in the testing protocols or
otherwise documented.
In addition to the DOD acknowledgement that an Aberdeen Test Center
subject matter expert found particles on the soft body armor, more
convincing evidence is the picture of the subject plate. Figure 6 of
our report clearly shows the tear in the fibers that were placed behind
the plate in question allowing the penetration of the particles found
by the Aberdeen Test Center subject matter expert. These particles can
only be fragments of the projectile or fragments of the armor material
that passed into the soft under garment used behind the protective
inserts (plates), confirming our observations of the event and the
subsequent incorrect scoring. The shot should have been scored a
complete penetration, and the penalty incurred would have caused the
design in question to fail First Article Testing.
12. DOD did not concur with our recommendation that the use of the
laser scanner needs to be reviewed by experts external to DOD due to
the lack of a full evaluation of the scanner's accuracy to measure
back- face deformations, to include an evaluation of the software
modifications and operation under actual test conditions. DOD asserted
that the laser scanner measurement device provides a superior tool for
providing accurate, repeatable, defensible back-face deformation
measurements to the deepest point of depression in the clay. We agree
that once it is properly certified, tested, and evaluated, the laser
may eliminate human errors such as incorrectly selecting the location
of the deepest point or piercing the clay with the sharp edge of the
caliper and making the depression deeper. However, as we stated, the
Army used the laser scanner as a new method to measure back-face
deformation without adequately certifying that the scanner could
function: (1) in its operational environment, (2) at the required
accuracy, (3) in conjunction with its software upgrades, and (4)
without overstating deformation measurements.
DOD asserted that the software upgrades did not affect the measurement
system of the laser scanner and that these software changes had no
effect on the physical measurement process of the back- face
deformation measurement that was validated through the certification
process. The software upgrades were added after the certification and
do include functions[Footnote 56] to purposely remove spikes and other
small crevices on the clay and a smoothing algorithm that changed back-
face deformation measurements. We have reviewed these software
functions and they do in fact include calculations that change the back-
face deformation measurement taken. Furthermore, Army officials told us
that additional upgrades to the laser scanner were made after First
Article Testing by Aberdeen Test Center to correct a software laser
malfunction identified during the subsequent lot acceptance testing of
its plates. According to these officials, this previously undetected
error caused an overstatement of the back-face deformation measurement
taken by several millimeters, calling into question all the
measurements taken during First Article Testing. Also, vendors have
told us that they have conducted several studies[Footnote 57] that show
that the laser scanner overestimates back-face deformation measurements
by about 2 millimeters as compared with measurements taken by digital
caliper, thereby over- penalizing vendors' designs and causing them to
fail lot acceptance testing.[Footnote 58] Furthermore, the laser
scanner was certified to an accuracy of 1.0 millimeters, but section
4.9.9.3 of the purchase descriptions requires a device capable of
measuring to an accuracy of ±0.1 millimeters. Therefore, the laser does
not meet this requirement making the certification invalid. The laser
scanner is an unproven measuring device that may reflect a new
requirement because the back- face deformation standards are based on
measurements obtained with a digital caliper. This raises concerns that
results obtained using the laser scanner may be more inconsistent than
those obtained using the digital caliper. As we stated in the report,
the Aberdeen Test Center has not conducted a side-by-side test of the
new laser scanner used during First Article Testing and the digital
caliper previously used during Preliminary Design Model testing. Given
the discrepancies on back-face deformation measurements we observed and
the overstating of the back-face deformation alleged by the vendors,
the use of the laser is still called into question. Thus, we continue
to support our recommendation that experts independent of DOD review
the use of the laser during First Article Testing and that a full
evaluation of the laser scanner is imperative to ensure that the tests
are repeatable and can be relied upon to certify procurement of armor
plates for our military personnel based on results of body armor
testing at the Aberdeen Test Center using the laser scanner.
Lastly, DOD stated that the laser scanner is used by the aeronautical
industry; however, the Army Test and Evaluation Command officials told
us that the scanner had to be customized for testing through various
software additions and mounting customizations to mitigate vibrations
and other environmental factors. These software additions and
customizations change the operation of the scanner.
13. DOD does not concur with our recommendation that experts examine,
among other items, "the exposure of clay backing material to rain and
other outside environmental conditions as well as the effect of high
oven temperatures during storage and conditioning," because it believes
that such conditions had no impact upon First Article Testing results.
As detailed in the report, we observed these conditions at different
points throughout the testing period. Major variations in materials
preparation and testing conditions such as exposure to rain and/or
violations of testing protocols merit consideration when analyzing the
effectiveness and reliability of First Article Testing.
As one specific example, we described in this report statistically
significant differences between the rates of failure in response to one
threat on November 13 and the failure rates on all other days of
testing but do not use the statistical analysis as the definitive
causal explanation for such failure. We observed one major
environmental difference in testing conditions that day, the exposure
of temperature-conditioned clay to heavy, cold rain in transit to the
testing site. After experts confirmed that such variation might be one
potential factor relating to overall failure rates on that day, we
conducted statistical tests to assess whether failures rates were
different on November 13 compared to other dates.
Our assertion that the exposure of the clay to rain may have had an
impact on test results is based not solely on our statistical analysis
of test results that day; rather, it is also based on our conversations
with industry experts, including the clay manufacturer, and on the fact
that we witnessed an unusually high number of clay calibration failures
during testing that comprised plate designs of multiple vendors, not
just the one design that DOD points to as the source for the high
failure rate. We observed that the clay conditioning trailer was
located approximately 25 feet away from the entrance to the firing
lane. The clay blocks, weighing in excess of 200 lbs., were loaded face
up onto a cart and then a single individual pulled the cart over
approximately 25 feet of gravel to the firing lane entrance. Once
there, entry was delayed because the cart had to be positioned just
right to get through the firing lane door. Army testers performed all
of this without covering the clay[Footnote 59] to protect it from the
rain and the cold, and once inside the clay had significant amounts of
water collected on it.
With respect to the unusually high number of clay calibration failures
on November 13, there were seven clay calibration drops that were not
within specifications. Some of these failed clay boxes were discarded
in accordance with the testing protocols; however, others were
repaired, re-dropped, and used if they had passed the second drop
series. These included one plate that was later ruled a no-test and
three plates for which the first shot yielded a catastrophic back-face
deformation. These were the only three first-shot catastrophic back-
face deformations during the whole test, and they all occurred on the
same rainy day and involved two different solutions, not just the one
that DOD claims performed poorly.
The failure rates of plates as a whole, across all plate designs, were
very high this day, and the failures were of both the complete
penetration and the back-face deformation variety. Water conducts heat
approximately 25 times faster than air, which means the water on the
surface cooled the clay considerably faster than the clay would have
cooled by air exposure alone. Moreover, Army testers lowered the
temperature of the clay conditioning trailers during testing on
November 13 and told us that the reason was that the ovens and clay
were too hot. This is consistent with what Army subject matter experts
and other industry experts told us--that the theoretical effect of
having cold rain collecting on hot clay may create a situation where
the clay is more susceptible to both complete penetrations because of
the colder, harder top layer and to excessive back-face deformations
because of the overheated, softer clay beneath the top layer. Finally,
the clay manufacturer told us that, although this is an oil- based
clay, water can affect the bonding properties of the clay, making it
more difficult for wet clay to stick together. This is consistent with
what we observed on November 13. After the first shot on one plate, as
Army testers were removing the plate from the clay in order to
determine the shot result, we observed a large chunk of clay fall to
the floor. This clay was simply swept off to the side by the testers.
In another instance, as testers were repairing the clay after the
calibration drop, one of the testers pulled a long blade over the
surface of the clay to smooth it. When he hit the spot where one of the
calibration drops had occurred and the clay had been repaired, the
blade pulled up the entire divot and the testers had to repair the clay
further.
Regarding our use of no-test data, we were strict in the instances
where we used this data, see our comment 24.
DOD stated that it was the poor performance of one solution in
particular that skewed the results for this day and that this solution
failed 70 percent of its shots against Threat D during First Article
Testing. DOD's statistic is misleading. This solution failed 100
percent of its shots (6 of 6) on November 13, but only 50 percent for
all other test days (7 of 14). Also, the fact that this solution
managed to pass the Preliminary Design Model testing but performed so
poorly during First Article Testing raises questions about the
repeatability of DOD's and the Army's test practices. Finally, DOD's
own analysis confirms that two of the four solutions tested on November
13 performed at their worst level in the test on that day. If the one
solution whose plate was questionably ruled a no-test on this day is
included in the data, then three of the four solutions performed at
their worst level in the test on this day.
DOD said that after testing Aberdeen Test Center completed the planned
installation of new clay conditioning chambers inside the test ranges
precluding any external environmental conditioning interacting with the
clay. We believe it is a step in the right direction that the Aberdeen
Test Center has corrected this problem for future testing, but we
continue to believe that an external entity needs to evaluate the
impact of introducing this new independent variable on this day of
First Article Testing.
14. DOD concurred that it should establish a written standard for
conducting clay calibration drops but non-concurred that failed blocks
were used during testing. DOD asserted that all clay backing material
used during testing passed the calibration drop test prior to use. We
disagree with this position because the calibration of the clay
required by the testing protocols calls for "a series of drops,"
meaning one series of three drops, not multiple series of three drops
as we observed on various occasions. DOD stated that, as a result of
our review and the concerns cited in our report, the Aberdeen Test
Center established and documented a revised procedure stating that only
one repeat of calibration attempt can be made and, if the clay does not
pass calibration upon the second attempt, it is reconditioned for later
use and a new block of clay is substituted for calibration. Based on
the testing protocols, this is still an incorrect procedure to ensure
the proper calibration of the clay prior to shooting. The testing
protocols do not allow for a repeat series of calibration drops. DOD
also says that, upon completion of testing under the current Army
solicitation and in coordination with the National Institute of
Standards and Technology, the office of the Director of Operational
Test and Evaluation and the Army will review the procedures for clay
calibration to include repeated calibration attempts and will document
any appropriate procedural changes. DOD goes on to say that the NIJ
standard as verified by personnel at the National Institute of
Standards and Technology does not address specifically the issue of
repeating clay calibration tests. However, the Aberdeen Test Center's
application of the Army's current solicitation's protocols during
testing, and not the NIJ standards, was the subject of our review.
In its comments, DOD acknowledged that the National Institute of
Standards and Technology officials recommend only one series of drops
for clay calibration, but the Aberdeen Test Center did multiple drops
during testing. We are pleased that DOD has agreed to partner with the
National Institute of Standards and Technology to conduct experiments
to improve the testing community's understanding of clay performance in
ballistic testing, but these conversations and studies in our opinion
should have occurred prior to testing, not after, as this deviation
from testing protocols calls the tests results into question. We
reassert that an external entity needs to evaluate the impact of this
practice on First Article Testing results.
15. DOD partially concurred with our recommendation and agreed that
inconsistencies were identified during testing; however, DOD asserted
that the identified inconsistencies did not alter the test results. As
stated in our response to DOD's comments on our first recommendation,
we do not agree. Our observations clearly show that (1) had the deepest
point been used during Preliminary Design Model testing, two designs
that passed would have failed and (2) had the Army not rounded First
Article Testing results down, two designs that passed would have
failed. Further, if the Army had scored the particles (which in their
comments to this report DOD acknowledges were imbedded in the shoot
pack behind the body armor) according to the testing protocols, a third
design that passed First Article Testing would have failed. In all,
four out of the five designs that passed Preliminary Design Model
testing and First Article Testing would have failed if testing
protocols had been followed.
16. DOD partially concurred with our recommendation that, based on the
results of the independent expert review of the First Article Testing
results, it should evaluate and recertify the accuracy of the laser
scanner to the correct standard with all software modifications
incorporated and include in this analysis a side-by-side comparison of
the laser measurements of the actual back-face deformations with those
taken by digital caliper to determine whether laser measurements can
meet the standard of the testing protocols. DOD maintains that it
performed an independent certification of the laser measurement system
and process and that the software changes that occurred did not affect
the measurement system in the laser scanner. However, as discussed in
comment 12, we do not agree that an adequate, independent certification
of the laser measurement system and process was conducted. Based on our
observations, we continue to assert that the software changes added
after certification did affect the measurement system in the laser.
17. DOD partially concurred with our recommendation for the Secretary
of the Army to provide for an independent peer review of the Aberdeen
Test Center's body armor testing protocols, facilities, and
instrumentation. We agree that a review conducted by a panel of
external experts that also includes DOD members could satisfy our
recommendation. However, to maintain the independence of this panel,
the DOD members should not be composed of personnel from those
organizations involved in the body armor testing (such as the office of
the Director of Operational Test and Evaluation, the Army Test and
Evaluation Command, or PEO Soldier.[Footnote 60]
18. DOD stated that Aberdeen Test Center had been extensively involved
in body armor testing since the 1990s and has performed several tests
of body armor plates. We acknowledge that Aberdeen Test Center had
conducted limited body armor testing for the initial testing on the
Interceptor Body Armor system in the 1990s and have clarified the
report to reflect that. However, as acknowledged by DOD, Aberdeen Test
Center did not perform any additional testing on that system for PEO
Soldier since the 1990s and this lack of experience in conducting
source selection testing for that system may have led to the
misinterpretations of testing protocols and deviations noted on our
report. According to a recent Army Audit Agency report,[Footnote 61]
NIJ testing facilities conducted First Article Testing and lot
acceptance testing for the Interceptor Body Armor system prior to this
current solicitation. Another reason Aberdeen Test Center could not
conduct source selection testing was that in the past Aberdeen Test
Center lacked a capability for the production testing of personnel
armor systems in a cost-effective manner; the test facilities were old
and could not support test requirements for a temperature-and humidity-
controlled environment and could not provide enough capacity to support
a war-related workload. The Army has spent about $10 million over the
last few years upgrading the existing facilities with state-of-the-art
capability to support research and development and production
qualification testing for body armor, according to the Army Audit
Agency. Army Test and Evaluation Command notes that there were several
other tests between 1997 and 2007, but according to Army officials
these tests were customer tests not performed in accordance with a
First Article Testing protocol. For example, the U.S. Special
Operations Command test completed in May 2007 and cited by DOD was a
customer test not in accordance with First Article Testing protocol.
The Aberdeen Test Center built new lanes and hired and trained
contractors to perform the Preliminary Design Model testing and First
Article Testing.
19. DOD stated that, to date, it has obligated about $120 million for
XSAPI and less than $2 million for ESAPI. However, the value of the 5-
year indefinite delivery/indefinite quantity contracts we cited is
based on the maximum amount of orders of ESAPI/XSAPI plates that can be
purchased under these contracts. Given that the Army has fulfilled the
minimum order requirements for this solicitation, the Army could decide
to not purchase additional armor based on this solicitation and not
incur almost $7.9 billion in costs.
DOD stated in its response that there are only three contracts.
However, the Army Contracting Office told us that there were four
contracts awarded and provided those contracts to us for our review.
Additionally, we witnessed four vendors participating in First Article
Testing, all of which had to receive contracts to participate. It is
unclear why the Army stated that there were only three contracts.
20. DOD is correct that there is no limit or range specified for the
second shot location for the impact subtest. However, this only
reinforces that the shot should have been aimed at 1.5 inches, not at
1.0 inch or at various points between 1.0 inch and 1.5 inches. It also
does not explain why the Army continued to mark plates as though there
were a range for this shot. Army testers would draw lines at
approximately 0.75 inches for the inner tolerance and 1.25 inches for
the outer tolerance of ESAPI plates. They drew lines at approximately
1.0 inch for the inner tolerance and 1.5 inches for the outer tolerance
of XSAPI plates. We measured these lines for every impact test plate
and also had Army testers measure some of these lines to confirm our
measurements. We found that of 56 test items,[Footnote 62] 17 were
marked with shot ranges wholly inside of 1.5 inches. The ranges of 30
other test items did include 1.5 inches somewhere in the range, but the
center of the range (where Army testers aimed the shot) was still
inside of 1.5 inches. Only four test items were marked with ranges
centered on 1.5 inches.
DOD may be incorrect in stating that shooting closer to the edge would
have increased the risk of a failure for this subtest. For most
subtests this may be the case, but according to Army subject matter
experts the impact test is different. For the impact test, the plate is
dropped onto a concrete surface, striking the crown (center) of the
plate. The test is to determine if this weakens the structural
integrity of the plate, which could involve various cracks spreading
from the center of the plate outward. The reason the requirement for
this shot on this subtest is written differently (i.e., to be shot at
approximately 1.5 inches from the edge, as opposed to within a range
between 0.75 inches and 1.25 inches or between 1.0 inches and 1.5
inches on other subtests) is that it is meant to test the impact's
effect on the plate. For this subtest and this shot, there may actually
be a higher risk of failure the closer to the center the shot occurs.
PEO Soldier representatives acknowledged that the purchase descriptions
should have been written more clearly and changed the requirement for
this shot to a range of between 1.5 inches and 2.25 inches during First
Article Testing. We confirmed that Army testers correctly followed shot
location testing protocols during First Article Testing by double-
checking the measurements on the firing lane prior to the shooting of
the plate. We also note that, although DOD stated the Preliminary
Design Model testing shot locations for the impact test complied with
the language of the testing protocols, under the revised protocol used
during First Article Testing several of these Preliminary Design Model
testing impact test shot locations would not have been valid. DOD
stated that there was no impact on the outcome of the test, but DOD
cannot say that definitively. Because shooting closer to the edge may
have favored the vendors in this case, the impact could have been that
a solution or solutions may have passed that should not have.
21. The Army stated that "V50 subtests for more robust threats—were
executed to the standard protocols." Our observations and analysis of
the data show that this statement is incorrect. Sections 2.2.3.h(2) of
the detailed test plans state:
"If the first round fired yields a complete penetration, the propellant
charge for the second round shall be equal to that of the actual
velocity obtained on the first round minus a propellant decrement for
100 ft/s (30 m/s) velocity decrease in order to obtain a partial
penetration. If the first round fired yields a partial penetration, the
propellant charge for the second round shall be equal to that of the
actual velocity obtained on the first round plus a propellant increment
for a 50 ft/s (15 m/s) velocity increase in order to obtain a complete
penetration. A propellant increment or decrement, as applicable, at 50
ft/s (15 m/s) from actual velocity of last shot shall be used until one
partial and one complete penetration is obtained. After obtaining a
partial and a complete penetration, the propellant increment or
decrement for 50 ft/s (15 m/s) shall be used from the actual velocity
of the previous shot."
V50 testing is conducted to discern the velocity at which 50 percent of
the shots of a particular threat would penetrate each of the body armor
designs. The testing protocols require that, after every shot that is
defeated by the body armor, the velocity of the next shot be increased.
Whenever a shot penetrates the armor, the velocity should be decreased
for the next shot. This increasing and decreasing of the velocities is
supposed to be repeated until testers determine the velocity at which
50 percent of the shots will penetrate. In cases in which the armor far
exceeds the V50 requirement and is able to defeat the threat for the
first six shots, the testing may be halted without discerning the V50
for the plate and the plate may be ruled as passing the requirements.
During Preliminary Design Model V50 testing, Army testers would achieve
three partial penetrations and then continue to shoot at approximately
the same velocity, or lower, for shots 4, 5, and 6 in order to
intentionally achieve six partial penetrations. Army testers told us
that they did this to conserve plates. According to the testing
protocols, Army testers should have continued to increase the charge
weight in order to try to achieve a complete penetration and determine
a V50 velocity. The effect of this methodology was that solutions were
treated inconsistently. Army officials told us that this practice had
no effect on which designs passed or failed, which we do not dispute in
our report; however, this practice made it impossible to discern the
true V50s for these designs based on the results of Preliminary Design
Model testing.
22. DOD agreed that Army testers deviated from the testing protocols by
measuring back-face deformation at the point of aim. DOD stated that
this decision was made by Army leadership in consultation with the
office of the Director of Operational Test and Evaluation, because this
would not disadvantage any vendor. We agree with DOD that this decision
was made by Army leadership in consultation with the office of the
Director of Operational Test and Evaluation. We did not independently
assess all factors being considered by Army leadership when they made
the decision to overrule the Integrated Product Team and the Milestone
Decision Authority's initial decision to measure to the deepest point.
DOD also stated that measuring back-face deformation at the point of
aim is an accurate and repeatable process. As we pointed out in our
previous responses, DOD's own comments regarding DOD's Assertion 3
contradict this statement where DOD writes that there were "potential
variances between the actual aim point and impact point during
testing." Furthermore, we observed that the aim laser used by Army
testers was routinely out of line with where the ballistic was
penetrating the yaw card,[Footnote 63] despite continued adjustments to
line up the aim laser with where the ballistic was actually traveling.
DOD stated that it is not possible to know the reference point on a
curved object when the deepest deformation point is laterally offset
from the aim point. We disagree. DOD acknowledges in its response that
PEO Soldier had an internally documented process to account for plate
curvature when the deepest point of deformation was laterally offset
from the point of aim. The use of correction factor tables is a well-
known industry standard that has been in place for years, and this
standard practice has been used by NIJ laboratories and is well- known
by vendors.
DOD and the Army presented several statistics on the difference between
aim point back-face deformation and deepest point back-face deformation
in testing and stated that the difference between the two is small. We
do not agree with DOD's assertion that a difference of 10.66
millimeters is small. In the case of Preliminary Design Model testing,
the difference between measuring at the aim point and at the deepest
point was that at least two solutions passed Preliminary Design Model
testing that otherwise would have failed. These designs passed
subsequent First Article Testing but have gone on to fail lot
acceptance testing, raising additional questions regarding the
repeatability of the Aberdeen Test Center's testing practices.
DOD asserts that the adoption of the laser scanner measurement
technique resolves the problems the Army experienced in measuring back-
face deformations completely. We would agree that the laser scanner has
the potential to be a useful device but when used in the manner in
which Aberdeen Test Center used it - without an adequate certification
and without a thorough understanding of how the laser scanner might
effectively change the standard for a solution to pass - we do not
agree that it resolved back-face deformation measurement issues.
Aberdeen Test Center officials told us that they did not know what the
accuracy of the laser scanner was as it was used during First Article
Testing.
23. DOD acknowledged the shortcoming we identified. DOD then asserted
that once the deviation of measuring back-face deformation at the point
of aim, rather than at the deepest point of depression was identified,
those involved acted decisively to resolve the issue. We disagree based
on the timeline of events described in our response to DOD's comments
on Preliminary Design Model testing, as well as on the following facts.
We were present and observed the Integrated Product Team meeting on
March 25 and observed that all members of the Integrated Product Team
agreed to start measuring immediately at the deepest point, to score
solutions based on this deepest point data, to conserve plates, and
then at the end of the testing to make up the tests incorrectly
performed during the first third of testing, as needed. We observed
Army testers implement this plan the following day. Then, on March 27,
Army leadership halted testing for 2 weeks, considered the issue, and
then reversed the unanimous decision by the Integrated Product Team and
decided to score to the point of aim.
The deviation of scoring solutions based on the back-face deformation
at the point of aim created a situation in which the Army could not
have confidence in any solution that passed the Preliminary Design
Model testing. Because of this, the Army had to repeat testing, in the
form of First Article Testing, to determine whether the solutions that
had passed Preliminary Design Model testing actually met requirements.
24. DOD did not concur with our finding that rain may have impacted the
test results. DOD stated that such conditions had no impact upon First
Article Testing results. Our statistical analysis of the test data
shows failure rates to be significantly higher on November 13 than
during other days of testing, and our observations taken during that
day of testing and our conversations with industry experts familiar
with the clay, including the clay manufacturer, suggest the exposure of
the clay to the cold, heavy rain on that day may have been the cause of
the high failure rates. Our analysis examined the 83 plates tested
against the most potent threat, Threat D. The testing protocols
required that two shots for the record be taken on each plate. We
performed a separate analysis for the 83 first shots taken on these
plates from the 83 second shots taken on the plates. These confirmed
statistically that the rate of failure on November 13 was significantly
higher than the rate of failure on other days. Further, of the 5 plates
that experienced first-shot catastrophic failures during testing, 3 of
them (60 percent) were tested on November 13 and all 3 of these were
due to excessive back-face deformation. Given that only 9 plates were
tested on November 13, while 74 were tested during all the other days
of testing combined, it is remarkable that 60 percent of all
catastrophic failures occurred on that one day of testing.
DOD objected to our inclusion of no-test data in its calculation of
first-and second-shot failure rates on November 13. We believe that the
inclusion of no-test data is warranted because the Army's exclusion of
such plates was made on a post hoc basis after the shots were initially
recorded as valid shots and because the rationale for determining the
need for a re- test was not always clear. Additionally, we conducted an
analysis excluding the no-test plates identified by DOD and that
analysis again showed that the failure rate on November 13 was
statistically higher than during the other days of testing, even after
the exclusions. Excluding the no-test plates, 38 percent of first shots
on November 13 (3 of 8) and 88 percent of second shots (7 of 8) failed.
In its response, DOD reports that Aberdeen Test Center's own
statistical analysis of test data for Threat D reveals that the
observed failure rate on November 13 is attributable to the "poor
performance" of one design throughout testing. DOD asserts that its
illustration indicates that "Design K was the weakest design on all
days with no rain as well as days with rain." DOD's data do not support
such a claim. As we have observed, excluding no-test plates, DOD's data
are based on 10 tests of two shots each for each of 8 designs (160
cases total). Each shot is treated as an independent trial, an
assumption we find tenuous given that a plate's structural integrity
might be affected by the first shot. To account for date, DOD
subdivides the data into cell sizes far too small to derive reliable
statistical inferences about failure rates (between 2 and 6 shots per
cell), as evidenced by the wide confidence intervals illustrated in
DOD's visual representation of its analysis.
Among evidence DOD presented to support its claim that Design K was the
weakest performing design on both November 13 and other days is failure
rate data for four designs that were not tested on the day in question.
For two of the three designs tested on November 13 there were only one
or two plates tested on November 13, far too few to conduct reliable
statistical tests on differences in design performance. For the other
type of plate tested on that day (Design L), the three plates tested
had a markedly higher failure rate (3 of 6 shots, or 50 percent) on
that day than on other days (when it had, in 14 shots, 5 failures, or a
36 percent failure rate). Design K had a failure rate of 6 of 6 shots
(100 percent) on the day in question, compared with 8 of 14 shots (57
percent)[Footnote 64] on other days. Overall, it is impossible to
determine from such a small set of tests whether the lack of
statistical significance between different designs' failure rates on
November 13 and other days results from small sample size or a
substantive difference in performance.
Overall, the Army Test and Evaluation Command's design-based analysis
cannot distinguish between the potential effects of date and design on
failure rates because sufficient comparison data do not exist to
conduct the kind of multivariate analysis that might resolve this
issue. Because the data alone are inadequate for distinguishing between
the potential effects of date and design, we continue to recommend that
independent experts evaluate the potential effects of variations in
materials preparation and testing conditions, including those occurring
on November 13, on overall First Article Testing results.
Additionally, DOD stated that the clay is largely impervious to water.
However, as stated in our report, body armor testers from NIJ-certified
private laboratories, Army officials experienced in the testing of body
armor, body armor manufacturers, and the manufacturer of the clay used
told us that getting water on the clay backing material could cause a
chemical bonding change on the clay's surface.
DOD stated that one of its first actions when bringing in the clay is
to scrape the top of the clay to level it. However, this only removes
clay that is above the metal edge of the box. Clay that is already at
or below the edge of the box is not removed by this scraping. We
witnessed several instances in which the blade would remove clay at
some points, but leave large portions of the clay surface untouched
because the clay was below the edge of the box.
25. See comment 11.
26. The DOD is correct that the one particular example regarding
deleting official test data only happened once. Fortunately, the
results of the retest were the same as the initial test. After we noted
this deficiency, Army officials told us that a new software program was
being added that would prevent this from occurring again. DOD also
stated that only two persons are authorized and able to modify the
laser scanner software. We did not verify this statement; however, we
assert that DOD needs to have an auditable trail when any such
modifications are made and that it should require supervisory review
and documentation or logging of these setting changes.
27. DOD acknowledged that the Army did not formally document
significant procedure changes that deviated from established testing
protocols or assess the impact of these deviations.
28. In our report we stated that the requirement to test at an NIJ-
certified laboratory was withdrawn because the Aberdeen Test Center is
not NIJ-certified. DOD's comments on this point do not dispute our
statement. Instead, DOD discussed NIJ certification and stated that it
does not believe that NIJ certification is appropriate for its test
facilities. However, we did not recommend that any DOD test facilities
be NIJ-certified or even that NIJ be the outside organization to
provide an independent review of the testing practices at Aberdeen Test
Center that we did recommend. However, we believe NIJ certification
would meet our recommendation for an independent review.
Regarding DOD's comments regarding NIJ certification, DOD asserted that
NIJ certification is not appropriate for its test facilities and
asserted that there are significant differences between NIJ and U.S.
Army body armor test requirements. NIJ certification of a test
laboratory and NIJ protocol for testing personal body armor primarily
used by law enforcement officers are two distinct and different issues.
Similar to a consumer United Laboratories laboratory certification, an
NIJ laboratory certification[Footnote 65] includes an independent peer
review of internal control procedures, management practices, and
laboratory practices. This independent peer review is conducted to
ensure that there are no conflicts of interest, and that the equipment
utilized in the laboratory is safe and reliable. This peer review helps
to ensure a reliable, repeatable, and accurate test, regardless of
whether the test in question is following a U.S. Army testing protocol
or a law enforcement testing protocol. NIJ-certified laboratories have
consistently proven to be capable of following an Army testing
protocol, which is demonstrated by the fact that NIJ-certified
laboratories have conducted previous U.S. Army body armor source
selection testing in accordance with First Article Testing protocol, as
well as lot acceptance tests. The slide DOD included in its comments is
not applicable here because it deals with the difference between
testing protocols - the protocols for Army Interceptor Body Armor tests
and the NIJ protocol for testing personal body armor primarily used by
law enforcement officers. NIJ certification of a laboratory and NIJ
certification of body armor for law enforcement purposes are two
different things.
29. DOD stated that we were incorrect in asserting that the Army
decided to rebuild small arms ballistics testing facilities at Aberdeen
Test Center after the 2007 House Armed Services Committee hearing.
Instead, DOD stated that the contract to construct additional test
ranges at the Aberdeen Test Center Light Armor Range was awarded in
September 2006 and that construction was already underway at the time
of June 2007 hearing. DOD also stated that this upgrade was not in
response to any particular event but was undertaken to meet projected
future Army ballistic test requirements. Army officials we spoke with
before testing for this solicitation told us that this construction was
being completed in order to perform the testing we observed. As of July
2007, the Light Armor Range included two pre-WWII era ballistic lanes
and four modern lanes partially completed. However, we noted that, as
of July 2007, the lanes we visited were empty and that none of the
testing equipment was installed; only the buildings were completed.
In addition to the physical rebuilding of the test sites, the Amy also
re-built its workforce to be able to conduct the testing. As stated on
page 4 of DOD's comments, PEO Soldier has instituted an effort to
transfer testing expertise and experience from PEO Soldier to the Army
Test and Evaluation Command. Prior to the start of testing we observed
that Aberdeen Test Center hired, transferred in, and contracted for
workers to conduct the testing. These workers were then trained by
Aberdeen Test Center and conducted pilot tests in order to learn how to
conduct body armor testing. We observed parts of this training, in
person, and other parts via recorded video. In addition, we spoke with
officials during this training and preparation process. From our
observations and discussions with Army testers and PEO Soldier
officials, we believe this process to have been a restarting of small
arms ballistic testing capabilities at Aberdeen Test Center. Based on
DOD's comments, we clarified our report to reflect this information.
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
William M. Solis, (202) 512-8365:
Acknowledgments:
In addition to the contact named above, key contributors to this report
were Cary Russell, Assistant Director; Michael Aiken; Gary Bianchi;
Beverly Breen; Paul Desaulniers; Alfonso Garcia; William Graveline; Mae
Jones; Christopher Miller; Anna Maria Ortiz; Danny Owens; Madhav
Panwar; Terry Richardson; Michael Shaughnessy; Doug Sloane; Matthew
Spiers; Karen Thornton; and John Van Schaik.
[End of section]
Footnotes:
[1] DOD Inspector General, DOD Testing Requirements for Body Armor, D-
2009-047 (Arlington, Va.: Jan. 29, 2009); and U.S. Army Audit Agency,
Body Armor Testing: Program Executive Office, Soldier, A-2009-0086-ALA
(Alexandria, Va.: Mar. 30, 2009).
[2] GAO, Defense Logistics: Army and Marine Corps' Individual Body
Armor System Issues, [hyperlink,
http://www.gao.gov/products/GAO-07-662R] (Washington, D.C.: Apr. 26,
2007); and Defense Logistics: Army and Marine Corps' Body Armor
Requirements, Controls, and Other Issues, [hyperlink,
http://www.gao.gov/products/GAO-07-911T] (Washington, D.C.: June 6,
2007).
[3] The designs submitted by that manufacturer also failed Preliminary
Design Model testing at Aberdeen Test Center.
[4] The armor plate contracts require First Article Testing, in
accordance with the Federal Acquisition Regulation, Subpart 9.3, to
ensure the contractor can furnish a product that conforms to all
contract requirements for acceptance. However, the standard Federal
Acquisition Regulation First Article Testing clause allows the
government to waive First Article Testing if a design has already been
demonstrated to meet the required specifications.
[5] Indefinite delivery/indefinite quantity contracts provide for an
indefinite quantity of supplies or services during a fixed period of
time. These types of contracts are generally used when agencies are
unable to predetermine, above a specified minimum, the precise
quantities of supplies or services that the government will require
during the contract period.
[6] We also issued two decisions on bid protests concerning testing
under the solicitation. Armorworks Enters., LLC.,B- 400394, B-400394.2,
Sept. 23, 2008, 2008 CPD para. 176 (protest denied in part and
dismissed in part) and Armorworks Enterprises, LLC, B-400394.3, Mar.
31, 2009, 2009 CPD para. 79 (protest dismissed).
[7] In addition to stopping bullets, body armor absorbs and dissipates
the force of the impact of these bullets. The amount of force absorbed
is determined by measuring the depth of the depression--called back-
face deformation--caused to the clay placed behind the body armor
during ballistic testing: the lower the back-face deformation, the more
force that is absorbed by the body armor. See figures 4 and 5 for
examples of back- face deformation.
[8] After testers realized they were incorrectly measuring back-face
deformation at the point of aim rather than at the deepest point,
testers began to measure to both points but used the point-of-aim
measure as the official measure, which according to Army officials was
necessary to maintain consistency throughout testing and to not
disadvantage any vendors. These two designs would have failed if the
deepest point measure recorded had been used as the official measure.
Because the deepest point was not measured during the first third of
testing, additional designs could have improperly passed.
[9] Prior to Preliminary Design Model testing a body armor manufacturer
whose design failed a prior test made public allegations that PEO
Soldier had an unfair bias against its design. In an attempt to remove
any appearance of bias against that manufacturer, PEO Soldier made a
decision to not provide an on- site presence during Preliminary Design
Model testing.
[10] Army and private laboratory officials told us that, on the basis
of the limited data they had previously collected, they were concerned
that the laser scanner may overstate back-face deformation measurements
by about 2 millimeters as compared with the measurements obtained by
using the digital caliper. We did not independently verify or validate
the data provided by these officials. Since standards are based on
measurements obtained with a digital caliper, results obtained using
the laser scanner may be inconsistent/different than those obtained
using the digital caliper.
[11] This design is also one of the ones that would have failed
Preliminary Design Model testing had back-face deformations been
measured to the deepest point as required by the testing protocols.
[12] A set of protective plates comprises two plates--one front and one
back plate.
[13] After it was discovered that back-face deformation was being
measured incorrectly, Preliminary Design Model testing was halted for 2
weeks so that Army officials could consult with senior Army leadership
on how to best resolve the issue.
[14] Testing was halted for other high-priority tests involving 2,000
plates from Iraq that were identified as potentially cracked by
nondestructive testing performed by the Army.
[15] From November 14 to November 19 First Article Testing was halted
to allow for higher-priority testing to be conducted. Nearly all the
ballistic testing was conducted between November 10 and December 4. The
testing conducted prior to November 10 was mainly physical
characterization of the plates, and the testing after December 4 was
limited to the retesting of a single plate that the Army had identified
as being tested incorrectly.
[16] Even though Aberdeen Test Center is not an NIJ-certified facility,
Aberdeen Test Center officials said they are actively keeping abreast
of NIJ standards, have made adjustments to their procedures based on
those standards, and consider those standards when evaluating their own
testing practices. Although there remains an active discussion in the
Army testing community as to whether Aberdeen Test Center should pursue
certification, Aberdeen Test Center currently has no plans to pursue
NIJ certification.
[17] Testing protocols require that clay be calibrated by dropping a 1-
kilogram cylindrical weight on the clay in three locations. If all
drops cause indentations between 22 and 28 millimeters, the clay is
acceptable for use.
[18] Specifications include factors such as firing the shot at proper
velocity, in conditions with correct humidity and temperature, and
using properly conditioned clay.
[19] A "pass" is any plate that is not a limited or catastrophic
failure.
[20] A "limited failure" for threats A, B, C, F, and Y is either (1)
complete penetration of hard armor (the plate), but partial penetration
of the soft armor (shoot pack) on any shot or (2) a back-face
deformation greater than 43 millimeters but less than 48 millimeters. A
limited failure on threats D and X is either (1) a complete penetration
of hard armor (plate), but a partial penetration of the soft armor
(shoot pack) on the first shot or (2) a complete penetration of both
the hard armor (plate) and the soft armor (shoot pack) on a second shot
or (3) back-face deformation on the first shot greater than 43
millimeters, but less than 48 millimeters or (4) back- face deformation
greater than 43 millimeters on a second shot.
[21] A "catastrophic failure" for threats A, B, C, F, and Y is either
(1) a complete penetration of the hard armor (plate) and the soft armor
(shoot pack) on any shot or (2) a back-face deformation on any shot
greater than or equal to 48 millimeters. A catastrophic failure for
threats D and X is either (1) a complete penetration of both the hard
armor (plate) and soft armor (shoot pack) on a first shot or (2) a
first shot back-face deformation greater or equal to than 48
millimeters.
[22] The "yaw card" is a piece of paper placed in the intended path of
the ballistic and is meant to measure the amount of yaw, or wobble, of
the ballistic as it travels through the air.
[23] Testing protocols use the acronym OTV for Outer Tactical Vest.
[24] Nine test items could not be measured either because they were
marked in a way that could not be measured or because the impact of the
bullet deformed the plate too severely.
[25] One of the designs that passed Preliminary Design Model testing
later failed First Article Testing because of its results during the
impact test. Thus, it is possible that this design may have passed
Preliminary Design Model testing due to shooting the plate at the wrong
location, resulting in additional testing costs.
[26] These drops comprise dropping a cylindrical metal apparatus onto
the clay backing material and measuring the amount of depression caused
by the drop.
[27] The Milestone Decision Authority (MDA) is the designated
individual with overall responsibility for a program. According to DOD
Directive 5000.01, the MDA shall have the authority to approve entry of
an acquisition program into the next phase of the acquisition process
and shall be accountable for cost, schedule, and performance reporting
to higher authority, including congressional reporting.
[28] When asked, Aberdeen Test Center officials could not produce a
memo documenting this procedure or how they knew that it was
consistently applied during the test.
[29] The Aberdeen Test Center dropped the procedure that measures
depression at the aim point location used during Preliminary Design
Model testing.
[30] We did not conduct an independent assessment of the
appropriateness of re-testing failed clay.
[31] We analyzed all V0, threat D shots. We excluded V50 shots, as well
as shots from all other threats either because those tests consistently
did not result in penalty points or because those threats were not
tested on November 13, 2008.
[32] Testing officials disputed the inclusion of one of the plates in
our analysis because it was ruled a no-test. We included this plate
because we had a complete set of data for the test item and it was
ruled a valid test on the lane, only to be discarded several days later
because testing officials believed one of the shots was "questionable."
Based on the Army's objection, we analyzed the data without this plate
or two other no-test plates and found 38 percent of first shots
resulted in penalties. Furthermore, our analysis revealed that the
proportion of failures or penalties on November 13, 2008, still
differed substantially and/or significantly from the proportion on all
other days.
[33] Roma Plastilina Number 1, manufactured by Chavant, Inc.
[34] We did not independently validate the information provided by
these officials.
[35] We reviewed a few test reports for body armor testing and found
instances where back-face deformation results were rounded and
instances where they were not rounded.
[36] We did not evaluate the validity of the certification; however, it
is worthy of note that the certification report states that the method
of analysis used was somewhat unusual, that some of the results were
discarded because of problems with the laser, and that changes were
made to the laser during the testing process.
[37] We did not independently evaluate the manufacturer's description
of the capabilities of the laser scanner.
[38] In addition, after First Article Testing was concluded Aberdeen
Test Center installed additional software upgrades needed to correct
errors discovered in subsequent tests. These errors, which were not
identified until after First Article Testing was concluded, affected
the First Article Testing results.
[39] We did not independently verify the level of accuracy of the
digital caliper. However, the manufacturer's stated accuracy is .01
millimeters for the digital caliper specifications we obtained.
[40] This solution is also one of the ones that would have failed had
back-face deformations been scored at the deepest point, rather than at
the point of aim, during Preliminary Design Model Testing
[41] We observed Kevlar fibers that were frayed and tattered.
[42] GAO, Internal Control: Standards for Internal Control in the
Federal Government, [hyperlink,
http://www.gao.gov/products/GAO/AIMD-00-21.3.1] (Washington, D.C.
1999).
[43] The one design that would have passed both the Preliminary Design
Model testing and the First Article Testing actually suffered a
catastrophic first-shot penalty during First Article Testing, on
November 13, 2008. However, Army testers later deemed this a
"questionable" shot and ruled it a no-test. The design subsequently
passed its re-test.
[44] NIJ Standard, Section 3.34 is consistent with this definition,
Ballistic Resistance of Body Armor, NIJ Standard- 0101.06, July 2008.
[45] Army protocols require only a series of three pre-shot calibration
drops. NIJ Section 4.2.5.6 requires that a series of five pre-shot and
a series of five post-shot calibration drops be within specification or
a new conditioned and calibration drop validated clay be used--
Ballistic Resistance of Body Armor, NIJ Standard-0101.06, July 2008.
[46] DOD members should not have veto power over non-DOD members.
[47] Testers marked an area of intended impact by drawing two long
lines, one marking the inner shot tolerance and the other marking the
outer shot tolerance. Both our measurements and those taken by Aberdeen
Test Center testers were taken by measuring the distance between the
two lines and the edge on a part of the test sample significantly
removed from where the shot actually impacted. We could only take these
measurements on hard plate samples because the flexible samples were
marked differently, in a way that we could not obtain an accurate
measurement.
[48] Ballistics testing was stopped on one occasion because of a higher
priority Army test that needed to be conducted at Aberdeen Test Center
that involved cracked plates shipped from Iraq as part of PEO Soldier's
non-destructive X-ray life cycle testing. Most of the First Article
Testing concluded on December 4, but one retest was conducted on
December 17.
[49] During Preliminary Design Model testing, the most current Army
Test Operating Procedure for testing body armor had not been updated
since 1975. Test Operations Procedure (TOP), 10-2-506 Ballistic Testing
of Personnel Armor Materials. January 6, 1975.
[50] Member of the Integrated Product Team (IPT).
[51] Omitted from this list of agencies agreeing that First Article
Testing was part of the original testing plan are (1) PEO Soldier, the
Army's materiel developer and product manager for individual protection
equipment being tested and the contracting officer, and (2) the U.S.
Army Research, Development, and Engineering Command's Contracting
Agency. Both entities told us that First Article Testing was going to
be waived.
[52] Form, fit, and function and the test for high ballistic
statistical confidence were not part of First Article Testing. Form,
fit and function tests involved having soldiers wear the body armor and
evaluate its comfort and suitability when performing deployment (war-
like) activities-- egression from armored vehicles, the double-time
run, moving through an obstacle course, and discharging their weapons.
[53] DODIG Report No. D-2008-067, March 31, 2008, DOD Procurement
Policy for Body Armor.
[54] Depending on the type of design (i.e., ESAPI or XSAPI) a design
can accumulate either 6 or 10 penalty points before being eliminated
from consideration. The designs in question were 1.0 point, or one
penalty, away from failing.
[55] The law-enforcement community relies on NIJ-certified laboratories
to conduct their body armor testing and ensure that their body armor
meets law enforcement levels of protection.
[56] Software upgrades were not part of the certification process. Some
of these software upgrades eliminate the deepest point of depression
measurement.
[57] A vendor test showed an approximately 2-millimeter overstatement
of back-face deformation measurements by the laser as compared to the
caliper.
[58] Lot acceptance testing provides additional ballistic testing that
ensures that the plates delivered meet requirements before they are
accepted. Two vendors whose designs passed Preliminary Design Model
testing and First Article Testing have failed lot acceptance testing
and in July 2009 submitted to the Army, in one case, a ruling on a
request for equitable adjustment and, in another case, a request to
waive contract penalties for late deliveries. These vendors have failed
several lot acceptance tests involving tens of thousands of plates that
have been rejected by the government because they failed this testing.
One vendor is asking for several millions of dollars in payment to
compensate for material, labor, and delays as a result of the failed
lots.
[59] According to Army officials, during subsequent lot acceptance
testing tests, Aberdeen Test Center technicians were covering the clay
boxes during transport from the conditioning ovens to the lanes.
[60] The Army Test and Evaluation Command performed these tests and the
office of the Director of Operational Test and Evaluation provided
oversight of Preliminary Design Model testing and First Article Testing
for the current solicitation to include determining the scope of
testing required and approving the test plans. PEO Soldier provided
subject matter experts to advise Army testers, developed the purchase
descriptions, and approved test plans. Therefore, these entities are
part of the program that needs to be reviewed and are not independent.
Additionally, any other individuals and organizations associated with
the Preliminary Design Model testing, First Article Testing, or lot
acceptance testing should also be excluded.
[61] U.S. Army Audit Agency, Body Armor Testing PEO Soldier; Audit
Report: A-2009-0086-ALA, 30 March 2009--Just before the current
solicitation, from January 2007 to June 2008, all 27 Army First Article
Testing for new designs associated with ESAPIs (four vendors), their
associated 1,024 lot acceptance quality assurance ballistic testing,
and the long-term environmental conditions testing were all performed
in an independent NIJ-certified testing facility.
[62] Nine test items were not able to be measured due either to the
absence of lines or due to damage caused by the impact of the
ballistic.
[63] The yaw card is a piece of paper placed in the intended path of
the ballistic and is meant to measure the amount of yaw, or wobble, of
the ballistic as it travels through the air. We observed that the hole
made by the bullet in the yaw card was routinely not in line with where
the aim laser was pointing.
[64] According to official test data, only 7 of these 14 shots were
failures (50 percent). This is due to the Army's practice of
incorrectly rounding down back-face deformations during First Article
Testing. One shot that resulted in a back-face deformation of 43.306
was officially rounded down to 43 and not penalized, but had Army
testers followed the protocols and not rounded this result down, 8 of
the 14 shots would have resulted in penalties.
[65] The U.S. Department of Justice offers this multi-departmental
voluntary compliance program.
[66] On June 21, 2007, the subject company, Pinnacle Body Armor, Inc.
was the subject of proposed debarment by the Department of the Air
Force. On July 16, 2009, the Armed Services Board of Contract Appeals
subsequently found the government's termination for cause of Pinnacle
Body Armor, Inc. justified and denied its appeal.
[67] Letter to Senators Levin and McCain, dated July 12, 2007.
[68] The overall concept consists of PDM testing (Phase I), FAT, and an
extended ballistic test to gather empirical data to support a new DoD
standard for body armor testing (Phase II). Phase II testing is nearing
completion as of the date of this communication.
[69] American Society for Testing and Materials, ASTM E-29, Standard
Practice for Using Significant Digits in Test Data to Determine
Conformance with Specifications. Approved for DoD use.
[70] The GAO noted in its report that they had reviewed past test
reports and found instances of rounding and instances of not rounding.
[71] Correction factors account for the curvature of the armor plates
when making a perpendicular measurement from a reference plane. This is
explained in a later section of this letter. This issue is also noted
in DoD IG Report, "DoD Testing Requirements for Body Armor," dated
January 29, 2009.
[72] Report #08-MS-25, "Quantum FARO Laser Scanning Body Armor Back-
Face Deformation" Warfighter Directorate, Applied Science Test
Division, ATC, dated September 23, 2008.
[73] "No Test" events result from test anomalies such as too high of a
striking velocity or impact to the test article at a location far
different from the intended.
[74] NIJ 0101.06, Ballistic Resistance of Body Armor, July 2008.
[75] The NIJ also removed specific thermal conditioning requirements
from NIJ 0101.06, instead indicating that, "Actual conditioning
temperature and recovery time between uses will be determined by the
results of the validation drop test..."
[76] NIJ 0101.03, page 7, Section 5.2.9.
[77] Test Operating Procedures (TOPs) are formal documents published by
ATEC that describe how tests are to be conducted.
[78] Testing with threat "c" nonetheless required to ensure that the
ballistic plate defeats all threats it is designed to defeat.
[79] Figure 3 in this letter is the "Figure 2" cited in the quote from
the Purchase Description.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E- mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black
and white. Pricing and ordering information is posted on GAO‘s Web
site, [hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
