Chemical Weapons
Lessons Learned Program Generally Effective but Could Be Improved and Expanded Gao ID: GAO-02-890 September 10, 2002The Army has been tasked to destroy 31,500 tons of highly toxic chemical agents by April 2007, the deadline set by an international treaty for the elimination of all chemical weapon stockpiles. To destroy the weapons, the Department of Defense (DOD) established the Army Chemical Demilitarization Program. The Army has destroyed over one-quarter of the U.S. stockpile as of March 2002. Originally, the Chem-Demil Program consisted only of the Chemical Stockpile Disposal Project, which was initiated in 1988 to incinerate chemical weapons at nine storage sites. In response to public concern about incineration, in 1994 Congress established the Alternative Technologies and Approaches Project to investigate alternatives to the baseline incineration process. The Chemical Stockpile Disposal Project operates a Programmatic Lessons Learned Program whose aim is to enhance safety, reduce or avoid unnecessary costs, and maintain the incineration schedule. This program has successfully supported the incineration project's primary goal to safely destroy chemical weapons and has captured and shared many lessons from past experiences and incidents. However, the Lessons Learned Program does not fully apply generally accepted knowledge management principles and lessons sharing best practices, thereby limiting its effectiveness. The program's management plan does not provide policy guidance for senior managers to help them in decision-making or daily operations. In addition, it does not have formal procedures to test or validate whether a corrective action has been effective in resolving its deficiency. Finally, the lessons learned database is difficult to search and does not prioritize lessons. The Lessons Learned Program has been effective in sharing knowledge among the different stakeholders within the Chemical Stockpile Disposal Project. However, as new components were created to destroy the stockpile, the scope of the Lessons Learned Program remained primarily limited to the incineration project. As a result, some components that could greatly benefit from timely and full sharing of lessons learned with the incineration project are not doing so.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-02-890, Chemical Weapons: Lessons Learned Program Generally Effective but Could Be Improved and Expanded
This is the accessible text file for GAO report number GAO-02-890
entitled 'Chemical Weapons: Lessons Learned Program Generally Effective
but Could Be Improved and Expanded' which was released on September 10,
2002.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products‘ accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
Report to Congressional Requesters:
United States General Accounting Office:
GAO:
September 2002:
CHEMICAL WEAPONS:
Lessons Learned Program Generally Effective but Could Be Improved and
Expanded:
Chemical Weapons:
GAO-02-890:
Contents:
Letter:
Results in Brief:
Background:
Lessons Learned Program Has Made Positive Contributions but Needs
Improvement:
Sharing of Lessons Learned Could Be Expanded:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Information on the Incineration Process and
Incidents at Three Sites:
Appendix II: Scope and Methodology:
Appendix III: Lessons Learned Process:
Appendix IV: Chemical Demilitarization Program Management
Developments,
1997-2001:
Appendix V: Comments from the Department of the Army:
Tables:
Table 1: Status of the Chemical Stockpile Disposal Project:
Table 2: May 2000 Agent Release at Tooele Chemical Agent Disposal
Facility:
Figures:
Figure 1: Chem-Demil Programmatic Lessons Learned Program Process:
Figure 2: Lessons Learned Stakeholders and Process Steps:
Figure 3: Chemical Demilitarization Program Organization Chart:
Abbreviations:
DFS: deactivation furnace system
DOD: Department of Defense
GAO: General Accounting Office
LIC: liquid incinerator
SDS: spent decontamination solution:
Letter:
September 10, 2002:
The Honorable Jeff Sessions
The Honorable Gordon Smith
The Honorable Ron Wyden
United States Senate:
The Honorable James V. Hansen
The Honorable Duncan Hunter
The Honorable Bob Riley
House of Representatives:
The Army has been tasked to destroy about 31,500 tons of highly toxic
chemical agents by April 2007, the deadline set by an international
treaty for the elimination of all chemical weapon stockpiles. Until
they are destroyed, the chemical agents will continue to pose a threat
to the thousands of people living and working near the disposal
facilities where the agents are being stored. To destroy the weapons,
the Department of Defense (DOD) established the Army‘s Chemical
Demilitarization (or Chem-Demil) Program. The Army has destroyed over
one-quarter (8,044 tons) of the U.S stockpile as of March 2002.
Originally, the Chem-Demil Program consisted only of the Chemical
Stockpile Disposal Project, also known as the baseline incineration
project, which was initiated in 1988 to incinerate chemical weapons at
nine storage sites. Then, in response to public concern about
incineration, Congress established the Alternative Technologies and
Approaches Project in 1994 to investigate alternatives to the baseline
incineration process. In 1997, Congress established the Assembled
Chemical Weapons Assessment Program to identify and test additional
technologies as alternatives to incineration. Today, five of the nine
storage sites use incineration; three others will use or plan to use
alternative technologies. The technology choice for the final site has
yet to be determined.
The Chemical Stockpile Disposal Project operates a Programmatic Lessons
Learned Program whose aim is to enhance safety, reduce or avoid
unnecessary costs, and maintain the incineration schedule. A lesson
learned is a set of rules or principles that summarizes past
experiences to help people better perform future tasks. The project‘s
goal is to capture and share lessons learned from experience so that
stakeholders--engineers, contractors, and program managers--working in
similar situations on new facilities can apply the knowledge. A lesson
learned is thus the product of a process through which lessons are
captured and shared with stakeholders.
After a chemical agent was accidentally released at one of the
project‘s facilities in May 2000, some Members of Congress and state
and local communities near disposal sites became increasingly concerned
about the overall safety at the Chem-Demil Program‘s incineration
facilities. In July 2000, you requested that we report on the status of
the Chemical Stockpile Emergency Preparedness Program and on the
Programmatic Lessons Learned Program. We issued a report in August 2001
on the Emergency Preparedness Program.[Footnote 1] For this second
report, we (1) assessed whether the Lessons Learned Program has
effectively captured and shared lessons to support the Chem-Demil
Program‘s goal to safely destroy the chemical stockpile and (2)
identified the extent to which lessons learned have been shared and
areas where sharing could be improved. You also asked us to provide
additional information on incidents at three sites and the corrective
actions taken following the incidents. The information is in appendix
I.
In performing our analysis, we used the underlying principles of
’knowledge management“ and lessons sharing best practices as the
criteria for assessing the program systems that capture and share
lessons learned. Both DOD and the Army endorse lessons learned
systems.[Footnote 2] Knowledge management includes four fundamental
principles: leadership that articulates management‘s vision and goals
(e.g. in written policies and guidance), processes (including
performance measurements) to turn vision into reality, technology that
allows implementation of goals and supports the processes, and a
culture of knowledge sharing and reuse. Together they create an
environment in which a lessons learned program can successfully
function.
We conducted our review from October 2001 to May 2002 in accordance
with generally accepted government auditing standards. See appendix II
for a description of our scope and methodology.
Results in Brief:
The Lessons Learned Program has successfully supported the incineration
project‘s primary goal to safely destroy chemical weapons. The program
has captured and shared many lessons from past experiences and
incidents. It has leadership that communicates the importance of the
lessons learned program in supporting the Chem-Demil Program‘s mission,
processes for capturing and sharing lessons, and a technology to
facilitate and support the program. It also has developed a culture
that promotes using lessons to foster safe operations. However, the
Lessons Learned Program does not fully apply generally accepted
knowledge management principles and lessons sharing best practices,
thereby limiting its effectiveness.
* The program‘s management plan does not provide policy guidance for
senior managers to help them in decision making or daily operations.
Guidance is needed especially if managers decide not to implement a
lesson learned. In at least one case, this resulted in cost avoidance
prevailing with serious safety, cost, and schedule consequences. The
program also does not define performance measures or provide incentives
for participation.
* The Lessons Learned Program does not have formal procedures to test
or validate whether a corrective action has been effective in resolving
a deficiency.
* The lessons learned database is difficult to search and does not
prioritize lessons. These shortcomings not only make it difficult to
verify or validate corrective actions but also may discourage some from
using the database, with potentially serious consequences.
The Lessons Learned Program has been effective in sharing knowledge
among the different stakeholders within the Chemical Stockpile Disposal
Project. However, as new components were created to destroy the
stockpile, the scope of the Lessons Learned Program remained primarily
limited to the incineration project. No policies or procedures were
established to ensure that lessons sharing would expand to all
components of the Chem-Demil Program. As a result, some components that
could greatly benefit from timely and full sharing of lessons learned
with the incineration project are not doing so. This can lead to higher
risk and costly duplication and delays. The Assembled Chemical Weapons
Assessment Program and the Alternative Technology and Approaches
Project in particular could find full participation in the program
useful because the majority of the processes they use are the same as
those used by the incineration project. Lessons sharing best practices
would dictate that all Chem-Demil Program components share important
information such as lessons learned because they are all part of the
same program with a common objective.
We are making recommendations to help improve the operation and overall
usefulness of the Programmatic Lessons Learned Program.
Background:
In 1985, Congress required the Department of the Defense to destroy the
U.S. stockpile of chemical agents and munitions and to establish an
organization within the Army to manage the agent destruction program.
Later, Congress also directed DOD to research and develop technological
alternatives to incineration for disposing of chemical agents and
munitions. These activities evolved into the Chem-Demil Program. The
Chem-Demil Program includes the Chemical Stockpile Emergency
Preparedness Program, created in 1988, to enhance the emergency
management and response capabilities of communities near the storage
sites in case of an accident. [Footnote 3] The Nonstockpile Chemical
Materiel Product was added in 1993 to destroy any chemical weapons or
materiel not included in the stockpile disposal program.
The Chemical Stockpile Disposal Project has or plans to use
incineration to destroy chemical agents at five sites: Johnston Atoll
in the Pacific Ocean; Anniston, Alabama; Pine Bluff, Arkansas;
Umatilla, Oregon; and Tooele, Utah. Tooele is the only site with a
facility currently operating. The three other stateside facilities are
scheduled to begin operations in fiscal years 2002-2003. The Johnston
Atoll facility has finished destroying its stockpile and is being
closed. The Alternative Technologies and Approaches Project will use
non-incineration methods (such as agent neutralization by chemical
treatment) to destroy agents in bulk containers at Newport, Indiana,
and Aberdeen, Maryland.[Footnote 4] The Assembled Chemical Weapons
Assessment Program is also researching alternative methods to destroy
agents in weapons at Pueblo, Colorado, and Blue Grass, Kentucky.
The Office of the Secretary of Defense and the Department of the Army
share management roles and responsibilities in the Chem-Demil Program.
The Program Manager of the Assembled Chemical Weapons Assessment
Program reports to the Under Secretary of Defense for Acquisition,
Technology, and Logistics. Thus, it is independent of the Program
Manager for Chemical Demilitarization, who reports to the Assistant
Secretary of the Army (Installations and Environment).
In 1997, the United States ratified the Chemical Weapons
Convention,[Footnote 5] a treaty committing member nations to dispose
of selected chemical agents and materiel by April 29, 2007. In
September 2001, the Army updated the life cycle cost estimate for the
Chem-Demil Program from $15 billion to $24 billion. The new cost
estimate extended the agent destruction schedule at four of the eight
stateside sites beyond the initial target date of April 2007.[Footnote
6] Despite setbacks experienced at Johnston Atoll, Tooele, Utah, and
Umatilla, Oregon, among others, the incineration program has
successfully destroyed over 25 percent of the original stockpile (see
table 1).
Table 1: Status of the Chemical Stockpile Disposal Project:
Percent of total stockpile destroyed; Johnston Atoll: ; 6; Tooele,
Utah: ; 19; Anniston,
Alabama: ; 0; Umatilla,
Oregon: ; 0; Pine Bluff,
Arkansas: ; 0.
Start of operations; Johnston Atoll: June 1990; Tooele,
Utah: Aug. 1996; Anniston,
Alabama: 4th quarter FY02; Umatilla,
Oregon: 4th quarter FY03; Pine Bluff,
Arkansas: 4th quarter FY03.
End of operations; Johnston Atoll: Nov. 2000; Tooele,
Utah: 4th quarter FY05; Anniston,
Alabama: 3rd quarter FY09; Umatilla,
Oregon: 2nd quarter FY09; Pine Bluff,
Arkansas: 3rd quarter FY09.
Current phase; Johnston Atoll: Closure; Tooele,
Utah: Operations; Anniston,
Alabama: Systemization[A]; Umatilla,
Oregon: Systemization[A]; Pine Bluff,
Arkansas: Construction.
[LEGEND] :
FY = fiscal year:
[A] Testing of each incineration system.
Source: Program Manager for Chemical Demilitarization.
[End of table]
The Lessons Learned Program was created in part because many different
contractors were involved in the incineration program, and a system was
needed to collect and preserve the institutional knowledge and acquired
experience.[Footnote 7] The program is intended to identify, capture,
evaluate, store, and share (implement) lessons learned during the
different phases of the chemical stockpile demilitarization process. It
collects two different kinds of lessons: ’design“ lessons covering
engineering and technical processes and ’programmatic“ lessons
involving management, quality assurance, emergency response, and public
outreach. As criteria for assessing the knowledge management processes
used by the Lessons Learned Program, we identified four of a number of
federal organizations that practice knowledge management and operate
lessons learned programs. In making our selections, we reviewed
literature and spoke with knowledge management experts to find
organizations recognized for their ability to share lessons or
effectively manage knowledge. We identified the following
organizations: the Center for Army Lessons Learned, the Department of
Energy, the U.S. Army Corps of Engineers, and the Federal Transit
Authority (for more details, see appendix II).
There are two levels of authority involved in developing lessons
learned from proposed engineering changes. A Configuration Control
Board composed of headquarters staff in the Office of the Program
Manager for Chemical Demilitarization has authority to approve, reject,
or defer engineering change proposals that involve costs above a set
limit or affecting multiple sites. The Field Configuration Control
Boards have authority over changes at their sites involving lower
costs. In September 2001, the Lessons Review Team (consisting of
headquarters staff) was established to screen all lessons and
engineering changes and provide the information needed to determine
which lessons require a response from sites. For more information on
the lessons learned process, see appendix III.
Lessons Learned Program Has Made Positive Contributions but Needs
Improvement:
The Lessons Learned Program has made valuable contributions in support
of the Chemical Stockpile Disposal Project‘s efforts to safely destroy
the chemical stockpile. It has generally operated consistently with
knowledge management principles and lessons sharing best practices and
has successfully captured and shared thousands of lessons. However, the
program does not apply or incorporate all knowledge management
principles and lessons sharing best practices. For example, the program
does not provide needed guidance for senior managers; it does not have
formal a validation procedure to determine whether a problem has been
fixed; and the database of lessons learned needs improvement.
Important Program Contributions:
The Lessons Learned Program has contributed to the Chem-Demil Program‘s
goal of destroying the chemical weapons stockpile while promoting
safety, maintaining schedule, and saving or avoiding costs. We found
that the Chem-Demil Program‘s management, through its leadership,
encourages headquarters, field staff, and contractor personnel in the
incineration program to use the Lessons Learned Program. It has
provided funding and has established processes to capture, evaluate,
store, and share lessons. It is committed to continuous improvement and
has provided the technology needed to support the lessons learned
process. Finally, it fosters a culture in which knowledge sharing is an
important element of day-to-day operations.
While it is difficult to quantify the benefits of each lesson,
available data indicate that lessons learned have generally helped
avoid on-the-job injuries (by using government-furnished-approved
tools that are better suited to specific tasks), reduce costs (by
improving the containers used to transport weapons), or maintain
schedules (by improving the design of a socket to disassemble weapons).
We also found that lessons from accidental releases of chemical agents
at Johnston Atoll and Tooele, Utah, were implemented at other
incineration sites under construction, thus incorporating improvements
into the design of those new facilities.
Program Lacks Guidance to Support Managers‘ Decision Making:
The Lessons Learned Program does not have guidance explaining how
senior managers (at headquarters) should use it in support of their
decision making process. Specifically, there is no guidance that
defines the procedures to be followed when an alternative to a lesson
is chosen or when a lesson is not implemented. Lessons learned guidance
for another federal government agency recommends that lessons be used
to optimize management decision making and to interact with other
management tools such as reviews, investigations, root-cause analyses,
and priorities.[Footnote 8]
We reviewed documentation of lessons learned from incidents at the
Johnston Atoll and Tooele, Utah facilities, and found that three other
facilities--Anniston, Umatilla, and Pine Bluff--had not implemented a
lesson that had evolved from problems with pipes in the pollution
abatement systems.[Footnote 9] The Tooele site had used a superior and
more expensive material (hastelloy) to fix their problem than the
material used at the other sites.[Footnote 10] Headquarters decided not
to implement the lesson at the three sites primarily because it would
have involved higher initial costs.[Footnote 11] This decision
ultimately caused serious safety concerns, higher costs, and delayed
the schedule. In February 2002, pipes at Anniston had failures similar
to those experienced at the first two sites. This raised safety
concerns and resulted in a 4-week delay to replace the pipes with
hastelloy. It is too early to determine whether the material used at
the Umatilla and Pine Bluff sites will have the same problems. Although
they need flexibility to manage the program, senior managers also need
guidance to help make decisions that allow them to consider the
potential impact of not implementing lessons learned. This process
would include safety and risk analyses that can provide criteria should
they decide not to adopt a lesson learned.
Program Lacks a Procedure to Validate Lessons Implemented:
There is no formal procedure to ensure that the lessons or corrective
actions that have been implemented have fully addressed a deficiency.
Chem-Demil Program guidance for engineering change proposals does
require that changes be tracked and reported after implementation, but
there is no similar requirement in the guidance for the Lessons Learned
Program (which includes programmatic lessons). Both contractor and
incineration project officials also confirmed that there are no
procedures for monitoring the effectiveness of corrective actions. As a
result, a problem could reoccur and affect safety and costs.
As shown in figure 1, the Lessons Learned Program process does not
contain the final validation stage (dashed line), which most knowledge
management systems and Army guidance consider as a necessary step. As
we previously reported, Army guidance states that lessons learned
programs should have a means for testing or validating whether a
corrective action has resolved a deficiency.[Footnote 12] The standard
issued for another federal lessons learned program[Footnote 13]
indicates that analyses should be made to evaluate improvements or to
identify positive or negative trends. The standard also states that
corrective actions associated with lessons learned should be evaluated
for effect and prioritized. Without such a validation procedure in the
architecture of the Lessons Learned Program, there is little assurance
that problems have been resolved, and the possibility of repeating past
mistakes remains.
Figure 1: Chem-Demil Programmatic Lessons Learned Program Process:
[See PDF for image]
Note: PLL (Programmatic Lessons Learned) is referred to in this report
as the Lessons Learned Program; in the figure, engineering change
proposal is referred to as ECP.
Source: Our analysis, based on data from PLL.
[End of figure]
Database Is Difficult to Use, Lessons Are Not Prioritized:
The lessons learned database includes about 3,400 issues, 3,055
engineering change proposals, and 2,198 lessons. But it is not easy to
obtain fast and ready access to relevant information. Furthermore, the
lessons in the database are not prioritized, making it difficult to
identify which lessons are most important and which need to be verified
and validated.
It is important that an organization employ appropriate technology to
support the participants of a lessons learned program. Having a
technology be available does not automatically guarantee its use or
acceptance. According to lessons sharing best practices,[Footnote 14]
the goal of technology is to (1) match a solution to users‘ needs, (2)
establish a simple content structure so that items may be found easily
and retrieved quickly, and (3) deliver only relevant information from
all possible sources. According to database users we interviewed and
surveyed, it is difficult to find lessons because the search tool
requires very specific key words or phrases, involves multiple menus,
and does not link lessons to specific events. As a result, some users
are reluctant to use the database and thus may not benefit from it when
making decisions that affect the program.[Footnote 15] Many users who
responded to our survey stated that they experienced difficulties in
searching the database, and some we interviewed described specific
problems with searches. One described the database as ’frustrating.“ We
tested the search tool and also had difficulty finding lessons linked
to specific incidents.
Users we interviewed made a number of suggestions to improve the
Lessons Learned Program‘s database, including:
* improving the search capability,
* organizing by subject matter,
* ranking or prioritizing lessons,
* creating links to other documents,
* providing a Web-based link to the database,
* periodically purging redundant data, and:
* making access screens more user-friendly.
Furthermore, because the database does not prioritize lessons, managers
may be unaware of some important areas or issues that need to be
monitored or lessons that need to be reviewed and validated. By
contrast, lessons learned processes used by the selected federal
agencies include periodic reviews of the usefulness of lessons and the
archiving of information that is no longer pertinent or necessary. The
processes also include prioritizing lessons by risk, immediacy, and
urgency. In 1998, the Army Audit Agency recommended that the database
be purged or archived of obsolete items and that current and future
lessons be prioritized. In September 2001, the Chem-Demil Program
created a Lessons Review Team to begin identifying ’critical“ lessons
(those requiring a response). But the team is not prioritizing lessons.
Some Knowledge Management Principles Are Not Applied:
Several other areas also did not adhere to knowledge management
principles and lessons sharing best practices. For example, the Chem-
Demil Program‘s management plan does not explain how the Lessons
Learned Program is to achieve its goals or define performance measures
to assess effectiveness. Knowledge management principles stress the
importance of leaders articulating how knowledge sharing will be used
to support organizational goals. Furthermore, the Chem-Demil Program
does not provide incentives to encourage involvement in the Lessons
Learned Program. Lessons sharing best practices and knowledge
management principles prescribe developing and using performance
measures to determine the effectiveness of a program. In addition, the
Lessons Learned Program currently surveys employees after workshops to
measure their satisfaction; however, these surveys are not sufficient
to assess the overall effectiveness of the program. The program is
attempting to identify ways to measure the cost and benefits derived
from lessons learned. Knowledge management principles also encourage
using performance evaluation, compensation, awards, and recognition as
incentives for participation in lessons learned programs. The lack of
incentives in the Lessons Learned Program may lead to missed
opportunities for the identification and sharing of lessons learned.
Sharing of Lessons Learned Could Be Expanded:
The Lessons Learned Program has shared thousands of lessons among the
five incineration sites through the different phases of construction,
testing, and destruction of chemical agents. However, as the Chem-Demil
Program evolved through the 1990s, and as the components using
alternative technologies were added, the scope of the Lessons Learned
Program did not expand to share lessons with the new components (see
app. IV for a history of the Chem-Demil Program‘s evolution). The
Lessons Learned Program remained primarily focused on the five
incineration sites. At the same time, each stockpile destruction
component developed its own separate lessons learned, but without any
program wide policies or procedures in place to ensure coordination or
sharing of information across components.[Footnote 16] We reported in
May 2000 that effective management of the Chem-Demil Program was being
hindered by a complex organizational structure and ineffective
coordination.[Footnote 17] This has created barriers to sharing.
Today, the four sites that are likely to use alternative technologies
are not full participants in the lessons learned effort:
* The Assembled Chemical Weapons Assessment Program does not fully
participate in the lessons learned process or activities. In at least
one instance, the Assembled Chemical Weapons Assessment Program
requested (from the Program Manager for Chemical Demilitarization), a
package of data including lessons on the pollution abatement system
filters, mustard thaw, and cost estimates. The data were eventually
provided, but they were too late to be used during a DOD cost data
review.[Footnote 18] This lack of access forced the program to submit
incomplete cost data for the review because it was unable to obtain
information from the incineration project in a timely manner.
* The Alternative Technologies and Approaches Project does have access
to the Lessons Learned Program‘s database, and it plans to develop its
own separate database that it will share with the Lessons Learned
Program only at ’key milestones.“ The project‘s information, however,
could be very valuable to other components of the Chem-Demil Program,
especially the Assembled Chemical Weapons Program, which also
researches alternative technologies. This plan could lead to lost
opportunities and duplication of efforts.
Many of the lessons learned by the incineration project could be used
by the other components of the Chem-Demil Program to promote safe,
cost-effective, and on-time operations. Many of the technical processes
(storing, transporting, unloading, and disassembling weapons) and
programmatic processes (regulatory compliance, management, public
relations practices) used by the Chemical Stockpile Disposal Project
are very similar to those used by the other programs. This is also the
case for processes used to develop operating destruction, or
throughput, rates and cost and schedule projections. In fact, the
majority of processes at incineration facilities are the same as those
used by the Assembled Chemical Weapons Assessment Program and the
Alternative Technologies and Approaches Project. Under these
circumstances, promoting a culture of knowledge sharing would enable
all components to capture and use organizational knowledge.
Furthermore, there is the possibility that the Pueblo, Colorado, site
(and possibly the Blue Grass, Kentucky, site) now managed by the
Assembled Chemical Weapons Assessment Program, which now reports to a
DOD office, may be transferred to the Army‘s Chem-Demil Program. If
this transfer of responsibilities does take place, it would be
important for the two programs to be already sharing information fully
and seamlessly. Even if the transfer does not take place, knowledge
management principles and lessons sharing best practices both dictate
that components of the same program should share information,
especially if they all have a common goal.
Conclusions:
The Lessons Learned Program has made important contributions to the
safe destruction of the nation‘s stockpile of chemical weapons. We
found that the program generally adheres to knowledge management
principles and lessons sharing best practices. However, the program‘s
full potential has not been realized. The program needs guidance to
help senior managers make decisions that allow them to weigh the
potential impact of not implementing lessons learned. This guidance
would be a set of procedures, including safety and risk analyses, to be
followed before deciding to counter a lesson learned. Without such
guidance, decision makers, in at least one case, chose lower cost over
safety and schedule, ultimately at the expense of all three. Also, the
Lessons Learned Program lacks procedures to validate the effectiveness
of implemented lessons. The lack of a validation step partially defeats
the purpose of the lessons learned process, which relies on the
confirmed effectiveness of solutions emerging from knowledge and
experience. If the effectiveness of a lesson cannot be validated over
time, problems may emerge again, with a negative impact on safety,
costs, and schedule.
Further, the information in the lessons learned database is not easily
accessible or prioritized. These drawbacks have frustrated users and
may discourage them from using the database. This could lead to wrong
or misinformed decisions that could affect safety. In addition, there
is no overarching coordination or sharing of information across all the
components of the Chem-Demil Program, which grew and evolved over time
without policies or procedures to ensure that knowledge would be
captured and communicated fully. As a result, fragmented or duplicative
efforts continue today, and the Assembled Chemical Weapons Assessment
Program in particular lacks access to important data maintained by the
Chemical Stockpile Disposal Project and the Alternative Technologies
and Approaches Project. In the case of the Chem-Demil Program, the
absence of policies and procedures promoting and facilitating the
broadest dissemination of lessons learned places the safety, cost
effectiveness, and schedule of the chemical weapons destruction at
risk.
Recommendations for Executive Action:
To improve the effectiveness and usefulness of the Chemical
Demilitarization Program‘s Lessons Learned Program, we recommend that
the Secretary of Defense direct the Secretary of the Army to:
* develop guidance to assist managers in their decision making when
making exceptions to lessons learned,
* develop procedures to validate, monitor, and prioritize the lessons
learned to ensure corrective actions fully address deficiencies
identified as the most significant, and:
* improve the organizational structure of the database so that users
may easily find information and develop criteria to prioritize lessons
in the database.
We also recommend that the Secretary of Defense direct the Secretary of
the Army to develop policies and procedures for capturing and sharing
lessons on an ongoing basis with the Alternative Technology and
Approaches Project and in consultation with the Under Secretary of
Defense (Acquisition, Technology, and Logistics) develop policies and
procedures for capturing and sharing lessons on an ongoing basis with
the Assembled Chemical Weapons Assessment Program.
Agency Comments and Our Evaluation:
The Army concurred with our five recommendations and provided
explanatory comments for each one. However, these comments do not
address the full intent of our recommendations. With regard to our
recommendation that it provide guidance to assist managers when
deciding to make an exception to a lesson, the Army stated that the
Lessons Review Team has guidance for characterizing the severity level
of lessons learned. However, as our report clearly points out, this
guidance is for site officials and is insufficient in assisting senior
managers at headquarters on important decisions involving costly
lessons that could potentially impact several sites. We believe that
good management practices require that senior managers make decisions
based on risk, safety, and cost analyses and that guidance should be
developed to support this decision-making process as we recommended.
In concurring with our recommendation to develop procedures to ensure
corrective actions fully address deficiencies, the Army stated that it
is initiating an effort whereby the system‘s contractors will be
responsible for validating, monitoring, and prioritizing lessons. The
Army‘s Lessons Learned Program currently does not validate the results
of corrective actions. Contracting this important function will require
monitoring by the Chem-Demil program to ensure that validation is
properly conducted as we recommended.
The Army stated that it has improved the Lessons Learned database to
make it easier to locate information. Converting the database to an
Internet-based program should also improve its accessibility and
utility. Although these actions address some users‘ concerns, the Army
needs to address all related user issues identified in our report in
order to improve the benefits of the database.
The Army concurred with our recommendation to develop policies and
procedures to capture and share lessons with the two alternative
technology programs. It stated that progress had been made toward
sharing lessons between the Alternative Technologies and Approaches
Project and the Lessons Learned Program at key milestones. The Army
also said it has shared the lessons database with the Assembled
Chemical Weapons Assessment Program. However, the Army should require,
as we recommended, that policies and procedures for capturing and
sharing lessons on an ongoing basis be established, instead of sharing
at key milestones and on a one-way basis. This approach would ensure
that both alternative technology programs fully participate in the
Lessons Learned Program and that the database is constantly enriched to
enhance safety, cost, and schedule based decisions for all components
of the Chem-Demil program.
The Army‘s comments are printed in appendix V. The Army also provided
technical comments, which we incorporated where appropriate.
We are sending copies of this report to interested congressional
committees, the Secretaries of Defense and of the Army; the Assistant
Secretary of the Army (Installations and Environment); the Under
Secretary of Defense (Acquisitions, Logistics, and Technology); the
Director, Federal Emergency Management Agency; and the Director, Office
of Management and Budget. We will also make copies available to others
upon request. In addition, the report will be available at no charge on
the GAO Web site at http://www.gao.gov:
Please contact me at (202) 512-6020 if you or your staff have any
questions regarding this report. Key contributors to this report were
Donald Snyder, Bonita Oden, Pamela Valentine, Steve Boyles, and Stefano
Petrucci.
Raymond J. Decker
Director, Defense Capabilities
and Management:
Signed by Raymond J. Decker:
[End of section]
Appendix I: Information on the Incineration Process and Incidents at
Three Sites:
There have been three releases of agent from operating incineration
facilities and one incident during construction that have generated
several lessons learned. The incineration process and the releases and
construction incidents are described below.
The Army‘s Baseline Incineration Process:
A baseline incineration process uses a reverse-assembly procedure that
drains the chemical agent from the weapons and containers and takes
apart the weapons in the reverse order of assembly. Once disassembled,
the chemical agent and weapon parts are incinerated in separate
furnaces and the gaseous and solid waste is treated in a separate
process. Liquid brine resulting from the treatment of exhaust gases in
the pollution abatement system is dried to reduce the volume and
transported to a commercial hazardous waste management facility.
The path to weapons disposal, in general includes six major steps.
Chemical weapons are stored in earth-covered, concrete-and steel
buildings called igloos. These igloos are guarded and monitored for any
signs of leaking weapons by the U.S. Soldier and Biological Chemical
Command.
Chemical weapons are taken from the igloos and transported to a
disposal plant in sealed on-site containers by the U.S. Soldier and
Biological Chemical Command. The sealed containers are resistant to
fire and impact.
When the on-site containers arrive at the disposal plant, workers check
them for leaking weapons before opening them. Chem-Demil crews then
load the weapons onto conveyors that carry the weapons through the
disposal process. When the weapons are loaded onto the conveyor, the
U.S. Soldier and Biological Chemical Command no longer has
responsibility for them.
From this point on, workers manage the disposal process from an
enclosed control room using advanced robotics, computer technology, and
video monitoring equipment. Automatic, robotic equipment drains the
chemical agent from the weapon and takes the weapons apart in explosive
proof rooms.
Once dismantled and drained, the individual weapon parts travel to
different furnaces in the plant, each designed for a specific purpose.
The liquid incinerator destroys the chemical agent, the deactivation
furnace destroys explosive materials, and the metal parts furnace heats
shell casings and other heavy metal parts to destroy any remaining
agent contamination.
The pollution abatement system cleans the air before it is released
into the environment.
Agent Release at Tooele:
The Tooele Chemical Agent Disposal Facility (Tooele plant) is located
on Deseret Chemical Depot in Tooele, Utah. The facility is designed to
dispose of 44.5 percent of the nation‘s original stockpile of chemical
weapons. Tooele plant is the first chemical weapons disposal facility
built within the continental United States. Construction of the Tooele
plant began in October 1989 and disposal operations began in August
1996. Operations at Tooele plant should be completed in 2008. The
Tooele plant incorporates systems originally tested and used at the
Chemical Agent Munitions Disposal System, also located at the depot.
These systems were first used on an industrial scale at the Army‘s
Johnston Atoll Chemical Agent Disposal System (Johnston Atoll plant) in
the Pacific Ocean. The Johnston Atoll plant was the first integrated
facility built to dispose of chemical weapons.
The sequence of events described in table 3 is based on documents from
the Utah Department of Environmental Quality--Division of Solid and
Hazardous Waste, U.S. Army Safety Center, Department of Health and
Human Services--Centers for Disease Control and Prevention, and a
program contractor. On May 8, 2000, the day shift was processing
rockets in the deactivation furnace system. The deactivation furnace
system lower tipping gate (used to control the feed of munitions to the
furnace) did not close properly and munitions/agent processing was
terminated. Workers in protective gear began to clean and repair the
gate and a strainer. A bag from the strainer, contaminated with GB
(nerve) agent, was left on top of the gate. This is believed to be the
source of the agent that was released. Vapors were drawn from the bag
through the furnace system.
During the initial attempt to re-light the afterburners following the
cleaning procedure, the agent monitoring equipment alarmed. During a
second attempt to re-light these burners another agent monitor alarmed.
In summary, a small amount of agent escaped through the common stack
during attempts to relight the furnace. (See table 2.):
The several corrective actions taken were based on 105 investigation
findings involving operations, training, and equipment. Lessons learned
from this incident include (1) modifying feed chute clean out
procedures, (2) providing operator refresher training, (3) installing a
deactivation furnace remote operated valve to isolate the deactivation
furnace during afterburner re-lights, and (4) redesigning deactivation
furnace feed chute.
Table 2: May 2000 Agent Release at Tooele Chemical Agent Disposal
Facility:
Date/time: May 8, 2000; Event description: Team C, working the day
shift, was processing M56 warheads in the deactivation furnace system
(DFS) and spent decontamination solution (SDS) in the liquid
incinerator (LIC) #1..
Date/time: 4:00 P.M.; Event description: The lower tipping feed gate
(from the explosive containment room) on the deactivation furnace
system was sticking.; Operators began preparation for a two-man entry
(in demilitarization protective ensemble level dress) to clear the jam
in the lower tipping feed gate..
Date/time: 6:00 P.M.; Event description: Team A relieved Team C, and
the problem with the lower tipping valve was briefed to the oncoming
shift.; The DFS chute sprays were on at the time of the operator
change..
Date/time: 8:10 P.M.; Event description: The pressure in the DFS rotary
kiln was lowered. This lower pressure in the kiln increased the airflow
rate through the system. The major problem was that the pressure began
to oscillate significantly.; This reduction lowered the time agent--
produced gases were exposed to heat in the DFS afterburner..
Date/time: 8:20 P.M.; Event description: A DFS Afterburner Exhaust Flow
Sensor alarm occurred indicating low pressure and high air through the
DFS incinerator and the pollution abatement system.; Operator has
trouble controlling pressure..
Date/time: 8:37 -9:30 P.M.; Event description: The entrants prepared to
use water to power-wash the debris that caused the tipping feed gate-
sticking problem. This water hose malfunctioned.; The entrants left the
explosive containment room, repaired the hose, and returned to
completed the clean up.; Once in the explosive containment room, the
entrants attempted to use a droplight to get a better view, the
droplight did not work. The entrants left, retrieved a working
droplight, and returned for a third time to the explosive containment
room to complete the tipping feed gate maintenance.; The entrants had
to leave the explosive containment room again, this time to repair a
clamp on the water hose.; The entrants cleaned out the Agent
Quantification System strainer and placed the strainer sock on the
upper feed gate. The sock contained about one pound of agent-
contaminated fiberglass fragments..
Date/time: 8:42 P.M.; Event description: The DFS Operator noticed
pressure fluctuations that began to affect the DFS induced draft fans.
(These fans pull air through the DFS incinerator and pollution
abatement system.).
Date/time: 8:48 P.M.; Event description: The DFS operator took manual
control of the kiln pressure controller and venturi plug valve.; The
wash down of the chute was completed by 9:30 p.m..
Date/time: 9:45 P.M.; Event description: The DFS operator has a
difficult time stabilizing the DFS..
Date/time: 9:59 P.M.; Event description: The DFS exhaust flow sensor
sends a malfunction signal to the control room, the flow sensor/meter
had been saturated with liquid during the entrants‘ maintenance
operation on the tipping feed gate.; This was followed by an alarm that
automatically shuts down the burner in the DFS kiln and in the DFS
afterburner.; Large draft pressure moved water into the meter.; The
temperatures in both burners dropped below permit levels.; The DFS
operators are unaware of a major agent source presence (the strainer
sock on the upper feed gate left by the entrants)..
Date/time: 10:26 P.M.; Event description: The DFS operators began
attempts to re-light the burners; they felt that re-lighting the
burners would be the safest course of action for preventing a release
of agent.; The DFS operators increased the combustion air in an attempt
to re-light the afterburner; by 10:48 p.m. a decision was made to stop
trying to re-light the burners..
Date/time: 11:18 P.M.; Event description: The operators shut down the
clean liquid pump. This was done to assist in drying out the flow
sensor/meter..
Date/time: 11:26 P.M.; Event description: The first stack agent monitor
alarm occurred.; This was ignored because the duct alarm did not signal
and it should have alarmed first.; At this time the site was masked
(workers were instructed to use protective masks).; The temperature in
the DFS kiln was approximately 204 F lower than what is required to
destroy agent and the temperature in the DFS afterburner was
approximately 1,250 F lower than the requirement..
Date/time: 11:27 P.M.; Event description: A second agent monitor alarm
occurred..
Date/time: 11:30 P.M.; Event description: The control room operator
notified the depot emergency operations center.; The depot commander at
the emergency operations center did not make contact with the Tooele
County emergency responders until nearly 4 hours after the first alarm
at 3:34 a.m. on May 9, 2000..
Date/time: 11:38 P.M.; Event description: The Depot Area Air Monitoring
System tubes for the common stack were removed for testing.; The
analysis confirmed the presence of GB chemical agent..
Date/time: 11:41 P.M.; Event description: A third agent monitor alarmed
occurred..
Date/time: 11:44 P.M.; Event description: The control room operator
directed a ’bottle-up“ of the DFS, in essence closing dampers, slowing
air flow in order to slow the loss of temperature to in the DFS.;
Residence time in the DFS afterburner climbed and the afterburner
temperature began to rise..
Date/time: May 9, 2000; 12:18 A.M.; Event description: Notice to unmask
the site was given..
Date/time: 12:23 A.M.; Event description: The DFS operator attempted a
second re-light of the DFS afterburner. A re-light was initiated..
Date/time: 12:28 A.M.; Event description: During the re-light, the
common stack and DFS agent monitors alarmed again.; The site was
masked..
Date/time: 12:32 A.M.; Event description: The DFS operator was directed
to ’bottle-up“ the furnace again..
Date/time: 1:07 A.M.; Event description: The site was unmasked..
Date/time: 1:17 A.M.; Event description: The depot emergency operations
center received notification that the Depot Area Air Monitoring System
analysis confirmed the presence of agent..
[End of table]
Agent Releases at Johnston Atoll:
In addition to reviewing lessons from Tooele incidents, we were briefed
on two incidents that occurred at Johnston Atoll, and we reviewed
relevant investigation reports for these incidents. Both incidents
resulted in corrective actions and generated several lessons learned.
On March 22, 1994, the liquid agent gun purge process began. The next
day workers dressed in protective gear removed the liquid agent gun,
and three lines had to be disconnected and capped (sealed). These three
lines to the liquid agent gun are the atomizing air, fuel oil, and the
agent line. During the disconnecting of the agent line, the liquid
incineration room agent monitoring system alarmed. Also, the agent
monitors in the common stack began to alarm. Operators turned off the
induction fan to divert room air out through plant exhaust to the
carbon filters.
Lessons learned from this incident include (1) replacing the fuel oil
purge system flow meter with an instrument that could be read in the
control room; an investigation found that the flow meter on the agent
purge line was not functioning (2) directing room air away from the
pollution abatement system to prevent contaminated air from escaping
through the duct work without going through the furnace and (3)
counseling workers on the importance of following approved standard
operating procedures.
On December 8, 1990, a laboratory analysis confirmed emission of
chemical agent from the common stack following a purging (flushing) of
the agent line. It was determined that the probable cause of the
release was that a quantity of agent GB (nerve) leaked from the agent
gun or feed line into the primary chamber of the liquid incineration
furnace, and the agent was swept downstream by the induced draft fan
(used to draw air through the plant) while the furnace was in a cool-
down cycle. It appears that the agent that leaked into the incinerator
and ultimately discharged to the atmosphere was from either valves in
the agent feed line to the primary chamber that were not totally sealed
or the agent remained in the agent line after it was purged and was
aspirated into the incinerator and subsequently the atmosphere. During
the incident, and due to a malfunctioning agent-sampling probe, the
agent-monitoring equipment in the common stack did not detect agent.
Lessons learned from this incident include (1) improving the process to
purge (flush) chemical agent from the feed line by adding a fuel oil
purge and increasing the purge cycle to ensure a complete purge, (2)
modifying the alarm system in the common stack to provide redundancy
and test the alarms more frequently, and (3) closing all four valves
after the agent line is purged and process activities involving the
liquid incineration feed system when the furnace is cooling down to the
charcoal filters.
Construction Incident at Umatilla:
On September 15, 1999, more than 30 construction workers were affected
by an irritating vapor in the air while working in the munitions
demilitarization building. This incident caused many workers to
experience respiratory irritation, sending them to the local hospital
where they were examined and released. Later that day, all construction
work stopped and approximately 800 contracted workers were sent home.
Investigations and analyses lead to the determination that chemical
agent was not involved; instead this was determined to be a
construction incident.
As construction progressed, the building became a ’closed-in“ area and
may not have been adequately ventilated. The building ventilation
system was not designed to control contaminants during construction; it
was only intended to control a release of chemical agent when
construction was complete and operations had begun. The release of 800
contracted-construction workers without informing them of the situation
that no chemical agents were involved, coupled with the slow release of
information to the press, eventually heightened public concern.
Lessons learned from this incident include (1) enhancing local
ventilation in the munitions demilitarization building, (2)
establishing and posting evacuation routes and response procedures
throughout the site, (3) installing a temporary public address system
at the construction site, and (4) ensuring there is adequate
communications between the site and any off-site facilities
particularly in the event of an incident.
Agent Exposure at Tooele:
On July 15, 2002, at the time we were drafting this report, an
individual working at the incineration facility in Tooele, Utah,
experienced a confirmed accidental chemical agent exposure. This
individual was performing maintenance on an agent purge line valve in
the liquid incinerator room and was exposed to residual agent present
in the agent purge line. The worker exhibited symptoms of chemical
agent exposure. Although the Army, DOD Inspector General, and the
facility‘s contractor are conducting investigations looking into the
events associated with the accidental exposure, it is too early to
report on lessons resulting from this incident. The Program Manager for
Chemical Demilitarization is awaiting the investigation reports and
will incorporate the corrective actions into lessons learned. According
to the Army, agent operations will not commence until all corrective
actions have been taken and the plant is deemed safe to operate.
[End of section]
Appendix II: Scope and Methodology:
To assess the Lessons Learned Program, we reviewed literature on the
principles of knowledge management and our previous reports on lessons
sharing best practices.
* To assess the leadership of the Lessons Learned Program, we
interviewed Chem-Demil Program managers, personnel, and the contractor
staff who manage the Lessons Learned Program. We also reviewed
management documents describing the program and we conducted 30
structured interviews[Footnote 19] with the Chem-Demil Program‘s
managers (headquarters and field level) and systems contractor staff at
three sites (Aberdeen, Maryland; Anniston, Alabama; and Tooele, Utah)
to determine how clearly management articulated its expectations about
using lessons learned. We did not select a statistical sample of
database users; therefore, our survey results cannot be generalized to
all Lessons Learned Program database users.
* To describe the lessons learned process, we reviewed documentation
relevant to the lessons learned process. We also interviewed personnel
from the office of the Program Manager for Chemical Demilitarization,
the Anniston, Alabama, site, and the contractor responsible for
managing the Lessons Learned Program.
* To learn how technology supports the Lessons Learned Program, we
reviewed the lessons learned process and identified the methods used to
gather, consolidate, and share information with stakeholders. We also
asked the staff we surveyed how effectively does the program‘s
technology tools support the lessons learned process.
* To determine whether the Chem-Demil Program fosters a culture of
knowledge sharing and use, we talked to program managers for each Chem-
Demil Program components, headquarters staff, and personnel from the
lessons learned contractor staff to determine how lessons are shared
and whether employees are encouraged to participate in the program. We
also asked the staff we surveyed how frequently they submitted
information to the program, whether they used the lessons, and whether
there were incentives to encourage participation.
To determine whether lessons learned contributed to the goals of the
destruction program; we documented and reviewed several important
lessons that program staff identified. We also traced several lessons
from incidents at Johnston Atoll and Tooele to verify that they had
been shared and implemented at the Anniston facility. We used
unverified Army data to assess whether the Lessons Learned Program
achieved its aim of reducing or avoiding unnecessary costs. To
determine if the Lessons Learned Program process conforms to other
programs‘ lessons sharing processes we identified four of a number of
federal organizations that practice knowledge management and operate
lessons learned programs. In making our selections, we reviewed
literature and spoke with knowledge management experts to find
organizations recognized for their ability to share lessons learned or
effectively manage knowledge. We obtained information from the Center
for Army Lessons Learned, the Department of Energy, the U.S. Army Corps
of Engineers, and the Federal Transit Authority. We interviewed
representatives from each organization about the processes they used
for identifying, collecting, disseminating, implementing, and
validating lessons learned information. We reviewed their lessons
learned program guidance to compare and contrast their practices with
the incineration project‘s Lessons Learned Program process. We also
interviewed an expert familiar with the program about the management of
the lessons learned process. To assess the search, linkage, and
prioritization of the database, we obtained documentation and
interviewed the contractor staff about the information in the database.
We tested the search feature of the database, including accessing
menus, keyword and category listings, and analyzed several lessons
learned we had obtained from our searches. We obtained opinions from
the staff we surveyed on the effectiveness of the lessons learned
database and their suggested areas of improvement. The respondents
included managers and others with an average of 9 years experience in
the Chem-Demil Program. The staff we surveyed routinely search the
database for lessons learned information. We did not select a
statistical sample of database users; therefore, our survey results
cannot be generalized to all Lessons Learned Program database users.
To assess the extent to which lessons learned have been shared, we
interviewed the Program Manager for Chemical Demilitarization and the
contractor responsible for operating the Lessons Learned Program. We
also attended status briefings for each Chem-Demil component. We
focused our work primarily on the stockpile destruction projects/
programs. We conducted interviews with officials from the Alternative
Technologies and Approaches Project, the Assembled Chemical Weapons
Assessment Program, and the Chemical Stockpile Disposal Project to
gather evidence on the commonality the alternative technology
components have with the incineration program and the extent to which
they share lessons learned information. To determine whether each
component participated in the Lessons Learned Program by either sharing
or receiving lessons learned information, we reviewed workshop minutes
from calendar years 2000 and 2001.
To describe the incidents at three sites, we attended briefings on the
incidents provided by officials from the incineration program, and
reviewed incident investigation reports and entries in the Lessons
Learned database. We identified key lessons from these sources and
toured the Anniston Chemical Disposal Facility, to determine whether
lessons learned had been shared and implemented. During our visit, we
observed that several lessons from the Tooele incident, among others,
were implemented.
[End of section]
Appendix III: Lessons Learned Process:
The Lessons Learned Program was established to collect and share
lessons learned within the incineration program. The Programmatic
Lessons Learned Program uses various methods to identify, review,
document, and disseminate lessons learned information among government
and contractor personnel. The program uses facilitated workshops to
introduce lessons and also takes lessons from engineering change
proposals. The Lessons Review Team reviews issues and determines
specific lessons to be implemented. These issues, engineering changes,
and lessons are stored in a database.
The program uses five distinct steps to develop lessons learned, as
shown in figure 2.
* Issues are raised through topics submitted to workshops (meetings of
headquarters and site personnel), critical document reviews (of changes
to program documents), engineering change proposals (technical changes
at one or more sites), quick reacts (immediate action), and express
submittals (information from a site.):
* Experts review issues to determine if a change should be initiated in
a workshop, an assessment (a study to support a management
recommendation for change), engineering change proposal review process
(a team at each site reviews changes at other sites), and directed
actions (requests for information on actions a site has taken.):
* Lessons are identified from workshops, assessment reports, and the
lessons review team (headquarters activity to segregate lessons into
response required or not required.):
* Issues and lessons are stored in the database.
* Lessons are then shared with stakeholders, including contractor
personnel, through access to the database, technical bulletins (a
quarterly publication with information of general interest to multiple
sites), programmatic planning documents (containing policies,
guidelines, management approaches, and minimum requirements), and site
document comparisons (new documents with baseline documents.):
Four primary elements of these steps are discussed below.
Figure 2: Lessons Learned Stakeholders and Process Steps:
[See PDF for image]
Note: PLL (Programmatic Lessons Learned) is referred to in this report
as Lessons Learned Program; in the figure engineering change proposal
is referred to as ECP.
Source: Project Manager for Chemical Stockpile Disposal, Programmatic
Lessons Learned Program Plan, Revision 3, April 2002.
[End of figure]
Lessons Learned Program Facilitated Workshops:
Facilitated workshops are the primary method for introducing lessons
learned into the Lessons Learned Program. Facilitated workshops are
meetings that offer an environment conducive for site and headquarters
personnel to speak openly about experiences. The intent of the
workshops is to allow program personnel familiar with particular
subjects to hold detailed discussions of issues relative to specific
subjects. All issues discussed in the workshops are entered into the
database and later reviewed to determine if the issues should become
lessons learned.
The facilitated workshop process begins with a memorandum that requests
site personnel from the Lessons Learned Program team to identify topics
they want to discuss in workshops. These topics are generally divided
into three basic categories: (1) valuable information provided to other
sites, (2) challenging issues and discussion of issues with other sites
in anticipation of possible recommendations, and (3) general topics to
discuss different approaches to a problem. After each workshop, a
feedback survey is sent to participants to determine user satisfaction
with workshops.
Engineering Change Proposal Review Process:
Engineering change proposals are the primary method of approving and
documenting design changes at the sites. Members of the Configuration
Control Board and the Field Configuration Control Boards are
responsible for reviewing and approving engineering change proposals
within certain dollar limits. The Configuration Control Board,
consisting of members from headquarters, is also responsible for
managing changes to items or products identified for configuration
control, such as facilities and equipment in order to maintain or
enhance reliability, safety, standardization, performance, or
operability. Each Field Configuration Control Board consists of members
from a site, and is responsible for controlling engineering changes
during construction, systemization, operations, and closure of
facilities. Engineering change proposals are discussed during bi-weekly
teleconferences where the sites can ask the originating site questions
about the proposed engineering change.
The Field Configuration Board is responsible for approving engineering
change proposals with an estimated cost of $200,000 or less. The
Configuration Control Board is responsible for approving proposals with
an estimated cost of $200,001 to $750,000. Proposals over $750,000 are
sent to the Project Manager for Chemical Stockpile Disposal for
approval. After approval, the engineering change proposals are reviewed
and input into the database and sent to the Lesson Review Team as part
of the review process.
Engineering changes are the primary source of design-related lessons
learned. Engineering change proposals are approved changes in the
design or performance of an item, a system or a facility. Such changes
require change or revision to specifications, engineering drawing, and/
or supporting documents. Consequently, the Program Manager for Chemical
Demilitarization developed a review process as a method to capture
these lessons in the Lessons Learned Program. The purpose of the
Engineering Change Proposal Review Process is to provide Chemical
Demilitarization sites with more control over lessons learned decisions
and incorporate lessons learned sharing under the Lessons Learned
Program. Additionally, the review process is structured to allow each
site the opportunity to review engineering changes being implemented at
other sites and consider the applicability to their site. The review
team consists of members from the sites, the Program Manager‘s office,
the Lessons Learned Program team, and the U.S. Army Corps of Engineers.
Lessons Review Team:
The Lessons Review Team, established in September 2001, is responsible
for reviewing issues discussed in facilitated workshops to determine
their potential impact and to determine if a specific site action is
required. Additionally, the review team reviews engineering change
proposals to determine if they are design-related lessons learned.
Issues are considered ’lessons learned“ when they have programmatic
interest and significant impact on safety, environmental protection, or
plant operations. The Lessons Review Team designates lessons learned as
mandatory, ’response required,“ and ’response not required.“ A lesson
is mandatory if the method of implementation has been or is directed
from the Program Manager for Chemical Demilitarization headquarters. A
lesson that is characterized as ’response required“ for means that the
given site must provide information to the home office on the action
taken to address the lesson. ’Response not required“ means that the
site is not required to provide information to the headquarters on the
action that the site has taken.
For mandatory lessons, the Lesson Review Team decision makers provide
specific guidance for implementation of lessons. Technical support
staff on the team conducts lesson reviews and provides recommendations
to the decision maker regarding lessons. A team member is responsible
for initial review of lessons and recommended designation, distribution
of materials before the meetings, and facilitation of the meetings.
Lessons Learned Program Database:
The Lessons Learned Program database is a repository for (1) issues
generated from facilitated workshops, (2) engineering change proposals,
(3) critical document reviews, (4) quick react/advisory system and
other lessons learned process data, and (5) programmatic and design
lessons learned. As of April 2002, the database contained 3,400 issues,
7,630 directed action, and 3,055 engineering change proposals.
The database was developed as a stand-alone program allowing users to
employ search utilities or category trees to retrieve lessons. The
program opens to the main screen, which consists of a search,
categories, and lessons screens. The lessons screen is a search
mechanism that utilizes a ’drop down menu“ enabling users to locate
lessons by selecting categories or subcategories to narrow the search
for lessons in a specific area. To summarize information and identify
lessons in the lessons learned database, the database contains
background information to support each lesson. The background
information provides a condensed history, as well as the status of each
lesson at the Chemical Demilitarization site.
[End of section]
Appendix IV: Chemical Demilitarization Program Management
Developments,
1997-2001:
The Departments of Defense and the Army made several changes to the
management structure of the Chem-Demil Program, principally in response
to congressional legislation. Originally the Program Manager for
Chemical Demilitarization reported directly to the Assistant Secretary
of the Army (Installations and Environment), who also oversees storage
of the chemical weapons stockpile. The U.S. Soldier and Biological
Chemical Command manages the stockpile. The Command also manages the
loading, delivery, and unloading of chemical weapons at the destruction
facility. After the estimated cost of the program reached a certain
dollar amount, as required by statute,[Footnote 20] the Army formally
designated it a major defense acquisition program. To manage this
program in the Army acquisition chain, it was then transferred to the
Assistant Secretary of the Army (Acquisition, Technology, and
Logistics). The Program Manager for Chemical Demilitarization continued
executing the program. In 1997, the Chemical Stockpile Emergency
Preparedness Program was removed from the Program Manager for Chemical
Demilitarization and transferred back to the Assistant Secretary of the
Army (Installations and Environment) where it is currently managed by
the U.S. Soldier and Biological Chemical Command. Also in 1997, the
Army and the Federal Emergency Management Agency signed a new
memorandum of agreement to better manage the on-and off-post emergency
response activities, respectively.
In the 1997 Defense Appropriations Act (sec. 8065),[Footnote 21]
Congress required the Assembled Chemical Weapons Assessment Program be
independent of the Program Manager for Chemical Demilitarization and
report directly to the Under Secretary of Defense (Acquisition and
Technology).[Footnote 22] The purpose of this legislation was to
separate the pilot program from the baseline incineration activities.
Achievement of this goal also meant that two program offices would
share responsibilities associated with disposal activities in Kentucky
and Colorado. However, the pilot program‘s legislation does not
specifically state whether or not the Program Manager for Chemical
Demilitarization will manage the assessment program once the
development of technology evaluation criteria, the technology
assessment, the demonstration, and pilot phases end.
In May 2000, we reported on the fragmented management structure and the
inadequate coordination and communication within the Chem-Demil
Program. [Footnote 23] We recommended that the Army should clarify the
management roles and responsibility of program participants and
establish procedures to improve coordination among the program‘s
various components.[Footnote 24] The Army, in December 2001,
transferred the Chemical Demilitarization Program to the Assistant
Secretary of the Army (Installations and Environment), bringing all
components of the program, except the Assembled Chemical Weapons
Program, under a single Army manager, as shown in figure 3. Another
significant management change occurred in April 2002 when the Program
Manager for Chemical Demilitarization retired after holding this
position for the past 5 years.
Figure 3: Chemical Demilitarization Program Organization Chart:
[See PDF for image]
Source: Offices of the Chemical Demilitarization and Assembled Chemical
Weapons Assessment Programs.
[End of figure]
[End of section]
Appendix V: Comments from the Department of the Army:
DEPARTMENT OF THE ARMY:
21 AUG 1002:
Mr. Raymond J. Decker Director:
Defense Capabilities and Management United States General Accounting
Office Washington, D.C. 20548:
Dear Mr. Decker:
On July 23, 2002, the Office of the Inspector General of the Department
of Defense forwarded a copy of the Draft GAO Report ’CHEMICAL WEAPONS:
Lessons Learned Program Generally Effective but Could Be Improved and
Expanded (GAO-02-890)“, for review and comment on the report and the
recommendations it contained. The Department of Defense has reviewed
this report and concurs with the GAO findings and recommendations.
The enclosure details specific comments on the Draft report.
Sincerely,
Mario P. Fiori:
Signed by Mario P. Fiori:
Enclosure:
GAO DRAFT REPORT - DATED 19 JULY 2002 GAO-02-890:
’CHEMICAL WEAPONS: Lessons Learned Program Generally Effective but
Could Be Improved and Expanded“:
DEPARTMENT OF THE ARMY COMMENTS TO THE RECOMMENDATIONS:
RECOMMENDATION 1: The GAO recommended that the Secretary of Defense
direct the Secretary of the Army to:
*develop guidance to assist managers in their decision making when
making exceptions to lessons learned;
*develop procedures to validate, monitor, and prioritize the lessons
learned to ensure corrective actions fully address deficiencies
identified as the most significant; and:
*improve the organizational structure of the database so that users may
easily find information and develop criteria to prioritize lessons in
the database.
(p. 15/GAO Draft Report):
ARMY RESPONSE:
a. Develop Guidance to Assist Managers. Concur:
Guidance was developed in September-October 2001. The Project Manager
for Chemical Stockpile Disposal (PMCSD) has chartered a Lessons Review
Team to assist managers in their decision-making. This assistance is
provided by first prioritizing the lessons (see response below for the
severity level definitions), then by requiring a response from the site
to PMCSD. A response from the site is a commitment to implement the
lesson learned or to provide documentation of why the lesson learned
should not or will not be implemented. Severity Level II lessons
require engineering/managerial judgment to determine whether a response
from the site will be required. Severity level III lessons do not
require a site response and are assigned to the site to determine the
disposition of the lesson. Exceptions made to lessons learned that are
of programmatic interest are made after extensive review by the Lessons
Review Team and the site involved.
b. Develop Procedures to Validate, Monitor and Prioritize the Lessons.
Concur:
Under the Lessons Review Team charter, guidelines and criteria have
been developed for those lessons that are of programmatic interest to
prioritize each lesson. Severity levels are assigned to each lesson.
Severity Level I - Could impact containment of chemical agent or
explosion; potentially expose workers to chemical agent; cause
significant harm to workers due to industrial activities; result in
release of toxic/hazardous material affecting public and/or worker
health, safety or the environment; or result in extensive damage to
equipment or long term stoppage of the process. Severity Level I
lessons will require a response from the site to PMCSD.
Severity Level II - Could adversely affect reliability, operability or
productivity or cause limited damage to equipment, facilities or
temporary process shutdown. Severity Level II requires engineering/
managerial judgment to determine whether a response from the site will
be required.
Severity Level III - Could result in minimal damage with minimal
monetary cost to repair or replace. Severity level III lessons do not
require a site response and are assigned to the site to determine the
disposition of the lesson.
Each site develops its plan of action including a schedule to
accomplish the plan; this is reported to the PMCSD. Each site validates
the workability of the solution. At this time sites do not report back
through the Programmatic Lessons Learned Program; however, the sites do
monitor all Engineering Change Proposals and lessons learned. Under a
new effort currently being developed the systems contractors will
assume more responsibility for the prioritization, monitoring, and
validation of lessons learned that are of programmatic interest. The
System Contractors will report this information to applicable PMCD
managers through a more efficient lessons learned information
management system.
c. Improve the Organization Structure of the Database. Concur:
The organizational structure of the Programmatic Lessons Learned
database has been improved, making it easier to locate information. The
Programmatic Lessons Learned database is currently formatted into a
Lessons Database and an Issues Database. The Lessons Database was
developed in FY01-02 for the specific purpose of making it easier to
locate information. Since the Lessons Database is relatively new
(development was finished in February 2002) current users may not be
familiar with it yet. All lessons are categorized at the highest level
by life cycle phases of a chemical demilitarization plant (i.e. Design,
Construction, Systemization, Operations, and Closure). Each phase is
then broken down into subcategories creating a category tree by which
users can quickly navigate to the area of interest.
A further enhancement is underway to convert the current Programmatic
Lessons Learned database to an Intemet-based program. This project is
scheduled for completion during the 2nd quarter, FY03.
RECOMMENDATION 2: The GAO recommended that the Secretary of Defense
direct the Secretary of the Army to develop policies and procedures for
capturing and sharing lessons on an ongoing basis with the Alternative
Technology and Approaches Project and in consultation with the Under
Secretary of Defense (Acquisition, Technology and Logistics) develop
policies and procedures for capturing and sharing lessons on an ongoing
basis with the Assembled Chemical Weapons Assessment Program.
ARMY RESPONSE: Concur.
PMCSD and the Project Manager for Alternative Technologies and
Approaches (PMATA) have made some progress toward sharing lessons
learned. PMATA has participated in the PMCSD Engineering Change
Proposal Review Process meetings and reviewed the Programmatic Lessons
Learned database for applicable lessons learned. The two sites under
the PMATA‘s responsibility have benefited from the lessons learned
database by avoiding previously identified programmatic shortcomings.
As key milestones are met, PMATA‘s increased participation in the
Lessons Learned Program will be encouraged.
The Lessons Database was completed in August 2001. Since then, the
information in the Lessons Database, originally compiled under the
Programmatic Lessons Learned Program, has been shared with the Program
Manager for the Assembled Chemical Weapons Assessment (PMACWA).
Continued and increased use of the Database will be encouraged.
[End of section]
FOOTNOTES
[1] Chemical Weapons: FEMA and Army Must Be Proactive in Preparing
States for Emergencies GAO-01-850 (Washington, D.C., Aug. 13, 2001).
[2] DOD, in its Knowledge Management Primer, provides managers and
practitioners with a framework for sharing knowledge. The Army uses
knowledge management principles in its Roadmap for Army Knowledge
Management.
[3] In our first report, we recommended that the Chemical Stockpile
Emergency Preparedness Program be more proactive, i.e., it should share
its lessons learned--especially those concerning emergency readiness
and response--with other stakeholders. This would include the Chemical
Stockpile Disposal Project.
[4] Although Pine Bluff, Arkansas, is an incineration site, the Army is
considering destroying a portion of the agent stored at Pine Bluff by
using an alternative method under the Alternative Technologies and
Approaches Project. No decision on whether an alternative technology
will be used at the Pine Bluff site has been reached.
[5] The Senate ratified the U.N.-sponsored Convention on the
Prohibition of the Development, Production, Stockpiling and the Use of
Chemical Weapons and on Their Destruction (known as the Chemical
Weapons Convention) in April 1997.
[6] In accordance with provisions of the treaty, the Army states that
an extension of the April 2007 deadline will be requested if and when
necessary.
[7] A lessons learned process is considered an integral part of most
knowledge management systems.
[8] U.S. Department of Energy Standard: Corporate Lessons Learned
Program Guidance (DOE-STD-7501-99).
[9] After the agent is destroyed in the incinerator, the pollution
abatement system cleans the air (gases produced during incineration)
before it is released into the environment.
[10] Initially the Johnston Atoll site reported experiencing
installation problems with the lower cost material. Later, however,
both the Tooele and Johnston Atoll sites informed the Lessons Learned
Program that a more expensive material (hastelloy) was the appropriate
solution to address the piping failures.
[11] Program Manager for Chemical Demilitarization headquarters
management made the decision to continue using the lower cost material
in part of the pollution abatement piping systems at the three future
sites; the recommended lesson emerged from a repeated problem.
Implementing the recommended lesson would have cost the Chem-Demil
Program more than $750,000 and involve multiple sites.
[12] Military Training: Potential to Use Lessons Learned to Avoid Past
Mistakes Is Largely Untapped , D.C., Aug. 9, 1995).
[13] U.S. Department of Energy Standard: Corporate Lessons Learned
Program Guidance (DOE-STD-7501-99). According to the standard, the
development process includes identification, documentation,
validation, and dissemination. The utilization and incorporation
process includes identification of applicable lessons, distribution to
appropriate personnel, identification of actions that will be taken as
a result of the lessons, and follow-up to ensure that appropriate
actions were taken.
[14] See footnote 8.
[15] In January 2002, we reported on problems related to the knowledge
management database tool used by the National Aeronautics and Space
Administration‘s lessons learned program, see NASA: Better Mechanisms
Needed for Sharing Lessons Learned, GAO-02-195 (Washington, D.C., Jan.
30, 2002).
[16] The Non-Stockpile Chemical Materiel Product maintains a separate
lessons learned database that is linked to the Lessons Learned
Program‘s database. The Chemical Stockpile Emergency Preparedness
Program maintains its own best practices on an Internet site, shares
lessons at national meetings, and does coordinate with the Lessons
Learned Program, especially for outreach and public relations efforts.
[17] Chemical Weapons Disposal: Improvements Needed in Program
Accountability and Financial Management (GAO/NSIAD-00-80, May 8, 2000).
[18] The Assembled Chemical Weapons Assessment Program submitted a
formal request for lessons and cost data through the Deputy Assistant
to the Secretary of Defense (Chemical/Biological Defense) and the
Deputy Assistant Secretary of the Army (Chemical Demilitarization).
[19] In this report, we refer to this population as ’surveyed staff“ to
distinguish from the general interviews.
[20] 10 U.S.C. 2430.
[21] Omnibus Consolidation Appropriations Act, 1997 (P.L. 104-208).
[22] The Under Secretary of Defense (Acquisition and Technology) is now
titled the Under Secretary of Defense (Acquisition, Technology, and
Logistics).
[23] The Cooperative Threat Reduction program, which assists Russia in
destroying over 40,000 tons of chemical agent stored there, is part of
the Chemical Demilitarization Program‘s mission but is funded
separately.
[24] In August 2001, we reported that the Chemical Stockpile Emergency
Preparedness Program did not share its lessons effectively. During this
review, the program‘s management demonstrated steps that had been taken
to address this issue.
GAO‘s Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO‘s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO‘s Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ’Today‘s Reports,“ on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select ’Subscribe to daily E-mail alert for newly
released products“ under the GAO Reports heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548:
