DOD Business Systems Modernization
Limited Progress in Development of Business Enterprise Architecture and Oversight of Information Technology Investments Gao ID: GAO-04-731R May 17, 2004The Department of Defense's (DOD) long-standing business systems problems adversely affect the economy, effectiveness, and efficiency of its business operations and have resulted in a lack of adequate transparency and appropriate accountability across all of its major business areas. In July 2001, DOD initiated a program to, among other things, develop a DOD business enterprise architecture (architecture). This effort is an essential part of the Secretary of Defense's broad initiative to "transform the way the department works and what it works on." Because DOD is one of the largest and most complex organizations in the world, overhauling its business operations and supporting systems represents a huge management challenge. In fiscal year 2003, DOD reported that its operations involved over $1 trillion in assets, nearly $1.6 trillion in liabilities, approximately 3.3 million military and civilian personnel, and disbursements of over $416 billion. To support its business operations, DOD reported that it relies on about 2,300 business systems, including accounting, acquisition, logistics, and personnel systems. The department requested about $19 billion--about $4.8 billion for business systems modernization and about $14 billion for operation and maintenance of these systems--in fiscal year 2004. Recognizing the importance of DOD's efforts to transform its business operations and systems through the use of an enterprise architecture, the Congress included provisions in the National Defense Authorization Act for Fiscal Year 2003 that were aimed at developing and effectively implementing a well-defined architecture. Specifically, section 1004 of this act required that DOD (1) develop, by May 1, 2003, a financial management enterprise architecture and a transition plan for implementing the architecture that meets certain requirements and (2) review financial system improvements with proposed obligations of funds in amounts exceeding $1 million to determine if those system improvements meet specific conditions that are called for in the act. The act also directed us to assess actions that DOD has taken to comply with these requirements. In July and September 2003, we reported on DOD's actions and made a number of recommendations to assist DOD in its efforts to effectively develop and implement an architecture and to guide and constrain its business systems investments. The act further requires that the Secretary of Defense submit an annual report not later than March 15 of each year from 2004 through 2007 to congressional defense committees on its progress in implementing the architecture, including the transition plan. Additionally, the act directs us to submit to congressional defense committees, within 60 days of DOD's report submission, an assessment of DOD's actions taken to comply with these requirements. DOD submitted its first annual report on March 15, 2004. This report is our assessment of DOD's March 15, 2004 report. We determined (1) the actions DOD has taken to address our previous recommendations regarding the development and implementation of the architecture and (2) the actions DOD is taking to ensure its ongoing and planned investments will be consistent with its evolving architecture.
Since our last review--and after 3 years of effort and over $203 million in obligations--we have not seen any significant change in the content of DOD's architecture or in DOD's approach to investing billions of dollars annually in existing and new systems. Few actions have been taken to address the recommendations we made in our September 2003 report, which were aimed at improving DOD's plans for developing the next version of the architecture and implementing the institutional means for selecting and controlling both planned and ongoing business systems investments. To DOD's credit, it has established, for example, a group under the Business Management Modernization Program (program) steering committee to facilitate communication and coordination across the domains for modernization program activities, including extending and evolving the architecture. However, DOD has not yet adopted key architecture management best practices and has not added the scope and detail to its architecture that we previously identified as missing. Additionally, the department does not have reasonable assurance that it is in compliance with the National Defense Authorization Act for Fiscal Year 2003, which requires DOD's Comptroller to review all system improvements with obligations exceeding $1 million. While DOD has recently issued a policy that assigns investment management responsibilities to the domains, the policy has not yet been implemented, and DOD has not clearly defined the roles and responsibilities of the domains, established common investment criteria, and conducted a comprehensive review of its existing business systems to ensure that they are consistent with the architecture. Further, each of the DOD components continues to receive its own funding and make its own parochial investment decisions. The department acknowledges that it still has much more to do, including developing the architecture to a necessary level of detail, defining specific performance metrics, and clarifying the roles and responsibilities associated with managing the domains' portfolios of business systems and ensuring that these systems comply with the architecture. The limited progress that DOD has made is due, in part, to the lack of clearly assigned, accountable, and sustained program leadership and to changes in the program direction and priorities. Our experience in reviewing other challenged architecture efforts shows that these efforts have suffered from limited senior management understanding of and commitment to an architecture and from cultural resistance to having and using one.
GAO-04-731R, DOD Business Systems Modernization: Limited Progress in Development of Business Enterprise Architecture and Oversight of Information Technology Investments
This is the accessible text file for GAO report number GAO-04-731R
entitled 'DOD Business Systems Modernization: Limited Progress in
Development of Business Enterprise Architecture and Oversight of
Information Technology Investments' which was released on May 17, 2004.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
May 17, 2004:
Congressional Defense Committees:
Subject: DOD Business Systems Modernization: Limited Progress in
Development of Business Enterprise Architecture and Oversight of
Information Technology Investments:
The Department of Defense's (DOD) long-standing business systems
problems adversely affect the economy, effectiveness, and efficiency of
its business operations and have resulted in a lack of adequate
transparency and appropriate accountability across all of its major
business areas. To help the department transform its operations, we
recommended[Footnote 1] in 2001 that DOD develop an enterprise
architecture to guide and constrain its almost $20 billion annual
investment in business systems and that it establish the investment
controls needed to implement this architecture. In July 2001, DOD
initiated a program[Footnote 2] to, among other things, develop a DOD
business enterprise architecture (architecture). This effort is an
essential part of the Secretary of Defense's broad initiative to
"transform the way the department works and what it works on.":
Because DOD is one of the largest and most complex organizations in the
world, overhauling its business operations and supporting systems
represents a huge management challenge. In fiscal year 2003, DOD
reported that its operations involved over $1 trillion in assets,
nearly $1.6 trillion in liabilities, approximately 3.3 million military
and civilian personnel, and disbursements of over $416 billion. To
support its business operations, DOD reported that it relies on about
2,300 business systems, including accounting, acquisition, logistics,
and personnel systems. The department requested about $19 billionóabout
$4.8 billion for business systems modernization and about $14 billion
for operation and maintenance of these systemsóin fiscal year 2004.
ecognizing the importance of DOD's efforts to transform its business
operations and systems through the use of an enterprise architecture,
the Congress included provisions in the National Defense Authorization
Act for Fiscal Year 2003[Footnote 3] that were aimed at developing and
effectively implementing a well-defined architecture. Specifically,
section 1004 of this act required that DOD (1) develop, by May 1, 2003,
a financial management enterprise architecture[Footnote 4] and a
transition plan for implementing the architecture that meets certain
requirements and (2) review financial system improvements with proposed
obligations of funds in amounts exceeding $1 million to determine if
those system improvements meet specific conditions that are called for
in the act. The act also directed us to assess actions that DOD has
taken to comply with these requirements. In July[Footnote 5] and
September 2003,[Footnote 6] we reported on DOD's actions and made a
number of recommendations to assist DOD in its efforts to effectively
develop and implement an architecture and to guide and constrain its
business systems investments.
The act further requires that the Secretary of Defense submit an annual
report not later than March 15 of each year from 2004 through 2007 to
congressional defense committees on its progress in implementing the
architecture, including the transition plan. Additionally, the act
directs us to submit to congressional defense committees, within 60
days of DOD's report submission, an assessment of DOD's actions taken
to comply with these requirements. (See enc. I for a copy of section
1004 of the act.) DOD submitted its first annual report on March 15,
2004. This report is our assessment of DOD's March 15, 2004 report. As
agreed with your offices, we determined (1) the actions DOD has taken
to address our previous recommendations regarding the development and
implementation of the architecture and (2) the actions DOD is taking to
ensure its ongoing and planned investments will be consistent with its
evolving architecture.
We performed our work from December 2003 through April 2004 in
accordance with U.S. generally accepted government auditing standards.
Details on our scope and methodology are in enclosure II.
Results in Brief:
Since our last review--and after 3 years of effort and over $203
million in obligations--we have not seen any significant change in the
content of DOD's architecture or in DOD's approach to investing
billions of dollars annually in existing and new systems. Few actions
have been taken to address the recommendations we made in our September
2003[Footnote 7] report, which were aimed at improving DOD's plans for
developing the next version of the architecture and implementing the
institutional means for selecting and controlling both planned and
ongoing business systems investments. To DOD's credit, it has
established, for example, a group under the Business Management
Modernization Program (program) steering committe[Footnote 8]eE to
facilitate communication and coordination across the domai[Footnote
9]ns for modernization program activities, including extending and
evolving the architecture. However, DOD has not yet adopted key
architecture management best practi[Footnote 10]ces and has not added
the scope and detail to its architecture that we previously identified
as missing. Further, DOD has not yet implemented an effective
management structure and processes to provide adequate control and
accountability over its $5 billion annual investment in business
systems modernization. Additionally, the department does not have
reasonable assurance that it is in compliance with the National Defense
Authorization Act for Fiscal Year 2003, which requires DOD's
Comptroller to review all system improvements with obligations
exceeding $1 million.
Regarding architecture management best practices, DOD has not yet
implemented key elements, such as assigning accountability and
responsibility for directing, overseeing, and approving the
architecture. In addition, it has not explicitly defined performance
metrics to evaluate the architecture's quality, content, and utility.
With respect to the architectural content, DOD's latest version of the
architecture does not include many of the key elements of a well-
defined "As Is," "To Be," and transition plan that we previously
reported as not being satisfied.[Footnote 11] For example, the "To Be"
environment does not provide sufficient descriptive content related to
future business operations and supporting technology to permit
effective acquisition and implementation of system solutions and
associated operational change. Similarly, DOD's verification and
validation contractor has concluded that this latest version of the
architecture retains most of the limitations that the initial version
had. Moreover, DOD has not yet defined specific plans, including
milestones, detailing how it intends to extend and evolve the
architecture to incorporate this missing content. For example, the
program's first objective for increment one--to enable asset
accountability, total force visibility, and an unqualified audit
opinion on DOD's fiscal year 2007 consolidated financial statements--is
not supported by a DOD-wide plan of action, and the individual
component plans for accomplishing this objective may not be linked to
the program's activities, including the architecture. According to DOD,
it will not have the plans on how the capabilities for increment one
will be achieved until August 2004.
DOD has also made limited progress in addressing our recommendations
aimed at establishing and implementing effective investment management
processes and, therefore, continues to lack effective management
oversight and control over ongoing business systems modernization
investments. While DOD has recently issued a policy that assigns
investment management responsibilities to the domains, the policy has
not yet been implemented, and DOD has not clearly defined the roles and
responsibilities of the domains, established common investment
criteria, and conducted a comprehensive review of its existing business
systems to ensure that they are consistent with the architecture.
Further, each of the DOD components continues to receive its own
funding and make its own parochial investment decisions. Moreover, DOD
has not yet established and implemented an effective process for
ensuring that system improvements with obligations exceeding $1 million
are submitted to DOD's Comptroller for review and to determine whether
they are consistent with the architecture, as required by the National
Defense Authorization Act for Fiscal Year 2003. Based upon a comparison
of limited information provided by the military services and defense
agencies, we identified a total of $863 million in obligations for
improvements that exceeded $1 million each but had not been submitted
to DOD's Comptroller for review and determination.
The department acknowledges that it still has much more to do,
including developing the architecture to a necessary level of detail,
defining specific performance metrics, and clarifying the roles and
responsibilities associated with managing the domains' portfolios of
business systems and ensuring that these systems comply with the
architecture. The limited progress that DOD has made is due, in part,
to the lack of clearly assigned, accountable, and sustained program
leadership and to changes in the program direction and priorities. For
example, from May 2003 to February 2004, there was no program manager
to identify, direct, and execute program activities. Our experience in
reviewing other challenged architecture efforts shows that these
efforts have suffered from limited senior management understanding of
and commitment to an architecture and from cultural resistance to
having and using one. Cultural resistance to change, organization
component parochialism, and stovepiped operations are all evident at
DOD.
Because many of our prior recommendations--for managing architecture
development, maintenance, and implementation and for controlling
ongoing and planned investments in business systems--remain open, we
are not making any new recommendations in this report, but we are
reiterating the 22 open recommendations that we made in our May
2001,[Footnote 12] February 2003,[Footnote 13] and September
2003[Footnote 14] reports. (Enc. III contains details on the status of
all of our prior recommendations, including our assessment of DOD's
actions.) It is imperative that DOD act swiftly to implement these
recommendations. If it does not, the prognosis for this program is
bleak, which in turn puts the department's business transformation
efforts in jeopardy.
In written comments, which are reprinted in enclosure IV, DOD agreed
with our assessment that the department's long-standing business
systems problems have resulted in a lack of adequate transparency and
appropriate accountability across all major business areas and that
development and use of an architecture is necessary to transforming its
business operations and supporting systems. In response to our
characterization that progress has been limited, DOD stated that, over
the past 3 years, progress has been slower than either GAO or DOD would
prefer but that it has been significant, despite appearances to the
contrary. In support of its view, DOD provided an extensive package of
information. We considered this information, most of which we had
previously evaluated. The willingness of DOD's leadership to tackle the
department's decades-old problems with its business operations and
supporting systems represents a major step forward. However, we remain
convinced that it is fair to characterize DOD's progress in developing
a well-defined architecture and implementing effective management
oversight and control over its business systems as limited. As a
result, we did not make any changes to this report.
Background:
Prior Reviews of DOD's Architecture Efforts Have Identified Challenges
and Weaknesses:
Over the last 3 years, we have reported[Footnote 15] that one of the
key elements to successfully meeting DOD's financial and related
business management challenges is establishing and implementing an
enterprise architecture, or modernization blueprint. An enterprise
architecture provides a clear and comprehensive picture of an entity,
whether it is an organization (e.g., federal department or agency) or a
functional or mission area that cuts across more than one organization
(e.g., financial management). This picture consists of snapshots of
both the enterprise's current or "As Is" operational and technological
environment and its target or "To Be" environment, as well as a capital
investment road map for transitioning from the current to the target
environment. These snapshots further consist of "views," which are
basically one or more architecture products that provide conceptual or
logical representations of the enterprise. We have made numerous
recommendations to assist DOD in successfully developing the
architecture and using it to gain control over its ongoing business
systems investments. Enclosure III contains details on the status of
all of our prior recommendations, including our assessment of DOD's
actions.
In May 2001[Footnote 16], we reported that the department did not have
an architecture for its financial and financial-related business
operations, nor the management structures, processes, and controls in
place to effectively develop and implement one. We reported that if the
department continued to spend billions of dollars on new and modified
systems, independently from one another and outside the context of an
architecture, this would result in more processes and systems that are
duplicative, not interoperable, and unnecessarily costly to maintain
and interface. We made eight recommendations aimed at providing the
means for effectively developing and implementing an architecture. The
Secretary of Defense established a program in July 2001 to develop and
implement an architecture. In April 2002, DOD entered into an agreement
with International Business Machines (IBM) pursuant to a governmentwide
General Services Administration contract, under which DOD issued task
orders for services to begin developing the architecture.
During the first year of DOD's architecture development, in 2002, we
reviewed the department's efforts and recognized that it was
undertaking a challenging and ambitious task and following some
architecture best practices and information technology (IT) investment
management processes and controls. We also identified challenges and
weaknesses in DOD's architecture efforts. For example, in a February
2003 report,[Footnote 17] we pointed out that DOD had not yet (1)
established a governance structure and process controls needed to
ensure ownership of and accountability for the architecture across the
department, (2) clearly communicated to intended stakeholders its
purpose, scope, and approach for developing the architecture, and (3)
defined and implemented an independent quality assurance process. We
also reported that DOD had yet to establish the necessary departmental
investment governance structure and process controls needed to
adequately align ongoing investments with its architectural goals and
direction. We made six recommendations aimed at enhancing DOD's ability
to further develop its architecture and guide and constrain its
business systems modernization investments.
Our March 2003[Footnote 18] report noted that the draft version of the
architecture did not include a number of items recommended by relevant
architectural guidance and that DOD's plans would not fully satisfy the
requirements of the National Defense Authorization Act for Fiscal Year
2003. For example, the draft architecture did not include a "To Be"
security view, which defines the security requirements, including
relevant standards to be applied in implementing security policies,
procedures, and controls. DOD officials agreed with our preliminary
assessment of the architecture and stated that subsequent versions of
the architecture would provide these missing details.
In July and September 2003,[Footnote 19] we reported that although DOD
had expended tremendous effort and resources in complying with
statutory requirements for developing and implementing a well-defined
architecture, the initial version of its architecture, including the
transition plan, did not adequately address the statutory requirements
and other relevant architectural requirements. For example, the "As Is"
environment did not include descriptions of the current business
operations in terms of entities and people who perform the functions,
processes, and activities and the locations where these are performed.
The "To Be" environment did not include descriptions of actual systems
to be developed or acquired to support future business operations and
the physical infrastructure that would be needed to support the
business systems. The transition plan did not include time frames for
phasing out existing systems within DOD's reported current inventory of
about 2,300 business systems. Overall, the department's initial version
of the architecture did not contain sufficient scope and detail either
to satisfy the act's requirements or to effectively guide and constrain
the departmentwide business transformation and systems modernization.
In our September 2003 report[Footnote 20], we reiterated the open
recommendations that we had made in our May 2001[Footnote 21] and
February 2003[Footnote 22] reports, and we made 10 new recommendations
aimed at improving DOD's plans for developing the next version of the
architecture and implementing the institutional means for selecting and
controlling both planned and ongoing business systems investments. To
date, DOD has yet to address 22 of our recommendations.
Funding Status of Architecture Program:
As of March 12, 2004, about $258 million had been appropriated to
support the program, of which the department reported having obligated
over $203 million and having made disbursements of $111 million since
the program began in 2002. Table 1 shows the reported status of program
funding by appropriation and fiscal year.
Table 1: Reported Funding Status of DOD's Business Management
Modernization Program as of March 12, 2004:
(Dollars in millions):
:
Appropriation; Appropriated; Obligated; Unobligated[A]; Disbursed;
Unliquidated[B].
Fiscal year 2002/2003 Research, Development, Test, and
Evaluation (RDT&E) -Defensewide (DW)[C];
Appropriated: $94.5;
Obligated: $94.5;
Unobligated[A]: $0.0;
Disbursed: $83.0;
Unliquidated[B]: $11.5.
Fiscal year 2003/2004 RDT&E -DW[D];
Appropriated: $67.2;
Obligated: $67.2;
Unobligated[A]: $0.0;
Disbursed: $11.9;
Unliquidated[B]: $55.3.
Fiscal year 2004/2005 RDT&E - DW[E];
Appropriated: $45.1;
Obligated: $8.5;
Unobligated[A]: $36.6;
Disbursed: $0.0;
Unliquidated[B]: $8.5.
Fiscal year 2003 Operations and Maintenance (O&M) -DW[F];
Appropriated: $24.9;
Obligated: $24.9;
Unobligated[A]: $0.0;
Disbursed: $16.1;
Unliquidated[B]: $8.8.
Fiscal year 2004 O&M-DW[G];
Appropriated: $26.1;
Obligated: $8.3;
Unobligated[A]: $17.8;
Disbursed: $0.0;
Unliquidated[B]: $8.3.
Total;
Appropriated: $257.8;
Obligated: $203.4;
Unobligated[A]: $54.4;
Disbursed: $111.0;
Unliquidated[B]: $92.4.
Source: Unaudited funding information contained in DOD's March 15, 2004
report to congressional defense committees. We did not validate the
accuracy and reliability of the funding information reported.
[A] Unobligated balances are the differences between funds appropriated
and obligated; they represent funds that have not been obligated for
expenditure. GAO's calculations are based on DOD's reported data.
[B] Unliquidated balances are the differences between the funds
obligated and disbursed; they represent funds that have been obligated
but have not yet been paid. GAO's calculations are based on DOD's
reported data.
[C] The fiscal year 2002/2003 RDT&E funds supported the initial
delivery of the architecture, transition plan, and change management
and communications initiatives.
[D] The fiscal year 2003/2004 RDT&E funds were used to refine the
architecture through business process modeling/reengineering for the
program's first increment and to fund test and evaluation activities,
verification and validation efforts, and engineering support.
[E] The fiscal year 2004/2005 RDT&E funds will be used to complete the
program's first increment and begin work on the second increment and to
fund the integration of the architecture, engineering support, and test
and evaluation activities.
[F] The fiscal year 2003 O&M funds were used for salaries, facilities,
supplies, and program management support contracts.
[G] The fiscal year 2004 O&M funds are being used for salaries,
facilities, supplies, and program management support contracts.
[End of table]
DOD Has Taken Few Actions to Address Our Previous Recommendations:
DOD has taken some steps since our last review, but it has not yet (1)
implemented key architecture management best practices,[Footnote 23]
such as assigning accountability and responsibility for directing,
overseeing, and approving the architecture to a committee or group
comprised of representatives from across DOD's major organization
components, (2) revised the architecture products to include the
missing scope and detail needed to guide and constrain the
implementation of system solutions, and (3) clearly defined near-term
and long-term plans for guiding its transformation activities. Until
the department addresses these key elements to fully satisfy relevant
architectural guidance, it will be challenged in its ability to produce
an architecture that is sufficient to guide and constrain its business
operations and systems modernization efforts. In response to our
recommendations, DOD has taken some actions, including establishing a
group to facilitate communication and coordination across the domains
for program activities; beginning to establish a configuration change
management process; recently issuing a policy governing the
development, maintenance, and implementation of the architecture; and
updating the initial version of the architecture. However, these
actions are not sufficient and do not fully address our concerns.
DOD Has Not Yet Implemented Key Architecture Management Best Practices:
DOD's actions have not been adequate to address architecture management
best practices or our previous recommendations. Since our last report,
the department has taken some actions in response to our
recommendations concerning the need to implement an effective
architecture management program. First, in September 2003, it formally
established the Domain Owners Integration Team (DO/IT), which reports
to the steering committee. The DO/IT is comprised of the various senior
executives from each domain and the Business Modernization and Systems
Integration office. The DO/IT is responsible for facilitating
communication and coordination across the domains for program
activities, including extending and evolving the architecture.
According to program officials, the DO/IT's role is continuing to
evolve. In particular, the specific actions or tasks it needs to
perform to carry out this responsibility have not been defined in
detail.
Second, DOD has taken steps to establish and implement a configuration
management process to ensure that all changes to the architecture
products are justified and are accounted for in a manner that maintains
documentation integrity. Specifically, the department has established a
configuration control board (CCB), developed a charter, and written
procedures governing this process. However, the CCB has been tasked
with reviewing changes to only some, but not all, of the architecture
products. For example, the CCB is not responsible for tracking changes
that are being made to the transition plan. In addition, both the
charter and the procedures governing this process are still in draft.
According to DOD officials, the charter is to be approved in June 2004,
but no time frame has been provided for final approval of the
procedures.
Third, DOD has recently issued an IT portfolio management policy
governing the development, maintenance, and implementation of the
architecture. This policy addresses, among other things, the roles,
responsibilities, and relationships of key players and program
participants, the value of an architecture, and the scope of the
architecture. However, the policy does not address accountability for
and approval of updates to the architecture, as called for by best
practices, nor does it address the issuance of waivers in those
instances when exceptions to the architecture are justified on the
basis of documented analysis.
Finally, DOD has developed high-level performance measures that are to
be used to develop the more specific results-oriented performance
metrics needed to enable it to evaluate program progress and benefits.
This means that DOD has yet to establish measurable, results-oriented
goals to evaluate and track, on an ongoing basis, specific program
progress, outcomes, and results. For example, it has not explicitly
defined performance measures to evaluate the quality, content, and
utility of subsequent major updates to its initial architecture. Given
that DOD has reported obligations of over $203 million since
architecture development efforts began 3 years ago, this is a serious
performance management weakness. It is critical that the department
establishes meaningful, tangible, and measurable program goals and
objectives--short-term and long-term--to enable the department to
determine what value it is receiving for its investment.
In addition, the department has yet to address other prior
recommendations related to architecture management best practices. Two
examples are provided below.
* DOD has not assigned accountability and responsibility for directing,
overseeing, and approving the architecture to a committee or group
comprised of representatives from across the department. Although it
previously established an executive and a steering committee, these
committees are advisory in nature, and they are currently not
accountable or responsible for directing, overseeing, and approving the
architecture. According to the department, it expects to revise the
committees' charters in June 2004 to include these responsibilities.
* DOD does not have an independent verification and validation function
to review the architecture products and management processes. The
department's current verification and validation contractor is not
independent[Footnote 24] and is responsible for reviewing the
architecture products and only selected program deliverables. According
to program officials, the department is planning to acquire the
services of a new contractor to perform this work, who will be tasked
with reviewing all the architecture products and all other program
deliverables. However, DOD officials also said that no decision has
been made to change the reporting structure of this function to make it
independent or to have this contractor review architecture management
processes. In addition, no milestones have been set for any of these
planned actions.
Several factors have contributed to the department's limited progress
in implementing an effective architecture management program, such as
changes to and a lack of accountability in program leadership,
direction, and priorities. As we previously stated, DOD has not yet
formalized responsibility for directing, overseeing, and approving the
architecture. Further, from May 2003 to February 2004, there was no
program manager to identify, direct, and execute program activities.
DOD hired a new program manager in February 2004. Initially, DOD had
planned to develop and implement its architecture in 1 year, but later
it adopted an incremental approach to developing the architecture.
About 1 year after adopting this approach, DOD has assigned labels to
these increments, but it has not yet defined the purpose of these
increments and plans of action to implement them.
Further, our research of successful organizations and experience in
reviewing other challenged enterprise architecture efforts show that
senior management's understanding of and commitment to an enterprise
architecture and overcoming cultural resistance to having and using one
are critical success factors. Cultural resistance to change, military
service parochialism, and stovepiped operations have all contributed
significantly to the failure of previous attempts to implement broad-
based management reforms at DOD. Until such barriers are addressed, and
effective architecture management structures and processes are
established, it is unlikely that DOD will be able to produce and
maintain a complete and enforceable architecture or implement
modernized systems in a way that minimizes overlap and duplication and
maximizes integration and mission support.
DOD's Architecture Products Remain Incomplete, with Minimal Content
Change:
Since our last review, DOD has not made significant changes to the
content of its architecture. The department has an updated version of
the architecture (version 2.0), which is currently being reviewed, but
it has not been approved. According to DOD, this version of the
architecture differs from the initial architecture (version 1.0) in
that it incorporates integrated "baseline reference business process
models," which, according to DOD, constitute a high-level process
framework consisting of processes that provide a business perspective
on the department's operations (e.g., execute budget and performance
plans) that cut across functions and organizations. DOD also stated
that version 2.0 includes the previously excluded requirements (e.g.,
revenue requirements) for the Joint Financial Management Improvement
Program (JFMIP)[Footnote 25] and partially addresses 21 of the 62
missing architecture content elements that we recommended be added.
According to DOD, the domains are currently validating all of the
requirementsóincluding the JFMIP requirements:
Further, with regard to the 21 missing elements, documentation provided
by DOD shows that it is either just beginning to initiate activities or
was planning to develop plans to address these missing elements. This
documentation also did not specify either how or when any of the 21
elements would be fully addressed. Instead, it showed that the
department, to date, has made only cosmetic changes to the
architecture, such as correcting typographical (i.e., misspelled words)
and grammatical errors, as well as deleting unnecessary text from
various architecture products. DOD has yet to incorporate the missing
scope and detail that we previously identified. For example, the "To
Be" environment does not provide sufficient descriptive content related
to future business operations and supporting technology to permit
effective acquisition and implementation of system solutions and the
associated operational change.
Program officials also stated that version 2.0 addresses some of the
recommendations made by its verification and validation contractor.
However, DOD did not provide documentation showing that it had
addressed any of these recommendations. For example, the verification
and validation contractor had previously reported that the "As Is"
information was insufficient to support realistic transition planning.
In April 2004, this contractor further reported that, while there have
been some improvements in both the overall completeness of definitions
and the structure of the architecture products, version 2.0[Footnote
26] retains most of the critical problems previously cited for version
1.0. Some examples are below.
* The utility of the architecture to stakeholders and the ability of
the architecture to support acquisition and portfolio investment
management decisions remain unclear.
* Supporting documentation that describes the analysis and rationale
for architecture choices represented within many of the architecture
products remains incomplete in many areas and, when available, is
poorly linked or referenced.
* The "As Is" environment, including business processes and existing
business application systems and supporting technology, is inadequate,
making it difficult for DOD to perform a gap analysis to support
development of a transition plan.
* The information assurance (security) representation remains spotty
and difficult to find even at the enterprise level, and it has some
unnecessary deviations from the department's accepted practices.
In addition, the verification and validation contractor stated that the
reference business process models included in version 2.0, which DOD
cited as the major difference from version 1.0, lacked the underlying
data and linkage to make it useful in reengineering existing processes.
The contractor also stated that these models would likely need to be
reworked based on the results of efforts that the domains currently
have under way.
According to DOD and contractor officials responsible for updating the
architecture products, the changes made to date have primarily affected
architecture management processes and not the architecture's content;
and this is the result of the change in the program's focus and
priorities, as discussed above. However, these officials also stated
that the Architecture Integration Teams (AIT), comprised of
representatives from the domains, are currently focused on developing
needed architectural contentóprimarily business processes, business
rules, and regulations for increment one.
Until the architecture is sufficiently complete and includes the
missing scope and detail, the department remains at risk for not
achieving its intended business transformation goals and of not having
an architecture that can be used to guide and constrain ongoing and
planned business systems investments to prevent duplicative and
noninteroperable systems.
DOD Has Yet To Explicitly Define Near-term and Long-term Plans for
Guiding Its Transformation Activities:
DOD has not yet developed either near-term or long-term plans for
developing the architecture that explicitly identify and establish a
baseline for the actions to be taken, milestones to be achieved, cost
estimates to be met, and targeted outcomes to be achieved. As we
previously stated, DOD has adopted an incremental approach to
developing the architecture, including the transition plan, and plans
to refine and extend the architecture in three increments. The
increments, as defined by DOD, are:
* Increment one. To enable asset accountability, total force
visibility, and an unqualified audit opinion of DOD's consolidated
fiscal year 2007 financial statements.
* Increment two. To focus on reducing acquisition cycle time and
streamlining the Planning, Programming, Budgeting, and Execution System
process between fiscal years 2004 and 2009.
* Increment three. To focus on providing total asset visibility and
total force management between fiscal years 2004 and 2010.
However, it is unclear what the increments individually or collectively
mean, and what they will provide or allow DOD to achieve in the near-
term and long-term, because DOD does not have detailed plans that
include performance measures for the quality, content, and utility of
the architecture. Although the three increments were identified in
November 2003, program officials do not expect to have a plan for
increment one until the next version of the transition plan is
completed in August 2004. According to program officials, the goals and
scope for the second and third increments were only recently approved
by the steering committee and, therefore, detailed plans of action and
milestones for developing these plans do not yet exist.
Currently, DOD has three initiatives under way to support increment
one. First, the program office is developing a plan of action for
increment one and intends to complete the plan by August 2004. Second,
the accounting and finance domain is conducting workshops to develop
needed business rules and requirements for extending and evolving
version 2.0 of the architecture. This domain also has two ongoing pilot
initiatives to acquire and implement production accounting systems by
the end of 2005 at an estimated cost of $135 million, which are
intended to provide JFMIP-compliant financial management systems to
support the department's general and working capital fund activities.
Last, DOD components are developing individual plans detailing their
respective efforts for supporting increment one. However, there is no
evidence that the program office is coordinating with the components
and that the components are coordinating amongst themselves. According
to a Defense Logistics Agency (DLA) official, there is currently no
direct link between DLA's plan and modernization program activities.
Because there are not yet detailed plans guiding the program's
activities, it is unclear whether and how these activities support each
other and whether they support the department's goal of achieving an
unqualified audit opinion in 2007.
DOD recognizes that it needs to develop detailed plans and establish
performance metrics to measure and track program progress in order to
determine (1) what it wants to accomplish by a certain point in time,
(2) what it has actually accomplished by that point in time, and (3)
how much it has spent. Changes in the direction of the program and a
lack of sustained leadership have hindered DOD's ability to do so. In
its March 15, 2004 progress report, DOD reported that it plans to
establish an initial approved program metrics baseline to evaluate the
cost, schedule, and performance of the program and that, beginning with
the fourth quarter of fiscal year 2004, it plans to begin formal
tracking and reporting of specific program goals, objectives, and
measures.
The lack of explicitly defined program plans also has made it difficult
to define measurable tasks that are or will be assigned to DOD
employees and the 289 program contractor employees who are involved in
architecture development, domain support, and program support
functions. In reviewing the contractor's work statements for
architecture development, we found that many of the tasks lacked the
specificity necessary to use them effectively to monitor the
contractor's progress. For example, tasks included such broad
statements as "maintain, extend, and integrate" the architecture. Since
DOD is acquiring services on the basis of direct labor hours at
specified fixed hourly rates and cost of materials, it is essential
that the department use efficient methods and effective cost controls
to measure the work being performed.[Footnote 27] It is also essential
because the proposed work statement that DOD plans to approve, at an
estimated cost of $43 million, explicitly states that work is
considered complete when the hours allotted have been expended, rather
than defining completion as the contractor's delivery of products that
meet predefined quality standards. According to DOD, to enable it to
better monitor the contractor's progress, it plans to reduce future
tasks to 15 staff-day increments (from 6 week increments) and to
require the contractor to provide more frequent, informal products to
program officials so they can monitor and track progress more
rigorously.
Without an explicitly defined program baseline, detailed plans, and
performance measures, it is difficult to validate or justify the $122
million that DOD has requested for fiscal year 2005 and the $494
million the department estimates it will need for fiscal years 2006
through 2009. In fact, DOD has been unable to show measurable progress
and meaningful utility of the architecture products to DOD stakeholders
for the $203.4 million that had been obligated for this program as of
March 2004.
DOD Has Taken Few Actions to Control Ongoing and Planned Investments in
Business Systems:
DOD continues to lack effective investment management oversight and
control over its numerous investments in business systems. While the
domains have been designated to oversee business systems investments,
the actual funding continues to be spread among DOD's components, which
continue to make their own parochial decisions regarding those
investments, including obligations in excess of $1 million for system
improvements, without having received the scrutiny of DOD's Comptroller
as required by the National Defense Authorization Act for Fiscal Year
2003.[Footnote 28] Because DOD lacks a departmentwide focus and
effective management oversight and control of its business systems
investments, it continues to invest billions of dollars in systems that
potentially will fail to deliver the integrated business systems
outcomes that are needed to provide DOD management with timely and
reliable financial information.
We previously recommended[Footnote 29] that DOD establish investment
review boards to better control its business systems investments (with
each board comprised of representatives from across the department) and
that the boards, consistent with recognized best practices, use a
standard set of investment review and decision-making criteria to
ensure compliance and consistency with the architecture. DOD agreed
with our recommendations and, in response, it issued in March 2004 an
IT portfolio management policy that assigns the domains responsibility
for IT portfolio management. However, the procedures to be followed to
implement the policy are still under development and no time frames for
completion have been provided. According to the IT portfolio management
policy, the department has 180 days (until mid-September 2004) to
develop these procedures. In addition, the department has yet to
formalize specific roles and responsibilities of the domains, develop
standard criteria for performing the system reviews, and assign
explicit authority for fulfilling roles and responsibilities. DOD
recognizes the need to clarify the roles and responsibilities
associated with managing the domains' portfolios of business systems
and ensure compliance with the architecture. However, it has yet to
establish time frames for completing these activities.
DOD has not yet implemented an effective investment management
structure and processes for controlling ongoing and planned business
systems investments, including one that meets the act's requirements
for ensuring that system improvements with obligations in excess of $1
million are consistent with the architecture. In an attempt to
substantiate that the obligations for business systems modernization
were in accordance with the National Defense Authorization Act for
Fiscal Year 2003, we requested that DOD agencies provide us with a list
of obligations greater than $1 million for fiscal year 2003[Footnote
30] and fiscal year 2004, as of December 2003. To ascertain whether
DOD's Comptroller had made the determination required by the act, we
compared a list of systems approvals provided by the program office
with the obligational data (by system) provided by the DOD activities.
As we previously testified,[Footnote 31] we identified $479 million in
obligations for business systems modernization reported by the DOD
military services that were not submitted to DOD's Comptroller for
review. Subsequently, we requested information from the defense
agencies and found $384 million in reported obligations for business
systems modernization that had not been submitted to DOD's Comptroller
for review. Examples of DOD system improvements with obligations in
excess of $1 million that were not submitted include the following:
* The Defense Finance and Accounting Service (DFAS) obligated about $19
million in fiscal year 2003 for the DFAS Corporate Database/DFAS
Corporate Warehouse. We previously reported[Footnote 32] that DOD had
yet to provide economic justification that its investment in this
system would result in tangible improvements to DOD's financial
management operations. As of April 2004, an economic analysis has yet
to be approved but, according to DOD officials, about $129 million had
been spent on the program through January 2004.
* The Defense Information Systems Agency obligated about $7 million in
fiscal year 2003 and about $2 million in fiscal year 2004 for the Wide
Area Workflow.[Footnote 33]
* DFAS obligated about $7 million in fiscal year 2003 for the Defense
Standard Disbursing System before it was terminated in December 2003
after approximately 7 years of effort and a reported investment of
about $53 million. We previously reported[Footnote 34] that continued
investment in this system had not been justified, because an economic
analysis had not been updated to reflect significant schedule delays.
DFAS noted that this system was terminated because a valid business
case for continuing the effort could not be made.
* DLA obligated about $5 million in fiscal year 2003 and about $2
million in fiscal year 2004 for the Defense Medical Logistics Standard
Support Program.
These and other examples demonstrate that DOD does not have reasonable
assurance that it is in compliance with the National Defense
Authorization Act for Fiscal Year 2003, which provides that obligations
in excess of $1 million for system improvements may not be made unless
DOD Comptroller makes a determination that the improvement is in
accordance with the criteria specified in the act. The act places
limitations on the legal authority of individual program and government
contracting officials to obligate funds in support of the systems for
which they are responsible, but DOD has yet to proactively manage its
investments to avoid violations of the limitations and to review and
approve investments in any meaningful way in order to comply with these
statutory limitations.
DOD acknowledges that the department does not have a systematic means
to identify and determine which system improvements should be submitted
to DOD's Comptroller for review and, in essence, is dependent upon
system owners coming forward to the domains and requesting approval.
Also, as we have testified,[Footnote 35] DOD components continue to
receive direct funding for their business systems and continue to make
their own parochial decisions regarding investments without the
scrutiny of DOD's Comptroller that is required by the act. The current
funding process has contributed to the proliferation of duplicative,
nonintegrated, and stovepiped business systems that are highly prone to
error and are unable to provide DOD management with timely, reliable
financial information. In March 2004,[Footnote 36] we offered a
suggestion for legislative action to address this issue, consistent
with our open recommendations to DOD that funds for its business
systems be appropriated directly to the domains in order to provide for
accountability, transparency, and the ability to prevent the continued
parochial approach to systems investments that exists today. The
legislation we envisioned--for which we will make a legislative
proposal in a related report to be issued soon--would define the scope
of the domains and establish functional responsibility for managing the
portfolio of business systems to the domains. The domains would
establish business systems investment review boards with DOD-wide
representation, including the military services and defense agencies.
Until DOD strengthens its process for selecting and controlling
business systems investments and sufficiently develops a well-defined
architecture that can be used to guide and constrain investment
decisions, it will likely continue to spend billions of dollars on
duplicative, nonintegrated, stovepiped systems that do not optimize
mission performance and accountability and, therefore, do not support
the department's transformation goals.
Conclusions:
Having and effectively using a well-defined architecture is essential
for guiding and constraining DOD's business transformation efforts and
moving the department away from nonintegrated business systems
development efforts. DOD's efforts to date to address our prior
recommendations aimed at accomplishing this have not been sufficient.
Specifically, despite 3 years of effort and over $203 million in
reported obligations, DOD's architecture remains insufficiently
defined, and the way in which the department makes business systems
investments decisions remains largely unchanged. As a result, billions
of dollars continue to be at risk of being spent on more systems that
are duplicative, are not interoperable, cost more to maintain than
necessary, and do not optimize mission performance and accountability.
The future of DOD's architecture development and implementation
activities is at risk; this in turn places the department's business
transformation effort in jeopardy of failing as other efforts by the
department have in the past. Therefore, it is imperative that DOD move
swiftly to implement our 22 open recommendations, which are aimed at
strengthening architecture management activities, adding missing
content to architecture products, and implementing investment oversight
and control measures.
Agency Comments and Our Evaluation:
In commenting on a draft of this report (reprinted in enc. IV), DOD
agreed with our assessment that its long-standing business systems
problems have resulted in a lack of adequate transparency and
appropriate accountability across all major business areas. It
confirmed that development and use of an architecture is necessary to
transforming its business operations and supporting systems. However,
in response to our characterization that progress has been limited, DOD
stated that, over the past 3 years, progress has been slower than
either GAO or DOD would prefer but that it has been significant,
despite appearances to the contrary. The willingness of DOD's
leadership to tackle the department's decades-old problems with its
business operations and supporting systems represents a major step
forward. However, we continue to believe that our characterization of
DOD's progress in developing a well-defined architecture and
implementing effective management oversight and control over its
business systems as limited is fair. Therefore, we are not making any
changes to our report.
DOD stated that we did not adequately consider or incorporate in our
report important completed and ongoing activities, plans, and
documentation, and provided an extensive package of information. We
disagree that we did not adequately consider or incorporate in our
report important completed and ongoing activities, plans, and
documentation. While our assessment focused primarily on the
department's completed, ongoing, and planned activities that it
reported to Congress on March 15, 2004, we also considered and
incorporated various activities, plans, and documentation through April
27, 2004, which included the vast majority of the information in the
package.
In addition, we acknowledged in this report most, if not all, of the
actions DOD represents in their comments as having been completed such
as issuing the IT portfolio management policy and establishing the DO/
IT. We also recognized many of DOD's ongoing activities, such as the
workshops being held by the accounting and finance domain and the
department's establishment of the AIT, which are currently focused on
developing needed architectural content for increment one. To gain an
understanding of the specific AIT activities, which began in January
2004, we met with the domains during our review and discussed various
collaborative efforts, such as their participation in validating
business rules and requirements to support the development of end-to-
end business processes. Regarding planned activities, we recognized
DOD's intent to revise the executive and steering committees' charters
to include responsibilities for directing, overseeing, and approving
the architecture, and to develop the procedures for implementing the IT
portfolio management policy. In its response, DOD stated that it plans
to complete these actions in June and December 2004, respectively.
Because we received information about certain activities after we
completed our fieldwork--or not at all--these activities are not
included in our evaluation. For example, DOD stated that the department
plans to release a management initiative during May 2004; establish an
architecture and program management maturity plan and supporting
metrics to address the quality, content, and utility of the
architecture; and establish a balanced scorecard and supporting metrics
focused on the organization's internal management processes. While we
inquired about the management initiative during our review, program
officials stated that they were unable to provide us a copy because DOD
does not allow the release of such draft documents outside the
department. In addition, program officials did not provide a
characterization of the focus or scope of the management initiative;
therefore we were unable to include it in our report. We made inquiries
about other initiatives during the course of our review, but were not
advised of the maturity plan and balanced scorecard. However, it is
important to note that carrying out activities does not necessarily
guarantee results, and thus progress, unless these activities are based
on an approved program plan that can be used to measure progress.
In addition, we disagree with DOD's other comments. Specifically, DOD
stated that the Comptroller is accountable for approval of updates to
the architecture as called for by best practices. As we previously
reported[Footnote 37] and reiterated in this report, best practices
recommend that the accountability and responsibility for directing,
overseeing, and approving the architecture be assigned to a committee
or group comprised of representatives responsible for the core business
areas across the department--not a single individual. The purpose of
assigning the accountability and responsibility to such a committee is
to obtain and ensure continued enterprisewide support of the
architecture effort, thereby, increasing the department's chances for
success.
In addition, DOD stated that its verification and validation function
is independent because the contractor responsible for these activities
currently reports to the Deputy Director, Program Support, who is not
directly involved with developing the architecture. DOD also stated
that to ensure this independence, reports prepared by this contractor
are provided directly to GAO and the steering committee. We disagree
that this function is sufficiently independent. According to best
practices, the contractor performing this role should report the
results of its work directly to the steering committee, as well as the
program office. Since the Deputy Director, Program Support reports
directly to the program management office and not the steering
committee, the function is not independent. In addition, the
verification and validation contractor agreed with us that its reports
are not provided directly to GAO or the steering committee. The
contractor also concurred with our statement that, in the one instance
in which it briefed the steering committee, the program office modified
the content of the presentation.
DOD disagreed with our prior position that funds for DOD business
systems be appropriated directly to the domains. We are reaffirming
that position as stated by the Comptroller General in his March 2004
testimony[Footnote 38] before the Subcommittee on Readiness and
Management Support, Senate Armed Services Committee. This matter for
congressional consideration and related changes in responsibility will
be discussed in additional detail in a report to be issued shortly.
We are sending copies of this report to interested congressional
committees; the Director, Office of Management and Budget; the
Secretary of Defense; the Under Secretary of Defense (Comptroller); the
Assistant Secretary of Defense (Networks and Information Integration)/
DOD Chief Information Officer; the Under Secretary of Defense
(Acquisition, Technology, and Logistics); the Under Secretary of
Defense (Personnel and Readiness); and the Director, Defense Finance
and Accounting Service. This report will also be available at no charge
on our Web site at http://www.gao.gov.
If you have any questions concerning this information, please contact
Gregory Kutz at (202) 512-9095 or kutzg@gao.gov or Randolph Hite at
(202) 512-3439 or hiter@gao.gov. GAO contacts and key contributors to
this report are listed in enclosure V.
Signed by:
Gregory D. Kutz:
Director, Financial Management and Assurance:
Signed by:
Randolph C. Hite:
Director, Information Technology Architecture and Systems Issues:
Enclosures:
List of Committees:
The Honorable John W. Warner:
Chairman:
The Honorable Carl Levin:
Ranking Minority Member:
Committee on Armed Services:
United States Senate:
The Honorable Ted Stevens:
Chairman:
The Honorable Daniel K. Inouye:
Ranking Minority Member:
Subcommittee on Defense:
Committee on Appropriations:
United States Senate:
The Honorable Duncan Hunter:
Chairman:
The Honorable Ike Skelton:
Ranking Minority Member:
Committee on Armed Services:
House of Representatives:
The Honorable Jerry Lewis:
Chairman:
The Honorable John P. Murtha:
Ranking Minority Member:
Subcommittee on Defense:
Committee on Appropriations:
House of Representatives:
Enclosure I:
SEC. 1004. [of Public Law 107-314] DEVELOPMENT AND IMPLEMENTATION OF
FINANCIAL MANAGEMENT ENTERPRISE ARCHITECTURE:
(a) REQUIREMENT FOR ENTERPRISE ARCHITECTURE AND FOR TRANSITION PLAN--
Not later than May 1, 2003, the Secretary of Defense shall develop--
(1) a financial management enterprise architecture for all budgetary,
accounting, finance, enterprise resource planning, and mixed
information systems of the Department of Defense; and:
(2) a transition plan for implementing that financial management
enterprise architecture.
(b) COMPOSITION OF ENTERPRISE ARCHITECTURE--
(1) The financial management enterprise architecture developed under
subsection (a)(1) shall describe an information infrastructure that, at
a minimum, would enable the Department of Defense to--
(A) comply with all Federal accounting, financial management, and
reporting requirements;
(B) routinely produce timely, accurate, and reliable financial
information for management purposes;
(C) integrate budget, accounting, and program information and systems;
and:
(D) provide for the systematic measurement of performance, including
the ability to produce timely, relevant, and reliable cost information.
(2) That enterprise architecture shall also include policies,
procedures, data standards, and system interface requirements that are
to apply uniformly throughout the Department of Defense.
(c) COMPOSITION OF TRANSITION PLAN--The transition plan developed under
subsection (a)(2) shall include the following:
(1) The acquisition strategy for the enterprise architecture, including
specific time-phased milestones, performance metrics, and financial and
nonfinancial resource needs.
(2) A listing of the mission critical or mission essential operational
and developmental financial and nonfinancial management systems of the
Department of Defense, as defined by the Under Secretary of Defense
(Comptroller), consistent with budget justification documentation,
together with--
(A) the costs to operate and maintain each of those systems during
fiscal year 2002; and:
(B) the estimated cost to operate and maintain each of those systems
during fiscal year 2003.
(3) A listing of the operational and developmental financial management
systems of the Department of Defense as of the date of the enactment of
this Act (known as 'legacy systems') that will not be part of the
objective financial and nonfinancial management system, together with
the schedule for terminating those legacy systems that provides for
reducing the use of those legacy systems in phases.
(d) CONDITIONS FOR OBLIGATION OF SIGNIFICANT AMOUNTS FOR FINANCIAL
SYSTEM IMPROVEMENTS--An amount in excess of $1,000,000 may be obligated
for a defense financial system improvement only if the Under Secretary
of Defense (Comptroller) makes a determination regarding that
improvement as follows:
(1) Before the date of an approval specified in paragraph (2), a
determination that the defense financial system improvement is
necessary for either of the following reasons:
(A) To achieve a critical national security capability or address a
critical requirement in an area such as safety or security.
(B) To prevent a significant adverse effect (in terms of a technical
matter, cost, or schedule) on a project that is needed to achieve an
essential capability, taking into consideration in the determination
the alternative solutions for preventing the adverse effect.
(2) On and after the date of any approval by the Secretary of Defense
of a financial management enterprise architecture and a transition plan
that satisfy the requirements of this section, a determination that the
defense financial system improvement is consistent with both the
enterprise architecture and the transition plan.
(e) CONGRESSIONAL REPORTS--Not later than March 15 of each year from
2004 through 2007, the Secretary of Defense shall submit to the
congressional defense committees a report on the progress of the
Department of Defense in implementing the enterprise architecture and
transition plan required by this section. Each report shall include, at
a minimum--
(1) a description of the actions taken during the preceding fiscal year
to implement the enterprise architecture and transition plan (together
with the estimated costs of such actions);
(2) an explanation of any action planned in the enterprise architecture
and transition plan to be taken during the preceding fiscal year that
was not taken during that fiscal year;
(3) a description of the actions taken and planned to be taken during
the current fiscal year to implement the enterprise architecture and
transition plan (together with the estimated costs of such actions);
and:
(4) a description of the actions taken and planned to be taken during
the next fiscal year to implement the enterprise architecture and
transition plan (together with the estimated costs of such actions).
(f) COMPTROLLER GENERAL REVIEW--Not later than 60 days after the
approval of an enterprise architecture and transition plan in
accordance with the requirements of subsection (a), and not later than
60 days after the submission of an annual report required by subsection
(e), the Comptroller General shall submit to the congressional defense
committees an assessment of the extent to which the actions taken by
the Department comply with the requirements of this section.
(g) DEFINITIONS--In this section:
(1) The term 'defense financial system improvement' means the
acquisition of a new budgetary, accounting, finance, enterprise
resource planning, or mixed information system for the Department of
Defense or a modification of an existing budgetary, accounting,
finance, enterprise resource planning, or mixed information system of
the Department of Defense. Such term does not include routine
maintenance and operation of any such system.
(2) The term 'mixed information system' means an information system
that supports financial and non-financial functions of the Federal
Government as defined in Office of Management and Budget Circular A-127
(Financial Management Systems).
(h) REPEAL--(1) Section 2222 of title 10, United States Code, is
repealed. The table of sections at the beginning of chapter 131 of such
title is amended by striking the item relating to such section.
(2) Section 185(d) of such title is amended by striking 'has the
meaning given that term in section 2222(c)(2) of this title' and
inserting 'means an automated or manual system from which information
is derived for a financial management system or an accounting system'.
Enclosure II: Scope and Methodology:
Consistent with the act and as agreed with congressional defense
committees' staffs, our review focused on (1) the actions the
Department of Defense (DOD) has taken to address our previous
recommendations regarding the development and implementation of the
architecture and (2) the actions DOD is taking to ensure its ongoing
and planned investments will be consistent with its evolving
architecture.
To determine what actions DOD has taken to address our previous
recommendations that relate to the development and implementation of
the architecture, we met with DOD and contractor officials to obtain an
update on the status of our prior recommendations. We used our
Enterprise Architecture Management Maturity Framework,[Footnote 39]
which describes the stages of management maturity, to update the status
of key elements of architecture management best practices that DOD had
not adopted. To make this determination, we reviewed program
documentation, such as program policies and procedures, communications
plan, and charters for the governance bodies; and we compared them to
the elements in the framework. We reviewed program documentation, such
as requests documenting proposed and actual changes to the architecture
and external[Footnote 40] requirements extracted from the updated
architecture. We also reviewed contractor task orders to determine the
work performed and planned by the prime contractor associated with
extending and evolving the architecture.
To determine what actions DOD is taking to ensure its ongoing and
planned business systems investments are consistent with its evolving
architecture, we met with DOD officials to obtain an update on the
status of our prior recommendations. We met with appropriate officials
from the offices of the Under Secretary of Defense (Comptroller) and
the Assistant Secretary of Defense (Networks and Information
Integration)/DOD Chief Information Officer, and with representatives
from the domains to discuss the status of various draft policies and
guidance that are aimed at improving the department's control of and
accountability for business systems investments. We also reviewed and
analyzed DOD's budget request for fiscal year 2004 to identify the
business systems investments that could be subject to the requirements
of the National Defense Authorization Act for Fiscal Year 2003, which
requires that all financial system improvements with obligations in
excess of $1 million be reviewed by DOD's Comptroller, who must make a
determination on whether the system improvements are in accordance with
criteria specified in the act. To assess DOD's compliance with the act,
we also obtained and reviewed departmental guidance, memorandums, DOD
Comptroller review decisions, and other documentation provided by the
program office. Additionally, we requested that DOD provide us with
data on obligations in excess of $1 million for business systems
modernization for fiscal years 2003 and 2004, as of December 2003. We
also received obligational data from some of the defense agencies. We
then compared the obligational data with the information we had
received from the program office to ascertain if the systems
modernization had been reviewed.
To augment our document reviews and analyses, we interviewed officials
from various DOD organizations and contractors, including the Office of
the Under Secretary of Defense (Comptroller); Office of the Under
Secretary of Defense (Acquisition, Technology, and Logistics); Office
of the Under Secretary of Defense (Personnel and Readiness);
International Business Machines; and MITRE Corporation.
We did not validate the accuracy and reliability of the funding
information contained in DOD's March 15, 2004, report. Also, the
unliquidated and unobligated calculations that we made were based on
DOD reported data. Further, we did not validate the accuracy and
reliability of the obligational data provided by the military services
and the defense agencies for systems modernization.
We requested comments on a draft of this report from the Secretary of
Defense or his designee. We received written comments from the Acting
Under Secretary of Defense (Comptroller), which we printed in enclosure
IV. We considered, but did not reprint the voluminous attachments that
DOD provided with its written comments.
We conducted our work primarily at DOD headquarters offices in
Washington, D.C., and Arlington, Virginia, from December 2003 through
April 2004, in accordance with U.S. generally accepted government
auditing standards.
Enclosure III: Status of Prior Recommendations on DOD's Business
Enterprise Architecture (BEA):
GAO-01-525: Information Technology: Architecture Needed to Guide
Modernization of DOD's Financial Operations. May 17, 2001.
Recommendation: (1) The Secretary of Defense immediately designate DOD
financial management modernization a departmental priority and
accordingly direct the Deputy Secretary of Defense to lead an
integrated program across the department for modernizing and optimizing
financial management operations and systems;
Status of recommendation: Implemented;
DOD‘s comments and our assessment: DOD‘s comments and our assessment:
[Empty].
Recommendation: (2) The Secretary immediately issue a DOD policy that
directs the development, implementation, and maintenance of a financial
management enterprise architecture;
Status of recommendation: Partially implemented;
DOD‘s comments and our assessment: DOD recently issued its Information
Technology (IT) Portfolio Management policy. This policy, in
conjunction with the overarching Global Information Grid[A] policy,
assigns responsibilities for the development, implementation, and
maintenance of the BEA.b However, it does not provide for having
accountability for and approval of updates to the BEA, processes for
BEA oversight and control, and review and validation of the BEA.
Recommendation: (3) The Secretary immediately modify the Senior
Financial Management Oversight Council's charter to;
designate the Deputy Secretary of Defense as the Council Chair and the
Under Secretary of Defense (Comptroller) as the Council vice-Chair;
* empower the Council to serve as DOD's financial management enterprise
architecture steering committee, giving it the responsibility and
authority to ensure that a DOD financial management enterprise
architecture is developed and maintained in accordance with the DOD
C4ISR Architecture Framework;
* empower the Council to serve as DOD's financial management investment
review board, giving it the responsibility and authority to (1) select
and control all DOD financial management investments and (2) ensure
that its investment decisions treat compliance with the financial
management enterprise architecture as an explicit condition for
investment approval that can be waived only if justified by a
compelling written analysis;
and;
* expand the role of the Council's System Compliance Working Group to
include supporting the Council in determining the compliance of each
system investment with the enterprise architecture at key decision
points in the system's development or acquisition life cycle;
Status of recommendation: Partially implemented;
DOD‘s comments and our assessment: The department had previously
established the executive and steering committees, which we reported
were advisory in nature. The department recently established the Domain
Owners Integration Team[C] and stated that these three bodies are
responsible for governing the program. However, these three groups have
not been assigned the accountability and responsibility for directing,
overseeing, and approving the BEA. According to DOD, it expects to
revise and finalize the executive and steering committees' charters to
include these responsibilities in June 2004; DOD issued an IT portfolio
management policy (March 2004) that assigns the domains[D]
responsibility for IT portfolio management. However, the procedures to
be followed to implement the policy are still under development and no
time frames for completion have been provided. According to the IT
portfolio management policy, the department has 180 days (until mid-
September 2004) to develop these procedures. In addition, the
department has yet to formalize specific roles and responsibilities of
the domains, develop standard criteria for performing the system
reviews, and assign explicit authority for fulfilling roles and
responsibilities. DOD has yet to establish time frames for completing
these activities.
Recommendation: (4) The Secretary immediately make the Assistant
Secretary of Defense (Command, Control, Communications & Intelligence),
in collaboration with the Under Secretary of Defense (Comptroller),
accountable to the Senior Financial Management Oversight Council for
developing and maintaining a DOD financial management enterprise
architecture; In fulfilling this responsibility, the Assistant
Secretary appoint a Chief Architect for DOD financial management
modernization and establish and adequately staff and fund an enterprise
architecture program office that is responsible for developing and
maintaining a DOD-wide financial management enterprise architecture in
a manner that is consistent with the framework defined in the CIO
Council's published guide for managing enterprise architectures. In
particular, the Assistant Secretary should take appropriate steps to
ensure that the Chief Architect;
* obtains executive buy-in and support;
* establishes architecture management structure and controls;
* defines the architecture process and approach;
* develops the baseline architecture, the target architecture, and the
sequencing plan;
* facilitates the use of the architecture to guide financial management
modernization projects and investments;
and;
* maintains the architecture;
Status of recommendation: Partially implemented;
DOD‘s comments and our assessment: The Assistant Secretary of Defense
(Networks and Information Integration)/DOD Chief Information Officer is
a member of the executive and steering committees; however, as
discussed previously, members' roles and responsibilities are advisory
in nature; DOD established a Financial Management Modernization Program
Office in July 2001. DOD has also appointed a chief architect and,
according to the department, it has adequate program funding and staff
for developing and maintaining its BEA. However, DOD has not yet
defined the roles and responsibilities for the Chief Architect.
Recommendation: (5) The Assistant Secretary of Defense (Command,
Control, Communications & Intelligence) reports at least quarterly to
the Senior Financial Management Oversight Council on the Chief
Architect's progress in developing a financial management enterprise
architecture, including the Chief Architect's adherence to enterprise
architecture policy and guidance from OMB, the CIO Council, and DOD;
Status of recommendation: Partially implemented;
DOD‘s comments and our assessment: The Assistant Secretary of Defense
(Networks and Information Integration)/ DOD Chief Information Officer
is a member of the steering committee, which is briefed monthly by the
program office on various program activities;
but not necessarily on the development and maintenance of the BEA's
content. The Chief Architect is a member of the program office.
Recommendation: (6) The Senior Financial Management Oversight Council
report to the Secretary of Defense every 6 months on progress in
developing and implementing a financial management enterprise
architecture;
Status of recommendation: Partially implemented;
DOD‘s comments and our assessment: The Deputy Chief Financial Officer
briefed the Secretary of Defense in November 2003 on behalf of DOD's
Comptroller, who chairs the executive committee.
Recommendation: (7) The Secretary reports every 6 months to the
congressional defense authorizing and appropriating committees on
progress in developing and implementing a financial management
enterprise architecture;
Status of recommendation: Partially implemented;
DOD‘s comments and our assessment: Senate Report 107-213 directs that
the department report every 6 months on the status of the BEA effort.
DOD submitted status reports on January 31 and July 31, 2003 and on
January 31, 2004. However, these reports were submitted by DOD's
Comptroller and were not signed by the members of the executive or
steering committees.
Recommendation: (8) Until a financial management enterprise
architecture is developed and the Council is positioned to serve as
DOD's financial management investment review board as recommended
above, the Secretary of Defense limit DOD components' financial
management investments to;
* deployment of systems that have already been fully tested and involve
no additional development or acquisition cost;
* stay-in-business maintenance needed to keep existing systems
operational;
* management controls needed to effectively invest in modernized
systems;
and;
* new systems or existing system changes that are congressionally
directed or are relatively small, cost effective, and low risk and can
be delivered in a relatively short time frame;
Status of recommendation: Not implemented;
DOD‘s comments and our assessment: DOD has not yet defined and
implemented an approach for selecting and controlling business systems
investments. In March 2004, the Deputy Secretary of Defense signed the
IT portfolio management policy and assigned responsibility to the
domains for IT portfolio management. However, the procedures to be
followed to implement the policy are still under development and no
time frames for completion have been provided. According to the IT
portfolio management policy, the department has 180 days (until mid-
September 2004) to develop these procedures. In addition, the
department has yet to formalize specific roles and responsibilities of
the domains, develop standard criteria for performing the system
reviews, and assign explicit authority for fulfilling roles and
responsibilities. DOD has yet to establish time frames for completing
these activities.
GAO-03-458: DOD Business Systems Modernization: Improvements to
Enterprise Architecture Development and Implementation Efforts Needed.
February 28, 2003.
Recommendation: (1) The Secretary of Defense ensure that the enterprise
architecture executive committee members are singularly and
collectively made explicitly accountable to the Secretary for the
delivery of the enterprise architecture, including approval of each
version of the architecture;
Status of recommendation: Partially implemented;
DOD‘s comments and our assessment: The department had previously
established the executive and steering committees, which we reported
were advisory in nature. The department recently established the Domain
Owners Integration Team and stated that these three bodies are
responsible for governing the program. However, these groups have not
been assigned the accountability and responsibility for directing,
overseeing, and approving the BEA. According to DOD, it expects to
revise and finalize the executive and steering committees' charters to
include these responsibilities in June 2004.
Recommendation: (2) The Secretary of Defense ensure that the enterprise
architecture program is supported by a proactive marketing and
communication program;
Status of recommendation: Not implemented;
DOD‘s comments and our assessment: DOD has a strategic communications
plan;
however, the plan has not yet been executed. According to DOD, the plan
is scheduled to be executed between June and December 2004.
Recommendation: (3) The Secretary of Defense ensure that the quality
assurance function;
* includes the review of adherence to process standards and reliability
of reported program performance,
* is made independent of the program management function, and;
* is not performed by subject matter experts involved in the
development of key architecture products;
Status of recommendation: Partially implemented;
DOD‘s comments and our assessment: According to DOD, the quality
assurance function does not yet address process standards and program
performance, nor is it yet an independent function. Further, DOD
subject matter experts continue to be involved in the quality assurance
function.
Recommendation: (4) The Secretary gain control over ongoing IT
investments by establishing a hierarchy of investment review boards,
each responsible and accountable for selecting and controlling
investments that meet defined threshold criteria, and each composed of
the appropriate level of executive representatives, depending on the
threshold criteria, from across the department;
Status of recommendation: Not implemented;
DOD‘s comments and our assessment: DOD has not yet defined and
implemented an approach for selecting and controlling business systems
investments. In March 2004, the Deputy Secretary of Defense signed the
IT portfolio management policy and assigned responsibility to the
domains for IT portfolio management. However, the procedures to be
followed to implement the policy are still under development and no
time frames for completion have been provided. According to the IT
portfolio management policy, the department has 180 days (until mid-
September 2004) to develop these procedures. In addition, the
department has yet to formalize specific roles and responsibilities of
the domains, develop standard criteria for performing the system
reviews, and assign explicit authority for fulfilling roles and
responsibilities. DOD has yet to establish time frames for completing
these activities.
Recommendation: (5) The Secretary gain control over ongoing IT
investments by establishing a standard set of criteria to include (a)
alignment and consistency with the DOD enterprise architecture and;
(b) our open recommendations governing limitations in business systems
investments pending development of the architecture;
Status of recommendation: Not implemented;
DOD‘s comments and our assessment: DOD has not developed standard
criteria for evaluating business systems investments and has not
established time frames for completing this activity.
Recommendation: (6) The Secretary gain control over ongoing IT
investments by directing these boards to immediately apply these
criteria in completing reviews of all ongoing IT investments, and to
not fund investments that do not meet these criteria unless they are
otherwise justified by explicit criteria waivers;
Status of recommendation: Not implemented;
DOD‘s comments and our assessment: In March 2004, the Deputy Secretary
of Defense signed the IT portfolio management policy and assigned
responsibility to the domains for IT portfolio management. However, the
procedures to be followed to implement the policy are still under
development and no time frames for completion have been provided.
According to the IT portfolio management policy, the department has 180
days (until mid- September 2004) to develop these procedures. In
addition, the department has yet to formalize specific roles and
responsibilities of the domains, develop standard criteria for
performing the system reviews, and assign explicit authority for
fulfilling roles and responsibilities. DOD has yet to establish time
frames for completing these activities. Further, the policy also does
not address the issuance of waivers in those instances when exceptions
to the BEA are justified on the basis of documented analysis.
GAO-03-1018: DOD Business Systems Modernization: Important Progress
Made to Develop Business Enterprise Architecture, but Much Work
Remains. September 19, 2003.
Recommendation: (1) The Secretary of Defense or his appropriate
designee;
define and implement an effective investment management process to
proactively identify, control, and obtain DOD Comptroller review and
approval of expenditures for new and ongoing business systems
investments exceeding $1 million while the architecture is being
developed and after it is completed, and which includes clearly defined
domain owners' roles and responsibilities for selecting and controlling
ongoing and planned
system investments;
Status of recommendation: Not implemented;
DOD‘s comments and our assessment: As discussed in this report, DOD has
not yet defined and implemented an effective investment management
process to proactively identify and control system improvements
exceeding $1 million in obligations. DOD officials have acknowledged
that the department does not have a systematic means to identify and
determine which system improvements should be submitted to the
Comptroller for review and, are dependent upon system owners coming
forward to the domains and requesting approval.
Recommendation: (2) The Secretary of Defense or his appropriate
designee implement the core elements in our Enterprise Architecture
Framework for Assessing and Improving Enterprise Architecture
Management that we identify in this report as not satisfied, including
ensuring that minutes of the meetings of the executive body charged
with directing, overseeing, and approving the architecture are prepared
and maintained;
Status of recommendation: Partially implemented;
DOD‘s comments and our assessment: DOD has taken some actions, but
these actions are not sufficient to fully address our previous
concerns. For example, DOD has;
* established the Domain Owners Integration Team;
* begun to establish a configuration management process;
* recently issued an IT portfolio management policy governing the
development, maintenance, and implementation of the BEA;
and;
* developed high-level performance measures that are to be used to
develop the more specific results-oriented performance metrics that are
needed to enable it to evaluate program progress and benefits;
However, as discussed previously, accountability and responsibility
have not been assigned to a committee or group comprised of
representatives from across the DOD component organizations--which
would be responsible for directing, overseeing, and approving the BEA.
DOD plans to update the executive and steering committees' charters to
include these responsibilities in June 2004. Also, the configuration
control board has been tasked with reviewing changes to only some, not
all BEA products (e.g., the board is not responsible for tracking
changes that are being made to the transition plan as part of the BEA
effort). In addition, both the charter and the procedures governing the
configuration management process are still in draft. According to DOD,
the charter is to be approved in June 2004, but no time frame has been
provided for final approval of the procedures. The portfolio management
policy does not address accountability for and approval of updates to
the BEA--as called for by best practices. In addition, the policy does
not address the issuance of waivers in those instances when exceptions
to the BEA are justified on the basis of documented analysis. The
department's current verification and validation contractor is not
independent and is responsible for reviewing the BEA products and only
selected program deliverables.
Recommendation: (3) The Secretary of Defense or his appropriate
designee update version 1.0 of the architecture to include the 340
Joint Financial Management Improvement Program requirements that our
report identified as omitted or not fully addressed;
Status of recommendation: Implemented;
DOD‘s comments and our assessment: [Empty].
Recommendation: (4) The Secretary of Defense or his appropriate
designee update version 1.0 of the architecture to include the 29 key
elements governing the "As Is" architectural content that our report
identified as not being fully satisfied;
Status of recommendation: Not implemented;
DOD‘s comments and our assessment: Of the 29 elements, DOD stated that
it plans to address 26, but it did not have detailed plans on how and
when they would be fully addressed. According to DOD, it is currently
addressing 2 of the 26--system and process inventories--which will be
part of the transition plan. With respect to the system inventory,
DOD's Comptroller recently testified that the actual number of DOD
systems could be twice as many as is currently reported;
2,274. For the 3 elements that DOD did not consider to be relevant to
the program, it did not provide any explicit analysis as to why the 3
would be irrelevant. Until DOD provides such an analysis, we consider
all 29 elements to be critical for the "As Is" architectural content.
Recommendation: (5) The Secretary of Defense or his appropriate
designee update version 1.0 of the business enterprise architecture to
include the 30 key elements governing the "To Be" architectural content
that our report identified as not being fully satisfied;
Status of recommendation: Not implemented;
DOD‘s comments and our assessment: According to DOD, it has partially
addressed 21 elements, but it plans to address all 30. However, DOD did
not provide any detailed plans showing how and when it would fully
address these elements. In addition, the information provided for the
21 elements shows that it had not addressed them but was planning to
address them or had just begun to take needed actions.
Recommendation: (6) The Secretary of Defense or his appropriate
designee update version 1.0 to ensure that "To Be" architecture
artifacts are internally consistent, to include addressing the
inconsistencies described in this report, as well as including user
instructions or guidance for easier architecture navigation and use;
Status of recommendation: Partially implemented;
DOD‘s comments and our assessment: DOD did not have information readily
available to show that it had addressed inconsistencies identified in
our report.
Recommendation: (7) The Secretary of Defense or his appropriate
designee update version 1.0 of the architecture to include (a) the 3
key elements governing the transition plan content that our report
identified as not being fully satisfied and (b) those system
investments that will not become part of the "To Be" architecture,
including time frames for phasing out those systems;
Status of recommendation: Not implemented;
DOD‘s comments and our assessment: According to DOD, the next
transition plan will be available in August 2004. However, DOD did not
have information readily available to show how or when it would fully
address our recommendations.
Recommendation: (8) The Secretary of Defense or his appropriate
designee update version 1.0 of the architecture to address comments
made by the verification and validation contractor;
Status of recommendation: Not implemented;
DOD‘s comments and our assessment: According to DOD, it has addressed
some of its verification and validation contractor's comments and plans
to address the majority of the others. However, DOD did not have
documentation readily available to show that it had addressed some of
the contractor's comments, and was unable to provide plans detailing
how and when it would fully address the remaining comments. According
to DOD, the verification and validation contractor's comments would be
addressed in BEA updates after version 2.0.
Recommendation: (9) The Secretary of Defense or his appropriate
designee develop a well-defined near-term plan for extending and
evolving the architecture and ensure that this plan includes addressing
our recommendations, defining roles and responsibilities of all
stakeholders involved in extending and evolving the architecture,
explaining dependencies among planned activities, and defining measures
of activity progress;
Status of recommendation: Not implemented;
DOD‘s comments and our assessment: As discussed in this report, DOD has
not developed well-defined near-term or long-term plans to be used to
guide day-to-day program activities to enable it to evaluate its
progress.
Recommendation: (10) The Secretary of Defense or his appropriate
designee limit the pilot projects to small, low-cost, low-risk
prototype investments that are intended to provide knowledge needed to
extend and evolve the architecture, and are not to acquire and
implement production version system solutions or to deploy an
operational system capability;
Status of recommendation: Not implemented;
DOD‘s comments and our assessment: The accounting and finance domain
has two ongoing pilot initiatives to acquire and implement production
accounting systems by the end of 2005--at an estimated cost of $135
million. This domain is also currently conducting workshops to develop
needed business rules and requirements for extending and evolving the
BEA in support of DOD's achieving objective of increment one, which
includes achieving an unqualified audit opinion on DOD's fiscal year
2007 consolidated financial statements. However, the department did not
provide detailed plans that showed how or if the workshop results would
serve as the basis for acquiring these production systems.
Source: GAO analysis of DOD data.
[A]DOD DEFINES THE GLOBAL INFORMATION GRID AS THE GLOBALLY
INTERCONNECTED, END-TO-end set of information, capabilities,
associated processes, and personnel for collecting, processing,
storing, disseminating, and managing information on demand to
warfighters, policymakers, and support personnel.
[B]IN MAY 2003, THE DOD COMPTROLLER CHANGED THE ARCHITECTURE NAME FROM
THE Financial Management Enterprise Architecture to the Business
Enterprise Architecture to reflect the transformation of departmentwide
business operations and supporting systems, including accounting and
finance, budget formulation, acquisition, inventory management,
logistics, personnel, and property management systems.
[C]IN SEPTEMBER 2003, DOD ESTABLISHED THE DOMAIN OWNERS INTEGRATION
TEAM, comprised of domain representatives, to facilitate communication
and coordination across the domains.
[D]DOD HAS SIX DEPARTMENTAL DOMAINS, WHICH ARE (1) ACCOUNTING AND
FINANCE, (2) acquisition, (3) human resources management, (4)
installations and environment, (5) logistics, and (6) strategic
planning and budgeting. It also has one enterprise information
environment mission area. The domains and the mission area, comprised
of the Under Secretaries of Defense and the Assistant Secretary of
Defense for Networks and Information Integration/DOD Chief Information
Officer, have authority, responsibility, and accountability within
their domains for business transformation, implementation of the BEA,
development and execution of the transition plan, portfolio management,
and establishment of a structure to ensure representation of the DOD
components and the appropriate federal agencies.
[End of table]
Enclosure IV:
Comments from the Department of Defense:
UNDER SECRETARY OF DEFENSE
1100 DEFENSE PENTAGON
WASHINGTON, DC 20301-1100:
MAY 10 2004
COMPTROLLER:
Mr. Gregory Kutz Director:
Financial Management and Assurance
United States General Accounting Office
Washington, DC 20548:
Dear Mr. Kutz:
Enclosed is the proposed response by the Department of Defense (DoD) to
the General Accounting Office (GAO) Draft Report, "DoD Business Systems
Modernization: Limited Progress in Development of Business Enterprise
Architecture and Oversight of Information Technology Investments,"
(GAO-04-731R), dated April 30, 2004.
The referenced draft General Accounting Office (GAO) report accurately
describes the size and scope of the Business Management Modernization
Program (BMMP). As stated in the opening paragraphs of the draft
report, "DoD is one of the largest and most complex
organizations in the world, overhauling its business operations and
supporting systems represents a huge management challenge."
Furthermore, the draft report pointedly describes the adverse effect of
long-standing business system problems, which have resulted in "a lack
of adequate transparency and appropriate accountability across all
major business areas.":
The DoD agrees with that assessment and concurs with the urgent need to
undertake the largest ongoing transformation in the public sector.
Since July 2001, we have been actively engaged in efforts to correct
these long-standing problems. However, GAO's draft report characterizes
DoD's progress so far as "limited." Though progress over the past 3
years admittedly has been slower than either GAO or DoD would prefer,
it has been significant, despite appearances to the contrary. The
report also underestimates development methodology of the Business
Enterprise Architecture (BEA), as well as the significant growth in
program maturity. The Department remains committed to working with GAO
to address these issues.
My point of contact for this matter is Dr. Paul Tibbits, Director for
Business Modernization and Systems Integration (BMSI). Dr. Tibbits may
be contacted by email: Paul.Tibbits@osd.mil or by telephone at (703)
607-3370.
Sincerely,
Signed by:
Lawrence J. Lanzillotta:
Acting
Enclosure: As stated:
Department of Defense (DoD) Response to General Accounting Office (GAO)
Draft Report, "DoD Business Systems Modernization: Limited Progress in
Development of Business Enterprise Architecture and Oversight of
Information Technology Investments" (GAO-04-731R):
DoD Response: The referenced draft General Accounting Office (GAO)
report accurately describes the size and scope of the Business
Management Modernization Program (BMMP). As stated in the opening
paragraphs of the draft report, "...DoD is one of the largest and most
complex organizations in the world, overhauling its business operations
and supporting systems represents a huge management challenge."
Furthermore, the draft report pointedly describes the adverse effect of
long-standing business system problems, which have resulted in "a lack
of adequate transparency and appropriate accountability across all
major business areas." The Department of Defense (DoD) agrees with that
assessment and concurs with the urgent need to undertake the largest
ongoing transformation in the public sector. Since July, 2001, we have
been actively engaged in efforts to correct these long-standing
problems. However, GAO's draft report characterizes DoD' s progress so
far as "limited." Though progress over the past 3 years admittedly has
been slower than either GAO or DoD would prefer, it has been
significant, despite appearances to the contrary. The report also
underestimates development methodology of the Business Enterprise
Architecture (BEA), as well as the significant growth in program
maturity.
GAO's report states that DoD has little to show for its three-year
effort and $203 million investment. However, GAO's assessment does not
adequately consider or incorporate in their report important completed
and ongoing activities, plans, and documentation. In addition to the
scope and complexity of the BMMP, DoD must solve difficult, persistent,
and tenacious culture, management and partnership barriers to defining
and using the BEA to drive business transformation, conditions for
which the GAO report seems to offer no allowances, although it
certainly acknowledges them.
The draft report asserts that GAO has not observed significant change
since their last review, which seems improbable considering the level
of effort expended during the past year. However, it is possible that
GAO's observations overlooked ongoing architecture development
activity or governance deliberations and therefore did not recognize
relevant improvements in their report. For example, the report should
highlight the significance of the Information Technology (IT) Portfolio
Management policy memorandum issued March 22, 2004. This document lays
the foundation of the Department's portfolio management strategy and
will be supplemented by the IT Portfolio Management Instruction,
currently under development, and other management initiatives.
In 2003-2004, DoD Chief Financial Officer (CFO) and Chief Information
Officer (CIO) collaborated on the DoD portfolio management policy. The
BMMP and its associated governance also enabled development of a
comprehensive directive to leverage the BEA and portfolio management
for near term IT investment decisions.
The DoD IT budget request for FY05 totals $29B. This budget includes
business applications, warfighting systems, shared infrastructure,
information assurance and related
technical activities. The business applications portion of the DoD IT
budget totals $5.0313 in FY05, which includes BMMP systems and IT
activities that support business functions of the Department.
Warfighting systems account for $7.813, shared infrastructure accounts
for $12.813, and Information Assurance and related technical activities
account for $3.113. DoD Domains clearly have responsibility for
developing and implementing business transition plans and portfolios
that will assure that the $4.813 annual business system expenditures
cited in the referenced GAO report are focused on the established
business transformation goals of the Department.
DoD's proposed Management Initiative is a key document that lays out
specific Domain responsibility and authority for DoD business
transformation. It is expected to be released in May 2004. The GAO
report does not it recognize the magnitude of the BMMP governance
accomplishments. For the first time in its history, DOD's business
executives are taking an integrated view of the business environment
and making collaborative decisions based on an enterprise view. In
addition, a major accomplishment and strong indicator of DoD
architecture management maturity is the series of completed and ongoing
Domain-led workshops and the subsequent collaborative Domain-led
Architecture Integration Team (AIT) sessions.
The portfolio management policy (Attachment 1), draft portfolio
management instruction and other related documents, provide clear
direction and accountability for management and oversight of IT
investment decisions. This accountability structure, organized along
functional business owner lines, is the key mechanism for achieving DoD
investment control frequently called for in GAO and other audit and
management reports. These policy documents will provide the Department
the visibility and information necessary to make informed investment
decisions.
The Department's BEA, containing the baseline "blueprint" of the
Department's vast, worldwide business activities (e.g., business
policies and rules, business process controls, information flows, and
business operational data collection and use), is one of the largest,
most complex, and most pervasive business architectures developed to
date, in either the public or private sectors. The DoD progress in the
past three years exceeds any experience in the private sector and
government.
There is no proven roadmap to guide our actions thus far. Despite the
absence of a direct precedent or example to follow, DoD continues to
take full advantage of guidelines, lessons learned and consultant
expertise in addition to the guidance provided by GAO.
BEA 5coue and Detail:
The BEA 1.1 Release provided the baseline and also illustrated the need
to approach architecture extensions in an incremental manner on
creating additional detail. As such, DoD is deliberately delivering
these extensions in increments, each scoped to take approximately a
year to define in the BEA. DoD identified and approved the focus areas
of the additional two increments (Two and Three) at the Domain Owner
Integration Team (DO/IT) meeting on March 22, 2004 and the BMMP
Steering Committee meeting on March 25, 2004 (Attachments 2 and 3). The
scope of increment one had already been approved.
* Increment One: Unqualified audit opinion (UAO), asset accountability
and total force visibility.
* Increment Two: Acquisition best practices, total asset visibility,
improved force management, improved military health care, and improved
environmental safety and occupational health.
* Increment Three: Improved planning, programming, budgeting and
execution, integrated total force management, and improved force
management.
The scope for each increment includes the processes, information,
requirements and business rules pertinent to accomplishing each of the
major focus areas. For the BEA, the Executive Summary Concept of
Operations (Attachment 4) defines the needed level of detail: "The BEA
shall depict architecture products in sufficient detail to provide an
unambiguous interpretation of cross-Domain business transactions".
Plans to Extend and Evolve BEA:
The BEA is an integrated plan for Departmental business transformation.
It includes work products consisting of models, diagrams, tables, and
narratives, all of which translate DoD's business activities into
meaningful representations of business processes with comprehensive
integration of relevant rules and regulations. These include the
initial all views (AV), system views (SV), technical views (TV) and
operational views (OV) of the business processes of the six key DoD
business Domains. The Domains encompass the following business areas;
Accounting and Finance, Acquisition, Human Resource Management,
Installations and Environment, Logistics, and Strategic Planning and
Budget. For the first time in the history of DoD, each Domain Owner is
now serving as the single authoritative source of interpretation of
rules and regulations for the Department in each of their respective
areas of cognizance. In addition to the Business Domains, the Assistant
Secretary of Defense (Networks and Information Integration)/Chief
Information Officer - ASD(NII)/CIO - is the leader of the Enterprise
Information Environment (EIE) Mission Area. This Mission Area is
composed of equipment, software and common information capabilities or
services for Enterprise use.
In the Fall of 2003, the BEA was extended to include the Reference
Business Process Model (RBPM), which proposed a desired condition for
the enterprise-level business processes for BEA 2.0. This Release was
published in February 2004.
For Increment One, all Domains conducted preparatory workshops to
define supporting business processes more fully. From October 2003 to
January 2004, the Accounting and Finance Domain conducted 26 workshops
(Attachment 5) to specifically analyze DoD business processes and
associated financial management requirements. These formed the basis
for extended analysis by the increment one Architecture Integration
Team (AITs), which focus on the following process categories:
* Team #1: Apportionment:
* Team #2: Receipts & Acceptance:
* Team #3: Commitments & Obligations:
* Team #4: Receivables, Payables & Disbursements:
* Team #5: Asset Accountability:
* Team #6: Purchase Card/Travel Card:
Each of the AITs is comprised of 8 to 21 team members consisting of
Domain Action Officers (DAOs) serving as a Team Lead, domain Subject
Matter Experts (SMEs) and Business Modernization Systems Integration
(BMSI) staff supporting the teams with process modelers, data modelers,
requirements analysts and accountants. Although they did not take
advantage of the DoD offer, GAO was invited to participate in any and
all the AIT sessions to witness the significance and depth of this
collaborative work. The AITs are key to assuring cross domain
integration, unambiguous interpretation of requirements across the
Department, and coherent representation of end-to-end processes that
will serve as the roadmap for subsequent IT investment and system
configuration decisions.
As described in the AIT methodology, DoD is developing the Enterprise
Business Process Model (EBPM), and end-to-end descriptions of DoD
business processes. This model will accurately identify financially
relevant requirements and business rules necessary to resolve all known
material weaknesses to support an Unqualified Audit Opinion and asset
accountability. These rules and requirements, along with their
associated data objects, are being linked to DoD business processes.
The EBPM will update the RBPM previously published in BEA Release 2.0.
The initial EBPM (in HTML) and updated Overview and Summary (AV-1) were
released on April 30, 2004 (BEA 2. 1) as scheduled and approved. This
Release represents a significant improvement of the BEA and can be
found on the BMMP Portal (portal navigation instructions are at
Attachment 6).
The near term schedules for completing Release 2.2 are shown in
Attachments 7 and 8. The EBPM defines the complete end-to-end process
of doing business in the Department, from planning to disposal. The
activity model developed for the baseline BEA will be aligned with the
enterprise processes (EBPM) to ensure architectural completeness, and
to facilitate its use. BEA Release 2.1 Overview and Summary (AV-1,
Attachment 9) explains that the content of BEA Release 2.2, scheduled
for July 2004, will support IT investment decisions for the FY06
Budget. Release 2.3 (October 2004) and 2.4 (January 2005) will augment
BEA 2.2 for enhanced portfolio management acquisition decision support
and business process re-engineering. Attachment 10 maps architecture
products to business utility and will serve to prioritize and focus
development of BEA products.
In BEA Release 2.2, DoD will concentrate on completeness and
correctness of the requirements repository, business rules, the
enterprise business process model, the conceptual data model, and
system functions, entities and information exchanges related to
Increment One. Release 2.3 will add the relevant realigned activity
model, and updated organizational nodes, roles, information and
activities.
In summary, since the publication of BEA Release 1.0 in May 2003, the
Department has:
* Completed an initial Transition Plan that identifies BMMP concept
strategy and delineates steps for future business transformation.
* Established the enterprise-wide DoD Portfolio Management policy.
Page 4 of 10:
* Established a DoD-wide BMMP Governance Structure, which defines
program strategies, performs strategic decision-making, and provides
oversight and control of portfolio management responsibilities with
respect to IT business system investments.
* Established an incremental
approach for extending the BEA into implementable parts that will drive
transformed business operations into DoD. BMMP Increments One, Two, and
Three have been approved by the BMMP Steering and Executive Committees.
* Established initial BMMP and BMSI program office performance metrics
(i.e., goals, objectives, measures and targets), and is near completion
of an integrated set of enterprise and domain business transformation
metrics. DoD plans to begin data collection and reporting in the 4`h
Quarter of FY04.
* Established the initial inventory of business IT systems.
* Identified the template and program entries to instruct the
implementation of a Standard General Ledger (SGL).
* Developed a schedule for completion of initial business process
reengincering and modeling for integration with the next extension of
the BEA.
Independent Verification and Validation Contractor:
GAO states that "DOD does not have an independent verification and
validation function to review the architecture products and management
processes. The department's current verification and validation
contractor is not independent and only reviews selected architecture
products." The current independent verification and validation (IV&V)
contractor is a neutral third party, a Federally Funded Research and
Development Center (FFRDC), free of organizational conflict of interest
(OCI) issues. The contractor is not directly involved with the BEA or
the Transition Plan development; hence they provide the requisite
independence. Under the recently revised BMSI organization structure,
the 1V&V contractor is not within the BEA or Transition Planning chain
of command. IV&V contractor activities are included in a separate
directorate with Quality Assurance and Risk Management. To assure
independence, reports of the IV&V contractor are provided directly to
GAO and the Steering Committees. (The list of IV&V items reviewed since
April 2003 and provided to GAO is included in Attachment 11):
The IV&V contractor provides input to evaluation criteria for major
deliverables and reviews the significant architecture-related
documentation developed by the BMSI office. These include; architecture
methodology, requirements management methodology, BEA Releases and
other significant documents. The BMSI office is in the process of
extending the IV&V support through a competitive acquisition process.
The contract award is anticipated by June 2004.
GAO and Verification and Validation Contractor Comments:
Attachment 12 is the verification and validation contractor's report on
BEA Release 2.0 as submitted to DoD and GAO. It supports DoD statements
that some of the contractor's recommendations were addressed in Release
2.0. Specific comments are on pages 17, 18, 19,
21, 22, and 23. DoD expects Release 2.2 and 2.3 of the architecture to
address the remaining IV&V comments. Attachment 13 is a matrix showing
consolidated comments mapped to disposition.
DoD is also addressing the 61 content elements identified by GAO.
Attachment 14 shows the evaluation criteria for the detailed action
plan for developing AS-IS BEA products. This action plan, scheduled for
completion June 28, 2004, will provide the timeline to reconcile and
address GAO content elements. Status for addressing the TO-BE
architecture products is at Attachment 15. Attachment 16, BEA
Information Assurance (IA) Guidance, illustrates DoD's plans to address
IA and Security and Attachment 17 contains the BEA Configuration
Management (CM) Plan as approved and implemented on January 28, 2004.
To ensure that BEA considers and implements all of the GAO comments
concerning the maturity of the architecture, DoD is using the GAO
Framework for Enterprise Architecture Management Maturity to measure
BMSI performance, as briefed to the BMMP Steering Committee.
DoD will measure BMSI performance relative to the quality of BEA
content, and its alignment with the Federal Enterprise Architecture
(FEA), using the Office of Management and Budget (OMB) Guidelines for
Enterprise Architecture Assessment Framework. To ensure consistent
mapping to FEA, DoD has developed a set of DoD enterprise reference
models (see Attachment 18) that map to each of the FEA reference
models.
BMMP Governance Structure:
The Department has established a BMMP Governance structure that
institutionalizes the participation of senior civilian and military
leaders from the Department's key business areas. We have created, and
are using, a hierarchy of governance committees to address and resolve
business transformation issues in the most efficient manner, at the
lowest level possible.
Through the Program's Executive Committee, the Secretary's senior
political leadership provides strategic direction and guidance. The
BMMP Executive Committee and Steering Committee are jointly chaired by
the DoD CFO and DoD CIO, and include representatives from both the
Office of the Secretary of Defense and the Military Services. The
Domain Owners Integration Team (DO/IT) are responsible for business
transformation by establishing and maintaining intra-Domain governance
processes, managing domain-specific Information Technology (IT)
investment portfolios, ensuring systems compliance with the BEA, and
assisting in the extension of the enterprise architecture..
The Governance Structure, at the appropriate level, has actively
engaged in developing the Department's enterprise level Portfolio
Management policies, and now is developing the guidance to the DoD
Components regarding the submission of IT exhibits for the FY06 budget.
With the release of the Department's Management Initiative, expected in
May 2004, greater detail regarding roles and responsibilities will be
explicitly articulated and Domain Owners will be provided additional
direction to manage DoD IT investments as portfolios. DOD's Management
Initiative will establish specific time frames (30 days from signature
date) for the issuance of initial Portfolio Management guidance, to
include the criteria for performing reviews of proposed IT investments.
Architectural compliance is noted as one of the criteria for decisions
on what investments to make, and to modify or terminate a system in the
recently issued Information Technology
Portfolio Management policy memorandum. This policy memorandum states
that the Under Secretary of Defense (Comptroller)(USD(C))/CFO is
responsible for developing and maintaining the DoD BEA as a component
of the overarching Global Information Grid (GIG) architecture. The
USD(C) is therefore accountable for the approval of updates to the BEA,
as called for by best practices.
The issue of waivers is addressed in DoD Directive 8100.1, "Global
Information Grid (GIG) Overarching Policy", dated September 19, 2002.
This directive is referenced in the Information Technology Portfolio
Management policy memorandum. DoD Directive 8100.1 states that the DoD
Chief Information Officer is responsible for establishing GIG
compliance and enforcement mechanisms to minimize duplication of IT.
The BEA Conops (Attachment 4) clarifies that the BEA is a component of
the GIG and, as such, the GIG waiver policy applies to the BEA as well.
Timeframes have been established for the development and implementation
of a DoD directive and instruction on IT Portfolio Management. The IT
Portfolio Management policy memorandum was signed by the Deputy
Secretary of Defense, March 22, 2004. This policy memorandum will be
issued as a directive within 180 days (mid-September 2004). An
accompanying IT Portfolio Management Instruction is under development,
and as stated earlier, will be issued 30 days from signature of the
Management Initiative. Coordination of this instruction will begin in
June 2004, with a scheduled completion date of December 2004, The
Department is not waiting for the publication of these documents to
begin the review process. The Steering Committee completed its first
round of IT Portfolio Management reviews for all Domains in March 2004.
Decisions from these and subsequent reviews this Summer and Fall will
guide the FY06 budget preparation process.
The Domains have a strong oversight responsibility for guiding
investment in IT. However, the GAO has recommended that funds for DoD
business systems be appropriated directly to the Domains in order to
provide for accountability, transparency, and the ability to prevent
"...a continued parochial approach to systems investments that exists
today.":
The Department does not concur with this recommendation, and does not
envision a governance structure in which Domain owners are assigned the
management responsibility for the planning, design, acquisition,
deployment, operation, maintenance, and modernization of individual DoD
business systems. Similarly, DoD does not envision a governance or
program management strategy in which funds are appropriated to Domain
leaders rather than the military service and Defense agencies to
operate, maintain, and modernize DoD business systems. The Department
does not consider this recommendation to be advisable for several
reasons.
The GAO's recommendation appears to have been made without
consideration of the Department's long standing recognition of the need
for operational expertise and perspective in the development of
business IT systems. The recommendation was made without regard for
collaborative consultation before advancing a concept that is radically
different from the portfolio management concepts being implemented in
DoD today. Finally, funds management by the Domains would require
creation of a completely new layer of bureaucracy to assure proper
management of appropriated funds.
Beginning with the FY06 DoD program budget development process, the
Domains will be acting as an Investment Review Board, and will provide
strong management oversight to the Department's business IT investment
decisions. Therefore the Domains will oversee, coordinate and guide the
business system investment decisions made by the DOD Components. IT
investments will be based on Domain approval. The DoD demonstrated in
last year's Program Review how the Logistics Domain realigned resources
from lower priority efforts into programs that better support the BEA.
In this case, the DoD moved $121M within the Army, $87M within the
Navy, and the $139M within the DLA - clearly a successful demonstration
of portfolio management. We are expanding that effort by working with
all the Domains to identify opportunities for similar realignments in
the FY06 budget.
DOD has the strongest requirements, acquisition and budgeting processes
of any Federal Agency. Our efforts to govern BMMP within these existing
processes, suggests that we will have an even more robust oversight
process for controlling portfolio investments. All Major
Automated Information Systems (MAIS) programs are subjected to the same
acquisition oversight as Major Defense Acquisition Programs (MDAP),
including adherence to Clinger-Cohen Act requirements. IT Capital
Planning and Investment Control have been instituted in DoD through
longstanding processes and executive-level forums. The Department uses
the Planning, Programming, Budgeting, and Execution System and the
Defense Acquisition System to select, manage, and evaluate IT
investments. DoD uses these existing management processes in lieu of
creating parallel processes to manage IT investments - ensuring the
integration of IT investments into the total DoD investment portfolio
in compliance with the BEA.
Several decision-making forums govern the movement of proposed IT
investments from one phase to the next, and ultimately determine
whether programs will be approved for funding. These bodies include the
Defense Resources Board, Defense Acquisition Board, IT Overarching
Integration Product Teams (OIPT), the DoD CIO Executive Board, and the
GIG Architecture Integration Panel. The roles of these bodies include
ensuring proposed IT investments support core missions, allocating
resources, overseeing and controlling IT investments, and overseeing
the development and configuration management of integrated
architectures.
Systems Review and Legislative Compliance:
The Department acknowledges that a more proactive approach is required
to ensure certification compliance with the legislative requirement of
the FY03 National Defense Authorization Act. To that end, we are
currently finalizing and issuing guidance that will significantly
improve the process by clearly delineating roles and responsibilities,
establishing time frames and making key system modifications to better
align budgets with actual IT expenditures. The guidance directs the
Business Domains to develop and submit business 1T system certification
compliance plans to the USD(C) within thirty days of issuance of
Management Initiative. The plan must include a schedule to accomplish
the reviews for all the systems comprising at least 80 percent of the
total business system funding by August 23, 2004. To facilitate
development of the plans the Department will use the Information
Technology Application (ITMA) and system repositories of the
Components, which serve as key components
in developing the Department's IT budget requirements, as the primary
source to identify Business IT systems needing USD(C) certification.
The completion of these reviews by Fall 2004 is required so that
Domains will have the opportunity to review component budget
submissions during the Program/Budget review cycle and make necessary
funding adjustments to the FY06 budget.
Both the BEA and the Business IT investment governance structure depend
significantly on the business system portfolio management process being
established within the Department's key business Domains. Beginning
with the FY06 budget development process, the Domains will provide
strong management oversight to the Department's near-term business
system investment decisions. Additionally, the Department has developed
standard architecture criteria by which the business systems can be
evaluated, to ensure they are being developed and managed in accordance
with BMMP. These criteria will be enhanced as the BEA matures.
Metrics:
The Department understands the importance of establishing meaningful,
tangible, and measurable program goals and objectives to guide and
measure the value of business transformation investments. The
establishment of a comprehensive performance measurement process for
the transformation of DoD business operations, the largest business
operations in the world, takes substantial time, resources, and
extensive cross-organizational coordination. BMMP has made substantial
progress establishing metrics (i.e., goals, objectives, measures, and
targets) to include business transformation metrics and BMSI Program
Management Office management metrics.
As approved by the Steering Committee and the DOAT, DoD is nearing
completion of an initial integrated set of enterprise and domain
business transformation metrics and plans to begin data collection and
reporting in 4 Quarter FY04. BMMP expects the integrated set of metrics
to encompass approximately 16 to 20 goals, 30 to 50 objectives, and 90
to 120 measures. These metrics reflect a major accomplishment because
they provide the basis for demonstrating the alignment of business
transformation investments with strategic outcomes.
Additionally, BMSI has established Architecture Integration Team (AIT)
performance metrics that track the number of requirements dispositioned
relative to the total number of requirements and the percent of process
steps reviewed relative to the total number of process steps. BMSI
tasking to contractors is quite clear, serving as the basis for use of
rigorous Earned Value Management metrics for BEA development. BMSI is
establishing an Architecture and Program Management Maturity Plan and
supporting metrics to address the quality, content, and utility of the
BEA. BMSI is establishing a balanced scorecard (BSC) and supporting
metrics focused on the organization's internal management processes.
Contracting Issues:
Under the General Services Administration (GSA) schedule through which
the BEA Blanket Purchase Agreement (BPA) was awarded, only two types of
contracts are permitted,
Time and Materials (T&M) and Firm Fixed Price (FFP). Due to the
complexity and evolutionary nature of the BEA development effort, it is
not prudent acquisition strategy to use a firm, fixed price vehicle.
Maintaining, extending and integrating the architecture can be
accomplished in a variety of ways using many different approaches, such
as the use of AITs.
The Government and the prime contractor establish acceptance criteria
prior to work beginning on each deliverable. The Government uses the
established criteria to evaluate the deliverables, after providing a
clear understanding of the work expectations. Government personnel
monitor the progress on a daily basis.
In addition, the Government holds weekly reviews with the Contractor to
monitor deliverables, performance (Earned Value Management System-
EVMS). The attached work statement for Call 0009 (Attachment 19)
includes additional EVMS requirements the Government is instituting to
more closely monitor performance.
Pilot Initiatives:
The Accounting and Finance Domain is serving as the program advocate
and oversight entity, to assure compliance with the BEA, in accordance
with the responsibilities assigned to the Domains, for two Component
acquisitions of Joint Financial Management Improvement
Program (JFMIP)-compliant financial management systems (Defense
Enterprise Accounting and Management System (DEAMS) and General Fund
Enterprise Business System (GFEBS)) to support the Department's general
and working capital fund activities. These initiatives are being funded
by the individual Components, not the Accounting and Finance Domain.
The following are GAO's comments on the Department of Defense's (DOD)
letter dated May 10, 2004.
GAO Comments:
See the "Agency Comments and Our Evaluation" section of this report.
We have reviewed multiple versions of the architecture development
methodology, including the latest version, which is currently draft.
Based on our review of the latest draft, we concluded that this
methodology does not provide a documented approach for performing
activities in a coherent, consistent, accountable, and repeatable
manner. Key DOD stakeholders also expressed concerns about the
methodology. For example, these stakeholders stated that it is an
"after-the-fact" description of work products rather than a
prescriptive document that details the approach to be followed to
develop the architecture.
The descriptions of increments 2 and 3 in DOD's comment letter are
different than the descriptions the program manager provided to us on
April 27, 2004, as reflected in this report. Because of the
inconsistencies in the responses, it remains unclear what the
department's focus is for increments 2 and 3. DOD's response also
stated that the executive committee approved all three increments;
however, DOD did not provide evidence of this approval.
As stated in our report, DOD's objective for achieving an unqualified
audit opinion is not supported by a DOD-wide plan of action nor are the
individual component plans linked to program activities.
As stated in our report, DOD does not expect to complete the next
version of its transition plan until August 2004. In addition, as we
previously reported,[Footnote 41] its initial transition plan was
basically a plan to develop a transition plan and did not possess the
attributes needed to guide its transformation efforts.
Attachment 11 does not list the deliverables the verification and
validation contractor has reviewed since April 2003.
The matrix does not show the anticipated disposition (e.g., date and
architecture version) of the verification and validation contractor's
comments.
This information was provided after fieldwork was completed.
These briefings were presentations given by each domain describing its
approach for conducting portfolio management reviews. The steering
committee did not review actual systems investments nor was there
detailed discussion about the domains' approaches.
The scope of our review did not include an evaluation of the
realignment of funds made by the logistics domain to determine an
effective demonstration of portfolio management.
DOD's characterization of its current process for oversight and control
of modernization investments is inaccurate. We along with the DOD
Inspector General continue to report on significant weaknesses in the
department's requirements, acquisition, and budgeting processes for
business systems modernization projects. For example, we
reported[Footnote 42] that estimated costs for the Defense Procurement
Payment System had increased by $274 million and the schedule had
slipped by almost 4 years after having already invested 7 years and
over $126 million. DOD Comptroller's noted that the project was being
terminated due to poor program performance and increased costs.
We look forward to receiving DOD's forthcoming guidance to contracting
and program officials that describes the process by which these
officials obtain DOD Comptroller's statutorily required determination
before making obligations exceeding $1 million for a financial system
improvement. However, given the obligation data provided to us by DOD
and included in our report, we are concerned that DOD plans no action
to review previous obligations for system improvements to ensure that
there have been no such violations since the enactment of the
limitation.
We continue to believe that contractor tasks have not been clearly
defined and that, because DOD is acquiring services on the basis of
time and materials, it is essential that the department use efficient
methods and effective cost controls to measure the work being
performed. While we did not take issue with DOD on its use of a time
and materials contract, we did state that DOD's failure to execute the
required Determination and Finding to evaluate the cost and performance
risks assumed by the government illustrates an overall lack of clear
architecture development plans.
According to a program official, DOD did not begin requiring acceptance
criteria prior to the contractor beginning work until August 2003.
Further, DOD has paid the contractor under the time and materials
contract even when the contractor did not meet all of the criteria for
the deliverable. For example, based on reports provided by DOD, the
department paid the contractor for delivery of version 2.0 of the
architecture even though this version did not meet 50 percent of the
acceptance criteria.
Enclosure V:
GAO Contacts and Staff Acknowledgments:
GAO Contacts Jenniffer Wilson, (202) 512-9192:
Cynthia Jackson, (202) 512-5086:
Acknowledgments In addition to the individuals named above, key
contributors to this report included Beatrice Alff, Francine
DelVecchio, Abe Dymond, Joanne Fiorino, Neelaxi Lakhmani, J.
Christopher Martin, Mai Nguyen, Michael Peacock, David Plocher,
Katherine Schirano, Darby Smith, and Bernard Trescavage.
(192114):
GAO-03-465.
FOOTNOTES
[1] U.S. General Accounting Office, Information Technology:
Architecture Needed to Guide Modernization of DOD's Financial
Operations, GAO-01-525 (Washington, D.C.: May 17, 2001).
[2] The Business Management Modernization Program is the department's
business transformation initiative; it encompasses defense policies,
processes, people, and systems that guide, perform, or support all
aspects of business management--including development and
implementation of the business enterprise architecture. The Under
Secretary of Defense (Comptroller) established a DOD-wide program
management office called Business Modernization and Systems Integration
(BMSI), to oversee and manage the program.
[3] Business systems include financial and nonfinancial systems, such
as civilian personnel, finance, health, logistics, military personnel,
procurement, and transportation, with the common element being the
generation or use of financial data to support DOD‘s business
operations.
[4] Bob Stump National Defense Authorization Act for Fiscal Year 2003,
Pub. L. No. 107-314, § 1004, 116 Stat. 2458, 2629 (Dec. 2, 2002).
[5] In May 2003, the DOD Comptroller changed the architecture name from
the Financial Management Enterprise Architecture to the Business
Enterprise Architecture to reflect the transformation of departmentwide
business operations and supporting systems, including accounting and
finance, budget formulation, acquisition, inventory management,
logistics, personnel, and property management systems.
[6] U.S. General Accounting Office, DOD Business Systems Modernization:
Summary of GAO's Assessment of the Department of Defense's Initial
Business Enterprise Architecture, GAO-03-877R (Washington, D.C.: July
7, 2003).
[7] U.S. General Accounting Office, DOD Business Systems Modernization:
Important Progress Made to Develop Business Enterprise Architecture,
but Much Work Remains, GAO-03-1018 (Washington, D.C.: Sept. 19, 2003).
[8] GAO-03-1018.
[9] The steering committee, made up of senior leaders from across the
department, is advisory in nature and is not accountable for directing,
overseeing, and approving the architecture.
[10] DOD has six departmental domains, which are (1) accounting and
finance, (2) acquisition, (3) human resources management, (4)
installations and environment, (5) logistics, and (6) strategic
planning and budgeting. It also has one enterprise information
environment mission area. The domains and the mission area, comprised
of the Under Secretaries of Defense and the Assistant Secretary of
Defense for Networks and Information Integration/DOD Chief Information
Officer, have authority, responsibility, and accountability within
their domains for business transformation, implementation of the
architecture, development and execution of the transition plan,
portfolio management, and establishment of a structure to ensure
representation of the DOD components and the appropriate federal
agencies.
[11] U.S. General Accounting Office, Information Technology: A
Framework for Assessing and Improving Enterprise Architecture
Management (Version 1.1), GAO-03-584G (Washington, D.C.: April 2003).
[12] GAO-03-1018.
[13] GAO-01-525.
[14] U.S. General Accounting Office, DOD Business Systems
Modernization: Improvements to Enterprise Architecture Development and
Implementation Efforts Needed, GAO-03-458 (Washington, D.C.: Feb. 28,
2003).
[15] GAO-03-1018.
[16] GAO-01-525; U.S. General Accounting Office, DOD Financial
Management: Important Steps Underway But Reform Will Require a Long-
term Commitment, GAO-02-784T (Washington, D.C.: June 4, 2002); and U.S.
General Accounting Office, Department of Defense: Further Actions
Needed to Establish and Implement a Framework for Successful Business
Transformation, GAO-04-626T (Washington, D.C.: Mar. 31, 2004).
[17] GAO-01-525.
[18] GAO-03-458.
[19] U.S. General Accounting Office, Information Technology:
Observations on Department of Defense's Draft Enterprise Architecture,
GAO-03-571R (Washington, D.C.: Mar. 28, 2003).
[20] GAO-03-877R and GAO-03-1018.
[21] GAO-03-1018.
[22] GAO-01-525.
[23] GAO-03-458.
[24] GAO-03-584G.
[25] Best practices recommend that the verification and validation
function be independent of the architecture program and report directly
to the steering committee.
[26] JFMIP requirements arise from various public laws, regulations,
bulletins, circulars, federal accounting standards, and leading
practices and are applicable governmentwide.
[27] The focus of this review was limited to the "As Is" and "To Be"
architecture products, because the transition plan has not been
updated.
[28] We also found that in issuing task orders to International
Business Machines (IBM) pursuant to the General Services Administration
contract, DOD did not execute a Determination and Finding to support
its use of Time-and-Materials task orders, as required by Federal
Acquisition Regulation 16.601, 48 C.F.R. § 16.601 (2003). This
Determination and Finding requirement guides agencies in evaluating the
cost and performance risks assumed by the government in awarding Time-
and-Materials contracts compared to other types of contracts that may
provide increased incentives for the contractor to control costs and
efficiency.
[29] Subsection 1004(d) of the Bob Stump National Defense Authorization
Act for Fiscal Year 2003, Pub. L. No. 107-314, 116 Stat. 2630, (Dec. 2,
2002), provides that any amount in excess of $1 million may be
obligated for Defense financial system improvements before approval of
its enterprise architecture and a supporting transition plan only if
the DOD Comptroller makes a determination that the improvement is
necessary for (1) critical national security capability or critical
safety and security requirements or (2) prevention of significant
adverse effect on a project that is needed to achieve an essential
capability. The act further provides that after the architecture and
transition plan are approved, the DOD Comptroller must determine,
before making obligations that exceed $1 million for system
improvements, that such improvements are consistent with the enterprise
architecture and the transition plan.
[30] GAO-01-525 and GAO-03-458.
[31] We requested the obligational data for fiscal year 2003 for the
period December 2, 2002, the date of enactment of the act, through
September 2003.
[32] GAO-04-626T.
[33] U.S. General Accounting Office, DOD Business Systems
Modernization: Continued Investment in Key Accounting Systems Needs to
be Justified, GAO-03-465 (Washington, D.C.: Mar. 28, 2003).
[34] Wide Area Workflow is a secure web-based system for electronic
invoicing, receipt, and acceptance.
[35] GAO-03-465.
[36] GAO-04-626T.
[37] U.S. General Accounting Office, Department of Defense: Further
Actions Needed to Establish and Implement a Framework for Successful
Financial and Business Management Transformation, GAO-04-551T
(Washington, D.C.: Mar. 23, 2004) and GAO-04-626T.
[38] GAO-03-458.
[39] GAO-04-551T.
[40] U.S. General Accounting Office, Information Technology: A
Framework for Assessing and Improving Enterprise Architecture
Management (Version 1.1), GAO-03-584G (Washington, D.C.: April 2003).
[41] External requirements are those that are obtained from
authoritative sources, such as the Joint Financial Management
Improvement Program, and constrain various aspects of the architecture.
[42] GAO-03-1018.
[43] GAO-03-465.
