Information Technology
Agriculture Needs to Strengthen Management Practices for Stabilizing and Modernizing Its Farm Program Delivery Systems Gao ID: GAO-08-657 May 16, 2008The U.S. Department of Agriculture (USDA) has experienced significant problems with its information technology systems that support the delivery of benefits programs to farmers. In October 2006, these systems began experiencing considerable delays while attempting to process a large number of transactions, and by January 2007, the systems became inoperable for 1 month. In response to these issues, USDA developed a near-term stabilization plan and long-term plans to modernize its delivery of these programs. GAO was asked to determine (1) the extent to which USDA's stabilization plan addresses key management issues, including consistently tracking reported problems, establishing performance metrics and goals, and defining roles and responsibilities and (2) the adequacy of USDA's assessment of existing product capabilities, as well as cost and schedule estimates for its new, long-term modernization investment. To address these objectives, GAO, among other things, compared USDA's plans with industry best practices. On March 25, 2008, GAO briefed the requesters' staff on the results of this review.
USDA's near-term plan to stabilize the agency's farm program delivery systems focused on technical issues such as expanding telecommunication capacity and acquiring a means for disaster backup and recovery; however, it did not address key managerial issues such as the department's inconsistent tracking of users' reported problems with the system. Additionally, USDA did not have system performance goals or dedicated staff to analyze and use system performance data, and the stabilization plan did not address these issues. Moreover, the plan did not clearly define the roles and responsibilities for the organizations involved in the stabilization effort in order to ensure proper accountability. While department officials indicated that they planned to address system performance management issues in a future version of the stabilization plan, they did not yet have plans to enable USDA to consistently track users' reported problems and to clarify roles and responsibilities. As a result, USDA could not be assured that its stabilization efforts would enable the department to reliably deliver farm benefit programs to its customers. Regarding USDA's proposed long-term investment known as MIDAS--Modernize and Innovate the Delivery of Agricultural Systems--officials had plans under way to obtain the necessary information for assessing the capability of products to integrate existing systems. However, business requirements were not used as a basis for the department's life-cycle cost estimate of $455 million for the modernization initiative. Instead, the estimate was based primarily on the cost estimate for another unrelated USDA IT investment. Similarly, the department had not adequately assessed its schedule estimate. According to department officials, they committed to accelerating the implementation of MIDAS from 10 years to 2 years in order to more quickly deliver a long-term solution to problems the department is experiencing with its existing program's delivery systems. However, business requirements were not considered when developing this schedule estimate. As a result, it was uncertain whether the department would be able to deliver the modernization initiative within the cost and schedule time frames it had proposed.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-08-657, Information Technology: Agriculture Needs to Strengthen Management Practices for Stabilizing and Modernizing Its Farm Program Delivery Systems
This is the accessible text file for GAO report number GAO-08-657
entitled 'Information Technology: Agriculture Needs to Strengthen
Management Practices for Stabilizing and Modernizing Its Farm Program
Delivery Systems' which was released on May 16, 2008.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
May 2008:
Information Technology:
Agriculture Needs to Strengthen Management Practices for Stabilizing
and Modernizing Its Farm Program Delivery Systems:
GAO-08-657:
GAO Highlights:
Highlights of GAO-08-657, a report to congressional requesters.
Why GAO Did This Study:
The U.S. Department of Agriculture (USDA) has experienced significant
problems with its information technology systems that support the
delivery of benefits programs to farmers. In October 2006, these
systems began experiencing considerable delays while attempting to
process a large number of transactions, and by January 2007, the
systems became inoperable for 1 month. In response to these issues,
USDA developed a near-term stabilization plan and long-term plans to
modernize its delivery of these programs. GAO was asked to determine
(1) the extent to which USDA‘s stabilization plan addresses key
management issues, including consistently tracking reported problems,
establishing performance metrics and goals, and defining roles and
responsibilities and (2) the adequacy of USDA‘s assessment of existing
product capabilities, as well as cost and schedule estimates for its
new, long-term modernization investment. To address these objectives,
GAO, among other things, compared USDA‘s plans with industry best
practices. On March 25, 2008, GAO briefed the requesters‘ staff on the
results of this review.
What GAO Found:
USDA‘s near-term plan to stabilize the agency‘s farm program delivery
systems focused on technical issues such as expanding telecommunication
capacity and acquiring a means for disaster backup and recovery;
however, it did not address key managerial issues such as the
department‘s inconsistent tracking of users‘ reported problems with the
system. Additionally, USDA did not have system performance goals or
dedicated staff to analyze and use system performance data, and the
stabilization plan did not address these issues. Moreover, the plan did
not clearly define the roles and responsibilities for the organizations
involved in the stabilization effort in order to ensure proper
accountability. While department officials indicated that they planned
to address system performance management issues in a future version of
the stabilization plan, they did not yet have plans to enable USDA to
consistently track users‘ reported problems and to clarify roles and
responsibilities. As a result, USDA could not be assured that its
stabilization efforts would enable the department to reliably deliver
farm benefit programs to its customers.
Regarding USDA‘s proposed long-term investment known as MIDAS”Modernize
and Innovate the Delivery of Agricultural Systems”officials had plans
under way to obtain the necessary information for assessing the
capability of products to integrate existing systems. However, business
requirements were not used as a basis for the department‘s life-cycle
cost estimate of $455 million for the modernization initiative.
Instead, the estimate was based primarily on the cost estimate for
another unrelated USDA IT investment. Similarly, the department had not
adequately assessed its schedule estimate. According to department
officials, they committed to accelerating the implementation of MIDAS
from 10 years to 2 years in order to more quickly deliver a long-term
solution to problems the department is experiencing with its existing
program‘s delivery systems. However, business requirements were not
considered when developing this schedule estimate. As a result, it was
uncertain whether the department would be able to deliver the
modernization initiative within the cost and schedule time frames it
had proposed.
What GAO Recommends:
GAO recommends that USDA develop specific plans to address management
weaknesses and develop reliable cost and schedule estimates. USDA
officials did not provide comments on a draft of this report; however,
in commenting on the draft briefing, they generally agreed with the
recommendations.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-657]. For more
information, contact Linda D. Koontz at (202) 512-6240 or
koontzl@gao.gov.
[End of section]
Contents:
Letter:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Briefing to Staff of Congressional Requesters on USDA's
Stabilization and Modernization Efforts:
Appendix II: GAO Contact and Staff Acknowledgments:
Abbreviations:
AS/400: Application System/400:
CRP: Conservation Reserve Program:
DCP: Direct and Counter-Cyclical Program:
e-LDP: Electronic Loan Deficiency Program:
FSA: Farm Service Agency:
IT: information technology:
ITIL: Information Technology Infrastructure Library:
ITS: Information Technology Services:
MIDAS: Modernize and Innovate the Delivery of Agricultural Systems:
NRCS: Natural Resources Conservation Service:
OMB: Office of Management and Budget:
RD: Rural Development:
SEI: Software Engineering Institute:
USDA: U.S. Department of Agriculture:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
May 16, 2008:
Congressional Requesters:
The U.S. Department of Agriculture (USDA) recently experienced
significant problems with its information technology (IT) systems that
support the delivery of benefits programs to farmers (farm program
delivery systems). In October 2006, these systems began experiencing
considerable delays while attempting to process a large number of
transactions, and by January 2007, the systems became inoperable for a
period of 1 month. Among other things, this outage led to significant
delays in USDA's delivery of benefits to farmers.
In response to these issues, USDA developed a near-term plan to
stabilize its farm program delivery systems. USDA also has a long-term
project to modernize its delivery of these programs. The project, which
has been in the planning stage since 2004, is known as MIDAS--Modernize
and Innovate the Delivery of Agricultural Systems--and is envisioned to
be an entirely new farm program delivery system.
At your request, we reviewed USDA's efforts to stabilize and modernize
its farm program delivery systems. Specifically, we (1) assessed the
extent to which USDA's stabilization plan addresses key management
issues, including consistently tracking reported problems,
establishing performance metrics and goals, and defining roles and
responsibilities; and (2) determined the adequacy of USDA's assessment
of existing product capabilities, as well as cost and schedule
estimates for its new, long-term modernization investment.
To address our first objective, we assessed USDA's near-term plans for
stabilizing farm program delivery systems to identify the activities
that the plan covers; analyzed system performance documentation,
including an independent validation and verification report and other
internal system performance assessments; compared the stabilization
plan with key practices for managing information systems; and
interviewed officials from USDA, the Farm Service Agency (FSA), the
Natural Resources Conservation Service, Rural Development, and end
users of the systems.
To address our second objective, we identified key practices and
lessons learned from previous GAO reports and other guidance, such as
the Software Engineering Institute's Capability Maturity Model
Integration--which encompasses defining requirements, assessing
existing product capabilities, and preparing cost and schedule
estimates. We also reviewed and analyzed FSA's business case, cost-
benefit analysis, and an analysis of alternative implementation options
for its modernization initiative. In addition to interviewing officials
from USDA and FSA to discuss their approach to planning the development
of MIDAS, we also interviewed officials from OMB to obtain their
perspective on USDA's plans to modernize its program delivery systems.
We conducted this performance audit at USDA in Washington, D.C., and
Kansas City, Missouri, and at the Office of Management and Budget in
Washington, D.C., from May 2007 to May 2008 in accordance with
generally accepted government auditing standards. Those standards
require that we plan and perform the audit to obtain sufficient,
appropriate evidence to provide a reasonable basis for our findings and
conclusions based on our audit objectives. We believe that the evidence
obtained provides a reasonable basis for our findings and conclusions
based on our audit objectives.
This report summarizes the information we provided to your staff during
our March 25, 2008, briefing. The full briefing, including our scope
and methodology, can be found in appendix I.
In summary, our briefing made the following points:
* USDA's near-term plan to stabilize the agency's farm program delivery
systems focused on technical issues, such as expanding
telecommunications capacity and acquiring a means for disaster backup
and recovery; however, it did not address key managerial issues, such
as the department's inconsistent tracking of users' reported problems
with the system. Additionally, USDA did not have system performance
goals or dedicated staff to analyze and use system performance data,
and the stabilization plan did not address these issues. Moreover, the
plan did not clearly define the roles and responsibilities for the
organizations involved in the stabilization effort in order to ensure
proper accountability. While department officials indicated that they
plan to address system performance management issues in a future
version of the stabilization plan, they did not yet have plans to
enable USDA to consistently track users' reported problems and to
clarify roles and responsibilities. As a result, USDA could not be
assured that its stabilization efforts would enable the department to
reliably deliver farm benefit programs to its customers.
* Regarding USDA's proposed long-term investment in MIDAS, officials
had plans under way to obtain the necessary information for assessing
the capability of products to integrate existing systems. However,
business requirements were not used as a basis for the department's
life-cycle cost estimate of $455 million for the modernization
initiative. Instead, the estimate was based primarily on the cost
estimate for another unrelated USDA information technology investment.
Similarly, the department had not adequately assessed its schedule
estimate. According to department officials, they committed to
accelerating the implementation of MIDAS from 10 years to 2 years in
order to more quickly deliver a long-term solution to problems the
department is experiencing with its existing program delivery systems.
However, business requirements were not considered when developing this
schedule estimate. As a result, it was uncertain whether the department
would be able to deliver the modernization initiative within the cost
and schedule time frames it had proposed.
Recommendations for Executive Action:
We recommend that the Secretary of USDA direct the department's Chief
Information Officer to work with FSA's Chief Information Officer to
develop specific plans for consistently tracking users' reported
problems and clearly defining roles and responsibilities for
Information Technology Services and the Farm Service Agency.
We also recommend that the Secretary of USDA direct the department's
chief information officer to work with FSA's chief information officer
to fully assess USDA's investment in MIDAS, including:
* establishing effective and reliable cost estimates using industry
best practices, including using key information such as business
requirements to develop the estimates; and:
* establishing a realistic and reliable implementation schedule for
MIDAS that is based on complete business requirements.
Agency Comments and Our Evaluation:
We solicited comments from USDA officials on a draft of this report;
however, the officials did not provide a response. We previously
received comments on a draft of the briefing slides (see app. I) via e-
mail, which represented the views of officials from FSA, USDA's Office
of the Chief Information Officer, and USDA's Office of the Chief
Financial Officer. These officials generally agreed with our findings,
conclusions, and recommendations. The department also provided
technical comments that we incorporated into the briefing slides, as
appropriate.
We are sending copies of this report to interested congressional
committees and the Secretary of Agriculture. We will also make copies
available to others on request. In addition, the report will be
available at no charge on the GAO Web site at [hyperlink,
http://www.gao.gov].
If you or your staffs have any questions about this report, please
contact me at (202) 512-6240 or at koontzl@gao.gov. Contact points for
our Offices of Congressional Relations and Public Affairs may be found
on the last page of this report. Key contributors to this report are
listed in appendix II.
Signed by:
Linda D. Koontz:
Director, Information Management Issues:
List of Requesters:
The Honorable Herb Kohl:
Chairman:
The Honorable Bob Bennett:
Ranking Member:
Subcommittee on Agriculture, Rural Development, Food and Drug
Administration, and Related Agencies:
Committee on Appropriations:
United States Senate:
The Honorable Rosa DeLauro:
Chairwoman:
Subcommittee on Agriculture, Rural Development, Food and Drug
Administration, and Related Agencies:
Committee on Appropriations:
House of Representatives:
The Honorable Jo Ann Emerson:
House of Representatives:
The Honorable Ray LaHood:
House of Representatives:
The Honorable Tom Latham:
House of Representatives:
[End of section]
Appendix I: Briefing to Staff of Congressional Requesters on USDA's
Stabilization and Modernization Efforts:
Information Technology: Agriculture Needs Stronger Management Practices
for Stabilizing and Modernizing Its Farm Program Delivery Systems:
Briefing for:
The Staff of the Senate Subcommittee on Agriculture, Rural Development,
Food and Drug Administration, and Related Agencies, Committee on
Appropriations, and:
The House Subcommittee on Agriculture, Rural Development, Food and Drug
Administration, and Related Agencies, Committee on Appropriations.
March 25, 2008:
Outline of Briefing:
* Introduction;
* Objectives, Scope, and Methodology:
* Results in Brief;
* Background;
* Extent to which USDA‘s Stabilization Plan Addresses Key Management
Issues;
* Adequacy of USDA‘s Assessment of Existing Product Capabilities and
Cost and Schedule Estimates for a New, Modernized Farm Program Delivery
System;
* Conclusions;
* Recommendations;
* Agency Comments and Our Evaluation.
Introduction:
The Department of Agriculture (USDA) recently experienced significant
problems with its information technology (IT) systems that support the
delivery of benefits programs to farmers”called farm program delivery
systems in this briefing. In October 2006, these systems began
experiencing considerable delays while attempting to process a large
number of transactions and, by January 2007, the systems became
inoperable for a period of one month. Among other things, this outage
led to significant delays in USDA‘s delivery of benefits to farmers.
In response to these issues, USDA developed a near-term plan to
stabilize its farm program delivery systems. USDA also has long-term
plans to modernize its delivery of these programs. The long-term
project, which has been in the planning stage since 2004, is known as
MIDAS”Modernize and Innovate the Delivery of Agricultural Systems”and
is envisioned to be an entirely new farm program delivery system.
Objectives, Scope, and Methodology:
As agreed, our objectives for this review were to:
* assess the extent to which USDA‘s stabilization plan addresses key
management issues including consistently tracking reported problems,
establishing performance metrics and goals, and defining roles and
responsibilities; and;
* determine the adequacy of USDA‘s assessment of existing product
capabilities and cost and schedule estimates for its new, long-term
modernization investment.
To address our first objective, we:
* reviewed relevant GAO and industry reports and guidance to identify
key management practices;
* reviewed and analyzed USDA‘s near-term plans for stabilizing farm
program delivery systems to identify the activities that the plan
covers;
* analyzed system performance documentation, including an independent
validation and verification report and other internal system
performance assessments;
* compared the stabilization plan with key practices for managing
information systems; and;
* interviewed officials from USDA, the Farm Service Agency (FSA), the
Natural Resources Conservation Service (NRCS), Rural Development (RD),
and end users of the systems.
To address our second objective, we:
* identified key practices and lessons learned from previous GAO
reports and other guidance such as the Software Engineering Institute‘s
(SEI) Capability Maturity Model Integration [Footnote 1] regarding
defining requirements, assessing existing product capabilities, and
preparing cost and schedule estimates;
* reviewed and analyzed FSA's business case, cost-benefit analysis, and
an analysis of alternative implementation options for its modernization
initiative;
* interviewed officials from USDA and FSA to discuss their approach to
planning the development of MIDAS, and;
* interviewed officials from OMB to obtain their perspective on USDA‘s
plans to modernize its program delivery systems.
We performed our work at USDA in Washington, D.C., and Kansas City,
Missouri, and at the Office of Management and Budget (OMB) in
Washington, D.C., from May 2007 to January 2008. We conducted this
audit in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our audit objectives.
Results In Brief:
Although USDA‘s plan to stabilize the agency‘s farm program delivery
systems focuses on technical issues such as expanding telecommunication
channels and acquiring a means for disaster backup and recovery, it
does not address key managerial issues. For example, the plan does not
address USDA‘s inconsistent tracking of users‘ reported problems with
the system, provide for the development of performance metrics and
goals to assess system performance, or clearly define organizational
roles and responsibilities. While USDA officials indicated that they
plan to address system performance management issues in a future
version of the stabilization plan, they do not yet have plans to enable
USDA to consistently track users‘ reported problems and to clarify
roles and responsibilities. As a result, USDA cannot be assured that
its stabilization efforts will enable the department to reliably
deliver farm benefit programs to its customers.
Regarding USDA‘s proposed long-term investment in MIDAS, officials have
plans under way to obtain the necessary information for assessing the
capability of products to integrate existing USDA systems; however,
they have not yet adequately assessed cost and schedule estimates by
using key information such as MIDAS business requirements to develop
the estimates. As a result, it is uncertain whether the department will
be able to deliver MIDAS within the cost and schedule time frames it
has proposed.
We are recommending that the Secretary of USDA direct the department‘s
Chief Information Officer to work with FSA‘s Chief Information Officer
to develop specific plans for consistently tracking users‘ reported
problems and clearly define the roles and responsibilities of
organizations involved in stabilizing USDA‘s farm program delivery
systems. We are also recommending that the Secretary direct the Chief
Information Officer to work with FSA‘s Chief Information Officer to
develop reliable cost and schedule estimates based on business
requirements.
We received comments on a draft of this briefing via e-mail from a
management analyst at USDA. According to the analyst, coordination has
occurred with officials from FSA, USDA‘s Office of the Chief
Information Officer, and USDA‘s Office of the Chief Financial Officer.
The analyst stated that these officials generally agreed with our
findings, conclusions, and recommendations. The department also
provided technical comments that we have incorporated, as appropriate.
Background:
One of USDA‘s major tasks is to manage and administer benefits to
farmers through programs that support farm and ranch production,
natural resources and environmental conservation, and rural
development. USDA‘s FSA is one of three service center agencies that
are responsible for administering these programs. In fiscal year 2008,
USDA estimates that it will spend approximately $29 billion in farm
loan, commodity, and conservation programs and $15 billion in rural
housing, utilities, and business development programs.
FSA supports the delivery of approximately 100 farm programs through
its 2,280 county-based service centers. Major programs include:
* The Direct and Counter Cyclical Program, which may be used by farmers
to offset the difference when the market price for a specific crop
falls below a defined target price.
* The Loan Deficiency Payment Program, which provides farmers short-
term funds to pay expenses when market prices fall below the ’loan“
price (also known as the loan rate, which is different from the
interest rate charged on marketing loans).
* The Noninsured Crop Disaster Assistance Program, which is designed to
provide aid for uninsured crops that are destroyed through natural
disasters.
* The Conservation Reserve Program, which is intended to reduce
erosion, protect streams and rivers, enhance wildlife habitats, and
improve air quality through incentive payments and cost sharing.
FSA currently uses two primary systems to process applications and user
data in support of the delivery of farm benefit programs. However, both
systems have shortcomings. One system consists of a distributed network
of IBM Application System/400 (AS/400) computers running software to
emulate [Footnote 2] IBM‘s System 36 computers, which were used in the
service centers in the 1980s. The use of emulation software allows USDA
to use the same program applications and data structures for these
computers as were previously developed for the System 36 computers, but
this configuration also limits the capabilities of the AS/400s. These
computers no longer fully meet business needs or internal control and
security requirements. Beginning in 2002, USDA also has used a
centralized ’Web farm“”an array of interconnected computer servers that
exchange data”to supplement the AS/400s with a Web-based interface for
specific programs. While FSA has been in the process of transitioning
specific farm program applications from the AS/400s to the Web farm, it
has encountered substantial performance problems with the Web farm.
FSA maintains 2,555 AS/400 computers (one or more for each service
center) with software applications to process most of its approximately
100 farm programs. The AS/400s store customer information and use it
locally for specific program delivery applications.
The Web farm stores customer data and hosts Web-based applications.
FSA‘s Web farm is located in Kansas City and is hosted on a network
known as the Common Computing Environment. [Footnote 3]
To date, FSA has transferred approximately 10 to 30 percent of its
programs to the Web farm, including several of the previously mentioned
farm programs:
* the Loan Deficiency Payment Program, also called the Electronic Loan
Deficiency Program (e-LDP) in September 2004;
* the soils database and processes for documenting participation offers
for the Conservation Reserve Program (CRP) in April 2004, [Footnote 4]
and;
* the enrollment processes for the Direct and Counter-Cyclical Program
(DCP) in October 2005. [Footnote 5]
There are two methods for applying for and obtaining benefits from USDA
farm programs. The method that is used depends on which system is
hosting the desired program:
* If the desired program is hosted on the Web farm, the customer has
the option of accessing it from a home or business computer or visiting
a local county service center, [Footnote 6] where staff use the Web
farm to complete the customer‘s transaction.
* If the desired program is hosted on the AS/400s, the customer must
visit, mail, or fax documents to a local service center, where the
staff use the AS/400s to complete the customer‘s transaction.
The following graphic illustrates the two methods for applying for and
receiving benefits.
Figure 1: Methods for Applying for and Obtaining Benefits:
[See PDF for image]
This figure is an illustration of methods for applying for and
obtaining benefits. The following information is depicted:
There are two methods shown:
First method:
1. Mandatory initial visit to service center to obtain approval for
participation in programs.
2. If desired program is hosted on the AS/400s: Staff assisted service
at the local USDA service center:
3. Access to AS/400.
4. Replicate data, utilizing the internet to:
5. Input to USDA Common Computing Environment; Kansas City Web farm
servers.
Second method:
1. Mandatory initial visit to service center to obtain approval for
participation in programs.
2. If desired program is one of the limited number of Web-enabled
applications hosted on the Web farm:
- Self-service access from home or business, or;
- Staff assisted service at the local USDA service center.
3. Access to Web-enabled applications utilizing the internet to:
4. 5. Input to USDA Common Computing Environment; Kansas City Web farm
servers.
Source: GAO analysis of USDA data.
[End of figure]
According to USDA officials, the AS/400s emulating the System 36
operating system are antiquated and no longer meet business needs.
* IBM first introduced the System 36 computers in 1983 and the AS/400s
in 1988. According to USDA, these computers are now obsolete and,
therefore, the company is no longer supplying replacement parts or
providing maintenance services to USDA for these machines.
* The AS/400 computers have limited storage. The storage for an AS/400
computer is 17 gigabytes. In comparison, today‘s personal computers
usually have approximately 20 to 30 times more storage. As a result, an
AS/400 can store only a limited number of files. Since many service
centers reach their full capacity for file storage on a daily basis,
USDA must take extra steps to monitor the status and, if necessary,
work with the service centers to remove files from the system when the
storage is overloaded.
* Applications running on the AS/400s are written in a legacy
programming language (COBOL). As a result, according to USDA officials,
they have had difficulty finding programmers who are knowledgeable in
this programming language to build and maintain additional applications
for the computer.
* AS/400s can store customer information only locally at the county
offices. Customers are unable to use different service centers to
complete their transactions.
USDA officials have also indicated that the AS/400s are not in
compliance with internal control and security requirements.
* According to officials, because of technological limitations, the
AS/400s emulating System 36 computers are not in compliance with
internal control requirements as specified in OMB Circular A-123,
’Management‘s Responsibility for Internal Control“ and security
requirements established under the Federal Information Security
Management Act. For example, officials stated that they currently have
limited mechanisms in place to prevent fraud and abuse when using the
AS/400s because the AS/400s do not allow USDA to run software that
would limit the access of service center employees to information
necessary to performing their duties.
As previously stated, the Web farm also has shortcomings. Specifically,
as USDA began to transfer more programs from the AS/400s to the Web
farm, performance issues developed and gradually became more severe,
ultimately leading to USDA‘s inability to deliver farm programs to many
customers in early 2007.
* Significant performance problems began in October 2006 and became
progressively worse when the use of the Web farm dramatically increased
in order to complete prior-year accounting transactions and to process
direct and counter-cyclical payment applications for the new accounting
year.
* On January 13, 2007, the majority of business applications running on
USDA‘s Web farm shut down for approximately a month. On February 14,
2007, USDA restored service based on the Web farm. According to USDA
officials, the Web farm continued to experience 4-6 hours of
unscheduled outages each week after this time. During outages, service
center staff were unable to provide reliable customer service due to
slow transaction speeds, Web page display errors, delays in updates to
Web-based applications, and had to continually re-enter Web-based
application data.
The following figure provides a timeline of events leading up to the
destabilization of the Web farm.
Figure 2: Timeline of Events Leading to the Destabilization of the Web
farm:
[See PDF for image]
This figure is a timeline of events leading to the destabilization of
the Web farm. Indicated dates are either approximate timeframe or
specific date. The following information is illustrated:
First quarter, 2002: USDA begins to develop Web farms (approximate
timeframe);
Fourth quarter, 2003: Web farms become operational (approximate
timeframe);
Second quarter, 2004: CRP added to the Kansas City Web farm
(approximate timeframe);
Third quarter, 2004: e-LDP added to Kansas City Web farm (approximate
timeframe);
Fourth quarter, 2004: ITS established (approximate timeframe);
Fourth quarter, 2005: DCP added to Kansas City Web farm (approximate
timeframe);
Fourth quarter, 2006: Completion of FY06 accounting transactions
increases usage (approximate timeframe);
Fourth quarter, 2006: Submission of DCP applications for FY07 increases
system usage (approximate timeframe);
Fourth quarter, 2006: Issues with Kansas City Web farm began
(approximate timeframe);
Early first quarter, 2007: Business activity running on Web farm shut
down (specific date);
Early first quarter, 2007: Expert performance tests conducted (specific
date);
Early first quarter, 2007: Marginal performance restored (specific
date).
Over the past two years, USDA has had various organizations and
individuals identify specific technical issues pertaining to the Web
farm. In addition, in January 2007, USDA officials assembled a group of
experts to conduct performance tests on the system to identify
additional technical issues that contributed to the performance
problems. The identified issues include:
* Inefficiently designed and structured databases caused benefits
transactions to be processed slowly during FSA‘s peak usage periods.
* USDA lacked 2 of the 5 commonly recommended testing environments,
which, if implemented, could have been used to identify the potential
adverse effects of adding new applications to the Web farm.
* Insufficient bandwidth of firewalls serving the county-based agencies
resulted in USDA‘s inability to accommodate increases in Web farm
traffic.
In May 2007, Congress appropriated $37.5 million to be used by USDA for
the stabilization of farm programs‘ existing delivery systems.
USDA‘s current IT support structure was established by merging IT staff
from FSA and the two other service center agencies into one
organization”Information Technology Services (ITS)”within the Office of
the Chief Information Officer. The new organization was designated to
support and maintain the Common Computing Environment. ITS provides
operations, maintenance, and help desk support for equipment,
telecommunications, and administrative applications.
While ITS supports the platforms and infrastructure across the service
center agencies, the responsibility for developing and operating
applications that support program delivery remains with the individual
service center agencies. For example, FSA has developed 143 different
applications to support more than 100 farmer benefit programs and is
responsible for maintaining these applications. For this reason, FSA
continues to maintain its own IT services division, headed by its own
chief information officer.
With regard to its long-term modernization program, USDA began planning
the MIDAS initiative in January 2004. It is aimed at reengineering
agency business processes and correcting weaknesses in aging IT systems
and, according to the MIDAS fiscal year 2007 business case, the
initiative was planned to be completed by fiscal year 2020. According
to USDA, MIDAS is intended to:
* improve the overall delivery of benefits to FSA customers through the
use of the Internet;
* modernize system operations to remediate IT weaknesses;
* correct financial material weaknesses and integrate with USDA‘s
financial modernization plan;
* provide flexibility in responding to changes in program requirements
as defined by new legislation, and;
* provide computing environments that comply with legislative
requirements such as the Federal Information Security Management Act.
From January 2004 through January 2006, officials reported that they
had spent $14 million defining requirements and conducting the initial
planning for MIDAS. However, USDA never completed the MIDAS
requirements development process because key program officials lost
confidence that the process would be an effective solution to meet
USDA's future business needs and consequently withdrew their support.
Subsequently, in the summer of 2006, USDA changed direction from
building a customized system to acquiring commercial off-the-shelf
enterprise resource planning software. [Footnote 7] USDA officials
stated that this approach would be more flexible in adapting to new
legislative requirements and would reduce overall IT operating costs
for the department.
The department is currently still in the planning phase for MIDAS. In
September 2007, USDA hired an executive program manager for the MIDAS
initiative. Officials have indicated that they have additional
requirements definition work under way to complete the foundational
requirements and expect to finish this work by the third quarter of
fiscal year 2008. [Footnote 8] Subject to the availability of funds,
they also plan to issue a request for proposals to acquire and
implement the system later in 2008.
USDA currently estimates that the life cycle cost of MIDAS will be $455
million and is planning for a two-year implementation schedule from the
time it awards a contract. Since an award date has not been
established, a specific implementation schedule has not been developed.
MIDAS has been on OMB‘s high risk IT project list [Footnote 9] since
fiscal year 2006.
Objective 1: Addressing Key Management Issues:
USDA has developed a stabilization plan, but has not addressed key
managerial issues with its farm program delivery systems.
To successfully implement any IT project, both technical as well as
managerial issues should be addressed. Regarding managerial issues, the
Project Management Institute [Footnote 10] and the Information
Technology Infrastructure Library (ITIL)[Footnote 11] both indicate
that good program management includes consistently tracking users‘
reported problems, developing reportable metrics for measuring
performance, and defining clear roles and responsibilities among
project teams. Instituting these good management practices can help
ensure that a solid foundation for achieving an IT project‘s objective
is established.
In April 2007, USDA established a plan for stabilizing its farm program
delivery systems. The plan, which focuses on technical issues
identified by USDA officials and outside experts, has the following
objectives:
* Expand telecommunication channels, acquire firewalls with greater
bandwidth, and optimize Web farm software and databases to address the
inefficient design and structure of the databases and thus improve the
efficiency of performing transactions.
* Conduct a review of the technical architecture of the Common
Computing Environment to improve FSA‘s understanding of all existing
components of its farm delivery systems.
* Build a data warehouse to centralize customer data to enable FSA to
more efficiently conduct program management oversight and generate
reports pertaining to farm programs.
* Implement monitoring tools, configuration management practices, and
the two missing testing environments to help enable USDA to be more
proactive in identifying performance problems.
* Acquire a means for disaster backup and recovery, which does not
currently exist.
* Invest in specialized staff training and development to help ensure
that users optimize the capabilities of these new tools and
applications.
USDA began to address these objectives in 2007 and plans to continue
the stabilization effort through 2010.
The plan also includes high-level descriptions of each of the sub-
projects intended to address the objectives. In addition, it includes
implementation schedules for the sub-projects and the management
structure for overseeing the project.
According to USDA officials, as of October 2007, they had spent
approximately $18 million to take steps towards achieving these
objectives. For example, they had expanded telecommunication channels,
acquired more sophisticated firewalls, and had a contractor prepare the
first draft of process flow diagrams of selected program delivery
processes.
USDA estimates it will spend an additional $131 million from fiscal
year 2008 through 2010 to complete its technical plan for stabilizing
the Web farm. Major tasks that remain include building a data warehouse
to centralize customer data and acquiring a means for disaster backup
and recovery.
While USDA has developed a stabilization plan that focuses on various
technical weaknesses relating to its farm programs‘ delivery system,
the plan does not address key managerial areas. Specifically:
* USDA has had difficulty ensuring that it consistently tracks problems
that users encounter with the Web farm. Rather than using a consistent
approach for reporting and resolving problems, which could enable USDA
to more accurately identify issues with the system, users have been
informally obtaining assistance from their colleagues to solve
problems. According to a report issued by an independent verification
and validation contractor in October 2007, there has been widespread
use of such informal networks for problem resolution. Moreover, certain
employees did not have access to USDA's customer service ticketing
system; therefore, problems that were reported by those without access
to the ticketing system were not being centrally tracked. As a result,
these problems are not being tracked and monitored consistently, thus
contributing to unreliable information about system performance.
Further, the current stabilization plan does not address the need to
improve problem tracking. USDA officials indicated that they
established a separate initiative in April 2007 to address this issue,
although they have not yet provided us with any details regarding it.
* ITS lacks measures for the performance of the Web farms. During an
assessment of the Kansas City Web farm in the fall of 2006, agency
officials reported that they did not have real-time statistics
available on the Web farm‘s performance, and that they needed
statistics such as data processing time, firewall activity, and
utilization of the telecommunications network. While officials have
indicated that in November 2007 they implemented automated tools to
monitor the system and to collect performance data, such as processing
and utilization time, ITS does not have dedicated staff to analyze and
use the collected performance data. Additionally, ITS has not
established performance goals for the Web farm.
While FSA‘s Chief Information Officer indicated that USDA officials are
aware of the performance management issues, they indicated that they
did not want to address them in their stabilization plan until they had
investigated them more thoroughly. The report by the independent
verification and validation contractor provided additional analysis and
offered recommendations to USDA for resolving the performance
management problems and other issues. For example, the contractor
recommended that USDA establish performance metrics for all ITS
business service areas and create a dedicated ITS team to conduct
performance management activities. In response to this study and our
inquiries about how the study‘s recommendations would be addressed,
USDA officials indicated that they plan to update the stabilization
plan by February 2008 to address the identified performance management
issues.
* USDA lacks clearly defined organizational roles and responsibilities
for stabilizing the Web farm. As previously mentioned, USDA‘s ITS
organization has overall responsibility for operating and maintaining
the Web farm hardware and the Common Computing Environment network;
FSA‘s ITS division is responsible for operating and maintaining FSA
applications for farm programs. However, USDA‘s stabilization plan
indicates that ITS and FSA are both ’owners“ of all planned improvement
activities and does not clearly establish the specific roles and
responsibilities of these respective organizations. As we have
previously reported, effective management of programs requires clear
definitions of roles and responsibilities. The extent to which these
are explicit and unambiguous goes a long way towards ensuring proper
accountability and performance. USDA officials indicated that they had
overlooked that aspect of their plan, and they do not yet have plans to
address the lack of clearly defined roles and responsibilities for
stabilizing the Web farm in the updated version of the stabilization
plan.
Until USDA addresses the inconsistent tracking of users‘ reported
problems and the lack of clearly defined roles and responsibilities, it
may not be able to establish a solid foundation for achieving and
sustaining stability in the farm program delivery systems. As a result,
the department faces the risk that its stabilization plan will not
ensure that it is able to successfully deliver benefits to farmers in
the future.
Objective 2: Assessment of Capabilities, Cost, and Schedule for
Modernization:
While the department has plans under way to obtain the information
necessary to assess the capabilities of commercial off-the-shelf
enterprise resource planning products for MIDAS, it has not:
* adequately assessed the cost of its proposed approach or;
* adequately assessed the schedule for its proposed approach.
Without developing reliable cost and schedule estimates using business
requirements to derive the estimates, it is questionable whether USDA
will be able to deliver MIDAS within the cost and schedule it has
proposed.
Objective 2: Assessment of Capabilities, Cost, and Schedule for
Modernization: Plans Under Way to Assess Product Capabilities:
USDA has not yet fully assessed the capabilities of commercial products
to integrate with key USDA systems, but has plans under way to do so.
SEI [Footnote 12] recommends that organizations planning to acquire a
new enterprise resource planning system understand the capabilities of
existing products. More specifically, it recommends that the
organization understand the ways in which the new system will interface
with legacy systems.
In June 2006, the department issued a request for information to learn
about vendors‘ enterprise resource planning product capabilities, and
the responses were used to develop USDA‘s investment justification
documentation. However, USDA did not request that vendors provide
detailed information regarding the capability of commercial off-the-
shelf products to integrate with key USDA systems, such as the
department‘s existing documents and records management system and
service center customer database. Officials indicated that they have
met with a few other federal agencies that have implemented enterprise
resource planning products to obtain lessons learned. Officials stated
that in order to save time, rather than issuing another request for
information, they plan to solicit this information in their request for
proposals.
USDA's inclusion of sufficient detail regarding the existing systems,
with which the commercial off-the-shelf enterprise resource planning
product needs to be integrated, may facilitate the agency's assessment
of proposals.
Objective 2: Assessment of Capabilities, Cost, and Schedule for
Modernization: Inadequate Assessment of Costs:
USDA has not adequately assessed the costs of MIDAS.
GAO‘s Cost Assessment Guide [Footnote 13] provides best practices for
establishing reliable cost estimates to be used in the software
acquisition process. Specifically, GAO's Cost Assessment Guide states
that key information such as requirements should be used to develop
cost estimates.
As previously mentioned, USDA had worked to define requirements for
MIDAS, but this effort was never completed. As a result, business
requirements were not used as a basis for their life cycle cost
estimate of $455 million. Instead, their estimate was based primarily
on the cost estimate for another unrelated USDA IT investment, the
Financial Management Modernization Initiative. [Footnote 14] Officials
also indicated that they had used information provided by vendors in
response to USDA‘s request for information. However, while the
department requested that vendors provide cost estimates as part of the
request for information for MIDAS, three vendors indicated that they
could not develop such estimates because USDA had not provided enough
specific information regarding its needs for hardware, software, and
labor in order for the vendors to provide an estimate. Other vendors
supplied a generic price list. As a result, the department had only a
limited basis for deriving the cost figures for MIDAS.
USDA officials acknowledge that they had limited information to use as
a basis for developing their cost estimate. However, the department
included the $455 million figure in its business case for justifying
the investment in MIDAS. According to USDA officials, they included
this figure because it was the best estimate they could derive given
the information they had.
Not until after the cost estimate was developed did USDA officials
begin analyzing the partially-defined requirements to determine what
requirements could be applied to the new approach of acquiring a
commercial off-the-shelf product for MIDAS. As previously mentioned,
officials have additional requirements definition work under way to
complete the foundational business requirements and expect to finish
this work by the third quarter of fiscal year 2008.
Without defined complete business requirements, significant questions
will remain regarding the accuracy and reliability of the MIDAS cost
estimate.
Objective 2: Assessment of Capabilities, Cost, and Schedule for
Modernization: Inadequate Assessment of Schedule:
USDA has not adequately assessed the implementation schedule for MIDAS.
According to SEI, [Footnote 15] a systematic assessment of business
requirements can also provide an organization with the opportunity to
conduct a realistic estimate for the project schedule.
In its fiscal year 2008 business case for MIDAS, the department
committed to accelerating its implementation of MIDAS from ten years to
two years. According to USDA officials, they decided to accelerate the
implementation in order to deliver a long-term solution to problems the
department is experiencing with its existing program delivery systems
more quickly.
As part of USDA‘s plan to reduce the time frame for implementing MIDAS
by 80 percent, officials plan to condense the requirements analysis
phase from four years to five months. Moreover, they plan to reduce the
analysis and design portion of the acquisition from three and a half
years to nine months.
However, according to USDA officials, similar to the cost estimation
process, the schedule estimates had been based on previous experience
and not on an understanding of the business requirements. Additionally,
officials said that they are unaware of any programs that have been
able to employ a similar product implementation within a two-year time
frame. While USDA has produced a high-level description of the
accelerated implementation, the department does not plan to establish a
detailed project schedule for MIDAS until it has selected a vendor for
the implementation.
The lack of a reliable basis for the two-year implementation schedule
for MIDAS significantly increases the risk that the department will not
be able to implement MIDAS within this time frame. Furthermore,
schedule overruns could lead to further problems such as cost overruns.
Conclusions:
While USDA‘s stabilization plan focuses on addressing technical issues,
the plan is inadequate because it does not address key managerial
weaknesses. USDA officials recently indicated that they plan to address
one of these three issues in a future version of their stabilization
plan; however, they do not have plans to address the inconsistent
tracking of users‘ reported problems and the lack of clearly defined
roles and responsibilities. As a result, the department cannot be
assured that its stabilization efforts will enable it to reliably
deliver farm benefit programs to its customers.
Additionally, officials have plans under way to obtain the information
necessary for assessing vendors‘ ability to integrate existing USDA
systems with commercial off-the-shelf enterprise resource planning
products; however, they have not yet adequately assessed cost and
schedule estimates by using key information such as business
requirements to develop the estimates. As a result, it is uncertain
whether the department will be able to deliver MIDAS within the cost
and schedule time frames it has proposed.
Recommendations:
We recommend that the Secretary of USDA direct the department‘s Chief
Information Officer to work with FSA‘s Chief Information Officer to
develop specific plans for consistently tracking users‘ reported
problems and clearly defining roles and responsibilities for ITS and
FSA.
We also recommend that the Secretary of USDA direct the department‘s
Chief Information Officer to work with FSA‘s Chief Information Officer
to fully assess USDA‘s investment in MIDAS, including:
* establishing effective and reliable cost estimates using industry
best practices, including using key information such as business
requirements to develop the estimates and;
* establishing a realistic and reliable implementation schedule for
MIDAS that is based on complete business requirements.
Agency Comments and Our Evaluation:
We received comments on a draft of this briefing via e-mail from a
management analyst at USDA. According to the analyst, coordination has
occurred with officials from FSA, USDA‘s Office of the Chief
Information Officer, and USDA‘s Office of the Chief Financial Officer.
The analyst stated that these officials generally agreed with our
findings, conclusions, and recommendations. The department also
provided technical comments that we have incorporated, as appropriate.
[End of section]
Appendix II: GAO Contact and Staff Acknowledgments:
GAO Contact:
Linda D. Koontz, (202) 512-6240 or koontzl@gao.gov:
Staff Acknowledgments:
In addition to the contact named above, John de Ferrari (Assistant
Director), Marisol Cruz, Neil Doherty, Nancy Glover, Josh Leiling,
James MacAulay, and Shannin O'Neill made key contributions to this
report.
[End of section]
Footnotes:
[1] Software Engineering Institute, CMMI Acquisition Model (CMMI-AM),
Version 1.1, (Pittsburgh, Penn.: May, 2005).
[2] Emulation software enables the use of programs not originally
intended for a particular computer system.
[3] The Common Computing Environment provides a network connecting the
three service center agencies and provides administrative applications”
such as common e-mail, telecommunications, and Microsoft Office
tools”to the three agencies.
[4] The AS/400s still host the application software for maintaining
contracts and annual CRP rental payments.
[5] The AS/400s still host the application software for payment
processes.
[6] Customers may also mail or fax documents to the service centers for
the staff to process.
[7] Enterprise resource planning refers to the use of commercial off-
the-shelf software that incorporates shared data from various lines of
business and that is consistent across an entire organization.
[8] According to USDA officials, detailed requirements development work
will be completed by the contractor following an award.
[9] High risk projects are projects requiring special attention from
oversight authorities and the highest level of agency management
because of one or more of the following four reasons: (1) the agency
has not consistently demonstrated the ability to manage complex
projects; (2) the projects have exceptionally high development,
operating, or maintenance costs; (3) the projects are addressing
deficiencies in the agencies‘ ability to perform an essential mission
program or function of the agency; or (4) the projects‘ delay or
failure would impact the agencies‘ essential mission functions.
[10] Project Management Institute, A Guide to the Project Management
Body of Knowledge (PMBOK), 3rd ed. (Newton Square, Penn.; 2004).
[11] The ITIL is a set of best practices guidance for IT service
management owned by the Office of Government Commerce within the
government of the United Kingdom.
[12] Smith, et. al., ’Enterprise Integration,“ The Architect, vol. 5,
no. 4 (Pittsburgh, Penn.; SEI, Fourth Quarter 2002), [hyperlink,
http://www.sei.cmu.edu/news-at-sei/columns/the_architect/2002/4q02/
architect-4q02.pdf] (accessed Jan. 4, 2008).
[13] GAO, Cost Assessment Guide: Best Practices for Estimating and
Managing Program Costs, Exposure Draft, GAO-07-1134SP Washington, D.C.:
July 2007).
[14] USDA's Financial Management Modernization Initiative is a
significant IT modernization effort that intends to address material
financial weaknesses through improving its general ledger and
administrative payment system.
[15] SEI, Software Acquisition Capability Maturity Model (SA-CMM)
Version 1.03, (Pittsburgh, Penn.: March 2002).
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
