Federal Student Aid
Progress in Integrating Pell Grant and Direct Loan Systems and Processes, but Critical Work Remains Gao ID: GAO-03-241 December 31, 2002To address system problems and other long-standing management weaknesses, in 1998, the Congress created a discrete unit within the Department of Education, the Office of Federal Student Aid (FSA). This office subsequently adopted a new approach to systems integration using middleware (a type of software that can allow an application to access data residing in different databases) and Extensible Markup Language (XML)--a flexible, nonproprietary set of standards that is intended to make it easier to identify, integrate, and process information widely dispersed among systems and organizations. FSA's first use of this approach is the Common Origination and Disbursement (COD) process for the Direct Loan, Pell Grant, and campus-based programs. GAO initiated a follow-up review to assess FSA's progress in implementing this process.
FSA has made progress in implementing the COD process. Specifically, it has implemented (1) a new information technology infrastructure that uses middleware to enable data exchange among disparate systems; (2) the initial version of the basic COD system, which replaces two existing systems and is being used by schools participating in the Pell Grant and Direct Loan programs; (3) middleware into existing systems to support the COD process; and (4) a common record based on XML that schools can use to submit student financial data for the Pell Grant and Direct Loan programs. However, the implementation of the COD process is behind schedule, and its ultimate success hinges on FSA's completing critical work, including addressing serious postimplementation operational problems, and having thousands of postsecondary schools implement the common record. Further, there are important elements to managing any information technology investment that FSA has not yet completed: Determining whether expected benefits are being achieved. FSA has only some of the metrics, baseline data, and tracking processes necessary to determine whether it is achieving all expected benefits. Tracking lessons learned. FSA has relied on an ad hoc approach for gathering and disseminating lessons learned related to schools' implementation of the common record. To address this issue, FSA plans to include lessons learned as part of an update to its school testing guide. However, this does not replace the need for an ongoing mechanism to capture and disseminate lessons learned, without which schools may encounter problems that could have been avoided or mitigated.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-03-241, Federal Student Aid: Progress in Integrating Pell Grant and Direct Loan Systems and Processes, but Critical Work Remains
This is the accessible text file for GAO report number GAO-03-241
entitled 'Federal Student Aid: Progress in Integrating Pell Grant and
Direct Loan Systems and Processes, but Critical Work Remains' which was
released on December 31, 2002.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products‘ accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
Report to the Secretary of Education:
United States General Accounting Office:
GAO:
December 2002:
Federal Student Aid:
Progress in Integrating Pell Grant and Direct Loan Systems
and Processes, but Critical Work Remains:
GAO-03-241:
GAO Highlights:
Highlights of GAO-03-241, a report to the Secretary of Education:
Why GAO Did This Study:
To address system problems and other long-standing management
weaknesses, in 1998, the Congress created a discrete unit within the
Department of Education, the Office of Federal Student Aid (FSA). This
office subsequently adopted a new approach to systems integration using
middleware (a type of software that can allow an application to access
data residing in different databases) and Extensible Markup Language
(XML)”a flexible, nonproprietary set of standards that is intended to
make it easier to identify, integrate, and process information widely
dispersed among systems and organizations.
FSA‘s first use of this approach is the Common Origination and
Disbursement (COD) process for the Direct Loan, Pell Grant, and campus-
based programs. GAO initiated a follow-up review to assess FSA‘s
progress in implementing this process.
What GAO Found:
FSA has made progress in implementing the COD process. Specifically, it
has implemented:
* a new information technology infrastructure that uses middleware to
enable data exchange among disparate systems;
* the initial version of the basic COD system, which replaces two
existing systems and is being used by schools participating in the Pell
Grant and Direct Loan programs;
* middleware into existing systems to support the COD process; and
* a common record based on XML that schools can use to submit student
financial data for the Pell Grant and Direct Loan programs.
However, the implementation of the COD process is behind schedule, and
its ultimate success hinges on FSA‘s completing critical work,
including addressing serious postimplementation operational problems,
and having thousands of postsecondary schools implement the common
record. Further, there are important elements to managing any
information technology investment that FSA has not yet completed:
* Determining whether expected benefits are being achieved. As
illustrated below, FSA has only some of the metrics, baseline data, and
tracking processes necessary to determine whether it is achieving all
expected benefits.
* Tracking lessons learned. FSA has relied on an ad hoc approach for
gathering and disseminating lessons learned related to schools‘
implementation of the common record. To address this issue, FSA plans
to
include lessons learned as part of an update to its school testing
guide.
However, this does not replace the need for an ongoing mechanism to
capture and disseminate lessons learned, without which schools may
encounter problems that could have been avoided or mitigated.
Highlights Figure:
[See PDF for image]
[End of image]
What GAO Recommends:
Among GAO‘s recommendations is that the Secretary of Education direct
the
Chief Operating Officer, FSA, to establish a process to capture and
disseminate lessons learned to schools.
In commenting on a draft of this report, FSA provided updated
information
and technical comments. GAO modified the report to address this new
information, as appropriate.
www.gao.gov/cgi-bin/getrpt?GAO-03-241.
To view the full report, including the scope and methodology, click on
the
link above.For more information, contact David A. Powner at (202) 512-
9286
or pownerd@gao.gov.
Contents:
Letter:
Results in Brief:
Background:
Progress in Implementing the COD Process, but Critical Work Remains,
and Benefits and Lessons Learned Are Not Being Tracked:
FSA Is Not Completely Tracking Actual COD Benefits or Lessons Learned
Related to Schools‘ Implementation of the Common Record:
Conclusions:
Recommendations:
Agency Comments and Our Evaluation:
Appendix I: Systems and Technologies Supporting the Common Origination
and Disbursement Process, as of
November 2002:
Appendix II: Comments from the Department of Education‘s
Office of Federal Student Aid:
GAO Comments:
Tables:
Table 1: Status of FSA Tracking of Actual Benefits of the COD System:
Table 2: Examples of Lessons Learned Related to School Migration to the
Common Record:
Abbreviations:
COD: Common Origination and Disbursement:
EAI: Enterprise Application Integration:
FSA: Office of Federal Student Aid:
IT: information technology:
XML: Extensible Markup Language:
Washington, DC 20548:
December 31, 2002:
The Honorable Roderick R. Paige
The Secretary of Education:
Dear Mr. Secretary:
Over the past decade, the Department of Education has spent millions of
dollars to modernize and integrate[Footnote 1] its disparate financial
aid systems in an effort to improve the administration of its programs
and provide more information and greater service to its customers--
students, parents, schools, and lenders. However, as we reported last
year, the department‘s efforts had achieved limited success.[Footnote
2] In January 1993, for example, the department awarded a 5-year, $39
million contract for the development and maintenance of the National
Student Loan Data System, which was to provide information on systems
across programmatic boundaries, yet it often lacks the most recent
information from other financial aid systems.
To address such systems problems and other long-standing management
weaknesses, in 1998, as you know, the Congress created the federal
government‘s first performance-based organization,[Footnote 3]
currently called the Office of Federal Student Aid (FSA).[Footnote 4]
Under the performance-based organization concept, FSA is a discrete
organizational unit within the Department of Education, led by its own
Chief Operating Officer. FSA focuses on managing the operation of the
student financial assistance programs, while the Department of
Education focuses on policy-making functions.
FSA subsequently adopted a systems integration approach that uses
middleware[Footnote 5] and Extensible Markup Language (XML)[Footnote 6]
technologies. In November 2001, we reported that in selecting
middleware, FSA adopted a viable, industry-accepted mechanism for
addressing its long-standing systems integration problems.[Footnote 7]
We also reported that FSA‘s first use of middleware and XML as part of
a process for delivering Direct Loan and Pell Grant aid to students was
expected in early 2002. This process, called the Common Origination and
Disbursement (COD) process, was expected to use a new system, also
called COD, and be supported by a new information technology (IT)
infrastructure and various existing systems.
In the summer of 2002, we initiated a follow-up review to our November
2001 report. Our objective was to assess FSA‘s progress in implementing
the COD process. In doing this work, we reviewed applicable FSA and
Accenture (the prime contractor) documentation, including FSA‘s
modernization blueprint, the COD business case, the COD system‘s
requirements, the COD and Enterprise Application Integration
implementation timelines, Accenture project status briefings, and test
guides and results. In addition, we reviewed reports by an independent
verification and validation contractor. We also interviewed applicable
officials from FSA IT and program offices and Accenture.
We performed our work at FSA headquarters in Washington, D.C., between
August and November 2002, in accordance with generally accepted
government auditing standards.
Results in Brief:
FSA has made progress in implementing the COD process. [Footnote 8]
Specifically, it has implemented (1) a new IT infrastructure that uses
middleware to enable data exchange between disparate systems, (2) the
initial version of the basic COD system, (3) middleware into existing
systems to support the COD process, and (4) a common record based on
XML that schools can use to submit student financial data for both the
Pell Grant and Direct Loan programs. Nevertheless, the implementation
of the COD process is behind schedule, and its ultimate success hinges
on FSA‘s completing critical work, including the implementation of the
basic COD system requirements, addressing serious postimplementation
operational problems, and having thousands of postsecondary schools
implement the common record.
Important elements of managing an IT investment are determining whether
expected benefits are being achieved and tracking lessons learned.
However, at this time, FSA is not fully tracking whether it is
achieving certain expected benefits, such as increased customer
satisfaction. In this instance, FSA COD officials stated that they did
not have approval from the Department of Education to perform the
survey that they had planned to use to validate this expected benefit,
but they are trying to identify alternative metrics. Without such data,
FSA lacks vital information on whether it is achieving all of its
investment goals. Regarding lessons learned, FSA relies on an ad hoc
approach to gathering and disseminating such information, which it
believes is an adequate approach. However, such an ad hoc process may
not ensure that all schools obtain these critical data. As a result,
schools may encounter problems that could have been avoided or
mitigated had they known of other schools‘ experiences.
We are making recommendations to improve the management of FSA‘s COD
program, including recommending that you direct FSA‘s Chief Operating
Officer to develop (1) metrics and baseline data that are related to
increased customer satisfaction and increased financial integrity and
(2) a tracking process to assess the extent to which the expected
results are being achieved.
In commenting on a draft of this report, FSA‘s Chief Operating Officer
provided updated information and technical comments, but did not
comment on our recommendations. Among the comments provided, the Chief
Operating Officer (1) stated that the draft report did not adequately
portray the level of COD progress that had been made, (2) agreed that
FSA was not tracking all expected benefits at this time and provided
new information and supporting documentation on its recent efforts in
this area, and (3) stated that FSA‘s ’informal“ process for
communicating lessons had worked well but planned to provide written
lessons learned as part of a planned update of its school testing
guide. We updated our report to reflect new FSA processes and
information, as appropriate. However, we believe that we have
accurately portrayed FSA‘s progress, particularly in light of its need
to address operational problems and facilitate the implementation of
the common record at thousands of postsecondary schools. In addition,
we continue to believe that FSA‘s ad hoc processes for capturing and
disseminating lessons learned do not provide assurance that it has
captured and disseminated lessons learned related to schools‘
implementation of the common record and could overlook important
improvements that could be made.
Background:
The Department of Education‘s FSA manages and administers student
financial assistance programs authorized under title IV of the Higher
Education Act of 1965, as amended. These postsecondary programs include
the William D. Ford Federal Direct Loan Program (often called the
Direct Loan program), the Federal Family Education Loan Program (often
called the Guaranteed Loan program), the Federal Pell Grant Program,
and campus-based programs.[Footnote 9] Annually, these programs
together provide about $50 billion in student aid to approximately 8
million students and their families.
During the past three decades, the Department of Education has created
many disparate information systems to support these various student
financial aid programs. In many cases, these systems--run on multiple
operating platforms using different network protocols[Footnote 10] and
maintained and operated by a host of different contractors--were unable
to easily exchange the timely, accurate, and useful data needed to
ensure the proper management and oversight of the various student
financial aid programs. For example, as we reported in 1997, neither
the National Student Loan Data System nor other systems were designed
for efficient access to reliable student financial aid information,
since many systems were incompatible and lacked data standards and
common identifiers.[Footnote 11] In addition, because FSA used three
separate systems to originate and/or disburse title IV funds, access to
student and school data was fragmented and unreliable. As a result, FSA
found it increasingly difficult to quickly access data to support day-
to-day operational and management decisions, and schools could not
easily access data to obtain a clear picture of the title IV student
aid that had been disbursed.
In September 1999, FSA issued its initial modernization
blueprint,[Footnote 12] which was subsequently updated in July 2000, to
transform the title IV student financial aid systems using technology.
COD is one of four school service business processes[Footnote 13] in
FSA‘s blueprint and is intended to implement a simplified process for
the operation of the Direct Loan and Pell Grant programs. According to
FSA‘s modernization blueprint, the common origination and disbursement
process is composed of seven steps involving students, the Department
of Education, and schools: (1) obtain applicant data, (2) determine
eligibility, (3) determine award, (4) notify the Department of
Education of the intent to disburse, (5) obtain funds from Education,
(6) disburse funds to student, and (7) close out. A common process to
support origination and disbursement is considered critical to FSA‘s
goal of achieving an enterprisewide solution that provides real-time
data to students, schools, and financial partners via Web portals.
To implement COD, FSA is using middleware and XML technologies.
Specifically, middleware is being used to integrate FSA systems that
support the COD process.[Footnote 14] Traditionally, systems
integration would require building separate point-to-point interfaces
between every two applications. Although this approach can be
effective, it creates several problems, such as (1) every connection
between two applications requires custom programming; (2) a lot of
connections have to be developed when there are multiple data sources;
and (3) whenever the logic or data in one application changes, the
accompanying interface often also needs to be altered. Middleware
represents an alternative means to the traditional approach, and it can
provide a quicker and more robust solution to systems integration. In
essence, middleware separates the business application from the
technical details of interapplication communications. Thus, middleware
can simplify and reduce the number of interfaces for multiple systems
because it can handle differences in data formats and record layouts.
As part of the COD process, XML is being used to consolidate multiple
legacy record formats previously used by schools to submit data on the
Pell Grant and Direct Loan programs.[Footnote 15] By using an XML-based
common record, schools can transmit one file with all of the student‘s
data instead of submitting separate legacy records with redundant
student and school information.
Appendix I provides a high-level depiction of the systems and
technologies supporting the COD process as of November 2002. As
depicted, the COD system can translate or convert legacy records by
using middleware. In addition, middleware has been built into several
existing systems so that they can establish connectivity and exchange
data with the COD system through a common IT infrastructure. This IT
infrastructure, called the Enterprise Application Integration (EAI)
bus, is also implemented using middleware to route data between systems
in a correct format. In addition, as part of the COD process, some
schools have begun submitting Pell Grant and Direct Loan data using the
XML-based common record.
FSA hired Accenture as its ’modernization partner“ to help carry out
its modernization blueprint, including the implementation of the COD
process. Accenture is the prime contractor[Footnote 16] providing
leadership of critical planning activities that are essential to the
success of FSA‘s modernization. Regarding the COD system part of FSA‘s
modernization, FSA also hired an independent verification and
validation contractor to review the initial release of this system,
which was completed earlier this year.
Progress in Implementing the COD Process, but Critical Work Remains,
and Benefits and Lessons Learned Are Not Being Tracked:
FSA has made progress in implementing the new COD process. In
particular, it has begun implementing (1) its middleware solution in
its IT infrastructure and various existing systems, (2) the COD system,
and (3) an XML-based common record. However, FSA‘s implementation of
COD
is behind schedule, and critical work remains to be completed. For
example, the basic COD system was to be completed by mid-October 2002;
however, only about three-quarters of the COD basic system requirements
had been implemented as of October 23, 2002. In addition, FSA is not
tracking whether it is achieving certain benefits because it is still
in the process of defining applicable metrics to measure progress.
Without such tracking processes, FSA lacks critical information about
whether it is achieving expected benefits. Finally, FSA lacks assurance
that it has captured and disseminated important lessons learned related
to schools‘ implementation of the common record because it believes
that its current ad hoc process is adequate. Accordingly, the thousands
of schools that have not yet implemented the common record may not
benefit from the experience of those that have.
FSA Has Begun Implementing COD Capabilities, but Critical Work Remains:
FSA has made progress in implementing COD. The following are
significant elements of the COD process that have been implemented:
* Deployment of the EAI bus. As a prerequisite to implementing COD, in
late October 2001, FSA deployed its middleware solution in an EAI
’bus“--an IT infrastructure that uses middleware to access data from
disparate systems, transform the data formats as necessary, and route
the data to the appropriate requesting systems, thus enabling data
exchange among disparate systems. The EAI bus provides the set of
technical capabilities necessary to integrate FSA‘s disparate systems.
* Initial implementation of basic COD system (release 1.x).[Footnote
17] On April 29, 2002, FSA went live with version 1.0 of the basic COD
system. As of mid-November 2002, FSA had released an additional five
sub-versions of the COD system (e.g., version 1.1). The COD system
replaces the Direct Loan Origination System and the Recipient Financial
Management System, and it currently processes files for all schools
participating in the Pell Grant and Direct Loan programs.[Footnote 18]
According to FSA, in the first 6 months of its operation, the COD
system processed just under 16 million transactions, representing Pell
Grant and Direct Loan awards totaling almost $10 billion to over 5
million recipients.
* Implementation of middleware in selected systems. As of mid-November
2002, FSA had built middleware into seven systems so that these systems
can interact with the COD system through the EAI bus. These systems
include (1) the Central Processing System, which determines students‘
eligibility and award levels, and (2) the National Student Loan Data
System, which contains loan-and grant-level information and is used by
schools to screen student aid applicants to identify ineligible
borrowers.
* Development and implementation of the common record. Using XML, FSA
developed and began implementing a common record that schools can use
to submit student financial aid data to the COD system. The common
record, designed with assistance from members of the National Council
of Higher Education Loan Programs and the Postsecondary Electronic
Standards Council, consolidates multiple legacy file formats previously
used by the Pell Grant and Direct Loan programs.[Footnote 19]
:
Although FSA has made progress in implementing the COD process,
critical work remains to be completed. First, FSA is behind schedule in
implementing the basic COD system. Although FSA had planned to complete
the basic COD system by mid-October 2002, only about three-
quarters[Footnote 20] of the COD basic system requirements had been
implemented as of October 23, 2002.[Footnote 21] For example, as of
early November 2002, one of the basic business functions that remains
to be implemented is to enable FSA to make automated adjustments in
batches to school current funding levels. FSA now estimates that most
of the remaining functionality will be completed by the end of
September 2003. According to FSA IT and program officials, the
implementation of the basic COD functionality was delayed to allow
adequate time for testing to ensure the quality of the system.[Footnote
22]
Second, as of November 19, 2002, Accenture reported several operational
problems that needed to be addressed. For example, in some cases, the
COD system was incorrectly processing school batch data that contained
multiple change records for an individual student. According to COD and
contractor officials, the causes of operational problems included
unclear requirements and software design defects. An independent
verification and validation contractor also found problems with the
requirements and design aspects of release 1.0. The COD Contracting
Officer‘s Representative characterized these operational problems as
very serious and stated that they could impede operations and the
delivery of future COD releases. This same official noted that FSA and
Accenture are currently undertaking efforts to address these problems.
For example, FSA has established production teams composed of agency
and contractor staff to address problems in specific areas. In
addition, FSA has established a continuous improvement process to more
rigorously manage its relationship with Accenture.
Third, fewer postsecondary schools than planned have implemented the
common record. FSA had estimated that 50 schools (out of about 5,500)
would implement the common record in fiscal year 2002. However, as of
November 26, 2002, only 22 schools[Footnote 23] had implemented and
tested the common record with FSA. FSA COD officials attributed the
fewer-than-expected number of schools using the common record to
schools and vendors not being ready to implement it. FSA expects that
the number of schools using the common record will be considerably
higher during the next award year (2003-2004) because, by April 2003,
it plans to implement and test the common record with EDExpress, a
software application FSA distributes free of charge to about 3,000
schools for use in submitting data. In addition, FSA expects that all
schools will be using the common record format by March 2004, in time
for the 2004-2005 award year.
FSA Is Not Completely Tracking Actual COD Benefits or Lessons Learned
Related to Schools‘ Implementation of the Common Record:
In its COD business case, FSA outlined five expected benefits:
(1) reduced cost, (2) increased customer satisfaction, (3) increased
employee satisfaction, (4) increased financial integrity, and (5) the
integration and modernization of legacy systems. An important aspect of
implementing an IT investment cited by the Office of Management and
Budget[Footnote 24] and our IT investment management guide[Footnote 25]
is evaluating the results of the investment by determining whether such
expected benefits are being achieved. However, as illustrated in table
1, at this time FSA has only some of the data necessary to determine
whether it is achieving all expected benefits. In particular, for the
increased customer satisfaction and financial integrity benefits, FSA
(1) has not fully defined the performance metrics to be used, (2) does
not have all baseline data, and/or (3) is not fully tracking whether
the benefits are being achieved. In these cases, FSA COD officials
stated that they were in the process of developing relevant metrics,
which would be tracked to measure the project‘s performance against
expected benefits. However, until FSA develops these data and begins
tracking actual benefits and comparing them with expected benefits, it
will lack vital data with which to demonstrate actual investment
results.
Table 1: Status of FSA Tracking of Actual Benefits of the COD System:
Expected benefit: Reduced cost; Metrics defined?: Yes--The COD business
case identifies decreased costs for systems operations, customer
service, and the virtual data center.; Baseline data available?: Yes;
Benefits tracked?: Yes[A].
Expected benefit: Increased customer
satisfaction; Metrics defined?: Partially--The COD business case
identifies the use of the American Customer Satisfaction Index scores
to demonstrate clarity of instructions, ease of submitting data, and
accuracy of records. However, FSA COD officials stated that at this
time, they did not have approval from the Department of Education to
perform this survey; therefore, they are trying to identify alternative
metrics that could be used to measure customer satisfaction
improvements due to COD.; Baseline data available?: Partially--
According to FSA, baseline data related to the American Customer
Satisfaction Survey exist, but other planned metric(s) are not yet
defined.; Benefits tracked?: No.
Expected benefit: Increased employee
satisfaction; Metrics defined?: Not applicable--The COD business case
identifies a metric for this expected benefit,
but in commenting on our draft report, FSA stated that it has decided
to discontinue the
use of employee satisfaction as a key performance measure and thus will
not be tracking this as an expected benefit of the
COD system.; Baseline data available?: Not applicable; Benefits
tracked?: Not applicable.
Expected benefit: Increased financial
integrity; Metrics defined?: Partially--FSA COD officials stated that
they have identified increased school compliance with its requirements
that schools substantiate the amount of funds withdrawn within certain
time frames as a metric. They also stated that they are currently
developing other performance metrics for increased financial integrity
but have not set a target
date for completing them.; Baseline data available?: Partially--
Baseline data related to the identified metric exist,[B] but other
planned metric(s) are not yet defined.; Benefits tracked?: Partially--
Tracking the identified metric only..
Expected benefit: Integration and modernization
of legacy systems; Metrics defined?: Yes--The COD business case
identifies a variety of quantitative and qualitative
measures, including the retirement of two legacy applications and the
simplification of testing and ongoing development.; Baseline data
available?: Yes; Benefits tracked?: Yes.
[A] In providing comments on a draft of this report, FSA‘s Chief
Operating Officer stated that after the completion of our review, FSA
had begun tracking this expected benefit against the metrics defined in
the business case. FSA also provided supporting documentation that
previously was not available. We did not validate these new data.
[B] In providing comments on a draft of this report, FSA‘s Chief
Operating Officer stated that estimated baseline data for the
identified metric are available, but FSA is working to develop more
formal baseline data for this performance metric.
Source: GAO analysis on the basis of FSA documentation.
[End of table]
FSA IT officials also stated that they plan to have a contractor
conduct a postimplementation review of the COD basic system in fiscal
year 2003, which is expected to look at the achievement of expected
benefits. While this is an important initiative that could provide FSA
with valuable information, it does not take the place of a continuing
and systematic process of tracking actual benefits.
According to our IT investment management guide, another critical
activity is establishing a process for developing and capturing lessons
learned in a written product or knowledge base and disseminating them
to decision-makers.[Footnote 26] Lessons-learned mechanisms serve to
communicate acquired knowledge more effectively and to ensure that
beneficial information is factored into planning, work processes, and
activities. Lessons learned can be based on positive experiences that
save money or on negative experiences that result in undesirable
outcomes.
FSA has recognized the importance of generating lessons learned in
certain areas. For example, it has implemented a process for developing
lessons related to managing the relationship between the agency and its
prime contractor. However, FSA lacks such a process for capturing or
disseminating lessons related to school migration to the common record.
FSA COD officials stated that lessons learned pertaining to school
migration to the common record are addressed through periodic
discussions during biweekly conference calls with schools undergoing
testing with FSA and during portions of various FSA-sponsored
conferences. FSA COD officials stated that they believed this process
for capturing and disseminating lessons learned was adequate.
However, by relying on such an ad hoc process, FSA lacks assurance that
it has captured and disseminated all key lessons learned related to
schools‘ implementation of the common record and could overlook
important improvements that could be made. In addition, schools that do
not attend the conferences may not receive and benefit from the lessons
identified in the initial phase of implementation. As a result, schools
may encounter problems that could have been avoided or mitigated had
they known of other schools‘ experiences. This could hamper FSA‘s
ability to facilitate the transition of schools to the new common
record and thus the agency‘s ability to fully implement the new COD
process and achieve the expected benefits.
In commenting on a draft of this report, FSA stated that it plans to
provide lessons learned as part of a planned update to its school
testing guide. While this is a positive step, it does not replace the
need for mechanisms to continuously capture and disseminate acquired
knowledge as schools implement the common record.
Table 2 includes examples of lessons learned provided by FSA at our
request that were drawn from schools‘ initial implementation of the
common record for the 2002-2003 award year. Such information would be
important for the thousands of schools that have not yet implemented
the common record so that they can avoid problems during the common
record implementation and testing processes.
Table 2: Examples of Lessons Learned Related to School Migration to the
Common Record:
Lesson learned: Several schools and vendors did not have the technical
staff to support the switch to XML.; Effect: FSA had fewer schools
participating in testing than initially expected..
Lesson learned: Many schools had only a technical contact or a business
contact, but not both.; Effect: Instructions FSA provided to schools
may not be translated properly between the business and technical
contacts..
Lesson learned: Schools did not sufficiently test their applications
before testing them with COD.; Effect: Schools and vendors required
more support from FSA than anticipated..
Lesson learned: Service-level expectations were not well-defined before
the start of testing and were not made available to all
parties involved.; Effect: Schools had unrealistic expectations of the
turnaround time required to process their records..
Lesson learned: Schools should use an XML tool to validate school XML-
based applications before testing with the COD system.; Effect: Schools
that did not participate in such validations had more errors and a
greater chance of their records being rejected upon submission to the
COD system..
Source: FSA.
[End of table]
Conclusions:
FSA has taken important steps toward achieving full implementation of
the new COD process. However, critical actions, such as completing the
basic functionality of the COD system and the implementation of the
common record at thousands of affected schools, must still be
undertaken. In addition, FSA has not yet fully established the metrics
and processes to track actual benefits related to all of its expected
benefits or the lessons that have been generated by the few schools
that have implemented the common record thus far. By not tracking
actual benefits, FSA lacks information that is critical to determining
whether it is meeting all of its goals. Further, not capturing and
disseminating information to schools regarding lessons learned could
make achieving these goals more difficult.
Recommendations:
To determine the extent to which the new COD process is achieving
expected results related to customer satisfaction and financial
integrity, we recommend that you direct FSA‘s Chief Operating Officer
to expeditiously develop metrics and baseline data to measure these
benefits and develop a tracking process to assess the extent to which
the expected results are being achieved.
To ensure that the schools that have not yet implemented the common
record benefit from the experiences of those that have, we recommend
that you direct FSA‘s Chief Operating Officer to establish a process
for capturing lessons learned in a written product or knowledge base
and for disseminating them to these schools.
Agency Comments and Our Evaluation:
In providing written comments on our draft report, FSA‘s Chief
Operating Officer provided technical comments and updated information,
but did not comment on our recommendations. Specifically,
* The Chief Operating Officer did not believe the report adequately
portrayed the level of COD progress that had been made. In particular,
she took issue with our using the completion of 75 percent of COD
requirements as an indication of progress. Although the Chief Operating
Officer did not disagree with the accuracy of this figure, she stated
that FSA‘s informal analysis indicated that between 85 to 90 percent of
COD functions had been implemented, which she believed was a better
gauge of progress. We believe that we have accurately portrayed FSA‘s
progress in implementing the COD process. First, since FSA‘s analysis
was ’informal,“ and FSA‘s supporting documentation had limited detail
that we could not validate, we do not agree that this should be the
primary basis for an analysis of COD‘s progress. Second, we included
both the percentage of COD‘s requirements that had been implemented and
FSA‘s estimate in our report. Nevertheless, we modified our report to
include additional data provided by FSA regarding the number of
transactions processed by the COD system to further indicate progress.
:
* The Chief Operating Officer agreed that the tracking of all of the
expected benefits is not in place at this time, but stated that work is
under way in this area. FSA also provided updated information and
supporting documentation related to the tracking of some of the
expected benefits. We made changes to the report reflecting this new
information, as appropriate.
:
* The Chief Operating Officer agreed that it is important and
beneficial to communicate lessons learned, but stated that FSA‘s
informal method for communicating lessons related to school migration
to the common record worked well in the first year of COD
implementation. FSA also noted that it plans to include lessons learned
in a planned update to its school testing guide. We modified the report
to reflect this initiative, but we do not agree that FSA‘s informal
method or its plan to include lessons learned in its testing guide is
adequate because these approaches do not provide a continuous process
for actively capturing and disseminating lessons learned. As a result,
some important lessons may be overlooked, and all schools may not be
aware of potential problems associated with implementing the common
record.
FSA‘s written comments, along with our responses, are reproduced in
appendix II.
We are sending copies of this report to the appropriate congressional
committees, the Secretary of Education, the Chief Operating Officer of
Education‘s Office of Federal Student Aid, and the Director of the
Office of Management and Budget. This report will also be available at
no charge on the GAO Web site at http://www.gao.gov.
If you have any questions on matters discussed in this report, please
contact me at (202) 512-9286 or Linda J. Lambert, Assistant Director,
at (202) 512-9556. We can also be reached by E-mail at pownerd@gao.gov
and lambertl@gao.gov, respectively. Other individuals making key
contributions to this report included Jason B. Bakelar and Anh Q. Le.
Sincerely yours,
Signed by David A. Powner:
David A. Powner
Director (Acting), Information Technology
Management Issues:
[End of section]
Appendix I Systems and Technologies Supporting the Common Origination
and Disbursement Process, as of November 2002:
[see PDF for figure]
[A] This is a temporary interface. DLOS is targeted to be retired in
fiscal year 2003.
[B] MQSeries is IBM‘s proprietary message format.
[C] This is an information technology infrastructure that enables data
exchange among disparate systems.
[End of image]
[End of section]
Appendix II Comments from the Department of Education‘s Office of
Federal Student Aid:
UNITED STATES DEPARTMENT OF EDUCATION:
Federal Student Aid Chief Operating Officer:
December 18, 2002:
Mr. David A. Powner Director (Acting),
Information Technology
Management Issues
United States General Accounting Office
Washington, DC 20548:
Dear Mr. Powner:
I am writing in response to your request for comment on the draft GAO
report to Secretary Paige on the implementation of the Common
Origination and Disbursement (COD) system by Federal Student Aid. The
draft report is entitled, ’Federal Student Aid: System Integration
Progressing, But Critical Work Remains.“ Our understanding was that the
purpose of this audit was refocused to specifically examine FSA‘s
progress in implementing the Common Origination and Disbursement System
(COD), and did not include an examination of FSA‘s system integration
plan nor progress against that plan. Thus, we suggest that GAO change
the title to make it clear that the audit was limited to an examination
of the implementation of COD. In addition, we ask you to consider the
information provided below as you finalize this report.
Highlights, Results in Brief and Background:
In the ’Highlights“ section, GAO makes reference to ’serious post-
implementation operational problems“. However, no such problems are
identified in the report. Further, in the ’Results in Brief‘ section on
page 2 of the report, while GAO states that FSA has made progress in
implementing the COD process, and describes four very significant
accomplishments, the information that follows is out of context, and
suggests that progress made has been minimal. Progress of significance
that should be reflected in the report is that in the first six months
of COD operations, the system has processed just under 16 million
transactions. Those transactions report Pell Grant and Direct Loan
awards totaling almost $10 billion to over 5 million recipients. While
we certainly acknowledge that the system has had problems, the fact is
that we have processed millions of records for billions of dollars, and
have given the right money to the right people.
In addition, the report on page 3 states that ’...the COD process is
behind schedule, and its ultimate success hinges on FSA‘s completing
critical work, including having thousands of post-secondary schools
implement the common record.“ This statement does not take into
consideration that all post-secondary educational institutions
participating in the Pell Grant and/or Direct Loan Programs
(approximately 5,500) are current users of the COD system,
whether common record users or not. We believe that the GAO report
should note that middleware functionality enables FSA to process all
Pell Grant and Direct Loan record formats so that our schools can
implement the XML formatted common record as their schedule permits.
The common record, while new and innovative is only one of the
important parts of the COD and it has always been our published
intention that schools migrate to using the common record over a three-
year period. Since the inception of the Commonline file format, which
was first introduced many years ago, it has been standard student aid
industry practice to have a planned phase-in of new file formats and
processes. This is widely viewed as risk mitigation, and in fact, the
industry supports that mitigation by supporting several versions of
file formats while its customers adapt to the new process. GAO seems to
be implying that phasing in the implementation of the common record was
actually an INCREASE in risk, while FSA feels it was prudent as a way
to DECREASE risk.
The report on page 3 also discusses in brief the findings of the review
that we shall address in the appropriate more detailed section of the
draft report. This discussion, as well as the Highlights section,
should be revised to reflect changes as a result of the comments that
follow.
Progress in Implementing COD Process, But Critical Work Remains and
Benefits and Lessons Learned Not Being Tracked:
On page 6 and again on page 8 of the report, GAO states that as of
October 22 only about three quarters of the COD basic system
requirements had been implemented. This is based on an arithmetic
calculation using the number of requirements implemented versus the
total number of requirements. We believe it is important to note the
relative functional value of the implemented and non-implemented
requirements, because in fact, if only 75 percent of the critical COD
functionality were in place it is doubtful that so many Pell Grants and
Direct Loans could have been processed through the current date.
As previously discussed with GAO staff, FSA COD IT development managers
performed an informal analysis of the percentage of COD functionality
that was implemented by the end of September 2002, as well as provided
an analysis of the requirements in the initial 1.0 COD software release
against the contract‘s 1.0 acceptance criteria. In both cases, the
development managers provided an estimate that between 85-90 percent of
the functionality was implemented and that the critical functionality
necessary to process records had been implemented.We have replicated
virtually all of the existing functionality of the two previous systems
into an integrated common system, as well as provided a number of
enhancements that please our customers and improve financial integrity.
It seems to FSA that our goal of integrating existing systems into a
common system has been met, although we recognize that there is support
functionality that still needs to be completed.
FSA Not Completely Tracking Actual COD Benefits or Lessons Learned
Related to Schools‘_ Implementation of the Common Record:
On pages 3, 6, and in detail beginning on page 9 of the report, the GAO
states that FSA is not tracking whether it is achieving certain
expected benefits (as discussed in the FSA COD system business case).
We agree that all of the tracking of the expected benefits is not in
place at this time but work on all is underway. The following is an
update to each of the items listed in Table 1 on page 11.
Reduced Cost. The Contracting Officer‘s Representative for the COD
system has been tracking monthly COD costs for the 6 months of COD
operations and has compared those costs to the baseline legacy system
costs. However, as noted in the report, those costs were not broken
down into the component business case costs of Operations, Customer
Service, and virtual data center costs at the time of the GAO review
team data collection. As part of our response, we are providing an
excel spreadsheet (Attachment 1) containing such a comparison for the
five months of COD operations in FY 02 with the understanding that this
is an on-going tracking of this expected benefit.
Increased Customer Satisfaction. Although FSA has for several years
conducted a valid and reliable survey of its customers, the Department
intends to modify the approach to, among other things, eliminate
duplicate surveys of the Department‘s customers. During this
transition, no survey was conducted in FY2002, so we have no data for
this indicator. However, we certainly intend to measure customer
satisfaction, and as you noted, we are in the process of identifying
alternative metrics that could be used to measure customer satisfaction
improvements due to COD and we agree that it may be necessary to alter
the COD contract to do so.
Increase Employee Satisfaction. FSA has decided to discontinue employee
satisfaction as a key performance measure for the organization and
therefore, will not be using this activity as an expected benefit of
the COD system.
Increased Financial Integrity. FSA is tracking the performance of
participating institutions against the key financial requirement of
reporting actual disbursements of Pell Grants and Direct Loans within
30 days of drawing the funds down. GAO was provided with the procedures
being employed by the COD Customer Service Center to monitor and act on
the resulting performance data related to this key financial integrity
measure. We are attaching a copy of the most recent weekly monitoring
report (Attachment 2). This report demonstrates for the week of 12/02/
02-12/06/02, a total of 421 of 5,500 (approximately 7.6%) participating
schools are outside the 30-day requirement. We believe that this result
is very noteworthy given that we have deliberately not aggressively
pursued the schools compliance with this requirement this early into
the implementation of COD. This 7.6% is compared to an estimate of 15-
20% of schools being non-compliant with that requirement for any given
period under the legacy systems. FSA is working to develop a more
formal baseline set of numbers for this performance measure.
Lessons Learned Related to School Testing. On page 12 in the report,
GAO took issue with the manner in which lessons learned during school
testing were being communicated to the 22 full participant schools. FSA
clearly understands the importance and benefits that can be derived by
communication of lessons learned. FSA employed a more informal method
of communicating lessons learned because of the limited number of full
participant schools in this inaugural year of COD implementation. This
process worked well for this year, but FSA is providing a more formal
method of disseminating this type of information to full COD
participant schools that will test in the upcoming 03-04 COD year. GAO
was provided with a set of slides used at the most recent FSA sponsored
Electronic Access Conference held this month. One of the slides was
labeled School Testing Lessons Learned (Attachment 3). In addition, the
03-04 School COD Testing Guide under development will have a section on
lessons learned from the schools and financial aid software vendors
that
tested with COD in the 02-03 COD year. We expect that document to be
available in January 2003.
Thank you for this opportunity to comment on the draft report on the
implementation of COD.
Sincerely,
Signed by Theresa S. Shaw:
Theresa S. Shaw:
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
See comment 3.
See comment 2.
See comment 1.
See comment 5.
See comment 4.
See comment 6.
See comment 8.
See comment 7.
The following are GAO‘s comments on the Department of Education‘s
Office of Federal Student Aid‘s letter dated December 18, 2002.
GAO Comments:
1. We revised the draft report title to clarify that our follow-up
review was focused on assessing FSA‘s progress in implementing the COD
process.
2. Information related to operational problems was contained in the
draft report. We asked the COD Contracting Officer‘s Representative to
characterize these problems, and he stated that they were very serious.
In addition, we confirmed the seriousness of these problems at the
conclusion of our review with FSA IT and program officials.
3. We modified this report to include data on the number of
transactions processed. We also modified our report to clarify that all
schools participating in the Pell Grant and/or Direct Loan programs
currently use the COD system.
4. We do not agree that the report implies that FSA‘s use of a phased-
in approach in implementing the common record increases risks. Instead,
this report notes that the implementation is not yet complete.
5. We believe that we have accurately portrayed FSA‘s progress in
implementing the COD process. First, since FSA‘s analysis was
’informal,“ and FSA‘s supporting documentation had limited detail that
we could not validate, we do not agree that the COD Development
Manager‘s functionality estimate should be the primary basis for an
analysis of COD‘s progress. Second, we included both the percentage of
COD‘s requirements that had been implemented and FSA‘s estimate in our
report.
6. We modified our report to reflect this updated information as
appropriate.
7. We do not agree that FSA‘s informal process for capturing and
disseminating lessons learned was adequate because (1) it may lead to
important lessons being overlooked and (2) all schools may not be aware
of potential problems associated with implementing the common record.
8. We modified this report to reflect that FSA plans to include lessons
learned in a planned update to its school testing guide. While this is
a positive step, it does not replace the need for mechanisms to
continuously capture and disseminate acquired knowledge as schools
implement the common record.
FOOTNOTES
[1] Information integration is defined by the National Institute of
Standards and Technology as the establishment of the appropriate
computer hardware/software, methodology, and organizational
environment to provide a unified and shared information management
capability for a complex business enterprise.
[2] U.S. General Accounting Office, Student Financial Aid: Use of
Middleware for Systems Integration Holds Promise, GAO-02-7 (Washington,
D.C.: Nov. 30, 2001).
[3] The Higher Education Amendments of 1998, which amended the Higher
Education Act of 1965, states that the responsibilities of the
performance-based organization include integrating the information
systems supporting federal student financial assistance programs;
implementing an open, common, and integrated system for delivery of
student financial assistance under title IV; and developing and
maintaining a student financial assistance system that contains
complete, accurate, and timely data to ensure program integrity.
[4] Financial aid programs are administered by an office previously
known as the Office of Student Financial Assistance, which was changed
to the Office of Federal Student Aid on March 6, 2002.
[5] Middleware is a type of software that can allow an application to
access data residing in different databases. In addition, middleware
can enable dissimilar systems to communicate and work together as if
they all resided on a single platform.
[6] XML is a flexible, nonproprietary set of standards that offers the
promise of making it significantly easier to identify, integrate, and
process information that is widely dispersed among systems and
organizations.
[7] GAO-02-7.
[8] Accenture was responsible for providing the project management,
software, testing, and operation of the technology elements of the COD
process. However, as used here, we ascribe the action to FSA because it
made the decision to accept and deploy the contractor‘s work.
[9] Campus-based programs, which include the Federal Work-Study
Program, the Federal Perkins Loan Program, and the Federal Supplemental
Educational Opportunity Grant Program, are administered jointly by FSA
and postsecondary educational institutions.
[10] For example, the FSA systems environment includes operating
platforms, such as IBM OS/390 mainframe, Sun Solaris on Sparc, and
Windows NT, and network protocols, such as the Transmission Control
Protocol/Internet Protocol and Systems Network Architecture.
[11] U.S. General Accounting Office, Student Financial Aid Information:
Systems Architecture Needed to Improve Programs‘ Efficiency, GAO/
AIMD-97-122 (Washington, D.C.: July 29, 1997).
[12] FSA‘s modernization blueprint describes the agency‘s business
requirements, business and technical architecture, and sequencing plan.
[13] The other three processes are program eligibility, program
support, and financial transactions.
[14] For additional information on middleware and how it works, see
GAO-02-7 and North Carolina Information Resource Management Commission,
North Carolina Statewide Technical Architecture, which can be found at
http://irm.state.nc.us/techarch/chaps/pdffiles/pdflist.htm.
[15] For additional information on XML and how it works, see U.S.
General Accounting Office, Electronic Government: Challenges to
Effective Adoption of the Extensible Markup Language, GAO-02-327
(Washington, D.C.: Apr. 5, 2002).
[16] Accenture also uses subcontractors to perform some of the
requirements related to this contract.
[17] ’X“ represents the sub-version number of the COD system. As of
mid-November 2002, FSA planned to deploy a total of seven sub-versions
of release 1.x.
[18] FSA plans to keep these two systems functioning until it (1)
completes the reconciliation of prior years‘ processing for direct
loans, which is expected to occur by the end of this fiscal year, and
(2) migrates the Recipient Financial Management System data to COD,
which is expected by the summer of 2003.
[19] The common record also includes data blocks for campus-based
programs to simplify school reporting. At this time, schools cannot
report these data to FSA because the campus-based program part of the
COD system is not yet available. According to a COD official, the
campus-based data blocks in the common record are expected to be
enabled in the summer of 2003.
[20] This percentage does not include functions initially planned to be
included in the COD system but subsequently cancelled.
[21] In addition, on September 4, 2002, the COD Development Manager
estimated that Accenture had met about 88 percent of the COD production
acceptance criteria.
[22] Once release 1.x is completed, FSA plans to implement other
functionality, including campus-based program reporting.
[23] In addition, as of mid-October 2002, five vendors had implemented
and tested the common record with FSA. Although FSA does not know how
many schools are serviced by these vendors because the vendors consider
this information proprietary, some of the 22 schools that have
implemented the common record use these vendors.
[24] Office of Management and Budget Circular A-130, Management of
Federal Information Resources (Nov. 30, 2000).
[25] U.S. General Accounting Office, Information Technology Investment
Management: A Framework for Assessing and Improving Process Maturity,
GAO/AIMD-10-1.23, Exposure Draft (Washington, D.C.: May 2000).
[26] GAO/AIMD-10-1.23.
GAO‘s Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO‘s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO‘s Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ’Today‘s Reports,“ on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select ’Subscribe to daily E-mail alert for newly
released products“ under the GAO Reports heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548:
