Immigration Services
Better Contracting Practices Needed at Call Centers Gao ID: GAO-05-526 June 30, 2005The U.S. Citizenship and Immigration Services (USCIS) bureau within the Department of Homeland Security (DHS) provides toll-free telephone assistance through call centers to immigrants, their attorneys, and others seeking information about U.S. immigration services and benefits. As the volume of calls increased--from about 13 million calls in fiscal year 2002 to about 21 million calls in fiscal year 2004--questions were raised about USCIS's ability to ensure the reliability and accuracy of the information provided at call centers run by an independent contractor. This report analyzes: (1) the performance measures established by USCIS to monitor and evaluate the performance of contractor-operated call centers; (2) how performance measures were used to evaluate the contractor's performance; and (3) any actions USCIS has taken, or plans to take, to strengthen call center operations.
USCIS developed seven performance measures intended to assess the performance and overall quality of responses provided by customer service representatives at contractor-operated call centers. These measures include how quickly calls were answered and the accuracy of information provided. The contract between USCIS and its contractor stipulated that the contractor could earn financial incentive awards if the average monthly performance met or exceeded the standards on a quarterly basis at each of four call centers. Conversely, financial deductions could be made if the standards were not met. USCIS did not finalize the terms regarding how the contractor's actual performance would be calculated, or scored, before awarding the contract. This limited USCIS's ability to exercise performance incentives (positive or negative) because the parties could not reach agreement on performance terms. USCIS suspended the use of financial incentives while the parties negotiated the issue. Agreement was not reached after 16 months, however, USCIS determined that the contractor had failed to meet standards for 4 of the 7 performance measures in the fourth quarter of 2004 and took action to reduce its payments for services. The contractor objected, citing the lack of agreement on the performance measurements and the impact of workload increases, but USCIS disagreed and stated it would reduce payment. In a separate but related matter, USCIS failed to meet contractual, regulatory, and GAO standards pertaining to how the contractor's performance would be documented--especially with respect to any deficiencies. Finally, USCIS exercised its option to extend the call center contract through May 2006, to allow time to solicit and award new call center contracts. USCIS said it intends to finalize performance measurement terms in the new contracts. USCIS used contractor performance data it collected over the course of the contract to identify opportunities to improve customer service and call flow, among other things. Several initiatives were launched as a result.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-05-526, Immigration Services: Better Contracting Practices Needed at Call Centers
This is the accessible text file for GAO report number GAO-05-526
entitled 'Immigration Services: Better Contracting Practices Needed at
Call Centers' which was released on June 30, 2005.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features, such
as text descriptions of tables, consecutively numbered footnotes placed
at the end of the file, and the text of agency comment letters, are
provided but may not exactly duplicate the presentation or format of
the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
June 2005:
Immigration Services:
Better Contracting Practices Needed at Call Centers:
GAO-05-526:
GAO Highlights:
Highlights of GAO-05-526, a report to congressional requesters:
Why GAO Did This Study:
The U.S. Citizenship and Immigration Services (USCIS) bureau within the
Department of Homeland Security (DHS) provides toll-free telephone
assistance through call centers to immigrants, their attorneys, and
others seeking information about U.S. immigration services and
benefits. As the volume of calls increased”from about 13 million calls
in fiscal year 2002 to about 21 million calls in fiscal year
2004”questions were raised about USCIS‘s ability to ensure the
reliability and accuracy of the information provided at call centers
run by an independent contractor.
This report analyzes: (1) the performance measures established by USCIS
to monitor and evaluate the performance of contractor-operated call
centers; (2) how performance measures were used to evaluate the
contractor‘s performance; and (3) any actions USCIS has taken, or plans
to take, to strengthen call center operations.
What GAO Found:
USCIS developed seven performance measures intended to assess the
performance and overall quality of responses provided by customer
service representatives at contractor-operated call centers. These
measures include how quickly calls were answered and the accuracy of
information provided. The contract between USCIS and its contractor
stipulated that the contractor could earn financial incentive awards if
the average monthly performance met or exceeded the standards on a
quarterly basis at each of four call centers. Conversely, financial
deductions could be made if the standards were not met.
USCIS did not finalize the terms regarding how the contractor‘s actual
performance would be calculated, or scored, before awarding the
contract. This limited USCIS‘s ability to exercise performance
incentives (positive or negative) because the parties could not reach
agreement on performance terms. USCIS suspended the use of financial
incentives while the parties negotiated the issue. Agreement was not
reached after 16 months, however, USCIS determined that the contractor
had failed to meet standards for 4 of the 7 performance measures in the
fourth quarter of 2004 and took action to reduce its payments for
services. The contractor objected, citing the lack of agreement on the
performance measurements and the impact of workload increases, but
USCIS disagreed and stated it would reduce payment. In a separate but
related matter, USCIS failed to meet contractual, regulatory, and GAO
standards pertaining to how the contractor‘s performance would be
documented”especially with respect to any deficiencies. Finally, USCIS
exercised its option to extend the call center contract through May
2006, to allow time to solicit and award new call center contracts.
USCIS said it intends to finalize performance measurement terms in the
new contracts.
USCIS used contractor performance data it collected over the course of
the contract to identify opportunities to improve customer service and
call flow, among other things. Several initiatives were launched as a
result.
USCIS Call Flow and Call Volume for Fiscal Year 2004:
[See PDF for image]
[End of figure]
What GAO Recommends:
To improve USCIS‘s evaluation of contractor performance, GAO recommends
that USCIS take steps to ensure that performance measurement provisions
are finalized before awarding new contracts and that performance
evaluation records are properly maintained. DHS generally agreed with
GAO‘s recommendations and indicated USCIS was taking steps to implement
them.
www.gao.gov/cgi-bin/getrpt?GAO-05-526.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Paul Jones at (202) 512-
8777 or jonespl@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
USCIS Established Performance Measures to Assess Contractor's Call
Center Performance:
USCIS Evaluated Contractor's Performance but Suspended Use of Financial
Incentives for More Than 2 Years Due to Performance Measurement
Dispute:
USCIS Has Used Results of Monitoring Efforts to Identify Opportunities
to Improve Customer Service and Call Flow at All Call Centers:
Conclusions:
Recommendation for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Criteria and Methodology Used by Contractor to Measure
Quality of Tier 1 Calls Monitored:
Appendix III: Criteria and Methodology Used to Measure Customer
Satisfaction with Customer Service Representatives:
Appendix IV: Criteria and Methodology Used by Independent Consulting
Firm to Measure Quality of Calls Monitored:
Appendix V: Mystery Shopper Scenario Example:
Appendix VI: Comments from the Department of Homeland Security:
Appendix VII: Comments from Pearson Government Solutions:
Appendix VIII: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Call Quality Monitoring or "Soft Skills"
Table 2: Accuracy of Information Provided:
Table 3: Accuracy of Capturing Information:
Figures:
Figure 1: Organization of USCIS's National Customer Service Center and
Call Centers:
Figure 2: Tier 1 Customer Service Representatives Processing
Immigration-Related Calls at a Contractor-Operated Call Center:
Figure 3: USCIS Call Flow and Call Volume for Fiscal Year 2004:
Abbreviations:
USCIS: U.S. Citizenship and Immigration Services:
COTR: contracting officer's technical representatives:
CSR: customer service representative:
DHS: Department of Homeland Security:
FAR: Federal Acquisition Regulation:
IIO: immigration information officer:
INS: Immigration and Naturalization Service:
NCSC: National Customer Service Center:
OFPP: Office of Federal Procurement Policy:
PRS: Performance Requirements Summary:
VA: Department of Veterans Affairs:
United States Government Accountability Office:
Washington, DC 20548:
June 30, 2005:
Congressional Requesters:
The U.S. Citizenship and Immigration Services (USCIS) bureau within the
Department of Homeland Security (DHS) provides telephone assistance to
customers calling about U.S. immigration services and benefits. USCIS
customers--immigrants and their attorneys or family members, employers,
and others--can call toll-free for information on such topics as how to
become U.S. citizens, obtain work visas, initiate the naturalization
process, obtain legal forms, and determine the status of applications
for benefits. This telephone assistance service helps customers avoid
making visits to USCIS offices, which could require customers to take
time off from work and travel long distances. Telephone calls for
assistance to USCIS have increased in recent years, from about 13
million in fiscal year 2002 to about 21 million in fiscal year 2004.
These calls are handled in a variety of ways: by an interactive voice
response system;[Footnote 1] by four call centers managed by a private
contractor (Pearson Government Solutions) engaged by USCIS; and by two
call centers operated directly by USCIS.
You expressed interest in how USCIS monitors and evaluates contractor-
operated call centers to ensure that they operate reliably and provide
accurate information to their growing number of customers. This report
addresses the following: (1) What performance measures did USCIS
establish to monitor and evaluate the performance of contractor-
operated call centers? (2) How were these performance measures used to
evaluate the contractor's performance? (3) What actions, if any, did
USCIS take or plan to take to strengthen call center operations?
To address these objectives, we obtained and analyzed information from
USCIS officials in Washington, D.C., from contractor officials in
Arlington, Virginia, and at one of four contractor-operated call
centers.[Footnote 2] At these locations, we interviewed officials and
collected and analyzed pertinent documentation, including descriptions
of monitoring and evaluation programs, contract requirements for call
center operations, and summaries of call center evaluation results. We
assessed the reliability of telephone call volume data provided to
USCIS by a telecommunications vendor, as well as USCIS and contractor
staffing data. To carry out our data reliability assessments, we (1)
reviewed information about the data, systems that produced the data,
and data quality control procedures, and (2) interviewed USCIS and
contractor officials knowledgeable about the data as necessary. We
determined that these data were sufficiently reliable for purposes of
this report.
We conducted our work between May 2004 and May 2005 in accordance with
generally accepted auditing standards. (See app. I for details about
our scope and methodology.)
Results in Brief:
USCIS developed seven performance measures that were intended to assess
the performance of and overall quality of responses provided by
customer service representatives at contractor-operated call centers.
These measures were described in a contract signed by both parties in
January 2002. Two performance measures, for example, were to assess the
accuracy of information provided to callers and how quickly calls were
answered. Based on provisions in the contract, the contractor could
earn financial incentive awards if certain standards or goals set for
each performance measure were met or exceeded on a quarterly basis at
each of the call centers--for instance, if callers waited an average of
30 seconds to 36 seconds for their calls to be answered. Conversely,
the contract provided for negative financial incentives (penalties),
whereby USCIS could deduct from payments owed to the contractor on a
quarterly basis if the standards were not met.
USCIS collected data on the contractor's performance on a monthly basis
since June 2002, when the contract was awarded. However, these data
were not used for the purpose of applying financial incentives (either
positive or negative) until the fourth quarter of 2004. The use of
financial incentives was suspended because USCIS awarded the contract
without finalizing how the contractor's actual performance would be
calculated, or scored, for the various performance measures. The two
parties negotiated on this issue over a period of about 16 months after
the contract was awarded before abandoning the effort. Thereafter,
USCIS determined that, for the fourth quarter of 2004, the contractor
had failed to meet standards for four of the seven performance
measures, and announced its intention to reduce payments to the
contractor for services. The contractor objected to USCIS's decision on
the grounds that the performance measurements had not been finalized
and that changes in call center workloads affected the basis for
applying financial incentives. USCIS disagreed and stated it would
reduce payment to the contractor. Although this dispute had not been
resolved, USCIS exercised its option to extend the current call center
contract through May 2006 to allow time to solicit and award new call
center contracts. The exercise of this option has no effect on the
contract's performance measurement terms, which is the source of the
parties' disagreement. In a separate but related matter, USCIS failed
to meet contractual, regulatory, and GAO standards pertaining to how
the contractor's performance was documented--especially with respect to
any deficiencies. For example, USCIS's failure to obtain the
contractor's written acknowledgement of USCIS-identified performance
deficiencies did not meet the notification procedures established by
the contract, documentation requirements of the Federal Acquisition
Regulations, and GAO's standards for internal control. Consequently,
USCIS did not maintain a complete and reliable record of the
contractor's performance needed to ensure accountability.
USCIS used the contractor performance data it collected monthly over
the course of the contract to identify opportunities to improve
customer service, help better respond to customer inquiries, and manage
the flow of calls into call centers. Several initiatives have been
launched as a result. For example, USCIS has implemented new call-
routing procedures designed, in part, to route customers' calls to
service representatives more quickly and reduce delays (on-hold time)
that arise when call volume is heavy. These and other initiatives were
undertaken by USCIS concurrent with its efforts to negotiate contract
differences with the contractor. It is too early to assess the impact
of these initiatives.
To improve USCIS's efforts for evaluating contractor performance and
encourage quality services at call centers, we recommend that the
Secretary of Homeland Security require the Director of USCIS take the
following two actions: (1) finalize contract terms related to specific
performance measurement requirements, before awarding new performance-
based call center contracts; and (2) maintain readily available written
records of performance assessments and performance evaluation meetings
with the contractor.
DHS and the contractor provided formal comments and technical comments
on a draft of this report, which we have incorporated, as appropriate.
DHS generally agreed with our recommendations. The contractor said the
report accurately summarizes the complex nature of CIS's call center
program and several challenges created by significant post-award
changes to that program.
Background:
The Homeland Security Act of 2002 established USCIS within
DHS.[Footnote 3] USCIS is responsible for several functions transferred
on March 1, 2003, from the former Immigration Services Division of the
Immigration and Naturalization Service (INS) under the Department of
Justice.[Footnote 4] These functions include providing services or
benefits to facilitate entry, residence, employment, and naturalization
of legal immigrants; processing applications for U.S.
citizenship/naturalization; and rendering decisions about immigration-
related matters.
Call Center Operations:
The USCIS Information & Customer Service Division is responsible for
operating the National Customer Service Center (NCSC), which was
established in 1997 to provide nationwide assistance by telephone to
customers calling about immigration services and benefits. When a
customer calls the NCSC toll-free number (1-800-375-5283), the call is
received by the interactive voice response system. The system features
automated, self-service options 24-hours a day, 7 days a week. If the
system cannot address a customer's concerns or needs or if a customer
requests live assistance, then the call is generally routed to one of
the four NCSC contract call centers, known as Tier 1.[Footnote 5] These
four centers are operated by the contractor, Pearson. If a question
posed by a customer is particularly complex or otherwise cannot be
answered at the Tier 1 level, the call is transferred to one of the two
USCIS-operated call centers, known as Tier 2.[Footnote 6] Figure 1
shows the organization of NCSC, including the call centers.
Figure 1: Organization of USCIS's National Customer Service Center and
Call Centers:
[See PDF for image]
[End of figure]
In fiscal year 2004, almost half of 21.1 million calls made to NCSC
were handled and completed by the interactive voice response system and
the rest were generally routed to Tier 1. Customer service
representatives (CSR) at Tier 1 respond to inquiries in English or
Spanish. The CSRs focus primarily on providing administrative
information to customers by using a series of scripts provided by
USCIS. For example, if a customer needs what USCIS considers basic
information, such as USCIS local offices' hours of operations,
eligibility requirements, and procedures to follow, such questions are
to be answered by CSRs at Tier 1 call centers using specific scripts.
In addition, CSRs are to refer customers to USCIS service centers and
local offices, for such things as changes of address and appointment
scheduling at USCIS application support centers. (Some of these tasks
may alternatively be performed by customers through the USCIS Web site-
-www.uscis.gov.) As of April 2005, the four Tier 1 call centers
employed over 450 CSRs.[Footnote 7] Figure 2 shows CSRs processing
calls at a Tier 1 call center.
Figure 2: Tier 1 Customer Service Representatives Processing
Immigration-Related Calls at a Contractor-Operated Call Center:
[See PDF for image]
[End of figure]
At the two USCIS-operated Tier 2 call centers, calls are handled by
immigration information officers (IIO)--immigration specialists with in-
depth knowledge of immigration laws, non-immigrant visas,
naturalization, asylum and refugee status, and other related policies
and procedures. As of April 2005, the Tier 2 call centers operated by
USCIS had 111 IIOs. About 5 percent, or about 590,000, of the calls
going to Tier 1 CSRs were rerouted to Tier 2 IIOs in fiscal year 2004.
Figure 3 shows the call volume handled by the interactive voice
response system, Tier 1 call centers, and Tier 2 call centers during
fiscal year 2004.
Figure 3: USCIS Call Flow and Call Volume for Fiscal Year 2004:
[See PDF for image]
[A] "Completed calls" refer to calls that have ended, not necessarily
to calls where customers' questions or issues have been resolved.
[End of figure]
Performance-Based Service Contract:
In January 2002, USCIS awarded a performance-based service contract for
the management of four Tier 1 call centers.[Footnote 8] In making this
award, USCIS obtained acquisition services from the Department of
Veterans Affairs (VA), and the contracting officer who signed and was
responsible for administering the contract was a VA employee working on
behalf of USCIS.[Footnote 9] The contracting officer's technical
representative (COTR), a USCIS employee, was also responsible for
administering the contract. In commenting on a draft of this report,
DHS noted that by agreement of both the VA and USCIS, on April 20,
2005, USCIS assumed responsibility for administering the contract. The
contract was awarded for a base year, beginning on June 1, 2002, plus 4
option years (1-year renewable extensions of the contract, three of
which have been exercised as of June 2005). Through calendar year 2004,
USCIS paid $64.6 million to the contractor for the Tier 1 call center
operations.
According to the Office of Federal Procurement Policy (OFPP) in the
Office of Management and Budget, performance-based service contracts
are designed to focus on results.[Footnote 10] Their purpose is to
ensure that contractors are given the freedom to determine how to meet
the government's performance objectives, that appropriate quality
levels are achieved, and that payment is made only for services that
meet these levels. This type of contract is to emphasize standards for
customer service and measurement of performance and may offer financial
incentives, both positive and negative, to encourage quality
performance. According to OFPP, call centers are suited to this type of
contract because, among other things, they emphasize achieving results
by meeting customer service standards. According to OFPP, with
performance-based service contracts, incentive payments made to an
independent contractor are to be contingent on the contractor's ability
to meet the government's performance standards; the contract does not
specify how those standards are to be met. Thus, the contractor retains
discretion in determining how to meet performance standards specified
in the contract, for example, how many CSRs to hire to ensure calls are
answered within a contractually specified time. Other elements
suggested for using a performance-based service contract include (1)
identifying the agency's needs and addressing those needs with
performance requirements that describe required service results; (2)
establishing performance standards that describe the required
performance level; and (3) establishing a quality assurance plan for
assessing contractor performance in order to ensure that the contractor
has performed in accordance with the standards.
USCIS Established Performance Measures to Assess Contractor's Call
Center Performance:
USCIS used a multi-faceted approach to monitor and evaluate the quality
of information and service provided by CSRs to customers calling
contractor-operated Tier 1 call centers.[Footnote 11] This approach
used seven performance measures. USCIS obtained performance data from
the contractor's monitoring of selected telephone calls; customer
satisfaction surveys; and a telecommunications vendor (telephone
company).[Footnote 12] In addition, USCIS used an independent
consulting firm to monitor CSRs' telephone calls and conduct a "mystery
shopper" program assessing CSRs' responses to customers.
USCIS Set Call Center Performance Measures for the Contractor to Meet:
In order to monitor and evaluate the performance of the four contractor-
operated Tier 1 call centers, USCIS planned to use seven performance
measures. These measures were to evaluate the quality of customers'
telephone interactions with CSRs; the accuracy of information provided
to callers over the telephone; the accuracy of callers' information
recorded by CSRs; callers' levels of satisfaction; how quickly CSRs
handled calls (two measures); and the number of calls abandoned by
customers put on hold. According to USCIS officials, USCIS established
the performance measures based on a review of industry standards for
both government and private-sector call center operations. The measures
were described in a section of the contract called the Performance
Requirements Summary (PRS).
Under the PRS, these performance measures comprised one of three
components upon which the contractor's performance score was based. The
other two components were the standard, or goal, set for each measure,
identifying the performance levels the contractor was expected to meet
(e.g., callers will wait an average of 30 to 36 seconds before their
calls are answered), and the performance calculation that USCIS would
use to analyze performance data (e.g., total delay of all calls divided
by the total number of calls).[Footnote 13] The PRS listing of the
seven performance measures included a "sample calculation" for each of
the measures, and stated that "actual calculations [are] to be
determined during Contract negotiations."[Footnote 14] USCIS officials
said they intended to negotiate and finalize the calculations after a 4-
to 6-month phase-in period, and the contract was awarded with this
provisional language.
As to the performance measures and their related standards or goals,
three of USCIS's performance measures are call quality monitoring,
accuracy of information provided, and accuracy of capturing
information. Data on these measures are to be collected by the
contractor's quality assurance staff, who are to randomly monitor two
calls per day for each CSR. (CSRs are not to know when they are being
monitored.) The data collected are to be reported to USCIS on a monthly
basis. Details on these three measures follow:
* Call quality monitoring. Calls are to be monitored by the
contractor's quality assurance staff to assess the CSRs' "soft skills,"
that is, their ability to interact with customers, establish customer
rapport, maintain composure during a call, speak with clarity and
professionalism, and other factors. Call quality monitoring data are to
be captured on a standardized form. CSR responses for each of nine
different "soft skills" are scored as percentages, with scores for the
most highly valued skills, such as "active listening"--that is, whether
the CSR was deemed to be attentive when listening to the customers--
given more weight than the scores for other skills. The nine scores
(i.e., percentages) are combined for a total "soft skills" score, with
100 percent as the highest possible score. The performance standard
stated in the PRS for this measure is that all calls monitored achieve
an average score of 90 percent to 95 percent after the nine "soft
skills" scores (i.e., percentages) for each call are combined. (See
app. II for additional details on the criteria and methodology used to
determine soft skills scores.)
* Accuracy of information provided. Calls are to be monitored by the
contractor's quality assurance staff to determine, among other things,
whether CSRs provided accurate and complete responses. Using a
standardized form, the staff score CSRs on five different efforts, such
as whether the CSR used software tools appropriately and whether, when
the callers were asked directly, they indicated that their needs had
been satisfied. The five efforts are scored as percentages, with more
weight given to the scores for certain efforts, such as "provides
complete response." The scores are then combined for a total "accuracy
of information provided" score, with 100 percent as the highest
possible score. The performance standard stated in the PRS for this
measure is that all calls monitored achieve an average score of 95
percent to 97 percent after the five accuracy scores (i.e.,
percentages) for each call are combined. (See app. II for additional
details on the criteria and methodology used to determine accuracy of
information provided.)
* Accuracy of capturing information. Calls are to be monitored by the
contractor's quality assurance staff to determine, among other things,
whether CSRs accurately record and verify the callers' information. The
staff assess this measure by scoring four efforts, including whether a
referral to a local USCIS service center or local office was completed
appropriately and correctly. The four efforts are scored as
percentages, with more weight given to the scores for certain efforts,
such as "verifies caller's information." The scores are then combined
for a total "accuracy of information provided" score, with 100 percent
as the highest possible score. The performance standard stated in the
PRS for this measure is that all calls monitored achieve an average
score of 95 percent to 97 percent after the four accuracy scores (i.e.,
percentages) for each call are combined. (See app. II for additional
details on the criteria and methodology used to determine accuracy of
capturing information.)
A fourth performance measure of call quality--customer satisfaction--
was assessed by an independent consulting firm. Customer satisfaction
surveys were conducted on a monthly basis to determine if customers
were satisfied with the service that CSRs provided.[Footnote 15] At
least 375 callers are to be randomly selected to be interviewed each
month from a population of 10,000 randomly identified callers who
called within the 30 days prior to the survey. To measure satisfaction
with CSRs, customer responses to four interview questions about CSRs
are compiled, and the overall percentage of respondents indicating
satisfaction is calculated. The performance standard stated in the PRS
for this measure is 80 percent to 85 percent of the customers surveyed
indicating overall satisfaction with the CSRs' service. (See app. III
for additional details on the criteria and methodology used to
determine customer satisfaction.)
Three other performance measures involve the collection of statistical
data by the telecommunications vendor for determining how quickly calls
are answered. The performance measures and standards in the contract
for assessing how quickly CSRs answered customers' calls are as
follows:
* Service level. The telecommunications vendor under contract with
USCIS is to collect information on the number of calls answered by CSRs
in 20 seconds or less, that is, the number of callers who spoke to a
CSR within 20 seconds after getting through the interactive voice
response system. The performance standard stated in the PRS for this
measure involves two factors: half-hour increments and the length of
time it took CSRs to answer calls. The standard is that for 80 percent
to 85 percent of the half-hour increments measured, 80 percent of the
calls are to be answered in 20 seconds or less.
* Average speed of answer. The telecommunications vendor under contract
with USCIS is to collect information on the length of time it takes for
CSRs to answer customers' calls after they are routed to Tier 1 by the
interactive voice response system; that is, how long callers are on
hold before a CSR answers their call. The performance standard stated
in the PRS for this measure is that, for all calls routed to Tier 1,
callers will wait an average of 30 seconds to 36 seconds.
* Abandoned calls. The telecommunications vendor under contract with
USCIS is to collect information on the number of calls abandoned by
customers after getting through the interactive voice response system
and waiting for a CSR to answer, that is, the number of times that
customers hang up the telephone while waiting for a CSR. The
performance standard stated in the PRS for this measure involves two
factors: half-hour increments and how frequently callers abandon their
calls. The standard is that for 85 percent to 95 percent of the half-
hour increments measured, 1 percent to 2 percent of the calls are
expected to be abandoned before a CSR answers.
The contract stated that the contractor would be eligible to earn
financial incentive awards if the average monthly performance met or
exceeded the standards on a quarterly basis at each call center, and
allowed USCIS to make deductions from payments to the contractor if the
average monthly performance fell below the standards.[Footnote 16]
According to the contract, the contractor is not eligible for an
incentive award for a particular quarter if one of the performance
standards is not met by one call center, and USCIS may make a deduction
from payments to the contractor in that case. In addition, USCIS may,
at its sole option, elect to include or waive financial incentives as
it deems appropriate.
USCIS Carried Out Additional Quality Assurance Efforts:
In addition to the performance data collected by the contractor's own
quality assurance staff, an independent consulting firm, and the
telecommunications vendor, USCIS took two additional steps to measure
call center performance for quality assurance purposes. First, to help
ensure that the contractor's scoring of call-quality performance
measures was reliable, USCIS used another independent consulting firm
to validate the results of the contractor's efforts by monitoring two
calls per month for each CSR. Data were gathered and provided to USCIS
on a monthly basis. (See app. IV for additional details on the criteria
and methodology used by the independent consulting firm to conduct call
monitoring.)
Second, in April 2003, USCIS engaged the same independent consulting
firm to carry out a "mystery shopper" program to assess the
completeness and accuracy of CSRs' answers to callers. Under this
program, an independent consultant places random calls--1,200 each
month--to Tier 1 call centers using various scripts provided by USCIS.
As of April 2005, the scripts used in these calls covered 32 different
scenarios, or types of calls, and 100 new scenarios were being
developed. The calls are conducted in English and Spanish. (See app. V
for an example of a mystery shopper scenario.)
USCIS Evaluated Contractor's Performance but Suspended Use of Financial
Incentives for More Than 2 Years Due to Performance Measurement
Dispute:
USCIS did not reach agreement with the contractor on how to apply the
performance measurement requirements described in the PRS before
awarding the performance-based service contract. USCIS suspended all
financial incentives, positive or negative, while the parties
negotiated this issue over a period of about 16 months without reaching
agreement. After negotiations were abandoned, USCIS determined that,
for the fourth quarter of 2004, the contractor had failed to meet four
of seven performance measures and merited a payment deduction. The
contractor disagreed on the grounds that the performance measurements
had not been finalized and that changes in call center workloads
affected the basis for applying financial incentives. In a separate
matter, USCIS failed to ensure that all contractual, regulatory, and
GAO standards pertaining to the documentation of the contractor's
performance were fulfilled.
USCIS Awarded the Call Center Contract without Finalizing Performance
Measurement Requirements:
The performance measurement requirements described in the PRS were not
completely finalized before the contract was awarded. The language
referring to "sample" calculations for determining how performance
would be measured remained in the contract after it was in force. In
commenting on a draft of this report, DHS said that at the time of the
contract award, USCIS management believed it was appropriate to let the
winning vendor have some input into the performance measurement
methodology since this contract represented a transition to performance-
based contracting for call center operations.
The negotiations between USCIS and the contractor on this issue began
in January 2003 (after a phase-in period) and continued intermittently
until April 2004, when they were abandoned. While negotiations were
taking place and after they were abandoned, USCIS obtained monthly data
relating to the contractor's performance on the seven performance
measures and compared those data to the standards.[Footnote 17] USCIS
considered the measures and standards themselves to be nonnegotiable;
the contractor, on the other hand, considered them as part of the
"sample calculations" and, thus, negotiable. For over 2 years, USCIS
did not use any of the resulting performance scores for the purpose of
calculating financial incentive awards or payment deductions under the
contract because the terms of the PRS remained unresolved between the
parties. The contractor maintained that the performance scores were
"potential scores" and were to be used by the parties in reaching an
agreement on how to structure the PRS.
On September 1, 2004, the contracting officer, representing USCIS, sent
a letter to the contractor advising that USCIS would begin evaluating
the contractor's performance and determining a financial incentive
award or payment deduction for the fourth quarter of the calendar year
(October 1 through December 31). USCIS officials told us they decided
to take this action because they had concluded that negotiations with
the contractor were unlikely to result in an agreement on the PRS. The
contractor objected to USCIS's decision to carry out this evaluation.
By letter dated November 29, 2004, the contractor stated that, under
the terms of the contract, USCIS could not unilaterally determine the
performance measurement requirements because all aspects of the
requirements were negotiable, including the performance standards.
The contractor further stated that an evaluation of its performance
must take into account certain changes that took place to the work
required under the contract. For example, the contractor stated that
the number of USCIS-provided scripts, containing information for CSRs
to address callers' inquiries, had grown to more than 2,300 pages from
approximately 400 script pages in June 2002.[Footnote 18] According to
the contractor, these changes significantly increased the average
amount of time needed to handle a call and affected the contractor's
ability to meet the performance standards imposed by USCIS. According
to the contractor, USCIS's unilateral imposition of performance
measurement requirements that did not account for the changed work
requirements was inconsistent with Federal Acquisition Regulation (FAR)
16.402(g), which provides that "[i]t is essential that the Government
and contractor agree explicitly on the effect that contract changes
(e.g., pursuant to the Changes clause) will have on performance
incentives."
Nevertheless, by letter dated February 11, 2005, USCIS's contracting
officer notified the contractor of the evaluation results for the
period of October through December 2004. The results showed that the
contractor met the standards for three of the seven performance
measures and did not meet the standards for the other four measures.
USCIS determined that, as a result of this performance, payments due to
the contractor for services would be reduced. The letter noted that the
contractor could submit its own data regarding performance during this
period. Following the review of any data submitted, USCIS would take
action to make the appropriate payment deduction, waive the payment
deduction, or pay an appropriate incentive award.
The contractor requested, by letter dated February 25, 2005, that USCIS
waive implementation of the financial incentives, both positive and
negative. The contractor reiterated its position that USCIS's
unilateral implementation of the performance measurement requirements
as currently written in the contract, without sufficient regard for
substantial changes to the contract and the changing nature of the
program, was not appropriate. The contractor stated that it was ready
to resume negotiations on this subject so that fair and equitable
financial incentives would be established. The contractor further
stated that it had determined the payment deduction was incorrectly
calculated by USCIS.
USCIS's contracting officer responded, by letter dated April 15, 2005,
that the government would not agree to waive implementation of the
financial incentives and a deduction would be made from the next
payment to the contractor. The letter stated that USCIS did not
unilaterally create and impose the performance measurement
requirements, which were included in the negotiated contract that USCIS
and the contractor agreed to. Regarding the contractor's assertion that
the average amount of time needed to handle calls had significantly
increased, the letter noted that the performance measurement
requirements would apply regardless of the average length of calls at
any given time.
According to FAR and OFPP guidance[Footnote 19] on performance-based
service contracting, the precise method for measuring performance
should have been agreed upon between USCIS and the contractor before
the contract was signed and implemented. FAR § 16.401 states that
performance-based service contracts should establish "reasonable and
attainable [performance] targets that are clearly communicated to the
contractor." According to OFPP, performance measurement techniques
(i.e., how performance will be assessed to determine whether standards
have been met) are essential elements of performance-based service
contracting and should be clearly stated. In addition, according to
OFPP, performance-based service contracts emphasize that all aspects of
an acquisition be structured around the purpose of the work to be
performed, that appropriate performance quality levels are achieved,
that payment is made only for services that meet these levels, and that
financial incentives are awarded to encourage quality performance.
USCIS Plans to Solicit New Tier 1 Call Center Contracts with Changes to
Improve Performance:
Although the disagreement between the two parties had not been
resolved, USCIS exercised its option to extend the current call center
contract for another year through May 31, 2006, to allow time to
solicit and award new call center contracts.[Footnote 20] The exercise
of this option has no effect on the contract's performance measurement
terms, which is the source of the parties' dispute. USCIS officials
said they plan to award new performance-based service contracts for
Tier 1 operations to two vendors, with the two vendors fully
operational by June 2006, to improve the handling of customers' calls
to Tier 1.[Footnote 21]
USCIS officials told us they intend for the new contracts to include
certain changes meant to improve Tier 1 call center operations and to
incorporate OFPP guidance on performance-based contracting. USCIS
officials told us that, unlike the current contract, the new PRS will
clearly specify how contractor performance will be assessed and will
not leave any terms open for post-award negotiation. In addition, USCIS
officials said the new contracts will include independent call
monitoring and the mystery shopper program as performance measurement
tools to assess the quality of the Tier 1 CSRs' responses to customers,
including the accuracy and reliability of the information provided. At
the time of our review, USCIS officials said that the solicitation was
going through DHS's contract review process and DHS had not issued the
solicitation for a new contract containing these changes. DHS said in
its comments on a draft of this report that the solicitation was with
the DHS Procurement Office for review and issuance.
USCIS and Contractor Did Not Fully Document Quality Assurance
Activities:
As part of its quality assurance responsibilities under the current
contract, USCIS is to keep written records of observations about the
contractor's performance based on periodic evaluations comparing
performance data to standards in the PRS. USCIS's contracting officer's
technical representative (COTR), who is responsible for administering
the contract, is to use these written observations to notify the
contractor if there are deficiencies--specifically, if the contractor
does not meet the performance standards. The contractor is required to
sign and date such observations to acknowledge that the COTR apprised
it of any deficiencies.
USCIS and contractor officials said they met at least quarterly
(monthly, since October 2004) to discuss performance, performance data,
and other items. USCIS officials said they provided the contractor with
documentation containing performance and other data to discuss at these
meetings. USCIS officials said some of this documentation identified
performance deficiencies. However, contractor officials said they
viewed the performance data as "potential scores" to be considered
during negotiations. To the extent that USCIS considered the
performance data as notification of deficiencies, it did not follow
contractual procedures requiring the COTR to obtain the contractor's
signature acknowledging notification of the deficiencies. In addition,
neither USCIS nor the contractor kept minutes of these meetings.
According to FAR § 46.104(c), the government should maintain, as part
of the performance records of a contract, suitable records reflecting
the nature of its contract quality assurance actions. With respect to
any performance deficiencies, the government's records should include,
among other things, the number and type of defects observed and any
actions to correct deficiencies. Further, according to GAO's standards
for internal control in the federal government, for an agency to run
and control its operations, it must have relevant, reliable information
relating to internal events.[Footnote 22] All transactions and other
significant events need to be clearly documented, and the documentation
should be readily available for examination. This information should be
recorded and communicated to management and others within the agency
who need it to carry out their responsibilities. GAO's standards
provide a framework for establishing and maintaining internal control
and for identifying and addressing major performance challenges.
Appropriate and effective internal control is a key factor in helping
agencies better achieve program results.
The contract also requires the contractor to provide a quality
assurance plan. The plan that was developed by the contractor describes
the contractor's approach and strategy for ensuring the delivery of
high-quality service. As part of the plan, the contractor is to conduct
formal, biweekly internal performance review meetings to help with the
identification and correction of performance deficiencies. These
meetings are to be attended by contractor and USCIS officials, with
contractor staff reporting on quality performance issues, and are to be
in addition to the quarterly (now monthly) meetings discussed above.
Minutes of the meetings are to identify action items, responsibilities,
and solution time frames, and the minutes are to be published for USCIS
review. However, a contractor official said that these meetings never
took place. According to both contractor and USCIS officials, the
quarterly meetings were used to discuss operations and performance and
to focus senior management attention on any performance issues.
Under FAR § 37.602-2, the government's quality assurance surveillance
plans should include actions to help ensure that the contractor carries
out its quality control obligations. By failing to ensure that the
contractor held and documented performance review meetings as required
by the contractor's quality assurance plan, USCIS did not meet its
quality assurance obligations under FAR § 37.602-2 and GAO's internal
control standards. In addition, USCIS's failure to obtain the
contractor's written acknowledgment of USCIS-identified performance
deficiencies did not meet the notification procedures established by
the contract, documentation requirements of FAR § 46.104(c), and GAO's
standards for internal control.
In its comments on a draft of this report, DHS noted that the contract
was administered by a component of the VA until April 20, 2005, and DHS
said the VA was provided with documentation discussed at quarterly and
monthly performance assessment meetings between USCIS and the
contractor. According to DHS, the lack of a clear understanding between
USCIS and the VA regarding their roles contributed to the fact that
formal documentation and evaluations were not always properly
maintained and formally transmitted to the contractor. DHS acknowledged
that the agency procuring a service is ultimately responsible for the
contract and, thus, USCIS should have clarified its and the VA's roles.
USCIS Has Used Results of Monitoring Efforts to Identify Opportunities
to Improve Customer Service and Call Flow at All Call Centers:
USCIS used contractor performance data, including the results of
surveys, call monitoring, and the mystery shopper program, to identify
opportunities to improve customer service, including improving call-
response times, help CSRs and IIOs better respond to customer
inquiries, and manage the flow of calls into call centers. Following
are examples of initiatives that USCIS recently implemented, or was
planning to implement, as of April 2005. It is too early to assess the
impact of these initiatives.
* USCIS implemented "intelligent call routing" for Tier 1. With
"intelligent call routing" in place since February 2004, telephone
calls routed from the interactive voice response system to Tier 1 are
now routed to the next available CSR, at any of the four Tier 1
contractor-operated call centers. Previously, telephone calls to Tier 1
were routed to the next available CSR in the call center that resided
in the same geographic region of the country as the caller. USCIS
officials said that by June 2005 they will also have implemented
"intelligent call routing" for calls transferred from Tier 1 to the
next available IIO at either of the two Tier 2 call centers.
* USCIS implemented "overflow routing." USCIS started its "overflow
routing" initiative in October 2004, enabling certain general call
types, identified as "English Other" and "Spanish Other," to be routed
directly from the interactive voice response system to Tier 2 USCIS-
operated call centers, bypassing the Tier 1 contractor-operated call
centers. Previously, all calls handled by IIOs at Tier 2 were first
routed from the interactive voice response system to Tier 1, where CSRs
then transferred the calls to Tier 2. USCIS officials said they expect
the change will result in 1 to 5 percent of all calls being routed
directly to Tier 2, which should help when Tier 1 CSRs cannot handle
the call volume.
* USCIS implemented interactive voice response system routing of
certain telephone calls to USCIS service centers. USCIS changed its
automated interactive voice response system in December 2004 so that
certain types of customers' telephone calls--for example, certain
issues concerning new permanent residents, cases already approved or
denied, and pending cases--are now routed directly to USCIS service
centers, bypassing CSRs at Tier 1. Previously, all customers' telephone
calls that needed to be handled by USCIS service centers were routed by
the interactive voice response system to Tier 1. Then, after talking
with the customers, the CSRs referred the customers to the service
centers (whose employees have access to case paperwork) via e-mail.
CSRs were allowed to transfer customers' telephone calls to service
center personnel only when customers requested emergency and expedited
handling of applications.
* USCIS implemented a portfolio management system. Private attorneys,
paralegals, and other representatives can use the USCIS Internet Web
site to check the status of their clients' immigration cases using a
USCIS receipt number. Under the system, USCIS also notifies the
representatives via e-mail when a case status changes; for example,
when actions are taken, such as the approval or denial of an
application. As of April 2005, over 300,000 customers, attorneys, and
other representatives had used this system.
* USCIS said it is planning to implement a referral management system.
Currently, Tier 1 CSRs send, via e-mail, service request referrals to
USCIS service centers and local offices for customers who call wanting
to change addresses, schedule and reschedule appointments at
application support centers, order forms, and resolve problems. After a
referral is made, NCSC does not know whether the service center or
local office responded to the customer in a timely manner or even
responded at all. To better monitor this process, USCIS plans to
implement a referral management system, with such service request
referrals placed in a database and assigned a tracking number. The
system is to (1) determine the proper service center or local office to
process the referral, (2) assign the case to an adjudicator, (3) update
the case on a daily basis, and (4) report once a month on case status.
The referral management system is planned to be accessible to customers
on USCIS's Internet Web site so they can make and track their own
service request referrals. In addition, customers without Internet
access are to be able to call on the telephone and CSR's will access
the USCIS Web site and create referrals for them. USCIS plans that the
referral management system will be fully operational during the summer
of 2005.
* USCIS is planning a customer service portal on USCIS's Web site.
USCIS has a long-term goal of giving customers Internet access to
information contained in the "scripts" used by Tier 1 CSRs to answer
customers' questions. USCIS plans to establish a customer service
portal on the USCIS Internet Web site, providing access to the
information. The goal is to let customers with Internet access look up
information themselves without having to call NCSC on the telephone,
navigate the interactive voice response system, and wait for CSR's to
answer. USCIS had not set a time frame for implementing this
initiative.
Conclusions:
Immigration call centers are a vital information referral source used
millions of times by immigrants and other interested parties seeking to
obtain needed documents, regulatory information, up-to-date status
information on immigration-related benefits and applications, and other
information. To ensure that it serves its customers effectively and
efficiently, USCIS appropriately used a performance-based contract, but
its failure to finalize all aspects of the performance requirements
before the contract was awarded hampered its ability to exercise
performance incentives in the contract. As a result, USCIS lost the
opportunity during the life of the contract to help ensure that it
received the maximum level of service from the contractor.
In addition, USCIS did not meet standards promulgated by federal
acquisition regulations, GAO, and the contract itself pertaining to
documenting the contractor's performance between 2002 and 2004, and
adequately documenting notification of the contractor when the
government perceived deficiencies in its performance. Failure to
generate adequate documentation could impair USCIS's ability to conduct
future contract negotiations and to preserve a complete and reliable
record of contract performance needed to ensure accountability.
Recommendation for Executive Action:
To improve USCIS's efforts for evaluating contractor performance and
encourage quality services at call centers, we recommend that the
Secretary of Homeland Security require the Director of USCIS take the
following two actions: (1) finalize contract terms related to specific
performance measurement requirements before awarding new performance-
based call center contracts; and (2) maintain readily available written
records of performance assessments and performance evaluation meetings
with the contractor.
Agency Comments and Our Evaluation:
DHS and the contractor provided formal comments and technical comments
on a draft of this report, which we have incorporated, as appropriate.
In its formal comments, DHS generally agreed with our recommendations.
DHS said the draft solicitation for the new contracts specifically
identifies performance requirements that are non-negotiable. DHS
further stated that, as recommended by GAO, written records of
performance assessments and performance evaluation meetings will be
maintained and readily available for review by all interested parties.
In its formal comments, the contractor provided additional language to
further clarify this report. The contractor said the report accurately
summarizes the complex nature of CIS's call center program and several
challenges created by significant post-award changes to that program.
DHS's and the contractor's formal comments are shown in appendixes VI
and VII, respectively.
As agreed, unless you publicly announce the contents of this report
earlier, we plan no further distribution of it until 30 days after its
issuance. At that time, we will send copies to the Director of USCIS
and other interested parties. We will also make copies available to
others upon request. In addition, the report will be available at no
charge on GAO's Web site at http://www.gao.gov.
If you or your staff have any questions regarding this report, please
contact me at 202-512-8777 or jonesp@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. Key contributors to this report are
listed in appendix VIII.
Paul L. Jones, Director Homeland Security and Justice Issues:
List of Requesters:
The Honorable John Conyers, Jr.:
Ranking Minority Member:
Committee on the Judiciary:
House of Representatives:
The Honorable Bennie G. Thompson:
Ranking Minority Member:
Committee on Homeland Security:
House of Representatives:
The Honorable Sheila Jackson Lee:
Ranking Minority Member, Subcommittee on Immigration, Border Security,
and Claims:
Committee on the Judiciary:
House of Representatives:
The Honorable Loretta Sanchez:
Ranking Minority Member, Subcommittee on Economic Security,
Infrastructure Protection, and Cybersecurity:
Committee on Homeland Security:
House of Representatives:
The Honorable Janice D. Schakowsky:
House of Representatives:
[End of section]
Appendix I: Scope and Methodology:
To determine what performance measures the U.S. Citizenship and
Immigration Services (USCIS) established to monitor and evaluate the
performance of contractor-operated call centers, we interviewed USCIS
headquarters officials in Washington, D.C; Tier 1 contractor officials
in Arlington, Virginia, and at a contractor-operated call
center;[Footnote 23] and an official representing an independent
consulting firm under contract to USCIS and located in Fairfax,
Virginia. We also collected and analyzed pertinent USCIS and contractor
documentation. We collected and analyzed information on the various
types of monitoring and evaluation programs used by USCIS, including
internal call monitoring, independent call monitoring, customer
satisfaction surveys, the mystery shopper program, and telephone call
data provided by a telecommunications vendor.
To find out how USCIS used the performance measures to evaluate the
contractor's performance, we interviewed USCIS headquarters officials
in Washington, D.C., and Tier 1 contractor officials in Arlington,
Virginia. We also collected and analyzed pertinent USCIS and contractor
documentation.
To determine what actions, if any, USCIS took or planned to take to
strengthen call center operations, we interviewed USCIS headquarters
officials in Washington, D.C., and Tier 1 contractor officials in
Arlington, Virginia, and at a contractor-operated call center. We also
collected and analyzed pertinent USCIS and contractor documentation.
We assessed the reliability of telephone call volume data provided to
USCIS by a telecommunications vendor, as well as USCIS and contractor
staffing data. To carry out our data reliability assessments, we (1)
reviewed information about the data, systems that produced the data,
and data quality control procedures, and (2) interviewed USCIS and
contractor officials knowledgeable about the data as necessary. We
determined that the call volume and staffing data were sufficiently
reliable for the purposes of this report.
We conducted our work between May 2004 and May 2005 in accordance with
generally accepted auditing standards.
[End of section]
Appendix II: Criteria and Methodology Used by Contractor to Measure
Quality of Tier 1 Calls Monitored:
Table 1: Call Quality Monitoring or "Soft Skills"
Criteria: Establish rapport;
Scoring range (0 - 3): 3;
Section score (Percent): 11.5%.
Criteria: Maintain composure;
Scoring range (0 - 3): 3;
Section score (Percent): 12.0%.
Criteria: Conversational style;
Scoring range (0 - 3): 3;
Section score (Percent): 11.5%.
Criteria: Active listening;
Scoring range (0 - 3): 3;
Section score (Percent): 12.5%.
Criteria: Efficient call flow;
Scoring range (0 - 3): 3;
Section score (Percent): 12.5%.
Criteria: Opening;
Scoring range (0 - 3): 3;
Section score (Percent): 10.0%.
Criteria: Call on hold;
Scoring range (0 - 3): 3;
Section score (Percent): 10.0%.
Criteria: Call transferred;
Scoring range (0 - 3): 3;
Section score (Percent): 10.0%.
Criteria: Closing;
Scoring range (0 - 3): 3;
Section score (Percent): 10.0%.
Criteria: Percent compliant;
Section score (Percent): 100%.
Source: USCIS contractor.
[End of table]
Table 2: Accuracy of Information Provided:
Criteria: Uses software tools appropriately;
Scoring range (0 - 3): 3;
Section score (Percent): 9.0%.
Criteria: Provides accurate response;
Scoring range (0 - 3): 3;
Section score (Percent): 28.0%.
Criteria: Provides complete response;
Scoring range (0 - 3): 3;
Section score (Percent): 27.0%.
Criteria: Provides USCIS content only;
Scoring range (0 - 3): 3;
Section score (Percent): 9.0%.
Criteria: Satisfies caller's needs;
Scoring range (0 - 3): 3;
Section score (Percent): 27.0%.
Criteria: Percent compliant;
Section score (Percent): 100%.
Source: USCIS contractor.
[End of table]
Table 3: Accuracy of Capturing Information:
Criteria: Completes referral when appropriate;
Scoring range (0 - 3): 3;
Section score (Percent): 12.0%.
Criteria: Completes referral record correctly;
Scoring range (0 - 3): 3;
Section score (Percent): 28.0%.
Criteria: Captures caller's information;
Scoring range (0 - 3): 3;
Section score (Percent): 30.0%.
Criteria: Verifies caller's information;
Scoring range (0 - 3): 3;
Section score (Percent): 30.0%.
Criteria: Percent compliant;
Section score (Percent): 100%.
Source: USCIS contractor.
[End of table]
[End of section]
Appendix III: Criteria and Methodology Used to Measure Customer
Satisfaction with Customer Service Representatives:
USCIS and an independent consulting firm jointly developed a telephone
survey to measure customer satisfaction with the three levels of NCSC
call center service--interactive voice response system, Tier 1 CSRs,
and Tier 2 IIOs. To carry out the survey each month, representatives of
the independent consulting firm call 375 randomly selected customers.
To assess the customers' satisfaction with the CSRs, the
representatives read several statements and ask questions for the
customers to rate their experiences with CSRs. For the customer
satisfaction performance measure required in the contract, USCIS
collects and summarizes data on the customers' responses to the four
statements below. The customers are asked to rate their agreement with
each of the statements using a scale of 1 to 7 (1 is strongly agree and
7 is strongly disagree).[Footnote 24]
1. The representative seemed to fully understand my questions.
2. The representative was polite.
3. The representative did not rush me.
4. The representative answered my questions promptly.
[End of section]
Appendix IV: Criteria and Methodology Used by Independent Consulting
Firm to Measure Quality of Calls Monitored:
An independent consulting firm scored CSRs on 23 separate quality
assurance factors as follows.
Service orientation:
Greeted customer;
Verified customer;
Established rapport;
Used customer name;
Expressed empathy;
Maintained composure;
Expressed commitment;
Offered additional assistance;
Terminated call appropriately.
Issue identification:
Obtained information;
Listed actively.
Issue resolution:
Took responsibility;
Provided thorough information;
Summarized actions.
Communications:
Projected enthusiastic tone;
Conveyed confidence;
Spoke with clarity;
Used appropriate language.
Call management:
Controlled call;
Used time efficiently;
Minimized "dead air";
Extended hold courtesies;
Transferred call appropriately.
[End of section]
Appendix V: Mystery Shopper Scenario Example:
[See PDF for image]
[End of figure]
[End of section]
Appendix VI: Comments from the Department of Homeland Security:
U.S. Department of Homeland Security:
Washington, DC 20528:
June 17, 2005:
Mr. Paul L. Jones:
Director, Homeland Security and Justice Issues:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Mr. Jones:
RE: Draft Report GAO-05-526, Immigration Services: Better Contracting
Practices Needed at Call Centers (GAO Job Code 440318):
The Department of Homeland Security (DHS) appreciates the opportunity
to review and comrnerrt on the Government Accountability Office's draft
report. As the auditors noted, United States Citizenship and
Immigration Services (USCIS) call centers are a vital information
referral source used millions of times by immigrants and other
interested parries seeking to obtain needed documents, regulatory
information, status on immigration-related benefits and applications,
and other information. The Department and and USCIS appreciate that GAO
recognized USCIS' use of contractor performance data, including the
results of surveys, call monitoring, and the mystery shopper program to
customer service. Management and staff at USCIS are proud of the
success of centers to date.
GAO concluded that to improve efforts for evaluating contractor
performance and encourage quality services at call centers, USCIS
officials should (1) finalize contract terms related to specific
performance measurement requirements before awarding new performance-
based call center contracts; and (2) maintain readily written records
of performance assessments and performance evaluation meetings with the
contractor. We generally agree with the GAO's recommendations and offer
comments on each recommendation.
In order to provide better context, we request that GAO mention early
in the body of the report that the contract in question was awarded and
administered by a component of the Veterans Administration for the
Immigration and Naturalization Service (INS) which included what is now
USCIS. The contract was awarded to Pearson Government Solutions (the
vendor) in January 2002, fourteen months before the DHS was established
This information is not found in the body of the draft report aside
from a footnote.
With respect to recommendation one, performance measures were, in fact,
detailed in the call center contract. The final methodology by which
performance against the group of stipulated critieria would be weighed
remained to be finalized. At the time of the award, INS management
belived it appropriate to let the winning vendor have some input into
the mechanics of the methodology since this contract represented a
stransition to performance based contracting for call center
operations. However, this approach resulted in subsequent discussions
with the vendor about performance that delayed application of the
incentive and disincentive clauses.
[See PDF for remainder of comment letter]
[End of section]
Appendix VII: Comments from Pearson Government Solutions:
PEARSON GOVERNMENT SERVICES:
June 15, 2005:
Mr. Paul L. Jones:
Director:
Homeland Security and Justice:
U.S. Government Accountability Office:
441 G Street NW:
Washington, DC 20548:
JOHN MCNAMARA CURTIS:
PRESIDENT AND CHIEF EXECUTIVE OFFICER:
PEARSON GOVERNMENT SOLUTIONS:
4250 N. FAIRFAX DRIVE:
SUITE 1200:
ARLINGTON,VA 22203:
TEL (703) 284-5602:
FAX (703) 284-5662:
mac.curtis@pearson.com:
www.pearsongov.com:
Dear Mr. Jones:
We appreciate the opportunity to comment on the GAO draft report (GAO-
05-526) regarding the USCIS management of call center contracts.
Overall, the report accurately summarizes the complex nature of CIS'
call center program, and several challenges created by significant post-
award changes to that program. However, we believe the report would
benefit from additional clarification on the following points:
Contract Changes Resulted in Challenging Negotiations:
The draft report explains that during a 16 month period following
contract award, CIS and Pearson Government Solutions negotiated on how
performance metrics would be scored. Pearson's objective during these
negotiations was to define performance metrics that were reasonable and
attainable given the significant and ongoing changes made to the
contract following the award. In the end, although the negotiations
were substantive they did not produce an agreement.
CIS Metrics Do Not Reflect Reflect Changes:
CIS unilaterally decided to apply performance measurements to the
contract that were unacceptable to Pearson. We objected for two
reasons: the contract requires mutually agreed upon performance
measurements; and CIS' performance measurements did not take into
account changes to the work required by the contract. For example, as
the report states, the number of scripts CSRs are required to use in
responding to callers increased from 400 to over 2,300. However, CIS's
performance measurements did not reflect this or other changes. As a
result CIS determined that PGS did not meet 4 of the 7 metrics for the
fourth quarter of 2004, triggering a payment reduction. Although we
object to this determination, we stand ready to work with CIS to
establish fair and equitable financial incentives.
Pearson Government Solutions manages citizen interaction programs for
governments worldwide. We understand what it takes for contact center
programs to be successful even with unexpected dynamic changes. As with
all contracts, the contractor and government must work together to
identify resources needed to respond to changes, agree on expected
outcomes, and ensure that performance measurements and contract
modifications reflect unexpected changes. The CIS program has had
challenges in this regard. However, despite these challenges, we have
worked closely with CIS to provide high-quality service and will
continue to do so.
In closing, I want to commend your team's efforts to thoroughly assess
and accurately summarize these complex issues. We appreciate the
opportunity to present these comments on the draft report. Please do
not hesitate to contact me or my staff if you have any questions.
Sincerely,
Signed by:
John McNamara Curtis:
President and Chief Executive Officer:
Pearson Government Solutions:
[End of section]
Appendix VIII: GAO Contact and Staff Acknowledgments:
GAO Contacts:
Paul L. Jones 202-512-8777:
Staff Acknowledgments:
In addition to the above, Darryl W. Dutton, Ronald G. Viereck, Brian J.
Lipman, Christine F. Davis, Amy L. Bernstein, and Michele C. Fejfar
made key contributions to this report.
FOOTNOTES
[1] An automated voice system with a touch-tone menu.
[2] The contractor considers the locations of its call centers to be
proprietary information.
[3] Pub. L. No. 107-296, § 451 (Nov. 25, 2002).
[4] Throughout this report, we use USCIS to refer to both the new USCIS
and the former Immigration Services Division of INS.
[5] If a customer attempts to obtain live assistance outside normal
business hours, the customer is told when to call back to obtain the
assistance.
[6] In some cases, customers' calls are routed directly from the
interactive voice response system to Tier 2, bypassing Tier 1.
[7] The contractor considers the precise number of CSRs at its call
centers and their locations to be proprietary information.
[8] The term "contract" in this report refers specifically to "Delivery
Order No. 591," which was awarded to the contractor following a
competitive procedure through the General Services Administration's
Federal Supply Schedule.
[9] The contract was awarded before DHS was established while what is
now USCIS was part of INS.
[10] OFPP and the National Performance Review coordinated a federal
interagency working group to develop generic performance-based concepts
for government agencies to use in the procurement of call center
services. The group developed a reference source to help program and
acquisition officials draft performance requirements and standards
related to call center performance-based contracts: Performance Based
Concepts for Telephone Call Center Contracting, Office of Federal
Procurement Policy (May 27, 1997).
[11] USCIS monitors and evaluates operations at both Tier 1 and Tier 2
call centers. This report focuses on USCIS's evaluation of Tier 1
contractor-operated call centers.
[12] This telecommunications vendor is also responsible for providing
the interactive voice response system and routing customers' telephone
calls from this system to Tier 1 and Tier 2 call centers.
[13] In this report, we collectively refer to these three components as
"performance measurement requirements."
[14] The "sample" terms of the contract were never finalized, leading
to a dispute between USCIS and the contractor that remains unresolved
(see p. 15). Examples of "sample calculations" listed for different
measures in the PRS were "total delay of all calls divided by total
number of calls" for determining the average length of time it takes
before calls are answered, and "total number of points achieved divided
by total number of points available for all quality monitoring scores"
for determining call quality. One sample calculation was listed for
each measure.
[15] The customer satisfaction surveys ask about customer satisfaction
with the interactive voice response system and IIOs, as well as CSRs.
[16] Up to 10 percent of payments can be added to or deducted from the
contractor's billed amounts, depending upon whether or not the
performance standards are met.
[17] USCIS started collecting data on the contractor's performance in
June 2002, the first month of the contractor's operations. We are not
presenting the performance data used by USCIS because the performance
measurement terms of the PRS were unresolved between USCIS and the
contractor as of April 2005.
[18] USCIS revised the scripts to provide new and updated immigration-
related information on a variety of topics. In its technical comments
on a draft of this report, DHS said that the contractor had been
compensated for the increased script content and average call duration.
[19] Performance Based Concepts for Telephone Call Center Contracting,
Office of Federal Procurement Policy (May 27, 1997).
[20] USCIS officials said that DHS will conduct the acquisition and
USCIS will administer the contract, without seeking inter-agency
assistance from the VA as it previously did with the current call
center contract.
[21] The current contractor may compete for the new contracts.
[22] GAO, Standards for Internal Control in the Federal Government,
GAO/AIMD-00-21.3.1 (Washington, D.C.: November 1999); Internal Control
Management and Evaluation Tool, GAO-01-1008G (Washington, D.C.: August
2001).
[23] The contractor considers the locations of its call centers to be
proprietary information.
[24] The complete scale for the four statements is as follows: 1-
Strongly agree; 2-Mostly agree; 3-Somewhat agree, 4-Mixed, Neither
agree nor disagree; 5-Somewhat disagree; 6-Mostly disagree; and 7-
Strongly disagree.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
