Data Mining

Data mining--a technique for extracting useful information from large volumes of data--is one type of analysis that the Department of Homeland Security (DHS) uses to help detect and prevent terrorist threats. While data-mining systems offer a number of promising benefits, their use also raises privacy concerns. GAO was asked to (1) assess DHS policies for evaluating the effectiveness and privacy protections of data-mining systems used for counterterrorism, (2) assess DHS agencies' efforts to evaluate the effectiveness and privacy protections of their data-mining systems, and (3) describe the challenges facing DHS in implementing an effective evaluation framework. To do so, GAO developed a systematic evaluation framework based on recommendations and best practices outlined by the National Research Council, industry practices, and prior GAO reports. GAO compared its evaluation framework to DHS's and three component agencies' policies and to six systems' practices, and interviewed agency officials about gaps in their evaluations and challenges.

As part of a systematic evaluation framework, agency policies should ensure organizational competence, evaluations of a system's effectiveness and privacy protections, executive review, and appropriate transparency throughout the system's life cycle. While DHS and three of its component agencies--U.S. Customs and Border Protection, U.S. Immigration and Customs Enforcement, and the U.S. Citizenship and Immigration Services--have established policies that address most of these key policy elements, the policies are not comprehensive. For example, DHS policies do not fully ensure executive review and transparency, and the component agencies' policies do not sufficiently require evaluating system effectiveness. DHS's Chief Information Officer reported that the agency is planning to improve its executive review process by conducting more intensive reviews of IT investments, including the data-mining systems reviewed in this report. Until such reforms are in place, DHS and its component agencies may not be able to ensure that critical data mining systems used in support of counterterrorism are both effective and that they protect personal privacy. Another aspect of a systematic evaluation framework involves ensuring that agencies implement sound practices for organizational competence, evaluations of a system's effectiveness and privacy protections, executive review, and appropriate transparency and oversight throughout a system's life cycle. Evaluations of six data mining systems from a mix of DHS component agencies showed that all six program offices took steps to evaluate their system's effectiveness and privacy protections. However, none performed all of the key activities associated with an effective evaluation framework. For example, four of the program offices executed most of the activities for evaluating program privacy impacts, but only one program office performed most of the activities related to obtaining executive review and approval. By not consistently performing necessary evaluations and reviews of these systems, DHS and its component agencies risk developing and acquiring systems that do not effectively support their agencies' missions and do not adequately ensure the protection of privacy-related information. DHS faces key challenges in implementing a framework to ensure systems are effective and provide privacy protections. These include reviewing and overseeing systems once they are in operation, stabilizing and implementing acquisition policies throughout the department, and ensuring that privacy-sensitive systems have timely and up-to-date privacy reviews. The shortfalls GAO noted in agency policies and practices provide insight into these challenges. Until DHS addresses these challenges, it will be limited in its ability to ensure that its systems have been adequately reviewed, are operating as intended, and are appropriately protecting individual privacy and assuring transparency to the public. GAO is recommending that DHS executives address gaps in agency evaluation policies and that component agency officials address shortfalls in their system evaluations. DHS concurred with GAO's recommendations and identified steps it is taking to address selected recommendations. The department also offered technical comments, which GAO incorporated as appropriate.

Recommendations

Our recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.

Director: David A. Powner Team: Government Accountability Office: Information Technology Phone: (202) 512-9286