SSA Disability Decision Making
Additional Steps Needed to Ensure Accuracy and Fairness of Decisions at the Hearing Level Gao ID: GAO-04-14 November 12, 2003Historically, the proportion of the Social Security Administration's (SSA) disability benefits claims that were approved has been lower for African-Americans than for whites. In 1992, GAO found that racial differences, largely at the Administrative Law Judge (ALJ) level, could not be completely explained by factors related to the decision-making process. This report examines how race and other factors influence ALJ decisions and assesses SSA's ability to ensure the accuracy and fairness of ALJ decisions.
GAO controlled for factors that are related to the disability decision-making process at the Administrative Law Judge level and found: (1) no statistically significant difference in the likelihood of being allowed benefits between white claimants and claimants from other, non-African-American racial/ethnic groups; and between white claimants and African-American claimants who were represented by attorneys; (2) statistically significant differences between white and African-American claimants who were not represented by attorneys (specifically, among claimants without attorneys, African-American claimants were significantly less likely to be awarded benefits than white claimants); and (3) other factors--including sex, income, and the presence of a translator at a hearing--also had a statistically significant influence on the likelihood of benefits being allowed. Due to the inherent limitations of statistical analysis, one cannot determine whether these differences by race, sex, and other factors are a result of discrimination, other forms of bias, or variations in currently unobservable claimant characteristics. Analytical, sampling, and data weaknesses in SSA's approach to quality assurance reviews limit its ability to ensure the accuracy and fairness of ALJ decisions. Analytic weaknesses: SSA analyzes ALJ decisions by various factors, such as SSA region, but not by the claimant's race. Sampling weaknesses: SSA currently excludes cases that have been appealed to the Appeals Council from the pool of ALJ cases that undergoes the quality assurance review. The exclusion of these cases could mean that the sample used by SSA in its quality assurance review is not representative of all ALJ decisions. While GAO did not find large differences in the sample of cases from 1997 to 2000 that it used for its analysis, the continued, systematic exclusion of cases that are under appeal could in the future result in an unrepresentative sample of all ALJ decisions. Data limitations: even if SSA wanted to conduct analyses by race/ethnicity, it would encounter difficulties doing so in the near future because, since 1990, SSA significantly scaled back its collection of race/ethnicity data. Although GAO had sufficient race data for its study, the scaled back collection of race/ethnicity data will impact SSA's future efforts to study ALJ benefit decisions by race. During GAO's review, however, SSA decided to collect race/ethnicity data for persons applying for Social Security benefits.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-04-14, SSA Disability Decision Making: Additional Steps Needed to Ensure Accuracy and Fairness of Decisions at the Hearing Level
This is the accessible text file for GAO report number GAO-04-14
entitled 'SSA Disability Decision Making: Additional Steps Needed to
Ensure Accuracy and Fairness of Decisions at the Hearing Level' which
was released on November 12, 2003.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States General Accounting Office:
GAO:
November 2003:
SSA Disability Decision Making:
Additional Steps Needed to Ensure Accuracy and Fairness of Decisions at
the Hearings Level:
GAO-04-14:
GAO Highlights:
Highlights of GAO-04-14, a report to congressional requesters
Why GAO Did This Study:
Historically, the proportion of the Social Security Administration‘s
(SSA) disability benefits claims that were approved has been lower for
African-Americans than for whites. In 1992, GAO found that racial
differences, largely at the Administrative Law Judge (ALJ) level,
could not be completely explained by factors related to the decision-
making process. This report examines how race and other factors
influence ALJ decisions and assesses SSA‘s ability to ensure the
accuracy and fairness of ALJ decisions.
What GAO Found:
GAO controlled for factors that are related to the disability decision-
making process at the Administrative Law Judge level and found:
* no statistically significant difference in the likelihood of being
allowed benefits between white claimants and claimants from other, non-
African-American racial/ethnic groups; and between white claimants and
African-American claimants who were represented by attorneys;
* statistically significant differences between white and African-
American claimants who were not represented by attorneys.
Specifically, among claimants without attorneys, African-American
claimants were significantly less likely to be awarded benefits than
white claimants; and
* other factors”including sex, income, and the presence of a
translator at a hearing”also had a statistically significant influence
on the likelihood of benefits being allowed.
Due to the inherent limitations of statistical analysis, one cannot
determine whether these differences by race, sex, and other factors
are a result of discrimination, other forms of bias, or variations in
currently unobservable claimant characteristics.
Analytical, sampling, and data weaknesses in SSA‘s approach to quality
assurance reviews limit its ability to ensure the accuracy and
fairness of ALJ decisions. For example:
* Analytic weaknesses: SSA analyzes ALJ decisions by various factors,
such as SSA region, but not by the claimant‘s race.
* Sampling weaknesses: SSA currently excludes cases that have been
appealed to the Appeals Council from the pool of ALJ cases that
undergoes the quality assurance review. The exclusion of these cases
could mean that the sample used by SSA in its quality assurance review
is not representative of all ALJ decisions. While GAO did not find
large differences in the sample of cases from 1997 to 2000 that it
used for its analysis, the continued, systematic exclusion of cases
that are under appeal could in the future result in an
unrepresentative sample of all ALJ decisions.
* Data limitations: even if SSA wanted to conduct analyses by race/
ethnicity, it would encounter difficulties doing so in the near future
because, since 1990, SSA significantly scaled back its collection of
race/ethnicity data. Although GAO had sufficient race data for its
study, the scaled back collection of race/ethnicity data will impact
SSA‘s future efforts to study ALJ benefit decisions by race. During
GAO‘s review, however, SSA decided to collect race/ethnicity data for
persons applying for Social Security benefits.
What GAO Recommends:
GAO recommends that SSA enhance its ALJ quality assurance reviews by
* incorporating cases that are appealed to SSA's Appeals Council in
the quality assurance review sample,
* conducting ongoing as well as in-depth analyses of ALJ decisions by
race and other factors, and
* publishing these results in its biennial reports.
Further, GAO recommends that SSA
* take action, as needed, to correct and prevent unwarranted allowance
differences; and
* establish an expert advisory panel to provide ongoing leadership,
oversight, and technical assistance with respect to ALJ quality
assurance reviews.
SSA agreed with GAO‘s recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-04-14.
To view the full product, including the scope and methodology, click
on the link above. For more information, contact Robert E. Robertson
at (202) 512-7215 or RobertsonR@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
Race and Other Factors Influence ALJ Decisions for Some Claimant
Groups:
SSA's Approach to Quality Assurance Reviews Limits Its Ability to
Ensure the Accuracy and Fairness of ALJ Decisions:
Conclusions:
Recommendations:
Agency Comments:
Appendix I: Scope and Methods:
Section 1: Databases and Information Sources:
Section 2: Data Reliability Tests:
Section 3: Weighting and Sampling Errors:
Section 4: Statistical Analysis:
Section 5: Limitations of Analysis:
Appendix II: SSA's Five-Step Sequential Evaluation Process for
Determining Disability:
Appendix III: Comments from the Social Security Administration:
Appendix IV: GAO Contacts and Acknowledgments:
GAO Contacts:
GAO Acknowledgments:
Other Acknowledgments:
Tables:
Table 1: Variables Used in Our Model of ALJ Decision Making:
Table 2: Percentage of Claimants Allowed Benefits at the Hearings Level
by Race and Region, 1997 to 2000:
Table 3: Data Used in Our Analyses:
Table 4: Statistically Significant Differences between Responder and
Nonresponder Groups, as Estimated with Logistic Regression:
Table 5: Tabulations of Statistically Significant Administrative
Factors (from Table 4) for Responders and Nonresponders:
Table 6: Results of Baseline and Final Models of ALJ Allowance
Decisions:
Table 7: Observed and Estimated Odds Ratios by Attorney Representation
and Race:
Table 8: Computations for Odds Ratios for Different Racial Groups That
Are Represented by an Attorney:
Table 9: Computations for Odds Ratios for Claimants of the Same Race
with and without Attorney Representation:
Table 10: Effect of Attorney Representation on ALJ Decisions for
Responders and Nonresponders:
Table 11: Effect of Attorney Representation on ALJ Decisions for
Responders and the Entire Sample:
Table 12: Effect of Attorney Representation on ALJ Decisions for
Responders and Nonresponders, by Race:
Table 13: Effect of Attorney Representation on ALJ Decisions for
Responders and the Entire Sample by Race:
Table 14: Summary Results of Oaxaca Decomposition:
Abbreviations:
ACAPS: Appeals Council Automated Processing System:
ALJ: Administrative Law Judge:
CCS: Office of Hearings and Appeals Case Control System:
DDHQ: Division of Disability Hearings Quality:
DDS: Disability Determination Service:
DI: Disability Insurance:
EAB: Enumeration at Birth:
HALLEX: Hearings, Appeals and Litigation Law Manual:
MEF: Master Earnings File:
NOSSCR: National Organization of Social Security Claimant
Representatives:
OHA: Office of Hearings and Appeals:
OQA: Office of Quality Assurance and Performance Assessment:
SGA: substantial gainful activity:
SSA: Social Security Administration:
SSI: Supplemental Security Income:
United States General Accounting Office:
Washington, DC 20548:
November 12, 2003:
The Honorable Charles B. Rangel:
Ranking Minority Member:
Committee on Ways and Means:
House of Representatives:
The Honorable Robert T. Matsui:
Ranking Minority Member:
Subcommittee on Social Security:
Committee on Ways and Means:
House of Representatives:
The Honorable Gene Green:
House of Representatives:
Historically, under the Social Security Administration's (SSA)
Disability Insurance (DI) and Supplemental Security Income (SSI)
programs, the proportion of benefit claims that were approved for
African-Americans has been lower than the proportion that were approved
for whites.[Footnote 1] In 1992, GAO conducted a statistical analysis
of disability benefit decisions and found that racial differences,
largely at the Administrative Law Judge (ALJ) level, could not be
completely explained by factors related to the decision-making process,
such as certain demographic characteristics of claimants (including
age, education, and sex) and their impairment types. In 2001, you asked
us to examine the steps SSA had taken to correct and prevent
unwarranted racial differences. You also asked us to examine whether
unwarranted racial differences currently exist within these programs.
This report is the second of two reports in response to your request.
In the first report, published in September 2002, we assessed steps SSA
took to investigate and correct potential unwarranted differences,
including SSA's study of racial differences in ALJ decisions.[Footnote
2] For its study, SSA used new data--which we will refer to as enhanced
data--developed as part of its recently established and ongoing quality
assurance review of ALJ decisions. The enhanced data contain
information, previously unavailable to GAO, such as an improved measure
of severity of the claimant's impairment. In our 2002 report, we stated
that we were unable to draw firm conclusions about racial differences
from SSA's study because of weaknesses we identified in SSA's sampling
and statistical methods. As a result, we recommended that SSA assess
the degree to which its enhanced data are representative of ALJ
disability decisions and make any needed changes to its sampling
protocol and statistical methods, as part of its ongoing quality
assurance review of ALJ decisions.
This report examines (1) how race and other factors influence ALJ
decisions and (2) limitations in SSA's ability to ensure the accuracy
and fairness of ALJ decisions. You asked us to examine racial
differences in DI and SSI decisions at the ALJ level, including
Hispanics and other ethnic groups. However, due to limitations with
SSA's race/ethnicity data, our examination was limited to African-
American claimants, white claimants, and claimants from other racial/
ethnic groups.[Footnote 3]
Given our previously reported concerns about the degree to which the
enhanced data are representative,[Footnote 4] we conducted tests at the
beginning of this review to determine whether the enhanced data were
sufficiently representative and reliable for our analyses.[Footnote 5]
Because these tests established that the enhanced data were of
sufficient quality for our analysis, we were able to analyze these data
to determine whether racial differences currently exist in ALJ benefit
decisions and whether differences in ALJ decisions are explained by
factors related to the decision-making process. To do this, we analyzed
SSA's enhanced data from 1997-2000 using statistical models of ALJ
decision making that we constructed. Specifically, we used multivariate
analysis to determine whether any differences by race/ethnicity could
be statistically attributed to factors related to ALJ decision
making.[Footnote 6] As shown in table 1, the variables we included in
our model can be grouped into three broad sets of factors that are
related to the decision-making process: (1) factors that represent the
criteria used in the disability decision-making process; (2) factors
that represent participants in the decision-making process; and (3)
factors that are not part of the decision-making process, but may
influence it.[Footnote 7] See appendix I for more information on our
statistical methods.
Table 1: Variables Used in Our Model of ALJ Decision Making:
Factors representing criteria in the decision-making process: Medical
variables:
Factors representing criteria in the decision-making process:
Impairments.
Factors representing criteria in the decision-making process: Severity
of impairment.
Factors representing criteria in the decision-making process: Alcohol
or drug abuse.
Factors representing criteria in the decision-making process:
Consultative examination requested.
Factors representing criteria in the decision-making process: Number of
impairments.
Factors representing criteria in the decision-making process: Number of
severe impairments.
Factors representing criteria in the decision-making process: Residual
functional capacity of claimant.
Factors representing criteria in the decision-making process: Mental
residual functional capacity of claimant.
Factors representing criteria in the decision-making process:
Nonmedical variables:
Factors representing criteria in the decision-making process:
Occupational type.
Factors representing criteria in the decision-making process: Years of
employment.
Factors representing criteria in the decision-making process:
Occupational skill level.
Factors representing criteria in the decision-making process:
Education.
Factors representing criteria in the decision-making process: Literacy.
Factors representing criteria in the decision-making process: Age
category.
Factors representing participants in the decision-making process:
Factors representing participants in the decision-making process:
Representation (by attorney or other).
Factors representing participants in the decision-making process:
Medical expert present at hearing.
Factors representing participants in the decision-making process:
Vocational expert present at hearing.
Factors representing participants in the decision-making process:
Translator present at hearing.
Factors representing participants in the decision-making process:
Claimant present at hearing.
Factors not part of the decision-making process, but may influence it:
Factors not part of the decision-making process, but may influence it:
Race.
Factors not part of the decision-making process, but may influence it:
Sex.
Factors not part of the decision-making process, but may influence it:
Earnings.
Factors not part of the decision-making process, but may influence it:
Type of
claim.
Factors not part of the decision-making process, but may influence it:
Year of decision.
Factors not part of the decision-making process, but may influence it:
Region.
Source: GAO analysis of SSA's enhanced data.
[End of table]
To obtain information on factors limiting SSA's ability to ensure the
accuracy and fairness of ALJ decisions, we interviewed SSA officials
and reviewed documentation concerning the agency's ongoing quality
assurance review of ALJ decisions. We also interviewed officials within
the Department of Health and Human Services' Centers for Medicare and
Medicaid Services to discuss their use of SSA race data.
We performed our work from August 2002 to September 2003 in accordance
with generally accepted government auditing standards.
Results in Brief:
When we controlled for factors that are related to the disability
decision-making process at the hearings level, including the severity
of the claimant's impairment, whether or not the claimant had attorney
representation, and the claimant's age and work experience, we found no
statistically significant differences in the likelihood of being
allowed benefits between whites and claimants from other, non-African-
American racial/ethnic groups. We did, however, find differences
between white and African-American claimants, but only among claimants
who were not represented by attorneys. That is, among claimants who
were represented by attorneys, white and African-American claimants
were equally likely to be allowed benefits, but among claimants who
were not represented by attorneys, African-American claimants were
significantly less likely to be awarded benefits than white claimants.
Moreover, claimants who were represented by persons other than
attorneys, such as legal aides, friends or family, were more likely to
be awarded benefits than claimants who are not represented; however,
among claimants represented by these nonattorneys, African-Americans
were less likely to be awarded benefits than whites. Besides race and
attorney representation, other factors that are not part of the
criteria used in the decision-making process also had a statistically
significant influence on the likelihood of benefits being allowed. For
example, male claimants, claimants with low incomes, or non-English-
speaking claimants who had a translator at a hearing were less likely
to be awarded benefits. Due to the inherent limitations of statistical
analysis, one cannot determine whether these differences by race, sex,
and other factors are a result of discrimination or other forms of
bias, or due to variations in currently unobservable claimant
characteristics, such as a lack of detailed information on medical
evidence needed to buttress impairment claims.
Analytical, sampling, and data weaknesses in SSA's approach to quality
assurance reviews limit its ability to ensure the accuracy and fairness
of ALJ decisions. As part of its ongoing quality assurance review, SSA
analyzes ALJ decisions by various claimant characteristics such as the
claimant's age and the region where the disability decision was issued,
but not by the claimant's race. This analytic omission limits SSA's
ability to identify, correct, and prevent unwarranted racial
differences in allowance rates. In addition, weaknesses in the review's
sampling methods present problems. For example, SSA currently excludes
cases that have been appealed to the Appeals Council from the pool of
ALJ cases that undergoes the quality assurance review. The exclusion of
these cases could mean that the sample used by SSA in its quality
assurance review is not representative of all ALJ decisions. While we
found the sample of cases that we used for our analysis to be
sufficiently representative, the continued, systematic exclusion of
appealed cases could, in the future, result in an unrepresentative
sample of all ALJ decisions. Finally, data limitations restrict SSA's
ability to ensure the accuracy and fairness of ALJ decisions. For
example, even if SSA wanted to conduct analyses by race/ethnicity, it
would encounter difficulties doing so in the near future because, since
1990, SSA has significantly scaled back its collection of race/
ethnicity data. Although we had sufficient race data for our study, the
scaled back collection of race/ethnicity data will impact SSA's future
efforts to study ALJ benefit decisions by race. During our review,
however, SSA decided to collect race/ethnicity data for disability
claimants and other individuals applying for Social Security benefits
and has set up a task group to explore implementation issues. In
addition, SSA officials recently informed us that they are considering
ways to include appealed cases in their quality assurance review.
To better ensure the accuracy and fairness of ALJ decisions by race/
ethnicity and other factors not related to criteria used in the
decision-making process, we recommend that SSA enhance its ALJ quality
assurance reviews by: incorporating cases that are appealed to SSA's
Appeals Council in the quality assurance review sample; conducting
ongoing as well as in-depth analyses of ALJ decisions by race and other
factors; and publishing these results in its biennial reports. We also
recommend that SSA take action, as needed, to correct and prevent
unwarranted allowance differences, and establish an expert advisory
panel to provide ongoing leadership, oversight, and technical
assistance with respect to ALJ quality assurance reviews.
In its written comments to our report, SSA agreed with our
recommendations and indicated that it intends to go further as it moves
forward with its recently proposed plan to improve the disability
determination process. SSA's comments and its proposed plan to improve
the disability determination process are printed in appendix III.
Background:
DI and SSI are the two largest federal programs providing cash
assistance to people with disabilities. Established in 1956, DI
provides monthly payments to workers with disabilities (and their
dependents or survivors) under the age of 65 who have enough work
experience to qualify for disability benefits. Created in 1972, SSI is
a means-tested income assistance program that provides monthly payments
to adults or children who are blind or who have other disabilities and
whose income and assets fall below a certain level.[Footnote 8] To be
considered eligible for either program as an adult, a person must be
unable to perform any substantial gainful activity by reason of a
medically determinable physical or mental impairment that is expected
to result in death or that has lasted or can be expected to last for a
continuous period of at least 12 months. Work activity is generally
considered substantial and gainful if the person's earnings exceed a
particular level established by statute and regulations.[Footnote 9] In
calendar year 2002, about 5.5 million disabled workers (age 18-64)
received about $55.5 billion in DI benefits, and about 3.8 million
working-age individuals with disabilities received about $18.6 billion
in SSI federal benefits.[Footnote 10]
To obtain disability benefits, a claimant must file an application
online,[Footnote 11] by telephone or mail, or in person at any Social
Security office. If the claimant meets the nonmedical eligibility
criteria, the field office staff forwards the claim to the appropriate
state Disability Determination Service (DDS) office. DDS staff--
generally a team comprised of disability examiners and medical
consultants--review medical and other evidence provided by the
claimant, obtaining additional evidence as needed to assess whether the
claimant satisfies program requirements, and make the initial
disability determination. If the claimant is not satisfied with this
determination, the claimant may request a reconsideration of the
decision within the same DDS.[Footnote 12] Another DDS team will review
the documentation in the case file, as well as any new evidence the
claimant may submit, and determine whether the claimant meets SSA's
definition of disability. In 2002, the DDSs made 2.3 million initial
disability determinations and over 484,000 reconsiderations.
If the claimant is not satisfied with the reconsideration, he or she
may request a hearing before an ALJ. Within SSA's Office of Hearings
and Appeals (OHA), there are approximately 1,150 ALJs who are located
in 140 hearing offices across the country. The ALJ conducts a new
review of the claimant's file, including any additional evidence the
claimant submitted after the DDS determination. At a hearing, the ALJ
may hear testimony from the claimant, medical experts on the claimant's
medical condition, and vocational experts regarding whether the
claimant could perform work he or she has done in the past or could
perform other jobs currently available in the national
economy.[Footnote 13] ALJs have an obligation to initiate the
development of evidence as needed and make every effort to obtain all
necessary evidence before the hearing. The hearings are recorded, and
the majority of claimants are represented at these hearings by an
attorney or a nonattorney representative, such as a legal aide, parent,
relative, or social worker. In addition, translators may be used for
claimants with limited proficiency in English. In fiscal year 2002,
ALJs made over 438,000 disability decisions.
If the claimant is not satisfied with the ALJ decision, the claimant
may request a review by SSA's Appeals Council, which is the final
administrative appeal within SSA. The Appeals Council may grant, deny,
or dismiss a request for review. If it agrees to review the case, the
Appeals Council may uphold, modify, or reverse the ALJ's action or it
may remand the case back to the ALJ level for an ALJ to hold another
hearing and issue a new decision. In fiscal year 2002, the Appeals
Council reviewed over 108,000 disability decisions, about 27,000 of
which were remanded.[Footnote 14]
SSA's Office of Quality Assurance and Performance Assessment (OQA)
conducts quality assurance reviews of ALJ decisions to promote fair and
accurate hearing decisions. These quality assurance reviews include an
evaluation of ALJ adjudicative and procedural issues. The findings and
information of these reviews are included in biennial reports and
assist the OHA in its pursuit of quality by identifying specific areas
of concern. These findings also support the "hearings decisional
accuracy rate" measure in SSA's annual performance plans and reports.
To conduct its quality assurance review, OQA selects a random sample
each month from the universe of ALJ decisions, stratifying the
selection of cases by region and decisional outcome (approval or
denial). Then, for each selected decision, SSA requests the case file
and a recording of the hearing proceedings from hearing offices and
storage facilities across the country.[Footnote 15] To collect the data
SSA uses in its review, SSA staff conducts a systematic review of each
case, including: a review of the ALJ decision by another ALJ (i.e., a
peer review), a review of the medical evidence provided at each level
of adjudication performed by one or more medical consultants,[Footnote
16] and a general review of the documentation and decision at each
adjudicative level by a disability examiner.
The peer review of an ALJ decision includes a reviewing judge's
assessment of whether the ALJ's ultimate decision to allow or deny
benefits is supported by substantial evidence.[Footnote 17] These
assessments are referred to in the quality assurance review as support
or accuracy rates. The peer review also includes judgments about the
fairness of the ALJ hearing, in which the reviewing judge evaluates a
number of issues, including abuse of discretion[Footnote 18] and error
of law.[Footnote 19] The results of the peer review, as well as the
results of the medical and general reviews, comprise SSA's enhanced
data.
Over the years, GAO and SSA have studied SSA's ability to administer
its disability programs in a fair and unbiased manner. In our 1992
report,[Footnote 20] we found that racial differences in ALJ allowance
rates were not explained by other factors related to the disability
decision-making process. We recommended, and SSA agreed, to further
investigate the reasons for the racial differences at the hearings
level and act to correct or prevent any unwarranted disparities. In
response to our recommendations, SSA conducted its own study of ALJ
allowance rates by race, using its enhanced data from 1992 to 1996.
Although the results were never published, SSA officials told us that
they found no evidence of unwarranted racial differences at the
hearings level. In our 2002 report,[Footnote 21] we assessed the steps
SSA had taken to study allowance rates by race, and we found that
methodological weakness precluded us from drawing conclusions on
whether unwarranted racial differences in ALJ allowance rates existed.
SSA's enhanced data indicate that racial differences exist in overall
allowance rates for disability benefits at the hearings level. As shown
in table 2, these differences in allowance rates by race exist to
varying degrees in almost every SSA region. However, differences in
allowance rates by race do not necessarily point to racial
discrimination because claimants from different racial/ethnic groups
may have other differences that influence allowance decisions.
Table 2: Percentage of Claimants Allowed Benefits at the Hearings Level
by Race and Region, 1997 to 2000:
Region: All regions; Numbers in percent: All: 59; Numbers in percent:
White: 63; Numbers in percent: African-American: 49; Numbers in
percent: Other race/ethnicity: 51.
Region: Region 1 Boston; Numbers in percent: All: 73; Numbers in
percent: White: 76; Numbers in percent: African-American: 66; Numbers
in percent: Other race/ethnicity: 62.
Region: Region 2 New York; Numbers in percent: All: 64; Numbers in
percent: White: 72; Numbers in percent: African-American: 51; Numbers
in percent: Other race/ethnicity: 57.
Region: Region 3 Philadelphia; Numbers in percent: All: 60; Numbers in
percent: White: 62; Numbers in percent: African-American: 59; Numbers
in percent: Other race/ethnicity: 37.
Region: Region 4 Atlanta; Numbers in percent: All: 60; Numbers in
percent: White: 65; Numbers in percent: African-American: 51; Numbers
in percent: Other race/ethnicity: 61.
Region: Region 5 Chicago; Numbers in percent: All: 55; Numbers in
percent: White: 59; Numbers in percent: African-American: 46; Numbers
in percent: Other race/ethnicity: 45.
Region: Region 6 Dallas; Numbers in percent: All: 54; Numbers in
percent: White: 61; Numbers in percent: African-American: 39; Numbers
in percent: Other race/ethnicity: 52.
Region: Region 7 Kansas City; Numbers in percent: All: 59; Numbers in
percent: White: 61; Numbers in percent: African-American: 51; Numbers
in percent: Other race/ethnicity: 45.
Region: Region 8 Denver; Numbers in percent: All: 59; Numbers in
percent: White: 61; Numbers in percent: African-American: 66; Numbers
in percent: Other race/ethnicity: 48.
Region: Region 9 San Francisco; Numbers in percent: All: 53; Numbers in
percent: White: 57; Numbers in percent: African-American: 49; Numbers
in percent: Other race/ethnicity: 45.
Region: Region 10 Seattle; Numbers in percent: All: 60; Numbers in
percent: White: 62; Numbers in percent: African-American: 53; Numbers
in percent: Other race/ethnicity: 51.
Source: GAO analysis of weighted enhanced data.
[End of table]
Race and Other Factors Influence ALJ Decisions for Some Claimant
Groups:
When we controlled for a comprehensive range of factors that could
affect disability decision making by ALJs, we identified a number of
variables, including race, which influence the likelihood that a
claimant is allowed benefits.[Footnote 22] Specifically, we found that
numerous variables representing medical and nonmedical criteria that
are used in the disability decision-making process had a statistically
significant influence on ALJ decisions. We also found that participants
in the decision-making process, such as attorneys and translators,
influenced ALJ decisions. In addition, our statistical model shows that
a claimant's race affects ALJ decisions for some but not all groups of
claimants. Finally, other factors that, like race, are not part of the
hearings process also affect ALJ decision making. For example, male
claimants and claimants with low incomes are less likely to be awarded
benefits. However, as with almost all statistical analyses, we cannot
be certain whether the differences we identified are due to unequal
treatment, limitations in our data, or some combination of the two.
Medical and Nonmedical Criteria Affect ALJ Decision Making:
Consistent with SSA's disability decision-making process, the results
of our statistical model show that a number of variables representing
key criteria used in the process have a statistically significant
effect on the likelihood of allowance. For example, claimants with 3 or
more impairments were more likely to be allowed than claimants with 1-
2 impairments, and claimants with 1 or more severe impairments were
more likely to be allowed than claimants with no severe impairments.
Moreover, claimants with the physical capacity to perform light work,
sedentary, and less than sedentary work were more likely to be allowed
than claimants with the physical capacity to perform heavy work.
Furthermore, claimants who did not have the mental capacity to perform
unskilled work were more likely to be allowed than claimants with the
mental capacity to perform such work. In addition, we found that
claimants who were 50 years old or older were more likely to be allowed
than claimants who were 18-24 years old. Finally, claimants with 10 or
more years of employment were more likely to be allowed than claimants
with less than 2 years of employment.
Participants in the Hearings Process also Influence ALJ Decisions:
Our statistical analyses also show that the presence of various
participants in the hearings process also affects ALJ allowances. For
example, claimants who were present at the hearing were more likely to
be allowed than claimants who were not present at the hearing. In
addition, claimants were less likely to be awarded benefits if a
vocational expert testified at their hearing than claimants who did not
have a vocational expert testify at their hearing. Also, claimants who
had translators at the hearing (i.e., for claimants who do not speak
English proficiently) were less likely to be awarded benefits than
claimants who did not have translators (i.e., who presumably do speak
English proficiently). Finally, claimants who were represented by an
attorney or a person who is not an attorney (such as a legal aide,
relative, or friend) were more likely to be allowed than claimants who
had no representative.[Footnote 23]
Effect of Race on ALJ Decisions Varies among Claimant Groups:
Our statistical analyses also show that, after controlling for a range
of factors, a claimant's race also affects ALJ decisions for some
groups of claimants. Specifically, we found no statistically
significant difference in the likelihood of being awarded benefits
between white claimants and claimants from other, non-African-American
racial/ethnic groups. However, this result is likely due to our
controlling for the presence of translators at hearings. Before
controlling for the presence of translators, claimants from other
racial/ethnic groups were less likely to be awarded benefits than white
claimants. After controlling for the presence of translators, there is
no statistically significant effect of the other race/ethnic claimants'
category on the likelihood of allowance. The relatively high incidence
of translators among claimants from other racial/ethnic backgrounds
explains why we found no statistically significant differences in the
likelihood of being awarded benefits between whites and claimants from
other racial/ethnic groups.[Footnote 24]
When we compared white claimants with African-American claimants, we
found statistically significant differences in the likelihood of
allowance, but only among claimants who had no representation.[Footnote
25] For example, among claimants with no representation, the odds of
being allowed benefits for African-Americans were about one-half the
odds of being allowed for whites.[Footnote 26] In contrast, among
claimants with attorney representation, we found no statistically
significant difference in the likelihood of allowances between whites
and African-Americans.[Footnote 27]
In addition, when we compared the effect of having attorney
representation with the effect of not having attorney representation,
we found that these effects also vary by race. That is, we found that
the effect of attorney representation is larger for African-American
claimants than it is for white claimants. Specifically, the odds of
being allowed benefits for African-American claimants with attorney
representation were more than 5 times higher than the odds of being
allowed for African-American claimants without attorney
representation. In comparison, the odds of being allowed benefits for
white claimants with attorney representation were three times higher
than the odds of being allowed benefits for white claimants with no
representation.[Footnote 28]
Finally, we used another statistical technique--the Oaxaca
decomposition--to analyze differences in ALJ allowances between
African-American and white claimants. Consistent with the results from
our other analyses, we found that, among claimants with attorney
representation, differences between African-Americans and whites can be
explained largely by differences in other factors included in our
model, whereas among claimants without attorney representation,
differences between African-Americans and whites were explained to a
lesser degree by differences in other factors in our model.[Footnote
29] These results are particularly important because a larger
percentage of African-American claimants do not have attorneys (39
percent) in comparison with white claimants (29 percent).
Although several possible explanations exist for why attorney
representation increases a claimant's likelihood of being awarded
benefits, we cannot empirically explain why the effect of attorney
representation is greater for African-Americans. According to two
attorneys affiliated with the National Organization of Social Security
Claimant Representatives (NOSSCR), attorneys increase the claimant's
likelihood of being awarded benefits by (1) providing assistance with
the development of evidence over and above SSA's efforts to develop
evidence[Footnote 30] and (2) preparing claimants to improve their
effectiveness and credibility as witnesses. Another possible
explanation for why attorney representation influences the likelihood
of being awarded benefits is that attorneys often screen cases to
select claimants with strong cases.[Footnote 31] However, given the
data available to us, we cannot empirically explain why attorney
representation has a stronger effect for African-American claimants
than for white claimants.
As mentioned earlier, claimants who are represented by persons other
than attorneys--such as legal aides, friends, or family--are also more
likely to be allowed than claimants with no representation. When we
conducted additional analyses on the effect these nonattorney
representatives had on allowances by race, we found, regardless of
race, claimants who were represented by nonattorneys had a greater
likelihood of being awarded benefits than claimants who were not
represented. Nevertheless, we also found that differences by race
persisted after controlling for nonattorney representatives.[Footnote
32]
Other Factors Not Part of the Decision-Making Process also Influence
ALJ Allowances:
Finally, our statistical analyses found that additional factors not
part of the decision-making process--including the claimant's earnings,
geographical location, and sex--influence the ALJ allowance decision.
For example, we found that claimants with higher levels of earnings
were more likely to be awarded benefits than those who have low
earnings levels. In particular, the odds of being allowed benefits for
claimants who earned over $20,000 per year were 3 times higher than the
odds of being allowed benefits for claimants who earned less than
$5,000 per year, and the odds of being allowed for claimants who earn
$5,000-$20,000 per year were 2 times higher than for claimants who earn
less than $5,000 per year. In addition, the odds of being allowed
benefits for claimants whose hearings took place in the Boston Region
were approximately 2 times higher than for claimants whose hearings
took place in other regions, after controlling for other
factors.[Footnote 33] Finally, the odds of being allowed benefits for
claimants who are men were approximately three-quarters as high as for
female claimants.
Data Limitations Prevent Definitive Conclusions Regarding the Cause of
Unexplained Racial Differences in ALJ Decisions:
The existence of persistent, unexplained differences by race and other
factors not used as criteria in the decision-making process--after we
controlled for as many factors as the data allowed--means that we
cannot rule out the possibility that claimant groups are being treated
unequally. However, two limitations, common to almost all multivariate
analyses, prevent us from definitively determining whether unexplained
differences in allowance decisions by claimant groups are due to
discrimination or other forms of bias in the decision-making process.
First, differences between claimant groups may be a result of a lack of
precision in some of the variables in the model. For example, when the
severity of a claimant's impairment is evaluated by the medical
examiners, they are placed in one of five categories. However, the
categories may not capture subtle differences in impairment severity.
This is true for many of the categorical variables in the
model.[Footnote 34] With more detailed information on severity and
other factors, we might have been able to better explain differences by
race. Second, differences that we see in the likelihood of being
awarded benefits between claimant groups may be the result of a lack of
data on certain factors that are relevant for our analysis. For
example, data on claimants' access to medical care are not available.
In the past, SSA developed data on the source of the claimant's medical
care--a proxy for the quality of the medical care and a factor that
determines the weight that is placed on a given piece of evidence.
However, SSA told us that it stopped developing these data due to
resource constraints. Other factors such as these, if included in the
model, might further explain some of the differences we found in ALJ
decisions by race, as well as other differences we found, for example,
by sex and income.
In addition, our model's results concerning the effect of attorney
representation on ALJ decisions might be somewhat inflated due to SSA's
systematic exclusion of certain cases--namely, the exclusion of denied
ALJ decisions that were appealed to the Appeals Council--from the
enhanced data we used for our study. An upward bias of this effect
could occur because the denied cases that were appealed (and,
therefore, excluded from our dataset) exhibited a higher rate of
attorney representation than the denied cases that were not appealed.
However, further analyses suggest that our estimates of the different
effects of attorney representation by race (that is, the larger effect
of attorney representation for African-Americans) are not likely to be
inflated. (See appendix I for a detailed discussion of our analyses of
this limitation.):
SSA's Approach to Quality Assurance Reviews Limits Its Ability to
Ensure the Accuracy and Fairness of ALJ Decisions:
Analytical, sampling, and data weaknesses in SSA's approach to quality
assurance reviews limit its ability to ensure the accuracy and fairness
of ALJ decisions. SSA does not analyze ALJ decisions by race, which
limits its ability to identify, correct, and prevent unwarranted racial
differences in allowance rates. In addition, weaknesses in the quality
assurance review's sampling methods and data availability present
problems.
SSA's quality assurance review of ALJ decisions includes numerous
analyses of ALJ decisions, including analyses of support rates and
whether an ALJ abused his or her discretion or committed an error of
law.[Footnote 35] In addition, SSA analyzes ALJ decisions by various
claimant characteristics such as the claimant's age and the region
where the disability decision was issued.[Footnote 36] However, SSA
does not currently analyze ALJ decisions by race.[Footnote 37] By not
analyzing ALJ decisions by race as part of its ongoing quality
assurance review, SSA is limited in its ability to identify, correct,
and prevent unwarranted racial differences in allowance rates. At the
time of our review, SSA had no plans to analyze decisions by race as
part of its ongoing quality assurance review of ALJ decisions.
Even if SSA decided to analyze ALJ decisions and related data by race,
weaknesses in the quality assurance review's sampling methods would
present problems. Specifically, SSA is limited in its ability to
conduct certain types of analyses by race because SSA does not take
measures to ensure the presence of a sufficient number of claimants in
each race/ethnicity category for its quality assurance reviews. As
noted in our previous report,[Footnote 38] since 1997, SSA no longer
stratifies the selection of ALJ decisions by race (i.e., by African-
American and non-African-American) when selecting a random sample of
cases--a practice that had helped to ensure that SSA had a sufficient
number of cases of African-American claimants in its sample to analyze
ALJ decisions by race. Unless SSA over-samples cases for African-
Americans and claimants from other racial/ethnic groups, certain
analyses by race/ethnicity cannot be performed. For example, due to the
low number of African-American claimants in SSA's enhanced data, we
were unable to analyze differences by race/ethnicity for those ALJ
decisions that were considered to be unsupported by the reviewing
judge. Furthermore, we were unable to analyze by race whether the ALJ
followed the appropriate procedures in deciding whether the claimant
was eligible for disability benefits.[Footnote 39] Because these
analyses for African-American cases would rely on a relatively small
number of decisions, conclusions related to race could be statistically
unreliable.
SSA also excludes cases that are appealed to the Appeals Council from
its quality assurance review--a sampling weakness that affects SSA's
entire quality assurance review process. SSA estimates that about 75
percent of ALJ denials are appealed. By excluding such cases, SSA may
be running the risk of using a nonrepresentative sample in its analyses
of ALJ decisions and, consequently, drawing incorrect conclusions about
the accuracy and fairness of ALJ decisions, although we did not find
large differences in the sample we used for our analysis.[Footnote 40]
For example, cases are often appealed on the basis of an alleged error
of law or abuse of discretion; therefore, SSA may be omitting cases
with information that could be valuable in assessing the fairness of
ALJ decisions.
According to SSA officials, SSA does not include appealed cases in its
ALJ quality assurance review because generally SSA has yet to render a
final decision for them. SSA believes that the Appeals Council decision
could be inappropriately influenced by information resulting from the
quality assurance review of these "live" cases. However, SSA officials
informed us that they are considering ways to include appealed cases in
their ALJ quality assurance review for which final decisions have been
rendered.[Footnote 41] According to SSA officials, this would require
establishing a special control system so that SSA can recover the files
and tapes after the cases have been reviewed at the Appeals Council and
have received a final decision.[Footnote 42] SSA officials said this
approach would also require removing any information regarding the
final decision from the files, so that the reviewing judge can assess
the cases without being influenced by this additional information. One
concern that SSA has about reviewing appealed cases that have received
a final decision is the 1-to 2-year time lag before the quality
assurance review could take place.[Footnote 43] SSA officials informed
us that reviewing cases 1 to 2 years after the original ALJ decision
could affect the quality of the data and the effectiveness of the
quality assurance review process.[Footnote 44] Another concern that SSA
has regarding this approach is that reviewing judges would know which
cases were appealed to the Appeals Council and might analyze appealed
cases differently from those cases that were not appealed.
In addition to having analytical and sampling weaknesses, SSA's quality
assurance reviews do not collect certain types of data that could be
useful in conducting its analyses of ALJ decisions. For example, SSA
does not collect information on the types and sources of medical
evidence in the claimant's file. Types of medical evidence could
include treatment records, narrative reports, results of laboratory or
clinical tests, and frequency of medical visits, and sources of medical
evidence could include treating physician, other specialist, hospital
(inpatient), and clinic or hospital (outpatient). This kind of
information, which was collected by SSA in the past, but is no longer
collected, could be used to study the impact of various types and
sources of medical evidence on the likelihood that a claimant would be
awarded benefits. For example, as part of its quality assurance review,
SSA would be able to analyze the relationship between claimants' access
to health care (as measured by the presence of a treating physician or
the number or length of doctor visits) and ALJ decisions to allow or
deny benefits. SSA would also be able to determine whether the extent
of medical evidence in the claimant's file is affected by attorney
representation, or the race, sex, or income of the claimant.
Additionally, since 1990, SSA has significantly scaled back its
collection of race/ethnicity data, leaving gaps for certain claimant
groups. As we noted in our previous report,[Footnote 45] SSA requests
information on race/ethnicity from individuals who complete a form to
request a new or replacement Social Security card. The race/ethnicity
field on this form is a voluntary field and the data collected are
self-reported. Although this process is still in place, only a small
portion of SSNs is issued in this manner today. Since 1990, SSA has
been assigning SSNs to newborns through its Enumeration at Birth (EAB)
program, and SSA does not collect race/ethnicity data through the EAB
program. In fiscal year 2002, approximately 90 percent of the 4.2
million original SSN cards issued to U.S. citizens were through the EAB
program. Consequently, SSA has not collected race data for those
individuals who obtained their SSNs through the EAB program and, under
its current approach, SSA would not generally collect these data in the
future.[Footnote 46] As future generations obtain their SSNs through
the EAB program, the number and proportion of claimants for whom SSA
lacks race/ethnicity data are likely to increase.
This lack of race data has implications on SSA's ability--and the
ability of other federal agencies that rely on SSA for race/ethnicity
data--to conduct certain types of analyses by race/ethnicity. Although
we had sufficient race data for our study,[Footnote 47] SSA's future
ability to identify, correct, and prevent racial differences in ALJ
decisions will be hampered by this growing lack of data for claimants
who received their SSNs through the EAB program. This growing lack of
data will also affect the ability of other federal agencies that rely
on SSA for race/ethnicity data, such as the Centers for Medicare and
Medicaid Services, to conduct research and produce reports to ensure
the fairness of their programs.
During our review, SSA decided to collect race/ethnicity data on
individuals applying for disability or other Social Security benefits
at the time of application. Previously, SSA did not collect race data
at the point of application for disability benefits since race is not a
criterion in the disability determination process. However, during our
review, SSA decided to collect data on race/ethnicity because,
according to SSA officials, the agency now views collecting and
analyzing these data as important for research purposes and to ensure
the race neutrality of its programs. SSA recently set up a task group
to explore implementation issues. Even though this decision to collect
race information has been made, SSA has not set a start date, and SSA
officials anticipate that implementation of this endeavor will be a
lengthy process.
Conclusions:
Our analyses of SSA's enhanced data from its quality assurance reviews
show that for claimants who are not represented by attorneys, there are
differences in the likelihood of being awarded benefits between
African-Americans and whites that cannot be explained by other factors
related to the disability decision-making process. Although our
empirical results cannot be used as proof that discrimination or some
other form of bias exists, the results also do not rule out this
possibility. As such, our findings raise important program integrity
issues for SSA in terms of its ability to ensure that disability
decisions are made accurately and fairly. Relatedly, the results of our
analyses raise questions regarding the role and influence that attorney
and nonattorney representatives have in the decision-making process;
although SSA does not require claimants to have representation, the
results of our analysis show that claimants with representation are
more likely to be awarded benefits than those without representation.
The lower likelihood of being awarded benefits for other claimant
groups, including non-English-speaking claimants with translators,
claimants with low income, and claimants who are men, also raise
questions about the fairness of SSA's disability decision-making
process. These findings point to the need for SSA's continued efforts
to understand racial and other differences in ALJ allowances. While SSA
may not have control over the sources of some of these differences,
understanding the sources of these differences is the key to taking the
necessary steps to demonstrate the neutrality of its decision-making
process and to eliminate and prevent unwarranted differences in
allowance rates.
SSA's approach to quality assurance reviews has limited its ability to
understand these differences and take appropriate action, if necessary,
in several ways. For example, because SSA does not over-sample cases
for African-Americans and claimants from other racial/ethnic groups and
analyze the ALJ decisions by race, it cannot determine whether
inaccuracies in ALJ decision making, such as errors of law and abuses
of discretion, occur with the same likelihood for claimants of
different racial/ethnic backgrounds. Additionally, by not including
cases appealed to the Appeals Council with those that undergo an ALJ
quality review, SSA's sample is potentially nonrepresentative of all
ALJ decisions. Moreover, the agency misses an opportunity to analyze
precisely those cases that are more likely to have had an alleged error
of law or abuse of discretion by the ALJ. Finally, SSA no longer
collects data on type and source of medical evidence that would allow
for more careful analyses of the accuracy and fairness of ALJ
decisions. Although SSA has significantly scaled back its collection of
race/ethnicity data since 1990, we applaud the agency's recent decision
to begin collecting these data at the point of application for
disability and other benefits, which will help to fill some of the gaps
in its race/ethnicity data.
Recommendations:
To improve SSA's ability to ensure the accuracy and fairness of ALJ
decisions, we recommend that the agency conduct ongoing analyses of ALJ
decisions by race/ethnicity, as well as by other claimant groups (such
as claimants with attorneys and nonattorneys, with translators, with
low incomes, from certain regions and claimants who are men). In doing
so, it should take the following steps to enhance its approach to
quality assurance reviews:
* Collect data on the types and sources of medical evidence in the
claimant's file to better understand the agency's and attorney's role
in the development of evidence.
* Analyze differences in support (accuracy) rates, in addition to
differences in allowance decisions.
* Over-sample the selection of ALJ decisions by African-American
claimants and, to the extent possible, other racial/ethnic groups to
ensure that SSA has a sufficient number of cases to conduct analyses of
ALJ decisions by race.
* Publish methods used and results as part of its biennial reporting on
the findings of its disability hearings quality review process.
* If needed, take actions to correct and prevent any unwarranted
differences in allowance and support rates among racial/ethnic and
other claimant groups.
To further ensure the accuracy and fairness of ALJ decisions for
various claimant groups, we recommend that SSA conduct in-depth
investigations of cases (e.g., case studies) to better understand
differences in ALJ allowances for certain claimant groups, including
claimants with and without an attorney. The results of these
investigations should also be published in the biennial reports. If
needed, SSA should take actions to correct and prevent any unwarranted
differences in allowance rates among these claimant groups.
To ensure that SSA uses a sample that is representative of all ALJ
decisions in its quality assurance review, we recommend that the agency
restructure its sampling process to incorporate cases that are appealed
to SSA's Appeals Council in the quality assurance review sample. These
appealed cases should be analyzed together with, rather than separate
from, the rest of SSA's quality assurance sample.
In light of the methodological complexities associated with analyzing
ALJ decisions, we recommend that SSA establish an advisory panel
comprised of external experts in a range of disciplines--including
statistics/econometrics, design methodology, law, medicine, vocational
training, and disability--to provide leadership, oversight, and
technical assistance with respect to conducting these and other quality
assurance reviews of ALJ decisions.
Agency Comments:
We provided a draft of this report to SSA for comment. In its written
comments, SSA said that our report was useful and timely and agreed
with all of our recommendations. SSA also indicated that it intends to
go further. For example, SSA noted that, as part of its overall plan to
improve the disability determination process, it intends to look at all
factors that may produce adverse impacts based on race, ethnicity,
national origin, or gender. In addition, SSA is currently developing
recommendations on how to collect meaningful data on race and
ethnicity. SSA's comments, as well as its recently proposed plan for
improving the disability determination process, are printed in appendix
III.
We are sending copies of this report to the Social Security
Administration, appropriate congressional committees, and other
interested parties. We will also make copies available to others on
request. In addition, the report will be available at no charge on
GAO's Web site at http://www.gao.gov.
If you or your staff have any questions concerning this report, please
call me or Carol Dawn Petersen, Assistant Director, at (202) 512-7215.
Staff acknowledgments are listed in appendix IV.
Robert E. Robertson:
Director, Education, Workforce, and Income Security Issues:
Signed by Robert E. Robertson:
[End of section]
Appendix I: Scope and Methods:
To determine whether decisions by Administrative Law Judges (ALJs) to
allow disability claims were affected by the race of the claimant, we
developed a model of ALJ decision making that tested for racial
differences after controlling for other factors related to the
disability decision-making process. These factors included (1) factors
that represent criteria in the decision-making process; (2) factors
that represent participants in the decision-making process; and (3)
factors that are not part of, but may influence, the decision-making
process. To conduct our analysis, we employed logistic regression
models and Oaxaca decomposition methods. We used data from the Social
Security Administration's (SSA) quality assurance review at the
hearings level, which we refer to as the enhanced data. The enhanced
data contain detailed information--some of which was previously
unavailable to GAO--on medical and vocational factors for a sample of
7,908 SSA claimants.
Prior to constructing these models, we conducted analyses related to
data quality. Given our previously reported concerns about the degree
to which the enhanced data are representative,[Footnote 48] we
conducted tests to determine whether the enhanced data were
sufficiently representative and reliable for our analyses.
Specifically, in these analyses, we sought to determine (1) whether the
more detailed medical and vocational information included in the
enhanced data set were sufficiently important to justify using this
restricted sample of claimants and (2) whether the sample of claimants
for which the enhanced data were available was representative of the
broader population of claimants.
We developed our analyses and models in consultation with GAO
methodologists, expert consultants, and SSA officials.[Footnote 49]
This appendix is organized into five sections: Section 1 describes the
data that were used in the analysis of potential racial disparities, as
well as data that were used in the analyses of data quality. Section 2
describes analyses and results related to our tests of data quality and
reliability. Section 3 provides background on the weighting scheme used
in the analysis, as well as details on sampling errors. Section 4
describes the variables that were included in our baseline and final
models and presents the results of these final models and the Oaxaca
decomposition analysis. Finally, Section 5 presents the limitations of
our analyses.
Section 1: Databases and Information Sources:
We used two types of SSA data to conduct our analyses: (1) the enhanced
data, which were derived from a sample of SSA claimants, and (2)
administrative data, which were derived from the universe of claimants.
The enhanced data are compiled by the Division of Disability Hearings
Quality (DDHQ) within SSA's Office of Quality Assurance (OQA). These
data are compiled as part of an ongoing quality assurance review of the
decision-making accuracy of ALJs. The review involves an examination of
the initial, reconsideration, and hearings level decisions by a medical
consultant, a disability examiner, and an ALJ.
The administrative data were obtained from several sources. For each
adjudicative level (the initial and reconsideration, hearings, and
Appeals Council levels), SSA has an electronic file that contains a
limited amount of data for each claimant. In addition to these three
datasets, we used earnings data from SSA's Master Earnings File (MEF).
We used these data for the various analyses that are described more
fully in later sections. In brief, we used the enhanced data for our
"severity analysis," which sought to determine whether the enhanced
data contained variables that were better measures of the claimant's
medical severity than the variables contained in SSA's administrative
files. We used the administrative data for our "nonresponder analysis,"
which sought to determine whether the enhanced data were
representative. Based on the results of the severity and nonresponder
analyses, we decided to use the enhanced data for our analysis of
potential racial disparities.
Table 3 presents the datasets that we used in our analyses, the
decision-making level to which the particular dataset pertains, the
analyses for which we used the particular dataset, and the years of
data and the specific variables that were used in our analyses.
Table 3: Data Used in Our Analyses:
Dataset: Enhanced data; Decision-making levels to which data generally
pertain: Hearings level[A]; Analyses conducted: Final analysis and
severity analysis; Years used in analyses: Oct. 1997-Sept. 2000;
Information that was used in analyses: Claimant's impairments, severity
of impairments, alcohol or drug abuse, consultative exam requested,
number of impairments, number of severe impairments, residual
functional capacity of claimant, mental residual functional capacity of
claimant, occupational type, years of employment, occupational skill
level, years of education, literacy, age, type of representation, other
hearing participants (vocational expert, medical expert, translator,
and claimant), sex, race, claim type, year of decision, region, and the
allowance decision at the hearing level.
Dataset: 831 data[B]; Decision-making levels to which data generally
pertain: Initial and reconsideration levels; Analyses conducted:
Nonresponder analysis; Years used in analyses: 1990-2000; Information
that was used in analyses: Claimant's age, sex, race, body systems
affected by the impairment(s) alleged at the initial and
reconsideration levels, occupational years, years of education, whether
the claimant obtained a consultative exam, and claim type.
Dataset: Office of Hearings and Appeals Case Control System (CCS)
data[B]; Decision-making levels to which data generally pertain:
Hearings level; Analyses conducted: Nonresponder analysis; Years used
in analyses: Oct. 1997-Sept. 2000; Information that was used in
analyses: Claimant's body system affected by the impairment(s) alleged
at the hearing level, type of representation, other hearing
participants (vocational expert, medical expert, translator and
claimant), and the allowance decision at the hearing level.
Dataset: Appeals Council Automated Processing System (ACAPS)[B];
Decision-making levels to which data generally pertain: Appeals Council
level; Analyses conducted: Nonresponder analysis; Years used in
analyses: 1997-2002; Information that was used in analyses: Indicator
of whether claimant appealed the allowance decision at the hearing
level and allowance decision at the Appeals Council level.
Dataset: Master Earnings File[B]; Decision-making levels to which data
generally pertain: N/A; Analyses conducted: Final analysis; Years used
in analyses: 1948-2002; Information that was used in analyses: Yearly
individual earnings.
Source: Social Security Administration.
[A] The enhanced data also contain variables pertaining to conditions
or actions taken at the initial and reconsideration levels for a sample
of claimants who have appealed to an Administrative Law Judge.
[B] The use of this database was restricted to only those observations
that had matches with the SSNs that were included in the enhanced data
or in the sample from which the enhanced data were developed.
[End of table]
Section 2: Data Reliability Tests:
To ensure that the SSA data were sufficiently reliable for our
analyses, we conducted detailed data reliability assessments of the
five datasets that we used. We restricted these assessments, however,
to the specific variables and records that were pertinent to our
analyses. We found that all of the datasets were sufficiently reliable
for use in our analyses.
Enhanced Data:
Our reliability assessment of the enhanced data included two steps.
First, to assess the general reliability of the enhanced data that we
used in our analysis, we interviewed officials from SSA's DDHQ about
procedures to ensure the enhanced data's reliability. On the basis of
discussions with DDHQ officials, we concluded that careful data entry
controls and processing procedures are applied in maintaining the
reliability of the enhanced data. Second, to assess the completeness of
the enhanced data that we used in our analyses, we conducted frequency
analysis of relevant fields. On the basis of the results of our
frequency tests of relevant data elements and our interviews with SSA
officials, we concluded that the enhanced data were sufficiently
complete and accurate for use in our final analyses.[Footnote 50]
SSA Administrative Files:
Our assessment of the reliability of the relevant data from SSA's
administrative files (831, CCS, and ACAPS) also involved several steps.
For each dataset, we assessed the general reliability of relevant data
(i.e., the specific variables and records that we would use in our
analyses) by interviewing SSA officials on their processes and
procedures to ensure data quality. To determine the completeness of the
data, we conducted frequency analyses of relevant fields. Finally, to
assess the accuracy of the relevant fields, we matched the enhanced
data with the data from the administrative files and compared the
values of the fields common to both data sets.
On the basis of our review of existing information, we concluded that,
while not optimal, adequate quality controls are in place to ensure the
reliability of the specific variables from SSA's administrative files
that we used in our analysis, and the results of our frequency tests
and our examination of matched data confirmed that we had sufficiently
complete and accurate data for use in our nonresponder
analyses.[Footnote 51]
With respect to earnings data from the MEF, SSA provided us with
complete earnings data for each person included in the enhanced data.
We were unable to test the accuracy of earnings data from the MEF
because comparable data were not available in the enhanced data.
However, SSA's OQA annually reviews the accuracy of the MEF earnings
data by extracting individual earnings from the reports submitted by
employers and self-employed individuals and by then comparing the
reported earnings to earnings posted to the MEF. To further ensure the
accuracy of these data, SSA also now mails Social Security statements
to individuals who have earnings and are age 25 years or older to
inform individuals about their earnings.
Additional Tests of Enhanced Data:
For our final analyses, the enhanced data have some significant
advantages over SSA's administrative files. Most importantly, the
enhanced data contain information on medical severity[Footnote 52] that
are not available in SSA's administrative files and were not available
to GAO when our agency issued a report in 1992 concerning similar
analyses.[Footnote 53] Data on medical severity are important because
severity is a key factor in the disability allowance decision. This and
other variables in the enhanced dataset are developed from a sample of
hearings claimants. However, as highlighted in our 2002 report, we were
concerned that the sample from which the enhanced data are developed
had the potential for being unrepresentative of the population of
hearings claimants.[Footnote 54]
The enhanced data may not be representative because SSA uses only a
fraction of the files that it selects for its sample of ALJ decisions.
SSA selects the sample for the enhanced data using an automated system
that selects a stratified random sample every month from the population
of claimants who had a hearing.[Footnote 55] However, over the period
that we examined (1997-2000), roughly 50 percent of the files that were
selected to be in the sample were not obtained. There were three
primary reasons for why files were not obtained:
* The files were still in use because claimants appealed the ALJ
decision to the next level, that is, to the Appeals Council.[Footnote
56]
* The files were misplaced or misfiled.
* The files were still in use because there were still pending payment
decisions for cases that were allowed.
In addition, not all of the files that were obtained underwent the
three reviews needed to be included in our sample (i.e., reviews by an
ALJ, a medical consultant, and a disability examiner). According to SSA
officials we interviewed, this was due to time and budget constraints.
After the monthly sample was selected, DDHQ requested the files from
various storage facilities and regional offices. As the files came in,
they were chosen to be reviewed by a medical team on a "first come,
first serve" basis--that is, files were selected until a sufficient
number (as deemed by DDHQ) of files for a given time period was
reached. The remaining files were not reviewed by a medical team.
Additionally, some of the files that were supposed to be reviewed by an
ALJ were not reviewed. In the end, of the 50,022 that were sampled from
1997 through 2000, only 9,082 files underwent all three reviews. For
purposes of exposition, we will call the sample of 9,082 files that
underwent all three reviews the "responders" and the sample of files
that were not obtained the "nonresponders."[Footnote 57]
Given our concerns about the degree to which the enhanced data were
representative, before we decided to use the data, we needed to
determine (1) whether the additional information contained in the
enhanced data were critical to our analyses (in terms of obtaining the
best possible estimates of the variables in our model of ALJ
decisions)[Footnote 58] and, if so, (2) whether the enhanced data were
representative of the population of claimants at the hearings level. To
answer these questions, we conducted (1) a "severity analysis" to
assess whether the additional information contained in the enhanced
data were critical to our analyses and (2) a "nonresponder analysis" to
test whether the enhanced data are representative. We developed these
statistical tests in consultation with our methodologists, our external
expert consultants, and SSA officials. The results of these analyses
indicated that the enhanced data were critical for our study and were
of sufficient quality for analyses of ALJ allowance decisions.
Severity analysis:
The goal of our severity analysis was to determine which data would
allow us to obtain the best possible estimates of the variables in our
model of ALJ decisions. Ensuring that we obtain such estimates requires
that we use data that are as precise as possible (i.e., those that best
capture the actual characteristics of the claimant and the case).
Imprecision in the measurement of variables that are statistically
significantly related to the disability determination process could
result in estimates of the differences between racial categories in
allowances that are inappropriately larger or smaller than the real
difference.
To determine whether variables in the enhanced data more precisely
measured severity and other factors that influence ALJ decisions than
variables in the 831 and CCS data, we conducted our severity
analysis.[Footnote 59] The specific objective of this test was to
determine (using regression analysis) whether the severity data in the
enhanced data increased the explanatory power of the model. If it did
not, we could use the severity data from SSA's administrative files,
which are available for all claimants, thus avoiding any problems of
representativeness.
To conduct our severity analysis, we compared two models of the ALJ's
disability decision (that is, the dependent variable is the ALJ's
decision to allow or deny disability benefits) for the same group of
claimants. Specifically:
* Model A contained only those independent variables from the enhanced
data that are also available in SSA's 831 and CCS files.[Footnote 60]
* Model B contained all of the independent variables in Model A, plus
several variables that are only available in the enhanced data,
including variables that measure medical severity at the hearings level
(impairment severity, number of impairments, number of severe
impairments, and residual functional capacity) as well as variables
that measure the occupational skill level of the claimant and whether
the claimant is literate.[Footnote 61]
To determine whether the additional variables in the enhanced data
improved our ability to explain allowance rates, we used logistic
regression analysis to estimate both of these models. We then compared
the predictive power of each model and the significance of the
additional variables in Model B.
In summary, we found that Model A (which excluded the additional
variables that are available in the enhanced data) explained roughly 27
percent of the variation in allowances, while Model B (which included
those additional variables) explained over 40 percent. The results of
this analysis show that the additional variables that are included in
Model B increase the overall explanatory power of the model.
Furthermore, the additional variables in Model B--such as the degree of
medical severity, the number of impairments, the number of severe
impairments, and measures of the claimant's residual functional
capacity and mental residual functional capacity--were all highly,
statistically significant predictors of the ALJ allowance decision.
Nonresponder analysis:
To determine whether the enhanced data were sufficiently
representative, we conducted our nonresponder analysis, which tested
whether the responders' cases (those that were included in SSA's
enhanced data) were statistically significantly different from the
nonresponders' cases (those that were excluded from SSA's enhanced
data). It is important to note that we can only compare the responders
and nonresponders on characteristics that are observable (that is, for
which data are available).[Footnote 62] Since we are controlling for
many of these same variables in our final model, differences we see in
observable characteristics in our nonresponder analysis are not
critical in and of themselves. However, if few differences exist
between responders and nonresponders in observable characteristics, it
is more likely (though not guaranteed) that few differences exist
between them in unobservable characteristics. Thus, if the nonresponder
analysis reveals little or no differences between the two groups we are
afforded some measure of confidence that the two groups are similar in
unobservable characteristics.
Our nonresponder analysis consisted of a series of tests to compare
responders and nonresponders with respect to (1) the allowance decision
and (2) characteristics that are related to the allowance decision,
including claimant characteristics and characteristics related to
administrative processes. To conduct these tests we used data available
from SSA's administrative files (831, CCS, and ACAPS).[Footnote 63] We
conducted both regression analyses and bivariate tests. Regression
methods and related test statistics were used to estimate differences
between responders and nonresponders after simultaneously controlling
for other factors that could influence nonresponse. Chi-squared tests
and t-tests were used to evaluate the differences in specific
characteristics when other characteristics were ignored. These
differences were estimated first for responders and nonresponders
overall, and then for responders and nonresponders within categories of
race, and then for responders and nonresponders within categories of
claimants who were allowed or denied at the hearings level.
The regression analysis showed no statistically significant differences
between responders and nonresponders in many factors that are related
to the decision-making process. Specifically, responders were not
statistically significantly different from nonresponders in most
medical, vocational, and demographic characteristics including body
system, age, sex, and race. However, the results of the regression also
showed that responders differed from nonresponders in some
administrative characteristics. Specifically, claimants who had
attorney or nonattorney representation or who had a medical expert
testify at the hearing, or had consultative exams were significantly
less likely to be responders. We also found small, but statistically
significant, differences in the year of the decision and the region.
Table 4 summarizes the results of the nonresponder regression analysis,
and presents these comparisons for (1) all responders and
nonresponders, (2) African-American responders and African-American
nonresponders, and (3) white responders and white nonresponders.
Table 4: Statistically Significant Differences between Responder and
Nonresponder Groups, as Estimated with Logistic Regression:
Variable or variable groups in the model: Medical, vocational, and
demographic characteristics:
Variable or variable groups in the model: Body system categories[A];
Statistically significant differences between: All responders and all
nonresponders: Medical, vocational, and demographic characteristics:
No[B]; Statistically significant differences between: African-
American responders and African-American nonresponders: Medical,
vocational, and demographic characteristics: No; Statistically
significant differences between: White responders and white
nonresponders: Medical, vocational, and demographic characteristics:
No.
Variable or variable groups in the model: Age group categories;
Statistically significant differences between: All responders and all
nonresponders: Medical, vocational, and demographic characteristics:
No; Statistically significant differences between: African-American
responders and African-American nonresponders: Medical, vocational,
and demographic characteristics: No; Statistically significant
differences between: White responders and white nonresponders:
Medical, vocational, and demographic characteristics: No.
Variable or variable groups in the model: Sex; Statistically
significant differences between: All responders and all nonresponders:
Medical, vocational, and demographic characteristics: No;
Statistically significant differences between: African-American
responders and African-American nonresponders: Medical, vocational,
and demographic characteristics: No; Statistically significant
differences between: White responders and white nonresponders:
Medical, vocational, and demographic characteristics: No.
Variable or variable groups in the model: African-American;
Statistically significant differences between: All responders and all
nonresponders: Medical, vocational, and demographic characteristics:
No; Statistically significant differences between: African-American
responders and African-American nonresponders: Medical, vocational,
and demographic characteristics: Not applicable; Statistically
significant differences between: White responders and white
nonresponders: Medical, vocational, and demographic characteristics:
Not applicable.
Variable or variable groups in the model: Years of education
categories; Statistically significant differences between: All
responders and all nonresponders: Medical, vocational, and demographic
characteristics: No[C]; Statistically significant differences
between: African-American responders and African-American
nonresponders: Medical, vocational, and demographic characteristics:
No[D]; Statistically significant differences between: White
responders and white nonresponders: Medical, vocational, and
demographic characteristics: No.
Variable or variable groups in the model: Administrative
characteristics:
Variable or variable groups in the model: Attorney representation;
Statistically significant differences between: All responders and all
nonresponders: Medical, vocational, and demographic characteristics:
Yes; Statistically significant differences between: African-American
responders and African-American nonresponders: Medical, vocational,
and demographic characteristics: No; Statistically significant
differences between: White responders and white nonresponders:
Medical, vocational, and demographic characteristics: Yes.
Variable or variable groups in the model: Nonattorney representation;
Statistically significant differences between: All responders and all
nonresponders: Medical, vocational, and demographic characteristics:
Yes; Statistically significant differences between: African-American
responders and African-American nonresponders: Medical, vocational,
and demographic characteristics: No; Statistically significant
differences between: White responders and white nonresponders:
Medical, vocational, and demographic characteristics: Yes.
Variable or variable groups in the model: Medical expert at hearing;
Statistically significant differences between: All responders and all
nonresponders: Medical, vocational, and demographic characteristics:
Yes; Statistically significant differences between: African-American
responders and African-American nonresponders: Medical, vocational,
and demographic characteristics: No; Statistically significant
differences between: White responders and white nonresponders:
Medical, vocational, and demographic characteristics: Yes.
Variable or variable groups in the model: Translator at hearing;
Statistically significant differences between: All responders and all
nonresponders: Medical, vocational, and demographic characteristics:
No; Statistically significant differences between: African-American
responders and African-American nonresponders: Medical, vocational,
and demographic characteristics: No; Statistically significant
differences between: White responders and white nonresponders:
Medical, vocational, and demographic characteristics: No.
Variable or variable groups in the model: Vocational expert at hearing;
Statistically significant differences between: All responders and all
nonresponders: Medical, vocational, and demographic characteristics:
No; Statistically significant differences between: African-American
responders and African-American nonresponders: Medical, vocational,
and demographic characteristics: No; Statistically significant
differences between: White responders and white nonresponders:
Medical, vocational, and demographic characteristics: No.
Variable or variable groups in the model: Supplemental Security Income
(SSI) claim; Statistically significant differences between: All
responders and all nonresponders: Medical, vocational, and demographic
characteristics: No; Statistically significant differences between:
African-American responders and African-American nonresponders:
Medical, vocational, and demographic characteristics: No;
Statistically significant differences between: White responders and
white nonresponders: Medical, vocational, and demographic
characteristics: No.
Variable or variable groups in the model: Consultative examination;
Statistically significant differences between: All responders and all
nonresponders: Medical, vocational, and demographic characteristics:
Yes; Statistically significant differences between: African-American
responders and African-American nonresponders: Medical, vocational,
and demographic characteristics: Yes; Statistically significant
differences between: White responders and white nonresponders:
Medical, vocational, and demographic characteristics: Yes.
Variable or variable groups in the model: Year of decision;
Statistically significant differences between: All responders and all
nonresponders: Medical, vocational, and demographic characteristics:
Yes; Statistically significant differences between: African-American
responders and African-American nonresponders: Medical, vocational,
and demographic characteristics: Yes; Statistically significant
differences between: White responders and white nonresponders:
Medical, vocational, and demographic characteristics: Yes.
Variable or variable groups in the model: Region; Statistically
significant differences between: All responders and all nonresponders:
Medical, vocational, and demographic characteristics: Yes;
Statistically significant differences between: African-American
responders and African-American nonresponders: Medical, vocational,
and demographic characteristics: No; Statistically significant
differences between: White responders and white nonresponders:
Medical, vocational, and demographic characteristics: Yes.
Source: GAO analysis of 831 and CCS data.
Note: Dependent variable is 1 if the claimant is a responder and 0 if
the claimant is a nonresponder.
[A] Body system categories represent the body system that was affected
by the claimant's impairment.
[B] Although the test for the effect of all of the body system
categories combined was not significant, the category for all
respiratory disorders was significant at the 95-percent confidence
level for this sample.
[C] Although the test for all of the education categories combined was
not significant, the category for less than 9 years of education was
significant at the 95-percent confidence level for this sample.
[D] Although the test for all of the education categories combined was
not significant, the category for between 12 and 16 years of education
was significant at the 95-percent confidence level for this sample.
[End of table]
To further explore the extent of the differences we identified in the
regression analysis, we conducted a series of statistical tests of
cross tabulations. The results of these tests confirm that--with
respect to the claimant's body system, age, sex, and race--the
responders did not differ significantly from the nonresponders. The
results also indicate that the statistically significant differences
between responders and nonresponders in allowances and several
administrative variables were not large in magnitude. Table 5 shows
that responders differed from nonresponders with respect to
statistically significant administrative factors from table 3 by 0 to 4
percentage points.
Table 5: Tabulations of Statistically Significant Administrative
Factors (from Table 4) for Responders and Nonresponders:
Variable: Attorney representation; Percent of responders in this
category: 70; Percent of nonresponders in this category: 73.
Variable: Nonattorney representation; Percent of responders in this
category: 11; Percent of nonresponders in this category: 11.
Variable: Medical expert at hearing; Percent of responders in this
category: 15; Percent of nonresponders in this category: 16.
Variable: Consultative examination requested; Percent of responders in
this category: 70; Percent of nonresponders in this category: 73.
Year of decision:
Year of decision: 1997; Percent of responders in this category: 8;
Percent of nonresponders in this category: 8.
Year of decision: 1998; Percent of responders in this category: 36;
Percent of nonresponders in this category: 33.
Year of decision: 1999; Percent of responders in this category: 33;
Percent of nonresponders in this category: 33.
Year of decision: 2000; Percent of responders in this category: 22;
Percent of nonresponders in this category: 26.
Region:
Region: 1. Boston; Percent of responders in this category: 12;
Percent of nonresponders in this category: 10.
Region: 2. New York; Percent of responders in this category: 10;
Percent of nonresponders in this category: 10.
Region: 3. Philadelphia; Percent of responders in this category: 10;
Percent of nonresponders in this category: 10.
Region: 4. Atlanta; Percent of responders in this category: 9;
Percent of nonresponders in this category: 10.
Region: 5. Chicago; Percent of responders in this category: 11;
Percent of nonresponders in this category: 10.
Region: 6. Dallas; Percent of responders in this category: 10;
Percent of nonresponders in this category: 10.
Region: 7. Kansas; Percent of responders in this category: 11;
Percent of nonresponders in this category: 10.
Region: 8. Denver; Percent of responders in this category: 10;
Percent of nonresponders in this category: 10.
Region: 9. San Francisco; Percent of responders in this category: 9;
Percent of nonresponders in this category: 10.
Region: 10. Seattle; Percent of responders in this category: 9;
Percent of nonresponders in this category: 11.
Source: GAO analysis of 831 and CCS data.
[End of table]
When we repeated the above analysis for subgroups of the sample--
African-American claimants, non-African-American claimants, claimants
who were allowed benefits, and claimants who were denied benefits--our
findings were generally consistent across most subgroups. That is, when
we compared responders and nonresponders who were African-American,
non-African-American, and who were allowed benefits, we found virtually
no differences in demographic, medical, and vocational characteristics,
and only small differences in administrative characteristics.
However, among the sample of claimants who were denied benefits, we
found a substantial difference in the rates of attorney representation
among responders and nonresponders. Specifically, 59 percent of
responders who were denied benefits were represented by attorneys and
67 percent of nonresponders who were denied benefits were represented
by attorneys. This means that claimants who were denied benefits and
had attorneys are underrepresented in the sample. Such under-
representation could result in inflated estimates of the effect of
attorney representation on allowances. Further analysis of denied
responders and nonresponders by race did not reveal variations in the
differences in attorney representation between responders and
nonresponders by race. (See below for our further analysis of this
effect by race.) Therefore, we are confident that, even though denied
claimants with attorneys are under-represented overall, our finding
indicating that the effect of attorney representation is greater for
African-American claimants than for white claimants is valid.
Ultimately, the small differences we found between responders and
nonresponders on only administrative factors, and the similarity of the
differences in responders and nonresponders for African-Americans and
whites, makes us reasonably confident that our estimates of the effects
of the factors on ALJ decisions are not severely biased by nonresponse.
At the same time, the statistical significance of the associations
between nonresponse and a number of administrative characteristics as
well as the cumulative effect of a number of small differences between
responders and nonresponders may be nontrivial. [Footnote 64]
Section 3: Weighting and Sampling Errors:
We conducted all of our analyses of the enhanced data using probability
weights because the enhanced data were based on a stratified sample
rather than the universe of hearings claimants. The weight for each
claimant equals the inverse probability of the claimant being selected
into the sample. To control for the effect of the stratified sampling
scheme on the estimates, we conducted all of our regression analysis
using computer software that adjusts the estimates according to the
weighting scheme.
Because the analysis was based on a sample, the reported estimates have
sampling errors associated with them.[Footnote 65] Sampling errors for
the estimates of allowance rates for whites, African-Americans, and
claimants from other racial/ethnic groups were calculated at the 95-
percent confidence level. This means that in 95 out of 100 chances, the
actual percentage would fall within the range defined by the estimate,
plus or minus the sampling error. For example, the estimate that 63
percent of claims filed by whites were allowed at the hearing level has
a sampling error of 2 percent. This means that a 95-percent chance
exists, or we can be 95-percent confident, that the actual percentage
falls between 61 percent and 65 percent. Similarly, for each variable
in our logistic regression model, a standard error was computed that
reflects the precision of the estimated odds ratio. The odds ratio for
each variable in the logistic regressions was considered to be
significantly different from 1.0 (1.0 implies no difference in the
odds) when the 95-percent confidence interval around the estimate of
the odds ratio did not contain 1.0. For example, the 95-percent
confidence interval for the variable indicating that a translator was
present at the hearing was 0.39 to 0.90. This interval did not contain
1.00 and, therefore, the translator variable is considered
statistically significant.
Section 4: Statistical Analysis:
To choose the appropriate variables for our model of ALJ decision
making, we reviewed pertinent literature and consulted with SSA
officials and outside experts.[Footnote 66] The final model included
variables that are either measures or approximate measures for (1)
factors that represent criteria used in decision-making process, (2)
factors that represent participants in the decision-making process, (3)
factors that are not part of the decision-making process but may have
an influence on it, and (4) interaction variables reflecting the
relationship between factors that are not criteria used in the
decision-making process.
A number of variables in our model are measures for medical and
nonmedical criteria used in 4 of the 5 steps of the disability
decision-making process.[Footnote 67] Specifically, the medical
factors that we controlled for included type of medical impairment
(such as disorders of the back and musculoskeletal disorders), the
degree of impairment severity, alcohol or drug abuse alleged,[Footnote
68] consultative examination requested, number of impairments, number
of severe impairments, residual functional capacity, and mental
residual functional capacity. The nonmedical factors that we controlled
for included occupational categories (blue collar, white collar, and
service sector), years employed, occupational skill level, educational
level, literacy, and age.
We also controlled for factors that represent participants in the
decision-making process. These variables include whether the claimant
was represented by an attorney or a nonattorney, such as a relative,
legal aide, or friend; whether a medical and/or vocational expert
testified at the hearing; whether a translator attended the hearing,
and whether the claimant attended the hearing.
Finally, we controlled for factors that are not part of the decision-
making process, but for which we have reason to believe may influence
the disability decision-making process. These variables include the
claimant's claim type,[Footnote 69] the year of the hearing decision,
and the SSA region.[Footnote 70] Other factors that we controlled for
include demographic factors such as sex, race, and earnings.[Footnote
71] Although these factors are not part of the ALJ decision-making
process,[Footnote 72] we included these variables in our analysis to
find out whether they are related to ALJ allowance decisions.
After estimating our initial model, we found several variables that did
not represent criteria but that had a statistically significant
influence on ALJ decisions. To investigate whether the effects of these
variables on ALJ decisions differed by the claimant's race, we
incorporated interaction terms into our model and tested their
significance, both simultaneously and sequentially. Specifically, to
test whether racial groups are treated differently when they are
represented by attorneys, we included an interaction term between race
and attorney representation. Similarly, we included an interaction term
to test whether racial groups are treated differently when they are
represented by persons other than attorneys. We also included
interaction terms between race and the following variables: sex,
earnings, translator, year of the decision,[Footnote 73] and region.
Logistic Regression:
We used logistic regression to estimate the model--an appropriate
technique when the dependent variable is binary, or has two categories,
such as benefits being allowed or denied.
On the basis of our initial analyses, we found that the interaction
term for race and attorney representation was the only statistically
significant interaction term in the model. We removed the remaining
insignificant interaction terms from the model because removing them
had little effect on our estimates of the variables left in the model.
We did not, however, remove insignificant variables that were not
interaction terms from our models since our primary objective was to
estimate the effect of race "net" of other factors we believed could
potentially influence the allowance decision, regardless of how small
or statistically insignificant they were.
The results of two of our models--our baseline model and our final
model containing the significant interaction term--are presented in
table 6. The first numerical column in table 6 presents the percentage
of claimants within each variable category. The second and third
columns present odds ratios that are estimated for each variable in our
baseline and final models, respectively.[Footnote 74] The
interpretation of the odds ratio for a particular variable depends on
whether the variable is a dummy variable or a categorical variable. For
dummy variables, a statistically significant odds ratio that is
greater/less than 1.00 indicates that claimants with that
characteristic are more/less likely to be allowed than claimants
without it. For categorical variables, a statistically significant odds
ratio that is greater/less than 1.00 indicates that claimants in that
category are more/less likely to be allowed than the claimants in the
comparison category.[Footnote 75]
Table 6: Results of Baseline and Final Models of ALJ Allowance
Decisions:
Categories for explanatory variables: Factors that represent criteria
in the decision-making process:
Categories for explanatory variables: Medical criteria: Impairments
(dummy variables):
Explanatory variables: Impairments (dummy variables): Disorders of the
back; Weighted percent of claimants in this category: 31%; Predicted
odds ratio for baseline model: 0.83; Predicted odds ratio for final
model: 0.83.
Explanatory variables: Osteoarthritis and allied disorders; Weighted
percent of claimants in this category: 10%; Predicted odds ratio for
baseline model: 0.84; Predicted odds ratio for final model: 0.84.
Explanatory variables: Other musculoskeletal disorders; Weighted
percent of claimants in this category: 18%; Predicted odds ratio for
baseline model: 0.63**; Predicted odds ratio for final model: 0.64**.
Explanatory variables: Mental retardation; Weighted percent of
claimants in this category: 1%; Predicted odds ratio for baseline
model: 0.83; Predicted odds ratio for final model: 0.80.
Explanatory variables: Mood disorders; Weighted percent of claimants
in this category: 24%; Predicted odds ratio for baseline model: 0.92;
Predicted odds ratio for final model: 0.92.
Explanatory variables: Schizophrenia; Weighted percent of claimants in
this category: 2%; Predicted odds ratio for baseline model: 0.97;
Predicted odds ratio for final model: 1.00.
Explanatory variables: Other mental disorders; Weighted percent of
claimants in this category: 17%; Predicted odds ratio for baseline
model: 0.59**; Predicted odds ratio for final model: 0.59**.
Explanatory variables: Diabetes; Weighted percent of claimants in this
category: 9%; Predicted odds ratio for baseline model: 1.16; Predicted
odds ratio for final model: 1.14.
Explanatory variables: Other endocrine disorders; Weighted percent of
claimants in this category: 4%; Predicted odds ratio for baseline
model: 1.13; Predicted odds ratio for final model: 1.11.
Explanatory variables: Ischemic heart; Weighted percent of claimants
in this category: 4%; Predicted odds ratio for baseline model: 1.17;
Predicted odds ratio for final model: 1.17.
Explanatory variables: Hypertension; Weighted percent of claimants in
this category: 5%; Predicted odds ratio for baseline model: 0.58**;
Predicted odds ratio for final model: 0.57**.
Explanatory variables: Other cardiovascular disorders; Weighted
percent of claimants in this category: 4%; Predicted odds ratio for
baseline model: 0.92; Predicted odds ratio for final model: 0.93.
Explanatory variables: Neurological disorders; Weighted percent of
claimants in this category: 14%; Predicted odds ratio for baseline
model: 1.11; Predicted odds ratio for final model: 1.11.
Explanatory variables: Respiratory disorders; Weighted percent of
claimants in this category: 7%; Predicted odds ratio for baseline
model: 0.93; Predicted odds ratio for final model: 0.93.
Explanatory variables: Neoplasms; Weighted percent of claimants in
this category: 2%; Predicted odds ratio for baseline model: 2.94**;
Predicted odds ratio for final model: 2.85**.
Explanatory variables: Other disorders; Weighted percent of claimants
in this category: 17%; Predicted odds ratio for baseline model: 1.39*;
Predicted odds ratio for final model: 1.39*.
Categories for explanatory variables: Medical criteria: Severity of
impairment (categorical variable):
Explanatory variables: Not severe; Weighted percent of claimants in
this category: 11%; Predicted odds ratio for baseline model: 1.00;
Predicted odds ratio for final model: 1.00.
Explanatory variables: Moderate; Weighted percent of claimants in this
category: 55%; Predicted odds ratio for baseline model: 1.30;
Predicted odds ratio for final model: 1.26.
Explanatory variables: Moderately severe; Weighted percent of
claimants in this category: 20%; Predicted odds ratio for baseline
model: 2.52**; Predicted odds ratio for final model: 2.46**.
Explanatory variables: Meets listing; Weighted percent of claimants in
this category: 11%; Predicted odds ratio for baseline model: 49.31**;
Predicted odds ratio for final model: 48.97**.
Explanatory variables: Insufficient medical evidence; Weighted percent
of claimants in this category: 3%; Predicted odds ratio for baseline
model: 3.71**; Predicted odds ratio for final model: 3.65**.
Categories for explanatory variables: Medical criteria: Medical
criteria: Drug abuse (dummy variable):
Explanatory variables: Alcohol or drug abuse; Weighted percent of
claimants in this category: 1%; Predicted odds ratio for baseline
model: 0.62; Predicted odds ratio for final model: 0.62.
Categories for explanatory variables: Medical criteria: Source of
medical care (dummy variable):
Explanatory variables: Consultative examination requested; Weighted
percent of claimants in this category: 15%; Predicted odds ratio for
baseline model: 1.07; Predicted odds ratio for final model: 1.06.
Categories for explanatory variables: Medical criteria: Number of
impairments (categorical variable):
Explanatory variables: 1-2 impairments; Weighted percent of claimants
in this category: 36%; Predicted odds ratio for baseline model: 1.00;
Predicted odds ratio for final model: 1.00.
Explanatory variables: 3-4 impairments; Weighted percent of claimants
in this category: 39%; Predicted odds ratio for baseline model:
1.49**; Predicted odds ratio for final model: 1.49**.
Explanatory variables: 5 or more impairments; Weighted percent of
claimants in this category: 25%; Predicted odds ratio for baseline
model: 2.08**; Predicted odds ratio for final model: 2.08**.
Categories for explanatory variables: Medical criteria: Number of
severe impairments categorical variable):
Explanatory variables: No severe impairments; Weighted percent of
claimants in this category: 14%; Predicted odds ratio for baseline
model: 1.00; Predicted odds ratio for final model: 1.00.
Explanatory variables: 1 severe impairment; Weighted percent of
claimants in this category: 47%; Predicted odds ratio for baseline
model: 1.77*; Predicted odds ratio for final model: 1.81*.
Explanatory variables: 2 severe impairments; Weighted percent of
claimants in this category: 26%; Predicted odds ratio for baseline
model: 2.33**; Predicted odds ratio for final model: 2.40**.
Explanatory variables: 3 or 4 severe impairments; Weighted percent of
claimants in this category: 13%; Predicted odds ratio for baseline
model: 2.36**; Predicted odds ratio for final model: 2.43**.
Categories for explanatory variables: Medical criteria: Residual
functional capacity (categorical variable):
Explanatory variables: Heavy or medium; Weighted percent of claimants
in this category: 17%; Predicted odds ratio for baseline model: 1.00;
Predicted odds ratio for final model: 1.00.
Explanatory variables: Light (nonexertional restrictions); Weighted
percent of claimants in this category: 26%; Predicted odds ratio for
baseline model: 1.89**; Predicted odds ratio for final model: 1.91**.
Explanatory variables: Light (exertional restrictions); Weighted
percent of claimants in this category: 7%; Predicted odds ratio for
baseline model: 3.53**; Predicted odds ratio for final model: 3.49**.
Explanatory variables: Sedentary; Weighted percent of claimants in
this category: 9%; Predicted odds ratio for baseline model: 2.42**;
Predicted odds ratio for final model: 2.42**.
Explanatory variables: Less than sedentary; Weighted percent of
claimants in this category: 8%; Predicted odds ratio for baseline
model: 13.69**; Predicted odds ratio for final model: 13.74**.
Explanatory variables: Not applicable (mental RFC or not severe);
Weighted percent of claimants in this category: 29%; Predicted odds
ratio for baseline model: 1.30; Predicted odds ratio for final model:
1.31.
Explanatory variables: Not determinable; Weighted percent of claimants
in this category: 4%; Predicted odds ratio for baseline model: 1.80*;
Predicted odds ratio for final model: 1.81*.
Categories for explanatory variables: Medical criteria: Mental
residual functional capacity (dummy variable):
Explanatory variables: Does not meet mental demands of unskilled work;
Weighted percent of claimants in this category: 8%; Predicted odds
ratio for baseline model: 30.97**; Predicted odds ratio for final
model: 31.97**.
Categories for explanatory variables: Nonmedical criteria:
Categories for explanatory variables: Nonmedical criteria:
Occupational categories (categorical variable)[A]:
Explanatory variables: White collar; Weighted percent of claimants in
this category: 28%; Predicted odds ratio for baseline model: 1.00;
Predicted odds ratio for final model: 1.00.
Explanatory variables: Service sector; Weighted percent of claimants
in this category: 23%; Predicted odds ratio for baseline model: 0.97;
Predicted odds ratio for final model: 0.96.
Explanatory variables: Blue collar; Weighted percent of claimants in
this category: 37%; Predicted odds ratio for baseline model: 1.06;
Predicted odds ratio for final model: 1.07.
Explanatory variables: No occupation; Weighted percent of claimants in
this category: 11%; Predicted odds ratio for baseline model: 1.08;
Predicted odds ratio for final model: 1.09.
Explanatory variables: Occupation not determinable; Weighted percent
of claimants in this category: 1%; Predicted odds ratio for baseline
model: 2.52; Predicted odds ratio for final model: 2.60.
Categories for explanatory variables: Nonmedical criteria: Years of
employment (categorical variable):
Explanatory variables: Less than 2 years of employment; Weighted
percent of claimants in this category: 22%; Predicted odds ratio for
baseline model: 1.00; Predicted odds ratio for final model: 1.00.
Explanatory variables: 2-4 years of employment; Weighted percent of
claimants in this category: 21%; Predicted odds ratio for baseline
model: 1.26; Predicted odds ratio for final model: 1.25.
Explanatory variables: 5-9 years of employment; Weighted percent of
claimants in this category: 22%; Predicted odds ratio for baseline
model: 1.34; Predicted odds ratio for final model: 1.34.
Explanatory variables: 10 or more years of employment; Weighted
percent of claimants in this category: 32%; Predicted odds ratio for
baseline model: 1.56**; Predicted odds ratio for final model: 1.56**.
Explanatory variables: Not determinable; Weighted percent of claimants
in this category: 3%; Predicted odds ratio for baseline model: 0.73;
Predicted odds ratio for final model: 0.74.
Categories for explanatory variables: Nonmedical criteria:
Occupational skill level (categorical variable):
Explanatory variables: Skilled; Weighted percent of claimants in this
category: 30%; Predicted odds ratio for baseline model: 1.00;
Predicted odds ratio for final model: 1.00.
Explanatory variables: Semiskilled; Weighted percent of claimants in
this category: 37%; Predicted odds ratio for baseline model: 0.88;
Predicted odds ratio for final model: 0.88.
Explanatory variables: Unskilled or has no skill; Weighted percent of
claimants in this category: 32%; Predicted odds ratio for baseline
model: 0.84; Predicted odds ratio for final model: 0.85.
Explanatory variables: No skill information available; Weighted
percent of claimants in this category: 1%; Predicted odds ratio for
baseline model: 1.07; Predicted odds ratio for final model: 1.04.
Categories for explanatory variables: Nonmedical criteria: Education
(categorical variable):
Explanatory variables: Under 6 years of education; Weighted percent of
claimants in this category: 5%; Predicted odds ratio for baseline
model: 1.00; Predicted odds ratio for final model: 1.00.
Explanatory variables: 6-11 years of education; Weighted percent of
claimants in this category: 31%; Predicted odds ratio for baseline
model: 1.00; Predicted odds ratio for final model: 0.99.
Explanatory variables: 12 years of education; Weighted percent of
claimants in this category: 45%; Predicted odds ratio for baseline
model: 0.92; Predicted odds ratio for final model: 0.91.
Explanatory variables: Greater than 12 years of education; Weighted
percent of claimants in this category: 18%; Predicted odds ratio for
baseline model: 1.02; Predicted odds ratio for final model: 1.02.
Explanatory variables: Not determinable; Weighted percent of claimants
in this category: 0.3%; Predicted odds ratio for baseline model: 0.91;
Predicted odds ratio for final model: 0.95.
Categories for explanatory variables: Nonmedical criteria: Literacy
(categorical variable):
Explanatory variables: Literate; Weighted percent of claimants in this
category: 96%; Predicted odds ratio for baseline model: 1.00;
Predicted odds ratio for final model: 1.00.
Explanatory variables: Illiterate; Weighted percent of claimants in
this category: 3%; Predicted odds ratio for baseline model: 1.20;
Predicted odds ratio for final model: 1.19.
Explanatory variables: Literacy not determinable; Weighted percent of
claimants in this category: 1%; Predicted odds ratio for baseline
model: 1.25; Predicted odds ratio for final model: 1.22.
Categories for explanatory variables: Age category[B] (categorical
variable):
Explanatory variables: 18-24 years old; Weighted percent of claimants
in this category: 2%; Predicted odds ratio for baseline model: 1.00;
Predicted odds ratio for final model: 1.00.
Explanatory variables: 25-44 years old; Weighted percent of claimants
in this category: 44%; Predicted odds ratio for baseline model: 1.13;
Predicted odds ratio for final model: 1.14.
Explanatory variables: 45-49 years old; Weighted percent of claimants
in this category: 21%; Predicted odds ratio for baseline model: 1.28;
Predicted odds ratio for final model: 1.29.
Explanatory variables: 50-54 years old; Weighted percent of claimants
in this category: 20%; Predicted odds ratio for baseline model:
2.28**; Predicted odds ratio for final model: 2.31**.
Explanatory variables: 55 years old or over; Weighted percent of
claimants in this category: 13%; Predicted odds ratio for baseline
model: 2.18**; Predicted odds ratio for final model: 2.19**.
Categories for explanatory variables: Factors that represent
participants in the decision-making process:
Categories for explanatory variables: Representation (categorical
variable):
Explanatory variables: No representation; Weighted percent of claimants in this category: 21%; Predicted odds ratio for baseline model: 1.00; Predicted odds ratio for final model: 1.00.
Explanatory variables: Attorney representation[C]; Weighted percent of claimants in this category: 67%; Predicted odds ratio for baseline model: 3.31**; Predicted odds ratio for final model: 2.93**.
Explanatory variables: Other representation; Weighted percent of claimants in this category: 12%; Predicted odds ratio for baseline model: 2.78**; Predicted odds ratio for final model: 2.75**.
Categories for explanatory variables: Other hearing participants (dummy variables);
Explanatory variables: Medical expert; Weighted percent of claimants in this category: 13%; Predicted odds ratio for baseline model: 1.01; Predicted odds ratio for final model: 1.00.
Explanatory variables: Vocational expert; Weighted percent of claimants in this category: 47%; Predicted odds ratio for baseline model: 0.41**; Predicted odds ratio for final model: 0.41**.
Explanatory variables: Translator; Weighted percent of claimants in this category: 4%; Predicted odds ratio for baseline model: 0.59*; Predicted odds ratio for final model: 0.59*.
Explanatory variables: Claimant present at hearing; Weighted percent of claimants in this category: 99%; Predicted odds ratio for baseline model: 2.51**; Predicted odds ratio for final model: 2.55**.
Categories for explanatory variables: Factors that are not part of the
decision-making process:
Categories for explanatory variables: Sex (dummy variable):
Explanatory variables: Male; Weighted percent of claimants in this
category: 47%; Predicted odds ratio for baseline model: 0.73**;
Predicted odds ratio for final model: 0.72**.
Categories for explanatory variables: Race (categorical variable):
Explanatory variables: White; Weighted percent of claimants in this
category: 65%; Predicted odds ratio for baseline model: 1.00;
Predicted odds ratio for final model: 1.00.
Explanatory variables: Other racial/ethnic groups; Weighted percent of
claimants in this category: 11%; Predicted odds ratio for baseline
model: 0.84; Predicted odds ratio for final model: 0.90.
Explanatory variables: African-American[D]; Weighted percent of
claimants in this category: 24%; Predicted odds ratio for baseline
model: 0.73**; Predicted odds ratio for final model: 0.50**.
Categories for explanatory variables: Earnings[E] (categorical
variable):
Explanatory variables: Less than $5,000 per year; Weighted percent of
claimants in this category: 49%; Predicted odds ratio for baseline
model: 1.00; Predicted odds ratio for final model: 1.00.
Explanatory variables: $5,000-$20,000 per year; Weighted percent of
claimants in this category: 37%; Predicted odds ratio for baseline
model: 1.96**; Predicted odds ratio for final model: 1.97**.
Explanatory variables: Greater than $20,000; Weighted percent of
claimants in this category: 14%; Predicted odds ratio for baseline
model: 3.24**; Predicted odds ratio for final model: 3.22**.
Categories for explanatory variables: Claim type (categorical
variable):
Explanatory variables: Supplemental Security Income (SSI); Weighted
percent of claimants in this category: 27%; Predicted odds ratio for
baseline model: 1.00; Predicted odds ratio for final model: 1.00.
Explanatory variables: Concurrent claim; Weighted percent of claimants
in this category: 34%; Predicted odds ratio for baseline model: 1.15;
Predicted odds ratio for final model: 1.16.
Explanatory variables: Disability Insurance (DI); Weighted percent of
claimants in this category: 39%; Predicted odds ratio for baseline
model: 1.12; Predicted odds ratio for final model: 1.13.
Categories for explanatory variables: Year of decision (categorical
variable):
Explanatory variables: 1997; Weighted percent of claimants in this
category: 9%; Predicted odds ratio for baseline model: 1.00; Predicted
odds ratio for final model: 1.00.
Explanatory variables: 1998; Weighted percent of claimants in this
category: 39%; Predicted odds ratio for baseline model: 1.22;
Predicted odds ratio for final model: 1.23.
Explanatory variables: 1999; Weighted percent of claimants in this
category: 33%; Predicted odds ratio for baseline model: 1.33;
Predicted odds ratio for final model: 1.33.
Explanatory variables: 2000; Weighted percent of claimants in this
category: 19%; Predicted odds ratio for baseline model: 1.35;
Predicted odds ratio for final model: 1.35.
Categories for explanatory variables: Region (categorical variable):
Explanatory variables: 1. Boston; Weighted percent of claimants in
this category: 3%; Predicted odds ratio for baseline model: 2.32**;
Predicted odds ratio for final model: 2.31**.
Explanatory variables: 2. New York; Weighted percent of claimants in
this category: 12%; Predicted odds ratio for baseline model: 1.10;
Predicted odds ratio for final model: 1.11.
Explanatory variables: 3. Philadelphia; Weighted percent of claimants
in this category: 11%; Predicted odds ratio for baseline model: 1.15;
Predicted odds ratio for final model: 1.15.
Explanatory variables: 4. Atlanta; Weighted percent of claimants in
this category: 26%; Predicted odds ratio for baseline model: 1.02;
Predicted odds ratio for final model: 1.02.
Explanatory variables: 5. Chicago; Weighted percent of claimants in
this category: 14%; Predicted odds ratio for baseline model: 1.08;
Predicted odds ratio for final model: 1.08.
Explanatory variables: 6. Dallas; Weighted percent of claimants in
this category: 14%; Predicted odds ratio for baseline model: 0.94;
Predicted odds ratio for final model: 0.93.
Explanatory variables: 7. Kansas; Weighted percent of claimants in
this category: 4%; Predicted odds ratio for baseline model: 1.05;
Predicted odds ratio for final model: 1.05.
Explanatory variables: 8. Denver; Weighted percent of claimants in
this category: 3%; Predicted odds ratio for baseline model: 1.06;
Predicted odds ratio for final model: 1.05.
Explanatory variables: 9. San Francisco; Weighted percent of claimants
in this category: 12%; Predicted odds ratio for baseline model: 0.89;
Predicted odds ratio for final model: 0.88.
Explanatory variables: 10. Seattle; Weighted percent of claimants in
this category: 3%; Predicted odds ratio for baseline model: 1.00;
Predicted odds ratio for final model: 1.00.
Categories for explanatory variables: Interaction variables:
Categories for explanatory variables: Race/attorney interaction term
(dummy variables):
Explanatory variables: Categories for explanatory variables: White
claimant with attorney; Weighted percent of claimants in this
category: 46%; Predicted odds ratio for baseline model: N/A; Predicted
odds ratio for final model: 1.00.
Explanatory variables: Claimant from other racial/ethnic group with
attorney; Weighted percent of claimants in this category: 6%;
Predicted odds ratio for baseline model: N/A; Predicted odds ratio for
final model: .87.
Explanatory variables: African-American claimant with attorney;
Weighted percent of claimants in this category: 14%; Predicted odds
ratio for baseline model: N/A; Predicted odds ratio for final model:
1.76**.
Source: GAO analysis of weighted enhanced data.
Notes: The dependent variable is 1 if the claimant is allowed and 0 if
the claimant is not allowed. Variables with an odds ratio of 1.00
represent the excluded category.
* Indicates that the variable is statistically significant at the 95-
percent confidence level.
** Indicates that the variable is statistically significant at the 99-
percent confidence level.
[A] White collar includes professional, technical, or managerial and
clerical and sales occupations. Service includes service occupations.
Blue collar includes all other occupations.
[B] Age reflects the age of the claimant on the hearing date.
[C] In the baseline model, the variable for attorney representation
indicates that, on average, the odds of allowance for claimants with
attorney representation are 3.3 times higher than those for claimants
with no representation. In the final model, the variable for attorney
representation indicates that the odds of allowance for white claimants
with attorney representation are 2.93 times higher than the odds of
allowance for white claimants without attorney representation. The
interpretation of the variable for attorney representation changes in
the final model because interaction terms between race and attorney
representation have been included in the final model. Section 4
explains the interpretation of the interaction terms in greater detail.
[D] In the baseline model, the variable for African-Americans indicates
that, on average, the odds of allowance for African-Americans are 0.73
times as high as the odds of allowance for white claimants. In the
final model, the variable for African-American indicates that the odds
of allowance for African-Americans without attorneys are 0.50 times as
high as the odds of allowance for white claimants without attorneys.
The interpretation of the variable for race changes in the final model
because interaction terms between race and attorney representation have
been included in the model. Section 4 explains the interpretation of
the interaction terms in greater detail.
[E] Earnings are computed as an average of the claimant's earnings for
the 5 years preceding the hearings level decision date.
[End of table]
Due to the presence of the interaction term between attorney
representation and race in the final model, one cannot interpret the
effect of race and attorney representation independent of each other.
Tables 7, 8, and 9 show how to derive and interpret odds ratios for
different race and attorney representation subgroups. Table 7 shows
that, first, the odds of allowance are computed for every race
subgroup. The odds of allowance are equal to the number of claims
allowed divided by the number of claims denied for a particular group.
For example, using the weighted enhanced data, we find that among white
claimants who were not represented by an attorney, 54,981 were allowed
and 57,667 were denied. Thus, the odds of being allowed for a white
claimant that was not represented by an attorney were 0.95 (54,981/
57,667).
The observed odds ratio compares the odds of one group against another.
The ratio is computed by dividing the odds of allowance of one group by
the odds of allowance for another group. For example, the odds of
allowance for African-American and white claimants who were not
represented were 0.49 and 0.95, respectively. Thus, the observed odds
ratio of an African-American claimant who was not represented compared
with a white claimant who was not represented was 0.52 (0.49/0.95). The
column entitled observed odds ratios presents these ratios for each
group, as they compare to whites. Both the odds of allowance and the
observed odds ratio are computed without controlling for other factors
that influence the allowance decision.
If we control for the other factors that influence the allowance
decision using regression analysis, we can estimate the odds ratios
"net" of the influence of other factors--the estimated odds ratio.
These are presented in the last column of table 7 and come from the
estimated odds ratios from the final model in table 6. Specifically,
the last column of table 7 shows that the estimated odds ratio for
claimants from other racial/ethnic groups who are not represented by an
attorney is 0.90, which is not significantly different from 1.00. This
means that after controlling for other factors, the likelihood of
allowance for claimants from other racial/ethnic groups without an
attorney is not significantly different from the likelihood of
allowance for white claimants who are not represented by attorneys (the
comparison group). In contrast, the odds ratio for African-Americans
without attorneys is statistically significantly different from 1.00.
The estimated odds ratio of 0.50 means that the odds of being allowed
benefits for African-Americans without attorneys are one-half as high
as the odds of being allowed benefits for whites without attorneys.
Among claimants who are represented by attorneys, the estimated odds
ratios for claimants from other racial/ethnic groups and for African-
American claimants are not statistically significantly different from
1.00 in comparison with white claimants. This means that among
claimants who are represented by attorneys, the likelihood of allowance
does not differ significantly by race.
Table 7: Observed and Estimated Odds Ratios by Attorney Representation
and Race:
Race: Not represented by an attorney:
Race: White; Allowed: 54,981; Denied: 57,668; Total: 112,649; Odds of
allowance: 0.95; Observed odds ratios: 1.00; Estimated odds ratios:
1.00.
Race: Other racial/ethnic background; Allowed: 11,196; Denied: 17,491;
Total: 28,687; Odds of allowance: 0.64; Observed odds ratios: 0.67;
Estimated odds ratios: 0.90.
Race: African-American; Allowed: 18,281; Denied: 37,028; Total: 55,309;
Odds of allowance: 0.49; Observed odds ratios: 0.52; Estimated odds
ratios: 0.50*.
Race: Represented by an attorney:
Race: White; Allowed: 191,225; Denied: 86,046; Total: 277,271; Odds of
allowance: 2.22; Observed odds ratios: 1.00; Estimated odds ratios:
1.00.
Race: Other racial/ethnic background; Allowed: 23,390; Denied: 15,326;
Total: 38,716; Odds of allowance: 1.53; Observed odds ratios: 0.69;
Estimated odds ratios: 0.78.
Race: African-American; Allowed: 50,932; Denied: 34,590; Total: 85,522;
Odds of allowance: 1.47; Observed odds ratios: 0.66; Estimated odds
ratios: 0.88.
Source: GAO analysis of weighted enhanced data.
*Statistically different from 1.00.
[End of table]
The last column of table 7 also shows the effect of race among
claimants who have attorneys. Using the estimated odds ratios from our
final model, table 8 shows how to compute these odds ratios. They are
computed by multiplying the odds ratio for the race variable[Footnote
76] by the odds ratio for the attorney/race interaction variable from
the final model (reported in table 6). For example, to derive the odds
ratio for African-American claimants with attorneys compared with white
claimants with attorneys, we multiplied the odds ratio for African-
American claimants (0.50) by the odds ratio for the interaction
variable between African-Americans and attorney representation (1.76).
Table 8: Computations for Odds Ratios for Different Racial Groups That
Are Represented by an Attorney:
Race: White; Odds ratio for race effect: 1.00; Multiplied by: Odds
ratio for race/attorney interaction term: 1.00; Equals the: Odds ratio
for claimants with attorneys who are a certain race relative to white
claimants with attorneys: 1.00.
Race: Other racial/ethnic background; Odds ratio for race effect:
0.90; Multiplied by: Odds ratio for race/attorney interaction term:
0.87; Equals the: Odds ratio for claimants with attorneys who are a
certain race relative to white claimants with attorneys: 0.78[A].
Race: African-American; Odds ratio for race effect: 0.50; Multiplied
by: Odds ratio for race/attorney interaction term: 1.76; Equals the:
Odds ratio for claimants with attorneys who are a certain race
relative to white claimants with attorneys: 0.88[A].
Source: GAO analysis of weighted enhanced data.
[A] Not statistically different from 1.00.
[End of table]
Taken alone, the odds ratio for the interaction variable for African-
Americans with attorney representation (1.76) indicates that the effect
of attorney representation is bigger for African-American claimants
than for whites. Specifically, the odds of being allowed benefits for
African-Americans with attorney representation are 1.76 times higher
than the odds of being allowed benefits for white claimants with
attorney representation. However, this does not mean that African-
American claimants with attorneys have higher odds of allowance than
white claimants with attorneys. Since African-Americans without
attorneys start with lower odds of allowance (0.50 times) than white
claimants without attorneys, the additional impact of attorneys for
African-Americans does not boost their odds of allowance above the odds
of allowance for white claimants with attorneys.[Footnote 77]
Using the estimated odds ratios from our final model, table 9 shows how
to compute the effect of attorney representation within a particular
race group--to compare the odds of allowance between claimants of the
same race who have attorneys with those that do not have attorneys. For
example, to derive the odds ratio for African-American claimants with
attorneys compared with African-American claimants without attorneys,
we multiply the odds ratio for attorney representation (2.93) by the
odds ratio for the interaction variable between African-Americans and
attorney representation (1.76). The product (5.16) means that the odds
of being allowed benefits for African-American claimants with attorneys
are 5.16 times higher than the odds of being allowed benefits for
African-American claimants without attorneys. In contrast, the odds of
being allowed benefits for white claimants with attorneys are 2.93
times higher than the odds of being allowed benefits for white
claimants without attorneys.
Table 9: Computations for Odds Ratios for Claimants of the Same Race
with and without Attorney Representation:
Race: White; Odds ratio for attorney representation: 2.93; Multiplied
by: Odds ratio for race/attorney interaction term: 1.00; Equals the:
Odds ratio for claimants with attorneys who are a certain race
relative to claimants without attorneys from the same race: 2.93*.
Race: Other racial/ethnic background; Odds ratio for attorney
representation: 2.93; Multiplied by: Odds ratio for race/attorney
interaction term: 0.87; Equals the: Odds ratio for claimants with
attorneys who are a certain race relative to claimants without
attorneys from the same race: 2.55*.
Race: African-American; Odds ratio for attorney representation: 2.93;
Multiplied by: Odds ratio for race/attorney interaction term: 1.76;
Equals the: Odds ratio for claimants with attorneys who are a certain
race relative to claimants without attorneys from the same race:
5.16*.
Source: GAO analysis of weighted enhanced data.
*Statistically different from 1.00.
[End of table]
In addition, the average effect of attorney representation is measured
with the odds ratio for the attorney representation variable in the
baseline model (before the interaction terms were added). Table 6 shows
that, on average, the odds of being allowed benefits for claimants with
attorney representation are 3.3 times higher than the odds of being
allowed benefits for claimants without attorney representation.
Due to the lower rates of attorney representation among denied
claimants in our sample, our estimate of the effect of attorney
representation may be inflated. Specifically, we found that the rate of
attorney representation was lower among responders who were denied
benefits (59 percent) than among nonresponders who were denied benefits
(66 percent).[Footnote 78] This difference in rates of attorney
representation between denied responders and denied nonresponders could
result in an overestimation of the effect of attorney representation on
ALJ decisions. This can be shown with an analysis comparing the
influence of attorney representation on ALJ decisions for responders
and nonresponders. Table 10 shows that among the responders, the odds
of allowance for claimants with and without attorneys were 1.97 and
0.69, respectively. The observed odds ratio comparing responders with
attorneys to responders without attorneys is 2.88--which means that,
the odds of allowance for responders with attorneys were 2.88 times
higher than the odds of allowance for responders without attorneys.
Similarly, among the nonresponders, the odds of allowance for claimants
with and without attorneys were 1.75 and 0.87, respectively. The
observed odds ratio comparing nonresponders with attorneys to
nonresponders without attorneys is 1.90. When we compare the size of
the effect of attorney representation for these two groups--that is,
2.88 for responders compared with 1.90 for nonresponders--we find that
the effect of attorney representation is 1.51 times higher among
responders than among nonresponders. Consequently, we conclude that, by
analyzing only responders, we are overestimating or inflating the
effect of attorney representation.
Table 10: Effect of Attorney Representation on ALJ Decisions for
Responders and Nonresponders:
Attorney representation: Responder:
Attorney representation: Has attorney; Allowed: 71,259; Denied: 36,092;
Odds of allowance: 1.97; Observed odds ratio of allowance: 2.88; Ratio
of odds ratios: 1.51.
Attorney representation: No attorney; Allowed: 17,442; Denied: 25,427;
Odds of allowance: 0.69; Observed odds ratio of allowance: [Empty];
Ratio of odds ratios: [Empty].
Attorney representation: Nonresponder:
Attorney representation: Has attorney; Allowed: 325,249; Denied:
196,796; Odds of allowance: 1.65; Observed odds ratio of allowance:
1.90; Ratio of odds ratios: [Empty].
Attorney representation: No attorney; Allowed: 87,825; Denied: 101,085;
Odds of allowance: 0.87; Observed odds ratio of allowance: [Empty];
Ratio of odds ratios: [Empty].
Source: GAO analysis of weighted CCS data.
[End of table]
A precise estimate of how greatly the size of the effect of attorney
representation is inflated by nonresponse would require complete
information about nonresponders, which we lack. Our best estimate
without more complete information on nonresponders is that the actual
effect of attorney representation in our sample of responders is higher
than in the entire sample (including responders and nonresponders), by
a factor of about 1.4. (See table 11.):
Table 11: Effect of Attorney Representation on ALJ Decisions for
Responders and the Entire Sample:
Attorney representation: Responder:
Attorney representation: Has attorney; Allowed: 71,259; Denied: 36,092;
Odds of allowance: 1.97; Observed odds ratio of allowance: 2.88; Ratio
of odds ratios: 1.41.
Attorney representation: No attorney; Allowed: 17,442; Denied: 25,427;
Odds of allowance: 0.69; Observed odds ratio of allowance: [Empty];
Ratio of odds ratios: [Empty].
Attorney representation: Entire Sample:
Attorney representation: Has attorney; Allowed: 396,508; Denied:
232,888; Odds of allowance: 1.70; Observed odds ratio of allowance:
2.05; Ratio of odds ratios: [Empty].
Attorney representation: No attorney; Allowed: 105,267; Denied:
126,512; Odds of allowance: 0.83; Observed odds ratio of allowance:
[Empty]; Ratio of odds ratios: [Empty].
Source: GAO analysis of weighted CCS data.
[End of table]
In order to determine the extent to which this overestimation affects
our finding that African-American claimants without attorneys were less
likely to be allowed than white claimants without attorneys, we
compared the effect of attorney representation on allowance decisions
for responders and nonresponders by race. As shown in table 12, among
African-Americans claimants, the observed odds ratio for responders
with attorneys versus responders without attorneys is 3.40 (in other
words, the odds of allowance for responders with attorneys were 3.40
times higher than the odds of allowance for responders without
attorneys), whereas the observed odds ratio for nonresponders is 2.06
(that is, the odds of allowance for nonresponders with attorneys were
2.06 times higher than the odds of allowance for nonresponders without
attorneys). The ratio of these two effects is 1.65. In other words, for
African-American claimants, the effect of attorney representation is
1.65 times higher for responders than for nonresponders. When we do a
similar computation for white claimants, we find that the effect of
attorney representation is 1.60 times higher for responders than for
nonresponders. The relatively small difference between 1.65 and 1.60
leads us to conclude that the over-estimation of attorney
representation does not vary by race.
Table 12: Effect of Attorney Representation on ALJ Decisions for
Responders and Nonresponders, by Race:
(Continued From Previous Page)
African-American claimants:
Responder; Attorney representation: Has attorney; Allowed: 16,223;
Denied: 8,150; Odds of allowance: 1.99; Observed odds ratio of
allowance: 3.40; Ratio of odds ratios: 1.65.
Attorney representation: Nonresponder: No attorney; Allowed:
Nonresponder: 3,499; Denied: Nonresponder: 5,973; Odds of allowance:
Nonresponder: 0.59; Observed odds ratio of allowance: Nonresponder:
[Empty]; Ratio of odds ratios: Nonresponder: [Empty].
Nonresponder; Attorney representation: Has attorney; Allowed: 75,505;
Denied: 45,700; Odds of allowance: 1.65; Observed odds ratio of
allowance: 2.06; Ratio of odds ratios: [Empty].
Attorney representation: No attorney; Allowed: 19,954; Denied: 24,862;
Odds of allowance: 0.80; Observed odds ratio of allowance: [Empty];
Ratio of odds ratios: [Empty].
White claimants:
Responder; Attorney representation: Has attorney; Allowed: 47,147;
Denied: 23,648; Odds of allowance: 1.99; Observed odds ratio of
allowance: 2.92; Ratio of odds ratios: 1.60.
Attorney representation: Nonresponder: No attorney; Allowed:
Nonresponder: 10,991; Denied: Nonresponder: 16,116; Odds of allowance:
Nonresponder: 0.68; Observed odds ratio of allowance: Nonresponder:
[Empty]; Ratio of odds ratios: Nonresponder: [Empty].
Nonresponder; Attorney representation: Has attorney; Allowed: 211,805;
Denied: 128,031; Odds of allowance: 1.65; Observed odds ratio of
allowance: 1.83; Ratio of odds ratios: [Empty].
Attorney representation: Attorney representation: No attorney;
Allowed: Allowed: 55,668; Denied: Denied: 61,478; Odds of allowance:
Odds of allowance: 0.91; Observed odds ratio of allowance: Observed
odds ratio of allowance: [Empty]; Ratio of odds ratios: Ratio of odds
ratios: [Empty].
Source: GAO analysis of weighted CCS data.
[End of table]
Table 13 shows that the over-estimation of attorney representation also
does not vary by race when we compare responders to the entire sample
of responders and nonresponders.
Table 13: Effect of Attorney Representation on ALJ Decisions for
Responders and the Entire Sample by Race:
African-American claimants:
Responder; Attorney representation: Has attorney; Allowed: 16,223;
Denied: 8,150; Odds of allowance: 1.99; Observed odds ratio of
allowance: 3.40; Ratio of odds ratios: 1.52.
Attorney representation: Entire sample: No attorney; Allowed: Entire
sample: 3,499; Denied: Entire sample: 5,973; Odds of allowance: Entire
sample: 0.59; Observed odds ratio of allowance: Entire sample: [Empty];
Ratio of odds ratios: Entire sample: [Empty].
Entire sample; Attorney representation: Has attorney; Allowed: 91,728;
Denied: 53,850; Odds of allowance: 1.70; Observed odds ratio of
allowance: 2.24; Ratio of odds ratios: [Empty].
Attorney representation: No attorney; Allowed: 23,453; Denied: 30,835;
Odds of allowance: 0.76; Observed odds ratio of allowance: [Empty];
Ratio of odds ratios: [Empty].
White claimants:
Responder; Attorney representation: Has attorney; Allowed: 47,147;
Denied: 23,648; Odds of allowance: 1.99; Observed odds ratio of
allowance: 2.92; Ratio of odds ratios: 1.47.
Attorney representation: Entire sample: No attorney; Allowed: Entire
sample: 10,991; Denied: Entire sample: 16,116; Odds of allowance:
Entire sample: 0.68; Observed odds ratio of allowance: Entire sample:
[Empty]; Ratio of odds ratios: Entire sample: [Empty].
Entire sample; Attorney representation: Has attorney; Allowed: 258,952;
Denied: 151,679; Odds of allowance: 1.71; Observed odds ratio of
allowance: 1.99; Ratio of odds ratios: [Empty].
Attorney representation: Attorney representation: No attorney;
Allowed: Allowed: 66,659; Denied: Denied: 77,594; Odds of allowance:
Odds of allowance: 0.86; Observed odds ratio of allowance: Observed
odds ratio of allowance: [Empty]; Ratio of odds ratios: Ratio of odds
ratios: [Empty].
Source: GAO analysis of weighted CCS data.
[End of table]
Based on this analysis, we conclude that (1) our estimates of the
effect of having an attorney on the likelihood to be allowed may be
inflated, but (2) our estimates of the relative effects of attorney
representation by race on the likelihood to be allowed should not be
biased.
Oaxaca decomposition:
To further test whether differences in allowance rates between African-
American and white claimants are the result of differences in their
race or in other characteristics, we employed a statistical technique-
-the Oaxaca decomposition--that is commonly used in analyses of
discrimination.[Footnote 79] The goal of this technique is to separate
the difference in allowance rates between African-Americans and whites
into two components: one that results from differences in
characteristics between African-Americans and whites and the second
that results from differential treatment by race.
Several steps were taken to develop the results for our final Oaxaca
decomposition analysis:
* First, we estimated two versions of our baseline model--one with only
the African-American claimants in the sample and one with only the
white claimants in the sample. This step provided us with two sets of
estimated regression coefficients--one set of coefficients for African-
Americans and the other set for whites.
* Second, we applied the estimated coefficients from the model for
African-Americans to the values of each variable for African-Americans
to produce a probability of allowance for African-Americans. We did the
same with the estimated coefficients for whites and the values of each
variable for whites to produce a probability of allowance for whites.
These estimated probabilities of allowance are similar to the allowance
rates for African-Americans and whites based on observed (or actual)
data; but, because the probabilities are predicted, they deviate
slightly from the observed allowance rates.
* Third, we used the coefficients from the model of whites and the
actual values for each variable for African-Americans to produce a new
probability of allowance. This probability reflects what the
probability of allowance would have been for African-Americans had they
been treated the same as whites in the allowance decision.
For our final Oaxaca decomposition analysis, we compared the results of
the steps above. Specifically, we compared (1) the African-American
probability of allowance predicted using the African-American model,
with (2) the African-American probability of allowance predicted using
the white model, with (3) the white probability of allowance predicted
using the white model. To the extent that the African-American
probability of allowance predicted using the white model departs from
the white probability of allowance predicted using the white model, we
can conclude that the difference between African-Americans and whites
can be explained by differences in characteristics. To the extent that
the African-American probability predicted using the white model
departs from that predicted using the African-American model, we
conclude that (1) the two models reflect different treatment of
African-Americans and whites and (2) the difference between African-
Americans and whites cannot be fully explained by differences in
characteristics. We performed these analyses on (1) the entire sample
of claimants, (2) the sample of claimants with attorney representation,
and (3) the sample of claimants without attorney representation. Table
14 presents the results of these analyses for each sample.
Table 14: Summary Results of Oaxaca Decomposition:
Entire sample; Predicted allowance rate for: African-Americans (with
African-American coefficients): 49%; Predicted allowance rate for:
African-Americans (with white coefficients): 53%; Predicted allowance
rate for: Whites (with African-American coefficients): 59%; Predicted
allowance rate for: Whites (with white coefficients): 63%;
Percentage of explained disparities[A]: 71%; Percentage due to unequal
treatment and/or factors not controlled for in model: 29%.
Claimants with attorneys; Predicted allowance rate for: African-
Americans (with African-American coefficients): 60%; Predicted
allowance rate for: African-Americans (with white coefficients): 62%;
Predicted allowance rate for: Whites (with African-American
coefficients): 68%; Predicted allowance rate for: Whites (with white
coefficients): 69%; Percentage of explained disparities[A]:
78%; Percentage due to unequal treatment and/or factors not controlled
for in model: 22%.
Claimants without attorneys; Predicted allowance rate for: African-
Americans (with African-American coefficients): 34%; Predicted
allowance rate for: African-Americans (with white coefficients): 40%;
Predicted allowance rate for: Whites (with African-American
coefficients): 43%; Predicted allowance rate for: Whites (with white
coefficients): 49%; Percentage of explained disparities[A]:
60%; Percentage due to unequal treatment and/or factors not controlled
for in model: 40%.
Source: GAO analysis of weighted enhanced data.
[A] The percentage of explained disparities is computed by dividing the
difference between the predicted allowance rate for whites (with white
coefficients) and the predicted allowance rates for African-Americans
(with white coefficients), by the difference between the predicted
allowance rate for whites (with white coefficients) and the predicted
allowance rate for African-Americans (with African-American
coefficients). For example, for the entire sample, the computation is
(63%-53%/63%-49%)=71%.
[End of table]
The results of the Oaxaca decomposition show that most of the
difference between African-Americans and whites can be explained by
differences in their characteristics. Specifically, we found that using
the entire sample, 71 percent of the difference in predicted allowance
rates between whites and African-Americans is due to differences in the
characteristics of African-Americans and whites. The remaining 29
percent is due to either unequal treatment in the disability decision-
making process or to factors that are not controlled for in the model
or to some combination of the two.
The results of the two subsamples can be interpreted in the same way as
the results from the entire sample. Specifically, the results for the
sample of claimants with attorneys show that 78 percent of the
difference in predicted allowance rates between whites and African-
Americans is due to differences in characteristics between African-
Americans and whites. The remaining 22 percent is due to either unequal
treatment in the disability decision-making process or to factors that
are not controlled for in the model or to some combination of the two.
In addition, when we use the sample of claimants without attorney
representation, we find that less of the difference between African-
Americans and whites is explained by differences in characteristics (as
compared with the entire sample or the sample of claimants with
attorneys). Specifically, the results show that 60 percent of the
difference in predicted allowance rates between whites and African-
Americans is due to differences in characteristics. The remaining 40
percent is due to either unequal treatment or to factors that are not
controlled for in the model or to some combination of the two. The
results of this technique buttress the conclusions we draw from our
final model, that is, among claimants without attorney representation,
substantial differences between African-Americans and whites cannot be
explained by differences in other factors.
Section 5: Limitations of Analysis:
Due to inherent limitations with our data and methods, we cannot
definitively determine whether unexplained differences in allowance
rates by race are due to unequal treatment during the decision-making
process.
First, many of the variables we used in our analyses had some degree of
measurement error, and this can be a potentially serious problem when
continuous variables are redefined and collapsed into categorical
variables. For example, the severity of the claimant's impairments
ranges along a very broad continuum. However, the data available for
these analyses rank the severity of claimant's impairments and place
them in a limited number of categories. Within a particular category,
however, there may be subtle and important variations in severity that
are completely unmeasured. Second, some variables were measured
imprecisely. For example, the earnings variable was derived using the
average of employment income earned by the claimant during the 5 years
previous to the hearings decision. This earnings variable did not
include investment income or earnings from other family members. Hence,
it does not necessarily reflect the claimant's total household income,
data that were not available.
Third, several factors, for which data were not available, could not be
controlled for in our model. For example, we were unable to control for
the extent to which claimants may differ in their access to and quality
of healthcare. Differences in access to and quality of healthcare are
reflected in, and thus related to, the quality of medical evidence in
the claimant's file--an important component of the decision-making
process. Credibility is also a key factor in the ALJ disability
decision-making process. However, we did not include a proxy for
credibility in our model because we did not have an independent
assessment of the claimant's credibility.[Footnote 80]
Finally, the choice of whether or not to appeal has a theoretical
potential to affect the analysis. However, due to a lack of data at the
initial level, we were unable to estimate, or control for, the
claimant's likelihood of appealing to the ALJ level.
Improving the precision of some of the variables that were included in
our model and including additional variables to control for other
factors might have improved our ability to account for the variation in
ALJ decisions. Although these limitations could have resulted in biased
estimates of our coefficients, the enhanced data we used were the best
data available for examining potential racial disparities in ALJ
disability decision making.
[End of section]
Appendix II: SSA's Five-Step Sequential Evaluation Process for
Determining Disability:
SSA's regulations provide for disability evaluation under a procedure
known as the "sequential evaluation process." For adult claimants, this
process requires a sequential review of the claimant's current work
activity, the severity of his or her impairment(s), and if necessary,
the claimant's residual functional capacity, his or her past work, and
his or her age, education, and work experience.[Footnote 81]
Step 1. Is the claimant working? If the claimant is working and the
claimant's average monthly countable earnings are above the substantial
gainful activity (SGA) level,[Footnote 82] SSA will find the claimant
not disabled, regardless of the claimant's medical condition, age,
education, and work experience, and deny the claim. If the claimant's
average monthly countable earnings are at or less than the SGA level,
SSA will look at the claimant's medical condition (step 2).
Step 2. Is the claimant's condition "severe?" The claimant's impairment
must significantly limit his or her physical or mental ability to do
basic work activities, such as walking, sitting, seeing, and
remembering. If it does not, SSA will deny the claim, regardless of the
claimant's age, education, and work experience. If it does, SSA will
look further at the claimant's medical condition (step 3).
Step 3. Is the claimant's medical condition in the list of "disabling"
impairments? If the claimant has an impairment that meets the duration
requirement and is on SSA's listing of impairments,[Footnote 83] the
claimant is considered "disabled" without considering age, education,
and work experience. If the medical condition is not on the list, SSA
considers whether the condition is of equal severity to an impairment
on SSA's list. If so, the claim is approved. If not, SSA considers
additional factors (step 4).
Step 4. Can the claimant perform past relevant work? If the medical
condition is severe, but not at the same or equal severity as an
impairment on SSA's list, then SSA will review the claimant's residual
functional capacity, and the physical and mental demands of work
performed in the past. If the claimant can do work performed
previously, SSA will deny the claim. If not, SSA considers other
factors (step 5).
Step 5. Can the claimant perform other types of work? If the claimant
cannot perform past work, SSA will consider the claimant's residual
function capacity, age, education, and past work experience to
determine whether he or she can perform other work that is available in
the national economy. If the claimant cannot perform other work, SSA
will approve the claim. If the claimant can perform other work, SSA
will deny the claim.
[End of section]
Appendix III: Comments from the Social Security Administration:
This flowchart is printed on pages 70-72.
SOCIAL SECURITY:
The Commissioner:
October 14, 2003:
Mr. Robert E. Robertson Director, Education, Workforce, and Income
Security Issues U.S. General Accounting Office Room 5T57:
441 G Street, NW Washington, D.C. 20548:
Dear Mr. Robertson:
Thank you for the opportunity to review the draft report, "SSA
Disability Decision Making: Additional Steps Needed to Ensure Accuracy
and Fairness of Decisions at the Hearings Level." The draft report is
useful and timely and fits into our overall goals of fairness and
accuracy. On September 25, 2003, I testified before the House Ways and
Means Subcommittee on Social Security and presented my approach to
improve the disability determination process. The proposed process
would shorten decision times, pay benefits much earlier to people who
are obviously disabled and test new incentives for people with
disabilities who wish to remain in, or return to, the workforce. I have
enclosed a copy of my testimony as well as flow charts depicting the
approach I described.
I agree with the recommendations in the report but intend to go
further. As part of our overall plan to improve the disability
determination process, we intend to look at all factors that may
produce adverse impacts based on race, ethnicity, national origin or
gender. And, we plan to introduce an in-line quality review process
that will, among other things, help us to assess those impacts at all
stages in the process. Finally, a few months ago, we convened an Agency
workgroup tasked with developing recommendations on how we can collect
meaningful data on race and ethnicity so we will have the information
we need to analyze any adverse effects of our program policies and
rules.
If you have any questions, please have your staff contact Candace
Skurnik, Director, Audit Management Liaison Staff at (410) 965-4636.
Sincerely,
Jo Anne B. Barnhart:
Signed by Jo Anne B. Barnhart:
Enclosures:
SOCIAL SECURITY ADMINISTRATION BALTIMORE MD 21235-0001:
Social Security Testimony Before Congress:
Testimony:
Mr. Chairman,
I want to thank you and the entire Subcommittee for your continuing
support for the people and programs of the Social Security
Administration, and most especially for your interest in and commitment
to improving the disability process. I also want to thank you for
holding this hearing which provides the opportunity for me to describe
my approach for improving the Social Security and Supplemental Security
Income disability process. Our disability programs are critically
important in the lives of almost 13 million of Americans. Claimants and
their families expect and deserve fair, accurate, consistent, and
timely decisions.
EDIB is a major agency initiative that will move all components
involved in disability claims adjudication and review to an electronic
business process through the use of an electronic disability folder.
Implementation of an electronic disability folder is essential for
process improvements. Therefore, structurally, my long-term strategy
for achieving process improvements is predicated on successful
implementation of our electronic disability system.
In designing my approach to improve the overall disability
determination process, I was guided by three questions the President
posed during our first meeting to discuss the disability programs.
* Why does it take so long to make a disability decision?
* Why can't people who are obviously disabled get a decision
immediately?
* Why would anyone want to go back to work after going through such a
long process to receive benefits?
I realized that designing an approach to fully address the central and
important issues raised by the President required a focus on two over-
arching operational goals: (1) to make the right decision as early in
the process as
possible; and (2) to foster return to work at all stages of the
process. I also decided to focus on improvements that could be
effectuated by regulation and to ensure that no SSA employee would be
adversely affected by my approach. My reference to SSA employees
includes state Disability Determination Service (DDS) employees and
Administrative Law Judges (ALJs).
As I developed my approach for improvement, I met with and talked to
many people --SSA employees and other interested organizations ,
individually and in small and large groups --to listen to their
concerns about the current process at both the initial and appeals
levels and their recommendations for improvement. I became convinced
that improvements must be looked at from a system-wide perspective and,
to be successful, perspectives from all parts of the system must be
considered. I believe an open and collaborative process is critically
important to the development of disability process improvements.
To that end, members of my staff and I visited our regional offices,
field offices, hearing offices, and State Disability Determination
Services, and private disability insurers to identify and discuss
possible improvements to the current process.
Finally, a number of organizations provided written recommendations for
changing the disability process. Most recently, the Social Security
Advisory Board issued a report prepared by outside experts making
recommendations for process change. My approach for changing the
disability process was developed after a careful review of these
discussions and written recommendations. As we move ahead, I look
forward to working within the Administration and with Congress, as well
as interested organizations and advocacy groups. I would now like to
highlight some of the major and recurring recommendations made by these
various parties.
The need for additional resources to eliminate the backlog and reduce
the lengthy processing time was a common theme. This important issue is
being addressed through my Service Delivery Plan, starting with the
President's FY 2004 budget submission which is currently before
Congress. Another important and often heard concern was the necessity
of improving the quality of the administrative record.
DDSs expressed concerns about receiving incomplete applications from
the field office; ALJs expressed concerns about the quality of the
adjudicated record they receive and emphasized the extensive pre-
hearing work required to thoroughly and adequately present the case for
their consideration.
In addition, the number of remands by the Appeals Council and the
Federal Courts make clear the need for fully documenting the
administrative hearing record.
Applying policy consistently in terms of: 1) the DDS decision and ALJ
decision; 2) variations among state DDSs; and 3) variations among
individual ALJs --was of great concern. Concerns related to the
effectiveness of the existing regional quality control reviews and ALJ
peer review were also expressed. Staff from the Judicial Conference
expressed strong concern that the process assure quality prior to the
appeal of cases to the Federal Courts.
ALJs and claimant advocacy and claimant representative organizations
strongly recommended retaining the de novo hearing before an ALJ.
Department of Justice litigators and the Judicial Conference stressed
the importance of timely case retrieval, transcription, and
transmission. Early screening and analysis of cases to make expedited
decisions for clear cases of disability was emphasized time and again
as was the need to remove barriers to returning to work.
My approach for disability process improvement is designed to address
these concerns. It incorporates some of the significant features of the
current disability process. For example, initial claims for disability
will continue to be handled by SSA's field offices. The State
Disability Determination Services will continue to adjudicate claims
for benefits, and Administrative Law Judges will continue to conduct
hearings and issue decisions. My approach envisions some significant
differences.
I intend to propose a quick decision step at the very earliest stages
of the claims process for people who are obviously disabled. Cases will
be sorted based on disabling conditions for early identification and
expedited action. Examples of such claimants would be those with ALS,
aggressive cancers, and end-stage renal disease. Once a disability
claim has been
completed at an SSA field office, these Quick Decision claims would be
adjudicated in Regional Expert Review Units across the country, without
going to a State Disability Determination Service.
This approach would have the two-fold benefit of allowing the claimant
to receive a decision as soon as possible, and allowing the State DDSs
to devote resources to more complex claims.
Centralized medical expertise within the Regional Expert Review Units
would be available to disability decision makers at all levels,
including the DDSs and the Office of Hearings and Appeals (OHA). These
units would be organized around clinical specialties such as
musculoskeletal, neurological, cardiac, and psychiatric. Most of these
units would be established in SSA's regional offices.
The initial claims not adjudicated through the Quick Decision process
would be decided by the DDSs. However, I would also propose some
changes in the initial claims process that would require changes in the
way DDSs are operating. An in-line quality review process managed by
the DDSs and a centralized quality control unit would replace the
current SSA quality control system. I believe a shift to inline quality
review would provide greater opportunities for identifying problem
areas and implementing corrective actions and related training.
The Disability Prototype would be terminated and the DDS
Reconsideration step would be eliminated. Medical expertise would be
provided to the DDSs by the Regional Expert Review units that I
described earlier.
State DDS examiners would be required to fully document and explain the
basis for their determination. More complete documentation should
result in more accurate initial decisions. The increased time required
to accomplish this would be supported by redirecting DDS resources
freed up by the Quick Decision cases being handled by the expert units,
the elimination of the Reconsideration step, and the shift in medical
expertise responsibilities to the regional units.
A Reviewing Official (RO) position would be created to evaluate claims
at the next stage of the process. If a claimant files a request for
review of the DDS
determination, the claim would be reviewed by an SSA Reviewing
Official. The RO, who would be an attorney, would be authorized to
issue an allowance decision or to concur in the DDS denial of the
claim. If the claim is not allowed by the RO, the RO will prepare either
a Recommended Disallowance or a Pre-Hearing Report. A Recommended
Disallowance would be prepared if the RO believes that the evidence in
the record shows that the claimant is ineligible for benefits. It would
set forth in detail the reasons the claim should be denied.
A Pre-Hearing Report would be prepared if the RO believes that the
evidence in the record is insufficient to show that the claimant is
eligible for benefits but also fails to show that the claimant is
ineligible for benefits. The report would outline the evidence needed
to fully support the claim. Disparity in decisions at the DDS level has
been a long-standing issue and the SSA Reviewing Official and creation
of Regional Expert Medical Units would promote consistency of decisions
at an earlier stage in the process.
If requested by a claimant whose claim has been denied by an RO, an ALJ
would conduct a de novo administrative hearing. The record would be
closed following the ALJ hearing. If, following the conclusion of the
hearing, the ALJ determines that a claim accompanied by a Recommended
Disallowance should be allowed, the ALJ would describe in detail in the
written opinion the basis for rejecting the RO's Recommended
Disallowance.
If, following the conclusion of the hearing, the ALJ determines that a
claim accompanied by a Pre-Hearing Report should be allowed, the ALJ
would describe the evidence gathered during the hearing that responds
to the description of the evidence needed to successfully support the
claim contained in the Pre-hearing Report.
Because of the consistent finding that the Appeals Council review adds
processing time and generally supports the ALJ decision, the Appeals
Council stage of the current process would be eliminated.
Quality control for disability claims would be centralized with end-of-
line reviews and ALJ oversight. If an ALJ decision is not reviewed by
the centralized quality control staff, the decision of the ALJ will
become a final agency action. If the centralized quality control review
disagrees with an allowance or disallowance
determination made by an ALJ, the claim would be referred to an
Oversight Panel for determination of the claim. The Oversight Panel
would consist of two Administrative Law Judges and one Administrative
Appeals Judge.
If the Oversight Panel affirms the ALJ's decision, it becomes the final
agency action. If the Panel reverses the ALJ's decision, the oversight
Panel decision becomes the final agency action. As is currently the
case, claimants would be able to appeal any final agency action to a
Federal Court.
At the same time these changes were being implemented to improve the
process, we plan to conduct several demonstration projects aimed at
helping people with disabilities return to work. These projects would
support the President's New Freedom Initiative and provide work
incentives and opportunities earlier in the process.
I believe these changes and demonstrations will address the major
concerns I highlighted earlier. I also believe they offer a number of
important improvements:
* People who are obviously disabled will receive quick decisions.
* Adjudicative accountability will be reinforced at every step in the
process.
* Processing time will be reduced by at least 25%. . Decisional
consistency and accuracy will be increased.
* Barriers for those who can and want to work would be removed.
Describing my approach for improving the process is the first step of
what I believe must be --and will work to make --a collaborative
process. I will work within the Administration, with Congress, the
State Disability Determination Services and interested organizations
and advocacy groups before putting pen to paper to write regulations. I
said earlier, and I say again that to be successful, perspectives from
all parts of the system must be considered.
Later today, I will conduct a briefing for Congressional staff of the
Ways and Means and Senate Finance Committees. I will also brief SSA and
DDS management. In addition, next week I will provide a
video tape of the management briefing describing my approach for
improvement to all SSA regional, field, and hearing offices, State
Disability Determination Services, and headquarters and regional office
employees involved in the disability program.
Tomorrow, I will be conducting briefings for representatives of SSA
employee unions and interested organizations and advocacy groups, and I
will schedule meetings to provide an opportunity for those
representatives to express their views and provide assistance in
working through details, as the final package of process improvements
is fully developed.
I believe that if we work together, we will create a disability system
that responds to the challenge inherent in the President's questions.
We will look beyond the status quo to the possibility of what can be.
We will achieve our ultimate goal of providing accurate, timely service
for the American people.
Note: A flowchart describing the process is available in pdf format.
A New Approach to SSA Disability Determination:
[See PDF for image]
[End of figure]
Reviewing Official-ALJ Work Flow:
[See PDF for image]
[End of figure]
Work Opportunity Demonstrations:
[See PDF for image]
[End of figure]
[End of section]
Appendix IV: GAO Contacts and Acknowledgments:
GAO Contacts:
Robert E. Robertson, (202) 512-7215 Carol Dawn Petersen, (202) 512-
7215:
GAO Acknowledgments:
In addition to those named above, the following GAO staff made
significant contributions to this report: Mark de la Rosa, Erin
Godtland, Michele Grgich, Stephen S. Langley III, and Ann T. Walker,
Education, Workforce, and Income Security Issues; Doug Sloane, Applied
Research and Methods. Also contributing to the report were: Gene
Kuehneman and Jill Yost, Education, Workforce, and Income Security
Issues; Jessica Botsford, Richard Burkard, David Plocher, and Dayna
Shah, General Counsel; Wendy Turenne and Shana Wallace, Applied
Research and Methods; Scott Farrow, Chief Economist; Robert Parker,
Chief Statistician; Ron Stroman, Office of Opportunity and Inclusion.
Other Acknowledgments:
We contracted with the following individuals for technical assistance:
* Judith Hellerstein, Associate Professor of Economics, Department of
Economics, University of Maryland.
* Joseph Kadane, University Professor and Professor of Statistics and
Social Sciences, Department of Statistics, Carnegie-Mellon University.
* Brent Kreider, Associate Professor of Economics, Department of
Economics, Iowa State University.
* Kajal Lahiri, Professor of Economics, and Professor of Health Policy,
Management and Behavior, Department of Economics, University at Albany,
State University of New York.
FOOTNOTES
[1] In 1992, GAO reported that DI allowance rates between 1961 and 1985
and SSI allowance rates between 1971 and 1989 were consistently lower
for African-Americans than whites. See U.S. General Accounting Office,
Social Security: Racial Difference in Disability Decisions Warrants
Further Investigation, GAO/HRD-92-56 (Washington, D.C.: Apr. 21, 1992).
[2] See U.S. General Accounting Office, SSA Disability Decision Making:
Additional Measures Would Enhance Agency's Ability to Determine Whether
Racial Bias Exists, GAO-02-831 (Washington, D.C.: Sept. 9, 2002).
[3] Changes in coding schemes over time limit our ability to analyze
Hispanic and other ethnic groups separately. Prior to 1980, race data
were collected for three categories: white, black, or other. In 1980,
SSA adopted new codes: "White," "Black," "Hispanic," "Asian or Pacific
Islander," and "American Indian or Alaskan Native." Because much of the
race data were collected before 1980, and were not recoded into the new
categories, "Hispanic," "Asian or Pacific Islander," or "American
Indian or Alaskan Native," we were unable to conduct our analyses using
these new categories.
[4] GAO-02-831.
[5] In conducting these tests, we compared the enhanced data with data
from SSA's administrative files. See appendix I.
[6] To construct the models, we reviewed pertinent literature and
consulted with SSA officials and outside experts.
[7] After estimating our initial model of factors affecting ALJ
decisions using logistic regression analysis, we identified race,
attorney representative, and several other factors that are not part of
the criteria used in the decision-making process but that had a
statistically significant influence on allowance decisions. We
constructed additional models that included combinations of these
variables to determine the influence of these variables on allowance
decisions. One of these interaction variables--controlling for African-
American claimants that had attorney representation--had a
statistically significant influence on allowance decisions and was,
therefore, included in our final model. To further analyze the
relationship between race and attorney representation on allowance
decisions, we employed a statistical technique--the Oaxaca
decomposition--that is commonly used in analyses of discrimination. See
appendix I for a description of this analysis.
[8] SSI also provides income assistance to the aged who have income and
assets below a certain level.
[9] The Social Security commissioner has the authority to set the
substantial and gainful activities level for individuals who have
disabilities other than blindness. In December 2000, SSA finalized a
rule calling for the annual indexing of the nonblind level to the
average wage index of all employees in the United States. The current
nonblind level is set at $800 per month. The level for individuals who
are blind is set by statute and is also indexed to the average wage
index. Currently, the level for blind individuals is $1,330 of
countable earnings.
[10] DI beneficiaries with low income and assets can also receive SSI
benefits. Of the 5.5 million DI beneficiaries, about .8 million also
received SSI in 2002. Thus, there was a total of 8.5 million working-
age beneficiaries in 2002, with 9 percent receiving both DI and SSI.
[11] SSA permits DI, but not SSI, claimants to file for benefits on-
line.
[12] While most claimants may request a reconsideration, at the time of
our study, SSA was testing an initiative that eliminates the
reconsideration step from the DDS decision-making process. In her
September 2003 testimony before Congress, SSA's Commissioner proposed
eliminating reconsideration as part of a large set of revisions to the
disability decision-making process.
[13] According to SSA's Hearings, Appeals and Litigation Law Manual
(HALLEX), Sec. I-2-5-30, the ALJ decides whether the testimony of a
medical or vocational expert is needed at a hearing.
[14] If the claimant is not satisfied with the Appeals Council
decision, the claimant may appeal to a federal district court. The
claimant can continue legal appeals to the U.S. Circuit Court of
Appeals and ultimately to the Supreme Court of the United States.
[15] Obtaining this documentation is complicated by the fact that files
are stored in different locations, depending on whether the case
involved an SSI or DI claim, and whether the ALJ decision was an
allowance or denial. For fiscal years 1999 and 2000, SSA obtained files
and tapes for 48 percent of the 33,484 records sampled. The case file
contains the application for benefits, disability information provided
by the claimant, DDS determinations, claimant's appointment of an
attorney/representative (if applicable), appeal request documentation,
medical evidence furnished at each level of the appeal, and the ALJ
decision. For ALJ allowance decisions, the file will also contain
documentation of benefit computation and payment.
[16] The number of medical consultants used depends on the number and
type of impairments alleged by the claimant.
[17] In the peer review process, ALJs use the standard of substantial
evidence that means that the ALJ should not overturn a decision if the
relevant evidence is what a reasonable mind might accept as adequate to
support a conclusion. In the original ALJ hearings process, ALJs use a
higher standard of preponderance of evidence that means that more than
half of the evidence must support a particular conclusion.
[18] According to SSA's HALLEX, Sec. I-3-3-2, abuse of discretion in a
judgment or conclusion involves an ALJ acting in a manner that is
imprudent, incautious, unwise, against precedent, and clearly against
logic.
[19] According to SSA's HALLEX, Sec. I-3-3-3, error of law covers six
broad issues: (1) misinterpretation of law or regulations; (2)
misapplication of the law, regulations, or rulings to the facts; (3)
failure to consider pertinent provisions of law, regulations, or
rulings; (4) failure to make a finding of fact, or to give reasons for
making a finding of fact, on an issue properly before the ALJ; (5) a
procedural error that affects due process (e.g., improper notice of
hearing, failure to notify the claimant of the right to question
witnesses; and (6) failure to rule on an objection raised at the
hearing.
[20] GAO/HRD-92-56.
[21] GAO-02-831.
[22] The complete results of our model are presented in appendix I.
[23] The category for nonattorney may include representatives from
legal aid organizations, which could include attorneys as well as
nonattorneys.
[24] About 25 percent of the claimants from the other racial/ethnic
group had translators at their hearings, and our analyses also show
that claimants who had translators at the hearing were less likely to
be awarded benefits than claimants who did not have translators.
[25] This discussion pertains only to claimants with no representation
as compared with claimants with attorney representation, and does not
pertain to claimants with nonattorney representatives such as legal
aides, relatives, and friends. Additional analyses showed that among
claimants with nonattorney representatives, African-Americans were
less likely to be awarded benefits than whites. However, this result
may be due to the low number of observations for claimants with
nonattorneys.
[26] The odds on claims being allowed are related to, but not quite the
same as, the probability of claims being allowed. Suppose that among
whites, 200 claims were allowed among a total of 300 filed. While the
probability of claims being allowed is estimated by dividing the number
of claims allowed by the number of all claims (i.e., 200/300= 0.66),
odds are estimated by dividing the number of claims allowed by the
number of claims not allowed (i.e., 200/100 = 2). If we found that
among African-Americans, 50 out of 100 claims were allowed, we would
calculate the odds of allowance to be 50/50 = 1.00, and the odds ratio
of African-Americans to whites would be 1.00/2.00 = 0.5. This implies
that the odds for African-Americans were only one-half those of whites.
While probabilities (P) and odds (O) are mathematically related (O = P/
[1-P]), odds have certain advantages over probabilities for these
statistical purposes, which is why we employ them.
[27] See appendix I for an explanation as to why this interaction term
was created and an explanation of how the specific result was
calculated.
[28] The effect of attorney representation for other race/ethnicity
claimants is not significantly different than for white claimants.
[29] See appendix I for a description and the results of our Oaxaca
decomposition analysis.
[30] Attorneys' efforts to obtain medical evidence might result in
better medical evidence than that obtained by SSA earlier in the
decision-making process because, for example: (1) attorneys often use
request forms that are tailored to the disability criteria and the
claimant's impairments to solicit specific information on the
claimant's medical history from medical providers and (2) attorneys pay
more for medical records than SSA.
[31] We were told by attorneys affiliated with NOSSCR that attorneys
typically screen their claimants to assess the strength of the
claimant's case. If the attorney believes the evidence does not support
an argument for the claimant's disability, as defined in SSA's
guidelines, the attorney is not likely to take the case. This may mean
that claimants with attorneys have stronger cases and are more likely
to be approved for benefits regardless of the additional assistance
provided by the attorney. Relatedly, ALJs--who may be aware that
attorneys choose stronger cases--may be more likely to view a claimant
with an attorney as having an impairment with such severity so as to
qualify the claimant for benefits.
[32] Additional analyses showed that among claimants with nonattorney
representatives, African-Americans were less likely to be awarded
benefits than whites. However, this result may be due to the low number
of observations for claimants with nonattorneys.
[33] The current model compares claimants in the Boston Region with
claimants in the New York Region (the reference category). However,
when we use any other region as the reference category, claimants from
the Boston Region are always significantly more likely to be awarded
benefits than claimants from the reference region.
[34] These variables include number of impairments, number of severe
impairments, physical and mental capacity, type of impairment,
occupational years, age, occupational categories, occupational skill
level, education, literacy, and earnings.
[35] The quality assurance review of ALJ decisions includes analyses of
the accuracy of ALJ decisions, in which the reviewing ALJs assess
whether the original ALJ's ultimate decision to allow or deny is
supported by substantial evidence--which is referred to in the quality
assurance review as support rates. This review also includes analyses
of the fairness of ALJ hearings in which the reviewing ALJs evaluate a
multitude of issues, including abuse of discretion and error of law.
[36] SSA's analysis of ALJ decisions is limited to descriptive
statistics; SSA does not use multivariate techniques--i.e., control for
other factors simultaneously--in its analysis of ALJ decisions.
[37] In addition to not analyzing AJJ decisions by race, SSA does not
analyze ALJ decisions by sex or income.
[38] GAO-02-831.
[39] In SSA's enhanced data that we used for our analysis, only 10
percent of the cases represented unsupported ALJ decisions, and only 13
percent of these were for African-Americans.
[40] As described in appendix I, we compared the characteristics of
claimants in SSA's enhanced data with the characteristics of claimants
that were originally sampled for but, for various reasons, were not
included in the enhanced data, and did not find large differences
between the two claimant groups. However, our results might be due to
the particular cases sampled and/or not included for various reasons
during the time period.
[41] SSA currently envisions selecting several hundred cases that were
originally excluded from the sample and reviewing them after the agency
has reached a final decision.
[42] A case is considered final by the agency when a claimant has
exhausted his or her right to appeal, and either SSA or the federal
courts have rendered a final decision. For example, a decision is
considered final when the Appeals Council dismisses cases or upholds,
modifies, or reverses the ALJ's action. If the Appeals Council remands
the case back to the ALJ level, the case is not considered final until
the ALJ decides on the case. Appeals to the federal court system would
further delay the final decision.
[43] For example, claimants have 60 days to appeal the ALJ decision to
the Appeals Council, after which the average number of days for
processing and deciding a case at the Appeals Council level is about
225 days. It takes, on average, an additional 250 days to reach a final
decision for cases that are remanded by the Appeals Council back to the
ALJ.
[44] The quality of data could be affected when policies and guidance
change over time. For example, reviewing ALJs may be using policies and
guidance that were not applicable when the original ALJ decided on a
case. For corrective action to be effective, it should be taken in a
timely manner. For example, if a belated quality assurance review finds
that a certain region does not make accurate and fair decisions for a
substantial number of its cases, corrective action might occur long
after the problem occurred.
[45] GAO-02-831.
[46] Under current procedures, SSA is unlikely to subsequently obtain
information on race and ethnicity for individuals assigned SSNs at
birth unless those individuals apply for a new or replacement Social
Security card, due to a change in name or a lost card.
[47] Since SSA's EAB program began in 1990, and our study used a sample
of adult disability claimants from 1997-2000, most claimants in our
sample preceded the EAB program. As a result, we had race data for most
of the claimants in our sample.
[48] See U.S. General Accounting Office, SSA Disability Decision
Making: Additional Measures Would Enhance Agency's Ability to Determine
Whether Racial Bias Exists, GAO-02-831 (Washington, D.C.: Sept. 9,
2002).
[49] We are grateful to four outside experts who assisted us with this
study. They are Judith Hellerstein, Associate Professor of Economics at
the University of Maryland; Joseph Kadane, Professor of Statistics and
Social Sciences at Carnegie-Mellon University; Brent Kreider, Associate
Professor of Economics at Iowa State University; and Kajal Lahiri,
Professor of Economics at the University at Albany, State University of
New York. We take full responsibility for any errors.
[50] See below for a discussion of the representativeness of the
enhanced data.
[51] In conducting these tests, we found that only one data field
(occupation from the 831 administrative file) did not pass all 3 of
these tests and was, therefore, excluded from the subsequent
nonresponder analyses.
[52] The data on medical severity in the enhanced data are developed
during DDHQ's disability examiner/medical consultant review--a process
that is independent from SSA's disability decision-making process. The
medical severity variables are proxies for information that the judge
would have seen during the hearing, but are not developed by the judge.
Thus, they are appropriate for use in a regression estimating the
judge's allowance decision.
[53] U.S. General Accounting Office, Social Security: Racial Difference
in Disability Decisions Warrants Further Investigation, GAO/HRD-92-56
(Washington, D.C.: Apr. 21, 1992).
[54] GAO-02-831.
[55] Specifically, 140 decisions from each region were selected per
month. Of the 140 decisions, 70 were denials and 70 were allowances.
[56] This usually occurs for cases that were denied, but can also occur
for allowances such as when the claimant disputes the date of onset.
[57] The nonresponders also include the sample of files that were
obtained, but did not undergo all three reviews.
[58] By best possible estimates, we mean unbiased estimates, combined
with small standard errors.
[59] Other factors that are available in the enhanced data, but are not
available in the administrative data, include variables on the
claimant's occupational skill level and whether the claimant is
literate.
[60] The enhanced data contain variables that are equivalent (or very
similar) to the variables in SSA's administrative files, such as
occupation, but are likely to be more complete and accurate than
administrative data, per our data reliability assessments. We used the
enhanced data for this analysis so that we would capture only the added
value of the variables that are available in the enhanced data in our
comparison. If we had used the 831 and CCS data in Model A and the
enhanced data in Model B, then Model B might also capture the effect of
the higher quality of the enhanced data.
[61] This model was the preliminary model of the ALJ decision-making
process, from which our final model was derived.
[62] Specifically, the variables that we compared include demographic
factors such as age, sex, and race; vocational factors such as years
employed and years of education; medical variables such as the body
system involved in the claimant's impairment (at the DDS level and at
the ALJ level) and whether they had a consultative exam; and
administrative variables including claim type, hearing participants
(attorney representation, nonattorney representation, vocational
expert present, medical expert present), ALJ allowance decision, the
final allowance decision (including Appeals Council decision if
claimant was denied at the ALJ level and appealed to the Appeals
Council), and regulation basis codes (indicating the step of sequential
disability decision-making process at which claimant was allowed or
denied).
[63] We did not use the enhanced data to conduct this analysis because
they were not available for nonresponders. Had we used the enhanced
data for nonresponders and SSA's administrative data for nonresponders,
it would have been difficult to separate the differences between
responders and nonresponders in characteristics with the differences
between the enhanced data and SSA's administrative data in quality.
[64] We conducted the nonresponder analysis with and without
probability weights. The results of both sets of analysis were
consistent.
[65] A sampling error is a variation that occurs by chance when a
model/analysis relies on a sample that was surveyed rather than the
entire population. The size of the sampling error reflects the
precision of the estimate--the smaller the sampling error, the more
precise the estimate.
[66] Four outside experts reviewed our methods and preliminary results
and provided us with helpful feedback. They are Judith Hellerstein,
Associate Professor of Economics at the University of Maryland; Joseph
Kadane, Professor of Statistics and Social Sciences at Carnegie-Mellon
University; Brent Kreider, Associate Professor of Economics at Iowa
State University; and Kajal Lahiri, Professor of Economics at the
University at Albany, State University of New York.
[67] See appendix II for a description of the 5-step decision-making
process.
[68] In 1996, the Contract With America Advancement Act provided that
individuals could not be found disabled for purposes of DI or SSI if
drug addiction or alcoholism was a "contributing factor material to the
determination of disability." Drug addicts and alcoholics who were
disabled as a result of other causes would still be eligible.
[69] Claim type includes SSI claims, DI claims, and concurrent claims
for both SSI and DI.
[70] The year of the decision might capture changes in decision making
that have occurred over time due to changes in national policy or in
the economic health of the country. In addition, region might capture
regional differences in culture, social norms, court decisions or
geographic variation in SSA's practices. In "A Structural Model of
Social Security's Disability Determination Process," in The Review of
Economics and Statistics, May 2001, 83(2): 348-61, Jianting Hu, Kajal
Lahiri, Denton R. Vaughan, and Bernard Wixon found evidence that
allowance rates at the initial level differed significantly by region
at Step 2 and 4 of the disability decision-making process. In
"Disability Insurance: Applications, Awards, and Lifetime Opportunity
Costs," Journal of Labor Economics, Oct. 1999, 784-827, Brent Kreider
found a significant relationship between region allowance rates and the
likelihood of allowance for an individual claimant.
[71] GAO/HEHS-94-94 found significant differences in allowance
decisions at the initial level by sex. GAO/HRD-92-56 found significant
differences in allowance decisions at the hearings level by race.
Additionally, in "A Structural Model of Social Security's Disability
Determination Process," in The Review of Economics and Statistics, May
2001, 83(2): 348-61, Jianting Hu, Kajal Lahiri, Denton R. Vaughan, and
Bernard Wixon found that sex and race played a statistically
significant role in Step 2 of the decision-making process. In SSA's
initial comments on our analysis, they suggested that we incorporate a
variable that controls for the claimant's earnings into our model.
[72] Although earnings are used in Step 1 of the decision-making
process to determine whether the claimant's earnings exceed the limit
required for eligibility (and to determine whether the claim type is
SSI or DI), earnings are not considered in Steps 2-5, which pertain to
the ALJ disability decision-making process.
[73] Although we had no compelling theoretical or empirical reason for
testing this particular interaction, we believed it would be useful to
determine whether any racial differences that we found in our initial
model were larger at the beginning of the 4-year period for which we
had data than they were at the end of the 4-year period.
[74] Odds (O) are mathematically related to but not the same as
probabilities (P), that is O = P/[1-P]. For further explanation of how
to interpret odds and odds ratios, see text after table 6.
[75] Comparison categories can be identified because they have an odds
ratio of exactly 1.00 and in our report, with the exception of region,
are presented first among the categories of a variable.
[76] Due to the presence of interaction terms between race and attorney
representation in the final model, the odds ratio for the race variable
in the final model represents the odds ratio for claimants of a
particular race who do not have attorney representation.
[77] The odds ratio for the interaction variable for claimants from
other racial/ethnic groups with attorney representation is not
significant. This indicates that the effect of attorney representation
on the odds of allowance for claimants from other racial/ethnic
backgrounds is not significantly different from the effect of attorney
representation on the odds of allowance for white claimants.
[78] This difference probably results from SSA's systematic exclusion
of cases that are appealed to the Appeals Council from the enhanced
data. According to attorneys that represent SSA claimants, attorneys
usually advise claimants who are denied at the ALJ level to appeal to
the Appeals Council. Therefore, claimants who are denied at the ALJ
level and appeal to the Appeals Council are likely to have higher rates
of attorney representation than claimants who are denied at the ALJ
level and do not appeal.
[79] For details on this technique see "Male-Female Wage Differentials
in Urban Labor Markets," by Ronald Oaxaca, in International Economic
Review, Volume 14, Issue 3 (Oct. 1973), 693-709.
[80] The original ALJ's assessment of the claimant's credibility cannot
be used as an independent variable because it is too highly correlated
with the final allowance decision and could distort other results in
our model.
[81] For children applying for SSI, the process requires sequential
review of the child's current work activity (if any), the severity of
his or her impairment(s), and an assessment of whether his or her
impairment(s) results in marked and severe functional limitations.
[82] The 2003 SGA level for claimants who are not blind is $800. The
2003 SGA level for persons who are blind is $1,330.
[83] SSA's Listing of Impairments describes, for each major body
system, impairments that are considered severe enough to prevent an
adult person from doing any gainful activity (or in the case of
children under age 18 applying for SSI, cause marked and severe
functional limitations). Most of the listed impairments are permanent
or expected to result in death, or a specific statement of duration is
made. For all others, the evidence must show that the impairment has
lasted or is expected to last for a continuous period of at least 12
months. The criteria in the Listing of Impairments are applicable to
evaluation of claims for disability benefits under both the Social
Security DI and SSI programs.
GAO's Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548:
