Aviation Safety
Certification and Approval Processes Are Generally Viewed as Working Well, but Better Evaluative Information Needed to Improve Efficiency Gao ID: GAO-11-14 October 7, 2010Among its responsibilities for aviation safety, the Federal Aviation Administration (FAA) issues thousands of certificates and approvals annually. These certificates and approvals, which FAA bases on its interpretation of federal standards, indicate that such things as new aircraft, the design and production of aircraft parts and equipment, and new air operators are safe for use in the national airspace system. Past studies and industry spokespersons assert that FAA's interpretations produce variation in its decisions and inefficiencies that adversely affect the industry. GAO was asked to examine the (1) extent of variation in FAA's interpretation of standards for certification and approval decisions and (2) views of key stakeholders and experts on how well these processes work. To perform the study, GAO reviewed industry studies and reports and FAA documents and processes; convened a panel of aviation experts; and interviewed officials from various industry sectors, senior FAA officials, and unions representing FAA staff.
Studies, stakeholders, and experts indicated that variation in FAA's interpretation of standards for certification and approval decisions is a long-standing issue, but GAO found no evidence that quantified the extent of the problem in the industry as a whole. Ten of the 13 industry group and company officials GAO interviewed said that they or members of their organization had experienced variation in FAA certification and approval decisions on similar submissions. In addition, experts on GAO's panel, who discussed and then ranked problems with FAA's certification and approval processes, ranked inconsistent interpretation of regulations, which can lead to variation in decisions, as the first and second most significant problem, respectively, with these processes for FAA's Flight Standards Service (which issues certificates and approvals for individuals and entities to operate in the national airspace system) and Aircraft Certification Service (which issues approvals to the designers and manufacturers of aircraft and aircraft parts and equipment). According to industry stakeholders, variation in FAA's interpretation of standards for certification and approval decisions is a result of factors related to performance-based regulations, which allow for multiple avenues of compliance, and the use of professional judgment by FAA staff and can result in delays and higher costs. Industry stakeholders and experts generally agreed that FAA's certification and approval processes contribute to aviation safety and work well most of the time, but negative experiences have led to costly delays for the industry. Industry stakeholders have also raised concerns about the effects of process inefficiencies on the implementation of the Next Generation Air Transportation System (NextGen)--the transformation of the U.S. national airspace system from a ground-based system of air traffic control to a satellite-based system of air traffic management. They said that the processes take too long and impose costs that discourage aircraft operators from investing in NextGen equipment. FAA has taken actions to improve the certification and approval processes, including hiring additional inspectors and engineers and increasing the use of designees and delegated organizations--private persons and entities authorized to carry out many certification activities. Additionally, FAA is working to ensure that its processes are being followed and improved through a quality management system, which provides a mechanism for stakeholders to appeal FAA decisions. However, FAA does not know whether its actions under the quality management system are achieving the intended goals of reducing inconsistencies and increasing consistency and fairness in the agency's application of regulations and policies because FAA does not have outcome-based performance measures and a continuous evaluative process that would allow it to determine progress toward these goals. Without ongoing information on results, FAA managers do not know if their actions are having the intended effects. GAO recommends that FAA develop a continuous evaluative process with measurable performance goals to determine the effectiveness of the agency's actions to improve its certification and approval processes. DOT did not comment on the recommendations but provided technical comments, which were included as appropriate.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Gerald L. Dillingham Team: Government Accountability Office: Physical Infrastructure Phone: (202) 512-4803GAO-11-14, Aviation Safety: Certification and Approval Processes Are Generally Viewed as Working Well, but Better Evaluative Information Needed to Improve Efficiency
This is the accessible text file for GAO report number GAO-11-14
entitled 'Aviation Safety: Certification and Approval Processes Are
Generally Viewed as Working Well, but Better Evaluative Information
Needed to Improve Efficiency' which was released on October 29, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
October 2010:
Aviation Safety:
Certification and Approval Processes Are Generally Viewed as Working
Well, but Better Evaluative Information Needed to Improve Efficiency:
GAO-11-14:
GAO Highlights:
Highlights of GAO-11-14, a report to congressional requesters.
Why GAO Did This Study:
Among its responsibilities for aviation safety, the Federal Aviation
Administration (FAA) issues thousands of certificates and approvals
annually. These certificates and approvals, which FAA bases on its
interpretation of federal standards, indicate that such things as new
aircraft, the design and production of aircraft parts and equipment,
and new air operators are safe for use in the national airspace
system. Past studies and industry spokespersons assert that FAA‘s
interpretations produce variation in its decisions and inefficiencies
that adversely affect the industry.
GAO was asked to examine the (1) extent of variation in FAA‘s
interpretation of standards for certification and approval decisions
and (2) views of key stakeholders and experts on how well these
processes work. To perform the study, GAO reviewed industry studies
and reports and FAA documents and processes; convened a panel of
aviation experts; and interviewed officials from various industry
sectors, senior FAA officials, and unions representing FAA staff.
What GAO Found:
Studies, stakeholders, and experts indicated that variation in FAA‘s
interpretation of standards for certification and approval decisions
is a long-standing issue, but GAO found no evidence that quantified
the extent of the problem in the industry as a whole. Ten of the 13
industry group and company officials GAO interviewed said that they or
members of their organization had experienced variation in FAA
certification and approval decisions on similar submissions. In
addition, experts on GAO‘s panel, who discussed and then ranked
problems with FAA‘s certification and approval processes, ranked
inconsistent interpretation of regulations, which can lead to
variation in decisions, as the first and second most significant
problem, respectively, with these processes for FAA‘s Flight Standards
Service (which issues certificates and approvals for individuals and
entities to operate in the national airspace system) and Aircraft
Certification Service (which issues approvals to the designers and
manufacturers of aircraft and aircraft parts and equipment). According
to industry stakeholders, variation in FAA‘s interpretation of
standards for certification and approval decisions is a result of
factors related to performance-based regulations, which allow for
multiple avenues of compliance, and the use of professional judgment
by FAA staff and can result in delays and higher costs.
Industry stakeholders and experts generally agreed that FAA‘s
certification and approval processes contribute to aviation safety and
work well most of the time, but negative experiences have led to
costly delays for the industry. Industry stakeholders have also raised
concerns about the effects of process inefficiencies on the
implementation of the Next Generation Air Transportation System
(NextGen)”the transformation of the U.S. national airspace system from
a ground-based system of air traffic control to a satellite-based
system of air traffic management. They said that the processes take
too long and impose costs that discourage aircraft operators from
investing in NextGen equipment. FAA has taken actions to improve the
certification and approval processes, including hiring additional
inspectors and engineers and increasing the use of designees and
delegated organizations”private persons and entities authorized to
carry out many certification activities. Additionally, FAA is working
to ensure that its processes are being followed and improved through a
quality management system, which provides a mechanism for stakeholders
to appeal FAA decisions. However, FAA does not know whether its
actions under the quality management system are achieving the intended
goals of reducing inconsistencies and increasing consistency and
fairness in the agency‘s application of regulations and policies
because FAA does not have outcome-based performance measures and a
continuous evaluative process that would allow it to determine
progress toward these goals. Without ongoing information on results,
FAA managers do not know if their actions are having the intended
effects.
What GAO Recommends:
GAO recommends that FAA develop a continuous evaluative process with
measurable performance goals to determine the effectiveness of the
agency‘s actions to improve its certification and approval processes.
The Department of Transportation provided technical comments, which
were included as appropriate.
View [hyperlink, http://www.gao.gov/products/GAO-11-14] or key
components. For more information, contact Gerald L. Dillingham, Ph.D.,
at (202) 512-2834 or dillinghamg@gao.gov.
[End of section]
Contents:
Letter:
Background:
Extent of Variation in Interpretation Is Unknown but Potentially Stems
from Factors Related to Performance-Based Regulations and FAA's
Processes:
Key Stakeholders and Experts Said the Certification and Approval
Processes Generally Work Well, but When They Do Not, It Can Be Costly
for Industry:
Conclusions:
Recommendations for Executive Action:
Agency Comments:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Summary of Responses from GAO Expert Panel:
Appendix III: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Participants in GAO's December 16, 2009, Expert Panel:
Table 2: Aviation Stakeholders Interviewed for This Study:
Figures:
Figure 1: Approvals Issued by FAA's Aircraft Certification Service,
Fiscal Years 2005-2009:
Figure 2: Organization of FAA's Aircraft Certification Service:
Figure 3: Key Phases in Aircraft Certification's Process for Approving
Aviation Products:
Figure 4: Certificates Issued by FAA's Flight Standards Service,
Fiscal Years 2005-2009:
Figure 5: Organization of FAA's Flight Standards Service:
Figure 6: Key Steps in Flight Standards' Process for Issuing
Certificates to Air Operators and Air Agencies:
Abbreviations:
CSI: Consistency and Standardization Initiative:
DOT: U.S. Department of Transportation:
FAA: Federal Aviation Administration:
ISO: International Organization for Standardization:
NextGen: Next Generation Air Transportation System:
ODA: organization designation authorization:
QMS: quality management system:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
October 7, 2010:
The Honorable John L. Mica:
Ranking Republican Member:
Committee on Transportation and Infrastructure:
House of Representatives:
The Honorable Pete Sessions:
House of Representatives:
The Federal Aviation Administration (FAA) is responsible for aviation
safety, in part by issuing certificates for new air operators, new
aircraft, and aircraft parts and equipment, as well as granting
approvals for such things as changes to air operations and aircraft,
and the design and production of aircraft parts and equipment. FAA
issues certificates and approvals based on the evaluation of aviation
industry submissions against standards set forth in federal aviation
regulations and related FAA guidance documents. Studies published over
the last 14 years have asserted that inconsistencies or variation in
FAA's interpretation and application of the regulations and guidance
hinders the efficiency of the certification and approval processes.
More recently, several aviation industry groups have asserted that
FAA's processes for carrying out these functions continue to result in
variation in decisions and inefficiencies, which can result in delays
and higher costs for their members.
You asked us to examine FAA's processes for the certification and
approval of national airspace users and products. To do so, we
addressed (1) the extent of variation in FAA's interpretation of
standards with regard to the agency's certification and approval
decisions and (2) key stakeholder and expert views on how well the
certification and approval processes work.
To fulfill these objectives, we reviewed relevant studies, reports,
and FAA documents and processes and, with the assistance of the
National Academy of Sciences, convened a panel of aviation industry
and other experts on December 16, 2009. Selected with the goal of
obtaining a balance of perspectives, the panel included FAA senior
managers; officials representing large and small air carriers,
aircraft and aerospace product manufacturers, aviation services firms,
repair stations, and aviation consultants; and academicians
specializing in aviation and organization theory. We also interviewed
trade groups and certificate and approval holders of various sizes
that represented a broad range of aviation industry sectors--including
air carriers, repair stations, and manufacturers. (See appendix I for
more information on our objectives, scope, and methodology.)
We conducted this performance audit from July 2009 to October 2010, in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.
Background:
Located in FAA's Office of Aviation Safety, the Aircraft Certification
Service (Aircraft Certification) and Flight Standards Service (Flight
Standards) issue certificates and approvals for the operators and
aviation products used in the national airspace system based on
standards set forth in federal aviation regulations. FAA inspectors
and engineers working in Aircraft Certification and Flight Standards
interpret and implement the regulations governing certificates and
approvals via FAA policies and guidance, such as orders, notices, and
advisory circulars.
Aircraft Certification:
Aircraft Certification's approximately 950 engineers and inspectors in
38 field offices issue approvals to the designers and manufacturers of
aircraft and aircraft engines, propellers, parts, and equipment,
including the avionics and other equipment required for the Next
Generation Air Transportation System (NextGen)--a federal effort to
transform the U.S. national airspace system from a ground-based system
of air traffic control to a satellite-based system of air traffic
management. These approvals are issued in three areas: (1) design--
including type certificates for new aircraft, engine, or propeller
designs,[Footnote 1] amended type certificates (issued only to the
type certificate holder) for derivative models, and supplemental type
certificates for major changes to existing designs by either the type
certificate holder or someone other than the original type certificate
holder; (2) production--including production certificates, which
certify a manufacturer's ability to build an aircraft, engine, or
propeller in accordance with an FAA-approved design, and parts
manufacturer approvals for spare and replacement parts; and (3) flight
approval--original airworthiness certificates and approvals for newly
manufactured aircraft, engines, propellers, and parts. Aircraft
Certification, along with Flight Standards, provides a safety
performance management system intended to assure the continued
operational safety of all aircraft operating in the national airspace
system and of U.S.-built aircraft operating anywhere in the world.
Aircraft Certification is also responsible for the appointment and
oversight of designees and delegated organizations that play a
critical role in acting on behalf of FAA to perform many certification
and approval activities, such as the issuance of design and
airworthiness approvals for aircraft parts.[Footnote 2]
Since 2005, Aircraft Certification has used project sequencing to
prioritize certification submissions on the basis of available
resources. Projects are evaluated against several criteria, including
safety attributes and their impact on the air transportation system.
In fiscal year 2009, Aircraft Certification issued 4,248 design
approvals, 2,971 production approvals, and 508 airworthiness
certificates. Figure 1 shows the Aircraft Certification approvals
issued for fiscal years 2005 through 2009. As of June 2010, according
to FAA, Aircraft Certification had a backlog of 47 projects.[Footnote
3] (According to a senior FAA official, the number of approvals
decreased from fiscal year 2006 to fiscal year 2007 because Aircraft
Certification implemented a new data collection system in fiscal year
2007 that improved data collection definitions and processes.) Figure
2 contains key information about Aircraft Certification's
organization, and figure 3 indicates key phases in Aircraft
Certification's product approvals process.
Figure 1: Approvals Issued by FAA's Aircraft Certification Service,
Fiscal Years 2005-2009:
[Refer to PDF for image: stacked vertical bar graph]
Fiscal year: 2005;
Design approvals: 13,093;
Production approvals: 2,547;
Airworthiness certificates: 957.
Fiscal year: 2006;
Design approvals: 14,240;
Production approvals: 2,935;
Airworthiness certificates: 946.
Fiscal year: 2007;
Design approvals: 4,853;
Production approvals: 2,499;
Airworthiness certificates: 788.
Fiscal year: 2008;
Design approvals: 4,456;
Production approvals: 2,642;
Airworthiness certificates: 597.
Fiscal year: 2009;
Design approvals: 4,248;
Production approvals: 2,971;
Airworthiness certificates: 508.
Source: GAO analysis of FAA data.
Note: Production approvals include new production certificates,
amended/revised production certificates, approved production
inspection system authorizations, parts manufacturer approval letters
(including supplements), and corrections to approved parts
manufacturer approvals.
[End of figure]
Figure 2: Organization of FAA's Aircraft Certification Service:
[Refer to PDF for image: illustration]
Headquarters offices and 4 geographic directorates for transport
airplanes, small airplanes, rotorcraft, and engines and propellers:
Design approval:
Key field offices:
* 14 Aircraft Certification offices.
Key functions:
* Issuing design approvals (e.g., type certificates, parts
manufacturer approvals) for aircraft and component designs that meet
standards;
* Oversight of designees and delegated organizations.
Key staffing:
* 640 engineers;
* 75 managers of engineers.
Production and flight approval:
Key field offices:
* 19 manufacturing inspection district offices;
* 4 manufacturing inspection satellite offices;
* Boeing certificate management office;
Key functions:
* Issuing original airworthiness certificates for aircraft and
production certificates for aircraft and components;
* Assisting aircraft certification offices with design approvals;
Key staffing:
* 221 inspectors;
* 22 managers of inspectors.
Source: FAA.
[End of figure]
Figure 3: Key Phases in Aircraft Certification's Process for Approving
Aviation Products:
[Refer to PDF for image: illustration]
1. Conceptual design:
Begin to develop the design concept for a product that may lead to a
viable certification project.
2. Certification basis:
Clarify the product design and apply certification standards to arrive
at the certification basis for the product.
3. Compliance planning:
Commit to a plan to manage the product certification project.
4. Implementation:
Work together closely to ensure that all agreed-upon product-specific
certification requirements are met.
5. Postcertification:
Close-out activities provide the foundation for continued
airworthiness activities and certificate management for the remainder
of the product‘s life cycle.
Source: FAA.
Note: During each phase, both the applicant and FAA staff are
involved. FAA staff include managers, engineers, inspectors, flight
test pilots, a chief scientist, and technical advisors, as well as an
aircraft evaluation group from Flight Standards.
[End of figure]
Flight Standards:
Flight Standards' nearly 4,000 inspectors issue certificates allowing
individuals and entities to operate in the national airspace system.
Flight Standards also issues approvals for programs, such as training
and minimum equipment lists.[Footnote 4] Flight Standards field office
managers in over 100 field offices use the Certification Services
Oversight Process to initiate certification projects within their
offices. According to FAA, the field offices are also assisted by a
headquarters-based office that provides experts on specific aircraft
and airlines. Accepted projects are processed on a first-in, first-out
basis within each office once FAA determines that it has the resources
to oversee an additional new certificate holder. Flight Standards
issued 599 air operator and air agency certificates in fiscal year
2009. These include certificates to commercial air carriers under 14
C.F.R. part 121, operators of smaller commercial aircraft under 14
C.F.R. part 135, repair stations under 14 C.F.R. part 145, and pilot
schools and training centers under 14 C.F.R. parts 141 and 142,
respectively. According to its Director, Flight Standards also issues
over 6,000 approvals daily. Figure 4 shows the number of air operator
and air agency certificates issued by Flight Standards in fiscal years
2005 through 2009.
Figure 4: Certificates Issued by FAA's Flight Standards Service,
Fiscal Years 2005-2009:
[Refer to PDF for image: stacked vertical bar graph]
Fiscal year: 2005;
Air operators: 331;
Air agencies: 393.
Fiscal year: 2006;
Air operators: 265;
Air agencies: 366.
Fiscal year: 2007;
Air operators: 265;
Air agencies: 391.
Fiscal year: 2008;
Air operators: 219;
Air agencies: 389.
Fiscal year: 2009;
Air operators: 312;
Air agencies: 287.
Source: GAO analysis of FAA data.
Note: Air agencies include aircraft repair stations and pilot schools;
air operators include operators of large and small commercial
passenger aircraft and agricultural operators.
[End of figure]
FAA officials noted that certification projects within and among the
categories of air operators and air agencies require various amounts
of FAA resources. For example, FAA indicated that an agricultural
operator certification requires fewer FAA resources than a repair
station certification. Additionally, certifications of small
commercial aircraft operations that are single pilot, single plane
require a different set of resources than operations that are dual
pilot and/or fly more aircraft. As of July 2010, Flight Standards had
1,142 certifications in process and a backlog of 489 applications.
[Footnote 5] According to an FAA official, Flight Standards has more
wait-listed applications than Aircraft Certification because it
receives numerous requests for certificates, and its certifications
are substantially different in nature from those issued by Aircraft
Certification.
Flight Standards is also responsible for assuring the continued
operational safety of the national airspace system by overseeing
certificate holders, monitoring (along with Aircraft Certification)
operators' and air agencies' operation and maintenance of aircraft,
and overseeing designees and delegated organizations. Flight Standards
inspectors were tasked with overseeing 13,089 air operators and air
agencies, such as repair stations, as of March 2010. Unless assigned
to a large commercial air carrier issued a certificate under part 121,
a Flight Standards inspector is typically responsible for overseeing
several entities that often perform different or several functions
within the system--including transporting passengers, repairing
aircraft, and training pilots.
Figures 5 and 6 contain key information about Flight Standards'
organization and certification process for air operators and air
agencies.
Figure 5: Organization of FAA's Flight Standards Service:
[Refer to PDF for image: illustration]
Headquarters and 8 regional offices:
Key field offices:
* 82 Flight Standards district offices;
* 19 certificate management offices for major air carriers and some
part 142 training centers.
Key functions:
* Initial certification of an air operator (e.g., a part 121 air
carrier) or air agency (e.g., a part 145 repair station) to perform a
specific aviation activity in the national airspace system;
* Approval of aviation users‘ initial programs and revisions to
ongoing programs that are part of their certification; these may
include training, operating manuals, and cockpit checklists;
* Oversight of designees and delegated organizations.
Key staffing:
* 1,416 part 121 inspectors;
* 2,552 other inspectors.
Other field offices:
* 5 aircraft evaluation groups coordinate and assist with aircraft
certification and continued airworthiness programs;
* 7 international field offices and units approve maintenance programs
and minimum equipment lists.
Source: FAA.
Note: A minimum equipment list is a list of all equipment on an
aircraft type. It details which equipment FAA has determined may be
inoperative under certain operational conditions and still provide an
acceptable level of safety.
[End of figure]
Figure 6: Key Steps in Flight Standards' Process for Issuing
Certificates to Air Operators and Air Agencies:
[Refer to PDF for image: illustration]
1. Preapplication:
The applicant delivers a preapplication statement of intent to an FAA
field office.
2. Project acceptance:
FAA field office and division managers determine if the certification
project can be accepted in light of available resources.
3. Project assignment:
The field office manager contacts the Air Transportation Oversight
System certificate management office for a part 121 certification. For
a non-part 121 certification, the field office manager assigns a
certification project manager and additional inspectors, as necessary.
4. Project schedule:
The field office manager determines if an accepted project can begin
at a time that meets the office‘s workload and is agreeable to the
applicant and notifies the applicant.
5. Formal application:
The applicant submits a formal application, including a schedule of
events. The district office and certification team review the
application, may hold a formal application meeting, and determine
whether to accept the application.
6. Document compliance:
The certification team reviews manuals and other documents and, if
they meet standards, approves them.
7. Demonstration and inspection:
The certification team inspects the applicant‘s facilities and
equipment and observes personnel in the performance of their duties.
The team emphasizes compliance with regulations and safe operating
practices.
8. Certification:
The certification project manager submits a report to the region for
concurrence with the certification team and issues the certificate.
Source: FAA.
Note: These steps are accomplished within a four-phase process for
part 121 certifications and a five-phase process for part 135 and
repair station certifications. For part 121, the phases include
application, design assessment, performance assessment, and
administrative functions. For part 135 and repair stations, the phases
are preapplication, formal application, document compliance,
demonstration and inspection, and certification.
[End of figure]
Extent of Variation in Interpretation Is Unknown but Potentially Stems
from Factors Related to Performance-Based Regulations and FAA's
Processes:
Extent of Variation in FAA's Interpretation of Standards for
Certification and Approval Decisions Is Unknown, but Stakeholders and
Experts Indicate That Serious Problems Occur Infrequently:
Studies we reviewed and aviation stakeholders and experts we spoke
with indicated that variation in FAA's interpretation of standards for
certification and approval decisions is a long-standing issue that
affects both Aircraft Certification and Flight Standards, but the
extent of the problem has not been quantified in the industry as a
whole. Inconsistent or variant FAA interpretations have been noted in
studies published over the last 14 years. A 1996 study by Booz Allen &
Hamilton, conducted at the request of the FAA Administrator to assess
challenges to the agency's regulatory and certification practices,
reported that, for air carriers and other operators, the agency's
regulations are often ambiguous; subject to variation in
interpretation by FAA inspectors, supervisors, and policy managers;
and in need of simplification and consistent implementation.[Footnote
6] A 1999 task force, convened at the request of the FAA Administrator
to assess FAA's certification process, found that the agency's
requirements for the various approvals--such as type certificates and
supplemental type certificates--varied substantially because of
differences in standards and inconsistent application of those
standards by different FAA field offices.[Footnote 7] While FAA has
put measures in place since these two reports appeared, a 2008
Independent Review Team, which was commissioned by the Secretary of
Transportation to assess FAA's safety culture and approach to safety
management, found that a wide degree of variation in "regulatory
ideology" among FAA staff continues to create the likelihood of wide
variation in decisions within and among field offices.[Footnote 8]
Industry officials and experts representing a broad range of large and
small aviation businesses told us that variation in interpretation and
subsequent decisions occurs in both Aircraft Certification and Flight
Standards, but we found no evidence that quantified the extent of the
problem in the industry as a whole.[Footnote 9] Specifically, 10 of
the 13 industry group and individual company representatives we
interviewed said that they or members of their organization
experienced variation in FAA's certification and approval decisions on
similar submissions; the remaining 3 industry representatives did not
raise variation in interpretations and decisions as an issue. For
example, an official from one air carrier told us that variation in
decisions occurs regularly when obtaining approvals from Flight
Standards district offices, especially when dealing with inspectors
who are newly hired or replacing a previous inspector. He explained
that new inspectors often task air carriers to make changes to
previously obtained minimum equipment lists or conformity approvals
for an aircraft.[Footnote 10] The official further noted that
inspector assignments often change for reasons such as transfers,
promotions, or retirement and that four different principal operations
inspectors were assigned to his company during the past 18 months.
Experts on our panel and most industry officials we interviewed
indicated that, though variation in decisions is a long-standing,
widespread problem, it has rarely led to serious certification and
approval process problems. Experts on our panel generally noted that
serious problems with the certification and approval processes occur
less than 10 percent of the time. However, when we asked them to rank
certification and approval process problems we summarized from their
discussion, they chose inconsistent interpretation of regulations,
which can lead to variation in decisions, as the most significant
problem for Flight Standards and as the second most significant
problem for Aircraft Certification.[Footnote 11] Panelists' concerns
about variation in decisions included instances in which approvals are
reevaluated and sometimes revised or revoked in FAA jurisdictions
other than those in which they were originally granted. Industry
officials we interviewed, though most had experienced it, did not
mention the frequency with which variation in decisions occurred.
However, 8 of the 13 said that their experiences with FAA's
certification and approval processes were generally free of problems
compared with 3 who said they regularly experienced problems with the
process.[Footnote 12]
FAA's Deputy Associate Administrator for Aviation Safety and union
officials representing FAA inspectors and engineers acknowledged that
variation in certification and approval decisions occurs. The Deputy
Associate Administrator noted that variation in interpretation and
certification and approval decisions occurs in both Aircraft
Certification and Flight Standards. He acknowledged that a
nonstandardized process for approvals exists and has been a challenge
for, and a long-term criticism of, the agency. Furthermore, he
explained that efforts were being made to address the issue, including
the establishment of (1) an Office of Aviation Safety quality
management system (QMS) to standardize processes across Aircraft
Certification and Flight Standards, (2) a process for industry to
dispute FAA decisions, and (3) standardization offices within Aircraft
Certification directorates. The first two efforts are discussed in
greater detail later in this report.
Industry Stakeholders Noted That Variation in Decisions Occurs as a
Consequence of Performance-Based Regulations and FAA's Exercise of
Professional Judgment:
Variation in FAA's interpretation of standards and certification and
approval decisions occurs as a result of factors related to
performance-based regulations and the use of professional judgment by
FAA inspectors and engineers, according to industry stakeholders. FAA
uses performance-based regulations, which identify a desired outcome
and are flexible about how the outcome is achieved. For example,
performance-based regulations on aircraft braking would establish
minimum braking distances for aircraft but would not call for a
particular material in the brake pads or a specific braking system
design. According to officials in FAA's rulemaking office, about 20
percent of FAA's regulations are performance-based. Performance-based
regulations, which are issued governmentwide,[Footnote 13] provide a
number of benefits, according to literature on the regulatory process.
[Footnote 14] By focusing on outcomes, for example, performance-based
regulations give firms flexibility in achieving the stated level of
performance; such regulations can accommodate technological change in
ways that prescriptive regulations that focus on a specific technology
generally cannot. For those certifications and approvals that relate
to performance-based regulations, variation in decisions is a
consequence of such regulations, according to one air carrier, since
performance-based regulations allow the applicant multiple avenues to
comply with regulations and broader discretion by FAA staff in making
certification and approval decisions. According to senior FAA
officials, performance-based regulations allow innovation and
flexibility while setting a specific safety standard. The officials
added that the benefits of performance-based regulations outweigh the
potential for erroneous interpretation by an individual inspector or
engineer. While agreeing with this statement, a panel member pointed
out that the potential for erroneous interpretation also entails a
risk of inconsistent decisions.
In addition, FAA oversees a large, diverse industry, and its
certification and approval processes rely, in part, on FAA staffs'
exercise of professional judgment in the unique situations they
encounter. In the opinion of senior FAA officials, some differences
among inspectors may be due to situation-specific factors that
industry stakeholders may not be aware of. According to officials from
Flight Standards, because differences may exist among regions and
district offices, operators changing locations may encounter these
differences.
Key Stakeholders and Experts Said the Certification and Approval
Processes Generally Work Well, but When They Do Not, It Can Be Costly
for Industry:
Stakeholders and Experts Said the Certification and Approval Processes
Contribute to System Safety and Work Well Most of the Time:
Many industry stakeholders and experts stated that FAA's certification
and approval processes contribute positively to the safety of the
national airspace system. For example, industry stakeholders who
participated in our expert panel ranked the office's safety culture
and record as the greatest strength of Flight Standards' certification
and approval processes and the third greatest strength of Aircraft
Certification's processes.
Industry stakeholders and experts also noted that the certification
and approval processes work well most of the time because of FAA's
long-standing collaboration with industry, flexibility within the
processes, and committed, competent FAA staff. In most instances,
stakeholders and experts said, when industry seeks certifications and
approvals, its experiences with FAA's processes are positive. For
example, two aviation manufacturers and an industry trade association
with over 400,000 members noted that most of their experiences or
their members' experiences were positive. Seventeen of 19 panelists
indicated positive or very positive experiences with Aircraft
Certification, and 9 of 19 panelists indicated positive experiences
with Flight Standards.[Footnote 15] Panelists ranked FAA's
collaboration with applicants highly--as the second greatest strength
of both Aircraft Certification and Flight Standards. In addition,
representatives of two trade associations representing over 190
aviation companies said that the processes provide flexibility for a
large, diverse industry. Additionally, panelists ranked FAA
inspectors' and engineers' expertise as the greatest strength of
Aircraft Certification and the third greatest strength of Flight
Standards, while officials from two industry trade groups cited the
inspectors' and engineers' competence and high level of expertise.
Industry Stakeholders Said Negative Experiences Are Infrequent but Can
Cause Costly Delays:
Industry stakeholders and experts noted that negative certification
and approval experiences, although infrequent, can result in costly
delays for them, which can disproportionately affect smaller
operators. While industry stakeholders indicated that negative
experiences occur in dealings with both Aircraft Certification and
Flight Standards, experts on our panel noted that negative experiences
are more likely to occur with Flight Standards than with Aircraft
Certification. For example, three experts noted that, overall,
industry's experience in obtaining certifications and approvals from
Flight Standards has been negative or very negative, while no experts
thought industry's experience with Aircraft Certification was
negative. The panelists indicated that negative experiences occur
during the processing of certifications and approvals and as
applicants wait for FAA resources to become available to commence
their certification or approval projects. For example, an aviation
industry representative reported that his company incurred a delay of
over 5 years and millions of dollars in costs when it attempted to
obtain approvals from Aircraft Certification and Flight Standards
field offices. Another industry representative indicated that it
abandoned an effort to obtain an operating certification after
spending $1.2 million and never receiving an explanation from FAA as
to why the company's application was stalled. One panelist indicated
that the negative experiences focus more on administrative aspects of
the certification and approval processes and not on safety-related
items.
The processing of original certifications and approvals in Aircraft
Certification and Flight Standards involves progressing through a
schedule of steps or phases. Responsibilities of both FAA and the
applicant are delineated. However, even with this framework in place,
industry stakeholders noted that the time it takes to obtain
certifications and approvals can differ from one FAA field office to
another because of differences in office resources and expertise. In
some cases, delays may be avoided when FAA directs the applicant to
apply at a different field office. Nevertheless, applicants who must
apply to offices with fewer resources can experience costly delays in
obtaining certifications or approvals.
Delays also occur when FAA wait-lists certification submissions
because it does not have the resources to begin work on them. Aircraft
Certification meets weekly to review certification project
submissions. If it determines that a submission is to be wait-listed,
the applicant is sent a 90-day delay letter and if, after the initial
90 days, the submission is still wait-listed, the applicant is sent
another letter. Additionally, Aircraft Certification staff and
managers periodically contact applicants to advise them of the status
of their submissions. Flight Standards also notifies applicants when
their certification submissions are wait-listed, and Flight Standards
staff are encouraged to communicate with applicants regularly about
the status of their submissions. However, according to an FAA notice,
staff are advised not to provide an estimate of when an applicant's
submission might be processed.[Footnote 16] While Aircraft
Certification tracks in a national database how long individual
submissions are wait-listed, Flight Standards does not. Without data
on how long submissions are wait-listed, Flight Standards cannot
assess the extent of wait-listing delays or reallocate resources to
better meet demand. Further, industry stakeholders face uncertainty
with respect to any plans or investments that depend on obtaining a
certificate in a timely manner.
Industry stakeholders have also raised concerns about the effects of
inefficiencies in the certification and approval processes on the
implementation of NextGen. As NextGen progresses, operators will need
to install additional equipment on their aircraft to take full
advantage of NextGen capabilities, and FAA's certification and
approval workload is likely to increase substantially. According to
our October 2009 testimony on NextGen, airlines and manufacturers said
that FAA's certification processes take too long and impose costs on
industry that discourage them from investing in NextGen equipment.
[Footnote 17] We reported that this inefficiency in FAA's processes
constitutes a challenge to delivering NextGen benefits to stakeholders
and that streamlining FAA's processes will be essential for the timely
implementation of NextGen. FAA is working to address the certification
issues that may impede the adoption and acceleration of NextGen
capabilities.[Footnote 18] Flight Standards has identified NextGen-
dedicated staff in each of its regional offices to support the review
and approval of NextGen capabilities within each region. Aircraft
Certification has created a team of experts from different offices to
coordinate NextGen approvals and identify specialists in Aircraft
Certification offices with significant NextGen activity. FAA also
plans a number of other actions to facilitate the certification and
approval of NextGen-related technology, including new procedures and
criteria for prioritizing certifications, updating policy and
guidance, developing additional communication mechanisms, and
developing training for inspectors and engineers. Since many of these
actions have either just been implemented or have not yet been
completed, it is too early to tell whether they will increase the
efficiency of FAA's certification and approval processes and reduce
unanticipated delays and costs for the industry.
Industry Stakeholders and Experts Told Us That the Efficiency of FAA's
Processes Is Hampered by Several Shortcomings; FAA Has Taken Some
Actions to Remedy Them:
Industry stakeholders also noted that the efficiency of the
certification and approval processes was hampered by a lack of
sufficient staff resources to carry out certifications and approvals
and a lack of effective communication mechanisms for explaining the
intent of the regulations to both FAA staff and industry. The
stakeholders said that these inefficiencies have resulted in costly
delays for them.
Stakeholders and experts said that, at some FAA offices, delays in
obtaining certifications and approvals were due to heavy staff
workloads, a lack of staff, or a lack of staff with the appropriate
expertise. Staff and managers at one FAA field office told us that in
the past a lack of staff had contributed to delays in completing
certifications. The relative priority of certifications and approvals
within FAA's overall workload also affects the availability of staff
to process certifications and approvals. According to FAA, its highest
priority is overseeing the continued operational safety of the people
and products already operating within the national airspace system,
[Footnote 19] but the same staff who provide this oversight are also
tasked with the lower-priority task of processing new certifications
and approvals. Additionally, Flight Standards field staff we contacted
said that the system under which their pay grades are established and
maintained provides a disincentive for inspectors to perform
certification work because the system allocates no credit toward
retention of their pay grades for doing certification work. Flight
Standards headquarters officials pointed out that there is an
incentive for field office inspectors to perform initial
certifications because once certificated the new entities add points
to an inspector's complexity calculation, which is used to determine
his or her pay grade.[Footnote 20]
FAA has addressed staff resource issues by increasing the number of
inspectors and engineers. Over the past 3 years, FAA has steadily
increased its hiring of Aircraft Certification engineers and Flight
Standards inspectors, thereby reducing the risk of certification
delays. According to agency data, FAA's hiring efforts since fiscal
year 2007 have resulted in an 8.8 percent increase in the number of
Aircraft Certification engineers and a 9.4 percent increase in the
number of Flight Standards inspectors on board. FAA hired 106
engineers in Aircraft Certification and 696 inspectors in Flight
Standards from the beginning of fiscal year 2007 to March 15, 2010.
FAA also hired 89 inspectors in Aircraft Certification from fiscal
year 2007 through August 2010.[Footnote 21] In addition, Flight
Standards headquarters staff are available to assist field staff with
the certification of part 121 air carriers--an average of 35 of these
staff were available for this assistance annually from 2005 through
2009, and they helped with 16 certification projects.
Furthermore, FAA delegates many certification activities to
individuals and organizations (called designees) to better leverage
its resources. As we previously reported, FAA's designees perform more
than 90 percent of FAA's certification activities. We have reported
that designees generally conduct routine certification functions, such
as approvals of aircraft technologies that the agency and designees
already have experience with, allowing FAA staff to focus on new and
complex aircraft designs or design changes.[Footnote 22] Panelists
ranked the expanded use of designees second and fifth, respectively,
among actions that we summarized from their discussions that would
have the most positive impact on improving Aircraft Certification's
and Flight Standards' certification and approval processes. FAA is
increasing organizational delegations under its organization
designation authorization (ODA) program and expects the ODA program
will allow more effective use of its resources over time.
Stakeholders pointed to a lack of effective communication mechanisms
as another problem with the certification and approval processes,
especially deficiencies in the guidance FAA issues and a lack of
additional communication mechanisms for sharing information on the
interpretation of regulations. Stakeholders said that the lack of
effective communication mechanisms can lead to costly delays when, for
example, methods or guidance for complying with regulations is not
clear. Stakeholders and experts had several issues with the FAA
guidance that interprets the regulations and provides supplemental
information to the industry. Stakeholders said there are sometimes
discrepancies between the guidance and the regulations. For example,
one stakeholder reported informing an FAA training course instructor
that a particular piece of guidance contradicted the regulations. The
instructor agreed that the contradiction existed but told the
stakeholder that FAA teaches to the guidance, not the regulations.
[Footnote 23] One employee group representing some FAA inspectors was
concerned that not all guidance has been included in an online system
that FAA has established to consolidate regulations, policy, and
guidance. FAA acknowledged that it is working to further standardize
and simplify the online guidance in the Flight Standards information
management system.
Stakeholders also identified a lack of opportunities for sharing
information about the interpretation of regulations and guidance. An
industry expert noted that FAA lacks a culture that fosters
communication and discussion among peer groups. Moreover, an industry
group with over 300 aviation company members suggested that FAA should
support and promote more agencywide and industrywide information
sharing in less formal, less structured ways to enhance communication.
Finally, according to an official of an employee group representing
some FAA inspectors, because their workloads tend to be heavy,
inspectors are less able to communicate with the companies they
oversee, and the reduced level of communication contributes to
variation in the interpretation of regulations. FAA officials
disagreed with these assertions and indicated that FAA staff
participate in numerous committees and conferences, share methods of
compliance in technical areas via forums with stakeholders, and
communicate resolutions to problems in various formats, such as by
placing legal decisions online.
FAA Has Taken Other Actions That Might Address Process Shortcomings
but Lacks Performance Data to Assess the Actions' Effectiveness:
Other FAA actions could identify and potentially address some of the
shortcomings in the agency's certification and approval processes as
follows:
* In 2004, FAA's Office of Aviation Safety introduced QMS, which is
intended to ensure that processes are being followed and improved and
to provide a methodology to standardize processes. QMS is expected to
help ensure that processes are followed by providing a means for staff
to report nonconformance with FAA procedures or processes and was
established as part of the office's effort to achieve certification by
the International Organization for Standardization (ISO).[Footnote 24]
Any employee can submit a report and check the status of an issue that
has been reported. From October 2008 to March 2009, approximately 900
reports were submitted, and 46 internal audits were completed. For
example, in July 2009, an FAA staffer noted that a required paragraph
on aging aircraft inspection and records review was missing from a
certificate holder's operations specifications.[Footnote 25] The issue
was resolved and closed in August 2009 when the missing paragraph was
issued to the certificate holder. Some FAA staff told us that QMS has
helped improve the processes because it requires management action to
respond to report submissions.
* To provide industry stakeholders with a mechanism for appealing
certification and other decisions, the Office of Aviation Safety
implemented the Consistency and Standardization Initiative (CSI) in
2004.[Footnote 26] Appeals must begin at the field office level and
can eventually be taken to FAA headquarters. CSI requires that FAA
staff document their safety decisions and that stakeholders support
their positions with specific documentation. Within Aircraft
Certification and Flight Standards, CSI cases at each appeal level are
expected to be processed within 30 working days. The total length of
the CSI process depends on how many levels of appeal the stakeholder
chooses. Aircraft Certification has had over 20 CSI cases, and Flight
Standards has had over 300. Most CSI cases in Aircraft Certification
involved clarification of a policy or an approved means of complying
with a regulation, while most of those submitted to Flight Standards
involved policy or method clarification, as well as scheduling issues,
such as delays in addressing a stakeholder's certification, approval,
or other issue. The large discrepancy between the number of cases
filed for the two services, according to FAA officials, may be due to
the fact that Aircraft Certification decisions are the result of
highly interactive, deliberative processes, which are not typical in
granting approvals in Flight Standards, where an inspector might find
the need to hand down a decision without prolonged discussion or
deliberation. Stakeholders told us that CSI lacks agencywide buy-in
and can leave stakeholders who use the program potentially open to
retribution from FAA staff.[Footnote 27] However, others noted that
CSI is beneficial because it requires industry stakeholders to use the
regulations as a basis for their complaints, which often leads to
resolution. According to one of our panelists, inconsistencies occur
when FAA does not start with the regulations as the basis for
decisions.
Although QMS and CSI are positive steps toward identifying ways to
make the certification and approval processes more efficient, FAA does
not know whether the programs are achieving their stated goals because
it has not established performance measures for determining program
accomplishments. One of the goals for QMS is to reduce inconsistencies
and increase standardization. A QMS database documents the reports
submitted and, through information in these reports, FAA says it has
identified instances of nonconformance and initiated corrective action
to prevent recurrence; revised orders to ensure they are consistent
with actual practice; and improved its processes to collect feedback
from stakeholders and take action on trends. However, FAA does not
know whether its actions have reduced inconsistencies because its
measures describe the agency's output--for example, number of audits
conducted--rather than any outcomes related to reductions in process
inconsistencies. FAA officials described CSI goals as promoting early
resolution of disagreements and consistency and fairness in applying
FAA regulations and policies. They provided us with data on the number
of CSI cases in both Aircraft Certification and Flight Standards, the
types of complaints, and the percentage of resolutions that upheld
FAA's original decision, but as with the overall QMS program, we could
find no evidence that FAA has instituted CSI performance measures that
would allow it to determine progress toward program outcomes, such as
consistency and fairness in applying regulations and policies. Outcome-
based performance measures would also allow QMS and CSI program
managers to determine where to better target program resources to
improve performance.
Conclusions:
FAA has taken actions to address variation in decisions and
inefficiency in its certification and approval processes, although the
agency does not have outcome-based performance measures and a
continuous evaluative process to determine if these actions are having
their intended effects. Because the number of certification and
approval applications is likely to increase for NextGen technologies,
achieving more efficiency in these processes will help FAA better
manage this increased workload, as well as its current workload. In
addition, while both Aircraft Certification and Flight Standards
notify applicants whether resources are available to begin their
projects, Flight Standards does not monitor how long applicants are
wait-listed and is therefore unaware how long projects are wait-listed
and unable to reallocate resources to better meet demand for
certification services.
Recommendations for Executive Action:
To ensure that FAA actions contribute to more consistent decisions and
more efficient certification and approval processes, we recommend that
the Secretary of Transportation direct the Administrator of FAA to
take the following two actions:
* Determine the effectiveness of actions to improve the certification
and approval processes by developing a continuous evaluative process
and use it to create measurable performance goals for the actions,
track performance toward those goals, and determine appropriate
process changes. To the extent that this evaluation of agency actions
identifies effective practices, consider instituting those practices
agency wide.
* Develop and implement a process in Flight Standards to track how
long certification and approval submissions are wait-listed, the
reasons for wait-listing them, and the factors that eventually allowed
initiation of the certification process. Use the data generated from
this process to assess the extent of wait-listing delays and to
reallocate resources, as appropriate, to better meet demand.
Agency Comments:
We provided a copy of a draft of this report to the Department of
Transportation (DOT) for its review and comment. DOT provided
technical comments, which we incorporated as appropriate.
As agreed with your offices, unless you publicly announce the contents
of this report earlier, we plan no further distribution until 21 days
from the report date. At that time, we will send copies to the
appropriate congressional committees, the Secretary of Transportation,
the Administrator of FAA, and other interested parties. The report
also will be available at no charge on the GAO Web site at [hyperlink,
http://www.gao.gov].
If you or your staff members have any questions or would like to
discuss this work, please contact me at (202) 512-2834 or
dillinghamg@gao.gov. Contact points for our offices of Congressional
Relations and Public Affairs may be found on the last page of this
report. GAO staff who made key contributions to this report are listed
in appendix III.
Signed by:
Gerald L. Dillingham, Ph.D.
Director:
Physical Infrastructure Issues:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
This report provides information on the Federal Aviation
Administration's (FAA) processes for granting certifications and
approvals to air operators, air agencies such as repair stations, and
designers and manufacturers of aircraft and aircraft components. It
describes the processes and discusses (1) the extent of variation in
FAA's interpretation of standards with regard to the agency's
certification and approval decisions and (2) key stakeholder and
expert views on how well the certification and approval processes
work. To address these objectives, we reviewed relevant studies,
reports, and FAA documents and processes; convened a panel of aviation
industry and other experts; and interviewed aviation industry members,
an expert, and FAA officials. We did not address FAA processes for
issuing certifications to individuals, such as pilots and mechanics.
Expert Panel:
We contracted with the National Academy of Sciences (the Academy) to
convene a panel on FAA's certification and approval processes on
December 16, 2009. The panel was selected with the goal of obtaining a
balance of perspectives and included FAA senior managers; officials
representing large and small air carriers, aircraft and aerospace
product manufacturers, aviation services firms, repair stations,
geospatial firms, and aviation consultants; and academics specializing
in aviation and organization theory. (See table 1.)
Table 1: Participants in GAO's December 16, 2009, Expert Panel:
Presenters:
John Allen, Director, Flight Standards Service, FAA.
Dorenda Baker, Director, Aircraft Certification Service, FAA.
Eric Byer, Vice President, Government and Industry Affairs, National
Air Transportation Association (NATA).
Walter Desrosier, Vice President, Engineering and Maintenance, General
Aviation Manufacturers Association (GAMA).
Joe White, Managing Director, Engineering, Maintenance and Materiel,
Air Transport Association of America (ATA).
Panelists:
Bill Ashworth, Consultant.
Ali Bahrami, Manager, Transport Airplane Directorate, FAA.
Chris Benich, Director, Aerospace Regulatory Affairs, Honeywell.
John Byrd, Government Affairs Manager, Management Association for
Private Photogrammetric Surveyors (MAPPS).
John Duncan, Manager, Flight Standards Air Transportation Division,
FAA.
Jacque Holloway, ODA Administrator, Cessna Aircraft Company.
Dennis Keith, President, Jet Solutions, L.L.C.
Todd LaPorte, Professor Emeritus of Political Science and Professor of
the Graduate School, University of California, Berkeley.
Jerry Mack, President, Mack Global Solutions.
Sarah MacLeod, Executive Director, Aeronautical Repair Station
Association (ARSA).
Kristine Marcy, Consultant, McConnell International.
Bill McCabe, President, The McCabe Group, L.L.C.
Roger Southgate, Director of Avionics Certification, Rockwell Collins.
John Strong, CSX Professor of Finance and Economics, The College of
William and Mary (Moderator).
Brad Tuttle, General Manager, 10 Tanker Companies.
Gregory Walden, Of Counsel, Patton Boggs L.L.P.
Bill Whitton, Vice President-Lead Administrator, Gulfstream ODA.
Mark Yerger, Vice President of Aircraft Engineering and Technical
Planning, FedEx.
Dan Zuspan, Director, Aviation and Regulatory Affairs, Boeing
Commercial Airplanes.
Source: GAO.
[End of table]
In the first session, FAA and industry officials presented their
organizations' perspectives on these processes and responded to
questions. The presenters then departed and did not participate in the
remaining sessions. In the next three discussion sessions, the
panelists--led by a moderator--shared their views on various aspects
of FAA's certification and approval processes. After the first two
discussion sessions, panelists voted in response to questions posed by
GAO. (See appendix II for the questions and responses.) The views
expressed by the panelists were their own and do not necessarily
represent the views of GAO or the Academy. We shared a copy of an
earlier draft of this report with all of the presenters and panelists
for their review and to ensure that we correctly captured information
from their discussions and, on the basis of their comments, made
technical corrections to the draft as necessary.
Industry Interviews:
We interviewed aviation industry certificate and approval holders,
trade groups, an industry expert, officials of unions that represent
FAA inspectors and engineers, and FAA staff in Aircraft Certification
and Flight Standards (see table 2). The industry and trade groups were
selected to provide a range of large and small companies and a variety
of industry sectors (e.g., aircraft and parts manufacturers, air
carriers, and repair stations). The interviews were conducted to gain
an understanding of the extent of variation in FAA's certification and
approval decisions and interviewees' views on FAA's certification and
approval processes. The FAA interviews provided an understanding of
the key aspects of FAA's certification and approval processes,
information on data collection and analysis related to the processes,
and current and planned process improvement efforts. In addition to
using information from the individual interviews, as relevant
throughout the report, we analyzed the content of the interviews to
identify and quantify the key issues raised by the interviewees.
Table 2: Aviation Stakeholders Interviewed for This Study:
Aviation certificate and approval holders:
Boeing:
Honeywell:
Jet Aviation:
Jet Solutionsv
Ryan International Airlinesv
Aviation industry expert:
Anthony Broderick, Independent Aviation Safety Consultant:
Aviation industry trade groups:
Aeronautical Repair Station Association (ARSA):
Aerospace Industries Association (AIA):
Aircraft Owners and Pilots Association (AOPA):
General Aviation Manufacturers Association (GAMA):
Management Association for Private Photogrammetric Surveyors (MAPPS):
National Air Carrier Association (NACA):
National Air Transportation Association (NATA):
National Association of Flight Instructors (NAFI):
National Business Aviation Association (NBAA):
Regional Airline Association (RAA):
FAA:
Aviation Safety Organization:
Aircraft Certification Service:
Aircraft Certification Service's Transport Airplane Directorate,
Renton, WA.
Flight Standards Service:
Flight Standards District Office, Washington, D.C.
FAA inspector and engineer unions:
National Air Traffic Controllers Association (NATCA):
Professional Aviation Safety Specialists (PASS):
Source: GAO.
[End of table]
[End of section]
Appendix II: Summary of Responses from GAO Expert Panel:
This appendix summarizes the responses the panelists provided to
questions we posed at the close of their discussion sessions. The
response options were based on the contents of their discussions. To
develop the rankings in questions 1, 2, and 12, we asked the
panelists, in a series of three questions, to vote for the option he
or she believed was the first, second, and third greatest, most
significant, or most positive. To rank order the items listed for
these questions, we assigned three points to the option identified as
greatest, most significant, or most positive; two points to the second
greatest, most significant, or most positive; and one point to the
third greatest, most significant, or most positive option. We then
summed the weighted values for each option and ranked the options from
the highest number of points to the lowest.
1. What is the greatest strength of the certification and approval
processes?
Strength ranking: 1;
Aircraft Certification: Inspector/engineer expertise;
Flight Standards: Safety culture and record.
Strength ranking: 2;
Aircraft Certification: Collaboration with applicants;
Flight Standards: Collaboration with applicants.
Strength ranking: 3;
Aircraft Certification: Safety culture and record;
Flight Standards: Inspector/engineer expertise.
Strength ranking: 4;
Aircraft Certification: Clear well-defined requirements and processes;
Flight Standards: Do not know/no basis to judge.
Strength ranking: 5;
Aircraft Certification: None;
Flight Standards: Flexibility of requirements.
Strength ranking: 6;
Aircraft Certification: Flexibility of requirements;
Flight Standards: Clear well-defined requirements and processes.
Strength ranking: 7;
Aircraft Certification: Do not know/no basis to judge;
Flight Standards: None.
Note: In total, there were only five responses for Aircraft
Certification for "greatest strength," so the analysis for Aircraft
Certification is based only on responses for the second and third
greatest strengths.
2. What is the most significant problem with the certification and
approval processes?
Problem ranking: 1;
Aircraft Certification: Delays;
Flight Standards: Inconsistent interpretations.
Problem ranking: 2;
Aircraft Certification: Inconsistent interpretations;
Flight Standards: Delays.
Problem ranking: 3;
Aircraft Certification: Dispute resolution process problems;
Flight Standards: Dispute resolution process problems.
Problem ranking: 4;
Aircraft Certification: Lack of communication;
Flight Standards: Lack of communication.
Problem ranking: 5;
Aircraft Certification: Duplication of approvals;
Flight Standards: Do not know/no basis to judge.
Problem ranking: 6;
Aircraft Certification: Do not know/no basis to judge;
Flight Standards: Not following procedures.
Problem ranking: 7;
Aircraft Certification: Not following procedures;
Flight Standards: Duplication of approvals.
3. What leading factor has contributed to problems with the
certification and approval processes?
Leading factor in process problems: Lack of FAA resources;
Aircraft Certification: 2;
Flight Standards: 1.
Leading factor in process problems: FAA's prioritization system for
managing certifications and approvals;
Aircraft Certification: 1;
Flight Standards: 0.
Leading factor in process problems: FAA's rulemaking process and
development of guidance (e.g., amount of time required to develop or
change regulations, etc.);
Aircraft Certification: 4;
Flight Standards: 3.
Leading factor in process problems: Culture of FAA (e.g., stove-
piping, resistance to change, etc.);
Aircraft Certification: 5;
Flight Standards: 7.
Leading factor in process problems: Organizational structure of FAA
(e.g., decentralization, varying procedures among local offices, etc.);
Aircraft Certification: 4;
Flight Standards: 2.
Leading factor in process problems: Lack of adequate dispute
resolution mechanisms;
Aircraft Certification: 0;
Flight Standards: 1.
Leading factor in process problems: Lack of consequences for erroneous
interpretations;
Aircraft Certification: 1;
Flight Standards: 3.
Leading factor in process problems: Do not know/no basis to judge;
Aircraft Certification: 0;
Flight Standards: 1.
Total responses:
Aircraft Certification: 17;
Flight Standards: 18.
4. How often do serious problems occur each year with the
certification and approval processes?
Rate of problem occurrences: Less than 10 percent of the time;
Aircraft Certification: 17;
Flight Standards: 7.
Rate of problem occurrences: 11 to 20 percent of the time;
Aircraft Certification: 0;
Flight Standards: 3.
Rate of problem occurrences: 21 to 30 percent of the time;
Aircraft Certification: 1;
Flight Standards: 1.
Rate of problem occurrences: 31 to 40 percent of the time;
Aircraft Certification: 0;
Flight Standards: 1.
Rate of problem occurrences: 41 to 50 percent of the time;
Aircraft Certification: 0;
Flight Standards: 1.
Rate of problem occurrences: 51 to 60 percent of the time;
Aircraft Certification: 0;
Flight Standards: 1.
Rate of problem occurrences: 61 to 70 percent of the time;
Aircraft Certification: 0;
Flight Standards: 0.
Rate of problem occurrences: 71 to 80 percent of the time;
Aircraft Certification: 0;
Flight Standards: 0.
Rate of problem occurrences: More than 80 percent of the time;
Aircraft Certification: 0;
Flight Standards: 0.
Rate of problem occurrences: Do not know/no basis to judge;
Aircraft Certification: 1;
Flight Standards: 5.
Total responses:
Aircraft Certification: 19;
Flight Standards: 19.
5. Overall, how positive or negative do you think industry's
experience has been in obtaining certifications and approvals from
Aircraft Certification and Flight Standards?
Characterization of process experience: Very positive;
Aircraft Certification: 2;
Flight Standards: 0.
Characterization of process experience: Positive;
Aircraft Certification: 15;
Flight Standards: 9.
Characterization of process experience: Neither positive nor negative;
Aircraft Certification: 2;
Flight Standards: 4.
Characterization of process experience: Negative;
Aircraft Certification: 0;
Flight Standards: 3.
Characterization of process experience: Very negative;
Aircraft Certification: 0;
Flight Standards: 1.
Characterization of process experience: Do not know/no basis to judge;
Aircraft Certification: 0;
Flight Standards: 2.
Total responses:
Aircraft Certification: 19;
Flight Standards: 19.
6. How would you assess the overall impact of the certification and
approval processes on the safety of the national airspace system?
Overall impact of processes: Very positive;
Aircraft Certification: 9;
Flight Standards: 5.
Overall impact of processes: Positive;
Aircraft Certification: 9;
Flight Standards: 11.
Overall impact of processes: Neither positive nor negative;
Aircraft Certification: 1;
Flight Standards: 2.
Overall impact of processes: Negative;
Aircraft Certification: 0;
Flight Standards: 0.
Overall impact of processes: Very negative;
Aircraft Certification: 0;
Flight Standards: 0.
Overall impact of processes: Do not know/no basis to judge;
Aircraft Certification: 0;
Flight Standards: 1.
Total responses:
Aircraft Certification: 19;
Flight Standards: 19.
7. Overall, how would you characterize efforts to improve the
certification and approval processes?
Characterization of process improvement efforts: Very effective;
Aircraft Certification: 1;
Flight Standards: 0.
Characterization of process improvement efforts: Effective;
Aircraft Certification: 10;
Flight Standards: 5.
Characterization of process improvement efforts: Neither effective nor
ineffective;
Aircraft Certification: 4;
Flight Standards: 7.
Characterization of process improvement efforts: Ineffective;
Aircraft Certification: 2;
Flight Standards: 4.
Characterization of process improvement efforts: Very ineffective;
Aircraft Certification: 0;
Flight Standards: 0.
Characterization of process improvement efforts: Do not know/no basis
to judge;
Aircraft Certification: 2;
Flight Standards: 2.
Total responses:
Aircraft Certification: 19;
Flight Standards: 18.
8. Overall, how would you characterize efforts to prioritize
certifications and approvals?
Characterization of process prioritization efforts: Very effective;
Aircraft Certification: 3;
Flight Standards: 0.
Characterization of process prioritization efforts: Effective;
Aircraft Certification: 10;
Flight Standards: 5.
Characterization of process prioritization efforts: Neither effective
nor ineffective;
Aircraft Certification: 3;
Flight Standards: 1.
Characterization of process prioritization efforts: Ineffective;
Aircraft Certification: 1;
Flight Standards: 5.
Characterization of process prioritization efforts: Very ineffective;
Aircraft Certification: 0;
Flight Standards: 3.
Characterization of process prioritization efforts: Do not know/no
basis to judge;
Aircraft Certification: 2;
Flight Standards: 5.
Total responses:
Aircraft Certification: 19;
Flight Standards: 19.
9. Overall, how would you characterize efforts to improve dispute
resolution through the Consistency and Standardization Initiative
(CSI)?
Characterization of dispute resolution improvement efforts: Very
effective;
Aircraft Certification: 0;
Flight Standards: 0.
Characterization of dispute resolution improvement efforts: Effective;
Aircraft Certification: 7;
Flight Standards: 2.
Characterization of dispute resolution improvement efforts: Neither
effective nor ineffective;
Aircraft Certification: 5;
Flight Standards: 4.
Characterization of dispute resolution improvement efforts:
Ineffective;
Aircraft Certification: 3;
Flight Standards: 6.
Characterization of dispute resolution improvement efforts: Very
ineffective;
Aircraft Certification: 0;
Flight Standards: 1.
Characterization of dispute resolution improvement efforts: Do not
know/no basis to judge;
Aircraft Certification: 4;
Flight Standards: 4.
Total responses:
Aircraft Certification: 19;
Flight Standards: 17.
10. Regarding efforts to improve dispute resolution through CSI, what
is the key factor hindering the progress of efforts?
Key hindrance factor: Lack of FAA-wide buy-in for efforts;
Aircraft Certification: 4;
Flight Standards: 3.
Key hindrance factor: Lack of national level data for assessing
efforts;
Aircraft Certification: 3;
Flight Standards: 1.
Key hindrance factor: Industry fear of retribution;
Aircraft Certification: 5;
Flight Standards: 7.
Key hindrance factor: Other;
Aircraft Certification: 3;
Flight Standards: 2.
Key hindrance factor: Do not know/no basis to judge;
Aircraft Certification: 4;
Flight Standards: 5.
Total responses: ;
Aircraft Certification: 19;
Flight Standards: 18.
11. What should be done to mitigate the effects of this factor?
Potential mitigation action: FAA should establish support for efforts;
Aircraft Certification: 3;
Flight Standards: 5.
Potential mitigation action: FAA should improve data collection and
analysis related to efforts;
Aircraft Certification: 3;
Flight Standards: 0.
Potential mitigation action: Eliminate potential for retribution;
Aircraft Certification: 6;
Flight Standards: 7.
Potential mitigation action: Other;
Aircraft Certification: 3;
Flight Standards: 3.
Potential mitigation action: Do not believe efforts are ineffective;
Aircraft Certification: [A];
Flight Standards: 1.
Potential mitigation action: Do not know/no basis to judge;
Aircraft Certification: 4;
Flight Standards: 3.
Total responses:
Aircraft Certification: 19;
Flight Standards: 19.
[A] This response option was not available to the panelists.
12. What action will have the most positive impact on improving the
certification and approval processes?
Action ranking: 1;
Aircraft Certification: Culture shift/improve staff accountability;
Flight Standards: Culture shift/improve staff accountability.
Action ranking: 2;
Aircraft Certification: Expand use of designees/organization
designation authorizations (ODA);
Flight Standards: Permit universal acceptance.
Action ranking: 3;
Aircraft Certification: Industry improve knowledge and expectations;
Flight Standards: Develop improved/different dispute resolution.
Action ranking: 4;
Aircraft Certification: Permit universal acceptance;
Flight Standards: Increase standardization of requirements.
Action ranking: 5;
Aircraft Certification: Develop improved/different dispute resolution;
Flight Standards: Expand use of designees/organization designation
authorizations (ODA).
Action ranking: 6;
Aircraft Certification: Improve rulemaking process;
Flight Standards: Industry improve knowledge and expectations.
Action ranking: 7;
Aircraft Certification: Increase standardization of requirements;
Flight Standards: Do not know/no basis to judge.
Action ranking: 8;
Aircraft Certification: Do not know/no basis to judge;
Flight Standards: Improve rulemaking process.
Action ranking: 9;
Aircraft Certification: Other;
Flight Standards: Other.
Source: GAO.
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Gerald L. Dillingham, Ph.D., (202) 512-2834 or dillinghamg@gao.gov.
Staff Acknowledgments:
In addition to the contact named above, Teresa Spisak (Assistant
Director), Sharon Dyer, Bess Eisenstadt, Amy Frazier, Brandon Haller,
Dave Hooper, Michael Silver, and Pamela Vines made key contributions
to this report.
[End of section]
Footnotes:
[1] A type certificate is a design approval issued by FAA when the
applicant demonstrates that a product complies with the applicable
regulations. As defined by 14 C.F.R. § 21.41, the type certificate
includes the type design, the operating limitations, the type
certificate data sheet, the applicable regulations, and other
conditions or limitations prescribed by the FAA Administrator. The
type certificate is the foundation for other FAA approvals, including
production and airworthiness approvals.
[2] Individuals appointed by FAA to act on its behalf are known as
individual or organizational--if connected with an organization--
designees, and delegated organizations are authorized to act on FAA's
behalf under the organization designation authorization program. These
appointments are allowed under 14 C.F.R. § 183.45.
[3] We did not obtain data on the number of certification projects
that are not approved by Aircraft Certification, but FAA officials
noted that projects are sometimes not completed because the
applications are withdrawn.
[4] A minimum equipment list is a list of all equipment on an aircraft
type. It details which equipment FAA has determined may be inoperative
under certain operational conditions and still provide an acceptable
level of safety.
[5] We did not obtain data on the number of certification projects
that are not approved by Flight Standards, but FAA officials noted
that some projects are not completed because the applications are
withdrawn.
[6] Booz Allen & Hamilton, Challenge 2000: Recommendations for Future
Aviation Safety Regulation, prepared for FAA, Office of Policy,
Planning and International Aviation (McLean, VA: Apr. 19, 1996).
[7] RTCA Task Force 4, Final Report of RTCA Task Force 4
"Certification" (Washington, D.C.: Feb. 26, 1999).
[8] Independent Review Team Appointed by Secretary of Transportation
Mary E. Peters, Managing Risks in Civil Aviation: A Review of FAA's
Approach to Safety (Washington, D.C.: Sept. 2, 2008). The authors
described two phenomena that confirmed for them the existence of
conflicting regulatory ideologies: (1) a high number of enforcement
actions from a small portion of an inspection team within an office
and (2) the description of enforcement-oriented inspectors as "rogue
inspectors" by both industry stakeholders and FAA management while
their own observations of several inspectors described as rogues found
them to be articulate, sophisticated, and professional.
[9] A recent attempt to quantify the issue was a 2009 industry survey
conducted by the National Air Transportation Association, which
represents about 2,000 aviation businesses, including fixed-base
operators, charter providers, aircraft management companies,
maintenance and repair organizations, flight training companies, and
airline service companies. However, this survey, conducted from July
to September 2009, suffered from a low general response rate, nonitem
response issues (e.g., some questions had a large number of
nonresponses), and a universe (population) that was not clearly
defined or identified.
[10] A conformity approval is a determination by FAA that an aircraft
was manufactured in accordance with and conforms to its type
certificate and is safe for operation.
[11] Panelists responded to this question using the following list of
answer choices: less than 10% of the time, 11 to 20% of the time, 21
to 30% of the time, 31 to 40% of the time, 41 to 50% of the time, 51
to 60% of the time, 61 to 70% of the time, 71 to 80% of the time, more
than 80% of the time, and do not know/no basis to judge. See appendix
II for the entire list of questions and responses.
[12] Two industry officials did not mention their overall experiences
in dealing with FAA.
[13] The Office of Management and Budget's regulatory guidance
contained in Executive Order 12866 suggests that if regulations are to
be adopted as justified by benefit-cost analysis, performance-based
regulations are generally preferred.
[14] See, for example, C. Coglianese, J. Nash, and T. Olmstead,
"Performance-Based Regulation: Prospects and Limitations in Health,
Safety, and Environmental Protection," Administrative Law Review 55
(2003) and P. May, "Performance-Based Regulation and Regulatory
Regimes" (paper prepared for the Global Policy Summit on the Role of
Performance-Based Building Regulations in Addressing Societal
Expectations, International Policy, and Local Needs), National Academy
of Sciences (Washington, D.C.: November 2003).
[15] See appendix II for the entire list of questions and responses.
[16] Notice 8000.311, Certification Services Oversight Process for
Original Organizational Certifications, 10/27/2005.
[17] GAO, Next Generation Air Transportation System: FAA Faces
Challenges in Responding to Task Force Recommendations, [hyperlink,
http://www.gao.gov/products/GAO-10-188T] (Washington, D.C.: Oct. 28,
2009).
[18] GAO, Next Generation Air Transportation System: Challenges with
Partner Agency and FAA Coordination Continue, and Efforts to Integrate
Near-, Mid-, and Long-term Activities Are Ongoing, [hyperlink,
http://www.gao.gov/products/GAO-10-649T] (Washington, D.C.: Apr. 21,
2010).
[19] FAA, AVS Work Plan for NextGen 2010, March 2010.
[20] Aviation safety inspectors, like many federal employees, are
classified and assigned pay grades under the General Schedule. Within
Flight Standards, inspectors with responsibility for the oversight of
certificate holders are allocated a certain number of points for each
pay grade. Additionally, the entities overseen by these inspectors are
allocated a point value based on the complexity of the certificate or
operation, and the combined point value for each inspector's oversight
responsibilities must meet or exceed the points allocated for the
inspector's grade. However, not all of the inspectors' duties receive
points in this system, and inspectors are subject to a downgrade if
entities in their portfolio relocate or go out of business. For
example, a grade 14 principal inspector needs to maintain an oversight
workload representing at least 2,500 points and would be assigned a
manageable number of operators or agencies that would accrue a score
of at least 2,500 points. The inspector would be required to maintain
that point score to retain his or her pay grade. However,
certification work provides no credit toward the inspector's total
points and retention of his or her pay grade, and the inspector could
be downgraded if one of his or her operators relocated to the
jurisdiction of another field office.
[21] Aircraft Certification did not collect hiring and attrition
numbers in 2007, so the number of inspectors hired in 2007 is the
number hired minus the losses from attrition.
[22] GAO, Aviation Safety: FAA Needs to Strengthen the Management of
Its Designee Programs, [hyperlink,
http://www.gao.gov/products/GAO-05-40] (Washington, D.C.: Oct. 8,
2004).
[23] Senior FAA officials indicated that it is agency policy to teach
what the regulations say because the regulations take precedence over
guidance.
[24] ISO is a network of the national standards institutes of 159
countries that develops and publishes international standards. ISO
9001:2008 specifies requirements for a quality management system.
According to a senior FAA official, each Aviation Safety office
originally had its own ISO certification. Flight Standards
headquarters achieved ISO 9001 certification in 2004 and, in 2006, the
Office of Aviation Safety achieved ISO 9001 certification.
[25] The operations specifications for an air operator contain the
authorizations, limitations, and certain procedures under which each
kind of operation, if applicable, is to be conducted.
[26] CSI, which is part of QMS, was originally called the Customer
Service Initiative.
[27] Although several industry stakeholders mentioned that they feared
or experienced retribution from FAA staff for submitting CSI
complaints, they provided no evidence that confirmed such retribution
actually occurred.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO‘s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO‘s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
