Skip to Main Content
FlightPushFLIGHTPUSH
NTSB Safety Recommendations

Learn

Dashboard My Courses Mastery Map Practice Flight Bag

Resources

CFR & AIM FAA Orders Handbooks & Guides Certification Standards Pilot/Controller Glossary Advisory Circulars Legal Interpretations
Safety Center
Articles Trust Center
Help Center FAQ Status
  1. Safety Center
  2. NTSB Safety Recommendations
  3. NTSB A-06-038
Loading
flightpush.co home

NTSB · Safety Recommendation

Safety Recommendation A-06-038

Recommendation

TO THE FEDERAL AVIATION ADMINISTRATION: Adopt Society of Automotive Engineers ARP5150 into 14 Code of Federal Regulations Parts 21, 25, 33, and 121 to require a program for the monitoring and ongoing assessment of safety-critical systems throughout the life cycle of the airplane. Safety-critical systems will be identified as a result of A-06-36. Once in place, use this program to validate that the underlying assumptions made during design and type certification about safety-critical systems are consistent with operational experience, lessons learned, and new knowledge.

Safety Issue and Context

Certification of systems that are critical to safety of flight has been an issue in several National Transportation Safety Board accident investigations of transport-category airplanes. In 1999, the Safety Board expressed concern about the Federal Aviation Administration's (FAA) certification process during its investigation of the rudder actuator in USAir flight 427. In 2000, the Board suggested the need for a directed examination of the certification process in the investigation of the center wing fuel tank in TWA flight 800.

Subsequent investigations of the horizontal stabilizer jackscrew in Alaska Airlines flight 26 1 and the rudder system in American Airlines flight 5874 also raised questions about the certification process used by the FAA to determine compliance with airworthiness standards. These four accidents resulted in 715 fatalities and accounted for 60 percent of the air carrier fatalities that occurred from 1994-2001 . In each of these accidents, the Board identified a safety-critical system that suffered a catastrophic failure, and issued certification-related recommendations to address system-specific design, operational, or maintenance issue.

Response History

December 9, 2014 · NTSB Correspondence · FAA

We are pleased that, in response to this recommendation, you developed the monitor safety analyze data (MSAD) Order, which you published on March 12, 2010, and which became effective on January 31, 2011. The MSAD Order describes the processes and procedures that you will use to identify and analyze the data necessary for effective continued operational safety oversight. On November 4, 2011, you issued FAA Policy Statement, PS-ANM-25-05, “Risk Assessment Methodology for Transport Category Airplanes,” which describes how you evaluate the fleet risk for both systems and structures.

The issuance of these documents and implementation of the practices they describe satisfies Safety Recommendation A 06 38, which is classified CLOSED—ACCEPTABLE ACTION.

October 28, 2013 · Recipient Response · FAA

-From Michael P. Huerta, Administrator: As mentioned in previous correspondence, Monitor Safety Analyze Data (MSAD) is the overarching process by which we will monitor and assess the safety critical systems identified as a result of implementing Safety Recommendation A-06-36. MSAD works in coordination with processes and programs to monitor the operational fleet. MSAD provides continued operational safety (COS) management guidance that applies the principles of SAE ARP5150 to airplane systems and structures. It ensures informed decision making by requiring a structured risk analysis process for potential safety issues and effective corrective actions through the use of root cause analysis.

Since our previous letter, the FAA has made progress on COS risk assessment guidelines and processes supporting MSAD, including:

• FAA Policy Statement: PS-ANM-25-05, Risk Assessment Methodology for Transport Category Airplanes, which evaluates the fleet risk for both systems and structures. We issued the policy statement on November 4, 20 11 , and it can be found at the following Web site: http://rgl.faa.gov/Regulatory_and_Guidance_Library/rgPol

Read Recommendation

Recommendation Facts

Status
Closed - Acceptable Action
Issued
May 17, 2006
Priority
CLASS II
Recipient
FAA
Adopted
May 17, 2006
Closed
December 9, 2014
Related Investigation
80707
Related Report
SR-06-02
icy.nsf/0/4E5A E8707164674A8625795 1 0061 F96B?OpenDocument&Highlight=ps-anm-25-05 • AC 39-8, Continued Airwot1hiness Assessments of Powerplant and Auxiliary Power Unit Installations of Transport Category Airplanes (CAAM), which provides the guidance requested by this safety recommendation.

The MSAD activity will strengthen the event monitoring and trending aspects of AC 39-8 for engine and auxiliary power unit instal1ation safety issues take into account the way airplanes are operated in service and use a data-driven approach to evaluate potential hazards. Since MSAD uses a standard taxonomy for organizing data, it can quickly identify emerging safety trends which can then be communicated to the appropriate parties. The implementation of MSAD, supplemented by the guidance of AC 39-8, allows the FAA to identify trends through data analysis and take appropriate action on aviation safety issues.

While the Board has expressed concerns about appropriate compliance timelines in AC 39-8, the FAA maintains that AC 39-8 sets suitable timelines. AC 39-8 is based on real-world data, collected from 19 years of engine-related, in-service events. The approach of AC 39-8 is to control risk to a level appropriate to the consequence of the event. Our methodology manages the risk of serious or severe consequences to published levels, although it does not completely eliminate the risk or the possibility of additional failures.

The Board has also expressed concerns about CAAM's risk calculation process. Through incorporation of experience gained and lessons learned from an unsafe condition, CAAM provides for continuous improvement in the effectiveness of the continued airworthiness assessment process for current and new products. The CAAM process is dynamic by design and risk calculation can change based on new information.

Through issuance of the overarching process of MSAD and the FAA Transport Airplane Directorate's risk assessment policy, I believe that the FAA has met the intent of this recommendation and consider our actions complete.

September 14, 2011 · NTSB Correspondence · FAA

In response to this recommendation, the FAA developed for its Aircraft Certification Service the monitor safety analyze data (MSAD) Order, which was published on March 12, 2010, and became effective on January 31, 2011. The MSAD Order describes the processes and procedures that the FAA will use to identify and analyze the data necessary for effective continued operational safety (COS) oversight. The FAA cited advisory circular (AC) 39-8 as an example of the detailed COS risk guidelines and processes supporting MSAD.

AC 39-8, “Continued Airworthiness Assessments of Powerplant and Auxiliary Power Unit Installations of Transport Category Airplanes (CAAM),” is used by the FAA and engine manufacturers for monitoring and assessing the continued airworthiness of engines installed in transport category airplanes throughout the product life cycle. On January 3, 2011, the FAA published for public comment a policy concerning the safety management process that it will use to evaluate for transport airplanes the fleet risk for both systems and structures.

In Safety Recommendation A-06-62, stated below, the NTSB expressed concern with the application of AC 39-8 to provide timelines to appropriately respond to identified safety problems with engines used in transport category aircraft. This recommendation was issued to the FAA as a result of the NTSB’s investigation of the June 2, 2006, uncontained failure of the high pressure turbine (HPT) stage 1 disk of an American Airlines Boeing 767-223(ER) General Electric CF6-80 engine during a high-power ground run for maintenance at Los Angeles International Airport, Los Angeles, California. A-06-62

Revise the engine-related airworthiness directive process to ensure that the compliance timelines are appropriately established.

The FAA initially responded that it believed the current version of AC 39-8 set appropriate compliance guidelines. On May 11, 2007, we asked the FAA to reconsider its position, citing our findings discussed in the letter that had transmitted Safety Recommendation A-06-62 to the FAA: that prior General Electric CF6-80 HPT stage 1 disk failures and crack propagation findings should have resulted in more aggressive inspection schedules than those mandated by the FAA. The FAA replied that it had reconsidered its position but continued to believe that the AC 39-8 process set appropriate compliance times.

The FAA further stated that the cracks cited by the NTSB were consistent with the risk analysis predictions made using the AC 39-8 methodology. In our September 23, 2009, letter, we continued to disagree that the methodology in AC 39-8 establishes appropriate compliance timelines and indicated that we believed the data in the letter that transmitted this recommendation showed that a shorter compliance schedule was needed. Consequently, Safety Recommendation A-06-62 was classified “Closed—Unacceptable Action.”

The NTSB agrees with the MSAD approach of using in-service data (that is, failures and non-failures) to predict the likelihood of occurrence of future events; however, we disagree with some of the assumptions used in the current CAAM process to assess the safety risk associated with a future occurrence. The CAAM process does not automatically assume that an engine uncontainment will result in a catastrophic event (and we agree that an uncontainment is not always a catastrophic event), but it applies a probability to the likelihood of a catastrophic chain of events occurring once an uncontainment has happened.

Our concern is that this method may not fully take into account the many variables associated with the airplane’s operation and that, as a result, the calculated risk assessment may underestimate the true risk of an accident outcome should another engine uncontainment occur before a permanent corrective action is taken.

The implementation of the MSAD Order and the development of a proposed policy that will evaluate for transport airplanes the fleet risk for both systems and structures constitute positive responses to this recommendation. We ask that, when developing and implementing these new systems, the FAA also consider the concerns we have previously expressed about the CAAM process.

The NTSB believes that MSAD will also address Safety Recommendation A-06-38 if, consistent with ARP 5150, MSAD takes into account the way airplanes are actually being operated in service and if it uses in-service data to evaluate previously unrecognized risks associated with the human/airplane system interaction in addressing Safety Recommendation A-06-37. For example, Safety Recommendations A-11-68 and -69 were recently issued to the FAA; they concern bounced landing recovery training for pilots of Boeing MD-11 airplanes.

In the letter issuing these recommendations, we noted that, since these airplanes entered service in 1990, there have been at least 14 MD-11 bounced landing events of such severity that the aircraft sustained substantial damage, including 4 events that resulted in complete hull losses. Of particular concern is that seven of these events have taken place over the last 2 years. We believe this is precisely the type of event that a system based on the MSAD Order should notice, and that the FAA should consider in determining what actions to take.

Accordingly, we ask the FAA to describe whether and how the new systems that the FAA is implementing have detected this problem, and what actions the FAA is considering as a result. Pending implementation of the policy published on January 3, 2011, concerning the process that it will use to evaluate for transport airplanes the fleet risk for both systems and structures, and information concerning how such a system would detect and respond to problems with the MD 11, Safety Recommendation A-06-38 is classified OPEN—ACCEPTABLE RESPONSE.

May 6, 2011 · Recipient Response · FAA

CC# 201100201: - From J. Randolph Babbitt, Administrator: Since our August 17, 2006 response, we have implemented plans to address this safety recommendation through the adoption of a standardized process for identifying the data necessary for effective continued operational safety (COS) oversight for each product. Through the implementation of Safety Management Systems, we expect to increase our ability to provide an ongoing assessment of safety-critical systems throughout the life cycle of the airplane. Within Aircraft Certification, Monitor Safety Analyze Data (MSAD) is the overarching process by which we will monitor and assess safety critical systems.

MSAD works in coordination with processes and programs to monitor the operational fleet. The MSAD Order (enclosure 3) was published on March 12, 2010 and became effective January 31, 2011. Goals for MSAD maturation are:

• Facilitation of standardized COS management by a process to quantify risks and apply resources to the largest contributors; • Inclusion of a standard operating procedure for monitoring airworthiness directive implementation; • Increased utilization of "root cause" process for more in-depth study of fundamental causes, application of corrective actions at the level of the most fundamental identifiable cause(s), and follow on investigation of the impacts of any corrective actions; and • Allow periodic cross issue analyses across multiple fleets to identify common fundamental causes or trends. Examples of detailed COS risk guidelines and processes supporting MSAD:

• AC 39-8, Continued Airworthiness Assessments of Powerplant and Auxiliary Power Unit Installations of Transport Category Airplanes (CAAM) provides the guidance required by this safety recommendation (enclosure 4). CAAM is used by the FAA and by engine manufacturers for n10nitoring and assessing the continued airworthiness of engines installed in transport category airplanes throughout the product life cycle. The MSAD activity will strengthen the event monitoring and trending aspects of AC 39-8 for engine and auxiliary power unit installation safety issues, and develop COS management guidance to apply to principles of SAE ARP5150 to airplane systems and structures;

• The Transport Airplane Directorate's continued operations safety managen1ent process has been developed to evaluate the fleet risk for both systems and structures. The policy was published for public comment on January 3, 2011.

The FAA believes that the intent of this safety recommendation will be satisfied for transport airplanes through the overarching process of MSAD, by guidance in AC 39-8 and the Transport Airplane Directorate's policy for managing COS.

I will keep the Board informed of the FAA's progress on these safety recommendations, and I will provide a further response by May 31, 2012.

October 27, 2009 · NTSB Correspondence · FAA

This recommendation was reiterated with the issuance of the A-09-113 through- 128 greensheet. This greensheet references a June 4, 2007 accident of a Cessna Citation 550, N550BP, which impacted Lake Michigan shortly after departure from General Mitchell International Airport, Milwaukee, Wisconsin (MKE). In its 2006 study of the aircraft certification process, the NTSB made two recommendations to the FAA concerning human/aircraft interaction issues in the certification of aircraft. Safety Recommendation A-06-37 asked the FAA to ?

amend the advisory materials associated with 14 CFR] 25.1309 to include consideration of structural failures and 7 human/airplane system interaction failures in the assessment of safety-critical systems.

October 27, 2009 · NTSB Correspondence · FAA

This recommendation was reiterated with the issuance of the A-09-113 through- 128 greensheet. This greensheet references a June 4, 2007 accident of a Cessna Citation 550, N550BP, which impacted Lake Michigan shortly after departure from General Mitchell International Airport, Milwaukee, Wisconsin (MKE). In its 2006 study of the aircraft certification process, the NTSB made two recommendations to the FAA concerning human/aircraft interaction issues in the certification of aircraft. Safety Recommendation A-06-37 asked the FAA to ?

amend the advisory materials associated with 14 CFR] 25.1309 to include consideration of structural failures and 7 human/airplane system interaction failures in the assessment of safety-critical systems.? Safety Recommendation A-06-38 asked the FAA to require a program for the monitoring and ongoing assessment of safety-critical systems throughout the life cycle of the airplane. Once in place, use this program to validate that the underlying assumptions made during design and type certification about safety-critical systems are consistent with operational experience, lessons learned, and new knowledge.

In response to these recommendations, the FAA indicated that it was incorporating more standardized consideration of human factors into new certification projects and planned to develop new regulations and advisory materials and a human factors design guide. Additionally, the FAA stated that it intended to formalize a process for monitoring and assessing safety critical systems throughout the life cycle of an airplane. Pending the FAA’s related actions, the NTSB classified Safety Recommendations A-06-37 and -38, "Open -- Acceptable Response" on September 12, 2007.

However, although this investigation revealed that many Citation pilots were aware of some of the human factors issues associated with the Citation cockpit design (such as the potential for inadvertently pressing the wrong button when engaging the autopilot or yaw damper), the NTSB is not aware of any plans by Cessna or the FAA to change the design to address these issues. The FAA indicated to the NTSB that it was developing guidance on human factors issues in aircraft for use by its certification staff; however, the NTSB is not aware that this new guidance has been developed and distributed, nor that it would address the many issues identified in this accident.

Therefore, the NTSB concludes that the circumstances of this accident demonstrate the importance of a program for the FAA to monitor and conduct ongoing assessments of safety-critical systems throughout the life cycle of an airplane; the FAA did not perform this task adequately for the Cessna Citation. Because the numerous human factors issues identified in the accident airplane’s cockpit design may have been corrected if current human factors principles had been applied after certification, the NTSB reiterates Safety Recommendations A-06-37 and 38 and classifies both OPEN -- UNACCEPTABLE RESPONSE.

September 12, 2007 · NTSB Correspondence · FAA

During the March 28, 2007 meeting, the FAA indicated that it planned to meet the intent of this recommendation by a structured risk assessment, without regard to a specific practice, such as ARP 5150. The FAA further indicated that it may proceed with rulemaking to require an SMS. The Safety Board notes that the intent of this recommendation was to formalize a process for monitoring and assessing safety-critical systems throughout the life cycle of an airplane, and ARP 5150 was suggested as an approach that already incorporates many of the features the Board regards as critical.

If the FAA requires an SMS that includes monitoring and assessing safety-critical systems throughout the life cycle of an airplane without specific regard to ARP 5150, this may constitute an acceptable alternative.

Pending the FAA’s adding requirements in Parts 21, 25, 33, and 121 for programs to monitor and assess safety-critical systems throughout the life cycle of airplanes, Safety Recommendation A-06-38 is classified OPEN -- ACCEPTABLE ALTERNATE RESPONSE.

August 17, 2006 · Recipient Response · FAA

Letter Mail Controlled 8/29/2006 8:26:32 AM MC# 2060429: - From Marion C. Blakey, Administrator: 8/17/06 The FAA agrees with the intent of these safety recommendations, and we have been working on several initiatives that we feel address the issues underlying them. For example, based on the outcome of our Certification Process Study (CPS), we launched a number of initiatives, including an initiative called Key Safety Information, which is closely related to the safety critical systems concept. Other initiatives address maintenance human factors as well as oversight of repairs, alterations, and maintenance.

In response to these recommendations, we have formed an internal team to evaluate current FAA initiatives. This team will assess the CPS initiatives, FAA’s research program, and rulemaking projects to determine relevance to the recommendations. The team will then identify any shortfalls and propose any refinements to our strategy, if necessary. We plan to complete that assessment by September 2006. We would like to meet with your staff after that time to discuss the results. A member of my staff will contact you to establish a meeting date. After those discussions, we will revise our plans for addressing the recommendations and provide you with a formal status update by January 31, 2007.

I will keep the Board informed of the FAA’s progress on these safety recommendations. Minutes from 3/28/07 Meeting with FAA on Recommendations related to Certification The FAA indicated that it planned to meet the intent of this recommendation by a structured risk assessment, without regard to a specific practice, such as ARP 5150. The FAA hopes that this is acceptable, without being restricted to only ARP 5150. The Safety Board indicated that the intent of this recommendation was for there to be a formalized process and not ad-hoc. The FAA indicated that they may proceed with rulemaking to require a SMS.

If the FAA requires such an SMS, the Board believes that alternative approaches may be acceptable, and that ARP5150 was suggested as an approach that already incorporates many of the features that the Board regards as critical.