Forecasting Return on Investment (ROI) for Naval

0 downloads 0 Views 279KB Size Report
Paper Number 2000-01-2094. Forecasting Return ... potential impact of maintenance error interventions. Collectively ... this reduction. Despite this trend, human factors are still ... mishap (e.g., an operator runs a forklift into an aircraft) or a latent ...
Paper Number 2000-01-2094

Forecasting Return on Investment (ROI) for Naval Aviation Maintenance Safety Initiatives CDR John Schmidt USN LT Dylan Schmorrow USN LtCol Robert Figlock USMC (Ret.) Naval Postgraduate School Copyright © 2000 Society of Automotive Engineers, Inc.

ABSTRACT

present in about 80% of these mishaps (Casey, 1999).

The Maintenance Extension of the Human Factors Analysis and Classification System (HFACS-ME) was employed to categorize errors present in 470 FY 90-97 Naval Aviation Maintenance Related Mishaps (MRMs). HFACS-ME identified common error types present in MRMs: maintenance supervision, crew coordination, maintainer error, and procedural violations. The data derived from classifying maintenance errors was used to develop mathematical models that were then employed to generate notional cost estimates associated with them. These models were then used to forecast the potential impact of maintenance error interventions. Collectively, the taxonomic analysis and model development served to identify common maintenance error forms, and consequently the optimal targets that have the most potential return on investment (ROI). An analysis of potential ROI tied to human error interventions revealed that reducing error as little as 10% can result in saving critical assets, lives, and millions of dollars each year.

NAVAL AVIATION HUMAN FACTORS QUALITY MANAGEMENT BOARD (HFQMB)

INTRODUCTION

Class AFMs/100,000 Flight Hours

Over the past several decades, Class A Flight Mishaps (FMs) that involve a loss of aircraft and/or life have declined (see Figure 1). Efforts to target aircrew error, a major causal factor, via engineering and administrative controls (e.g., low altitude warning systems & aircrew coordination training, respectively) have contributed to this reduction. Despite this trend, human factors are still

50

HFACS was developed at the Naval Safety Center to analyze human errors leading to Naval Aviation mishaps (Weigmann & Shappell, 1997). It integrates an array of models to depict factors that are precursors to accidents. HFACS was adapted (Schmidt, Schmorrow, & Hardee, 1998) to classify factors that lead to Maintenance Related Mishaps (MRMs). The “Maintenance Extension” consists of Supervisory, Maintainer, Working Conditions,

ORGANIZATIONAL CLIMATE Supervisory Conditions

42.0 Aircrew Acts

30

14.7

20

7.8

10 50'S

Maintainer Conditions

Working Conditions

Maintainer Acts

40

0

Given the acknowledged success of engineering and administrative controls in attacking aircrew error, human factors efforts are now expanding to cover maintenance. The Naval Aviation HFQMB formed an Aviation Maintenance Working Group (AMWG) to consider means to counter maintainer error. The AMWG adopted the same three prong approach used earlier by the HFQMB to address aircrew error (Nutwell & Sherman, 1997): mishap data analysis to identify hazards and risks, benchmarking to uncover best practices and find process improvements, and organizational climate assessments to evaluate safety posture.

60'S

70'S

80'S

Maintenance Conditions

MISHAP 3.7 2.3 90'S

Decade

Figure 1. Average Class A FM Rate by Decade

Figure 2. HFACS Maintenance Extension Model

Flight Aircraft Total Related Ground Class A 50 0 13 63 Class B 17 6 34 57 Class C 90 29 231 350 Total 157 35 278 470 Notably 51 people died in the Class A MRMs, 40 were attributed to FMs and 11 to AGMs (Aircraft Ground Mishaps). The financial costs for the 470 MRMs totaled over $800 million in FY98 dollars (see Table 3). The average cost for these mishaps was just over $2 million per MRM in FY98 dollars (see Table 4). Table 3 FY 90-97 MRM Costs (FY98$M) Flight

Table 1. HFACS-ME Taxonomy 2nd Order Organization Squadron

Maintainer Conditions

Medical

Crew Coordination Readiness

Working Conditions

Environment

Equipment

Workspace

Maintainer Acts

Error Violation

3rd Order Hazardous Operations Inadequate Documentation Inadequate Design Inadequate Supervision Inappropriate Operations Failed to Correct Problem Supervisory Violation Adverse Mental State Adverse Physical State Physical/Mental Limit Communication Assertiveness Adaptability/Flexibility Preparation/Training Qualification/Certification Violation Lighting/Light Exposure/Weather Environmental Hazard Damaged/Broken Unavailable Dated/Uncertified Confining Obstructed Inaccessible Attention/Memory Rule/Knowledge Skill/Proficiency Routine Infraction Exceptional

Class A Class B Class C Total

Flight Class A Class B Class C Overall

799 21 17 837

16,579 514 164 8,261

Flight Related 0 393 56 116

Aircraft Ground 260 362 43 91

Total 13,537 412 59 2,168

HFACS-ME ANALYSIS OF FY 90-97 MRMs Two Naval Officers, well versed in the HFACS-ME taxonomy and experienced in maintenance operations, reviewed the causal factors present in the 470 FY 90-97 MRM cases. The two judges independently reviewed each MRM case, and its respective HFACS-ME codes were entered into a spreadsheet for subsequent tabulation and analysis. Each MRM causal factor was given only one HFACS-ME code, and codes were only assigned to issues clearly identified as having had contributed to a mishap. Cohen’s kappa was calculated as a measure of inter-rater agreement. A kappa of .75 was obtained, indicating an overall “excellent” level of agreement. Codes that were disputed were discussed and resolved on the spot or after conferring with a third party. Each HFACS-ME category frequency count was

90 80

Pe rcentage of MRMs

Table 2 FY 90-97 Naval Aviation MRMs

Total

Table 4 FY 90-97 AVE MRM Costs (FY98$K)

FY 90-97 MRMs During FY 90-97 there were 470 MRMs (see Table 2). A total of 63 (13%) were major Class A MRMs involving the loss of an aircraft and/or life; of the remaining 407 MRMs, 265 (65%) involve ground and flightline activities.

796 8 6 810

Aircraft Ground 3 11 9 23

70

Er rors

Ma inta Act iner s

1st Order Supervisory Conditions

Flight Related 0 2 2 4

60 50 40 30 20 10

Wo Co rking ndi tion s Ma in Co tainer ndi tion s

In HFACS-ME “conditions” may be latent and can impact maintainer performance, contributing to an active failure --an unsafe Maintainer Act. Such failures can lead to a mishap (e.g., an operator runs a forklift into an aircraft) or a latent Maintenance Condition for aircrew to handle (e.g., an improperly rigged landing gear or over-torqued hydraulic line). Supervisory Conditions tied to poor design, procedures, etc. can also lead directly to an adverse Maintenance Condition. HFACS-ME provides for three orders of maintenance error: first, second, and third, reflecting decomposition from a molar to a micro perspective (see Table 1).

Flight

Su Co perviso ndi tion r y s

and Maintainer Acts (see Figure 2).

Violations

0

F igur e 3 . HF ACS-M E Pro file fo r FY90 -9 7 M RM s

totaled and respective percentages were calculated for subsequent comparison (see Figure 3). The judges' classifications determined the following profile of human errors in the MRMs: Supervisory Conditions- 60% of Naval Aviation reported a Squadron Condition, and 32% had an Organizational Condition (not shown).

data, was determined to be satisfactory in describing the MRMs. A variable Poisson process model was used to generate monthly hypothesized MRM means for the mishap data. Hypothetical expected number of MRMs per year were obtained by summing the hypothetical monthly means. The number of expected MRMs for FY 98-02 were calculated (see Table 5) Table 5. Expected MRMs for FY98-02

Working Conditions– 6% of Naval Aviation MRMs reported an Environment, Equipment, or Workspace Condition. Maintainer Condition - 22% of Naval Aviation MRMs reported a Medical, CRM, or Readiness Condition. Maintainer Acts - 79% of Naval Aviation MRMs reported an Error, whereas 39% had a Violation. Clearly, latent conditions in the form of Supervisory, Maintainer, and Working Conditions are present that can impact maintainers in performing their jobs. Furthermore, it is apparent that some form of intervention must be used to control these conditions and prevent errors and violations. Consequently, it becomes imperative to prioritize areas for intervention based on which efforts are likely to have the best return on investment (ROI).

Mishap Classification Flight Flight-Related Aircraft-Ground Class-A Class-B Class-C Composite

Unfortunately, most traditional approaches to calculating ROI depend on implementing intervention strategies and monitoring associated change (Phillips, 1999). This requires an effort not only be adopted, but also fully implemented, researched, and supported before a determination of its worth can be made. It is contended that through forecasting processes estimates of potential ROI can be made and used to prioritize and select interventions. STOCHASTIC MODELING OF FY 90-97 MRMs Model fitting was used to reveal the underlying arrival process for MRMs (Schmorrow, 1998). A variable Poisson process model, a method to generate an estimate based on a function fitted to historical mishap

FY99

FY00

FY01

FY02

11.6 1.8 20.2 4.1 5.3 20.0 33.5

10.4 1.5 18.1 3.6 5.0 17.0 30.0

9.4 1.3 16.2 3.2 4.8 14.4 26.8

8.5 1.1 14.6 2.8 4.5 12.2 24.0

7.7 0.9 13.0 2.4 4.3 10.3 21.5

Expected costs of MRMs for FY98 and the five-year period FY98-02 were calculated (see Table 6). Costs are assumed to be independent and identically distributed. The composite dollar value shown is an average of the cost total for mishap type, class, and frequency. Table 6. Expected MRM Costs in FY98$M Mishap Classification Flight Flight-Related Aircraft-Ground Class-A Class-B Class-C Composite

INTERVENTION STRATEGY ROI CALCULATION Presently, several intervention measures have been identified through benchmarking of best industry practices. Two have been developed for implementation: Maintenance Operational Risk Management (MORM) and Groundcrew Coordination Training (GCT). However, Fleet-wide dissemination of MORM and GCT is not without cost, and in times of declining budgets and manpower shortages the notion of adding another training program or administrative requirement is not exactly palatable. Demonstrating the potential ROI from engaging in such efforts can go a long way in convincing decision-makers to buy into these programs.

FY98

FY 98 Expected Costs 95.6 .2 1.9 55.3 .2 1.2 72.6

FY 98 - FY 02 Expected Costs 394.1 .8 7.5 217.0 9.9 4.4 294.4

Cost savings were based on the expected number of future mishaps, associate mishap costs, and likelihood a human error played a role in the expected mishap. Cost saving estimates were calculated based on a reduction in occurrence of 10%, 20%, and 30% (See Table 7). Table 7. Potential Cost Savings (FM98$M) %

Years

Supervisory

CRM

Error

Violation

10

1

4.4

0.9

6.0

2.8

5

18.1

3.5

24.8

11.3

1

8.8

1.7

12.1

5.5

5

36.1

7.0

49.6

22.6

1

13.2

2.6

18.1

8.3

5

54.4

10.5

74.3

33.9

20

30

CONCLUSIONS

HFACS-ME was effective in capturing the latent conditions and active failures present in MRMs. The insights gained provide a solid perspective to identify potential intervention strategies. The taxonomic analysis indicated that both MORM and GCT are well suited to combat most common maintenance error forms, and subsequently, modeling and associated forecasting revealed that reducing error as little as 10% can result in a significant savings of lives and resources.

REFERENCES Casey, R. (1999). School of Aviation Safety Brief. Monterey,CA: Naval Postgraduate School. Nutwell, R. & Sherman, K. (1997). Changing the way we operate, Washington, DC: Naval Aviation News. 79(3), 12-16 Phillips, J. (1999). HRD trends wordwide. Houston, TX: Gulf Publishing Co. Schmidt, J., Schmorrow, D., & Hardee, M. (1998). A preliminary human factors analysis of Naval Aviation maintenance related mishaps. SAE AEMR Conference Proceedings. Long Beach, CA. Schmorrow, D. (1998). A human error analysis and model of Naval Aviation maintenance related mishaps. Monterey, CA: Naval Postgraduate School Weigmann, D. & Shappell, S. (1997). Human factors analysis of postaccident data: Applying theoretical taxonomies of human error. The International Journal of Aviation Psychology, 7(1), 67-81.