May 20, 2018 - systems engineers consider it to be a risky process (Frank, Harel, & Orion, 2014). The magnitude of the risk and cost to fix issues increases if ...
Methodology to Evaluate Proposed Leading Indicators of Space System Performance Degradation Due to Contamination
by Elaine Ellen Seasly
B.S. in Chemical Engineering, May 2000, University of Arizona M.S. in Patent Law, May 2014, University of Notre Dame
A Praxis submitted to The Faculty of The School of Engineering and Applied Science of The George Washington University in partial fulfillment of the requirements for the degree of Doctor of Engineering
May 20, 2018
Praxis directed by Muhammad F. Islam Professorial Lecturer of Engineering Management and Systems Engineering
ProQuest Number: 10751853
All rights reserved
INFORMATION TO ALL USERS The quality of this reproduction is dependent upon the quality of the copy submitted. In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if material had to be removed, a note will indicate the deletion.
ProQuest 10751853 Published by ProQuest LLC (2018 ). Copyright of the Dissertation is held by the Author.
All rights reserved. This work is protected against unauthorized copying under Title 17, United States Code Microform Edition © ProQuest LLC.
ProQuest LLC. 789 East Eisenhower Parkway P.O. Box 1346 Ann Arbor, MI 48106 - 1346
The School of Engineering and Applied Science of The George Washington University certifies that Elaine Ellen Seasly has passed the Final Examination for the degree of Doctor of Engineering as of March 13, 2018. This is the final and approved form of the praxis.
Methodology to Evaluate Proposed Leading Indicators of Space System Performance Degradation Due to Contamination
Elaine Ellen Seasly
Praxis Research Committee: Muhammad F. Islam, Professorial Lecturer of Engineering Management and Systems Engineering, Praxis Director Amirhossein Etemadi, Assistant Professor of Engineering and Applied Science, Committee Member Ebrahim Malalla, Visiting Assistant Professor of Engineering and Applied Science, Committee Member D. Brooke Hatfield, President, Raven Research, Committee Member
ii
© Copyright 2018 by Elaine Seasly All rights reserved
iii
Acknowledgments I would like to thank my advisors, Dr. Muhammad Islam and Dr. Brooke Hatfield for their valued support during the praxis review and defense process, and Dr. Amirhossein Etemadi and Dr. Ebrahim Malalla for their detailed review and valuable feedback. I would also like to thank Dr. Jason Dever and Dr. Steven Stuban for providing guidance and support throughout the research process. All of the aforementioned advisors ensured I stayed on track and developed a body of work that went from an idea, to an experiment, to a research project, and finally, a completed praxis. I am thankful and appreciative of their time spent in assisting me from the very beginning all the way to the end. I would also like to thank several colleagues that provided assistance throughout this research: Gugu Rutherford at NASA Langley for identifying and bringing the idea of portable Raman spectroscopy as an analysis tool for contamination inspections forward for development, Mark Thornblom at NASA Langley for encouraging the exploration and development of new ideas, Walter Wrigglesworth III at Wyzkyds LLC for software programming expertise and support throughout the analysis process, Joyce Corriere at Virginia Space Grant Consortium for her encouragement and checking up on me when I was working extremely late and long hours in the lab, and members of the NASA Langley Systems Integration and Test Branch for their support. I owe thanks to my parents, Patricia and Larry, for their encouragement and support throughout this entire effort. Finally, I must thank Arthur Martin for his genuine caring and always positive encouragement. He not only listened to and encouraged my ideas, but kept me grounded and from going over the edge. I am very fortunate to have such
iv
wonderful support in my life.
v
Abstract Methodology to Evaluate Proposed Leading Indicators of Space System Performance Degradation Due to Contamination Leading indicators can be utilized to monitor a system and detect if a risk is present or increasing over time during key development phases such as integration and test. However, no leading indicator is perfect, and each contains inherent holes that can miss signals of risk. While the Swiss cheese model is a well-known framework for conceptualizing the propagation of risks through holes in system defenses, research is lacking on characterizing these holes. There are many choices for leading indicators, and to select an appropriate indicator for a system, engineering managers need to know how well the indicator will detect a signal of risk and what it can miss. A methodology was developed to quantify holes in proposed leading indicator methods and determine the impact to system performance if the methods miss detecting the risk. The methodology was executed through a case study that empirically evaluated two different techniques for detecting and monitoring molecular contamination risk to space system hardware performance during systems integration: visual inspections and portable Raman spectroscopy. Performance model results showed the impact the presence of a contaminant film had on space system surfaces, and how each indicator method missing the detection of a film could impact the system. Results from the methodology provide an understanding of the limitations in the risk detection and monitoring techniques of leading indicators to aid engineering managers in effectively selecting, deploying, and improving these techniques. vi
Table of Contents Acknowledgments........................................................................................................... iv Abstract ........................................................................................................................... vi List of Figures .................................................................................................................. x List of Tables................................................................................................................. xiii List of Acronyms ......................................................................................................... xiv Chapter 1: Introduction ................................................................................... 1 1.1
Background and Problem Definition..................................................................... 2
1.2
Purpose of the Study ............................................................................................. 4
1.3
Significance of the Study ...................................................................................... 5
1.4
Scope & Limitations.............................................................................................. 8
1.5
Organization of the Document .............................................................................. 8 Chapter 2: Literature Review........................................................................ 10
2.1
Part I: Applicable Engineering Management Topics .......................................... 10
2.1.1 Systems Integration ............................................................................................. 10 2.1.2 Risk Management ................................................................................................ 14 2.1.2.1
Traditional 5x5 Risk Matrix........................................................................ 14
2.1.2.2
New Perspectives in Risk Management ...................................................... 15
2.1.2.3
Detecting Signals and Warnings of Risk .................................................... 16
2.1.2.4
Missing Signals and Warnings of Risk ....................................................... 17
2.1.3 Leading Indicators ............................................................................................... 18 2.2
Part II: Case Study of Spacecraft Contamination................................................ 20
2.2.1 Contamination as a Risk...................................................................................... 23
vii
2.2.2 Examples of Past Spacecraft Issues due to Molecular Contamination ............... 26 2.2.3 Molecular Contamination Requirements ............................................................ 29 2.2.4 Current Indicator Method: Visual Inspections .................................................... 32 2.2.5 Proposed Indicator Method: Portable Raman Spectroscopy ............................... 34 2.2.6 The Chemistry of Common Spacecraft Molecular Contaminants ...................... 36 2.2.7 Spacecraft Surface Performance Modeling: STACK Program ........................... 37 Chapter 3: Methods ......................................................................................... 39 3.1
Proposed Evaluation Methodology ..................................................................... 39
3.2
Hypothesis ........................................................................................................... 44
3.3
Evaluation Requirements .................................................................................... 47
3.4
Experimental Materials and Sample Preparation ................................................ 47
3.5
Experimental Measurements ............................................................................... 50
3.5.1 Visual Inspections ............................................................................................... 51 3.5.2 Portable Raman Spectroscopy............................................................................. 53 3.6
Data Collection & Analysis ................................................................................ 55
3.6.1 Visual Inspections ............................................................................................... 56 3.6.2 Portable Raman Spectroscopy............................................................................. 58 3.6.3 Data Transformation and ANOVA Assumptions ............................................... 63 3.7
Performance Modeling of Molecular Contamination Effects ............................. 66 Chapter 4: Results ........................................................................................... 68
4.1
Experimental Results........................................................................................... 68
4.2
Hypothesis Test Results ...................................................................................... 70
4.3
Performance Model Results ................................................................................ 71
viii
Chapter 5: Discussion, Conclusions & Recommendations ....................... 77 5.1
Discussion ........................................................................................................... 77
5.2
Conclusions ......................................................................................................... 79
5.3
Implications of the Study for the Spacecraft Industry......................................... 79
5.4
Recommendations for Future Research .............................................................. 81
References ..................................................................................................................... 85 Spacecraft Performance Model Results .............................................. 94
ix
List of Figures Figure 1-1: Research Question........................................................................................... 4 Figure 1-2: Focus of this Research .................................................................................... 6 Figure 2-1: Systems Engineering V-Model ..................................................................... 11 Figure 2-2: Parts Problems found during Product Development Phases (GAO, 2011)... 12 Figure 2-3: Cost to Extract Defects during Lifecycle Cost over Time (INCOSE, 2010) 13 Figure 2-4: 5x5 Risk Matrix (NASA) .............................................................................. 15 Figure 2-5: Reason’s Swiss Cheese Model (Reason, 1997) ............................................ 18 Figure 2-6: Particulate Contamination on a Thermal Vacuum Chamber Window (NASA/Elaine Seasly) ............................................................................................... 21 Figure 2-7: Molecular Contamination Film on a Cryogenically Cooled Surface (NASA/Elaine Seasly) ............................................................................................... 22 Figure 2-8: UV Reflectance of Flight and Spare Hubble Pickoff Mirrors (Tveekrem et al., 1996) .................................................................................................................... 27 Figure 2-9: Darkening of Expose-R Windows after Space Exposure for 22 Months (Demets et al., 2014) .................................................................................................. 29 Figure 2-10: Visual Inspection of a James Webb Space Telescope Mirror Segment (NASA/Chris Gunn) .................................................................................................. 33 Figure 2-11: Raman Spectra of a Chemical Warfare Agent at Different Excitation Wavelengths (Guicheteau et al., 2011) ...................................................................... 36 Figure 3-1: Holes in Risk Indicators Missing Signals of Risk ........................................ 40 Figure 3-2: Proposed Indicator Evaluation Methodology with Filter Metaphor and Case Study ................................................................................................................. 43
x
Figure 3-3: Evaluation of Independent Variable Effects on Dependent Variables for Equal Comparison of Indicator Methods ................................................................... 46 Figure 3-4: Step 1 of the Proposed Indicator Evaluation Methodology .......................... 47 Figure 3-5: Steps 2 and 3 of the Proposed Indicator Evaluation Methodology ............... 48 Figure 3-6: Steps 4 and 5 of the Proposed Indicator Evaluation Methodology ............... 51 Figure 3-7: Contamination Samples Prepared for Visual Inspection Experiments (NASA/Elaine Seasly) ............................................................................................... 53 Figure 3-8: Experimental Setup of Portable Raman Spectrometer System (NASA/Elaine Seasly) ............................................................................................... 54 Figure 3-9: Raman Spectra of Acetaminophen Tablet Standard ..................................... 55 Figure 3-10: Steps 6, 7 and 8 of the Proposed Indicator Evaluation Methodology ......... 56 Figure 3-11: Signal Processing of Raman Spectra of Hydrocarbon Droplet on Foil Substrate. (a) Pre-processed Raman Spectra. (b) Post-processed Raman Spectra. ... 60 Figure 3-12: Post-processed Raman Spectra for Contaminant Droplets on Foil Substrate. (a) Silicone (b) Hydrocarbon (c) Fluorocarbon (d) Glycol (e) Ester. ....... 61 Figure 3-13: Johnson Transformation of the Visual Inspection Data .............................. 64 Figure 3-14: Johnson Transformation of the Portable Raman Spectroscopy Data .......... 65 Figure 3-15: Step 9 of the Proposed Indicator Evaluation Methodology ........................ 67 Figure 4-1: Evaluation Scenario of Independent Variable Effects on Dependent Variables for Hypothesis Test Results ....................................................................... 71 Figure 4-2: Example Spacecraft Performance Modeling Results. (a) Silicone on Aluminum Foil (b) Hydrocarbon on Aluminum Foil (c) Silicone on Glass (d) Hydrocarbon on Glass (e) Silicone on Solar Reflector MLI (f) Hydrocarbon on
xi
Solar Reflector MLI. .................................................................................................. 75 Figure 4-3: Silicon Wafer Mirror Reflectivity Performance Impact Model Results for Visual Inspection ....................................................................................................... 76
xii
List of Tables Table 1-1: Leading Indicator Challenges ........................................................................... 3 Table 2-1: Particulate and Molecular Performance Degradation Effects on Spacecraft Surfaces (Tribble, 2000) ............................................................................................ 23 Table 2-2: Considerations and Risks in Performing Contamination Response Actions . 26 Table 3-1: Experimental Materials for Spacecraft Contaminants.................................... 48 Table 3-2: Experimental Materials for Spacecraft Surface Substrates ............................ 49 Table 3-3: Determination of Maximum Film Thickness Missed in Visual Inspections for Fluorocarbon Film on Aluminum Kapton® Tape................................................. 58 Table 3-4: Determination of Maximum Film Thickness Missed with Portable Raman Spectroscopy for Fluorocarbon Film on Aluminum Kapton® Tape .......................... 63 Table 3-5: Satisfaction of ANOVA Assumptions with Transformed Data ..................... 66 Table 4-1: Maximum Contaminant Film Thickness Missed for Visual Inspections ....... 69 Table 4-2: Maximum Contaminant Film Thickness Missed for Portable Raman Spectroscopy .............................................................................................................. 69 Table 4-3: Hypothesis Test Results ................................................................................. 71 Table 4-4: Contaminant Performance Impact on Space System Surfaces ....................... 72
xiii
List of Acronyms ACS
American Chemical Society
ALADIN
Atmospheric Laser Doppler Instrument
ANOVA
Analysis of Variance
ASTM
American Society of Testing and Materials
CALCRT
Calculation of Reflectance and Transmittance
CC
Contamination Control
CCD
Charge Coupled Device
CDR
Critical Design Review
CLID
Contaminant Induced Laser Damage
DOD
Department of Defense
DV
Dependent Variable
FTIR
Fourier Transform Infrared
GAO
Government Accountability Office
GC/MS
Gas Chromatography/Mass Spectroscopy
GLAS
Geoscience Laser Altimeter System
IEST
Institute of Environmental Sciences and Technology
INCOSE
International Council on Systems Engineering
IPA
Isopropyl Alcohol
ISO
International Organization for Standardization
ISS
International Space Station
IV
Independent Variable
LIDAR
Light Detection and Ranging
xiv
LITE
LIDAR In-Space Technology Experiment
LLC
Limited Liability Company
MLI
Multilayer Insulation
MOLAl
Mars Orbiter Laser Altimeter
NASA
National Aeronautics and Space Administration
NVR
Non-volatile Residue
PDR
Preliminary Design Review
PSM
Practical Software and Systems Measurement
ROSINA
Rosetta Orbiter Spectrometer for Ion and Neutral Analysis
STD
Standard
UV
Ultraviolet
VC
Visibly Clean
VCHS
Visibly Clean Highly Sensitive
WFPC
Wide Field Planetary Camera
xv
Chapter 1: Introduction In any given failure review board, a common question raised is: “Could we have found this problem sooner?” After failure occurs, signs that risks and issues were growing into major problems leading to the eventual failure may be obvious in hindsight. The challenge is to shift the thinking from a reactive nature of looking for evidence of risk growth after the incident or failure occurred to a proactive nature. In a proactive effort, risks are identified, monitored for growth or change with detection methods, analyzed to determine trends, and predictive modeling is performed to estimate future system performance given current trend magnitude and direction. The ultimate goal is to provide engineering managers with enough information to understand the risk to the system performance and catch risks and issues early enough in the system lifecycle when course corrections are still possible to execute. Leading indicators are methods for detecting signals and warnings of risk and providing predictions of future system states. However, faced with several candidate indicator methods, engineering managers need to know how well a potential indicator will perform for their system. Challenges exist in selecting the appropriate leading indicator for a system including how well an indicator method will detect signals of risk, how to measure and determine leading indicator effectiveness, and what signals an indicator can miss and how that will affect the system. This research examines these challenges and proposes an evaluation methodology to characterize and compare proposed leading indicators for a system. This work demonstrates through case study research how different indicator methods can be evaluated to 1
determine if they are being compared on an equal “apples-to-apples” basis despite very different detection and measurement techniques being utilized by the indicator methods. The evaluation methodology is executed through empirical testing and statistical analysis and illustrates how two different indicator methods can be compared, the risk signal detection for each indicator method can be defined, and the impact to system performance can be determined. 1.1
Background and Problem Definition Leading indicators are designed to detect signals and warnings of risk and
predict future system performance with enough accuracy and lead time to allow engineering managers to make any required course-corrections. Leading indicators have been developed and utilized in several high-risk fields such as process and industrial safety where accidents can cause wide-spread damage including loss of human life. An early example of such accidents were the regular explosion of 19th century steam engines due to a lack of understanding of steam pressure, steam temperature, and steel boiler wall thickness in contributing to explosions (Swuste & Blokland, 2015). Once these properties were understood, operational measures could be put in place to monitor steam temperature and pressure, safety valves could be developed and installed, and boiler manufacturing processes could be improved to decrease the probability of boiler explosion (Swuste & Blokland, 2015). Leading indicators have been adopted by the field of systems engineering to determine the effectiveness of systems engineering on a project (INCOSE, 2010). These system engineering leading indicators have been detailed and documented in the International Council on Systems Engineering (INCOSE) Systems Engineering 2
Leading Indicators Guide. Currently, Version 2.0 of this guide provides 18 systems engineering leading indicator categories for indicators that can be deployed throughout a system’s lifecycle. As stated by this guide, “Leading indicators aid leadership in delivering value to customers and end users, while assisting in taking interventions and actions to avoid rework and wasted effort” (INCOSE, 2010). Regardless of the field of use, engineering managers that would like to realize the benefits of leading indicators are often faced with a myriad of choices for leading indicator methods. Given these numerous choices, it can be challenging for engineering managers to evaluate and compare candidate indicator methods on an equal basis and ensure they are selecting an effective indicator for their system. Bias towards an indicator may exist, or an indicator method may be discarded from consideration if the system effects on the indicators are not known. Additional challenges identified in developing and utilizing leading indicators are provided in Table 1-1. Table 1-1: Leading Indicator Challenges Challenges Accurately detecting signals of risk and separating them out from system noise (Paté-Cornell, 2012) Determining if it is possible to know if a risk signal is being missed or ignored prior to problems, accidents, and failures occurring (Aven, 2015a) Determining how to “derive effective leading indicators” (Leveson, 2015) Determining how to “select the measurements to be made based on the system insight they will provide” (Roedler & Rhodes, 2010) Determining how to “select the right leading indicator for system improvement” (Forest & Kessler, 2013) Determining how to “quantify the accuracy and precision of the leading indicator and determine its effectiveness for the system of interest” (Roedler & Rhodes, 2010) Determining how to “best implement and use the leading indicators in a given program context” (Rhodes et al., 2009)
3
1.2
Purpose of the Study As discussed in the previous section, there are several challenges the
engineering manager faces when exploring different indicator methods for their system. Consideration of these challenges led to the development of the following research question (Figure 1-1):
Figure 1-1: Research Question
The purpose of this study is to answer this research question through testable hypotheses stemming from data collected during experimental testing. To answer this research question, a leading indicator evaluation methodology was developed and executed through case study research of molecular film contamination on space system surfaces. This case study evaluated two different indicator methods: the existing standard method of visual inspection of space system hardware, and a proposed indicator method of portable Raman spectroscopy. Visual inspection relies on detection of contamination with the human eye, while portable Raman spectroscopy detects the presence of contamination through spectrographic techniques. Through the evaluation methodology, each indicator method was demonstrated and quantified in terms of the maximum molecular contaminant film thickness the method can miss for detecting different contaminants on different spacecraft surfaces (substrates). The resulting impact to space system performance was determined through predictive modeling. Statistical analysis was performed to 4
determine if contaminant type or substrate type (the independent variables) had a significant effect on the maximum film thickness missed by each indicator method (the dependent variables). The null hypotheses developed were as follows:
H10: Substrate type will have no significant effect on the maximum film thickness missed for visual inspections.
H20: Contaminant type will have no significant effect on the maximum film thickness missed for visual inspections.
H30: Substrate type will have no significant effect on the maximum film thickness missed for portable Raman spectroscopy.
H40: Contaminant type will have no significant effect on the maximum film thickness missed for portable Raman spectroscopy.
The results of this analysis determined if the metric of maximum film thickness missed is a metric that allows equal comparison of the two indicator methods, as posed in the research question. 1.3
Significance of the Study This research focuses on the intersection of three main engineering
management concepts as shown in Figure 1-2: systems integration, risk management, and leading indicators. The systems integration phase of the system lifecycle was chosen for this research because it is the critical phase where system components first come together and begin to interact, and the first opportunity for integrators to detect signals and warnings of risks. The earlier a risk can be discovered, the sooner an engineering manager can identify, deploy, and execute mitigation and response actions before the risk grows into a major problem. This, in turn, saves cost and 5
effort by avoiding rework of system elements.
Figure 1-2: Focus of this Research
As stated by the INCOSE Systems Engineering Leading Indicators Guide, “…it is wise to employ a measurement to quantify the performance (accuracy and precision) of a leading indicator” (Roedler & Rhodes, 2010). This work illustrates through case study research how two different indicator methods for a system can be characterized with empirical testing, metrics stemming from the resulting data can be compared on an equal basis, and results can be expressed in terms of system performance to support engineering managers in evaluating indicators. The case study provides an example of execution of the evaluation methodology for the case of molecular film contamination on space system surfaces. The evaluation methodology developed in this research allows for engineering managers to determine if they have an indicator method that is a leading indicator (as opposed to a lagging indicator) and aids engineering managers in determining if it 6
will be effective for their systems. This reduces the potential for lost time, effort, and wasted cost from deploying ineffective indicator methods. The evaluation methodology can be repeated for any indicator method. The methodology executed in this research evaluates an existing indicator method and a proposed indicator method for a system. However, the methodology can be expanded in future work as a tool to create requirements and design criteria for future indicator methods, thus providing an opportunity for development of future engineering management tools. Molecular contaminant films were chosen as the focus of this study because of the tenacious nature of such contaminants and the potential impact to space system performance. Contaminant films present on solar arrays can cause a reduction in the transmission of light through the array, creating a reduction in power which can impact the charging of on-board batteries. Radiators and thermal blankets can experience degradation in emissivity which negatively impacts the ability of the spacecraft to heat and cool itself and maintain thermal balance. The optics of sensitive instruments can experience a decrease in the reflection or transmission of light, which negatively impacts instrument calibration and science measurements. This research provides a means of characterizing limitations in indicator methods for detecting molecular contamination films on space system surfaces through empirical testing, and translating the results in terms of impact to space system performance through performance modeling. By understanding these limitations, indicator methods can be more effectively deployed in the system integration process to detect signals of molecular contamination risk.
7
1.4
Scope & Limitations This study relies on case study research of molecular film contamination
present on space system surfaces to execute the proposed leading indicator evaluation methodology. The goal of this case study is to illustrate to potential future users how the methodology can be executed to compare different leading indicator methods and provide valuable information to engineering managers to aid in selecting an effective indicator for their system. However, the case study is limited to the case at hand and the independent and dependent variables defined in the study. This particular study explored five molecular film contaminants on five spacecraft substrate surfaces at 25 different contaminant concentration levels (with one control and one blank) for a total of 675 samples analyzed by each indicator detection method. The case study only considered two indicator detection methods of visual inspections and portable Raman spectroscopy. Additionally, only one metric was evaluated for the indicator methods: maximum molecular contaminant film thickness missed by each detection method. 1.5
Organization of the Document This document is organized in six chapters. This chapter provided a brief
overview of the background and problem definition, purpose and significance of the study, how the study will utilize engineering management tools, introduced the research question to be answered and hypotheses to be tested, and provided the scope and limitations of the study. Chapter 2 provides a literature review of two key areas: 1) applicable engineering management topics of systems integration, risk management, and leading indicators, and 2) a background of spacecraft 8
contamination to provide supporting information for the case study. Chapter 3 presents the proposed evaluation methodology, research goals, and testable hypotheses. This chapter also details the research methods including experimental materials and sample preparation, experimental measurements with visual inspections and portable Raman spectroscopy, data collection and analysis, and the performance modeling of molecular contamination effects on space system performance. Chapter 4 presents all results including experimental results, hypothesis test results, and performance modeling results. Finally, Chapter 5 discusses conclusions, implications of the study for the spacecraft industry, and opportunities for future research.
9
Chapter 2: Literature Review The literature review is divided into two parts. Part I focuses on relevant engineering management topics of systems integration, risk management, and leading indicators. Part II provides a review and background on research pertinent to the chosen case study of molecular contamination of spacecraft surfaces. 2.1 Part I: Applicable Engineering Management Topics The following sections summarize past work performed and the state of current knowledge in engineering management. This knowledge includes systems integration and reducing rework during this process, risk management including new perspectives in addressing system risk and looking for signals and warnings of risk, and use of leading indicators to provide predictions on the future state of systems to assist engineering managers in risk management and avoidance of rework. 2.1.1 Systems Integration INCOSE defines a system as “a combination of interacting elements organized to achieve one or more stated purposes” (INCOSE, 2010). It follows that systems integration is a process that “combines system elements to form complete or partial system configurations in order to create a product specified in the system requirements” (INCOSE, 2010). The classic systems engineering V-Model shown in Figure 2-1 illustrates the location of the systems integration stage in the development cycle of a project (Forsberg & Mooz, 1991). The project definition phase (left side of the V) involves the planning activities of concept of operations development, requirements and architecture definition, and detailed design processes. These plans 10
are manifested in the project test and integration phase (right side of the V). This phase begins with systems integration, test, and verification and proceeds through verification and validation to operation and maintenance. Therefore, the systems integration stage is the first opportunity for the components of a system to come together, interface, and interact.
Figure 2-1: Systems Engineering V-Model
As individual components are built up into sub-assemblies and assemblies, requirements and interfaces are verified through tests, analysis, inspections, and demonstrations (Madni & Sievers, 2014). Performance characteristics of higher level assemblies depend on the performance of lower-level components (Marchant, 2010). This process can be quite extensive as system complexity increases. As such, problems can remain hidden and appear later in the project lifecycle. Issues created at a lower level of integration can propagate through the system, often undetected, until they manifest into problems. As the system matures in the project test and integration phase, problems become “more difficult an expensive to isolate and 11
correct” (Madni & Sievers, 2014). Parts quality problems in government programs were reviewed by the U.S. Government Accountability Office (GAO) during a study in 2011 and found problems discovered during final system integration and testing created significant contributions to program cost increases and schedule delays (GAO, 2011). This is illustrated in Figure 2-2, where the significance of consequences from problem parts increases as the system progresses from design and fabrication to assembly and test. The goal becomes to find issues earlier in the system lifecycle rather than later.
Figure 2-2: Parts Problems found during Product Development Phases (GAO, 2011)
When problems are discovered during the systems integration stage, nonconformances to requirements may result and rework may have to occur to rectify and resolve issues. Rework is “work that occurs when a prior decision that was assumed to be final for that project is changed because it was later found to be defective" (Kennedy, Sobek, & Kennedy, 2014). The cost of rework increases over the system lifecycle as the system matures as shown in Figure 2-3, and “can increase from 3 to 1000 times the original development costs depending on when they are 12
discovered” (Kennedy et al., 2014).
Figure 2-3: Cost to Extract Defects during Lifecycle Cost over Time (INCOSE, 2010)
Systems integration is often viewed by responsible parties as a troublesome process; project managers view it as a complex process to be managed, while systems engineers consider it to be a risky process (Frank, Harel, & Orion, 2014). The magnitude of the risk and cost to fix issues increases if the project team is reactive and waits until issues and problems are observable. By the time risks manifest to a level that can be visibly detected and observed during hardware inspections, it may be too late and the system may have already experienced degraded system performance (Aven & Krohn, 2014). Not only do known unknowns need to be identified during systems integration planning (Frank et al., 2014), but emphasis must be placed on effective risk management during systems integration to avoid costly defects and rework (INCOSE, 2010).
13
2.1.2 Risk Management INCOSE states, “The purpose of the Risk Management Process is to identify, analyze, treat and monitor the risks continuously” (INCOSE, 2010). Risk is used in systems engineering to determine the level of effort and investment of resources to be applied during the development phase to prevent problems and failures in the future operational system (Leveson, 2015). Of particular interest is the monitoring of risks during the systems integration process as components and elements begin to interact. The following literature review will summarize the traditional 5x5 risk matrix based on probabilistic analysis and introduce new perspectives in risk management that focus on supporting the engineering manager in detecting signals and warnings of risk. 2.1.2.1 Traditional 5x5 Risk Matrix Risk assessments are performed as part of the risk management process in a project lifecycle. The Department of Defense (DoD) and National Aeronautics and Space Administration (NASA) utilize 5x5 risk matrices to identify and score risks as shown in Figure 2-4. Probability of occurrence is plotted on the y-axis while severity of consequences is plotted on the x-axis. Both use a one to five scoring scale with one being the lowest and five being the highest. Risks with high likelihood and high consequence are ranked “high” and appear in the red zones, while risks with low likelihood and consequence are ranked “low” and appear in the green zones (Madni & Sievers, 2014). Risks are tracked throughout the project, and the status of risks are presented at reviews such as preliminary design and critical design reviews. Risk mitigation plans may be developed throughout the project to 14
reduce the probability of occurrence and/or the severity of consequence for each high and medium ranked risk.
Figure 2-4: 5x5 Risk Matrix (NASA)
One issue with the traditional 5x5 risk assessment is that the assessments are subjective and limited to the experience and knowledge of assessors. An assessor with deep knowledge of a subject may be able to provide more realistic scoring of a risk based on previous experience over another assessor that may have little experience or knowledge. Aven and Krohn argue, “probability is just one tool for describing uncertainty and the concept of risk should not be limited to this tool” (Aven & Krohn, 2014). Probability assessments may provide a narrow view of risk, and key aspects of risk can be missed. New perspectives in risk management have recently emerged to focus on the strength of knowledge with the goal of providing a broader view and a better understanding of risk for a system (Aven & Krohn, 2014). 2.1.2.2 New Perspectives in Risk Management The knowledge used to develop probability assessments of risk is “just as important as the probability itself” (Aven & Krohn, 2014) which has led to a new 15
focus on knowledge in risk management. Better risk analysis can be performed by drawing on quality data and information and reducing uncertainties, which depend on the knowledge of the assessor and engineering manager as a decision maker. Therefore, the more information an engineering manager has concerning a risk, and the more they understand the impact to the system performance, the better the risk assessment. “Management of risk is thus to be read as management of risk and performance, and a new way of thinking about risk, as a new way of thinking about risk and performance” (Aven & Krohn, 2014). From this, improved mitigation plans can be developed to adequately reduce risk and ensure the system will operate and perform as intended. Measures made on system elements can provide information to support engineering managers in risk analysis. However, each measurement comes at a cost, and the measurement must provide real information value to be effective (Roedler & Rhodes, 2010). Decision makers may need to choose between measurement techniques and methods to ensure they are effective in monitoring risks (Aven, 2015b). Ideally, measures performed to monitor for risks will detect signals of risk to allow engineering managers enough time to course-correct and deploy mitigation activities. 2.1.2.3 Detecting Signals and Warnings of Risk To reduce risk, the risk must first be properly understood. This can be done through “systematic observation and recording of near-misses and precursors” (Paté-Cornell, 2012). Signals and warnings of risks may emerge prior to the risks evolving into problems, incidents and accidents. For the systems assembly, 16
integration and test phase, engineering managers need information on risk growth or change as soon as possible to provide the greatest amount of response time. This begins with accurate detection of the signal followed by interpretation of the information. At this point, additional information may be required or immediate action may take place depending on “the quality of the signal, the lead time, and the consequences of an event” (Paté-Cornell, 2012). Signal detection and identification ranges between early detection of faint signals that may be hidden in system noise, and clear trends with statistically significant results. However, by the time a signal or warning has become visible and statistically significant, system performance may have already become significantly degraded (Aven & Krohn, 2014). To be proactive in risk monitoring, leading indicators may be developed for a system. The challenge becomes knowing how effectively a measure will monitor a risk to indicate a risk is growing or changing over time. Addressing this challenge is a major focus of this research. 2.1.2.4 Missing Signals and Warnings of Risk While monitoring signals for risks of accidents and failures is desirable, these important signals can be weak and hidden in system noise (Leveson, 2015). Additionally, it can be difficult to know if a signal is being missed or ignored prior to an accident or failure occurring (Aven, 2015a). This can be illustrated in Reason’s Swiss cheese model shown in Figure 2-5 (Reason, 1997). Independent barriers and safeguards are often employed to prevent a problem from occurring, especially in high-risk or high-consequence fields. In this model, independent safety barriers are illustrated as slices of Swiss cheese, as no barrier is perfect and contains holes 17
(Qureshi, 2007). Failures and accidents happen when the holes in the barriers become perfectly aligned, allowing hazards to slip through (Paté-Cornell, 2012). In such a scenario, signals and warnings may be present indicating the alignment of holes in the barriers, or movement, opening, or closing of the holes (Sipilä, Auerkari, Holmström, & Vela, 2014). These signals and warnings may be very weak, especially early in the systems assembly, integration and test phase.
Figure 2-5: Reason’s Swiss Cheese Model (Reason, 1997)
2.1.3 Leading Indicators Indicators for issues can describe current situations as current indicators or future situations as leading indicators (Rhodes, Valerdi, & Roedler, 2009). Leading indicators are of interest because they provide perspective on the “to be” state rather than the “as-is” state. From this information, engineering managers can determine if it is acceptable for a system to continue on the predicted trend or if changes need to be made to avoid problems and move to a different predicted state. The purpose of leading indicators is to “aid leadership in delivering value to customers and end users, while assisting in taking interventions and actions to avoid rework and wasted effort”(Roedler & Rhodes, 2010). To assist the engineering community in 18
developing leading indicators and realizing this benefit, the Massachusetts Institute of Technology, INCOSE, and Practical Software and Systems Measurement (PSM) created the Systems Engineering Leading Indicators Guide. This document provides guidance on identifying and implementing leading indicators based on trends throughout the systems engineering process. A leading indicator is defined as being “composed of characteristics, a condition and a predicted behavior” (Roedler & Rhodes, 2010). Indicators monitor elements of a system through measures and analysis with the goal of providing actionable information to the decision maker before risks develop into problems. The characteristic and condition of interest are empirically measured and analyzed for trends which are input into predictive tools such as models and simulations. The accuracy of the measurements and robustness of the predictive tools influence the confidence a decision maker has in the information provided by the indicator. The leading indicator needs to provide information in a timely manner for the decision maker to evaluate the information, develop an action plan, and deploy responsive actions. The leading indicator should also allow for improvement by allowing adjustability to new measurement techniques and information sources, and improving predictive elements through advancements in models and simulations. Indicators are effective when they are developed specifically for a system (Leveson, 2015), contributing to system knowledge as a system is developed, integrated, and operated. A leading indicator cannot simply be chosen from a list of candidates, rather, it must be designed and evaluated to ensure system risks can be effectively detected 19
and monitored to provide actionable information to engineering managers as decision makers. While there are many benefits to incorporating leading indicators for a system, there are also several challenges for an engineering manager to identify, select, and deploy leading indicators, which were previously presented in Table 1-1. Faced with several choices for measurements, monitoring techniques, and metrics, an engineering manager needs to know how well a candidate leading indicator will perform for their system. This research focuses on this issue for leading indicators and addressing the identified challenges and gaps in current leading indicator knowledge. 2.2 Part II: Case Study of Spacecraft Contamination Space systems are complex systems comprised of extremely large number of parts and sub-assemblies that must work together to perform complex operations in the harsh environment of space, often for long periods of time (years to decades of operation). Examples of space systems include satellites, planetary landers, orbiters, and rockets. The assembly, integration and test phase can take several years to complete, and throughout this phase in the system lifecycle risks can begin to emerge as problems. One of the major environmental risks that spacecraft systems must address is hardware contamination (Ott et al., 2006). Contamination is any undesired foreign material in or on a solid, or in a gas or liquid (Tribble, 2000). For spacecraft systems, contamination is typically divided into two main types: particulate contamination and molecular contamination. Particulate contamination is solid “undesirable foreign material of miniature size with observable length, width, and thickness” (Tribble, 2000). Common 20
examples include dust, skin flakes, fibers, paint chips, and metallic particles. Particulate contamination can occur from processes including but not limited to material wear and degradation, metal oxidation (rust), people, and processes occurring in an environment. An example of particulate contamination on the viewport window of a thermal vacuum chamber is shown in Figure 2-6.
Figure 2-6: Particulate Contamination on a Thermal Vacuum Chamber Window (NASA/Elaine Seasly)
Molecular contamination is any “undesirable foreign film matter” (Tribble, 2000) on a surface. Such films typically condense on surfaces in the form of droplets rather than a uniform film (Tveekrem, Leviton, Fleetwood, & Feinberg, 1996) (Luey & Coleman, 2006). Common examples of this type of contamination include fingerprints, oils, greases, and residues. Sources of molecular contamination include but are not limited to condensed films from materials outgassing volatile 21
compounds, transfer contamination during handling and assembly processes, and residues from surface treatment and cleaning processes. An example of molecular contamination is shown in Figure 2-7. This photo shows a molecular film condensed on a cryogenically cooled surface in a thermal vacuum chamber.
Figure 2-7: Molecular Contamination Film on a Cryogenically Cooled Surface (NASA/Elaine Seasly)
Degradation effects of particulate and molecular contamination depend on the spacecraft sub-system and surface affected. Spacecraft surfaces that can experience the most performance degradation due to particulate and molecular contamination include solar arrays, thermal control surfaces, and optics. Performance degradation effects for each are summarized in Table 2-1 and include decreased signal strength from changes in reflectivity, transmission, and emissivity.
22
Table 2-1: Particulate and Molecular Performance Degradation Effects on Spacecraft Surfaces (Tribble, 2000) Contaminant Type
Solar Arrays
Particulate
Surface obscuration (power reduction)
Molecular
Decrease in signal transmission causing a power reduction
Thermal Control Surfaces Change in solar absorptance and/or emissivity Change in emissivity resulting in a change in spacecraft heating/cooling
Optics Surface obscuration (reduced signal strength) Stray light scattering Decrease in reflectivity or transmission Absorption of signal Stray light scattering from droplets
As will be discussed in detail in Section 2.2.4, visual inspections are routinely performed throughout the assembly, integration and test phase to detect the presence of particulate or molecular contamination. Studies have been performed to quantify visibly detectable particles under standard inspection conditions (Tribble, 2000). However, little data exists on visual inspection detection of molecular contaminant films despite visual inspections being utilized as standard quality control processes. Therefore, this research will target this gap for this case study and focus on molecular contamination. 2.2.1 Contamination as a Risk Contamination is almost always a risk for space system hardware. Particles and molecular films are always present in any system because it is impossible to remove all unwanted matter. Therefore, the amount of contamination is always a non-zero value. The risk matrix for contamination essentially reduces to a 1x5 matrix, with the probability of occurrence at 100%. The goal becomes to keep the severity of consequence from moving to the right in the matrix shown in Figure 2-4 to a higher risk level. Since contamination cannot be completely eliminated, efforts are required 23
to mitigate and control contamination to acceptable levels to keep contaminants from negatively impacting system performance. The entire field of contamination control engineering is devoted to these efforts which must identify and mitigate contamination risks throughout the entire space system lifecycle. “Effective contamination control is essential for the success of most aerospace programs because the presence of contamination, even in small quantities, can degrade the performance of sensitive spacecraft hardware” (Rampini, Grizzaffi, & Lobascio, 2003). During systems integration, components are assembled and begin to interface and interact. This interaction is the physical pathway for contamination to be generated and transferred throughout the system. If the contamination remains undetected at lower level assemblies, it can propagate through the system later in the integration phase as the system increases in complexity. Worst case, the contamination remains undetected during assembly, integration and test and causes major system degradation or failure during system operation. As stated by Holmes, “Molecular contamination can occur at various points during integration, test and space flight operations and can possibly go undetected until after launch” (Holmes et al., 2016). If contamination is detected during systems integration, engineering managers must determine appropriate response actions that should be taken to mitigate the issue. Such actions may include performing cleaning operations to remove the contamination, reworking or replacing the contaminated hardware, or doing nothing and allowing the contamination to remain in place. Each action carries 24
considerations and risks that the engineering manager must weigh in the decision making process. Examples of such considerations and risks for contamination response actions are summarized in Table 2-2. Another challenge for engineering managers is the consideration of when in the integration phase the contamination was discovered. Contamination discovered at the component-level or in lower-level subassemblies can be easier and less costly to remove compared to contamination discovered at the full assembly or completed system level. Late in the system integration phase, surfaces and components may be physically inaccessible for performing cleaning or rework operations. Additionally, cleaning and rework operations can create additional risk including damage of sensitive surfaces and components, transference of contaminants to hidden areas where they can redistribute at a later time, and adding additional or new contaminants from cleaning solvents or tools. Engineering managers need the capability to detect contamination risks as early as possible in the system integration phase to allow for maximum response time in developing mitigation strategies and taking course-corrective actions.
25
Table 2-2: Considerations and Risks in Performing Contamination Response Actions Action Clean
Don’t Clean
Rework or Replace
Considerations and Risks Ability to access surfaces for cleaning Ability to clean effectively and remove contamination Transfer of contamination to other surfaces during cleaning Damaging parts or surfaces due to cleaning process Leave contamination in place & risk system degradation later in lifecycle Impact to system reliability, maintainability, and usability Able to access component for replacement Schedule risk due to time required for rework Availability of spare components to replace contaminated components
2.2.2 Examples of Past Spacecraft Issues due to Molecular Contamination Molecular contamination, also known as non-volatile residue (NVR), is “one of the most serious problems for satellites” (Yokozawa, Baba, Miyazaki, & Kimoto, 2012) and can cause severe performance degradation once space systems are in operation. “On-orbit NVR is influenced by a complex of factors, including the history of environmental exposure, ConOps, venting, surfaces on which residue may spread, and the physical properties of materials used in the system” (Marchant, 2010). The following examples illustrate degraded system performance experienced on past spacecraft missions due to molecular contamination. In 1993, the Wide Field and Planetary Camera I (WFPC-1) was replaced on the Hubble Space Telescope during a servicing mission. The returned hardware provided an opportunity to analyze mirror degradation effects from 3.5 years of onorbit contamination exposure. The pickoff mirror of the WFPC-1 was discovered to have been heavily contaminated with a 450 angstrom thick molecular contaminant film of hydrocarbons, esters, and silicones (Tveekrem et al., 1996). Ultraviolet (UV) 26
reflectance measurements of the contaminated mirror were made and compared to measurements of a spare mirror that was identical to the flight mirror but not flown in space. These results for incident light normal to the mirror surface are shown in Figure 2-8. In this figure, mirror reflectance is shown on the y-axis over the wavelength range given on the x-axis. A perfectly reflective mirror would have a reflectance of 1.0. The highest degradation for the flight mirror occurred from 120 nm to 130 nm (almost completely degraded), with 50% reflectance reduction occurring all the way to approximately 140 nm. These measurements showed significant degradation of optical surfaces in the UV wavelength had occurred from molecular contamination outgassed from on-board Hubble components (Tveekrem et al., 1996).
Figure 2-8: UV Reflectance of Flight and Spare Hubble Pickoff Mirrors (Tveekrem et al., 1996)
Space-flight laser systems are perhaps the most sensitive to molecular contamination as molecular interactions with the laser beam can cause contaminant induced laser damage (CILD). As described by Riede, “In the past, various space-borne 27
laser systems and predecessors of the Atmospheric Laser Doppler Instrument (ALADIN) system (LITE - LIDAR In- Space Technology Experiment (1994), MOLA1 Mars Orbiter Laser Altimeter (1996), GLAS - Geoscience Laser Altimeter System (2003)) suffered from performance loss or failure with organic and/or silicone contamination being suspected as possible causes” (Riede, Allenspacher, & Schroder, 2005). LITE was a short-term mission deployed on the space shuttle and experienced performance degradation in both on-board lasers (Abdeldayem, Dowdye, & Canham, 2006). The GLAS system, comprised of three identical lasers, experienced failure of the first laser after 36.7 days on-orbit, failure of the second laser after 121 days, and degradation of the third laser to the point it was at 50% power for the last 167 days of operation (Abdeldayem et al., 2006). Experiments are flown on the International Space Station (ISS) to determine the degradation effects of the space environment on materials and biological samples. One such experiment, “Expose-R,” was a series of suitcase-sized containers with numerous compartments containing different material and biological samples. The containers were deployed on the ISS and exposed to solar energy for 22 months. After the exposure period, windows on compartments that had been connected to space vacuum had turned brown while windows on compartments pressurized with argon remained clear (Demets et al., 2014). This visible browning effect is shown in Figure 2-9, where windows 8, 10, 5, and 23 turned brown under space vacuum exposure while windows 6, 20, and 18 in the pressurized argon compartments remained clear. This experiment verified molecular contaminant film deposition from internal compartment materials had condensed on the interior window surfaces and darkened after exposure to UV radiation. 28
Figure 2-9: Darkening of Expose-R Windows after Space Exposure for 22 Months (Demets et al., 2014)
2.2.3 Molecular Contamination Requirements Product cleanliness levels for particulate and molecular contamination are defined by IEST-STD-CC1246E: Product Cleanliness Levels – Applications, Requirements, and Determination (IEST, 2013). Molecular contamination requirements are specified as concentration of NVR in mass per 0.1 m2 on surfaces or mass per 0.1 L in fluids. Cleanliness levels were previously designated by alphabetical letters beginning with Level A for 1 mg/0.1 m2 or 1 mg/0.1 L, Level B for 2 mg/0.1 m2 or 2 mg/0.1 L, etc. Increasing letters indicated increasing concentrations of contamination. Levels cleaner than Level A followed a fractional designation (i.e. Level A/2 for 500 g/0.1 m2 or 500 g /0.1 L, Level A/5 for 200 g/0.1 m2 or 200 g /0.1 L, etc.). Revision E of the standard introduced new designation levels “specified by calling out R, followed by the value of the 29
maximum allowed mass (mg) of NVR per unit area (1 cm2) or volume (0.1 cm3)” (IEST, 2013). Therefore, Level A/5 became R2E-1, Level A/2 became R5E-1, Level A became Level R1, Level B became R2, etc. The Revision E designation levels are relatively new and the contamination community is still updating and transitioning documentation to include these levels. Because of this, and the fact that previous work in the literature refers to prior designation levels, the previous designation levels of A/5, A/2, A, B, etc., will be used in this work. Hardware NVR levels are quantitatively determined by four main sampling techniques: witness mirrors, witness plates, sampling of surfaces or fluids with swabs and wipes, or rinsing of surfaces with solvent (Colony, 1985). Witness mirrors and witness plates are used when direct sampling of a hardware surface is not possible. The mirror or plate is placed physically close to the critical hardware and experiences the same integration environment and processes. The mirror or plate can then be analyzed after exposure. In NVR analysis, the surface of interest is rinsed, swabbed, or wiped with a solvent to collect deposited molecular contaminants. Wipes and swabs are analyzed via ASTM E1560 (ASTM, 2011), and surfaces exposed to integration environments that are solvent rinsed are analyzed via ASTM E1235 (ASTM, 1995). In these methods, the collected solvent is evaporated in a controlled manner, and the remaining residue is weighed to gravimetrically determine the NVR. Spectroscopic analysis such as Fourier-Transform Infrared (FTIR) spectroscopy or gas chromatography mass spectrometry (GC/MS) can then be performed on the NVR sample to determine chemical constituents and aid in identification of contaminant sources. The drawback to NVR analysis is the time 30
required to obtain results. Surfaces in integration environments must be exposed for long periods of time (weeks to months) to deposit enough NVR to be detected by laboratory analytical balances. Sample preparation, processes, and analysis can take several days to weeks to complete. System assembly, integration, and test activities typically continue in parallel while NVR analysis is performed. By the time the NVR analysis is complete and indicates if a problem with contamination has occurred the system may have achieved a higher level of integration or completed critical testing. Villahermosa, Weiller, Virji, and Taylor (2008) describe the two approaches taken by programs to define molecular contamination requirements for hardware: “On one end, programs have utilized a ‘best effort’ approach where contamination requirements are specified based on the lowest levels a manufacturer can produce and maintain…The other extreme takes the approach of achieving the ‘best possible’ levels of cleanliness that often push the state-of-the industry” (Villahermosa, Weiller, Virji, & Taylor, 2008). There are downfalls to both of these approaches. A “best effort” approach provides little incentive for a manufacturer to improve processes to achieve higher levels of cleanliness. A “best possible” approach can be difficult to verify, as measurement and verification methods must be able to accurately detect contaminant levels lower than those specified by the requirement. This, in turn, increases costs for equipment and analysis. Most importantly, both approaches do not take into account system performance degradation effects due to different types of molecular contaminants. Ideally, a “best performance” approach for setting requirements would be taken to define 31
cleanliness levels based on system sensitivity to different molecular contaminants, with the impact to performance modeled and known for each. 2.2.4 Current Indicator Method: Visual Inspections INCOSE defines the inspection verification activity as, “An examination of the item against applicable documentation to confirm compliance with requirements. Inspection is used to verify properties best determined by examination and observation (e.g., paint color, weight, etc.)” (INCOSE, 2010). Visual inspections are continuously performed on hardware throughout the system assembly, integration, and test phase to ensure no visible changes have occurred due to processes performed on the hardware. During these processes, inspectors look for issues such as defects, damage, missing parts, incomplete processes, workmanship quality, and contamination. Visual inspections are common throughout several industries including the manufacture of “…piston rings, acoustical tiles, aircraft components, pipelines, highway bridges, television panels, contact lenses, printed circuit board assemblies, airport baggage, pharmaceuticals, and food products” (See, 2015). Visual inspections are typically 70% to 80% accurate (See, 2015). Visual inspections are common practice for observing the presence of contamination on space system surfaces. An example of a visual inspection performed on the mirror segment for the James Webb Space Telescope is shown in Figure 2-10. Some benefits to performing visuals inspections are that no physical contact with surfaces is required for observations to be made, results are instantaneous, multiple observers can be utilized, and inspections can be performed numerous times. Visual inspection conditions for spacecraft surfaces are defined by 32
IEST-STD-CC1246E, which specifies the type of incident light (white light or black light), light intensity, and viewing distance from the surface to be observed. For example, Level VC-0.5-1000, also known as “Visibly Clean Highly Sensitive (VCHS),” specifies 100 ft-candles white light inspection of a surface approximately 6 to 18 inches away from the observer. A surface is determined to be “visibly clean” when no particulate or molecular contamination is observed under the specified lighting conditions. Ultraviolet black light inspection may also be performed to view particles or molecular films that may fluoresce under UV light. The ability to detect the presence of contamination on surfaces varies with surface type and contaminant type. Particles and droplets are easier to detect on polished and reflective surfaces than rough surfaces. As stated by Colony, “Sometimes NVR may be detected simply by visual observation. This is most likely with highly reflective surfaces viewed under strong lighting conditions” (Colony, 1985).
Figure 2-10: Visual Inspection of a James Webb Space Telescope Mirror Segment (NASA/Chris Gunn)
Unfortunately, visual inspection results are subjective based on the experience and ability of observers. Only surfaces that can be seen and physically accessed can be inspected. Additionally, no information is provided on the level of cleanliness. 33
As stated by Huang et al., “Visual inspection of surfaces is easy to perform, but it might not provide accurate or objective information about the levels of cleanliness achieved” (Huang et al., 2015). Although the IEST “visibly clean” criteria is used as an inspection standard in space systems assembly, integration, and test, no empirical data exists on quantifying visible detection of molecular contaminant films on spacecraft surfaces. By the time a molecular film can be visibly detected, it may have already reached a level that can cause significant system performance degradation. This can be problematic for engineering managers, as the indicator method may be providing lagging information that limits response time and the ability to mitigate issues before they become problems for the system. 2.2.5 Proposed Indicator Method: Portable Raman Spectroscopy The Raman scattering effect of molecules was observed by Professor C.V. Raman in 1928, resulting in him being awarded the Nobel prize for this discovery (Carron & Cox, 2010). In Raman spectroscopy, monochromatic light (usually in the form of laser light) illuminates a sample. The scatter of light from the sample results from the molecular make-up of the sample, and depends on the molecular vibrations inherent to each molecule. The resulting spectra can be used to determine the molecular composition of a sample, thus identifying the molecular “fingerprint” (Cooper, Marshall, Jones, Abdelkader, & Wise, 2014). Similar to visual inspection, if a laser beam can reach the sample then a Raman spectrum can be generated (Carron & Cox, 2010). Raman spectroscopy has been utilized in several industries including the detection of “illicit drugs, explosives, chemical weapons and precursors” (Shand, 2008). 34
In the last 10 to 15 years, development of portable spectrometers has made analysis in the field possible, and no longer requires laboratory-scale equipment and environments (Wang, 2015). These systems allow samples to be analyzed in situ, providing instantaneous results in a non-contact manner, similar to visual inspections. This is promising technology for space system integration, where it may be possible to probe spacecraft or witness surfaces and obtain Raman spectra of contaminants throughout the assembly, integration and test phase. This would allow for monitoring of contamination risks of spacecraft surfaces in real time. Studies by Mandrile et al. (2015) have shown positive detection and identification of surface molecular contamination with Raman spectroscopy through experiments with silicone oil and paraffin oil (Mandrile et al., 2015). Building from this groundwork, the case study in this research focuses on five contaminants and five substrates common to space system integration. One of the key decisions in selecting a portable Raman spectrometer for sample analysis is determining the excitation laser wavelength. An example of the Raman spectra of the chemical warfare agent Soman at different excitation wavelengths is shown in Figure 2-11. Shorter wavelengths have higher Raman efficiency, but can cause sample fluorescence (Wang, 2015). This fluorescence effect can be seen for 262 nm and 532 nm in Figure 2-11, where molecular identification peaks disappear in the signal. In some cases, burning and damage can occur upon excitation of the sample with the laser. Longer laser wavelengths, such as 1064 nm, can reduce sample fluorescence (Wang, 2015). For this reason, a 1064 nm excitation wavelength was chosen for this research study. 35
Figure 2-11: Raman Spectra of a Chemical Warfare Agent at Different Excitation Wavelengths (Guicheteau et al., 2011)
2.2.6 The Chemistry of Common Spacecraft Molecular Contaminants The spectral absorbance properties, or the interaction with light for a material at the molecular level, depend on the chemistry of the contaminant (Yokozawa et al., 2012). As will be shown in Chapter 4, experiments in this research were designed to study the effects of different contaminants on spacecraft performance. As such, five contaminants of interest were chosen based on historical test data and past incidents of spacecraft contamination: silicones, hydrocarbons, fluorocarbons, esters, and a glycol-based polymer. Silicones are known as one of the most problematic spacecraft contaminants. Silicones easily spread over surfaces due to low surface tension and readily outgas, causing issues such as adhesive bonding problems, electrical connection failures, and optical performance degradation (Wolfgong, 2011). Silicones also leave oily deposits on surfaces they contact, which creates cross-contamination issues for material and hardware handling (Wolfgong & Wiggins, 2010). Silicones and aromatic hydrocarbons are also known to damage optics in one-micron lasers 36
(Abdeldayem et al., 2006). Data obtained from space systems on orbit also identified common spacecraft contaminants. Outgassing data from the Rosetta Orbiter Spectrometer for Ion and Neutral Analysis (ROSINA) identified hydrocarbon and fluorine contaminants after six years in of space exposure (Hassig et al., 2011). Possible sources of the hydrocarbons included polycarbonates from the spacecraft structure, while fluorine sources included brazing residues, fluorocarbons in the spacecraft structure, and fluorine contained in tapes and lubricants (Hassig et al., 2011). Hydrocarbons, esters, and silicones were identified in the deposited molecular film on the pickoff mirror of the Hubble Space Telescope (Tveekrem et al., 1996). As previously discussed, the LITE, MOLAI, and GLAS on-orbit laser systems all “suffered from performance loss or failure with organic and/or silicone contamination being suspected as possible causes” (Riede et al., 2005). Statistical data was obtained in a study performed by NASA Goddard Spaceflight Center for the most common molecular contaminants found in 167 thermal vacuum test of spaceflight hardware (Chen et al., 1997). Silicones, hydrocarbons, esters, fluorine-containing compounds, and polymers were all identified as frequently-occurring chemical species in deposited molecular contaminant films. The fifth contaminant chosen for this study, a glycol-based polymer, was chosen as a representative polymer from a commercially-available optical cleaning polymer film (Gushwa & Torrie, 2014). 2.2.7 Spacecraft Surface Performance Modeling: STACK Program As discussed in Section 2.2, molecular contamination can cause light 37
absorption, scatter, and obscuration leading to changes in reflective, transmissive, and emissive spacecraft surfaces. To model these effects, the Arnold Engineering Development Center created the modeling program “STACK” in 1994. This model, written in the Interactive Data Language (IDL) replaced the previous contamination modeling program “Calculation of Reflectance and Transmittance,” or CALCRT (Palmer, Williams, Budde, & Bertrand, 1994). In CALCRT, the refractive and absorptive indices (n and k values) of molecular contaminant films were measured from experiments and input into the model to calculate the effects of these films on different optical materials in terms of transmission and reflection losses (Wood et al., 2007). STACK expanded upon CALCRT and allowed a “stack” of contaminant films to be modeled rather than just a single contaminant film. The user defines the substrate material and thickness, defines each contaminant layer and thickness, and inputs the contaminant optical constants (complex refractive index n and k values) for each layer. STACK models the change in reflectivity for reflective surfaces such as mirrors, change in transmission for transmissive optics such as lenses and windows, and the change in emissivity for thermal control surfaces such as radiators. With this tool, engineering managers can predict the effect of molecular contaminant films on the performance of different space system surfaces.
38
Chapter 3: Methods The motivation of this research is to aid engineering managers in catching risks earlier in the systems integration phase before they are realized in a fully integrated or deployed system. This can be accomplished by utilizing leading indicators to detect signals and warnings of risk. However, there are many choices for leading indicators, and to select an appropriate indicator for a system, engineering managers need to know how well the indicator will detect a signal of risk and what it can miss. This research proposes and experimentally executes a methodology to evaluate candidate leading indicators through a systems engineering approach of test requirements definition, indicator characterization, analysis, and prediction of system performance to assist engineering managers in indicator selection. Execution of the methodology is accomplished through a case study of molecular contamination film accumulation on sensitive spacecraft surfaces and the resulting impact to space system performance. 3.1 Proposed Evaluation Methodology Reason’s Swiss Cheese model of risk describes failures occurring after holes in layers of barriers perfectly line up allowing issues and problems to slip through (Paté-Cornell, 2012). This model uses the metaphor of Swiss cheese to describe the barrier layers as the cheese containing inherent holes. Similar to barriers, indicators can contain holes and miss signals and warnings of risk (Aven, 2015a). This process is illustrated in Figure 3-1. Swiss cheese contains holes that are random in size and location. However, the holes of an indicator can be designed, engineered, measured, rated, and improved, 39
much like a filter. An ideal indicator will have holes small enough to not allow key signals of risk to be missed. To create an effective indicator, one must know the size of the hole required. The allowable size of the hole depends on the system in which it is to be used, similar to a filter designed for a specific fluid. This filter metaphor will be presented in parallel with the proposed indicator evaluation methodology to help illustrate the methodology concept.
Figure 3-1: Holes in Risk Indicators Missing Signals of Risk
The proposed methodology for evaluating candidate leading indicator methods is shown in Figure 3-2. The figure is divided into three swim lanes. The first (pink) lane illustrates the evaluation methodology, while the second (blue) lane shows the previously described filter metaphor and third (green) lane shows the specific case study for this research of molecular contamination film accumulation on spacecraft surfaces. The process flow for the methodology is as follows:
Step 1: The first step of the methodology defines the evaluation requirements in terms of the aspect of the risk to be monitored and measured and the system performance metrics of interest. In terms of the filter metaphor, this first step defines the filter and what it should remove. In the case study of spacecraft surface molecular contamination, 40
the risk to be monitored is molecular contamination, and the system performance metrics of interest are changes in reflectivity, transmission, and emissivity.
Step 2: The second step is to identify standard tests to be performed to compare indicator methods. These standard tests should provide data that can be tied to the performance metrics of the system. In the case study, this standard test method is the IEST-STD-CC1246E which defines both molecular contaminant film concentrations and visual inspection criteria.
Step 3: The third step is to identify test subjects for evaluating indicator methods. In the case study, the test subjects (dependent variables) are five substrates of representative spacecraft surfaces and five contaminant materials for molecular contamination.
Step 4: The fourth step of the methodology is to define the pass/fail test criteria for the indicator methods. This defines the size of the hole in the filter and how well it has to perform. For the case study, the pass/fail criteria was detection of each concentration of molecular contaminant film for each substrate. Together, these steps define the characteristic and conditions of the leading indicators.
Step 5: In the fifth step of the methodology, each candidate indicator method is tested and characterized, and quantitative data is gathered. This step tests the filter in the system of interest. For the case study, the current indicator method of visual inspection and proposed indicator 41
method of portable Raman spectroscopy are tested.
Step 6: In the sixth step of the methodology, the results of the testing are analyzed and translated to be codified for comparison. This translation and codification process is key to the methodology, as this step puts all results on the same baseline for comparison. Thus, although indicator methods may have inherently different measurement techniques, the results can be compared on the same level to ensure an equal comparison.
Step 7: The seventh step evaluates the results of the indicator methods in terms of the pass/fail criteria defined in Step 4. The information from Step 6 and Step 7 combined identify and quantify the holes in the filter for each method by determining the risk signal each method misses and by how much.
Step 8: In the eighth step, the results of each indicator method are compared against each other. This compares the holes in the filter for each method. In the case study, the combination of Step 7 and Step 8 define and compare the maximum contaminant film thickness missed for each contaminant and substrate combination for visual inspection and portable Raman spectroscopy.
Step 9: In the ninth and final step of the methodology, the predictive element of leading indicators is incorporated. Predictive modeling and simulation is performed to determine the impact of the results to the system performance. In the filter metaphor, this step determines the 42
impact to the system if something slips through a hole in the filter. For the chosen case study, spacecraft performance modeling is performed with the STACK program as described in Section 2.2.7.
Figure 3-2: Proposed Indicator Evaluation Methodology with Filter Metaphor and Case Study
43
3.2 Hypothesis The research goals led to the development of the following research question: How can multiple indicator methods that use different detection mechanisms and measurement techniques be evaluated and compared equally to aid engineering managers in selecting a leading indicator for their system? To answer this question, testable hypotheses were formulated from the selected case study. The case study evaluates two different indicator methods: the current standard method of visual inspection and the proposed method of portable Raman spectroscopy. Visual inspection relies on human observers to visually detect the presence of contaminant films on surfaces, while portable Raman spectroscopy relies on a laser excitation source and charge coupled device (CCD) detector to identify molecular species on surfaces. Each method was evaluated by detecting 25 concentration levels of five different contaminants on five different spacecraft substrate surfaces. The evaluation criteria or “hole in the filter” of interest in this research was the maximum contaminant film thickness that each method can miss. Testable hypotheses were developed to answer the research question and compare the two different indicator methods of visual inspection and portable Raman spectroscopy. The null hypotheses developed were as follows:
H10: Substrate type will have no significant effect on the maximum film thickness missed for visual inspections.
H20: Contaminant type will have no significant effect on the maximum film thickness missed for visual inspections.
H30: Substrate type will have no significant effect on the maximum 44
film thickness missed for portable Raman spectroscopy.
H40: Contaminant type will have no significant effect on the maximum film thickness missed for portable Raman spectroscopy.
To ensure both indicator methods are compared equally, a two-way analysis of variance (ANOVA) without replication was performed. ANOVA compares the means of a sample from a population when there are more than two levels of a single factor (Montgomery, 2013). A one-way ANOVA considers one variable, while a two-way ANOVA considers two groups or samples as variables. In the case study, the dependent variables were visual inspection and portable Raman spectroscopy, while the independent variables were substrate type and contaminant type. The five maximum film thicknesses missed for each contaminant and substrate combination in each method created a two-way, five level ANOVA. In this two-way ANOVA, the effect of two independent variables on each dependent variable was tested. The results indicate if an effect is statistically significant on the maximum film thickness missed for each dependent variable. For the indicator methods to be compared equally, either both methods have to have no effect from either independent variable, or both methods must have the same effect from the independent variables as shown in Figure 3-3. For example, visual inspection and portable Raman spectroscopy must either have no statistically significant effect due to contaminant type or substrate type or have the same effect to be compared equally. If substrate type has an effect on the maximum film thickness missed for visual inspections while contaminant type has an effect on the maximum film thickness missed for portable Raman spectroscopy, then the indicator methods 45
cannot be compared as it is not a true “apples-to-apples” comparison, and additional research should be performed to understand why an effect is experienced with the indicator method.
IV = Independent Variable DV = Dependent Variable
Figure 3-3: Evaluation of Independent Variable Effects on Dependent Variables for Equal Comparison of Indicator Methods
46
3.3 Evaluation Requirements The first step of the methodology of defining the evaluation requirements is shown in Figure 3-4. The risk of interest to be monitored during spacecraft integration and measured in the evaluation tests is molecular contaminant film accumulation on spacecraft surfaces. The system performance metrics to quantify the degradation due to the presence of different molecular contaminant films on spacecraft surfaces include changes in transmission for optical lenses and windows, changes in reflectivity for mirrors, solar panels, and witness plate optics, and changes in emissivity for thermal control surfaces. These changes in system performance metrics will be modeled based on the experimental results obtained for each indicator method.
Figure 3-4: Step 1 of the Proposed Indicator Evaluation Methodology
3.4 Experimental Materials and Sample Preparation In Steps 2 and 3 of the evaluation methodology shown in Figure 3-5, the standard test to be performed to compare indicator methods is defined with the test subjects. As described in Section 2.2.6, five spacecraft contaminants of interest were chosen for study, and laboratory materials were selected based on representative contaminant chemistries as shown in Table 3-1. As described in Section 2.2 of the literature review, transmissive, reflective, and emissive 47
spacecraft surfaces can experience performance degradation due to molecular contaminant films. Five substrates were chosen to represent common spacecraft surfaces as shown in Table 3-2. The aluminum foil, multilayer insulation (MLI), and aluminized tape were cut into one inch by one inch square coupons. The microscope slides were standard three inch by one inch rectangular slides, and the silicon wafers were two inch diameter wafers.
Figure 3-5: Steps 2 and 3 of the Proposed Indicator Evaluation Methodology
Table 3-1: Experimental Materials for Spacecraft Contaminants Representative Contaminant Chemistry Silicone Hydrocarbon Fluorocarbon Ester Glycol polymer
Experimental Material Dow Corning® 704 diffusion pump fluid Kurt J. Lesker® Company KJLSS 15-1920-70 vacuum pump fluid Miller-Stephenson MS-122AD release agent Loctite® 242™ Threadlocker methacrylate ester sealant Photonic Cleaning Technologies, LLC First ContactTM polymer
48
Table 3-2: Experimental Materials for Spacecraft Surface Substrates Representative Substrate Aluminum foil
Experimental Material Reynolds Wrap®, dull side
Glass microscope slides
VWR® Micro Slides
Multilayer insulation (MLI) solar reflector material
DuPont™ Kapton® polyimide film Sheldahl® First Surface Aluminum Coated Polyimide Tape University Wafer, 50.8 mm diameter N-type phosphorousdoped single side polished
Aluminized Kapton® tape
Silicon wafers
Spacecraft Surface Metallic surfaces Transmissive optics such as lenses Thermal blankets Radiators
Reflective optics, solar panels, witness plate optics
Molecular contaminant films were prepared to IEST-STD-CC1246E NVR levels from mixture solutions of isopropyl alcohol (IPA) and the contaminants from Table 3-1. Molecular contaminant films were deposited on substrates by depositing a single droplet of prepared NVR solution on a pre-weighed substrate sample. The diameter of the droplet was measured with calipers with a measurement accuracy of +/- 0.001 inch after the droplet had spread over the substrate surface but before evaporation of the IPA took place. The mass of the NVR droplet was also recorded prior to IPA evaporation. NVR film thickness was calculated from the droplet area, NVR concentration, and solution density. This calculation requires an assumption of a uniform film deposited on the substrate surface. In reality, the NVR may exist as a non-uniform film, or a conglomeration of droplets. However, by assuming a uniform film for the purposes of determining the film thickness, this provides a conservative worst-case metric of the NVR film that the detection methods could miss. A blank substrate sample with no NVR deposited was included for each contaminant. Additionally, a control sample made from an IPA droplet deposited on 49
a substrate sample was included to account for any impurities present in the IPA. Finally, a substrate sample with a droplet of 1.0 mg/ml solution was created as the most concentrated sample. A total of 27 samples for each contaminant-substrate combination were coded and randomized for this study: 24 NVR levels defined by IEST-STD-CC1246E, a 1.0 mg/ml concentrated sample, a blank substrate sample, and an IPA control sample. This created a total of 675 samples for this study. Laboratory materials for solution preparation included American Chemical Society (ACS) grade IPA, and glass bottles that were pre-cleaned with deionized water, rinsed with IPA, and then thoroughly dried. Substrates were cleaned with IPA and Anticon® Gold HeavyWeight™ cleanroom wipes prior to deposition. All sample preparations took place in an ISO Class 7 cleanroom environment certified to ISO 14644-1 (ISO, 2015) to minimize adding contamination to the samples from personnel or from the laboratory environment and negatively impacting measurements. Personnel gowning protocol for all sample preparation, handling, and visual inspections included bouffant cap, face mask, safety glasses, frock, powder-free latex gloves, and shoe covers. 3.5 Experimental Measurements The experimental measurements of the case study executed Steps 4 and 5 of the indicator evaluation methodology as shown in Figure 3-6. During the design of experiments, the pass/fail criteria for evaluation tests were defined. In this case study, the pass/fail criteria was detection or no detection of the molecular contaminant film on the substrate for each sample. After samples were prepared, samples underwent visual inspection tests followed by portable Raman spectroscopy 50
tests. Visual inspections were performed first in case any measurements made by portable Raman spectroscopy resulted in physical or visual changes to the samples due to any interactions with the excitation laser. This allowed data to be collected on the same samples by the two different techniques.
Figure 3-6: Steps 4 and 5 of the Proposed Indicator Evaluation Methodology
3.5.1 Visual Inspections Visual inspection experiments were performed to VC-0.5-1000 criteria per IEST-STD-CC1246E. In this standard, this criteria level refers to “visually clean” (VC), a viewing distance of 0.25 to 0.5 meters (6 to 18 inches), and lighting intensity of 1000 lumens/m2 (100 ft-candles). A surface is considered to be visually clean to this level if it is free of visible contamination when viewed under these conditions. Samples were placed face-up on a grid of cleanroom paper with grid squares identifying sample numbers. The sample grids were placed on cleanroom tables in an ISO Class 7 cleanroom environment, as shown in Figure 3-7. To control the viewing distance from the samples, vertical stands were placed on the tables with marks delineating the 6 inch and 18 inch distances from the samples. Standard cleanroom room lighting was used to illuminate the samples. The minimum 100 ft-candle light intensity was verified by measuring 51
the lighting with a light meter placed on the table surface with the samples. Visual inspections were performed by five different observers that volunteered to participate in the study. These observers were engineers and technicians skilled and experienced in spacecraft integration and test at NASA Langley Research Center and had performed hardware inspections as part of their regular job functions. Observers followed cleanroom gowning protocol for an ISO Class 7 cleanroom as described in Section 3.4. Prior to the inspections, each observer was provided the same instructions: to review the IEST-STD-CC1246E inspection criteria for VC-0.5-1000, to observe each sample with their eye level between the 6 inch and 18 inch marks on the vertical stand, to inspect the surfaces as they normally would inspect spaceflight hardware, and to avoid touching or handling the samples. Each observer was free to control their viewing angle to the surface by moving around the cleanroom tables, and viewed surfaces with the unaided eye (corrected vision was allowed). Each observer performed visual inspections in separate sessions with only the lead experimenter present to record results. Observers verbally called out a Yes/No response for each sample to indicate if they detected or did not detect the presence of a molecular film, and the lead experimenter recorded results on a data sheet. This allowed the observers to focus on performing the visual inspections without having to shift their viewing gaze away from the samples and interrupt the inspections to record data.
52
Figure 3-7: Contamination Samples Prepared for Visual Inspection Experiments (NASA/Elaine Seasly)
3.5.2 Portable Raman Spectroscopy Portable Raman spectroscopy measurements involve the use of a Class 3b laser source, which requires a laser safety certified laboratory. After visual inspections were complete, samples were placed in clean petri dishes, bagged, and transported to a laser safety certified ISO Class 5 cleanroom environment. Personnel gowning protocol for the ISO Class 5 cleanroom included bouffant cap and face mask under a cleanroom hood, laser goggles for a 1064nm laser, coveralls, shoe covers under cleanroom boots, and powder-free nitrile gloves. Sample measurements were taken with a commercial portable Raman spectrometer purchased from B&W Tek. The instrument used was an i-Raman® EX (Model No. BWS485-1064S-05) with a 1064 nm excitation laser. This system is comprised of a spectrometer box containing the excitation laser and detector, a fiber optic cable extending from the box and connecting to a handheld probe, and an 53
adjustable XYZ stage for positioning the probe above the samples as shown in Figure 3-8. The spectrometer connects directly to a laptop for data acquisition. The system requires a 5.4 millimeter working distance from the end of the probe to the sample surface, so a distance regulator provided by B&W Tek was installed at the end of the probe to maintain this distance.
Figure 3-8: Experimental Setup of Portable Raman Spectrometer System (NASA/Elaine Seasly)
An acetaminophen tablet was used as a Raman standard per ASTM E1840 (ASTM, 2014) to verify proper laser operation before sample scanning commenced each day. An example screen shot of this standard scan from the spectrometer’s BWSpecTM software is shown in Figure 3-9. Scanning conditions for each sample substrate was setup by scanning a substrate blank (Sample No. 27 in all cases) to determine laser power and scan duration settings. By varying these conditions, the best signal with the lowest noise could be obtained, signal fluorescence could be minimized, and sample burning or damage could be avoided. A droplet of 54
contaminant (no IPA dilution) was deposited on the blank substrate and scanned to obtain the relevant identifying peaks for the contaminant. This way, the contaminant could be characterized to determine where identifying peaks occurred on the Raman shift or wavelength spectrum. A dark scan was taken with the probe shutter closed prior to each sample scan to subtract out laser system noise. Each individual sample of the contaminant-substrate combinations was then scanned. The center of each individual sample droplet was scanned with the probe, similar to how actual spacecraft surfaces would be scanned during systems integration.
Figure 3-9: Raman Spectra of Acetaminophen Tablet Standard
3.6 Data Collection & Analysis After test data is collected from each indicator method, Step 6 of the evaluation methodology is to analyze and translate the results to codify them for comparison as shown in Figure 3-10. This is followed by Step 7, to evaluate the results of the indicator methods in terms of the pass/fail criteria as defined previously in Step 4. The metric for comparison of indicator methods in this case study was the maximum contaminant film thickness missed by each indicator method for each contaminant55
substrate combination. This quantifies the “hole in the filter” for each method and how much signal of risk each method allows to slip through the system. At this point, the 2-way ANOVA testing is performed on the results to determine if the results can be equally compared against each other. If the results of the ANOVA testing show both methods do not experience an effect from one or both of the independent variables, or experience the same effect, then the results of the methods can be compared against each other in Step 8 as it is proven to be an equal “applesto-apples” comparison. If, not, a different metric can be chosen and the ANOVA testing repeated to ensure the metric chosen allows for an equal comparison between indicator methods.
Figure 3-10: Steps 6, 7 and 8 of the Proposed Indicator Evaluation Methodology
3.6.1 Visual Inspections The Yes/No data collected from the visual inspections for each observer were codified as a “1” for a positive detection of a contaminant film, and a “0” for no film detection. This information was coded in an Excel spreadsheet and the results from all five observers for each contaminant-substrate combination were summed and then sorted in descending order by film thickness. The highest film thicknesses 56
missed by at least one observer was flagged and recorded as the maximum film thickness missed. An example of this process is shown in Table 3-3 for a fluorocarbon film on an aluminum Kapton® tape substrate. In this case, an NVR film thickness of 147.2 nm was the highest film thickness missed for this contaminant-substrate combination. In this case study, the worst-case film thickness missed was chosen as the metric, regardless of the number of observers that missed a positive detection of a sample. Additional metrics based on the data obtained may be analyzed in future studies.
57
Table 3-3: Determination of Maximum Film Thickness Missed in Visual Inspections for Fluorocarbon Film on Aluminum Kapton® Tape
Sample Designation
NVR Thickness (nm)
Number Observers that Positiviely Detected
Substrate
NVR
Aluminum Kapton® Tape
Fluorocarbon
F26
147.23
3
Aluminum Kapton® Tape
Fluorocarbon
F12
43.53
3
Aluminum Kapton® Tape
Fluorocarbon
F18
21.22
3
Aluminum Kapton® Tape
Fluorocarbon
F17
14.37
4
Aluminum Kapton® Tape
Fluorocarbon
F5
10.01
3
Aluminum Kapton® Tape
Fluorocarbon
F21
6.05
2
Aluminum Kapton® Tape
Fluorocarbon
F3
5.58
2
Aluminum Kapton® Tape
Fluorocarbon
F24
3.70
2
Aluminum Kapton® Tape
Fluorocarbon
F2
1.93
2
Aluminum Kapton® Tape
Fluorocarbon
F1
0.94
2
Aluminum Kapton® Tape
Fluorocarbon
F4
0.59
4
Aluminum Kapton® Tape
Fluorocarbon
F8
0.32
4
Aluminum Kapton® Tape
Fluorocarbon
F11
0.14
3
Aluminum Kapton® Tape
Fluorocarbon
F9
7.06E-02
3
Aluminum Kapton® Tape
Fluorocarbon
F7
2.56E-02
3
Aluminum Kapton® Tape
Fluorocarbon
F15
1.49E-02
3
Aluminum Kapton® Tape
Fluorocarbon
F13
6.71E-03
4
Aluminum Kapton® Tape
Fluorocarbon
F10
3.00E-03
4
Aluminum Kapton® Tape
Fluorocarbon
F14
1.14E-03
4
Aluminum Kapton® Tape
Fluorocarbon
F23
7.67E-04
2
Aluminum Kapton® Tape
Fluorocarbon
F20
3.20E-04
4
Aluminum Kapton® Tape
Fluorocarbon
F16
1.34E-04
4
Aluminum Kapton® Tape
Fluorocarbon
F22
2.65E-05
4
Aluminum Kapton® Tape
Fluorocarbon
F19
1.64E-05
4
Aluminum Kapton® Tape
Fluorocarbon
F6
1.36E-05
3
Aluminum Kapton® Tape
Fluorocarbon
F25
0
4
Aluminum Kapton® Tape
Fluorocarbon
F27
0
3
3.6.2 Portable Raman Spectroscopy Once the Raman spectra of a material is obtained, signal peaks must be identified based on intensity and wavelength to chemically identify the constituents. It can be challenging for any analyst to separate and identify signal peaks from system noise when reviewing raw spectra. Signal margin, threshold and three notch 58
regions can be defined by the user to define signal peaks from noise and subtract known sources of noise from the system. A dark scan was taken between each sample, so the detector noise was subtracted from each sample signature based on this scan. An example of this signal processing is provided in Figure 3-11 for a hydrocarbon contaminant droplet on an aluminum foil substrate. In Figure 3-11(a), the raw, pre-processed Raman spectrum is given while the post-processed spectra is given in Figure 3-11(b). After the signal processing is applied, the noise floor is forced to zero, peaks that may have been hidden in the noise begin to emerge, and prominent peaks are intensified. Figure 3-12 shows the Raman spectra for each contaminant droplet on aluminum foil after signal processing. This figure illustrates how each contaminant chemistry produces a unique Raman signal response by the peak position in the wavelength spectrum and the peak intensity. As contaminant concentration and film thickness decreases, peak intensity will also decrease until the peaks are no longer present and the contaminant film can no longer be detected.
59
Figure 3-11: Signal Processing of Raman Spectra of Hydrocarbon Droplet on Foil Substrate. (a) Pre-processed Raman Spectra. (b) Post-processed Raman Spectra.
60
Figure 3-12: Post-processed Raman Spectra for Contaminant Droplets on Foil Substrate. (a) Silicone (b) Hydrocarbon (c) Fluorocarbon (d) Glycol (e) Ester.
Peak discrimination and correlation algorithms are commonly used to aid analysts in identifying peaks in spectral data (Carron & Cox, 2010). Such algorithms 61
can help remove ambiguity in interpreting spectra. For this research, a custom peakfinding algorithm was developed to determine the point when peaks could no longer be detected as the molecular contaminant film thickness decreased. The algorithm considers each individual signal value over the scanned spectral range. If the individual signal value is above a user-defined signal margin, above the minimum defined threshold, and does not appear in a substrate notch range defined by the user, then the signal value is flagged as a peak and assigned a value of “1” for positive detection. If any of these criteria are not met, the signal value is not a peak and assigned a peak value of zero. This codifies the Raman results in a similar Yes/No logic as the visual inspection method. The individual peak values of 0/1 are then summed over the entire spectral range. If no peaks over the spectral range could be identified, the film thickness was determined to be a thickness that was missed by the Raman detection method. The worst-case film thickness for a contaminant-substrate combination was then defined as the highest film thickness where no peaks over the spectral range could be identified. An example of this codification, evaluation, and determination for portable Raman spectroscopy is provided in Table 3-4 for a fluorocarbon contaminant film on an aluminum Kapton® tape substrate. Note the results in Table 3-4 are codified and can be compared to the results for visual inspections presented previously in Table 3-3. Final results for these maximum film thickness values and the resulting ANOVA analysis performed on these values will be presented in Chapter 4 and discussed further in Chapter 5.
62
Table 3-4: Determination of Maximum Film Thickness Missed with Portable Raman Spectroscopy for Fluorocarbon Film on Aluminum Kapton® Tape Sample Designation
NVR Thickness (nm)
Raman Result 1 = Positive Detection 0 = Negative Detection
Substrate
NVR
Aluminum Kapton® Tape
Fluorocarbon
F26
147.23
1
Aluminum Kapton® Tape
Fluorocarbon
F12
43.53
0
Aluminum Kapton® Tape
Fluorocarbon
F18
21.22
0
Aluminum Kapton® Tape
Fluorocarbon
F17
14.37
0
Aluminum Kapton® Tape
Fluorocarbon
F5
10.01
0
Aluminum Kapton® Tape
Fluorocarbon
F21
6.05
0
Aluminum Kapton® Tape
Fluorocarbon
F3
5.58
0
Aluminum Kapton® Tape
Fluorocarbon
F24
3.70
0
Aluminum Kapton® Tape
Fluorocarbon
F2
1.93
0
Aluminum Kapton® Tape
Fluorocarbon
F1
0.94
0
Aluminum Kapton® Tape
Fluorocarbon
F4
0.59
0
Aluminum Kapton® Tape
Fluorocarbon
F8
0.32
0
Aluminum Kapton® Tape
Fluorocarbon
F11
0.14
0
Aluminum Kapton® Tape
Fluorocarbon
F9
7.06E-02
0
Aluminum Kapton® Tape
Fluorocarbon
F7
2.56E-02
1
Aluminum Kapton® Tape
Fluorocarbon
F15
1.49E-02
0
Aluminum Kapton® Tape
Fluorocarbon
F13
6.71E-03
0
Aluminum Kapton® Tape
Fluorocarbon
F10
3.00E-03
0
Aluminum Kapton® Tape
Fluorocarbon
F14
1.14E-03
0
Aluminum Kapton® Tape
Fluorocarbon
F23
7.67E-04
0
Aluminum Kapton® Tape
Fluorocarbon
F20
3.20E-04
0
Aluminum Kapton® Tape
Fluorocarbon
F16
1.34E-04
0
Aluminum Kapton® Tape
Fluorocarbon
F22
1.64E-05
0
Aluminum Kapton® Tape
Fluorocarbon
F19
2.65E-05
0
Aluminum Kapton® Tape
Fluorocarbon
F6
1.36E-05
0
Aluminum Kapton® Tape
Fluorocarbon
F25
0
0
Aluminum Kapton® Tape
Fluorocarbon
F27
0
0
3.6.3 Data Transformation and ANOVA Assumptions One of the assumptions for ANOVA testing is that the data be normally distributed. A distribution identification analysis was performed in the statistical analysis software program Minitab 18 for both the visual inspection and portable 63
Raman spectroscopy results for maximum film thickness missed. The original data sets for visual inspections and portable Raman spectroscopy were not normal as assessed by the Anderson-Darling test. Both data sets had a p-value of less than 0.005, indicating a poor fit to a normal distribution (alpha = 0.05). The best fit for both data sets was a log-logistic fit, and a Johnson Transformation was performed to normalize the data. The results of the Johnson Transformation for the visual inspection data is shown in Figure 3-13, and the transformed Raman data is shown in Figure 3-14. The Anderson-Darling p-value for the transformed visual data was 0.879 and for the transformed Raman data was 0.247, indicating the transformed data now follows a normal distribution (alpha = 0.05).
Figure 3-13: Johnson Transformation of the Visual Inspection Data
64
Figure 3-14: Johnson Transformation of the Portable Raman Spectroscopy Data
The transformed data for both visual inspections and Raman spectroscopy were checked against all of the assumptions for ANOVA as shown in Table 3-5 (alpha = 0.05 for Grubbs’, Anderson-Darling, and Levene’s tests). All assumptions for ANOVA were satisfied by the transformed data.
65
Table 3-5: Satisfaction of ANOVA Assumptions with Transformed Data ANOVA Assumptions (“Laerd Statistics,” 2015) 1) One dependent variable that is measured at the continuous level. 2) Two independent variables consisting of categorical and independent groups.
Transformed Visual Inspection Data
Max film thickness missed for Visual is measured and continuous.
Contaminant type and substrate type are the independent variables.
Contaminants and substrates 3) Independence of observations.
4) No significant outliers.
5) Dependent variable should be approximately normally distributed.
6) Homogeneity of variances: Variance of the dependent variable should be equal.
are independent (no contaminant appears in the substrate group and vice versa).
Grubbs’ test: p-value = 0.537 No outlier at the 5% level of significance.
Anderson-Darling test: pvalue = 0.879 Transformed data is normally distributed.
Levene’s test: Visual vs. Substrate p-value = 0.307. Visual vs. Contaminant pvalue = 0.785 All variances are equal.
Transformed Portable Raman Spectroscopy Data
Max film thickness missed for Raman is measured and continuous.
Contaminant type and substrate type are the independent variables.
Contaminants and substrates are independent (no contaminant appears in the substrate group and vice versa).
Grubbs’ test: p-value = 1.000 No outlier at the 5% level of significance.
Anderson-Darling test: pvalue = 0.247 Transformed data is normally distributed.
Levene’s test: Raman vs. Substrate p-value = 0.171. Raman vs. Contaminant pvalue = 0.381 All variances are equal.
3.7 Performance Modeling of Molecular Contamination Effects The final step of the evaluation methodology determines the impact of the results on system performance through predictive modeling and simulation as shown in Step 9 of Figure 3-15. This step incorporates the predictive behavior of leading indicators and translates the analysis results so the engineering manager can understand the impact to system performance. Returning to the filter metaphor, this step determines the impact of allowing a signal of risk to slip through the hole in the filter. In the case study explored, the results of maximum film thickness missed for each contaminant-substrate combination for visual inspections and portable Raman 66
spectroscopy were modeled with the STACK program to determine the impact to spacecraft performance as described in Section 2.2.7. To ensure the substrate was considered as a base material (no transmission of light through the material), the substrate film thickness was set to 25 micrometers, or 10 times greater than any deposited contaminant film thickness. Light was modeled at a 90 degree angle of incidence for transmissive and reflective surfaces to simulate the highest possible energy incidence on optics. Light was modeled at a 45 degree angle of incidence for emissive surfaces to simulate average spacecraft orientation during orbit for radiators and solar cells. The reflectivity of polished bare aluminum from 300 nm to 2500 nm was modeled to check for correct model functionality, and results matched those of published values from industry (“Optical Reference Laboratory Reflectance Standards,” n.d.). Final performance model results will be presented in Chapter 4 and discussed further in Chapter 5.
Figure 3-15: Step 9 of the Proposed Indicator Evaluation Methodology
67
Chapter 4: Results Execution of the indicator evaluation methodology presented in Chapter 3 produced results divided into three major categories: experimental results of maximum contaminant film thickness missed for each indicator detection method, hypothesis test results to determine if substrate type or contaminant type had an effect on the maximum contaminant film thickness missed for each indication detection method, and performance model results to illustrate the effect of the maximum contaminant film thickness missed for each indicator detection method on spacecraft system performance. 4.1 Experimental Results The experimental results of maximum contaminant film thickness missed for each contaminant-substrate combination for visual inspections is provided in Table 4-1 and portable Raman spectroscopy is provided in Table 4-2. Values in each table are highlighted to compare results between visual inspections and portable Raman spectroscopy. A higher film thickness missed (larger hole in the filter) is highlighted in pink, a lower film thickness missed (smaller hole in the filter) is highlighted in green, and equivalent thicknesses missed between the two methods are in gray. For example, for a silicone film detected on aluminum foil, visual inspection missed a maximum of 1.58 nm while portable Raman spectroscopy missed a maximum film thickness of 18.50 nm. Because visual inspections missed a smaller film thickness, the visual inspection value is highlighted in green while the portable Raman spectroscopy value is highlighted in pink. For a fluorocarbon film detected on aluminum foil, both visual inspection and portable Raman spectroscopy missed a 68
maximum film thickness of 16.78 nm, so that value is highlighted in gray. Table 4-1: Maximum Contaminant Film Thickness Missed for Visual Inspections Contaminant
Silicone Film Thickness (nm) 1.58 1.12 3.15 2.07
Hydrocarbon Film Thickness (nm) 3.51 1.98 8.42 0.96
Fluorocarbon Film Thickness (nm) 16.78 7.00 8.59 147.23
Glycol Film Thickness (nm) 32.62 1.66 23.58 6.88
Ester Film Thickness (nm) 8.94 3.06 1.60 0.03
Substrate
Aluminum Foil Glass Slide Solar Reflector MLI Aluminum Kapton® Tape Silicon Wafer 2.31 1.74 22.25 18.91 5.65 Shaded cells indicate values compared between visual inspections and portable Raman spectroscopy. Pink = higher level, green = lower level, gray = equivalent level.
Table 4-2: Maximum Contaminant Film Thickness Missed for Portable Raman Spectroscopy Contaminant
Silicone Film Thickness (nm) 18.50 19.16 12.36 2.80
Hydrocarbon Film Thickness (nm) 9.67 43.66 230.14 204.47
Fluorocarbon Film Thickness (nm) 16.78 26.26 126.23 43.53
Glycol Film Thickness (nm) 4.12 26.70 2.70 200.63
Ester Film Thickness (nm) 3.26 28.73 14.41 39.00
Substrate
Aluminum Foil Glass Slide Solar Reflector MLI Aluminum Kapton® Tape Silicon Wafer 19.25 9.72 22.25 18.91 22.34 Shaded cells indicate values compared between visual inspections and portable Raman spectroscopy. Pink = higher level, green = lower level, gray = equivalent level.
It should be noted that a problem occurred with the portable Raman spectrometer during the experimental testing. An issue with detector functionality was discovered during one of the daily checks with the acetaminophen standard. At this point in the experiment, approximately half of the samples had been scanned. The spectrometer equipment was shipped back to the manufacturer for evaluation and repair. The detector was completely replaced by the manufacturer, and upon receipt of the repaired unit, an improvement in sensitivity and detection signal was observed. Therefore, all contaminant and substrate samples that had been scanned were re-scanned so that all samples would be measured with the same device. All data obtained and reported herein originated from the repaired and improved 69
portable Raman spectrometer. 4.2 Hypothesis Test Results To review, the null hypotheses were presented in Section 3.2 and are re-stated below:
H10: Substrate type will have no significant effect on the maximum film thickness missed for visual inspections.
H20: Contaminant type will have no significant effect on the maximum film thickness missed for visual inspections.
H30: Substrate type will have no significant effect on the maximum film thickness missed for portable Raman spectroscopy.
H40: Contaminant type will have no significant effect on the maximum film thickness missed for portable Raman spectroscopy.
The null hypotheses were developed to determine if the metric of maximum film thickness missed experienced a significant effect due to contaminant type or substrate type for each indicator detection method evaluated (visual inspections and portable Raman spectroscopy). As described in Sections 3.2 and 3.6.3, a two-way ANOVA without replication was performed to determine if the indicator methods can be compared equally. The results of this statistical analysis are provided in Table 4-3. Both substrate type and contaminant type do not have a statistically significant effect on the maximum film thickness missed for portable Raman spectroscopy, but contaminant type did produce a statistically significant effect on the maximum film thickness missed for visual inspections as illustrated in Figure 4-1. Based on these results, an equal comparison cannot be made between the two 70
indicator detection methods when reviewing the results of Table 4-1 and Table 4-2 without exploring the effect of contaminant type on visual inspections further. Table 4-3: Hypothesis Test Results
Visual Inspections Portable Raman Spectroscopy
Effect Substrate (H10) Contaminant (H20) Substrate (H30) Contaminant (H40)
F4,24
p-value
1.28 4.81 1.66 0.92
0.320 0.010 0.208 0.478
Reject null? ( = 0.05) No Yes No No
IV = Independent Variable DV = Dependent Variable
Figure 4-1: Evaluation Scenario of Independent Variable Effects on Dependent Variables for Hypothesis Test Results
4.3 Performance Model Results Although the results of Table 4-1 and Table 4-2 were obtained and a statistically significant effect of contaminant type was found for the maximum film thickness missed by visual inspections as a result of the ANOVA analysis, this information alone may not be enough for an engineering manager to select the appropriate indicator method for their system. For this case study, the impact of the results on system performance needed to be quantified through modeling the effects of contaminant films on the transmission, reflectivity, and emissivity of space system surfaces. The performance impact effect on each type of space system 71
surface (substrate) is provided in Table 4-4. Table 4-4: Contaminant Performance Impact on Space System Surfaces Representative Substrate Aluminum foil Glass microscope slides Multilayer insulation (MLI) solar reflector material Aluminized Kapton® tape Silicon wafers
Spacecraft Surface Metallic surfaces
Performance Impact Change in reflectivity
Transmissive optics such as lenses Thermal blankets
Change in transmission
Radiators Reflective optics, solar panels, witness plate optics
Change in emissivity Change in reflectivity
Change in emissivity
The impact of maximum film thickness missed for visual inspections and portable Raman spectroscopy were modeled in the STACK program to obtain the changes in transmission, reflectivity, and emissivity for each contaminant-substrate combination. Example results for each performance impact effect are provided in Figure 4-2. Changes in reflectivity for the maximum film thickness missed for silicone and hydrocarbon films on aluminum foil are provided in Figure 4-2(a) and Figure 4-2(b), respectively. In Figure 4-2(a), a maximum of 1.58 nm silicone film missed for visual inspections was modeled against an 18.5 nm film missed for portable Raman spectroscopy. Very little change in reflectivity of aluminum foil was observed over the modeled wavelength region for the 1.58 nm film. However, a potentially significant change in reflectivity could result for the 18.5 nm film depending on the wavelength of incident light. For example, a 0.65% change could occur at 827 nm for an 18.5 nm thick silicone film on aluminum foil. From approximately 1500 nm to 2480 nm, a 0.11% change could occur. The impact to system performance depends on the operational wavelength region of interest to an engineering manager. These results can, in turn, be incorporated into full system
72
performance models to determine the overall impact to the system. A different performance impact is observed for the same substrate but different contaminant as shown in Figure 4-2(b) for a hydrocarbon film on aluminum foil. For this contaminant-substrate combination, a 3.51 nm maximum film thickness missed for visual inspections was modeled against a 9.67 nm maximum film thickness missed for portable Raman spectroscopy. Again, the impact to performance depends on the operational wavelength region of interest to an engineering manager. The hydrocarbon results for visual and Raman presented in Figure 4-2(b) trended similarly to each other, with visual decreasing from a maximum reflectivity change of 0.05% and Raman decreasing from a maximum reflectivity change of 0.15%. Example results for transmission changes on a glass slide for maximum silicone film thicknesses missed and hydrocarbon film thicknesses missed for each indicator method are shown in Figure 4-2(c) and Figure 4-2(d), respectively. Example results for emissivity changes on a solar reflector MLI surface for maximum silicone film thicknesses missed and hydrocarbon film thicknesses missed for each indicator method are shown in Figure 4-2(e) and Figure 4-2(f), respectively. All performance model results from the STACK program are provided in Appendix A. An example of the impact of the maximum contaminant missed for each contaminant on a single substrate for a given method is provided in Figure 4-3. In this example, the effect of missing the maximum film thickness of each contaminant on a silicon wafer mirror for visual inspections is shown. Most contaminants trend together. However, the visual inspection method missing a 22.25 nm fluorocarbon film produced a noticeable effect. Silicon wafers are used as witness surfaces during 73
the integration process, and a typical requirement used in the spacecraft industry is no more than 1% change in reflectivity on the witness mirror throughout integration. Depending on the wavelength region of interest, missing a 22.25 nm fluorocarbon film during visual inspections could result in a significant performance impact. In the visible wavelength region of 390 to 700 nm, the fluorocarbon film could cause a 0.7% to 1.9% change in mirror reflectivity, which may exceed the typical requirement of no more than 1% change in witness mirror reflectivity. This may be significant and result in performance issues for systems that operate in this region, such as laser systems that operate at 488 nm or 532 nm.
74
Figure 4-2: Example Spacecraft Performance Modeling Results. (a) Silicone on Aluminum Foil (b) Hydrocarbon on Aluminum Foil (c) Silicone on Glass (d) Hydrocarbon on Glass (e) Silicone on Solar Reflector MLI (f) Hydrocarbon on Solar Reflector MLI.
75
Figure 4-3: Silicon Wafer Mirror Reflectivity Performance Impact Model Results for Visual Inspection
76
Chapter 5: Discussion, Conclusions & Recommendations Faced with several candidate indicator methods, engineering managers need to know how well a potential indicator will perform for their system. This body of work focused on answering the following research question: How can multiple indicator methods that use different detection mechanisms and measurement techniques be evaluated and compared equally to aid engineering managers in selecting a leading indicator for their system? To answer this question, an evaluation methodology to characterize and compare proposed leading indicators for a system was developed. The methodology was executed through a case study of molecular contaminant films on space system surfaces. The case study evaluated two different indicator methods: the current standard method of visual inspection and the proposed method of portable Raman spectroscopy. Data was collected on 675 samples of five contaminants and five substrates. Results for each indicator method were quantified in terms of maximum contaminant film thickness missed by each method. From this data, testable hypotheses were formulated and statistically evaluated to determine if an equal comparison between the two methods could be made. Performance modeling determined the impact of each method not detecting a contaminant film on space system performance for each substrate in terms of changes in transmission, reflectivity, and emissivity. 5.1 Discussion The maximum film thickness missed for each method was quantified and presented in Table 4-1 and Table 4-2. The goal is to have a smaller “hole,” or a 77
smaller film thickness missed by each method. Reviewing the tables of results, the visual inspection method missed a lower film thickness than the portable Raman spectroscopy method for the majority of contaminant-substrate combinations. However, as a result of the statistical testing of the hypotheses, it was discovered that contaminant type had a statistically significant effect on the maximum film thickness missed for visual inspections. Contaminant type and substrate type did not have a significant effect on maximum film thickness missed for portable Raman spectroscopy. This may explain why a difference between the two methods was observed and an overall lower film thickness was missed for visual inspections over portable Raman spectroscopy. Additional research is required to explore this observation and effect further. However, for an engineering manager considering both methods, it is important for them to know and understand that this effect exists. The performance model results show the impact the presence of a contaminant film has on space system surface, and how an indicator method missing the detection of a film will impact the system. This is valuable information for an engineering manager to consider for their specific system. Depending on the performance requirements, performance budget, and margin allocated for the system, these results may be significant to the overall performance of the system. The performance model results in combination with the hypothesis test results provide additional knowledge to an engineering manager beyond the experimental results alone when making a decision on selecting and deploying leading indicator methods.
78
5.2 Conclusions The case study was successful in executing the developed evaluation methodology for characterizing and comparing candidate leading indicator methods with two different measurement and detection techniques. A significant effect of contaminant type was discovered for the maximum film thickness missed for visual inspections that was not present for portable Raman spectroscopy. The subsequent performance model results converted the metrics of maximum film thickness missed by each indicator method into meaningful information for an engineering manager to consider when evaluating each method. Overall, this process translated measurements made by each indicator method into information an engineering manager can then consider and utilize in creating mitigation and response action plans for risk. Once the evaluation methodology has been executed and baseline information is obtained, the process can be repeated for different metrics for the indicator methods, or to compare additional candidate indicators. The initial evaluation is a foundation from which additional studies, considerations, and trades can extend. By executing the evaluation methodology for a specific case study, the utility and benefits of the methodology are illustrated in an example system context. 5.3 Implications of the Study for the Spacecraft Industry Currently, visual inspection is the standard method for detecting molecular contaminant films on spacecraft surfaces during integration and testing, where a surface is considered to be “visibly clean” if no film is observed on a surface when viewed under the lighting and distance conditions specified by IEST-STD-CC1246. Despite being used as a standard, no empirical data previously existed on how well 79
this method did or did not perform in detecting different contaminant films on different surfaces. As a result of this study, the amount this method could miss in terms of maximum film thickness for different contaminants on different spacecraft surfaces has been determined, and the corresponding potential impact to space system performance modeled. In some cases, “visibly clean” may not be clean if a molecular film that cannot be visibly detected has built up to a level to impact system performance. As a result, this may impact how contamination control engineers set hardware cleanliness requirements and when inspections are performed. Ideally, performance modeling should take place up front in the requirements development process to determine system sensitivity to certain contaminants. Based on this information, the contamination control engineer may set more stringent requirements for specific contaminant chemistries, and be able to determine which contaminants may be detected visually. This is a shift from conventional molecular contamination requirements development, which may not take contaminant chemistry or surface type into account, and only considers a single maximum contaminant film thickness allowed for the system. For contaminants that have a lower probability of visual detection, the contamination control engineer may need to increase the number of witness plates deployed and analyze the plates more frequently to quantitatively determine molecular contaminant film levels for contaminants of concern. This study has also illustrated how no detection method is perfect, and each method contains inherent holes that can miss detecting molecular contaminant films. The key is to recognize these holes exist, and to deploy detection methods in a 80
manner to keep the holes from lining up, thus increasing the probability of catching a signal of contamination risk. In the specific case study performed in this research, it is recommended that spacecraft surfaces continue to be visually inspected, but also followed up with portable Raman spectroscopy. There is value in immediately identifying any contaminant films that may be present, or determining if a particular contaminant of concern is not present. For example, silicones are known to be problematic for spacecraft surfaces. A surface can be visibly inspected and then scanned with the portable Raman spectrometer to determine if there is any chemical signature of silicone present. This provides immediate information while more timeconsuming processes, such as witness plate analysis, are performed in parallel. This increases the probability of providing the engineering manager with additional information that can be used in the decision making process. In the case of silicone contamination, the engineering manager may determine operations should stop and critical hardware should be covered, or tests should be postponed to avoid exposing critical hardware while the source of the issue is investigated. 5.4 Recommendations for Future Research There are two main areas of opportunity recommended for additional research stemming from this work: improvements and additions to the case study and additional research in the engineering methodology. The scope of the study was limited to molecular contaminant film levels defined by IEST-STD-CC1246E. Therefore, the results reported are not absolute for each method and are relative based on the IEST standard. Higher maximum contaminant film thickness values missed for each indicator method beyond the scope of this study are certainly 81
possible. Previously, no empirical data existed on the levels of contamination that visual inspections or portable Raman spectroscopy could miss. Based on the experimental results of this study, a better understanding has been gained and samples of specific film thicknesses can now be prepared to perform factorial experiments. Results from factorial experiments can then be incorporated into ANOVA analysis with replication to expand on the statistical testing performed in this work, which was performed without replication. To improve on the experimental testing, different contaminants and different substrates could be explored, and film thicknesses could be deposited via methods such as spin casting or vapor deposition and quantified through ellipsometry to determine the absolute maximum film thickness missed by each method. Additional metrics beyond maximum film thickness missed could be evaluated for each method, such as percentage of contaminated samples correctly detected, or film thickness most accurately detected. Also, detection results for portable Raman spectroscopy can change with a different evaluation metric or different user-defined criteria in the peak-finding algorithm, and additional studies can explore these variables further. Now that the methods of visual inspections and portable Raman spectroscopy have been initially characterized and quantified through the experimental testing in this work, additional development into the techniques can be performed. Variations of visual inspections can be characterized and quantified to determine the effects on the results of the study. Such variations can include different lighting intensity and viewing distances as specified in IEST-STD-CC1246E, utilizing different wavelengths of light such as white light and ultraviolet light, and exploring different 82
observers depending on training, familiarity with hardware surfaces to be inspected, etc. Based on the results of this study, additional research can be conducted on portable Raman spectroscopy detection of contaminant films on space system surfaces. A major area for improvement and research would be to enhance the substrate surface for higher Raman signal and lower noise, thus designing a witness surface that would be ideal for this indicator method for detecting contaminant films during space system integration. Finally, additional system performance modeling can be conducted to improve the case study research. For this case study, the effect of only one contaminant on a substrate at a time was modeled. In reality, contaminant films of different materials may deposit on a substrate in layers. Future work could utilize the STACK software program and model the effects of different contaminants stacked as layers on one another on a substrate. Such results could be compared to the single contaminant layer research results presented in this work to determine if the contaminant layer effects are significant to space system performance. Performance modeling is also a source for additional research in the engineering methodology. The performance modeling aspect of the methodology could be performed up front to determine system sensitivity to a particular risk. These results could then guide evaluation criteria for finding an appropriate indicator method, or could provide input into design criteria for creation of new or improved indicator methods. This, in turn, can provide an opportunity for development of future systems engineering tools. For example, system requirements could be set through the aforementioned “best performance” approach where system 83
sensitivity to a particular risk is identified and modeled and requirements are set based on the impact to system performance. Complimentary leading indicators could then be identified and developed to monitor this risk and predict future system performance. Based on the research performed in this work, it is recommended that system requirements and leading indicators be chosen and developed in parallel, rather than at different phases of the system lifecycle. A key reason for incorporating leading indicators into the risk management strategy of a system is to detect signals and warnings of risk early enough in the system lifecycle to allow engineering managers enough time to course-correct and deploy any required risk mitigation activities. Future research could explore this further by determining, based on the empirical results of the evaluation methodology, how much time each indicator method can save and how much response time can be provided to engineering managers.
84
References Abdeldayem, H. A., Dowdye, E., and Canham, J. (2006). Roles of contamination and nonlinear effects in failing space-flight lasers. Applied Physics B, 84(3), 447–452. https://doi.org/10.1007/s00340-006-2242-x ASTM. (1995). ASTM E1235: Standard Test Method for Gravimetric Determination of Nonvolatile Residue (NVR) in Environmentally Controlled Areas for Spacecraft. West Conshohocken: American Society for Testing and Materials. https://doi.org/10.1520/E1235-08.2 ASTM. (2011). ASTM E1560: Standard Test Method for Gravimetric Determination of Nonvolatile Residue from Cleanroom Wipers. West Conshohocken: American Society for Testing and Materials. https://doi.org/10.1520/E1560-11.2 ASTM. (2014). ASTM E1840: Standard Guide for Raman Shift Standards for Spectrometer Calibration. West Conshohocken: American Society for Testing and Materials. https://doi.org/10.1520/E1840-96R07.2 Aven, T. (2015a). Implications of black swans to the foundations and practice of risk assessment and management. Reliability Engineering & System Safety, 134, 83–91. https://doi.org/10.1016/j.ress.2014.10.004 Aven, T. (2015b). The concept of antifragility and its implications for the practice of risk analysis. Risk Analysis : An Official Publication of the Society for Risk Analysis, 35(3), 476–83. https://doi.org/10.1111/risa.12279 Aven, T., and Krohn, B. S. (2014). A new perspective on how to understand, assess and manage risk and the unforeseen. Reliability Engineering & System Safety, 121, 1–
85
10. https://doi.org/10.1016/j.ress.2013.07.005 Carron, K., and Cox, R. (2010). Qualitative analysis and the answer box: a perspective on portable Raman spectroscopy. Analytical Chemistry, 82(9), 3419–25. https://doi.org/10.1021/ac901951b Chen, P., Hedgeland, R., Montoya, A., Roman-Velazquez, J., Dunn, J., Colony, J., and Petitto, J. (1997). Statistical Evaluation of Molecular Contamination During Spacecraft Thermal Vacuum Test. Retrieved from http://ntrs.nasa.gov/search.jsp?R=19980237964 Colony, J. A. (1985). Analysis of Nonvolatile Residue (NVR) from Spacecraft Systems. NASA Technical Memorandum 86184. Greenbelt, MD: NASA. Retrieved from http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/19850008710.pdf Cooper, J. B., Marshall, S., Jones, R., Abdelkader, M., and Wise, K. L. (2014). Spatially compressed dual-wavelength excitation Raman spectrometer. Applied Optics, 53(15), 3333–3340. https://doi.org/10.1364/AO.53.003333 Demets, R., Bertrand, M., Bolkhovitinov, A., Bryson, K., Colas, C., Cottin, H., … Schuster, M. (2014). Window contamination on Expose-R. International Journal of Astrobiology, 14(1), 33–45. https://doi.org/10.1017/S1473550414000536 Forsberg, K., and Mooz, H. (1991). The Relationship of System Engineering to the Project Cycle. In Proceedings of the National Council for Systems Engineering (NCOSE) Conference (pp. 57–65). Chattanooga, TN. Retrieved from http://www.damiantgordon.com/Videos/ProgrammingAndAlgorithms/Papers/The Relationship of System Engineering to the Project Cycle.pdf Frank, M., Harel, A., and Orion, U. (2014). Choosing the Appropriate Integration 86
Approach in Systems Projects. Systems Engineering, 17(2), 213–224. https://doi.org/10.1002/sys.21264 GAO. (2011). United States Government Accountability Office (GAO) Space and Missile Defense Acquisitions: Periodic Assessment Needed to Correct Parts Quality Problems in Major Programs. GAO-11-404. Washington, D.C. Retrieved from http://www.gao.gov/assets/330/320127.pdf Guicheteau, J. A., Christesen, S. D., Tripathi, A., Emmons, E. D., Wilcox, P. G., Emge, D. K., … Fountain III, A. W. (2011). Proximal and point detection of contaminated surfaces using Raman spectroscopy. In Proc. SPIE 8189, Optics and Photonics for Counterterrorism and Crime Fighting VII; Optical Materials in Defence Systems Technology VIII; and Quantum-Physics-based Information Security, 818902. International Society for Optics and Photonics. https://doi.org/10.1117/12.899243 Gushwa, K. E., and Torrie, C. I. (2014). Coming clean: understanding and mitigating optical contamination and laser induced damage in advanced LIGO. In G. J. Exarhos, V. E. Gruzdev, J. A. Menapace, D. Ristau, and M. Soileau (Eds.), SPIE Laser Damage (p. 923702). International Society for Optics and Photonics. https://doi.org/10.1117/12.2066909 Hassig, M., Altwegg, K., Balsiger, H., Calmonte, U., Jackel, A., Schlappi, B., … Rubin, M. (2011). Spacecraft outgassing, a largely underestimated phenomenon. In 2011 2nd International Conference on Space Technology (pp. 1–4). IEEE. https://doi.org/10.1109/ICSpT.2011.6064657 Holmes, W., McKenney, C., Barbier, R., Cho, H., Cillis, A., Clemens, J.-C., … Williams, J. (2016). Modeling effects of common molecular contaminants on the Euclid 87
infrared detectors. In H. A. MacEwen, G. G. Fazio, M. Lystrup, N. Batalha, N. Siegler, and E. C. Tong (Eds.) (p. 99042R). International Society for Optics and Photonics. https://doi.org/10.1117/12.2233778 Huang, Y.-S., Chen, Y.-C., Chen, M.-L., Cheng, A., Hung, I.-C., Wang, J.-T., … Chang, S.-C. (2015). Comparing visual inspection, aerobic colony counts, and adenosine triphosphate bioluminescence assay for evaluating surface cleanliness at a medical center. American Journal of Infection Control, 43(8), 882–6. https://doi.org/10.1016/j.ajic.2015.03.027 IEST. (2013). IEST-STD-CC1246E: Product Cleanliness Levels – Applications, Requirements, and Determination. Arlington Heights: Institute of Environmental Sciences and Technology. INCOSE. (2010). International Council on Systems Engineering (INCOSE) Systems Engineering Handbook. (C. Haskins, K. Forsberg, M. Krueger, D. Walden, and R. D. Hamelin, Eds.) (v. 3.2). International Council on Systems Engineering. ISO. (2015). ISO 14644-1: Cleanrooms and associated controlled environments - Part 1: Classification of air cleanliness by particle concentration. Geneva: International Organization for Standardization. Kennedy, B. M., Sobek, D. K., and Kennedy, M. N. (2014). Reducing Rework by Applying Set-Based Practices Early in the Systems Engineering Process. Systems Engineering, 17(3), 278–296. https://doi.org/10.1002/sys.21269 Laerd Statistics. (2015). Retrieved March 16, 2018, from https://statistics.laerd.com/ Leveson, N. (2015). A systems approach to risk management through leading safety indicators. Reliability Engineering & System Safety, 136, 17–34. 88
https://doi.org/10.1016/j.ress.2014.10.008 Luey, K. T., and Coleman, D. J. (2006). Formation of contaminant droplets on surfaces. In O. M. Uy, S. A. Straka, J. C. Fleming, and M. G. Dittman (Eds.), Proc. SPIE Vol. 6291, Optical Systems Degradation, Contamination, and Stray Light: Effects, Measurements, and Control II (p. 62910G–1–62910G–13). International Society for Optics and Photonics. https://doi.org/10.1117/12.678563 Madni, A. M., and Sievers, M. (2014). Systems Integration: Key Perspectives, Experiences, and Challenges. Systems Engineering, 17(1), 37–51. https://doi.org/10.1002/sys.21249 Mandrile, L., Giovannozzi, A. M., Pennecchi, F., Saverino, A., Lobascio, C., and Rossi, A. M. (2015). Direct detection and quantification of molecular surface contaminants by infrared and Raman spectroscopy. Analytical Methods, 7(6), 2813–2821. https://doi.org/10.1039/C4AY02850C Marchant, A. B. (2010). Obstacles to the flow of requirements verification. Systems Engineering, 13(1), 1–13. https://doi.org/10.1002/sys.20127 Montgomery, D. C. (2013). Introduction to Statistical Quality Control (7th ed.). Hoboken, NJ: John Wiley & Sons, Inc. Optical Reference Laboratory Reflectance Standards. (n.d.). Retrieved May 8, 2017, from http://opticalreferencelab.com/calibrated-specular-reflectance-standards/ Ott, M. N., Jin, X. L., Chuska, R., Friedberg, P., Malenab, M., and Matuszeski, A. (2006). Space Flight Requirements for Fiber Optic Components; Qualification Testing and Lessons Learned. In H. G. Limberger and M. J. Matthewson (Eds.), Photonics Europe (pp. 619309-619309–15). International Society for Optics and 89
Photonics. https://doi.org/10.1117/12.669880 Palmer, K. F., Williams, M. Z., Budde, B. A., and Bertrand, W. T. (1994). Optical Analysis Methods for Material Films Condensed on Cryogenic Surfaces of Spacecraft. In Technical Report AEDC-TR-94-3, AD-A284014. Defense Technical Information Center, Fort Belvoir, Va. Paté-Cornell, E. (2012). On “Black Swans” and “Perfect Storms”: Risk Analysis and Management When Statistics Are Not Enough. Risk Analysis, 32(11), 1823–1833. https://doi.org/10.1111/j.1539-6924.2011.01787.x Qureshi, Z. H. (2007). A Review of Accident Modelling Approaches for Complex Critical Sociotechnical Systems. 12th Australian Workshop on Safety Related Programmable Systems (SCS’07), Adelaide, 86, 47–59. Retrieved from http://delivery.acm.org/10.1145/1390000/1387046/p47qureshi.pdf?ip=146.165.52.212&id=1387046&acc=PUBLIC&key=0834BCF3EEC8 C5C1.4D4702B0C3E38B35.4D4702B0C3E38B35.4D4702B0C3E38B35&CFID=7 47903449&CFTOKEN=87225175&__acm__=1491495263_7e23ffb61a1afd43b127 72aae4d4 Rampini, R., Grizzaffi, L., and Lobascio, C. (2003). Outgassing Kinetics Testing of Spacecraft Materials. Materialwissenschaft Und Werkstofftechnik, 34(4), 359–364. https://doi.org/10.1002/mawe.200390075 Reason, J. (1997). Managing the Risks of Organizational Accidents. Aldershot, Hants: Ashgate. Rhodes, D. H., Valerdi, R., and Roedler, G. J. (2009). Systems engineering leading indicators for assessing program and technical effectiveness. Systems Engineering, 90
12(1), 21–35. https://doi.org/10.1002/sys.20105 Riede, W., Allenspacher, P., and Schroder, H. (2005). Laser-induced hydrocarbon contamination in vacuum. In Proc. SPIE Vol. 5991, Laser-Induced Damage in Optical Materials. International Society for Optics and Photonics. https://doi.org/10.1117/12.638765 Roedler, G., and Rhodes, D. H. (2010). Systems Engineering Leading Indicators Guide, Version 2.0. Cambridge, MA. See, J. E. (2015). Visual Inspection Reliability for Precision Manufactured Parts. Human Factors, 57(8), 1427–42. https://doi.org/10.1177/0018720815602389 Shand, N. C. (2008). Tackling Field-portable Raman Spectroscopy of “Real World” Samples. In G. Owen (Ed.), Proc. SPIE 7119, Optics and Photonics for Counterterrorism and Crime Fighting IV, 71190J. International Society for Optics and Photonics. https://doi.org/10.1117/12.802345 Sipilä, J., Auerkari, P., Holmström, S., and Vela, I. (2014). Early warning indicators for challenges in underground coal storage. Risk Analysis : An Official Publication of the Society for Risk Analysis, 34(12), 2089–97. https://doi.org/10.1111/risa.12273 Swuste, P., and Blokland, P. J. (2015). Process safety indicators , a review of literature. Journal of Loss Prevention in the Process Industries, 40(August 2016), 162–173. https://doi.org/10.1016/j.jlp.2015.12.020 Tribble, A. C. (2000). Fundamentals of Contamination Control. (A. R. Weeks, Ed.). Bellingham, Washington: SPIE Press, The Society of Photo-Optical Instrumentation Engineers. https://doi.org/10.1117/3.387881 Tveekrem, J. L., Leviton, D. B., Fleetwood, C. M., and Feinberg, L. D. (1996). 91
Contamination-induced degradation of optics exposed to the Hubble Space Telescope interior. In Proc. SPIE 2864, Optical System Contamination V, and Stray Light and System Optimization (pp. 246–257). International Society for Optics and Photonics. https://doi.org/doi:10.1117/12.258316 Villahermosa, R. M., Weiller, B. H., Virji, S., and Taylor, D. P. (2008). Managing Contamination-enhanced Laser Induced Damage (CLID). In S. A. Straka (Ed.), Optical Engineering + Applications (pp. 706908-706908–9). International Society for Optics and Photonics. https://doi.org/10.1117/12.802313 Wang, P. (2015). Review and recent progress of handheld spectrometry at Thermo Fisher Scientific. In M. A. Druy, R. A. Crocombe, and D. P. Bannon (Eds.), Proc. SPIE 9482, Next-Generation Spectroscopic Technologies VIII, 948204. International Society for Optics and Photonics. https://doi.org/10.1117/12.2178140 Wolfgong, W. J. (2011). The Silicone Conundrum Part II: “Low Outgassing” Silicones. Journal of Failure Analysis and Prevention, 11(3), 215–221. https://doi.org/10.1007/s11668-011-9442-3 Wolfgong, W. J., and Wiggins, K. (2010). The Silicone Conundrum. Journal of Failure Analysis and Prevention, 10(4), 264–269. https://doi.org/10.1007/s11668-010-93538 Wood, B. E., Garrett, J. W., Meadows, G., Raghavan, V., Lay, E., Bertrand, W. T., … Montoya, A. F. (2007). Updated Version of the NASA SEE Program Spacecraft Contamination and Materials Outgassing Effects Knowledgebase. In AIAA 2007907, 45th AIAA Aerospace Sciences Meeting and Exhibit (pp. 1–15). Reno, NV: American Institute of Aeronautics and Astronautics. Retrieved from 92
http://highorder.berkeley.edu/proceedings/aiaa-annual-2007/paper1170.pdf Yokozawa, H., Baba, S., Miyazaki, E., and Kimoto, Y. (2012). Evaluation of bakeout effectiveness by optical measurement of a contaminated surface. In S. A. Straka, N. Carosso, and J. Egges (Eds.), SPIE Optical Engineering + Applications (p. 84920A). International Society for Optics and Photonics. https://doi.org/10.1117/12.929453
93
Spacecraft Performance Model Results
Change in Reflectivity due to Silicone Film on Aluminum Foil
Change in Reflectivity due to Hydrocarbon Film on Aluminum Foil
94
Change in Reflectivity due to Fluorocarbon Film on Aluminum Foil
Change in Reflectivity due to Glycol Polymer Film on Aluminum Foil
95
Change in Reflectivity due to Ester Film on Aluminum Foil
Change in Transmission due to Silicone Film on Glass Slide
96
Change in Transmission due to Hydrocarbon Film on Glass Slide
Change in Transmission due to Fluorocarbon Film on Glass Slide
97
Change in Transmission due to Glycol Polymer Film on Glass Slide
Change in Transmission due to Ester Film on Glass Slide
98
Change in Emissivity due to Silicone Film on Solar Reflector MLI
Change in Emissivity due to Hydrocarbon Film on Solar Reflector MLI
99
Change in Emissivity due to Fluorocarbon Film on Solar Reflector MLI
Change in Emissivity due to Glycol Polymer Film on Solar Reflector MLI
100
Change in Emissivity due to Ester Film on Solar Reflector MLI
Change in Emissivity due to Silicone Film on Aluminum Kapton® Tape
101
Change in Emissivity due to Hydrocarbon Film on Aluminum Kapton® Tape
Change in Emissivity due to Fluorocarbon Film on Aluminum Kapton ® Tape
102
Change in Emissivity due to Glycol Polymer Film on Aluminum Kapton ® Tape
Change in Emissivity due to Ester Film on Aluminum Kapton ® Tape
103
Change in Reflectivity due to Silicone Film on Silicon Wafer
Change in Reflectivity due to Hydrocarbon Film on Silicon Wafer
104
Change in Reflectivity due to Fluorocarbon Film on Silicon Wafer
Change in Reflectivity due to Glycol Polymer Film on Silicon Wafer
105
Change in Reflectivity due to Ester Film on Silicon Wafer
106
