tions, the safety lifecycle could apply to a wide range of application domains. ... with its unique development lifecycle, safety processes will need to fit around ...
Safety Lifecycle for Developing Safety Critical Artificial Neural Networks Zeshan Kurd, Tim Kelly Department of Computer Science University of York, York, YO10 5DD, UK. {zeshan.kurd, tim.kelly}@cs.york.ac.uk
Abstract. Artificial neural networks are employed in many areas of industry such as medicine and defence. There are many techniques that aim to improve the performance of neural networks for safety-critical systems. However, there is a complete absence of analytical certification methods for neural network paradigms. Consequently, their role in safety-critical applications, if any, is typically restricted to advisory systems. It is therefore desirable to enable neural networks for highly-dependable roles. Existing safety lifecycles for neural networks do not focus on suitable safety processes for analytical arguments. This paper presents a safety lifecycle for artificial neural networks. The lifecycle focuses on managing behaviour represented by neural networks and to provide acceptable forms of safety assurance. A suitable neural network model is outlined and is based upon representing knowledge in symbolic form. Requirements for safety processes similar to those used for conventional systems are also established. Although developed specifically for decision theory applications, the safety lifecycle could apply to a wide range of application domains.
1 Introduction Typical uses of ANNs (Artificial Neural Networks) in safety-critical systems include areas such as industrial process and control, medical systems and defence. An extensive review of ANN use in safety-related applications has been provided by a U.K. HSE (Health & Safety Executive) report [1]. However, ANNs in these safety systems have been restricted to advisory roles with the continued absence of safety arguments. There are many techniques for typical neural networks [2, 3] which aim to improve generalisation performance. However, these performance-related techniques do not provide acceptable forms of safety arguments. Any proposed ANN model must allow for trade-offs between performance and safety. Attributes or features of ANN models must satisfy safety arguments whilst ensuring that measures incorporated do not diminish the advantages associated with ANNs. A promising approach outlined in section 2 is to utilise ‘hybrid’ neural networks. The term ‘hybrid’ in this case, refers to a neural network representing knowledge in symbolic form. This knowledge is represented by the internal structure of the ANN (within its weights, links and neurons). This has greater potential for ‘transparency’ or white-box style analysis. White-box analysis can potentially result in strong arguments
about the knowledge and perhaps behaviour represented by the network. Overcoming the black-box restriction will provide strong safety arguments not only about ‘what it can do’ but also about ‘what it cannot do’. Developing such ‘hybrid’ neural networks will involve frameworks unlike conventional software development. This is primarily because of the learning characteristics of ANNs. Learning helps the network determine desired behaviour as opposed to some complete specification determined at the initial stage of development. Along with its unique development lifecycle, safety processes will need to fit around ANN development. A safety lifecycle is established in section 3 which illustrates where safety processes are performed during development. Typical processes required for the chosen ANN model may need to be adapted to deal with knowledge in symbolic forms. Although a fully defined ANN model is beyond the scope of the paper, the general concept of ‘hybrid’ ANNs will be considered. Certification of the ANN model can then be presented in the form of a safety case. Section 4 highlights the nature of potential safety arguments and how they resemble those associated with conventional software development. Finally, section 5 emphasises the challenge of maintaining feasibility of the ANN model by considering the performance vs. safety trade-off.
2 A Suitable ANN Model for Safety Any potential safety case must overcome problems associated with classic neural networks i.e. multi-layered perceptrons. Some typical problems may be concerned with determining ANN structure and topology. For example, internal structure of ANNs may influence the generalisation and learning performance [4]. Another problem lies in determining training and test sets. These sets must represent the desired function using a limited number of samples. Dealing with noise during training is also problematic. Given these problems with training, it is essential to ensure that the network does not deviate from the required target function. Another issue related to the learning or training process is the ‘forgetting’ of previously learnt samples [5]. This can lead to poor generalisation and long training times [5]. Other problems during training may result in learning settling in local instead of a global minimum [5]. This will result in poor performance or sub-optimal solutions. Another typical problem encountered during ANN training is deciding upon appropriate stopping points for the training process (such as the total number of training samples). Although this could be aided by cross-validation [6], it relies heavily on test sets to provide an accurate representation of the desired function. A key problem associated with typical ANNs is the inability to perform white-box style analysis over the behaviour represented by the ANN. Consequently, an overall error over some test set is often described as the generalisation performance [5]. Moreover, there is a lack of explanation mechanisms of how each ANN output is produced. This inability to analyse the internal behaviour also makes it difficult to identify and control potential ‘hazards’ associated with the ANN. This results in the ANN unable to provide analytic arguments for the satisfaction of safety requirements.
2.1 Requirements for Suitable ANN Model Analytical and test arguments are needed for any proposed ANN model. To support analytical arguments, there must be means to analyse the behaviour using the internal structure (in terms of weights and neurons) of ANNs. Some common approaches to analysing the behaviour of ANNs include: • •
Pedagogical Approach: This uses a black-box style analysis to build a behavioural description of the ANN. One particular approach is known as sensitivity analysis [7] and investigates the effects perturbing inputs have on outputs. Decomposition Approach: This is a white-box style analysis which focuses on the internal structure of the ANN to determine a behavioural description. This requires that knowledge about the problem (gathered by the ANN) is represented in a highly structured form. Typical examples for eliciting knowledge are rule extraction algorithms [8-10]. Extraction algorithms can help determine knowledge or behaviour represented by the ANN. This knowledge may be in terms of symbolic information (such as if-then rules).
The decomposition approach is particularly attractive for safety applications. This provides the required white-box style analysis to allow a more detailed and comprehensive view of the ANN. A particular type of ANN model that attempts to utilise this approach combines symbolic knowledge with neural learning paradigms. A common view is that symbolic approaches focus on producing discrete combinations of features (such as rules) [9]. On the other hand, the neural approach adjusts continuous, nonlinear weighting of their inputs to perform learning. By combining symbolic and neural paradigms, learning can be exploited to refine or modify symbolic knowledge. Figure 1 illustrates a typical model for combining symbolic knowledge with neural (connectionist) learning. Initial Symbolic Information
Final Symbolic Information
INSERT
Examples
EXTRACT
Initial Neural Network
REFINE
Final Neural Network
Fig. 1. Framework for combining symbolic and neural learning taken from [9]
The term ‘hybrid’ is associated with combining symbolic knowledge with neural learning. There are three main stages associated with the framework for ‘hybrid’ ANNs. The first stage involves gathering initial symbolic information. This initial information is prior knowledge known about the solution and may be in the form of rules. This initial knowledge is important since it will help neural learning achieve
desired performance using less time resources [11]. The initial rule set is then ‘inserted’ into a neural network structure. This can be achieved through rule-to-network algorithms [9] which map parts of each rule into the internal structure of the ANN (such as weights and neurons). Once the ANN has represented all initial symbolic information the next stage involves refinement or learning. Refinement uses training samples to modify or add new knowledge to the initial symbolic information. This is the key motivation for adopting the neural learning concept. Training data may be simulated or consist of data sampling gathered from real-world interaction. Typical algorithms used for the learning process may involve variations of back-propagation algorithms [9]. During refinement or learning, the topological structure of the ANN may require adaptation to accommodate changes in the initial rule set. Once learning is complete, the final stage involves eliciting and evaluating symbolic knowledge from the final ANN. This process attempts to determine all changes or additions made to the initial symbolic information. There is an abundance of rule extraction algorithms [8] to elicit all symbolic knowledge stored in the network. Once the final symbolic information has been determined these rules may be evaluated and potentially used in expert systems [12]. ‘Hybrid’ ANNs have been demonstrated to enable quicker learning and overall better generalisation performance [9, 12] for a wide range of problems. ‘Hybrid’ ANNs present a useful tool for theory-refinement problems. They exploit the neural learning paradigm widely associated with machine learning in artificial intelligence. It also uses symbolic information to provide a highly structured representation of the function represented by the network. Given the nature of symbolic knowledge, possible applications are associated with decision theory domains. ‘Hybrid’ ANNs are well established models [8, 13] and many have been used in safetyrelated problems [12, 14]. Medical diagnosis [15] is one example of how ‘hybrid’ ANNs can be applied to real world problems. In this particular example a set of initial rules are derived through theoretical knowledge and practical experiences (this may involve representing causes of a particular disease). These rules are then translated into a neural network and global learning is performed. That is, the ANN can adapt its knowledge to variations of the data domain. This is performed by learning new knowledge (through input-output samples) leading to possible modification of the architecture or topology of the ANN. The learning process will produce a new set of knowledge (with the aim of improving diagnosis of a particular disease). Along with learning, the generalisation capability can also be preserved with the ‘hybrid’ ANN. Conventional ANNs generalise outputs given inputs based upon the activations that occur within its architecture. Very little is known how certain outputs were produced. On the other hand, the ‘hybrid’ ANN has the advantage of representing some analysable form of knowledge. It is this knowledge that is the ‘output’ of the network which can be utilised for generalisation. For safety-critical applications, ‘hybrid’ ANNs encompass the potential to overcome many problems associated with typical ANNs. Given the decomposition-style approach offered by rule extraction algorithms, techniques devised for black-box representations will not be relied upon. Analytical arguments about the solution provided by the ‘hybrid’ ANN may be based upon symbolic information. This enables a
higher level of abstraction as opposed to dealing with low-level representation such as weights, links and neurons. For example, rather than using an overall output error (produced over some test set), the network can be analysed in terms of the rules it embodies. Consequently, this provides extra comprehensibility for analytical processes. However, analytical processes have to overcome some of the potential problems associated with symbolic information. One typical problem is associated with managing large sets of rules. When evaluating symbolic knowledge, the quantity of rules may make it hard to identify and mitigate potential ‘hazards’. Work is currently ongoing for a complete definition of the ‘hybrid’ ANN that is suitable for safety-critical applications. 2.2 Systems Development Lifecycle There are very few existing ANN development lifecycles for safety-related applications [3, 4]. One particular lifecycle [4] is directly intended for safety-critical applications however there are several problems associated with its approach. One problem is that it relies on determining the specification and behaviour at the initial phase of development. This is not practical since the prime motivation for ANN learning is to determine its behaviour given very limited initial data. Another problem is that it emphasises control over non-functional properties. Typical examples of non-functional properties include usage of spatial or temporal resources. Instead, focus should be upon constraining functional properties such as learning or the behaviour of the ANN. Documenting each development phase (process-based arguments) are generally regarded as weak arguments for providing assurance for conventional software [16]. However, the lifecycle [4] emphasises process-based arguments such as ‘Data Analysis Document’ that represents all training samples used during training. Other problems associated with the lifecycle involve iterating training without clear analytical tools to determine performance or safety. The iteration also attempts to vary network topologies, input neurons and other factors. This is no clear indication of how these ANN variations will contribute to safety or performance. Instead, an appropriate ANN model should be determined which will allow time-efficient training, acceptable performance and provide suitable safety assurance. There is another ANN development lifecycle model for safety-critical applications [3]. This lifecycle attempts to overcome several problems associated with the above model. The lifecycle acknowledges that a precise functional specification is not possible at the initial stages of development. It also emphasises that the role of data (training and test samples) requires extra tasks such as data gathering and modelling. The undefined iteration during training is also tackled. However, the lifecycle [3] does not define a suitable ANN model for overcoming the above problems. It relies on using typical ANN models resulting in black-box analytical techniques (using network error over test sets as an indication of performance and safety). Consequently, it inherits many of the problems associated with typical ANNs. Suitable safety processes that attempt to identify and mitigate hazards are also neglected. This results in poor product-based arguments about the safety (existence of potential hazards) of the ANN.
To overcome limitations associated with current approaches, Figure 2 presents a lifecycle for developing ‘hybrid’ neural networks. This lifecycle ensures that the learning mechanism is preserved whilst allowing analysis and control over the network (greater potential for safety assurance).
Fig. 2. Development Lifecycle for Hybrid Neural Networks The development lifecycle is divided into three main levels starting from a symbolic representation, down to a ‘hybrid’ ANN. Levels in the lifecycle are as follows: 1. Symbolic Level: This level is associated only with symbolic information. It is separated from the neural learning paradigm and deals with analysis in terms of symbolic knowledge. Typical examples may include the gathering and processing of initial knowledge. Other uses may involve evaluating extracted knowledge gathered post-learning. 2. Translation Level: This is where symbolic knowledge and neural architectures are combined or separated. This is achieved though processes such as rule insertion [9] or extraction algorithms [8]. All transitions from the symbolic level to neural learning level involve rule insertion algorithms. All transitions from the neural learning level to symbolic involve rule extraction algorithms. 3. Neural Learning Level: This level uses neural learning to modify and refine initial symbolic knowledge. Neural learning is achieved by using specific training samples along with suitable learning algorithms [9, 12, 17]. The development of ‘hybrid’ ANNs initially starts by determining requirements. These requirements are a representation of the problem to be solved possibly in the form of informal descriptions. The intention of conventional software development is to have a complete set of requirements. On the other hand the initial requirements for ‘hybrid’ ANNs are intentionally incomplete (hence the use of learning). ‘Incompleteness’ of these requirements could be caused by insufficient data or a result of noise. A common factor for ‘incompleteness’ is if the expert doesn’t have enough knowledge about
the problem as would be the case in typical real-world applications. Consequently, the requirements do not represent the entire target function or solution. Requirements then need to be translated into symbolic knowledge by experts or those involved in development. A set of rules are devised and are labelled as ‘sub-initial’ knowledge. These rules are then converted into a form compatible for incorporation into the ANN structure [9] resulting in ‘initial’ symbolic knowledge. This knowledge is then ‘inserted’ into the ANN model using suitable rules-to-network algorithms [9]. Once ‘inserted’, the neural learning commences (no longer with random initial weights). Refinement and modification of the initial knowledge occurs using a two-tier learning model: 1. Dynamic Learning: This uses specific learning algorithms to refine the initial knowledge. It uses some training set and attempts to reduce error by adding or removing rules. This may result in architectural (topological) changes to the network [15] hence the term ‘dynamic’. For example, requiring additional hidden units for adding new rules. 2. Static Learning: This phase is concerned with refining and modifying the set of rules represented in the network (by adjusting ANN link weights). No architectural or topological changes are allowed at this stage. Instead, the learning algorithm concentrates on changing elements of each rule (such as adding and removing antecedents). Although the ANN system states varies over time (like dynamical systems), this phase is considered ‘static’ since only network weights are adjusted. This learning phase can be used in critical roles and interfaced with real-time data. Learning takes place in two distinct phases. This will be shown to be particularly advantageous in producing both analytical and test-based safety arguments. The model attempts to preserve the learning ability, which is a major contributor to its performance. The lifecycle is unlike conventional software as the ANN works towards a specification (or desired function) rather than from it. Finally, the network can be utilised in an area of expertise determined by its symbolic knowledge.
3 Using Adapted Safety Processes Software can be regarded as safety-critical if it directly or indirectly contributes to the occurrence of a hazardous system state (“system hazard”) [18]. A hazard is a situation, which is potentially dangerous to man, society or the environment [18]. When developing safety-critical software, there are a set of requirements which must be enforced. These requirements are formally defined in a safety standard such as Defence Standard 00-55 [19]. One of the main tools for determining requirements is the use of a safety case to encapsulate all safety arguments for the software. A safety case is defined in Defence Standard 00-55 [19] as: “The software safety case shall present a well-organised and reasoned justification based on objective evidence, that the software does or will satisfy the safety aspects of the Statement of Technical Requirements and the Software Requirements specification.”
Some of the main components in formulating safety requirements are hazard analysis and mitigation. Function Failure Analysis (FFA) [20] is a predictive technique to identify and refine safety requirements. It focuses upon system functions and deals with analysing effects of failures to provide a function, incorrect function outputs and determining actions to improve design. Another technique is Software Hazard and Operability Study (SHAZOP) [21] which uses ‘guide words’ for qualitative analysis to identify hazards not previously considered. It attempts to analyse all variations of the system based on these guide words and can uncover causes, consequences, indications and recommendations for particular identified hazards. Arguing the satisfaction of the safety requirements is divided into two types of arguments. These two types are analytical and test-based arguments which are defined in a widely accepted Defence Standard 00-55 [19]. Arguments from testing (such as white-box or black-box) can be generated more easily than analytical arguments for typical ANNs. There are several safety processes performed during the lifecycle of safety-critical systems e.g. ARP 4754/4761 [20]. Preliminary Hazard Identification (PHI) is the first step in the lifecycle, it forms the backbone of all following processes. It is a predictive style technique and aims to identify, manage and control all potential hazards in the proposed system. Risk Analysis and Functional Hazard Analysis (FHA) analyses the severity and probability of potential accidents for each identified hazard. FHA can be considered as confirmatory or evaluative analysis. Preliminary System Safety Assessment (PSSA) [20] ensures that the proposed design will refine and adhere to safety requirements and will help guide the design process. System Safety Analysis (SSA) demonstrates through evidence that safety requirements have been met. It uses inductive and deductive techniques to examine the completed design and implementation. Finally, the Safety Case [22] generated throughout development delivers a comprehensible and defensible argument that the system is acceptably safe to use in a given context. The intentions of these principles (safety processes) are widely accepted. The challenge is to find how these safety processes can support the ANN in safety-critical applications. 3.1 Safety Development Lifecycle With the unique development lifecycle of ‘hybrid’ ANNs, it is not obvious where each safety process should be initiated. Existing safety processes need to be adapted to deal with symbolic representation. Therefore, adapted safety processes are heavily influenced by the ANN model and application domain. For example, control theory applications may result in ANNs representing control algorithms. Safety processes must then deal with this level of detail as opposed to individual weights and neurons. This is achieved by providing suitable translation algorithms [8, 9]. The aim of the lifecycle is to gather initial knowledge about the problem and to utilise a two-tier learning process. Certification must be produced post-dynamic learning and will argue how potential hazards have been identified, controlled and mitigated. The safety lifecycle also emphasises how static learning can be performed postcertification, allowing future changes to the symbolic representation. A safety lifecycle
for the hybrid ANN is illustrated in Figure 3. Several safety processes have been superimposed on the development lifecycle.
Fig. 3. Safety Lifecycle for Hybrid Neural Networks Although detailed description of how each adapted safety process is beyond the scope of this paper, the aims and objectives for each process is highlighted. Initially, an overall description of the ANN role is determined. This process has been described in the previous section as ‘requirements’ and involves black-box descriptions of the role. Preliminary Hazard Identification (PHI) deals with initial symbolic knowledge and is used to determine the initial conditions of the network. The role of this process is the initial consideration of possible system level hazards (using a black-box approach). This results in a set of rules partially fulfilling the desired function. Typical characteristics of this process must include: 1. 2. 3. 4.
Ability to represent knowledge in rule form. Generate possible system level hazards and help understand system context. Ability to remove, add, refine rules or parts of it. The process must generate a set of rules suitable for translation into the ANN model.
This process may use knowledge gathered from domain experts, empirical data or any other source. It is important that PHI is performed at this stage so that any future refinements can be analysed in relation to the initial rules. This may also help relieve the workload imposed on hazard analysis which is performed at a later stage. Using the initial knowledge, PHI generates an initial set of rules partially describing the desired function. The next phase is to perform a predictive-style technique using FHA. This is involved in the assessment of the rule set by using perturbation-style analysis. FHA
presents a systematic white-box technique that may help identify rules that do not represent the desired function. The FHA approach may also help understand how symbolic knowledge could lead to potential hazards. Furthermore, FHA may also help build an understanding for the importance of certain rules. During the FHA safety process, assertions of new rules may be made to help prevent specific hazards. Identifying criticality of certain rules may result in ‘targeted’ training for the next phase. For example, specific training sets may be devised to focus on certain rules or areas of concern. Full details of this safety process are beyond the scope of this paper. Once the initial conditions of the ANN have been set, safety processes must identify, control and mitigate potential hazards during learning. All other safety processes will be performed during dynamic learning and before the static learning phase. After the dynamic learning phase has learned some training set (by reaching some error threshold), FHA is performed again. However this time, the approach of FHA is not only predictive but also evaluative. FHA is performed over the entire set of rules represented by the network. These can be elicited using rule extraction algorithms by exploiting decomposition approaches. FHA initially uses an evaluative approach by analysing the rule set and determines the existence of potential hazards. During analysis the following events may occur: 1.
2.
Initiating further training – Present a new training set that reflects desired direction of learning. This can be seen as guiding the design process using specific training sets. Training samples may be based upon particular areas of concern determined by critical or important rules. This iteration of re-training and reanalysing (using FHA) will contribute to the performance of the system (entire rule set). This iteration will end based upon performance results determined by FHA (by analysing the rule set). Modify rule set – Manual assertions or modifications may be made to mitigate or control identified hazards.
It is important to remember that the function (or rule set represented by the ANN) may only partially satisfy the desired function. This is a realistic assumption since it may be infeasible to determine whether totality of the desired function has been achieved (supporting the motivation of neural learning). Certification can be provided based upon this ‘partial-but-safe’ approach. Although the approach so far deals with mitigating hazards associated with the rule set and improving performance, it must also control future learning. Once sufficient performance has been gained (by the rule set) the predictive role of FHA is initiated. This attempts to utilise an exploratory safety analysis technique to identify how rules may be refined during static learning. FHA is based upon the preconsideration of possible adaptation of rules. This may involve analysing the rule set and determining how potential hazards may be introduced if parts of rules are modified. Measures to mitigate potential hazards must then be incorporated into the network by translating rule conditions into network weights. For example, a typical condition may state that a particular antecedent of a rule may not be removed during static learning. Violating such rule conditions would result in a potentially hazardous state. Future paths of learning are therefore constrained and will provide assurance about
future states of the ANN. This enables static learning to be performed without the need for further safety assurance. It is not feasible to allow dynamic learning postcertification. A prime reason for this is that predictive or evaluative analysis for identifying possible hazards is not feasible if the rule set is modified (by adding new rules) during deployment. Some of the requirements for the FHA are as follows: 1. The initial symbolic information may be incomplete or may lead to potential hazards. It is common for neural network problems to suffer from incomplete or sparse initial data. Analytical processes must also be able manage symbolic knowledge for predictive or evaluative analysis. 2. Perturbation-style analysis must attempt to discover potential hazards (in the rule set) possibly utilising a HAZOP-style [21] analysis. Furthermore, constraints must be derived for static learning (by translating conditions for rules into weights and links). This must attempt to foresee possible evolutionary states and to specify what can or can’t be performed. This requires an extra amount of domain knowledge and is applicable for applications that can provide it. 3. The ability to determine whether additional rules will create new hazards or not and if they invalidate or conflict with existing rules. In other words, there must be a clear understanding about non-monotonic behaviour. Finally, certification of the ANN will be provided using a safety case. This will be derived from the last evaluation performed by FHA and will rely on the completeness of constraints. Each safety process will be heavily involved in providing solutions or evidence for potential safety arguments. FHA will also demonstrate how all hazards in the symbolic representation have been mitigated. Safety processes associated with the ‘hybrid’ ANN model, exploit the translation level to achieve transparency (in terms of the rules represented by the network). This enables FHA to provide assurance that in operation (despite permitting static learning), constraints maintain safety for any future learning. Constraints over learning allow certification to be finalised before the static learning phase begins. Once certification is completed, the ANN model can then be deployed. The ‘hybrid’ ANN model and safety lifecycle demonstrates how safety can be assured particularly when used for theory-refinement applications. Consequently, the knowledge represented by the network can be used for highly-dependable roles in safety-critical applications.
4 Potential Safety Arguments The safety case is a living document that is developed and maintained throughout the systems lifecycle. This contributes to safety by justifying modifications and refinements that are made during neural learning. Various attributes associated with ‘hybrid’ ANN development (such as the translation level), contribute to safety case arguments. Although providing a complete safety case is beyond the scope of this paper, the main goals to overcome are hazard identification and mitigation. When considering ANNs, the most problematic area is learning, since it has the ability to change itself (this can
also be considered as changing its own implementation). The strategy is to exploit the two-tier learning process so that safety arguments about future refinement can be provided for smaller changes. Preliminary Hazard Identification is the backbone for arguments related to the functional behaviour. It provides assurance that potential hazards in the initial set of rules have been identified. Since these rules will undergo changes, this process may not contribute as evidence but aid later hazard analysis processes. The ANN model facilitates this process by representing knowledge in the form of rules. Hazard analysis will provide the strongest form of safety arguments. The arguments are mainly related to guiding training, ensuring there are no faults that could lead to potential hazards and to define all possible safe refinement. In terms of hazard identification, rule extraction algorithms will provide solutions for determining the rules represented by the network. Hazard analysis will provide assurance that all hazards have been identified (within the rule set) and measures have been taken to mitigate them. Learning algorithms and focussed training can provide assurance about the performance or the completeness of the rule set. Finally, a set of controls or constraints determined by the hazard analysis process will provide assurance for all paths during static learning. These constraints over the ANN can be determined at high-level (symbolic level) and incorporated using translation algorithms. This has the potential to provide safety assurance about the constraints in terms of meaningful representations. It is evident for the presented lifecycle that all adapted safety processes contribute to safety argument solutions. For conventional software development it is argued that emphasis should be on product-based arguments rather than demonstrating that certain processes have been performed (and assuming that the final software or system is safe based on this fact) [16]. Conventional software safety emphasises producing evidencebased frameworks instead of blindly performing regulatory process-based approaches. This requires deriving product or process arguments that have a clear affect on mitigating or controlling hazards. The safety lifecycle presented in this paper focuses on producing product-based safety arguments. This is achieved by exploiting a suitable ANN model and utilising adapted safety processes.
5 Feasibility Issues: Performance vs. Safety Trade-off The aim is to maintain feasibility by providing satisfactory performance whilst generating acceptable safety assurance. Certain attributes of the model may need to be limited or constrained so that particular safety arguments are possible. Typical examples may include permissible learning during development but not whilst deployed. This may provide safety arguments about the behaviour of the network without invalidating them with future changes to implementation. But, this could lead to over-constraining the performance of ANNs. One particular type of ANN model which demonstrates poor safety assurance employs diversity [2]. This is an ensemble of ANNs where each is devised by different methods to encapsulate as much of the target function as possible. Results demonstrate improvement in generalisation performance over some test
set but lack in the ability to analyse and determine the function performed by each member of the ensemble. The safety lifecycle defined in this paper attempts to generate acceptable safety arguments providing assurance comparable to that achieved with conventional software. Some of the typical motivations for using neural networks involve dealing with problems whose complete algorithmic specification is not possible during initial stages of development (hence the use of learning). They can also deal with large input spaces and generalise outputs given novel inputs. The hybrid ANN outlined in this paper effectively exploits learning by working towards a solution using the two-tier learning model. The model can be considered for theory refinement applications where the problem directly drives the network (through training samples) to arrive at a solution. The ANN model also presents the opportunity to not only analyse but also control the behaviour or learning of the network. The twotier learning process may provide safety assurance but it limits learning during deployment by not allowing topological changes (adding or removing rules). The hybrid ANN can also represent rules in a structured way allowing white-box style analysis. Consequently, rather than managing weights and neurons, analysis can take place at a higher and more comprehensible level (symbolic information). However, some of the limitations of the proposed system are that rules may have restrictions (e.g. rules must be non-recursive) [13]. Some typical problems associated with knowledge reasoning may include managing many rules and predicting consequences for changes made to rules. The knowledge or rules are also crisp so no approximate answers are presented. This however, may be a desirable property for safety-critical systems where certainty plays a crucial role for highly dependable outputs.
6 Conclusions To justify the use of ANNs within safety critical applications will require the development of a safety case. For high-criticality applications, this safety case will require arguments of correct behaviour based both upon analysis and test. Previous approaches to justifying ANNs have focussed predominantly on (test-based) arguments of high performance. In this paper we present the ‘hybrid’ ANN as a potential model and outlined a suitable development lifecycle. We then present a safety lifecycle, which emphasises the need for adapting software safety processes for the chosen ANN model. Detailed analysis of the ANN model, safety processes and potential supporting arguments are beyond the scope of this paper. We however, discussed the benefits of performing white-box style analysis over the ‘hybrid ANN’ and the types of potential safety arguments that can be generated. The requirements and aims of adapted safety processes were also highlighted including the necessity to constrain neural learning and other factors. The development and safety lifecycles contribute to suitable safety arguments and allow neural networks for highly dependable roles.
References 1. 2. 3. 4.
5. 6. 7. 8.
9. 10. 11. 12.
13. 14.
15. 16.
17. 18. 19. 20. 21. 22.
Lisboa, P., Industrial use of safety-related artificial neural networks. Health & Safety Executive 327, (2001). Sharkey, A.J.C. and N.E. Sharkey, Combining Diverse Neural Nets, in Computer Science, University of Sheffield: Sheffield, UK (1997). Nabney, I., et al., Practical Assessment of Neural Network Applications, Aston University & Lloyd's Register: UK (2000). Rodvold, D.M. A Software Development Process Model for Artificial Neural Networks in Critical Applications. in Proceedings of the 1999 International Conference on Neural Networks (IJCNN'99). Washington D.C. (July 1999) Principe, J.C., N.R. Euliano, and W.C. Lefebvre, Neural and Adaptive Systems: Fundamentals Through Simulations: John Wiley & Sons (2000). Kearns, M., A Bound on the Error of Cross Validation Using the Approximation and Estimation Rates, with Consequences for the Training-Test Split, AT&T Bell Laboratories. Kilimasaukas, C.C., Neural nets tell why. Dr Dobbs's, (April 1991) 16-24. Andrews, R., J. Diederich, and A. Tickle, A survey and critique of techniques for extracting rules from trained artificial neural networks, Neurocomputing Research Centre, Queensland University of Technology (1995). Shavlik, J.W., A Framework for Combining Symbolic and Neural Learning, Computer Science Department, University of Wisconsin: Madison (1992). Cristea, A., P. Cristea, and T. Okamoto, Neural Network Knowledge Extraction. Série Électrotechnique et Énergétique. 42(4) (1998) 477-491. Gemen, S., E. Bienenstock, and R. Doursat, Neural Networks and the Bias/Variance Dilemma. Neural Computation. 4 (1992) 1-58. Taha, I. and J. Ghosh, A Hybrid Intelligent Architecture and Its Application to Water Reservoir Control, in Department of Electrical and Computer Engineering, University of Texas: Austin, TX (1995). Wermter, S. and R. Sun, Hybrid Neural Systems, New York: Springer (January 2000). Sordo, M., H. Buxton, and D. Watson, A Hybrid Approach to Breast Cancer Diagnosis, in School of Cognitive and Computing Sciences, University of Sussex: Falmer, Brighton (2001). Osorio, F., INSS: Un Systeme Hybride Neuro-Symbolique pour l'Apprentissage Automatique Constructif, in INPG - Laboratoire, LEIBNIZ - IMAG: Grenoble - France (1998). Weaver, R.A., J.A. McDermid, and T.P. Kelly. Software Safety Arguments: Towards a Systematic Categorisation of Evidence. in International System Safety Conference. Denver, CO (2002) Rummelhart, D.E., G.E. Hinton, and R.J. Williams, Learning representations by backpropagating errors. Nature. 323 (1986) 533-536. Leveson, N., Safeware: system safety and computers: Addison-Wesley (1995). MoD, Defence Standard 00-55: Requirements for Safety Related Software in Defence Equipment, UK Ministry of Defence (1996). SAE, ARP 4761 - Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne Systems and Equipment, The Society for Automotive Engineers (1996). MoD, Interim Defence Standard 00-58 Issue 1: HAZOP Studies on Systems Containing Programmable Electronics, UK Ministry of Defence (1996). Kelly, T.P., Arguing Safety – A Systematic Approach to Managing Safety Cases, in Department of Computer Science, University of York: York, UK (1998).
