Model-Checking and its Relevance Padmanabhan Krishnan Department of Computer Science, University of Canterbury, Christchurch, New Zealand Email:
[email protected] Abstract This article is like a tutorial. It introduces an automatic technique to verify programs called model checking. There are various ways this task can be performed. We select a point in the spectrum and explain the key ideas. These include modelling programs as automata, specifying the required behaviour in terms of a temporal logic and using the output generated by a model checker to debug the program. We also provide a variety of references on the different approaches to model checking.
1.
Introduction
In his invited talk J. Cuellar [8] discusses the features necessary for a new technology to gain industrial acceptance. The new technology he specifically considers is formal methods or the use of mathematically sound (well understood) techniques for developing software. His presentation concludes with a statement that model checking is an example of a successful technology. The main reason for this conclusion is that the benefits of model checking are clear. The costs of using model checking is not high. Also it can be easily integrated into existing design processes. In other words, model checking does not interfere with the normal design process nor does it demand a high level of expertise for it to be used successfully. A similar claim is made by C. Heitmeyer [11]. It is argued that formal analysis should be as automatic as possible and provide good feedback. Again model checking is used as an example of a promising technology. This paper provides an introduction to model checking. Here we describe the various concepts involved at a very high level. There are two (perhaps conflicting) aims; the first is to convince the reader, who may not be an expert in theoretical computer science or mathematics, that model checking is indeed a useful as well as an accessible technique. The second is to convince senior students that topics related to automata theory can really be used in practice. Our exposition will demonstrate the various steps required to perform the model checking task. So what is model checking. Simply put, model checking is the automatic process of checking if a program satisfies a given property. The term model checking comes from
logic. The meaning of a logical formula is defined in terms of what is called a model. For us, programs act as models. That is, they are a representation for some real world object. The aim of reliable software development is to ensure that the program we write has appropriate behaviour. In other words, it satisfies the requirements imposed on it. The properties that the program (model) needs to satisfy are expressed in some logic. Hence model checking is the process of checking that the program is a model for the formula (i.e., it satisfies the formula). There are two principal advantages of model checking. The first is that it is exhaustive. Hence it is superior to simulation, debugging etc. Usually in simulation and debugging, the programmer has various scenarios in mind. These scenarios are then tested for expected behaviour. However, in a large system, there could a number of scenarios that the programmer may have missed. Such behaviours could arise due to various reasons. For example, the programmer might make certain incorrect assumptions. Also there could be subtle interaction between the various components which leads to unexpected behaviours (race conditions are typical examples). As model checking is exhaustive, all possible behaviours are tested for conformance. Hence it is superior to the traditional simulation or debugging techniques. The second is that if the right choice of programs and logic is made, the process can be completely automated. Because the process is automatic, the user of the tool does not need to understand how the verification process proceeds. If the verification process succeeds, they can go to the next task. If it fails, the process will provide hints on why the program failed to satisfy the property. The user can use these hints to debug the program or perhaps conclude that the program was indeed correct and the property was incorrectly constructed. This approach is not very attractive if the language to express the properties is not sufficiently powerful. A developer will not be interested in spending a lot of time developing the properties to be tested. In practice, the properties that one is usually interested in are fairly compact when compared to the program. A typical example is an air traffic control system. Two of the simplest properties it needs to satisfy are “at no time should two places occur the same location” and “every plane that wishes to take off or land eventually does so”. These properties can be expressed very compactly in various logics. However, a program which actually achieves this is in general quite complicated. Other typical properties include deadlock avoidance, reachability of desirable states, unreachability of unsafe states, etc. There are various approaches to model checking. They depend on various technical issues such as the choice of logic, the representation of the program etc. The various choices made have pragmatic implications. But we will not concern ourselves with this as yet. We consider a simple form of the problem statement and develop this in a precise fashion. The reader is referred to [9, 20] for technical discussions related to
model checking. In this introduction, we discuss the issues using intuitive ideas. These ideas are only to illustrate the process. In practice, these ideas have to be refined and more efficient techniques used. Hence the reader should not use our presentation to develop their own model checker. In any case we do not expect the readers to build model checkers, but we hope that they at least try to use a model checker. The purpose of this tutorial is to show what a model checker does and hence provides hints on what to expect from a typical model checking tool.
2.
Programs as Automata
The main technical material required for this tutorial is the notion of a state transition diagrams (or automata). This then gives a deeper understanding of the model checking process. Those who have studied finite automata formally can skip this section. There are also commercial tools from vendors like I-Logix (http://www.ilogix.com) that use automata at a reasonable level of abstraction. Other formalisms like Message Sequence Charts can also be viewed as automata. The use of automata models for describing requirements is described in [16]. Here we take a very general view and explain key ideas using imperative program fragments. We represent the essence of a snapshot of a program execution as a collection of variables along with the values they hold. Call this the state of the computation. The behaviour of the program is then described by a sequence of states that are visited. Consider the code shown in Figure 1. i :=0; b := true; j := 10; while b do read(i); if (i > j) then b := false; else j := j+ i; endif; endwhile; Figure 1: Sample Program.
A representation of a behaviour of this program on the inputs 5 followed by 20 is shown in Figure 2. In this representation, there are three states of interest. This idea can be applied to any program to obtain a suitable automaton. In fact automata based methods like Statecharts [10] can be found in a variety of application domains.
i = 0, b = true, j = 10 /
i = 5, b = true, j = 15 /
i = 20, b = f alse, j = 15
Figure 2: Sample Behaviour.
{p1 }
i = 0, b = true, j = 10
/
i = 7, b = true, j = 17 {p2 }
∅
i = 3, b = true, j = 21 o
i = 1, b = true, j = 18
{p1 ,p2 }
i = 25, b = f alse, j = 21
Figure 3: Labelled Behaviour.
As a general model is not amenable to automatic techniques, we will assume that the set of variables used is finite and the domain of values for each variable is only finite. This ensures that the state transition system we obtain is finite. In the above example, we have not placed any labels on the transitions. It is useful to have labels on the transitions where the labels indicate some observation. For example, let the focus of our attention be whether the variable i is a multiple of 3 (call this property p1 ) and the variable j is odd (call this property p2 ). Now the labels will indicate whether these properties hold or not. Consider another input sequence where the input is 7, 1, 3 and 25. The labelled transition system is shown if Figure 3. This transition system is very similar to tracing variables while debugging. As model checking is nothing but exhaustive debugging, we will use these labels in model-checking process. Given our brief description and example, we hope that the reader is convinced that a program can be represented by a state transition diagram. Furthermore, we can label these transitions by the properties that the states satisfies. As automata models can be created at most stages of design model checking can be used at various stages of software design. For example, a software architecture can be viewed as an automaton [15]. Hence one can verify properties satisfied by the design. This helps the early detection of design errors.
3.
Logic and Automata
There are a number of logics that can be used to express properties. These include linear time temporal logic (LTL), computation tree logic (CTL), CTL with recursion (CTL∗ ), modal µ-calculus etc. Each logic has its strengths and weakness. Each logic also requires the program automaton to be interpreted in a particular way. In this tutorial we consider LTL to express the various properties. This is because it is fairly simple to understand and all behaviours are represented in terms of sequences. We assume a set P which contains the basic state properties. In the example considered in the earlier section p1 and p2 will be elements of P . The syntax of the logic is defined by the following grammar. φLT L ::= (p ∈ P )
¬α
(α ∨ β)
α
αU β
A state formula is automatically a behavioural formula. In order to establish this, the current state must have this property. The connective ¬ represents the usual boolean negation while ∨ represents the usual boolean or. Using this, other boolean connectives such as and, equivalence, logical implication can be encoded. The two new connectives describe behavioural properties. The formula α will be satisfied by a program which on the next step (i.e., after one transition) can satisfy α. Note that if α is a state formula, α indicates that the program will make α true in one step. The connective U (for until) gives power to this logic. To satisfy the formula α U β requires the following. Firstly there is some state reachable from the initial state where β holds. Furthermore, in all the intermediate steps α holds. Hence a natural language reading of α U β is “β holds eventually and also α holds until β holds”. Based on this various one can derive other useful connectives. For example by defining 3α to be T rue U α, we get eventually α. That is because T rue always holds. By defining 2α to be ¬(3(¬α)) we get henceforth. That is, 2α requires that α holds at each and every state of the computation. Hence it represents an invariant property. One can use graphical techniques to capture these logical properties [18]. For example, figure 4 captures the intuition behind the formula p Uq. p o
/
q Figure 4: Until: Pictorially
One could also use natural language to express these properties. For example, one can actually write Eventually p instead of 3p or Always p instead of 2p. The key point is that the logical formulae are fairly concise.
{p}
s0
0123 7654
{q},{p,q} /
s1
0123 7654 '&%$ !"#
Figure 5: Formula Automaton
The logic is quite powerful and one can express various properties. For example, 2(3p) expresses the property that p will occur infinitely often. In other words InfinitelyOften p is an abbreviation for Always (Eventually p). The formula 3(2p) expresses the property that p will eventually hold for ever. That is, p stabilises or Stable p is Eventually (Always p). The formula 2(p ⇒ 3q) expresses the fact that for every p, there is a q that occurs after it. Note that this does not require a q for every p. One could have a sequence of p’s followed by a single q that satisfies this property. Such a property is useful to model the following situation. An impatient user may press a key numerous times before any action is taken. One usually wants to take a single action for all the key presses. One could also define specific macros [14] which are intuitively well understood. We have shown a few like Always, Stable etc. In the context of computer hardware, properties such as after an instruction is f etched, it will eventually be decoded can be specified. Another useful macro is Bef ore p q which states that q cannot hold till p holds. Furthermore once p holds q will eventually hold. The set of properties one is interested in can be perceived as having a safety component and a liveness. A safety property indicates that nothing bad happens while a liveness property indicates that something good happens. In an air traffic control system, collision avoidance is a safety requirement (written as Always noCollision) while planes taking off and landing is a liveness issue (written as requestToLand => Eventually landed). Such a classification helps the construction of the formulae. A key technical result is that for every logical formula there is a corresponding automaton that generates all its models. That is, there is a mechanical way of converting a formula to a program. The program will be given a sequence of properties that are true and if the sequence will satisfy the formula, the program will accept the sequence. For example, the automaton shown in figure 5 corresponds to the formula “eventually q”.
4.
Model Checking
We have so far looked at how programs and properties are viewed in terms of finite state automata. We now focus on the process of model checking a particular program. As model checking is exhaustive, every program behaviour must satisfy the formula. In other words, every program behaviour must not satisfy the negation of the formula
in question. More formally, model checking can now be modeled as a language inclusion problem. That is, all runs of the program (i.e., now viewed as a sequence of sets of properties) should be valid models for the formula. Hence all program generated sequences must be accepted by the formula automaton. In other words L(P ) must be contained in L(Aα ). This can be viewed in another way. No run of the the program should be a model of ¬α. Mathematically, L(P ) ∩ L(A¬α) must be the empty set. Here we do not describe the intersection construction in detail. We present the basic idea which can be used to understand the counter examples various model checkers generate. This construction is the key to obtaining a counter example if the model checking fails. If the state space of the program is represented by Q1 and the state space of the negated formula automaton is represented by Q2 , the state space of the intersection automaton is represented by the set of triples belong to Q1 × Q2 × F l where F l is a set of flags used for technical reasons. The transition from one state to the other o (i.e., (q1 , q2 , f ) −→ (q10 , q20 , f 0 ) will occur if both the individual state components can make that move. That is, the program can evolve from state q1 to state q10 exhibiting the observation o (which is a collection of properties) and the formula automaton can also evolve from state q2 to state q20 exhibiting o. Depending on the states and the the observation the flag f 0 is derived from f . A valid behaviour represented by this automaton can be modeled as a path from the initial state to a “satisfactory” state. Therefore, behaviour in the intersection automaton o0 o1 on (q11 , q21 , f 1 ) −→ . . . (q1n , q2n , f n ) −→. will be characterised by (q10 , q20 , f 0 ) −→ This represents a behaviour of the program that satisfies the negation of the formula. Hence one can construct a run of the program that violates the specification. In order to obtain this we just drop the second and third components (components related to the formula automaton and the flags which are of no interest to the program developer), thereby obtaining an invalid behaviour of the program. Hence we will get o0 o1 on a sequence of the form q10 −→ q11 −→ . . . q1n −→ The programmer now has to translate this sequence of states back to the original program. The complexity of this step depends on the how the program was translated to the automaton. In general this is not too difficult as one usually needs to look only a certain aspect of this run. Also if the translation of the program into its automata form is understood, the reverse process is not too hard. Now the program can be debugged using this counter example. It is possible to extend the above technique to generate a class of counter examples. Hence the process of model checking could yield a number of counter examples where each counter example represents a different run of the program. If the model checking succeeds, no such run can be produced. That is, there is no
Direction 1
Direction 2
Figure 6: Traffic Flow
. run of the program that can be used to satisfy the negation of the formula in question. Technically, the language common to both is empty. While model checking does not do away with the iterative nature of program testing, the nature of the testing is exhaustive at each step. Hence one would detect bugs early in the design phase.
5.
A Simple Example
We now present a simple example to illustrate the model checking process. We focus on the actions the user needs to take to effectively use model checking. For this exercise we use the model checker component of the tool STeP [3]. The example is a simplified traffic intersection controller. We will assume a main two-way street where cars can go straight, left or right. We will not consider traffic from the side street merging into the main street. The possible behaviour is shown in figure 6. We number the directions for ease of reference. To model this, we have a queue for each traffic flow. For example, the queue q1s represents the queue of cars approaching from direction one wishing to go straight. Hence we have six such queues; one queue for each direction of travel. For the purposes of model checking, we will assume that the queues can hold up to two cars. We have a traffic light (which can be red or green) for each traffic flow. Hence the variable tl1s will be associated with the queue q1s. Each queue is controlled by a process. A typical process is shown in Figure 7. This process is completely non-deterministic. Cars can arrive at the queue (in which case the counter is incremented). If the queue is not empty the appropriate traffic light is made green and all the other ones which may result in collision is made red. If the light is green and there is any car in the queue, the counter is decremented. Note
Qs1 :: [ loop forever do [ > or 0 /\ tl1s = Green) then q1s := q1s -1 >> or 0 then [ tl1s := Green; tl2r := Red]>> ] ] Figure 7: Queue: Example Program.
that the process associated with the flow s1 is in direct competition with the process associated with flow r2. Hence there is a possibility for various race conditions to occur. The finite automata representation used by STeP is shown in Figure 8. The key thing to note is that there is one control state (denoted by pi0). This represents the start of the loop. There are three possible transitions (corresponding to the three if statements. If the preconditions are false no change occurs. Part of the state transition diagram (with values) is shown in Figure 9. This is based on represented on the following notation. We denote the states as triples (the first value is q1s the second is tl1s and the third is tl2r. We also use the following abbreviations for the properties, viz., C1 = {q1s < 2} C2 = {q1s > 0, tl1s = Green} and C3 = {q1s > 0}. Note that the values of tl2r etc. will be altered by the process controlling the appropriate queue. The arrows marked C2 coming into states [1, green, red] and [2, green, red] are self loops indicating that while the queue is greater than 0, no change to the lights occurs. The first property we check is a safety one. That is, at all times if tl1s is green then tl2r is red. This is specified as 2(˜((tl1s = Green)/\(tl2r=Green))). That is, both the traffic lights are not green at the same instant. The model checker verifies that this property holds. This fact indicates that the three states marked [0/1/2, green, green] will never be reached. Hence they are spurious states in our diagram and can be removed. The next property is a liveness requirement. We test if one will eventually reach a
Transition $1 Just: enable pi0 = 0 /\ ˜(q1s < 2) assign pi0 := 0 ; enable pi0 = 0 /\ q1s < 2 assign q1s := q1s + 1,pi0 := 0 Transition $2 Just: enable pi0 = 0 /\ ˜(0 < q1s /\ tl1s = Green) assign pi0 := 0 ; enable pi0 = 0 /\ 0 < q1s /\ tl1s = Green assign q1s := q1s - 1,pi0 := 0 Transition $3 Just: enable pi0 = 0 /\ ˜(0 < q1s) assign pi0 := 0 ; enable pi0 = 0 /\ 0 < q1s assign tl1s := Green,tl2r := Red,pi0 := 0 Figure 8: Transitions in STeP.
C1
[0, red, red]
/
C2
[0, green, red]
C1 .
C3
/
C1
[1, green, red]
.
C3
.
n
C3
[1, green, green]
[2, green, red] O
C2
[0, green, green]
C2
n
O
C1
[2, red, red] C2
C2
n
C1
[1, red, red]
C2 C1 .
n
C3
[2, green, green]
Figure 9: Automaton Representation.
situation where traffic in both directions can turn left. Note that there is no collision possible when two cars from opposite directions are turning left. This is written in our logic as 3(((tl1l = Green)/\(tl2l = Green)). The model checker verifies that this formula does not hold. Hence there is a behaviour that violates it. This is because the system is not fair. While the above condition may hold, there is no guarantee that it will indeed hold. The model checker produces a run of the program (a long output that takes careful analysis) which violates this property. While we do not reproduce the entire counter example, we give a taste of what the output looks like. The entire state space is represented as a tuple of values. This is based on the ordering q1l, q1r, q1s, q2l, q2r, q2s, tl1l, tl1r, tl1s, tl2l, tl2r, tl2s. For example, the sequence [1, 0, 0, 1, 1, 2, 1, 0, 0, 0, 0, 1] states the q1l is equal to one, q2s is equal to two etc. Each transition step in the counter example is written as a triple. An example is shown below. state: [0,0,2,0,1,0,0,0,0,1,1,0] trans: $5$r state: [0,0,2,0,1,0,0,0,1,1,0,0] This shows what the values of the various variables are and what transition is applied to change the state. The transitions are numbered in the counter example, but in a different part of the output the transition is explained in detail. The transition numbered $5$r corresponds to value of tl1s being changed from Red to Green and the value of tl2r being changed from Green to Red. The values 0 and 1 indicated the index in the enumeration in the program. This corresponds to the third option associated with Qs1 shown earlier. By stepping through the various transitions one can study the behaviour of the program that violates the specification. One must keep in mind that STeP is a prototype tool. Hence it the time to perform the verification is reasonably high. Also its output is not the most readable. To be fair to STeP, it is far more than a model checker. It has various theorem proving features (where theorems can be interactively proven) which is beyond the scope of what we need. Commercial tools with better user interfaces and structured output can be purchased. Other tools (some of which are quite powerful) are discussed in the next section. In conclusion, the program was developed in a Pascal like language. This step is not very different from the usual process. The construction of the logical formula requires some training. However, they are usually very intuitive and the translation into the syntax required for the logic is not too hard. The model checking process is automatic. The use of the counter example to debug the program takes some time to understand.
6.
Other Relevant Work
We now cite a variety of results from the literature to convince the reader that model checking can be used to handle realistic projects. While we have presented the basic idea behind model checking, there are a number of approaches. For example, [5] present a technique where large programs can be model checked very efficiently. The program models and the logic are slightly different. Technically, they are based on branching structures (instead of linear structures) and computation tree logics (CTL). There are many concrete examples of model checking in the literature. For example, [21] show how they can verify hyperdocuments using a trace based approach. Verification of real-time systems has been been addressed in [22] while the verification of audio protocol has been reported in [2]. The article [6] presents the model checking of certain types of concurrent C++ applications using the model checker SPIN [12]. As one uses a restricted version of C++ to model the systems, it is possible to verify a prototype and then extend the prototype without extensive recoding. The following figures give an approximate idea of performance of various good model checkers. While interpreting the figures one must keep in mind that the actual numbers are influenced by the speed of the processor and the available memory. Usually, as the amount of available memory increases there is a drastic reduction in time taken to perform the verification. In [7] the authors report the verification of distributed mutual exclusion circuit. It is a token ring network with a number of nodes. For a circuit with 7 nodes, verification of safety properties takes around 325 seconds while verification of liveness properties takes 97800 seconds. The specification has about 108 states. [4] report using statecharts to model systems and then using model checking to verify various properties. They report that the verification of a collection of properties associated with a central car locking mechanism (involving 12 parallel automata each having 3 to 20 states) took about 1785 seconds. They also report that the verification of an airline storage management system took 966 seconds. [17] verify a gear controller where timing requirements are crucial. There are less than 100 untimed states but with the various timing constraints, the number of states is quite large. Verification of 46 properties took only 2.99 seconds. [19] use SDL to model certain aspects of the GSM (mobile communication) protocol. The modelling of this protocol has about 108 to 1013 states. To verify certain properties took a maximum of 30 seconds for the smaller model and up to 120 seconds for the larger model. The main reason for the vast difference in the times is the relative importance of
data (specific values) against control. If the verification is mainly concerned with the control aspects the times are quite small. However, when specific data values are of interest, the model checking process takes reasonable time. The key is to balance the level of data and the level of control. An important extension to the basic idea is that of verification by modules. As large systems are developed in modules, it is more efficient to verify the individual modules. However, this process should guarantee that the required property holds when the modules are combined. Such a system is described in [1]. Here again the logic used is slightly different. One cannot have the full generality of CTL and hence have to consider alternating temporal logic. These examples show that one can adapt the basic idea behind model checking to a wide variety of systems. From a users perspective, the key is to identify the size of the problem and the type of model checking desired. Once these choices are made, the specifics of the technique and tool to be used is more or less straightforward.
7.
Conclusion
This article has introduced the basic ideas involved in model checking. These include the modelling of programs as automata, the modelling of logical properties as automata and using the model checking process to obtain counter examples. We conclude the paper by presenting the effort required by the students to understand and use this process. The author has taught model checking for about five years. This is not a comprehensive survey. They are a summarisation of the notes taken by the author while marking assignments, exams and in answering student queries. The basic idea of programs as automata takes only an hour or two. This is mainly because nearly all the students have formally studied finite automata. A few who have not studied finite automata find it a bid hard to begin with but within about ten hours are quite comfortable with it. The logic presented in this paper takes about three hours to explain. Numerous examples are used to illustrate the usage. In the course the technical details of model checking process is explained due to which they can understand the counter examples easily. Without the technical details the students find it hard to understand the counter example. However, with the author’s help they were able to interpret the counter examples. This was sufficient to debug their programs. In any case most students had an idea of why their programs failed. All in all about 20 hours were sufficient to get a firm understanding of the process. If the technical details (formal description of the intersection construction) are ignored, 15 hours suffice. In conclusion we paraphrase [8] in that the automatic technique of model checking can be easily incorporated into the standard design cycle.
Acknowledgements The author thanks his various cosc402 students for their valuable comments on the course. Special thanks to Yvonne Simmons and Warwick Irwin for their comments.
References 1. R. A LUR , T. H ENZINGER , F. M ANG , S. Q ADEER , S. R AJAMANI , AND S. TASIRAN, MOCHA: modularity in model checking, in Hu and Vardi [13], pp. 521–525. 2. J. B ENGTSSON , W. G RIFFIOEN , K. K RISTOFFERSEN , K. L ARSEN , F. L AR SON , P. P ETTERSON , AND W. Y I (1996) Verification of an audio protocol with bus collision using UPPAAL, in Proc. Eighth International Worshop on Computer Aided Verification, LNCS1102, Springer Verlag, pp. 244–256. 3. N. S. B JORNER , A. B ROWNE , E. C HANG , M. C OLON , A. K APUR , Z. M ANNA , H. S IPMA , AND T. U RIBE (1996) STeP: Deductive algorithmic verification of reactive and real-time systems, in Proc. Eighth International Worshop on Computer Aided Verification, LNCS1102, Springer Verlag, pp. 415–418. 4. U. B ROCKMEYER AND G. W ITTICH (1998) Tamagotchis need not die – Verification of STATEMATE designs, in Tools and Algorithms for the Construction and Analysis of Systems (TACAS), B. Steffen, ed., vol. LNCS 1384, Lisbon, Portugal, Springer Verlag, pp. 217–231. 5. J. R. BURCH , E. M. C LARKE , K. L. M C M ILLAN , AND L. J. H WANG (1992) Symbolic Model Checking: 1020 States and Beyond, Information and Computation, 98 , pp. 142–170. 6. T. C ATTEL (1998), Modelling and Verification of sC++ Applications, in Tools and Algorithms for the Construction and Analysis of Systems (TACAS), B. Steffen, ed., vol. LNCS 1384, Lisbon, Portugal, Springer Verlag, pp. 232–248. 7. E. C LARKE , O. G RUMBERG , AND K. H AMAGUCHI (1997) Another Look at LTL Model Checking, Formal Methods in System Design, 10 , pp. 47–71. 8. J. C UELLAR, Formal methods in an industrial environment, in Hu and Vardi [13], pp. 57–60. 9. E. A. E MERSON(1990) Temporal and modal logic, in Handbook of Theoretical Computer Science: Formal Models and Semantics, J. van Leeuwen, ed., Elsevier, pp. 995–1072.
10. D. H AREL(1987) Statecharts: A Visual Formalism for Complex Systems, Science of Computer Programming, 8 , pp. 231–274. 11. C. H EITMEYER (1998), On the Need for Practical Formal Methods in Proceedings of the Symposium on Formal Techniques in Real-Time and Fault-Tolerant Systems, A. P. Ravn and H. Rischel, eds., vol. LNCS 1486, Lyngby, Denmark, Springer Verlag, pp. 18–26. 12. G. J. H OLZMANN (1997) The Model Checker SPIN, IEEE Transactions on Software Engineering, 23 . 13. A. H U AND M. VARDI, eds. (1998) Proc. Tenth International Worshop on Computer Aided Verification, LNCS1427, Vancouver, Canada, Springer Verlag. 14. P. K RISHNAN (1994) A Case Study in Specifying and Testing Architectural Features, Microprocessors and Microsystems, 18 , pp. 123–130. 15.
(1998), Independence in Architectural Descriptions, in SE:E&P Software Engineering: Education and Practice, IEEE, pp. 68–75.
16. N. L EVESON , M. H EIMDAHL , H. H ILDRETH , AND J. R EESE (1994) Requirements specification for process control systems, IEEE Transactions on Software Engineering, 20 , pp. 694–707. 17. M. L INDAHL , P. P ETTERSSON , AND W. Y I (1998) Formal Design and Analysis of a Gear Controller, in Tools and Algorithms for the Construction and Analysis of Systems (TACAS), B. Steffen, ed., vol. LNCS 1384, Lisbon, Portugal, Springer Verlag, pp. 281–297. 18. L. E. M OSER , Y. S. R AMAKRISHNA , G. K UTTY, P. M. M ELLIAR -S MITH , AND K. K. D ILLON (1997) A Graphical Environment for the Design of Concurrent Real-Time Systems, ACM Transactions on Software Engineering Methodology, 6 , pp. 31–79. 19. F. R EGENSBURGER AND A. BARNARD (1998) Formal Verification of SDL Systems at the Siemens Mobile Phone Department, in Tools and Algorithms for the Construction and Analysis of Systems (TACAS), B. Steffen, ed., vol. LNCS 1384, Lisbon, Portugal, Springer Verlag, pp. 439–455. 20. C. S TIRLING (1992) Modal and temporal logics, in Handbook of Logic in Computer Science, S. Abramsky, D. M. Gabbay, and T. S. E. Maibaum, eds., Oxford Science Publications, pp. 477–563.
21. P. D. S TOTTS , R. F URUTA , AND C. R. C ABARRUS (1998) Hyperdocuments as Automata: Verification of Trace Based Browsing Properties by Model Checking, ACM Transactions on Information Systems, 16. 22. J. YANG , A. M OK , AND F. WANG (1997) Symbolic model checking for eventdriven real-time systems, ACM Transactions on Programming Languages and Systems, 19 , pp. 386–412.
