A Generic Tool for Automatic Protocol Conformance Testing with Application to ATM Equipment* M. Alvarez-Campana, E. Vázquez, J. Vinyes Dept. Ingeniería de Sistemas Telemáticos Universidad Politécnica de Madrid Address: ETSI Telecomunicación, E-28040 Madrid, Spain. Tel: +34 1 3367330, Fax: +34 1 3367333 E-mail:
[email protected],
[email protected],
[email protected]
Abstract Protocol conformance testing provides a extremely valuable mechanism in order to reduce incompatibility risks when interconnecting equipment from different manufacturers. This is achieved by defining a common set of test cases that an implementation must pass to be considered compliant with a particular standard. The benefits of the conformance testing methodology have led major Standardization Organizations to its application to an increasing number of communication protocols (e.g. Asynchronous Transfer mode, Integrated Services Digital Network, Signaling System No. 7, etc.) Thus, nowadays it is common practice to find the definition of Abstract Test Suites (ATSs) along with the technical specifications of a protocol. Unfortunately, things are not that easy when it turns to apply an ATS to a real system. On the one hand, an ATS may consist of a considerable number of test cases. On the other hand, simple they may seem, their preparation and execution may require a long time for the test operator. Typical tasks involved in passing a test case are configuring the test equipment, building and sending the adequate protocol data units, monitoring the responses from the system under test, interpreting the results, etc. Furthermore, the possibility of introducing human errors in this process must not be neglected. This paper describes a generic conformance testing tool aimed to solve the above problems. The tool provides a general platform that allows the operator to select an ATS and execute all or part of the test cases without requiring manual intervention. The tool has been designed so that new ATSs can be easily added as needed. As an application, the ATM Forum ATS for ATM End Systems has been implemented within the tool.
∗
This work has been partially supported by the Spanish CICYT project CEPAT (Certificación de Protocolos ATM).
1. Introduction Standardization in the Telecommunication industry has play a major role in fostering the interoperability between equipment from different manufacturers. In practice, however, it is often the case that implementations drawn from the same standard do not perform right when connected together. Incompatibilities may be due to different causes: programming bugs, misinterpretations of the standard, systems supporting only a subset of the features described in the standards, etc. From the above discussion, it seems clear how valuable would be a methodology allowing users, providers and manufactures to know in advance whether two products will interoperate. The ideal solution would be to perform interoperability testing between the particular systems to interconnect. These tasks could be done by a certification laboratory so that efforts would not be duplicated. In practice, however, interoperability testing becomes unfeasible as the number of implementations grows. As it can be imagined, performing interoperability testing for every possible pair of existing products can be exhausting, even for a certification laboratory. That is the reason why another methodology, known as protocol conformance testing, is used in practice. Protocol conformance testing helps to minimize the chances of incompatibility by defining the test cases that an implementation must pass to be considered compliant with a particular standard. Although less rigorous than interoperability testing, conformance testing has the key advantage of only requiring each product to be tested once for each specification it supports. The benefits of protocol conformance testing have been widely recognized to the point that a standardized methodology, IS 9646 [1], has been internationally agreed. The ISO conformance methodology is being increasingly adopted by the major standardization bodies (e.g. ITU-T [2], ETSI [3], …) and other relevant organizations in the standardization field (e.g. the ATM Forum). Thus, nowadays it is common practice to find the definition of the conformance test suites along with the technical specifications of communication protocols. Examples of fields where the IS 9646 methodology has been applied are the Integrated Services Digital Network, the Signaling System No. 7, and the Asynchronous Transfer Mode, to cite just a few of them. The ISO conformance testing methodology has an additional attractive for the people who actually have to perform the testing. This is the possibility of considering the automation of the testing process. The IS 9646 specification contains some characteristics specially adequate for that purpose. To date, however, known works have not exploited the idea to its full extent. The work described in this paper represents an attempt to complete the automation of the whole conformance testing process. The rest of the paper is organized as follows. In section 2, we review the ISO conformance testing methodology. The possible tasks subject to automation are discussed in section 3, focusing in those aspects not covered by the known tools. Section 4 describes the generic tool for automatic conformance testing that has been developed by the authors. As an application case, section 5 describes how the tool has been used to test ATM equipment. Finally, in section 6 we summarize the main contributions of our work and outline some possible future research directions.
2
2. ISO Conformance Testing This section provides a general overview of the ISO Conformance Testing methodology. The objective is not to provide a detailed description of the standard but to introduce the terminology that will be used through the rest of the paper as well as a general background for the beginners. For a tutorial introduction to ISO Conformance Testing see [4,5]. The IS 9646 provides a consistent framework for the specification and execution of conformance testing for implementations of protocol standards. For that purpose, it defines the conceptual architecture shown in figure 1.
Upper Tester (UT)
[N] ASPs
PCO Test Coordination Procedure
[N] PDUs
Layer N+1
Implementation Under Test (IUT)
Layer N
[N-1] ASPs
PCO
Lower Tester (LT)
Layer N-1
Figure 1 - Conformance Testing Model
The particular part of the system that is to be tested is called the IUT (Implementation Under Test). The Upper (UT) and Lower Testers (LT) can be viewed as the Means of Testing (MOT) that are connected to the IUT at the upper and lower layer service interfaces, respectively. The connection is done trough the corresponding PCOs (Points of Control and Observation), where the exchange of ASPs (Abstract Services Primitives) takes place. Both testers are depicted together as surrounding the IUT to reflect the fact that a test coordination procedure between them is necessary. The abstract conformance testing architecture may adopt in practice different realizations. For example, it often happens that the IUT cannot be isolated from the SUT (System Under Test). This may be due either to physical limitations (the SUT inners cannot be accessed) or simply to the fact that there are not real service layer boundaries in the SUT. As a result, the tester will not be able to use PCOs adjacent to the layer under test. In these cases, the problem is usually solved by connecting a separate piece of test equipment to the SUT, either directly or remotely via a network, as depicted in figure 2. Several abstract testing methods (ATMs) based on this approach are described in IS 9646. 3
System Under Test (SUT)
Tester Test Coordination Procedures Lower Tester (LT) PCO
[N] PDUs
Layer N+1 Implementation Under Test (IUT)
Layer N
Service Provider Layer N-1
Figure 2 - Conformance Testing using a separate piece of test equipment
Assuming the test method has been decided, it arises the issue of determining the test cases that must be applied to the IUT. To this respect, IS 9646 specifies that it should be used the corresponding Abstract Test Suite (ATS) defined for the particular protocol standard that is being considered. An ATS is a standard itself that specifies the set of test cases that a protocol must pass to be considered conformant. As pointed out in the introduction, many ATSs have already been or are being defined by standardization organizations. ATSs are described using an informal language called Tree and Tabular Combined Notation (TTCN), described in part 3 of the IS 9646, which allows test cases to be modeled using an event tree approach. Although it is not a particularly elegant language, TTCN has the advantages of being easy to learn and supporting a machine processable form. This last feature can be exploited for deriving an Executable Test Suite (ETS) from an ATS, as discussed in section 3. The IS 9646 methodology defines also some aspects related with the test realization process. Figure 3 illustrates the three basic stages involved in the process, those being: • Preparation for testing. In this stage, the adequate ATM and ATS are selected taking into account the capabilities of the IUT. These capabilities are reflected in a standardized document called the Protocol Implementation Conformance Statement (PICS). The information contained in the PICS is also used to complete the Protocol Implementation Extra Information for Testing (PIXIT). This is a document that includes information not specified by the base standard, but indispensable for testing purposes (e.g. particular values for configuring the IUT and the tester). Once the required information have been completed, the next step is the preparation of the IUT and the tester. • Test operations. This stage is where the conformity of the IUT to the protocol standard is actually investigated. In a first phase, the PICS is analyzed in order to determine whether the static conformance requirements are satisfied. In case the IUT does not meet these 4
requirements, it is declared as non conformant and the testing process moves to the report production stage. If the static conformance review is satisfactory, a selection of test cases from the ATS is made. The resulting set of test cases will depend on the information contained in the PICS and the PIXIT. These information will also determine the particular parameter values (e.g timers, addresses, ...) to use when executing the test cases. At that point of the process, is when everything is ready to begin the execution phase or test campaign. The test campaign consists in running the selected test cases one after the other, registering the resulting verdicts (pass, fail or inconclusive). As a general rule, it is stated that if an error happens during a test campaign, it should be started from scratch. However, there may be cases where the interruption does not invalidate previously executed test cases. Under that assumption, it could be considered to resume the test campaign at the point it halted. • Test report production. The goal of the last stage in the conformance assessment process is the production of two documents. These are the System Conformance Test Report (SCTR) and the Protocol Conformance Test Report (PCTR). The format of both documents is defined in IS 9646. The SCTR provides an overall summary of the test results for the protocols for which the SUT has been tested. It includes also general information such as the name and version of the SUT or the dates of testing. The PCTR reflects a more detailed information of the testing performed for a particular protocol. This information includes a listing of the test cases that have been run together with their verdicts.
Begin
Preparation for Testing
PICS PIXIT
- Selection of ATM and ATS - Preparation of IUT and tester
IUT ATM + ATS MOT (ETS)
Test operations - Static conformance review - Test selection/parameterization - Test campaign
Static Conformance Results
Test Results Test Report Production
End
SCTR PCTR
Figure 3 - Conformance Assessment Process
5
3. Conformance testing automation As it was mentioned in the introduction, one of the advantages of conformance testing is that it reduces the number of test sessions when compared to interoperability testing. That is because conformance testing only requires the IUT to be tested with one other system, namely the test system. There is, however, an additional advantage that may be considered even more attractive, specially to the people actually involved in the testing process. That is the possibility of automating the testing process. It can be easily understood why automation is desirable by analyzing the costs associated to some of the main tasks involved in the conformance testing process. Let us consider that a testing laboratory is asked to test a given protocol implementation according to the corresponding standardized ATS. Assuming that no automatic facilities are available, the operator would typically proceed as follows. The first step would be to visually inspect the PICS and the PIXIT to perform the static conformance review. In case the static requirements were satisfied, the operator would proceed to select the applicable test cases from the ATS standard document. Once arranged the testing scenario, which would include the preparation of the IUT and the (manually operated) test equipment, it would start the execution of the test cases. Typically, the execution of each test case would involve the following steps: configuring the test equipment, building and sending the adequate PDUs, monitoring and decoding the responses from the system under test, interpreting and writing down the results, etc. Obviously, the above procedure can be quite tedious if manually performed, specially considering that an ATS may consists of a considerable number of test cases. Furthermore, the possibility of introducing human errors during the testing process should not be neglected. From the above discussion, it seems clear how advantageous would be to automate, at least partially, the conformance testing procedure. The benefits would be obvious: shorter testing times, lower costs, higher confidence in the results, repeatability of the tests, etc. Fortunately, the IS 9646 specification contains some characteristics that make the automation idea feasible. There are two main areas within the conformance testing methodology where automation can be considered. They are discussed in the following sections. 3.1 Implementation of the Abstract Test Cases Test cases defined by an ATS are abstractly specified so that they cannot be directly executed by a real test equipment. As mentioned before, trying to manually run a single test case can be an exhausting and error-prone task. It seems clear then that any attempt to automate the execution of test cases becomes essential. Fortunately, conformance test cases are specified in TTCN, a language that supports a machine processable form (TTCN-MP). This fact has led to the development of compilers from TTCN to conventional programming languages (e.g. to C language)[6,7]. This kind of tools can be used to obtain the skeleton of the program that models the dynamic behavior of a test case. The generated code may also include calls to functions from a library supporting general tasks such as handling of timers, tracing of messages (e.g. to display a verdict), etc. Sometimes this can be complemented with the use of tools for handling of
6
PDUs (e.g. an ASN.1 compiler for encoding/decoding application layer PDUs). By using the appropriate combination of tools, the process of deriving the an Executable Test Suite (ETS) from the ATS can be highly automated. 3.2 Management of Conformance Test Sessions To date, most of the known efforts to automate the conformance testing methodology have focused in the ATS to ETS derivation process. There is, however, another area where automation can be helpful as well. It has to do with several aspects related to the management of conformance test sessions. A conformance test session can be defined as a realization of the ISO conformance assessment process for a particular system with regard to a particular protocol standard. Therefore, a given system may be involved in as many testing sessions as protocols it supports. Similarly, there may exist as many testing sessions for a protocol as potential implementations. It cannot be imagined a testing laboratory required to perform all possible tests sessions. However, it can be asked to test the same protocol for several systems, or several protocols for the same system. The number of test sessions can be even higher if we consider the possibility of system and protocol upgrades. As a result, the laboratory may be confronted to the problem of managing a considerable number of test sessions. Several aspects within the management of test sessions can be considered for automation. One of them can be the use of databases to store information about individual systems (e.g. name and version number), protocol implementations (e.g. the PICS), abstract test suites (e.g. the TTCN specification of test cases), test sessions (e.g. the test verdicts), etc. The information could be used to automatically perform a number of tasks such as the review of static conformance requirements, the generation of standard reports (i.e. the SCTR and the PCTR), etc. A further automation degree can be obtained by integrating the databases and the test equipment. The result can be a powerful conformance testing platform. The advantages of this integrated approach are analyzed in the following section, where a generic tool for automatic conformance testing developed by the authors is described.
4. The Generic Conformance Testing Tool This section presents the main features of a generic protocol conformance testing platform developed in accordance to the ideas just discussed. The platform incorporates a number of facilities for assisting the test operator through the conformance assessment process. More specifically, it provides an automated testing environment both for the execution of test cases and for the management of test sessions. The tool has been designed following an open approach so that new protocol test suites can be easily added as required. This allows the operator to select an executable test suite off the shelf and perform a test session on a given protocol implementation. This fact contrasts with other known realizations, which are generally protocol-specific. The general architecture of the conformance testing platform is shown in figure 4. The tool has three main parts: the user interface, the test management module and the test execution modules. They functions performed by each module are described in the following paragraphs. 7
Conformance Testing Tool
User
User Interface Module (UIM)
Test Execution Modules (TEMs) ATS ‘c’ ATS ‘b’
Test Management Module (TMM)
Commands
ATS ‘a’ (e.g. ATM_UNI_ES)
Responses
Test Session Database Test Equipment (e.g. ATM Analyzer)
Figure 4. Architecture of the Conformance Testing Tool
• User Interface Module (UIM). The UIM allows the operator to interact with the conformance testing platform through a window-based graphical interface. The user interface provides the test operator with a number of controls (menus, buttons, text windows, etc.) for performing test sessions. The meaning of the controls and how they are used during a typical test session, will be presented later. By using the UIM controls, the user can select an ATS, load an IUT, request the execution of a test case, save and print test results, etc. These actions are actually carried out by the underlying module, the TMM, described below. • Test Management Module (TMM). This module represents the heart of the conformance testing platform, being responsible of managing all aspects related to the test sessions. The TMM functions can be classified into two major categories: database handling functions and test driving functions. The TMM makes use of the database approach discussed in section 3.2. The database handling functions are used to store/retrieve information related to an ATSs, an IUTs, or a test session. These functions are automatically invoked as a consequence of the stimuli received from the UIM or the TEMs (see below). The test driving functions are used to control the execution of test cases during a test session. When the user requests a test case to be executed, the TMM invokes the corresponding TEM. During the test execution, the TMM monitors the responses sent back by the TEM, which may be trace messages (e.g. to keep track of the TTCN code) or the test verdict (pass, fail, inconclusive). The TMM processes these responses in order to notify the UIM and to update the database. The repertory of messages exchanged between the TMM and the TEMs is shown in table 1.
8
Table 1. TMM/TEM Interface Message Repertory
Direction
Category
TMM toTEM
Control
Name EXEC STOP
Control
STOP
Verdicts
PASS FAIL INCONCLUSIVE
TEM to TMM
NOTAPPLY
Traces
TTCN PDU INFO ERROR
Meaning request the test case execution to be started request the test case execution to be aborted notifies that the test case execution has ended verdict pass verdict fail verdict inconclusive undefined verdict, selection reference failed ttcn code pdu contents debug info error message
• Test Execution Modules (TEMs). Using the IS 9646 terminology, a TEM is equivalent to an ETS. Each TEM concentrates all aspects that are specific to the ATS it implements. Therefore, there will be a separate TEM for each ATS supported by the tool. As mentioned above, the execution of the test cases is driven by the TMM. The execution itself is performed by the TEMs, which implement the dynamic behavior of the test cases and interact with the test equipment. Therefore, each TEM depends both on the ATS and on the particular piece of test equipment that is being used. Section 5 describes an example of TEM developed for the conformance testing tool. Figure 5 shows the general aspect of the UIM. Typically, right after starting the user interface, the user will load an ATS and an IUT. This is done through the pull-down menus Test Suite and Implementation, which open the appropriate temporary file selection windows. Once the ATS and the IUT have been selected, the tool checks for the existence of a previous test session. If that is the case, the tool recovers the status of the test session from disk, displaying the corresponding verdicts, log traces, etc. If no test session exist for the selected ATS-IUT combination, a new one is created. The top window of the interface shows a summary of the test cases together with the verdicts (either pass, fail, or inconclusive) for the cases that have been already executed. Otherwise, the entry in the verdict column for test cases not executed will be either empty or showing a Don’t apply message. The latter means that the test case cannot be executed because the IUT (i.e. its PICS or PIXIT) does not meet the selection reference clauses described in the ATS. The user can select individual test cases by clicking on the corresponding line. The line is then highlighted and a textual description of the selected case is displayed in the middle window.
9
T e st S uite T est suite :
Im plem e nta tion
S ession
H elp
Im ple m e ntation:
A TT MM__ UU NNII __EE SS
SSa m pp l e __ IIUU TT
1
Ver_Gen_GFC0_VP
Inconclusive
13/12/97
09:31:06
2 3
Ver_Gen_GFC0_VC Ver_Gen_Unass_Cell
Inconclusive Pass
13/12/97 16/12/97
09:31:10 12:56:59
4
Ver_Disc_OAM_Seg_VP_LB
Don´t apply
13/12/97
09:31:14
T e st ca se na m e: V e r_ G e n_ U na s s_ C e ll P urpose : V e rify that if th e IU T ha s a p h ysica l la ye r th at h a s sy nc h ro no u s c ell tim e slo t, it g e n e ra te s a n d a d d s u na s sig n e d ce ll to th e a s sign e d c e ll s tre am to b e tra n sm itte d, tra n sfo rm in g a n o nco n tin u o us ce ll stre a m o f assig ne d ce lls into a co n tin u ou e s strea m of a ssig n e d a n d un a ss ig ned ce lls. Test case Ver_Gen_Unass_Cell started at 16/12/97 12:56:56 ---------------------------------------------------------| Label | Behaviour Description | Constraints Ref | V | ---------------------------------------------------------| | START T_Test | | | | | LT_PCO?CELL | CELL_UNASSIGNED | P | ---------------------------------------------------------Test case Ver_Gen_Unass_Cell ended at 16/12/97 12:56:59
Exec C ancel Log O n
B a tc h O ff
E la p s e d tim e :
00:00:03
Figure 5. Graphical User Interface
The execution of individual test cases can be initiated by clicking twice in the corresponding line or by pushing down the Exec button. During the test execution, the bottom window shows traces of the actions performed by the TEM. The user can individually activate or deactivate three types of tracing information: TTCN code, PDU contents and execution progress messages. The execution of a test case can be aborted at any time by pushing the cancel button. The operator can also request the execution of several test cases in batch mode. This operation mode requires the Batch toggle button to be set to on. The set of test cases to execute can be selected either manually or by specifying one ore several matching criteria from a list. Examples of predefined categories are tests not executed, test requiring/not requiring operator intervention, test with verdict FAIL, .etc. One the selection has been done, the execution can be started by pressing the Exec button. This action causes the selected test cases to be sequentially executed by the tool. The tool includes a number of additional features such as the test report generation (SCTR and PCTR), handling of PICs and PIXIT, etc. This section has tried only to summarize some of the main features of the tool. The purpose is not to provide a detailed description of the tool (which can be found in [8]) but to emphasize the huge potential that can be obtained by using an automated conformance testing environment. The benefits can be further understood by considering the application of the tool to a real case. This is precisely the subject of the following section, which describes the implementation of an ATS for testing ATM equipment.
10
5. Application Case: Testing of ATM End Systems As mentioned in the introduction, the number of protocols for which standardized ATSs have been or are being defined is continually increasing. This trend has reached, as it could be expected, to the family of protocols related to the Asynchronous Transfer Mode (ATM). So, one of the major organizations in the standardization of ATM protocols, the ATM Forum, has specified to date several ATSs [9]. This section describes the implementation of one of the ATM Forum’s ATSs, namely the Conformance ATS for the UNI 3.0 ATM Layer of End Systems [10]. The ATS, from now on ATM-UNI-ES, defines a total of thirty seven test cases. They are intended for testing the ATM Layer for end systems at the UNI 3.0 [11]. The test cases are grouped into three categories as follows: • General. This test group deals with the ATM cell structure and encoding at the UNI. The objective is to verify general connectivity issues as, for example, that the IUT encodes all GFC bits to zero in a VPC. • Cell Discrimination. The purpose of this test group is to verify the ability of the IUT to perform cell discrimination based on VCI values and PTI values. • Management Plane. This test group tries to determine whether the IUT performs the management plane functions according to the specification. The test cases verify things such as the OAM cell formats, fault management flows, ... The preparation of a typical test case requires building the ATM cells to be sent with the header and the payload contents as specified in the ATS. The execution requires sending the adequate cells and monitoring whether the cells received in response are the expected ones and they arrive in time. Although most ATM analyzers include facilities to build, send, capture, and examine ATM cells, it is evident that a manual testing procedure may result overwhelming. As an alternative, an executable version of the ATM-UNI-ES ATS has been implemented and integrated as a TEM within the generic conformance testing tool described in the previous section. The TEM allows test cases to be automatically executed, reducing manual interventions to a minimum. In particular, by selecting the batch mode, a test session can be performed in less than five minutes. Figure 6 shows the test configuration considered† during the development of the TEM. As proposed in the ATM-UNI-ES specification, the configuration is based on the remote test method discussed in section 2 (see figure 2). The PCO is placed immediately below the protocol layer under test, that is, between the ATM layer and the Physical layer. Consequently, test cases are based in the exchange of ATM cells between the test equipment and the IUT. Recall that the remote test method considers a separate piece of test equipment connected to the SUT. In our case, an ATM board inserted in the computer containing the conformance testing tool (UIM, TMM, and TEM) was used as test equipment.
†
Note that, although not shown in the figure, the test equipment can be also indirectly connected to the ATM end system, via one or several intermediate systems.
11
Tester (ATM Test Equipment)
Tester
PCO
ATM
System Under Test (ATM End System)
IUT PDUs ATM Cells
PHY
ATM PHY
Figure 6. Test Configuration for the ATM-UNI-ES ATS
The software architecture of the ATM-UNI-ES TEM consists of four modules, as shown figure 7. The Test Interface Module (TIM) contains all the aspects that are not specific of the ATM-UNI-ES. They include the communication with the TMM based the exchange of messages discussed in section 4 (see table 1), and the access to the test session database in order to read the values of the IUT parameters required for the execution of the test cases. The execution of the test cases is driven by the TIM, which uses the TBM (Test Behavior Module) and the TDM (Test Declarations and Constraints Module). These two modules contain the ATM-UNI-ES specific functions resulting from the translation of the TTCN specification into C language. The translation was manually performed because of the relative simplicity of the ATS. In the future, a semiautomatic translation procedure based on existing TTCN tools can be considered for adding new ATSs. ATM_UNI_ES Test Execution Module Test Management Commands Module (TMM) Responses IUT parameters
Test Case Test Interface Module Selection Reference (TIM)
Test Session Database ATM Board Configuration
Test Behavior Module (TBM)
Test Declarations and Constraints Module (TDM)
ATM Board API
ATM Board
Figure 7. Architecture of the ATM-UNI-ES Test Execution Module
12
The TBM implements the dynamic part of the ATM-UNI-ES specification, that is, the state machines that model the behavior of the test cases. During the test case execution, the TBM makes use of the TDM, which contains the declarations (e.g. constants, parameters, variables, PDU types,...) and the constraints parts (i.e. the PDU constraints). The execution of a test case usually involves the exchange of two or three ATM cells between the ATM board and the SUT. The Application Programming Interface (API) allows the TBM to interact with the ATM board for sending and receiving ATM cells. The steps that take place during the execution of a test case can be described as follows. The process is initiated by the TMM, which invokes the ATM-UNI-ES TEM with the IUT name and the Test Case name as arguments. The former is used by the TIM to retrieve the parameter values of the IUT from the test session database. Before proceeding with the test case execution, the selection reference clause is evaluated by the TDM for the current combination of parameter values. If the selection reference fails, the TIM sends the messages NOTAPPLY and STOP to the TMM, and the test case execution ends. In case the selection reference clause is matched, the TIM uses the API to configure the ATM board for capturing cells on the appropriated VPI/VCI pairs. The values of the VPI/VCI pairs are determined on real-time from the IUT parameter values. T T C N s p e c i f i c a t io n Label
LB1
B e h a v io r D e s c r ip tio n
C o n s tr a in t s R e f
LT_PCO!OAM_LB START T_Test LT_PCO?OAM_LB LT_PCO?CELL GOTO LB1 ?TIMEOUT T_Test LT_PCO?OTHERWISE
OAM_SEG_F4_LB1(VPIvpc) OAM_SEF_F4_LB0(VPIvpc) CELL_UNASSIGNED
V e r d ic t
PASS
FAILED FAILED
E q u iv a le n t C f u n c t io n int Bhvr_Ver_OAM_F4_Coding() { static int state = S0_Ver_OAM_F4_Coding; if(state==S0_Ver_OAM_F4_Coding){ api_send_cell(OAM_SEG_F4_LB1(VPIvpc)); start_timer(T_Test); state = S1_Ver_OAM_F4_Coding; return(BHVR_RUN); } if(state==S1_Ver_OAM_F4_Coding){ if(api_get_cell(&cell)){ if(tdm_constraint(cell,OAM_SEG_F4_LB0(VPIvpc))){ verdict = PASS; return(BHVR_END); } if(tdm_constraint(cell,CELL_UNASSIGNED)){ return(BHVR_RUN); } verdict = FAIL; return(BHVR_END); } if(time_out()){ verdict = FAIL; return(BHVR_END); } return(BHVR_RUN); } }
Figure 8. Implementation of test case dynamic behavior
13
Once the ATM board has been configured, the TIM invokes the TBM to perform the test case dynamic behavior. This means successive calls to the C function that implements the TTCN specification of the test case, until the verdict is obtained. As an example, the function corresponding to the abstract test case Ver_OAM_F4_Coding is shown in figure 8. Trace messages have been omitted for clarity. The thirty seven test cases contained in the ATM-UNIES ATS have been modeled in a similar manner.
6. Summary and conclusions Nowadays, we are assisting to a spectacular grow in the number of communication networks based on the interconnection of multivendor equipment. In practice, however, real systems may not interoperate when connected together, despite all the claims made by the manufacturers about their products being in conformance to the standards. In most cases, the incompatibility problems lie precisely in the inaccuracy of such claims. The ISO Conformance Testing methodology was developed to provide a consistent framework for determining whether a given equipment complies with the standards, and to what extent. This is achieved by specifying the test cases that implementations must pass to be considered as conforming with a particular standard. The success in passing the test cases cannot be used to determine the degree of interoperability for a particular implementation. It can however give more confidence than another product which do not pass all or some of the test cases. An additional advantage of the ISO Conformance Testing is the possibility of automating the conformance assessment process. Two main areas of automation can be identified in this process. One has to do with the implementation of Executable Test Suites (ETSs) so that test cases can be automatically run without requiring the operator to interact with the test equipment. The other automation area, which can be considered equally important but which has received less attention, has to do with the management of test sessions. It covers the many aspects involved when considering the conformance assessment process for systems supporting several protocols or for different implementations of a same standard protocol. This paper has shown the benefits of generalizing the automation of the conformance testing methodology rather that individually considering its application to specific protocols. A generic conformance testing tool following this approach has been implemented. The tool provides a graphical user interface which allows the test operator to perform test sessions for different systems and standard protocols, reducing manual interventions to a minimum. During the test session, the user can request the execution of one or several test cases by simply selecting them with the mouse. The traces of the execution and the resulting verdicts can be visualized on real-time and saved for posterior retrieval. In particular, they can be used to automatically generate and print standard test reports. One of the key features of the tool is its flexibility for supporting new protocol test suites. As an example, the ATM Forum ATS for ATM End Systems has been developed and integrated within the tool. Although the implementation has
14
been done by hand, the use of TTCN compilers is being considered for future ATSs. Future research works can consider the improvement of the conformance testing platform and the implementation of new ATS, either manually or by using TTCN tools. Note that the platform is prepared to handle ATSs for any protocol, not only ATM. Examples of protocols that can be included in the future are Signaling System No. 7, GSM, ISDN, etc.
References [1]
ISO, “OSI - Conformance Testing Methodology and Framework”, ISO/IEC 9646, 1991. [2] ITU-T, “OSI conformance testing methodology and framework for protocol Recommendations for ITU-T applications”, Recommendations X.290 to X.296, 1995. [3] ETSI, “Conformance Testing Specifications: Standardisation Methodology”, ETS 300406, 1994. [4] K.G. Knightson, “OSI Protocol Conformance Testing - IS 9646 Explained”, McGraw-Hill Series on Computer Communications, 1993. [5] B. Baumgarten, A. Giessler, “OSI Conformance Testing Methodology and TTCN”, North Holland, 1994. [6] Telelogic AB, “ITEX - Interactive TTCN Editor and Executor”, Version 3.1, http://www.telelogic.se, 1997. [7] Expert Telecoms, “TTCN.Expert Compiler”, Expert Telecoms GbmH, http://www.expert.de, Stuttgart. [8] M. Alvarez-Campana, E. Vázquez. “Protocol Conformance Testing Tool for ATM Systems”, DIT-UPM, Mayo 1996. [9] “List of Approved ATM Forum Technical Specifications”, ATM Forum’s WWW Server, http://www.atmforum.com/atmforum/specs/ approved.html, December 1997. [10] “Conformance Abstract Test Suite for the UNI 3.0 ATM Layer of End Systems”, ATM Forum Technical Committee, Doc. af-test-0041.000, January 1996. [11] “ATM User-Network Interface Specification, Version 3.0”, ATM Forum, 1993.
15
