Distributed and parallel simulation in an interactive environment S. Pawlettaa, T. Pawlettab and W. Drewelowa a
Institute of Automatic Control, Department of Electrical Engineering, Univ. of Rostock, D-18051 Rostock, Germany* b
Chair of Applied Computer Science, Dept. of Mechanical Engineering, Univ. of Wismar, P.O. Box 1210, D-23952 Wismar, Germany
Multiprocessing systems are available for a growing number of users, but their exploitation for distributed and parallel simulations is still at the beginning because most today’s engineering software tools do not support distributed and parallel processing. In order to overcome this lack, a general approach to extend interactive environments by the missing features is presented. An interactive Master/Slave-paradigm is introduced, and examples of the main application areas are given.
1. INTRODUCTION Many simulation experiments are characterized by a considerable runtime expenditure on one side and a medium- or high-grained logical problem structure (e.g. Monte-Carlo-Studies, optimization of simulation models) on the other side. This peculiarity is more and more aggravated owing to increasing complexity of the examined models and the used analysis methods. Therefore, there is a need to take advantage of the problem specific concurrency by means of structural parallelization. The suitable hardware basis in the form of multiprocessing systems (computer networks, multicomputers and multiprocessors) is available for a growing number of users. Another difficulty of complex simulation tasks leads to the need of distributed processing. That means, some tasks cannot be solved in a single and isolated simulation system. The reasons are diverse, but they usually fit into two general categories. Tasks of the first category are simulation problems where data have to be processed or provided on-line (e.g. real-time and hardware-in-the-loop simulations). The second category results from the limited set of methods in a real simulation package. For instance, most packages do not support combined simulations (discrete/continuous, equation/rule based etc.). In such cases, the integration of different simulation tools in a distributed system can offer a solution. A third demand of the modern simulation methodology is the support of interactive working in all phases of a simulation study (modelling, simulation, result processing and visualization). In that area, engineering tools like MATLAB/SIMULINK, MATRIXX, CTRL-C etc. have established standards which exceed markedly the capabilities of classical programming environments. Unfortunately, these tools do not support distributed and parallel processing. *
email:
[email protected]
In this situation, it is the most elegant way out to extend existing interactive environments by the missing features. This way has two unquestionable advantages: • The user can apply distributed and parallel processing in an accustomed environment. • The immense expenditure which has been invested in the development of sophisticated engineering packages is well-preserved. Because most today’s approaches for distribution and parallelization base on classical programming languages (e.g. FORTRAN, C), the usual paradigms (master/slave, single program multiple data) disregard interactive concepts. Therefore, modified paradigms for an interactive usage have to be developed. The necessary primitives for communication and process control are provided by programming libraries like PVM [1], PSI [2] etc. Inside the engineering tool a high-level and easy to use communication interface has to be implemented which supports interactive development, validation and execution of distributed and parallel applications. The main application areas of the presented concept are parallelisation of runtime expensive, medium- and high-grained simulation tasks, distributed processing in heterogeneous environments and integration of different simulation tools.
2. AN INTERACTIVE ENVIRONMENT FOR DISTRIBUTED AND PARALLEL SIMULATIONS When the CSSL-standard was introduced in 1968, the typical simulation experiment was the single simulation run with a certain model. Newer approaches, for instance the concept of models, methods and experiments introduced by Breitenecker [3], take into account that the classical simulation in the time domain is only one basic method side by side with others. Extensive model based studies require complex methods for time and frequency domain analysis, optimization, statistics and visualization. Today, this variety of methods is available by technical computing environments like MATLAB [4]. Such packages are interactive systems whose basic data type is usually a matrix that does not require any declaring or dimensioning. The matrix oriented notation leads to a short and compact problem formulation. Due to the possible mixture of programming and interactive/interpretative execution, the necessary time effort for implementing and testing a solution is very small compared with traditional compilation based approaches. MATLAB high-level communication interface
other components
extension external interface communications library network operating system
real-time operating system
heterogeneous multiprocessing system
Figure 1. Extension of MATLAB to a distributed and parallel environment
Figure 1 shows the principle of the extension of such systems to distributed and parallel environments. Prerequisite is a programming interface of the system to external software. In case of MATLAB it is provided by the MEX-interface. It permits the dynamic linkage of subroutines (MEX-files) produced from C or FORTRAN source code. In this way, a high-level communication interface can be constructed inside of MATLAB by a set of MEX-files. The abstraction degree of the interface must fit to the system and is mapped onto the primitives of a communication library. 2.1. Parallel paradigm for interactive working A communication interface can MASTER SLAVES be realized as message passing or shared memory interface. The start message passing approach models interactions by explicit sending and spawn slaves start start receiving of messages. In the shared memory model an abstract send id's receive id's receive id's global tupel space is used [5]. In send data the following, only the message receive data receive data passing model is considered. The most important programprocess data process data ming paradigms for distribution and parallelization based on message send data send data receive data passing are Master/Slave (s. fig. 2) and SPMD (single program exit exit exit multiple data). The basic constructs of the Master/Slave-paradigm are spawn to create slaves, send Figure 2. Master/Slave-paradigm and receive to handle messages. MASTER
SLAVES
matlab startup matlab startup spawn input data eval(*.m)
put(M)
output data
aeval(*.m)
matlab startup
matlab startup
matlab exit
matlab exit
quit M=putback('M') matlab exit interactive
Figure 3. Interactive working with MATLAB (principle)
quit matlab exit
Figure 4. Interactive Master/Slave (principle)
Because both approaches are pure programming paradigms and do not consider interactive working, it is not useful to build up a message passing interface for an interactive environment with only those constructs. A modified Master/Slave-paradigm is given in figure 4. For a better understanding, the usual sequential work with the MATLAB interpreter is shown in figure 3. eval(*.m) symbolizes the call or the processing of a built-in-, M-, MEX-function or an Mscript. aeval(*.m) is the parallel equivalent of eval. Different to eval, aeval works asynchronously that means aeval does not return a result, but it returns immediately after the function is called in the slave instance. 2.2. High-level message passing interface for MATLAB A message passing interface for MATLAB, which is mapped alternatively onto the communication libraries PVM [1] and PSI/IPC [2,6], has been developed by the authors. The libraries PSI/IPC can realize communication by a number of transports (Message-Queues, Shared-Memory, TCP, UDP) of different operating systems (BSD, System V, OS9 etc.). All functions of the message passing interface support matrix and vector arguments, optional and default arguments and alternative argument signatures. Thereby, the functions are very powerful. The fundamental functions and arguments are briefly described in figure 5. Examples, which demonstrate the usage of the interface in an interactive way and for programming (according to Master/Slave- and SPMD-paradigm), are given in [7]. I = spawn(n)
creates n MATLAB-instances or other processes on the desired node(s) and returns identifier
put(I,M)
delivers one ore more matrices to one ore more receivers
M=putback(I,’M’)
requests one or more matrices from one or more processes
M=get()
receives any matrices or desired matrices or matrices from desired senders etc.
aeval(I,’*.m’)
asynchronous function call in MATLAB-instances or other processes
i=myid
returns the identifier of the calling MATLAB-instance
b=parent
returns TRUE for father instance and FALSE for son instance
Figure 5. MATLAB message passing interface (extract).
3. APPLICATIONS 3.1. Parallelization of runtime expensive simulation tools For the purposes of comparing parallel simulation techniques three simulation tasks are published in [8]. The problem granularity of the tasks ranges from a high-grained MonteCarlo-Study via a medium-grained calculation of a PDE to a fine-grained simulation of coupled predator-prey populations. All test problems were solved in the presented environment on a cluster of 20 SUN Classic workstations under Solaris 2 connected via Ethernet (10 Mbit/s). The performance results are published in [7] .
Another application example is the parameter optimization of an wheat-ontogenesis model (s. fig. 6). The details of that optimization problem with embedded simulations are described in [9]. The optimization has a vast consumption of calculation power because n ontogenesis simulations have to be performed in each iteration step where n is the number of climate data records, which ranges between 10 and 100. Owing to the high-grained problem structure, the speed-up factor of parallel solutions grows almost linear with the number of employed processors.
climate data
p
n simulation runs with n different climate data records YM Y
calculation of the residual and loss functions
f
observed data popt.
optimization method
Figure 6. Parameter optimization of an ontogenesis model 3.2. Distributed processing in heterogeneous environments In this field, simulation is often a sub-task in a more complex problem. Figure 7 shows a typical distributed application in the area of automatic control. The data sampling from a plant runs on a real-time system and the collected data are delivered to an interactive environment on a standard system. There, the user can perform plant identification, controller design and simulation of the closed loop. After that, the found parameters are down-loaded to an actual controller. The realization of this application is described in [10]. Modelling Controller Design Solaris 2 SUN Data Sampling
Controller
OS9 VME-BUS
OS9 VME-BUS
Figure 7. Distributed data sampling and process control 3.3. Integration of different simulation tools Own investigations in the field of agricultural systems [11] have shown that many simulation problems cannot be solved within single simulation systems. One example are models which are described by differential equations and logical rules. In order to solve such tasks, MATLAB and a rule-based simulator (PROLOG-application) were coupled via the described message passing interface [12].
Another example are structure variable systems [13] which are not supported by MATLAB. The connection to a special simulator, which is available as C++-application [14], allows the user to simulate that system class in the accustomed environment. On the other side, all methods necessary for data pre- and post-processing and for visualization are supported by MATLAB and need not be provided by the special simulator.
4. CONCLUSIONS The introduced extension of interactive engineering tools provides for a wide range of applications (not only simulations) an easy approach to distributed and parallel processing. The necessary time for implementing and testing a distributed and parallel version of a problem decreases significantly. Principles of a high-level shared memory interface for interactive environments will be published in the near future.
REFERENCES 1. 2. 3.
4. 5. 6. 7. 8. 9.
10. 11. 12. 13. 14.
Geist, A. et al.: PVM 3 User's Guide and Reference Manual. Technical Report ORNL/TM-12187, Oak Ridge National Laboratory, Oak Ridge, May 1994 Pawletta, S.: Design and Implementation of a Distributed and Parallel Simulation Environment for Complex Experiments. Pre-print CS-03-94, University of Rostock, 1994 Breitenecker, F. and Solar, D.: Models, methods, experiments - modern aspects of simulation languages. In: Proc. 2nd European Simulation Conf., Antwerp (SCS, San Diego, CA, 1986) 195-199 MATLAB - External Interface Guide. The Mathworks Inc., Natick, January 1993 Carriero, F. C. and Gelernter, D.: How to write parallel programs. The MIT Press, Cambridge, MA, 1991 Meyer, H.: A C++ Class Library for Process Interaction and Communication. Technical Report DB-92-2, University of Rostock, 1992 Pawletta, S., Pawletta, T. and Drewelow, W.: Comparison of Parallel Simulation Techniques - MATLAB/PSI. Simulation News Europe, (13), 1995, pp. 38-39 Comparison of Parallel Simulation Techniques. Simulation News Europe, (10), 1994 Pawletta, S., Pawletta, T. and Ewert, F.: Model validation of agricultural systems by parameter optimization. In Jaeschke, A. et al. (eds.): Informatik fuer den Umweltschutz, Springer, Berlin, 1993, pp. 133-141 Krueger, T.: Distributed Data Sampling and Process Control under OS9. Diploma Thesis, Institute of Automatic Control, University of Rostock, 1994 Pawletta, T.: Design and implementation of a modular hierarchical simulation runtime system for ecological models. Pre-print CS-02-94, University of Rostock, 1994 Nekien, T.: Coupling of rule based and equation based model descriptions. Diploma Thesis, Department of Computer Science, University of Rostock, 1993 Pawletta, T. and Pawletta, S.: CAST based simulation of structure variable systems. Rostocker Informatik Berichte, University of Rostock, (1) 1995 Pawletta, T. and Pawletta, S.: Design of a simulator for structure variable systems. Proc. of the 5th International IMACS-Symposium on System Analysis and Simulation, Berlin, Gordon & Breach Publishing House
