A MATLAB Toolbox for Distributed and Parallel Processing S. Pawlettaa, W. Drewelowa, P. Duenowa, T. Pawlettab and M. Suessea a
Institute of Automatic Control, Department of Electrical Engineering, Univ. of Rostock, D-18051 Rostock, Germany* b
Chair of Applied Computer Science, Dept. of Mechanical Engineering, FH Wismar, P.O. Box 1210, D-23952 Wismar, Germany
Multiprocessing systems, especially networked computers, are available for a growing number of users, but their exploitation for distributed and parallel processing is often a very painful task, because MATLAB and similar packages disregard the support of such concepts on the application layer. To fill this lack a MATLAB toolbox is presented which allows interactive development, validation and execution of distributed and parallel applications based on multiple MATLAB instances and other non-MATLAB software. The typical fields of application of the toolbox are illustrated by examples.
1. INTRODUCTION Up to now, there is no parallel version of MATLAB or of a similar package available for practical use. The main reasons are the fine granularity of the most MATLAB functions which can not be parallelised effectively on distributed memory computers, the small number of processors in shared memory computers, conflicts with MATLAB's sophisticated memory model and architecture and last but not least the limited number of customers with parallel machines [1]. But many scientific and engineering problems coded as MATLAB applications are characterized by a considerable run-time expenditure on one side and a medium- or highgrained logical problem structure (e.g. Monte Carlo studies, complex simulations, teaching neural networks) on the other side. This peculiarity is more and more aggravated owing to increasing complexity of the used algorithms and methods. Therefore, there is a need to take advantage of the problem specific concurrency by means of structural parallelisation. A suitable hardware basis in the form of networked computers is available for a growing number of users. Another class of problems leads to the need of distributed processing. That means, some tasks cannot be solved in a single and isolated MATLAB environment. The reasons are diverse, but they usually fit into two general categories. Tasks of the first category are problems where data have to be processed or provided on-line (e.g. measurement processing, hardware-in-the-loop simulations). The second category results from the always limited set of methods in a real software package. Here, the integration of autonomous software tools with MATLAB in a distributed system offers a solution. *
email:
[email protected]
This situation was the motivation for the authors to start the development of the DP (distributed and parallel application) toolbox in 1992. The DP toolbox provides an easy to use high-level communication interface. It enables the user to develop interactively distributed and parallel applications which are running on multiple MATLAB instances and which include autonomous non-MATLAB components. The interactions between the tasks of an application are mapped onto the primitives of a programming library for interprocess communication and control. The main application areas of the presented DP toolbox are parallelisation of runtime expensive, medium- and high-grained calculation problems, distributed processing in heterogeneous environments and integration of different software tools with MATLAB.
2. THE DP TOOLBOX Figure 1 shows the principle how the DP toolbox extends MATLAB to a distributed and parallel environment . Prerequisite is MATLAB’s programming interface (MEX) to external software [4]. It opens the way for a linkage of MATLAB with a communication library, which are usually implemented in C. The supported platforms range from standard to real-time operating systems depending on the used communication library. In the moment the DP toolbox can use the PVM [2] and the PSI [3] library. MATLAB high-level communication interface
other autonomous components
DP Toolbox external interface (MEX) communication library (PVM, PSI, ...) standard operating systems (Unix, MS-Windows, ...)
real-time operating systems (OS9, LynxOS, ...)
heterogeneous multiprocessing systems (networked computers, multicomputers)
Figure 1. Extension of MATLAB to a distributed and parallel environment Actually, the DP toolbox realizes a two layer interface (s. fig. 2). The first layer provides the complete function set of the underlying communication library in MATLAB. The abstraction degree is not increased at this level and the maximum flexebility of the communication library is kept up. For a convenient interactive usage a second layer provides a much more abstract interface consisting of only a few but very powerful functions.
high-level interface DP Toolbox low-level interface MATLAB MEX-Interface PVM or PSI library
Figure 2. Two layer interface of the DP toolbox 2.1. Parallel paradigm for interactive working Generally, a communication MASTER SLAVES interface can be realized as message passing or shared memory start interface. The message passing approach models interactions by spawn slaves start start explicit sending and receiving of messages. In the shared memory send id's receive id's receive id's model an abstract global tupel send data space is used [5]. In the following, receive data receive data only the message passing model is considered. process data process data The most important programming paradigms for distribution and send data send data receive data parallelization based on message passing are Master/Slave (s. fig. 3) exit exit exit and SPMD (single program multiple data). The basic constructs of the Master/Slave-paradigm are Figure 3. Master/Slave paradigm spawn to create slaves, send and receive to handle messages. MASTER
SLAVES
matlab startup matlab startup
spawn
input data eval(*.m)
put(M)
output data
aeval(*.m)
quit
matlab startup
matlab startup
matlab exit
matlab exit
M=putback('M') matlab exit
interactive
Figure 4. Interactive working with MATLAB (principle)
quit matlab exit
Figure 5. Interactive Master/Slave (principle)
Because both approaches are pure programming paradigms and do not consider interactive working, it is not useful to build up a message passing interface for an interactive environment like MATLAB with only those constructs. A modified Master/Slave paradigm is given in figure 5. For a better understanding, the usual sequential work with the MATLAB interpreter is shown in figure 4. eval(*.m) symbolizes the call or the processing of a built-in-, M-, MEXfunction or an M-script. aeval(*.m) is the parallel equivalent of eval. Different to eval, aeval works asynchronously that means aeval does not return a result, but it returns immediately after the function is called in the slave instance. 2.2. The high-level communication interface The DP toolbox provides a high-level message passing interface which supports application development according to the introduced interactive Master/Slave principle and to the conventional paradigms Master/Slave and SPMD. All functions of the message passing interface support matrix and vector arguments, optional and default arguments and alternative argument signatures. Thereby, the functions are very flexible and powerful. The fundamental functions and arguments are briefly described in figure 6. I = spawn(n)
creates n MATLAB-instances or other processes on the desired node(s) and returns identifier
put(I,M)
delivers one ore more matrices to one ore more receivers
M=putback(I,’M’)
requests one or more matrices from one or more processes
M=get()
receives any matrices or desired matrices or matrices from desired senders etc.
aeval(I,’*.m’)
asynchronous function call in MATLAB-instances or other processes
i=myid
returns the identifier of the calling MATLAB-instance
b=parent
returns TRUE for father instance and FALSE for son instance
Figure 6. Message passing interface provided by the DP toolbox (extract). To give an impression of the usage of the DP toolbox a simple example should be discussed in greater detail. Consider, the simulation of the step response of a damped mass-spring system is coded as the MATLAB function [xmean]=sim_mss(dd). The function argument dd is the damping factor which should be used as model parameter. If dd is a vector, a simulation is done with every component of dd and the average step response xmean is calculated. In a traditional way a Monte Carlo study with e.g. 1000 randomly chosen damping factors is carried out in MATLAB by the following commands: dd=rand(1000,1); xmean=sim_mss(dd); plot(xmean);
That can be done interactively, but it is absurdly because the study consumes too much time. In a parallel fashion multiple MATLAB instances are used (e.g. 20): nslaves=20; slaves=spawn(nslaves);
The damping factors are placed in a matrix and each column is put to another MATLAB instance: dd=rand(1000/nslaves,nslaves); for i=1:nslaves put(slaves(i),dd(:,i)) end
By calling aeval each MATLAB instance is requested to process its part of the whole task. After this the results can be collected and the entire step response is calculated: aeval(slaves,'xmean=sim_mss(dd);') xmean=0; for i=1:nslaves xmean=xmean+putback(slaves(i),'xmean')/nslaves; end plot(xmean);
If necessary, the parallel Monte Carlo study can be coded as MATLAB program in sense of the Master/Slave programming paradigm. sim_master.m: nslaves=20; slaves=spawn(nslaves,'sim_slave'); dd=rand(1000/nslaves,nslaves); for i=1:nslaves put(slaves(i),dd(:,i)) end xmean=0; for i=1:nslaves xmean=xmean+get(slaves(i))/nslaves; end plot(xmean) sim_slave.m: dd=get; put(-1,sim_mss(dd))
Sometimes the SPMD (single program multiple data) programming paradigm is preferred. A SPMD example program coded with the DP toolbox is published in [7].
3. APPLICATIONS 3.1. Parallelization of runtime expensive calculations For the purpose of comparing parallel simulation techniques three simulation tasks are published in [6]. The problem granularity of the tasks ranges from a high-grained Monte Carlo study via a medium-grained calculation of a PDE to a fine-grained simulation of coupled predator-prey populations. All test problems were solved in MATLAB with the DP toolbox on a cluster of 20 SUN Classic workstations under Solaris 2 connected via Ethernet (10 Mbit/s). The performance results are published in [7]. Another application example is the parallelization of a stability test for linear uncertain systems by the method of convex decomposition [8], which is also applicable for controller design. The
method bases on the repeated solution of Lyapunov-Equations and provides a graphical presentation of stability domains in the space spawned by the uncertain parameters (s. fig. 7). In the sense of parallelization the method represents a difficult load balancing problem, because the repeated division of the original cube in subdomains leads to a decomposition tree with branches of different complexity (s. fig 8). Therefore, static load balancing will produce pure results. In [9] a dynamic load balancing algorithm is described which has been implemented using the DP toolbox. 2
1.5
Figure 7. Well-known stability triangle of a discrete second order system
1
0.5
0
-0.5
-1
-1.5
-2 -1.5
-1
-0.5
0
0.5
1
1.5
(-,-)
(0,0)
(00,01)
(000,011)
(000,111)
(00,11)
(0,1)
(10,01)
(100,011)
(10,11)
Figure 8. Decomposition tree
(1,0)
(1,1)
(00,01)
(00,11)
(10,01)
(10,11)
(100,111) (1,0)
(0,0)
(11,01) (01,01) (10,01) (00,01)
(11,11) (01,11) (10,11)
(100,011) (000,011) (100,111) (000,111)
3.2. Distributed processing in heterogeneous environments Figure 7 shows a typical distributed application in the area of automatic control. The data sampling of a plant runs on a real-time system and the collected data are delivered to
MATLAB on a standard system. There, the user can perform plant identification, controller design and simulation of the closed loop. After that, the found parameters are down-loaded to an actual controller. The realization of this application is described in [10]. Modelling Controller Design MATLAB Unix Data Sampling
Controller
OS9 VME-BUS
OS9 VME-BUS
Figure 9. Distributed data sampling and process control 3.3. Integration of autonomous software with MATLAB Investigations in the field of agricultural systems [11] have shown that some simulation problems cannot be solved within MATLAB effectively. One example are models which are described by differential equations and logical rules. In order to solve such tasks, MATLAB and a rule-based simulator (PROLOG-application) were coupled via the DP toolbox [12]. Another example are structure variable systems [13] which are not supported by MATLAB. The connection to a special simulator, which is available as C++-application [14], allows the user to simulate that system class in the accustomed MATLAB environment. On the other side, all methods necessary for data pre- and post-processing and for visualization are supported by MATLAB and need not be provided by the special simulator.
4. CONCLUSIONS The presented DP toolbox provides an easy approach to distributed and parallel processing for a wide range of applications. The necessary time for implementing and testing a distributed and parallel version of a problem decreases significantly compared with other techniques [15] because the usual fast prototyping of MATLAB is kept up. Principles of a high-level shared memory interface for MATLAB and experiences with distributed and parallel SIMULINK applications will be published in the near future.
REFERENCES 1. Moler, C.: Why there isn't a parallel MATLAB. MATLAB News and Notes, spring, 1995 2. Geist, A. et al.: PVM 3 User's Guide and Reference Manual. Technical Report ORNL/TM-12187, Oak Ridge National Laboratory, Oak Ridge, May 1994 3. Pawletta, S.: Design and Implementation of a Distributed and Parallel Simulation Environment for Complex Experiments. Pre-print CS-03-94, University of Rostock, 1994 4. MATLAB - External Interface Guide. The Mathworks Inc., Natick, January 1993 5. Carriero, F. C. and Gelernter, D.: How to write parallel programs. The MIT Press, Cambridge, MA, 1991 6. Comparison of Parallel Simulation Techniques. Simulation News Europe, (10), 1994
7. 8.
9. 10. 11. 12. 13. 14.
15.
Pawletta, S., Pawletta, T. and Drewelow, W.: Comparison of Parallel Simulation Techniques - MATLAB/PSI. Simulation News Europe, (13), 1995, pp. 38-39 Kiendl, H.; Ossadnik, H.: Robustness Analysis and Synthesis of Linear Uncertain Systems by the Method of Convex Decomposition. Workshop on Control of Uncertain Systems, Bremen, Germany, 1989 Suesse, M.: Parallel Processing in the Interactive Environment MATLAB. Diploma Thesis, Institute of Automatic Control, University of Rostock, 1995 Krueger, T.: Distributed Data Sampling and Process Control under OS9. Diploma Thesis, Institute of Automatic Control, University of Rostock, 1994 Pawletta, T.: Design and implementation of a modular hierarchical simulation runtime system for ecological models. Pre-print CS-02-94, University of Rostock, 1994 Nekien, T.: Coupling of rule based and equation based model descriptions. Diploma Thesis, Department of Computer Science, University of Rostock, 1993 Pawletta, T. and Pawletta, S.: CAST based simulation of structure variable systems. Rostocker Informatik Berichte, University of Rostock, (1) 1995 Pawletta, T. and Pawletta, S.: Design of a simulator for structure variable systems. Proc. of the 5th International IMACS-Symposium on System Analysis and Simulation, Berlin, Gordon & Breach Publishing House Breitenecker, F.: Comparison of Parallel Simulation Techniques. Proc. of the European Simulation Congress, EUROSIM 95, Vienna, Elsevier Science B.V., 1995
