IEEE TRANSACTIONS ON SOFTWARE ENGINEERING,
VOL. 26,
NO. 3,
MARCH 2000
193
Guest Editors' Introduction: Special Issues on Architecture-Independent Languages and Software Tools for Parallel Processing Domenico Talia, Member, IEEE Computer Society, Pradip K. Srimani, Fellow, IEEE, and Mehdi Jazayeri, Senior Member, IEEE
æ
T
HE demand for information processingÐboth in terms of volume of information and the variety of processing Ðis insatiable and ever increasing. Many previously infeasible systems are considered conventional today. Basic physical limitations on electronic circuits continually force high performance computations to look toward the exploitation of parallelism. Just as the fastest cycle times are approaching the fundamental barriers imposed by physical device limitations, different kinds of parallel and distributed systems are emerging. These trends are sure to affect general purpose computing. Parallel processing systems more and more demonstrate their effectiveness as tools that deliver high-performance in solving complex problems. Unfortunately, in using parallel computers, designers of parallel programs are often forced to consider low-level issues such as synchronization, process mapping, and architecture details. These issues arise mainly from the different architectural features that characterize parallel computers. The class of parallel systems includes many different architecturesÐfrom single-instruction-multiple-data (SIMD) machines to distributed-memory, multiple-instruction-multiple-data (MIMD) computers and workstation clusters. This architectural heterogeneity leads to the development of parallel software that lacks portability. A serious impact of this lack of portability is that software developers cannot rely on the usability of their software on new parallel architectures as they emerge. The concept of parallelism, either at the architecture or the algorithm level, is a significant departure from the traditional sequential approach. The sequential approach attempts to reduce processing time by carrying out each operation faster and by eliminating redundancies in computation. In parallel processing, we save time by
. D. Talis is with ISI-CNR, DEIS, UNICAL, 87036 Rende, Italy. E-mail:
[email protected]. . P.K. Srimani is with the Department of Computer Science, Colorado State University, Ft. Collins, CO 80523. E-mail:
[email protected]. . M. Jazayeri is with the Distributed Systems Department (184-1), Technical University of Vienna, A-1040 Vienna, Austria. E-mail:
[email protected]. For information on obtaining reprints of this article, please send e-mail to:
[email protected], and reference IEEECS Log Number 110996.
identifying concurrently executable subtasks in a given task and executing them on multiple processors simultaneously; often, we introduce redundancies to speed up the overall parallel execution time (by reducing communication overhead between processors). The first step in the development of a parallel program is the identification of sections that can be executed in parallel and how these should be parallelized. But, unlike the von Neumann model for sequential computing, there is no general purpose model for parallel computation. There is not a single computation model that can guide the choices of parallel programmers in designing, implementing, and evaluating the performance of parallel software. Therefore, a program decomposition that is efficient on a parallel machine might become inefficient on a different parallel machine. Languages and tools for parallel processing are designed to separate software development concerns from effective parallel execution concerns and provide both abstraction and standardization. Abstraction arises because the operations that the tools provide are higher-level than those of the underlying architectures, simplifying the structure of software and reducing the difficulty of its construction. Standardization arises because software construction can assume a standard interface over a long time period, regardless of developments of newer parallel computer architectures. At the same time, a parallel programming language defines an environment for the implementation effort (compiler, linker, and run-time system) directed to each parallel computer. High-level languages and tools are essential to free the application developers from the architecture specific implementation details and thus make their job more productive and efficient. Design and adoption of architecture-independent programming models, languages, and software engineering tools (integrated tool environments including testing, debugging, validating, cost, and process models) are necessary for cost effective use of general purpose parallel computing. This approach will also allow development of portable and standardized parallel software. The challenge for high level languages and tools is to provide their benefits without requiring substantial effort in the construction of a supporting environment and without imposing large execution overhead for a large number of parallel architectures.
0098-5589/00/$10.00 ß 2000 IEEE
194
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING,
Currently, to use parallel machines software, developers are required to first develop algorithms that are different than the ones they use on sequential machines and, then, to express their new algorithms using new parallel languages or parallel extensions to existing languages. A consequence of the slow emergence of new languages, lack of distributed software engineering environments and programming techniques, and lack of any standards is that parallel software designers are faced with ineffective programming models and tools and are often compelled to program at a low level with specific target machines in mind. This results in software that is neither portable nor reusable. In the world of sequential programs, there has been steady progress in the development of languages and techniques to write portable and reusable code. Through the use of careful programming practice, even programs in ªmidlevelº languages such as C can be written so that they can be ported between machines with little or no loss of performance. In the parallel world, portability often comes at a severe cost in terms of performance. The architectural and paradigm differences between parallel computers lead to a large set of different parallel computing models, languages, and tools (that often are architecture-dependent) that is bewildering to the programmer. At best, programs are portable within a certain type of machines, i.e., a program written with a standard message passing library might be portable from one MIMD machine to another, but, in general, the performance of a parallel application may change drastically when moved from one parallel machine to another. One approach to portable parallel programs is the use of standard procedure libraries. These include libraries for message passing and shared-memory operations, such as MPI, PVM, and OpenMP. These libraries are able to solve the portability problem because they run on many platforms, from a single PC to massively parallel computers from different vendors. In particular, in some cases (e.g., PVM), processes that compose a parallel application can run on different machines as parts of the same program. That is, a process can run on a workstation and another process can run on a supercomputer at the same time. For these and other reasons, toolkits such as MPI, PVM, and OpenMP are widely used. While these libraries support portability, they offer a low-level programming model and do not offer any methodology for software development, testing, and evaluation. In fact, using these libraries, programmers must program all of the process decomposition, placement, and communication explicitly. Another approach is to use a new programming language designed specifically with parallel processing in mind. In recent years, there have been several efforts to provide more practical programming languages that have led to the development of new languages for parallel processing. Examples of these languages are HPF, parallel C++, emerging skeleton-based languages, compositional languages, Java, and declarative languages like SISAL. These languages are important contributions toward high-level software tools for architecture-independent parallel programming. There is still a need, however, for general
VOL. 26,
NO. 3,
MARCH 2000
purpose programming models, cost models, design tools, as well as management tools. In order to be useful, parallel programming tools must address both issues of abstraction and effectiveness. In particular, parallel software should incorporate a software development methodology and should be architectureindependent. The first requirement implies the need for methodologies for testing and debugging of parallel programs and methodologies for achieving program correctness. Such methodological approaches have been advocated for sequential programming as well and there has been much progress. These approaches seem essential for parallel programming, even if they must remain goals rather than practice in the medium term. A parallel software methodology must address the following main problems: .
identification of parallelism: analysis and discovery of problem-domain and solution-domain parallelism; . parallel process structuring: how to decompose a problem into a set of parallel actions; . communication and synchronization: how the parallel actions cooperate to solve a problem; . global computation design and evaluation: how to view the global properties of a parallel program (as a whole) to improve its structure and evaluate its computational costs; . process-to-processor mapping: how to assign the processes that compose a program to the processors that compose a parallel computer; . validation and testing of parallel software: how to assess the behaviors and the efficiency of a parallel program. Parallel programming languages and environments should support the user in all the phases of the parallel program development process. They should provide constructs, mechanisms, and techniques that support a methodology for parallel software design which is able to address the problems listed above. The second requirement, architecture independence, deals with system heterogeneity. Architecture independence assures that programs can migrate between parallel machines without having to be redeveloped or, indeed, modified in any nontrivial way. This requirement implies that languages and tools must abstract from the features of any particular style of parallel computer. If parallel computation is to be useful, it must be possible to insulate software from changes in the underlying architecture, even when these changes are substantial. This characteristic may help portability across current parallel computers, but also toward future parallel architectures that will be designed in the coming years. We proposed two special issues of the IEEE Transactions on Software Engineering to address architecture-independent programming models and software tools (analysis, synthesis, and comprehension) for parallel processing, to disseminate experience reports on platform-independent parallel software development (which is already empirically validated), and to present the current state of highlevel parallel languages and software tools. There are many journals, as well as many conferences, that regularly devote
TALIA ET AL.: GUEST EDITORS' INTRODUCTION: SPECIAL ISSUES ON ARCHITECTURE-INDEPENDENT LANGUAGES AND SOFTWARE...
space to discuss issues in parallel processing models and tools; mostly, these are available to and readable by only specialists in parallel processing. The focus very often is not on the software engineering aspects of the parallel processing; very few structured techniques, principles, and tools are available for parallel programming. Many practitioners, as well as researchers, do not have easy access to the research on requirement analysis techniques, design, implementation, and testing issues in distributed software development because major papers have appeared in diverse places and experience reports are not widely available. Results that have evaluated different techniques or have brought out their limitations are not well-known. We hope that these special issues will stimulate interchange between researchers and practitioners in parallel processing and software engineering. These special issues are expected to improve the knowledge of the user community in parallel software development and provide innovative ideas on practical high-level programming of high performance parallel computers. They include papers that: .
provide new theoretical and/or experimental insight in developing real-life applications using either new high level languages or architecture-independent distributed software development environments; . discuss new distributed/parallel software engineering tools, distributed software management tools, and cost models. We received 58 submissions on topics ranging from abstract models to practical tools and languages, from transformation methods to experimental evaluation of tools. We selected 12 papers that cover a wide range of the areas of interest. In this first issue, we included six papers. The first three papers present architecture-independent languages and methodologies for high-level programming of parallel software. The paper ªZPL: A Machine Independent Programming Language for Parallel Computersº by B.L. Chamberlain, S-E. Choi, E.C. Lewis, C. Lin, L. Snyder, and W.D. Weathersby presents the ZPL programming language. It provides a complete description of the language and evaluates it in detail with respect to its design criteria. It also discusses the relationship of ZPL to a number of other parallel programming languages and provides results from studies on the performance obtained on diverse parallel machines using this approach. The paper by S.B. Baden and S.J. Fink, ªA Programming Methodology for Dual-Tier Multicomputers,º presents a programming paradigm and software implementation that supports the development of block-structured scientific programs on hierarchical parallel computers. Emphasis is on practical modern architectures consisting of networked clusters of symmetric multiprocessors. The authors describe, in many details, the problems encountered and the possible and implemented solutions into the KeLP2 model. The actions being taken are also justified by a number of test results presented in the paper. T. Braunl's paper, ªParallaxis-III Architecture Independent Data Parallel Processing,º presents an architectureindependent language for data parallel programming that has evolved over the past 10 years. After describing the
195
language features, the paper presents the compilers for data parallel systems, as well as for a sequential simulation system and a data parallel graphics debugger that allows efficient source level analysis for parallel programs. The next two papers deal with automatic extraction of parallelism from programs written using sequential languages. The paper ªEfficient Interprocedural Array DataFlow Analysis for Automatic Program Parallelizationº by J. Gu and Z. Li, presents techniques for efficient interprocedural array data-flow analysis and documented experimental results of its implementation in a research parallelizing compiler. The paper shows that the proposed analysis, which is based on the concept of guarded array regions, is both more precise and more efficient than those used in other systems. The paper ªAutomatic Mapping of System of N-Dimensional Affine Recurrence Equations (SARE) onto Distributed Memory Parallel Systems,º by A. Marongiu and P. Palazzari, describes a procedure for automatic parallel code generation in the case of algorithms for a set of affine recurrence equations. The authors introduce a new projection technique in order to map the original iteration space into the processor-time space and also demonstrate how it is being done on an SIMD parallel system. The final paper in this issue deals with formal design methodologies for efficient parallel software development which can contribute to improve performance of programs on different parallel architectures. The paper by S. Gorlatch, ªToward Formally Based Design of Message Passing Programs,º presents a promising formal methodology for modeling and optimizing message passing programs composed of collective communication operations, within the BMF framework. The author estimates, analytically, the impact of the design rules on the target performance and discusses through several MPI examples, the utility of the proposed transformations. The other six papers will appear in the April 2000 issue of TSE and as a special section in the May 2000 issue. Two papers continue to deal with formal design methodologies for efficient parallel software development. T. Rauber and G. Reunger's paper, ªA Transformation Approach to Derive Efficient Parallel Implementations,º presents TwoL (Two Level parallelism), an approach to parallel software development in which the user expresses the parallel structure of an application in an initial, high-level, specification as a hierarchical combination of two levels of parallelism: task parallelism and data parallelism. Then, systematic transformations of the initial specification are carried on in a semiautomatic way in order to generate a real parallel program for a given target machine. The paper ªClustering Algorithm for Parallelizing Software Systems in Multiprocessor Environmentº by D. Kadamuddi and J.J.P. Tsai, describes a clustering technique and associated algorithms for identification of a module task group which may be placed on independent processors. The proposed technique also facilitates the parallelization of applications by avoiding performance degradation due to synchronous communications. Simulation results on some standard cases are discussed.
196
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING,
Dealing with architecture-independent languages and methodologies is the paper ªA Design Methodology for Data-Parallel Applicationsº by L.S. Nyland, J.F. Prins, A. Goldberg, and P.H. Mills, which describes a methodology of software development based on prototyping for designing and developing data parallel programs and discusses tool support for the software development process. Primary issues addressed by the proposed approach are algorithm choice, correctness, and efficiency, followed by data decomposition, load balancing, and message-passing coordination, depending on the architecture model. The last three papers discuss data structures and tools for simulation of parallel programs and an approach to optimization of parallel I/O systems. The paper ªAsynchronous Parallel Simulation of Parallel Programsº by S. Prakash, E. Deelman, and R. Bagrodia, presents a technique that uses application-level analysis to reduce the amount of synchronization required in parallel simulation of parallel programs. This technique has been embedded in the MPISIM simulator, a library for the execution driven simulation of task and data parallel programs. Validation and performance results for a set of applications that include the NAS Parallel Benchmark suite are discussed. The paper ªPriority Queues and Sorting Methods for Parallel Simulationº by M. Grammatikakis and S. Liesche, describes the design, implementation, and experimental analysis of parallel priority queues for device and network simulation. The authors consider distributed splay trees using MPI, concurrent heaps using shared memory atomic locks, and a new, more general concurrent data structure based on distributed sorted lists which is designed to provide dynamically balanced work allocation and efficient use of shared memory resources. Performance for all three data structures has been evaluated on a Cray-T3E. In the paper ªAutomated Tuning of Parallel I/O Systems: An Approach to Portable I/O Performance for Scientific Applications,º Y. Chen and M. Winslett describe a method for automatic tuning of parameters of parallel I/O systems to run scientific applications on a multiprocessor system. Selection of such parameters manually requires both time and familiarity with the target machine. Therefore, any support for automatic selection of parameters will be worthwhile. The authors describe some of the algorithms used to select parameters and have given performance results on an IBM SP, using the Panada I/O library, to compare default settings to the parameters computed by their program. We would like to express our gratitude to the authors of all submitted papers and the reviewers for their contributions to these special issues. We would also like to thank Dr. Richard A. Kemmerer, the TSE editor-in-chief at the time of the development of these issues, for his support of the special issues. Domenico Talia Pradip K. Srimani Mehdi Jazayeri
VOL. 26,
NO. 3,
MARCH 2000
Domenico Talia received the Laurea degree in physics from the University of Calabria. He is a senior researcher at the ISI-CNR-Institute of Systems Analysis and Information Technology of the Italian National Research Council and a lecturer on the Faculty of Engineering of the University of Calabria. He has been a researcher at CRAI (Consortium for Research and Applications of Informatics). His main research interests are parallel computation, parallel programming languages, cellular automata, and parallel data mining. Talia is a member of the editorial board of the IEEE Computer Society Press, a member of the editorial board of the journals Parallel and Distributed Practices and Future Generation Computer Systems, and a member of the Advisory Board of Euro-Par. Talia is a speaker in the IEEE Computer Society Distinguished Visitors and Chapter Tutorial Speaker Programs. He has published three books and more than 80 papers in international journals and conference proceedings. He is member of the ACM and the IEEE Computer Society. Pradip K. Srimani received his PhD degree in computer science from the University of Calcutta, Calcutta, India, in 1978. He has served on the faculty of the Indian Statistical Institute, Calcutta, Gesselschaft fuÈr Mathematik und Datenverarbeitung, Bonn, West Germany, Indian Institute of Management, Calcutta, India, and Southern Illinois University, Carbondale, Illinois. He has been a professor of computer science at Colorado State University in Ft. Collins, Colorado, since 1990. His research interests include reliable systems, parallel algorithms, fault-tolerant computing, networks, and graph theory applications. He is a fellow of the IEEE and a member of the ACM. He is currently the editor-in-chief of the IEEE Computer Society Press and is a member of the editorial boards of IEEE Software and IEEE Transactions on Knowledge and Data Engineering. He has been serving as a Distinguished Visiting Speaker and Chapter Tutorial Speaker for the IEEE Computer Society for the past several years. He has guest edited special issues for Computer, IEEE Software, the Journal of Computer & Software Engineering, the Journal of Systems Software, VLSI Design, the International Journal of Systems Science, etc.; he has also served many conferences in various capacities. Mehdi Jazayeri holds degrees from the Massachusetts Institute of Technology and Case Western Reserve University. He is a professor of computer science and heads the Distributed Systems Group at the Technical University of Vienna. He is interested in component-based software engineering of distributed systems. He has worked at Hewlett-Packard Laboratories, Synapse Computer Corporation, Ridge Computers, and TRW Vidar, all located in the Silicon Valley. He spent two years in Pisa, Italy, to set up and manage a joint research project on parallel systems between Hewlett-Packard and the University of Pisa. He has been an assistant professor of computer science at the University of North Carolina at Chapel Hill, adjunct professor at Georgia Institute of Technology, University of Santa Clara, and San Jose State University. He was a Fulbright Scholar at the University of Helsinki (1979) and a visiting professor at the Politecnico di Milano (1988). He was a principal investigator on an Esprit project on Software Architectures for Embedded Systems and is currently involved in several European Union projects dealing with component technologies for e-commerce and other Internet-based software systems. He has coauthored two textbooks, Fundamentals of Software Engineering (Prentice Hall, 1991), and Programming Language Concepts (John Wiley, 1998). He has been a consultant to the US government and to multinational companies in the areas of software engineering and architecture. He is a senior member of the IEEE, and a member of the ACM, the Austrian Computer Society, and the German Computer Society.
