Deploying Legacy Applications on Grids Jiˇr´ı Denemark Faculty of Informatics Masaryk University Brno Czech Republic
[email protected]
Archit Kulshrestha Center for Computation & Technology Louisiana State University U.S.A.
[email protected] Gabrielle Allen Center for Computation & Technology Louisiana State University U.S.A.
[email protected]
Abstract As Grid Computing technologies mature, computational grids are becoming increasingly common, making a large number of varied computational resources spread around the world potentially available to scientists and engineers. Whereas much effort has been spent on the development of core Grid services, less attention has been paid to how Grids will be used by general applications. Most existing or legacy scientific applications are not designed to support the use of Grid technologies inherently. In this paper, we describe how legacy applications can be deployed on a Grid without making complex changes to the original source code, and provide as a case study a migration scenario for a legacy parallel astrophysics application. This scenario was demonstrated at the SuperComputing 2004 Conference held at Pittsburgh.
1. Introduction Grids promise researchers a powerful environment in which they can collaboratively utilize distributed computational resources and distributed data [1, 2]. Grid computing will provide seamless, transparent, secure access to information technology resources such as hardware, software, scientific instruments, and services. Like electrical power, this access will be dependable, consistent, and pervasive. Grid computing’s focus on the ability to support computation across administrative domains sets it apart from traditional distributed computing. A primary concept of the Grid is coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations [1]. This sharing can be file exchange, access to
computers, software, data, and other resources, as is required by a range of current day collaborative problemsolving strategies. One of the major strengths and major challenges of the Grid is its diverse nature in terms of the involved hardware and software technologies. There is a wide and growing range of Grid services under development. Globus Toolkit [3], currently the de facto standard for Grid Computing, provides an environment that can be used to securely access remote system resources such as data and computational cycles. This is provided by services such as the Replica Location Service (RLS) which deals with data files, Grid Resource Allocation and Management (GRAM) which deals with jobs and resources, the Monitoring and Discovery Service (MDS) which deals with information, and the Grid Security Infrastructure (GSI) which provides security. Grid APIs have been written to ease the task of the application programmers to develop powerful Grid applications that are short and effective. These provide a common programming interface to the numerous Grid technologies that exist such as Globus, Condor etc. The complexity of these systems, both in terms of administration and application usability, remains high. This aspect, the usability for application programmers, is the focus of the Grid Application Toolkit (GAT) [4, 5], developed in the EU GridLab project. The GAT incorporates an API designed to help Grid programmers access Grid capabilities directly from their source code. New applications can code against this generic and middleware independent API, providing a new generation of Grid-aware applications able to dynamically and adaptively exploit their environment, and make their own intelligent decisions based on information about their environment. However a lot of so-called “legacy” applications developed prior to the emergence of Grid technolo-
gies do not lend themselves to this style of Grid computing. These applications could however greatly benefit from the capabilities that Grid technologies provide. In this paper show how APIs such as provided by the GAT can not only help application programmers write Grid aware applications but to also make “legacy” applications Grid aware and allow them to benefit from the new capabilities of the Grid. There are many such “legacy” applications that can benefit from Grids. One class includes computation intensive applications written by researchers using the Fortran programming language, and using MPI to distribute work among many processors. Much scientific computation involves running a large number of iterations to model a time dependent problem. These applications often write out their results as files and can resume computation by reading in the previous output. Additional support information can be read in from support files that we call checkpoint files. These generally have stored in them metadata that helps resume the legacy application. These could be filenames, simulation parameters or time-step information etc. In many cases it is difficult or impossible to modify source code due to reasons such as closed source programs, or complex code that is impractical to rewrite using a new approach. Also researchers would like to be able to run their old code out of the box on the Grid. In this work we show the migration of a “legacy” Fortran MPI code for neutron star simulation across the GridLab Testbed. This performed with the help of a wrapper code which uses the GAT API. This wrapper code enables the legacy application, which fulfils some simple requirements, to listen to checkpoint requests and prepare the application to migrate, or move to a more appropriate resource, given some reason to do so. In Section 2 we define job migration and checkpointing. Section 3.1 describes the legacy application used for this work, while Section 3.2 describes general requirements for applications. Section 3.3 describes the architecture and implementation of migration. Section 3.4 discusses results and related work. Finally in Section 4 we provide concluding remarks and discuss future work.
2. Job Migration and Checkpointing Job migration refers to relocating a running application to a different resource or to a different batch queue on the same machine. There are several motivations for migrating jobs. Present day supercomputers have time limited queues for running jobs and when the time limit expires the job is killed. If further execution is required the job must be restarted. Hence it is extremely important to save the state of the job either periodically or just before the job is terminated, and then resume from this saved state.
In a Grid context there are further motivations that make migration even more desirable. As opposed to isolated computing environments, when an application is deployed in Grid environment it can be moved amongst several computational resources during its execution. Various reasons can be found for such a movement, e. g., a more suitable (faster, with more storage capacity, broader network bandwidth, . . . ) resource was found or on the other hand some other application could make use of system resources better than the one currently assigned that system or a machine which the application is running on is going to be switched off and so on. Migrating many small jobs from a supercomputer to smaller machines could free the resource so that a large job could be deployed there. The dynamic nature of the Grid makes it essential for applications to be able to migrate successfully to better resources or to just survive and continue to run on a lesser system. Also such mechanisms would give resource management systems the power to apply complex scheduling algorithms and move around applications hence producing a highly efficient and optimized computing and storage environment. Before a running application can be stopped and moved to some other machine the current status of its execution needs to be saved so that the computation can be resumed from the same point where it was stopped. This process is called checkpointing. There are two ways of generating checkpoint files: application level and system level. System level checkpointing is completely handled by operating system in which the job is running and it needs no special support from application. However, both the operating system on the source machine and the one on the target machine must be similar and special tools are needed to help in checkpointing applications. Currently very few operating systems supports this kind of checkpointing. Application level checkpointing requires the application to be able to write its state to a file and when it is restarted it has to read that file and resume its execution. This approach is more portable and can also be more efficient because the application in contrast to the OS knows exactly what information is required to resume the computation. Current day application level checkpointing generally writes out parameters and other values that can be used to recreate data structures rather than saving all the information in memory and trying to load it again. This approach is more flexible as it can deal with changing platforms, system types and architectures easily, however requires more work for application developers. The Global Grid Forum in their effort to provide a standard API for application level checkpointing and recovery have come up with a proposed architecture [6] for grid checkpoint recovery that represents a basic, common foundation on which more sophisticated and applicationspecific checkpoint recovery solutions can be built.
3. Deploying Legacy Applications 3.1. Legacy Application In this paper we demonstrate the Grid enabling of legacy applications by taking as a case study a physics code that models the non-linear development of a secular barmode instability in rotating neutron stars. This instability is driven by gravitational radiation-reaction forces. From studying the results of simulations of this code, astrophysicists can understand more about the physics of neutron stars and the likely production of gravitational waves. The legacy application code was written by Shangli Ou and other members of Prof Tohline’s Astrophysics group at LSU as part of his doctoral research work. The neutron star simulation code is written in Fortran and is typically run in a domain decomposition parallel mode, distributing three-dimensional coordinate space across multiple processors. MPI is used for message passing between processors, and the code was already capable of providing regular checkpoint files from which a job can be later recovered. Figure 1 contains a visualization of the output from a sample run which clearly shows the degeneration in shape of the spinning neutron star, and the appearance of a bar mode. We describe in this paper how legacy applications can be made Grid aware by demonstrating that this neutron star code can be migrated from one system to another and the simulation resumed on the new machine from right where it left off.
3.2. Legacy Application Requirements Grid enabling legacy applications essentially means two things that we must be able to achieve. One is to be able to start the application remotely and observe its progress and the other is to be able to move or “migrate” this application from one system to another. In order to be able to do this we need some amount of support functionality from the target application. It should be noted that such functionality is already available in many applications that are compute intensive and these requirements do not mandate any change, or minimal change, to the way the legacy application functions. In order to be able to successfully migrate a legacy application it should be able to do the following: 1. Generate a checkpoint file: A checkpoint file is a file that stores information about the state of an application while it is running. Usually checkpoints are generated periodically in legacy applications and no external checkpoint request is issued. The checkpoint file merely contains the current output, states and any information necessary to start the computation from where it was last stopped. Such a technique has been
Figure 1. Images produced from the output of our case study legacy application showing the development of a bar mode instability in a rotating neutron star. This application is written in Fortran, and uses MPI for message passing. Simulations are typically deployed on supercomputers, such as at NCSA and LSU, using some hundreds of processors. This application already had the ability to checkpoint and recover, and was Grid-enabled for migration by altering only details about the checkpoint data files. Visualization provided by Joel Tohline and Shangli Ou (LSU). in use since the times of batch execution and is available in many legacy applications. 2. Portability in case of cross-platform migration: If the migration is to be possible to a system that is different from the one where the application is originally running, it must be ascertained that the application runs on the target platform and has all the necessary libraries. In some cases the application would have to be compiled from source and this would mean having all the necessary compilers and other support software. All the other tasks needed to be performed for successfully migrating the application are performed by a wrapper code written using the GAT API that sits around the legacy application like an envelope acting as the legacy applications interface to the Grid environment.
3.3. Implementation In the Grid community the demand for experience with a wide variety of case studies with real world applications using emerging Grid infrastructure is steadily increasing. Over the last years a lot of Grid middleware has advanced from pure experimental and research state to mature and widely supported software systems. The complexity of these systems, both in terms of administration and application usability, remains high. One of these aspects, the usability for application programmers, is the focus of Grid Application Toolkit [4, 7] as developed in the EU GridLab project [8]. The purpose of the Grid Application Toolkit (GAT) is to provide a complete application oriented abstraction layer
to the underlying Grid middleware. Application oriented here means, that the toolkit provides those functionality (capabilities) the application programmer needs. Such capabilities are often composed of several Grid operations, and may involve more than one Grid service, different protocols etc. Exactly that complexity should be hidden from the application programmer. GAT presents to the application programmer a uniform interface to Grid technologies. The GAT is designed to handle that diversity of Grid middleware. The API exposed to the application is, as far as possible, independent of the Grid middleware incarnation used. In general terms, the capabilities provided by the GAT API implementation, a library, are handled by the GAT Engine. The GAT Engine picks from currently available adaptors, which implement that specific capability. The adaptors are lightweight modular software elements which interface to the specific Grid middleware to be used. The actual Grid technologies which implement the GAT API functionalities are included into the GAT by means of a plugin architecture. This modular design allows the application programmer to worry about the application code and their own science and the Grid technology experts to worry about the plugins which talk to Grid services over whatever protocols they have implemented. In order to enable a legacy application to benefit from Grids we designed and implemented a wrapper code that envelopes the legacy application and provides an interface between the application and Grid environment. The wrapper is written in C and uses GAT calls for transparent access to various underlying Grid subsystems. On current Grids built upon application oriented services and toolkits developed under the GridLab project, the execution of each application is handled by the Grid Resource Management and Brokering Service (GRMS) [9]. This service is mainly responsible for finding suitable resource for an application, providing information about the status of execution, and migrating application to other machine. When submitting a new job to GRMS, the user has to specify a number of processors the application will run on. Other resource restrictions, such as OS and CPU architecture, can also be defined as part of a submission request. All of these restrictions remain the same when migrating the application, although GRMS can explicitly be instructed to change them. The basic workflow of application migration with GRMS is illustrated in Figure 2. At some point after starting the application, GRMS is asked or decides by itself to migrate the job (running application) to some other machine. Before GRMS can stop the job, it sends a checkpoint request to the Mercury Grid Monitoring System service running on the machine where the job was started and the monitoring service, in turn, delivers the request to the application. After receiving the checkpoint request, the ap-
Figure 2. Workflow for application migration using GRMS.
plication saves its current state to a file and registers this checkpoint file to a logical file and exits. Then GRMS replicates the logical file on target host and starts the application there. The application reads its checkpoint file and resumes its computation from the point where it was stopped. Even though the wrapper remains very simple (about 330 lines of code), it is designed to be quite general so that it can be used for varied sorts of legacy applications with minimal need for modifying the wrapper code. The main responsibilities of the wrapper are (i) starting the application, (ii) registering callback function for checkpoint requests, (iii) registering checkpoint files created by the application with a logical file service, and (iv) cleanly terminating the application. All the specific information needed to start and stop a particular application and register its checkpoint file in a logical file service is passed into the wrapper as command line arguments. If more complex actions are desired, shell scripts can be defined and executed by the wrapper for such actions. Since the wrapper does not apply any restrictions on data being registered as checkpoint files, it can easily be instructed to register both checkpoint files and application data so that they are automatically replicated on the target machine during the migration process. The complete scenario of legacy application deployment
based on the basic processes detailed in Section 2 can be described as follows: 1. The wrapper which will start the legacy application is submitted to a resource management service. 2. Once the wrapper is started it registers a callback function for receiving requests for checkpointing from a monitoring system. 3. Since the wrapper has to wait for a checkpoint request, the legacy application cannot be just executed within the wrapper process, and thus it is started as a separate process. After starting the application, the wrapper just waits either for the application to finish or for a checkpoint request to be delivered. 4. If a checkpoint request is delivered to the wrapper, via the callback function from the monitoring system, the specified action is executed. The action can be anything from a simple “do nothing” to a sophisticated shell script which takes the most recent set of checkpoint files, verifies their correctness and packs them into a single file expected by the wrapper. 5. Now, the checkpoint file is ready to be registered to a logical file of a name passed as an argument to the wrapper. 6. After creating the logical file, the wrapper stops itself and the legacy application as well because GRMS expects the job to stop after creating the checkpoint. 7. When the wrapper stops, GRMS knows that it can start replicating the logical file to some other host and then it simply restarts the wrapper on that machine. 8. The freshly started wrapper checks for the checkpoint file and runs a specified action to prepare the file for the legacy application. The action usually undoes what was done in step 4, that is, for example, unpacks the whole set of checkpoint files and moves them to a place where the legacy application expects them. 9. From this point, the scenario continues with step 3.
3.4. Results The paper is inspired from the successful demonstration of the legacy application migration scenario at SC2004 in Pittsburgh. The astrophysics code discussed earlier in this paper was used for the demonstration and was able to migrate between machines in the GridLab testbed. In this paper we demonstrated that a legacy application can be made Grid-aware and can advantage from the facilities of the Grid environment. This however needs some amount of
support both from the application and from the Grid. The current way in which many clusters are configured for instance make it difficult to migrate the legacy application to or from it. Some of the problems that we faced when trying to migrate the legacy application are as follows: 1. Internal cluster nodes often do not have public IP addresses. They are generally a part of a small subnet with only the head node visible to the outside world. The problem is that some cluster configurations even forbid internal nodes to create network connection outside the private subnet of the cluster. This makes it impossible to register checkpoint files with logical file service if the application is not running on head node. So once the application is migrated to such a cluster, there is no chance to move it anywhere else. 2. The Grid Environment is a complex environment that needs to be installed on all the nodes of the cluster and all the components should work together to make the migration scenario possible. For instance the GridLab monitoring service “Mercury” has a local monitoring component that must be running on all cluster nodes. There are several other services that must function correctly to be able to successfully migrate the legacy application. GRMS for instance must be able to find resources and submit jobs to them and the logical file service should be able to register files and replicate them when needed so the legacy application finds them in the right place when needed. 3. Keeping in view the queuing systems that exist on current day clusters such as PBS, the Resource management system must be capable of submitting jobs to these queueing systems and also have more advanced capabilities such as being able to specify the queue names and then be able to track the job after submission. GRMS fortunately provides us with these facilities and makes the task of migration much easier.
4. Conclusions and future work In this paper we demonstrated with the help of the neutron star simulation code that legacy applications can indeed be “Grid-enabled” and that Grid tasks such as migration can be applied. This saves an enormous amount of effort on the part of programmers who have code that runs on legacy clusters and would like to start using the Grid to be able to efficiently utilize CPU cycles, storage resources and human time. The main advantage of our approach is the separation of scientific computation from Grid-aware code. These two parts are formed by separate programs. The scientist who wants to deploy an application (which fulfils requirements mentioned in Section 3.2) on Grid does
not need to know much about Grid technologies. Generally, the only thing the scientist needs to do is to use our wrapper (without making any changes to it) and instruct it how to do application specific actions such as starting, stopping, and packing several checkpoint files into a single one. The ability to Grid-enable applications can be used to provide the benefits of Grid computing to legacy applications without having to modify the applications themselves. A large number of such applications are currently in use such as those in Petroleum Engineering. Applications such as reservoir and drilling simulations have been around since before the advent of Grid computing and these applications are examples of the typical legacy applications that we have shown can be made Grid-aware. The Ubiquitous Computing and Monitoring System (UCoMS) [10] project aims at using Grid Technologies together with Sensor Networks and Petroleum Engineering code to find answers to the potential Oil and Natural gas problems of the twenty first century. Also among the projects that can benefit from this work is the SURA Coastal Ocean Observing Program (SCOOP) that uses traditional wave and ocean modelling code written in Fortran and C to predict weather patterns and coastal phenomena. Information gathered from a large array of sensors, placed on the Gulf of Mexico, is used to feed simulations that forecast various coastal phenomena such as tidal waves, hurricanes etc. Grid enabling such applications can prove to be a great help in both speeding up and increasing the reliability of these applications. During our work on the wrapper and the preparation of migration demo for SC 2004, we identified some common issues the user has to deal with using shell scripts passed to the wrapper. This functionality will be placed directly to the wrapper code so that it does not need to be dealt with repeatedly. The wrapper can also be easily modified to replicate application data previously registered to a logical file on the target machine on demand thus removing the need to register such data as a part of a checkpoint. This paper complements other work such as GriddLeS [11] and other work that tries to expose legacy applications on the web such as [12].
Acknowledgements The authors would like to thank Shangli Ou, Joel E. Tohline (Dept. of Physics, Louisiana State University) and Lee Lindblom (Caltech) for sharing with us their Astrophysics code for neutron star simulation and would like to specially acknowledge Shangli Ou for helping us understand the code and providing the much needed software support without which this task would have been a daunting task to accomplish. The authors would also like to thank Tom Goodale and Hartmut Kaiser for discussions which have aided the development of this work. We are also grate-
ful to many GridLab colleagues for practical help and advice, particularly Andre Merzky and Miroslav Ruda. Jiˇr´ı Denemark would also like to thank the Center for Computation and Technology at LSU and his supervisor Ludˇek Matyska (Masaryk University Brno) for supporting his visit and giving him the opportunity to work with the great team of people working there. This research was supported by DOE DE-FG0204ER46136 and the CCT.
References [1] I. Foster, C. Kesselman, and S. Tuecke, “The anatomy of the grid: Enabling scalable virtual organizations,” Intl J. Supercomputer Applications, vol. 15, no. 3, 2001. [2] I. Foster, C. Kesselman, J. Nick, and S. Tuecke, “The physiology of the grid: An open grid services architecture for distributed systems integration,” Open Grid Service Infrastructure WG, Global Grid Forum, vol. June, 2002. [3] “Globus Project Home Page”, http://www.globus. org. [4] G. Allen, K. Davis, K. N. Dolkas, N. D. Doulamis, T. Goodale, T. Kielmann, A. Merzky, J. Nabrzyski, J. Pukacki, T. Radke, M. Russell, E. Seidel, J. Shalf, and I. Taylor, “Enabling applications on the Grid: A GridLab overview,” International Journal of High Performance Computing Applications, 2003, Special issue on Grid Computing: Infrastructure and Applications, http://www.gridlab.org/Resources/ Papers/ijhpca gridlab 2003.pdf. [5] Kelly Davis, Tom Goodale, and Andre Merzky, GAT API Specification: Object Based, http://www.gridlab.org/Software/ GAT/Documents/Gridlab-1-GAS-0004. ObjectBasedAPISpecification.pdf. [6] Derek Simmel, Thilo Kielmann, and Nathan Stone, “An architecture for grid checkpoint recovery services and a gridcpr api,” 2003, http://gridcpr.psc.edu/GGF/ docs/GridCPR001.pdf. [7] “Grid Application Toolkit web-page,” http://www. gridlab.org/WorkPackages/wp-1. [8] “GridLab: A Grid application toolkit and testbed project home page,” http://www.gridlab.org. [9] “Gridlab Resource Management System web-page,” http: //www.gridlab.org/GRMS. [10] “Ubiquitous COmputing and Monitoring System webpage,” http://www.ucoms.org. [11] D. Abramson and J. Komineni, “Interprocess Communication in GriddLeS: Grid Enabling Legacy Software,” Technical report, School of Computer Science and Software Engineering, Monash University, Australia. [12] T. Bodhuin and M. Tortorella, “Using Grid Technologies for Web-Enabling Legacy Systems,” Eleventh Annual International Workshop on Software Technology and Engineering Practice (STEP’03), Amsterdam, The Netherlands, 2003.
