xDU: A Java-based Framework for Distributed Programming and Application Interoperability Samir B. Gehani Gregory D. Benson Systems Research Group Department of Computer Science University of San Francisco fsgehani,
[email protected]
Abstract Despite several language features and development tools geared toward assisting the implementation of Java-based distributed applications, writing such applications is still cumbersome. Middleware approaches, such as Java RMI and CORBA, place a heavy burden on the programmer to specify unnatural and restricted interfaces for existing classes. To address this problem, we have developed xDU, a simple and intuitive Java-based framework for writing general-purpose distributed applications. In addition to a simple distributed programming model, xDU supports interoperatility with non-xDU (and non-Java) application via CORBA clients. The new programming model is realized by augmenting Java source code with xDU syntax. The annotations can be applied directly to non-distributed applications to achieve distributed functionality. An xDU translator converts xDU source into standard Java and CORBA transport code. The xDU run-time environment allows an xDU application to utilize participating network nodes dynamically and transparently. This paper describes the xDU programming model and corresponding syntax, and our current implementation of the xDU translator and run-time environment. Keywords: tools and environments for software development, distributed programming, Java, CORBA
1 Introduction Many research efforts in distributed systems have focused on devising programming models that facilitate the development of programs that can execute on two or more networked computers. This research has resulted in a multitude of distributed programming languages [2], distributed programming frameworks (e.g., ACE [11]), and middleware (e.g., CORBA [9] and DCOM [8]). Some of these models support explicit forms of code placement, data placement, and remote invocation, while
others support varying degrees of implicit placement and invocation. Distributed programming languages generally have integrated syntax and semantics for expressing a distributed computation, whereas distributed programming frameworks usually augment an otherwise non-distributed language with library calls and data types that support distributed execution. The language approach often leads to concise programs that are easy to develop and maintain, but such languages are currently limited to research efforts that do not have industry support. In addition, the language approach usually leads to insular programming environments in which a distributed application can only interacted with internal components; language approaches do not usually support interoperability with other applications. The frameworks and middleware approaches can leverage off standard languages and provide interoperability between clients running on different platforms, but they often lead to complicated code. Recently, middleware systems — such as CORBA [9], RMI [12], and DCOM [8] — have achieved industry acceptance as a common way to build distributed programs. These middleware systems support the notion of distributed objects and remote invocation. In addition to remote object creation and invocation, each of these systems provides a location service that allows clients and servers to establish connections. Middleware approaches help solve the fundamental problem of defining standards by which distributed programs can communicate. One can think of middleware as a high-level protocol for program interaction. These protocols are necessary to allow distinct programs to communicate in meaningful ways at the language level (e.g., the object level). However, these middleware systems require a programmer to provide a substantial amount of infrastructure that can heavily influence the design of a distributed program. In addition, it is often difficult to transform a nondistributed application into a middleware-based application without significant modifications. Even very simple middleware-based programs require a large amount of scaffolding and program contortions. As
a result, programmers often setup remote interfaces, then “code around” these interfaces. It is often non-trivial to change the remote interfaces and all of the code that depends on these interfaces. Due to their complexity, middleware systems introduce a steep learning curve for programmers that want to write simple distributed applications. We believe that current middleware approaches to distributed programming, while effective, are still too complicated for most programmers who simply want to write software that can take advantage of several nodes in a network. To address these deficiencies, we have developed xDU, a simple framework for Java-based distributed programming. The basic idea behind xDU is to provide a programming model that more closely matches object creation and invocation in standard Java. We introduce the concept of duroot classes. A duroot class is defined just as a regular Java class with some minor syntactic annotations. Objects instantiated from duroot classes can potentially be created on a remote node. A programmer invokes methods on duroot objects the same way as normal objects. By declaring duroot classes, the programmer instructs the xDU run-time environment to dynamically partition the application across network nodes. In addition, because xDU generates pure CORBA IDL, non-xDU applications can interoperate with an xDU application. An xDU translation tool automatically generates transport- or middleware-specific code from duroot annotations. The programmer does not have to be concerned with transport details such as initialization, location, and remote object binding. The goal is provide a more natural programming model such as those found in many distributed programming languages, but target industry accepted programming languages and middleware protocols. The xDU framework is based on the following principles:
Simplify distributed programming for general purpose Java applications. Provide a simple run-time infrastructure for program execution. Provide interoperability with external applications using standard CORBA. Our initial experience suggests that xDU is easy to use and generates code that performs as well as handwritten middleware-based applications. The rest of this paper is organized as follows. Section 2 presents a detailed description of the xDU programming model and source code annotations. Section 3 gives a simple example that shows how to write and execute an xDU application. Section 4 describes the current implementation of the xDU translator and run-time environment. Section 6 summarizes research and production systems that have influenced the xDU framework as well as projects that have goals similar to xDU. Section 7 makes some concluding remarks.
111111 000000 0000 1111 000000 111111 0000 001111 11 000000 111111 0000 1111 00 11 0000 1111 001111 11 0000 0000 1111 0000 1111 00 11 11 00 00 11 00 11 00 11 00 11 00 11 00 11 00 11 00 11 00 11 00 11 00 11 00 11 00 11 00 11 00 11 00 11 00 11 00 11 00 11 00 11 0000 1111 0000 1111 0000 1111 0000 1111 0000 1111 0000 1111 111 000 0011 11 00 11 0000 1111 111 000 00 11 00 00 11 00 0011 11 00 11 00 11 00 11 00 11 00 11
0000 1111 0000 1111 0000 1111 0000 01101111 0000 1111 0000 10101111 00000 11111 0000 1111 00000 11111 000 111 000 111 000 111 000 111 000 111 000 111
main
KEY
1111 0000 0000 1111 00 11 0000 1111 00000 11111 00 11 0000 1111 00000 11111 00 11 0000 1111 00000 1111 11111 00 11 0000 0000 1111 00000 11111 1011111111 00000000 11111111 00 11 0000 1111 0000 1111 00000 1111 101011111 00000000 0000 0000 1111 L 10 0000 1111 0000 1111 0000 1111
Locally instantiated object
1111 0000 0000 Remotely instantiated 1111 0000 1111 0000 1111 duroot 0000 1111 0000 1111 0000 1111 0000 1111 1111 0000 instantiated 0000 1111 L Locally 0000 1111 duroot 0000 1111 0000 1111 Distributable Unit (DU)
Figure 1: Communication Between Distributable Units in an xDU Application
2 The xDU Framework Typically, an object-oriented application is designed such that a group of objects work closely together through interobject communication to perform a particular task for the application. We consider such a group of objects to be an application unit. Such units have the property that they can be logically separated from the rest of the application in a modular fashion to perform their task(s). In xDU, we allow a programmer to define such units as distributable. Every distributable unit (DU) has a root object that will communicate with the rest of the application. We term the object identified to be the interface for the DU as the duroot. A programmer using xDU must select classes in a target application such that the application will benefit if these classes are instantiated remotely. These classes must be annotated as duroot classes by replacing the class keyword with duroot. While any class can be annotated as a duroot, it is the programmer’s responsibility to select duroot classes with appropriate access granularity. That is, while duroot classes can be used just as regular classes, the programmer must take into consideration that accessing duroot objects may require network communication. If the advantage of running a DU on a separate processor is greater than the interprocess communication cost then the programmer should select the root of the DU to annotate as a duroot. Figure 1 shows an abstract view of a running xDU application. Each DU is running on a separate network node and has a duroot that communicates with other DUs. As shown in the figure, a duroot class can be instantiated locally if the programmer explicitly uses the xDU newlocal keyword instead of the new keyword. To support interoperability, xDU will generate pure CORBA IDL for all duroot classes that do not have meth-
ods that accept constructed types as parameters. In turn, the generated IDL can be used by a non-xDU (and non-Java) application to communicate with an xDU application. To build and execute xDU applications, a programmer uses the xDU translator, xducc, and the xDU run-time environment, xdurte. The xducc tool will generate pure Java source and CORBA code from Java source with xDU annotations. An IDL compiler is used to generate CORBA stubs and skeletons. Finally, the generated sources are compiled to Java bytecode and linked with the xDU runtime environment library. In order to execute an xDU application, the xDU runtime environment must be in place. The run-time environment consists of xDU daemons that identify participating nodes and any middleware-specific daemons (e.g., the transient name service of a CORBA implementation). The xDU model currently has some restrictions. First, to decide if a duroot object will be instantiated locally or remotely, xducc alters all method signatures that accept a duroot as a parameter or return an object of a duroot class to reflect the transport specific interface. Such methods will not be able to handle locally instantiated duroots. Second, with the newlocal keyword method overloading allows for all methods that accept or return a duroot to have two interfaces but a common implementation. This will solve the problem mentioned above whereby a duroot object may be accessed locally or remotely at run time, but local or remote duroot instantiation must be decided at compile time.
public class AppModule { AppModule() { } public int processRequest(String stringArg, int intArg) { // request processing code } }
Figure 2: An Abstract Application Module
public class AppClient{ AppClient() { } public int go() { String stringArg = "Test String"; int intArg = 3.14; AppModule am = new AppModule(); return am.processRequest(stringArg, intArg); } public static void main(String args[]) { AppClient ac = new AppClient(); int err = ac.go(); } }
3 Example This section provides some code examples that show how to program with xDU, how xDU source is translated into pure Java and CORBA, and how the translated CORBA IDL can be used by non-xDU applications to interoperate with xDU applications. Our example is based on an abstract application in which a client class makes use of a module class that provides a service. Consider a application that consists of two classes: AppModule (see Figure 2) and AppClient (see Figure 3). The AppModule class provides a service that is used by AppClient and perhaps other classes. Initially we have written our application as a non-distributed program. In order to turn this application into a a distributed in xDU, all that is needed is to replace the class keyword in Figure 2 with duroot. The xDU translator is used to generate pure Java and CORBA. The translated application will take advantage of participating network nodes by transparently creating AppModule objects on the remote nodes. For example, the AppClient in Figure 3 is translated to the code in Figure 4. The translation for AppModule is a bit more complicated because the code provides all of the CORBA code to establish an AppModule object as a remote object.
Figure 3: An Abstract Application Client
public class AppClient { AppClient() { } public int go() { String stringArg = "Test String"; int intArg = 3.14; idlgen.IAppModule am = idlgen.IAppModuleHelper. narrow(Allocator.New("AppModule")); return AppModuleMarshal. processRequest(stringArg, intArg, am); } public static void main(String args[]) { AppClient ac = new AppClient(); int err = ac.go(); } }
Figure 4: The xDU Version of AppClient
A non-xDU application can utilize an AppModule object by using the generated CORBA IDL (see Figure 5). Note that DUMethodSwitch is used internally by xDU and can be ignored by external clients. module idlgen { typedef sequence ByteArray; interface IAppModule { long processRequest( in string stringArg, in long intArg); ByteArray DUMethodSwitch( in ByteArray args, in long methodID); }; };
The first pass over the abstract syntax tree identifies all duroots and generates IDL interfaces for all public methods and data fields. The second pass translates the four translations identified above to pure Java, CORBA, and xdurte calls. The xDU Run-time Environment: xdurte The purpose of the xDU run-time environment is to support instantiation of new duroots on participating network nodes, dynamic utilization of new participating network nodes, and dynamic discovery of xdurte critical services. An application translated by xDU can dynamically utilize participating nodes. Nodes can participate in the distributed infrastructure by running an xDU Agent that represents the machine. The duroot instantiations are distributed among the participants.
Figure 5: The xDU Version of AppClient The xDU framework also supports remote objects that have methods with arguments that are constructed types. Such objects cannot be accessed by external applications because we generate custom marshaling code for the complex data types.
4 Implementation The implementation of xDU includes a source translator, xducc, and a run-time environment, xdurte. The xDU Translator: xducc The xducc tool parses xDU Java source into an abstract syntax tree. The parser and abstract syntax tree builder are based on a modified Java 1.1 grammar using JavaCC and JJTree [7], respectively. We modified the Java 1.1 grammar by adding a production that recognizes the duroot keyword. This keyword is parsed by a modified version of the class production. That is, the production expands to the same sequence of expansion units as an unmodified class declaration with the class terminal replaced by the duroot terminal. JJTree allows a programmer to annotate an input grammar to generate abstract syntax trees. This annotated grammar is fed to JavaCC, which generates a parser and abstract syntax tree builder. Using our new grammar and JJTree directives, xducc recognizes xDU annotations in Java source code. In particular, we can distinguish between regular Java classes and duroots. To convert a non-distributed application to one that is distributed, four key translations have been identified: duroot declaration, object instantiation, method declaration and method invocation. We use the Visitor design pattern from [5] to walk the abstract syntax tree. Most nodes are passed through unmodified, while some nodes require special translation to support the xDU annotations.
Dependencies Between xdurte and xducc The xducc translator generates transport specific code as well as code that calls the xdurte run-time environment. In the case of object instantiation, the modified input source code contains calls to the Allocator to instantiate a remote object and to obtain a reference to this newly instantiated remote object. In addition, each object must be served by a corresponding object server. The companion object server is generated by xducc. An Allocator determines the next Agent on which to instantiate a duroot object based on information from the Agent Manager. It then proceeds to supply the Agent on the remote host with type information about the object it needs instantiated. The Agent depends on the existence of the xducc generated object server. It instantiates the appropriate object server by dynamically resolving the class of the object server using Java reflection. Once an object server is created, it hosts an object instance and registers it with the name service. Thus, the calling Allocator receives an identifier for the newly registered object. It resolves this identifier and returns the remote object reference for the newly instantiated duroot object. Implementation Restrictions The current implementation of the xDU framework imposes the following restrictions.
Passing duroot object references A duroot object reference may be passed between methods of classes instantiated within a DU. However, a duroot object reference may not be passed across DU boundaries. This restriction is imposed because of a limitation of the target transport, CORBA. Distributed garbage collection Our current implementation of the xDU framework does not perform distributed garbage collection.
Constructors with arguments The current version of our translator, xducc, does not handle instantiations of duroots whose constructors require arguments.
5 Experimental Results To demonstrate the usability and performance of xDU, we developed three types of test programs: a method invocation benchmark, a distributed prime number finder, and a generic readers/writers application. For the latter two program types we developed three implementations from an initial non-distributed, multi-threaded version: an RMI version, a CORBA version, and an xDU version. For each test program type, we provide a discussion of each implementation, performance results, and a discussion of the results. In the interest of space, we do not include the source code for each implementation. , The xDU versions are not much more complicated than the equivalent nondistributed, multi-threaded versions, whereas the RMI and CORBA versions are considerably more complicated. The test platform consists of two Pentium II 350MHz machines running Linux 2.2.10 and Blackdown JDK 1.2 [4]. The machines are connected with 100Mbit Ethernet adaptors. For all the experiments, the machines and the network were lightly loaded. Method Invocation Our first set of test programs are designed to determine the cost of local method invocation, remote method invocation with RMI, and remote method invocation with xDU. For the latter two implementations we first create a remote object, then invoke a method on the remote object. To vary the size of the input parameter we used byte arrays of different sizes. We do not provide a separate benchmark for remote invocation with CORBA, as xDU uses CORBA as the transport, so the performance of CORBA and xDU remote invocation are the same. Arg. size (bytes)
void 1 32 128 512
Exec. time (ms) Non RMI xDU -distributed 0.04 0.90 1.65 0.05 1.20 1.99 0.05 2.00 1.90 0.05 1.76 1.93 0.06 1.85 2.61
Table 1: Method Invocation Table 1 provides the results of our method invocation benchmark programs. Remote invocation is clearly more expensive than local invocation. RMI and xDU are comparable. Since xDU uses CORBA as its transport, these benchmarks are really comparing RMI invocation to CORBA invocation. The xDU translator generates pure
CORBA code for this benchmark. We have not yet determined the source of the fluctuations in the RMI time.
Prime Number Finder Our next test program is a prime number finder. The program is designed to demonstrate speedup of a distributed version of an application with coarse-grain parallelism when compared to a nondistributed version. A segmented version of the Sieve of Eratosthenes algorithm (segmented sieve) is used to count the number of primes below a specified threshold, n. Here we compare the execution time of a non-distributed version of the sieve algorithm to distributed versions written in RMI, CORBA, and xDU. n
101 102 103 104 105 106 107 108 109
Non -distributed 10 20 20 20 30 140 1,171 22,062 258,962
Exec. time (ms) RMI CORBA 528 528 528 530 541 645 1,646 14,117 151,534
31 32 33 36 55 163 1,199 14,005 152,040
xDU 27 27 28 30 44 156 1,214 13,851 152,185
Table 2: Segmented Sieve Table 2 provides the results of our RMI, CORBA, and xDU distributed and non-distributed multi-threaded sieve programs. We measured the cost of method invocations and data marshaling. When the threshold value under which to calculate the number of primes, n, is 108 and greater the distributed versions of the sieve application compute faster than the non-distributed version of the same application. This is due to the parallelism in the distributed sieve algorithm and the fact that at 108 , the computation time dominates the communication time. For small values of n the cost of method invocation is higher if RMI is the transport used in the distributed sieve application than if CORBA is used. The transport employed by xDU in this implementation is CORBA. Hence, the results for the CORBA and xDU versions of the distributed versions are close. Both, the CORBA and xDU versions perform method invocations on CORBA remote object references. However, the data passed between the network nodes is marshaled differently. In xDU we marshal the data by flattening constructed data types and arrays into byte streams (arrays of bytes). In this test, arrays are required to be passed between distributed nodes. The results of this experiment suggest that xDU marshals constructed types more efficiently than the CORBA implementation we used.
Iterations Avg. of 500 Write calls Avg. of 500 Read calls
Exec. time (ms) RMI CORBA xDU 1010 1052 1010 1010 1022 1010
Table 3: Readers-Writers Application Readers-Writers Our readers-writers test program measures the cost of passing constructed (non-primitive) data types between distributed nodes. As mentioned above, xDU marshals constructed types differently than CORBA. The xDU mechanism involves streaming constructed types and arrays into an array of bytes. Thereafter, CORBA is used as the transport to invoke a low-level method of a duroot that unmarshals the data from the byte stream at the distributed node. A writer iteratively looks up the remote stub reference to the database, and writes to it. The time taken to read from and write to the remote database is measured for each version of this experiment. The results in table 3 show that the cost to marshal constructed types is more expensive in the CORBA version of the distributed readers-writers application than the xDU version of the same application. We believe this is because our simple approach to marshaling data as byte streams at the level of the application is more efficient than the marshaling done in the CORBA implementation, where specific marshaling code is used in the generated CORBA stubs and skeletons. The cost of marshaling constructed types in the xDU version of the readers-writers application is close to the cost in the RMI version of the same application. We confirm that using the xDU framework to distribute an application will therefore incur little or no cost.
6 Related Work Many recent Java-based research project are investigating how to improve distributed program performance and ease of program construction. In particular, JavaParty [10] is similar to xDU in that it provides a new keyword, remote, that is used just as the duroot keyword. JavaParty provides a source translator and run-time system to build and execute distributed applications. However, unlike xDU, JavaParty targets Java RMI and supports object migration. Also in contrast to xDU, JavaParty was designed to support insular distributed programs; it does not directly support interoperability with external applications. That is, remote objects in these systems cannot be accessed by external applications. The design of the xDU framework has been influenced by other distributed programming languages. Two such languages are SR [1] and Emerald [3]. SR supports the notion resource, which encapsulates data and functions sim-
ilar to a Java class. Unlike a Java class, a resource can be created both locally and remotely. Similar to SR resources, Emerald objects can be created on remote machines and accessed transparently using object references. Unlike SR resources, Emerald objects can be created on remote machines, both explicitly by the programmer and implicitly by the Emerald run-time system. In addition, Emerald objects can migrate from one machine to another and object references will determine the current location of an object at run time. More recently, a system called Coign was developed to automatically distribute COM-based applications [6]. Coign not only decomposes a COM application and executes different components on different machines, but a run-time profiling tool can determine how to efficiently partition an application to minimize network communication between components. Eventually, we want to add profiling support so that xdurte can make intelligent object placement decisions.
7 Conclusions This paper has presented the xDU framework and a prototype implementation. Our initial experience indicates that the xDU programming model is both simple and powerful. We believe xDU has achieved our goal of making it easier for programmers to write distributed applications in Java. By simply annotating Java classes that are candidates to be distributed as duroot classes, an xDU translation tool automatically generates all of the CORBA-specific interfaces and xdurte code that enables remote object creation and invocation. The xdurte run-time environment dynamically creates duroot objects on participating network nodes. In addition, xDU and non-xDU applications can interoperate using the xDU-generated CORBA IDL. Our initial experience indicates that it is straightforward to turn a non-distributed application into a distributed one using xDU. We have written some simple benchmark applications to test the performance of xDU. Our remote invocation benchmark shows that xDU remote invocation performance as well as CORBA and RMI remote invocation. This is not surprising as we use CORBA as our transport. In addition, we have written a distributed prime number finder that exhibits linear speedup. The current xDU model does have some shortcomings we plan to address in future versions. First, the programmer cannot be completely oblivious when using duroot classes. The programmer must design duroot classes with the knowledge that method invocations will potentially have high latencies if the duroot object is placed on a remote node. Second, duroot objects cannot migrate from one node to another. Once a duroot object has been created on a remote node, it will stay there until it terminates.
Acknowledgments We greatly appreciate input on earlier versions of this paper from Peter Pacheco, David Wolber, Meghna Mehta, and Thomas Alexander. We also thank Meghna for her contributions to the RMI experiments. We thank the anonymous referees for their helpful suggestions. This work was partially supported by a University of San Francisco faculty development research grant.
References [1] G. R. Andrews and R. A. Olsson. The SR Programming Language: Concurrency in Practice. Benjamin/Cummings, Redwood City, California, 1993. [2] H. E. Bal, J. G. Steiner, and A. S. Tanenbaum. Programming languages for distributed computing systems. ACM Computing Surveys, 21(3):261–322, September 1989. [3] A. Black, N. Hutchinson, E. Jul, and H. Levy. Object structure in the Emerald system. In Proceedings of Object-Oriented Programming Systems, Languages and Applications, pages 78–86, November 1986. In ACM SIGPLAN Notices, 21(11). [4] S. Byrne. Blackdown - a free virtual machine to run Java code. http://www.blackdown.org/. [5] Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. Design Patterns: Elements of Reusable Object-Oriented Software. AddisonWesley, Reading, MA, 1995. [6] G. C. Hunt and M. L. Scott. The Coign automatic distributed partitioning system. In Proceeding of the 3rd Symposium on Operating Systems Design and Implementation, pages 187–200, New Orleans, Louisiana, February 1999. USENIX. [7] Metamata Inc. JavaCC, JJTree, JJDoc, 1996. http: //www.metamata.com. [8] Microsoft Corporation, Redmond, Washington. Distributed Component Object Model (DCOM), 1999. http://www.microsoft.com/com/ tech/dcom.asp. [9] Object Management Group, Inc., Framingham, Massachusettes. Common Object Request Broker: Architecture and Specification, Revision 2.30, June 1999. http://www.omg.org/corba/ corbaiiop.html. [10] M. Philippsen and M. Zenger. Javaparty - transparent remote objects in java. Concurrency: Practice and Experience, 9(11):1125–1242, November 1997.
[11] Douglas C. Schmidt. The adaptive communication environment, an object-oriented network programming toolkit for developing communication software. In Sun User Group Conference, December 1993. [12] Sun Microsystems, Inc. Java Remote Method Invocation (RMI), 1999. http: //www.javasoft.com/products/jdk/ 1.2/docs/guide/rmi/.
