Next-Generation Virtual Worlds

Virtual World Architectures Track

Editor: Craig W. Thompson • [email protected]

Next-Generation Virtual Worlds Architecture, Status, and Directions It seems likely that the Web as we know it will undergo several paradigm shifts in the coming years: its content will expand from 2D documents to also accommodate 3D virtual worlds; with new technologies like RFID and smartphones, we’ll be able to link to a place or thing (in the real or virtual world) and “go there”; and we’ll associate more metadata annotations with things in the real and virtual worlds to create increasingly smart semantic worlds. But to realize these promises, the virtual world and ubiquitous computing communities will need architectures that are standard enough to become pervasive and flexible enough to expand beyond the applications we have today. This article describes some of the motivations and requirements for common core architectures for virtual worlds and introduces a track of articles to follow in later issues of IEEE Internet Computing that aim to change the way humans interact with the Web and their world.

Craig W. Thompson University of Arkansas

60

F

or the past 20 years, the wonderful World Wide Web has been document-centric — that is, most websites display their increasingly dynamic content in the form of webpages (of course, pictures and videos are also embedded on these pages). Critically, anyone can create Web content that anyone else can view, worldwide. At the same time, if you ask a 20-yearold university student, “How long have you been playing video games?,” the answer is liable to be 22 years. Facebook aside, many students will tell you

Published by the IEEE Computer Society

that they spend 10 to 20 hours a week playing video games, most of which are 3D multiplayer games with limited content extensibility. But change is afoot. Imagine the Web in 5 to 20 years. Many websites are still document-centered, but many others are 3D worlds. Conventional URLs are extended to indicate a virtual world region plus local XYZ coordinates (and possibly time, role, and other attributes). A user follows a 3D link into this virtual world, becomes an avatar, moves about and interacts, and possibly changes focus

1089-7801/11/$26.00 © 2011 IEEE

IEEE INTERNET COMPUTING

Next-Generation Virtual Worlds

to other websites or virtual worlds before coming back to this one. This all happens in a conventional browser, accessible from a smartphone. As technology quickly improves, it becomes easier to build virtual worlds — real time and realism improve, and many new applications are enabled. Eventually, in 10 years or so, it’s possible for children to script a movie like Avatar when they’re supposed to be doing homework. There’s a sense of manifest destiny that this vision will occur, but to get there, we must develop a better understanding of the architecture of virtual worlds and of driving virtual world applications. The recent X10 Workshop on Extensible Virtual Worlds (March 2010, held in-world in Second Life; http://vw.ddns.uark. edu/X10) concluded that gaming architectures and virtual world architectures are similar in several ways (multi-user, 3D rendering, depending on a physics engine) and different in only a few (games have a specific objective and cache content on the client machine, whereas virtual worlds let users create new content), but, that said, there are significant differences among different virtual world architectures, and many are developing in isolation without a clear understanding of others. Because much is at stake, the virtual world development community recognizes the need to share architectural knowledge and determine the opportunities for convergence and interoperability that will be needed for standards that enable a universal 3D Web to become a reality.

Very Brief History of the Internet, Web, Gaming, and Virtual Worlds

In the beginning, there was the real world — a very high-def, 3D virtual world. Although virtual worlds can explore alternate realities, the real world sets a high bar in terms of quality and diversity, so considering the real world as a kind of virtual world helps when considering pervasive computing applications that can occur in both real and virtual worlds. The following very brief history leaves out the development of many Internet and communication protocols, as well as early computer games like Pong, hundreds of video games, and dozens of social virtual worlds. But it argues the point that it’s time for Web technologies to converge with virtual world technologies. Such a merger can usher in many new kinds of entertaining applications as well as so-called “seriJANUARY/FEBRUARY 2011

ous games” — applications interesting to society, business, and government communities. Way back around 1970, Internet pioneers developed a common set of networking protocols for how to send information among heterogeneous computers. The Web as we know it was “invented” around 1990. Somewhat simplistically, a Web browser on any end-user client machine connects to any of millions of servers, which is where Web content is stored. In the late 1990s, massively multiplayer online games (MMOG) were developed and soon commercialized; especially popular were role-playing games (MMORG) like Everquest and World of Warcraft (with close to 12 million monthly subscribers worldwide). The emergence of game consoles like Xbox in 2001 and improved broadband brought gaming to the mass market. In the 1990s, the US Department of Defense (DoD) announced requirements for distributed 3D simulations — for instance, war game simulations involving simulated tanks and aircraft. Different organizations needed to supply different models, necessitating an interoperability standard. High Level Architecture (HLA) was developed and later mandated as a standard specification to meet this need (see http://ieeexplore.ieee. org/xpl/freeabs_all.jsp?arnumber=5553440). In HLA, simulation entities called federates are connected into a federation via a runtime infrastructure (RTI); an object model template (OMT) specifies information messages communicated between simulations. The 3D virtual world Second Life (http:// secondlife.com) launched in 2003, with 18 million accounts registered by 2010 (1.3 millions logins in the past two months). A free client program called a viewer (very much like a browser) enables users (called residents) to interact with each other’s avatars, which can walk, fly, teleport, build objects, and own land and objects. A scripting language attaches behaviors to objects, which can have associated digital rights that make it possible for one resident to create an object and allow another to copy it. In Second Life, active content is associated with regions of land that are in turn associated with servers. This means the Second Life model isn’t unlike the Web model: clients anywhere can interact with any server and follow links or teleport to that server. A key difference is that Second Life’s developer, Linden Labs, owns all the servers. 61


Open Research Directions and Requirements

Second Life can be viewed as a baseline in providing a commercially popular virtual world platform for free use worldwide. But, as a platform, Second Life is proprietary and has several limitations (fixed-sized regions tied to servers limit the number of avatars and primitive objects that can occupy a region, a homegrown scripting language, limited content import/ export, and more). Many alternative virtual worlds have been developed to remove various limitations: OpenSimulator (open source), Unity3D, Blue Mars, Open Wonderland, Open Cobalt, and dozens more. 3D virtual worlds let users model real and fantasy worlds. Some of these worlds are extensive, with tens of thousands of avatar residents, and require grids of thousands of machines. Although it isn’t hard to believe that a 3D Web can someday gracefully complement today’s document-centric Web and that such virtual worlds will model the Earth in credible detail, it’s slow in happening. Educators and serious gamers want to experiment with virtual worlds but often find the platforms difficult to extend. What are the roadblocks, and how can we accelerate the pace of progress to realize the vision? This article introduces a special track of IEEE Internet Computing on virtual world architectures that will explore how to take 3D virtual worlds to a level where they’re a pervasive complement to the Web. It will cover topics such as • visions for virtual worlds — 3D social networking, 3D Web, second Earth, semantic world, holodeck, matrix, complete model of the past, and immortal avatars; • experience reports related to virtual world applications involving socializing, meetings, training, healthcare, retail, logistics, real estate, museums, archaeology, defense, disaster relief, with fan out to communities such as K-12 education; • analysis of current virtual world architecture stacks — trade-offs related to scalability (many avatars, models with varying accuracy), extensibility (services, plug-ins, capabilities), evolvability as capabilities are added or removed, performance studies, new security models, and privacy-assurance architectures; • plans for populating and provisioning vir62

www.computer.org/internet/

•

•

•

•

tual worlds — rapidly populating a large space by importing (or generating) terrain, buildings, and objects, and/or using GIS and CAD standards to build on, help build, and keep models updated via image stitching and image-to-model technologies; border crossings — connecting enclaves that have different properties and enabling avatars and their inventories to move across boundaries; novel services and capabilities of related areas — games, GISs, Web technologies, artificial intelligence, cloud computing, serviceoriented architectures, social networking, mobile computing, robotics, sensor networks, and the Internet of Things; other novel services and capabilities — search engines, spatial queries, ontologies, workflows, planners, microlicensing, and micropayments; and integration with the real world — mirror worlds that tie the real and virtual worlds together using technologies such as smartphones and RFID.

One way to understand future requirements for a virtual world is to consider what’s lacking in the most popular current virtual world — that is, Second Life, which I do in the following sections. Another litmus test is to compare virtual worlds to the real world.

Populating Virtual Worlds 3D virtual worlds typically consist of representations for terrain, avatars, objects, textures, and other forms of digital assets. It’s desirable for end users to be able to develop content (as permitted in several virtual worlds) but also to import and share assets from already mature communities (maps from the GIS community, building plans from architects, models of objects from the CAD community, and realistic face and body animations). Because of Second Life’s internal economy (residents buy and sell created objects) and its proprietary object format based on prims (primitive graphics objects), Second Life doesn’t make it easy to import or export content. In contrast, several other virtual worlds have embraced mesh architectures, use the Collada standard for representing objects (https://collada.org), and can import 3D content from Web repositories such as Google’s 3D warehouse (http://sketchup.google. IEEE INTERNET COMPUTING


com/3dwarehouse/). Generally, you want to download map data for terrains, CAD models of products, city plans, and so on to populate a city or build a virtual Mars, and the ability to do this in minutes to hours instead of months to years.

Accuracy

they can operate on a variety of platforms, from high-performance gaming engines to older PCs to smartphones over cellular connections. This could require expansion joints in the form of throttling and caching mechanisms throughout the architecture.

Second Life is cartoonish, with approximate object models not calibrated for accuracy. It can be hard to position avatars because their movements aren’t precise. But other virtual world applications require accuracy, such as tolerance-sensitive CAD models, building plans, and military applications requiring precision ballistics. The DoD Information Assurance Certification and Accreditation Process (DIACAP) model certifies models to ensure they meet accuracy requirements. Today’s gaming engines focus on realism such as “cloth simulation” so that a cape hangs naturally.

Extensibility

Scalability

Educators and serious gamers want to experiment with virtual worlds but often find the platforms difficult to extend.

Second Life’s residents might not have the latest graphics card or game engine, so Linden Labs has set limitations on 50 avatars and 15,000 objects per fixed-sized region of land (256 m × 256 m). Generally, one (or a small number) of regions are assigned to a server, and a collection of servers forms a grid. Second Life supports a separate grid for teenagers, for instance, because the main grid has a fair amount of adult content (somewhat segregated). The open source OpenSimulator virtual world doesn’t enforce avatar and object count restrictions, so lag can become an issue. Just considering scalability, it’s clear that some virtual worlds are restricted. If you want to model a hospital, the object count might run to the hundreds of thousands; if you want to model a stadium on game day, the avatar count could be tens of thousands. But these aren’t the only dimensions of scalability: you might also want to model the microscopic and the cosmic. Different virtual world systems have explored alternative means of scaling. One approach is to partition virtual world services so that different services (login, inventory, and so on) run on different servers. Another approach uses octrees (http://en.wikipedia.org/wiki/Octree) so that region size can grow or shrink dynamically depending on the activity in a region. Related to both accuracy and scalability, another requirement for virtual worlds is that JANUARY/FEBRUARY 2011

Where Second Life provides a fixed platform with applications built using its proprietary scripting language, it seems likely that virtual world applications may or may not require a certain functionality. Some virtual worlds, like Open Wonderland, provide plug-in architectures to meet this need. Since the line between virtual world applications and virtual world platforms is fuzzy, a well-defined collection of extensibility mechanisms would help accommodate a wide variety of application types via a library of extension services.

Workflow Service One extension that would be useful for some applications is a workflow service. Virtual worlds provide a variety of scripting languages (Second Life’s proprietary Linden Scripting Language or more mature languages like C#, Lua, JavaScript, and others) for adding behaviors to objects. But it’s often difficult to coordinate multiple objects — for instance, two avatarbots carrying a stretcher (avatarbots are avatars controlled by programs instead of people). More generally, it would be nice if it were easier to define workflow recipes that describe the roles of doctor, nurse, and patient in a catheter operation. It would be useful to represent parameters and variations in such operations, to provide for exceptions (as when a doctor drops a scalpel), and even to tie leaf-level workflow operations to higher-level planning rules and reasons for actions. Mixed initiative simulations involving avatarbots and avatars controlled by humans are also needed for training. 63


Ontology Service Another useful extension would be an ontology service. Whereas humans can tell a door from a castle using image recognition and domain knowledge built into their brain, virtual world engines might know how to render objects but not what they are functionally. In the same way that the Semantic Web enables machines to process metadata on webpages, there’s a need to associate metadata with virtual world objects. Optional naming or labeling fields aren’t further used by virtual world engines, but such metadata could be used for a variety of purposes: associating descriptions, prices, owner’s manuals, discoverable APIs, and many other uses. It’s interesting that, in the real world, objects aren’t labeled explicitly, but if they were (think RFID), similar benefits would accrue. Indeed, the same ontology service could be used by both virtual worlds and the real world. For instance, it would be convenient to use a cell phone to control a nearby real-world thermostat or turn off your real oven by visiting your virtual home. Not all applications need workflow and ontology services (or other services that might be useful in many but not all applications). Being able to plug in such services when needed would be useful.

Searching Virtual Worlds Virtual worlds contain many objects and locations, and some individual worlds provide search services within a single world. It would be useful to use search spiders or similar means to wander through virtual worlds, cataloging objects of interest across all available worlds to create a cross-world search service. Statistics on average number of objects or scripts per region would help us understand virtual worlds; searching and location-aware ontologies would help us find objects and “places like this one.” As with the ontology service, virtual world search methods are also applicable to real-world content gathered by vehicles or backpack crawlers.

Interoperability One of the most critical areas needing agreement is interoperability. Because many implementations of virtual worlds will exist in a 3D Web, there’s a need for base-level agreement on standards. There are many kinds of interoperability. For instance, it seems useful for an avatar to move from one virtual world to another, 64

www.computer.org/internet/

retaining identity and possibly appearance, clothing, inventory of objects, and even currency. As we know from the scalability discussion, boundary crossings might not be allowed or might lose information. Other forms of interoperability include conversations in chat or voice across world boundaries. Similar to the real world, we can expect other differences as an avatar crosses from one enclave to another, such as currency exchange, permissions, rules, policies, and quality-of-service differences.

Borrowing from Related Technologies Virtual world technologies don’t (and shouldn’t) have to invent the entire stack of component technologies. Where possible, they should build on existing standards as mix-in technologies. The X10 workshop identified the following technologies as related to virtual worlds: virtual reality, mixed reality, gaming, serious games, mobile and pervasive computing, GIS, Collada, graphics, chat, audio, social networks, telepresence, search engines, ontology, knowledge discovery, Semantic Web, Internet of Things, RFID, identity management, smart homes, artificial intelligence, simulation, World Wide Web, 3D Web, service-oriented architectures, e-healthcare, e-business, e-education, e-science, training, and entertainment. If we carefully consider each of these technical areas, we might find existing standards and technologies that could benefit virtual worlds — or new functionality that could be included in them.

Challenging Assumptions

Another way to understand the range of virtual world technologies is to consider the range of applications that could be enabled by virtual worlds — meetings, smart homes, war games, archaeology sites, healthcare, and virtual tourism. Each of these can add to the architectural requirements list. So, too, can movies like Avatar and TV shows like Caprica, where avatars move across the boundary from virtual to the real world. One other way virtual worlds and pervasive technologies might challenge us involves longheld assumptions about telepathy, ghosts, multiple personalities, immortality, humanity, and even heaven. While we can’t quite read each other’s thoughts at a distance as in telepathy, telepresence is becoming more common and acceptable through meetings, classes, and conIEEE INTERNET COMPUTING


ferences held in virtual worlds. While we don’t believe in ghosts, by using mirror worlds we can make a change at a remote location in a virtual world that directly affects the real world. Individuals are increasingly used to playing many roles and are comfortable with male, female, animal, and robot avatars or skins. Combining avatars who look like a person with chat bots programmed with a person’s experience provides a new genre of interactive (auto) biography and a kind of immortality. Considering human capabilities, we can identify long lists of ways to augment them (super hearing, X-ray vision) and try these out in virtual worlds (http://v w.ddns.uark.edu/content/designdoc. htm#_Toc202796960). Finally, when we consider heaven and how old we’ll be and what we’ll look like, surely we’ll be 22 years old and beautiful (just as everyone is in Second Life), with San Diego-blue skies every day (again, as in Second Life). Well, maybe not — as Shel Silverstein once sang, “After you’ve ‘bin havin’ steak for a long time, beans, beans taste fine” (www.lyricsbay.com/beans_taste_fine_lyrics -shel_silverstein.html), that is, variety spices up the real world.

T

he 3D virtual world community is actively working in the direction of developing core standards. A long-lived Architecture Working Group in Second Life was recently complemented with an IETF Virtual World Region Agent Protocol1 effort (VWRAP; http://tools. ietf.org/wg/vwrap); the Web 3D Consortium (www.web3d.org) is developing some needed 3D standards; and a newer IEEE Metaverse Standards working group is developing a reference architecture for virtual worlds (www.metaverse standards.org). Most of this article has been about the manifest destiny of convergence of Web and virtual world technology into a 3D Web, but this megatrend is only part of the story. A complementary story involves the Internet of Things. Viewing the real world as one more virtual world, we can see that if we label consumer objects with RFID tags (giving them explicit identity), and, if smartphones are extended with RFID readers as they have been with GPS, many crossover applications might be possible. As humans walk about in the real world, their avatars could walk about in a mirrored virtual world. Like JANUARY/FEBRUARY 2011

search spiders, their smartphones could track their location (yes, there’s already an app for that) as well as the RFID-labeled objects nearby, enabling the building and updating of a parallel virtual world. If you purchased a lamp and brought it home, a virtual lamp would be installed in your virtual home. In a plug-andplay-like way, the virtual lamp could be used to control the real lamp using interface controls uploaded from the cloud to your smartphone remote control. We’ll need architectural standards to make this a reality, but it’s likely the case that the protocol stack will be similar for a real-world Internet of Things and a 3D Web full of virtual worlds. Reference 1. J. Bell, M. Dinova, and D. Levine, “VWRAP for Virtual Worlds Interoperability,” IEEE Internet Computing, vol. 14, no. 1, 2010, pp. 73–77. Craig W. Thompson is the Charles Morgan Chair in the Department of Computer Science and Computer Engineering at the University of Arkansas. His research interests include artificial intelligence, databases, middleware architectures, virtual worlds, RFID, and pervasive computing. Thompson has a PhD in computer science from the University of Texas at Austin. He is an IEEE Fellow. Contact him at [email protected].

Selected CS articles and columns are also available for free at http://ComputingNow.computer.org.

IEEE Computer Graphics and Applications magazine is indispensable reading for people working at the leading edge of computer graphics technology and its applications in everything from business to the arts.

Visit us at www.computer.org/cga

65