PC Clusters for Virtual Reality

Transcription

1 See discussions, stats, and author profiles for this publication at: PC Clusters for Virtual Reality ARTICLE JANUARY 2008 DOI: /VR Source: DBLP CITATIONS 23 READS 97 3 AUTHORS: Luciano Pereira Soares Insper Institute of Education and Research 42 PUBLICATIONS 129 CITATIONS Bruno Raffin National Institute for Research in Computer 103 PUBLICATIONS 885 CITATIONS SEE PROFILE SEE PROFILE Joaquim Armando Jorge Inesc-ID 359 PUBLICATIONS 2,306 CITATIONS SEE PROFILE Available from: Joaquim Armando Jorge Retrieved on: 08 April 2016

2 The International Journal of Virtual Reality, 2008, 5(3):1-5 1 PC Clusters for Virtual Reality Luciano P. Soares, Bruno Raffin and Joaquim A. Jorge 1 Abstract In the late 90's, the emergence of high-performance 3D commodity graphics cards paved the way to the use of PC clusters for high-performance Virtual Reality (VR) applications. Today PC clusters are broadly used to drive multi-projector immersive environments, among other high-performance VR tasks such as tracking and sound synthesis. This survey presents the different approaches that have been developed to use PC clusters for VR applications. We review taxonomies and the most common software and hardware tools that enable us to take advantage of clusters. We also discuss some new trends for the near future. Index Terms PC Clusters, Virtual Reality. I. INTRODUCTION PC clusters became popular in the mid 90's for batch processing of high-performance applications. Rendering farms are an example of batch processing. Today, PC clusters are very common among the machines in the top 500 supercomputer ranking [1]. Their success is mainly due to: their low cost, because they are mainly built with commodity components produced for the mass market; their modularity, which enables building a cluster adapted to the user's needs regarding components, size or performance; their compliance with standards, which favors software and hardware interoperability; their upgradeability, since the commodity market produces new and more powerful devices often; the availability of a large range of open-source software solutions, which enables customizing any given software layer, if and when it proves necessary. In the late 90's, the emergence of high-performance 3D commodity graphics cards paved the way to the use of PC clusters for high-performance Virtual Reality (VR) applications. It was first motivated by the need to have multiple video outputs to drive multi-projector immersive environments. One PC was not able to support multiple video outputs, via one or even several graphics cards, for high-performance 3D graphics. The clustering approach was not new, as it was used in the early 90's to drive the first Cave with a cluster of SGI workstations [2]. Since these early uses, PC cluster technology and software approaches have evolved vastly, giving rise to different solutions with different advantages and drawbacks regarding performances, portability, ease of use, etc. In the present work, we survey the different approaches that have been developed to use PC clusters for Virtual Reality Maniscript Received on August 2, Luciano P. Soares, lpsoares@tecgraf.puc-rio.br Bruno Raffin,. bruno.raffin@imag.fr Joaquim A. Jorge, jaj@inesc-id.pt applications. We first focus on parallel rendering, i.e. how to take advantage of the graphics power of a PC cluster to drive a multi-projector environment. The solutions range from a duplication of the application on each node with the appropriate synchronizations and data communications to keep the different copies coherent, to graphics-oriented parallelization based on sort-first, sort-middle or sort-last paradigms [3]. We also discuss low-level synchronization issues, i.e. swap-lock and gen-lock, in conjunction with stereo-rendering constraints. We put these different approaches in perspective with the evolution of PC and network components. We also study the trends in video projecting technologies, as they influence the tasks PCs have to perform, such as high image resolution and image blending. But PC cluster benefits go beyond parallel rendering. Having several computer nodes enables to distribute tasks not directly related to rendering, taking advantage of additional computing or I/O capabilities. Thus, one issue is how to coordinate and distribute the different tasks on a cluster. We survey existing approaches from parallel scene graphs to sequential and parallel task coupling. An overview of the most advanced applications regarding their ability to take advantage of the potential of large PC clusters is also presented. PC clusters basics are reviewed in section 2. Section 3 discusses the different levels of parallelism that may be present in a VR application. Section 4 presents the image lock constraints that a cluster should comply with when driving a multi-projector environment. Section 5 reviews some of the most common software tools supporting clusters, while section 6 presents some hardware compositing devices. Display technology issues are discussed in section 7, before our conclusions. 2.1 Hardware II. BASICS A PC cluster is a set of interconnected PCs dedicated to process high-performance parallel applications, usually gathered in a single room. This architecture can range from low-end single CPU PCs connected through a common Ethernet or Gigabit-Ethernet network to high-end multi-cpu PCs connected through some high-performance network like Myrinet or Infiniband. In recent years, the PCI-Express bus has been probably the most important hardware evolution for PC clusters. It enables a significant increase in data transfer bandwidth for two important components: The bandwidth of high-performance networks like Myrinet was at first limited by PCI and after that by PCI-X busses rather than by their own interconnection technology. The

3 2 The International Journal of Virtual Reality, 2008, 5(3):1-5 availability of PCI-Express opened the way to very high bandwidth. For instance, the Myri-10G cards from Myrinet achieve a point-to-point bandwidth of 1.2 GBytes/s. The bandwidth for transferring data from memory to the graphics card also takes advantage of the PCI-Express bus. It has higher performance than the AGP bus in both directions. AGP 1X has a speed of 264 Mb/s; the last AGP version, AGP 8X, has a speed of 2 Gb/s. The speed of PCI-Express depends on the number of lanes. Usually for graphics cards a x16 slot having a speed of 4Gb/s is used. Theoretical studies [4] show that a single PC node can drive up to 20 displays at a combined resolution of 38 megapixels, although this solution presents several bottlenecks and it is not feasible in the present. 64-bit processor architectures (AMD Opteron, Intel Itanium or 64-bit Xeon) allow addressing beyond 4 GBytes of memory, the limit imposed by 32-bit architectures. This is useful for memory-intensive applications. Also, 64-bit architectures have been providing higher performance than their 32-bit counterparts. However, experiencing a real application performance increase may not be straightforward. Part of the code may need to be reworked (especially with the Intel Itanium processor). Graphics cards have become more versatile, and it is possible to program part of the graphics pipeline through programmable shaders to achieve high-quality and high-performance final rendering results. As PCs usually provide several PCI-Express slots, it is now possible to install several graphics cards, while only one AGP slot was previously available. Also notice that most commodity graphics cards now provide two digital video outputs GPGPU Since the graphics cards are very fast vectorial processing units, they can be used for non-graphical computation; this procedure is known as GPGPU (General-Purpose Computation on GPU). For example, the GPU can run the entire physical and visual computation of a simulation avoiding transferences with the CPU. In a cluster, several graphics cards are available and can be used to compute heavy problems and data sets. Nvidia has released CUDA [5] (Compute Unified Device Architecture), which makes the development of GPGPU applications easier. It uses standard C language for software development oriented to SIMD (Single Instruction Multiple Data) [6]. GPGPU has been promising to increase computational power to teraflops in a single box. In order to implement a dedicated GPGPU cluster, Nvidia also released the Tesla architecture [7]. Since there is no reason to implement video-output internal hardware for GPGPU, this solution can reduce costs and rack space, since it is just one rack unit high. But it is important to know that programming in GPGPU architectures is different from standard CPUs, and porting programs is not an easy task. 2.2 Software One difficulty with clusters is to install and maintain a coherent system on all computer nodes. A classical approach is to rely on a disk-cloning mechanism. From time to time, when incoherencies appear between machines, for instance because a user has changed a configuration on a set of machines and forgot to propagate the change on all machines, the node's disks are cloned from a reference computer. Linux is the main operating system for clusters. Several Linux cluster distributions, such as Rocks [8], provide tools to ease the installation and maintenance of clusters. They usually provide fast cloning mechanisms and some packages for classical cluster tools like the message-passing library MPI or the cluster-performance monitoring tool Ganglia [9]. Since Linux kernels and additional packages are always being updated, this system is quite used in research centers that want the latest technologies available as soon as possible. Though these tools ease cluster management, the user still has to see a cluster as a set of PCs, each one running its own OS. Ongoing efforts are focused on developing single-image operating systems for clusters, such as Kerrighed [10]. The goal is to provide the view of a unique SMP machine on top of a cluster of standard PCs. It is also possible to configure Microsoft Windows to drive a PC cluster for VR. Some kind of remote display to configure the nodes, such as VNC or Microsoft Remote Desktop, is usually necessary, making its use more difficult. Nonetheless, some device drivers are better or only available for this operating system. It is important to note that the industry in general relays on several Windows applications for Virtual Reality on PC workstations. 2.3 Set-up Clusters can be organized in many ways. Usually 19" racks are very convenient, but unfortunately rack-mount workstations are not common and, due to the graphics card size, the nodes are often 4Us or 5Us wide, using much space in racks. That is why many clusters for VR are organized on simple shelves (Fig. 1). Fig. 1. Cluster organized on simple shelves. Keyboard and mouse are not really necessary for a PC cluster, but can help debug problems and manage the computer nodes; usually KVMs (Keyboard, Mouse and Video) can help

4 The International Journal of Virtual Reality, 2008, 5(3):1-5 3 this access. A KVM USB is even better, since just one cable for each node for keyboard and mouse signals is necessary. Console access can help cluster control as well as other VR devices. There are some special computers [11] that have several serial ports that can connect to all cluster nodes, creating an alternative access, which is mainly necessary in case of network failures. Sometimes routing the video signals is necessary, especially if there are many video generators in the VR facility. There are several video switches available in the market, but due to the high resolution usually required in VR it is important to choose a video switch that is able to handle the bandwidth used. 2.4 Clusters versus Dedicated Supercomputers The difference between a PC cluster and a dedicated supercomputer is getting thinner (at least for small- to medium-sized configurations). They currently use the same CPUs and GPUs, and high-performance networks that provide about the same performance. They run the same operating system (Linux). The main difference lies in the vendor's ability to deliver turn-key solutions with strong hardware/software integration, validated configurations and quality user support. This was not always the case. For instance, Silicon Graphics (SGI) computers were very common in Virtual Reality facilities in the 90's. At that time these computers had graphics cards [12] [13] powerful enough to display high-quality 3D content in real time. They achieved parallel rendering through special busses (Triangle and later Vertex), not available on commodity graphics cards. The Onyx computers have a distributed memory with specific hardware to implement a cache coherency protocol, on top of which a virtual shared memory was built [14]. The SGI operating system, IRIX, is a single-image UNIX-like OS. Another important project in the field of parallel rendering and clustering is PixelFlow [15]. It was developed to support parallel rendering in a complete dedicated hardware. It has an array of renderers that compute a full image with a fraction of the global primitives needed. At the end a high-performance image-compositing network combines the images from the array nodes in real time. PixelFlow can be coupled to a parallel supercomputer for immediate-mode rendering. It also works in retained mode, the geometries being stored inside its hardware. For image compositing PixelFlow uses binary trees. One important aspect about PixelFlow is its support for deferred shading in compositing, enabling high-quality graphics without increasing image-composition bandwidth or redundancy. III. PARALLELISM FOR VR: A TAXONOMY We can distinguish four main components more or less intricate in a VR application: Input: Inputs include static data stored into files as well as dynamic data captured by various sensors, such as a tracker, a wall-clock time or random number generators. Simulation: A VR application can involve several simulations ranging from solid-object collision-detection algorithms to fluid simulations. Animation: The properties of objects present in a scene are updated according to input and simulation results. Rendering: For user feedback, an output is computed and rendered on the appropriate devices. The most common outputs are images, sounds and forces. The goal when using a cluster is to take advantage of the additional resources available to alleviate performance bottlenecks. To achieve this goal it is necessary to split processing amongst the different cluster nodes. There are basically two different approaches: Data parallelism, where several instances of the same task are executed concurrently but on different datasets. Task parallelism, where different tasks are executed concurrently. Parallelism implies data communication and synchronization to ensure proper task coordination. In particular, data redistribution (or sorting) steps are required to make data computed at source tasks available to the target tasks. Depending on the nature and amount of data to be redistributed, the cost can vary significantly. For instance, input events retrieved by a position tracker are limited to a few bytes, while graphics primitives can be significantly larger. Below we discuss the main approaches used for the different components of a VR application and the required redistribution steps. 3.1 Input Task parallelism is the most common approach. Because input devices are usually independent, they can be distributed on different nodes [16]. For instance, a tracker is executed on one node, while a wand is executed on a different one. This also enables a better use of the connectors available on the different PCs. Data parallelism is used in some cases for computation- and data-intensive input devices like multi-camera environments [17] [18]. 3.2 Simulation Task parallelism can be used to execute several simulations concurrently. Large simulations, like collision detection or fluid dynamics, may be internally parallelized. For instance, data parallelism is a classical approach for parallelizing fluid simulations. The space where the fluid can evolve is split into regular cells that are then cyclically distributed by blocks [19]. 3.3 Animation The data are distributed onto different nodes (data parallelism), each node being responsible for updating its data [20]. As some data may be replicated on several nodes to save on communication, care must be taken to manage coherency between the different copies when a data set is modified. 3.4 Rendering Data parallelism is the main approach for rendering. Let us

5 4 The International Journal of Virtual Reality, 2008, 5(3):1-5 focus on graphics rendering. The standard taxonomy for parallel rendering distinguishes three broad classes, depending on where parallelism occurs in the graphics pipeline [3]: Sort-first: Each task is assigned a sub-section (tile) of the entire image to render. Then each task processes independently the graphics primitives that project into its tile until the final image is obtained. This approach is very classical on clusters. Data distribution can occur at the input layer only, all subsequent tasks working locally for their own tiles. Because this scheme only requires carrying lightweight data (input events) over the network, it is widely used for VR applications. However, an important part of the data and computation (mainly simulations and animations) are repeated on each task, thus limiting the benefits of using a cluster for multi-projector rendering and input-event capture parallelization. Henceforth we will call this approach the replication approach. The other classical scheme consists in having the application executed on a single node up to the generation of graphics primitives. Then primitives are distributed to different nodes, each one in charge of computing a tile. Though this helps avoiding most of the replications of the previous approach, performance is impaired by the cost of distributing the graphics primitives, whose amount is proportional to the complexity of the scene. For better scalability, load balance policies [21] are also important; otherwise the overall speed will be defined by the slowest rendering node. Sort-middle: Each task is assigned a set of graphics primitives that it processes up to rasterization. Then, data are sorted according to the tile they belong to before rasterization is performed. Because commodity graphics cards integrate both geometry processing and rasterization without giving users the ability to retrieve data before rasterization, this approach is seldom used on clusters. A sort-middle approach can be used on clusters when geometric transformations are performed on CPU [22]. Sort-last: The geometry dataset is split and sent to different tasks. In a final step, the images computed by each task are redistributed for compositing. Image compositing can be performed by dedicated hardware reading the images from the video output [23], or through software solutions reading back the images in the frame buffer of graphics cards and using the cluster network to move the data [24]. The complexity is then proportional to the image resolution rather than to the scene. It is also easier to achieve load-balancing, as there is no locality constraints on graphics primitives when they are initially distributed to the different tasks. This solution has been used mainly for scientific visualization, where datasets tend to be very large [25]. 3.5 Techniques There are many possible techniques to achieve the parallelism explained above. We are going to present some of the mostly used ones. Since there is no standard taxonomy for these techniques, names used here can be different in other publications. Sample division: Used for anti-aliasing. It is possible to sample different renderings and combine them in a final image. Since each node renders a sample, the anti-aliasing procedure is quite fast and does not decrease the graphical frame rate. Time division: In this method, the frame rate is split amongst the nodes, and then each node in the cluster sends the following animation frame in a sequence, increasing the frame rate if more computer nodes are connected. Some predictions about future frames are necessary, which creates some frame delay; however, as the increase in frame rate is linear, if two nodes are used instead of one, the frame rate doubles. Image division: In this approach each node renders a screen region. o Static partitioning: In split mode each node renders a screen region, the final image being the side-by-side composition of each image rendered by each node. o Interleaved: In this approach each node renders the complete frustum, but in a resolution squeezed horizontally or vertically with a subpixel shift. The final image is the composition of all images onto a high-resolution one. o Dynamic partitioning: In this mode the image is drawn only in an area of the entire frame buffer; the remaining area is not rendered. Then the final image is the composition of one image over the others. It is possible to change the area on the flight, achieving a better load balance. Eye division: This mode is interesting for active-stereo systems, since the left and the right eyes can be rendered by different nodes at a lower frequency and then be combined into an active-stereo signal. Scene division: The geometry is spread across the node, then the final images are combined using the Z value (depth buffer). This method enables distributing the geometry among the nodes in the cluster, and using depth information to combine the whole geometry into a final image. Volume division: Used for volume-rendering applications, this approach uses the volume depth information to compose the final image. It can significantly reduce texture memory usage in graphics cards. Operational Decomposition: Operational parts, such as view-frustum culling operations, are split in the cluster in order not to overload the master node processor. IV. CONSISTENCY AND LOCKING PC clusters were first used in VR to drive multi-projector environments (Fig. 2). The first issue is to ensure that the image streams displayed by the different projectors are coherent, even though they have been computed on different nodes: the images displayed with the different projectors should appear as a single high-resolution image. This image-lock constraint can be decomposed into three synchronization levels: gen-lock, frame-lock and data-lock, presented as follows.

6 The International Journal of Virtual Reality, 2008, 5(3): Gen-lock Fig. 2. Multi-projection display wall. Gen-lock ensures that all video signals generated by the cluster are compatible regarding the color and black level as well as synchronization, which is the most important factor in PC clusters. Synchronization can occur at a pixel, line or frame level. A frame-level gen-lock is mandatory for active stereo. If gen-lock is not properly ensured, the user may see with the same eye a right- and a left-eye image displayed by two different projectors, which would affect the stereo quality. Systems that do not use active stereo usually do not require gen-lock to obtain a good-quality image. 4.2 Frame-lock Frame-lock, also called swap-lock, ensures that the images computed on each PC node are released at the same time, i.e., the buffer swaps are synchronized. Failing to ensure a proper frame-lock results in discrepancies, where images that are displayed at the same time correspond to different rendering steps. 4.3 Data-lock The goal of data-lock is to guarantee that the data used to compute the images for the different projectors are coherent. For instance, all images related to the same time frame must be computed from 3D objects that are at the same position or have the same color. Data-lock is a complex issue that can be tackled at different levels of the application. 4.4 Implementations There are several possibilities to achieve image-lock. Some require specialized hardware, but usually it is possible to synchronize using only software techniques Hardware Commodity graphics cards usually do not support hardware gen-lock and frame-lock, although some high-end models do provide such features. NVidia's Quadro family [26] and 3DLabs [27] implement these synchronizations relying on an additional daughter board through a RJ-45 or DB9 chain connection. 3DLabs also allows maintaining a minimum swap period between frames, allowing the application as a whole to maintain an acceptable frame rate. Unfortunately, 3DLabs no longer supports this kind of graphics cards. ATI says that their FireGL V7350 [28] is gen-lock ready, but it depends on a daughter card that at the time of this writing is not yet available. Springer et al. [29] propose to modify commodity graphics cards to control the gen-lock. This is achieved by retrieving the gen-lock signal from a pin on the master node's board, disabling each graphics card's slave-node gen-lock and feeding them with the gen-lock signal from the master node Software A synchronization barrier, executed on a classical Fast-Ethernet network just before the buffer swap, is sufficient to ensure proper frame-lock [30] [31] [32] [33]. As gen-lock concerns video signal generation, it is more associated to hardware and therefore much more difficult to ensure with a software-only approach. However, some software approaches that do not require any hardware modification of the graphics card have been developed to implement a frame-level gen-lock and enable active stereo on PC clusters. Rather than gaining total control over signal generation, which would make the software deeply dependent on the graphics card specification, SoftGenLock [34] applies continuous small modifications to converge and maintain gen-locked video signals. The gen-locked signal is propagated to the different machines using the parallel port, a low-latency device present on almost all PCs. It results in software that only requires access to a few specific registers on a graphics card: it can be ported with minimal effort to potentially any graphics card. Another implementation, based on SoftGenLock's [34] core code, has been developed at HLRS [35]. It supports the Linux Kernel 2.6. WinSGL [36] is a software gen-locking approach for Microsoft Windows. It uses the parallel port as well, but requires an external signal generator to synchronize the video refreshments. It uses third-party software utilities to access the graphics card as well as DirectDraw routine calls. Although less precise than SoftGenLock, it is more flexible regarding the use of graphics cards. V. SOFTWARE TOOLS FOR CLUSTERS In this section we review different software tools for VR that support PC clusters. Most of them are open source. This list is not exhaustive, but its covers the most common and advanced uses of parallelism for VR applications. Each tool is positioned according to the parallelism it enables. A complementary classification can be found in [37]. 5.1 CaveLib CAVELib [38] was developed at the Electronic Visualization Lab to drive the first Cave [2]. Initial versions ran on a cluster of SGI machines, using a replication approach. Some fixed data,

7 6 The International Journal of Virtual Reality, 2008, 5(3):1-5 such as a navigation matrix and input device values, were shared throughout the cluster. Further data sharing could be implemented by transferring blocks of memory between nodes. Nowadays it supports PC clusters following a similar replication approach. 5.2 VR Juggler VR Juggler [39] is a software framework for developing portable VR applications. PC cluster support was first provided by Net Juggler Error! Reference source not found., a library based on MPI for message exchange. VR Juggler II relies on a client/server paradigm where inputs are executed on servers while clients take care of parallel rendering [40]. Both approaches use a replication approach. 5.3 Syzygy Syzygy is a software library dedicated to VR applications running on PC clusters [41]. Syzygy supports networked input devices and sound rendering. It includes two application frameworks, both based on a master/slave paradigm. The first one relies on a classical duplication paradigm, while the second proposes to distribute the data from the master at the scene-graph level (or animation level according to our classification). Syzygy provides a special protocol to transport the scene-graph primitives. The replication approach is normally used when a scene-graph approach is not appropriate, such as for volume rendering. The entire cluster configuration information is stored in a networked accessible database managed though a distributed operating system called Phleet. It enables a dynamic application reconfiguration to recover from a slave crash, for instance. 5.4 DIVERSE DIVERSE [42] is a modular collection of complementary software packages designed to facilitate the creation of device-independent virtual environments. DgiPf is the DIVERSE graphics interface for OpenGL Performer. A program using DgiPf can run on platforms ranging from fully immersive systems such as CAVEs to generic desktop workstations without modification. On clusters, DIVERSE relies on a replication paradigm. 5.5 Jinx Jinx [43] is a fully distributed virtual-environment browser, which has special support for commodity PC clusters and immersive-visualization devices. It is used to develop Virtual Reality applications based on the X3D format. It uses a replication approach to provide cluster support. 5.6 OpenSG OpenSG [44] [20] is a portable scene-graph system. It allows multiple asynchronous threads to independently manipulate the scene graph without interfering with one another. As scene-graph data can get very large, a distinction between structural and content data has been introduced, with a method to replicate the latter only if necessary. OpenSG also runs on PC clusters and is implemented as an extension of the multi-threaded model. Changes in the environment are propagated when they are applied to another node. OpenSG has a Multi Display Window mode, used to render one virtual window on a number of cluster servers, allowing the use of OpenSG in a CAVE configuration. OpenSG can also be used with a replication approach when combined with other tools such as VR Juggler. 5.7 Chromium Chromium [45], the successor of WireGL [46], proposes a stream processing framework for OpenGL graphics primitives. A network of Stream Processing Units (SPUs) enables the application of different transformations to the primitive stream. Chromium is mainly used for sort-first and sort-last parallel rendering. SPUs implement various optimizations to reduce the amount of data to be sent over the network. Chromium enables the execution of an unmodified OpenGL application by intercepting the graphics primitives and broadcasting them to rendering SPUs, each one in charge of its own image tile. Several optimizations [47] [48] were developed to increase the performance of Chromium. Also commercial solutions based on a similar approach are available today, such as TechViz [49] or Tungsten Graphics, [50] for instance. 5.8 OpenGL Multipipe SDK OpenGL Multipipe SDK [51] was developed by SGI and provides a uniform API to manage the access of scalable graphics applications across several graphics subsystems, also providing a customizable image-compositing interface, which facilitates the development of parallel OpenGL-based applications. It is able to choose and adapt the decomposition strategy for a given problem domain and graphics environment at run time. 5.9 Basho Hinkenjann et al. [52] implemented a solution that runs on AVANGO [53] and OpenGL/Performer [54] called Basho [55]. It pursues a retained-mode approach for the distribution of geometry data with a minimal network load. It consists of a front-end for scene-graph distribution and load balancing, and a back-end that performs rendering and compositing. The image combiner works in cascade. For load balance, it uses statistical information from the rendering nodes, adapting the balance as necessary. It also supports different kinds of renders, from the 3D engine of graphics cards to ray-tracing algorithms Covise Covise [56] relies on a data-flow model for coupling components that can be distributed to the different nodes of a cluster. It also supports collaborative work and immersive environments using a replication approach. It allows building advanced VR applications such as collaborative volume rendering [57] OpenMask OpenMask [58] [59] is a multi-threaded and distributed

8 The International Journal of Virtual Reality, 2008, 5(3):1-5 7 middleware library for animation and simulation. It enables the definition of tasks that are distributed on a cluster and communication through point-to-point messages and signals. Task update frequency is fixed and controlled by the OpenMask scheduler. Parallel rendering is provided by external tools like OpenSG Flowvr and Flowvrrender Flowvr [60] [61] is a middleware library dedicated to parallel VR applications. It relies on an extended data-flow model. An application is composed of modules exchanging data through a Flowvr network. From the Flowvr point of view, modules are not aware of the existence of other modules. A module only exchanges data with the Flowvr daemon that runs on the same node. The set of daemons running on the PC cluster are in charge of implementing the Flowvr network that connects modules. The daemons take care of moving data between modules using the most efficient method. Daemons can also load plug-ins for implementing advanced message-handling operations such as filtering, sampling, frustum culling, broadcasts, etc. This approach enables the development of a pool of modules that can next be combined in different applications, without having to recompile the modules. Modules can be multi-threaded or parallel applications relying on other parallelization tools like MPI [62], with Flowvr providing the necessary features to implement a parallel code coupling. Flowvrrender [63] [64] is a library built on top of Flowvr. It implements a sort-first retained-mode parallel rendering. Instead of relying on OpenGL commands, it defines a shader-based protocol using independent batches of polygons as primitives. Differently from OpenGL-based sort-first approaches like Chromium, Flowvrrender does not support unmodified OpenGL applications, but provides a more flexible and efficient framework for parallel rendering. Flowvr and Flowvrrender allow building complex distributed applications combining different parallelization codes at the input, simulation, animation and rendering levels [65] NAVER NAVER [66] is a framework focused on clusters of low-cost PCs. It is based on component nodes that take care of rendering, devices and applications. It uses a non-standard script language to describe the virtual environment. It also has a synchronization mechanism for the cluster that supports different devices such as wheelchair and speech recognition Cluster Display Middleware One common task in display walls is to display standard applications, like slide players and spreadsheets. This is not a simple task, since the cluster is based on different computers with their window managers XDMX XDMX (Distributed Multi-head X11) [67] is an X server that will run on a computer cluster as a single desktop. It is based on a front-end X server that controls multiple back-end X servers. XDMX takes care of all input events, such as mouse and keyboard, and output events, such as rendering a window. Although this approach is very convenient, it does not take full advantage of the parallelism available in each graphics card in the cluster, but some applications can be modified and when necessary use all the PC cluster resources SAGE SAGE (Scalable Adaptive Graphics Environment) [68] enables the visualization of multiple applications in high-resolution displays, with the capability of dynamically moving and resizing application windows freely, which is controlled across a UI client application that can run in a simple computer. However, to use SAGE it is necessary to modify the application that will run in the display wall in order to actually parallelize the application in the cluster nodes. VI. HARDWARE COMPOSITING DEVICES If rendering is the performance bottleneck and software solutions for parallel rendering do not always provide the expected results, it is still possible to take advantage of hardware compositing devices. These devices are not commodity components but most of them can be installed on commodity PC clusters. 6.1 Cluster Level Compositing A common approach consists in designing additional boards that take as input signal the video signal of standard graphics cards [69] [70]. The data are then combined in real time to provide as output one or several video signals. A daughter board can be installed on each PC to retrieve the video signals. Data are exchanged between boards through a dedicated network. Boards can be chained or use a switch like the Sepia boards, which rely on an Infiniband network. Another approach consists in having an external compositing device. 6.2 PC Level Compositing: SLI and CrossFire Back in 1996, Voodoo proposed to have two graphics cards installed in the same PC to share graphical processing for one display. Each card rendered half of the image's scan lines, which where interleaved when generating the video signal. Nowadays, NVidia [71] and ATI [72] developed similar solutions, but with better load-balancing algorithms and different operational modes. Both solutions support Split Frame Rendering (SFR), Alternative Frame Rendering (AFR) and Anti-aliasing. In SFR mode, one card computes the upper half of the image and the other the lower half. In AFR mode one card computes all odd frames, the other even ones. The Anti-aliasing mode splits the anti-aliasing workload between the two graphics cards. ATI also supports a Supertiling mode where rendering tiles are defined and distributed dynamically to the GPUs to ensure a better load balancing. This mode is only available for DirectX applications for the moment. ASUS launched a graphics card that uses two NVidia 7800 GT chips in one board [73]. This was achieved through

9 8 The International Journal of Virtual Reality, 2008, 5(3):1-5 PCI-Express x32 and using NVidia's SLI system. Right now Nvidia has a ready solution in the GeForce 7950GX2 graphics card. It uses PCIe x16, which is easily available in motherboards. But this kind of solution still requires applications specially developed to this architecture, otherwise it achieves small benefits. Standard GeForce graphics cards use PCI-Express x Special Compositing Hardware For large clusters, it may not be economically viable to add dedicated boards to each node in order to perform compositing tasks. To address this, several manufacturers have developed specialized processors which sit between the cluster nodes and the display devices. We survey several such solutions in this section ORAD ORAD has special hardware designed for parallel rendering in clusters called DVG [74]. This card composes images from different nodes, so some sort of parallel rendering without network limitations is possible. DVG basically connects the video signal of each node in a chain using standard graphics cards Lightning-2 Lightning-2 [23] was developed to fully implement sort-first and sort-last algorithms on PC clusters, enabling system scalability in computer nodes and displays. One good point of this solution is the fact that it does not require any modification to graphics cards or even to the computer nodes, but to the application. It has a bus that composes images in real time using the Z channel. This enables the visualization of geometries from the computer nodes to be unified in final high-resolution images with a high frame rate. The image chain is pipelined and constructed with point-to-point communication links, providing a theoretically unlimited scalability. The images are combined inside Lightning-2, and each pixel has 32 bits (24 bits for RGB and 8 used to control compositing operations). The image and depth information comes in the same DVI bus Sepia Sepia is another image-combining architecture for parallel rendering that supports interactive rates [69]. It implements the combining logic in a FPGA device that communicates with a network infrastructure. This system is able to use a standard network because it can handle network packages of any size with extremely low processing and transmissions latencies. It works using a chain topology. Thanks to the node-to-node flow control, even a system with high node count the data flow resumes with a latency of one node. Also it uses on-the-fly compression, reducing the required network bandwidth. It accesses the images in the graphics card memory through DMA, which limits the use of PCI graphics cards. It is also able to make good use of the versatile reconfigurable board PCI Pamette SGI InfinitePerformance SGI InfinitePerformance [70] is a scalable graphics compositor from SGI capable of receiving two or four DVI digital video inputs and combining them into a single video output. Dynamic load balancing can be achieved for graphics compositor tiling while an application is running. The compositor has a mode that uses one input as the right-eye data source and another input as the left-eye data source. The compositor output alternates between the two inputs on its video output and provides adequate signal for active stereo. 6.4 NVidia Quadro Plex Fig. 3. Quadro Plex in a Rack. 2 Quadro Plex [75] is a solution from Nvidia that allows up to 4 graphics cards to be connected into one single PCI-Express X16 slot of a host computer. If the host computer has two PCI-Express X16 slots, it is possible to even install two Quadro Plex. Quadro Plex consists of an external node that hosts the graphics cards. It is already planned for racks, making it very suitable for cluster configurations (Fig. 3). The connection between the host computer and the Quadro Plex is based on a VHDCI (Very High Density Cable Interconnect) connector. In a cluster configuration all video outputs can be gen-locked. The host computer can be any AMD Opteron, Intel Pentium or Xeon architecture, making Quadro Plex a very flexible solution. For the time being, the use of clusters is still necessary for larger systems that need several video inputs, but it is quite possible that in the future more graphics cards can be connected to a single host computer, which will create an easy user and development environment. Three-way SLI motherboards [76] are available for the game market, and probably in the future they will support professional graphics cards. VII. DISPLAY TECHNOLOGY The first immersive environments used large and expensive CRT projectors. While PC clusters were emerging as a relevant alternative to dedicated supercomputers, it also appeared that commodity projectors based on LCD or DLP technologies could be used too [77], and coupling them could lead to high-end solutions applied to different domains [78]. 2 Image Courtesy of NVIDIA, Inc. (c) Copyright 2007.

10 The International Journal of Virtual Reality, 2008, 5(3): Projectors Different projection technologies are available today. Usually they fit into four main categories: CRT (Cathode Ray Tube) projectors utilize three tubes, one for each color (RGB). The three colors are converged into a light beam that draws the image sequentially pixel by pixel. CRTs do not have a fixed number of pixels. As the number of pixels to be drawn increases, the refresh rate decreases. One problem regarding CRT projectors is that they require periodic calibrations. Today CRT technology tends to be abandoned in favor of other ones. LCD (Liquid Crystal Display) projectors contain three separate LCD glass panels, one for each color (RGB). As the light passes through the LCD panels, their opacity is controlled to modulate the amount of color pixels receive. LCD is one of the main technologies used for commodity projectors. They require little maintenance. DLP (Digital Light Processing) projectors are based on an optical semiconductor called DMD chip, which was invented in 1987 by Texas Instruments [79]. The DMD chip is an array of micro-mirrors (one per pixel) that tilt to control pixel brightness. Color is provided either by associating three DMD chips, one per color (RGB), or by using one DMD and a color wheel. Commodity DLP projectors use one DMD, while high-end ones use three DMDs. DLP projectors require little maintenance. LCoS (Liquid Crystal on Silicon) projectors rely on a new technology that uses LCD filters with a mirror under them. This technology is used today for high-end projectors such as JVC D-ILA and Sony's SRX projector, [80] which achieves a resolution of 4096 x 2160 pixels, and the most recent Barco LX-5 [81], with 4096x2400 pixels. Digital projection is therefore the technology that drives modern display walls, since the analogical technology of CRT is no longer used. Digital projectors are usually very bright, but they have issues like screen-door effects and rainbow effects, and lamp replacements can be rather expensive. LCoS technology has been increasing in popularity because it can produce much higher-resolution images for lower prices. These projectors need up to four high-resolution video signals, and PC clusters can be used to feed one single projector. Different technologies can also be used for display, such as PDP (Plasma Display Panel), LED (Light Emitting Diode) or even VFC (Vacuum Fluorescent Display). Several specialized groups are working on display walls composed of LCD monitors, which are disassembled and positioned side by side. LambdaVision [82] developed by EVL, for instance, is based on a 11x5 tiled LCD display wall that achieves a resolution of 100 megapixels. This technique cannot avoid the borders between the monitors, but it can be cheaper, does not require room in the back for the projection, and a user in front of the screen will not create a shadow. 7.2 Stereo Stereo is a fundamental feature to provide a sense of immersion. Usually stereo technologies can be classified into three categories: Active stereo. Left- and right-eye images are displayed alternatively. Swap occurs during vertical blanking, i.e. when no pixel is generated before the next video retrace starts. The user wears glasses with shutters opening alternatively to let him see the right-eye images with the right eye only and vice-versa. Gen-lock is mandatory to ensure that all projectors display the left- or right-eye image synchronously. The video refresh rate must be high enough (usually about 120 Hz) to ensure that the user does not perceive flickering. Active stereo is available with specific CRT and three-chip DLP projectors. Single-chip DLP projectors supporting active stereo may be soon available [83]. In all cases, quality active stereo is only supported by projectors that have been adapted for that use. Passive stereo. Left- and right-eye images are displayed concurrently by two different projectors. Passive stereo can be obtained with almost any projector. Frame-lock is usually sufficient. No specific hardware is required on the cluster. Although there are several technologies for passive stereo, like Anaglyph, that separate the left and right eye with a red and a blue or green color filter, there are two main different approaches used to separate the images for each eye in high-quality Virtual Reality applications: o Linear and circular polarization. The left- and right-eye images are polarized differently, and the user wears glasses with polarizing filters to block the image of the opposite eye. Light can be linearly or circularly polarized. Filters for linear polarization are cheaper than the ones for circular polarization, but image separation is lost when the user tilts his head. Both techniques require special screens that maintain the polarization, usually high-gain screens. o Infitec. The spectral colors are split into six ranges, two for each primary color. Each eye is associated to one range for each color. Specific filters on the projectors and glasses are used to separate right- and left-eye images. Infitec does not impose any constrain on screen materials, and allows users to freely tilt their heads (Fig. 4). However, a color shift occurs that requires correction using hardware or software techniques. Fig. 4. Infitec glasses. 3 Auto-stereo technologies are becoming more popular since they do not require users to wear glasses. Today no projection technology supports auto-stereo. Despite the attempts to build display walls with auto-stereo displays [84], they are not yet used in immersive environments. 3 Image Courtesy of Infitec.

11 10 The International Journal of Virtual Reality, 2008, 5(3): Image Calibration When using a large number of projectors, the challenge is to ensure a high-quality image calibration, i.e. a correct geometric projector alignment as well as photometric uniformity. Fig. 5 presents a display wall calibrated automatically. Due to the screen's high gain it is still possible to see the boundaries between projectors, although edge-blending software compensation creates smooth transitions between screen tiles. Fig. 5. Automatic geometric alignment in a 16-channel display wall. When projections overlap, it is necessary to inform the application or the graphics card that there is a region in the original image that must be replicated among the video outputs for a perfect matching between the projections. Recent graphics card drives support this feature, enabling the video output images of a single node to be readjusted by means of edge blending and image overlapping to fit in the projection region [85]. Promising approaches propose automatic calibration techniques. These approaches use a camera to take pictures of predefined patterns. From these pictures, information on corrections to be applied to the images displayed by each projector is extracted [86] [87] [88] [89]. Corrections are then applied by graphics cards at each projector. Though quality results can be obtained for geometric calibration, photometric uniformity is more difficult to obtain, partly because camera captors are not very precise in measuring color discrepancies. High dynamic range techniques can significantly improve the photometric calibration quality [90]. An alternative consists in computing color calibration from values obtained by a spectroradiometer, but this is a very expensive device. VIII. HARDWARE FOR PC CLUSTERS AND VR Many aspects regarding cluster hardware are important in a PC cluster for Virtual Reality application. Here we are going to cover the most relevant ones. 8.1 Network The network system is very important, and often it is a bottleneck of the cluster application. Depending on the kind of application, decisions regarding the network can change. There are several possibilities for networking connections. The most efficient, such as Myrinet connections, used for instance at the Brown University 4-side Cave [91], have compatibility and price drawbacks. On the other hand, more commodity systems such as Gigabit Ethernet do not always fulfill all requirements easily [52]. Also some architectures use special networking systems, like ServerNet Colorado, used by Sepia [69]. 8.2 I/O Since Virtual Reality requires devices to interact with the virtual environment, it is important to connect the I/O device to the cluster. The most common interaction abstraction is tracking. Many different approaches are used; usually the application synchronizes the input devices amongst the cluster [92] [43]. DIVERSE [42] creates IP network-interaction devices, enabling the creation of device-independent applications. Syzygy, [41] otherwise, has a communication layer with integrated administration. 8.3 Sound Beside the images, sound, audio and multimedia are very important in VR solutions [93]. The adapters can be connected to the cluster in many ways to achieve sound spatialization in the VR simulation. The most common approaches are using tracked headphones for each user or distribution sound speakers around the simulated environment for sound immersion. Some VR software already have their own internal sound engine, such as Syzygy or VRJuggle. TRIPS [94] is a solution developed for OpenSG that uses a different computer node to compute sound; this node runs Microsoft Windows, which the authors claim to have better driver support. IX. CONCLUSION PC clusters are one of the solutions of choice to provide both the computing and I/O power required by advanced VR applications and facilities. These can range from single CPU PCs using video-game graphics cards connected through a standard network to high-end multi-cpu PCs equipped with professional graphics cards and connected via a dedicated high-performance network. Among other challenges, software tools have to be adapted and developed for these architectures. The main difficulty lies in developing software solutions that enable taking advantage of the performance offered by clusters while keeping the complexity of application development, deployment and execution as low as possible. Today such solutions are available for distributed rendering while others are emerging to provide extra computing capabilities for processing input data from sensor networks, or handling multi-modal applications involving 3D graphics, spatialized sound, haptics systems, multiple simulations, etc. The size of clusters used for VR is still moderate (a few dozen nodes), whereas much larger configurations have been developed for other problem domains, such as simulation in

12 The International Journal of Virtual Reality, 2008, 5(3): particle physics and astronomy. In the future, we may see more applications using such larger machines. Then, issues regarding algorithm parallelization, algorithm coupling and software engineering will have to be addressed to handle the increased complexity of new applications. Even though developers can take advantage from the many developments arising out of high-performance or distributed computing, VR applications tend to be more heterogeneous than classical high-performance applications coupling different codes for sound, graphics, haptics, physical-based simulations, etc. and with stronger constraints on performance latencies and update frequencies. Beside clusters, processor-level parallelism will also become more present as new architectures emerge. It will be common to have single-box machines with multiple CPUs, GPUs and FPGAs. For instance, today it is possible to have a PC with eight GPUs and eight dual-core CPUs. The Cell processor [ref missing!] including eight cores is also available. Intel and AMD have already released four-core CPUs. And looking further to the future Intel has a project for 80 cores in one chip [95]. The extra computing power available could greatly influence the development of the next generation of VR applications. REFERENCES [1] Top 500 supercomputers. Available: [2] C. Cruz-Neira, D. J. Sandin, T. A. DeFanti, R. V. Kenyon, and J. C.Hart. The Cave Audio Visual Experience Automatic Virtual Environment. Communication of the ACM, 35(6):64 72, [3] S. Molnar, M. Cox, D. Ellsworth, and H. Fuchs. A Sorting Classification of Parallel Rendering. IEEE Computer Graphics and Applications, 14(4):23 32, July [4] S. C. Eric Penner, Ryan Schmidt. A GPU cluster without the clutter: A drop-in scalable programmable-pipeline with several GPUs and only one PC, [5] Cuda. Available: [6] M. J. Flynn, Very high-speed computing systems, Proc. IEEE, vol. 54, pp , Dec [7] Tesla. Available: [8] Rocks cluster distribution. [9] M. L. Massie, B. N. Chun, and D. E. Culler. The ganglia distributed monitoring system: Design, implementation, and experience. In Parallel Computing, volume 30, July [10] Kerrighed. [11] M. Saito. Console management in the data center. Technical report, Cyclades, [12] J. S. Montrym, D. R. Baum, D. L. Dignam, and C. J. Migdal. Realityengine graphics. In SIGGRAPH 93: Proceedings of the 20 th annual conference on Computer graphics and interactive techniques, pages , [13] J. S. Montrym, D. R. Baum, D. L. Dignam, and C. J. Migdal. Infinitereality: a real-time graphics system. In SIGGRAPH 97: Proceedings of the 24th annual conference on Computer graphics and interactive techniques, pages , New York, NY, USA, ACM Press/Addison-Wesley Publishing Co. [14] Origin Servers. Technical report, Silicon Graphics, April [15] ] J. Eyles, S. Molnar, and J. Poulton. PixelFlow: High-Speed Rendering Using Image Composition. In Proceedings of ACM SIGGRAPH 92, pages , Chicago, USA, July [16] VR Juggler. Available: [17] S. Wrmlin, E. Lamboray, O. Staadt, and M. Gross. 3D Video Recorder: A System for Recording and Playing Free-Viewpoint Video. Computer Graphics Forum, 22(2): , [18] J. Allard, E. Boyer, J.-S. Franco, C. Ménier, and B. Raffin. Markerless Real Time 3D Modeling for Virtual Reality. In Proceedings of the Immersive Projection Technology Workshop, Ames, Iowa, May [19] J. Allard, V. Gouranton, E. Melin, and B. Raffin. Parallelizing Prerendering Computations on a Net Juggler PC Cluster. In Immersive Projection Technology Symposium, Orlando, USA, March [20] M. Roth, G. Voss, and D. Reiners. Multi-threading and clustering for scene graph systems. Computers & Graphics, 28(1):63 66, [21] M. Roth, P. Riess, and D. Reiners. Load balancing on cluster-based multi projector display systems. In Proceedings of the International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision (WSCG), Plzen, Czech Republic, [22] J. L. Williams and R. E. Hiromoto. Sort-middle multi-projector immediate-mode rendering in chromium. In IEEE Visualization Conference, Minneapolis, USA, October [23] G. Stoll, M. Eldridge, D. Patterson, A. Webb, S. Berman, R. Levy, C. Caywood, M. Taveira, S. Hunt, and P. Hanrahan. Lightning-2: A High-Performance Display Subsystem for PC Clusters. In Proceedings of ACM SIGGRAPH 01, 2001 [24] X. Cavin, C. Mion, and A. Filbois. Cots cluster-based sort-last rendering: Performance evaluation and pipelined implementation. In IEEE Visualization Conference, Minneapolis, USA, October [25] K. Liang, P. Monger, and H. Couchman. Interactive Parallel Visualization of Large Particle Datasets. In Proceedings of the Eurographics Parallel Graphics and Visualization Symposium, pages 23 30, Grenoble, France, June [26] Nvidia quadro fx 3000g solutions for advanced visualization. Technical brief, NVIDIA Corporation, [27] New wildcat realizm graphics technology. White paper, 3DLabs, [28] Ati firegl v [29] J. Springer, A. Simon, and J. Plate. Supporting commodity clusters using avango s generic scene graph distribution. In VR-Cluster 03-Workshop on Commodity Clusters for Virtual Reality, IEEE Virtual Reality Conference 2003, Los Angeles, March [30] J. Allard, V. Gouranton, L. Lecointre, E. Melin, and B. Raffin. Net Juggler: Running VR Juggler with Multiple Displays on a Commodity Component Cluster. In IEEE VR, pages , Orlando, USA, March [31] M. Bues, R. Blach, S. Stegmaier, U. Häfner, H. Hoffmann, and F. Haselberger. Towards a Scalable High Performance Application Platform for Immersive Virtual Environments. In Immersive Projection Technology and Virtual Environments 2001, pages , Stuttgart, Germany, May Springer. [32] B. Schaeffer. Networking Management Frameworks for Cluster-Based Graphics [33] M. P. Guimarães, P. A. Bressan, and M. K. Zuffo. Frame lock synchronization for multiprojection immersive environments based on pc graphics clusters. In Proceedings of the 5th SBC Symposium on Virtual Reality, [34] J. Allard, V. Gouranton, G. Lamarque, E. Melin, and B. Raffin. Softgenlock: Active Stereo and Genlock for PC Cluster. In Proceedings of the Joint IPT/EGVE 03 Workshop, Zurich, Switzerland, May [35] U. Wösner and M. Aumller. Software-based genlock for active stereo nvidia cards. [36] M. Waschbüsch, D. Cotting, M. Duller, and M. Gross. Winsgl: Software genlocking for cost-effective display synchronization under Microsoft windows. In Proceedings of the 6th Eurographics Symposium on Parallel Graphics and Visualization, Braga, Portugal, [37] A. Streit, R. Christie, and A. Boud. Understanding next-generation vr: classifying commodity clusters for immersive virtual reality. In GRAPHITE 04: Proceedings of the 2nd international conference on Computer graphics and interactive techniques in Australasia and South East Asia, pages , New York, NY, USA, ACM Press. [38] D. Pape, C. Cruz-Neira, and M. Czernuszenko. CAVE User s Guide. Electronic Visualization Laboratory, University of Illinois at Chicago, [39] A. Bierbaum, C. Just, P. Hartling, K. Meinert, A. Baker, and C. Cruz-Neira. VR Juggler: A Virtual Platform for Virtual Reality Application Development. In IEEE VR 2001, Yokohama, Japan, March [40] A. Bierbaum, A. Bierbaum, and C. Cruz-Neira. Clusterjuggler: A modular architecture for immersive clustering. In

13 12 The International Journal of Virtual Reality, 2008, 5(3):1-5 VR-Cluster 03-Workshop on Commodity Clusters for Virtual Reality, IEEE Virtual Reality Conference 2003, Los Angeles, March [41] B. Schaeffer and C. Goudeseune. Syzygy: Native PC Cluster VR. In IEEE VR Conference, [42] J. Kelso, L. E. Arsenault, S. G. Satterfield, and R. D. Kriz. Diverse: A framework for building extensible and reconfigurable device independent virtual environments. In Proceedings of IEEE Virtual Reality 2002 Conference, [43] L. P. Soares and M. K. Zuffo. Jinx: an x3d browser for vr immersive simulation based on clusters of commodity computers. In Proceedings of the ninth international conference on 3D Web technology, Monterey, California, USA, April [44] D. Reiners. Opensg: A scene graph system for flexible and efficient realtime rendering for virtual and augmented reality applications, [45] G. Humphreys, M. Houston, R. Ng, S. Ahern, R. Frank, P. Kirchner, and J. T. Klosowski. Chromium: A Stream Processing Framework for Interactive Graphics on Clusters of Workstations. In Proceedings of ACM SIGGRAPH 02, pages , [46] G. Humphreys, M. Eldridge, I. Buck, G. Stoll, M. Everett, and P. Hanrahan. WireGL: A Scalable Graphics System for Clusters. In Proceedings of ACM SIGGRAPH 2001, [47] M. Lorenz and G. Brunnett. Optimized Visualization for Tiled Displays. Eurographics/ACM SIGGRAPH Symposium Proceedings, Parallel Graphics and Visualization 2006, pages , [48] J. Binns, G. Gill, M. Hereld, D. Jones, I. Judson, T. Leggett, A. Majumder, M. McCroy, M. Papka, and R. Stevens. Applying geometry and color correction to tiled display walls, [49] TechViz. Available: [50] Tungsten graphics. Available: [51] P. Bhaniramka, P. C. D. Robert, and S. Eilemann. Opengl multipipe sdk: A toolkit for scalable parallel rendering. In Proceedings of IEEE Visualization IEEE, October Proceedings of IEEE Visualization [52] A. Hinkenjann, M. Bues, S. Schupp, and T. Olry. Mixed-mode parallel real-time rendering on commodity hardware. In Proceedings of 5th Symposium on Virtual Reality, Fortaleza, Brazil, [53] H. Tramberend. Avocado: A distributed virtual reality framework. In Proceedings of the IEEE Virtual Reality, page 14, [54] J. Rohlf and J. Helman. Iris performer: a high performance multiprocessing toolkit for real-time 3d graphics. In Proceedings of the 21st annual conference on Computer graphics and interactive techniques, pages , [55] F. Mannuss and A. Hinkenjann. Towards better quality in virtual environments. In Proceedings of Eurographics Workshop on Virtual Environments, [56] A.Wierse, U.Lang, and R. Rhle. Architectures of Distributed Visualization Systems and their Enhancements. In Eurographics Workshop on Visualization in Scientific Computing, Abingdon, [57] U. Wösner, J. Schulze, S. Walz, and U. Lang. Evaluation of a Collaborative Volume Rendering Application in a Distributed Virtual Environment. In Eighth Eurographics Workshop on Virtual Environments, [58] D. Margery, B. Arnaldi, A. Chauffaut, S. Donikian, and T. Duval. Openmask: {Multi-Threaded Modular} animation and simulation {Kernel Kit }: a general introduction. In Simon Richir, Paul Richard, and Bernard Taravel, editors, VRIC 2002 Proceedings, pages ISTIA Innovation, June [59] OpenMASK. Available: [60] FlowVR. Available: [61] J. Allard, V. Gouranton, L. Lecointre, S. Limet, E. Melin, B. Raffin, and S. Robert. FlowVR: a Middleware for Large Scale Virtual Reality Applications. In Proceedings of Euro-par 2004, Pisa, Italia, August [62] W. Gropp, E. Lusk, and A. Skjellum. Using MPI: Portable Parallel Programming with the Message-Passing Interface. Scientific and Engineering Computation Series. The MIT Press, [63] J. Allard and B. Raffin. A shader-based parallel rendering framework. In IEEE Visualization Conference, Minneapolis, USA, October [64] J. Allard and B. Raffin. Distributed physical based simulations for large vr applications. In IEEE Virtual Reality Conference, USA, March [65] J. Allard, C. Méier, E. Boyer, and B. Raffin. Running large vr applications on a pc cluster: the flowvr experience. In Proceedings of EGVE/IPT 05, Denmark, October [66] C. Park, H. Ko, and T. Kim. Naver: a pc-cluster-based vr system. In Z. Pan and J. Shi, editors, Third International Conference on Virtual Reality and Its Application in Industry. Edited by Pan, Zhigeng; Shi, Jiaoying. Proceedings of the SPIE, Volume 4756, pp (2003), pages , April [67] K. E. Martin, D. H. Dawes, and R. E. Faith, Available: [68] B. Jeong, L. Renambot, R. Jagodic, R. Singh, J. Aguilera, A. Johnson, and J. Leigh. High-performance dynamic graphics streaming for scalable adaptive graphics environment. In SC 06: Proceedings of the 2006 ACM/IEEE conference on Supercomputing, Tampa, FL, USA, IEEE Computer Society. [69] L. Moll, M. Shand, and A. Heirich. Sepia: Scalable 3d compositing using pci pamette. In Proceeding of the Seventh Annual IEEE Symposium on Field-Programmable Custom Computing Machines, pg. 146, April [70] F. Razo. SGI InfinitePerformance: Scalable Graphics Compositor Owner s Guide, [71] S. Nvidia. Available: [72] A. Crossfire. Available: [73] T. Kreiss. Two s company, four s a wow! sneak preview of nvidia quad gpu sli. Toms Hardware, December Available: [74] G. Otrage. Orad s dvg: Solutions for scalable graphics clusters, Aug Graphics Hardware Conference. [75] Quadro plex. Available: [76] P. Miller. NVIDIA 3-Way SLI review roundup Available: [77] K. Li, H. Chen, Y. Chen, D.W. Clark, P. Cook, S. Damianakis, G. Essl, A. Finkelstein, T. Funkhouser, A. Klein, Z. Liu, E. Praun, R. Samanta, B. Shedd, J. P. Singh, G. Tzanetakis, and J. Zheng. Early Experiences and Challenges in Building and Using A Scalable Display Wall System. IEEE Computer Graphics and Application, 20(4): , [78] P. Yin, X. Jiang, J. Shi and R. Zhou, Multi-screen Tiled Displayed, Parallel Rendering System for a Large Terrain Dataset, In International Journal of Virtual Reality, Vol. 5, Num. 4, pp.47-54, December [79] M. R. Douglass. Dmd reliability: A mems success story. In Proceedings of SPIE, volume 4980, pages 1 11, San Jose, California, [80] SR Projector, Operating Instructions, [81] Barco LX-5. 4,500 lumens LCoS 10 megapixel projector Available: ment=4068 [82] B. Jeong, R. Jagodic, L. Renambot, R. Singh, A. Johnson, and J. Leigh. Scalable graphics architecture for high-resolution displays. In Proceedings of IEEE Information Visualization Workshop, Minneapolis, MN, USA, October [83] A. Hopp. Using a single spatial light modulator for stereo images of high color quality and resolution. In Proceedings of EGVE/IPT 05, Denmark, October [84] D. J. Sandin, T. Margolis, J. Ge, J. Girado, T. Peterka, and T. A. De-Fanti. The varriertm autostereoscopic virtual reality display. ACM Trans. Graph., 24(3): , [85] Nvidia powerwall. [86] R. Raskar and J. van Baar. Low cost projector mosaic. In Proceedings of the Asian Conference on Computer Vision 2002, Jan [87] A. Majumder, Z. He, H. Towles, and G. Welch. Achieving color uniformity across multi-projector displays. In Proceedings of the 11 th IEEE Visualization, page 17, October [88] A. Majumder and R. Stevens. Color Non-Uniformity in Projection Based Displays : Analysis and Solutions. Trasactions of Visualization and Computer Graphics, [89] C. Jaynes, B. Seales, K. Calvert, Z. Fei, and J. Griffioen. The Metaverse - A Collection of Inexpensive, Self-configuring, Immersive Environments. In 7th International Workshop on Immersive Projection Technology/Eurographics Workshop on virtual Environments, Zurich, Switzerland, May [90] A. Majumder and R. Stevens. Perceptual photometric seamlessness in projection-based tiled displays. ACM Transactions on Graphics. January Volume 24, Issue1. Pages [91] D. Lemmerman and A. Forsberg. Adapting event-based applications for synchronization in VR clusters. In Proceedings of the 8th International Immersive Projection Technology Workshop, Iowa State University, Ames, IA, May 2004.

14 The International Journal of Virtual Reality, 2008, 5(3): [92] J. Allard, B. Raffin, and F. Zara. Coupling Parallel Simulation and Multi-display Visualization on a PC Cluster. In Euro-par 2003, Klagenfurt, Austria, August [93] R. R. A. Faria, L. Thomaz, L. Soares, B. Santos, M. Zuffo, and J. A. Zuffo. Audience - audio immersion experience in the caverna digital. In Proceedings of the 10th Brazilian Symposium on Computer Music, Belo Horizonte, Minas Gerais, Brazil, October [94] C. F. Tomas Neumann and D. Fellner. Trips a scalable spatial sound library for opensg. In D. Reiners, editor, Proceedings OpenSG Symposium 2003, pages 57 63, Darmstadt, Germany, Apr [95] T. Krazit. Intel pledges 80 cores in five years. CNet News.com, September Luciano Pereira Soares holds PhD in Electrical Engineering from Polytechnic School, University of São Paulo, Brazil. He was a postdoctoral research at IST, Instituto Superior Técnico in Portugal, INRIA, Institut National de Recherche en Informatique et en Automatique in France and ISCTE, Instituto Superior de Ciências do Trabalho e da Empresa in Portugal. His research interests include real-time 3D computer graphics and cluster computing. Specifically, he is studying techniques to drive immersive virtual environments using distributed systems. He is currently researcher at the Tecgraf, Computer Graphics Technology Group in PUC, Pontifical Catholic University of Rio de Janeiro working in several projects at Petrobras, Petróleo Brasileiro. He worked as support engineer at Silicon Graphics, application engineer at Alias Wavefront and as project manager at the Integrated Systems Laboratory. Bruno Raffin is research scientist at INRIA Rhône-Alpes Grenoble. He received a Ph.D. on parallel computing from the Université d Orléans in After a 2 year postdoc in USA working on large parallel computers, he returned to France to work on solutions to run multi-display virtual environments with PC clusters equipped with commodity graphics cards. This work led to the Softgenlock and Net Juggler Software libraries. Today his research activity focuses on high performance interactive computing, virtual reality and new interaction techniques. He develops the FlowVR software suite and manages the Grimage experimental platform gathering multiple PCs, projectors and cameras. He has co-chaired the 2004 and 2006 Eurographics Symposium on Parallel Graphics and Visualization and participated to the program committee of several international conferences. Joaquim A. Jorge holds PhD and MSc degrees in Computer Science from Rensselaer Polytechnic Institute, Troy New York, awarded in 1995 and 1992 respectively, and a BSEE degree from Instituto Superior Técnico at Universidade Técnica de Lisboa, Portugal in His research interests include calligraphic and multimodal user interfaces and visual environments. He is currently Associate Professor of Computer Graphics and Multimedia at the Department of Information Systems and Computer Engineering at the Instituto Superior Técnico. He has co-authored over 110 internationally refereed papers on Computer Graphics and User Interfaces. Prof. Jorge is a member of the Eurographics Association and is also affiliated with ACM (SM 07), SIGGRAPH and SIGCHI, as well as National Representative to IFIP TC13. He has served on the International Program Committees for over 100 international conferences and has organized or helped organize over 25 international scientific events. He is Editor-in-Chief of Computers & Graphics Journals and serves on the Editorial Boards of five other scientific publications, including Computer Graphics Forum.