Magic View: An Optimized Ultra-large Scientific Image Viewer for SAGE Tiled-display Environment

Transcription

1 2013 IEEE 9th International Conference on e-science Magic View: An Optimized Ultra-large Scientific Image Viewer for SAGE Tiled-display Environment Yihua Lou, Haikuo Zhang, Wenjun Wu, Zhenghui Hu State Key Laboratory of Software Development Environment Beihang University Beijing, China {louyh, wwj, zhk, Abstract Massive amount scientific data often need to be visualized in ultra-large images for scientific discovery. Although ultra-high resolution tiled-display environments have been widely used, there still lacks of proper image viewers that can display ultra-large images with billions of pixels in tileddisplay environments. To address the problem, we propose Magic View, an optimized ultra-large scientific image viewer for SAGE tiled-display environment. It can achieve real-time interactive performance in viewing images with billions of pixels. Our experiments show that the performance of Magic View are at lease 8x better than Juxta View, another ultralarge image viewer for SAGE. Keywords-Ultra-large image; SAGE; Tiled-display; Image viewer; Magic View I. INTRODUCTION Nowadays, massive amount scientific data that need to be analyzed, processed and visualized are increasingly generated by advanced scientific simulations and experiments. In order to make valuable scientific discovery from these data, they should be visualized in ultra-large size images with billions of pixels. Such visualization cannot be adapted in a typical desktop display, and ultra-high resolution tiled-display environments are required. Such environments can demonstrate much richer information than a regular desktop display environment and have been widely used for collaborative scientific research and education over recent years. With a tiled display system, it is possible for researchers to easily view and manipulate juxtaposed ultralarge scientific images at the same time. For example, scientists in the satellite ground station of Beihang University receives tens of gigabytes of image data every day from several satellites, including the earth observation satellites such as TERRA and AUQA, the NOAA Polar Earth Orbiting weather satellites such as NOAA 17/18/19, and the China s Feng Yun series of polarorbital weather satellites such as FY-1D and FY-3A. The data are then analyzed and processed by a crowd-sourcing data processing system proposed in [1]. Although a SAGEenabled tile-display wall with about 30M pixels is available for displaying ultra-large images, we found that Juxta View the imagery application from SAGE project can t smoothly handle the processed billion-pixel-level images with the support of fast pan, fast zoom, and easily switching the currently viewing image. In this paper, we will propose a new software tool called Magic View to address the requirements. Magic View is fully optimized for displaying ultra-large scientific images with billions of pixels and supporting key features such as fast pan, fast zoom and dynamic image switch. The rest of this paper is organized as follows: Section 2 gives a brief introduction of the SAGE system and Magic Input Framework. We will review some related works in section 3. In section 4 and section 5, we describe the architecture, core algorithms and important optimization mechanisms of Magic View. Section 6 presents experimental results of Magic View and compares its performance with Juxta View. And section 7 gives the conclusion. II. SYSTEM BACKGROUND A. SAGE Overview SAGE (Scalable Adaptive Graphics Environment), a middleware developed by EVL-UIC, is used to support stream pixels displays driven by a computer cluster. Through SAGE, these displays are treated as a single virtual desktop, regardless of their configuration and combination. Remote applications can stream their visual contents to the windows on the virtual desktop. As described in [2], SAGE includes four major components: Free Space Manager, SAGE Application Interface Library (SAIL), SAGE Receiver and SAGE User Interface (SAGE UI). Free Space Manger is the central controller of SAGE, which connects with other SAGE components via TCP sockets. It defines a set of window manipulation commands that can be used by other SAGE components. SAIL is an API library for applications and through the APIs exposed by SAIL, an application (referred as SAGE-enabled application later in this paper) can communicate with the Free Space Manager to create windows in the SAGE virtual desktop and stream pixels to SAGE Receivers that renders these pixels in the windows. SAGE UI provides a simple GUI for end users to manipulate SAGE windows, launch applications, and start/stop SAGE /13 $ IEEE DOI /eScience

2 B. Magic Input Framework Magic Input is a multi-user interaction framework proposed in [3], which provide a unified input interface to interact with the SAGE-enabled applications through remote input devices including keyboard, mouse and somatosensory devices like Wii Remote and Kinect. By supporting the somatosensory devices, users can use touch-free gestures as inputs as well as traditional key-based inputs in Magic Input. Magic Input consists of the following three major parts: Input Manager, Input Client and Application Layer. Input Manager is Magic Input s central controller, which receives input events from Input Clients, processes them according to pre-configured strategies, and then forwards them to appropriate SAGE-enabled applications. Input Client is responsible for collecting events generated by actual input devices, converting them to input events, sending input events to the Input Manager, and receiving feedbacks from Input Manager. Application Layer is an I/O substrate for SAGE-enabled applications, which receives input events forwarded by Input Manager and these input events are delivered to the application as if they were generated by a locally attached input device. III. RELATED WORKS There are many desktop image viewers such as ACDSee, Adobe Photoshop and Gimp, which support various image formats and offers general image manipulation functions like pan and zoom. However, these applications are mainly designed and optimized for handling small images with millions or fewer of pixels, and their abilities of handling extremely large images beyond billions of pixels are highly restricted by the available memory in a single computer. A cluster based image viewer for tiled-displays is TimV [4], which is developed at San Diego Supercomputing Center based on OpenGL, MPI, and ImageMagick libraries. However, as the ImageMagick library itself is not optimized for processing ultra-large images, TimV cannot handle ultralarge images well. Argonne National Labs also developed a cluster based image viewer [5] that can display large images through a projector array and support basic image viewing functions such as pan or zoom. However, it lacks of some important features such as selecting an arbitrary point for zooming and dynamically switching the currently viewing image. Juxta View [6], another cluster-based image viewer initially developed to view images through LambdaRAM, is then modified to become a SAGE-enabled application. The core idea of Juxta View is pre-splitting an ultra-large image into small image tiles, and only the image tiles that will be displayed are loaded and processed. However, despite of using the LibTIFF[7] library for reading pixels from TIFF images, Juxta View does not utilize any enhanced TIFF features such as multiple images per file and tiled images. And it only supports the striped format TIFF files, which makes it not so efficient. In addition, it relies on a statically configuration file to specify the paths of the pre-split image tiles, thereby having no feature for switching the currently viewing image on-the-fly. Figure 1. System architecture Magic Carpet [8] implements enhancement in Juxta View by supporting image pyramid method to accelerate the viewing process instead of doing zoom operations at runtime for some specified resolution levels. However, Magic Carpet always requires a complex pre-process procedure to create the pyramid of an image in order to support zoom operations, and it adopts a file-system structure similar to that used by Juxta View for keeping the pre-processed results, which may also leads to a higher latency. Besides, its preprocessed results can only be stored in DXT1 format, DXT3 format or uncompressed format. As the former two formats are lossy formats of low and fixed compression ratios, it cannot achieve the best space-quality balance in many real cases. Recently, a new tiled-display system named by Display- Cluster [9] is proposed and provides a new imagery tool for viewing ultra-large pictures. DisplayCluster improves Magic Carpet by supporting both image pyramid method and runtime zoom operations. However, it also uses the less-efficient method of storing pre-processed result in groups of image tiles. IV. MAGIC VIEW We proposed a new solution, the Magic View, to provide flexible and optimized ultra-large images viewing functionality. A. Overview Magic View is designed to provide a full-screen, fast viewing experience for ultra-large images through SAGE. As such images usually have the size with more than billions of pixels and more than gigabytes, it is difficult to process them 263

3 Figure 2. The standalone GUI version Figure 3. The Magic Input integrated version (The red arrow pointer inside the red circle on the screen is provided by the Magic Input framework) in a single machine. Therefore, Magic View is designed as a MPI-based parallel application consisting of one controller process and several renderer processes, with each renderer process being responsible for rendering pixels on one nonintersected rectangle area of the virtual desktop. The system architecture of Magic View is shown in Fig. 1. In the architecture, each SAGE display node hosts a separate Magic View renderer process, and a dedicated machine is used to host the Magic View controller process. Magic View has the following key features: Support TIFF files larger than 4GB and storing multi-resolution image pyramid in one TIFF file. Support viewing multiple images one by one with panning and zooming. Support any zoom ratios between two consequent zoom levels. B. Components 1) Magic View Controller Magic View Controller is the central control module of Magic View, acting as the master MPI process. When started, it reads configuration files and broadcast the configurations to renderer processes through MPI interface. Then it enters into a daemon loop, waiting for commands from clients and broadcasting the received commands to renderer processes. 264

4 2) Magic View Renderer Magic View Renderer is the main working part of Magic View. When started, it receives configuration data from the controller process through MPI and then enters into a loop, listening for commands issued from the controller process. When a drawing command is received, it parses the command to retrieve the location of the TIFF image, reads the pixel data, performs zooming actions on the pixel data if necessary, and then sends the processed pixel data through SAIL to SAGE Receivers. 3) Magic View Client Magic View Client is the user interface of Magic View. It converts user operations into commands, and sends the commands to the Magic View Controller. A command includes the path of the image file to display, a viewport rectangle that indicates the visible part of the image and a drawing rectangle representing the window in the virtual desktop. The client can either have a standalone GUI (Fig. 2), or be integrated with the Magic Input Framework (Fig. 3). Both versions of the client support common image manipulation operations like pan, zoom and changing the currently displayed image. And the two versions are used in different situations. If user interaction in front of the tileddisplay is required, the Magic Input integrated version should be a better choice as it supports collaborative viewing. However, if the tiled-display is mainly used to show images without interaction, the standalone GUI version can be used as it can provide additional functions such as changing image display order and free panning on the image. C. Parallel Image Extraction As there are multiple renderer processes in a Magic View system, the image pixels can be extracted in parallel. Each process will determine the pixels to be extracted from the image file when it receives a drawing command, according to the pre-configured rendering rectangle and the two rectangles from the command. To implement such algorithm, we first define some helper functions. Assume R is a rectangle with (x 1, y 1) as the left-top coordinate and (x 2, y 2) as the right-bottom coordinate, then we define the following equations: L(R) = x 1, T(R) = y 1, R(R) = x 2, B(R) = y 2, W(R) = x 2 -x 1, H(R) = y 2 -y 1. Assume there are N renderer process labeled from 1 to N, and the rendering rectangle of the nth renderer process is R 1. If the nth renderer process receives a command with a drawing rectangle R 2 and an extraction rectangle R 3, then the pixels to be extracted can be determined by the following steps: 1. Intersect the rendering rectangle R 1 and the drawing rectangle R 2, resulting a temporary rectangle R'. If either the width or the height of R' is zero, then there is no pixels to be extracted. Otherwise, go to the next step. 2. The region of pixels to be extracted can be described by a rectangle R defined by the following formulas: (1) (4) After the extraction region is calculated, the renderer process can load pixels from the TIFF image file through the APIs provided by LibTIFF, and zoom the loaded pixels if necessary. V. OPTIMIZATION A. Space Optimization Given the huge size of an ultra-large image, it is clearly not efficient, and sometimes even impossible to load all the pixels of the image into the memory. Thus, Magic View takes a load-on-demand approach that only fetches the pixels within the visible portion of the image required by the user. This approach needs to consider two cases: displaying the original image and zooming. 1) Display images in original size If an image is displayed in its original size, the loaded pixels can be copied to the back buffer provided by SAIL directly. As there are two storage formats, the tiled format and the striped format, in the TIFF specification, the renderer process supports both of them by treating the striped format as a specialized tiled format. Suppose the size of image is w h, the size of a tile or strip is w' h', and the extraction region is R, then the loading algorithm can be described by the following pseudocode: Tiles per row := (w + w' - 1) / w'; For y = T(R) To B(R) Step h' (y mod h') For x = L(R) To R(R) Step w' (x mod w') Tile number := (y / h') Tiles per row + (x / w'); Read the tile from file to the tile buffer; Copy the pixels inside the display portion to back buffer; For the tiled format TIFF files, each read operation will get a small square of pixels, whereas for striped format TIFF files, one or more rows of pixels will be obtained through one read operation. Therefore, for most ultra-large images whose horizontal sizes are much wider than the tile-display wall, there will be fewer pixels read from tiled format files than that from the striped format files, which may lead to less process latency and higher performance. 2) Display mages with zooming If an image is not displayed in its original size, the procedure may be more complex. Normally, all the pixels to be displayed should be initially loaded into memory, and then they can be zoomed to fit the display size. However, if the image is ultra-large, and the zoom ratio is too high, display nodes have no enough memory to load the pixels entirely. Therefore, Magic View has to optimize the memory usage for zooming operation by using two different optimized procedures. One is the two-step zooming procedure that can support any zoom ratios between two consequent zoom levels, and the other is the simple-sub- (2) (3) 265

5 sampling zooming procedure that can only support integer zoom ratios. a) Two-step Zooming Firstly the image is zoomed in horizontal rows and stored in an intermediate buffer. Secondly, the stored image is zoomed again in vertical rows. Suppose that the width of image (in striped format) or the portion to be displayed (in tiled format) is w, the height of the portion to be displayed is h, the size of a tile or strip is w' h', the width of the rendering rectangle is w 1, then there will be only two extra buffers needed, with one of w h' pixels and the other of w 1 h pixels. As the one-step zooming procedure requires an extra buffer of w h pixels, and normally the conditions h' << h and w 1 << w are both satisfied, the extra space required by the two-step zooming procedure will be much less than that required by the normal one-step zooming procedure. The following pseudo-code describes the twostep zooming procedure: // The horizontal zooming step For y = T(R) To B(R) Step h' (y mod h') For x = L(R) To R(R) Step w' (x mod w') Tile number := (y / h') Tiles per row + (x / w'); Read the tile from file to the tile buffer; Copy the pixels inside the display portion to buffer1; Zoom the pixels in buffer1 and store the result in buffer2; // The vertical zooming step Zoom the pixels in buffer2 and store the result in back buffer; b) Integer Subsampling Zooming Although the two-step zooming can support any integer and non-integer zooming ratios, it requires an extra buffer to keep the intermediate results during the zooming procedure. However, in many cases, it is enough to support only integer zooming ratios, in which we can adopt the integer subsampling zooming procedure. The biggest advantage of this method is to avoid any extra buffers for storing intermediate results. Instead, it performs zooming directly between the tile buffer and the back buffer. Assume that the horizontal zoom ratio is Z x and the vertical zoom ratio is Z y, with all the other definitions being the same as those described above, then the following pseudo-code describes the integer subsampling zooming procedure: For y = T(R) To B(R) Step h' (y mod h') For x = L(R) To R(R) Step w' (x mod w') Tile number := (y / h') Tiles per row + (x / w'); Read the tile from file to the tile buffer; Copy only the top-left pixel in each Z x Z y rectangle inside the display portion to back buffer; B. Time Optimization Besides the space optimization, Magic View also performs time optimization to achieve higher performance. Since TIFF files support fast random access in both striped format and tiled format, the maximal time consumption before rendering the image will be the zooming operation. Therefore, Magic View uses the multi-resolution image pyramid to eliminate the actual run-time zooming procedures. An image pyramid consists of several pre-zoomed images as well as the original ultra-high resolution image with each one corresponding to a detail level. However, different to many other image viewers such as DisplayCluster which require the image pyramid to be stored as groups of image tile files, Magic View supports storing the image pyramid of one ultrahigh resolution image in a single TIFF file. The space requirement of the time optimization is not a fixed value, which is determined by the number of pre-stored detail levels. When a zoom-in or zoom-out operation is required by the user, the renderer process reads the corresponding prezoomed image from the image pyramid in the TIFF file instead of actually zooming it. Therefore, the timeconsuming zooming operation is converted to a fast filereading operation. The only disadvantage of this method is that only some particular zoom levels of an image is available. However, in many cases, only few zoom levels of an image is required, which makes this method very useful in the real scenarios. VI. EXPERIMENTS AND ANALYSIS A. Environment Preparation All of the experiments are done in our own tiled-display environment, which is a 7 4 tiled-display wall with a resolution of driven by 7 Dell workstations. Each workstation drives four monitors with the resolution of and is equipped with one Quad-core Intel Xeon E5506 CPU, 4GB RAM, one 500GB hard disk formatted by ext4 file system, two NVidia Quadro FX 3800 GPUs and one Emulex 10Gbps Ethernet adapter. All the comparisons will be done between Magic View and Juxta View. To minimize the influence of the network latency, we use a special deployment for both Juxta View and Magic View. Such deployment is done by deploying one working or renderer processes on each display node, and make sure that each process only be responsible for the same region as the SAGE Receiver in display node be responsible for. In such deployment, the process on each render node needs only send the pixels to the SAGE Receiver on the same machine and the network is only used for transmitting messages. By doing so, the latency of transmitting pixels through network will be eliminated, which can make the experiment results more accurate to reflect the performance of the two programs themselves. Similarly, the client program is also deployed on the master node to eliminate the network latency between them. We compare the performance between Magic View and Juxta View by measuring the execution time of arbitrary pan and zoom operations on the NASA s Blue Marble, the thumbnail of which is shown in Fig. 4. For Magic View, we build two versions of the image with the parameters shown in Table I. For Juxta View, we split the whole image into three image tile sets of different sizes with the parameters shown in Table II. As Juxta View supports only four integer zoom levels when displaying the test image, only the largest four sub-images are used by Magic View in the image pyramid mode. 266

6 Figure 4. NASA s Blue Marble, a 24-bit RGB TIFF image with the original size of pixels. B. Experiments and Results Before the experiment, we build a test set consisting of hundreds of randomly selected pan and zoom operations. Each zoom operation in the set will change the current zoom level by one, e.g. enlarging or reducing the display ratio by two in both directions, and each pan operation will be executed at current zoom level. Then we apply the test set on Magic View and Juxta View respectively, and record the execution time interval between the initial moment when the client issues a command to the controller and the next moment when the client receives the acknowledgement of execution result from the controller. And the first fifty values of execute time of pan operations at original size, pan operations at zoom level 3 and zoom operations are used for comparison. The results are shown in Table III and Table IV. TABLE I. PARAMETERS OF IMAGE FILES USED BY MAGIC VIEW Parameter Image version 1 Image version 2 Image format Tiled Striped Image file size 3.25 GB 4.58 GB Tile size N/A Rows per strip N/A 1 Pixel format Compression algorithm 24-bit RGB Deflate Number of sub-images 10 Resolutions of downsampled subimages [0]: [1]: [2]: [3]: [4]: [5]: [6]: [7]: [8]: [9]: TABLE II. PARAMETERS OF IMAGE FILES USED BY JUXTA VIEW Parameter Tile set 1 Tile set 2 Tile set 3 Image size of each tile file Number of tile files 57,122 3, Total size of tile files 3.43 GB 3.05 GB 2.95 GB Image format Striped Rows per strip 1 Pixel format Compression algorithm Configuration Tiled image Striped image TABLE III. 24-bit RGB Deflate RESULTS FOR MAGIC VIEW Operation Execution time (ms) Min Max Avg Pan / Original Pan / Pyramid L Pan / Integer subsampling L Zoom / Pyramid Zoom / Integer subsampling Pan / Original Pan / Pyramid L Pan / Integer subsampling L Zoom / Pyramid Zoom / Integer subsampling

7 Configuration tiles tiles tiles TABLE IV. RESULTS OF JUXTA VIEW Operation Execution time (ms) Min Max Avg Pan / Original Pan / L Zoom Pan / Original Pan / L Zoom Pan / Original Pan / L Zoom C. Result Analysis From the Table III and Table IV we can easily find that Magic View can achieve at least 8x lower exexution time than Juxta View by using pyramid method and tiled format TIFF. And from the two tables, we can find the critical design decisions that may influence the execution time of displaying an ultra-large image. The first choice is whether the image is stored in the tiled format or striped format. From Table III, we can see that the execution time of displaying a tiled format image in its original size is about 120ms in average, while the execution time of displaying a striped format image in its original size will be more than 4600ms in average. The performance disparity is mainly caused by the different amount of pixel data to be processed. In our experiment, each renderer process will be responsible for a display region of pixels and the original size of the image is If the image is stored in the tiled format with the tile size of pixels, then there will be a total amount of 5.1 million pixels to be processed. However, if the image is stored in the striped format, the pixel data is increased to million pixels Thus, the execution time of displaying a striped format image is certainly much higher than that of showing a tiled format image. The second choice is whether pyramiding method is applied in the imagery management. Through the results in Table III, if we compare the average execution time of zoom operations on a tiled format image, the rendering process on a pyramid image is about 10x faster than the process on a non-pyramid image. The performance improvement is mainly due to the removal of the latencies in run-time resize operations. For a pyramid image, a zoom operation is actually a normal file read operation without complex arithmetic operations; while for a non-pyramid image, a zoom operation involves both a file reading step and an actual resize step that includes many complex arithmetic calculations. Therefore, using pyramid image will greatly decrease the execution time. The third design decision is about how the tiled format images are organized. The number of tile files for an image affects the display performance as well. As discussed above, the execution time increases as the amount of data to be processed goes up. Therefore, for a tiled format image, or an image that is pre-split into group of tile files, using smaller tile size will reduce data amount in the display procedure. Because there will be fewer invisible pixels at the edges of the display area when using a smaller tile size. However, it is interesting to find out in Table IV that the shortest execution time is actually achieved with the tile size of instead of The reason is that the number of files in the tile set is nearly 60,000 while the number in the tile set is less than 4,000. And as Magic View uses only one TIFF file to store all the tiles of all the pyramid levels, Magic View can achieve much higher reading performance than Juxta View when they use the same tile size. From the above analysis, we can find that by combining the usage of image pyramiding method, tiled format imagery and consolidation of all image tiles in one TIFF file, Magic View can provide real-time interactive performance on ultralarge images with billions of pixels at any zoom levels, while Juxta View cannot provide similar performance even in the original size. VII. CONCULSION In this paper, we proposed Magic View, an optimized SAGE-enabled ultra-large image viewer. It takes advantage of the multi-resolution image pyramid method and tiled format images to enhance the performance. And for nonpyramid images, two improved zooming procedures are introduced to reduce the memory consumption when zooming ultra-large images. Our experiments show that Magic View achieves at least 8x faster response time than Juxta View for viewing the same ultra-large image and can provide real-time interactive performance on ultra-large images with billions of pixels. We are now working on some new features to Magic View, including viewing multiple images simultaneously, adding image tagging support and allowing collaborative viewing. ACKNOWLEDGMENT This work was supported by the State Key Laboratory of Software Development Environment Funding No. SKLSDE- 2013ZX-03. REFERENCES [1] Z. Hu and W. Wu, A Satellite Data Portal Developed for Crowdsourcing Data Analysis and Interpretation, in Proc. e-science, [2] B. Jeong, L. Renambot, R. Jagodic, R. Singh, J. Aguilera, A. Johnson and J. Leigh, High-performance dynamic graphics streaming for scalable adaptive graphics environment, in Proc. Supercomputing, [3] Y. Lou, W. Wu and H. Zhang, Magic Input: A Multi-user Interaction System for SAGE Based Large Tiled-display Environment, in Proc. ICMEW, [4] Tiled Image Viewer (TimV), [5] J. Blinns, M. E. Papka, R. Stevens, Cluster-based image viewer, 268

8 [6] N. K. Krishnaprasad, V. Vishwanath, S. Venkataraman, A. G. Rao, L. Renambot, J. Leigh, A. E. Johnson, JuxtaView - a tool for interactive visualization of large imagery on scalable tiled displays, in Proc. Cluster Computing, [7] LibTIFF - TIFF Library and Utilities, [8] Magic Carpet, [9] G. P. Johnson, G. D. Abram, B. Westing, P. Navr'atil and K. Gaither, DisplayCluster: An Interactive Visualization Environment for Tiled Displays, in Proc. Cluster Computing,