12 Window Systems - A window system manages a computer screen. - Divides the screen into overlapping regions. - Each region displays output from a particular application. X window system is widely used in most Unix machines. We will use the X11 window system for the following discussion. The Goals of X - To provide low-level graphics workstation services - windows - 2-D color bitmap graphics - multiprocess access to workstation screen (Multiple clients connected to a display server) - low-level input processing - To have local-area network transparency - applications on same or different network node of the workstation - portability of application software - multiple displays - open architecture - To provide standard programming interfaces, eg. Xlib library interface - To provide user interfaces - xterm terminal emulator - other tools such as editors and calculators - programming toolkits - To encourage further development - Inexpensive, widely available source code Features of the X11 - Hardware and operating system independent. - Developed jointly by MIT and DEC. - The X11 provides programmer interfaces through Xlib and X Toolkits. - Xlib provides basic functions for drawing and for manipulating the windows. Eg., XcreateWindow(display, parentwin, x,y,width,height,..); XSetLineAttributes(display, gc, linewidth, linestyle,..); XdrawLine(display, win, gc, x1,y1,x2,y2); - X Toolkits provide a higher level of interface to Xlib through widget. A widget is a routine for creating and using user-interface components. A widget set may include menus dialog boxes scrollbars and command buttons. 1
Server and Client The server is a program running on the workstation. It acts as an intermediary between user programs (or clients) and the workstation resources (keyboard, mouse, and screen.) Client applications need not know the hardware of the server resources. The server performs the following tasks: - Allows access by multiple clients. - Allows client applications to share resources such as screen space and graphics processor. - Interprets network messages (or requests) and acts on them. Requests include move window and draw polygon. - Sends user inputs to clients by sending network messages, eg. key presses, button presses, and pointer motion. - Maintains complex data structures including windows and fonts so that the server can perform its tasks efficiently. A Client is an application which requests services from the server. A client may or may not be running on the same machine as the server (network transparency). All communication between a client and a server uses the X Protocol. 2
Window Manager In X11, a window manager is just another client, managing a particular screen at any one time. Tasks of the window manager: - Controls the behavior of the screen such as the functionality of each mouse button. - Controls the layout of the windows such as title bars and borders. - Controls the size and displacement of windows. - Manage icons. Windows and Window Hierarchy Dimensions of a window: Hierarchical organization of windows: - A window will obscure any overlapping windows that are behind it. - All sub-windows are bounded by their parent windows. - Any part of a sub-window going outside of its parent window will be clipped. - Any thing to be drawn but are outside of the specified window will be clipped too. - A window s shape may become irregular when obscured by other windows. - To simplify the clipping operation, an obscured window is broken into rectangular windows: Clipping an object against Window B is simplified to clipping the object against each of the windows a,b,c. 3
13.Virtual Reality (VR) A definition is from the book Silicon Mirage by Steve Aukstakalnis and David Blatner: Virtual Reality is a way for humans to visualize, manipulate and interact with computers and extremely complex data. Methods for human-computer interaction are called computer interfaces, and virtual reality is just the newest in a long line of interfaces. General features of virtual reality: - It emphases in the interaction with the computer through gestures and human senses. - Instead of using screens and keyboards, people can put displays over their eyes, gloves on their hands, and headphones on their ears. - The computer controls what the user senses and the user, in turn, can control the computer. - The user may have a feeling of entering into a different world (virtual world). - We may also want to obtain interactive information related to the real world (Augmented VR). Levels of Virtual Reality 3 levels of VR: - Passive: Not much control over the environment. Eg., watching a movie or TV show, reading a book, and listening to the radio. During reading, though we can choose a particular book, we cannot change its story. - Exploratory: Instead of just seeing a3d space, we can move around in it (flying or walking). - Interactive: Can explore and interact with a virtual world. Eg., reaching out and grabbing a virtual book, or moving furniture around in a virtual room. The term VR normally refers to either 2 nd level or 3 rd level of VR. 4
Human and Computer Communications People communicate through variety of senses: - Vision: Visual effects play a very important role in human communication. - A lot of research has been done on improving the realism of the computer generated images and the speed of generating them. - Also, a lot of research has been done on improving the computer vision technology to understand the user s world through vision. (Far more difficult than the previous one.) - Sound: A direct way for human to exchange ideas, and let people sense the surrounding, eg. the presence of an approaching car and its approximate distance. Similar to vision, it is easier to generate sound than to recognize it. - Other Senses: Other senses include smell and touch. - To produce different kind of smell as well as to identify them are both difficult to do. These factors are rarely considered. - There are existing equipment which can detect forces from or exert forces onto a user. However, this kind of equipment is usually intrusive in that the user can feel their presence even through they are not in operation at a particular moment. Eg. one can feel the presence of the glove when he/she is wearing it. Types of VR Systems - Different approaches to let users communicate with the computer. - Common ones: Immersive VR, Non-immersive VR, Augmented VR, and Telepresence. Immersive VR The ultimate VR system is to completely immerse the user to a computer simulated environment. - Because visual effect is important in human communication, and 3D graphics hardware is getting cheaper, existing immersive VR systems emphasis on visual immersion. - A head Mounted Display (HMD) with position tracker provides a virtual 360 vision. - Some systems may use multiple large projection screens to visually surround the user. - Other systems may provide even the same physical environments. Eg., flight simulators and car simulators in which the users are confined to a small room. 5
Non-immersive VR Non-immersive VR systems do not provide a 360 of vision and the user is not required to wear an HMD or be inside a simulation room. => limited reality. - Usually uses traditional monitors for visual output. - Some may use an additional equipment (such as the CrystalEyes 3D stereo glasses) to provide stereo vision. In the CrystalEyes glasses, the 2 lenses allow light to pass through alternatively in synchronization with the displayed images to provide stereo vision. The user is inside the virtual world only when he is looking at the computer screen. When he looks away from it, he is immediately back to the real world. Augmented VR It uses transparent glasses onto which data, diagrams, animation or video can be projected so that the user is able to see the real world with additional information provided by the computer. - There are 2 common types of displays available. One is a pair of glasses. A small projector is attached to it so that video images can be projected onto the pair of glasses. - The other one is the flight helmet also used by the pilots of the fighter jets. A small projector is attached to the helmet so that additional information from the computer about the environment can be projected onto the transparent screen. Telepresence Telepresence links the sensors and manipulators of a remote robot to a human operator. - Cameras, microphones and other sensors are attached to the remote robot. Signals from these sensors are sent to the human operator in a control room. The operator may be wearing an HMD and other devices which render the remote signals. - The human operator is also wearing some sensors which sense the movement of the operator. The signals are sent to the remote robot to control its movement. Useful in performing some dangerous operations such as putting out the fire in a nuclear plant, or in performing some otherwise too expensive operations such as Mars exploration. 6
Position Sensors - 2D mouse: Difficult to use in 3D world. Only returns relative (not absolute) mouse movement. - Mechanical tracker: A mechanical device is attached to the object. The movement of the object results in the movement of the mechanical device, which generate the positional signal. This kind of devices are fast and accurate but has restrictions on motion. - Ultrasonic tracker: An ultrasonic transmitter is attached to the object to be tracked and a few sensors are placed at different positions. By detecting the time it takes for the sensors to receive the ultrasonic signal from the transmitter, it is possible to locate the position of the object. However, ultrasonic systems are of low resolution and are susceptible to noise. - Light pattern: By putting a pattern of light transmitters on the object and cameras on the wall (or vice versa), it is possible to locate the position of the object by detecting the patterns of light captured by the cameras. Limitations: it requires image processing, which is not reliable enough, and a dedicated room. - Magnetic tracker: Magnetic tracing is one of the widely used methods. - A magnetic transmitter is placed at fixed location. A magnetic sensor is attached to the object. - Both the transmitter and sensor contain 3 coils at right angles to each other. - Sensor s position and orientation affect the amount of current induced to it by the transmitter. - As such, the sensor can be used to detect the 3D position (x,y,z) and orientation of an object. - Advantage: Magnetic tracking devices are cheap and accurate. - Limitation: 1) The accuracy of these devices is affected by nearby metallic materials. 2) It is generally believed that strong magnetic fields are harmful to human body. A Typical Immersive System 7
Time Lag Problem All VR systems suffer from the time lag problem. - When one moves his head, the magnetic sensor senses the change and generates a signal. There is a delay from the head movement to the generation of the signal. =>A - When the head tracking system receives the signal from the sensor, it processes the signal to obtain the new position and orientation of the sensor. There is also a delay in this process. =>B - When the tracking system sends out the information, the computer receives the information, processes it and then updates the database. There is a delay in this process. =>C - After the database is updated, the image generator renders an updated image from the updated database. There is also a delay in this process. =>D - A+B+C+D : The time the head moves to the time the image is shown on screen = total time lag. - This time lag causes an uncomfortable feeling to the user. - Some work is being done to reduce such problem by predicting what may the head be few steps ahead. Problem: the prediction may not always be correct. Virtual Reality vs Computer Graphics - In CG, we concern about the output image quality and also the speed in generating the image. - In VR, we concern about the quality of the images. However, with the limitation in current hardware technology, we are more concerned about the speed in generating the images. - VR: at least 10 images per second and the time lag problem must be within an acceptable level. - The Time critical rendering issue: given the time we have, how can we generate the best picture. - Eg., if time is not enough, we may render objects with flat shading instead of Gouraud shading. - We may also use simpler models for rendering less important objects. - The image coherence issue: The time between 2 consecutive frames is small, the change in the scene is small too. Hence, given the information about the previous frame, we can make use of it and minimize the amount of calculation in generating the next image. - Eg., if an object is far away in the last frame, it is likely to be still far away in the current frame. Current Areas of Research - Multi-resolution modelling: This uses a simpler model for rendering when the object is far away from the viewer. This is to reduce the number of polygons needed to be rendered. - Fast rendering - Collision detection: This detects if 2 objects collide with each other. - Visibility culling: This determines the visibility of a surface or an object. 8
14. Multimedia Multimedia is the integration of media technologies including sound, video, document, network and graphics with the computer technology. There are 2 major areas of interest in multimedia: - Multimedia technology: the technology itself. - Multimedia applications: the application of the technology Multimedia Technology The integration of media technologies may be considered as putting all the devices that handle different media together with the computer being the central controlling device: With a high-speed network, it is possible to provide video and audio services through the network: - These systems allow us to watch TV, to have video-conferencing with someone in remote location, to read an electronic document, etc... - Under such a situation, we can also have a large central database to provide variety of information services. 9
One of the most demanding media technologies is the handling of video. Consider color images of video quality 512x512 in resolution: - The memory size needed to store a single image, M single, is: M single = 512 x 512 x 3 Bytes = 0.75 M Bytes - Assuming 30 frames per second. The memory size needed for a second of image sequence, M sec : M sec = 0.75 M Bytes x 30 = 22.5 M Bytes - A movie typically lasts for 90 minutes. The memory size needed, M movie, is: M movie = 22.5 M Bytes x 60 x 90 = about 118.7 G Bytes For color images of movie quality, eg. resolution of 1600 x 900 (16:9) with 24 frames per second: M single = 1600 x 900 x 3 Bytes = 4.12 M Bytes M sec = 4.12 M Bytes x 24 = 98.88 M Bytes M movie =98.88 M Bytes x 60 x 90 = about 521.4 G Bytes With such a huge amount of memory, handling of videos digitally becomes a driving force to improve other technologies such as - database (to store the images) - network (to transfer the images from the server to the computer) - data compression (to compress / uncompress the images before / after transmission or storage). - Other related issues (eg. the demand for high resolution displays) Related Technologies 1. Display Technology: With windowing systems being widely used in both PCs and workstations, and the consumer community pushing for high definition television (HDTV), display technology is advancing rapidly. Monitors of resolution 1600x1280 are commonly available. At the same time hardware circuitries are also available which can read the frame buffer fast enough to update the screen near to 100Hz. 2. Network Technology: A local computer network connects many computers together so that a small community can share resources such as disk space, printing facilities, and processing power. With most of the local networks in the whole world being connected together through the INTERNET, it is now possible to share some of the resources in a global scale. A fast Ethernet network for video transmission is required. Recent developments in this area include the 100 Mbits/s fast Ethernet network and the over 1 Gbits/s fiber optical network. 3. Database Technology In a multimedia environment, there may be many different types of data that conventional data management systems are not prepared to handle. The data objects can span an enormous range of size and format. Such a multimedia database contains a mixture of structured data (relations with fixed fields) and of unstructured data (text and images). All these factors demand for new models of data management systems. Because not all multimedia systems are built using an object paradigm, the new models must also provide object-oriented facilities for handling of data objects. 4. Image Compression The needs for fast network to transmit digital videos and for large database to store them demand for better and faster image compression methods. 10
5. Sound Technology There are two major issues: music synthesis and speech processing. Music synthesizing technology is becoming matured. In addition, the protocol called MIDI is also widely accepted by manufacturers as a standard for networking musical instruments. Speech technology can be divided into 2 categories: Speech synthesis and Speech recognition 6. Memory Technology A multimedia system may need to have access to a large volume of information. This demands for more RAMs and secondary storage. With the continuous improvement in memory technology, both RAMs and secondary storage devices are getting cheaper and cheaper. Multimedia Applications Applications of the multimedia technology are continuously to be proposed and developed. Some of them are: 1. Video conferencing: allows 2 or more people at different physical locations to communication with each other via voice and video through telephone or high speed networks. 2. Video services: are usually provided to users who have their machines connected to a high speed network. In a cable network environment, subscribers receive different video services from a central server(s) maintained by the cable company. 3. Video shopping: is a kind of services usually provided in a cable network environment in which the user can purchase some products simply by pressing a button on the remote controller while watching an advertisement on the cable TV. 4. Interactive environment for collaboration: is the use of multimedia technology to provide an interactive environment for collaboration through work space sharing, and video and audio communication. 5. CD-ROM / DVD-ROM publishing: is the distribution of CDROMs and DVDs which store the interactive multimedia catalogues or interactive books. 11