- Modifying the histogram by changing the frequency of occurrence of each gray scale value may improve the image quality and enhance the contrast.

Transcription

1 11. Image Processing Image processing concerns about modifying or transforming images. Applications may include enhancing an image or adding special effects to an image. Here we will learn some of the image processing functions. The description will be based on grey-scale images. However, extending the techniques to color images are usually straight-forward. Histogram Modifications - A histogram is a plot of the frequency of occurrence, p(g), against the gray level value, g. - The distribution of gray scale values affects the quality of the image. The following diagram shows some examples: (a) If most of the gray level values in an image are small, the image will be dark. (b) If most of the gray level values in an image are large, the image will be bright. (c) If most of the gray level values in an image are within a small region, the image will have a low contrast. Therefore, both (a) and (b) are also considered as low contrast images.) (d) A high contrast image has a wide range of gray level values. - Modifying the histogram by changing the frequency of occurrence of each gray scale value may improve the image quality and enhance the contrast. - Brightness Correction is used to increase the brightness of a dark image as follows: Pixel(x,y) = Pixel(x,y) + brightness Brightness correction is therefore equivalent to shifting the histogram horizontally. - Contrast Correction is used to modify the contrast of an image as follows: Pixel(x,y) = contrast * (Pixel(x,y) gmean) + gmean Where gmean is the mean gray scale value of the original image. Hence contrast correction is equivalent to scaling the histogram. We can reduce the contrast of an image by setting the factor contrast in the above equation to a value smaller than 1 and we can increase the contrast by setting it to a value higher than 1. 1

2 Editing of Intensity / Color of an Image - Very often, it may be desirable to be able to edit the gray scale values in a linear as well as non-linear way. - To do this, a curve is usually provided which indicates the relationship of the original and new values. - This curve can be edited interactively by moving the control points. - A color image is composed of red, green and blue channels. - Three curves are usually provided so that each of the channels can be modified independently. Warping - Warping is the use of image mapping functions to produce geometric distortions of images. - The basic technique is that the user specifies the contour of the object in the image to be transformed and the final contour that the object is to fit into. - A texture mapping operation is performed to map the source object into the destination shape. Morphing - Morphing is used to provide a smooth transition from one image to another and thus create an illusion of image transformation. - A simple morphing may use a fade or dissolve process that gradually decreases the contribution of the source image to the output image while increases the contribution of the destination image as follows: - Depending on applications, the decrement (or increment) in contribution of the source (or destination) image into the final image may not have to be linear as in the diagram. - More advanced morphing techniques may use a warping process in addition to the fade / dissolve to slowly bring into alignment the shapes of the start and end pictures. 2

3 12. Window Systems A window system manages a computer screen and divides the computer screen into overlapping regions. Each region displays output from a particular application. Since the X window system is widely used in most of the Unix machines, we will use the X11 window system as a basis for this discussion. The Goals of X - To provide low-level graphics workstation services - windows - 2-D color bitmap graphics - multiprocess access to workstation screen - low-level input processing - To have local-area network transparency - applications on same or different network node of the workstation - portability of application software - multiple displays - open architecture - To provide standard programming interfaces, eg. Xlib library interface - To provide user interfaces - xterm terminal emulator - other tools such as editors and calculators - programming toolkits - To encourage further development - Inexpensive, widely available source code Features of the X11 - It is a hardware and operating system independent window system developed jointly by MIT and DEC. - It is based on a client-server model. - Multiple clients can be connected to a display server simultaneously. - The X11 provides programmer interfaces through Xlib and X Toolkits. - Xlib provides basic functions for drawing and for manipulating the windows. Eg., XcreateWindow(display, parentwin, x,y,width,height,..); XSetLineAttributes(display, gc, linewidth, linestyle,..); XdrawLine(display, win, gc, x1,y1,x2,y2); - X Toolkits provide a higher level of interface to Xlib through widget. A widget is a routine for creating and using user-interface components. A widget set may include menus dialog boxes scrollbars and command buttons. 3

4 Server and Client The server is a program running on the workstation. It acts as an intermediary between user programs (or clients) and the resources of the workstation such as keyboard, mouse, and screen. It contains all device-specific code so that client applications do not need to know the hardware of the server resources. The server performs the following tasks: - Allows access by multiple clients. - Allows client applications to share resources such as screen space and graphics processor. - Interprets network messages (or requests) and acts on them. Requests include move window and draw polygon. - Sends user inputs to clients by sending network messages, eg. key presses, button presses, and pointer motion. - Maintains complex data structures including windows and fonts so that the server can perform its tasks efficiently. In X window system, the term display is often used to refer to server. The term screen refers to the physical hardware screen and a display may have more than one screen. A Client is an application which requests services from the server. A client may or ma not be running on the same machine as the server (network transparency). All communication between a client and a server uses the X Protocol. Window Manager In X11, window manager is just another client. However, there can only be one window manager managing a particular screen at any one time. The tasks of the window manager include: - Controls the behavior of the screen such as the functionality of each mouse button. - Controls the layout of the windows such as title bars and borders. - Controls the size and displacement of windows. - Manage icons. The following diagram shows the structure of the X11 window system: 4

5 Windows and Window Hierarchy The following diagram shows the dimensions of a window; Windows in X are organised in a hierarchical manner. - The top window, which covers the whole screen surface, is called root window and is opened automatically. - A window will obscure any overlapping windows that are behind it. - Any sub-windows are bounded by its parent window. - Any part of a sub-windows goes outside of its parent window will be clipped. - Any thing to be drawn but are outside of the specified window will be clipped too. - Since rectangle is easy to do clipping, most window systems use rectangle as the basic window shape. - Although a window may originally be rectangular, it may become irregular when it is obscured by other windows (eg. window A in the previous diagram). - In order to simplify the clipping operation, the X approach is to break an obscured window into rectangular windows as shown in the following diagram: - In the diagram, window B is divided into 3 rectangular regions a, b, and c. Clipping an object against Window B can now be done by clipping the object against each of the 3 rectangular regions and display any parts of the object which are inside any of the regions. 5

6 13.Virtual Reality (VR) There are many definitions for the term Virtual Reality. The following definition is from the book Silicon Mirage by Steve Aukstakalnis and David Blatner: Virtual Reality is a way for humans to visualize, manipulate and interact with computers and extremely complex data. Methods for human-computer interaction are called computer interfaces, and virtual reality is just the newest in a long line of interfaces. General speaking, virtual reality has the following features: - It emphases in the interaction with the computer through gestures and human senses. - Instead of using screens and keyboards, people can put displays over their eyes, gloves on their hands, and headphones on their ears. - The computer controls what the user senses and the user, in turn, can control the computer. - With such an interaction, the user may have a feeling of entering into a different world (virtual world). - Instead of entering into a different world, sometimes, we may want to be able to obtain interactive information related to the existing world. We call this Augmented VR. Levels of Virtual Reality There are 3 levels of VR. Each of them offers slightly different features and each progressive step is much more difficult to create. - Passive: In passive VR, we do not have much control over the environment that we are experiencing. It includes watching a movie or TV show, reading a book, and listening to the radio. Taking reading as an example. While we may be able to choose a particular book to read, we cannot change the story of it. - Exploratory: The 2 nd level of VR is that of being able to explore a virtual world. Instead of just seeing a3d space, we can move around in it, whether by flying or walking. - Interactive: The 3 rd level of VR is that of being able to explore and interact with a virtual world. We may do this by reaching out and grabbing a virtual book, or moving furniture around in a virtual room. The term VR normally refers to either 2 nd level or 3 rd level of VR. 6

7 Human and Computer Communications Human communicate through variety of senses. Similar kind of communication may be necessary in order for a person to communicate with a computer in the same way as to another person. - Vision: Visual effects play a very important role in human communication. - A lot of research has been done on improving the realism of the computer generated images and the speed of generating them so that the computer can present its own world to the user. - In the same way, a lot of research has been done on improving the computer vision technology so that the computer can understand the user s world through vision. (This has proven to be far more difficult than the previous one.) - Sound: Audio can provide a direct way for human to exchange ideas. It also helps a person to sense its surrounding, eg. the presence of an approaching car and its approximate distance. Similar to vision, it is easier to generate sound than to recognise it. - Other Senses: Other senses include smell and touch. - To produce different kind of smell as well as to identify them are both difficult to do. These factors are rarely considered. - There are existing equipment which can detect forces from or exert forces onto a user. However, this kind of equipment is usually intrusive in that the user can feel their presence even through they are not in operation at a particular moment. Eg. one can feel the presence of the glove when he/she is wearing it. Types of VR Systems A major distinction of VR systems from traditional computer systems is on how the user communicates with the computer. There are many types of VR systems developed for different kinds of applications. Four of them are Immersive VR, Non-immersive VR, Augmented VR, and Telepresence. Immersive VR The ultimate VR system is to completely immerse the user to a computer simulated environment. - Because visual effects play such an important role in human communication and 3D graphics hardware is getting cheaper and cheaper, existing immersive VR systems emphasis on visual immersion. - A head Mounted Display (HMD) with position tracker is usually used to provide a virtual 360 vision. - Some systems may use multiple large projection screens to visually surround the user. - Other systems may provide even the same physical environments. These are usually limited to applications such as flight simulators and car simulators in which the users are confined to a small room. 7

8 Non-immersive VR Unlike the immersive VR, non-immersive VR systems do not provide a 360 of vision and the user is not required to wear an HMD or be inside a simulation room. Hence, non-immersive VR is also called limited reality. - Most low cost systems use traditional monitors for visual output. - Some systems may use an additional equipment (such as the CrystalEyes 3D stereo glasses) to provide stereo vision. In the CrystalEyes glasses, the 2 lenses allow light to pass through alternatively in synchronization with the displayed images to provide stereo vision. These systems are slightly more expensive than those using traditional monitors along and they require the user to wear a pair of glasses. With these systems, the user is inside the virtual world only when he is looking at the computer screen. As soon as he looks away from it, he is immediately back to the real world. Augmented VR Augmented VR is the use of transparent glasses onto which data, diagrams, animation or video can be projected so that the user is able to see the real world with additional information provided by the computer. - There are 2 common types of displays available. One is a pair of glasses. A small projector is attached to it so that video images can be projected onto the pair of glasses. - The other one is the flight helmet also used by the pilots of the fighter jets. A small projector is attached to the helmet so that additional information from the computer about the environment can be projected onto the transparent screen. Telepresence Telepresence links the sensors and manipulators of a remote robot to a human operator. The communication is in both ways. - Cameras, microphones and other sensors are usually attached to the remote robot. Signals from these sensors are sent to the human operator in a control room. The operator may be wearing an HMD and other devices which render the remote signals. - The human operator is also wearing some sensors which sense the movement of the operator. The signals are sent to the remote robot to control its movement. This technology is very useful in performing some dangerous operations such as putting out the fire in a nuclear plant, or in performing some otherwise too expensive operations such as Mars exploration. 8

9 Position Sensors In order to explore or to interact with a virtual world, it is important for the computer to know the user s physical coordinate. The following are some of the methods: - 2D mouse: While 2D mouse is always available, it is difficult to use in 3D world. In addition, it only returns relative (not absolute) movement of the mouse. - Mechanical tracker: A mechanical device is attached to the object. The movement of the object results in the movement of the mechanical device, which generate the positional signal. This kind of devices are fast and accurate but has restrictions on motion. - Ultrasonic tracker: An ultrasonic transmitter is attached to the object to be tracked and a few sensors are placed at different positions. By detecting the time it takes for the sensors to receive the ultrasonic signal from the transmitter, it is possible to locate the position of the object. (Some products may use a few transmitters and a single sensor to detect the location of an object.) However, ultrasonic systems are of low resolution and are susceptible to noise. - Light pattern: By putting a pattern of light transmitters to the object and cameras to the wall (or vice versa), it is possible to locate the position of the object by detecting the patterns of light captured by the cameras. Limitations are that it requires the use of the image processing technology, which Is not reliable enough, and a dedicated room. - Magnetic tracker: Magnetic tracing is one of the widely used methods. - Basically, a magnetic transmitter is placed at a fixed location and a magnetic sensor is attached to the object. - Both the transmitter and sensor contain 3 coils at right angles to each other. - The position and orientation of the sensor affect the amount of current induced to the sensor by the transmitter. - As such, the sensor can be used to detect the 3D position (x,y,z) and orientation of an object. - Very often, time multiplexing is used so that more than one sensor can be connected to a single transmitter system. - Advantage: Magnetic tracking devices are cheap and accurate. - The major limitation of magnetic tracking device is that the accuracy of these devices is affected by nearby metallic materials. - Another limitation is that it is generally believed that strong magnetic fields are harmful to human body. A Typical Immersive System The configuration of a typical immersive VR system includes a Head Mounted Display (HMD), an electronic glove, a set of position tracking devices and a graphics workstation. - The HMD contains a pair of LCD screens and a pair of optical lenses. Together they provide the user with stereo-vision. The HMD also contains a pair of speaker to provide stereo sound. - An electronic glove is a glove with a lot of sensors around it to detect the bending of fingers. - Magnetic systems are commonly used for position tracking. 9

10 - Normally, a magnetic sensor will be attached to the HMD and another sensor will be attached to the electronic glove so that the positions and orientations of the user s head and hand can be detected. - The graphics workstation is expected to generate stereo images in real-time, which are to be sent to the 2 display screens inside the HMD. - As we move our head, the 2 display screens will show images corresponding to what we are supposed to see at each particular head position. - As we move our hand, the computer will also know where our hand is in 3D space so that we can grab objects inside the virtual world. Time Lag All VR systems suffer from the time lag problem. Consider the following VR system: - When we move our head, the magnetic sensor senses the change and generates a signal. There is a delay from we move our head to the generation of the signal. - When the head tracking system receives the signal from the sensor, it processes the signal to obtain the new position and orientation of the sensor. There is also a delay in this process. - When the tracking system sends out the information, the computer receives the information, process it and then updates the database. There is a delay in this process. - After the database is updated, the image generator renders an updated image from the updated database. There is also a delay in this process. - Hence from the time the head moves to the time the image on the screen is updated constitutes to the total system lag. - This time lag causes an uncomfortable feeling to the user because we are used to an instant visual response in real life. - Some work is being done to reduce such problem by predicting what may the head be few steps ahead. This method of course has its own problem as the prediction may not always be correct. 10

11 Virtual Reality vs Computer Graphics - In computer graphics, we are concerned about the quality of the output image and we may also be concerned about the speed in generating the image. - In VR, we are also concerned about the quality of the images. However, with the limitation in current hardware technology, we are more concerned about the speed in generating the images. - In VR, in order to give the user an interactive feeling, we must be able to generate at least 10 images per second and the time lag problem must be within an acceptable level. - To be able to generate at least 10 images per second, time critical rendering becomes an important issue of study: given the time we have, how can we generate the best picture possible. - Eg., if time is not enough, we may render objects with flat shading instead of Gouraud shading. - We may also use simpler models for rendering less important objects. - Another issue of study is image coherence. - Since the time between 2 consecutive frames is very small, the change in the scene is small too. - Hence, given the information we have about the scene from the previous frame, we can make use of it and minimize the amount of calculation in generating the next image. - Eg., if an object is far away from the viewer in the last frame, it is likely that it is still far away in the current frame. Current Areas of Research - Multi-resolution modelling: This uses a simpler model for rendering when the object is far away from the viewer. This is to reduce the number of polygons needed to be rendered. - Fast rendering of surfaces: Usually curve surfaces are broken down into polygons for rendering. This area tries to speed up the polygonization process. - Fast rendering of deformable surfaces: Again, this area tries to speed up the polygonization process of deforming surfaces. - Collision detection: This detects if 2 objects collide with each other. - Visibility culling: This determines the visibility of a surface or an object. Applications of the VR Technology - Surgical training - Architectural walk-through - Telepresence - To assist the disables to communicate. - Visualization - VR games 11

12 13. Multimedia Multimedia is the integration of media technologies including sound, video, document, network and graphics with the computer technology. There are 2 major areas of interest in multimedia: - Multimedia technology: the technology itself. - Multimedia applications: the application of the technology Because multimedia covers a wide range of disciplines, we will only briefly look at this technology. Multimedia Technology The integration of media technologies may be considered as putting all the devices that handle different media together with the computer being the central controlling device as shown in the following diagram: However, when the multimedia workstation is connected to a high-speed network, it is possible to have most of the video and audio services provided through the network as in the following diagram: 12

13 - These systems allow us to watch TV, to have video-conferencing with someone in remote location, to read an electronic document, etc... - In a networked multimedia system, all these multimedia services can be provided by some central servers so that any computers in the network can have access to these services. In such a situation, a multimedia computer only needs to be equipped with a video interface to display / capture real-time images and an audio interface for sound input and output. - Under such a situation, we can also have a large central database to provide variety of information services. One of the most demanding media technologies is the handling of video. Consider color images of video quality 512x512 in resolution: - The memory size needed to store a single image, M single, is: M single = 512 x 512 x 3 Bytes = 0.75 M Bytes - Assuming that we have 30 frames per second. The memory size needed to store a second of the image sequence, M sec is: M sec = 0.75 M Bytes x 30 = 22.5 M Bytes - A movie typically lasts for 90 minutes. The memory size needed to store the whole movie, M movie, is: M movie = 22.5 M Bytes x 60 x 90 = about G Bytes For color images of movie quality, eg. resolution of 1600 x 900 (16:9) with 24 frames per second: M single = 1600 x 900 x 3 Bytes = 4.12 M Bytes M sec = 4.12 M Bytes x 24 = M Bytes M movie =98.88 M Bytes x 60 x 90 = about G Bytes With such a huge amount of memory, handling of videos digitally becomes a driving force to improve other technologies such as database (to store the images), network (to transfer the images from the server to the computer) and data compression (to compress / uncompress the images before / after transmission or storage). Other related issues include the demand for high resolution displays. Related Technologies 1. Display Technology: With windowing systems being widely used in both PCs and workstations, and the consumer community pushing for high definition television (HDTV), display technology is advancing rapidly. Monitors of resolution 1600x1280 are commonly available. At the same time hardware circuitries are also available which can read the frame buffer fast enough to update the screen near to 100 times a second. 2. Network Technology: A local computer network connects many computers together so that a small community can share resources such as disk space, printing facilities, and processing power. With most of the local networks in the whole world being connected together through the INTERNET, it is now possible to share some of the resources in a global scale. A fast Ethernet network for video transmission is required. Recent developments in this area include the 100 Mbits/s fast Ethernet network and the over 1 Gbits/s fiber optical network. 13

14 3. Database Technology In a multimedia environment, there may be many different types of data that conventional data management systems are not prepared to handle. The data objects can span an enormous range of size and format. Such a multimedia database contains a mixture of structured data (relations with fixed fields) and of unstructured data (text and images). All these factors demand for new models of data management systems which can provide the typical join and union operations for structured data queries, and unstructured text, image, and sound queries. Because mot multimedia systems are built using an object paradigm, the new models must also provide object oriented facilities for handling of data objects. All these issues are in various stages of research. 4. Image Compression The needs for fast network to transmit digital videos and for large database to store them demand for better and faster image compression methods. 5. Sound Technology There are two major issues: music synthesis and speech processing. Music synthesizing technology is becoming matured. In addition, the protocol called MIDI is also widely accepted by manufacturers as a standard for networking musical instruments. Speech technology can be divided into 2 categories: Speech synthesis and Speech recognition 6. Memory Technology A multimedia system may need to have access to a large volume of information. This demands for more RAMs and secondary storage. With the continuous improvement in memory technology, both RAMs and secondary storage devices are getting cheaper and cheaper. Multimedia Applications Applications of the multimedia technology are continuously to be proposed and developed. Some of them are: 1. Video conferencing: allows 2 or more people at different physical locations to communication with each other via voice and video through telephone or high speed networks. 2. Video services: are usually provided to users who have their machines connected to a high speed network. In a cable network environment, subscribers receive different video services from a central server(s) maintained by the cable company. 3. Video shopping: is a kind of services usually provided in a cable network environment in which the user can purchase some products simply by pressing a button on the remote controller while watching an advertisement on the cable TV. 4. Interactive environment for collaboration: is the use of multimedia technology to provide an interactive environment for collaboration through work space sharing, and video and audio communication. 5. CD-ROM publishing: is the distribution of CDROMs which store the interactive multimedia catalogues or interactive books. 14