2 About Pressure Sensing Pressure sensing is a mechanism which detects input in the interface of which inputs are sense of touch. Although the example

Transcription

1 A Framework of FTIR Table Pressure Sensing for Simulation of Art Performance Masahiro Ura * Nagoya University Masashi Yamada Mamoru Endo Shinya Miyazaki Chukyo University Takami Yasuda Nagoya University Figure 1: Result of Simulation Abstract In brushworks, slight differences of pressure between the brush and the paper influences significantly to the expressiveness of artworks. To reflect artists sensitivity and techniques, it is deemed desirable to simulate the detail factors in real artwork environments. In this study, to realize various simulated artworks on the table top interface, we produce a framework of pressure sensing based on FTIR tables. We especially evaluate the effectiveness of hand and brush drawing in the framework and reproducibility on it. Keywords: table-top interface, FTIR, pressure sensing * ura@nagoya-u.jp {myamada, endoh, miyazaki}@sist.chukyo-u.ac.jp yasuda@is.nagoya-u.ac.jp 1 Introduction There are many attempts to simulate artwork and artistic techniques with a computer; they are classified into two broad categories. One is giving priority to the output result as a work [Hertzmann 2003], and the other is giving priority to not only the result but also the reproduction of the production process [Nakakoji et al. 2007]. The latter is intended that users can simulate the production process for artworks which have difficult construction of production environments, and to provide users with practice environments of artwork which the production process is performed. For the reproduction of the production process, it is important for each technique to be expressible based on the actual production environment and metaphor of techniques, and to offer an interface that can reflect artist's sensibility. The interface that enables intuitive inputs includes table-top. Although table-top is used a lot as the interface of the cooperative working system, it would appear that tabletop is effective as the interface in simulation of artwork because many production environments of artworks are flat. However many table-top applications are not appropriate for artwork that has delicate difference of input which influences a work because they are designed to use tips of fingers and pens as input interface. Therefore it is necessary to achieve input equal to reality to simulate artwork production by using tabletop. In this study, we propose a framework to realize pressure sensing on table-top which used Frustrated Total Internal Reflection (FTIR). In this framework, we realize to support production techniques in various artworks by treating input shapes as pointers which can be tracked. We evaluate the effectiveness of the proposed method and reproducibility of artwork by developing simulators of artwork with hands and brushes using the framework as shown in Fig

2 2 About Pressure Sensing Pressure sensing is a mechanism which detects input in the interface of which inputs are sense of touch. Although the example in which this is used is the device like Phantom that can obtain force feedback, the cases of pressure sensing that uses tabletop are reported [Kamiyama et al. 2005]. Thus, though purpose-built hardware is generally used for pressure sensing, smith and his colleagues propose a method of realizing the pressure sensing at a low price by putting silicon on FTIR table [Smith et al. 2007]. In addition, they develop the paint tool that can input various shapes such as hands and brushes as a demonstration of this interface. However, because the input shape is reflected in drawing without modification, the input cannot be tracked. Therefore, because the supplementation processing is not executed when the distance of the input is large, the intended result might not be obtained. it influences the perfection level of the work if the input is not appropriately reflected, because The difference of a slight input appears as a difference of power of expression in the art work. Therefore, it is necessary to treat input shapes as pointer which can be tracked to correspond to various art productions. 3 Pressure Sensing by FTIR Table We develop an environment to realize pressure sensing which consists of two or more tracking pointers. 3.1 Making of a FTIR Table Therefore we propose a method that recognizes shapes of hands which touched on the table from images that can be acquired in the interface as pointers, and enabling two or more pointing and tracking pointers. The steps are described in the following: 1) Binalize and labeling the inputted image from the camera, and extracting area pixels and outline pixels from the image. 2) Calculating the mean values of outline pixels of each area, and setting the values to a central point of pointers. 3) Comparing the position of a center point of each pointer with the position of each pointer of the last frame, and processing the one to which the distance is shortest as the same pointer. 4) Giving new ID to a new pointer. As a result, the tracking of the pointer is enabled [Hara et al. 2009]. The parameters of these pointers are maintained by the structure shown in Fig. 3. Figure 4 shows an example of recognizing the pointer. The left figure is the image before processing and the right figure is the image after processing. In the right figure, gray is the surface of contact, white is the outline and the number is the ID of the pointer. struct pointer { int id; // pointer ID list<pixel> area; // list of area pixels list<pixel> outline; // list of outline pixels point center; // central point }; Figure 3: Structure of the Pointer We use FTIR to realize contact sensing that acquire the shape grounded to the table as inputs. By using property of total internal reflection of infrared rays, gray scale images that parts where hand touch on the table is white and the other is black can be obtained from a camera. Moreover, by putting a film of thin silicon to the acrylic board with Smith's proposed method, the change of input shapes according to the pressure of contact can be obtained. Figure 2 shows the hardware configuration of the FTIR table. Silicon Infrared Ray LED Acrylic Board Infrared Ray LED Figure 3: 4: Example of Recognizing Pointer Projector (mirror-reversed projection) Figure 2: Configuration of FTIR table 3.2 Making of a Framework PC Camera (only infrared rays pass through) One of the libraries for constructing multi-touch environment is Touchlib [David]. It can recognize pointing by two or more fingers and track these pointers. However it treats all the input shape as a circle; therefore, when inputting various shapes by using various tools such as hands and brushes are assumed, there is a limit to reflect input. On the other hand, though the touched area can be treated as inputted shape in the case of Smith's method, there are still problems such as no appropriate reflection of input when the speed of operation exceeds the frame rate of the camera input because the pointer is not tracked. In the FTIR table, because it is difficult to match coordinates of the projected image of the projector and the acquired image of the camera completely by arranging the equipment, the calibration is executed by software processing as input and output are corresponding. When input is treated as a circle like Touchlib, the input shape can be treated by the calculating formula by information on a central point and radius. Therefore, calibration can only be executed by processing projective transformation to central coordinates of each input. In this case, computational complexity is projective transformation of the number of times of pointers. On the other hand, when using the inputted shape as the shape of the pointer like the proposal method, if only central coordinate is converted like Touchlib and the inputted shape is arranged at the center, the more the distortion of the camera and the projector, the more the difference between the inputted shape and the outputted shape; therefore, it is necessary that the pixels of the input shape are targeted of the projective transformation to match the shape. However, it is clear that the computational complexity increases several times compared with the case of a round pointer, and this become a factor to generate the delay in processing. 119

3 Then, we reduce computational complexity by using the texture function which is provided in 3D graphics library. Figure 5 shows an example of calibration. A gray quadrangle in the left figure shows the acquisition area in the camera, and a black quadrangle shows the projection domain of the projector. The texture is set by conforming coordinates of four corners of acquisition area to coordinates of four corners of projection domain like the right figure. As a result, a stable processing speed can be expected though the accuracy of the image is less than the actual resolution. Camera Artist Sand Canvas Light Figure 6: Environment of Sand Animation Screen Audience Simulation of Artistic Techniques 4 Simulation of Artworks We simulate artworks on the produced environment. The simulated artworks are Sand Animation which produces animation by operating sand by hand, and Rainbow Art which can be painted colorful strokes like a rainbow with brushes. In these cases, we confirm reproducibility and effectiveness of the proposed method by comparing a simulated artwork with an actual artwork and measuring the processing speed in each case. The production procedures of works are provided, and the works are produced based on the storyboard made beforehand. The experiments are executed on the environment shown in Table 1. OS CPU Memory Graphic Card Camera 4.1 Sand Animation Figure 5: Calibration by using Texture Table 1: Environment of Experiments Windows Vista Businnes Intel Core2 Duo E GHz 4GB ATI Radeon HD 4800 (RV770) Frame rate: 30fps, Resolution: 640x480pixel Many production techniques of Sand Animation are executed by touching the hand to a tabular canvas. In this chapter, we survey the production environment and techniques from the works of Ferenc Cako who is a Sand animation artist [Cako], and simulate the behavior of sand and the production techniques based on the result [Hara et al. 2009] About Sand Animation Sand Animation is the name of an art technique for producing animation work with sand and the performance art of the production process. It can produce a unique warmth expression such as sepia tone by using sand and light. Its works are produced with the environment showing in Fig. 6. The artist produce works by operating sand on a canvas. In the case of showing the production process as performance art, a picture of the canvas is taken with a camera, and the image from the camera is projected to the screen. There are several drawing techniques for production in Sand Animation. We reproduced the behavior of sand in each technique based on the result of surveying Cako s works. The movement or removal of arranged sand, additional sand, and so on are divided into three "Put", "Sprinkle" and "Shave", and simulated respectively, because there are several production techniques to compose animation Put: Holding sand in hands, and drawing lines and points like putting it on canvas Sprinkle: Painting a certain area by sprinkling sand to canvas by separating hand and canvas. When inertia is large, sand flies widely, and becomes an expression with speed and power. Shave: Shaving arranged sand by fingers and palms. The backlight leaks from the space made after shaving, and a clear line is expressed by the contrast. In actual Sand Animation, hands do not touch the canvas when executing Put and Sprinkle. However, because the pointer cannot be recognized if hands do not contact the FTIR table, these operations are inputted by touching hands to the table at the present stage. Moreover, we treat sand by its height field which each cells manage sand height in simulation. Thickness and texture are expressed by changing the drawing in proportion to the amount of sand. The color of the state without sand - the color of amount 0 of sand - is RGB (255,240,120), and the color when the backlight is intercepted in the amount of 100 or more sand is RGB (40,0,0). In addition, we use dithering method to express the texture of sand. In this method, threshold is set to each pixel of the field at random beforehand, and sand variability is expressed by improving brightness when the amount of sand in the height field is less than the threshold. (1) Simulation of Put Adding only the amount of sand by formula k r l e 1 in the circle the radius is r and the center is the pointer's central point p. k 1 is a controlling constant, l is a distance of targeted pixel and p, e is a random numbers value. Moreover, a natural drawing which amount of sand is not the same is obtained by increasing and decreasing the added amount of sand at random. (2) Simulation of Sprinkle We separately treat the process of flying and sprinkling sand. First, calculating the distance l with the central point p of a pointer in the current frame and the central point q of a pointer in the previous frame. Then, setting constant distance t. If l is within the range of t, assuming there is no inertia force and processing of sprinkle is executed. This processing arrange sand more widely and more thinly in the circle than Put The supplementation process is executed if distance l is large. 120

4 (3) Simulation of Shave This technique moves arranged sand to the traveling direction of hands. Therefore, we realize the simulation by processing of moving sand on area pixels of pointer to outline pixels of pointer. First of all, if the amount of sand of each pixel stored in the area list area of the pointer is v or more, only v decreases the amount of sand from the pixel. If it is smaller than v, the amount of sand is adjusted to 0. Then, decreasing s which is the total amount of sand, and adding the sand amount s / n ( n is a number of outline pixel) to each outline pixels. Because sand is generated in the outline on the other side's traveling direction in each process when the distance is large, sand is not distributed to an outline pixel that overlaps with the pointer area in the previous supplementation point Execution Example Figure 7 shows Sand Animation produced based on the storyboard, and Figure 8 shows "Shave" is executed in simulation. The left figures of Fig. 7 are works by simulation, and the right figures are works by actual sand. There is a difference in background color because one is display capture and the other is an actual photo of sand; however, it confirms that the work that used various techniques is drawn similar to the same drawing. Figure 8: Example of Simulating Sand Animation Processing Time We measure time from touching a table by hand to the recognition of these as pointers in each operation of Put, Sprinkle and Shave, and time until the processing of one frame is completed. As a result, time required to recognize the pointer is 500 ms on average, and this is equal to the delay generated when the camera displays the acquired image; therefore, the processing time of the pointer recognition is small enough. In addition, execution time is smaller than 1/30s that is the update interval of the frame in 30fps in each operation as shown in Table 2. For these reasons, we confirmed processing in simulation has no delay. Table 2: Processing Time condition dimensions (px) Processing Time (ms) Put Sprinkle Shave Finger Five Fingers Side of Hand Palm Two Palms Rainbow Art We show an example to simulate artwork which uses a tool for drawing. For the confirmation of the reproducibility of details, we simulate Rainbow Art that painters can draw colorful pictures such as a rainbow About Rainbow Art Figure 7: Produced Sand Animation based on the Conte Rainbow Art is a drawing tool for kids designed by Laszlo Szekely [Szekely 1994]. The works of it is produced by painting paper with platy sponge brushes. The sponge brushes are oblong, and lines like rainbows can be drawn by putting two or more colorant to the brush. As a result, everyone can draw colorful pictures easily. A shape that can be drawn is decided to some degree on the characteristic of the brush, and a lot of works are produced in combination of the sector, quadrangle, and the straight line. The sizes of brushes vary, and can adhere up to six colors maximum. Not only one person but multiple people occasionally produce work on the same canvas as a recreation. Figure 9 shows the tool and the usage example. 121

5 Figure 9: The Tool and the Usage Example of Rainbow Art Simulation of Artistic Techniques A brush has characteristics which colorant is prolonged and a surface is drawn by moving the brush head which colorant is adhered to on the painted surface by keeping contact. Therefore, in the case of broad brush used in rainbow art, the interpretation that the movement stroke becomes surface is approved when considering the end points of the brush head are connected mutually to be one line. Moreover, it can be thought that the line is composed by not a single color like a usual brush but multiple colors in Rainbow Art. Then, by treating the input as a line by using a central point and an outline of the input acquired by the framework, we realize drawing on tabletop with the brush. First of all, setting the contour point p which is furthest away from central point c in the inputted area as the end point of brush. Next, q which is symmetrical to p in central point c is set as another end point of the brush, and segment l which has p and q as both ends is considered to be a contact line of the brush. Moreover, to enable tracking, setting both end points p and q of segment l as markers, and treating markers which the distance from previous frames coordinates to current coordinates is the shortest as the same marker. As a result, it is possible to keep information of the brush's direction even if the position and the angle of the brush are changed. Figure 10 shows a drawing of brush strokes according to marker's movement. By considering drawn surface by moving the brush as quadrangles from two points of the previous frame's pointer and two points of the current frame's pointer, stroke is expressed. The stroke is expressed by these consecutive quadrangles. Figure 11: Generating Coloring by using Gouraud Shading Execution Example Figure 12 shows Rainbow Art produced based on the storyboard, and Figure 13 shows work produced in simulation. The left figures of Fig. 12 are works by simulation, and the right figures are actual Rainbow Art works. From these figures, it is confirmed that coloring and shapes are of the same drawing in each work. In addition, processing speed of simulation is less than the frame update interval as well as the simulation of Sand Animation. These results show the effectiveness of simulation. However, paying attention to detail, a blur is caused in the drawn stroke in the simulator. It is thought that putting pressure to the support surface is not constant unlike actual production, because both silicon of the interface and the sponge brush absorb pressure. Moreover, in actual production, because the color adheres to the brush, the color can be confirmed by watching when newly drawn. On the other hand, the intended drawing might not be obtained in simulation, because direction information of the brush is lost when moving the brush away from the table. Figure 10: The Tool and the Usage Example of Rainbow Art Although the colorant used for the rainbow art is similar to watercolor pigment, it becomes a gradation color by mixing with the adjoined color that adheres to the brush. Therefore, peculiar coloring of rainbow art is reproduced by using gouraud shading. Figure 10 shows the example of drawing a brush stroke that consists of three colors. Segment pq represents the contact area of a previous frame and segment p q represents the contact area of a current frame. In this case, because the contact area is composed of three colors, the turning point of the color is segment mm that consists of midpoint m of segment pq and midpoint m of segment p q. Then, the brush stroke is drawn by combining quadrangle pmm p and mq q m. As a result, a gradation which is smooth and without irregularity is expressed. Figure 12: The Tool and the Usage Example of Rainbow Art 122

6 Figure 13: The Tool and the Usage Example of Rainbow Art 5 Conclusion In this study, we produced the framework for pressure sensing which enabled tracking of a pointer based on FTIR table. We also confirmed the reproducibility and effectiveness of it by simulating actual artworks by using the framework. Acknowledgments I would like to thank Marvin Lenoar for his constant support. This work was supported in part by Grant-in-Aid for Scientific Research from the Ministry of Education, Culture, Sports, Science and Technology of Japan, research grant from Kayamori Foundation of Informational Science Advancement, research grant from Artificial Intelligence Research Promotion Foundation, and research grant from The Hori Information Science Promotion Foundation. References HERTZMANN, A A Survey of Stroke-Based Rendering. IEEE Computer Graphics and Applications, vol.23, no.4, NAKAKOJI, K., JO, K., YAMAMOTO, Y., NISHIHARA, Y., AND ASA- DA, M Reproducing and Re-experiencing the Writing Process in Japanese Calligraphy. In Proceedings of 2nd IEEE Tabletop Workshop, IEEE Xplore, SMITH, J. D, GRAHAM, T. C. N., HOLMAN, D., AND BORCHERS, J Low-Cost Malleable Surfaces with Multi-Touch Pressure Sensitivity. In Proceedings of 2nd IEEE Tabletop Workshop, IEEE Xplore, KAMIYAMA, K., VLACK, K., MIZOTA, T., KAJIMOTO, H., KAWA- KAMI, N., AND TACHI, S., Vision-Based Sensor for Real- Time Measuring of Surface Traction Fields. IEEE Computer Graphics and Applications, vol.25, no.1, WALIN, D. Touchlib: an opensource multi-touch framework. CAKO, F. Sand Animation. HARA, K., URA, M., YAMADA, M., ENDO, M., MIYAZAKI, S., AND YASUDA, T An Artistic Technique of Sand Animation and Its Simulation. In Proceedings of NICOGRAPH HARA, K., URA, M., YAMADA, M., ENDO, M., MIYAZAKI, S., AND YASUDA, T Framework of Pressure Sensing by using FTIR Table and Its Application to Art Simulation. Technical report of IEICE, vol.109, no.466, SZEKELY, L US Patent No. 5,318,