An Intelligent System for Overlaying Texts on Background Images Based on Computational Aesthetics

Transcription

1 An Intelligent System for Overlaying Texts on Background Images Based on Computational Aesthetics Chien-Yin Lai, Sheng-Wen Shih, Jen-Shin Hong National Chi-Nan University Department of Computer Science and Information Engineering 1, University Rd., Puli, Nantou Hsien, 545 Taiwan Tel: (+886) ext. 4010, Fax: (+886) Abstract A computational model on the aesthetic appeal of colored text-overlaid images is proposed and experimentally investigated. Five photos were used to compose a set of test images overlaid with a paragraph of Chinese texts as the stimuli. The positions of the text overlay were determined by varying the balance in order to validate computational aesthetic quantification algorithms with subjective ratings. The stimuli were rated by 10 subjects in an experiment using the ratio-scale magnitude estimation method against a benchmark image for each photo. The experiment established a relationship between a higher averaged visual balance and the aesthetic appeal of text-overlaid images. Based on the result, we have implemented a prototype system that compute the optimal position for automatically overlaying a paragraph of texts on a given background image. Keywords: computational aesthetics, interface design, compositional attributes

2 1. Introduction Researchers and practitioners have begun to emphasize considerations of aesthetics in interface design and to investigate its impact on the overall experiences of users when interacting with a computer system. Evidence in support of the importance of aesthetics in various aspects of computing has emerged recently. For example, recent empirical studies suggested that the aesthetic aspect of various computing systems serves an important role in shaping user attitudes in general (e.g., Hassenzahl, 2004; Karvonen, 2000; Lindgaard and Dudek, 2002, 2003; Tractinsky, 1997; Tractinsky et al., 2000; etc.). It was found that subjective evaluations of usability and aesthetics are correlated depending on the user s background, goal, task, and application type (Ben-Bassat et al., 2006; De Angeli et al., 2006; Hartmann et al., 2007). In the context of multimedia web page design, aesthetics was also found to be an important determinant of user preferences of the sites. For example, Schenkman and Jonsson (2000) found that beauty was a primary predictor of overall impression and preferences of web sites, while van der Heijden (2003) linked perceived attractiveness of a website to the usefulness, ease of use and enjoyment perceived by users. Overall, these previous studies illustrate the complexity and intricacy of the issues involved with respect to the effect of aesthetics in visual interfaces of general information systems. Although the exact mechanisms that link the affective and cognitive evaluations of the visual interfaces are not yet clear, it is certain that aesthetics has played a non-negligible role in general visual interface designs for computer systems. The type of visual interface addressed in this study is the text-overlaid image which usually consists of a large-size background image with a small number of texts overlaid on it. An example application is an online greeting card (or postcard) service that dynamically positions user-input texts over a selected background image. In such applications, there is a need to automatically overlay the texts on the pictorial image because the image and texts are generally randomly selected and composed together on-the-fly. Existing services of this kind provide only a small number of style templates upon which the user s message is positioned at a pre-determined location on the background image. Another example is the dynamic text formatting used in PowerPoint-like slide presentations. When a user switches a background image for the slides, it is preferred that the layout of the texts in a slide can be automatically reorganized according to the selected background image. Applications using this kind of interface typically are graphic-intensive multimedia applications that attempt to use the background images to generate appropriate affects. To consider the overall aesthetics of the final presentation, there is a need for appropriate computation models to measure aesthetics. To address this issue, this study was designed to develop a quantitative metrics of perceived aesthetics of text-overlaid images. Such metrics of aesthetics can be potentially applied in an optimization engine that helps to dynamically compose multimedia presentations on-the-fly, based on optimization algorithms that maximize the objective functions of aesthetics. Research on visual interfaces with aesthetic considerations has traditionally defined visual aesthetics using various qualitative design principles but without knowing quantitatively where an optimal design space exists. Indeed, there have been only a small number of studies that aimed to develop computation models for the aesthetics of visual interfaces. Bauerly and Liu (2006) proposed computation models of symmetry and balance. Evaluation experiments conducted showed that symmetry is closely related to the aesthetic appeal of pages with abstract black image blocks. Contrarily, the number of image blocks was shown to be the main composition element related to the aesthetic appeal of pages with realistic images. Overall, previous studies aiming to develop computation models of visual aesthetics of layout designs focused on pages composed of large chunk of texts interlaced with a small number of relatively small-sized images. This study, in contrast, considers pages where small

3 amounts of text are to be overlaid upon large-sized color background images. Previous computation models of aesthetics are not applicable in such a context mainly because they did not consider the color information for objects in a page. For developing an intelligent system for composing texts and background images, there is a need to develop a computational model to measure the overall aesthetics of text-overlaid images. As the first step towards this goal, this study mainly concerns the visual balance that we believe play the major role in the visual aesthetics of text-overlaid images. In the following sections, we first describe the computation model for balance based on pixelated and segmented images. Then we investigate the relationships between the visual appeals of text-overlaid images with the computed values of balance. Finally, a prototype intelligent image composing system is described. Conclusions and future work are addressed finally. 2. Image quantification method The algorithm for computing balance was designed in an attempt to mimic the human cognitive representations of the attributes of balance. In the following we describe essential principles for developing this model that is based mainly on the concept of visual weight. Arnheim (1974, 1988) described a formally balanced composition as a perceptual phenomena based on balancing the visual weight of objects in a page. Several major factors such as size, position, contrast, shape, and texture, were qualitatively shown to affect the visual weight of an object. One of the most important factors is size, for example, other factors being equal, the larger the object the heavier it is perceived to be. Another important factor is distance to center axis : an object near the center can be counterbalanced by smaller objects placed off-center. In addition, the contrast of an object against the surrounding region also significantly affects its visual weight. In principle, the contrast of an object generally depends on the colors or textures of surrounding objects. For example, a red spot within a green region is visually heavier than one within a pink region; a stick in a pile of sand is visually heavier than one within a pile of wood. As the first step towards a comprehensive model for computing visual balance, this study considered only color contrast of the objects to calculate the visual weight. The modeling of texture contrast is currently underway. The present study considers pages consisting of a large-size background image overlaid with a small number of texts. There are a number of issues of concern with respect to the calculation of the visual weight of the color pixels in such a context. First of all, an image that can feasibly be used as a background image in applications of this kind usually has a relatively homogeneous and spacious background region within which the texts may be overlaid. Since the background region typically does not attract visual attention, the original pixels in the background region should be considered to have a null visual weight and thus discarded in the computation for balance (Fig. 1). Secondly, the visual weight of a pixel in the object regions should be proportional to its color contrast to the background region (note that texture contrast is not considered in this study). A pixel with high color contrast against the background region is heavier than one with low contrast. For example, given a blue background region of an image, a yellow pixel is considered to have a larger visual weight than a dark green pixel in the same position. Based on this principle, the visual weight of a pixel in object regions is based on its size, distance to a center axis, and color contrast against the color of the background region. Later in this section we will propose a computationally-efficient approach to computing the color contrast between two pixels based on their color distance. Thirdly, in a text-overlaid image, the texts are perceptually quite different from other bitmap pixels. In other words, pixels of the text regions present a higher contrast against the background region than non-textual pixels of the same color at the same position. Thus our

4 model considers each character of text as a rectangular block with dimension and color equal to those of the character (Fig. Fig. 1). Fourthly, since it is not clear how sensitive is human perception to subtle le variations on the positions of the objects, our computation model for balance is based on the pixel blocks of a pixelated version of the text-overlaid overlaid image. In our experiment we compute balance based on different granularities of pixelation operations operation on the text-overlaid images.. Computational measures of balance based on different pixelation levels are compared to subjective ratings to determine an optimal method of pixelation for our application. Fig. 1 gives an example of an original text-overlaid overlaid image and its pixelatedd version on the basis of which balance is computed. overlaid image Fig. 1. An example of a text-overlaid (left) and a pixelated version of it (right) in which the detected background region has been erased. We propose a computational model to quantify the balance of a text-overlaid overlaid image. The model is based on several essential operations including image segmentation, background region determination, image pixelation by down-sampling sampling in the spatial domain, color differences calculation,, and balance calculation and normalization. The following sections discuss the details, principles, and approaches for these operations Background region detection by image segmentation techniques Inn the application addressed in this study, a target et image usually has an obvious background region such that the text can be overlaid without occluding the main subjects in the image. The background region of this t kind of image is relatively easy to identify using conventional image segmentation algorithms. algorith techniques segmentation In computer vision techniques, refers to the process of partitioning a digital image into multiple regions (sets of pixels) to simplify its representation into something that is more meaningful and easier to analyze by computer. Each of the pixels in a region is similar with respect to some characteristic or computed property such as color, intensity, or texture. Adjacent regions are significantly different with respect to the same characteristic(s). The result of image segmentation is a set of regions that collectively cover the entire image. Currently, there is no segmentation algorithm that can perfectly match human perception in wide contexts. We chose the algorithm developed in Comaniciu and Meer (1997) because it is easy to implement, computationally efficient, and does not require prior supervised learning. These matters are crucial to real-life real applications that require dynamic composition of text-overlaid overlaid images on-the-fly. on Once the image is segmented by the proposed algorithm, the largest connected region is defined as the background region. Pixels of the original image in the background region are considered to have a null visual weight and are ar discarded in the balance computation. computation Texts are placed inside the detected background region Overlay texts on the selected background region The texts should be appropriately positioned and overlaid on the background region to avoid possibly occluding the main object(s) of a given image. The font size and the color of texts used for a given image are selected based on the concerns for readability readabili and overall aesthetics. To color the texts, among the main colors of the original image we select the one with the highest contrast to that of the background region. Such an approach usually generates a final text-overlaid text image with rather harmonious color olor combinations that generally appears aesthetic appealing. In addition, a shadow effect is superimposed on each character of the texts to improve readability (Hall and Hanna, 2004). Fig. 1 shows an example of a photo with a paragraph

5 of shadowed Chinese characters overlaid on the background region Pixelations of the text-overlaid images In our application, there are two main concerns that call for an image resolution reduction in the model. First, because the target applications using layout optimization engines usually require real-time computations of the aesthetics measures, it is preferable that the computation model is based on a resolution-reduced image instead of the original one. Second, the typical resolution of a digital color image displayed on a current LCD monitor is around 72 pixels per inch. It is uncertain whether human perceptions of balance are sensitive to minute variations of the visual object positions at the pixel level. Therefore, this study proposes a computation model based on pixelated images with reduced image resolution. Pixelation is a common image-manipulation technique by which an image is blurred by displaying it at a markedly lower resolution. Usually it is done by a spatial down-sampling that reduces image resolution while maintaining image dimensions. The pixelated image becomes rather blocky. There are different ways to combine the pixels to achieve the desired image resolution. We apply a most intuitive and computationally-efficient approach by filling each pixel block with the average color of all the pixels in it. Balance is calculated based on the pixel blocks in the pixelated images Color contrast based on color distance in HSV space We propose a computation model of balance based on quantifiable color contrast measured by the color distances of the pixel blocks. Following is a description of the main principles for computing color distances based on the hue, saturation, and value of a color. A color space is defined as a model for representing color in terms of different numbers of values. The HSV model is known to correspond closely with the way humans describe and interpret color and offers much better perceptual uniformity than RGB (Paschos, 2001). The HSV color difference space is a three-dimensional space with approximately uniform visual spacing in terms of color difference judgments. Two colors with a large distance in the HSV color space appear quite differently to human eyes. Therefore, in this study, the computation of the color distance of two pixel blocks is based on the Euclidean differences between two colors in the HSV color space. The three-dimensional HSV color coordinate system is a cylinder. It is divided cylindrically by having hue form a circle. Hue varies from 0 o to 360 o while the corresponding colors vary from red through yellow, green, cyan, blue, magenta, and back to red. Saturation varies from 0 (unsaturated) to 1.0 (fully saturated). Value varies from 0 to 1.0 with the corresponding colors becoming increasingly brighter. The color difference between two colors, (H 1, S 1, V 1 ) and (H 2, S 2, V 2 ), is generally given as (Smith and Chang, 1996): C= 1 5 V 1 V (S 1 cos H 1 S 2 cos H 2 ) 2 + (S 1 sin H 1 S 2 sin H 2 ) 2 (1) A human perceives quite differently two colors with a C close to 1. In contrast, the C of two perceptually-similar colors is close to 0. Therefore, a pixel block with a large C against the color of the background region should appear to be more contrasted and visually heavier than one with a small C Computation model of balance Following Bauerly and Liu (2006), we propose the following formulations for the normalized horizontal balance (B H ) and the vertical balance (B V ). The balance point, b, is the Cartesian coordinate of the center of the visual weights of all pixel blocks in a given image. In other words, the balance point is the one for which the summations of the visual weight of all pixel blocks in each direction (horizontal or vertical) are equal. For a black-and-white image, the visual balance can easily be found by assigning each pixel block a constant visual weight. For a color image, the visual weight of a color pixel block is proportional to its distance from the center axis and to its color contrast (i.e., the color distance) against the background region.

6 A color block with a large color distance from the background color has a larger visual weight than one with a smaller color distance. The HSV color difference between a color block at (i, j) with a color of (H ij, S ij, V ij ) and the background color (H B, S B, V B ) can be computed based on the Eq. 1 in section 2.4. Notations: x b : y b : the x-coordinate of the balance point b the y-coordinate of the balance point b w: width of image h: height of image C ij-b : the color difference between a block at (i,j) and the background W i : summation of the visual weights of all color pixel blocks in the i th column W j : summation of the visual weights of all color pixel blocks in the j th row h = j=1 C ij-b (2) w = i=1 C ij-b (3) w i=1 i x b =0 (4) h j=1 j y b =0 (5) B H = 1 2 x b 1 (6) w B V = 1 2 y b 1 (7) h According to the above formulations, B H and B V range between 0 and 1. The closer to 1 the value of B H (B V ), the better the horizontal (vertical) visual balance obtained Methods Participants Ten subjects participated in the evaluation experiment. All subjects had normal or corrected-to-normal vision and normal color vision. The subject population (mean age 23.3 years, range 20-29, 7 male and 3 female) was composed of graduate students of the Department of Computer Science and Information Engineering of National Chi-Nan University, Taiwan. The experimental procedure for each subject typically took 45 min to 1 h Stimuli Five photos, depicted in Fig. 2, comprised the stimuli (test images) of the experiment. Each of the five photos contains a clearly perceptible background region that is sufficiently spacious for placing the given texts in various distinct positions. For each photo, five different positions in the background region were selected to overlay a paragraph of Chinese characters, generating five test images with varied values of balance. The positions of the texts in the test images were designed to leave at least a reasonable margin to the photo edges and the boundaries of the main objects in the photo. 3. Experiment An experiment was conducted to determine whether there is a relationship between the computational metrics and subject ratings of aesthetic appeal of text-overlaid images. Realistic photos overlaid with a paragraph of shadowed Chinese texts were shown to human subjects. Numerical values representing balance were computed based on pixelation with pixel blocks each measuring 5x5 pixels. Fig. 2. The photos used in the experiment and their corresponding segmented images. The background region for each photo was removed.

7 Fig. 3. All the stimuli (test images) applied in the experiment. The benchmark image for each photo is marked by the red lines. For each photo the test images are ranked by the average aesthetic scores given by subjects in the evaluation experiment. For each photo, we assigned one test image, which was considered to have a moderate value of aesthetic appeal, as the benchmark image. Other test images were compared to the given benchmark image. Fig. 3 lists all the test images for each photo, ranked by the subjective aesthetic score obtained in the experiment described below. The photos and texts were shown to the subjects in advance so as to reduce the influence of the image contents on the evaluation Procedure The experiment was conducted in a well-lit room. Each participant sat at a desk and viewed all test images on a 22-inch LCD monitor at 1680x1050 pixel resolution, with all test images measuring 600x400 pixels. An image presentation system, written in ASP.NET, was implemented to display the test images and record the scores given by subjects. Subjects rated the aesthetic appeal of the test images of each photo compared to the benchmark image. Each test image was displayed on the LCD screen next to the benchmark image. The presentation sequences of the test images were randomly ordered. The rating method applied is the magnitude estimation method. Subjects were instructed to rate the aesthetic appeal or balance of each test image against a benchmark image that was rated as a 10. For example, if the test image is twice as appealing as the benchmark image, it is rated as a 20. Higher ratings correspond to higher degrees of balance or aesthetic appeal. The subjects were instructed to make their judgments solely on the basis of the overall layout of the text-overlaid test images, not on the content of the photos or the Chinese texts.

8 In addition, subjects were asked to perform their evaluations based on the initial quick impressions on the test images. This procedure is based on the investigation addressed in Tractinsky et al. (2006), which provided evidence in support of the premise that aesthetic impressions of web pages are formed quickly Results and discussions We investigated the relationships between the subjective ratings of aesthetic appeal and computational measures of balance. Fig. 3 ranks the subjective aesthetic scores of the test images of each photo, with the least aesthetically appealing image shown on the left and the most aesthetically appealing one on the right. Quantitative analyses were conducted to investigate the possible relationships between the subjective aesthetic ratings and computational measures of balance. Table 1 summarizes the adjusted R 2 and β Coefficient values for each photo. Table 1. Adjusted R 2 values of linear regression functions for the subject ratings of aesthetic appeal and computational measures based on visual balance. Aesthetic rating vs. computation measures B V B H B avg Photo 1 Adjusted R p-value (**) 0.004(***) 0.024(**) β Coefficient Photo 2 Adjusted R p-value (***) β Coefficient Photo 3 Adjusted R p-value (***) β Coefficient Photo 4 Adjusted R p-value (**) 0.014(**) β Coefficient Photo 5 Adjusted R p-value (**) β Coefficient * p < 0.1, ** p < 0.05, *** p < 0.01 direction does not show a linear relationship with the subjective aesthetic scores. However, from observation of the aesthetic rankings for each of the test images for each photo, it appears that the top ranked images typically have better overall balances. Therefore, it was hypothesized that the balance of an image may have a logarithmic relationship to its aesthetic rating. Indeed, in the results, the high values for the adjusted R 2 and positive β Coefficient indicate that average balance, given as Bavg= ( BV + BH ) / 2, appears to be positively proportional to the aesthetic ratings. Based on this result, we propose a heuristic aesthetic measurement (A) based on the averaged balance, given as: LOG( A) = β B + β (8) 1 avg 0 where β 1 is the coefficient of linear regression equations. In practice, the optimal position to overlay texts on a background image would be that which gives the largest B avg. 4. A prototype system for composing text-overlaid images Based on the above analysis of the balance-based computation aesthetics of text-overlaid images, we have implemented an image composing system which automatic chooses the optimal position in a given image to place a give textual paragraph. Fig. 4 shows the overall framework. There are several essential modules in the overall framework. First, the background extraction module detects the background region of the image using the segmentation-based approach described in section 2.1. Secondly, the text coloring module determines the color of the texts described in section 2.2. Results for B V and B H show that the adjusted R 2 and β Coefficient varies significantly across different photos. It appears that the computed balance measure in either vertical or horizontal

9 Fig. 4. The system framework for composing text-overlaid images based on computational aesthetics. Finally, the optimization engine calculates the best position with which the best average visual balance of the overlaid image is achieved. Note that the texts are only allowed to place in the background region of the image are aligned with the boundary of the objects when approaching the objects. The maximum number of characters in one line was set to 25 to avoid too lengthy lines. In this initial attempt to implement the optimization engine, we chose a naïve approach based on exhaustive search to the best position among all the candidates positions. Fig. 5. The most and least aesthetic images among the composed images by the prototype system. Fig. 5 lists the output of the system for the most/least aesthetic text-overlaid images for all the selected photos. For example, for the fourth photo, the main object of the photo is the leaf lying on the left bottom corner. When the texts are placed on the left side, the computated average balance is clearly poor and thereby the aesthetic value is poor accordingly. Contrarily, when putting the text on the right-upper corner, the average balance is good and so is the aesthetic quality of the image. 5. Conclusions and future works We created methods to quantitatively analyze the aesthetics of a text-overlaid image such that a best position for overlaying texts on a

10 background image can be obtained automatically. Results from the experiment show a strong relationship between the averaged balances and overall aesthetic appeal was shown in the experiments. This was reflected in the higher ratings of those more-balanced test images for each photo used in the experiment. The findings on correlation between the aesthetic appeal ratings and computational measures of the average balance of the pixelated image can be directly applied in applications that require an optimization engine for automatically composing a text-overlaid image. In principle, the optimal position for text that has a high probability of making those images more aesthetically appealing would be the one that gives the largest Bavg= ( BV + BH ) / 2. The initial results obtained by our prototype optimization engine show a promising future. Immediate directions for future work include 1) investigate the roles of other composition elements (e.g., visual symmetry) on the overall visual appeals for text-overlaid images, 2) investigate the optimal levels of pixilation for computing different composition elements, and 3) develop advanced algorithms to speed up the optimization engine such that real-life online applications for image-text composition applications can be achieved. References Arnheim, R., Art and Visual Perception. University of California Press, Berkeley. Arnheim, R., The Power of the Center. University of California Press, Berkeley. Bauerly, M., Liu, Y., Computational modeling and experimental investigation of effects of compositional elements on interface and design aesthetics. International Journal of Human-Computer Studies 64(8), Ben-Bassat, T., Meyer, J., Tractinsky, N., Economic and subjective measures of the perceived value of aesthetics and usability. ACM Transactions on Computer-Human Interaction 13(2), Comaniciu, D., Meer, P., Robust analysis of feature spaces: color image segmentation. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Puerto Rico, June De Angeli, A., Sutcliffe, A., Hartmann, J., Interaction, usability and aesthetics: What influences users' preferences?. In: Proceedings of the 6th ACM Conference on Designing Interactive Systems, PA, June Hall, R.H., Hanna, P., The impact of web page text-background colour combinations on readability, retention, aesthetics, and behavioral intention. Behaviour & Information Technology 23(3), Hartmann, J., Sutcliffe, A., De Angeli, A., Investigating attractiveness in web user interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, California, April 28 May Hassenzahl, M., The interplay of beauty, goodness, and usability in interactive products. Human-Computer Interaction 19(4), Karvonen, K., The beauty of simplicity. In: Proceedings of the 2000 ACM Conference on Universal Usability, Virginia, November Lindgaard, G., Dudek, C., User satisfaction, aesthetics and usability: Beyond reductionism. In: Proceedings of the IFIP 17th World Computer Congress, Montreal, Lindgaard, G., Dudek, C., High appeal versus high usability: Implications for user satisfaction. In: Proceedings of the HF2002 Human Factors Conference, Melbourne, November Lindgaard, G., Dudek, C., What is this evasive beast we call user satisfaction? Interacting with Computers 15(3), Paschos, G., Perceptually uniform color spaces for color texture analysis: an empirical evaluation. IEEE Transactions on Image Processing 10(6),

11 Schenkman, R.N., Jonsson, U., Aesthetics and preferences of web pages. Behaviour & Information Technology 19(5), Smith, J.R., Chang, S.-F., VisualSEEk: A fully automated content-based image query system. In: Proceedings of the 4th ACM International Conference on Multimedia, Massachusetts, November Tractinsky, N., Aesthetics and apparent usability: Empirically cultural and methodological issues. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Atlanta, March Tractinsky, N., Cokhavi, A., Kirschenbaum, M., Sharfi, T., Evaluating the consistency of immediate aesthetic perceptions of web pages. International Journal of Human-Computer Studies 64(11), Tractinsky, N., Katz, A.S., Ikar, D., What is beautiful is usable. Interacting with Computers 13(2), Van der Heijden, H., Factors influencing the usage of websites: The case of a generic portal in the Netherlands. Information and Management 40(6),