PhonePaint: Using Smartphones as Dynamic Brushes with Interactive Displays Jian Zhao Department of Computer Science University of Toronto jianzhao@dgp.toronto.edu Fanny Chevalier Department of Computer Science University of Toronto fanny@dgp.toronto.edu Karan Singh Department of Computer Science University of Toronto karan@dgp.toronto.edu Copyright is held by the author/owner(s). Abstract We present PhonePaint, a novel technique that allows users to sketch on interactive displays using a smartphone as a configurable paintbrush. PhonePaint provides a casual and ubiquitous painting experience that is suitable for a broad audience, and offers a rich sketching interaction where the stroke parameters are dynamically controlled by the built-in sensors of the phone. We describe a proof-of-concept implementation of PhonePaint with minimal hardware added to a regular smartphone. We conducted a light-weight qualitative evaluation of the prototype. The results show that participants enjoyed the drawing process with PhonePaint and were able to create interesting visuals with minimal training. Author Keywords Sketch Interface, Painting, Touch Devices ACM Classification Keywords H.5.2 [Information Interfaces and Presentation]: User Interfaces Introduction Painting, drawing and sketching is a ubiquitous form of creative expression, be it to communicate ideas through visuals, share artistic illustrations, or simply doodle thoughts. The considerable differences
between traditional painting with brushes and digital painting with mice have long been a barrier to creative and freeform sketching on computers. However, the practice of digital painting has drastically increased with the release of interactive displays and styluses, which offer a more engaging and direct user experience. With these digital supports, one category of drawing applications is typically designed for simple, portable and quick sketching needs. As such, they are often limited to the most primitive drawing tools, confining users to a rough and rudimentary style. Conversely, another category of sketching interfaces aims to offer a richer experience. These, which are mostly attached to stationary devices, are typically employed by professional software, where brush types and many of their parameters can be fine tuned through a complex graphical user interface. Such systems, although offering the best flexibility, can be difficult and intimidating for new users to learn, and thus discouraging for novice or casual users. In this paper, we are interested in exploring if and how a light-weight and everyday device such as a smartphone can provide an easy and portable as well as customizable and rich enough drawing experience with interactive displays. We propose PhonePaint, a novel technique designed to support an engaging tangible-input painting process through the flexible control of brush parameters with the interactive affordances of the device, namely its accelerometer, gyroscope and touch display. PhonePaint allows simultaneous drawing at two levels: with the device as a brush on an interactive display, and directly manipulating the phone to dynamically control brush parameters and appearance while drawing strokes. Related Work There exists a large body of research on digital painting systems that aim to reproduce and extend the traditional drawing experience with real brushes. One approach is to algorithmically simulate the ordinary paintbrushes with software, such as Adobe Photoshop and Illustrator. The software simulations can be used with input devices other than a mouse, e.g., a stylus, to further enhance the user experience of painting via tangible inputs. For example, FluidPaint [7] allows artists to paint with real wet brushes in which the footprints of the brush are captured by a transparent surface illuminated by infrared LEDs. Another approach is to augment the tangible brush input device with extra and special hardware. IntuPaint [8] uses a self-illuminated LED brush whose bristles are made of semi-transparent nylon fibers. Conté [9], also equipped with a LED, enables easy mode switching (e.g., color selection) by using its different corners and faces in contact with a tabletop surface. I/O brush [4] uses a tiny camera that allows to pick colors from the real-world. Researchers have explored the use of portable devices as styluses on interactive displays. Hardy et al. [1] proposed a technique allowing users to pair a mobile device with a display and perform selections through a NFC/RFID tag grid. PhoneTouch [5] further refines the selection by correlating the phone accelerometer and tabletop input events. Beside selection tasks, there exist studies on using a smartphone with the surface in a stylus-like fashion. Jang et al. [2] extended smartphone with an optical sensor allowing users to perform flick-based browsing, selections and text input. Schmidt et al. [6] proposed
a TOUCH SCREEN b ACCELEROMETER Figure 1: Example of dynamic brush manipulations using the phone built-in sensors: (a) finger touch location can be used to control the stroke width (up-down) and opacity (left-right), and (b) tilt angles can be used to change the stroke perpendicular opacity gradient. an interaction style for mobiles and surfaces based on direct manipulations with the phone as a tangible stylus, facilitating seamless interaction across device boundaries. McAdam et al. [3] compared a dial manipulation task with 3 interaction types and found that people thought the tangible use of the surface was easier than direct touch. In summary, there has been a consequent effort on augmenting tangible input devices to improve the digital painting experience on the one hand. On the other hand, smart phones has proven to be good candidates to serve as a stlyus on interactive display. However, as far as we know, no work to date has explored bringing those two pieces of the puzzle together. With PhonePaint, we propose to use of the smart phone device as a tangible paintbrush, leveraging the sensor data to dynamically control stroke parameters. The PhonePaint Design The design of PhonePaint is motivated by the many features of a smartphone that we believe are essential for an effective tangible device to serve as a brush for digital painting on interactive displays. Prevalence and Portability. The ubiquitous usage of smartphones nowadays makes the potential brushes prevalent [3]. Smartphones are portable and easy to be associated with other digital devices, therefore people can perform painting on interactive displays at different locations, including private and public places. Personalization and Collaboration. The phone touch display can be used as a dedicated interface for parameter configurations (e.g., a settings panel made of sliders and buttons), allowing personalized sketching styles adjusted on the brush device itself [6]. Unlike traditional styluses, this affordance naturally supported by smartphones enables a potential convenient multi-user collaborative painting experience. Dynamic Brush Manipulation. Figure 1 illustrates some examples of leveraging the built-in phone sensors to dynamically manipulate brush parameters while painting strokes. First, finger manipulation on the phone touch display offers two extra degrees of freedom (x and y coordinates) for directly varying brush properties, e.g., stroke width, color hue or transparency. Second, additional built-in sensors of the phone can be utilized for dynamic brush interactions. For example, the device orientation interpreted can be used for intuitive mode switching between drawing and erasing. Mapping tilt angles to transparency gradients of stroke colors provides a more realistic painting experience (mimicking that only part of the brushes bristles is in contact with the canvas). Cross-Device Data Transfer. The brush can benefit from additional features usually supported by smartphones, including the inter-device communication and camera capabilities [6]. For instance, brush settings and sketches can be fluidly transferred between different phones or between the phone and the display, to support easy sharing, recording and collaboration. The digital camera can collect additional input from the surrounding environment, such as using real-world imagery as a color or texture palette [4]. Proof-of-Concept Implementation We created a research prototype using a HTC Touch Pro 2 as the PhonePaint brush and a Microsoft Surface as the painting display. At this stage, we chose to implement features that are critical for the painting
(a) (b) Figure 2: Proof-of-concept implementation of PhonePaint: (a) adding minimal hardware to the phone, i.e., two LEDs and a battery, and (b) the brush parameters configuration panel. process. Features such as data transfer and camera usage, secondary to our goal, are left for future work. The Phone Brush Ideally we aim not to add any extra hardware with the phone to maintain its ubiquitousness. However, limited by current settings of smartphones, we attached minimal hardware two infrared LEDs (5mm diameter, 1.2V) in order to make the footprints of the phone detectable by the surface (Figure 2-a). The LED on the top of the phone is used to track the device in the drawing mode. When the phone is flipped, the one on the bottom indicates the location of the eraser. An AA battery was used to supply power for the two LEDs. The physical dimensions of the phone brush are 116 59.2 17.3mm and its total weight (including the extra hardware) was 195g We developed a customized system for configuring the brush parameters and sending data to the surface via the Bluetooth network. Figure 2-b shows the configuration user interface of PhonePaint, in which the user can manipulate: stroke colors and color gradients, brush initial width, and mapping mechanisms between finger touch locations on the phone display and the chosen brush parameters. The phone accelerometer and gyroscope data was used to: 1) compute the phone tilting angles for dynamically changing the opacity gradient perpendicular to the stroke drawing direction, and 2) distinguish whether the brush is in the drawing mode or the erasing mode as described in the previous section. While our prototype only implements associations of sensor data with a few drawing parameters, other parameter mappings can be easily added in a similar fashion. The Painting Surface The Microsoft Surface (30 inch diagonal, 1024 768 pixels) was raised to a height of 30cm to allow users to comfortably reach the full surface of the display, when standing beside it. Since an important affordance of PhonePaint is to incorporate the brush settings directly on the brush device, the interactive display itself was limited to a simple drawing canvas. The surface, thus, only received data sent by the phone, and interpreted the brush parameters to render strokes, using the tracking information of the phone footprint coordinates. To facilitate the evaluation of PhonePaint, we developed a simple graphic user interface as shown in Figure 3-a, which we discuss in detail below. User Study We conducted a light-weight evaluation with the purpose of gathering initial user feedback and studying the usability of PhonePaint. Six volunteers (3 females), aged 22-31, participated in the study. Participants were members of the university community (i.e., graduate students and researchers). All are right-handed, and none of them had prior professional artistic training. First, participants performed 24 trials of structured drawing tasks for reproducing example strokes on a blank canvas, as shown on the top-left corner of the surface (Figure 3-a). Textual instructions were also provided for describing the conditions required for completing the trial. Creative manipulations of various brush properties (e.g., stroke colors, gradients or widths) was encouraged as long as the required instructions were obeyed. Subjects could restart a trial as many times as they desired until satisfaction (no time limit was imposed).
(a) (b) Figure 3: Participants using PhonePaint (a) to reproduce a given stroke shown on the top-left corner and, (b) for freestyle drawing. For the example strokes, we controlled 4 independent variables: the stroke type (straight or curvy line), the color mode (unicolor or three-point gradient), opacity gradient mode (with or without phone tilt angles to change stroke opacity gradient) and touch display control mode (not using the touch display, using touch positions to change the brush transparency, or change the brush width), for a total of 24 conditions. After completing all the trials, participants were given 10min for free-form painting (Figure 3-b), after which they filled out a post-study questionnaire. We also conducted a short informal interview to collect their general feedback. The whole study took about 30min. Results and Discussions Qualitative Assessment Figure 4 shows the results of the post-questionnaire. The overall user reactions were very encouraging. Participants generally thought that PhonePaint was flexible, novel and compelling to control multiple stroke parameters in real-time with phone sensors (Q1: µ = 3.8). They were eager to have such application on their personal phones (Q5: µ = 4.4). But we observed larger variances on Q4 (σ = 1.0) and Q6 (σ = 1.1) related to aspects of collaborative and public drawing, which may due to participants various personal preferences about privacy issues. Task Observations and Initial User Feedback During structured drawing time, we observed that participants were able to familiarize themselves with the PhonePaint brush after a very short time practice. All participants completed the task fluidly asking for only little assistance on setting the brush parameters with the configuration panel (Figure 2-b). All the trials were achieved fairly quickly by participants (average completion time 18s including the configurations of brush parameters), except for the ones trials requiring the use of the phone touch display and the phone tilt angle that were more time consuming (30s on average). Overall, participants were able to easily and accurately reproduce the given sample strokes, indicating that PhonePaint is a suitable candidate as a configurable brush for digital painting. Q1. I find that the control of the stroke colors, width and opacity is flexible Q2. I enjoy the process of painting with the phone Q3. I feel creative when I use this system Q4. It would be more fun if I could draw with others with this system Q5. I would like to have an application like this one on my phone Q6. I would like to have access to sketching displays everywhere Q1 Q2 Q3 Q4 Q5 Q6 Score 1 2 3 4 5 Figure 4: Average usability assessment measured on a 5-point Likert scale (1: strongly disagree, 5: strongly agree) In free-form drawing part, participants showed enthusiasm and reported that the whole drawing process to be enjoyable, interesting and engaging. Some more talented artists were able to produce very compelling and interesting sketches like those of Figure 5. Three participants were very excited and even requested more time to finish their artwork, looking at the sketches and saying: I should show it to my friends. All the participants particularly liked using the tilt angle to modify dynamic brush parameters: It s lovely. I can draw fancy strokes by just rotating my wrist. Two participants tried almost all the features of PhonePaint by themselves, including different ways of controlling the dynamic brush parameters. In contrast, some participants sticked to a set of few features, such
Figure 5: Examples of free-form sketches performed by participants as changing the stroke opacity gradient with tilt angles. For these participants, the experimenter sporadically suggested exploration by giving some hints. This may be because of participants different personal preferences and prior experiences. A notable issue was that several participants found it difficult to use the touch display for dynamic brush parameter control and it became harder when using in combination of tilting angles. However, some participants used the second hand on the touch screen for more accuracy, which could overcome this issue to some extent, similar to the interaction in [6]. Participants also reported that PhonePaint can become cumbersome and tiring for fine-stroke sketching. Similar issues have been pointed out in [2, 3]. However, such fine-drawing tasks tend to be rare in non-professional and casual sketches which are typically used for quick illustrations. PhonePaint is designed to provide a portable and easy sketching experience with a rich dynamic manipulation of stroke parameters. In this light, we believe the prevalence and ubiquitousness of the approach is of a higher importance. During our study, participants felt that the phone and the tabletop surface worked well as an unit, and the design of PhonePaint provided them with an interface easy to grasp and use. Conclusion and Future Work We have introduced PhonePaint, a technique designed for informal ubiquitous sketching, where a smartphone is used as a paintbrush in a novel way to draw on an interactive displays. PhonePaint aims at improving the users experience of digital painting through direct manipulations on the brush itself to control brush parameters, by leveraging the phone built-in sensors. Our initial study suggests that PhonePaint is an effective and convenient tool for perform basic everyday drawing tasks. In the future, we wish to explore other interaction techniques to improve the overall user experience of digital painting using a similar system. One possibility could be enabling haptic feedback to convey additional information of stroke parameter settings. We also plan to extend PhonePaint to support collaborative drawing. Finally, we want to conduct further evaluations on both multi-user and individual painting scenarios to better understand the capabilities of this idea. References [1] R. Hardy and E. Rukzio. Touch & interact: touch-based interaction of mobile phones with displays. In Proc. of MobileHCI, pp. 245-254, 2008. [2] B. Jang, M. Lee, and G. J. Kim. CornerPen: smart phone is the pen. In Proc. of int. conf. on Human-computer interaction. Part 3:76-83, 2011. [3] C. McAdam and S. Brewster. Using mobile phones to interact with tabletop computers. In Proc. of ITS, pp. 232-241, 2011. [4] K. Ryokai, S. Marti, and H. Ishii. I/O brush: drawing with everyday objects as ink. In Proc. of CHI, pp. 303-310, 2004. [5] D. Schmidt, F. Chehimi, E. Rukzio, and H. Gellersen. PhoneTouch: a technique for direct phone interaction on surfaces. In Proc. of ACM UIST, pp. 13-16, 2010. [6] D. Schmidt, J. Seifert, E Rukzio, and H. Gellersen. A cross-device interaction style for mobiles and surfaces. In Proc. of DIS, pp. 318-327, 2012. [7] P. Vandoren, L. Claesen, T. Van Laerhoven, J. Taelman, C. Raymaekers, E. Flerackers, and F. Van Reeth. FluidPaint: an interactive digital painting system using real wet brushes. In Proc. of ITS, pp. 53-56, 2009. [8] P. Vandoren, T. Van Laerhoven, l. Claesen, J. Taelman, C. Raymaekers, and F. Van Reeth. IntuPaint: Bridging the gap between physical and digital painting. In IEEE workshop on tabletops and interactive surfaces, pp. 65-72, 2008. [9] D. Vogel and G. Casiez. Conté: multimodal input inspired by an artist s crayon. In Proc. of UIST, pp. 357-366, 2011.