IN THE UNITED STATES PATENT AND TRADEMARK OFFICE PATENT TRIAL & APPEAL BOARD

Transcription

1 DOCKET NO: US IN THE UNITED STATES PATENT AND TRADEMARK OFFICE PATENT TRIAL & APPEAL BOARD PATENT: 8,174,506 INVENTOR: TAE HUN KIM et al. TITLE: METHOD OF DISPLAYING OBJECT AND TERMINAL CAPABLE OF IMPLEMENTING THE SAME TRIAL NO.: IPR2017- DECLARATION OF DR. DANIEL J. WIGDOR, PH.D. 1. I, Dr. Daniel J. Wigdor, Ph.D., make this declaration in connection with this petition for inter partes review of U.S. Patent No. 8,174,506 ( the 506 patent, attached as Exhibit 1001 to the petition). I am over 21 years of age and otherwise competent to make this declaration. Although I am being compensated for my time in preparing this declaration, the opinions herein are my own, and I have no stake in the outcome of the inter partes review proceeding. 1 BlackBerry Exhibit 1004, pg. 1

2 I. QUALIFICATIONS 2. A detailed record of my professional qualifications, including a list of publications, awards, and professional activities, can be found in my curriculum vitae, which is attached as Appendix A to this declaration. An appendix to my curriculum vitae also lists each matter in which I have provided testimony, either though declaration, deposition, or trial in the last 5 years. 3. I am currently an Associate Professor of Computer Science and Sloan Research Fellow at the University of Toronto, where I have joint appointments at the Department of Computer Science and the department of Mathematical and Computational Sciences. I will be taking a sabbatical later this year to visit Cornell Tech in New York City as a Visiting Associate Professor, where I will be conducting research and teaching classes in human computer interaction. 4. Before joining the faculty at the University of Toronto in 2011, I was a researcher at Microsoft Research, the user experience architect of the Microsoft Surface table, and a company-wide expert in user interfaces for new technologies. 2 BlackBerry Exhibit 1004, pg. 2

3 5. While studying to obtain my Ph.D. degree at the University of Toronto, which pioneered much of the early work on touch sensitive devices and, in particular, invented the first multi-touch computing device, I was a fellow at the Initiative in Innovative Computing at Harvard University and conducted research for Mitsubishi Electric Research Labs (MERL). While at MERL, I was part of the DiamondSpace project that developed software for the DiamondTouch multi-touch device. DiamondTouch is a multi-input touch sensitive device that allows multiple people, simultaneously, to interact with the display. 6. In particular, I was responsible for conducting research regarding user interfaces for use on multitouch devices such as the DiamondTouch. For example, I was responsible for the design and development of user interface software that ran on the DiamondTouch display and that responded to user input by detecting changes in capacitance measurements of the touch sensor. 7. My work regularly involved designating areas of the display to correspond to buttons and other user interface elements and writing software to map the sensed capacitance variations of each sensor area 3 BlackBerry Exhibit 1004, pg. 3

4 or element to a user interface object, including pre-processing, filtering of input, and detecting gestures. My work further included the creation of applications, as well as general-purpose tools that would process input and enable application software. 8. In my work at MERL, I developed such software not only for the DiamondTouch but for several other touchscreen technologies as well, such as the FingerWorks igesture Pad, mobile phones, and digital whiteboards, among others. I was also responsible for the design of hardware devices, such as a two-sided DiamondTouch and mobile devices. 9. I hold Hon. B.Sc., M.S., and Ph.D. degrees in computer science and have published extensively, with about 70 peer-reviewed, technical publications. Of these, approximately 16 are technical papers which relate directly to the design of touch sensitive interfaces and implementation of the same into electronic devices. 1 I have given over 70 invited talks, including multiple keynote lectures. 1 This includes papers numbered C.7, C.9, C.11, C.13, C.15, C.16, C.17, C.18, C.24, C.26, C.29, C.32, C.33, C.39, C.41, and C.44 in my curriculum vitae. 4 BlackBerry Exhibit 1004, pg. 4

5 10. I have used my education and experience working in the field of computer science, and my understanding of the knowledge, creativity, and experience of a person having ordinary skill in the art in forming the opinions expressed in this report, as well as any other materials discussed herein. II. MATERIALS CONSIDERED 11. In forming my opinions, I read and considered the 506 patent and its prosecution history, along with the prosecution history of parent U.S. Patent No. 7,663,610 ( the 610 patent ), the exhibits listed in the Exhibit List filed with the petition for inter partes review of the 506 patent, as well as any other material referenced herein. III. LEGAL PRINCIPLES 12. For the purposes of this declaration, I have been informed about certain aspects of patent law that are relevant to my analysis and opinions, as set forth in this section of my declaration. A. A Person Having Ordinary Skill in the Art 13. I understand that the disclosure of patents and prior art references are to be viewed from the perspective of a person having ordinary skill in the art ( POSITA ) at the time of the alleged invention. 5 BlackBerry Exhibit 1004, pg. 5

6 Unless I state otherwise, I provide my opinion herein from the viewpoint of a POSITA at the earliest alleged priority date for the 506 patent, which I have been informed is March 30, The 506 patent relates to the field of user interface ( UI ) design. The particular technical issue that the 506 patent purports to address is the efficient arrangement and display of objects on a mobile terminal. I understand from my own research, work experience, and conversations with colleagues at Microsoft, Mitsubishi Electric Research Labs, and my former students that user interface design involves many considerations, including but not limited to: (1) knowledge of human capabilities, perception, and experience, (2) an understanding of context of use of the UI (such as device size, input technology, and circumstances of use), and (3) the task(s) to be performed. A POSITA in the field of user interface design and engineering would be skilled in the best practices in designing user interfaces under each of those considerations. 15. The various references that I discuss below are informative of the level of skill of a POSITA and are of the type that are reasonably 6 BlackBerry Exhibit 1004, pg. 6

7 relied upon by experts in my field to form opinions on the subject of user interface design and engineering. 16. Long before the 506 patent application was filed, touchbased portable electronic devices having reconfigurable user interfaces were known. These include, for example, the Palm Pilot devices (Ex. 1012), the IBM Simon (Ex. 1013), and the ATT EO (Ex. 1014), all devices released in the 1990 s and all capable of various degrees of reconfiguration of their user interfaces. 17. Touch-sensitive display technology was also well-known prior to the filing date. The first known examples of such systems were first developed in the 1960 s, such as the Sketchpad device (Ex. 1015) developed at Lincoln Labs, as well as the Programmed Man-Machine Interface (Ex. 1016) developed in the U.K. Later, multitouch technologies allowed for multiple simultaneous touches to the screen to be detected. The first example of such a system was the Flexible Human Machine Interface (Ex. 1017) developed at the University of Toronto in the early 1980 s. 18. The prior art also demonstrates that companies such as Apple, Inc. and IBM were already seeking patent protection on user 7 BlackBerry Exhibit 1004, pg. 7

8 interfaces for portable devices, such as mobile phones and PDAs, for displaying objects or icons more intuitively and efficiently in order to make better use of the limited display screen area. (Ex. 1005, 1:20 54, Figures 1-3; Ex. 1006, 1:16 62.) 19. Further, the prior art evidences that a POSITA knew to change the appearance of objects or icons or output a sound or vibration to indicate a change in operation associated with the object, such as the movability of the object on a touchscreen, i.e., the object being in a fixed position or the position fixation of the object being released. (Ex. 1006, Abstract, 1:9 13, 2:30 34; Ex. 1007, Abstract, [0049].) 20. Based on these factors, I have concluded that a POSITA was sufficiently skilled in the general design of user interfaces for touchsensitive displays and experienced in arranging their layouts to most efficiently use available space in view of display screen areas limited in size. Further, the POSITA was sufficiently skilled and experienced to recognize the value in enabling a user to reconfigure the interface to suit their particular needs. 8 BlackBerry Exhibit 1004, pg. 8

9 B. Claim Construction 21. I understand that claim construction is the process of determining a patent claim s meaning. I also have been informed and understand that the proper construction of a claim term is the meaning that a POSITA would have given to that term. 22. I understand that claims in inter partes review proceedings are to be given their broadest reasonable interpretation in light of the specification, which is what I have done when performing my analysis in this declaration. I provide comments on specific terms below. 23. Shape (Claims 1 and 10): Under its broadest reasonable interpretation in light of the specification, shape means appearance. The 506 patent describes changing the appearance of an object, for example, by displaying an icon or graphical indication adjacent to or attached to the fixed object, labeled Not Fixed or Movable, to visually inform the user that the object is either fixed to a position and non-movable or has been released and is movable. (Ex. 1001, 5:31 38, 6:13 19.) 9 BlackBerry Exhibit 1004, pg. 9

10 C. Anticipation 24. I understand that a patent claim is unpatentable as anticipated if a POSITA during the relevant timeframe would have understood a single prior art reference to teach every limitation contained in the claim. The disclosure in a reference does not have to be in the same words as the claim, but all of the requirements of the claim must be described in enough detail, or necessarily implied by or inherent in the reference, to enable a POSITA looking at the reference to make and use at least one embodiment of the claimed invention. D. Obviousness 25. I understand that a patent claim is invalid if the differences between the patented subject matter and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person of ordinary skill in the art. 26. When considering the issues of obviousness, I understand that I am to do the following: (i) determine the scope and content of the prior art; (ii) ascertain the differences between the prior art and the claims at issue; (iii) resolve the level of ordinary skill in the pertinent art; and (iv) consider objective evidence of non-obviousness. Moreover, I 10 BlackBerry Exhibit 1004, pg. 10

11 have been informed and I understand that so-called objective indicia of non-obviousness (also known as secondary considerations ) like the following are also to be considered when assessing obviousness: (1) commercial success; (2) long-felt but unresolved needs; (3) copying of the invention by others in the field; (4) initial expressions of disbelief by experts in the field; (5) failure of others to solve the problem that the inventor solved; and (6) unexpected results. I also understand that evidence of objective indicia of non-obviousness must be commensurate in scope with the claimed subject matter. I am not aware of any objective indicia of non-obviousness relevant to the claims of the 506 patent. 27. Put another way, my understanding is that not all innovations are patentable. Even if a claimed product or method is not disclosed in its entirety in a single prior art reference, the patent claim is invalid if the invention would have been obvious to a person of ordinary skill in the art at the time of the invention. In particular, I understand that a patent claim is normally invalid if it would have been a matter of ordinary innovation within the relevant field to create the claimed product at the time of the invention. 11 BlackBerry Exhibit 1004, pg. 11

12 28. In determining whether the subject matter as a whole would have been obvious at the time that the invention was made to a person having ordinary skill in the art, I have been informed of several principles regarding the combination of elements of the prior art: a. First, a combination of familiar elements according to known methods is likely to be obvious when it yields predictable results. b. Second, if a person of ordinary skill in the art can implement a predictable variation in a prior art device, and would see the benefit from doing so, such a variation would be obvious. In particular, when there is pressure to solve a problem and there are a finite number of identifiable, predictable solutions, it would be reasonable for a person of ordinary skill to pursue those options that fall within his or her technical grasp. If such a process leads to the claimed invention, then the latter is not an innovation, but more the result of ordinary skill and common sense. 12 BlackBerry Exhibit 1004, pg. 12

13 29. I understand that the teaching, suggestion, or motivation test is a useful guide in establishing a rationale for combining elements of the prior art. This test poses the question as to whether there is an explicit teaching, suggestion, or motivation in the prior art to combine prior art elements in a way that realizes the claimed invention. Though useful to the obviousness inquiry, I understand that this test should not be treated as a rigid rule. It is not necessary to seek out precise teachings; it is permissible to consider the inferences and creative steps that a person of ordinary skill in the art (who is considered to have an ordinary level of creativity and is not an automaton ) would employ. IV. THE 506 PATENT 30. The 506 patent, which was filed December 28, 2009 and issued May 8, 2012, is a continuation of U.S. Patent No. 7,663,610, filed March 30, 2007, which claims foreign priority to Korean Patent Application No , filed March 30, The 506 patent describes a method of displaying a graphical object, which can be any item that can be displayed on a touchscreen of a terminal and fixed to a particular position on the screen in response to a touch action of a terminal user. (Ex. 1001, 1:15 20, 1:61 64, 4:37 40.) 13 BlackBerry Exhibit 1004, pg. 13

14 In particular, a user s touch action on the touchscreen releases the object from a fixed position, such that it can be moved from a first position to a second position on the touchscreen. (Id. at 5:57 61.) Further, the user interface is designed to give feedback to the user when the object is no longer fixed in place. (Id. at 6:13 19.) The movable object can then be dragged from the first position to the second position if another touch action is carried out on the object. (Id. at 2:10 18.) Thereafter, the moved object can be fixed in the second position by any of the various types of touch actions illustrated in FIGs. 5A 5D below. (Id. at 5:7 30.) 14 BlackBerry Exhibit 1004, pg. 14

15 32. The entire process of displaying and interacting with the onscreen object is described, for example, in the flowchart in FIG. 2 below. 15 BlackBerry Exhibit 1004, pg. 15

16 V. Summary of Select Prior Art 33. Based on my extensive experience in the design and engineering of user interface technologies, it is my opinion that all claim elements of at least claims 1, 2, and 8 11 of the 506 patent were known in the prior art and used for the same purpose. I provide a brief summary of each of the prior art references relied upon in the petition below. 16 BlackBerry Exhibit 1004, pg. 16

17 1. Van Os 34. I understand that Van Os (U.S. Patent No. 7,509,588) was filed on July 24, 2006 and claims the benefit of priority to provisional U.S. Patent Application No. 60/755,368 ( the 368 provisional application ) (Ex. 1009), filed on December 30, Hence, Van Os is available as prior art under 102(e). I reviewed the 368 provisional application and find that all of the subject matter of Van Os relied upon in this petition is fully supported by the 368 provisional application, unless stated otherwise. 35. Van Os describes a portable electronic device, such as a mobile phone, which is controlled via a touch-sensitive display. The device displays graphical objects on a portion of the touch-sensitive display. The device also includes an interface reconfiguration mode in which the objects may be repositioned or moved. (Ex. 1005, Abstract, 1:27 31, 1:58 62.) A user may enter or exit the interface reconfiguration mode by performing a predefined user action, such as selecting one or more appropriate physical buttons on the portable device, by a gesture, and/or by selecting one or more soft buttons (such 17 BlackBerry Exhibit 1004, pg. 17

18 as one or more icons that are displayed on the touch-sensitive display). (Id. at 4:14 21.) 36. As a general overview, during a position adjustment process the user first places the portable device into reconfiguration mode where fixed positions of icons are released, then drags and drops a desired icon or icons to a new position, and then places the device back into a mode in which the icons positions are fixed. (Ex. 1005, Figure 1.) To begin, as shown in Figure 2B below, the user initiates the interface reconfiguration mode via a first predefined user action. As a result, the one or more icons displayed in the user interface are animated to simulate floating above a plane of the display of the device, notifying the user that the fixed positions of the icons have been released such that they can be moved. (Id. at 3:36 43.) The varying of the positions of the one or more icons may intuitively indicate to the user that the positions of the one or more icons may be reconfigured by the user. (Id. at 3:49 51.) 18 BlackBerry Exhibit 1004, pg. 18

19 37. The user then makes contact, either directly or indirectly, with one of the animated icons (at position 226, Figure 2C below) and moves the point of contact across the display surface, i.e., [t]he user may drag the respective icon to a second position. (Ex. 1005, 3:58 59.) The contact and the motion are detected by the portable communication device 200, and, as a consequence, the displayed icon game, as illustrated in Figure 2C below, is moved accordingly. (Id. at 5:63 6:4.) 19 BlackBerry Exhibit 1004, pg. 19

20 Once the user moves the game icon to position 228 and lifts his finger to break contact with the display surface, the game icon is displayed at the position 228. See Figure 2D of Van Os. (Id. at 6:5 8.) 20 BlackBerry Exhibit 1004, pg. 20

21 38. The user is then able to exit the interface reconfiguration mode by performing a second predefined user action, for example, by selecting or deselecting a physical button on the portable device. As a result, animation of the icons ceases, and the icons in the GUI 210 are displayed in stationary or fixed positions. (Ex. 1005, 4:61 67, Figure 2E.) 21 BlackBerry Exhibit 1004, pg. 21

22 2. Cox 39. I understand that Cox (U.S. Patent No. 6,462,760) issued on October 8, 2002 and is prior art under 102(b). 40. Cox describes a user interface directed to conserving space on a computer display screen by using an icon s visual appearance to associate the icon with a particular operation. In response to input from 22 BlackBerry Exhibit 1004, pg. 22

23 a user, the icon s appearance changes to indicate that the icon is now associated with an alternative operation. (Ex. 1006, Abstract, 1:9 13.) The icon s appearance can be changed by changing textual information disposed on the icon, by changing the color of the icon, or by changing the geometry (e.g., size or shape) of the icon. (Id. at 2:30 34.) 41. FIG. 5A of Cox below illustrates one embodiment in which a pushbutton icon 74 used by the user interface module 60 includes a textual label that associates the pushbutton icon 74 with a particular operation. In the example from the figure, the text indicates a Find operation that may be used as part of a text replacement function in a word processing program. (Ex. 1006, 5:6 12.) If an alternate operation is requested by the user, then the user interface module 60 causes the pushbutton icon 74 to change appearance at block 84. This is illustrated in FIG. 5B where the textual label on the pushbutton icon 74 is changed from Find to Replace. The color, geometry (e.g., shape or size), or other aspect of the pushbutton icon s 74 appearance could also be changed to associate the pushbutton icon 74 with an alternative operation. (Id. at 5:34 43.) 23 BlackBerry Exhibit 1004, pg. 23

24 3. Woolley 42. I understand Woolley (U.S. Patent Publication No. 2003/ ) published on January 23, 2003 and is prior art under 102(b). 43. Woolley describes a kiosk touchpad with a touch-sensitive surface, which includes a speaker for giving audio feedback to indicate, for example, when a selection has been made. (Ex. 1007, Abstract.) As illustrated in FIG. 4 of Woolley below, the touch pad 21 includes several components, including a cover plate 31, a touch-sensitive surface 26, 24 BlackBerry Exhibit 1004, pg. 24

25 and electronic components 45 mounted on the reverse side thereof, including a speaker 71. (Id. at [0049].) 44. Woolley further suggests employing multiple forms of sensory feedback simultaneously, such as auditory, tactile, and visual, to assist the user. (Ex. 1007, [0049].) 4. Apple Newton MessagePad Handbook ( MessagePad Handbook ) 45. I understand that the Apple Newton MessagePad Handbook ( MessagePad Handbook ) was published in 1995 and is prior art under 102(b). I personally recall the Apple Newton MessagePad, hereinafter simply referred to as the MessagePad, being sold circa In fact, I sold these devices myself while working at an electronics store. I distinctly remember the Newton logo, the hand-drawn lightbulb, on the packaging of the MessagePad, along with on the Handbook that came with the device, similar to that which appears in Exhibit Further, 25 BlackBerry Exhibit 1004, pg. 25

26 as evidenced by a 1996 IEEE article (Ex. 1010), which references the MessagePad Handbook dated 1995, and the Computer History Museum (Ex. 1011), which lists the MessagePad Handbook with a catalog number and a date of 1996, the MessagePad Handbook and the MessagePad, itself, were made publicly available at least by The MessagePad is a type of portable electronic device, a personal digital assistant ( PDA ). As shown below, a series of pictorially depicted objects are arranged along a lower edge of the touch screen of the MessagePad. These objects are buttons that can be used to access different applications of the MessagePad, such as an address book ( Names ), a calendar ( Dates ), Extras, etc. (Ex. 1008, ) 26 BlackBerry Exhibit 1004, pg. 26

27 27 BlackBerry Exhibit 1004, pg. 27

28 As discussed in the MessagePad Handbook, tapping the Extras button, for example, opens the Extras Drawer, which contains a plurality of icons or graphical objects displayed in fixed positions. (Ex. 1008, ) A selection action allows the user to reposition the icons and reconfigure the interface. You can select icons and move them around in the Extras Drawer to change their position. (Id. at 160.) 28 BlackBerry Exhibit 1004, pg. 28

29 47. As shown in the example below, to move an icon from a first position to a second position, the user first selects the icon by holding a pen on or near the icon until a heavy mark appears under the pen and the MessagePad makes a squeaking sound. (Ex. 1008, 160.) Next, the user draws a mark over or around the icon to select it. (Id. at 160, 212.) Then, to actually move the icon, the user drags the icon from the first position to the second position. (Id. at 160.) 29 BlackBerry Exhibit 1004, pg. 29

30 48. Finally, to fix the moved icon in the second position on the touch screen, the user simply taps the pen anywhere outside the selection to deselect the item. (Ex. 1008, ) VI. ANALYSIS 49. It is my opinion that claims 1, 2, and 8-11 of the 506 patent are obvious and/or anticipated over the prior art, as detailed in the grounds of rejection below. At the request of counsel, I have divided the claims into elements denoted as [a], [b], [c], etc. to correspond to the discussion of the same elements in the petition for inter partes review, as follows: 30 BlackBerry Exhibit 1004, pg. 30

31 Claim 1: [a] A mobile terminal comprising: [b] a touch screen configured to display at least one object and to sense first and second touching actions on the at least one object; and [c] a control unit configured to display the at least one object as being fixed and not moveable, [d] to release the at least one object according to the second touching action such that the at least one object can be moved, [e] to move the at least one object from a first position to a second position on the touch screen corresponding to the first touching action on the at least one object, and [f] to fix the moved object to the second position on the touch screen based on a manipulation of a position fixing button that is separate from the first and second touching actions, [g] wherein the first touching action is a dragging and dropping touching action on the touch screen, [h] wherein the second touching action includes touching the at least one object for a predetermined time period, and [i] wherein the control unit is further configured to display differently a shape of the object when the position fixation of the object is released. 31 BlackBerry Exhibit 1004, pg. 31

32 Claim 2: The mobile terminal of claim 1, further comprising: an output unit configured to output at least one of a sound and a vibration to indicate that the moved object is fixed to the second position according to a control of the control unit. Claim 8: [a] A mobile terminal comprising: [b] a touch screen configured to display at least one object and to sense first and second touching actions on the at least one object; and [c] a control unit configured to display the at least one object as being fixed and not moveable, [d] to release the at least one object according to the second touching action such that the at least one object can be moved, [e] to move the at least one object from a first position to a second position on the touch screen corresponding to the first touching action on the at least one object, and [f] to fix the moved object to the second position on the touch screen based on a manipulation of a position fixing button that is separate from the first and second touching actions, [g] wherein the first touching action is a dragging and dropping touching action on the touch screen, and 32 BlackBerry Exhibit 1004, pg. 32

33 [h] wherein the second touching action includes touching the at least one object for a predetermined time period. Claim 9: The mobile terminal of claim 8, further comprising: an output unit configured to output at least one of a sound and a vibration to indicate that the moved object is fixed to the second position according to a control of the control unit. Claim 10: [a] A method of controlling a mobile terminal, the method comprising: [b] displaying at least one object on a touchscreen of the mobile terminal; [c] sensing, via a sensor of the touch screen, first and second touching actions on the at least one object; [d] displaying the at least one object as being fixed and not moveable; [e] releasing the at least one object according to the second touching action such that the at least one object can be moved, [f] moving the at least one object from a first position to a second position on the touch screen corresponding to the first touching action on the at least one object, and 33 BlackBerry Exhibit 1004, pg. 33

34 [g] fixing the moved object to the second position on the touch screen based on a manipulation of a position fixing button that is separate from the first and second touching actions, [h] wherein the first touching action is a dragging and dropping touching action on the touch screen, [i] wherein the second touching action includes touching the at least one object for a predetermined time period, and [j] wherein the method further comprises displaying differently a shape of the object when the position fixation of the object is released. Claim 11: The method of claim 10, further comprising: outputting at least one of a sound and a vibration to indicate that the moved object is fixed to the second position. A. Ground 1: Claims 1 and 10 Are Obvious Over Van Os and Cox Claim 1[a]: A mobile terminal comprising: 50. The 506 patent describes the mobile terminal as a mobile phone, a PDA..., a PMP (portable multimedia player), a game player, and the like. (Ex. 1001, 3:1 3.) 34 BlackBerry Exhibit 1004, pg. 34

35 51. Van Os discloses a portable electronic device, including but not limited to a handheld computer, a tablet computer, a mobile phone, a media player, personal digital assistant (PDA) and the like. (Ex. 1005, 7:9 18.) Figure 2A of Van Os below, for example, illustrates a portable communication device Thus, it is my opinion that the Van Os and Cox combination renders obvious claim element 1[a]. 35 BlackBerry Exhibit 1004, pg. 35

36 Claim 1[b]: a touch screen configured to display at least one object and to sense first and second touching actions on the at least one object; and 53. The portable electronic device described in Van Os includes a touch-sensitive display or touch screen which responds to physical contact by a stylus or one or more fingers at one or more contact points. (Ex. 1005, 3:43 48.) Figure 2A of Van Os above illustrates the portable communication device 200 with the touch-sensitive display with a GUI 210. The display surface is transparent to allow various icons or graphical objects 222 to be displayed to a user. (Id. at 5:8 11.) 54. Van Os describes sensing of touching actions on the touchsensitive display 412 as follows: [t]he touch-sensitive display 412 communicates with the processing system 404 via the touch sensitive screen controller The touch-sensitive display 412 and the touch screen controller detects contact... on the touch-sensitive display 412 and converts the detected contact into interaction with user-interface objects, such as one or more soft keys, that are displayed on the touch screen when the contact occurs. (Ex. 1005, 9:34 36, ) 36 BlackBerry Exhibit 1004, pg. 36

37 55. Van Os further discloses that the touch-sensitive display 412 and the touch screen controller 432 detect contact and movement or release thereof using any of a plurality of touch sensitive technologies including but not limited to... proximity sensor arrays or other elements for determining one or more points of contact with the touchsensitive display 412. (Ex. 1005, 9:63 10:3.) 56. Regarding the claimed first touching action, Vas Os discloses that the user may drag the respective icon 222 from a first position and drop it at a second position: [t]he user may drag the respective icon to a second position. (Ex. 1005, 3:58 59, 4:53 60, Figure 2C.) Van Os further describes the first touching action as follows: [t]he user makes contact, either direct or indirect, with one of the icons that is moving at a position 226 and moves the point of contact across the display surface with GUI 210. The contact and the motion are detected by the portable communication device 200. As a consequence, the displayed icon, in this example corresponding to a game [(see Figure 2C)], is moved accordingly. (Id. at 5:65 6:4, Figure 2C.) 37 BlackBerry Exhibit 1004, pg. 37

38 57. Van Os also discloses a first predefined user action that initiates the interface reconfiguration process, placing the user interface into a mode in which the icons may be repositioned. Such process is initiated by selecting one or more appropriate physical buttons on the portable device, by a gesture (such as making contact and swiping one or more fingers across the touch-sensitive display or making contact and holding for more than a predefined time period) and/or by selecting 38 BlackBerry Exhibit 1004, pg. 38

39 one or more soft buttons (such as one or more icons that are displayed on the touch-sensitive display). (Ex. 1005, 4:14 67.) In my opinion, this first predefined user action discussed in Van Os corresponds to the claimed second touching action of the 506 patent because such action in Van Os releases the fixed positions of the icons such that they may be moved and repositioned. Further, similar to the 506 patent, the user is notified that the icons are released and movable by changing an appearance of the icons. That is, one or more of the icons displayed in the user interface are animated to simulate floating above a plane of the display of the device, notifying the user that the fixed positions of the icons have been released such that they can be moved. (Id. at 3:36 43.) The varying of the positions of the one or more icons may intuitively indicate to the user that the positions of the one or more icons may be reconfigured by the user. The user may modify, adapt and/or reconfigure the positions of the one or more icons. (Id. at 3:49 53.) 58. Thus, it is my opinion that the Van Os and Cox combination renders obvious claim element 1[b]. 39 BlackBerry Exhibit 1004, pg. 39

40 Claim 1[c]: a control unit configured to display the at least one object as being fixed and not moveable, 59. Van Os provides a block diagram of the portable communication device 400 in Figure 4, which includes, for example, one or more computer-readable mediums 402, a processing system 404, an Input/Output (I/O) subsystem 406, radio frequency (RF) circuitry 408 and audio circuitry 410. These components may be coupled by one or more communication buses or signal lines 403. (Ex. 1005, 7:9 19.) Figure 4 further illustrates that the processing system 404 includes a controller 420 for controlling the operations of the portable device. (Id.) 40 BlackBerry Exhibit 1004, pg. 40

41 60. Further, as explained above for claim element 1[b], [t]he touch-sensitive display 412 communicates with the processing system 41 BlackBerry Exhibit 1004, pg. 41

42 404 via the touch sensitive screen controller The touch-sensitive display 412 and the touch screen controller detects contact... on the touch-sensitive display 412 and converts the detected contact into interaction with user-interface objects, such as one or more soft keys, that are displayed on the touch screen when the contact occurs. (Ex. 1005, 9:34 36, ) 61. Figure 2E of Van Os below illustrates the portable communication device 200 after the interface reconfiguration mode has been terminated or has terminated (due to a time out). The icons 222 in GUI 210 are visually presented in stationary or fixed, non-moveable positions. (Ex. 1005, 4:61 63, 6:41 45, Figure 2E.) Accordingly, the controller 420 of Van Os is configured to display at least one object or icon as being fixed and not moveable. 42 BlackBerry Exhibit 1004, pg. 42

43 62. Thus, it is my opinion that the Van Os and Cox combination renders obvious claim element 1[c]. Claim 1[d]: to release the at least one object according to the second touching action such that the at least one object can be moved, 63. I understand claim element 1[d] to require that the second touching action releases the object such that it can be moved. 43 BlackBerry Exhibit 1004, pg. 43

44 64. As explained above for claim element 1[b], in my opinion, the first predefined user action discussed in Van Os corresponds to the claimed second touching action. The first predefined user action causes the positions of one or more of the plurality of displayed icons to be varied, which notifies the user that the position fixation of the icon(s) has been released such that the one or more icons can be reconfigured or moved. (Ex. 1005, 3:49 51.) The varying of the positions of the one or more icons may intuitively indicate to the user that the positions of the one or more icons may be reconfigured by the user. The user may modify, adapt and/or reconfigure the positions of the one or more icons. (Id. at 3:49 53.) In the above passage of Van Os, I interpret reconfigured / reconfigure to mean moved / move. 65. As an example of how the positions of the one or more icons are varied, Van Os discloses animating the icons to simulate floating on a surface corresponding to a surface of the display (Ex. 1005, 3:40 43), which again, informs the user that the icons have been released from a fixed position. 66. Further, as discussed above for claim element 1[b], the user may initiate the first predefined user action by selecting one or more 44 BlackBerry Exhibit 1004, pg. 44

45 appropriate physical buttons on the portable device, by a gesture (such as making contact and swiping one or more fingers across the touchsensitive display or making contact and holding for more than a predefined time period) and/or by selecting one or more soft buttons (such as one or more icons that are displayed on the touch-sensitive display). (Ex. 1005, 4:14 67.) 67. Thus, it is my opinion that the Van Os and Cox combination renders obvious claim element 1[d]. Claim 1[e]: to move the at least one object from a first position to a second position on the touch screen corresponding to the first touching action on the at least one object, and 68. I understand claim element 1[e] to require that the first touching action on the object moves the object from a first position to a second position on the touch screen. 69. As explained above for claim element 1[b], Van Os describes a position adjustment process which includes dragging an object or icon from a first position and dropping it at a second position on the touch screen: [t]he user may drag the respective icon to a second position. (Ex. 1005, 3:58 59, Figure 2C.) This corresponds to the claimed first touching action. 45 BlackBerry Exhibit 1004, pg. 45

46 70. Van Os further describes the position adjustment process as follows: [a] point of contact with the touch-sensitive display at a first position of a respective icon is detected (116). Movement of the point of contact to a second position is detected (118). Movement of the respective icon to the second position is displayed and the respective icon is displayed at the second position (120). (Ex. 1005, 4:54 60, Figure 1.) 46 BlackBerry Exhibit 1004, pg. 46

47 71. Thus, it is my opinion that the Van Os and Cox combination renders obvious claim element 1[e]. Claim 1[f]: to fix the moved object to the second position on the touch screen based on a manipulation of a position fixing button that is separate from the first and second touching actions, 72. I understand claim element 1[f] to require a manipulation of a position fixing button to fix the moved object to the second position on the touch screen, where such manipulation action is separate from the first and second touching actions. 73. Van Os discloses a second predefined user action that terminates the interface reconfiguration process and fixes the positions of the one or more moved icons to the second position when a user selects or deselects a physical button on the portable device or a soft button. In my opinion, the second predefined user action in Van Os corresponds to the claimed fix[ing] the moved object to the second position on the touch screen based on a manipulation of a position fixing button. (Ex. 1005, 4:54 67.) 74. Further, the second predefined user action in Van Os is separate from the dragging and moving of the icon from a first position and dropping it at a second position on the touch screen, which 47 BlackBerry Exhibit 1004, pg. 47

48 corresponds to the claimed first touching action, and the first predefined user action, which corresponds to the claimed second touching action, as explained above for claim elements 1[b], 1[d], and 1[e]. 75. Thus, it is my opinion that the Van Os and Cox combination renders obvious claim element 1[f]. Claim 1[g]: wherein the first touching action is a dragging and dropping touching action on the touch screen, 76. As explained above for claim element 1[e] and shown in Figure 2C of Van Os above, for example, Van Os discloses moving or adjusting a position of the icon or object by dragging it from a first position and dropping it at a second position on the touch screen, which corresponds to the claimed first touching action. The user may drag the respective icon to a second position. (Ex. 1005, 3:58 59, Figure 2C.) Figure 1 of Van Os further describes the position adjustment process as follows: [a] point of contact with the touch-sensitive display at a first position of a respective icon is detected (116). Movement of the point of contact to a second position is detected (118). Movement of the respective icon to the second position is displayed and the respective icon is displayed at the second position (120). (Id. at 4:54 60, Figure 1.) 48 BlackBerry Exhibit 1004, pg. 48

49 49 BlackBerry Exhibit 1004, pg. 49

50 77. Thus, it is my opinion that the Van Os and Cox combination renders obvious claim element 1[g]. Claim 1[h]: wherein the second touching action includes touching the at least one object for a predetermined time period, and 78. As explained above for claim element 1[d], Van Os discloses that the first predefined user action, which corresponds to the claimed second touching action, initiates an interface reconfiguration process (step 112 in Figure 1) and varies the position(s) of one or more of the plurality of displayed icons (step 114 in Figure 1), releasing the icons or objects from a fixed or stationary position(s) such that they can be repositioned or moved. (Ex. 1005, 4:47 56, Figure 1.) 79. In particular, as discussed above for claim element 1[b], the user may initiate the first predefined user action by selecting one or more appropriate physical buttons on the portable device, by a gesture (such as making contact and swiping one or more fingers across the touch-sensitive display or making contact and holding for more than a predefined time period) and/or by selecting one or more soft buttons (such as one or more icons that are displayed on the touchsensitive display). (Emphasis added.) (Ex. 1005, 4:14 67.) 50 BlackBerry Exhibit 1004, pg. 50

51 80. I understand Van Os disclosure of making contact and holding for more than a predefined time period to correspond to the claimed touching the at least one object for a predetermined time period. In particular, Van Os discloses that the icons may be activated when the user makes contact at the position of the icon... (Ex. 1005, 5:34 35.) Accordingly, as would be understood by a POSITA, the first predefined user action in Van Os includes the user touching an icon or object for a predetermined time period. 81. Thus, it is my opinion that the Van Os and Cox combination renders obvious claim element 1[h]. Claim 1[i]: wherein the control unit is further configured to display differently a shape of the object when the position fixation of the object is released. 82. As explained above for claim element 1[d], Van Os discloses that the first predefined user action initiates an interface reconfiguration process (step 112 in Figure 1) and varies the position(s) of one or more of the plurality of displayed icons (step 114 in Figure 1), releasing the icons or objects from a fixed or stationary position(s). (Ex. 1005, 4:47 56, Figure 1.) As an example of how the positions of the icons may be varied, Van Os discloses animating the one or more 51 BlackBerry Exhibit 1004, pg. 51

52 icons to simulate floating of the one or more icons on a surface corresponding to a surface of a display in the portable device. (Emphasis added.) (Id. at 3:36 53.) 83. Under the broadest reasonable interpretation, I understand the shape of the object to mean the appearance of the object. As such, Van Os discloses changing an appearance of the icon when the fixed position of the icon is released by animating the one or more icons to simulate floating. 84. Cox further teaches that, in response to input from a user, the icon s appearance changes to indicate that the icon is now associated with an alternate operation. (Ex. 1006, Abstract, 1:9 13.) The icon s appearance can be changed by changing textual information disposed on the icon, by changing the color of the icon, or by changing the geometry (e.g., size or shape) of the icon. (Id. at 2:30 34.) 85. For example, FIG. 5A of Cox below illustrates a pushbutton icon 74 which includes text disposed thereon that associates the pushbutton icon 74 with a particular operation. The text describes a Find operation that may be used as part of a text replacement function in a word processing program. (Ex. 1006, 5:6 12, FIG. 5A.) If 52 BlackBerry Exhibit 1004, pg. 52

53 an alternative operation is requested by the user, then the user interface module 60 causes the pushbutton icon 74 to change appearance at block 84. This is illustrated in FIG. 5B where the text disposed on the pushbutton icon 74 is changed from Find to Replace to associate the pushbutton icon 74 with an alternative operation. (Id. at 5:34 43, FIG. 5B.) 86. Changing the shape of an object or icon when an alternate operation is requested by the user, as disclosed in Cox, as opposed to changing the appearance of the icon by animating or varying the position of the icon when the user initiates the interface reconfiguration 53 BlackBerry Exhibit 1004, pg. 53

54 mode, as discussed in Van Os, is simply a matter of design choice between known alternatives for a POSITA because both are ways to intuitively inform the user that the position fixation of the object is released. Van Os itself teaches that [t]he varying of the positions of the one or more icons may intuitively indicate to the user that the positions of the one or more icons may be reconfigured by the user. The user may modify, adapt and/or reconfigure the positions of the one or more icons. (Ex. 1005, 3:49 53.) Further, based on my extensive experience in the field of user interface design and engineering, changing the visual appearance of an icon to indicate a change in mode was common in human computer interaction at the time of the alleged invention. 87. Further, a POSITA would have a reasonable expectation of success in making that modification because Van Os user interface would perform equally well with either animated icons or icons that change in shape because both provide the same predictable result of changing the icons appearance for the same purpose of informing the user that the position fixation of the icon has been released and is ready for reconfiguration. Indeed, a change in an icon s shape is itself an 54 BlackBerry Exhibit 1004, pg. 54

55 animation, as the user first sees an original shape of the icon, and then later, a new shape. 88. Thus, it is my opinion that the Van Os and Cox combination renders obvious claim element 1[i]. Claim 10 of the 506 Patent 89. Claim 10 is directed to a method ( [a] method of controlling a mobile terminal ) corresponding to the functions recited in apparatus claim 1 ( [a] mobile terminal ) discussed above. Accordingly, claim 10 is obvious for the same reasons as claim 1. Thus, it is my opinion that the combination of Van Os and Cox renders obvious claim 10. Claim 10[a]: A method of controlling a mobile terminal, the method comprising: 90. See analysis of claim elements 1[a] and 1[c] supra in 50-52, Thus, it is my opinion that the combination of Van Os and Cox renders obvious claim element 10[a]. 55 BlackBerry Exhibit 1004, pg. 55

56 Claim 10[b]: displaying at least one object on a touchscreen of the mobile terminal; 91. See analysis of claim element 1[b] supra in Thus, it is my opinion that the combination of Van Os and Cox renders obvious claim element 10[b]. Claim 10[c]: sensing, via a sensor of the touch screen, first and second touching actions on the at least one object; 92. See analysis of claim element 1[b] supra in Thus, it is my opinion that the combination of Van Os and Cox renders obvious claim element 10[c]. Claim 10[d]: displaying the at least one object as being fixed and not moveable; 93. See analysis of claim element 1[c] supra in Thus, it is my opinion that the combination of Van Os and Cox renders obvious claim element 10[d]. Claim 10[e]: releasing the at least one object according to the second touching action such that the at least one object can be moved; 94. See analysis of claim element 1[d] supra in Thus, it is my opinion that the combination of Van Os and Cox renders obvious claim element 10[e]. 56 BlackBerry Exhibit 1004, pg. 56

57 Claim 10[f]: moving the at least one object from a first position to a second position on the touch screen corresponding to the first touching action on the at least one object; and 95. See analysis of claim element 1[e] supra in Thus, it is my opinion that the combination of Van Os and Cox renders obvious claim element 10[f]. Claim 10[g]: fixing the moved object to the second position on the touch screen based on a manipulation of a position fixing button that is separate from the first and second touching actions, 96. See analysis of claim element 1[f] supra in Thus, it is my opinion that the combination of Van Os and Cox renders obvious claim element 10[g]. Claim 10[h]: wherein the first touching action is a dragging and dropping touching action on the touch screen, 97. See analysis of claim element 1[g] supra in Thus, it is my opinion that the combination of Van Os and Cox renders obvious claim element 10[h]. 57 BlackBerry Exhibit 1004, pg. 57

58 Claim 10[i]: wherein the second touching action includes touching the at least one object for a predetermined time period, and 98. See analysis of claim element 1[h] supra in Thus, it is my opinion that the combination of Van Os and Cox renders obvious claim element 10[i]. Claim 10[j]: wherein the method further comprises displaying differently a shape of the object when the position fixation of the object is released. 99. See analysis of claim element 1[i] supra in Thus, it is my opinion that the combination of Van Os and Cox renders obvious claim element 10[j]. B. Ground 2: Claims 2 and 11 Are Obvious Over Van Os, Cox, and Woolley 100. I understand claim 2 of the 506 patent to require that at least one of sound and a vibration is outputted to indicate that the moved object is fixed to the second position. 58 BlackBerry Exhibit 1004, pg. 58

59 Claim 2: The mobile terminal of claim 1, further comprising: an output unit configured to output at least one of a sound and a vibration to indicate that the moved object is fixed to the second position according to a control of the control unit As I discussed above for claim element 1[f] in Section VI.A, Van Os discloses selecting or deselecting a physical button on the portable device or a soft button to terminate the interface reconfiguration process and fix the position of one or more moved icons to the second position. (Ex. 1005, 4:54 67.) Figure 2E of Van Os below, for example, illustrates the portable communication device 200 after the interface reconfiguration mode has been terminated, with the icons in GUI 210 visually presented in stationary positions. (Id. at 6:41 44.) 59 BlackBerry Exhibit 1004, pg. 59

60 102. Van Os further discloses providing audio and/or tactile feedback when an icon is moved to a new position, such as an audible chime and/or a vibration (Ex. 1005, 6:16 19); however, such feature is not described in the Van Os provisional application, U.S. Provisional Patent Application No. 60/755,368 ( the 368 provisional application ) (Ex. 1009). 60 BlackBerry Exhibit 1004, pg. 60

61 103. Woolley describes a kiosk touchpad with a touch-sensitive surface, which includes a speaker for auditory feedback to indicate, for example, when a zone has been selected. (Ex. 1007, Abstract.) As illustrated in FIG. 4 of Woolley below, the touch pad 21 includes, at least, a cover plate 31, a touch-sensitive surface 26, and electronic components 45 mounted on the reverse side thereof, including a speaker 71 for auditory feedback. (Id. at [0049].) 104. Woolley further suggests employing multiple forms of sensory feedback simultaneously: auditory, tactile, and visual. (Ex. 1007, [0049].) 105. Accordingly, in my opinion, it would have been obvious to a POSITA at the time of the invention to add auditory or tactile feedback to Van Os user interface because those features would have predictably resulted in the same advantage of providing the user with additional 61 BlackBerry Exhibit 1004, pg. 61

62 modes of feedback to confirm that the position fixing button has been selected and that a moved icon has been fixed to the second position. It is well understood in the field of user interface design, and specifically, with respect to human computer interaction, that adding additional modalities of feedback can only contribute to increased awareness. Further, disclosure of audio and tactile feedback in Van Os provides additional evidence of the level of skill in the art and further demonstrates that a POSITA would have a reasonable expectation of success in combining Van Os and Woolley Thus, it is my opinion that the Van Os, Cox, and Woolley combination renders obvious claim 2. Claim 11: The method of claim 10, further comprising: outputting at least one of a sound and a vibration to indicate that the moved object is fixed to the second position Claim 11 is directed to a method ( [t]he method... further comprising: outputting at least one of a sound and a vibration... ) corresponding to the function ( an output unit configured to output at least one of a sound and a vibration... ) recited in claim 2 discussed above. Accordingly, claim 11 is obvious for the same reasons as claim BlackBerry Exhibit 1004, pg. 62

63 Thus, it is my opinion that the Van Os, Cox, and Woolley combination renders obvious claim 11. C. Ground 3: Claim 8 Is Anticipated By Van Os 108. Claim 8 of the 506 patent is also directed to a mobile terminal, identical to claim 1 except that it does not include claim element 1[i]. Accordingly, it is my opinion that Van Os teaches every element of claim 8 as set forth above for claim elements 1[a]-1[h]. Claim 8[a]: A mobile terminal comprising: 109. See analysis of claim element 1[a] supra in Thus, it is my opinion that Van Os anticipates claim element 8[a]. Claim 8[b]: a touch screen configured to display at least one object and to sense first and second touching actions on the at least one object; and 110. See analysis of claim element 1[b] supra in paragraph Thus, it is my opinion that Van Os anticipates claim element 8[b]. Claim 8[c]: a control unit configured to display the at least one object as being fixed and not moveable, 111. See analysis of claim element 1[c] supra in Thus, it is my opinion that Van Os anticipates claim element 8[c]. 63 BlackBerry Exhibit 1004, pg. 63

64 Claim 8[d]: to release the at least one object according to the second touching action such that the at least one object can be moved, 112. See analysis of claim element 1[d] supra in Thus, it is my opinion that Van Os anticipates claim element 8[d]. Claim 8[e]: to move the at least one object from a first position to a second position on the touch screen corresponding to the first touching action on the at least one object, and 113. See analysis of claim element 1[e] supra in Thus, it is my opinion that Van Os anticipates claim element 8[e]. Claim 8[f]: to fix the moved object to the second position on the touch screen based on a manipulation of a position fixing button that is separate from the first and second touching actions, 114. See analysis of claim element 1[f] supra in Thus, it is my opinion that Van Os anticipates claim element 8[f]. Claim 8[g]: wherein the first touching action is a dragging and dropping touching action on the touch screen, and 115. See analysis of claim element 1[g] supra in Thus, it is my opinion that Van Os anticipates claim element 8[g]. 64 BlackBerry Exhibit 1004, pg. 64

65 Claim 8[h]: wherein the second touching action includes touching the at least one object for a predetermined time period See analysis of claim element 1[h] supra in Thus, it is my opinion that Van Os anticipates claim element 8[h]. D. Ground 4: Claim 9 Is Obvious Over Van Os and Woolley 117. Since claim 9 is identical to claim 2 discussed above, claim 9 is obvious for the same reasons. Claim 9: The mobile terminal of claim 8, further comprising: an output unit configured to output at least one of a sound and a vibration to indicate that the moved object is fixed to the second position according to a control of the control unit As I discussed above for claim element 1[f] in Section VI.A, Van Os discloses selecting or deselecting a physical button on the portable device or a soft button to terminate the interface reconfiguration process and fix the position of one or more moved icons to the second position. (Ex. 1005, 4:54 67.) Figure 2E of Van Os below, for example, illustrates the portable communication device 200 after the interface reconfiguration mode has been terminated, with the icons in GUI 210 visually presented in stationary positions. (Id. at 6:41 44.) 65 BlackBerry Exhibit 1004, pg. 65

66 119. Van Os further discloses providing audio and/or tactile feedback when an icon is moved to a new position, such as an audible chime and/or a vibration (Ex. 1005, 6:16 19); however, such feature is not described in the Van Os 368 provisional application (Ex. 1009) Woolley describes a kiosk touchpad with a touch-sensitive surface, which includes a speaker for auditory feedback to indicate, for example, when a zone has been selected. (Ex. 1007, Abstract.) As 66 BlackBerry Exhibit 1004, pg. 66

67 illustrated in FIG. 4 of Woolley below, the touch pad 21 includes, at least, a cover plate 31, a touch-sensitive surface 26, and electronic components 45 mounted on the reverse side thereof, including a speaker 71 for auditory feedback. (Id. at [0049].) 121. Woolley further suggests employing multiple forms of sensory feedback simultaneously: auditory, tactile, and visual. (Ex. 1007, [0049].) 122. Accordingly, in my opinion, it would have been obvious to a POSITA at the time of the invention to add auditory or tactile feedback to Van Os user interface because those features would have predictably resulted in the same advantage of providing the user with additional modes of feedback to confirm that the position fixing button has been selected and that a moved icon has been fixed to the second position. It is well understood in the field of user interface design, and specifically, 67 BlackBerry Exhibit 1004, pg. 67

68 with respect to human computer interaction, that adding additional modalities of feedback can only contribute to increased awareness. Further, disclosure of audio and tactile feedback in Van Os provides additional evidence of the level of skill in the art and further demonstrates that a POSITA would have a reasonable expectation of success in combining Van Os and Woolley Thus, it is my opinion that the Van Os and Woolley combination renders obvious claim 9. E. Ground 5: Claims 1, 8, and 10 Are Obvious Over The MessagePad Handbook Claim 1[a]: A mobile terminal comprising: 124. The 506 patent describes the mobile terminal as a mobile phone, a PDA..., a PMP (portable multimedia player), a game player, and the like. (Ex. 1001, 3:1 3.) 125. The MessagePad is a PDA, which features a variety of personal-organization applications. (Ex. 1008, ) The illustration from the MessagePad Handbook below, for example, shows pictures of the various applications the MessagePad offers. (Id. at ) 68 BlackBerry Exhibit 1004, pg. 68

69 69 BlackBerry Exhibit 1004, pg. 69

70 126. Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 1[a]. Claim 1[b]: a touch screen configured to display at least one object and to sense first and second touching actions on the at least one object; and 127. As discussed above with respect to claim element 1[a], the MessagePad is a PDA which includes a touch-sensitive screen. As described in the MessagePad Handbook, [o]ne of the main ways you interact with your MessagePad is by using the pen... to tap items, 70 BlackBerry Exhibit 1004, pg. 70

71 such as buttons or words, or to write information. To tap, you simply touch the pen to the screen. (Ex. 1008, 8.) 128. As would be understood by a POSITA, the touch-sensitive display technology used in the MessagePad includes a resistive touchscreen for sensing one or more points of contact with the touchsensitive screen. Touch based on resistance relies on the physical deformation of a sensor as the user touches the display to track the position of contact, as it moves. The output of such resistive sensing system is the same as that of an electrostatic capacitive-type sensing system, as disclosed, for example, in the 506 patent. [T]he touchscreen 140 can be configured with one of a contact electrostatic capacitive type, a pressure resistive overlay type, an infrared beam detective type, a surface acoustic wave type, an integral strain gauge type, a piezoelectric type, and the like. (Ex. 1001, 3:66 4:3.) 129. Below is a sample screen shot of the MessagePad s touch screen displaying a plurality of icons or objects, such as an In Box, Out Box, Calls, etc. (Ex. 1008, 160.) 71 BlackBerry Exhibit 1004, pg. 71

72 130. In order to move or reposition at least one of the icons above, the MessagePad Handbook discloses that the user simply drags the icon from a first position and drops it at a second position, which corresponds to the claimed first touching action. (Ex. 1008, 160.) To move an icon, select it... and drag the icon where you want to place it. (Id.) If not implicit, it would have been obvious to a POSITA that the icon is dropped at the second position by lifting the pen from the screen, as this is the logical way to stop the dragging function It is standard practice in user interface design that operations are terminated by changing a state of the input device. As an example, a click with a mouse constitutes a depressing and releasing action of the mouse button. Also with the mouse, a drag is a 72 BlackBerry Exhibit 1004, pg. 72

73 movement of the mouse that takes place between a depressing and releasing action of the mouse button. Accordingly, with respect to the MessagePad, when the user is instructed to drag the icon where you want to place it, although it is not explicitly stated in the MessagePad Handbook, the user has only two options available after the icon has been dragged to the second position, the user can either continue sliding the pen along the screen or lift it from the screen. If the user continues with the first option, the system has no way of knowing that the input is part of the drag operation. Hence, as would be understood by a POSITA, the only way for the user to terminate the drag operation is to lift the pen from the screen Regarding the claimed second touching action, before the icon can be dragged from the first position to the second position, the user must select the icon. To select an icon, hold down the pen on or near the item until a heavy mark appears under the pen (the MessagePad makes a squeak), and draw the mark over or around the icon... The icon is selected. (Ex. 1008, 160.) Accordingly, as would be understood by a POSITA, the user s action of selecting the icon by highlighting it, either by drawing a mark over or around the icon, 73 BlackBerry Exhibit 1004, pg. 73

74 corresponds to the claimed second touching action, as such action releases the icon from a fixed position such that it can be moved when the user drags the icon Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 1[b]. Claim 1[c]: a control unit configured to display the at least one object as being fixed and not moveable, 134. Regarding a control unit, the MessagePad Handbook discloses that [t]his product is powered by Advanced RISC Machine s high performance, power efficient, 32-bit RISC microprocessor. (Ex. 1008, iii.) As would be understood by a POSITA, the RISC microprocessor is a microprocessor based on reduced instruction set computing and is a type of central processing unit, or CPU. Such microprocessor is responsible for controlling the input, processing, and output of the MessagePad Further, a sample screen shot of the MessagePad s touch screen below is displaying several icons or objects as being fixed and not moveable in the Extras Drawer. (Ex. 1008, 160.) Out of the eleven icons displayed, only one of them is not fixed, as evidenced by the highlighted 74 BlackBerry Exhibit 1004, pg. 74

75 icon. As discussed above with respect to claim element 1[b], highlighting an icon, either by drawing a mark over or around the icon, corresponds to the claimed second touching action, which signifies that the icon has been released from a fixed position such that it can be moved when the user drags the icon Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 1[c]. Claim 1[d]: to release the at least one object according to the second touching action such that the at least one object can be moved, 137. I understand claim element 1[d] to require that the second touching action releases the object such that it can be moved. 75 BlackBerry Exhibit 1004, pg. 75

76 138. As explained above for claim element 1[b], before an icon can be moved and dragged from a first position to the second position, the user must select the icon, as illustrated in the sample screen shot below. To select an icon, hold down the pen on or near the item until a heavy mark appears under the pen (the MessagePad makes a squeak), and draw the mark over or around the icon... The icon is selected. (Ex. 1008, 160, 212.) 139. As would be understood by a POSITA, the user s action of selecting the icon, either by drawing a mark over or around the icon, corresponds to the claimed second touching action, as such action 76 BlackBerry Exhibit 1004, pg. 76

77 releases the icon from a fixed position such that it can be moved when the user drags the icon to a second position Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 1[d]. Claim 1[e]: to move the at least one object from a first position to a second position on the touch screen corresponding to the first touching action on the at least one object, and 141. I understand claim element 1[e] to require that the first touching action on the object moves the object from a first position to a second position on the touch screen As explained above for claim 1[b], in order to move or reposition at least one of the icons, the MessagePad Handbook discloses that the user drags the icon from a first position and drops it at a second position, which corresponds to the claimed first touching action. (Ex. 1008, 160.) To move an icon, select it... and drag the icon where you want to place it. (Id.) If not implicit, it would be obvious to a POSITA that the icon is dropped at the second position by lifting the pen from the screen, as this is the logical way to stop the dragging function. 77 BlackBerry Exhibit 1004, pg. 77

78 143. It is standard practice in user interface design that operations are terminated by a change in a state of the input device. As an example, a click with a mouse constitutes a depressing and releasing action of the mouse button. Also with the mouse, a drag is a movement of the mouse that takes place between a depressing and releasing action of the mouse button. Accordingly, with respect to the MessagePad, when the user is instructed to drag the icon where you want to place it, although it is not explicitly stated in the MessagePad Handbook, the user has only two options available after the icon has been dragged to the second position, the user can either continue sliding the pen along the screen or lift it from the screen. If the user continues with the first option, the system has no way of knowing that the input is part of the drag operation. Hence, as would be understood by a POSITA, the only way for the user to terminate the drag operation is to lift the pen from the screen Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 1[e]. 78 BlackBerry Exhibit 1004, pg. 78

79 Claim 1[f]: to fix the moved object to the second position on the touch screen based on a manipulation of a position fixing button that is separate from the first and second touching actions, 145. I understand claim element 1[f] to require a manipulation of a position fixing button to fix the moved object to the second position on the touch screen, where such manipulation action is separate from the first and second touching actions Once an object has been released and is movable, to fix that object in position the MessagePad Handbook discloses deselecting that icon by tapping the pen anywhere outside the selection. (Ex. 1008, ) As would be understood by a POSITA, the user s action of deselecting the icon fixes the moved icon in position on the touch screen, as this is diametrically opposite to the user s action of selecting the icon, which, as discussed above in claim elements [b] and [d], releases the icon from a fixed position such that it can be moved when the user drags the icon to a second position Further, a POSITA would understand that any portion of the touch screen outside of the selected icon is essentially one large position fixing button, which can be tapped or manipulated by the user to deselect and fix the moved icon on the touch screen. A skilled artisan 79 BlackBerry Exhibit 1004, pg. 79

80 would understand that in the field of user interface design, a button is an arbitrary region of a screen that, when tapped, results in the execution of a function. Whether the arbitrary region of the screen is drawn as a visible button is inconsequential Moreover, as would be understood by a POSITA, it also would have been an obvious design choice to have the user fix the moved icon in the second position by manipulating an actual button, as the MessagePad Handbook already teaches that buttons can be provided on the touch screen to select the device s functions, such as opening documents, scrolling, and undoing actions. (Ex. 1008, ) 80 BlackBerry Exhibit 1004, pg. 80

81 149. Additionally, as discussed above for claim 1[b], in order to move or reposition at least one of the icons, the MessagePad Handbook discloses that the user drags the icon from a first position and drops it at a second position, which corresponds to the claimed first touching action. (Ex. 1008, 160.) Also, before the icon can be dragged from the first position to the second position, the user must select the icon by drawing a mark over or around the icon, which corresponds to the claimed second touching action. As such, the user s action of deselecting the icon, or alternatively, manually selecting a button to fix 81 BlackBerry Exhibit 1004, pg. 81

82 the moved icon in position on the touch screen, is separate from the claimed first and second touching actions Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 1[f]. Claim 1[g]: wherein the first touching action is a dragging and dropping touching action on the touch screen, 151. As explained above for claims 1[b] and 1[e], in order to move the object from a first position to a second position on the touch screen, the user drags the icon from a first position and drops it at a second position, which corresponds to the claimed first touching action. (Ex. 1008, 160.) To move an icon, select it... and drag the icon where you want to place it. (Id.) As explained above, if not implicit, it would have been obvious to a POSITA that the icon is dropped at the second position by lifting the pen from the screen, as this is the logical way to stop the dragging function It is standard practice in user interface design that operations are terminated by a change in a state of the input device. As an example, a click with a mouse constitutes a depressing and releasing action of the mouse button. Also with the mouse, a drag is a movement of the mouse that takes place between a depressing and 82 BlackBerry Exhibit 1004, pg. 82

83 releasing action of the mouse button. Accordingly, with respect to the MessagePad, when the user is instructed to drag the icon where you want to place it, although it is not explicitly stated in the MessagePad Handbook, the user has only two options available after the icon has been dragged to the second position, the user can either continue sliding the pen along the screen or lift it from the screen. If the user continues with the first option, the system has no way of knowing that the input is part of the drag operation. Hence, as would be understood by a POSITA, the only way for the user to terminate the drag operation is to lift the pen from the screen Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 1[g]. Claim 1[h]: wherein the second touching action includes touching the at least one object for a predetermined time period, and 154. As explained above for claims 1[b] and 1[d], in order to release the object such that it can be moved, which corresponds to the claimed second touching action, the user must select the icon. To select an icon, hold down the pen on or near the item until a heavy mark appears under the pen (the MessagePad makes a squeak), and 83 BlackBerry Exhibit 1004, pg. 83

84 draw the mark over or around the icon... The icon is selected. (Emphases added.) (Ex. 1008, 160, 212.) 155. Accordingly, as would be understood by a POSITA, the user s action of hold[ing] down the pen on or near the item until a heavy mark appears under the pen, indicates that the user is touching the icon or object for a predetermined time period, as the heavy mark used to highlight the icon does not appear under the pen instantaneously. Indeed, because it is clear throughout the disclosure of the MessagePad Handbook that the holding action is meant to differentiate the 84 BlackBerry Exhibit 1004, pg. 84

85 indicated action from other inputs, it is necessary that a period of time be allowed to elapse before entering the selection mode; otherwise, the squeak noise described in the Handbook would occur with every input At the very least, it would be obvious to a POSITA to use a predetermined time period so that a user of the device can anticipate how long an object has to be touched in order to highlight it and also how to avoid inadvertent highlighting of the object. Further, a POSITA would understand the importance of providing the user with a set of input gestures that are physically distinct from one another to avoid ambiguous inputs. In this case, it is essential that a selection action not be confused with a drawing input. Without a predetermined period of time the pen must be held on or near the object or icon, there would be no way of distinguishing an input intended to initiate a selection from an input intended to initiate drawing Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 1[h]. 85 BlackBerry Exhibit 1004, pg. 85

86 Claim 1[i]: wherein the control unit is further configured to display differently a shape of the object when the position fixation of the object is released As explained above for claims 1[b] and 1[d], in order to release the position fixation of the object such that it can be moved, the user must select the icon. Select an icon by placing the pen on or near the icon until a heavy mark appears under the pen and you hear a squeak. Then draw the highlighting mark over or around the icon. (Emphasis added.) (Ex. 1008, 160, 212.) 159. Accordingly, as would be understood by a POSITA, the user s action of draw[ing] the highlighting mark over or around the icon, changes a shape or appearance of the icon or object, intuitively 86 BlackBerry Exhibit 1004, pg. 86

87 informing the user that the position fixation of the icon is released such that the icon can be dragged to another position. Without such visual feedback to indicate this change in state to the user, the user would be unable to differentiate one mode of operation from another, not knowing when the icon had been released from a fixed position so that it may be moved Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 1[i]. Claim 8 of the 506 Patent 161. Claim 8 of the 506 patent is also directed to a mobile terminal, identical to claim 1 discussed above except that it does not include claim element 1[i]. Accordingly, it is my opinion that the MessagePad Handbook renders obvious every element of claim 8 as set forth above for claim elements 1[a]-1[h]. Claim 8[a]: A mobile terminal comprising: 162. See analysis of claim element 1[a] supra in Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 8[a]. 87 BlackBerry Exhibit 1004, pg. 87

88 Claim 8[b]: a touch screen configured to display at least one object and to sense first and second touching actions on the at least one object; and 163. See analysis of claim element 1[b] supra in Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 8[b]. Claim 8[c]: a control unit configured to display the at least one object as being fixed and not moveable, 164. See analysis of claim element 1[c] supra in Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 8[c]. Claim 8[d]: to release the at least one object according to the second touching action such that the at least one object can be moved, 165. See analysis of claim element 1[d] supra in Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 8[d]. 88 BlackBerry Exhibit 1004, pg. 88

89 Claim 8[e]: to move the at least one object from a first position to a second position on the touch screen corresponding to the first touching action on the at least one object, and 166. See analysis of claim element 1[e] supra in Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 8[e]. Claim 8[f]: to fix the moved object to the second position on the touch screen based on a manipulation of a position fixing button that is separate from the first and second touching actions, 167. See analysis of claim element 1[f] supra in Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 8[f]. Claim 8[g]: wherein the first touching action is a dragging and dropping touching action on the touch screen, and 168. See analysis of claim element 1[g] supra in Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 8[g]. 89 BlackBerry Exhibit 1004, pg. 89

90 Claim 8[h]: wherein the second touching action includes touching the at least one object for a predetermined time period See analysis of claim element 1[h] supra in Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 8[h]. Claim 10 of the 506 Patent 170. Claim 10 is directed to a method ( [a] method of controlling a mobile terminal ) corresponding to the functions recited in apparatus claim 1 ( [a] mobile terminal ) discussed above. Accordingly, claim 10 is obvious for the same reasons as claim 1. Thus, it is my opinion that the MessagePad Handbook renders obvious claim 10. Claim 10[a]: A method of controlling a mobile terminal, the method comprising: 171. See analysis of claim elements 1[a] and 1[c] supra in , Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 10[a]. 90 BlackBerry Exhibit 1004, pg. 90

91 Claim 10[b]: displaying at least one object on a touchscreen of the mobile terminal; 172. See analysis of claim element 1[b] supra in Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 10[b]. Claim 10[c]: sensing, via a sensor of the touch screen, first and second touching actions on the at least one object; 173. See analysis of claim element 1[b] supra in Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 10[c]. Claim 10[d]: displaying the at least one object as being fixed and not moveable; 174. See analysis of claim element 1[c] supra in Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 10[d]. Claim 10[e]: releasing the at least one object according to the second touching action such that the at least one object can be moved; 175. See analysis of claim element 1[d] supra in Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 10[e]. 91 BlackBerry Exhibit 1004, pg. 91

92 Claim 10[f]: moving the at least one object from a first position to a second position on the touch screen corresponding to the first touching action on the at least one object; and 176. See analysis of claim element 1[e] supra in Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 10[f]. Claim 10[g]: fixing the moved object to the second position on the touch screen based on a manipulation of a position fixing button that is separate from the first and second touching actions, 177. See analysis of claim element 1[f] supra in Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 10[g]. Claim 10[h]: wherein the first touching action is a dragging and dropping touching action on the touch screen, 178. See analysis of claim element 1[g] supra in Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 10[h]. 92 BlackBerry Exhibit 1004, pg. 92

93 Claim 10[i]: wherein the second touching action includes touching the at least one object for a predetermined time period, and 179. See analysis of claim element 1[h] supra in Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 10[i]. Claim 10[j]: wherein the method further comprises displaying differently a shape of the object when the position fixation of the object is released See analysis of claim element 1[i] supra in Thus, it is my opinion that the MessagePad Handbook renders obvious claim element 10[j]. F. Ground 6: Claims 2, 9, and 11 Are Obvious Over The MessagePad Handbook and Woolley 181. I understand claim 2 of the 506 patent to require that at least one of sound and a vibration is outputted to indicate that the moved object is fixed to the second position. 93 BlackBerry Exhibit 1004, pg. 93

94 Claim 2: The mobile terminal of claim 1, further comprising: an output unit configured to output at least one of a sound and a vibration to indicate that the moved object is fixed to the second position according to a control of the control unit As I discussed above for claim 1[f] in Section VI.E, and as would be understood by a POSITA, the user s action of deselecting the icon on the MessagePad, by tapping the pen anywhere outside the selection, where any portion of the touch screen outside of the selected icon is considered to be one large position fixing button, fixes the moved icon in position on the screen. (Ex. 1008, ) In my opinion, it also would have been an obvious design choice to have the user manually select a button to fix the moved icon in position on the touchscreen because the MessagePad Handbook teaches that buttons can be provided on the touch screen to select the device s functions, such as opening documents, scrolling, and undoing actions. (Id. at ) 183. Interestingly, I note that although the MessagePad Handbook does not explicitly disclose outputting additional feedback, such as sound or vibration, to indicate that the moved object is fixed to the second position, the MessagePad does make a squeak when the user selects an icon for releasing it from a fixed position, that is by 94 BlackBerry Exhibit 1004, pg. 94

95 hold[ing] down the pen on or near the item until a heavy mark appears under the pen... and draw[ing] the mark over or around the icon... (Ex. 1008,160.) 184. Woolley further describes a kiosk touchpad with a touchsensitive surface, which includes a speaker for auditory feedback to indicate, for example, when a zone has been selected. (Ex. 1007, Abstract.) As illustrated in FIG. 4 of Woolley below, the touch pad 21 includes, at least, a cover plate 31, a touch-sensitive surface 26, and electronic components 45 mounted on the reverse side thereof, including a speaker 71 for auditory feedback. (Id. at [0049].) 185. Woolley also suggests employing multiple forms of sensory feedback simultaneously: auditory, tactile, and visual. (Ex. 1007, [0049].) 95 BlackBerry Exhibit 1004, pg. 95

96 186. Accordingly, in my opinion, it would have been obvious to a POSITA at the time of the invention to add auditory or tactile feedback to the MessagePad s user interface because those features would have predictably resulted in the same advantage of providing the user with additional modes of feedback to confirm that the position fixing button has been selected and that a moved icon has been fixed to the second position. It is well understood in the field of user interface design, and specifically, with respect to human computer interaction, that adding additional modalities of feedback can only contribute to increased awareness, informing the user that a subsequent input will yield a different result than it might have in the previous mode Alternatively, a POSITA would have been motivated to use the MessagePad s existing auditory feedback, currently used for informing the user that the icon has been released from the fixed position, i.e., the MessagePad makes a squeak, for instead informing the user that the position fixing button has been selected and that the icon has been fixed to the second position. (Ex. 1008, 160.) As I have discussed above, any type of feedback indicating a change in the mode of operation of a device is important because it informs the user that a 96 BlackBerry Exhibit 1004, pg. 96

97 subsequent input will yield a result that is different than it might have in the previous mode. As discussed above, the MessagePad squeaks when the user transitions from a normal, stationary mode of operation to a mode in which the icons may be repositioned. It would be an obvious design choice for a POSITA to use of that same type of output when transitioning back from the repositioning mode to the normal, stationary mode, again notifying the user of a change in the mode of operation Thus, it is my opinion that the MessagePad Handbook and Woolley combination renders obvious claim 2. Claim 9: The mobile terminal of claim 8, further comprising: an output unit configured to output at least one of a sound and a vibration to indicate that the moved object is fixed to the second position according to a control of the control unit Since claim 9 is identical to claim 2 discussed above, claim 9 is obvious for the same reasons. Thus, it is my opinion that the MessagePad Handbook and Woolley combination renders obvious claim BlackBerry Exhibit 1004, pg. 97

98 Claim 11: The method of claim 10, further comprising: outputting at least one of a sound and a vibration to indicate that the moved object is fixed to the second position Claim 11 is directed to a method ( [t]he method... further comprising: outputting at least one of a sound and a vibration... ) corresponding to the function ( an output unit configured to output at least one of a sound and a vibration... ) recited in claims 2 and 9 discussed above. Accordingly, claim 11 is obvious for the same reasons as claims 2 and 9. Thus, it is my opinion that the MessagePad Handbook and Woolley combination renders obvious claim 11. VII. CONCLUDING STATEMENTS 191. In signing this declaration, I understand that the declaration will be filed as evidence in a contested case before the Patent Trial and Appeal Board of the United States Patent and Trademark Office. I acknowledge that I may be subject to cross-examination in the case and that cross-examination will take place within the United States. If cross-examination is required of me, I will appear for cross-examination within the United States during the time allotted for cross-examination. 98 BlackBerry Exhibit 1004, pg. 98

99 192. I declare that all statements made herein of my knowledge are true and that all statements made on information and belief are believed to be true; and further, that these statements were made with knowledge that willful false statements and the like so made are punishable by fine or imprisonment, or both, under 18 U.S.C I declare under penalty of perjury under the laws of the United States of America that the foregoing is true and correct Executed on Dr. Daniel J. Wigdor, Ph.D. 99 BlackBerry Exhibit 1004, pg. 99