DETC REPRESENTATION: METRICS FOR ANALYZING SKETCHES: A CRITICAL SURVEY

Transcription

1 Proceedings of the ASME 2012 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference IDETC/CIE 2012 August 12-15, 2012, Chicago, Illinois DETC REPRESENTATION: METRICS FOR ANALYZING SKETCHES: A CRITICAL SURVEY Shraddha Joshi Graduate Research Assistant Department of Mechanical Engineering Clemson University Clemson, SC shraddj@g.clemson.edu Joshua D. Summers Professor and CoES IDEaS Professor Department of Mechanical Engineering Clemson University Clemson, SC joshua.summers@clemson.edu ABSTRACT Sketches are used in design research experiments by various researchers to draw inferences with respect to creativity, representation use, solution quality, and other research goals. Numerous different metrics, such as quantity, sketch quality, and solution quality, are used by the researchers to analyze sketches. This paper presents a survey of the metrics used in the design research community. Characteristics developed by the authors to evaluate these metrics are used to compare the metrics. Further, an initial attempt to identify a relation between type of research approach and characteristics of the metrics is then reported to test the hypothesis that a mature research field would include canonically accepted metrics for experimental studies. The findings indicate that current literature does not provide any formal definition of metrics that are widely accepted in the community. The findings from correlations indicate a general trend of using objective metrics for qualitative research and subjective metrics for mixed method approach. Ultimately, this work lays the foundation for a more systematic approach to evaluating engineering design sketches through critical selection of metrics. 1. MOTIVATION STUDYING SKETCH ANALYSIS METRICS Many design researchers use sketches for answering research questions such as how does the number of sketches influence the project performance, does one ideation method provide better performance in novelty, or simply what is the role of sketching in design. Analyzing sketches is a challenging task because of the subjectivity that may be involved in the process. Also, the way the sketches are analyzed can highly influence the results and the associated inferences. While, analyzing the sketches itself is a challenging task, identifying the metrics that may be and should be used to analyze the sketches poses a bigger challenge. A variety of metrics used to analyze the sketches are available in literature but guidelines as to how a researcher should select from the pool of these metrics for analyzing the sketches seems to be missing. Further, for the case where the researcher needs to develop a new metric to analyze sketches for their specific research needs, guidelines for the characteristics that should be possessed by the metric are not currently available. Such guidelines, if made available, can be useful, especially to novice researchers, in selecting from current pool of metrics or developing new metrics. The overall goal of this research is to develop guidelines for researchers using sketch analysis that can enable selecting appropriate metrics from the available pool or developing new metrics for their research. To this end, this paper first identifies the characteristics of metrics and the metrics relation with the type of research conducted. Thus, the research seeks to address the following questions: RQ-1 What generic trends can be observed from reviewing the metrics currently used to analyze sketches? RQ-2 What are the characteristics of these metrics? Is there a relation between the characteristics of metrics used for evaluating the sketches and the type of research? It would be beneficial to know the characteristics that the metrics used for evaluating the sketches need to possess so that these can be used either while selecting the metrics from the current pool or while developing the new metrics. Selecting metrics to analyze sketches is not straightforward issue due to various factors involved in the decision such as research questions, time and other resources available for analysis, and sensitivity of results to the selected metrics. However, if the 1 Copyright 2012 by ASME

2 relation between the characteristics and type of research is known, then this knowledge can serve as preliminary guidance to the novice researchers to select appropriate metrics to evaluate the sketches for their research. It may be noted here that the scope of this paper is limited to identifying the metrics currently used in the literature and relating the characteristics of the metrics to the type of research. While the papers under review used sketches for various purposes, this research does not aim at studying the roles of sketches in design. 2. RESEARCH APPROACH As mentioned previously, this research aims at identifying general trends from review of metrics currently used for analyzing the sketches. The next step is to draw relation between the characteristics of metrics currently used to evaluate the sketches and the type of research. The end goal is to be able to provide guidelines for appropriate selection of metrics from the current pool available. To that end, a review of literature is conducted. Specifically, the papers were largely drawn from research groups that used sketches to answer research questions. This review led to the development of a pool of metrics currently used by the researchers as listed in Table 1. Since this is a preliminary study, types of research methods used by the researchers were classified broadly into quantitative, qualitative and mixed methods as discussed in detail in section 3.2. The authors then developed a list of characteristics to examine the metrics. Finally, the characteristics of metrics are mapped to the type of research method and the observations are discussed in the paper. The specific scope of interest for this study is limited to research that studies handmade sketch representations found within the mechanical engineering domain. Studies that focus on either novice or experienced engineers are included. The investigation is not limited to experimental studies, but also includes historical and empirical studies that directly evaluate generated sketches. Oxford dictionary defines sketch as unfinished or rough representation that assist to create a more finished drawing [1]. Sketch is also defined as graphical representation created using either freehand or computers used in early design stages [2]. Further, a metric is defined as measurement standard [1]. Metric can also be defined as a measure of X where X could be creativity, quality, quantity or variety [3]. It is within this context that the evaluation proceeds. 3. METRICS FOR SKETCHES AND TYPE OF RESEARCH Sketches are widely used in the design research community for addressing various research questions [4,5,6,7,8,9,10,11,12,13]. To support this previous research, many metrics were developed by the respective researchers to analyze sketches from many different sources. Section 3.1 discusses the metrics currently used in literature. Further, a variety of research methods such as case study, user study, protocol study, and document analysis are used in the design research community. These methods can be broadly classified in to quantitative, qualitative, and mixed methods [14,15]. Section 3.2 discusses the type of research approaches followed by researchers Metrics used in current research While sketches are broadly used by researchers to answer their research questions, analyzing these sketches poses a challenge. If the metrics used for analyzing the sketches are not appropriately selected, it can lead to subjectivity and thus affecting the findings of the study. Wide varieties of metrics have been used by researchers to analyze the sketches. Table 1 provides a list of these metrics with their descriptions as found in the literature. A brief discussion on each approach follows. Pool of Metrics Table 1: Metrics to analyze sketches Description Reference Complexity Defined in terms of line, shading and annotation [4,5] Size Scale Ranging from a thumbnail drawing to A3 size drawing [4] Information Calculated based on size and content complexity [4] Drawing Media Use of soft/hard pencil, ball point pen, color marker, fine liner pen, mixed or freehand media, computer [4] generated Subject matter Entire artifact or subsystem, exploded assembly, artifact feature, artifact in operation, free body diagram Visuals that are in the same Part of multiple grouping on a page but are very object different from one another in type or subject Signals the presence of arrows or Motion directional lines that are show indicator movement on a static plane of paper Applied forces Visuals that are shown with force arrows Part of set Signals a grouping of multiple visuals that are related to each other Views isometric, orthogonal, multiple Quantity of Sketch Sketch Count Demonstration Does it look like bicycle and could it of grasp of work mechanically? concept [5,7,8] Accuracy of proportions Correctness of proportions, 3D perspective Quality [6,7,8,9,10, 16] Does it look like hand [5,7,8] Does it have correct proportions and 3D perspective? High medium low based on solutions for requirements [5,7,8] [10] 2 Copyright 2012 by ASME

3 Linkograph Variety Novelty Quantity of Function Quality Representation 2D or 3D Annotation Representation media Novelty Variety Quality Quality Novelty Accuracy of communication Quantity Quality Finding links of idea with earlier ideas by using evidence from sketch content, context in which ideas are generated Sort solutions in bins and measured by percentage of total bins that teams solution occupy 1-(number of similar concepts/total number of concepts) Measured based on the functions by counting individual ideas generated by individuals and teams Three point scale based on technical feasibility and difficulty Whether sketch is 2D or 3D, single or multiple view 'Support' marks on paper, such as text, calculations and dimensions. Sketches may have no, one type or multiple type of annotations [11,17] [12] [12] [12] [12] [13] [5,13] Tangible, digital or mixed [13] a priori measured based on expected ideas for each function a posteriori measured based on occurrences of same idea for same function Measured by considering the number of branches at various levels of genealogy tree Measured by using quality score table Based on satisfaction of identified functions Measured using function and sub function breakdown Measured based on matching features between ideas Measured by counting ideas that fulfilled functions Measured as improvement in the [16] [16] [16] [18] [18] [18] [19] [19] sketch with addition of ideas McGown, et.al [4] conducted a case study to understand the sketching behavior of students so that the findings could be used to develop computer tools to improve effectiveness and efficiency of conceptual design activity. Case study is a preferred method of research while addressing how and why type of questions and for real life context where the researcher has minimal control over events [20,21]. Four students were observed during the period from November 1996 to February 1997 while they worked on assigned design projects. Data collection entailed sketches drawn from design notebooks and anecdotal evidence through observation notes. The drawing media used by the student, such as hard pencil, soft pencil, ball point pen, fine liner pen, or color marker, is used to characterize the sketches. Figure 1 illustrates a sample sketch from the design notebook. McGown developed a complexity scale based on the details of the lines, shading, and annotation. Further, to assess the amount of information in each sketch, a size scale was used in addition to complexity. The size scale ranged from thumbnail sketches to drawings covering an A3 page. Figure 1: Sample sketches from design notebook [4] Westmoreland conducted a case study using document analysis to understand the types of visual representations used by the students and its correlation with the design phase. A document analysis research method is defined as one in which documents such as books, journals, newspapers, photographs or even electronic reports are systematically studied and analyzed [22] While she was analyzing different visual representations such as sketches, line drawing, CAD renderings, and photographs, the metrics developed were applicable to sketches and thus considered for this study. For analyzing the sketches from design journals, while reusing McGown s complexity scale [4] and Yang s scale [8], Westmoreland developed a wide variety of other metrics such as subject matter, part of multiple object, views, annotations, subject matter such as system, subsystem or artifact and force and motion indicators. A demonstration of the coded data from her analysis can be seen in Figure 2. The metrics were coded as alphabets (A, B, C ) while 0 and 1 refer to whether or not the sketch had that particular aspect. For example, considering the sketches of Figure 2, whether or not the participant had multiple views was indicated by letter M. The code M1 in the sketches below indicates that the sketches have multiple views shown. Further details of each code can be found in the paper. 3 Copyright 2012 by ASME

4 Figure 2: Coded sketches from Westmoreland study Yang conducted a study to understand the correlation between sketching and design outcomes focusing on the conceptual design stage [6]. This study was conducted in the project based design course and sketches generated in the design notebooks were analyzed to draw correlation between design outcomes and sketching. The metric for analyzing the sketches primarily consisted of quantity of sketches. In addition to counting the number of sketches and dimensioned sketches generated by each student, number of concepts generated in morph charts and the grades of students were also used to draw conclusions. The findings from this study indicate that higher number of sketches and more dimensioned sketches correlate significantly with design outcomes as observed from the grades of the students. It may be noted here that the paper did not provide sample sketches from the study and thus fully understanding how the metrics were used to code the data was left to the interpretation of the readers. In another study conducted by Yang and Cham [7,8], sketches generated by students were used to understand the influence of designers sketching ability and role of sketching instruction on design outcome. They first conducted a survey of sketching ability of students as a part of which, the students were assigned to draw sketches to emphasize on aspects of sketching such as mechanical recall, drawing facility and novel visualization. The sketches from these tasks were evaluated using metrics such as demonstration of concept, accuracy on proportions and overall realism. Figure 3 illustrates sample sketches from mechanical recall task that involved sketching a bicycle from memory. Figure 3: Mechanical Recall Sketch [7,8] Further, to study the role of sketching instruction, the pool consisted of students with and without sketching instruction. In order to correlate the sketching ability with the design outcome, sketches were then collected from student logbooks for a four week class project. As in the previous study by Yang [6], quantity of individual sketches generated was used as a metric to evaluate the sketches alongside of the grades. Another study conducted by Yang [9] uses quantity as a metric to evaluate the sketches to draw correlations between quantities of ideas generated in conceptual design stage and design outcomes. Again, sample sketches from the actual study were not provided in the paper. It may be noted here that quantity as a metric is also used by user study conducted by Ramchandran [10]. Linsey et. al. [12] conducted experimental study to understand the effect of idea representation and viewing methods while using different idea generation techniques. The problem consisted of designing a peanut shelling device. The experiments were conducted with twelve teams of five students over a period of two weeks. Sketches generated by the teams were then evaluated using four metrics: quantity, variety, novelty and quality. It may be noted here that while Yang used sketch count as a measure of quantity [6,8], Linsey used a count of sketches that fulfilled desired functions as a measure of quantity [12]. For measuring the variety and novelty, the solutions were distributed into bins. Variety was then measured using percentage of bins that team s solution occupied while novelty was measured as (1-frequency of ideas). She also developed a three point scale for measuring quality of sketches. Illustration of the use of this scale can be seen in Figure 4. So a low quality solution is the one which is technically not feasible such as training a squirrel or genetically engineering a peanut. A low quality solution receives a score of 0, while a high quality solution is the one that is technically feasible and receives a score of 2. Figure 4: Sample sketch from Linsey study [12] In another study conducted by Linsey et.al. [19], to understand the influence of display and representation method on idea generation, quantity was used as a metric to evaluate the ideas represented in words or sketches. An elaborate set of rules was developed by the researchers to correctly measure the quantity of ideas developed by the participants. In addition to quantity, quality of sketches was evaluated by considering the improvements made in the sketches developed by the participants. Schmidt et.al. [16] conducted a user study at four different universities to investigate the improvement in idea generation with use of TRIZ, increased emphasis on sketching and use of 4 Copyright 2012 by ASME

5 smart pen. The metrics of quantity, quality, novelty and variety were used to analyze the ideas developed by the students. McKoy et. al. [18] also used quality and novelty in their user study to understand the influence of textual versus graphical representations on effectiveness of idea generation. The study was conducted with 89 participants from undergraduate class. The participants were alternately given two design problems. They were also given initial solutions in textual and graphical representations and were asked to generate more solutions for each design problem while providing interpretations for given initial solutions. While quality was measured based on satisfaction of identified functions, novelty was measured using function and subfunction breakdown. In addition another metric-accuracy of communication determined by comparing features between two expressions was also developed. The findings from the study indicate that graphical representations have a greater positive impact on idea generation when compared to textual representations. Again, the paper did not provide sample sketches from the actual experiment. In his user study for understanding the role of functions and interactions in the product design, Ramchandran used quantity and quality as metrics to evaluate the sketches [10]. While quantity entailed counting the number of sketches, he developed a high, medium and low scale for quality based on the pre-developed solutions to fulfill the given requirements. The study was conducted with participant pool that consisted of both graduate and undergraduate students. After giving introductory lecture on function model and function interaction model, the students were given a problem to design a burrito folder and asked to sketch solutions using one of the two models. The participants were also given specific instructions for generating sketches. The sketches were then collected and analyzed for quantity and quality. Figure 5 illustrates a sample sketch of burrito folder collected from the user study. The findings indicate that use of function model resulted to high quantity solutions, while high quality solutions were generated when using function interaction model. Figure 5: Sample sketch of burrito folder [10] Lugt conducted a study to understand how sketching affects the idea generation process in group meetings [11]. He conducted experiment using four teams with each team consisting of five product design students. Teams were asked to generate ideas using brain sketching and brain storming with post-its for the design problem making travelling by car fun for children. He used linkography as a metric to draw connections between ideas generated by different participants. Although, it is not clear how this metric was used for analyzing the sketches collected from the experiment as examples of the same are not discussed in the paper. The findings indicate that sketching positively supports re-interpretation of individual thinking process and enhances access to ideas previously generated. Another study conducted by Lau et. al [13] to explore the sketching behavior of designers, uses the sketches from design journals as data. The journals were collected from graduate students enrolled in a two semester long multidisciplinary design class. Metrics to analyze these sketches consisted of representation media (tangible, digital or mixed), representation as 2-dimensional or 3-dimensional and annotations. The findings from the study indicate that over the period of time from 2004 to 2006, the number of sketches in hybrid journals increased as compared to tangible journals and textual annotations were predominantly used as compared to dimensions. From this survey, it is evident that quantity appears to be most frequently used metric; however the context in which it is used may differ. While some researchers define quantity as count of the number of sketches [6,7,8,9,10], others define it as count of only those sketches which satisfy a specific function or requirement [12,19] Type of Research One of the goals for this study is to be able to identify a relation between characteristics of the metrics used to evaluate sketches and the type of research methods. Researchers in engineering design use different types of research methods such as surveys, user study experiments, case study, document analysis, interviews and protocol studies to answer their research questions. These methods are broadly classified as quantitative, qualitative and mixed methods [14,15,23,24]. Quantitative research aims at confirming hypothesis by using numerical data collected using methods such as surveys or experiments [14,15]. The research questions are typically closed-ended [14,15]. Qualitative research is characterized by the use of textual or graphical data to answer the research questions that are open-ended [14,15,24] such as understanding a phenomenon or process. Data collection methods for qualitative research entail case study, interviews, document analysis and protocol studies [21,23]. methods include a combination of quantitative and qualitative research methods [14,15]. Table 2 illustrates a summary of the types of research, approach for gathering data and source of collected data used by the researcher in the papers under review for this study. 5 Copyright 2012 by ASME

6 Table 2: Summary of research methods Reference Research Goal/Questions Type of research Research Approach [4] Studying designer activity to understand the sketching behavior of designers Qualitative Observation [6] What types of visual representations do students use? Does a particular type of visual correlate with a design phase? What insights can be gained from researching report visual contents to help improve design education? Is the quantity of design concept linked to design outcome? Is the timing of concept generation associated with design outcome? Is the type of sketch linked to the design outcome? What is the role of novice engineer's design experience on design outcome? Qualitative Qualitative Data Source Sketches from design notebook, observer notes Visual representation such as sketches, CAD, line drawings and photographs from senior design reports Student design log books, data from morph charts [13] Explores sketching behavior of designers and role of sketching in design process Qualitative [7] [8] How is sketching ability linked to design process and what is its association with sketch fluency and design outcome? Does someone who can draw well also draw more while engaged in engineering design process? Does better sketching also mean better design? Does teaching sketching skill correlate with better design? What is the nature of sketching skill in the context of engineering design? How is sketching ability linked to the design process, in particular, with sketch fluency and design outcome? What is the role of sketch instruction in design outcome? How does sketching skill relate to how a team accomplishes its group work? Survey and Survey and Sketches in design journals Data from survey and student design logbooks Data from survey and student design logbooks [9] Hypothesis 1-The quantity of concepts generated at the beginning of a design project correlates with design outcome. Hypothesis 2 The quantity of sketches generated during a project correlates with its design outcome. Hypothesis 3-Increased sketching at the beginning of the project, rather than at the end, correlates with better design outcome Survey and Data from survey and student design logbooks [10] How are product functions and interactions and associated representations useful in design? How do the function model and function interaction model affect the creativity of the designers within conceptual design? Sketches from user study [11] [12] Explore whether the functions of sketching as proposed in design thinking research can also be relevant for idea generation meetings RQ1 How does the techniques being tested influence the quantity, novelty and variety of ideas? RQ2 - Does the representation method of ideas interplay with the display method, or are they independent? RQ3- Are certain representations better for producing or improving the quality of solutions? Do certain representations cause bias toward certain type of ideas? Sketches from user study Sketches from user study [18] What is the influence of design representation on the effectiveness of idea generation? Which representational language for ideas is more suitable during conceptual design review? Sketches and textual descriptions of data [16] [19] RQ1 Can TRIZ improve the ideation performance of engineering students? RQ2 Can emphasizing sketching improve ideation performance of engineering students? RQ3 Can technology enabled journaling improve ideation performance of engineering students? RQ1 What type of display method will result in the generation of a greater quantity of ideas? RQ2 Does there exist interaction between the form of representation and how ideas are displayed? RQ3 What type of concept representation, words, sketches, or a combination will produce a higher quantity of results? RQ4 For methods such as 6-3-5, C-Sketch, and Gallery, how do the contributions of the individuals before the ideas are shared with the group compare with the number of ideas the group generates by building from these initial ideas? RQ5 As individuals add and combine ideas, what effect is there on the quality of the concepts? Design ideas and sketches Sketches and textual descriptions, post session survey 6 Copyright 2012 by ASME

7 As evident from the table, most researchers under review used qualitative and mixed research methods. Researchers using qualitative method to answer their research questions, primarily used document analysis for data collection [4,6,13]. The data collected was graphical in nature and entailed the sketches generated by the students in their design journals. The researchers who preferred mixed method to answer the research questions used a combination of experiments or survey with graphical and textual data collection in form of sketches or descriptions of solutions as opposed to pure numerical data [7,8,9,11,12]. It may be noted here that while the type of research method was not explicitly mentioned by the researchers, the classification provided here is based on interpretations made by using the definitions of quantitative, qualitative and mixed methods from literature. For instance, user study conducted by Linsey et at. [12,19], uses sketches and textual descriptions of ideas in addition to surveys as a source of data collection. Thus, the research approach is considered as mixed. 4. CHARACTERISTICS OF METRICS After conducting review of literature to identify the metrics currently used and classifying the research methods into qualitative, quantitative and mixed methods, the next step was to develop characteristics to evaluate the metrics and then develop a relation between these characteristics and types of research. A list of characteristics for evaluating the metrics used for analyzing the sketches was developed by the authors. It may be noted that these characteristics were empirically developed. Further study needs to be conducted to validate these characteristics. These include objective versus subjective, explicit versus implicit, qualitative versus quantitative, manual versus automated, time to evaluate and skill required to use the metrics. The list of characteristics with their definition can be seen in Table 3. A metric for analyzing the sketches is objective if the results of analysis do not vary when evaluated by different judges while it is subjective if the results vary by changing the judges. An example of such metric would be quantity measured by counting the number of sketches. The count of sketches and thus quantity should not vary when evaluated by different judges. However, quantity may become subjective if the count of the sketches is based on whether the sketch fulfills certain function or requirement. In this case, whether the sketch is counted or not depends on judge s interpretation of fulfillment of function or requirement. A metric is explicit or implicit depending on whether the judges do not or do need to make interpretations while evaluating the sketches respectively. An example of metric that is explicit would be quantity measured by counting the number of sketches. The judges do not need to make any interpretations while counting the number of sketches. However, the metric quality is implicit as evaluating the sketches for quality will depend on judge s interpretation of quality. The metric is quantitative if the results are expressed in numbers. Some examples would be expressing the evaluation results in number of lines used, number of functions fulfilled or number of features possessed by the sketch. However, if the metric is expressing the characteristics of sketch then it is qualitative. Examples of qualitative metrics may include drawing media or amount of shading used in the sketch. Table 3 Characteristics for metrics evaluation Characteristics Objective Subjective Explicit Implicit Quantitative Qualitative Manual Automated Time to evaluate Definition Results independent of evaluators results depends on people who evaluate No inference need to be made while evaluating inferences need to be made to analyze the sketches Results are expressed in numbers such as number of lines, features, functions etc. That which is not quantitative - such as characteristics of sketch, it can be binary(yes or no) Requires analysis to be done manually and cannot be automated can be automated Time required to analyze a single data instance Can analysis be performed by novice or does it requires expertize? A metric can be manual or automated depending on Skill required whether the analysis using the metric needs to be done manually or can be automated. For example, sketches are manually evaluated for quality and thus the metric quality is manual. It may be noted that none of the metrics found during the review were automated. However, metric complexity based on amount of shading can be automated based on the number of pixels in the sketch. Further, it may also be noted that having a metric which is automated can highly reduce the time and subjectivity involved while analyzing the sketches. The characteristics time to evaluate refers to the amount of time it would take to evaluate a single data instance using that metric. Currently the information about how much time it takes to evaluate the sketch using a particular metric is not available in the literature. However, this information if made available can be used to appropriately select the metric based on the resources, especially time, available to analyze the data. The characteristic- skill required refers to the expertise that should be possessed by the judges who analyze the data. For example, counting the number of sketches does not require a high level of expertise; however it requires expertise to use metrics such as linkography to analyze the sketches. While the 7 Copyright 2012 by ASME

8 information regarding the skill required is currently not available in literature, if made available it will help the researcher to select the judges with appropriate skills for analyzing the data. 5. FINDINGS After identifying the metrics currently used to analyze the sketches, they were evaluated against the characteristics described in section 7. The observations made from this evaluation are described in section 5.1. Then, correlations were drawn between the characteristics of metrics and type of research and section 5.2 discusses the details of the findings Observations from review of metrics From the discussions in section 3.2, it is evident that a wide variety of metrics are currently used by researchers in design research community to evaluate the sketches. After conducting a review of these metrics following observations can be made: There does not seem to be a clear consensus between researchers while using a specific metric. Thus though they are using the same metric, it is defined differently. An example of such metric is quantity. While some researcher use quantity as a measure of counting the number of sketches [6,7,8,9,10], others use it as counting sketches which satisfy a function or a requirement [12]. Another example is the metric of quality. While some researchers define quality as measure of whether or not the solution is technically feasible or difficult as evident from the sketch [12], others define the sketch as high or low quality based on whether or not it satisfies a function or requirement [10]. Different metrics are used by researchers who are addressing similar research questions. For example, while aiming to understand the sketching behavior of the designers, McGown et al. [4] use metrics such as complexity based on lines, shading and annotations, sketch size and amount of information, Lau et al. use metrics such as representation in 2D or 3D, amount of annotations and representation media. The information regarding the time and skill level required to use a particular metric is currently not available in the literature. These observations lead to questions such as what criteria should a researcher keep in mind while selecting a metric for analyzing sketches? Does using the same metric in slightly different manner affect the findings from the research? What influence does type of research approach used have on the selection of metric to evaluate the sketches? Of these questions, a preliminary effort is made by the authors to address the last question which is drawing relation between type of research and characteristics of metrics. The findings from this preliminary study are discussed in section Relating characteristics with type of research This research is the first attempt towards drawing relations between type of research and characteristics of the metrics used to analyze sketches. After identifying the metrics currently used in the literature and evaluating them against the developed characteristics, the next step was to relate the characteristics with the type of research. The details of these can be found in Table 4. It may be noted that current review of 14 research papers is not sufficient to draw significant correlations; however some observations were made which are discussed in this section. Of the papers under review, majority of the papers used mixed method approach to answer their research questions. While the research methods were limited to document analysis, surveys and experiments, the data source mainly consisted of design journals or notebooks of the participants and in some cases, data from survey and observer notes. No instances of methods such as case study or protocol study were found in the reviewed literature. In most cases, the metrics used for qualitative research were objective in nature. This means the results were independent of the judges analyzing the data. Examples include use of metrics such as size scale, quantity of sketches, use of drawing media for analyzing sketches to understand the sketching behavior of designers using design notebooks or journals as data source [4,6,13]. Further, the metrics which were objective were also explicit in all cases. This means that no inferences were required to be made while analyzing the sketches using those metrics. Examples include quantity measured by counting sketches, use of drawing media such as pen, pencil, marker or sketch size. For mixed research approach, which consisted a combination of qualitative and quantitative methods, in most cases the metrics used were subjective in nature. In some cases this was also identified by the researchers and they demonstrate use of inter-rater reliability to prove that this subjectivity did not influence the findings from their research. For example, Linsey et. al. [19] conducted a user study to understand the influence of display and representation methods on idea generation and analyzed the sketches using quantity as a metric. In this case the metric quantity was subjective as it was measured using a set of rules developed by the authors. However, they used inter-rater reliability to demonstrate that the subjectivity did not influence the findings of the research. Some researchers who adopt a mixed method approach use combination of objective and subjective metrics to analyze the data. For example, Ramchandran [10], conducted a user study to understand the influence of function and function interaction model on the creativity of designers. 8 Copyright 2012 by ASME

9 He used combination of two metrics quantity and quality to analyze the sketches. Quantity is objective while quality is subjective and results may change by changing the judges. However, by testing for inter-rater reliability he demonstrated that the findings of his study were not influenced by subjective nature of the metric-quality. Further, the metrics that were subjective were also implicit in nature. This means that it required interpretations to be made by the judges while analyzing the data. For example, Linsey [12] developed a three point scale for quality based on the technical feasibility and difficulty. While analyzing, whether or not the sketch demonstrates a concept which is technically more or less feasible or difficult depends on the interpretations made by the judges. Thus the metric quality is both subjective and implicit. Most metrics which were subjective were also qualitative in nature. This means that they indicated the quality of the sketch. An example of one such metric is complexity measured by lines and shading. Amount of shading illustrates the sketch quality in terms of level of detail and analyzing the sketch by using this metric depends on the judge s interpretation of amount of shading used All the metrics currently used in the literature are manual. No automation is currently used for analyzing sketches. However some metrics such as those based on line, shading, drawing media can be automated. This will significantly reduce the time of analyzing the data. From these observations, a general trend seems to be that of using objective metrics for qualitative research and subjective metrics tested for inter-rater reliability for mixed method approach. However, an extensive survey of literature needs to be done to draw statistically significant correlations. 6. SUMMARY This paper presents a review of the metrics currently used by researchers for analyzing sketches. After identifying the metrics currently used, the paper presents characteristics to evaluate the metrics and reports it s relation with the type of research approach. While, the current review of papers is not sufficient to identify significant correlations, this study reports some interesting observations made during the review. The findings show that researchers investigating similar type of research questions use completely different metrics to evaluate the sketches. There seems to be a lack of consensus between researchers using the same metric. Thus, the current literature lacks formal definitions for metrics used to analyze sketches. There is a general trend of using metrics that are objective, explicit and quantitative for qualitative research approach and metrics that are subjective, implicit and qualitative in nature for mixed research approaches. The subjectivity of the metric is also identified by the researchers and they demonstrate the use of inter-rater reliability to prove that the findings are not influenced. However, as previously mentioned a more extensive review needs to be conducted to draw significant correlations. The current literature lacks information such as time required or desired skill level to analyze the data using particular metric. This information if made available in the literature can be useful for novice designers while selecting a particular metric from the available pool. 7. FUTURE WORK As evident from the findings for this research, current review is not sufficient to draw meaningful correlations between the characteristics of the metrics and type of research. Thus, the future work entails conducting a more extensive review of literature from design research community, specifically focusing on research using sketches as data source. This extensive survey will then be used for drawing meaningful correlations between the types of research and characteristics of metrics used. Then, based on these correlations, systematic guidelines for selecting or developing new metrics for sketch analysis will be provided. As previously mentioned, the characteristics to evaluate the metrics were empirically developed by the authors. A study needs to be conducted to validate the sufficiency of these characteristics. In addition, a pilot study will be conducted to analyze the sketches using different metrics to answer the same research question will be conducted to understand the sensitivity of the findings on the type of metrics used. 8. REFERENCES [1] Angus Stevenson, Oxford Dictionary of English.: Oxford University Press, [2] Song Shuang and Alice M. Agogino, "Insights on Designers Sketching activities in new product design teams," in Proceedings for ASME IDETC, Salt Lake City, 2004, pp [3] Jami J. Shah, Noe Vargas-Hernandez, and Steve M Smith, "Metrics for Measuring Ideation Effectiveness," Design Studies, vol. 24, pp , [4] Alistair McGown, Graham Green, and Paul A Rodgers, "Visible ideas: information patterns of conceptual sketch activity," Design Studies, vol. 19, no. 4, pp , Sophoria Westmoreland, Ashley Ruocco, and Linda Schmidt, " of Capstone Design Reports: Visual Representations," Journal of Mechanical Design, vol. 133, no. 5, pp. 1-7, May [6] Maria Yang, "Concept Generation and Sketching: Correlations with design outcome," in Proceedings of ASME Design Engineering Technical Conference, Chicago, IL, USA, 2003, pp [7] Jorge G Cham and Maria C Yang, "Does Sketching Skill relate to good design?," in Proceedings of ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Long Beach, California, USA, 2005, pp [8] Maria C Yang and Jorge G Cham, "An of 9 Copyright 2012 by ASME

10 Sketching Skill and its Role in Early Stage Engineering Design," Journal of Mechanical Design, vol. 129, pp , May [9] Maria C Yang, "Observations on concept generation and sketching in engineering design," Research in Engineering Design, vol. 20, no. 1, pp. 1-11, December [10] Raveesh Ramchandran, "Understanding the Role of Functions and Interaction in the Product Design," Clemson, [11] van der R Lugt, "How sketching can affect the Idea Generation Process in Design Group Meetings," Design Studies, vol. 26, no. 2, pp , [12] Julie S Linsey et al., "An Experimental Study of Group Idea Generation Techniques: Understanding the roles if idea Representation and Viewing Methods," Journal of Mechanical Design, vol. 133, pp. 1-15, March [13] Kimberly Lau, Lora Oehlberg, and Alice Agogino, "Sketching in Design Journals: An of Visual Representations in the Product Design Process," in ASEE Engineering Design Graphics Division Mid-Year Conference, Berkeley, CA, 2009, pp [14] John Cresswell, Research Design - Qualitative, Quantitative and Methods Approaches. Thousand Oaks, CA: Sage Publications, Inc., [15] Burke Johnson and Larry B. Christensen, Educational research: quantitative, qualitative, and mixed approaches. Thousand Oaks, CA: Sage Publications, [16] Linda C Schmidt, Noe Vargas Hernandez, Gul Kremer, and Julie Linsey, "Pilot of systematic ideation study with lessons learned," in ASME International Design Engineering Technical Conferences and Computers and Informtion in Engineering Conference, Montreal, 2010, pp [17] Gabriela Goldschmidt and Maya Weil, "Content and Structure in Design Reasoning," Design Issues, vol. 14, no. 3, pp , [18] Felicia L McKoy, Noe Vargas-Hernandez, Joshua D Summers, and Jami J Shah, "Influence of design representation on effectiveness of idea generation," in Proceedings of ASME Design Engineering Technical Conference and Computers and Information in Engineering conference, Pittsburgh, [19] Julie Linsey, M G Green, J T Murphy, K L Wood, and A B Markman, ""Collaborating to Success": An experimental Study of Group Idea Generation Techniques," in ASME - International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Long Beach, 2005, pp [20] Robert Yin, Case Study Research- Design and Methods, 3rd ed. Thousand Oaks, USA: Sage Punlications, Inc, [21] Sudhakar Teegavarapu, "Foundations of Design Method Development," Clemson, [22] Glenn A. Bowen, " as Qualitative Research Method," Qualitative Research Journal, vol. 9, no. 2, pp , [23] Sudhakar Teegavarapu and Joshua D Summers, "Case study Method for Design Research," in Proceedings of ASME International Design Engineering Technical Conferences & Computers and Information in Engineering Conference, New York, [24] Maura Borrego, Elliot P Douglas, and Catherine T Amelink, "Quantitative, Qualitative and Research Methods in Engineering Education," Journal of Engineering Education, vol. 98, no. 1, pp , Copyright 2012 by ASME

11 Reference Type of research method Table 4 Relating type of research with characteristics of metrics Metric Objective / subjective Explicit / Implicit Quantitative / Qualitative Manual / Automated Complexity based on line shading and annotation [4] Qualitative Size Scale Objective Explicit Quantitative Manual Drawing media Objective Explicit Qualitative Manual Information content Subject matter - system, subsystem, artifact Objective Explicit Qualitative Manual Part of multiple object - in same grouping on page but different from one Qualitative another in type or subject Motion indicator Objective Explicit Qualitative Manual Applied forces Objective Explicit Qualitative Manual Part of set - multiple visuals related to each other Views - isometric, orthogonal, multiple Objective Explicit Qualitative Manual [6] Qualitative Quantity - sketch count, dimensioned sketch counted separately Objective Explicit Quantitative Manual Representation - 2D or 3D Objective Explicit Qualitative Manual [13] Qualitative Annotations Objective Explicit Qualitative Manual Media - Tangible, digital or mixed Objective Explicit Qualitative Manual Demonstration of grasp of concept Accuracy of proportions [7] Correctness of proportions, 3D perspective Quantity - sketch count Objective Explicit Quantitative Manual Demonstration of grasp of concept Accuracy of proportions [8] Correctness of proportions, 3D perspective Quantity - sketch count Objective Explicit Quantitative Manual [9] Quantity - sketch count Objective Explicit Quantitative Manual Quantity-sketch count Objective Explicit Quantitative Manual [10] Quality - high medium low based on solutions for requirements Subjective but IRR tested Implicit Qualitative Manual [11] Linkography Subjective but IRR tested Implicit Quantitative Manual Variety Subjective but IRR tested Implicit Qualitative Manual [12] Novelty Subjective but IRR tested Implicit Qualitative Manual Quantity (based on function count) Subjective but IRR tested Implicit Quantitative Manual Quality Subjective but IRR tested Implicit Qualitative Manual [16] [18] [19] Novelty Subjective Implicit Quantitative Manual Variety Subjective Implicit Quantitative Manual Quantity Objective Explicit Quantitative Manual Quality Subjective Implicit Quantitative Manual Quality-based on satisfaction of identified functions Novelty-using function and sub-function breakdown Accuracy of communication Quantity Subjective but IRR tested Implicit Quantitative Manual Quality 11 Copyright 2012 by ASME