Expertise in Complex Decision Making: The Role of Search in Chess 70 Years After de Groot

Cognitive Science 35 (2011) 1567 1579 Copyright Ó 2011 Cognitive Science Society, Inc. All rights reserved. ISSN: 0364-0213 print / 1551-6709 online DOI: 10.1111/j.1551-6709.2011.01196.x Expertise in Complex Decision Making: The Role of Search in Chess 70 Years After de Groot Michael H. Connors, a Bruce D. Burns, a Guillermo Campitelli b a Department of Psychology, University of Sydney b School of Psychology and Social Science, Edith Cowan University Received 14 September 2010; received in revised form 7 March 2011; accepted 14 March 2011 Abstract One of the most influential studies in all expertise research is de Groot s (1946) study of chess players, which suggested that pattern recognition, rather than search, was the key determinant of expertise. Many changes have occurred in the chess world since de Groot s study, leading some authors to argue that the cognitive mechanisms underlying expertise have also changed. We decided to replicate de Groot s study to empirically test these claims and to examine whether the trends in the data have changed over time. Six Grandmasters, five International Masters, six Experts, and five Class A players completed the think-aloud procedure for two chess positions. Findings indicate that Grandmasters and International Masters search more quickly than Experts and Class A players, and that both groups today search substantially faster than players in previous studies. The findings, however, support de Groot s overall conclusions and are consistent with predictions made by pattern recognition models. Keywords: Chess; Decision making; Expertise; Pattern recognition; Problem solving; Psychology; Reasoning; Search; Skill acquisition and learning; Thinking; Verbal protocol 1. Introduction Across many different domains, experts make difficult and complex decisions under conditions of uncertainty and time pressure. These experts can be viewed as having either superior analytical skills in generating and evaluating alternatives (search), or a greater ability to recognize situational characteristics and promising options based on stored knowledge (pattern recognition; Klein & Peio, 1989). Although both these elements are undoubtedly Correspondence should be sent to Michael H. Connors, Macquarie Centre for Cognitive Science, Macquarie University, Sydney, NSW 2109, Australia. E-mail: michael.connors@mq.edu.au

1568 M. H. Connors, B. D. Burns, G. Campitelli Cognitive Science 35 (2011) essential, current theories emphasize the role of pattern recognition in expert decision making (Feltovich, Prietula, & Ericsson, 2006). One of the most important sources of evidence for this view is de Groot s (1946, 1978) study of expert chess players. This study is one of the most influential and frequently cited in expertise research and indeed the entire psychological literature (see Bilalić, McLeod, & Gobet, 2008; Charness, 1992; Gobet & Charness, 2006). Vicente and Brewer (1993) reported that de Groot s work is in the 99.9th percentile in terms of frequency of citation. de Groot presented chess players of varying skill, including some of the best players of the time, with an unfamiliar position and asked them to think aloud as they analyzed the position and chose what move they would make. de Groot found that, contrary to popular belief, the strongest players did not think further ahead than weaker players. World-class Grandmasters (GMs) searched at a similar depth and considered a similar number of options as mere chess experts. The GMs, nevertheless, made better decisions than the experts, choosing better moves and examining moves more relevant to the position. In a second task, the chess players were shown a chess position for a brief period of time (2 15 s) and tested on their recall. GMs proved to have a much greater recall of chess positions than the chess experts. Subsequent research by Chase and Simon (1973a, 1973b) revealed that more skillful players displayed this much superior recall only when the positions were meaningful and typical of ordinary play, but not when pieces were randomly arranged on the board. These findings suggested that what accounted for differences in skill, at least above a certain level of proficiency, was recognizing patterns based on previous experience, rather than a real time search through various options or a difference in general capacities. This work inspired a number of studies which found evidence that pattern recognition underlies many other domains of expertise (Feltovich et al., 2006; Richman, Gobet, Staszewski, & Simon, 1996). Various theories of expertise have been proposed to account for these findings. Arguably the most influential of these are chunking and template theories, which hold that expertise is made possible by the ability to recognize patterns in the task environment on the basis of past experience (Gobet & Simon, 1996b, 1996c, 2000). In chess, experts recognize perceptual chunks typical and distinctive configurations of pieces that they have acquired through practice and study, and stored in long-term memory (Chase & Simon, 1973a, 1973b). When an expert recognizes a chunk, it prompts them to think of a move or strategy based on previous experience and so allows them to make a superior decision. A contrary view, however, was taken by Holding (1985, 1992), who denied the importance of pattern recognition and instead emphasized the role of search. Holding proposed that chess masters play better than novices because they search more deeply and quickly than novices and also better evaluate the products of their search using their general knowledge. In many ways, however, the contrast between pattern recognition and search models is a false opposition as both elements are clearly important. Nevertheless, the relative contribution of search and pattern recognition in expertise has remained a contentious issue in the literature. Many changes have occurred in chess since de Groot conducted his study in the 1930s and 1940s, such as increased access to databases, shorter time limits in tournaments, and the introduction of chess computers and online play (see Gobet, Campitelli, & Waters, 2002). These changes have led van Harreveld, Wagenmakers, and van der Maas (2007) to argue

M. H. Connors, B. D. Burns, G. Campitelli Cognitive Science 35 (2011) 1569 that the cognitive underpinnings of expertise have also changed. van Harreveld et al. suggest that in the 1930s and 1940s, chess players tended to rely on pattern recognition and general rules of thumb, such as the importance of controlling the center of the board. van Harreveld et al. claim that these principles are now outdated and that search is much more important: modern chess on a high level no longer relies on rule-oriented and principleoriented thinking (fast processes) but focuses on concrete analysis of the position at hand (slow processes) (p. 596). In effect, then, van Harreveld et al. suggest that de Groot s seminal results were an artifact of the cultural and historical context in which the study was set. Given the importance of de Groot s findings, van Harreveld et al. s suggestions deserve empirical examination. Unfortunately, however, there is comparatively little recent empirical data on chess players thinking to test van Harreveld et al. s claims. In spite of the large impact that de Groot s study has had on cognitive psychology, there has only been one direct replication of de Groot s study, conducted by Gobet (1998). This study was limited in that it did not include any GMs, which makes it difficult to compare players at the highest levels of skill, and that it only used one position (other research has found that players search may vary somewhat across positions; e.g., Bilalić, McLeod, & Gobet, 2009; Campitelli & Gobet, 2004; Wagner & Scurrah, 1971). In addition, Gobet collected his data in 1986 and many of the changes in chess have occurred since then (see Gobet et al., 2002). Even though each domain of expertise may have idiosyncratic properties, it is important to understand the role of search in chess because chess research has been a cornerstone of the argument that practice and pattern recognition are the keys to expertise (e.g., Klein, 1989, 1999). The present study thus sought to replicate de Groot s study using his think-aloud method. This procedure has since acquired strong empirical support for its validity (Ericsson, 2006; Ericsson & Simon, 1993). Masters (GMs and International Masters) and intermediate players (class players and experts) were asked to analyze two chess positions and the search characteristics of the two groups were compared. An additional analysis compared the search characteristics of players in this study with those in de Groot s and Gobet s studies. The memory experiments that de Groot conducted were not repeated because they have recently been replicated many times (e.g., Gobet & Clarkson, 2004; Gobet & Simon, 1996b, 1998a). 2. Method 2.1. Participants Twenty-two male chess players, including six GMs (Elo 2450-2600), five International Masters (Elo 2350-2500), six Experts (Elo 2000-2200), and five Class A players (Elo 1800-2000), volunteered to participate in this study without remuneration. The GMs and International Masters were assigned to the group masters. The Experts and Class A players were assigned to the group intermediates. Players were recruited from chess tournaments around Australia or through personal contacts. The sample included top chess professionals from six different countries. For Experts and Class A players, the national Australian Chess Federation (ACF) rating was used to measure chess skill because not all had an international

1570 M. H. Connors, B. D. Burns, G. Campitelli Cognitive Science 35 (2011) FIDE rating (for the 16 players with both FIDE and ACF rating, the correlation between the two was.98). The mean age of the players was 29.9 (SD = 11.2) years, with a range from 18 to 52, and did not differ between groups, t(20) = 1.851, p =.079. All players were fluent in English. Sample summary statistics are shown in Table 1. 2.2. Materials Two chess positions (de Groot s positions A and C) were presented to players on a standard tournament chess set (see Fig. 1). These positions were used because de Groot collected most of his data with them. A digital recorder was used to record the players verbal protocols. 2.3. Procedure The procedure followed de Groot s (1978) original methodology, while adopting certain recommendations offered by Ericsson and Simon (1993) for verbal protocol studies. The Table 1 Ratings and age of chess players (standard deviations are in parentheses) Skill Group N Rating Range Mean Rating (SD) Mean Age (SD) Masters 11 2,350 2,600 2454.7 (54.3) 34.1 (11.1) Intermediates 11 1,800 2,200 2001.6 (102.6) 25.7 (10.1) Position A Position C Fig. 1. The two chess positions used in this experiment. de Groot s Position A (white to move) is on the left; Position C (black to move) is on the right.

experimenter gave players a warm-up task to familiarize them with the protocol procedure. Players were given a simple mathematical multiplication problem and asked to verbalize their thought processes as they completed it with pen and paper (Ericsson & Simon, 1993). All players completed this easily. The experimenter then told players that he was going to show them two chess positions one after the other. They were to decide on a move while verbalizing their thinking aloud, wherever possible using algebraic chess notation. The experimenter emphasized to players that they should not try to explain or justify their thinking, but simply say aloud whatever they thought about while choosing a move and to treat it as if they were analyzing the position in a tournament game. The experimenter told players to talk continuously and that if they were silent for more than 3 s, they would be prompted with the words keep on talking (Ericsson & Simon, 1993). The experimenter also told players that when they were ready, they were to physically make the move on the board. There was no time limit for players to arrive at a decision. Although this differed from tournament conditions, all players used an amount of time that would typically be used for analyzing a critical position in tournament play. The experimenter then showed players the two positions one after the other. The verbal protocols from both positions were recorded on a digital recorder and were later transcribed for analysis. 2.4. Measures M. H. Connors, B. D. Burns, G. Campitelli Cognitive Science 35 (2011) 1571 2.4.1. Outline As in previous studies, a problem behavior graph was extracted from each transcribed protocol (Newell & Simon, 1972). From this, the following search measures were determined. The reader is referred to the supplementary online appendix for an example of how the search variables were extracted (see also Charness, 1981; Gobet, 1998; de Groot, 1978; Newell & Simon, 1972; Wagner & Scurrah, 1971). 2.4.2. Quality of chosen move Each move the players chose was assigned a numerical value from 0 to 5. For the current study, this numerical value was based on the analysis of the chess computer program fritz 9. When comparing across studies, it was not possible to use this criterion because Gobet gave slightly different values to some moves and only the raw data (i.e., the numerical values) from his study were available, so Gobet s values were used instead for this purpose. 2.4.3. Total time to choose a move The time from when players first viewed the position to when they made their final move was measured in minutes. This reflects both amount and speed of search and is used to calculate other measures. 2.4.4. Number of nodes searched Nodes are the discrete board positions resulting from possible moves that are considered during search. The number of nodes searched equals the total number of moves considered in the search space, including repetitions of the same position and no-moves (when a

1572 M. H. Connors, B. D. Burns, G. Campitelli Cognitive Science 35 (2011) player s turn to move is skipped in analysis and moves for just one player are considered). It is a measure of the total amount of search performed. 2.4.5. Rate of generating nodes per minute This variable is obtained by dividing the number of nodes by the total time taken in search. It is a measure of search speed. 2.4.6. Number of base moves This variable is the number of moves the players considered that were immediately playable in the position in front of them. It is a measure of search breadth. 2.4.7. Number of episodes An episode is defined as a sequence of moves involving a base move (a move immediately playable in the position) and a series of moves that follows. It is another measure of search breadth. 2.4.8. Maximal depth This variable measures the single longest sequence of moves from the initial position considered by a player. It is expressed in plies. Plies are half-moves, one turn taken by one of the players. 2.4.9. Mean depth This variable is calculated by taking the longest sequence of moves in each episode and dividing by the total number of episodes. Like maximal depth, it is expressed in plies. 3. Results 3.1. Current study analysis Results from the current study are shown in Table 2. The results were analyzed in a 2 (Skill: masters vs. intermediates) 2 (Position: A vs. C) mixed anova. For the between-subjects effect of skill, there were two significant differences between masters and intermediates: Masters chose better moves overall than intermediates, F(1, 20) = 5.664, p =.027, g p 2 =.221, and also searched faster than intermediates, generating significantly more nodes per minute, F(1, 20) = 10.184, p =.005, g p 2 =.337. In addition, there was a trend for masters to search at a greater maximal depth than intermediate players, F(1, 20) = 3.982, p =.060, g p 2 =.166. There was no difference between masters and intermediates in the total time taken, F(1, 20) = 2.103, p =.162, g p 2 =.095; number of nodes searched, F(1, 20) =.166, p =.688, g p 2 =.008; number of base moves, F(1, 20) = 2.623, p =.121, g p 2 =.116; number of episodes, F(1, 20) =.203, p =.658, g p 2 =.010; or mean depth of search, F(1, 20) = 3.053, p =.096, g p 2 =.132. Importantly, however, statistical power was an issue for these five variables; analysis showed that power ranged from.071 to.338 for

the variables where no statistical difference was found. The Skill Position interactions were not significant for any of the eight dependent variables (all Fs < 1.690, all ps >.208, all g p 2 <.078). 3.2. Historical trend analysis M. H. Connors, B. D. Burns, G. Campitelli Cognitive Science 35 (2011) 1573 Table 2 Means and standard deviations (in parentheses) of the data from the current study An additional analysis compared the absolute values of search characteristics across studies in a 2 (Skill: masters vs. intermediates) 3 (Study: de Groot, Gobet, current study) between-subjects anova. We refer to this as the historical trend analysis (see Table 3). This analysis used anova to compare across studies, based on Gobet (1998), who used this statistical procedure to compare his results to de Groot s. This approach is justified because all three studies used the same procedure and stimuli. For this analysis, the participants in de Groot s and Gobet s studies were categorized as either masters (de Groot s GMs and IMs, Table 3 Means and standard deviations (in parentheses) of de Groot s (1946) and Gobet s (1998) data, and the data from the current study Position A de Groot (1946) Gobet (1998) Current Study Masters (n =7) Masters (n =11) Position A Intermediates (n =7) Intermediates (n =11) Masters (n =12) Intermediates (n =36) Masters (n =11) Position C Masters (n =11) Intermediates (n =11) Quality of move 4.2 (1.3) 3.3 (1.9) 3.7 (1.5) 3.1 (1.7) Total time (minutes) 11.4 (6.7) 17.5 (12.0) 15.1 (8.5) 20.1 (12.7) Number of nodes visited 145.3 (114.6) 138.6 (78.7) 216.6 (147.7) 190.6 (117.2) Nodes per minute 11.6 (4.2) 8.4 (2.6) 14.2 (2.7) 9.6 (2.8) Number of base moves 5.6 (2.8) 8.6 (3.7) 6.6 (3.5) 7.4 (4.0) Number of episodes 23.1 (14.9) 30.1 (13.2) 35.5 (23.6) 34.0 (18.7) Maximal depth (plies) 11.7 (4.8) 9.4 (2.3) 14.1 (4.0) 12.0 (2.7) Mean depth (plies) 4.1 (1.3) 3.3 (1.0) 4.9 (1.5) 4.2 (1.0) Intermediates (n =11) Quality of move 4.6 (0.8) 2.7 (1.0) 4.6 (0.8) 2.7 (1.4) 4.2 (1.3) 3.3 (1.9) Total time (minutes) 11.6 (4.4) 15.4 (7.7) 11.3 (7.4) 16.7 (8.1) 11.4 (6.7) 17.5 (12.0) Number of nodes visited 51.6 (39.0) 34.3 (16.0) 48.3 (44.1) 40.1 (25.6) 145.3 (114.6) 138.6 (78.7) Nodes per minute 4.1 (1.7) 2.4 (1.0) 4.0 (1.5) 2.9 (2.8) 11.6 (4.2) 8.4 (2.6) Number of base moves 5.4 (3.1) 4.3 (2.4) 3.3 (2.7) 5.4 (2.7) 5.6 (2.8) 8.6 (3.7) Number of episodes 9.6 (6.2) 7.9 (3.9) 9.3 (7.4) 10.5 (5.7) 23.1 (14.9) 30.1 (13.2) Maximal depth (plies) 8.6 (3.6) 6.3 (2.7) 9.1 (3.8) 8.0 (4.7) 11.7 (4.8) 9.4 (2.3) Mean depth (plies) 5.4 (1.3) 4.5 (0.9) 5.0 (2.4) 3.7 (1.8) 4.1 (1.3) 3.3 (1.0) Note. The results from the current study for Position A are repeated from Table 2.

1574 M. H. Connors, B. D. Burns, G. Campitelli Cognitive Science 35 (2011) and Gobet s masters) or intermediates (de Groot s and Gobet s Experts and class players). These categories allowed easy comparison across studies and maximized the statistical power of the analyses. The historical trend analysis used Position A because de Groot only reported raw data for this position and Gobet only used this position in his study. In terms of skill levels, there were four significant differences between masters and intermediate players on search variables across the three studies. First, masters chose better quality moves than intermediate players, F(1, 78) = 21.053, p <.001, g p 2 =.213. Second, masters searched faster, generating more nodes per minute, than intermediate players, F(1, 78) = 8.591, p =.004, g p 2 =.099. Third, masters took less time to decide on a move than intermediate players, F(1, 78) = 6.103, p =.016, g p 2 =.073. Finally, masters showed a deeper mean depth of search than intermediates, F(1, 78) = 5.365, p =.023, g p 2 =.064. This depth effect, however, was due to the presence of Class B players in Gobet s data, a group less skillful than the groups in the current study. Re-analysis of the data without this group removed this effect but did not alter the effects found for quality of move, speed, or time. There was no statistically significant difference between masters and intermediate players in terms of the number of nodes they searched, F(1, 78) =.560, p =.456, g p 2 =.007; number of base moves, F(1, 78) = 3.500, p =.065, g p 2 =.043; number of episodes, F(1, 78) =.965, p =.329, g p 2 =.012; or in their maximal depth of search, F(1, 78) = 3.284, p =.074, g p 2 =.040. Power remained an issue for these four variables as it ranged from.115 to.455, but the effect sizes were small. Comparing across the studies, there were four significant differences, including total number of nodes searched F(2, 78) = 23.122, p <.001, g p 2 =.372; number of nodes generated per minute, F(2, 78) = 45.850, p <.001, g p 2 =.540; number of episodes, F(2, 78) = 28.810, p <.001, g p 2 =.425; and number of base moves, F(2,78) = 6.540, p =.002, g p 2 =.144. Given the size of these effects, it is important to emphasize that these variables were measured the same way across the three studies. For the first three variables, post hoc analysis (Scheffé) showed significant contrasts between the current study and both de Groot s (ps <.001) and Gobet s (ps <.001) studies. Players in the current study searched more nodes, searched more quickly, and searched more episodes than players in the previous two studies. For the number of base moves, however, post hoc analysis showed no significant difference between the current study and de Groot s (p =.082), but a significant difference between the current study and Gobet s (p =.013). There were no other differences between studies (all Fs < 2.730, all ps >.071, all g p 2 <.065). The Skill Study interactions were not significant for any of the eight dependent variables (all Fs < 2.456, all ps >.092, all g p 2 <.059). 4. Discussion Given the importance of de Groot s findings to theories of expert decision making, this study sought to replicate de Groot s study. Our analyses showed that skilled players differ from less skilled players in their search in more ways than previously thought. Masters searched faster and possibly also more deeply than intermediates. However, these mean

M. H. Connors, B. D. Burns, G. Campitelli Cognitive Science 35 (2011) 1575 differences existed in de Groot s sample as well, though they were not reported. This indicates that there has not been an underlying change in the relative roles of search and pattern recognition in chess expertise, although it also suggests that statistical power was an issue in the previous studies. Overall, the findings support the robustness of both de Groot s conclusions and pattern recognition models. The findings thus suggest that pattern recognition remains critical to chess skill despite other evidence that chess has changed. In particular, the historical trend analysis showed large differences in the absolute values of speed of search and amount of search across studies. These differences provide some support for van Harreveld et al. s (2007) claim that chess has changed. 4.1. Relative differences between masters and intermediates Both the current study analysis and the historical trend analysis revealed that masters generated more nodes per minute than intermediate players. This indicates that while masters may search more quickly than intermediates, this difference has remained constant over time. The previous studies conducted by de Groot (1978) and Gobet (1998) showed a trend for more skillful players to search more quickly than less skillful players, but the differences did not reach statistical significance. This suggests that power may have been an issue in these studies. Importantly, both search and pattern recognition models predict that stronger players generate moves faster during search than weaker players, so the current findings are consistent with both models (see Gobet, 1997; Gobet & Simon, 1998b). With respect to depth of search, however, there were no clear differences between masters and intermediates. In terms of maximal depth of search, both the current study analysis and the historical trend analysis showed that there was a trend for masters to search more deeply than intermediate players, though this did not reach statistical significance. For mean depth of search, no difference in skill level was found in the current sample, nor in the historical trend analysis once Class B players in Gobet s sample, a group weaker than the groups in the current sample, were removed. The fact that the differences in search depth between masters and intermediate players players several standard deviations apart on the Elo rating scale were not statistically significant and the effect sizes were small suggests that other factors, such as pattern recognition, continue to play a more important role than search depth in chess expertise. Indeed, pattern recognition models predict that more skillful players have a greater number of chunks and templates than less skillful players and this will in turn promote a deeper, more efficient search. The weak indication of a relationship between skill and depth of search rather than the strong, almost linear relationship proposed by search models seems consistent with the pattern recognition account. Finally, results showed no differences between masters and intermediates in terms of the number of nodes searched, number of episodes, and number of base moves. This indicates that amount or breadth of search is not sufficient for expert performance and that, again, other factors, such as evaluation made possible by pattern recognition, play an important role. Importantly, however, the analyses that found marginal or no effects all suffered from chronically low power, even with the large sample size of the historical trend analysis, so it is possible that other differences in search between masters and intermediates exist but were

1576 M. H. Connors, B. D. Burns, G. Campitelli Cognitive Science 35 (2011) not detected. Even if this were the case, the effects would appear less strong than other kinds of differences that have been reported in the literature, such as memory for patterns (Chase & Simon, 1973a, 1973b). Overall, then, our analyses show evidence that masters search more quickly and possibly also more deeply than intermediates, even if the two groups do not differ on other measures of search. This shows that the two skill groups differ in more ways than originally reported, although the overall trend of the data remains the same. These findings are thus somewhat analogous to the results from the memory recall experiments for random positions, which have likewise shown that masters perform slightly better on this task than weaker players (Gobet & Simon, 1996a), contrary to the original report that there was no difference in skill when recalling random positions (Chase & Simon, 1973a, 1973b). Nevertheless, the differences in search we found seem relatively minor compared to the large differences between masters and intermediates shown in the memory recall experiments for meaningful patterns (Chase & Simon, 1973a, 1973b). The results thus support de Groot s overall conclusions and are entirely consistent with predictions made by pattern recognition models (e.g., Gobet, 1997; Gobet & Simon, 1998b). The findings add to a large body of research from other domains of expertise supporting this view (e.g., Hodges, Starkes, & MacMahon, 2006; Norman, Eva, Brooks, & Hamstra, 2006). 4.2. Changes in the absolute value of speed and number of nodes searched Results show that the absolute values for speed of search and number of nodes searched were much higher than previous studies, irrespective of skill level. These two measures are related: As a result of greater speed, players generate more nodes in their overall search. The present study found that players are capable of searching more than 10 nodes per minute. This is faster than de Groot and others thought, but it is consistent with values reported by Campitelli and Gobet (2004). The findings are also consistent with Gobet s (1997) SEARCH model, a formal model of template theory, which predicts very similar values for speed of search to what were found in this study. This provides further support for pattern recognition accounts. Nevertheless, the differences across studies require explanation. It is unlikely that methodological differences could account for these differences. As de Groot hand-recorded his protocols, it is possible that he may have underestimated some values. Gobet, however, tape-recorded his protocols and reported very similar values to de Groot, so the differences between studies cannot simply be attributed to technological differences. Variability in coding also cannot explain these differences because the same criteria for coding were used and these criteria are very strictly defined. Indeed, both search speed and number of nodes searched are measured in terms of the total number of moves a player mentions, as well as the amount of time taken in the case of speed, and neither of these base measures is subject to the interpretation of a coder. Finally, the previous studies did not use a warm-up task for the verbal protocol, so it is possible that part of the difference might be attributable to this. However, it seems unlikely that this could explain a roughly three-fold increase in the number of nodes visited, especially given that players spent on average more than 10 min considering the position in all studies, whereas the warm-up

exercise took only a couple of minutes. So presumably even if they spent the first couple of minutes warming up in previous studies, they were equivalent to the current study after that. Instead the data suggest that some aspects of thinking in chess have changed in the 20 years since Gobet conducted his study. Several major changes have occurred in chess during this period. These include, for example, the introduction of shorter time limits and the abolition of adjournments in tournaments (Gobet et al., 2002), which both suit players capable of thinking quickly. For example, in 1986 (when Gobet, 1998, collected his data) the World Championship match required players to make their first 40 moves in 150 min; however, in the 2004 FIDE World Championship players had to make their first 40 moves in 90 min. Other changes over this period include the popularization of chess computers and ChessBase, a computer database of opponent s games, which together assist players to prepare for games and encourage quicker decision making. As a result of these changes, it is possible that tournament chess today might select for players capable of thinking quickly. If this is the case, then it shows the robustness of the importance of pattern recognition in expert decision making, which seems to have remained constant over time. 4.3. Conclusion M. H. Connors, B. D. Burns, G. Campitelli Cognitive Science 35 (2011) 1577 On balance, therefore, the results of this study support de Groot s original conclusions. Although masters may differ from intermediates in their search in more ways than previously thought, the results fit within the broader trend of research emphasizing the importance of pattern recognition in expert performance. The factors underpinning expert decision making, however, continue to be an important area of research, and it is likely that studies involving chess will continue to inform the field. Acknowledgments We would like to thank Michael Lip, Charles Zworestine, and Brian Jones for their help in recruiting participants, and all the chess players who very kindly volunteered their time to participate in this study. We would also like to thank Amanda Barnier, Max Coltheart, Art Markman, John Sutton, Lena Quinto, and three anonymous reviewers for helpful comments on an earlier draft of this paper. References Bilalić, M., McLeod, P., & Gobet, F. (2008). Expert and novice problem solving strategies in chess: Sixty years of citing de Groot (1946). Thinking and Reasoning, 14, 395 408. Bilalić, M., McLeod, P., & Gobet, F. (2009). Specialization effect and its influence on memory and problem solving in expert chess players. Cognitive Science, 33, 1117 1143. Campitelli, G., & Gobet, F. (2004). Adaptive expert decision making: Skilled chess players search more and deeper. Journal of the International Computer Games Association, 27, 209 216.

1578 M. H. Connors, B. D. Burns, G. Campitelli Cognitive Science 35 (2011) Charness, N. (1981). Search in chess: Age and skill differences. Journal of Experimental Psychology: Human Perception and Performance, 2, 467 476. Charness, N. (1992). The impact of chess research on cognitive science. Psychological Research, 54, 4 9. Chase, W. G., & Simon, H. A. (1973a). Perception in chess. Cognitive Psychology, 4, 55 81. Chase, W. G., & Simon, H. A. (1973b). The mind s eye in chess. In W. G. Chase (Ed.), Visual information processing (pp. 215 281). New York, NY: Academic Press. Ericsson, K. A. (2006). Protocol analysis and expert thought: Concurrent verbalizations of thinking during experts performance on representative tasks. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 223 241). Cambridge, UK: Cambridge University Press. Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data, Rev edn. Cambridge, MA: The MIT Press. Feltovich, P. J., Prietula, M. J., & Ericsson, K. A. (2006). Studies of expertise from psychological perspectives. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 41 67). Cambridge, UK: Cambridge University Press. Gobet, F. (1997). A pattern-recognition theory of search in expert problem solving. Thinking and Reasoning, 3, 291 313. Gobet, F. (1998). Chess players thinking revisited. Swiss Journal of Psychology, 57, 18 32. Gobet, F., Campitelli, G., & Waters, A. J. (2002). Rise of human intelligence: Comments on Howard (1999). Intelligence, 30, 303 311. Gobet, F., & Charness, N. (2006). Expertise in chess. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 523 538). Cambridge, UK: Cambridge University Press. Gobet, F., & Clarkson, G. (2004). Chunks in expert memory: Evidence for the magical number four or is it two? Memory, 12, 732 747. Gobet, F., & Simon, H. A. (1996a). Recall of rapidly presented random chess positions is a function of skill. Psychonomic Bulletin & Review, 3, 159 163. Gobet, F., & Simon, H. A. (1996b). Templates in chess memory: A mechanism for recalling several boards. Cognitive Psychology, 31, 1 40. Gobet, F., & Simon, H. A. (1996c). The roles of recognition processes and look-ahead search in time-constrained expert problem solving: Evidence from grandmaster level chess. Psychological Science, 7, 52 55. Gobet, F., & Simon, H. A. (1998a). Expert chess memory: Revisiting the chunking hypothesis. Memory, 6, 225 255. Gobet, F., & Simon, H. A. (1998b). Pattern recognition makes search possible: Comments on Holding (1992). Psychological Research, 61, 204 208. Gobet, F., & Simon, H. A. (2000). Five seconds or sixty? Presentation time in expert memory. Cognitive Science, 24, 651 682. de Groot, A. D. (1946). Het denken van den schaker [The thinking of the chess player]. Amsterdam, Netherlands: N.V. Noord-Hollandsche Uitgevers Maatschappij. de Groot, A. D. (1978). Thought and choice in chess, 2nd ed. The Hague, Netherlands: Mouton Publishers. van Harreveld, F., Wagenmakers, E.-J., & van der Maas, H. L. J. (2007). The effects of time pressure on chess skill: An investigation into fast and slow processes underlying expert performance. Psychological Research, 71, 591 597. Hodges, N. J., Starkes, J. L., & MacMahon, C. (2006). Expert performance in sport: A cognitive perspective. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 471 488). Cambridge, UK: Cambridge University Press. Holding, D. H. (1985). The psychology of chess skill. Hillsdale, NJ: Erlbaum. Holding, D. H. (1992). Theories of chess skill. Psychological Research, 54, 10 16. Klein, G. A. (1989). Recognition-primed decisions. In W. Rouse (Ed.), Advances in man-machine systems research (Vol. 5, pp. 47 92). Greenwich, CT: JAI Press.

M. H. Connors, B. D. Burns, G. Campitelli Cognitive Science 35 (2011) 1579 Klein, G. A. (1999). Sources of power. Cambridge, MA: MIT Press. Klein, G. A., & Peio, K. J. (1989). Use of a prediction paradigm to evaluate proficient decision making. American Journal of Psychology, 102, 321 331. Newell, A., & Simon, H. A. (1972). Human problem-solving. Englewood Cliffs, NJ: Prentice-Hall. Norman, G., Eva, K., Brooks, L., & Hamstra, S. (2006). Expertise in medicine and surgery. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 339 353). Cambridge, UK: Cambridge University Press. Richman, H. B., Gobet, F., Staszewski, J. J., & Simon, H. A. (1996). Perceptual and memory processes in the acquisition of expert performance: The EPAM model. In K. A. Ericsson (Ed.), The road to excellence: The acquisition of expert performance in the arts and sciences, sports, and games (pp. 167 187). Mahwah, NJ: Erlbaum Associates. Vicente, K. J., & Brewer, W. F. (1993). Reconstructive remembering of the scientific literature. Cognition, 46, 101 128. Wagner, D. A., & Scurrah, M. J. (1971). Some characteristics of human problem-solving in chess. Cognitive Psychology, 2, 454 478. Supporting Information Additional Supporting Information may be found in the online version of this article on Wiley Online Library: Appendix S1. Extraction of Search Variables From a Verbal Protocol. Please note: Wiley-Blackwell is not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.