Playing the Name Game to identify academic patents in Germany

Transcription

1 Scientometrics (2014) 101: DOI /s x Playing the Name Game to identify academic patents in Germany Anja Schoen Dominik Heinisch Guido Buenstorf Received: 30 September 2013 / Published online: 28 August 2014 Ó Akadémiai Kiadó, Budapest, Hungary 2014 Abstract Identifying academic inventors is crucial for reliable assessments of academic patenting and for understanding patent-based university-to-industry technology transfer. It requires solving the who is who problem at the individual inventor level. This article describes data collection and matching techniques applied to identify academic inventors in Germany. To manage the large dataset, we adjust a matching technique applied in prior research by comparing the inventor and professor names in the first step after cleaning. We also suggest a new approach for determining the similarity score. To evaluate our methodology we apply it to the EP-INV-PatStat database and compare its results to alternative approaches. For our German data, results are less sensitive to the choice of name comparison algorithm than to the specific filtering criteria employed. Restricting the search to EPO applications or identifying inventors by professor title underestimates academic patenting in Germany. Keywords Name matching algorithm Academic patents Patent database German universities Kürschners Gelehrtenkalender JEL classification C81 I23 O30 O34 MSC classification A. Schoen (&) TUM School of Management, Technische Universität München, Munich, Germany schoen@wi.tum.de D. Heinisch G. Buenstorf Institute of Economics, University of Kassel, Kassel, Germany heinisch@uni-kassel.de G. Buenstorf buenstorf@uni-kassel.de D. Heinisch G. Buenstorf International Centre for Higher Education Research (INCHER-Kassel), University of Kassel, Kassel, Germany

2 528 Scientometrics (2014) 101: Introduction Measuring academic patenting activities has increasingly gained in importance since university-to-industry technology transfer became a third mission for universities (Etzkowitz and Leydesdorff 2000). However, for several reasons, it is not easily done. Not all patents by academic inventors are university-owned, and individual-level data must be used to identify those that are not. This requires reliable ways of identifying academic inventors in the patent record as well as suitable matching techniques. Further problems arise from the heterogeneity of patents and the presence of alternative patenting strategies. Since patents can be filed at different patent offices, only considering applications at a single office risks underestimation of patenting activities as well as biased results (de Rassenfosse et al. 2014). This article contributes to the research on identifying academic patents by adjusting prior matching techniques to the German case and comparing results from various matching techniques to identify invention-based technology transfer in Germany. Our basic approach is to match a complete list of active professors at German universities with the PatStat patent database provided by the European Patent Office (EPO). In comparing professor and inventor names, we use three different algorithms: the 2-gram, the tokenbased, and the simple-string matching algorithms. To assess the performance of our matching approach, we also searched for the professor title in the patent data, which is the conventional approach used in earlier research on academic patenting in Germany. We compare results from the different methods and analyze the impact on the observed invention-based technology transfer activity. Our findings suggest that in the case of German academic inventions, a search for EPO applications would yield incomplete and potentially biased results. In particular, findings on the ownership structure and inter-university comparisons are affected by (not) taking into account applications at other offices, most notably the German Patent and Trademark Office (GPTO). We also find that searching for the professor title in PatStat substantially underestimates patenting activity among German academics. By comparison, the choice of algorithm in the initial name comparison has a less pronounced impact on the filtered professor-inventor pairs. However, manual control of professor-patent pairs was still necessary because the PatStat database does not contain sufficient information for perfect matching. The remainder of the article is structured as follows: The next section discusses the interest in academic patenting and the difficulty faced in obtaining reliable data on academic patents. The Data inputs section describes the data sources for our empirical analysis, before the alternative matching techniques, and their results are presented in the Matching methodology section. Differences in the observed level of patenting activity are discussed in the How do the different Name Game methods affect results? section. Finally, the Discussion and conclusion section provides implications for scholars and policymakers as well as concluding remarks. Academic patenting: the need for better data University managers and policy makers around the world are increasingly emphasizing knowledge and technology transfer as a third mission of universities and other public research organizations (PROs) (e.g., OECD 2003). Even though it is conventional wisdom that technology transfer takes a variety of pathways (Cohen et al. 2002), the recent policy (and likewise scholarly) attention has focused on academic patenting as a means to enhance the societal use of new knowledge from public research.

3 Scientometrics (2014) 101: Today, most universities and PROs have adopted measures to facilitate and support patenting activities by their faculty and staff, and in most countries the resulting intellectual property (IP) is owned by these organizations. Large numbers of university patent applications are increasingly taken as evidence of successful transfer activities. However, even when other channels of technology transfer are disregarded, there are at least two problems that need to be addressed before patent counts can be used as meaningful performance indicators. 1 First, while university-owned patents are relatively easy to identify by searching for university names in patent databases, not all patents on inventions made in public research are owned by the respective universities and PROs. This is true even in the U.S. where university ownership of most patents was established under the Bayh-Dole Act (Thursby et al. 2009). In Europe, only counting university- and PRO-owned patents would lead to an even more drastic underestimation of academic patenting (Lissoni et al. 2008; von Proff et al. 2012). In many European countries (including Germany before 2002), university inventors, and not their employers, traditionally owned the IP in their inventions. This was known as the professors privilege in the law of employee inventions. As a consequence of this institutional setup, inventors themselves frequently show up as the applicants of European academic patents (e.g., Lissoni et al. 2008). Even more widespread, however, is the practice that patents on technologies (co-) invented by university researchers are filed by private-sector firms. Based on inventor-level patent applications filed at the EPO, Lissoni et al. (2008) found that companies accounted for 64.4, 73.9, and 83.1 % of all academic patents filed in France, Italy and Sweden, respectively. Even after the end of the professors privilege, some European universities are quite willing to leave the ownership in patents to private-sector collaboration partners. In this way, they do not have to carry the administrative and financial burden of the patenting process, and the risk of antagonizing potentially important collaboration partners is minimized. Counts of university-owned patents (or applications) are therefore a poor indicator of transfer performance (cf. also the analysis in Schoen and Buenstorf (2013) for Germany). To obtain a more complete picture of academic patenting, university inventors have to be identified in the patent data. As will be argued below in more detail, this is a non-trivial task. Second, there is substantial scope for differences in universities patenting strategies. In Europe, patents can be filed at national patent offices or (directly) at the EPO. More specifically, (first) applying at the GPTO is a widespread pattern in Germany. Many applications are never extended to the European level. Accordingly, just identifying EPO patents (or EPO applications) may lead to further biases in estimating academic patenting activities or comparing the patent portfolios of individual universities. The relevance of this concern has been assessed by de Rassenfosse et al. (2014) using data for Belgian universities as well as for German university-owned patents. They show that the choice of the filing route is nonrandom, and a global count of patents is indispensable to obviate selection bias. This problem of strategic patenting at different patent offices is further aggravated by the fact that patents are highly heterogeneous, and incomplete identification of academic patents is likely to lead to biased results. As is well known, there are dramatic differences in the quality and value of individual patents, and the distribution in values is highly skewed (e.g. Harhoff et al. 2003). This also holds for academic inventions. For instance, studying the licensing activities of the German Max Planck Society from 1980 to 2004, 1 Improving data and methods to deal with these challenges was the objective of the project Academic Patenting in Europe Inventor Database, which was supported by the European Science Foundation and provided the context for the present study. Results from this project have been published in Lissoni (2013).

4 530 Scientometrics (2014) 101: Buenstorf and Geissler (2012) found that about 63 % of all patented inventions did not find licensees, and more than half of all royalty income from successfully licensed technologies was obtained from a single invention. Geuna and Nesta (2006) likewise highlight the skewed nature of licensing income (which proxies for the value of academic patents) in a broader empirical context. Faced with these problems, sophisticated strategies to identify academic patents were developed in prior research. Various techniques have been adopted to match scholar names in official university staff listings and in patent databases. In this way, academics from several European countries (including Italy and France) have been identified among the inventors listed in EPO patent applications (e.g., Lissoni et al. 2006; Raffo and Lhuillery 2009; Mejer 2011). However, because of the potential selection bias stemming from the relevance of GPTO applications, these techniques require substantial modification to be useful in the German context. Adding to the difficulties, official staff listings of German university researchers are not available. Prior research on academic patenting in Germany therefore dodged the who-iswho problem by searching for the professor title in patent databases (Czarnitzki et al. 2007; Schmoch 2007; von Proff et al. 2012). However, this identification technique leads to new problems, as not all university professors use their title in patent applications, whereas several groups of individuals in Germany may (legitimately) use a professor title although they are not full-time university researchers (e.g., Honorarprofessoren, privatesector employees or self-employed individuals who used to work at universities, and physicians that work at teaching hospitals). Only recently have researchers begun to analyze academic patenting activities in Germany based on matching data at the individual level (Schmoch et al. 2012). In this article, we present a new approach to the identification of academic patents originating from German universities. It is based on matching individuals listed as university professors with inventors from all priority patents in the PatStat database and helps mitigate the problems listed above. Data inputs In this section, we describe the datasets of German university professors and of German inventors that provide the empirical foundation of our analysis. The German PROFLIST The starting point of our identification of academic inventors in Germany is a list of professors names from Kürschners Gelehrtenkalender, a commercial directory of university researchers that has been regularly published since (The print version is currently in its 23rd edition.) Researchers are listed after completing their Habilitation or when they assume a professorship at a German research university. Accordingly, Kürschners Gelehrtenkalender is restricted to the top level of the German academic hierarchy. Junior researchers (doctoral students and postdocs, etc.) and technical staff are excluded, as are professors working in Fachhochschulen and individuals employed in positions oriented toward teaching (such as Akademische Räte and lecturers). Kürschners Gelehrtenalender is well-suited for our purposes because the data are collected in a two-step procedure. Publicly available basic information is first assembled for the full population of relevant researchers, resulting in virtually perfect coverage. This

5 Scientometrics (2014) 101: basic information includes researchers full names, titles, gender, positions, (university) addresses, and research fields. In the second step, the publisher contacts the identified researchers and asks them to submit further information. This information, which is only available for a subset of researchers, includes private addresses, dates of birth, doctoral degree years, and previous university positions. Our analysis is based on a version of the dataset that includes entries for researchers working at German universities in April 2010 and comprises 45,307 entries. To minimize homonym problems for identifying academic inventors, the list of professor names used in the matching procedure was limited to disciplines with a high propensity for patenting [biology, chemistry, engineering, medicine, (unspecified) natural sciences, pharmaceutics, physics, and physical as well as technical chemistry] (cf., Lissoni et al. 2006). After such restrictions, 16,046 names remained on the list (hereafter, referred to as PROFLIST). The DE-INV database To identify German university-invented patents, raw patent data were collected from PatStat [the European Patent Office (EPO) Worldwide Patent Statistic Database, version April 2010]. To keep the search effort manageable, we initially only included inventors with a German place of residence 2 or without specified country information. We then expanded the database to inventors with the same standardized name id 3 to ensure that no relevant inventors were excluded. Moreover, we restricted our database to inventors who patented at least once in 2006 or Given these limitations, the number of relevant inventors in PatStat was reduced from over 28.5 million to 369,284. In the following, we refer to this database as DE-INV. Matching methodology The challenge of searching for academic inventors in patent databases was addressed in Lissoni et al. (2006) for Italy, France, and Sweden; Raffo and Lhuillery (2009) for Switzerland 4 ; Lissoni et al. (2009) for Denmark; Mejer (2011) for Belgium; and Schmoch (2007), Czarnitzki et al. (2007), von Proff et al. (2012), and Schmoch et al. (2012) for Germany. Most research projects (except those that identified patents invented by German professors) are based on European patent applications, which are reclassified by inventors and academics. 5 Prior research used the following sequence of operations in the matching procedure: (1) cleaning, (2) inventor inventor matching, (3) inventor inventor filtering, and (4) professor-inventor filtering, (5) manual control. In steps 2 and 3 only information on the inventors is used, in step 4 information on the academics is added. Our methodology differs in that, after cleaning, we directly compare the PROFLIST names to the names in 2 The country of residence is identified by using the country code information provided by PatStat. 3 The standardized name id is provided by PatStat (doc_std_name_id) and allocates a standardized name to applicants and inventors with exactly the same or very similar names. However, the limited quality of the id does not allow us to use it as unique inventor id. 4 Raffo and Lhuillery (2009) examined patents by professors at Ecole Polytechnique Fédérale de Lausanne (EPFL). 5 Using European patent data is advantageous because more complete address information is available in PatStat. Addresses are provided for 94 % of the inventors listed on EPO patents but only for 0.2 % of German patent inventors.

6 532 Scientometrics (2014) 101: the DE-INV database. In the third step, we calculate similarity scores for the individual inventors that matched with professor names in the second step. As with other projects, the fourth step involves professor-inventor pair filtering. The final step consists of a manual control of the professor-patent pairs. Both approaches are depicted in Table 1. Changing the order has several advantages for large countries like Germany. Most importantly, prior name matching limits the number of inventor pairs subsequently considered. We only had to filter inventors with similar or identical names to the professors we searched for. This drastically reduced the computation time for step 3, allowing us to include patent applications worldwide. As noted above, this is crucial because many German patent applications are filed only at the national patent office and there are concerns about selection bias when only patents from a single office are taken into consideration. The following subsections explain each step in more detail. 6 Professor-inventor name matching The objective of the professor-inventor name matching step was to generate a list with identical or similar inventor names for each professor. To reduce the number of missing correct matches (false negatives or error type I), the procedure was designed to include as many inventor-professor pairs as possible (Lissoni et al. 2010). Different matching algorithms can be applied in this phase, and each has advantages and disadvantages. The simple-string matching algorithm has been applied most frequently in prior research (Raffo and Lhuillery 2009). It has a high precision rate and is relatively easy to use. However, compared with other algorithms it has an inferior recall rate. Token-based and n-gram algorithms have much better recall rates with the same level of precision. After testing different algorithms based on their precision and recall rates, Raffo and Lhuillery (2009, p. 1619) concluded: [w]hen precision is targeted by scholars, the weighted Token algorithm is thus the dominant choice. Instead, when researchers aim at general identification of a patent portfolio rather than a sampling view and agree to give up some precision in order to re-integrate false negatives, the weighted 2-gram algorithm is a good choice. Based on this result, we used the 2-gram algorithm, the token-based algorithm, and the simple-string matching algorithm for the professor-inventor name comparison. 7 The 2-gram algorithm decomposes the text string into elements with two characters through a moving window (e.g., Raffo and Lhuillery 2009). The similarity score applied is the dice coefficient. The coefficient takes values between zero and one; with a score of one indicating that two text strings are identical. We calculated the similarity between professor and inventor names (name and surname) and extracted results with similarity scores above or equal to The token-based algorithm splits a text string into elements. The Jaccard similarity coefficient is used as the similarity indicator (cf. Cohen et al. 2003). We excluded results with coefficients below Simple-string matching identifies inventors 6 For a more detailed description of the cleaning stage please refer to Appendix 1. 7 To calculate the Jaccard similarity coefficient, we used the Microsoft Fuzzy Lookup add-in for Microsoft Excel (downloadable at: last accessed on February 8, 2014). 8 The matching procedure allows us to consider misspelled names and consequently reduce false negative matches (error type I). The algorithm, as discussed before, has the following disadvantage: the similarity score comprises PROFLIST names included in the names stored in DE-INV and vice versa. This results in false positive matches (error type II). To reduce the false positives, we delete matches wherein the length of the professor first name and surname differs from the inventor first and second names by more than two characters.

7 Scientometrics (2014) 101: Table 1 Overview of matching approaches Matching approach by prior research Step 1: cleaning Step 2: inventor inventor matching Pairwise comparison of inventor 1-5 Step 3: inventor inventor filtering Inventor 2 and 3 as well as 4 and 5 are identical Step 4: professor-inventor filtering Step 5: manual control Professor A is matched to unique inventor b (inventor 2 and 3) Inventor 1 Inventor a Inventor 1 Unique inventor a Unique inventor a / ALBERTO EINSTEIN ALBERTO EINSTEIN Inventor 2 Inventor b Inventor 2 Unique inventor b Unique inventor b Professor A ALBERT ALEBRT EINSTEIN ALBERT EINSTEIN EINSTEIN Inventor 3 Inventor b Inventor 3 Unique inventor b A EINSTEIN A EINSTEIN Inventor 4 Inventor c Inventor 4 Unique inventor c Unique inventor c / Donald Duck Donald Duck Inventor 5 Inventor d Inventor 5 Unique inventor c D Duck D Duck Matching approach applied Step 1: cleaning Step 2: professor-inventor name matching Professor A matched to inventor 1-3 Professor A ALEBRT EINSTEIN Step 3: inventor inventor filtering Step 4: professor-inventor filtering Step 5: manual control Inventor 2 and 3 are identical Professor A is matched to unique inventor b (inventor 2 and 3) Inventor 1 Inventor 1 Unique inventor a Unique inventor a / ALBERTO EINSTEIN ALBERTO EINSTEIN Inventor 2 Inventor 2 Unique inventor b Unique inventor b Professor A ALBERT ALEBRT EINSTEIN ALEBRT EINSTEIN EINSTEIN Inventor 3 Inventor 3 Unique inventor b A EINSTEIN A EINSTEIN

8 534 Scientometrics (2014) 101: with exactly the same name as the professor. 9 Table 2 summarizes the results from the three algorithms. The 2-gram algorithm identified 42,887 inventors with names similar to 5,599 professors. The token-based algorithm matched 33,007 inventors with 5,119 professors, and the simple-string algorithm matched 18,093 inventors with 3,805 professors. A closer look at the different results reveals that the 18,093 inventors from the simplestring algorithm were also identified by the other two approaches. In addition, 3,614 inventors were identified by the 2-gram algorithm and the token-based algorithm. We found 21,180 inventors only when using the 2-gram algorithm and 11,300 only through the token-based algorithm. Inventor-inventor filtering In the next step, we performed pairwise comparisons of all inventor names in the same name group to identify identical inventors. 10 We based the comparison on a series of variables, such as name, title, geographical location (street, city, and postal code), technical content (IPC class), application dates, and relational data (co-inventor, applicants, and patent family). To determine whether two inventors are the same person, prior research used cumulative similarity scores and a defined threshold value (Lissoni et al. 2006; Trajtenberg et al. 2006). However, this implies that criteria must be weighted, and the threshold value must be determined a priori. Because we do not have a benchmark dataset, weights and threshold values would be arbitrary. To circumvent this problem, we did not weight the criteria but assigned each variable either the value zero (not fulfilled) or one (fulfilled). Where different addresses (street, city, and postal code) were provided for both inventors, we assigned the value minus one. Thus, divergent information is treated differently from missing information. Negative values (minus one) were also assigned to inventors with common German surnames and inventor pairs whose first application dates differ by more than 20 years. Taken alone, all extracted filter criteria performed relatively poorly. However, using specific combinations of classifiers substantially increased the predictive power. To find combinations of the criteria that can identify (true) positive inventor inventor pairs with high levels of precision, we proceeded as follows. First, we used a positive test sample of inventor pairs that we trusted to be true positive matches. These inventor pairs were characterized by perfectly matching name and address information (N = 3,910). We also generated a negative test sample with 3,000 inventor-inventor pairs by randomly matching inventors. Subsequently, we searched for combinations of criteria that identify as many positive matches as possible in the positive test sample and as few as possible false positive matches in the negative test sample. Address information could not be harnessed in this step since by definition it is identical for all inventor pairs in the positive test sample. To include address information, we used the combinations of criteria identified previously to create an extended test sample. Using this extended sample, we again searched for combinations of criteria that identified as many positive matches as possible and as few false positive matches as possible. In addition, we determined a threshold value by using the 9 We used a slightly modified procedure for the simple-string name comparison. Instead of comparing the full name-strings we first separated them into their components. Afterwards, we re-sampled the portions with first name and surname combinations. These name combinations were used for the simple-string name comparison. 10 The term name group denotes the inventors matched with a professor in the first step.

9 Scientometrics (2014) 101: Table 2 Results from the professor-inventor name comparison Algorithm # Inventors # Professors 2-Gram 42,887 5,599 Token 33,007 5,119 Simple-string 18,093 3,805 information about already identified combinations of criteria. 11 Finally, we employed the combinations thus found to perform the filtering step for the full sample. All combinations identified in this process can be found in Appendix 2. For example, whenever an inventor pair has an (eight digit) IPC class, co-inventor, and applicant in common, we assume that these inventors are the same person (cf. Table 3, combination 1). If the inventor pair just shares IPC class (six digits), co-inventor, and applicant, they must also have the same name to be considered the as identical (cf. Table 3, combination 2). Without this additional criterion the combination would include too many false positive matches in the negative sample. Therefore, the precision is increased by including the exact name. If one of the criteria combinations illustrated in Appendix 2 is satisfied by an inventor pair, we assumed that the two inventors are the same person and assigned a unique inventor id to them. Within name groups we imposed transitivity: if inventor A is assumed to be identical to inventor B, and inventor B is assumed to be identical to inventor C, inventor A and inventor C are considered the same person (Trajtenberg et al. 2006). Table 4 presents the results from the inventor inventor filtering step. Based on the assumption that all inventors with the same unique inventor id are the same person, we aggregated the information for the respective inventors. This is advantageous because more information is available for a unique inventor than for each inventor alone. However, such combining is disadvantageous because false positive matches may distort the results in the next step, professor-inventor filtering. Therefore, we were as conservative as possible in the inventor inventor filtering step. To test our approach, we compared inventor inventor filtering from our algorithm to the clean database provided by the ESF-APE-INV project 12 (Lissoni et al. 2010). In the first step, we matched professor names to inventor names using the token-based algorithm. We found 32,004 inventors with similar names to 7,567 professors. In the second step, we applied inventor inventor filtering for each name group. This reduced the number of inventors to 26,949 unique inventors. In comparison, the MassacratorÓ algorithm employed in the ESF-APE-INV project reduced the names to 25,286 unique inventors. Three different outcomes are possible: (1) the two algorithms assume the same inventors to be the same person, (2) one inventor group identified by an algorithm is split into more groups by another algorithm, or (3) different inventors are assumed to be the same person. The latter case indicates that an algorithm yields wrong results, while the second case indicates that one algorithm is more conservative than another. Table 5 summarizes the comparison results based on the number of unique inventors. 11 The identified combinations of criteria generated values of three. Therefore, we conservatively assumed that true positive matches would produce at least values greater than four. 12 Project on academic patenting in Europe (APE-INV) funded by the European Science Foundation (ESF).

10 536 Scientometrics (2014) 101: Table 3 Example for criteria combinations Criteria Combination 1 Combination 2 Name Name 1 Common surname Geographical information Street City Postal code (5 digits) Postal code (2 digits) Technical contents IPC 8 (8-digit level) 1 IPC 6 (6-digit level) 1 IPC 4 (4-digit level) Application dates First application dates differ more than 20 years Relation data Co-inventor 1 1 Applicant 1 1 Patent family Notes: 1 denotes a required condition Table 4 Results from inventor inventor filtering Algorithm # Inventors # Unique inventors 2-gram 42,887 23,568 Token 33,007 17,219 Simple-string 18,093 9,991 Table 5 Comparison of our (token-based) algorithm vs. the MassacratorÓ Case Description # Of unique inventors 1 Same inventors grouped together 20,527 2 Unique inventor group is split in more groups by the method presented 4,810 Unique inventor group is split into more groups by MassacratorÓ Different inventors grouped together 773 (2.9 %) The results show that only 2.9 % of the inventors identified may be false matches. The relatively large number of cases (4,810; 17.9 %) where our algorithm splits the unique inventor groups identified by the MassacratorÓ algorithm into several groups demonstrates that our algorithm is more conservative. Professor-inventor filtering This stage aimed to identify and exclude false positives among the matches and, thus, increase the precision rate (Raffo and Lhuillery 2009). To identify and retain true matches for professor-inventor pairs in the database, we used filtering methods that consider biographical and geographical information as well as information on the applicants and

11 Scientometrics (2014) 101: technical content of patents (cf. Table 6). For the latter, we adapted the concordance table for IPC classes and Scopus research areas developed by Schmoch et al. (2012) to the research areas indicated in Kürschners Gelehrtenkalender. To find criteria combinations that identify (true) positive professor-inventor pairs, we used the same approach as in the inventor inventor filtering step. The true positive test sample was generated by considering the professor-inventor pairs with exact matches of names and addresses. The negative test sample was generated by matching the names of 361 professors from the field of German studies (who are unlikely to patent) to the DE-INV database. As described above, we tried to find combinations of criteria that identify as many true positive matches as possible and as few false positive matches as possible. We learned from the two test sets that the following patterns (cf. Table 6) are sufficient to identify true professor-inventor pairs. Based on this approach, each professor was compared to the unique inventors in the name groups. If one inventor-professor pair was assumed to be the same person, the professor was assumed to be the same person as all of the inventors with the same unique inventor id. The impact of the algorithms applied in the first step on the filtering results is illustrated in Table 7. In total, 8,064 professor-inventor pairs were assumed to be correct matches by at least one of the three algorithms. While 4,654 (57.7 %) of the professor-inventor pairs were identified by all three algorithms, 3,410 (42.3 %) of the professor-inventor pairs were missed using the simple-string algorithm in the professor-inventor name comparison step (cf. Fig. 1). Manual control Because we do not have benchmark data, assessing the data quality was not straightforward. As discussed in Raffo and Lhuillery (2009), a manual examination is necessary to eliminate false positive matches. We followed this recommendation and proceeded as follows. First, to limit the manual search effort, we only considered inventors listed on at least one priority patent (application) in the years 2006 and Continuations, translations of patent applications, PCT applications in the national phase, and artificial applications were also excluded from the dataset. These restrictions reduced the number of relevant patent documents to be checked from 17,465 to 4,922; the corresponding number of professor-inventor pairs was reduced to 2,269 (1,288 professors). Second, we eliminated the names of researchers below the professor rank (primarily Privatdozenten), retired professors incorrectly included in the dataset, professors at public research organizations, as well as individuals who are not full-time employees at German research universities (primarily Honorarprofessoren and physicians employed at university-affiliated teaching hospitals). 13 Moreover, we deleted professors who changed universities between 2006 and This step reduces the dataset to 1,639 professor-inventor pairs (911 professors). Third, we manually examined the names of the matched professors and inventors. The dataset was again reduced to 1,169 professor-inventor pairs (852 professors). Fourth, we controlled each professor-patent pair for the fit between patent title and the professor s research field and the overlap of co-inventor(s) and co-author(s). In addition, a web search 13 We also disregarded professors from German Federal Armed Forces universities and professors affiliated with the Charité medical center. 14 We could have taken these steps at the beginning of matching; however, we included these professors in another sample. The results are available upon request.

12 538 Scientometrics (2014) 101: Table 6 Criteria combinations for professor-inventor filtering Title information Applicant information Professor title Doctor title Diploma title University as applicant Name information Geographical information Technical content Whole name Surename First name Second name Name affix Private address University address Private postcode University postcode Private city University city IPC-research area compatible V V V V V V V V V V V V V V V 1 V V V V 1 V V V V 1 V V V V Notes: 1 denotes a required condition V denotes an or condition (at least one criteria must be fulfilled)

13 Scientometrics (2014) 101: Table 7 Results for professor-inventor filtering Algorithm # Identified inventors # Unique inventors # Identified professors 2-gram 7,248 2,070 1,621 Token 6,646 1,978 1,583 Simple-string 4,654 1,806 1,444 for applicant and inventor names was conducted to control for direct relationships between applicants and inventors. Where we found such direct relationships (e.g. the applicant is a firm and the inventor was identified as employee of this firm) the respective professorinventor pair was eliminated from the sample. The results of the manual cleaning step are summarized in Table 8. Figure 2 illustrates that 648 (83.1 %) professor-inventor pairs were correct matches using each algorithm. The increase in recall rate from using the 2-gram algorithm (tokenbased algorithm) compared with the simple-string matching amounts to 16.2 % (15 %). How do the different Name Game methods affect results? As detailed in the Academic patenting: the need for better data section, prior research on academic patenting in Europe has mostly been based on the matching of inventor names with names from official university researcher and staff listings. In this paper we reported our own efforts to use a similar approach in Germany. As official name listings could not be obtained for this context, we turned to Kürschners Gelehrtenkalender, a commercial directory of university professors that turned out to be remarkably complete in its coverage. To address the problem of incomplete and potentially biased identification of academic patenting caused by strategic choices of patent authorities, which is particularly troubling in Germany, our approach differed from the earlier work in that we did not constrain the search to EPO applications, but considered all worldwide priority patent applications. The implications of the extended population of applications are substantial. With 837 of 1,066 patents (78.5 %; this and all following numbers refer to results obtained using the 2-gram algorithm), the majority of the patent applications originating from German universities were initially filed at the GPTO. Only 199 (18.7 %) were filed directly at the EPO, and 460 (43.2 %) were subsequently extended to the EPO. Accordingly, restricting the search to only EPO applications would have missed more than 38 % of all academic patent applications. University-invented EPO applications and extensions differ substantially from nonextended GPTO patents in terms of their ownership structure (cf. Table 9). While almost 35 % of all EPO applications are owned by private-sector firms, corporate ownership accounts for only one-sixth of the non-extended GPTO applications. Again, only considering EPO documents would lead to erroneous conclusions about the share of universityowned academic patents. 15 Another concern is that, because of different patenting strategies and shares of university-owned patents, incomplete identification of academic patents may bias comparisons at the university level. Our data suggest that this may indeed be the case. For example, consider the following numbers: The Technical University of Dresden, well known as the 15 In contrast, we did not find statistically significant differences between the groups regarding forward or backward citations.

14 540 Scientometrics (2014) 101: Fig. 1 Professor-inventor pairs identified most patent-active German university, accounts for a total of 90 patent applications in our dataset, whereas we only find a total of 29 patent applications for the University of Duisburg-Essen. If we restrict our attention to applications filed at or extended to the EPO, the difference is still appreciable but much less pronounced (28 for Dresden vs. 19 for Duisburg-Essen). The above comparisons suggest the importance of not restricting the search for academic patents to a single patent office. To do so for the German case, researchers have hitherto identified academic inventors based on their professor title (e.g., Czarnitzki et al. 2007; Schmoch 2007; von Proff et al. 2012). We next explored the differences between the results of our algorithm and those obtained by a simple search for the professor title. We found 501 inventors with a professor title in the DE-INV database. To reduce the number of false positive matches and to compare the results, we manually examined whether we could identify these inventors in the PROFLIST. Indeed, we found that only 152 inventors were true positive matches for 126 professors. This is a substantial discrepancy, which is mostly due to individuals legitimately using a professor title, but not (or no longer) being employed at a German university. In most cases, the respective individuals work at nonuniversity PROs. We then applied the further cleaning steps described in the Manual control section. The matching and filtering procedure produced 64 priority patents for 55 German professors in 2006 and This comparison shows that using the professor title as the single filtering criterion drastically reduces the number of academic patents. The resulting number of patenting professors is only approximately 5 % of the number implied by our algorithm. This indicates that few patenting German professors provide title information on patent applications or that the title information is not included in PatStat. Interestingly, the professor title is provided almost exclusively for professors listed as an inventor on EPO patents. We conclude from this comparison that a PatStat-based search for German academic inventors using the professor title would severely underestimate the number of academic patents. In addition, the professor title cannot be used as an unambiguous indicator for academics employed at German universities, but manually examining the professor-inventor pairs is still necessary. In contrast, the differences among the observed patent portfolios produced by the alternative matching algorithms outlined above are modest. The token-based algorithm and 2-gram algorithm produced almost identical results. While simple-string matching generated a lower recall rate, the manual search effort is much lower. This result indicates that, for Germany, the specific algorithm applied for the name comparison step is of considerable importance, but not to the same extent as has been found for other countries. Fewer

15 Scientometrics (2014) 101: Table 8 Results after manual cleaning Algorithm # Identified inventors # Unique inventors # Identified professors 2-Gram Token Simple-string Fig. 2 Professor-inventor pairs identified after manual control spelling errors in German data than in other countries may provide a (partial) explanation. Notably, due to the high levels of missing address information (compared with EPO application data), inventor inventor and professor-inventor filtering produces more inaccurate datasets for Germany. However, using only EPO priority patents is not a viable alternative because it reduces the patent portfolio by approximately 80 % (cf. Fig. 3). Discussion and conclusion This article examined the influence of different approaches and matching algorithms on identifying patenting activities of German university professors. Our analysis was based on two data sources: the PatStat database and the Kürschners Gelehrtenkalender. Whereas previous studies used the professors title to find German academic inventors (e.g., Schmoch 2007; Czarnitzki et al. 2007, 2012; von Proff et al. 2012), studies on other European countries (notably, Lissoni et al. 2008, 2009) systematically matched professor names with inventor names in patent data. We developed a substantial modification of the latter approach that made it applicable to the German context where GPTO applications matter and official faculty and staff listings of universities are absent. The presented method has already been employed to analyze how patent and organizational variables relate to ownership patterns of academic patents in Germany (Schoen and Buenstorf 2013; cf. also the other contributions in Lissoni 2013, for approaches to identify academic patents in other European countries). Beyond our own empirical context, the matching approach detailed above is useful for all cases where a relatively small number of individuals have to be identified in large datasets. In terms of methods, several lessons for future work on inventor datasets can be derived from our study. First, we found that the results are highly sensitive to the filtering technique. The various algorithms used in the name comparison stage led to substantial

16 542 Scientometrics (2014) 101: Table 9 Proportion of patents identified and ownership type Ownership type Priority patents GPTO without extension Priority patents GPTO extended to EPO Priority patents EPO University-owned 218 (56.8 %) 200 (44.2 %) 67 (33.7 %) Firm-owned 64 (16.7 %) 147 (32.5 %) 69 (34.7 %) Others 102 (26.6 %) 106 (23.4 %) 63 (31.7 %) Total 384 (100 %) 453 (100 %) 199 (100 %) gram Token Simple-string Professor title # Priority patents # EPO priority patents Fig. 3 Patent portfolios generated by the different Name Games differences in the number of inventors identified. However, subsequent filtering steps reduced such differences to a small fraction. This result implies that the algorithm used for name matching is not as important as the specific filtering steps, which in turn suggests that misspellings, among other concerns, are not as common in German patent data as in data for other countries. At the same time, given the incomplete information on inventors for GPTO patents in PatStat, more sophisticated filtering methods are necessary. By systematically combining weak criteria, accuracy can be improved substantially. Classification models require training samples that provide information on true and false matches independent of the filters used. We consider our own approach only as a first step in this direction. Without access to a training sample for German academic inventors, we adopted a data adaptive approach by building test samples out of the data. Although this improved our ability to identify academic inventors, the automatic matching approach remains limited. A manual examination of the results is indispensable and the associated effort makes the application to large datasets difficult. Therefore, building test and training samples to improve filtering techniques should be one of the major aims for future research. Additional data are available that could not be used in the above analysis, such as detailed information on name frequency, co-author information as well as patent citations. Even if individual criteria only have small predictive power (e.g., country information), they can generate a higher precision level for a statistical model if sensibly combined with other criteria.

17 Scientometrics (2014) 101: In terms of substantive results, our approach provides evidence that restricting the search for German academic patents to the EPO yields incomplete findings. A search for professor titles in PatStat also underestimates academic patenting activity. Our results moreover confirm conclusions from prior work on other European countries, which showed that identifying university-owned patents is insufficient to evaluate academic patenting in Europe. Overall, we conclude on a cautionary note: Empirically assessing academic patenting in the German innovation system is difficult, and good indicators are not easily constructed. Policies based on seemingly straightforward indicators, such as universityowned patents, may be seriously misguided. Acknowledgments The authors are grateful for comments received from the participants at the Name Game Workshops in Brussels and Leuven. Financial support by the European Science Foundation (project ESF-APE-INV) is gratefully acknowledged. Appendix 1 Cleaning The cleaning stage was aimed at reducing noise for the relevant data without losing relevant information in subsequent stages (Raffo and Lhuillery 2009). The number of false positive matches should be as low as possible. Raffo and Lhuillery (2009) tested the impact of different cleaning algorithms on the matching results using the simple-string match algorithm as a standard. The results showed that, separately, each cleaning algorithm produces a small improvement. However, together, the precision rate was increased 7 % and the recall rate by 64 %. 16 Based on these results, the cleaning stage in this project comprises nine steps and was applied to DE-INV and PROFLIST. In addition to capitalization and comma deletion, these nine steps include eliminating title information in the name field and saving it in a separate file. 17 Replacing umlauts (ä, ö,ü) was especially important for Germany. Moreover, we separated the address field into street, postal code, and country information as well as extracted initials for names. Where either the postal code or the city was specified, we supplemented the missing information. 16 The precision rate is computed by dividing the number of true positive matches by the sum of true positive and false positive matches. The recall rate is computed by dividing the number of true positive matches by the sum of true positive and false negative matches (Lissoni et al. 2010). 17 Cleaning was based on SQL scripts developed by Julio Raffo ( We adapted the scripts and determined the order for execution. We executed the different steps in the following order. (1) Replacement of umlauts (2) Elimination of invalid characters (3) Replacement of accented letters (4) Conversion in upper case (5) Removal of title information and saving in a separate variable (6) Removal of double spaces as well as spaces at the beginning or end (7) Removal of commas (8) Replacement of abbreviations (9) Removal of letters not included in the Latin alphabet

18 544 Scientometrics (2014) 101: Appendix 2 See Table 10. Table 10 Identified criteria combinations inventor inventor filtering Name Geographical information Technological information Application dates Relational data Name Common family name Street City Postal code (5 digits) Postal code (2 digits) IPC 8 (8-digit level) IPC 6 (6-digit level) IPC 4 (4-digit level) First application dates differ by more than 20 years Coinventor Applicant Patent family generation of positive test sample applied to full dataset generation of extended test sample X X X X 1 X 1 V 1 1 V 1 V V X 1 V 1 V V 1 X X V V V V V V V V X 1 X V V V V V V V V X X 1 V V V V V V V 1 X X 1 1 X Notes: 1 denotes a required condition; X denotes that classifier is not allowed to be negative; V denotes an or condition (at least one criteria must be fulfilled)