Taxonomic Name Recognition (TNR) in Biodiversity Heritage

Transcription

1 Taxonomic Name Recognition (TNR) in Biodiversity Heritage Library L L Qin Wei, Chris Freeland, P. Bryan Heidorn Missouri Botanical Garden

2 Co-author Chris Freeland Director of Biodiversity Heritage Library IT division manager of Missouri Botanical Garden 10/26/08 02:01 AM TNR in BHL 2

3 Biodiversity Heritage Library(BHL) Ten major natural history museum libraries, botanical libraries, and research institutions have joined to form the BHL. The group is developing a strategy and operational plan to digitize the published literature of biodiversity held in their respective collections. This literature will be available through a global biodiversity commons U 1 0 U More information about BHL could be found at 10/26/08 02:01 AM TNR in BHL 3

4 Participating institutions( ( (ar ) American Museum of Natural History (New York, NY) The Field Museum (Chicago, IL) Harvard University Botany Libraries (Cambridge, MA) Harvard University (Cambridge, MA) Marine Biological Laboratory / Woods Hole Oceanographic Institution (Woods Hole, MA) Missouri Botanical Garden (St. Louis, MO) Natural History Museum (London, UK) The New York Botanical Garden (New York, NY) Royal Botanic Gardens, Kew (Richmond, UK) Smithsonian Institution Libraries (Washington, DC) 10/26/08 02:01 AM TNR in BHL 4

5 Open Access O O BHL Project strives to establish a major corpus of digitized publications on the Web drawn from the historical biodiversity literature. This material will be available for open access and responsible use as a part of a global Biodiversity Commons. We will work with the global taxonomic community, rights holders, and other interested parties to ensure that this legacy literature is available to all. 10/26/08 02:01 AM TNR in BHL 5

6 10/26/08 02:01 AM TNR in BHL 6

7 TNR in BHL T NR i n BHL T NR i A significant aspect of BHL is the incorporation of algorithmic Taxonomic intelligence provided by ubio.org.. r g mi c i n t e l l i g e. r g mi c As materials are scanned, the image files are processed through ABBY FineReader or PrimeOCR to create text derivatives. Those text files are then submitted to ubio s TaxonFinder web service to identifies strings in the text that match the characteristics of scientific names. s c i e n t s c i e n t i f i c n a 10/26/08 02:01 AM TNR in BHL 7

8 Two TNR algorithms w 2 2 o 2 o T NR a l g o TaxonFinder is developed by ubio and it uses statistical models that were created from the validated organism names that are in NameBank. These models aim to describe the structure and frequency of common character sequences of organism names, such that TaxonFinder can infer whether an unknown word has a similar structure as a known organism name. a k n o wn o r g a n a k n Online only n l i n e o n 10/26/08 02:01 AM TNR in BHL 8

9 Two TNR algorithms FAT, short for Finds All Taxonomic names, was developed aiming to automatically extract all the taxonomic name from the biological literature. It then use the parts already classified to build lexica and statistics (dictionary lookup), which will be used to classify the rest of the text. (Sautter et al) b e u s e d t o c l a s s i f y t h e Offline usage and customized dictionaries( Of f l Of f l i n e ) 10/26/08 02:01 AM TNR in BHL 9

10 10/26/08 02:01 AM TNR in BHL 10

11 Digitalization Process Di g i Di g Libraries images to texts process text BHL Web OCR TNR Text Database Ubio Database BHL (Text Mining Database) Unstructured Data f i n d Text Mining find names Structured Data f i n d 10/26/08 02:01 AM TNR in BHL 11

12 Digitalization Process Di g i Di g Libraries OCR Error TNR Error BHL Web OCR TNR Text Database Ubio Database BHL (Text Mining Database) Unstructured Data Text Mining Authority File Error Structured Data 10/26/08 02:01 AM TNR in BHL 12

13 Sample Characteristics a m Number of Pages Average Number of Tokens Average Number of Names /26/08 02:01 AM TNR in BHL 13

14 Evaluation Measures v a l u a v Precision is the proportion of matching strings that are valid names. In our case,the precision means the capability of the algorithm to exclude the non-valid name in the result. n a me Recall is the proportion of valid names in the whole database that were returned as true positives. It means the capability of finding all valid names from the database. f r o m In this evaluation, we also use a single measure F- score which is a harmonic mean of R and P: F-score=2(Precision*Recall)/(Precision+Recall) 10/26/08 02:01 AM TNR in BHL 14

15 Sample Language Distribution Di s t r i b u t LanguageDistribution /26/08 02:01 AM TNR in BHL 15

16 Ground Fact r o u n d Total: 3003 valid names Unique Name One Word Two Words Three Words More Words No. of Valid names Percentage 89.01% 35.76% 40.63% 17.25% 6.36% 10/26/08 02:01 AM TNR in BHL 16

17 OCR Overall Performance OC OCR Ov Total 3003 Wrong OCR 1056 Error Rate 35.16% 10/26/08 02:01 AM TNR in BHL 17

18 Error Breakdown Error Percentage more 31% onechar 31% fourchar 6% threechar 11% twochar 21% 10/26/08 02:01 AM TNR in BHL 18

19 OCR-Is language matters? OCR Error Rate lat unk ger eng dut spa fre swe por cze ita Language 10/26/08 02:01 AM TNR in BHL 19

20 TOP OCR error patterns 1 Insert Space 8 n->v 2 Omit Space 9 l->i 3 e->c 10 r->i 4 u->i 11 u->ii 5 u->n 12 h->l 6 i->l 13 h->ii 7 c->e 14 e->o 10/26/08 02:01 AM TNR in BHL 20

21 NameBank For TaxonFinder, NameBank impleteness error rate is 6% NoBankID IsValidName Found Total /26/08 02:01 AM TNR in BHL 21

22 Without_OCR_Error* No. of Names (identified by biologist) Error Analysis No. of Names Found by algorithms Correct Exact Match Precision P r e c i Recall Re c a l 36.62% 23.34% F-score 38.47% 25.77% With_OCR_Error* TaxonFinder FAT No. of Names (identified by biologist) No. of Names Found by algorithms Correct Precision P r e c i 43.77% 32.25% Recall e c a l 25.82% 17.21% F-score TaxonFinder FAT % 28.20% 34.80% 24.73% 10/26/08 02:01 AM TNR in BHL 22

23 Overall Performances Step OCR TaxonFinder* NameBank Total No. of Names Error Rate 35.16% 41.09% 3.06% 79.32% 10/26/08 02:01 AM TNR in BHL 23

24 Conclusion o n c l Our result indicate that TaxonFinder is slightly better than FAT. But even TaxonFinder only got an F-score of 38.47% which is relatively lower compared to other Named entity recognition results. For instance, the best system entering Message Understanding Conferences (MUC) scored 93.39% of F-score while human annotators scored 97.60% and 96.95%. We could see that there is a large space we could improve the algorithm to get better result. ( ( mp r ( mp r o v ) 10/26/08 02:01 AM TNR in BHL 24

25 Future Work F u t u r e W Artificial Intelligent Retrieval is the trend ( Ar t i f i c Ar t i f ) How could we achieve it? ( Ho w c ) Experiments on machine learning methods x p e r i m x p Using other external sources, e.g. ontologies Us i n g Us i n g Automatic OCR correction Au t o ma t i c OC Fuzzy matching algorithms in IR F u z z y ma 10/26/08 02:01 AM TNR in BHL 25

26 References [1] S. Rice, J. Kanai, and T. Nartker. An evaluation of OCR accuracy. In UNLV Information Science Research Institute Annual Report, pages 9-20, 1993 [2] Koning, D., N. Sarkar, and T. Moritz TaxonGrab: Extracting Taxonomic Names from Text. Biodiversity Informatics 2: [3] Sautter, G., K. Bohm, and D. Agosti A combining approach to find all taxon names (FAT) in legacy biosisystematics literature. Biodiversity Informatics 3, ( [4] McCray, A. T., A. R. Aronson, A. C. Browne, T. C. Rindflesch, A. Razi, and S. Srinivasan UMLS knowledge for biomedical language processing. Bulletin Medical Library Association 81: /26/08 02:01 AM TNR in BHL 26

27 Questions? 10/26/08 02:01 AM TNR in BHL 27

28 Thanks! 10/26/08 02:01 AM TNR in BHL 28