Founders and Survivors Linkage Strategy

Size: px

Start display at page:

Download "Founders and Survivors Linkage Strategy"

Elwin Ward
5 years ago
Views:

1 Founders and Survivors Linkage Strategy John Bass, University of Tasmania Sandra Silcot, University of Melbourne Len Smith, Australian National University

2 Founders and Survivors Prosopography Database relational genealogical database integrated with an XML source database individual life histories compiled dynamically from diverse sources, linked by a combination of machine matching and human judgment Independent link management module

3 Study Population 74,377 convicts transported to Van Diemen s Land in the 19th Century, and their descendants. Over 1 million records (and counting) Convict ships are our basic grouping unit

4 The database On-going researchable prosopography database updateable, shareable, maintainable, sustainable. Open source multi-user Web based software Backbone is Yggdrasil, GENTECH-compliant open source relational genealogy and prosopography database (Kristensen Data flows from sources to events to participants to people Supported by and integrated with an open source XML database which manages the source data. Individual life histories made up of a chain of dynamically linked assertions or factoids judged to relate to a single person

5 Sources Parish and census records from the home communities in Britain and Ireland court and imprisonment records paper panopticon of convict system records: voyage and disembarkation records, work assignments, conduct and punishment records, requests to marry, records of exit from the system newspaper reports vital registration and shipping records military enlistment and service records etc etc Information constantly expanding as transcription proceeds

7 Linkage strategy Ongoing researchable database of individual life histories requires links which are definite and persistent. but links can evolve over time as more information becomes available Two-step approach: let machines do what they are good at, let humans do what they are good at computer matching to identify definite links where programmed matches not found or ambiguous, present possible links to a human for review and final decision Where place names available, user can map them interactively to assist matching People aggregated on the fly in database queries as the set of linked sources referring to them Facilitated by LKT, independent link management module

9 How was it done?

10 Linking Convict Records About a dozen source manuscript record types, and an overall index Known ontological relationship between some record types eg if a person appears in a conduct record they should also be in the indent and description lists which document entry into the convict system. Linkage assisted by Police Numbers - identify individuals within the convict system primary sources, and Archive Identification Numbers, secondary finding aids assigned to the convicts records by TAHO, the Tasmanian Archives and Heritage Office. Manual correction needed when numbers missing or duplicated

11 Ships as Natural Grouping Ship on which convict arrived in Tasmania persisted as basic element of their identity & administrative processes even after leaving the convict system About 330 ships, each carrying between 250 and 400 convicts Provides a natural blocking factor with very small size

12 Establishing links standardise names to conventions used in Police Numbers compare each record type with each other type, search for matching individuals on basis of forename and surname, plus any fields the sources have in common all within ship block. If computer finds an exact match, link stored in LKT If not, name comparisons made progressively more fuzzy, until a match is found or process exhausted. If only one pair matches, link made. If multiple possible links are identified, or if no link can be found, a human finishes the comparison. Chain of links constitutes an individual s reconstructed life history Links incorporated dynamically - new information included by adding a link to the source in the link management module.

16 Assign Rectype Assign and preserve PERMANENT UNIQUE identifiers to each observation. Transcriptions & inherited databases -- a record type e.g. c31a = con31 transcription "a use a unique id present in the file i.e. a record number configuration data: date, metadata, control totals etc.

17 Data cleaning Ingest workflow: data standardised and cleaned --- person names * forenames broken into parts ie. multiple forenames ("sequences" a strngth of XML) * surname standardisation as per original O'Conner --> conner_o * INITIAL here is C -- vital for police numbers --Police number: initial + numeric part --Dates and Julian day Codebook generated -- shows all fields, all values and frequencies -- iterate cleansing routines and add recoding entries

19 Recoding & Error Correction Recoding step as part of process errors/typos corrected, part of workflow cleaned values by frequency System generated attributes: soundex, nysiis on all nameparts ship blocking values tagging: minorship, arrivedfree, colonialconviction, death

20 Normalize: TEI prosopography model Person: Identification attributes and traits EventList Event: type, date, place A wrapper for event specific attributes Event types:. birth, conviction, voyage, vdlconvict, musters(1830,1832,1833,1835), freedoms (various subtypes), freedomrevokations (various), death TEIp5: Person, Places, Dates, Events with very powerful ways of addressing manuscripts and facsimilies

22 Data dictionary Maintained a data dictionary as we went along incorporating new sources. same event + element names where they were the same thing across files different elements only where meaning is different enables significant variations to be retained and event data to be merged across sources

23 Matching-Linking worked in ship blocks ( very small numbers) Customised pairwise linkage based on domain knowledge exploit what was common; data was good but attributes varied across sources: names, year/places of birth; police no (not unique, needed initial) if a person in one series then they were KNOWN to be in the other look for exact, then loosen/sanity check, if 1 left do link ITERATE/LOOSEN until candidates found Feed links made to LKT leave multiples, orphans to humans Challenges and problems: Transcriptions chosen to minimise transcription time/cost Bits and pieces from different series made up a population very complex underestimated effort needed for population establishment + linkage distinction between MATCHING and LINKAGE! Lesson: if not dealing with censuses (which by definition provides a (in theory!) known set complete closed population, choose your transcriptions so that you establish your BASE population, and sensibly segment to suit your sources to enhance blocks for matching.

24 Mitigation population PER SHIP by choosing a base series which formed the population for THAT ship Not perfect but good enough Add research to existing records i.e. data is prelinked Life course inconsistencies become readily apparent This enables the ships research to largely avoid the need for additional matching at convict level

25 Linking to World War 1 descendants Research question requires linking convicts to their descendants who served in the Australian Infantry Force in the First World War reconstruct a genealogy using multiple approaches and linking both backward and forward in time. Intergenerational linkage is more complex and involves elements of both group and graph linkage. Incomplete transcription and unstructured transcripts sometimes require text mining and other special measures to extract information required for linkage.

26 ERT (Enlistment Records Tasmania) List of Tasmanian-born AIF Enlistees provided by the National Archives of Australia. Includes name, place of birth, and name and address of next of kin Did not capture the AIF record identifier systematically: we mine it from unstructured transcribed text.

27 Link ERT to CCC (Community Contributed Content). In related project volunteers recorded additional information about all convicts on particular ships: death marriages and children descendants (typically 2nd generation) who enlisted in the First World War. Match descendants who had enlisted in the Army according to CCC to list of Tasmanian-born AIF Enlistees and add additional textual genealogy information by mining unstructured CCC text to extract eg registration year, registration number, and surnames

28 ERT to KGB (Kippen-Gunn Births, from previous research). If CCC contains birth data, use it to link descendants to linked database of Tasmanian Births Deaths and Marriages If no CCC data, use name, place of birth and next of kin to link the Tasmanian-born AIF Enlistees to their birth.

29 KGB to KGM (Kippen-Gunn Marriages, from previous research) Use linkages within the Tasmanian Births Deaths and marriages dataset to identify the mother and father of the Tasmanian-born AIF Enlistees find the mother and father in the file of Tasmanian marriages and births. If necessary repeat process for the previous generation.

33 Link Convict Permission to Marry to Civil Registration Within convict records, link convicts to file of Permissions to Marry to identify potential married couples Establishes definitive link between the convict records and the Tasmanian civil registration records Police Numbers not included in transcription, so machine match using ship and convict name, where unambiguous match we link otherwise refer to original images to find the Police Number and use this for matching.

35 Evaluation and future directions. Works satisfactorily but too complex too much tailoring for individual tasks programmer has to determine what match attempts are possible for specific record types and sources. Make adaptive using semantic ontologies and other strategies augment the capacity for automated linkage reduce the amount of human intervention required. Move matching decisions to ontology-aware machine process Require development of formal ontology of our sources Eg use a standard such as CIDOC-CRM well established in museum circles proposed for use in prosopography Yggdrasil relational data base model appears semantically identical to CIDOC- CRM.

36 Future Directions Semantic web technology such as RDF, the model for data interchange and OWL, a knowledge representation language, will (we hope) make it possible to accommodate a wider variety of sources, collaborators and partners. Transition to RDF and ontology-based processing is a logical next step typed unique identifiers integral to our entire approach & generating RDF is almost trivial by exporting key assertions and identifiers using RDF, various datasets can be brought together to enable federated searching and matching across different geographically dispersed datasets Will enable us to focus on provision of specific web services which provide generalised functionality and more sophisticated natural language processing.

37 Conclusion Ongoing project to develop sustainable software for intergenerational prosopography databases which can grow as new sources are added Enable users to escape from the limits of the desktop and commodity software Key features are unique record ids people as aggregations of linked assertions from documented sources Let machines do what they do best, people do what they do best But boundary can move Semantic web will enhance linkage by matching across a wider range of databases. But complexity of sources will be an issue

38 Try it out AP20 ANDS Project in GitHub

39 Funding This project was funded by the Australian Research Council and the Australian National Data Service.

Quebec population resources: towards an integrated infrastructure of historical microdata ( )

Quebec population resources: towards an integrated infrastructure of historical microdata (1621-1965) Hélène Vézina BALSAC, Université du Québec à Chicoutimi Claude Bellavance Centre interuniversitaire