Danish Demographic Database - principles and methods for cleaning and standardization of data

Transcription

1 Danish Demographic Database - principles and methods for cleaning and standardization of data Nanna Floor Clausen Danish Data Archive, Odense, Denmark Danish National Archives nc@dda.dk Abstract In this article is presented the work that has been done and is still being done in cleaning and standardizing seven completely transcribed Danish censuses. The transcriptions are done by volunteers and with the use of different applications. The information in the censuses is not standardized and the transcriptions have further added variation to the original data making cleaning and standardization a tiresome though interesting task. 1 Credits This document is based on the work done by H. J. Marker and me from 2001 on cleaning and standardizing the data. The work on cleaning and standardizing the first completed census 1801 has been published in e.g. newsletters from the Danish Data Archive and elements from these articles are integrated in this article. 2 Introduction The Danish Demographic Database 1 is a collection of transcribed historical sources that are created in a structured way or can be transcribed into structured data. In 1992 a group of historians, amateur historians and genealogists met and founded the Source Entry Project. The background was the fact that many genealogists already transcribed sources for themselves. The rationale for the Source Entry Project was that this work and effort ought to be coordinated and shared for the common benefit. The Danish Data Archive (hereafter DDA) took charge of the organizing of the project and together with representatives from historians and the genealogical 1 societies formed a steering committee. The transcription of the sources was and is done by volunteers. Data definitions and design for each source has been done by the steering committee and in close cooperation with the Danish Data Archive (in practice H.J. Marker 2 and me). From the beginning definitions were made for a large range of sources, but as access to the censuses and parish registers was made easy 3 these sources gained priority. From 1993 until today several source entry applications have been in use and they have all been developed by volunteers. The volunteers are free to decide what they want to transcribe so there are data to a greater or lesser extent from all census years and pasishes. In the beginning of 2000 it became clear that if all volunteers became focused on the 1801 census it would be possible to complete it in This goal was achieved and since then additional 6 censuses have been completed. The work on cleaning and standardization of the following censuses form the foundation of this article: 1787, 1801, 1803 (Duchy of Schleswig), 1834, 1840, 1845, 1850, 1880 and 1885 that was only taken in Copenhagen. 3 The data historical censuses 3.1 Questions in the censuses Over the years the census-questions have changed, sometimes new questions have been 2 H. J. Marker, director of the Swedish National Data Service. Before that he was a senior researcher at DDA from and is one of the primary founders of the Data Entry Project. 3 The Danish Data Archive received a complete set of Xerox-copies of the original census ledgers from the National Archives which the volunteers could borrow for free. This meant that the volunteers did not have to use the reading rooms of the regional archives. Parish Registers were on microfilm.

2 added and at other times questions were removed. The censuses are very well structured and in Denmark the original censuses have been preserved. The Danish National Archives has scanned and published all the sources and made them accessible on the web free for all. 4 This is likewise done for the parish registers and is presently being done for other popular sources (for the genealogists) like military proscription rolls, probate registers and birth protocols. New types of sources are added on a regular basis (and according to funding). Each census holds information on Source identifies census year Place names: county, district, parish and in the cities name of street and house number: these fields identify the place of living for each individual Name - full name of all members of the household Age - from 1880 the achieved age Marital status married or not Occupation - stated as in the sources In 1787 and 1801 the field Position in household was a separate field whereas in the later censuses it was included first or last in the occupation column. In 1845 important new fields were added: the field Place of Birth, Number of families in the farm or house and what we have called Handicaps (the information required in this field varied in the censuses from information on being insane to being blind or deaf). In 1855 the field Religion was added. Gender after 1880 this was stated in a separate column A volunteer is obliged to transcribe a whole parish with the exception of Copenhagen where it is allowed to transcribe the streets separately (due to the large amount of people in a parish). DDA supplies each parish with a unique number (kip- No) when the parish is taken out for transcription. This number is added to each record. Each record is furthermore given a unique record ID within the parish/street. The combination of these ID s is used to identify each record across all the censuses. The coordination of the project has resulted in the development of unique numeric codes for 4 each parish and in the later years for streets in Copenhagen. Many parishes and streets have the same name or almost the same name so it might lead to a volunteer using the wrong name for the parish he is transcribing. The numbering of the parishes used the numbers of the municipalities defined by the interior ministry as a starting point and added record numbers to these codes. The list is constantly expanded as new areas and streets are selected for transcription. The counties and districts are also given unique numeric codes. The table with the names of districts, counties and parishes and their ID s are integrated into the data entry programs as a drop down list. The table controlling the geographical codes, codes for place names, includes codes for urbanisation: is it a rural area, a town or Copenhagen. It also includes a code for which part of Denmark it belonged to. In 1864 Denmark lost the southern part of Jutland and the Duchies apart from small areas in that region. The codes for urbanization are necessary when we analyse the data because Copenhagen had its own legal status. 3.2 Data entry In order to understand the work on cleaning and standardizing the data it is necessary to explain how the data are transcribed. The applications used for transcription have changed over time but the principles and instructions are almost unchanged. Right from the beginning in 1992 the structure of the entry project was the definition of a variety of structured sources. The definitions are published along with instructions on how to transcribe the sources. The most important principle is the distinction between source reading and source interpretation. The text in the source must be transcribed exactly as it is even when the census taker might have made misspellings. This may be the case when stating names or occupations. The volunteer may add!! (always two!!) and in a commentary field write the correct name or spelling. Often the census takers used curly brackets indicating the same term; the term stated once must be given to each record within the brackets. Another way of stating repeated information is by writing do or ditto. The volunteer must enter do or ditto but in square brackets [] is entered the text the do substitutes. The same is done when place of birth is stated as the source often says: in this parish so the data will e.g. hold: in this parish [Abild]

3 The original sources from 19 th century are written in gothic handwriting often using the census schemas in the extreme making it very difficult to read the text. The volunteer is instructed to enter as much as possible and replace the illegible with two??. One? may appear in the sources but not two consecutive question marks. If text is implicit, e.g. the place name or house number, this information should be added in the relevant data field. Often children are stated only with their first names and the last names implicit. They should not be entered unless it is obvious and then in square brackets. Missing information, like place of birth which should have been stated, cannot be entered but it should be noted in the commentary field that the information is missing. A major exception from the source reading principle relates to marital status. Only six possible values are allowed plus the two question marks for illegibility. If a researcher might be interested in the original spellings of e.g. being married he will have to look in the original sources. 3.3 Data entry applications DDA did not have the resources to develop any applications but luckily some volunteers made the first application and since then new generations of applications have been developed by supporters of our source entry project. Presently online transcription is emerging along with the traditional offline transcriptions. The applications have allowed us to make stricter recommendations and to simplify the acquisition of the data; the data are now accepted in only two formats. General information is entered only once and other fields with repeated information are repeated automatically by the application. The programs are freely available on the condition that the user informs DDA what he/she wants to transcribe in order to avoid redundancy and he/she promises to deposit the transcription at DDA. 3.4 Data cleaning and standardization When a census is complete we make a copy of the data thus making a version to be used for research and keeping one for the general public on our website. The public database is by the way divided into county tables - one for each county. During this process each record is supplied with a new record ID unique for the whole census. We add a field holding information on the number of persons for each record ID. The default is one. This number is updated during the cleaning process when the source mentions empty house or gives the number of persons if a group of persons has been grouped in the census. 5 Gender: With the 1880 census gender became a field in the census schemas but nevertheless the volunteers are instructed to always give information on the gender. Too often this is not the case and for practically all the completed censuses this information has been given in app. 50% of the records. The logical possible values for gender are male, female or unknown. But the fact is that before the cleaning and standardization there are possibilities where blank and NULL is not included. In the Danish censuses we also have transcriptions from the Duchies that are made in German. Some values are due to information entered in wrong fields, parentheses (which some volunteers use extensively) and the like. This is simple to clean. The bigger task is in applying gender when it is missing. The variation in women s name is smaller than for men so the standardization is begun with women. The first part is done by looking at the names and then by looking at position in household. A name with the suffix datter (daughter) is presumed to be a female. Names beginning with typical female names like Anna, are likewise presumed to be females. Most names are typical for one of the genders making the task simpler. The rest group with missing codes was standardized using household position, like hans kone (his wife), hans datter (his daughter). If the household position contains the suffix mand (man) it is normally coded with a M. Information about occupation was mostly done for men and this information was also used when coding gender. The small number of records not standardized when the preceding procedures had been carried out had to be standardized more or less one by one. In the 1880 census only 41 records of nearly 2 mio records do not have a code for gender. Another result of this first cleaning is the correction of errors and better grouping into households. This is done during all the following cleaning procedures apart from the specific task of making correct households. 5 In 1801 a record simply stated 25 negroes

4 Age: Age is given as a text which gives room for many wrong entries. Often numbers like age are the simplest text to read in the sources and an easy way to identify a person when looking for a person in the sources but there is a great variety in these numbers. The completed censuses have in general almost 800 different values. All text wrongly entered in the field results in data being corrected; all ages more than 100 have been controlled against the sources manually. The data from the age field is converted to a new float field. Records that it is not possible to convert are cleaned. This relates to ages like 1½ which will be 1.5. The records containing!! or?? are either controlled or the extra characters ignored. All ages below 1 year like 2 weeks are converted to a fraction of a year. This kind of stating age in the sources ought not to appear as before 1880 a child of e.g. 2 weeks should be given the age of 1 year. From 1880 till 1901 age was given as achieved age and after 1901 age was given as date of birth. Age statistics show the different way of recording age. In our analyses we convert to modern age by adding 1 year to the ages in the censuses before 1880 but in 1880 many census takers still used the former procedure. Population pyramids clearly demonstrate the tendency to age heaping around the ages ending in 5 and 10. The tendency is decreasing from 1787 when it is almost extreme until 1880 when it has almost disappeared. The tendency to age heaping was much more evident for women, especially elder women, than it was for men. 6 Marital status: The guidelines for data transcription state that the information for marital status should be normalized but nevertheless marital status was given in a large variety but fairly simple to standardize. The present version of the data entry program has a drop-down list for the status. We have transformed the information to one of six codes: Code Marital status 0 Unexplained 1 Not married 2 Married 3 Widow 4 Separated 6 See pdf (Widowhood in Denmark 1801) where this is explained. N. Clausen, Divorced For children with missing information, living at home and at the age below 15 they are coded with not married. This is normally also the case for servants. When a servant is married it is clearly stated in the sources and the transcriptions. Position in household and households: The censuses for 1787 and 1801 has the information on position in household in a separate field; the other censuses has this information in the same field as occupation; either as the first or last information. The standardization of this field is very complex and there are more ways to standardize the data. We have chosen to make a sort of hierarchy having everybody in the household given a relation to the head of household. A table with possible relationships was created. This table holds fields for numeric code, term, (hierarchical) order, number of steps (from head of household), generation in relation to head of household, family (family member or not) and the term in English. Currently there are 367 different codes for position in household. Examples of terms are: head of household (order: 1), spouse of head of household (order 2), Common child of head of household, Separate child of head of household, Separate child of spouse of head of household, Separate child of fiancée of head of household, Separate child of sweetheart of head of household, Foster-child of head of household. Examples of the more exotic ones: Parent of spouse of in-house by head of household, Servant of in-house by head of household, Quartered by head of household to Member or prisoner, Spouse of member or prisoner, Child of member or prisoner. The head of household is 0 steps from head of household and the spouse to head of household is 1 step from head of household. The member or prisoner is also 1 step from head of household but has 0 in the field family member. As a curiosum I can give the position with 7 steps from head household: Child of great-grandchild of sibling of parent of head of household. The head of household and his spouse is 0 generation from the head of household. The children are 1 generation from head of household and parent of head of household is -1 generation from head of household. It is not possible to give this

5 information on generation on all the codes, only on codes for family. A table holding the codes and terms for family membership enables us to code for unknown, family, staff and family to staff. The work on standardizing the position in household is initially done by coding the records that are entered correctly which is the majority. The rest group is done via a large sql-job which has been developed making use of our experience with the data. The job takes care of correcting spelling to e.g. one term for servants, siblings etc. and correcting typos. There is a sql-job for each census-year but of course with most of the code in common. The next step adds the codes mentioned above for those positions having wrong spellings. Example: exec procedure with these parameters: 2, 1, '[hans] kone', 'm', null meaning that '[hans] kone' (his wife) will have the code 2 for spouse instead of 1 and that the head of household is a male ( m ) instead of null as gender. This job is run iteratively and each iteration results in still more records coded correctly. The remainder of the records is coded manually and represents many hours of work with repetition of the mentioned sql-job. The work of standardizing position in household is at the same time very useful in the division of the records into households. If there are e.g. two heads of household in a household something is wrong as when there is no head of household. If a woman is his wife and listed as head of household then the household is not correctly divided. When the census was taken the census taker rather clearly made a division between each household. In some censuses, like in 1845, there is a field to hold the number of families in each farm or house. In others like the one from 1801, each family was given a number. Furthermore there was often made a line between each household. The applications have a special field for holding the household number and the number is automatically repeated until the next family begins - stated by the volunteer. This is done in app. 50% of the records; the rest we have to divide using the information in the censuses and sometimes by referring to the original sources. These records are grouped into household by making use of the kipno, place name, title and position in household as head of household. If a household had more than one head of household the household was split into two households. If a household did not have a head of household these households were controlled manually. Some of the households without head of household could be a hospital, servants living on a farm etc. In order to distinguish the different types of households we have made a referencing table with the following household types: Code Description 0 Empty 1 Family 2 Institution 3 Soldiers 4 Without head of household Each household is given a new household ID and the household IDs are stored in a separate table combining the reference to household ID and household type. These tables are used for the many analyses that can be done on households. We make other analyses and statistics on the data and this results in yet more updates to the household divisions. Occupation: In 1787 and 1801 this field holds information on occupation alone. After 1801 the census schema asked title, occupation, handicraft or what else a person lives by. Later censuses comprise information on occupation and position in household which makes it more complicated to standardize the data. When we began this task we created a table holding a standardized term of an occupation (occupational codes). This standardized term was given an internal DDA code and a HISCO 7 code. The reason for making the DDA codes is that HISCO is not detailed enough for many occupations. The HISCO project is gradually expanding in detail but we continue updating the table with the DDA codes. A farmer and an estate owner have the same HISCO code (61110) whereas we consider them as two different occupations and categories. HISCO holds codes for status and where applicable they have been added to our table. A table for each census with a subset of the records having information in the occupation field was created. It holds 4 fields: a reference ID to the full census table, a standardised term for the occupation, the DDA code for the occupation 7 Codes for historical occupations based on ISCO. Used for comparative research.

6 and the number of occupation stated in the census. The standardization of occupations in the cases where more occupations are listed can be done in many ways. The instruction in the census schemas lists title before occupation although you do not make a living of a title by itself. We have decided to split each element into a record in the occupation table and in order to keep the sequence from the census they are given a number. Normally a person have one occupation like farmer but often a farmer is listed as bonde og gaardbeboer (peasant and farm resident). This person would then have two records: Bonde Gaardbeboer The first field is the record ID, the second number refers to the order in the census and the last numbers are DDA codes for the occupation. Both refer to the same HISCO code. The DDA codes allows analyzing what people live by when they do not have a standard occupation, like living on alms, pensions or elderly having a contract with the farmer where they live ( aftægt ). When we make statistics each person s occupation are included with the same weight as it is not possible to see what occupation made the biggest income. Being a cottager and smith are calculated with 50% for each occupation. Being a lodger and receiving alms are also giving 50% each but in the analyses it must be taken into account whether a lodger is an occupation or not. The outcome depends of what you want to express. The standardization of the occupations reduces the variation given in the sources but as we preserve both the transcriptions and the standardized records we do not lose any information. The recoding is done in many sql-steps comprised into a single sql-job, that takes care of misspelling, eliminates references to gender when it is not needed (like servants), splitting of the occupations if more than one is given etc. An integrated task of the coding is removing occupations that are not occupations or an allowance like being a daughter or deaf. The sql job is used for each census and for each census more lines are added as more kinds of occupations are listed during the 19 th century. It is the hope that by using and expanding the same sql job (replacing the text for year of census) the task will be less tiresome for the censuses not yet coded. We have only begun on standardising occupation for the censuses after and 1801 are completely standardized. Data from the Danish Demographic Database is on the way to be included into the North Atlantic Population Project (NAPP) 8 which again will be integrated with the IPUMS project 9. This project has developed a set of occupational codes close to the HISCO codes. The codes to be used in this project have been added as an extra field into the occupational codes-table. The concordance between the DDA codes and NAPPcodes was done by using the HISCOcodes and the DDAcodes in combination. As all three kinds of codes are in the same table it is now possible to make cross overs from one code system to the other. The system is likewise well-prepared for future updates. 4 Discussion on standardization of Danish censuses In this article I have tried to demonstrate the work being done on the Danish, completely transcribed censuses. Norway had the census from 1801 transcribed many years before Denmark which made it still more interesting to also have the Danish 1801 census complete. The standardisation of the Norwegian census and the Danish census has been done using different principles. We have standardized the data very thoroughly into the finest details preserving the references and link to the transcriptions. This is very timeconsuming but when it is done we can generate the data into any format with only little effort. A problem mentioned above is the mixture of titles and occupations. When a person s only occupation is e.g. lodger we have to assume something about the income but this is not evident from the census. Being a parish executive officer ( sognefoged ) did not generate an income but can be seen as a sign on the importance of the person as a farmer. The granularity is demonstrated in the statistics in a way that might lead to greater precision in interpretation of the data than the data actually reflect. E.g. the division of each listed occupation into a record accumulates to decimal numbers. Like: there are % of farmers instead of saying, 25 % of farmers (the numbers are fictive). The coding of position in household is relatively simple using our sql-jobs for the vast part of the records but time-consuming for the re

7 maining records. The benefit of the coding is the potential for making all kinds of detailed analyses on the household structures and composition but of course the result must be weighed against the available resources. The standardization of the data has required the development of a large collection of sqltables, views and stored procedures that can be reused and updated for each census year. The standardisation tables are used for all censuses which means that once a record is coded all the extra information from these tables are added to this record. The problem is the few resources that we have resulting in a postponement of the standardization of all the censuses. References N.F. Clausen. SSHA The elderly in 1787 and 1801; 1.pdf N.F. Clausen, H.J. Marker. SSHA 2012 Marriage patterns in DK 19 th century N.F. Clausen. ESSHC Widowhood in Denmark 1801 H.J.Marker.2008.Counting Danes H.J. Marker. 2002: Completion of the 1801 census, H.J. Marker : Klargøring af FT1801 Metode & data, nr 92, 2006: