SURVEY ON POLICE INTEGRITY IN THE WESTERN BALKANS (ALBANIA, BOSNIA AND HERZEGOVINA, MACEDONIA, MONTENEGRO, SERBIA AND KOSOVO) Research methodology Prepared for: The Belgrade Centre for Security Policy
Contents Contents... 3 Face-to-face Survey CAPI (Computer-assisted personal interviewing)... 4 omnibus surveys... 4 CATI survey (Computer assisted telephone interviewing)... 4 Questionnaire... 4 Sampling methodology... 5 Sampling procedure CAPI survey... 8 Sampling procedure CATI survey... 10 Fieldwork procedures - F2F survey... 11 Fieldwork procedures - Telephone (CATI) surveys... 14 Data processing and analysis... 16 Analysis and reporting... 17
3
FACE-TO-FACE SURVEY CAPI (COMPUTER-ASSISTED PERSONAL INTERVIEWING) Survey on general population was conducted using face-to-face method on representative sample of 1000 citizens aged 18+ in Albania, Bosnia and Herzegovina, Montenegro, Kosovo and Serbia (without Kosovo). The survey evenly covered the entire territory of each of these six countries, both rural and urban settlements, with mixed gender male and female, mixed ages, mixed education level in proportion reflecting population. The size and structure of the sample make possible generalization of the obtained findings to entire population of citizens of these countries. Namely, we can say with a degree of certainty, that findings of surveys conducted on representative sample of citizens reflect situation in the entire population. OMNIBUS SURVEYS In countries where there are monthly omnibus surveys, questions were included into omnibus survey (Serbia and Montenegro), while in countries where there are no omnibus surveys in June (Albania, Bosnia and Herzegovina and Kosovo), ad hoc F2F survey was conducted. BRIEFLY ABOUT OMNIBUS SURVEYS Face-to-face omnibus surveys are being conducted in regular monthly intervals, which allow clients to obtain relevant information and to share costs of the fieldwork with other participants in omnibus survey. Information obtained in this way is representative for the entire population, while the questions and topics for the Omnibus can determine Client according to their needs. Omnibus is a type of quantitative research whose results are interpreted according to the statistical significance of quantified parameters, and it provides us with a comprehensive picture of the market at any given point in time. Because of its dynamic and consistent methodology, Omnibus is a great tool for tracking trends and identifying changes in the market and public opinion. CATI SURVEY (COMPUTER ASSISTED TELEPHONE INTERVIEWING) Survey on general population was conducted using CATI method on representative sample of 1000 citizens aged 18+ in Macedonia. The survey evenly covered the entire territory of each of these six countries, both rural and urban settlements, with mixed gender male and female, mixed ages, mixed education level in proportion reflecting population. The size and structure of the sample make possible generalization of the obtained findings to entire population of citizens of these countries. Namely, we can say with a degree of certainty, that findings of surveys conducted on representative sample of citizens reflect situation in the entire population. QUESTIONNAIRE The survey instrument duration is estimated to be about 20 minutes. The BCSP will be provided with questionnaire in the English and Serbian language drafted by the POINT network immediately after signing the Contract. ISM team would translate the questionnaire to local languages. 4
SAMPLING The public opinion survey is planned to be performed on general population 18+ in Albania, Macedonia, Bosnia and Herzegovina, Kosovo, Montenegro and Serbia. In total 6000 interviews will be performed, 1000 in each country. In each country the sample will be nationally representative. The sampling universe will be based on the data from Census and estimated population dynamics. The type of sample will be stratified three-staged random representative sample. The three stages of the sample are the following: Primary sampling units (PSU) will be Polling station territory in all countries. Polling stations territories enable the most reliable sample selection, due to the fact that for these units the most complete data are available (dwelling register - addresses). Polling stations territories are defined by street(s) name(s) and dwelling numbers. Each polling station territory comprises approximately 200 households (approximate size of enumeration areas), with exception of the settlements with less than 300 HH which are defined as one unit. Secondary sampling units (SSU) will be households: One household comprises people living in the same apartment and sharing the expenditure for food. Tertiary sampling units (TSU) will be respondent: Respondent will be adult member of the household (a person above 18 years). The stratification in each country will be done according to region and type of settlement. The purpose of stratification is optimization of the sample plan, and reducing the sampling error. The stratification is done based on criteria of optimal geographical and cultural uniformity. The exact strata which will be used in each country are given in the Table 1. In Table 1 the specification of the sampling methodology in each country (sampling universe, frame, sample type, strata, sampling stages, etc.) are briefly described. In countries where CATI survey would be performed, the type of sample will be stratified two-staged random representative sample. The two stages of the sample are the following: Primary sampling units (PSU) will be households: One household comprises people living in the same apartment and sharing the expenditure for food. Secondary sampling units (SSU) will be respondent: Respondent will be adult member of the household (a person above 18 years) chosen according to quota criteria. In both type of survey (CATI and CAPI) the stratification in each country will be done according to region and type of settlement. The purpose of stratification is optimization of the sample plan, and reducing the sampling error. The stratification is done based on criteria of optimal geographical and cultural uniformity. The exact strata which will be used in each country are given in the Table 1. In Table 1 the specification of the sampling methodology in each country (sampling universe, frame, sample type, strata, sampling stages, etc.) are briefly described. 5
Table 1. Sampling methodology per each country Albania Bosna and Hercegovina Serbia Kosovo Montenegro Macedonia Data Collection Face-to-face in home Face-to-face in home Face-to-face in home Face-to-face in home Face-to-face in home Telephone (CATI) survey method CAPI CAPI CAPI (Omnibus) CAPI CAPI (Omnibus) Sample universe Population 18+ Population 18+ Population 18+ Population 18+ Population 18+ Population 18+ Sampling Frame List of polling stations territories List of polling stations territories List of polling stations territories List of polling stations territories List of polling stations territories Based on the data of Telephone (book) database, Census from 2002, vital statistics and the migrations data, as well as IPSOS estimates of population and households for 2011 Source of data Electoral Commission Central Election Commission, IDDEEA Electoral Commission Electoral Commission Electoral Commission Strata Strata allocation Areas of the population excluded from the sample Type of sample 3 traditional regions: North, Central and South; Type of settlement: urban/ other Proportional to Census 2011. data and estimated population dynamics Inhabitants of poorly accessible, remote parts of the country, 0,2% of the population Three stage random representative stratified sample 6 regions: West Republic of Srpska, East Republic of Srpska, Cazin region, Herzegovina, North Federation B&H, and Sarajevo with surrounding (Federation B&H); Type of settlement: urban / other Proportional to Census 2013. data and RSO s figures and estimated population dynamics Inhabitants of poorly accessible, remote parts of the country, 0.3% of the population Three stage random representative stratified sample 4 regions: Vojvodina, Belgrade, Sumadija and West Serbia, South and East Serbia; Type of settlement: urban / other Proportional to Census 2011. data and estimated population dynamics Inhabitants of poorly accessible, remote parts of the country, 1.6% of the population Three stage random representative stratified sample 5 regions: Pristina (Central), North-West, South-West, North and South-East Kosovo; Type of settlement: urban / other Proportional to Census 2011. data and estimated population dynamics None Three stage random representative stratified sample 3 regions: North, Central and South; Type of settlement: urban / other Proportional to Census 2011. data and estimated population dynamics Inhabitants of poorly accessible, remote parts of the country, 1.5% of the population Three stage random representative stratified sample 4 traditional regions: Skopje, Northwest & Kumanovo, Southwest, and East & Central; Type of settlement: urban/ other Proportional to Census 2002. data and estimated population dynamics Inhabitants of poorly accessible, remote parts of the country, 0.8% of the population Two stage random representative stratified sample 6
Sampling stages: Primary sampling units PSU: Number of PSUs in sample universe: Polling station territory Polling station territory Polling station territory Polling station territory Polling station territory Households 5.504 4.709 8.246 2.256 1.149 251.844 Selection procedure: PPS PPS PPS PPS PPS Simple random sample Secondary sampling units SSU: Number of SSUs in sample universe: Selection procedure: Tertiary sampling units TSU: Number of TSUs in sample universe: Households Households Households Households Households Household member, 18+ 722.262 1.163.387 2.487.886 297.090 192.242 1.497.013 SRSWoR SRSWoR SRSWoR SRSWoR SRSWoR Quota Household member, 18+ Household member, 18+ Household member, 18+ Household member, 18+ Household member, 18+ 2.060.324 3 226 570 5 920 374 1 147 779 474 903 Selection procedure: Random selection Random selection Random selection Random selection Random selection Number of sampling points 125 125 125 125 125 Sample size 1000 completed 1000 completed 1000 completed 1000 completed 1000 completed questionnaires questionnaires questionnaires questionnaires questionnaires Weighting variables Gender, age, education, Gender, age, Gender, age, education, Gender, age, Gender, age, education, region, type of education, region, type region, type of education, region, region, type of settlement settlement of settlement settlement type of settlement 1000 completed questionnaires Gender, age, education, nationality, region, type of settlement 7
SAMPLING PROCEDURE CAPI SURVEY Step 1 Stratification Population data is used to make initial strata. When national representative samples are being planned, two variables are always used for creating strata and estimating its size and proportion in sample stratum and urban/rural. Stratum and urban/rural variables are used at the same time for population and, furthermore, sample stratification, so samples are stratified in two dimensions. Step 2 Sample Strata Estimation After creating initial strata and calculating their proportion in population and in sample, respondents are distributed to initial strata using proportional criterion percentage of respondents per stratum is proportional to its size, i.e. percentage of respondents in sample strata is the same as percentage of citizens in population strata. When planning national representative samples, we are actually matching population strata proportions in order to get the sample structure that is identical to population structure. Number of respondents in cell (strata=region*type_of_settlement) is calculated using the following formula: No_of_respondents_in_strata_(cell) = Percentage_of_citizens_in_strata_(cell) / 100 * No_of_respondens_in total Step 3 PSU Allocation Based on planned sample size and number of respondents per sampling point, number of sampling points is estimated using following formula: No_of_sampling_points = No_of_respondents_in_strata_(cell) / No_of_respondents_per_sampling_point Usually, we are planning a little bit more sampling points than number we get using this formula. Since this calculation provides numbers with decimal places, we are using rounding to estimate exact number of sampling points per stratum. Step 4 Drawing Polling Station Territories When we have exact allocation of sampling points, i.e. when we know number of sampling points in each strata cell, we are drawing polling stations territories within strata. One polling station territory represents one sampling point. Polling stations are selected with Probabilities Proportional to Size (using Lahirie PPS Cumulative algorithm), meaning that bigger (with more households) polling stations have greater chance to be selected. This information (Step1 to Step4) suggests that our sampling framework do not consist of citizens, but of polling station territories. Polling stations territory division is official. Physical (geographical) borders of polling station territories are well known and all these data are stored in database we use for sampling purpose. These data are also used as a manual for interviewer, giving him/her information where to start and where to go. 8
Step 5 Household Selection Households are selected by random route technique starting from given addresses (SRSWoR). Interviews at the starting point address are not performed. From a starting point, they move in the direction of increasing house numbers and take the right side of the street. They skip three entrances (house numbers) and in the FOURTH house in a row from the starting point, they look for the first household where they can interview one person that is the Start. From the Start, moving on the right side of the street and in the direction of increasing ordinal numbers, they choose every FOURTH house number that is, they enter the FOURTH ADDRESS IN A ROW, then the EIGHT ADDRESS IN A ROW etc. At every cross-road, they turn right and keep choosing every FOURTH house number. If it is an apartment building with up to four floors, interviewer chooses up to two apartments in which they try to do interviews. If it is an apartment building with five or more floors, they are allowed to choose three apartments. In case of villages the Step is reduced to three. Step 6 Respondent Selection Household s member with same probability random selection from the list of household member aged 18 and more years. 9
SAMPLING PROCEDURE CATI SURVEY Step 1 Stratification Population data is used to make initial strata. When national representative samples are being planned, two variables are always used for creating strata and estimating its size and proportion in sample stratum and urban/rural. Stratum and urban/rural variables are used at the same time for population and, furthermore, sample stratification, so samples are stratified in two dimensions. Step 2 Household Selection On the basis of the alocation created in accordance with population data, the selection of households is done by a random selection of households from the phone database by the data collection software. Step 3 Respondent Selection Within households, the respondent is selected using the corresponding quotas. More specifically, for each strata surveyed, in creating allocation, the quotas in terms of gender and age categories are created in the way it reflects gender and age structure of the population of a given strata. 10
FIELDWORK PROCEDURES - F2F SURVEY ALBANIA, BOSNIA AND HERZEGOVINA, KOSOVO, SERBIA AND MONTENEGRO SELECTION AND TRAINING OF THE INTERVIEWERS Interviewers and supervisors are recruited from the most experienced workers and interviewers from local agencies. Criteria for selection of the interviewers and supervisor are qualifications, communication skills and experience in similar fieldwork and familiarity with the area in which the survey is carried out. The training for the supervisors and regional coordinators is held in headquarters office by project manager and field force manager. Also, all regional coordinators hold the same training for interviewers from their specific regions. The training includes: Detailed explanation of the objectives of the survey Detailed instructions of respondent selection and fieldwork procedure (please see below the explanation of the respondent selection and fieldwork procedure for each of the target groups) Interviewer deportment and respondent confidentiality Quality control by interviewers supervisors and the project management team Detailed explanation of the questionnaire, question by question, including routing and filtering, and a comprehensive discussion of directive and non-directive probing Interviewing method with special focus on sensitive questions and possible difficulties during the interview, including instructions related to guaranteeing confidentiality of responses. The use of other survey materials (for example, show cards) Discussion of any problems or respondent queries that may arise Practice interviews, each interviewer with other interviewers, each interviewer role-playing as both interviewer and respondent Logistics of the survey The training includes formal classroom training and mock interviews, as well as audio-visual training tools. Interviewers are provided with detailed written instructions of respondent selection and fieldwork procedure. After the training each interviewer receives fieldwork pack. Fieldwork pack will be consisted of: CAPI devices Interviewers manual on local language Letter of introduction Identification cards 11
Regional coordinators responsibilities: to prepare strategy for research implementation, particularly bearing in mind specificity of the region involved; to brief each interviewer about the results of their work, and go through each problem; to give additional instructions to the interviewers during the research implementation; in case of any difficulties, to contact fieldwork manager; in case that no solution for the current problems in research implementation can be found, to contact central office and act according to given information; to execute control with selected respondents Interviewers' responsibilities: to interview respondents designed for the sample; to introduce him/her to respondent and ask permission to carry out the interview; to interview respondents within the specified time limit; in case that they cannot find a solution for current problems in research implementation, to contact their regional coordinators and act according to given information; to submit report on daily work to the regional coordinator at the end of each day of the fieldwork; to regard all data gathered in interviewing of households as business secret. DATA COLLECTION Method of data collection will be face to face interviews in respondent's home. Instead of using printed questionnaire, we suggest to use CAPI (Computer-assisted personal interviewing) method for this survey. Since the entire questionnaire will be automatically recorded in database, data entry will not be performed. There are several advantages of this method of data collection: There is no data entry process costs are lower Logic control is automatically performed - data can be checked immediately because the program will perform some internal validity checks Allows creating skip patterns in the survey software and item grid rotations. This way routing errors, that is, errors in the sequence of questions, skipping and branching, will not occur Daily upload of finalized questionnaire allows more accurate control of the sample Provides greater interviewer control and offers protection against unwanted interviewer behaviour 12
For data collecting process Ipsos uses SM-S software (developed by Ipsos Strategic Marketing programmers). This program is used for all types of quantitative surveys that ISM conducts. Using SM-S software allows automatically generation of data file in SPSS format. Beside this, it has the advantage as data entry and data collection software because it enables identification of all (user defined) inconsistencies, so 100% logic control is performed during data collection process. In this way additional check of fieldwork quality is performed. Based on the questionnaires on local language, Ipsos Strategic Marketing programmers will prepare data collection software. Each interviewer team synchronizes database each day, that is, uploads questionnaires competed during that day. FIELDWORK QUALITY CONTROL PROCEDURES Methods for ensuring quality control in the field In order to ensure high quality of fieldwork the following procedures are applied: Every interviewer keeps interviewer's diary which contains all basic information on the concrete interview realization Fieldwork control by back checking in direct contact with respondent Fieldwork control will be realized on 20% of the sample. Fieldwork control verifies the following aspects of interviewers' work: Fact that the interview has actually taken place Approximate duration of the interview Proper administration of sections of the questionnaire Interviewers' general adherence to professional standards 13
FIELDWORK PROCEDURES - TELEPHONE (CATI) SURVEYS MACEDONIA DATA COLLECTION SOFTWARE PREPARATION After finalization of questionnaire in local languages, data collection script is programmed. For data collection purposes SM-S, data entry and data collection program developed by our agency is used. This software automatically generates SPSS database format, but has the advantage as program because it enables identification of all (user defined) inconsistencies, so that 100% logic control is performed during data collection process. Once the questionnaire has been scripted but before fielding begins, project manager will test the survey script to ensure the exact question wording, skip patterns and order of questions/answer choices matches questionnaire. This includes a word-for-word check of the survey instrument against the survey document. SELECTION AND TRAINING OF INTERVIEWERS Upon data collection software scripting, and before fieldwork start, selection and training of interviewers is done. The training for the supervisors and regional coordinators is held in headquarters office by project manager and field force manager. The training includes: Detailed explanation of the survey objectives Sample design Interviewer deportment and respondent confidentiality Quality control by interviewers supervisors and the project management team Detailed explanation of the questionnaire Interviewing method with special focus on sensitive questions and possible difficulties during the interview, including instructions how to proceed in these situations. The use of other survey materials Discussion of any problems or respondent queries that may arise Practicing interviews, each interviewer with other interviewers, each interviewer role-playing as both interviewer and respondent Logistics of the survey Beside this `face to face` briefing, all interviewers will receive written instructions in electronic format which will be available on the screen of data collection program, so it can be accessed anytime interviewers experience certain difficulties in data collection process. Interviewers are recruited from the telephone interviewers from local. Criteria for selection of the interviewers are qualifications, communication skills and experience in similar surveys. 14
DATA COLLECTION After training of the interviewers, data collection is carried out through a telephone survey. Telephone (CATI) interviewers collected information over the phone, using a system that provides maximum control of the process and implementation of the survey sample (SM-S software platform developed by Ipsos Strategic Marketing). The program, which is used for telephone surveys, allows CATI manager to control the data collection process. During a telephone interviewing, after completing the questionnaire by interviewers, data is stored on the server and can be accessed at any time which allows control the number of completed questionnaires and the realization of the planned sample. FIELDWORK QUALITY CONTROL PROCEDURES Fieldwork control will be realized on 15% of the sample. Fieldwork control verifies the following aspects of interviewers' work: Fact that the interview has actually taken place Approximate duration of the interview Proper administration of sections of the questionnaire Interviewers' general adherence to professional standards 15
DATA PROCESSING AND ANALYSIS LOGICAL CONTROL OF DATA, CODING AND POST-STRATIFICATION After data collection is done, logical control of data, coding of open-ended questions and checking the consistency of answers are performed. Within SPSS statistical package basic data processing is conducted in order to prepare the data for further analysis. As noted above, using CAPI survey instead of PAPI reduces the number of errors that could occur during data collection process. Therefore, the process of data cleaning is more simplified and takes less time than it would in case PAPI survey was performed. At first, before starting any other procedure, sample realization is checked. This means that we are checking appropriate application of sampling plan, including selection of respondents according to preset criteria, number of respondents within each strata and sampling point. Since sampling allocation is carefully monitored during fieldwork, we can say that cases where there is no proper application of sampling plan are very rare. Next step in data cleaning process is logic control. As noted above, great part of logic control is automatically performed. But after data collection is done, some additional controls are performed. These controls include checking of data quality or checking percentage of missing data within each questionnaire. It is important to underline that those cases where there is a lot of missing data are quite rare, since fieldwork control is performed during fieldwork and within this control quality of individual questionnaires is checked. After fieldwork is done, same control is performed. The most frequent procedure is excluding questionnaires with more that 50% of missing data. In addition, average duration of questionnaire is monitored during and after fieldwork. After logic control is done, corrections of structure of the realized sample to ensure that it is identical with structure of population are made statistically, with the method of post stratification or weighting. Structure of sample is usually slightly distorted and it deviates from population structure, particularly in case of field surveys, because there are certain groups of people who refuse to participate in the survey more frequently than others. For this reason, it is necessary to make correction of observed deviations to ensure that the obtained sample fully represents the given population. Weighting procedures: Rim-weighting (ranking ratio) procedure is used. Coding procedure is done by ISM permanent coding team under the supervision of the project leader. 16
ANALYSIS AND REPORTING After logical control of data, coding and post-stratification, Ipsos Strategic Marketing performs data tabulation, data processing and report preparing. Data processing is based on specialized software Kal, developed by Ipsos Strategic Marketing for analytical purposes. In practice, Ipsos usually delivers technical, statistical report in Word format, with detailed tables of cross tabulation all questions with relevant socio-demographic variables, comparison between subsamples defined by sociodemographic variables categories and graphic view of the results. Analysis covers the whole area in which the research is conducted, as well as subsamples defined by sociodemographic variables. Data processing method, types of realized analysis and models of presentation will be defined by Ipsos in partnership with the Client. Example of Ipsos Strategic Marketing table 17
Ipsos Strategic Marketing Gavrila Principa 8 Belgrade, Serbia Tel: +381 11 3284 075 www.ipsos.com 18