Representativeness of Knowledge Bases with the Generalized Benford's Law. Arnaud Soulet, Arnaud Giacometti, Béatrice Markhoff and Fabian M.

Transcription

1 Representativeness of Knowledge Bases with the Generalized Benford's Law Arnaud Soulet, Arnaud Giacometti, Béatrice Markhoff and Fabian M. Suchanek University of Tours Telecom ParisTech

2 Reliability of queries on Knowledge Bases statistical query [Auer et al., 2007] How many cities are small (<1k inhabitants) in France/Yemen? ISWC Monterey, CA 2

3 Reliability of queries on Knowledge Bases statistical query [Auer et al., 2007] Does Yemen really not have any small cities? ISWC Monterey, CA 3

4 Reliability of queries on Knowledge Bases crowdsourcing statistical query Voluntary bias [Callahan and Herring, 2011;Wagner et al., 2015] ISWC Monterey, CA 4

5 Reliability of queries on Knowledge Bases crowdsourcing statistical query Voluntary bias [Callahan and Herring, 2011;Wagner et al., 2015] We do not know the KB biases, but the statistics can give us a hint? ISWC Monterey, CA 5

6 Missing facts Yemeni city: Population: Sanaa Aden Taiz 1,937,451 missing 760, ,222 Several methods for estimating the completeness for facts [Darari et al., 2016; Galarraga et al., 2017; Lajus and Suchanek, 2018; Razniewski et al., 2015; Razniewski et al., 2016] ISWC Monterey, CA 6

7 Missing facts Missing entities + missing facts Yemeni city: Population: Sanaa Aden Taiz Haid almissing Jazil 1,937, ,923 missing 615,222 missing few Missing facts due to missing entities are ignored! ISWC Monterey, CA 7

8 Big cities Big cities Completeness = #present facts / (#present facts + #missing facts) Assuming that K is an ideal KB (= correct + complete): K K 1 K K 2 Small cities Small cities What is the best KB between K 1 and K 2 for statistical queries? ISWC Monterey, CA 8

9 Big cities Big cities Completeness Representativeness Assuming that K is an ideal KB (= correct + complete): K K 1 K K 2 Small cities Small cities More complete, less representative Less complete, more representative Representativeness is more important than completeness for statistics! ISWC Monterey, CA 9

10 Representativeness of Knowledge Bases A KB K is representative of K iff the distribution remains the same for all uniform-sampling invariant measures. K <1k inhab. 1k inhab. K <1k inhab. 1k inhab. ISWC Monterey, CA 10

11 Representativeness of Knowledge Bases A KB K is representative of K iff the distribution remains the same for all uniform-sampling invariant measures. K <1k inhab. 1k inhab. K The ideal knowledge base K is unknown! <1k inhab. 1k inhab. Challenge: How to estimate the representativeness? ISWC Monterey, CA 11

12 Example: population of capitals Abidjan Bangkok Conakry Kingston Mogadishu Santiago Abuja Beijing Dakar Kinshasa Montevideo Seoul Accra Belgrade Damascus Kuala Lumpur Nairobi Sofia Addis Ababa Berlin Dhaka Lilongwe Niamey Taipei Algiers Bogota Doha Lima Ouagadougou Tashkent Amman Brasilia Erbil London Paris Tbilisi Ankara Brazzaville Freetown Luanda Phnom Penh Tegucigalpa Antananarivo Bucharest Havana Lusaka Prague Tokyo Ashgabat Budapest Islamabad Madrid Pyongyang Tripoli Bahawalpur Buenos Aires Jakarta Managua Quito Tunis Baku Cairo Kabul Maputo Riyadh Ulaanbaatar Bamako Caracas Khartoum Mexico City Sana'a Vienna ISWC Monterey, CA 12

13 Example: population of capitals ISWC Monterey, CA 13

14 Example: population of capitals What is the distribution of the first significant digit of capital inhabitants? ISWC Monterey, CA 14

15 Benford s law Population of cities Benford s law ISWC Monterey, CA 15

16 Benford s law 0.30 Population of cities 0.30 Discharge of rivers 0.30 Length of rivers ISWC Monterey, CA 16

17 Benford s law 0.30 Population of cities 0.30 Discharge of rivers 0.30 Length of rivers P first digit X = d = log d [Newcomb, 1881;Benford, 1938] ISWC Monterey, CA 17

18 The Generalized Benford s Law Population of cities Discharge of rivers Length of rivers α 0 α 0 α 0 P first digit X = d = 1 + d α d α 10 α 1 [Hürlimann, 2014] ISWC Monterey, CA 18

19 The Generalized Benford s Law Population of cities Discharge of rivers Length of rivers α 0 α 0 α 0 Actors per movie α= Persons per birth place α= Out-degree of wikipedia pages α= ISWC Monterey, CA 19

20 Key idea of our method representativeness = compliance with the Generalized Benford s Law = #present_facts #present_facts+#missing_facts_for_compliance Population in France Population in Yemen Representativeness = 97% Representativeness = 79% DBpedia ISWC Monterey, CA 20

21 Our method in supervised context facts of r on K distribution of the fsd facts of r on K Benford s law Using the known distribution of the first significant digit ISWC Monterey, CA 21

22 Our method in supervised context facts of r on K distribution of the fsd Population in Yemen facts of r on K Benford s law present facts 101 missing facts Representativeness: Computing the minimum number of facts for retrieving Benford s law = = 79% ISWC Monterey, CA 22

23 Our method in unsupervised context facts of r on K distribution of the fsd facts of r on K ideal distribution is unknown! GBL with α=0.12 Learning the parameter α of the Generalized Benford s Law ISWC Monterey, CA 23

24 Our method in unsupervised context facts of r on K facts of r on K ideal distribution is unknown! distribution of the fsd GBL with α= present facts 78 missing facts Representativeness: = = 82% Computing the minimum number of facts for retrieving Benford s law Population in Yemen ISWC Monterey, CA 24

25 Experimental study Evaluation protocol 1. Take a correct and complete relation as gold standard 2. Degrade the completeness by discarding facts 3. Approximate the representativeness Gold standard: population in French cities according to govt statistics Degradation: Most-populated: remove the least populated cities Least-populated: remove the most populated cities Random: remove cities randomly ISWC Monterey, CA 25

26 Population of French cities Representativeness is an upper bound of completeness Most/least-populated degradation: tight bound if number of cities > 22k Random degradation: the representativeness is high Representativeness approximates well the bias ISWC Monterey, CA 26

27 Population of French cities Learning the parameter α does not perturbate the approximation ISWC Monterey, CA 27

28 Auditing DBpedia (France) 1,487 relations (out of 2,920) have a distribution statistically compliant with the GBL Representativeness: 72% Present facts Missing facts ISWC Monterey, CA 28

29 Conclusion The representativeness is more important than the completeness for achieving statistics. First use of Benford s law for approximating the proportion of missing data The approximate representativeness based on the GBL is an upper bound of the true representativeness and the true completeness. Future work: How to correct sparql queries with representativeness information? How to scale up the approach to audit the LOD? ISWC Monterey, CA 29