Novelty, Information and Surprise

Transcription

1 Günther Palm Novelty, Information and Surprise 4i Springer

2 Contents Introduction References xiii xxii Part I Surprise and Information of Descriptions 1 Prerequisites from Logic and Probability Theory Logic and Probability of Propositions Mappings, Functions and Random Variables Measurability, Random Variables, and Expectation Value Technical Comments 10 References 10 2 Improbability and Novelty of Descriptions Introductory Examples Definition and Properties Descriptions Properties of Descriptions Information and Surprise of Descriptions Information and Surprise of a Random Variable Technical Comments Exercises 32 References 34 3 Conditional and Subjective Novelty and Information Introductory Examples Subjective Novelty Conditional Novelty Information Theory for Random Variables Technical Comments Exercises 45 References 46 v

3 vi Contents Part II Coding and Information Transmission 4 On Guessing and Coding Introductory Examples Guessing Strategies Codes and Their Relation to Guessing Strategies Kraft's Theorem Huffman Codes Relation Between Codewordlength and Information Technical Comments Exercises 60 References 62 5 Information Transmission Introductory Examples Transition Probability Transmission of Information Across Simple Channels Technical Comments Exercises 72 Reference 74 Part III Information Rate and Channel Capacity 6 Stationary Processes and Their Information Rate Introductory Examples Definition and Properties of Stochastic Processes The Weak Law of Large Numbers Information Rate of Stationary Processes Transinformation Rate Asymptotic Equipartition Property Technical Comments Exercises 87 References 88 7 Channel Capacity Information Channels Memory and Anticipation Channel Capacity Technical Comments Exercises 94 References 95 8 How to Transmit Information Reliably with Unreliable Elements (Shannon's Theorem) The Problem of Adapting a Source to a Channel Shannon's Theorem 98

4 Contents vii 8.3 Technical Comments Exercises 101 References 101 Part IV Repertoires and Covers 9 Repertoires and Descriptions Introductory Examples Repertoires and Their Relation to Descriptions Tight Repertoires Narrow and Shallow Covers Technical Comments Exercises 120 References Novelty, Information and Surprise of Repertoires Introductory Examples Definitions and Properties Finding Descriptions with Minimal Information Technical Comments Exercises 138 References Conditioning, Mutual Information, and Information Gain Introductory Examples Conditional Information and Mutual Information Information Gain, Novelty Gain, and Surprise Loss Conditional Information of Continuous Random Variables Technical Comments Applications in Pattern Recognition, Machine Learning, and Life-Science Exercises 156 References 157 Part V Information, Novelty and Surprise in Science 12 Information, Novelty, and Surprise in Brain Theory Understanding Brains in Terms of Processing and Transmission of Information Neural Repertoires Experimental Repertoires in Neuroscience The Burst Repertoire The Pause Repertoire The Coincidence Repertoire The Depolarization Repertoire Neural Population Repertoires: Semantics and Syntax 173

5 viii Contents 12.5 Conclusion Technical Comments Coincidence Coincidental Patterns Spatio-Temporal Patterns 179 References Surprise from Repetitions and Combination of Surprises Combination of Surprises Surprise of Repetitions Technical Comments 194 References Entropy in Physics Classical Entropy Modern Entropies and the Second Law The Second Law in Terms of Information Gain Technical Comments 204 References 204 Part VI Generalized Information Theory 15 Order- and Lattice-Structures Definitions and Properties The Lattice 2 of Descriptions Technical Comments 214 Reference Three Orderings on Repertoires Definition and Basic Properties Equivalence Relations Defined by the Orderings The Joins and Meets for the Orderings The Orderings on Templates and Fiat Covers Technical Comments Exercises 228 References Information Theory on Lattices of Covers The Lattice e of Covers The Lattice 3rf of Finite Fiat Covers The Lattice 9i of (Clean) Repertoires The Lattice T of Templates The Lattice 13 of Partitions Technical Comments Exercises 235 References 235

6 Contents ix Appendices A Fuzzy Repertoires and Descriptions 237 A.1 Basic Definitions 238 A.2 Definition and Properties of Fuzzy Repertoires 240 Reference 242 Glossary 243 Index 245