ARTIFICIAL NEURAL NETWORKS Learning Algorithms, Performance Evaluation, and Applications

Transcription

1 ARTIFICIAL NEURAL NETWORKS Learning Algorithms, Performance Evaluation, and Applications

2 THE KLUWER INTERNATIONAL SERIES IN ENGINEERING AND COMPUTER SCIENCE

3 ARTIFICIAL NEURAL NETWORKS Learning Algorithms, Performance Evaluation, and Applications by N. B. Karayiannis University oj Houston A. N. Venetsanopoulos University of Toronto ~. " SPRINGER SCIENCE+BUSINESS MEDIA, LLC

4 Library of Congress Cataloging-In-Publlcation Data Karayiannis, N. B. (Nicolaos B.), Artificial neural networks : learning algorithms, performance evaluation, and applications / N. B. Karayiannis, A. N. Venetsanopoulos. p. cm. -- (The Kluwer international series in engineering and computer science ; SECS 0209) lnc1udes bibliographical references and index. ISBN ISBN (ebook) DOI / Neural networks (Computer science) 2. Machine learning. 3. Algorithms. 1. Venetsanopoulos, A. N. (Anastasios N.), Il. Title. III. Series. QA K dc CIP Copyright 1993 by Springer Science+Business Media New York Originally published by Kluwer Academic Publishers in 1993 Softcover reprint of the hardcover 1 st edition 1993 AII rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photo-copying, record ing, or otherwise, without the prior written permission of the publisher, Springer Science+ Business Media, LLC. Printed on acid-free paper.

5 To Vivian & Vaso

6 EL,ap 'f/8vvato f/wutov TWV op,avwv KfAWU(UV 7j 7r poalu(}avopwov a7rouadv TO avtov EP,OV OV8EV av E8ft ovu TOl<; apxltektou[,v V7r'f/PfTWV OVU Toi<; &U7rOraL<; 80VAWV. APLUTOTEA'f/<;, IIoALTLKa A4, 1253b al. If a tool could do its job after obeying a command or its own feeling neither the a.rchitects (experts) would require assistants nor the masters slaves. Aristotle, Politics A4, 1253b al

7 Acknowledgments Table of Contents 1 Introduction. 1 Xlll 1.1 Overview Book Organization 4 2 Neural Network Architectures and Learning Schemes Introduction Feed-forward Neural Networks Feed-back Neural Networks Self-organizing Neural Networks Discussion. 75 Appendix A Appendix A Appendix A Appendix A Appendix A ELEANNE: Efficient LEarning Algorithms for Neural NEtworks Introduction Recursive Least-squares Algorithms 3.3 Efficient Learning Algorithms for Single-layered Neural Networks. 3.4 Efficient Learning Algorithms for Multi-layered Neural Networks 3.5 Computational Considerations 3.6 Experimental Results

8 x 3.7 Discussion. Appendix A3.1 Appendix A3.2 Appendix A3.3 Appendix A3.4 4 Fast Learning Algorithms for Neural Networks 4.1 Introduction A Generalized Training Criterion 4.3 Fast Learning Algorithms for Single-layered Neural Networks. 4.4 Fast Learning Algorithms for Multi-layered Neural Networks 4.5 Experimental Results 4.6 Discussion. Appendix A4.1 Appendix A4.2 Appendix A4.3 Appendix A4.4 5 ALADIN: Algorithms for Learning and Architecture DetermINation 5.1 Introduction Training Criteria 5.3 Neural Networks with one Hidden Layer 5.4 Neural Networks with Multiple Hidden Layers 5.5 Experimental Results 5.6 Discussion. Appendix A5.1 Appendix A

9 Xl 6 Performance Evaluation of Single-layered Neural Networks Introduction Optimal Least-squares Training of Single-layered Neural Networks Capacity Considerations Output Nonlinearities and Network Performance Discussion Appendix A Appendix A Appendix A Appendix A High-order Neural Networks and Networks with Composite Key Patterns Introduction High-order Neural Networks Neural Networks with Composite Key Patterns Capacity Considerations Discussion Applications of Neural Networks: A Case Study Introduction General Methodology for the Development of Neural Network Systems Application of Neural Networks in Environmental Protection Discussion

10 xu 9 Applications of Neural Networks: A Review Introduction Optimization Problems Image Compression Recognition of Handwritten Signatures, Characters, and Digits Text to Speech Conversion Classification Applications Medical Diagnosis Prediction of Secondary Structures of Proteins Weather Forecasting Financial Predictions Other Applications FUture Trends and Directions. 371 References Subject Index. 413 Author Index. 435

11 Acknowledgments We are deeply indebted to the many people who have made the completion of this book possible. Dr. Nicholas DeClaris read an early version of the manuscript and motivated several improvements. Drs. Ben Jansen, Periklis Ktonas, Haluk Ogmen, and Erkki Oja carefully read portions of the manuscript and provided valuable suggestions. Particular thanks go to Buddy Marshall, who had the patience to read the entire manuscript. His interesting feed-back improved the manuscript in many ways. Demetri Androutsos, Gokalp Bayramoglu, Yezdi Kharas, Akbar H. Marand, Arun Padmanabhan, Pin-I Pai, Panagiotis Pandelis, and Costa S. Partsinevelos helped us by proofreading portions of the manuscript. Vivian Karayiannis, Arun Padmanabhan, Pin-I Pai, and Panagiotis Pandelis helped with the preparation of the final version of the manuscript. We would also like to thank Dr. Stuart Long, chairman of the Department of Electrical Engineering of the University of Houston, for providing the facilities used in the preparation of this book. Finally, special thanks go to Vivian Karayiannis for spending the summer and several weekends at home. Her love, patience and encouragement made this book possible. Nicolaos B. Karayiannis Houston, Texas Anastasios N. Venetsanopoulos Toronto, Canada

12 ARTIFICIAL NEURAL NETWORKS Learning Algorithms, Performance Evaluation, and Applications