Parallel Computing in Optimization

Transcription

1 Parallel Computing in Optimization

2 Applied Optimization Volume 7 Series Editors: Panos M. Pardalos University of Florida, U.S.A. Donald Hearn University of Florida, U.SA. The titles published in this series are listed at the end of this volume.

3 Parallel Computing in Optimization Edited by Athanasios Migdalas LinJcijping Institute of Technology Panos M. Pardalos University of Florida and Sverre Stor0y University of Bergen KLUWER ACADEMIC PUBLISHERS DORDRECHT I BOSTON I LONDON

4 A C.l.P. Catalogue record for this book is available from the Library of Congress. ISBN-13: : / e-isbn-13: Published by Kluwer Academic Publishers, P.O. Box 17,3300 AA Dordrecht, The Netherlands. Kluwer Academic Publishers incorporates the publishing programmes of D. Reidel, Martinus Nijhoff, Dr W. Junk and MTP Press. Sold and distributed in the U.S.A. and Canada by Kluwer Academic Publishers, 101 Philip Drive, Norwell, MA 02061, U.S.A. In all other countries, sold and distributed by Kluwer Academic Publishers, P.O. Box 322, 3300 AH Dordrecht, The Netherlands. Printed on acid-free paper All Rights Reserved 1997 Kluwer Academic Publishers Softcover reprint of the hardcover 1st edition 1997 No part of the material protected by this copyright notice may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording or by any information storage and retrieval system, without written permission from the copyright owner.

5 "there is no rest for the messenger till the message is delivered" The Rescue: A Romance in the Shallows (by Joseph Conrad)

6 CONTENTS PREFACE 1 MODELS FOR PARALLEL ALGORITHM DESIGN: AN INTRODUCTION Afonso Ferreira and Michel Morvan 1 Introduction 2 Shared memory model: PRAM 3 Distributed memory models: DMM 4 The coarse grained multicomputer model: CGM 5 Summary 6 EXERCISES 2 PARALLEL ALGORITHMS AND COMPLEXITY Martin Furer 1 Introduction 2 Models of Parallel Computers 3 Limits of Parallelism 4 Classification of some Important Graph Problems 5 Basic Techniques 6 Parallel Algorithms Toolbox 7 Approximating the Minimum Degree Spanning Tree Problem 8 EXERCISES 3 A PROGRAMMER'S VIEW OF PARALLEL COMPUTERS TorS revik 1 Introduction xvii Vll

7 Vlll PARALLEL COMPUTING IN OPTIMIZATION 2 The Memory Hierarchy 59 3 Communication Network 65 4 Future trends 68 5 EXERCISES 69 4 SCALABLE PARALLEL ALGORITHMS FOR SPARSE LINEAR SYSTEMS Anshul Gupta, George Karypis, and Vipin Kumar 73 1 Introduction 73 2 Parallel Direct Cholesky Factorization 77 3 Multilevel Graph Partitioning 84 4 EXERCISES 91 5 OBJECT ORIENTED MATHEMATICAL MODELLING AND COMPILATION TO PARALLEL CODE Niclas Andersson and Peter Fritzson 99 1 Introduction Object Math Background to Parallel Code Generation Definitions Towards a Parallelising Compiler Equation System Level Equation Level Clustered Task Level Explicit Parallelism Summary EXERCISES PARALLEL ALGORITHMS FOR NETWORK PROBLEMS Olof Damberg, Athanasios Migdalas and Sverre Stor y Introduction Parallel processing paradigms The shortest path problem Linear problems over bipartite graphs 190

8 Contents IX 5 Convex problems over singlecommodity networks Convex problems over multicommodity networks EXERCISES PARALLEL BRANCH AND BOUND - PRINCIPLES AND PERSONAL EXPERIENCES Jens Clausen Introduction Sequential B&B Parallel B&B Personal Experiences with GPP and QAP Ideas and Pitfalls for Parallel B&B users EXERCISES PARALLELIZED HEURISTICS FOR COMBINATORIAL SEARCH Kristina Holmqvist, Athanasios Migdalas, Panos M. Pardalos Heuristics for Combinatorial Search Local Search Simulated Annealing Tabu Search Genetic Algorithms Greedy Randomized Adaptive Search Procedures Conclusions EXERCISES PARALLEL COST APPROXIMATION ALGORITHMS FOR DIFFERENTIABLE OPTIMIZATION Michael Patriksson Introduction Sequential Cost Approximation Algorithms Synchronized Parallel Cost Approximation Algorithms Partially Asynchronous Parallel Cost Approximation Algorithms328 5 Concluding Remarks EXERCISES 335

9 x PARALLEL COMPUTING IN OPTIMIZATION 10 PARALLEL COMPUTATION OF VARIATIONAL INEQUALITIES AND PROJECTED DYNAMICAL SYSTEMS WITH APPLICATIONS Anna Nagurney Introduction The Variational Inequality Problem Projected Dynamical Systems Variational Inequality Applications Projected Dynamical Systems Applications Summary and Conclusions EXERCISES PARALLEL ALGORITHMS FOR LARGE-SCALE STOCHASTIC PROGRAMMING Hercules Vladimirou and Stavros A. Zenios Introduction Stochastic Programs with Recourse Algorithmic Approaches Algorithmic Comparisons Conclusions EXERCISES PARALLEL CONTINUOUS NON-CONVEX OPTIMIZATION Kristina Holmqvist, Athanasios Migdalas and Panos M. Pardalos Introduction Local Search Heuristics Deterministic and Stochastic Refinements of Local Search Summary of General Principles for Local Search Parallelization Exact Methods: Deterministic Approaches EXERCISES 516

10 Contents Xl 13 DETERMINISTIC AND STOCHASTIC LOGARITHMIC BARRIER FUNCTION METHODS FOR NEURAL NETWORK TRAINING Theodore B. Trafalis and Tarek A. Tutunji 1 Introduction 2 Newton-type and Logarithmic Barrier Methods 3 Application to Neural Network Training 4 Ill-Conditioning 5 Computational Results 6 Conclusions and Future Research 7 EXERCISES INDEX

11 CONTRIBUTORS Niclas Andersson PELAB Computer Science Department Linkoping Institute of Technology S-581 Linkoping SWEDEN nicangida.liu.se Jens Clausen DIKU Department of Computer Science University of Copenhagen Universitetsparken 1 DK-2100 Copenhagen 0 DENMARK clausengdiku.dk Olof Damberg Division of Optimization Department of Mathematics Linkoping Institute of Technology S Linkoping SWEDEN oldamgmath.liu.se Afonso Ferreira CNRS LIP-ENS Lyon Lyon Cedex 07 FRANCE ferreiraglip.ens-lyon.fr Peter Fritzson PELAB Computer Science Department Linkoping Institute of Technology S-581 Linkoping SWEDEN petfrgida.liu.se Martin Fiirer Department of Computer Science and Engineering 220 Pond Laboratory Pennsylvania State University University Park Pennsylvania USA furergcse.psu.edu Anshul Gupta Department of Computer Science University of Minnesota Minneapolis, MN USA aguptagcs.umn.edu Kristina Holmqvist Division of Optimization Department of Mathematics Linkoping Institute of Technology S Linkoping SWEDEN krholgmath.liu.se

12 xiv CONTRIBUTORS George Karypis Department of Computer Science University of Minnesota Minneapolis, MN USA karypisgcs.umn.edu Vipin Kumar Department of Computer Science University of Minnesota Minneapolis, MN USA kumargcs.umn.edu Athanasios Migdalas Division of Optimization Department of Mathematics Linkoping Institute of Technology Linkoping SWEDEN samiggmath.liu.se Michel Morvan LITP/IBP Universite Paris Paris Cedex 07 FRANCE morvanglitp.ibp.fr Anna Nagurney School of Management Department of General Business and Finance University of Massachusetts Amhers, Massachusetts USA annagumass. edu Panos M. Pardalos Department of Industrial and Systems Engineering University of Florida Gainesville, FL32611 USA pardalosgmath.ufl.edu Michael Patriksson Division of Optimization Department of Mathematics Linkoping Institute of Technology S Linkoping SWEDEN mipatgmath.liu.se Sverre StorfllY Department of Informatics University of Bergen Thorml/lhlensgate 55 N-5020 Bergen NORWAY Sverre.StoroyGii.uib.no Tor Sfllrevik Parallab Department of Informatics University of Bergen, Thorml/lhlensgate 55 N-5020 Bergen NORWAY Tor.SorevikGii.uib.no Theodore Trafalis School of Industrial Engineering 202 West Boyd, Room 124 Oklahoma USA trafalisgmailhost.ecn.uoknor.edu Tarek A. Tutunji Halliburton Energy Services 1100 Everman Road Forth Worth, Texas USA

13 Contributors xv Hercules Vladimirou Department of Public and Business Administration University of Cyprus 75 Kallipoleos Street P.O. Box Nicosia CYPRUS herculesoatlas.pba.ucy.ac.cy Stavros Zenios Department of Public and Business Administration University of Cyprus 75 Kallipoleos Street P.O. Box Nicosia CYPRUS zeniossojupiter.cca.ucy.cy

14 PREFACE During the last three decades, breakthroughs in computer technology have made a tremendous impact on optimization. In particular, parallel computing has made it possible to solve larger and computationally more difficult problems. This volume contains mainly lecture notes from a Nordic Summer School held at the Linkoping Institute of Technology, Sweden in August In order to make the book more complete, a few authors were invited to contribute chapters that were not part of the course on this first occasion. The purpose of this Nordic course in advanced studies was three-fold. One goal was to introduce the students to the new achievements in a new and very active field, bring them close to world leading researchers, and strengthen their competence in an area with internationally explosive rate of growth. A second goal was to strengthen the bonds between students from different Nordic countries, and to encourage collaboration and joint research ventures over the borders. In this respect, the course built further on the achievements of the "Nordic Network in Mathematical Programming", which has been running during the last three years with the support ofthe Nordic Council for Advanced Studies (NorFA). The final goal was to produce literature on the particular subject, which would be available to both the participating students and to the students of the "next generation". The school was funded by NorFA, while activities related to the preparation of the application to NorFA, and the co-ordination of the lecturers and the editors were supported in part by the Linkoping Center for Transfer of Information Technology (CENIIT). The National Supercomputer Center (NSC) in Linkoping freely supplied its computer resources and laboratories to the participating lecturers and students. More than 30 students and seniors from the Nordic countries participated in the course. xvii

15 xviii PARALLEL COMPUTING IN OPTIMIZATION An international group of invited speakers presented lectures dealing with novel programming and algorithmic aspects of parallel computing as well as technical advances in parallel optimization. In addition, the Summer School provided a concentrated forum to discuss progress on enhancing the number of optimization applications that take advantage of emerging parallel computing architectures. The volume contains 13 chapters. These can be divided into three groups. The first two chapters, contributed by A. Ferreira & M. Morvan and M. Fiirer respectively, discuss theoretical models for parallel algorithm design and their complexity. T. SI/Jrevik, on the other hand, in chapter 3 gives the perspective of the programmer practicing parallel algorithm development on real world platforms. Solving systems of linear equations efficiently is of great importance on their own because they arise in many scientific and engineering applications but also because algorithms for solving many optimization problems need to call system solvers as subroutines. The fourth chapter by A. Gupta, G. Karypis and V. Kumar is devoted to scalable parallel algorithms for sparse linear systems, while the fifth chapter by N. Andersson and P. Fritzson discusses computer based mathematical modeling aspects of applications such as bearing and automatic generation of parallel code for the solution of the resulting system of equations. Chapters six to thirteen are dedicated to optimization problems and methods. More specifically, in Chapter six O. Damberg, A. Migdalas and S. Storl/Jy give an introductory survey of parallel algorithms for network problems with emphasis on linear bipartite assignment problems and a thorough discussion on the implementation of a parallel solver for the traffic assignment problem. In Chapter seven, J. Clausen reviews the principles of sequential Branch-and-Bound (B& B) and sketches the main trends in parallel B& B and the problems experienced with the latter approach. In chapter eight, K. Holmqvist, A. Migdalas and P. M. Pardalos provide an introductory survey in the parallelization of modern heuristic methods for combinatorial optimization problems. In Chapter nine, M. Patriksson presents a unified analysis of decomposition algorithms for differentiable optimization problems. Chapter ten, written by A. Nagurney, presents parallel algorithms for the computation of solutions to finite dimensional variational inequality problems and projected dynamical systems. H. Vladimirou and S. A. Zenios present in the eleventh chapter the state-ofthe-art in parallel algorithms for stochastic programming. The twelfth chapter, authored by K. Holmqvist, A. Migdalas and P. M. Pardalos, surveys exact and heuristic algorithms of deterministic and stochastic type for global optimization

16 Preface XIX problems. Finally in Chapter thirteen, T. Trafalis and T. Tutunji present deterministic and stochastic logarithmic barrier function algorithms for neural network training. Although the entire spectrum of Parallel Computing in Optimization cannot be covered in a volume, it is hoped that these chapters communicate the richness and diversity in this important research domain with an ever increasing practical application area. We would like to take this opportunity to thank the authors, the anonymous referees, CENIIT for funding the coordination process, and NorFA for funding the Summer School and helping us produce this book with state-of-the-art chapters. A. Migdalas, P. M. Pardalos and S. Stor~y August, 1996