Filter Design With Time Domain Mask Constraints: Theory and Applications

Filter Design With Time Domain Mask Constraints: Theory and Applications

Applied Optimization Volume 56 Series Editors: Panos M. Pardalos University of Florida, U.S.A. Donald Hearn University of Florida, U.S.A. The titles published in this series are listed at the end of this volume.

Filter Design With Time Domain Mask Constraints: Theory and Applications by Ba-Ngu Vo The University of Melbourne, Australia Antonio Cantoni The University of Western Australia, Australia and KokLay Teo The Hong Kong Polytechnic University, Hong Kong SPRINGER-SCIENCE+BUSINESS MEDIA, B.V.

A C.I.P. Catalogue record for this book is available from the Library of Congress. ISBN 978-1-4419-4858-8 ISBN 978-1-4757-3409-6 (ebook) DOI 10.1007/978-1-4757-3409-6 Printed on acid-free paper All AII Rights Reserved 2001 Springer Science+Business Media Dordrecht Originally published by Kluwer Academic Publishers in 2001 Softcover reprint of the hardcover 1 st edition 2001 No part of the material protected by this copyright notice may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording or by any information storage and retrieval system, without written permission from the copyright owner

v CONTENTS List of Figures... XI List of Tables................... xv Preface....................... xvii CHAPTER 1 INTRODUCTION................. 1 1.1 Applications... 2 Pulse compression...................................... 2 TV waveform equalization... 5 Data channel equalization................................ 6 Design to meet standards................................. 9 Deconvolution... 12 1.2 Envelope Constrained Filtering... 13 1.3 Historical Notes... 18 1.4 Road Map... 19 CHAPTER 2 FILTERING WITH CONVEX RESPONSE CONSTRAINTS 21 2.1 Analog Filtering with Convexly Constrained Responses... 22 The cost functional... 24 The feasible region... 27 2.2 Filter Design with Envelope Constraints... 28 2.3 Convex Programming... 32 2.4 Finite Dimensional Analog CCR Filters... 34 2.5 Discrete-time CCR Filter Design... 37 2.6 Continuous-time CCR Filtering via DSP Approach... 40 Problem formulation for hybrid filter... 41 Feasible region... 42

vi 2.7 Finite Dimensional Hybrid Filter for CCR Filtering... 50 FIR digital processor.... 51 2.8 Appendix... 53 CHAPTER 3 ANALYSIS AND PROBLEM CHARACTERIZATION..... 61 3.1 Duality of Quadratic Program... 62 3.2 Dual Problems for EC Filtering... 67 Unconstrained dual problem... 71 3.3 Dual Problem for Finite Dimensional Filter... 76 3.4 Semi-Infinite Programming... 79 Optimality conditions... 81 Transformation technique via dual parametrization... 84 Getting the primal solution... 87 3.5 Linearly Constrained Quadratic SIP and EC Filters... 87 Finite dimensional dual problem with mxn support points... 89 Finit~ dimensional dual problem with n support points... 90 Finite-dimensional EC filters... 91 3.6 Appendix... 94 CHAPTER 4 DISCRETE-TIME EC FILTERING ALGORITHMS..... 103 4.1 Discrete-time EC Filtering Problem... 104 4.2 QP via Active Set Strategy... 106 QP with linear inequality constraints... 106 QP with inequality constraints... 111 4.3 Iterative Algorithm via The Primal-Dual Method... 116 Non-smooth dual problem... \... 118 Steepest ascent with directional differentials... 119 4.4 Iterative Algorithm using Augmented Cost.... 126 Approximation results... 129

vii Update equations... 130 4.5 Tapped Delay Line FIR Filters............................. 136 4.6 Discrete-time Laguerre Networks... 144 4.7 Appendix A... 149 4.8 Appendix B............................................ 156 CHAPTER 5 NUMERICAL METHODS FOR CONTINUOUS TIME EC FIL TERING... 161 5.1 Continuous-time EC Filtering... 162 Analog filters........................................ 162 Hybrid filters... 163 5.2 Non-iterative Method... 164 5.3 Primal Dual Method...................................... 167 Discretization of dual problem for analog filters............. 167 Discretization of dual problem for hybrid filters..., 170 Finite filter structures.................................. 171 Steepest ascent algorithm............................... 172 5.4 Penalty Approach for Semi-Infinite Programming... 175 Approximations by conventional constrained problems....... 177 Approximations by unconstrained problems................ 179 Approximating convex problems... 181 Affine functional inequality constrained problems........... 182 SIP with quadratic cost and affine constraints... 184 Application to finite dimensional EC filters... 186 5.5 Laguerre Networks in Continuous-time EC Filtering... 188 Application to channel equalization... 190 5.6 Hybrid filter with FIR Digital Components... 194 Linear interpolator..., 195 Butterworth and Bessel post-filtering... 196

viii Chebyshev and elliptic post-filtering... 199 5.7 Appendix... 203 CHAPTER 6 ROBUST ENVELOPE CONSTRAINED FILTERING..... 219 6.1 Constraint Function... 220 6.2 Transformation to Smooth Problem... 224 Problem conversion... 224 6.3 Application to Analog ECUI Filtering Problem... 228 Finite dimensional filter for analog problem... 231 Example using Walsh functions... 234 Approximations for finite dimensional filter... 237 6.4 Discrete-time... 238 6.5 Hybrid ECUI Filtering... 239 Approximations for hybrid filters... 241 Finite dimensional hybrid filters... 242 6.6 Constraint Robustness... 247 Characterization of filter structure... 250 Finite dimensional filter... 252 Numerical example with Laguerre filter... 253 6.7 EC Filtering with Uncertain Implementation... 256 Examples with finite dimensional filters... 259 6.8 Appendix... 265 APPENDIX A MATHEMA TICAL BACKGROUND..... 275 A.l Topological Space... 275 A.2 Metric Spaces... 279 A.3 Vector Spaces... 282 A.4 Normed Spaces... 285 A.5 Inner Product Spaces... 286

IX A.6 Linear Operators... 289 A.7 Linear Functionals and Dual Spaces... 292 A.8 Measures and Integration... 295 APPENDIX B OPTIMIZATION THEORY... 301 B.l Projection Theorem... 301 B.2 Hahn-Banach Theorem... 302 B.3 Positive Cones and Convex Mappings... 303 B.4 Gateaux and Frechet Differentials........................... 305 B.5 Lagrange Multipliers... 310 References... 313 Index... 325

xi List of Figures Figure 1.1.1. Pulse Compression... 3 Figure 1.1.2. Pulse shape constraints for radar/sonar problem... 4 Figure 1.1.3. K-rating mask for equalization of TV channel.... 6 Figure 1.1.4. Model of a data channel..... 7 Figure 1.1.5. Mask for constraints at sampling instances... 8 Figure 1.1.6. Mask for handling timing jitter... 8 Figure 1.1.7. Impulse response of coaxial cable for various lengths... 10 Figure 1.1.8. DSX3 pulse template, coaxial cable response and filter output... 11 Figure 1.1.9. Pre- shaping of pulse... 11 Figure 1.1.10. ANSI T1.403 for T1-1.544 Mb/s... 12 Figure 1.2.1. Receiver model and output mask... 14 Figure 1.2.2. Magnitude response constraint... 15 Figure 1.2.3. Antenna receiver... 17 Figure 2.2.1. EC filtering with uncertain input.... 30 Figure 2.4.1. Parallel filter structure... 36 Figure 2.4.2. Transversal filter structure... 36 Figure 2.6.1. Configuration for digital processing of continuous-time signal... 41 Figure 3.2.1. Configuration of optimal analog EC filter... 74 Figure 3.2.2. Configuration of optimal hybrid EC filter... 76 Figure 4.3.1. Configuration for an adaptive EC filter... 117 Figure 4.4.1. Penalty allocator... 126 Figure 4.4.2. Flow chart for line search... 132 Figure 4.5.1. A tapped delay line FIR filter... 136 Figure 4.5.2. Optimum EC filter output for a 13-bit Barker-coded input...... 139 Figure 4.5.3. Augmented cost and noise gain of Barker coded example... 140 Figure 4.5.4. Optimum EC filter output for a rectangular input... 141 Figure 4.5.5. Augmented cost and noise gain of rectangular input example... 141 Figure 4.5.6. Optimum and sub-optimum EC filter and their outputs... 143 Figure 4.5.7. Augmented cost and noise gain of DSX-3 example... 143 Figure 4.6.1. Discrete-time Laguerre network... 144 Figure 4.6.2. Signals and output mask for Laguerre and FIR EC filter... 147 Figure 5.2.1. Bound on slopes... 165

XlI Figure 5.4.1. Lower bound... 179 Figure 5.5.1. B lock diagram of a Laguerre network... 189 Figure 5.5.2. Augmented cost function for coaxial cable example... 191 Figure 5.5.3. Magnitude responses of equalizer, unequalized and equalized cable...... 192 Figure 5.5.4. DSX3 pulse template, coaxial cable response and filter output... 192 Figure 5.5.5. Augmented cost function for fourth order circuit example... 193 Figure 5.5.6. DSX3 pulse template, filter input and output...... 193 Figure 5.5.7. Magnitude responses of equalizer, unequalized and equalized circuit..... 194 Figure 5.6.1. Rectangular pulse response of post-filter... 195 Figure 5.6.2. FIR-linear-interpolator filter output.... 196 Figure 5.6.3. FIR-Butterworth and FIR-Bessel filter outputs... 197 Figure 5.6.4. Magnitude responses of 5th order Butterworth, Bessel and Linear Interpolator... 198 Figure 5.6.5. Magnitude responses of unequalized and equalized cable... 198 Figure 5.6.6. FIR-Chebyshev-l and FIR-elliptic filter outputs... 200 Figure 5.6.7. Magnitude responses of 5th order Chebyshev-l and elliptic filters.200 Figure 5.6.8. Magnitude responses of unequalized and equalized cable.. :... 201 Figure 5.6.9. Magnitude responses of elliptic and Chebyshev-2 filters... 201 Figure 5.6.10. FIR-Cbebysbev-2 and FIR-elliptic filter output... 202 Figure 5.6.11. Magnitude responses of unequalized and equalized cable... 202 Figure 5.7.1. Illustration for lemma 5.7.1... 208 Figure 6.3.1. 13 -bit Barker-coded signal with input mask... 234 Figure 6.3.2. Filter outputs... 235 Figure 6.3.3. Impulse response of optimal ECUI filter... 235 Figure 6.3.4. DSX3 example... 236 Figure 6.3.5. Impulse response of optimal ECUI filter... 237 Figure 6.5.1. Response of filter to nominal Barker Coded input...... 245 Figure 6.5.2. Response of filter to perturbed inputs... 245 Figure 6.5.3. Filter's response to nominal input... 246 Figure 6.5.4. Filter's response to perturbed inputs... 246 Figure 6.6.1. Weighted constraint robustness margin... 249 Figure 6.6.2. Laguerre filter - EC approach... 254 Figure 6.6.3. Magnitude response EC approach... 255 Figure 6.6.4. Laguerre filter - EC approach with constraint robustness... 255

xiii Figure 6.6.5. Magnitude response with constraint robustness... 256 Figure 6.7.1. Response of perturbed filter... 260 Figure 6.7.2. Response of perturbed robust filter... 260 Figure 6.7.3. Signal Energy = 5.73... 261 Figure 6.7.4. Signal Energy = 6.25, - 9% increase... 262 Figure 6.7.5. Magnitude response of perturbations... 263 Figure 6.7.6. Magnitude response of perturbations of robust filter... 264

xv List of Tables Table 4.5.1. Simulation results for 13-bit Barker-coded signal example... 139 Table 4.5.2. results for rectangular pulse example... 142 Table 4.5.3. Simulation results for DSX-3 pulse template... 142 Table 4.6.1. Laguerre and FIR filters... 146 Table 4.6.2. Performance of Laguerre filters for various dominant pole values.. 148 Table 5.6.1. Simulation data for linear interpolator, Butterworth and Bessel postfilters... 197 Table 5.6.2. Simulation data for Chebyshev type 1 and elliptic post-filters... 199

Preface In the signal processing literature, filter design is a well-established area and there has been a considerable amount of work devoted to it. Whether it's classical or at the leading edge of research, the literature primarily concentrates on frequency domain constraints (such as passband ripples, stopband attenuation). However, very little attention has been directed towards filter design with time domain constraints in the form of envelopes or masks, which have become increasingly important in the performance specification of modern telecommunication systems. Time domain filter design problems are commonly known in the literature as time-domain synthesis problems and deal with finding a network to give a prescribed response for a given excitation, both of which are specified as functions of time. This class of problems arises in pulse-shaping circuits, pulse transmission systems, delay network, transmission channel equalization, video distribution systems, and the like. In time domain synthesis problems, often the performance criterion is the mean square error between the filter output and some desired signal. In many practical signal processing problems, this soft least-square approach is artificial. The approach may also yield unsatisfactory results because large narrow excursions from the desired shape occur, and the norm of the filter can be large. In addition, the choice of an appropriate weighting function is not obvious. Moreover, the solution can be sensitive to the detailed structure of the desired pulse, and it is usually not obvious how the desired pulse should be chosen so as to obtain the best possible solution. The distinctive feature of the hard envelope-constrained filter formulation is that the output waveform is specified to lie within an envelope defined by a set of inequality constraints, rather than attempting to match it with a specific desired pulse. Therefore, in the hard envelope-constrained filter formulation, we deal with a whole set of allowable outputs, and the objective is to seek the one for which the corresponding filter results in minimum enhancement of filter input noise. The hard envelope-constrained filter formulation is more relevant than the soft least-square approach in a number of xvii

xviii signal processing fields. For example, in TV channel equalization, it is required that the shaped signal simply fits into a prescribed envelope called a K-mask. In digital transmission, the performance of a digital link is often specified in terms of a mask applied to the received test signal. The envelope-constrained approach is directly applicable for shaping test signals into given masks. For pulse compression applications in sonar and radar detection, envelope-constrained filters can also be used to suppress sidelobes. This monograph has evolved from work on envelope constrained filter design problems that has origins dating back to the late 60's. Over the years, there have been numerous fundamental contributions reported in the literature. In this monograph, our modest intention is to present a unified formulation that covers digital, analog and hybrid (consisting of both analog and digital parts) filters. This formulation considers not only envelope-constrained filters but also problems with more general response constraints and wider range of cost functions. Motivated by practical realization issues, finite dimensional approximations to this general problem are investigated and a number of realizable filter structures are suggested. Furthermore, the important issue of designing envelope constrained filters that are robust to implementation errors are addressed. A class of iterative algorithms is developed for solving this envelope-constrained filtering problem with finite filter structures. Convergence properties of these algorithms are addressed. In continuous-time, this technique yields optimum EC filters with analog or hybrid realizations. This is achieved without discretizing the constraints of the filter. Practical real examples are included to illustrate the effectiveness of each of the algorithms developed in this monograph. Most of the material contained in this monograph is based on the research carried out by the authors and their collaborators during the last several years. This monograph can also be viewed as an application of constrained optimization theory to a class of engineering problems in signal processing. It takes the reader through the three phases of problem solving: problem formulation, characterization of solution, and numerical techniques for solving the problem. In the formulation phase (Chapter 2) we see how a raw engineering problem is formulated by taking into account real world constraints. We also see that the first itera-

xix tion of the formulation yields a solution, which is not directly usable. The next iteration in the formulation process is to improve and refine the formulation and the mathematical model. The solution characterization phase (Chapter 3) marches through the mathematical analysis of optimization theory to derive the forms that the solutions take on. Interpretations of what these results mean in signal processing terms are then made. Numerical algorithms for solving these problems are perhaps the most important phase in engineering, because if we can't find the solution to the problem, the other two phases are of little use to the designer. This phase is covered in Chapters 4,5 and 6. Chapters 4 and 5 present algorithms for discrete-time and continuous-time problems respectively. We look at the discrete-time problem first because it is simpler. This also makes it easier to grasp the continuous-time problem. Chapter 6 considers the robustness problem. This monograph is intended for both engineers and applied mathematicians. The authors believe that both engineers and applied mathematicians can make further contributions to the subject. With the background acquired from this monograph, engineers will learn the mathematical rigor in problem formulation and the development of solution methods. On the other hand, applied mathematicians will find this monograph as a stepping stone towards a research area in signal processing. Moreover, the authors also believe that the monograph will be useful as a reference to practicing engineers and scientists as well as mathematicians. The background required for understanding the mathematical formulation, the algorithms and the application of these algorithms to solve practical problems is advanced calculus. However, to analyze the convergence properties of these algorithms, some results in real and functional analysis and optimization theory are required. For the convenience of the reader, essential mathematical concepts and facts in real and functional analysis are stated without proof in Appendix A, while optimization theory, in particular, convex mathematical programming, are reviewed in Appendix B. To ensure readability of the monograph amongst both engineers and applied mathematicians, the proofs are given in the appendix of each chapter so as not to disrupt the

xx continuity of flow. Those who are interested in the mathematical details can take excursions to the appendices. The authors believe that in this way both classes of reader can gain a better appreciation of the subject. It is a pleasure to express our gratitude to R. Evans, W. X. Zheng, G. Lin, Z. Zang, C. H. Tseng, and H. H. Dam. They have made a significant contribution to the material presented in this monograph through collaborative research projects. We wish to thank Australian Telecommunications Research Institute of Curtin University of Technology and the Co-operative Research Centre for Broadband Telecommunications and Networking for the stimulating research environment and we also appreciate the financial supports provided by the Australian Research Council, the Department of Electrical and Electronic Engineering of the University of Western Australia, and the Department of Electrical and Electronic Engineering of the University of Melbourne. Also the financial supports of the Research Committee, the Centre for Multimedia Signal Processing of the Department of Electronics and Information Engineering, the Department of Applied Mathematics, all with The Hong Kong Polytechnic University, and the Research Grant Committee of Hong Kong are gratefully acknowledged. Furthermore, we wish to express our appreciation to John R. Martindale, Senior Publishing Editor of Kluwer Academic Publishers, for his encouragement, enthusiasm and collaboration. Our sincere thanks also go to our families for their support, patience and understanding. B.Vo A. Cantoni K. L. Teo February 2001