UNIVERSITY OF CINCINNATI

Transcription

1 UNIVERSITY OF CINCINNATI Date: April I, Vikram Srinivasan, hereby submit this work as part of the requirements for the degree of: Master of Science in: Electrical Engineering It is entitled: HDL Descriptions of Artificial Neuron Activation Functions This work and its defense approved by: Chair: Dr.Carla Purdy Dr.Hal Carter Dr.Robert Ewing

2 HDL Descriptions of Artificial Neuron Activation Functions A thesis submitted to the Division of Research and Advanced Studies of the University of Cincinnati in partial fulfillment of the requirements for the degree of Master of Science in the Department of Electrical and Computer Engineering and Computer Science of the College of Engineering 2005 by Vikram Srinivasan B.E., Sri Venkateswara College of Engineering, 2002 Committee Chair: Dr.Carla Purdy

3 Abstract A significant development in recent times has been the remarkable ability to create artificial intelligent systems which are capable of thinking and making decisions independently. To a large extent, this has been a fruit of modeling systems based on existing prototypes like the human body. Conforming to this definition, artificial neural networks mimic biological neural networks in their structure and capacity. Such designs have been physically fabricated in silicon with the advent of Very Large Scale Integration (VLSI) and there is an increasing demand to make larger networks on a smaller scale, by reducing component sizes. In keeping with this trend, Hardware Description Languages offer a powerful medium for designers to express their ideas in code-style; a far less complicated option vis-à-vis graphic design tools like MAGIC. Although automated synthesis of HDL codes has not advanced significantly, there is no doubt that such design codes are easy to develop, understand, simulate and optimize. Almost all commercial practices adopt the HDL style of design today and we are not too far from coming up with complementary synthesis technologies. In our efforts to fabricate entire artificial neural networks as systems on chips, we must first develop component models in HDL. This thesis attempts to describe commonly used neuronal activation functions like the Linear Threshold, Sigmoid and Gaussian functions in Verilog-AMS. This is a popular HDL with Mixed Signal extensions to analog and digital design. These models have been described in their structural and behavioral sense. The results have been compared to existing SPICE simulations to verify accuracy. Disparities between simulation results in certain models can be observed upon comparison, which can be attributed to a difference in MOS transistor descriptions between SPICE and Verilog models. Although we are yet to fully tap the potential of a convenient designing language like Verilog-AMS, a library of neurons can enable us to develop HDL codes for entire networks, which will ultimately be efficient, convenient and cost-effective for automatic synthesis when we optimize our designs completely with the powerful features this language has to offer.

4

5 Acknowledgements In your hands lies the outcome of many stimulating and thoroughly enjoyable meetings that I was fortunate enough to have with Dr. Carla Purdy, my advisor in this research and my mentor in many ways. From her, I have learnt new definitions for capacity and capability and I cannot over-stress the many ways in which I am indebted to her. For their unfailing support and unsinkable faith in my potential at times when I needed reassurance, I thank my friends in my research group. Jyothiram, Shruthi.N, Aravind, Madhuri, Eashwari, Rajesh and Shruti.B, you have made it a fun trip reaching my destination and saved me from moments when I was close to losing my sanity. To friends outside my lab-priya, Subash, Sai, Vijay, Sharada, Ratna, Sandhya, Arun, Raja, Hari and Divakar, I cannot forget the numerous instances when I banked on you for support, confidence, suggestions and criticisms. When my efforts were slacking, you were forthright with your chidings and when obstacles overwhelmed me, you helped me put in more than I ever could. To my family, I owe my existence. The totality of experiences in my life holds no value in their absence. And to the University of Cincinnati, I am obliged, for making my stay in Cincinnati a cherished one. Graduate school has been one of the best experiences in my life and I am pleased I chose this fine institution. For the time they took to review this compilation and offer their suggestions, ideas and comments, I am grateful to the members of my advisory panel: Dr. Hal Carter and Dr. Robert Ewing. And I cannot end this note without mentioning the immense help I received from some fine individuals. To Guneet Singh, my guiding light when I was lost in the world of Verilog-AMS, to Lakshminarayanan.R for unflinchingly resolving my queries in analog design, to Mohit Jain for being a pillar of support, without you all, I could not have seen this day. To you, I dedicate this thesis.

6 CONTENTS List of Figures 4 List of Tables 7 1. INTRODUCTION Introduction to Artificial Intelligence and Neural Systems The Nervous System Artificial Intelligence and Intelligent Systems Artificial Neural Networks Motivation for this Thesis Organization of this Thesis NEURAL NETWORKS Biological Neural Networks Central Nervous System (CNS) Peripheral Nervous System (PNS) The Neuron The Artificial Neuron Electrical Operation of a Neuron Artificial Neural Networks Feed-Forward and Feedback Networks Single Layer and Multi Layer Networks Cellular Neural Networks VLSI Implementation of Neural Networks Applications of Neural Networks Business Medicine Cardio-Vascular Model Electronic Nose ANALOG-MIXED SIGNAL (AMS) DESIGN System Representation Behavioral Domain Structural Domain Physical Domain Digital Design Metrics in Digital Design Analog Design Techniques and Principles in Analog Design A Comparative Study 39 1

7 3.5 Mixed Signal Modeling Hardware Description Languages (HDLs) HDL Features Verilog-AMS The Top-Down Design Structure Aspects of Verilog-AMS VHDL-AMS Methods of Declaration Comparison with Verilog-AMS Importance of HDLs MODEL DESIGNS Goal Definition Network Components The MOS Transistor Structure Principle of Operation The CMOS Technology The CMOS Inverter The Operational Amplifier Common Op-amp Configurations CMOS Structure of an Op-amp Neuron Models Processing Element Design Unit Step Function Linear Threshold Function-Fixed Threshold Linear Threshold Function-Adjustable Threshold Sigmoid Function-Fixed Gain Sigmoid Function-Adjustable Gain Gaussian Function Equivalent HDL Descriptions Behavioral Models Structural Models Design Space SIMULATIONS Verilog-AMS Overview The Source Code The Test Bench Header File Module Ports File Write Operation 78 2

8 Component Instantiation Inputs Behavioral Models The Unit Step Function The Linear Function-Fixed Threshold The Linear Function-Variable Threshold The Sigmoid Function-Fixed Gain The Sigmoid Function-Variable Gain Gain= Gain= Gain= Summary Structural Models The Unit Step Function The Linear Function-Fixed Threshold The Linear Function-Variable Threshold The Sigmoid Function-Fixed Gain The Sigmoid Function-Variable Gain The Gaussian Function Voltage Derivative Plots Summary Aspect Ratio Variations Operational Amplifier Models Gaussian Model CONCLUSIONS AND FUTURE WORK Summary and Conclusions Possible Improvements The BSIM Model Resolving Jump Discontinuities An Improved Gaussian Function Model Design Space Exploration Network Integration 132 BIBLIOGRAPHY 133 APPENDIX APPENDIX

9 List of Figures 2.1 Diagram of the central nervous system Diagram of the peripheral nervous system Photograph of a neuron from the CNS Diagram of the neuron Direction of nerve message transmission Control flow in a neuron Structures of the biological neuron and artificial neuron Circuit schematic of an artificial neuron Electrical circuit of a neuron Feedforward and feedback ANNs Multi-Layer neural networks Gajski and Kuhn s Y-chart representing levels of abstraction Digital, analog-discrete and analog-continuous signals Illustration of key principles, concepts and techniques in analog design Structure of Verilog-AMS Key features of Verilog-AMS and VHDL-AMS Types of FETs MOS transistor behavior Physical structure of an nmos transistor Top view of an n-type metal oxide semiconductor field effect transistor I-V characteristics of an n-type MOSFET for VG=5V, 4V, 3V, 2V The CMOS inverter The Op-amp symbol and power supply connections and equivalent circuit Block diagram of op-amp The non-inverting amplifier mode configuration The inverting amplifier mode configuration The voltage follower configuration Op-amp schematic with transistor sizes Unit step function Analog comparator circuit for unit step function Linear threshold function Non-inverting amplifier circuit for linear function Circuit to realize a linear function with adjustable threshold Sigmoid function graph Inverting amplifier cascaded network to realize the sigmoid function Amplifier circuit and CMOS representation of the configuration to generate a sigmoid activation function with adjustable gain Graph of the Gaussian function proposed Gaussian function circuit Test bench for the linear function (fixed threshold) Circuit diagram with test parameters for the unit step function (behavioral model) Schematic of the unit step function behavioral model using SimVision 81 4

10 5.4 Example waveforms of the unit step function behavioral model using SimVision SPICE plot of output vs input voltages for the unit step function behavioral model Verilog-AMS and SPICE plots of output vs input voltages for the unit step function behavioral model Circuit diagram with test parameters for the linear function-fixed threshold behavioral model Schematic of the linear function fixed-threshold behavioral model using SimVision Example waveforms of the linear fixed-threshold behavioral model using SimVision SPICE plot of output vs input voltages for the linear fixed threshold behavioral model Verilog-AMS and SPICE plots of output vs input voltages for the linear fixed threshold behavioral model Circuit diagram with test parameters for the linear function-variable threshold behavioral model Schematic of the linear function variable-threshold behavioral model using SimVision Example waveforms of the linear variable-threshold behavioral model using SimVision SPICE plot of output vs input voltages for the linear variable threshold behavioral model Verilog-AMS and SPICE plots of output vs input voltages for the linear variable threshold behavioral model Circuit diagram with test parameters for the sigmoid-fixed gain behavioral model Schematic of the sigmoid fixed gain behavioral model using SimVision Example waveforms of the sigmoid fixed gain behavioral model using SimVision SPICE plot of output vs input voltages for the sigmoid fixed gain behavioral model Verilog-AMS and SPICE plots of output vs input voltages for the sigmoid fixed gain function behavioral model Circuit diagram with test parameters for the sigmoid variable gain (100) behavioral model Schematic of the sigmoid variable gain (100) behavioral model using SimVision Example waveforms of the sigmoid variable gain (100) behavioral model using SimVision SPICE plot of output vs input voltages for the sigmoid variable gain (100) behavioral model Verilog-AMS and SPICE plots of output vs input voltages for the sigmoid variable gain (100) behavioral model Circuit diagram with test parameters for the sigmoid variable gain (500) behavioral model Schematic of the sigmoid variable gain (500) behavioral model using SimVision Example waveforms of the sigmoid variable gain (500) behavioral model using SimVision SPICE plot of output vs input voltages for the sigmoid variable gain (500)behavioral model Verilog-AMS and SPICE plots of output vs input voltages for the sigmoid variable gain (500) behavioral model Circuit diagram with test parameters for the sigmoid variable gain (1000) behavioral model Schematic of the sigmoid variable gain (1000) behavioral model using simvision Example waveforms of the sigmoid variable gain (1000) 5

11 behavioral model using SimVision SPICE Plot of output vs input voltages for the sigmoid variable gain (1000) behavioral model Verilog-AMS and SPICE plots of output vs input voltages for the sigmoid variable gain (1000) behavioral model Circuit diagram with test parameters for the Unit step function (structural model) Schematic of the unit step function structural model using SimVision Example waveforms of the unit step function structural model using SimVision SPICE plot of output vs input voltages for the unit step function structural model Verilog-AMS and SPICE plots of output vs input voltages for the unit step structural model Circuit diagram with test parameters for the linear fixed threshold function (structural model) Schematic of the linear fixed threshold function structural model using SimVision Example waveforms of the linear fixed threshold function structural model using simvision SPICE plot of output vs input voltages for linear fixed threshold structural model Verilog-AMS and SPICE plots of output vs input voltages for the linear fixed threshold structural model Circuit diagram with test parameters for the linear functionvariable threshold structural model Schematic of the Linear variable threshold function structural model using SimVision Example waveforms of the linear variable threshold function structural model using SimVision SPICE plot of output vs input voltages for the linear variable threshold structural model Verilog-AMS and SPICE plots of output vs input voltages for the linear variable threshold structural model Circuit diagram with test parameters for the sigmoid-fixed gain structural model Schematic of the sigmoid fixed gain function structural model using SimVision Example waveforms of the sigmoid fixed gain function structural model using SimVision SPICE plot of output vs input voltages for the sigmoid fixed gain structural model Verilog-AMS and SPICE plots of output vs input voltages for the sigmoid fixed gain structural model Circuit diagram with test parameters for the sigmoid variable gain (100) structural model Circuit diagram with test parameters for the Gaussian function structural model Schematic of the Gaussian function structural model using SimVision Example waveforms of the Gaussian function structural model using SimVision SPICE plot of output vs input voltages for the Gaussian function structural model Verilog-AMS and SPICE plots of output vs input voltages for the Gaussian function structural model Verilog-AMS plots of output voltage and derivative for sigmoid behavioral model Verilog-AMS plots of output voltage and derivative for sigmoid structural model Verilog-AMS plots of output voltage and derivative for Gaussian structural model Plots of output current vs input voltage for aspect ratio variations in the Gaussian model Gaussian synapse circuit schematic 131 6

12 List of Tables 3.1 Analog and digital design Unit step function behavioral model simulation results Linear fixed threshold behavioral model simulation results Linear variable threshold behavioral model simulation results Sigmoid fixed gain behavioral model simulation results Sigmoid variable gain (100) behavioral model simulation results Sigmoid variable gain (500) behavioral model simulation results Sigmoid variable gain (1000) behavioral model simulation results Unit step function structural model simulation results Linear fixed threshold structural model simulation results Linear variable threshold structural model simulation results Sigmoid fixed gain structural model simulation results Gaussian function structural model simulation results Simulation results of aspect ratio variations in op-amp models( ± 5V DC supply) Simulation results for aspect ratio variations for the Gaussian model 126 7

13 Chapter 1 Introduction 1.1 Introduction to Artificial Intelligence and Neural Systems The human body is arguably the most complex structure that can ever be conceived in the universe. So intricate is its construction and so complicated are its operations that it has been established that modern day super-computers are no match in terms of performance or organization. At the lowest level, its anatomy is comprised entirely of cells- billions of them which are microscopic in nature. Cells are clustered into tissue structures and groups of tissues shape an organ, resulting in basic organsystems. While each of these organ-systems is vital for existence, we confine our focus to the prime controller within the human body-the nervous system The Nervous System The nervous system is the primary controller of physiological operations in the human body. It is responsible for regulatory actions and serves as a network of communication channels within the human body. This system is essential to maintain homeostasis and facilitates us to be cognitive of our internal and external environments through its receptive sensors. Broadly, the nervous system is comprised of the brain, a myriad of brain cells (called neurons) and connecting nerves. The actions performed by this system can be classified into one of 3 typessensory, integrative and motor. Changes in internal or external environment (called stimuli ) are sensed or detected by receptors, thus compiling sensory input data. This data is carried to the brain, which is the central controller of the human body. The process is done through nerve impulses, which are electrical signals. These signals carry information which enables the brain to decide the appropriate response that needs to be implemented. For example, this might be in the form of storage in memory, creation of thought, recollection of information stored previously or 8

14 evolving specific sensations. This continuous, on-going process integrates data from the sensory input with information stored in the brain to produce an appropriate response. The motor operation is carried out by muscles and glands in our body. The brain sends specific instructions by dispatching signals to muscles (for external stimuli) and glands (for internal stimuli). This enables our body to perform the required action--moving a body part or producing secretions. [1] While the process described above certainly seems to be easy to comprehend, it is unimaginable to perceive the speed of operation and the complexities involved with such a wide range of stimuli being sensed within and outside our body. The 3-fold process of sensing input and integrating it to produce a motor output takes place incessantly throughout the lifespan of the human body. The natural question that arises in our mind is- how does the brain do it? While we might very well consider ourselves to be the most intelligent and developed species on this planet, it is ironic that we are baffled by the very functioning of our own body. We seem to perform effortlessly tasks of recognizing patterns and identifying shapes, forms, objects and odors that we perceive through our senses-tasks that are barely capable of being automated. We can assume without exaggeration that a single human brain performs more tasks than the world s stock of supercomputers. Modern day computers are extremely effective in producing precise results to precise questions at an impressive speed. Our nervous system however, performs ill-defined computations with distorted data to produce output that is fairly good, but not entirely accurate. [2,pp3-9] It is only logical to conclude that, in our effort to produce automated systems to perform comparably with the human body, we need to first understand how the body works, in order to design our system on close lines to perform simulations that are permissible by physical restrictions. This brings us to the study of artificial intelligence. 9

15 1.1.2 Artificial Intelligence and Intelligent Systems It is not without reason that we call ourselves homo sapiens- man the wise. Our mental capabilities are fundamental for our existence. The computational actions that are probably most imperative are the ones we take for granted, with little knowledge of their basis. The study of artificial intelligence deals with the fundamental operation of our own intelligence system, in an effort to comprehend its functioning, not merely to establish results but in fact to develop artificial systems capable of performing in a similar fashion. In this aim, it differs from most other studies of intelligence. While the task outlined seems to be certainly daunting, one might argue that other fields of science also raise the possibility of developing nearly-impossible phenomena (eg:anti-gravity, travel at speeds faster than light etc). However, our efforts are encouraged considerably by the fact that we have existing proof of realizing our dream- something that is lacking in most other fields. Our very bodies are living proof that such intelligent systems are not impossible to create. We know what lies at the end; we need to use it to develop the means to get there. This is the foundation of artificial intelligence and the study of intelligent systems. [3, pg 3] An intelligent system can be safely defined as a system capable of operating in an environment with cognitive capability to detect and respond to changes in that environment. The system does not require external control for its operation, and this suggests that the system should be capable of learning by itself. Hence, if a certain situation produced favourable results at one point of time and unfavourable results at a later point due to a difference in system reactions, the system should rationally deduce the more appropriate response, if the same situation arises in future. It seems obvious to expect then that the system should also have an ability to retain information and results in memory. It is also insufficient for such a system to make informed decisions if it cannot implement them. [4] 10

16 1.1.3 Artificial Neural Networks Since we have established what an intelligent system should be capable of performing, it facilitates our quest that we model such systems based on living examples like the human body. Predominantly, the field of artificial intelligence has dealt with mimicking the structure of the human body s nervous system and genetic engineering. Thus the concepts of evolving artificial neural networks and genetic algorithms arose. We shall look closely into what the former is. Artificial Neural Networks refer to computing systems whose central theme is borrowed from the analogy of biological neural networks. [5, chapter 1] Just as the biological neural network works with nerve cells (neurons), its artificial counterpart works with neurons to cater to specific functions like classifying data or recognizing patterns. These networks can be trained to learn operations by iterations. Thus a training algorithm is developed to teach an artificial neural network. The primary reason why neural networks are studied is because they have a remarkable ability to process imprecise data and deliver meaningful results. They can detect patterns or trends that are otherwise unnoticeable by other computer techniques. A neural network that has been trained to process a particular type of data may well be considered an expert in analyzing that data type. Further, it can enable us to speculate how the system would perform in different situations. Other key advantages can be summarized in the following points. 1. Adaptive Learning The ability to learn how to respond, given an initial data set and training experience. 2. Self-Organization The ability to create its own organization of data received during training. 3. Real Time Operation The ability to run processes in parallel. Special hardware devices can be designed to make use of this feature. 11

17 4. Fault Tolerance via Redundant Information Coding The ability to retain some network capabilities in the event of partial damage to the network (and subsequent degradation of performance). [6] The driving force behind artificial neural network research is the fact that biological neural networks evolved without any notion of mathematics or engineering analysis. But over the course of evolution, a vast array of physical phenomena have been studied and analysed. We can use this to our advantage to implement complex functions. Thus it is evident that the capabilities of well-developed artificial networks can be incredible, given that we have an excellent prototype (the human body) and powerful developmental tools (principles and languages). [2] 1.2 Motivation for this Thesis Neural Networks are rapidly emerging as intelligent and complete solutions for computational tasks in a variety of real-world applications. There is an increasing demand to manufacture such entire systems in silicon and this has been a result of growing trends in Very Large Scale Integration (VLSI). With this tool, we can pack millions of microscopic devices on a small silicon chip and make the design process custom-made for specifications. Hardware implementation of intelligent systems enjoys prime importance today and this is well-deserved since such systems are extremely fast, robust and portable. Fabricating neural networks in silicon requires appropriate hardware and software. Designs built using digital logic are noise-tolerant with a high degree of accuracy. They can also be manufactured easily using existing automated synthesis procedures. Analog designs offer the advantage of reducing the amount of hardware required and implementing dynamic systems. Thus, our efforts will be wellrewarded when we adopt mixed signal modeling-a combination of digital and analog techniques which encapsulates the advantages of both styles [52]. Subsequently, we need a language which can be used to define such mixed-signal designs. Here arises the need for using Hardware Description Languages. The Verilog-AMS language has been employed in this thesis to describe commonly used neuronal activation functions and compile a library of such models. Apart from obtaining a better understanding of the powerfulness of this language, the exercise is an effort to explore rapid 12

18 prototyping of mixed signal systems in hardware. Automatic synthesis of such designs is not sufficiently developed today. Designing in HDL will now spur efforts to come up with complementary technologies to realize such models. Since HDLs offer immense flexibility in terms of optimization and modification, we have a convenient and economic way to explore our options in adopting various scales of technologies and architectures. 1.3 Organization of this Thesis This thesis attempts to study the operation of some neuron models and describe the same in a hardware description language. How is this relevant? While we have made significant progress in developing artificial neural networks, we are far from realizing them comfortably in the physical sense. Two basic factors hamper our progress in that aspect. Neural networks are far more complex in terms of connectivity than standard hardware. In the past, the technology to realize such complex circuits was absent. Sufficient knowledge of the organizational principles of these systems was unknown. It is only in recent times that we have made advancements in understanding the functioning of biological neural networks. Chapter 2 elaborates the basic structure of neural networks-biological and artificial. It also covers the basic building block of the nervous system-the neuron. Recently, cellular neural networks are proving to be a revolutionary concept and an experimentally proven paradigm. A section of the chapter is devoted to shedding more light on this new technology. Altogether, Chapter 2 has been included to present a sample of the vast amount of information we now possess to realize artificial neural networks. Until recently, we lacked the means to design complicated network circuits. However, we have with us today, a powerful technology in the form of Very Large Scale Integration (VLSI), which can be used to integrate millions of devices on a single wafer of silicon. VLSI can include digital and analog components. The latter is significantly complicated when compared to the former, reasons for which will 13

19 be explained in detail in Chapter 3.To realize powerful artificial neural networks, we need to understand the complexities involved in designing circuits with analog components and develop solutions to the problems faced. This chapter compares design characteristics in digital and analog VLSI. It also gives an introduction to the Verilog-AMS tool that has been used in developing neuronal descriptions Chapter 4 has been dedicated to descriptions of network components like the metal-oxide semiconductor (MOS) transistor and the operational amplifier (op-amp), apart from explaining the neuron activation function models used in this thesis. The MOS transistor is the basic unit of all analog circuits while the op-amp is the functional unit of most neuronal circuits. A thorough understanding of their structure and function will help analyze the working of circuit models built with them. With a sufficient description of the tools we have at our disposal as well as the knowledge we possess, we look at the actual neuronal models in Chapter 5. These models have been developed in the Verilog- AMS hardware description language (HDL) and their functioning has been illustrated with simulations and graphical charts. An important focus of this thesis is a comparison of these models with their counterparts in a VLSI graphic layout design tool like MAGIC. Simulations of behavioral and structural models obtained using Verilog-AMS have been compared to those obtained by simulating the corresponding structural MAGIC models with SPICE. The results have been compared and contrasted. We find that our design task is sufficiently simplified by adopting a code-style approach with the Verilog-AMS, which also offers acceptable performance. While we observe deviations in some cases from ideal behavior or structural model behavior (SPICE simulations), we acknowledge that such factors exist as the difference in levels of MOS models used (in SPICE and Verilog) as well as an improvement in the understanding of the features of the language and simulator. Automated synthesis of analog HDL designs is not currently possible. However, a compilation of model descriptions such as the one attempted here will enable us to optimize our designs when we possess complete working knowledge of the language and simulator. Equipped with such codes, we explore what possibilities lay ahead of us in Chapter 6. 14

20 Chapter 2 Neural Networks 2.1 Biological Neural Networks The nervous system has the prime responsibility of monitoring and maintaining optimal conditions in our internal and external environment. A brief introduction to the nervous system is presented, before its artificial counter-part is examined. The human biological nervous system is divided into the following sub-systems: Central Nervous System (CNS) The central nervous system is comprised of the brain and the spinal cord (Fig 2.1). Much is unknown about the functioning of the human brain. An interesting question that is yet to be answered is how the brain trains itself in accordance with the functioning of an intelligent system. The brain is the control unit of the human body. It receives information from other parts of the body through electrical impulses and sends responsive instructions after processing this information. Different parts of the brain handle various cognitive functions like touch, speech, memory, recognition, hearing etc. The brain is connected through nerve cells to the sensors and actors in the body. Fig 2.1: Diagram of the central nervous system from [8] 15

21 The spinal cord runs along the dorsal side in the human body. It connects the brain to other parts of the body. It is enclosed in a vertebrae structure which forms the vertebral column, while the brain itself is enclosed in the human skull. Fluid and tissue act as insulation for the brain and the spinal cord Peripheral Nervous System (PNS) The division of the nervous system that connects the brain and spinal cord to other parts of the human body is the peripheral nervous system (Fig 2.2). This is almost entirely comprised of nerve tissues. Nerves are made up of billions of nerve cells called neurons. Fig 2.2: Diagram of the peripheral nervous system from [9] The peripheral nervous system has 2 kinds of pathways: The sensory (afferent) pathways which provide information from body parts to the CNS. The motor (efferent) pathways which carry signals from the brain to the muscles and glands. Most sensory information is usually carried below the level of conscious awareness. When it crosses this level, it contributes to perception of the external environment. 16

22 2.1.3 The Neuron The neuron is the basic functional unit of the nervous system. A microscopic image of a neuron is presented in Fig 2.3. The human brain alone has over 100 billion neurons. All of them resemble one another in their structure to a considerable extent. Fig 2.4 illustrates this basic structure. Information from neighboring neurons travels through dendrites to the cell body. The cell body contains the nucleus, mitochondria and Golgi bodies. The axon conducts messages away from the cell body. These 3 parts are characteristic of any neuron. [7] Fig 2.3: Photograph of a neuron from the CNS taken from [22] 17

23 Fig 2.4: Diagram of the neuron from [10] Signals sent by the neuron travel across the axon (Fig 2.5). At the end of every branch lies a structure called the synapse. The synapse converts this electrical signal into an activity which inhibits or excites the axon of a neighboring neuron. When a neuron receives excitatory input sufficiently larger than its own inhibitory input, it sends a spike of electrical activity through its axon. Learning occurs when the effectiveness of the synapse changes (thus affecting the influence of one neuron on another). [6] Fig 2.5: Direction of nerve message transmission from [11] 18

24 The basic operation of a neuron is explained in Fig 2.6. Input 1 Input 2 SUM Threshold Function OUTPUT SIGNAL Input 3 Fig 2.6: Control Flow in a neuron According to Zell, All inputs received by the neuron are summed up before they are passed onto a threshold function. The output signal is produced by processing the summed input with the threshold function. The processing time is about 1ms per cycle and the transmission speed is about ms. [12] Zell goes on to describe the 100-step rule. The brain works in both a parallel and serial way. The parallel and serial nature of the brain is readily apparent from the physical anatomy of the nervous system. That there is serial and parallel processing involved can be easily seen from the time needed to perform tasks. For example a human can recognize the picture of another person in about 100 ms. Given the processing time of 1 ms for an individual neuron this implies that a certain number of neurons, but less than 100, are involved in serial; whereas the complexity of the task is evidence for a parallel processing, because a difficult recognition task can not be performed by such a small number of neuron. This phenomenon is known as the 100-step-rule. [12, pg 24] Fault tolerance in biological systems is also very important. Biological neural systems usually have a very high fault tolerance. Experiments with people with brain injuries have shown that damage of neurons up to a certain level does not necessarily influence the performance of the system, though tasks such as writing or speaking may have to be learned again. This can be regarded as re-training the network. [13] 19

25 2.2 Artificial Neurons The artificial neuron essentially has been modeled on the same lines as our current understanding of the biological neuron with multiple inputs and a single output. This neuron undergoes training and usage phases. During training sessions, the neuron is taught to fire (or not fire) for specific input patterns. Thus when in use, if an input pattern is recognized, its associated output will become the current output. Unrecognized input patterns call for firing rules to determine appropriate action. [6] The close structural parallelism between the biological and artificial neurons is evident in Fig 2.7. Fig 2.7: Structures of the biological neuron (upper) [6] and artificial neuron (lower) [14] In accordance with Fig 2.7, an artificial neuron can be modeled as a multi-linear threshold device with weighted interconnections or synapses. 20

26 Variable resistors are used to represent the synapses while a high gain, non-linear amplifier usually represents the cell body (Fig 2.8). The dynamics of each neuron is governed by an ordinary first-order differential equation by applying Kirchoff s Current Law at the input node of the amplifier. Fig 2.8: Circuit schematic of an artificial neuron from [14] Ci du dt i n = GijV j=1 j U i + U ρ i io ( ) V = f (1) i U i The above equations explain the voltage flow for a neural network with n neurons where Ui is the input voltage to the i th amplifier, Uio is the external noise to the i th amplifier and Vi is the output voltage of the i th amplifier. Ci is the input capacitance of the i th amplifier while ri is the input resistance of the i th amplifier. Gij forms the conductance of the resistive interconnect between the i th and j th amplifiers while f is the transfer function of the non-linear amplifier. Signals received from other neurons in the form of potentials across the resistive interconnects are collected by summing currents. Each synaptic weight or resistive interconnect is modeled as a passive resistor with conductance G. Based on its input neutral voltages U, the processing element (PE) produces an output V according to its non-linear transfer function f. The output signal is then propagated to other neurons. [14] 21

27 2.2.1 Electrical Operation of an Artificial Neuron In a single neuron, the synaptic weights are stored as charge on a capacitor. The operating principle is illustrated in Fig 2.8. It is divided into two phases: 1. Pre-Charge Phase All the switches in the synapses are set towards the buffer and the pre-charge signal in the neuron is active. In each synapse, the output capacitor is charged through the weight buffer to the same voltage as the storage capacitor. The neuron consists of a comparator and a storage latch. The pre-charge signal closes a switch between both inputs of the comparator. This pre-charges the post-synaptic signal to a reference voltage that constitutes the zero level of the network. Fig 2.9: Electrical circuit of a neuron from [15] 2. Evaluate Phase Here, the sum of all the synapses is compared to the pre-charge voltage. The neuron fires if the synapse signal exceeds this pre-charge voltage. This neuron state is stored in the flip flop at the moment when the phase changes from evaluate to pre-charge. In the evaluate phase, if the presynaptic signal is active, the output capacitor is connected with the post-synaptic signal by the synapse switch. The pre-synaptic signals are generated by the input neurons depending on the network input and feedback information. This cycle can be repeated a fixed number of times to restrict the networks to a maximum layer number and limit the processing time for an input pattern. The network can also 22

28 run continuously while the input data changes from cycle to cycle, thus making it useful in signal processing applications. [15] 2.3 Arificial Neural Networks Artificial neural networks (ANNs) are intelligent systems that have been designed to mimic the performance of a biological neural network. While existing ANNs are no match for the superb functioning of a human biological neural network, significant advancements have been made recently in developing powerful artificial networks with impressive speed and robustness. As Foo et al explain, Artificial networks are trained by successive examples in a real-world situation. As the ANNs adapt to changes in their environment, they develop their own internal rules. One advantage of ANNs is their ability to handle fuzzy or incomplete data. These networks are used for a variety of applications including pattern recognition, robotic control and combinatorial optimization. [14] Feed-Forward and Feedback Networks Feed forward artificial neural networks (Fig 2.9) allow signals to travel from input to output without a path for feedback. Thus the output cannot affect the input in the same layer. These are simple networks used in pattern recognition. They are also called top-down or bottom-up networks. Fig 2.10: Feed-forward (right) and feedback (left) ANNs from [6] 23

29 Feedback networks (Fig 2.9) have loops in them which enable signals to travel both ways. These powerful networks change their state continuously until they reach equilibrium. They remain in this state until the input changes. These networks are also called recurrent or interactive networks. [6] Single and Multi Layer Networks A single layer network is composed of m neurons, each having n inputs. The system performs a mapping of the n-dimensional input space to the m-dimensional output space. To train the network, the same learning algorithms as for a single neuron can be used. This type of network is widely used for linear separable problems only. Multi-layer networks solve the classification problem for non-linear sets by employing hidden layers, whose neurons are not directly connected to the output. Fig 2.10 shows a two-layer and a three-layer network architecture. Fig 2.11: Multi-layer neural networks from [13] The additional hidden layers are geometrically interpreted as hyperplanes. These enhance the separation capacity of the network. The problem of training the hidden units for an unknown desired output is solved using the back-propagation (BP) algorithm. 24

30 The training occurs in a supervised style. The basic idea is to present the input vector to the network, calculate the output of each layer in the forward direction and the final output of the network. For the output layer, the desired values are known and therefore, the weights can be adjusted as for a single layer network; in the case of the BP algorithm according to the gradient descent rule. To calculate the weight changes in the hidden layer, the error in the output layer is back-propagated to these layers according to the connecting weights. This process is repeated for each sample in the training set. One cycle through the training set is termed an epoch. The number of epochs needed to train the network depends on various parameters, especially on the error calculated in the output layer. [13] 2.4 Cellular Neural Networks A recent addition to the family of artificial neural networks is the cellular neural network (CNN). Invented by Leon.O.Chua and Lin Yang in Berkeley in 1988, the CNN is an array of analog dynamic processors or cells. The computer architecture is the CNN Universal Machine [18], with its various physical implementations. At the same time, Analogic CNN computers mimic the anatomy and physiology of many sensory organs with an additional capability of stored programmability. [17] Processors in a cellular neural network accept and generate continuous-time analog signals with the inputs playing a vital role. The CNN becomes a framework for complex system performance and computations, thus opening a startling new world of analogic computing with, in one implementation, 16,000 fully programmable processors integrated into a visual microprocessor. [17, pp 1-6] The CNN universal machine can be considered as a high-speed supercomputer integrated onto a single chip with a complicated algorithm sequence coded into it. An interesting feature of the CNN is its dual nature. While it is predominantly analog in its operation, it has an analog-digital input-output interface which enables it to be used as a digital signal processing element too. The facility of a special operating system, a compiler and a high-level language to work with this network, make it very attractive for use in image processing. [19] As defined by its inventors, the CNN can be considered to possess the following features: 25

31 It is an N-dimensional regular array of elements (cells) The cell grid can be for example a planar array with rectangular, triangular or hexagonal geometry Cells are multiple input-single output processors, all described by one or just some few parametric functionals A cell is characterized by an internal state variable, sometimes not directly observable from outside the cell itself Multiple connection networks can be present, with different neighborhood sizes A CNN dynamical system can operate both in continuous or discrete time CNN data and parameters are typically continuous values CNNs operate typically with multiple iterations; i.e. they are recurrent networks. The CNN s main characteristic is the locality of the connections between the units: the main difference between CNN and other neural network paradigms being that information is directly exchanged just between neighboring units. Better global processing is also obtained this way. Communications between units which aren t directly connected are obtained by passing through other units. It is possible to consider the CNN paradigm as an evolution of the cellular automata paradigm. Moreover it has been demonstrated that the CNN paradigm is universal, being equivalent to the Turing machine. [20] Thus cellular neural networks are better suited for VLSI implementation than general neural networks. They also have practical dynamic ranges as opposed to severe range restrictions with general networks. Given its recent conception, the full potential of this powerful theory is yet to be exploited. [21] 2.5 VLSI Implementation of Neural Networks In our quest to develop intelligent systems, we have been driven by the assumption that a complete understanding of the structural components of a system will suffice to understand and describe the operation of the entire system. Biology, however, has not provided us with such systems to model. All biological systems have a considerable level of complexity which is not due to the intricacies of the individual components as much as it is due to the convoluted ways in which these components interact with one another. In other words, it would be presumptuous to believe that an understanding of every 26

32 nerve and synapse in a neural network is sufficient to understand how that system performs neural computations. Performance is characteristic of the organization principles of the system, which is obscure in any single component. With the rapid strides that we have made in trying to understand neural biology, we can take pride in the fact that there is nothing about the operation of a neuron that is incomprehensible. There exist two reasons why we have not been able to realize intelligent systems to the extent that we would like to, given the wealth of information we possess- lack of appropriate technology and knowledge of operating principles. Very Large Scale Integration (VLSI) has evolved as a powerful technology to help us in this process. Tens of millions of metal oxide semiconductor (MOS) transistors can be integrated on a single silicon wafer, providing us with a two-fold advantage of high density and high operation speed. It should be borne in mind that complementary MOS (CMOS) technology has been primarily considered to be ideal for digital circuits and that such a technology might not be suitable for analog designs like neural networks. This reservation is further strengthened by the precision and reliability of neural systems which seem to fall short of even rudimentary digital systems. But as Mead explains, This lack of precision is considerably offset by system-level redundancy. This is an important emergent property of neural systems. If neural system design in VLSI is based on the same principles, it stands to reason that system malfunction due to individual device failure should not be a cause of concern; on the contrary, the feasibility of realizing resultant robust systems on the scale of a silicon wafer might not be far-fetched. [2, pp1-10] Another reason why this approach is best suited for our goal is the close parallelism between biological and analog systems. The active devices in both technologies (synapses and transistors) occupy barely 2% of the entire system space; the rest occupied by wires. Thus if we attempt to reduce the quantity of connections, we can create systems with more computation per unit area. The constraints in this process are similar in both systems: Limited connections ( wires ) Power Required robustness 27

33 Required reliability The key then, is to develop operations which are natural to silicon and explore the possibility of using them for known neural functions. Thus silicon technology delivers the dual advantage of Providing a synthetic element for computational neuroscience to facilitate hypothetical tests Developing an engineering discipline to design collective systems for required computations [2, pp1-10] The usage of device physics (as opposed to digital logic) to design systems would allow us to make full use of this parallelism, particularly with neural networks, since weight adjustments can be carried out by compensating for inevitable device fluctuations. [15] Thus, instead of merely analyzing biological operations, it would be more fruitful to use them as guidelines in extracting maximum utility out of VLSI implementation resources to develop distributed parallel networks (since the biological prototype can also be viewed in the same fashion as involving processing elements using resources in their implementation media). Careful studies of ANNs and VLSI have revealed a high degree of complementation between the two fields; one provides an escalating demand for an increasing number of processing elements to develop solutions for real-world data-processing while the other provides a means of integrating increasingly large number of elements on a single chip. The former provides acceptable performance with high tolerance while the latter can make best use of this ability to increase overall yield of the system. [16, pp1-15] In the next chapter, we look at commonly faced issues in analog and digital VLSI design before we examine the tools that have been chosen for our purpose in this thesis. 2.6 Applications of Neural Networks Ever since their conception, the prospects of applying neural networks for real-world problems have been exciting. Rewarding experiments have further encouraged efforts. While a wide array of applications call for the utilization of neural networks, their usefulness in business and medicine are presented here. 28

34 2.6.1 Applications in Business The inherent ability of artificial neural networks to recognize patterns and trends makes them ideal for forecasting applications including Sales forecasting Target marketing Risk management Customer research Quality management There also exist favorable prospects in applying ANNs for database mining and resource allocation tasks Applications in Medicine Presently, artificial neural networks are being extensively used for modeling body parts and detecting diseases based on symptoms and behaviors. This is yet another application of their pattern recognition capability. An added advantage for this application is the fact that neural networks train themselves to recognize symptoms to identify a particular disease. Thus the actual process is irrelevant as compared to the expected system performance. What is required though is a good set of examples which will cover every possible symptom to make an accurate diagnosis Cardio-vascular Model A neural network can, for example, be made to develop a model of a cardio-vascular system based on existing physical conditions of an individual. Such a model should have relations between physiological variables defined clearly for the specimen individual. Once the system has adapted to the specific nature of that individual, it will become a model of that particular body. Our requirement, however, calls for a system that can adapt itself to any specimen; a feature that neural networks are particularly useful for. 29

35 Sensor fusion is yet another key feature of ANNs that is useful for this purpose. The network can deduce complex relations between various physiological variables, instead of monitoring them individually. Thus, while the sensors as such monitor only a particular quantity individually, the network can detect complex medical conditions by fusing this data Electronic Nose Electronic nose technology developed with neural networks is useful in long-distance medical treatment or telemedicine through a communication link. The actual odour in a remote site can be detected and transferred electronically to the clinical center where it can be recreated for analysis. 30

36 Chapter 3 Analog-Mixed Signal (AMS) Design 3.1 System Representation The design of a circuit system is a definition that has become multi-fold in the field of integrated circuit technology. Any system can be broken down to its components and to the most basic logic gates and MOS transistors. It has become possible to automate the process of describing a system from the topmost to the bottom-most level, enabling us to obtain circuit descriptions that can then be amended, if required, for chip manufacture. The Y-Chart proposed for a digital system by Gajski and Kuhn in Fig 3.1 is an excellent representation of the design domains that exist in IC design. Fig 3.1: Gajski & Kuhn s Y-Chart representing levels of abstraction from [31] Any design can be described at various abstraction levels and the concentric circles denote common levels. From highest to lowest, these might include Architectural 31

37 Algorithmic Functional Logical Switch Circuit levels Behavioral Domain The behavioral domain describes the function of the system; the operation it actually performs. Thus a behavioral description of a circuit describes how it might respond for a particular set of inputs. At every stage, the method of expressing activity is different; from algorithmic steps to hardware register description. Ideally, it is desired to convert a system from the highest possible level into its most simplified system design using minimum time and eliciting maximum system performance. For the task at hand, the designer tries to use features of hardware description languages to best describe the manner in which that operation should be performed. An algorithm is essential for this purpose and the designer attempts a technology-independent description. Speed specifications in terms of gate times can be specified. Functional simulations are performed with a behavioral description to ensure its adherence to system specifications and the designer then moves to the structural domain Structural Domain In the structural domain, the interconnections between components in the system are described. These are so designed to best obtain the output from the system. Thus, no attempt is made to describe the actual operation, as was done in the behavioral domain. Essentially, the structural representation consists of a list of instances of defined models and their interconnections. The levels of abstraction here include the module level, the gate, circuit and switch levels. Again, this representation is independent of technological specifications since mere instances of structures are defined. No mention is made about the actual gates that will be used or the speeds that 32

38 these gates will operate with. This representation is used with the assumption that a clear definition of individual module operations will make the entire system work smoothly, when objects of such modules are created and interconnected. Simulations will ensure the validity of the structural description and we are now ready to manufacture this circuit in silicon Physical Domain Once we have a proper knowledge of how a system is connected and how it functions, we need to know how to realize it in the physical sense, including what the required connectivity is, to implement this system. This is entirely governed by technological specifications starting from photo-masking to circuit element interconnections. In the physical description, no importance is given to the description of a structure or its operation. Instances of modules are represented as bounded process-boxes with specified ports for inputs and outputs. Here, the original connectivity can be re-designed to implement geometric changes and obtain accurate performance estimations. The simulations performed after this stage will yield valuable information about overall processing time and heat dissipation. [32, pp 20-40] With knowledge of the various representations of a system, we can now see what kinds of design methodologies exist. Integrated circuits are either digital or analog, depending on the nature of the signal that flows in them. Recently, a new category of mixed signal models has arisen, which incorporates digital and analog circuits. A discussion on these classes is now presented. 3.2 Digital Design Digital signals are discrete-event signals which possess discrete values at any instant in time. The transition they make from one value to another value is abrupt. Transitions occur typically only between 2 clearly defined levels-high and low, true and false, zero and one. Thus in contrast to analog signals, ideal digital signals can take only one of two values at any given time instant. This is why they are also called discrete signals. [23, chapter 1] 33

39 Since digital signals can only exist in one of two states, the characteristic feature of bi-polar transistors and field-effect transistors (FET) to operate in one of two states makes them ideal operating elements in circuits which are implemented using digital logic. Digital circuit design has evolved rapidly since the advent of integrated circuit (IC) technology. The first IC chips had a small gate count and were manufactured using Small Scale Integration (SSI) technology. Slowly, circuit designers started using Medium Scale and Large Scale (MSI & LSI) Integration technologies with the increase in the number of gates that could be packed in a single chip of silicon. These developments also called for automating the design process simultaneously, since it was becoming progressively difficult to integrate thousands of transistors and verify their operation. Electronic Design Automation (EDA) techniques helped with the advent of Very Large Scale Integration (VLSI). Designers could now pack as many as 100,000 transistors on a chip and computer programs were available to test these circuits. Small building blocks were first designed and then used in replication and expansion to realize higher level blocks until the top-level design was completed. Logical simulators helped weed out flaws in the design process before subsequent architecture levels could be realized, thus optimizing and economizing the process. [24, chapter 1] Predominantly, bipolar devices were used in digital design before the advent of MOS transistors. Bipolar transistors are very inexpensive and they can change their state almost 20,000,000 times every second. However, their need for a constant 5V power supply and large power losses have proved to be major disadvantages. These devices almost became obsolete ever since metal-oxide semiconductor (MOS) transistors were introduced. Although MOS devices can be damaged by static electricity and need more than 2 voltage sources sometimes, they operate at extremely high speeds and use less power than any other device family [25]. With increasing focus on high performance, new technologies like the BiCMOS (bipolar and CMOS technologies on the same die) technology are in effect. Silicongermanium and Super-conducting technologies are also in use today. [24, chapter 1] 34

40 3.2.1 Metrics in Digital Design Any design process needs to be analyzed at every stage to incorporate the best possible operations with minimum cost and maximum utility. A number of factors come into play when one tried to optimize a digital circuit integration process. It should also be borne in mind that the nature of factors might vary depending on the specific application that the chip is being designed for. Thus, key factors in one situation might not be so influential for another application. Commonly monitored factors include Cost of the Circuit: This includes fixed and variable (recurrent) costs. Fixed costs may include the cost of automating the entire process and indirect costs like company overhead, etc. Variable costs are governed primarily by the number of chips being manufactured. They include the costs of testing every chip, packaging, die costs, etc. Functionality and Robustness These refer to the performance of the chip in the required environment. Unavoidable aberrations in design tend to make the measured behavior visibly different from the expected chip behavior. Internal and external noise can also contribute to such unwanted changes. A number of parameters such as noise margins and voltage-transfer characteristics define the robustness of the system. Performance of the System The performance of a digital system is usually interpreted as processing speed. Thus the performance of every single transistor gate governs the performance of the entire system and the importance of precision with respect to time cannot be over-stressed. It is here that the concept of propagation delay is introduced. This is a measure of the delay a signal undergoes when passing through a gate. Power and Energy Consumption A large number of system specifications are influenced by the amount of thermal power dissipated by the system. Thus this factor affects feasibility, cost and reliability. Energy consumption defines a 35

41 limit on the number of computations that can be performed; thereby on the number of devices that can be integrated as well. The ideal gate should perform operations with minimum delay and minimum energy consumption. 3.3 Analog Design Analog signals vary continuously with time. Their possible values are not discrete, as are digital signals. Analog signals are used to represent quantities which vary continuously over a time period, like temperature or pressure. They are usually represented in a range of current or voltage values. Piecewise continuous (analog discrete event signals) analog signals might hold the signal at a particular value for a small time period before changing to a new value. Continuous analog signals perform the transition more smoothly and without periods of retention. The kinds of digital and analog signals described so far have been illustrated in Fig 3.2. [23,chapter 1] Fig 3.2: Digital, analog-discrete and analog-continuous signals from [23] Before we look at the features of analog design, we need to understand what analog design is essentially. While the analysis of a circuit involves a close examination of the flow in the circuit to understand what its output and properties are, the design of a circuit is the exact opposite. Here, we come up with an actual circuit given the kind of output and properties we want this circuit to possess. Thus there is more scope for innovation and optimization. 36

42 Analog integrated circuit design uses active and passive components on the same substrate and in close proximity, thus utilizing component matching as a design tool. This process also provides the designer with more liberty to control the geometry of these devices. Computer-controlled simulations can be performed on such integrated chips instead of manual testing. This also eliminates the need for breadboards and gives the designer immense freedom to monitor signals at various instances and modify the circuit as desired. [28, pp 1-8] Analog design is significantly complicated by the very nature of the devices used in it and the level of integration required with such devices on a single substrate. Therefore, the analog designer seeks to maintain structure and organization in his design process, while leaving room for flexibility and ease to make modifications to the designs if so required. This is imperative, since designs get complicated with increase in the number of devices and the number of ways they interact with one another electrically ans physically. A brief outline of techniques and principles in analog design is presented in the following sub-section. 37

43 3.3.1 Techniques and Principles in Analog Design Fig 3.3: Illustration of key principles, concepts and techniques in analog design from [29] The key players in analog design are depicted in Fig 3.3. Principles and laws are established scientific guidelines which govern the behavior of systems. These might be very convenient for use in modeling but their ease of use is considerably reduced when they need to be coupled with one another to account for physical layouts. Concepts include soft laws which are usually true for most systems. These might be asserted after a common observation in many systems. A good example would be the concept of component-matching. Here, attention is paid to details like physical layouts, using common areas and replicating structures. Techniques are tools used to implement our concepts which are backed by principles, ensuring that the overall goal is achieved without logic or design violations. Thus assumptions are included in techniques and these are made to simplify the task at hand, without compensating any portion of the problem. Simulations are performed to validate the assumptions made. [29] 38

44 3.4 A Comparative Study In recent times, analog design has become a more active area due to significant advances made in mixed-signal (analog and digital circuits on a single chip) technology and hardware implementation of such circuits for data processing. Neural network paradigms have been identified as most resourceful for this purpose, and their study has been enhanced considerably. Table 3.1 points out some key differences between analog and digital design. It must be borne in mind that the field of analog design has not seen as much progress as one would like, given the complexities involved in its devices and their interactions. Much is yet to be understood about their operations and the manner in which these circuits can be physically realized. [30, pp ] Analog Design Digital Design Signals have a continuum of values for amplitude and time Irregular blocks Customized Components have a continuum of values Requires precise modeling Difficult to use with CAD Designed at the circuit level Longer design times Successful designs need many attempts Difficult to test Signals have just 2 states Regular blocks Standardized Components have fixed values Modeling can be simplified Amenable to CAD methodology Designed at the system level Short design times Success at first attempt possible Amenable design for testing Table 3.1: Analog and digital design from [30] 39

45 It is visibly clear that design automation is not as advanced with analog design as it is with digital design. This has become the case largely because of the much more complex behavior of analog components, especially when they are combined into systems. Most digital systems are constructed as a combination of data path and control path (typically a finite state machines or FSM) using standard cell libraries. These are modules based on simple Boolean logic. This unifies and lends homogeneity to digital design and the corresponding synthesis tools can use register-transfer level (RTL) descriptions to produce gate-level descriptions. This is made feasible by the fact that gates and registers (used to realize digital design) have a common input-output model and are derived from simple cell-types which can be parameterized easily and which have adjustable performance trade-off. Mathematical descriptions created by logic synthesizers undergo transformations to produce designs optimal in speed, power and area. To achieve this, equivalence transformations applied to the mathematical descriptions reduce the number of gates (thus minimizing the area) and the depth of logic (thus maximizing the speed). Since speed-power trade-off is easily made in gates, the drive ability of each gate can now be adjusted to provide minimum power while meeting speed requirements. On the contrary, in analog design, the absence of a unifying mathematical structure such as Boolean logic complicates design. The possibility of a common I/O model would increase costs and introduce additional circuitry, thus increasing power dissipation. Parametric optimization is also not an easy task in analog design due to the multitude of performance metrics involved and the scarcity of trade-offs. [23, chapter 1] 3.5 Mixed Signal Design It is evident that analog circuit design is much more complicated than digital design, thus explaining the relative shortcoming in progress made. Analog designers do not have an equivalent to RTL design to synthesize their design and implement a functionally and physically correct model. The necessity to do this procedure manually makes it prone to errors. 40

46 The present trend in integrated circuit design is to work with mixed-signals to overcome significant setbacks in analog design. Mixed-signal systems involve components of analog and digital nature and focus is on their interface. The levels of abstraction that we have observed for analog and digital systems exist for mixed-signal systems too. [33, chapter 1] Mixed-signal design aims to reduce the number of device components needed to realize a system. The effective chip area is also decreased this way. Complexities and possible solutions that cannot be reached in either design process can now be realized in mixed-signal design. Important device components in this design methodology include analog-digital converters (ADC) and digital-analog converters (DAC). The conflict in technology is not easily resolved and a compromise on the number of analog and digital components should be arrived at. There exists no single, well-defined solution to make this decision. Simulations in mixed-design need to take into account high speed (for digital parts) and reasonable (for analog parts). As far as layouts are concerned, certain difficulties like simulating non-intentional interactions between devices exist. [34] In spite of these drawbacks, mixed-signal design is gaining popularity as it provides the designer with freedom to combine two entirely different forms of design and thus save on time, space and silicon. The most common tools used to describe circuits in mixed-signal design are hardware description languages (HDL). The focus in mixed signal design lies on developing digital models with high precision and analog models with acceptable precision and improved processing speeds. 3.6 Hardware Description Languages (HDLs) HDL Features Hardware description languages came into existence when designers realized the need to describe circuits in textual code instead of graphical form. HDLs allow designers to run processes simultaneously; a feature that is characteristic of digital circuit elements. Logical synthesis extracts information about gate-connections from RTL descriptions of circuits and functionalities are thus 41

47 implemented. HDLs are extensively used for system-level design including simulations of field programmable gate arrays (FPGAs) and programmable array logic (PAL). [24, chapter 1] Hardware description languages have two main applications in circuit modeling; simulation and synthesis. A model described in a HDL needs to be verified for proper functioning by testing it in an environment with test inputs and observing the performance. Thus, a key feature of HDLs in this aspect is the ability to make explicit descriptions with useful syntax and semantic definitions. While performing synthesis, it is assumed that the components of the model are abstract descriptions in that HDL and bear no physical form. Here, we need HDLs to describe only descriptions that can be realized physically. As far as various design methods are concerned, only digital design is automatically synthesized currently. Here, the HDL is used to describe the behavior of the circuit in RTL and a gatelevel description is obtained by synthesizing this. However, automation of analog or mixed-signal designs is not yet well-developed. The two main HDLs used in mixed-signal design are Verilog-AMS and VHDL-AMS. The AMS (analog mixed signal) extensions were given to their original names, which were meant for digital design. As such, although these two languages are used for the same purpose, their styles, strengths and weaknesses are different. A choice of language is made depending on the language used to describe the corresponding digital design. [23, chapter 1] Verilog-AMS Verilog-AMS is the language formed by the union and feature extension of Verilog-HDL and Verilog-A languages, as shown in Fig 3.4. The former component allows description of digital components while the latter is used to describe analog circuit elements. Verilog-AMS has the additional capacity to describe mixed-signal circuits also. Fig 3.4: Structure of Verilog-AMS from [23] 42

48 Since it integrates features of both languages, Verilog-AMS can be conveniently used by analog and digital designers. It offers a single language with a single simulator. It also equips analog modules with digital compilation speed; a major advantage that digital design enjoys over analog design. [23] The Top-Down Design Structure Traditionally, analog designers have adhered to a bottom-up design methodology where individual components and blocks are defined and then integrated to form the larger system. While blocks are being defined, inputs and outputs defined are for each block and a transistor-level implementation of every block is obtained. Thus, each block is verified for functionality as an individual unit. Then system integration is performed and verified for proper functioning of the entire system. A number of inefficiencies can be spotted in this procedure. An unfair tradeoff occurs between verification and processing time. Simulating such an integrated system will consume a lot of time, which can only be reduced by decreasing the amount of verification. This is definitely not an advisable option. Economic considerations are explored at the architectural level and there is little scope to do this in the bottom-up design. Errors spotted while integrating the system can only be rectified by re-designing blocks all over again. This process requires an extremely reliable communication system between designers Critical steps in the process can only be done in a serial fashion and this increases time consumption. On the contrary, Verilog-AMS utilizes the top-down approach which is visibly more efficient. System designers and circuit designers work in the same programming environment; thus enhancing communication and scope for modification. Verilog-AMS also uses a common representation of a design at the system and circuit level, enabling individual blocks to be co-simulated with the entire system. 43

49 Verification is performed as early as possible until the highest possible level is reached. Thus errors are caught in advance and simulations of high-level designs take shorter time. Defining a near-perfect architecture first, before going to lower levels, allows for complete refinement of corresponding higher levels at every stage and reduction of overall re-design. Documenting designs in code is easier to use for updates and verification, as compared to other forms. Storing executable blocks and plans saves considerable time when a design is reopened in future. [23] Aspects of Verilog-AMS Verilog-AMS is conveniently designed in the object-oriented programming fashion and therefore, its coding style bears a striking resemblance to programming in other languages of the like (e.g.,:c++). In the coding process, to facilitate descriptions of various physical parameters and models, the language offers a set of disciplines ; a discipline being a collection of related physical signal types or natures. Models are designed by describing their structure or functionality in specific descriptions titled modules. These are the basic building blocks of the language. Modules describe every aspect of the model being constructed. This requires the usage of constants (called parameters ) and variables. There is a wide variety of data types which are present in the both categories. Connections to a model are made at points called ports. The nature of signal and direction of signal flow can be specified for such ports. If the model being designed consists of analog components, the analog description is defined in a block labeled analog. A model that has been described in such a module needs to be tested under experimental conditions to measure its performance. This is where test benches are used. Required test conditions are described in this file and the test bench provides a virtual simulation network for the test module. Instances of one or multiple modules are created in the test bench and power sources are defined here. In some cases, actions which further need to be taken during/after simulation (e.g., writing data to a file) can also be described. 44

50 Verilog-AMS allows good flexibility to the designer not only in behavioral descriptions but in structural descriptions as well. Instances of Verilog-A, digital Verilog and Verilog-AMS can be interconnected with their netlists in a single module. Thus connections between analog and digital instance ports can be done easily. The connecting modules in that case are called interface elements. Such modules provide access to features restricted to regular modules. [23, chapter 1] Verilog-AMS can be used to describe electrical and non-electrical models. The concepts of nodes, branches and ports are used to support conservative and signal-flow descriptions. Natures are collections of attributes which describe the units of measured variables [35] The following (from [35]) is a summary of some key features of Verilog-AMS. It provides a common design language for analog and digital signals and a mix of both. Both analog and digital signal values may be accessed (read operations) from any context (analog or digital) Digital signal values may be set (write operations) from any context outside of an analog procedural block The discipline declaration is extended to digital signals A new construct, connect statement, is added to facilitate auto-insertion of user defined connection modules between the analog and digital domains When hierarchical connections are of mixed type (i.e. analog signal connected to digital port or vice versa) then user defined connection modules are automatically inserted to perform signal value conversion A common platform for analog and digital modeling enables reusability of code and ease of design. With such features, this language promotes the design of systems involving digital and analog interchanges. It is understandable that Verilog-AMS is growing to become a widely used tool in mixed signal modeling. [36] 45

51 3.6.3 VHDL-AMS The VHDL-AMS language is yet another hardware description language capable of performing mixedsignal simulations. The language was obtained by extending the features of the very high speed integrated circuit (VHSIC) hardware description language (VHDL). As such, VHDL has predominantly been considered as the standard language for designing digital systems and therefore, it was chosen to be the base for building the AMS extension Methods of Declaration The structural and behavioral domains of modeling that have been introduced earlier can be realized in this hardware language. The kernel of a model is the entity, which is similar to a module in Verilog- AMS. The declaration of an entity involves the definition of the inputs and outputs to the model, their natures and the kind of value they can possess. The structural or functional behavior of the model is described in another unit called the architecture of the system. Thus multiple architectures can exist for a single entity unit, depending on the manner in which the user seeks to implement it. The architecture begins with a declaration of variables that are pertinent to that particular architecture (and hence not defined in the entity). Process statements describe digital design while simultaneous statements describe the analog portion of the model. The actual behavior of the system is described between the begin and end architecture statements and is marked by the key word process is. At any point of time, the process can be made to represent the state of the system by enabling action to be triggered by a clock signal. In this wait state, temporary values are not lost. While attempting to make a structural description of a system, it is necessary to remember that entity declarations of all components in the system need to be made in the code as opposed to the single entity declaration in the functional description. Variables in VHDL-AMS are represented as quantities, terminals or signals. Quantities usually represent analog values while signals and terminals represent arbitrary values and nodes of energy domains respectively. After declaring and defining every component through its entity and architecture, the structural model of the system is defined by creating instances of these entities and mapping the ports accordingly. 46

52 While structural and behavioral models can be defined exclusively, the option of defining a mixed model with component instantiation and process statements is not ruled out. Thus, hybrid models with structural and behavioral definitions in the same code are permissible, and, in many situations, desirable. An important part of any model involves testing it in a simulated environment which matches the actual performance environment. The concept of a test bench has been introduced earlier, and it is pertinent to models described in VHDL-AMS too. It defines the inputs and outputs to the system and the components it is expected to possess. Thus component instantiation is performed here, along with power source and signal definitions. Simulating a model with its test bench involves a 3-fold process of analysis, elaboration and execution. The analysis of the model-test bench pair is an examination of the logical authenticity of the code to ensure that it adheres to the syntax and semantics of the language. In the elaboration stage, the actual functional units are created with their behavior, terminals and signals well defined. At this stage, the model is completely defined by a set of processes and signals with an expression set to govern its operation. Digital portions of the code are executed in discrete time units while analog portions are executed by an analog solver at analog solution points in continuous time. [33, chapter 1] Comparison with Verilog-AMS Both HDLs can be used to realize mixed signal systems involving continuous and discrete signals. They are a result of extensions to their original digital hardware description languages (Verilog-HDL and VHDL). Verilog-AMS has distinct methods of defining free and flow quantities while VHDL-AMS allows mathematical formulations implicitly and explicitly. Models described in both languages are simulated in a similar fashion through the stages of analysis, elaboration and execution. There are analog kernels in both languages and piecewise behavioral descriptions. Both languages are capable of describing conservative (based on Kirchoff s laws) and nonconservative (signal flow) systems. While VHDL has options to manually describe interface units between discrete and continuous components, Verilog-AMS does this job automatically. 47

53 While Verilog-AMS has an exclusive analog description language base (Verilog-A), no such counterpart exists in VHDL-AMS. A model in VHDL-AMS is defined as an entity which can be operated by multiple architectures governing its operation and behavior. In Verilog-AMS, multiple definitions involving a model need to be defined as individual modules. Verilog-AMS is more compatible with SPICE with a list of preferred parameter and port names for SPICE elements. This is absent in VHDL-AMS. Relating tolerance definition between models and their SPICE simulations is more generic in VHDL-AMS than in Verilog-AMS. The former also provides an option to group unknown quantities into classes of tolerance ranges. [37] A complete analysis of features in both languages is displayed in Fig

54 Fig 3.5: Key features of Verilog-AMS & VHDL-AMS from [37] 49

55 3.6.5 Importance of HDLs Ease of abstraction is the biggest advantage that HDLs bring to hardware design. Designs need not be restricted by fabrication technology and changes in technology can be incorporated every time a design is synthesized. Synthesis will incorporate area and timing optimization. Optimizing and improving the RTL code of a design to meet specifications goes a long way towards reducing design cycle time. Errors are rectified early in the process and the probability of identifying bugs in the physical layout stage is minimized. Given that software programming is widely accepted and used, using HDLs for design descriptions is synonymous to writing in code. The description made is more concise, unlike complicated gate-level descriptions. [24, chapter 1] The attractive option that a pre-defined analog subset exists in Verilog (in terms of the Verilog-A domain) is the key reason why we have decided to adopt the Verilog-AMS hardware description language for our modeling task, as opposed to VHDL-AMS. The Verilog-AMS language is also relatively easier for designers to understand and start using for modeling. 50

56 Chapter 4 Model Designs 4.1 Goal Definition We have seen that hardware description languages are excellent tools for analog design, given the simplicity of definition and the profound flexibility that the designer enjoys. It is then our ultimate goal to develop entire artificial neural networks using HDLs to describe systems on chips. In this thesis, we focus on developing HDL descriptions of neuron models based on various activation functions. Such descriptions have been illustrated in the behavioral and the structural sense in the following sections. 4.2 Network Components Before we proceed to the actual models, a general introduction to some neural network circuit components is presented. These are the building blocks that can be seen as an integral part of all the models that have been developed in this thesis. Predominantly, we focus on the MOS transistor and the operational amplifier, which have proved to be the fundamental network components in structural and behavioral descriptions The MOS Transistor Structure A field-effect transistor (FET) is a uni-polar device that conducts electricity by the action of an electric field on the flow of electrons through a single type of material. The various types of FETs are illustrated in Fig.4.1. N and P channel FETs exist in depletion and enhancement modes. 51

57 (Depletion) (Enhancement) (Depletion) (Enhancement) Fig 4.1: Types of FETs from [38] The MOSFET derives its name from the insulating semiconductor (usually SiO2) layer between the gate and channel, which decreases gate current and increases gate control voltage range. The gate is usually connected to the source internally. The insulated gate is on the opposite side of the substrate. A biasing voltage will either attract or repel majority charge carriers across the PN junction, as shown in Fig 4.2. This changes the polarity of the gate-source voltage and therefore causes the channel to grow wide (enhancement) or narrow (depletion). Fig 4.2: MOS transistor behavior from [38] For an N-channel MOSFET (NMOS), a positive gate voltage with respect to the source and substrate (VGS>0) will repel holes into the substrate and cause an enhancement of the channel, apart from decreasing channel resistance. Alternatively, a negative gate voltage will increase the channel width by attracting holes into it, thereby increasing channel resistance. The converse operation is noticed in P-channel MOSFETs (PMOS). Since a single MOS can act in one of two phases, the construction of a MOS depends on the channel size in the resting state when VGS=0. A depletion mode device (Normally-ON device) will conduct when there is no bias voltage applied. Its resting state channel will become smaller when a reverse bias is applied. An enhancement mode device (Normally-OFF device) does not conduct in resting state since 52

58 it does not have a channel. Increasing Forward bias will form a current-conducting channel. Electrons are the majority charge carriers in an NMOS while holes are the majority carriers in a PMOS. [38] Principle of Operation An enhancement-mode NMOS transistor (Fig 4.3) has a moderately-doped p-type substrate with the source and drain regions being heavily diffused with n +. The p-type substrate channel is covered by a thin insulating SiO2 gate oxide layer, over which lies the polysilicon gate electrode. The high degree of symmetry lends to no distinction between source and drain areas while the high strength of the oxide layer makes the application of high gate fields feasible. [32,chapter 1] Fig 4.3: Physical structure of an nmos transistor from [32] The top-view of such an nmos transistor is illustrated in Fig.4.4. It is to be pointed out that the term gate length refers to the distance between the source and the drain below the gate and not the actual gate dimension. A fair amount of overlap is provided to enable smooth conduction. The voltage applied to the gate controls the electron flow from source to drain. When a positive voltage is applied, electrons are attracted to the dielectric-semiconductor interface, thus forming a conducting channel called the inversion layer. As such, the gate oxide blocks carrier flow and therefore, no gate current is required at the interface to maintain the inversion layer. Effectively, the current flowing between drain and source is controlled by the gate voltage applied. The I-V curves for various gate voltages are shown in Fig 4.5. [38] 53

59 Fig 4.4: Top View of an n-type metal oxide semiconductor field effect transistor from [39] Fig 4.5: I-V characteristics of an n-type MOSFET for V G=5V (top), 4V, 3V, 2V(bottom) from [39] As can be seen in Fig 4.5, the drain current increases linearly with applied voltage and then saturates at a maximum value. When the drain current is maximum, the drain end charge density becomes zero and with decrease in current, the density changes sign. With accumulation of holes on the surface, the change in the inversion layer goes to zero and reverses its sign. However, these holes cannot contribute to the drain current since the reversed-biased p-n diode between the drain and the substrate 54

60 blocks any flow of holes into the drain. Instead the current reaches its maximum value and maintains that value for higher drain-to-source voltages. The additional drain-to-source voltage is accommodated by a depletion layer located at the drain end of the gate. This behavior is referred to as drain current saturation. [38] The CMOS Technology With a basic NMOS transistor, a low input throws the transistor into the non-conducting state and the output is high. Conversely, a high input will make the transistor conduct and the output low. Thus, the transistor acts like a virtual switch. In the complementary metal oxide semiconductor (CMOS) technology, both the PMOS and the NMOS are used to build models. The former conducts when the input is low and the latter conducts when the input goes high. CMOS technology has become the industry standard today since it has almost zero static power dissipation, when compared to bipolar or NMOS circuits. Therefore, more devices can be integrated into a single chip, which is a tremendous advantage over bipolar and NMOS technology The CMOS Inverter The Inverter is one of the most basic CMOS gates. It has a simple structure consisting of a single PMOS and NMOS transistor connected as shown in Fig 4.6. When the input to the inverter is grounded (or LOW), the output is pulled to the VDD value (and thus made HIGH) by the p-channel transistor. Similarly, the n-channel transistor turns the output LOW (electrical ground) when the input is HIGH. 55

61 Fig 4.6: The CMOS inverter from [40] When the input is HIGH (or equal to VDD), the PMOS transistor does not conduct. However, the NMOS transistor provides a path from ground to output and therefore, the output of the device is 0. If the input level is low, the PMOS transistor provides a path from VDD to output while the NMOS is blocked. Therefore, the device output is logical 1 or HIGH. [41] Structural models developed in this thesis have been entirely based on circuits using the CMOS transistor. The core device electronics of the device and the equations that govern the same are far too complicated to warrant a description here. However, they have been included in the codes compiled, as a good reference The Operational Amplifier Innumerable circuits are modeled using the operational amplifier (op-amp) as the fundamental device. The op-amp is a 2 input device which operates on dual DC power supply. It has a very high open-loop gain. In the presence of feedback, the feedback network determines what the closed-loop gain is going to be. Fig 4.7: The Op-amp symbol and power supply connections (a) and equivalent circuit (b) from [42] The op-amp has an Inverting (-ve) and non-inverting (+ve) input terminal. As shown in Fig 4.7, their voltages with respect to electric ground are denoted by VN and VP. The output voltage is VO. Reference ground for op-amps is established by the dual power supply of VCC and VEE, typical values being +15V and -15V. Structurally, the op-amp consists of 3 stages (Fig 4.8). The differential amplifier stage is 56

62 followed by a gain stage and finally by an output buffer. The buffer stage is used for resistive loads and not used when the op-amp drives purely capacitive loads. Fig 4.8: Block diagram of op-amp from [43] The output of an op-amp is the difference between the 2 input voltages (VP-VN) multiplied by the openloop unloaded gain (a). Vo = avd = a(vp-vn) (1) Thus the op-amp can be considered as a device with a single-ended output port and a double ended input port. The nature of the op-amp to amplify the differential input voltage lends it the name difference amplifier. [42, chapter 1] Common Op-Amp Configurations In this section, we look at some common configurations that the op-amp is used in. The models developed later have incorporated these very same configurations too. Non-Inverting Amplifier In this configuration, a single input voltage is given to the positive non-inverting feedback network connects the negative inverting terminal and ground (Fig 4.9). terminal. A 57

63 Fig 4.9: The non-inverting amplifier mode configuration from [44] The output voltage in this case, is given by Vo = Vin * (1 + R2/R1) (2) Since the gain (1 + R2/R1) is positive in this case, the polarity of the output voltage is the same as that of the input voltage. Hence there is no phase reversal, thus making the amplifier non-inverting. Inverting Amplifier Here, the input voltage is given to the negative inverting terminal while the positive terminal remains grounded. The Gain is given by R2/R1. The negative sign indicates that there is a 180 degree phase shift, making the amplifier inverting. Fig 4.10: The inverting amplifier mode configuration from [44] 58

64 Vo = (- R2/R1) * Vin (3) Voltage Follower In the voltage follower configuration, the input voltage is given to the positive non-inverting terminal and the output is fed back to the negative terminal. Thus, the ground resistance (R1) is 0 while the feedback resistance (R2) is infinity. Fig 4.11: The voltage follower configuration from [44] Vo = AVi = Vi A = 1 (4) In this configuration, the op-amp presents a view of an open circuit from the input side and a short circuit from the output side. The op-amp now behaves as a voltage buffer between the source and a connected load. [42, chapter 1] CMOS Structure of an Op-amp So far, we have seen a general description of the op-amp and its configurations. We now take a look at a gate-level description of an op-amp. This has been based on a design explained by Baker et al in [43, chapter 25]. The model in Fig 4.12 lays out the CMOS structure of the various stages in an op-amp. Transistor dimensions (width/length ratios) have also been marked for each device. 59

65 Fig 4.12: Op-amp schematic with transistor sizes from [43] This op-amp has been tested to display an open-loop gain of It forms the basis of all our structural models, as will be explained below. 4.3 Neuron Models It has already been mentioned that research is active to synthesize artificial intelligent systems that can match or excel the functioning of their biological models. The field of VLSI has further spurred advancements in this area with the advent of integrating millions of electronic components on a small silicon chip. It is along these lines that we wish to model and synthesize entire artificial neural networks as systems on chips. To do so, a good start would be to model neuron components first and test their functioning. Once we are successful in doing so, we can then integrate copies of these components to form entire networks. We have just seen that hardware description languages (HDLs) are useful tools in circuit design for reasons of convenience, concurrent simulation and easy optimization. The aim of this thesis is to 60

66 develop neuron models in Verilog-AMS and analyze the relative capability of the language to deliver simulation results comparable to existing SPICE simulations Processing Element Design In Chapter 2, we have seen that the electronic neuron consists of synapses and a cell body which are the resistive interconnects and the processing element (PE) respectively. Referring back to Fig 2.7, we can see that a neuron receives electrical signals from neighboring neurons in the form of potentials across its interconnects. These are summed up and passed as an input to the threshold function in the PE. The PE will then generate an output voltage based on its transfer function. This voltage will then be transmitted to other neurons. [14] In our design task, we then focus on modeling resistive interconnects and the PE. Foo et al have displayed numerous designs of neuronal components utilizing simple analog devices in [14] and this forms the basis of a large part of our design. While resistive interconnects can be easily modeled using many variable linear conductance devices (e.g., JFET) we concentrate on describing the PE. The PE is a key component of a neural network. It receives voltage inputs through its synapses from other neurons. The sum of these inputs may or may not fire the neuron depending on its ability to exceed a certain threshold value. Commonly used activation functions include unit step function linear function sigmoid function Gaussian function Therefore our task of designing PEs reduces to developing circuits which utilize the above mentioned functions to evaluate thresholds for firing. 61

67 4.3.2 Unit Step Function This is the simplest activation function that can be used for a neuron s operation. The function is evaluated as zero for all values of x less than 1(unity) and as 1 for all values of x greater than 1. A graph of this function and its mathematical representation are shown in Fig f (x) = 0 : x <=1 = 1 : x > 1 Fig 4.13: Unit step function from [45] According to Foo et al, This high-gain step function is used for configuring associative memory in the Hopfield neural networks, Hamming networks and Boltzmann machines. [14] This function is easily realized by an analog voltage comparator as shown in Fig The input voltage at the positive terminal of the op-amp comparator is compared with the constant reference voltage at the negative terminal. The output is driven to high (+Vcc) once the input crosses this threshold. Until such time, the output is at 0. Fig 4.14: Analog comparator circuit for unit step function from [14] 62

68 4.3.3 Linear Threshold Function-Fixed Threshold The linear threshold function has a ramp type output as shown in Fig The output is at a fixed low or a fixed high state if the input is below or above the lower and upper threshold values respectively. For input values within the range of the thresholds, the output is linearly proportional, thus giving a ramp wave. Fig 4.15: Linear threshold function from [14] According to Foo et al, The Linear Threshold activation function is commonly used in the single-layer feed forward networks such as ADALINE and single-layer perceptrons. This function is implemented by using the op-amp in its non-inverting configuration, as has been explained in section (Fig 4.16). The voltage gain of this function and the bounds on the same are given by the following formulas. [14] Fig 4.16: Non-inverting amplifier circuit for linear function from [14] 63

69 Voltage Gain : AV = Rf + Rg Rg Lower Saturation Point : LSP = VccRg Rg + Rf Upper Saturation Point : USP = VccRg Rg + Rf Linear Threshold Function Adjustable Threshold The threshold of the linear function can be made adjustable by using a twin-cascaded inverting operational amplifier. This is illustrated in Fig The circuit also offers the added advantage of varying the range of linearity. The equations for output voltage and threshold limits are given below. R Output Voltage : Vo = Vi R 2 1 R Vref 1 + R 2 1 R1 + R Lower Saturation Point : LSP = Vref R2 2 R Vcc R 1 2 R1 + R Upper Saturation Point : USP = Vref R2 2 R + Vcc R 1 2 Fig 4.17: Circuit to realize a linear function with adjustable threshold from [14] 64

70 4.3.5 Sigmoid Function-Fixed Gain The sigmoid function is a widely used activation function in Sigma-Pi and Hopfield types of neural networks. Mathematically, it is expressed as 1 f ( x) = (5) 1 + e x The graph of the sigmoid function resembles an S-shaped curve as shown in Fig Fig 4.18: Sigmoid function graph from [14] To realize this function, a high-gain inverting amplifier can be cascaded with a unity gain inverting amplifier, as in Fig The output voltage of this configuration is given by V = 0.5tanh(λU) where λ is the voltage gain and U is the input voltage to the circuit. [14] Fig 4.19: Inverting amplifier cascaded network to realize the sigmoid function from [14] 65

71 The continuous derivative values make the sigmoid function widely applicable in back-propagation networks, for purposes such as character recognition, sonar target recognition image classification signal encoding knowledge processing etc. [46] Sigmoid Threshold Function-Adjustable Gain The sigmoid function can be used in a more convenient way which offers the flexibility of changing the gain of the circuit. This requires an unbuffered voltage comparator with positive feedback loops to be connected to a pair of inverters with negative feedback loops. The comparator generates the sigmoid function and the gain of the circuit is controlled by the negative loops of the inverters. The gain can be controlled by varying the ratio r which is defined as R 8 R10 r = = (6) R R where the resistances in the ratios are the resistances in Fig This illustrates the amplifier circuit as well as its equivalent CMOS circuit

72 Fig 4.20: Amplifier circuit (top) and CMOS representation of the configuration to generate a sigmoid activation function with adjustable gain from [14] Gaussian Function In one dimension, the Gaussian function is the probability function of the normal distribution given by [47] (7) The graph of the function resembles a bell-shaped curve as illustrated in Fig

73 Fig 4.21: Graph of the Gaussian function from [48] Although the Gaussian function responds to a very small range of input values, the back-propagation training of Gaussian-based networks is far more effective than that of sigmoid-based networks. These networks also function at much higher speeds comparably. One such circuit, unbiased in the sub-threshold region, is proposed by Choi et al in [46]. This circuit is illustrated in Fig 4.22 below. Fig 4.22: Proposed Gaussian function circuit from [46] The input voltage is applied to the gate terminal of one transistor in the differential pair, and the weight value is stored on the total capacitance at the gate terminal of the other transistor. The two currents in the differential pair are expressed as 68

74 I 1 = I x β 4 4I x ( V V ) ( V V ) 2 in w β 2 in w I 2 = I x β + 4 4I x ( V V ) ( V V ) 2 in w β 2 in w with the input voltage in a finite region of V in 2I x Vw. β 2 W Here, Ix is the reference current and β = µc ox L M2. is the trans-conductance value of transistors M1 and The output of the synapse cell can be determined by Iout = a.ix-(i6+i9) where a is the drain current ratio of transistor M12 to M10. When the difference between Vin and Vw turns negative, then I1 is greater than Ix and I2 is lower than Ix. In this situation, I6=I1-bIx and I9=0. Here, b is the drain current ratio of transistor M4(M7) to M11. On the other hand, when Vin-Vw>0, then I1 is less than Ix while I2 is greater than Ix. In this new situation, I6 equals zero and I9=I2-bIx. When the input voltage Vin is comparable to the synapse weight Vw, transistors M5, M6, M8 and M9 turn off and the output current is mainly contributed by transistor M12. [46] 4.4 Equivalent HDL Descriptions It is our aim to develop a library of neuron models in a hardware description language. The functioning of such models will also be compared to the functioning of their SPICE counterparts. In this thesis, the models described above have been designed in the Verilog-AMS HDL. While the models are based on the designs illustrated in 4.3, it has to be mentioned that the actual circuit component parameters have 69

75 been fixed according to the existing SPICE models developed by Athreya in [49]. Simulations and further discussions are compiled in Chapter 5. In all these models, the description language used has been Verilog-AMS. This language has been developed with the intent to let designers of analog and mixed signal systems and integrated circuits create and use modules that encapsulate high-level behavioral descriptions as well as structural descriptions of systems and components. The terminals of a module and external parameters applied to the module can be used to mathematically describe its behavior. The structure of each component can be described in terms of interconnected sub-components. These descriptions can be used in many disciplines such as electrical, mechanical, fluid dynamics, and thermodynamics. Verilog-AMS HDL is defined to be applicable to both electrical and non-electrical systems description. By using the terminology for conservative and signal-flow descriptions using the concepts of nodes, ports, branches etc, Verilog-AMS is capable of supporting such descriptions. The solution of analog behaviors which obey the laws of conservation fall within the generalized form of Kirchhoff s Potential and Flow laws (KPL and KFL). Both of these are defined in terms of the quantities associated with the analog behaviors. [35] The description codes developed in Verilog-AMS have been simulated using the Cadence-AMS Simulator 2.0 on a Solaris workstation. This is a mixed-signal simulator capable of compiling Verilog- AMS codes. Transient analysis of signals in circuits has been performed using the SimVision 4.10-s022 [59] tool, which is a unified graphical environment to test all Cadence simulators. We now take a closer look at the HDL design process Behavioral Models The design process was initiated by developing behavioral models for every circuit, which could replicate the performance of the circuit in HDL simulations. In this aspect, we are concerned with the functioning of the circuit and the signals associated, as opposed to the internal components of the circuit. 70

76 Thus, for example, while modeling the linear fixed threshold circuit (Fig 4.16), we would be using models of an op-amp and a resistor to build the circuit. Next, we would want to see how this circuit behaves for different inputs and parametric values. We experiment by changing the input given to the non-inverting terminal, as well as by changing the values of the feedback resistance-pair. For this particular model, since we have a formula that tells us what the ideal gain of the configuration should be, we can verify the performance of our model by running a simulation with circuit parameters from the corresponding SPICE simulation [49] and observing the output to note any deviations. It needs to be reiterated here that all circuit parameters for the models compiled in this thesis have been based on existing SPICE simulation values in [49]. Such behavioral models can only be built by using models of individual circuit components. While one can locate resources online which have sizeable compilations of such models, it is good design practice to try and develop individual models by oneself to fully understand the powerful features of the language for efficient utilization. As far as possible, the component models used in these neuron circuits have been created independently, using the Verilog-AMS Language Reference Manual [50] and other publications [23, 24] as references. The operational amplifier is a highly complex device to be described completely in terms of its functioning. The model used in this thesis for behavioral designs is the one offered by Cadence in the standard libraries of version 4.1. This standard directory is available for licensed access in the Cadence toolset. A good number of op-amp parametric values have been specified in this model and these have been left unchanged in all models. Simulations performed with behavioral descriptions have been compiled in 5.2. Their corresponding source codes and test benches have been compiled in Appendix Structural models The other form of description focuses more on the individual components as opposed to the functioning of the entire system. Care is taken to design each component as carefully as possible, so that the interconnections that follow will result in proper functioning. While simple analog components like 71

77 resistors and capacitors have only been defined by their behavior, CMOS-based models including inverters and op-amps have been described in both styles. Thus, going back to the same example of the linear fixed threshold circuit (Fig 4.16), we would now use the same model of the resistance, but an entirely different model of the op-amp (Fig 4.12). A similar process of making interconnections and testing the robustness of the system is carried out eventually. While a behavioral description of the op-amp was used in our behavioral models, we now use a description of a MOS transistor provided by Cadence in its libraries. This is a Level-1 MOS and it should be noted that this has been used to merely demonstrate the ability to use Verilog-AMS for defining such structural models of complex circuits through its syntax. The ultimate advantage of using this HDL will be evident when a complete neural network can be constructed using digital and analog component libraries. For specific technologies, it has been recommended [51] to make models using specific Generic Process Design Kits (PDK) which cater to such technologies. For practical implementation, it is recommended that process parameters be obtained for the required target process using a more accurate MOS model like the bsim3v3 or bsim4 [56] (built-in models in Spectre and AMS-Designer). Models can be illustrated for technologies from 0.18 microns to 0.9 microns. The designs compiled in this thesis have consistently used 0.8 micron process. The MOS code used in our models can be easily altered to realize a PMOS or an NMOS by specifying the appropriate parameter, as has been done in the codes documented. The structural model simulations have been compiled in 5.3 and the corresponding source codes and test benches can be located in Appendix II. A special mention must now be made about the Gaussian function circuit (4.3.7). Since this model is entirely composed of MOS transistors, a behavioral description does not exist for this circuit. Only a structural description has been attempted and documented, while dual descriptions exist for all other models. This model will be discussed extensively in Chapter 5. 72

78 4.5 Design Space Many parameters have a marked effect on the functioning of a circuit. Some of these might be internal to the system while others could be extraneous parameters. Nonetheless, a designer s task is considerably complicated by having to deal with them and deduce optimized solutions. Op-amps use a DC supply voltage, typically anywhere from a few volts on up to 30 V or more. The opamp's output will be solely governed by its inputs if the power supply is a perfect DC voltage source [55]. Since no voltage source is completely ideal, a power supply s quality affects the performance of an op-amp. In our models, we have seen the effects of changing the power supply. Simulations offer freedom in choosing higher voltages for testing. Such an option is neither economical nor recommended for practical experiments. Thus considerable care needs to be taken in choosing a power supply. With simulations, we have the liberty to experiment with a range of values before deciding on what fits our needs best. Another primary issue in circuit fabrication is the area the fabricated chip occupies. Aspect ratio is the key factor here and there is growing impetus to make devices smaller and smaller. Changes in aspect ratio demonstrated that corresponding effects were not easily discernible. However, this can be attributed to the version of the MOS model used. Access to commercial Process Design Kits (PDKs) will enable a designer to come up with more viable solutions in terms of choosing the design technology as well as device sizes. On the downside, since HDL codes are not immediately synthesizable at present, one cannot ascertain the best possible feature size for a fully satisfactory end product without any risk. As far as circuit components are concerned, our choices are primarily governed by custom design and therefore, not as restricted. Particularly while making behavioral models, system behavior and deviations from desired results define the value of resistors, capacitors and other devices. Extraneous factors play a relatively smaller role here. 73

79 With simulations, we have the capacity to recreate the exact test setting for a system. This includes noise, interference and other effects. Thus, a more complete test environment definition will produce the best possible performance results, comparable to actual functioning. A major advantage that HDL modeling offers is the top-down design methodology. The entire system can be described at the highest level first. Individual blocks can then be decided and models of lower level components can be made once a working concept of the modules has been established. This lets circuit-level designers and system-level designers work together or independently, at any stage of development. 74

80 Chapter 5 Simulations 5.1 Verilog-AMS Overview In this chapter, we take a look at the simulations performed with the behavioral and structural models developed. The complete listing of program codes is present in the Appendices. Before we proceed to the simulations, a brief overview of programming styles in Verilog-AMS is presented. These concepts have been repeatedly used in our models and the explanations provided help to understand how simulations work. It is helpful to recollect that in Section , we have been introduced to the style of simulation in Verilog-AMS; a source code contains behavioral or structural definitions of all components in a circuit and instances of such components are created in the test bench and relevant interconnections, signals and other test conditions supplied The Source Code Models in Verilog-AMS are essentially constructed by creating instances of pre-defined structures, making relevant interconnections to form a complete circuit and supplying the necessary inputs to simulate the test environment. The first step therefore, is to create a source code. This code has complete definitions of all modules which are used in a particular model. A module is a behavioral or structural definition of a component or a group of components, depending on the hierarchy. Thus, for example, an RLC filter circuit would have modules to describe the resistance, capacitance and inductance as source codes. It is not necessary to have a single file which contains all these modules. It is sufficient if the required modules are present in the working library. For the source codes of our behavioral models, apart from other component descriptions, we use the description of the operational amplifier provided in its standard libraries by Cadence. The structural models have been constructed by using the Level-1 MOS model (provided at the same location) to describe a typical op-amp. This has been discussed in Section The op-amp thus constructed does not have an explicit definition of its working parameters. Instead, it is a gate-level description of 75

81 the actual physical architecture. These codes can be found in the LDV-4.1 version of the Cadence package The Test Bench It is not sufficient to merely come up with descriptions of model structures without any means of gauging their effectiveness. Hardware description languages offer the facility to simulate actual working environments where models can be tested to examine their performance, before real-time synthesis. This is achieved by creating test benches. The system developed is provided with the required inputs and other circuit parameters. In some cases, it might be required to run a simulation in a specific fashion, to achieve a target purpose. Such conditions, options and restrictions are defined in test benches. As an example, we look at the test bench developed for the unit step function (Fig 5.1) to get a better understanding of the concept (refer to Section 4.3.2) and to discuss some common coding styles. This example has been chosen for the purpose of illustration since it is the simplest model that has been simulated and the test bench contains a number of frequently-used commands and statements. The code is explained in detail below. //Unit Step Function TestBench //Behavioral description //Vikram Srinivasan Jan 31'05 `include "disciplines.vams" `timescale 10ps/1ps module opampstep(vout,vref,vin_p,vin_n,vspply_p,vspply_n); inout vout,vref,vin_p,vin_n,vspply_p,vspply_n; electrical vout,vref,vin_p,vin_n,vspply_p,vspply_n; ground vref; integer results; //Writing simulation data into text file initial begin results=$fopen("unitstep.txt"); $fdisplay(results,"unit Step Function Results"); forever # $fdisplay(results,"%t %f %f", $time, V(vin_p), V(vout)); $fwrite(results,"\n"); 76

82 $fclose(results); end //port-mapping an op-amp instance opamp opamp1(vout,vref,vin_p,vin_n,vspply_p,vspply_n); //positive terminal input vsource#(.type("pulse"),.val0(-2.5),.val1(2.5),.rise(1m))vin1(vin_p,vref); //negative terminal input vsource #(.type("dc"),.dc(1))vin2(vin_n,vref); vsource #(.type("dc"),.dc(2.5))vin3(vspply_p,vref); //vdd vsource #(.type("dc"),.dc(-2.5))vin4(vspply_n,vref); //vss endmodule Fig 5.1: Test bench for the unit step function Header File `include disciplines.vams The disciplines.vams file provided by Verilog-AMS contains a collection of commonly used natures and disciplines. Physical signal types related to one another are called natures and a collection of such natures is a discipline. The disciplines.vams file contains the electrical, thermal and mechanical disciplines among others. This is included in a code to enable the creation of modules which use such signal types. [23], chapter Module module opampstep(vout,vref,vin_p,vin_n,vspply_p,vspply_n); Modules are the fundamental building blocks of models in Verilog-AMS. They define the base structure of the component, the terminals it has, the input and output ports and other nodes which form a part of the component. The definition of a module is made by the key word module followed by the component name and a list of ports. It ends with the keyword endmodule. Thus, in the example provided, this statement defines the structure of an op-amp with an output port, a reference node, 2 input terminals and dual power supply terminals. [23], chapter 3 77

83 Ports inout vout,vref,vin_p,vin_n,vspply_p,vspply_n; electrical vout,vref,vin_p,vin_n,vspply_p,vspply_n; ground vref; The ports defined thus are now given a specific description of the manner in which they should operate. Input ports can sense a signal but not affect it. Output ports can affect the signal but not sense it. An Inout port can sense a signal as well as affect it. This illustrates clarity of design. The ports in our example have also been categorized as electrical since they are going to operate on signals of that nature. [23], chapter File Write Operation //Writing simulation data into text file initial begin results=$fopen("unitstep.txt"); $fdisplay(results,"unit Step Function Results"); forever # $fdisplay(results,"%t %f %f", $time, V(vin_p), V(vout)); $fwrite(results,"\n"); $fclose(results); end This paragraph defines a method to sample the signal at the positive input terminal (vin_p) and at the output (vout) at regular time steps and write this information into a text file. A file is accessed using an integer called the multichannel descriptor (results). This integer is used to perform operations with that particular file. The description of the process is contained by the key words begin and end [23], chapter 5. For our models, since AMS-Simulator only provides transient analysis, we have adopted this technique to sample data at incremented time steps and write this into a text file to make plots of output and input voltages using a graphing tool like MS-Excel. 78

84 Component Instantiation opamp opamp1(vout,vref,vin_p,vin_n,vspply_p,vspply_n); By passing this command, an instance opamp1 of the opamp module is created with a defined port list. It should be mentioned that the order in which port names are passed for the instance should be in the same order that has been used to define the parent module. This forms a one-to-one mapping and this process is termed port mapping. In this example, the same names have been used for the parent module and the instance Inputs vsource #(.type("pulse"),.val0(2.5),.val1(2.5),.rise(1m))vin1(vin_p,vref); vsource #(.type("dc"),.dc(1))vin2(vin_n,vref); This is an example of the definition of voltage sources and inputs. The nature of input is specified in the.type parameter. For every nature, there are certain associated parameters which need to be supplied as well. In our example, the ramp input is defined as type pulse. It increases from -2.5V to +2.5V with a 1 millisecond time step. This voltage source has been termed vin1 and it is connected between the positive input terminal (vin_p) and electrical ground (vref). For the dc source, the amplitude of the signal has been specified in the.dc parameter. It is hoped that this section will provide useful information in understanding the model codes documented in the Appendices. A presentation of simulation results is made in the following sections. 5.2 Behavioral Models For all behavioral descriptions, the source code is a single file that contains functional descriptions of all components used-the resistor, the capacitor, the op-amp etc. This file is the opampcad.vams file which is documented in Appendix 1. In this section, each discussion consists of a circuit diagram with test parameters, circuit schematic using SimVision, example waveforms from the Cadence-AMS Simulator for these parameters, corresponding HSPICE simulation and a graphical analysis of results using the Verilog-AMS model. 79

85 The U version of HSPICE and a Level-49 model of the MOS transistor has been used in the models from [49]. The Verilog-AMS models have been simulated on the Cadence-AMS Simulator version 2.0 and SimVision 4.10-s022 using a behavioral model of the op-amp and an early Level-1 MOS structural model available in the standard libraries of the Cadence toolkit. In all our simulations, we have focused on experimenting with and verifying the functionality of each model. This is done by making a comparison of threshold voltages between HSPICE simulations and Verilog simulations. Where possible, a comparison has been made with theoretical threshold values that have been derived using equations. Since we have used a behavioral description of the op-amp, only those models are tested in this section, which are op-amp based circuits. The Gaussian model is composed of MOS transistors alone and is not an op-amp circuit. The Verilog-AMS code for a transistor is a structural description and hence, the Gaussian model is discussed with the structural models and not with the behavioral models. An interesting observation is the difference in saturation voltages that we see, between the SPICE and Verilog behavioral models. This can be explained by the fact that our behavioral models describe the functionality of the circuits in terms of equations. Therefore, they are more likely to be accurate and closer to perfect performance. Hence their saturation voltages are only slightly lower than supply voltages. However, the SPICE models are structural in terms of the MOS transistors involved. Device operation is more relevant here and the performance observed will naturally not match that of a behavioral model. Since we are also adopting a different power supply for the structural models, the associated compensation network would definitely need to be adjusted. It should also be borne in mind that structural model design for SPICE simulations was made for a Level-49 MOS model while a lower level model has been used in Verilog-AMS. Thus saturation voltages might not be comparable between SPICE and Verilog The Unit Step Function The circuit diagram with test parameters used to generate the unit step function is illustrated in Fig 5.2. In the following two figures, we look at the circuit schematic of this model and sample waveforms obtained for these test values, using the SimVision tool. In Fig. 5.5, we look the corresponding HSPICE simulation obtained from [49] and Fig 5.6 shows a plot of output voltage vs input voltage using the Verilog-AMS and SPICE simulation data. 80

86 Parameter V in V ref V cc/v ss Test Value -2.5 to 2.5V ramp 1V +2.5V/-2.5V Fig 5.2: Circuit diagram with test parameters for the unit step function (behavioral model) [14] Fig 5.3: Schematic of the unit step function behavioral model using SimVision 81

87 Fig 5.4: Example waveforms of the unit step function behavioral model using SimVision (2.5V supply) Fig 5.5: SPICE plot of output vs input voltages for the unit step function behavioral model 82

88 Unit Step Function Behavioral Model Output Voltage (V) Input Voltage (V) Verilog-AMS Behavioral SPICE Structural Fig 5.6: Verilog-AMS and SPICE plots of output vs input voltages for the unit step function behavioral model The model is found to behave excellently, transitioning from low state to high state when the input voltage is 1V. Table 5.1 illustrates a comparison between theoretical threshold, SPICE and Verilog simulations. Parameter Theoretical SPICE Verilog-AMS Value Simulation Simulation Threshold Voltage 1V 1V 1V Table 5.1: Unit step function behavioral model simulation results 83

89 5.2.2 The Linear Function-Fixed Threshold Parameter V in R f V cc/v ss R g Test Value -2.5 to 2.5V ramp 800kΩ +2.5V/-2.5V 400kΩ Fig 5.7: Circuit diagram with test parameters for linear function-fixed threshold behavioral model [14] The circuit diagram with test parameters used to generate the linear function-fixed threshold model is illustrated in Fig 5.7. In the following two figures, we look at the circuit schematic of this model and sample waveforms obtained for these test values, using the SimVision tool. In Fig. 5.10, we look the corresponding HSPICE simulation obtained from [49] and Fig 5.11 shows a plot of output voltage vs input voltage using the Verilog-AMS simulation data and SPICE simulation data.. 84

90 Fig 5.8: Schematic of the linear function fixed-threshold behavioral model using SimVision Fig 5.9: Example waveforms of linear fixed-threshold behavioral model using SimVision (2.5V supply) 85

91 Fig 5.10: SPICE plot of output vs input voltages for the linear fixed threshold behavioral model Linear Function (Fixed Threshold) Behavioral Model 3 Output Voltage (V) Input Voltage (V) Verilog-AMS Behavioral SPICE Structural Fig 5.11: Verilog-AMS and SPICE plots of output vs input voltages for the linear fixed threshold behavioral model 86

92 Theoretical Threshold Voltage Values: Vth(lower and upper) = VccR1 ± R + R 2.5* 400 = ± = V, -0.83V 1 2 Threshold Voltage Theoretical Value SPICE Verilog-AMS Simulation Simulation Lower -0.83V -0.4V -0.8V Higher +0.83V +0.6V +0.8V Range (High-Low) 1.66V 1V 1.6V Table 5.2: Linear fixed threshold behavioral model simulation results The behavioral model clearly exhibits a good performance comparable to theoretical analysis, as can be seen from Table 5.2. The disparity between saturation values as well as threshold values can be attributed to the fact that the SPICE model is a structural model while the Verilog model in this case, is a behavioral model The Linear Function- Variable Threshold The circuit diagram with test parameters used to generate the linear function-variable threshold model is illustrated in Fig In the following two figures, we look at the circuit schematic of this model and sample waveforms obtained for these test values, using the SimVision tool. In Fig. 5.15, we look the corresponding HSPICE simulation obtained from [49] and Fig 5.16 shows a plot of output voltage vs input voltage using the Verilog-AMS simulation data and SPICE simulation data.. 87

93 Parameter Test Value V i -2.5 to +2.5V ramp V ref 1.5V V cc V ss R 1 R 2 R V -2.5V 40kΩ 1000kΩ 100 kω Fig 5.12: Test circuit for the linear function-variable threshold behavioral model [14] Fig 5.13: Schematic of the linear function variable-threshold behavioral model using SimVision 88

94 Fig 5.14: Example waveforms of linear variable-threshold behavioral model using SimVision (2.5V supply) Fig 5.15: SPICE plot of output vs input voltages for the linear variable threshold behavioral model 89

95 Linear Function (Variable Threshold) Behavioral Model 3 Output Voltage (V) Input Voltage (V) Verilog-AMS Behavioral SPICE Structural Fig 5.16: Verilog-AMS and SPICE plots of output vs input voltages for the linear variable threshold behavioral model Theoretical Threshold Voltage USP, LSP = R1 + R2 R1 Vref ± Vcc R2 R2 = ± = 1.4V, 1.9V Table 5.3 illustrates results which indicate that the Verilog model performs better than the SPICE model when compared to theoretical calculations, since it is a behavioral model. Threshold Voltage Theoretical Value SPICE Verilog-AMS Simulation Simulation Lower 1.4V 1.1V 1.4V Higher 1.9V 1.9V 1.9V Range (High-Low) 0.5V 0.8V 0.5V Table 5.3: Linear variable threshold behavioral model simulation results 90

96 5.2.4 The Sigmoid Fixed Gain Function Parameter Test Value V i -2.5 to +2.5V ramp V ref 1.5V V cc V ss Op-amp 1: Series Res. Op-amp 1 : Feedback Res R V -2.5V 40kΩ 1000kΩ 100 kω Fig 5.17: Circuit diagram with test parameters for the sigmoid-fixed gain behavioral model [14] The circuit diagram with test parameters used to generate the sigmoid-fixed gain behavioral circuit model is illustrated in Fig In the following two figures, we look at the circuit schematic of this model and sample waveforms obtained for these test values, using the SimVision tool. In Fig. 5.20, we look the corresponding HSPICE simulation obtained from [49] and Fig 5.21 shows a plot of output voltage vs input voltage using the Verilog-AMS simulation data and SPICE simulation data.. 91

97 Fig 5.18: Schematic of the sigmoid fixed gain behavioral model using SimVision Fig 5.19: Example waveforms of the sigmoid fixed gain behavioral model using SimVision (2.5V supply) 92

98 Fig 5.20: SPICE plot of output vs input voltages for the sigmoid fixed gain behavioral Model Sigmoid Function (Fixed Gain) Behavioral Model 3 Output Voltage (V) Input Voltage (V) Verilog-AMS Behavioral SPICE Structural Fig 5.21: Verilog-AMS and SPICE plots of output vs input voltages for the sigmoid fixed gain function behavioral model Table 5.4 illustrates results which indicate that the Verilog model performs better than the SPICE model, since it is a behavioral model. But the Verilog model seems to be making sharp transitions, rather than the smooth ones required for the sigmoid function. This phenomenon needs further study. 93

99 Threshold Voltage Theoretical Value SPICE Verilog-AMS Simulation Simulation Lower Not available -0.8V -0.1V Higher Not available 1.0V 0.2V Range (High-Low) Not available 1.8V 0.3V Table 5.4: Sigmoid fixed gain behavioral model simulation results The Sigmoid Function-Variable Gain Gain=100 The circuit diagram with test parameters used to generate the sigmoid function-variable gain model (gain=100) is illustrated in Fig In the following two figures, we look at the circuit schematic of this model and sample waveforms obtained for these test values, using the SimVision tool. In Fig. 5.25, we look the corresponding HSPICE simulation obtained from [49] and Fig 5.26 shows a plot of output voltage vs input voltage using the Verilog-AMS simulation data and SPICE simulation data. The maximum deviation and RMS deviation values have also been computed based on this comparison with SPICE simulation results. Parameter V i V cc/ V ss Op-amp1 Series Res:R 1 Op-amp1: Feedback Res:R 2 Inverter Feedback Res:R 3 Inverter Series Res:R 4 Test Value -2.5 to +2.5V ramp +2.5V/-2.5V 40kΩ 1000kΩ 100kΩ 1kΩ Fig 5.22: Circuit diagram with test parameters for the sigmoid variable gain (100) behavioral model [14] 94

100 Fig 5.23: Schematic of the sigmoid variable gain (100) behavioral model using SimVision Fig 5.24: Example waveforms of the sigmoid variable gain (100) behavioral model using SimVision (2.5V supply) 95

101 Fig 5.25: SPICE plot of output vs input voltages for the sigmoid variable gain (100) behavioral model Sigmoid Function (Variable Gain) Gain=100 Behavioral Model 3 Output Voltage (V) Input Voltage (V) Verilog-AMS Behavioral SPICE Structural Fig 5.26: Verilog-AMS and SPICE plots of output vs input voltages for the sigmoid variable gain (100) behavioral model 96

102 Table 5.5 illustrates results which indicate that the SPICE model s performance is matched closely by that of the Verilog model. Threshold Voltage Theoretical Value SPICE Verilog-AMS Simulation Simulation Lower Not available -0.1V 0.05V Higher Not available 0.1V 0.2V Range (High-Low) Not available 0.2V 0.25V Table 5.5: Sigmoid variable gain (gain=100) behavioral model simulation results Gain=500 The circuit diagram with test parameters used to generate the sigmoid function-variable gain model (gain=500) is illustrated in Fig In the following two figures, we look at the circuit schematic of this model and sample waveforms obtained for these test values, using the SimVision tool. In Fig. 5.30, we look the corresponding HSPICE simulation obtained from [49] and Fig 5.31 shows a plot of output voltage vs input voltage using the Verilog-AMS simulation data and SPICE simulation data. Parameter Vi Vcc Vss Op-amp1 Series Res:R1 Op-amp1: Feedback Res:R2 Inverter Feedback Res:R3 Inverter Series Res:R4 Test Value -2.5 to +2.5V ramp +2.5V -2.5V 40kΩ 1000kΩ 500kΩ 1kΩ Fig 5.27: Test circuit for the sigmoid variable gain (500) behavioral model [14] 97

103 Fig 5.28: Schematic of the sigmoid variable gain (500) behavioral model using SimVision Fig 5.29: Example waveforms of the sigmoid variable gain (500) behavioral model using SimVision (2.5V supply) 98

104 Fig 5.30: SPICE plot of output vs input voltages for the sigmoid variable gain (500) behavioral model Sigmoid Function (Variable Gain) Gain=500 Behavioral Model 3 Output Voltage (V) Input Voltage (V) Verilog-AMS Behavioral SPICE Structural Fig 5.31: Verilog-AMS and SPICE plots of output vs input voltages for the sigmoid variable gain (500) behavioral model Table 5.6 illustrates results which indicate that the SPICE model s performance is matched fairly well by that of the Verilog model. 99

105 Threshold Voltage Theoretical Value SPICE Verilog-AMS Simulation Simulation Lower Not available 0.00V 0.1V Higher Not available 0.15V 0.25V Range (High-Low) Not available 0.15V 0.15V Table 5.6: Sigmoid variable gain behavioral model (gain=500) simulation results Gain=1000 Parameter V i V cc V ss Op-amp1 Series Res:R 1 Op-amp1: Feedback Res:R 2 Inverter Feedback Res:R 3 Inverter Series Res:R 4 Test Value -2.5 to +2.5V ramp +2.5V -2.5V 40kΩ 1000kΩ 1000kΩ 1kΩ Fig 5.32: Test Circuit for the sigmoid variable gain (1000) behavioral model [14] The circuit diagram with test parameters used to generate the sigmoid function-variable gain model (gain=1000) is illustrated in Fig In the following two figures, we look at the circuit schematic of this model and sample waveforms obtained for these test values, using the SimVision tool. In Fig. 5.35, we 100

106 look the corresponding HSPICE simulation obtained from [49] and Fig 5.36 shows a plot of output voltage vs input voltage using the Verilog-AMS simulation data and SPICE simulation data. Fig 5.33: Schematic of the sigmoid variable gain (1000) behavioral model using SimVision 101

107 Fig 5.34: Example waveforms of the sigmoid variable gain (1000) behavioral model using SimVision (2.5V supply) Fig 5.35: SPICE plot of output vs input voltages for the sigmoid variable gain (1000) behavioral model 102

108 Sigmoid Function (Variable Gain) Gain=1000 Behavioral Model 3 Output Voltage (V) Input Voltage (V) Verilog-AMS Behavioral SPICE Structural Fig 5.36: Verilog-AMS and SPICE plots of output vs input voltages for the sigmoid variable gain (1000) behavioral model Table 5.7 illustrates results which indicate that the SPICE model s performance is not entirely comparable to that of the Verilog model. Threshold Voltage Theoretical Value SPICE Verilog-AMS Simulation Simulation Lower Not available -0.05V 0.1V Higher Not available 0.15V 0.3V Table 5.7: Sigmoid variable gain behavioral model (gain=1000) simulation results Summary Thus so far, in this section, we have seen a comparison of threshold values obtained theoretically as well as from SPICE and Verilog simulations. While the functional aspect of our Verilog models has been borne in mind, we have observed that these models perform better than SPICE models. This is primarily due to the fact that the SPICE models are structural and defined in terms of their MOS layouts while the Verilog models in this section have been described in terms of their behavior. Therefore, they attempt to perform in an ideal fashion, without taking real world physical effects and circuit parasitics into account. 103

109 5.3 Structural Models A similar presentation is made with results and graphs for the structural models developed. While a 2.5V DC supply was used to simulate the behavioral models, the transistors in structural models required a higher voltage to exhibit satisfactory performance. Hence, a power supply of 5V DC was used, which is still definitely acceptable. The source code for all op-amp models (step, linear and sigmoid) is the opampcmos.vams file while the source code for the Gaussian model is the moscad.vams file. It should be mentioned here that the Gaussian model has only a structural definition and not a functional definition as it is entirely composed of MOS devices. All files can be found in Appendix The Unit Step Function The circuit diagram with test parameters used to generate the unit step function structural model is illustrated in Fig In the following two figures, we look at the circuit schematic of this model and sample waveforms obtained for these test values, using the SimVision tool. In Fig. 5.40, we look at the corresponding HSPICE simulation obtained from [49] and Fig 5.41 shows a plot of output voltage vs input voltage using the Verilog-AMS simulation data and SPICE simulation data. The maximum deviation and RMS deviation values have also been computed based on this comparison with SPICE simulation results. Parameter Test Value V in V ref V cc/v ss -2.5 to 2.5V ramp 1V +5V/-5V Fig 5.37: Circuit diagram with test parameters for the unit step function (structural model) [14] 104

110 Fig 5.38: Schematic of the unit step function structural model using SimVision Fig 5.39: Example waveforms of the unit step function structural model using SimVision (5V supply) 105

111 Fig 5.40: SPICE plot of output vs input voltages for the unit step function structural model Unit Step Function Structural Model 6 Output Voltage (V) Verilog-AMS Structural SPICE Structural -6 Input Voltage (V) Fig 5.41: Verilog-AMS and SPICE plots of output vs input voltages for the unit step structural model 106

112 Parameter Theoretical Value SPICE Verilog-AMS Simulation Simulation Threshold Voltage 1V 1V 1V Table 5.8: Unit step function structural model simulation results The output of the Verilog structural model exhibits a transition to high saturation state when the input voltage is exactly 1V. This is in accordance with the SPICE model behavior and with theoretical threshold voltage The Linear Function-Fixed Threshold Parameter V in R f V cc/v ss R g Test Value -2.5 to 2.5V ramp 800kΩ +5V/-5V 400kΩ Fig 5.42: Circuit diagram with test parameters for linear function-fixed threshold structural model [14] The circuit diagram with test parameters used to generate the linear function-fixed threshold model is illustrated in Fig In the following two figures, we look at the circuit schematic of this model and sample waveforms obtained for these test values, using the SimVision tool. In Fig. 5.45, we look the corresponding HSPICE simulation obtained from [49] and Fig 5.46 shows a plot of output voltage vs input voltage using the Verilog-AMS and SPICE simulation data. 107

113 Fig 5.43: Schematic of the linear fixed threshold function structural model using SimVision Fig 5.44: Example waveforms of the linear fixed threshold function structural model using SimVision (5V supply) 108

114 Fig 5.45: SPICE plot of output vs input voltages for the linear fixed threshold structural model Linear Function (Fixed Threshold) Structural Model 6 Output Voltage (V) Input Voltage (V) Verilog-AMS Structural SPICE Structural Fig 5.46: Verilog-AMS and SPICE plots of output vs input voltages for the linear fixed threshold structural model 109

115 Theoretical Threshold Voltage Values: Vth(lower and upper) = VccR1 ± R + R 1 2 5* 400 = ± = V, -1.67V Threshold Voltage Theoretical Value SPICE Verilog-AMS Simulation Simulation Lower -1.67V -1.3V -1.35V Higher +1.67V +1.2V +1.35V Range (High-Low) 3.34V 2.5V 2.7V Table 5.9: Linear fixed threshold structural model simulation results As can be seen from Table 5.9, the Verilog-AMS structural model closely follows the SPICE simulation results in terms of threshold voltages The Linear Function-Variable Threshold The circuit diagram with test parameters used to generate the linear function-variable threshold structural model is illustrated in Fig In the following two figures, we look at the circuit schematic of this model and sample waveforms obtained for these test values, using the SimVision tool. In Fig. 5.50, we look the corresponding HSPICE simulation obtained from [49] and Fig 5.51 shows a plot of output voltage vs input voltage using the Verilog-AMS simulation data and SPICE simulation data. 110

116 Parameter Test Value Vi -2.5 to +2.5V ramp Vref 1.5V Vcc +5V Vss -5V R1 R2 R3 40kΩ 1000kΩ 100 kω Fig 5.47: Test circuit for the linear function-variable threshold structural model [14] Fig 5.48: Schematic of the linear variable threshold function structural model using SimVision 111

117 Fig 5.49: Example waveforms of the linear variable threshold function structural model using SimVision (5V supply) Fig 5.50: SPICE plot of output vs input voltages for the linear variable threshold structural model 112

118 Linear Function (Variable Threshold) Structural Model 6 Output Voltage (V) Input Voltage (V) Verilog-AMS Structural SPICE Structural Fig 5.51: Verilog-AMS and SPICE plots of output vs input voltages for the linear variable threshold structural model Theoretical Threshold Voltage USP, LSP = R1 + R2 R1 Vref ± Vcc R2 R2 = ± = 1.15V, 2.15V Table 5.10 illustrates results which show a difference between Verilog and SPICE models. While the Verilog model is an elementary structural model, it would also help to remember that circuit parasitics and external circuit components (in SPICE) could affect performance too. Threshold Voltage Theoretical Value SPICE Verilog-AMS Simulation Simulation Lower 1.15V 0.9V 1.25V Higher 2.15V 2.1V 2.10V Range (High-Low) 1V 1.2V 0.85V Table 5.10: Linear variable threshold structural model simulation results 113

119 5.3.4 The Sigmoid Fixed Gain Function Parameter Test Value V i -2.5 to +2.5V ramp V ref 1.5V V cc +5V V ss -5V Op-amp 1: Series Res. 40kΩ Op-amp 1 : Feedback Res 1000kΩ R kω Fig 5.52: Test circuit for the sigmoid-fixed Gain Structural model [14] The circuit diagram with test parameters used to generate the sigmoid-fixed gain structural circuit model is illustrated in Fig In the following two figures, we look at the circuit schematic of this model and sample waveforms obtained for these test values, using the SimVision tool. In Fig. 5.55, we look at the corresponding hspice simulation obtained from [49] and fig 5.56 shows a plot of output voltage vs input voltage using the Verilog-AMS simulation and SPICE simulation data. 114

120 Fig 5.53: Schematic of the sigmoid fixed gain function structural model using SimVision Fig 5.54: Example waveforms of the sigmoid fixed gain function structural model using SimVision (5V supply) 115

121 Fig 5.55: SPICE plot of output vs input voltages for the sigmoid fixed gain structural model Sigmoid Function (Fixed Gain) Structural Model Output Voltage (V) Input Voltage (V) Verilog-AMS Structural SPICE Structural Fig 5.56: Verilog-AMS and SPICE plots of output vs input voltages for the sigmoid fixed gain structural model 116

122 Table 5.11 illustrates results which indicate that the threshold voltages with the Verilog model are fairly close to the SPICE model values. Again, a better performance can be expected by using higher-level transistors in the Verilog design. Threshold Voltage Theoretical Value SPICE Verilog-AMS Simulation Simulation Lower Not available -0.2V -0.15V Higher Not available 0.2V 0.2V Range (High-Low) Not available 0.4V 0.35V Table 5.11: Sigmoid fixed gain structural model simulation results The Sigmoid Function-Variable Gain Parameter Test Value V i -2.5 to +2.5V ramp V cc +5V V ss -5V Op-amp1 Series Res:R 1 40kΩ Op-amp1: Feedback Res:R kΩ Inverter Feedback Res:R 3 100kΩ Inverter Series Res:R 4 1kΩ Fig 5.57: Test circuit for the sigmoid variable gain (100) structural model [14] 117

123 Simulations for the structural model of the sigmoid function circuit with variable gain could not be performed with the circuit specifications illustrated in Fig The op-amp requires a minimum of a 5V supply voltage, which explains the change in the test circuits between the behavioral and the structural models. While attempting simulations, it was observed that convergence could not be achieved due to the presence of jump discontinuities. These are common in circuits which have positive feedback loops (such as the one in this case). Capacitors can slow down these jumps and attempts were made to include capacitances at all nodes and ground them. However, while this helped to complete a simulation, it required capacitances as large as 10 µf and this could adversely affect circuit behavior. Spectre provides a convenient way to include grounding capacitances by setting the cmin parameter [51]. This option will place capacitances at all nodes. A similar option was not found on the AMS- Designer at the time this thesis was written. A better understanding of how amplifiers work with jump discontinuities would aid in avoiding this problem and figuring out an alternative solution. However, models with positive feedback of this nature (with inverting terminal grounded) are few in number and the amount of information on this topic is limited at present The Gaussian Function Parameter Test Value Ix 30µA Capacitor 8pF Vcc,Vss Vin +2.5V,-2.5V -3 to +3V ramp Vw -1V Fig 5.58: Circuit diagram with test parameters for the Gaussian function structural model [46] 118

124 The circuit diagram with test parameters used to generate the Gaussian function structural circuit model is illustrated in Fig In the following two figures, we look at the circuit schematic of this model and sample waveforms obtained for these test values, using the SimVision tool. In Fig. 5.61, we look at the corresponding HSPICE simulation obtained from [49] and Fig 5.62 shows a plot of output voltage vs input voltage using the Verilog-AMS and SPICE simulation data. Fig 5.59: Schematic of the Gaussian function structural model using SimVision 119

125 Fig 5.60: Example waveforms of the Gaussian function structural model using SimVision (2.5V supply) Fig 5.61: SPICE plot of output vs input voltages for the Gaussian function structural model 120