Evolving Finite State Machines for the Propulsion Control of Hybrid Vehicles JONAS HELLGREN and MATTIAS WAHDE Div. of Machine and Vehicle Design, Div. of Mechatronics Chalmers University of Technology, SE-42 96, Gothenburg, SWEDEN Abstract: Hybrid Vehicles (HVs) are proposed as a candidate to conventional vehicles. Characteristic for HVs is the use of a temporary energy buffer. But the control of the Primary Power Unit (PPU), e.g. an otto engine or a fuel cell, is much more complex than in conventional vehicles. This paper deals with a method to design a control algorithm for the PPU in a HV. Here, a Finite State Machine (FSM) is used as the control architecture and is trained using an evolutionary algorithm. A control sequence is derived from a simplified vehicle model and used to train the FSM. The evolved FSM is then used to control a much more accurate computer model of a vehicle. A special technique, which is introduced in this paper, has been developed, which considerably increases the speed of evolution. A realistic example of an implementation of a control algorithm in a virtual prototype is also presented. Key-words: Hybrid vehicle, Control strategy, Control algorithm, Propulsion control, Finite State Machine, Evolutionary algorithm. Introduction The major difference between conventional vehicles and Hybrid Vehicles (HVs) is the presence of a buffer for temporary storage of energy in HVs. The buffer enables regenerative breaking and the Primary Power Unit (PPU) to be more or less decoupled from the wheels. This gives a potential to reduce fuel consumption and emissions. In this paper a series hybrid bus is in focus. The term series hybrid implies that the PPU can work independently of the wheels, since there is no mechanical connection between the wheels and the PPU. Driveline Synthesis (DS) [] is a method that automatically suggests a driveline configuration, e.g. a series hybrid vehicle, with components (PPU and buffer) of predetermined sizes. Also a control strategy (not implementable) is derived from DS. The aim of this work is to generate a method that creates a control algorithm, that is implementable, based on the result from DS. If this is done, it can be shown that the suggested driveline can be controlled in practice. In addition, the possibility to validate the DS is hereby achieved. Developing a good control algorithm for HVs is a very complex problem. Such an algorithm should be: intelligent, i.e. control the vehicle in a competent way. The objective can be low fuel consumption, low emissions, small wear of components, driveability or a combination of issues; implementable, i.e. possible to use it in practice. Only signals that are possible and, preferably, easy to measure or estimate should be used. The control algorithm also needs to be robust, i.e. insensitive to unpredictable events in the environment. In general, control strategies for HVs are divided into Load following and On/off strategies. In the case of a Load following strategy, the power from the PPU is closely related to the power demand from the chassis. If the strategy is of On/ off type the power from the PPU is related to the State Of Charge (SOC) in the buffer; at some specific SOC, the PPU is turned on or off. The SOC is a measure of the available energy in the buffer. In practice, a combination of the strategies might be the most appropriate choice. There are several alternatives available for performing the control of a HV. A neural network has been used for a HV project within NASA [2], an analytic expression has been developed by [3] and a strategy based on Fuzzy logic is carried out in [4].
The design process is illustrated, in general terms, in Figure. The overall purpose is to transform customer demands into a HV. The problem in focus in this paper is introduced in box C: evolving an Finite State Machine (FSM) that can control a virtual prototype. By imitating a control sequence an FSM is trained to perform well in a specific situation. The control strategy from DS is not implementable because it contains abstract information such as the recommended power from the PPU is 00 kw and the maximum change of SOC is 4%. The control sequence is not implementable either, since it does not explicitly recommend what action to take in a specific situation. Acc [m/s 2 ] Pch [kw] Velocity[km/h] 0 Time[s] 0.5 35 30 25 20 5 0 Fig. 2: Driving cycle. 0 0.5 00 0 00 200 5 Demands (driving cycle, number of passengers, etc.), conditions (fuel price, discount rate, etc.) DS Virtual prototype A D Control strategy Suggested driveline FSM Generate control sequence Evolve FSM Control sequence Fig. : Overview of the design process. An FSM is proposed as the control architecture. The following control signals are chosen here: the acceleration of the vehicle (Acc), the power demand from the chassis (Pch) and the SOC. This choice is debatable, but it is beyond the scope of this paper to find the best choice. The output is the desired power from the PPU (P PPU ). Figure 2 shows the driving cycle. The control sequence used here to evolve FSMs is illustrated in Figure 3. Note that, in order to achieve a more robust control algorithm, the acceleration, deceleration, and time for moving at constant speed differ between different parts in the driving cycle. By training the control algorithm to handle several situations, a more robust result is obtained. B C SOC Pppu [kw] 0.8 0.6 0.4 0.2 60 40 20 Time[s] Fig. 3: Control sequence, P PPU is the output signal. 2 Structure of proposed FSM The structure of the FSM is influenced by the type of transition described in [5] and the modular approach proposed by [6]. The main principle is that when a specific state is active an output related to that state will be generated. The state will remain active unless some condition is fulfilled, which occurs when an input signal (here represented by a floating-point number) reaches a predefined value. The modular approach implies that an FSM can be divided into separate modules containing several states each. Modularity is introduced in order to: obtain FSMs that are easier to interpret, obtain a simpler structure with fewer states and transitions.
Figure 4 shows an example of the structure of an FSM. The rules for transitions between modules are described in Appendix. 3 2 Mod (main) Mod 2 Mod 3 Mod Mod 2 Mod 3 2 Mod 3 Mod Mod 2 = State = Transition = Other module Fig. 4: Example of FSM. The first state in module is the start state. Each state is associated with one or more transitions. A transition occurs when a condition is satisfied. The transitions in a state have different priorities, as shown in Figure 5. The number of transitions per state may differ between states and is determined by the evolutionary process. Out = 34 Transition (with highest priority) ins < 0.3 ins 3 < 8 ins 2 >.2 Mod/State= / Mod/State= /2 This line is normally not drawn Mod/State= 2/ Fig. 5: An example of transitions associated with a state. Ins i denotes the input signals. 3 Training of FSMs An evolutionary algorithm has been used for obtaining the FSMs. Evolutionary algorithms are very useful tools in optimization problems where the search space is large or has a complicated structure, and they function even in the absence of a differentiable error surface, i.e. in situations where a gradient cannot be defined. For a general introduction to the use of evolutionary algorithms in connection with FSMs, see [7]. Initially, a population of random FSMs is generated. The FSMs are evaluated by measuring their ability to produce the correct output signal for a given input signal. Those FSMs, which perform best, are more likely to be selected, and thus propagate their features to subsequent generations of FSMs. Our evolutionary algorithm resembles the evolutionary programming approach used by Fogel [7]. The FSMs are evolved with the aim of imitating a control sequence, as indicated in Figure. The control sequence is obtained using an inverse dynamic vehicle model, i.e. a model in which the state of the system is computed from the output of the system. In the case of a HV, examples of states are the power from the PPU and the SOC in the buffer. The output can be the driving cycle, generally described as speed as a function of time. The procedure for training FSMs consists of the following main steps (described in detail below): preprocessing a list containing transition data, (hereafter: LOT, short for List Of Transition data). The motivation for creating a LOT is to speed up the evolution of the FSMs. This is achieved since the LOT supports the evolutionary process in its search for a proper transition by providing a set of elementary building blocks that can be used by the evolutionary algorithm. Running an evolutionary algorithm, with support from the LOT. 3. Performance measure In evolutionary algorithms, a performance measure known as fitness is defined. In this case, individuals that imitate the control sequence accurately and have a simple structure are given high fitness. The following fitness function is used here: where: ε fitness = --------------------- + ε f( Σ) the difference between the output from the FSM and the desired (given) output
f Σ f increases linearly with Σ 3.2 Preprocessing a LOT The algorithm for generating a LOT is described in Figure 6. As an example, consider the training sequence in Figure 7. For the first step (A), the following transition data is added to the LOT: condition = ins > 2.99; first output signal = 4; second output signal = 5; input signals = 3,. A simple evolutionary algorithm (different from the one used when evolving the FSMs) generates the condition part of the transition data. The remaining three parameters in the transition data are used to avoid duplicates of transition data elements. At step B, data will also be added to the LOT. At step C, no element will be added to the LOT, since this step is similar to step A (the first and second output signals, as well as the input signals are equal to the corresponding transition data in step A). At step D two elements will be added to the LOT, since both ins and ins 2 change. Start the total number of transitions in the FSM. This reflects the complexity of the FSM. Find next step in output signal End of output signal? no Similar in/output signals found in LOT? no * Generate transition data * Add transition data to LOT yes yes Stop yes Fig. 6: Algorithm for generating a LOT. no More input signals that change during step? 5 output signal A B C D ins ins 2 Fig. 7: Example of out/input signal sequence for the training of an FSM. 3.3 Evolving FSMs In the evolutionary process, the existing FSMs are changed using mutations, which are made by randomly selecting a mutation operator from the set: {add module, delete module, add state, delete state, add transition, delete transition, modify transition}. Table contains a brief description of all the available operators. The probability for an operator to be chosen is roughly inversely proportional to the change in the structure of the FSM caused by the operator. For the operators add transition and modify transition there is a possibility to choose a transition randomly, since there might be situations when no proper transition data is present in the LOT. If a transition is defined with support from the LOT, only elements in the LOT whose first output signal (defined in section 3.2) equals the output of the active state can be selected. The allowed destination module and state are functions of the second output signal in the transition data. Only the maximum number of modules, states and transitions are defined in advance. Thus, the structure of the best FSM is a result of the evolutionary process. The parameter settings used for evolving FSMs are defined in Appendix 2.
OPER- ATOR Add module Table : Operators for evolving FSMs. P is the probability for an operator to be chosen. Delete module Add state Delete state Add transition Delete transition Modify transition TASK (P) Adds a module to the FSM. The numbers of states and numbers of transitions in the states are random. (/2) Deletes a randomly chosen module. Transitions pointing to the deleted module are reassigned to a random module (of remaining modules). (/2) When adding a state a random number of transitions are also created. The output signal from the state can only be one of the values present in the given output signal. (/2) Like the case of deleting a module, transitions have to be reassigned to a random state (of remaining states). (/2) A transition is added to a randomly chosen state. The priority of the added transition is random. (2/2) This operator is similar to add transition but deletes a transition instead. (2/2) A randomly chosen transition in the FSM is selected and changed. (4/2) 4 Implementation in a virtual prototype When an FSM has been evolved, it can be used in a virtual prototype, consisting of an accurate vehicle model developed in Modelica [8] (a language for modelling physical systems). This is done by transforming the FSM to a file format that can be interpreted by the virtual prototype. Since the model is fully dynamic, its behavior is very similar to that of a real vehicle. A virtual driver gives a torque request to the electric motor. The torque request is related to the difference between actual and desired speed. As stated in section, Acc, Pch, and SOC are used as control signals. In practice, the acceleration may be interpreted as the torque demand of the driver. The power from chassis and the SOC can be measured or estimated. In this work, a series hybrid city bus is chosen as an example. Table 2 gives data about the bus, and Figure 8 illustrates the layout of the model. Table 2: Data for the series hybrid city bus. TERM Total weight Type and size of the PPU Type and size of the buffer Type and size of the electric motor DESCRIPTION 7 000 kg Diesel engine, 00 kw Super capacitor, 300 kg, 300 kw, 3 MJ Asynchronous motor, 00 kg, 200 kw Fig. 8: Model of a series HV. The PPUCtrl component interprets the data files containing the FSM. 5 Results According to Figure 9, an almost perfect FSM is found. There is, however, a slight deviation from the desired output signal. This can be explained by the fact that some time elapses between the fulfillment of a condition and the generation of the new output signal. For clarity, only a part of the sequence is shown, but the deviation (called Error, in Figure 9) is very small also in the rest of the sequence.
Outsignal, OutFSM [kw] Error [kw] 60 40 20 40 20 0 20 50 200 250 300 350 400 450 500 50 200 250 300 350 400 450 500 Time [s] Fig. 9: In the top panel, the output signal from the evolved FSM (dashed) is compared to the desired output signal. The difference (Error) between the output from the evolved FSM and the desired output signal is shown in the lower panel. Figure 0 shows the driving cycle and the chassis power versus the power from the PPU (equal to the output signal from the FSM). The SOC, as a function of time, is also shown in Figure 0. As is evident from Figure 0, the FSM controls the PPU in a sensible and desired way. It is interesting that the PPU supports the buffer during acceleration and delivers low power during deceleration. This agrees well with common sense. Figure illustrates the evolved FSM. States one and two (with output signals equal to 9.77 and 25.77, respectively) can be interpreted as standstill states since they become active after deceleration. Whether state one or two becomes active after deceleration depends on the SOC. In the same way, states three and four (with output signals equal to 60.5 and 39.44) can be interpreted as motion states. The two remaining states, five and six, are deceleration (with output signal equal to 7.3) and acceleration states (with output signal equal to 60.5). The fuel consumption of the HV is, according to the simulation, 7 liters/0 km. In [2] it is stated that the fuel consumption of a conventional diesel bus is 9 liters/0 km for the same driving cycle. This indicates that HVs have a potential to reduce substantially fuel consumption and that the control algorithm behaves intelligently. Velocity [m/s], SOC [%] Power [W] 7.3 39.44 SOC Output signal from FSM Time [s] Chassis power Velocity Fig. 0: Results from a fully dynamic simulation performed in Modelica. acc.> 0.33 acc.< 0. Pch.> 8.0 SOC.< 0.34 60.5 60.5 Pch.> 20.7 Pch.< 26.2 acc.< 0.22 SOC.> 0.45 Fig. : Shape of the evolved FSM. It contains one module with 6 states. The output signal is given in kw. 6 Discussion The method described in this paper and illustrated in Figure, is very promising since the driveline and the control algorithm are automatically generated. When the computer is the designer there is no risk of negative influence due to human limitations and/or prejudices. The special technique of preprocessing transitions, 9.77 25.77 SOC.< 0.77 SOC.> 0.83 acc.> 0. acc.> 0.2
that is introduced in this paper, considerably decreases the time needed to evolve the FSMs. If the operators do not use the LOT, the number of evaluations needed increases approximately by a factor 00. A major advantage compared with, for example, neural networks is that FSMs can be interpreted and modified in an intuitive and physical way. 7 Future work The results obtained so far are very encouraging. Some future developments will now be introduced and briefly described. 7. Introducing more complex driving cycles In order to meet more realistic scenarios, the method described in this paper will be applied to more complex driving cycles which may include, for instance, a highway part. 7.2 Increasing the robustness In order to achieve more robust FSMs, that are able to cope with any realistic driving situation, the evolutionary algorithm should be applied to a larger training data set, consisting of data from several different situations. For example, the performance of the FSM shown in Figure will be less than perfect if the vehicle is going uphill. 7.3 Introducing adaptability The use of FSMs gives the possibility to improve the control algorithm after the design of the FSM, i.e. when the vehicle is in duty. The principle is to change some appropriate characteristic of the FSM, for example the output signal for some specific state. For instance, a successful change may result in lower fuel consumption. Since the change is based on real measurements, it can compensate for simplifications made in the vehicle model. 7.4 Introducing other HV types and signals The aim is to develop implementable control algorithms for other HV types in a way similar (in principle) to that described in this paper. In [9], different HV types are described further. Also, the affect of choosing other signals should be investigated. Acknowledgements The authors would like to thank several colleagues, especially Bengt Jacobson, for interesting discussions during the work. Also Margit deserves a thank for careful reading of the manuscript and useful comments. References [] J. Hellgren and B. Jacobson, A Systematic Way of Choosing Driveline Configuration and Sizing Components in Hybrid Vehicles, SAE 00FTT_23, 2000. [2] J.C. Brown, Baseline Testing of the Hybrid Electric Transit Bus, NASA/ TM-999-208890. 999. [3] C. Andersson, Simulation and verification of a hybrid bus, Nordic Workshop on Power and Industrial Electronics, Aalborg, Denmark, 2000. [4] E. Cerruto, A. Consoli, A. Raciti, A. Testa, Energy Flows Management in Hybrid Vehicles by Fuzzy Logic Controller, IEEE, 994. [5] M. Wahde and H. Sandholt, Evolving Complex Behaviors on Autonomous Robots, Mechatronics Forum International Conference, 2000. [6] K. Chellapilla and D. Czarnecki, A Preliminary Investigation into Evolving Modular Finite State Machines, Proc. of the 999 Congress on Evolutionary Computation, Washington D.C, USA, Jul. 6-9, 999. [7] L.J. Fogel, Intelligence through simulated evolution, Wiley, New York, 999. [8] Modelica, www.modelica.org, 200. [9] H. Wallentowitz and J.W. Biermann, Strukturvarianten von Hybridantrieben, VDI BERICHTE, NR. 459, 999.
Appendix - Rules for transition In a transition to module i start with state in this module if: condition : module i- calls module i (i- < i, i.e. i- has higher rank) or condition 2: module i+ calls module i (i+ > i, i.e. i+ has lower rank) and module i has not called module i+ after module i- calls module i+ (where i- is a module of higher rank than both i and i+). Consider now the FSM in Figure 4. In this figure, module has higher rank than module 2 which, in turn, has higher rank than module 3. An example of the first condition above occurs if the FSM jumps to module 2 from module. As a result of this particular transition, the FSM will always end up in state of module 2. The second condition can be illustrated in a similar way, but this will not be done here. If neither of the two conditions above are satisfied, the FSM will simply return to the location from which is previously left module i, and resume execution from this point. As an example, consider again the FSM in Figure 4. If the FSM jumps to module 3 from state 3 in module, then, if it returns to module, it will follow the link leading back to state 3, rather than jumping to state in module. Appendix 2 - Parameter settings Number of individuals: 00 Number of subpopulations (to prevent the solution from getting stuck in a local minimum): 0 Probability for defining a transition randomly: 0.5 Maximum number of states: equal to the number of different output signals Maximum number of modules: 3 Maximum number of transitions per state: 5