Neural pattern recognition with self-organizing maps for efficient processing of forex market data streams Piotr Ciskowski, Marek Zaton Institute of Computer Engineering, Control and Robotics Wroclaw University of Technology Abstract. The paper addresses the problem of using Japanese candlestick methodology to analyze stock or forex market data by neural nets. Self organizing maps are presented as tools for providing maps of known candlestick formations. They may be used to visualize these patterns, and as inputs for more complex trading decision systems. In that case their role is preprocessing, coding and pre-classification of price data. An example of a profitable system based on this method is presented. Simplicity and efficiency of training and network simulating algorithms is emphasized in the context of processing streams of market data. 1 Introduction Neural networks have been widely used in solving problems from financial domain ([1]). Among such tasks as credit assessment, fraud detection, option pricing etc., a wide spectrum of applications is devoted to price timeseries analysis and forecasting. Solutions found in literature use various types of neural nets to forecast future values of the analyzed instrument or to provide a good trading decision. An example of methodology of building trading systems using neural nets may be found in [4]. Neural nets have also been used for pattern classification of stock market data, some methods even used Japanese candlesticks (e.g. [3]), however in not so straight and clear form as our method. Our approach does not consider forecasting itself, we do not use neural nets as black boxes to predict future from a set of present and past market data (prices, indicators etc.). We aim at dividing the decision process into functional blocks or stages, for which different specialized types of neural nets may be used as assistance. The idea is close to modeling intellectual activity of humans, in this case technical analysts, for whom the visual and geometrical analysis of patterns on charts is only one specific (often unintentional) activity in the whole decision process, which either precedes other activities, influences them, or provides a context for them. In this paper we focus on only one such activity - the analysis of single candles and small groups of candles on the chart. Particularly, self organizing maps This work is supported by KBN grant in the years 2006-2009
2 Piotr Ciskowski, Marek Zaton will be used as tools for recognizing patterns in price timeseries and for coding the recognized formations as inputs for further steps of decision making. Their role in the whole decision process may be considered as preprocessing, compression, or initial classification of candle data. Along that task, self organizing nets will provide us with clear and coherent maps of formations for the analyzed instrument, preserving the density, topology and similarity of candle formations on a particular chart. Further in the paper we will present a trading system, based on a feedforward neural net, taking signals from the above mentioned self organizing maps. The design and performance of that net is not the main scope of our work, however. We would like to focus on the Japanese candlestick methodology, self-organizing maps as a tool for recognizing patterns, and joining these two techniques for efficient decision tools. We will operate on the largest and most fluent, unregulated market for exchanging derivatives on currencies, commodities, metals, stocks and indices - the forex market, while the proposed solution may be applicable to all markets. 2 Neural nets for processing price data streams We are looking at the problem of stock/forex market pattern recognition also from the viewpoint of processing intensive data streams. Indeed the price data, here called the quotes, arrive at the trading platform in the form of an intensive stream. The prices on the most popular and fluent instruments (EURUSD and other major currency pairs, most popular commodities, or stock indices) change every few seconds or even a few times in each second. The methods of analyzing these data must be simple and fast. For that reason the candlestick analysis is generally a right tool, as it performs analysis of past candles only once at the beginning of each candle (for the rest of the interval of the current candle it waits until the candle is eventually formed). That may be 1 day, 1 hour, but also 1 minute, depending on the trader s strategy. Additionally one trading platform may work on many instruments and intervals, for all of which a fast response is needed. Moreover, mobile trading platforms are gaining more and more popularity. For this kind of applications fast and simple algorithms, consuming less computational power, are the key points to successful use. Self organizing maps, described in this paper, provide such efficient tools, they may also be used as a preprocessing stage for simplifying further steps of other decision algorithms. In this case, along with pre-classification of single candles or short formations, they provide data reduction - the four prices of each candle are reduced to a pattern position on the map. 3 Japanese candles for analyzing price data Two main ways of analyzing markets are: fundamental and technical analysis. The latter relies only on the charts - geometrical, visual, and recently also computational analysis of the history of prices. Its principles fit exactly into the idea of neural net training. Technical analysis assumes that: - price reflects all
Neural pattern recognition of forex data streams 3 available information, - history repeats itself, - prices move in trends. For technical analysts the patterns recognized in the past are a valuable clue to predict probable scenarios of market moves in the future. These scenarios are built on psychological analysis of traders and on statistical analysis of past data. Although technical analysis is to some extent subjective to the person performing it, it is much more appropriate for neural automation than fundamental analysis. In addition, historical training data are easy to collect. For many years technical analysts used only the Western techniques to analyze stock and forex markets. These methods were based on traditional charts (linear or bar), a set of measures called technical indicators, theories such as Elliot waves or Fibonacci levels, and formations drawn by prices on the two mentioned kinds of charts. In 1990s the Eastern methodology of Japanese candles was introduced and popularized among Western investors, mostly due to Steve Nison ([2]). It is based on a special kind of a chart. open high close low wicks high open body close low high close open low 30% 45% 25% Fig. 1. The structure of a Japanese candle (both middle), compared to a bar (left). Percentage notation of one candle s structure supplied to the SOM inputs (right) Let us briefly compare the three types of charts used to visualize the market. The simplest linear chart shows only the closing prices for each interval, providing an image of longer market moves, without showing the structure of single candles. Bar and candlestick charts visualize the open, high and low price of each interval. The candle chart is more precise and capable, due to adding color to each bar - white for bullish and black for bearish. The structure of bar and candle is compared in fig. 1. The latter shows the market s struggle, condition and mood during each interval. One-, two- and three-candle formations, built upon psychological background (often called the crowd psychology ), may signal important moments, such as turns, beginnings and ends of impulse and corrective waves etc. A few examples of trading signals based on candle formation analysis are presented in fig. 2. This illustrative example, although performed a posteriori (with the knowledge of each formation s consequences), shows that candle formations occur not incidentally in specific moments in time. In reality, we do not know the right hand side of the chart while analyzing the current candle. We study the charts not to predict the future, but to provide us with possible scenarios of market behavior, to chose the most probable of them, while still be prepared (with trade management techniques) for the other ones. The analysis should be based on various methods and indicators. Pattern recognition of Japanese candle formations should be one of them. A reliable method for au-
4 Piotr Ciskowski, Marek Zaton tomatic recognition and classification of these formations would provide us with a strong confirmation of other signals and important clues for optimal decisions. Such a tool will be provided by self organizing maps. Fig. 2. Candle chart, with example of trading signals based on Japanese candlestick formations The tradition of Japanese candles is long and the number of formations that have been recognized is large. There are about a dozen major formations - well known continuation and reversal patterns, among them: hammer, shooting star, dark cloud, morning star (no. 1, 4, 8, 7 in fig. 2). The names of many of them are very illustrative and poetic. For an in depth geometrical, psychological and tactical analysis of patterns, see [2]. All candle formations may be analyzed not sooner than after their last candle is formed, that is at the beginning of the next candle. It is also taken as a rule that candle formations should be confirmed by other signals or should confirm signals coming from other techniques (e.g. Fibonacci levels or Elliot waves). Generally speaking, the more signals occur at some time, the higher the probability of a specific price move. 4 Neural pattern recognition of Japanese candles Most applications of neural nets to analyze market timeseries are based on Western methods - that is on different indicators calculated on price data. These data are easy to calculate or to obtain from trading platforms. Traders and researchers use neural nets in their attempts to find dependencies between values of some factors (e.g. indicators) and future prices or proper trading decisions. The analysis of Japanese candles is more geometrical and visual, closer to modeling human
Neural pattern recognition of forex data streams 5 perception and intuition rather than strict functional dependency. Most technical indicators are delayed in time as based on moving averages. The analysis of candle formations is focused on current market behavior - mostly on the previous candle right after it finishes forming its shape. The emphasis is put not only on closing or average values for each timeframe, but also on the structure of the market movement during each period. We chose self organizing maps (SOMs) for the task of recognizing and coding Japanese candle formations. First of all, these nets learn in unsupervised manner, so no human recognition will be needed to classify formations while preparing training data. The preprocessing of training data will include only a special way of scaling. Secondly, these nets discover clusters of data points in training data. They imitate the density and topology of these data. Therefore they will adapt only to the formations present on charts for the selected currency and timeframe. It is even possible that they will discover new patterns, not described as formations yet, characteristic only for the given instrument. Similar formations will be placed closely to each other on the map. After training the map and labeling its neurons, the winners positions will clearly indicate the recognized pattern and will be easy to code for further steps of the decision process. Another reason for using SOMs is that their training and functioning algorithms are very simple, providing good performance on streams of data. 5 Self organizing maps for candle pattern recognition We use self organizing maps in their traditional form - a layer of linear neurons with connections to all inputs, arranged on a plane. The neighborhood function uses hexagonal neighborhood topology. When an example is shown to the net during training, the winner - determined by the maximum output value - and its neighbors adapt their weights moving them closer to the inputs. Therefore the neurons neighboring each other in the map learn similar patterns. An important issue is how the way candle data are applied to the SOM. The basic data of each candle are: O - open, C - close, H - highest, and L - lowest price during the candle s interval. These four values define the candle s position on a chart and its shape - the proportions of its body, upper and lower wicks. We have decided to describe the candle s shape in percents - the length of its body and both wicks as parts of the whole candle s length, as shown in fig. 1 (right). The sign of the body s length defines its direction - positive for bullish and negative for bearish candle. Additionally the price movement between the last two bars should is given on net s inputs to indicate the difference between two adjacent candle s positions. We have also used SOMs for discovering patterns in two- and three- candle formations. In that case two and three candle windows were supplied on the nets inputs using the same notation. All neural nets presented in this paper were implemented using MATLAB Neural Network Toolbox. Self Organizing Feature Map from that package was trained in a standard unsupervised way. For one-candle formations nets of 16, 25 and 36 neurons in the layer were trained, for two-candle formations - nets of 49,
6 Piotr Ciskowski, Marek Zaton Fig. 3. Topology of patterns recognized by the 1-candle SOM 81 and 100 neurons, for three-candle formations - nets of 100 and 225 neurons. In all cases the smallest structures provided the best performance. After training the maps were labeled. The sets of known formations were supplied on nets inputs. The map of the 1-candle SOM is presented in fig. 3. The net grouped similar patterns on the map placing formations of market s hesitation (e.g. the doji and spinning tops ) in the middle of it, bullish patterns in the top part of the map (ideal hammer in the upper left corner), while bearish patterns in the lower part of the map (ideal shooting star in the lower left corner). The left hand side of the map contains patterns with longer wicks, while the right hand side - candles built mostly by their bodies. 6 A sample trading system The maps described above provide us with coherent visual maps of candlestick formations. They also allow us to classify and code the analyzed window of timeseries for further steps of decision algorithms. The system presented in this section uses the outputs of SOMs analyzing 1-, 2- and 3-candle windows, translated to the direction of the recognized pattern (-1 for bearish, +1 for bullish and 0 for neutral), along with the value of a short term exponentially weighted moving average of the price, as inputs to a feedforward net learning trading decisions. These decision were set arbitrarily by a human trader using a trend following strategy and visual analysis of the chart. The analyst knew the future, that is saw both the left and right hand side of the analyzed time point on the chart, therefore his decisions were correct by definition. The multilayer percpetron network is very simple - built of 4 sigmoid units in the hidden layer and one linear in the output layer. The structure of the neural decision part of the trading system is presented in fig. 4. First, the three SOMs were trained to recognize patterns, as described earlier. Then the feedforward net was trained on the same period of time with 300
Neural pattern recognition of forex data streams 7 SOM 1 candle... +1= BUY 0=WAIT -1= SELL SOM 2 candles... +1= BUY 0=WAIT -1= SELL +1= BUY 0=WAIT -1= SELL SOM 3 candles... +1= BUY 0=WAIT -1= SELL Fig. 4. The structure of the neural decision part of the trading system desired trading signals, with standard backpropagation algorithm. 70% of data was used for training, while 15% for validating and 15% for testing. The net reached 1.02913 10 13 value for the mean squared error and the correlation coefficient of 0.999999. These values illustrate only the performance of the neural decision subsystem. However, good decisions (entry points for transactions) may be worthless if used with wrong strategy of securing an open trade and its profit. For a full view of system s profitability on the market, it should be tested on-line on real market and its performance should be measured with such characteristics as: gross and net profit, profit factor, number and value of profit and loss transactions, maximal drawdown etc. In our case, we used simple but strict money and risk management rules, so that their influence did not dominate the role of the neural decision part. The system was tested on historical data (EURUSD, 1 hour interval) in Metatrader s strategy tester. The results of the test are presented in fig. 5 and in table 1. Testing period of 2.5 years covered bullish and bearish markets, and periods of stagnation. Starting account balance was 10000 USD, with the leverage of 1:100. Money and risk management strategy defined the following rules: 1 open trade at a time (stop and reverse system), maximum of 30% of capital involved in margin for one trade and maximum of 2% of capital exposed to risk in one trade. The total profit achieved was 390%. The results show both the effectiveness of the system and the role of proper money and risk management. The number of profit trades was 6 time smaller than the number of loss trades, while the value of the former exceeded the value of the latter by 1.5. This was due to the fact that loss trades were closed quickly at stop losses, while profit trades were kept for a long time. The system presented stable and monotone performance, without large drawdowns in capital. The proportion of profit to loss trades does not deprecate the effectiveness of neural decision module. It was able to point out the most important moments for long lasting and profitable trades, while the loss trades happened mostly due to delayed market reaction to some formations (followed by a so-called second chance ). In that cases good trades open too early may be closed on strict stop losses due to market hesitation. More sophisticated strategies including re-entries and multiplying long profitable trades may provide further improvement to the simple strategy presented here.
8 Piotr Ciskowski, Marek Zaton account balance transactions Fig. 5. Results of system backtest on EURUSD - account equity backtest period 13.10.2006-11.06.2009 trades profit in one trade profit 25 90 901.50 USD max profit 16 058.00 USD loss 159-51 842.00 USD max loss 2 454.00 USD all 184 39 059.50 USD average profit 3 606.06 USD total profit 390.59 % average loss 326.05 USD Table 1. The results of trading system s backtesting on historical data 7 Conclusions The problem of efficient processing of forex market data is addressed in the paper. It was shown that the Japanese candles, appropriately coded, provide a superb method of presenting timeseries data to neural nets. Self organizing maps provide clear coherent maps of candlestick formations for the analyzed instrument and timeframe. The classification performed by the maps is simple and efficient and may be used as input for the final trading decision algorithm. An example of a profitable trading system based on that methodology was presented and analyzed. References 1. McNelis P.D., Neural Networks in Finance: Gaining predictive edge in the market, Elsevier, 2005 2. Nison S., Japanese Candlestick Charting Techniques, Second Edition, Prentice Hall Press, 2001 3. Sheng-Tun Li and Shu-Ching Kuo, Knowledge discovery in financial investment for forecasting and trading strategy through wavelet-based SOM networks, Expert Systems with Applications, vol. 32, no. 2, pp. 935-951, Elsevier, 2008 4. Vanstone B. and Finnie G., An empirical methodology for developing stockmarket trading systems using artificial neural networks, Expert Systems with Applications, vol. 36, no. 3, pp. 6668-6680, Elsevier, 2009