For many years various problems of analysis of financial markets are used as testing grounds for most advanced new technologies of Artificial Intelligence. Interest in such applications of Artificial Intelligence is kept hot by both, developers of Artificial Intelligence systems as well as potential users of these systems: financial organizations and individual investors, operating in securities markets. Their reasoning are the following. On one hand, forecast of dynamics of a market – the system, which corrects its behavior depending on its own history – is probably one of the most difficult existing tasks. Thus it is considered as a challenge to the might of mathematics and modern computer technology. On the other hand, even a small improvement in portfolio management efficiency against the market average can result in very substantial perks.
In today’s world hundreds, if not thousands of different intelligent portfolio management decision support systems (PMDSS) are in use. Their work is based on combination of sophisticated statistical methods with rules distilled out of knowledge acquired from experienced traders, as well as on some other approaches. Majority of these systems are based on Neural Networks, referred to as computer models of brain, which forecast outcomes of future situations by establishing weighed links between network elements (neurons). Weights are assigned to links while analyzing fragments of a previous history of the market, represented by prices and volumes of trades, as well as some other indexes.
Why all this does not work in Russia
Despite of a variety of approaches, majority of PMDSS are based on the analysis of market history. However, in order for the model built on the basis of analysis of the previous experience to work well in the future, it is necessary that the market would not undergo drastic changes for an extended period of time, as well as that events taking place at present time would have some approximate analogs in the past. For example, most of Neural Network systems require for their training market histories being available for no less than three to five years. In other words it is assumed that over some period of time internal laws of market dynamics will not differ substantially from those that were in operation for the last three to five years. It is quite obvious, that in the case of Russian financial markets, and the Short Term Government Bonds (STGB) market in particular, these conditions are not satisfied. Majority of these markets exist for only two-three years, while the STGB market is rapidly evolving in addition: total volume of securities circulating in this market grow constantly and significantly, rules of operation in this market change frequently.
Along with all other Russian securities markets, the STGB market is influenced heavily by political instability. This brings in focus yet another drawback of Neural Networks. It is extremely difficult to understand, and hence to control, why Neural Network takes one or another decision. This reduces reliability of portfolio management in crisis situations when influence of some external factors, not taken into account by the system, becomes strong. One additional peculiarity of Russian markets is the lack of developed system of indexes characterizing various financial instruments, markets and economy as a whole, which are traditionally used by PMDSS for assessment of current market situations. These are the main features that make application of traditional PMDSS systems to Russian financial markets ineffective.
Mining Data for Knowledge
So the task of developing an optimal portfolio management strategy is very complicated, and the same task for Russian financial markets is twice more difficult. Is there a way out? In fact there is one. It is called Data Mining and Knowledge Discovery. This name designates a bundle of methods from a branch of Artificial Intelligence that began its explosive growth very recently. These methods allow one to acquire from “raw” data (in our case that would be a market history description) previously hidden knowledge about relations and patterns in behavior of the investigated object.
The process of knowledge search is accompanied by a process of careful verification of the significance of discovered knowledge. This way even for a small amount of data interesting partial results can be obtained. A set of variables suited most for description of the system is chosen automatically. Obtained results are presented in text and graphic forms which are easily understood by a human.
Attractiveness index of a bond
What is a recipe for applying Data Mining technology to analysis of securities market? Let us first consider a simplified problem. Suppose, you can tell the future and know for certain, which bond issue price will grow the most during the next trading session. Then you can buy this bond beforehand each time, and thus rip the maximally possible profit for any given period of time. Similarly, if a Data Mining system were able to predict, based on the previous history, this “best bond issue”, it would become a powerful tool in the hands of investor. More precisely, the system should find a function of parameters of various bond issues (their prices, number of days to bond redemption, dividend amount, etc.), as well as a market as a whole (average profitability, its change, etc.), that represents an index of attractiveness of each bond issue to investor at a given moment. The larger the attractiveness index for a bond issue, the larger price increase in the next trading session is predicted by the system.
In fact, this is a much more complicated task than it seems at first. One does not know not only a set of parameters on which the attractiveness index depends, but also even an approximate form of this dependence. These facts preclude simple solution of this task by even the most sophisticated statistical methods such as used in SAS or Statgraphics software packages, because to apply them the user know half-correct final solution. Statistics can only help finding regression coefficients for a known relation or determining correctness of an existing hypothesis. On the contrary, methods of Data Mining allow one to find a solution
automatically from scratch.
Problem Formulation
Utilization of the Data Mining technology allows a non-traditional formulation of task of predicting dynamics of various securities in the market. The idea is to use market history for testing different attractiveness index functions built for various securities. To each such function there corresponds a certain trading strategy. For example, in our simplified problem the strategy is to have in your portfolio the bond issue with the largest attractiveness index at any moment. When the optimal trading strategy is applied to a given period of market history, a corresponding profit for this period can be calculated. Thus to each tested attractiveness index function there corresponds a certain profit for a given market history period. The objective is to find a function maximizing the profit.
This is very different from the traditional problem formulation when the task is solved in two steps: first, a model for the market dynamics is built, and then, on its basis, an optimal strategy is calculated. Accuracy is reduced at each step of the process. The Data Mining technology allows to solve the task in just one step: an optimal trading strategy, described in terms of attractiveness indices, is built directly from raw data describing market history. This allows to gain some accuracy against the traditional approach. And since securities market is a system governed by very vague and imprecise rules, this gain in accuracy turns out to be crucial for successful trading.
The way we operate
The task is split into two parts. For managing bond portfolio we use a version of PolyAnalyst – Data Mining system developed by Megaputer Intelligence, Ltd. – specially tuned to analysis of the bond market. And actual trading is performed with the help of our system SmartBroker. PolyAnalyst develops a model for calculating attractiveness indices of different securities and exports it to SmartBroker. SmartBroker has an access to constantly updated bond market information (for example, it can obtain data from GKO-Invest – the system well known in Russian financial circles). On the basis of the model acquired from PolyAnalyst and a current market situation SmartBroker suggests buying or selling certain securities. Meanwhile PolyAnalyst periodically recalculates the model taking into account recent market statistics.
There is a very important question: After what period a currently used model becomes inefficient and should be recalculated? Our experience shows that even without abrupt changes in the market situation a model loses its predictive power after roughly two-three months. Thus one should have an updated model prior to an old one’s expiration time. And one should not forget that from time to time market situation changes dramatically due to some new external factors. In such cases a current model instantaneously ceases to work. An example of such an occasion in Russian financial markets history is the situation preceding summer election-96 when political uncertainty became an absolutely dominant factor. Obviously, a fireproof early recognition of such structural shifts is a major requirement to any technology used for portfolio management. When a developing transition is detected, the most cautious trading strategy should be used temporarily.
The following mechanisms have been developed for detecting exceptional market situations. A set of propositions describing the current market mode is formulated along with each recalculated model. A break of any of these propositions means that the current model might become inadequate. The propositions are chosen so that they would be, on one hand, maximally limiting but, on the other, applicable to any situation of the current market history which was used for the model development. As most elementary propositions one could use: “market average return does not exceed 95% annually” (actual numbers!) or “difference in return on long and short bond issues does not exceed 60% annually”, etc. When market leaps into another state, we receive a signal of some propositions breakdown. In this case we switch to a cautious strategy and restructure our portfolio in favor of shorter bond issues whose redemption time is around the corner.
The bottom line – strategy effectiveness
The lack of established indices characterizing average market return makes the assessment of a strategy effectiveness a nontrivial issue. In order to estimate how well the described technology works we use the following effectiveness indicator. We can choose a time interval such that an STGB is issued at the beginning of this period and is redeemed at the end. If one buys this STGB and simply holds it until the redemption date, he receives some guaranteed income. One would certainly hope to do better when when some portfolio management strategy is utilized. On the other hand, looking backwards one can determine the maximum possible income for the same period, assuming that all trades are performed at prices averaged over the trading session. The income actually received when utilizing the discussed strategy lies somewhere in the interval between the guaranteed and maximum possible income. The share of this interval provided by our portfolio management strategy, expressed in percent, is called the strategy effectiveness indicator.
Efficiency = (PA- Garant)/(Ideal - Garant)
On average this indicator stays near 30%, sometimes exceeding 60%. Shaded bars show the guaranteed (white), actual PolyAnalyst (gray), and ideal (black) income measured monthly as the annual return in percent (measured on the left scale). The shadowed area corresponds to the period of the presidential election instability. In the beginning of this period actual trading according to the described technology had been suspended since the system issued a warning that current market state propositions became false. This warning system allowed us to avoid losses. The displayed calculated effectiveness points are purely theoretical values. When sufficient amount of data characterizing the new crisis condition of the market has been collected, the model was recalculated and put in use again. As a result of this recalculation the new model can be applied successfully in deep crisis situations as well, since the system “knows” now what the crisis looks like and what it should do in such situations.
Reviewer
Professor Lev Meshalkin
Central Institute of Mathematics in Economy
Moscow, Russia