Data mining for strategies
For the past several months, I have been busy working on software to aide in “data mining” the market. With the hope of eventually finding repeatable patterns. The basic premise is to devise software capable of finding patterns that provide an exploitable edge. Because I have been spending all my free time working on this software, I have been falling behind on the pursuits outlined within this journal.
I am hoping to get back into the swing of things, and eventually incorporate the software I have been working on as a way of further discovering what works and what does not work. The software is great at finding an “edge containing pattern” if one exists in the provided data set. However, the most challenging aspect is figuring out what data to provide and the basic constructs of the pattern you are interesting in.
That is why its important to continue down the path that I started within this journal. To most effectively find patterns within the market, you have to have a way of weeding out some of the useless indicators and ideals. Without performing this degree of pre-filtering, the software can not run fast enough to sift through all of the data.
As this software becomes more polished, I will release a version here in this journal as well as utilize it in further weeding out the garbage indicators. For example, by focusing on a single indicator and feeding it into the software, you can “fairly” quickly see if any edge is possible. If the software is not able to extract any repeatable patterns (resulting in an edge), then that indicator is most likely useless in your trading. Unfortunately, the software is not quite polished enough for prime time. In the mean time, I will continue to test indicators the way I previously approached the issue while slowly migrating towards utilizing this data mining software. Eventually replacing most of my analysis of indicators with results found through the data mining software approach.
Share
MACD Part 1: Zero Line (Long Only)
In this series we will test the MACD (Moving Average Convergence Divergence) indicator. Like our other indicator series we will test the results against our 5-minute base line results found here: Long Baseline, Short Baseline, and Risk/Reward Baseline
MACD is commonly used to enter trades based on its position relative to the Zero Line. Our first test will enter long whenever MACD is above the zero line.
Results: Exit at 5 Bars
These results turned out very similar to what we saw with RSI when above 50. Very few combinations were even positive with no results coming close to demonstrating a tradeable edge.
Results: Exit Using 1:2 Risk/Reward Ratio
Not a single combination resulted in a net profit. Clearly it is not wise to buy when MACD is above the Zero Line.
Now we will test the MACD below the Zero Line. Based on the above results, we expect being below the Zero Line to outperform.
Results: Exit at 5 Bars
No combination was able to surpass our targets for total profit. However, a large number of combinations surpassed our percent profitable threshold. This indicates a potential edge, however the edge is likely too small to surpass trading costs such as commission. May be useful as a filter to filter out losing trades.
Best Combination: Fast = 2, Slow =4 Smooth =4
Results: Exit Using 1:2 Risk/Reward Ratio
No combination came close to our thresholds. This combined with the failure to reach profit thresholds using 5 bar exit, further underscores this strategy as containing a very weak if any edge.
Summary
There may be a slight edge when entering trades while below the Zero Line. However, by itself it is not tradeable. In future articles we will look at ways of using this filter to improve our overall edge.
Source Code:
Related Posts:
Share
Exit Strategy Part 1
Entries are important, but exits are where you make your money. A strategy can not be complete without fine tuning both sides. In this article series I will explore various exit strategies to determine which are effective and which are not. The end goal is to create an exit strategy that makes a random entry strategy profitable.
To succeed the exit strategy must remain profitable over 100 continuous runs with an average profit factor over 1.2. The random entry strategy will target 5,000 trades over the testing period. This is a rather lofty goal. In fact, it may not even be possible. However, we can learn a lot about various exit strategies and dynamics by attempting it.
Time Based Exits
This is a very simple concept. Exit after x number of bars have passed. We use this method often while evaluating entries because it provides a good reference point.
Results:
As we already knew, this method fails to make a random entry strategy profitable. The best results were only able to achieve 44 of 100 runs profitable and -$2,000 average profit.
Trailing Stops – Ticks
I will use NinjaTrader’s SetTrailStop() method set to CalculationMode.Ticks.
Results:
No combination produced a single profitable run.
Close > Close 1 Bar Ago
The strategy will exit if Close > Close 1 Bar Ago.
Results:
This was a a big surprise. The strategy was actually profitable on 88 of 100 runs with an average profit of +$8,823. PF = 1.05, WIN % 68
Close < Close 1 Bar Ago
The strategy will exit if Close < Close 1 Bar Ago.
Results:
Only profitable in 5 runs
Close < Low 1 Bar Ago
The strategy will exit if Close < Low 1 Bar Ago.
Results:
15/100 Runs
Close > High 1 Bar Ago
The strategy will exit if Close > High 1 Bar Ago.
Results:
Profitable in 92/100 runs. Average Profit +$15,931 PF = 1.07 Win % 68
RSI(30) > threshold
The strategy will exit when RSI is above a certain threshold. When testing RSI we determined there was an edge to short when RSI went above a threshold. Therefore, we can assume it may be a good time to exit long positions.
Results:
As expected the strategy performed well. Profitable in 82 runs. Average Profit +$10,131 PF = 1.04 Win % 71
Summary:
So far we have only looked at very simple exit logic and already see that reaching our goal may be possible. Future articles will be one article per exit strategy as we begin to tackle much more complex exit strategies.
Related Posts:
Share
RSI Part 9: Risk/Reward
Before temporarily moving away from RSI, I want to evaluate each method one more time using the risk/reward test discussed in this article: Risk/Reward Testing
Crossing Above a Threshold:
Surprisingly many parameter combinations performed very well. Unfortunately, no combinations were able to exceed our desired targets.
Crossing Below a Threshold:
This method performed horrible based on this metric. This is what I thought might happen, and the reason I wanted to perform this test. Buying when oversold is a high percentage play if your nimble. But from a risk/reward standpoint its a hard strategy to trade.
Normal Divergence:
This method still did not contain any tradeable edge. Performs very poorly.
Hidden Divergence:
No combination was able to exceed our lofty targets, however several combinations missed by only fractions of a percent. This further underscores the fact that RSI Hidden Divergence contains a tradeable edge.
Related Posts:
Share
Testing setup: Risk/Reward
I believe any tradeable edge should find trades that quickly become profitable. However, there are many ways to make money. We should also evaluate a strategy based on its risk/reward profile. A strategy may not be profitable 3 or 5 bars into the future like we have been testing, but may still signify a turning point with low risk. Before evaluating strategies based on risk/reward we have to establish a baseline to eliminate bias in our data set.
In this article, we will once again be running random simulations. We will use a 1:2 risk:reward ratio. Entries will be random based on the target number of trades (500,1500,5000). The strategy will set a profit target 3 *ATR(14) above the closing price and a stop loss 1.5 * ATR(14) below the closing price. The strategy will exit when the first target is hit or end of the day. Whichever comes first. After 100 random simulations we will then average all of the results and produce targets like we did in this article: Testing Setup
Baseline Results for 5,000 trades (Long):
Percent Profitable: 34.18%
Expected Return: -$29,377
Target Percent: 36.86%
Target Return: +$38,493
Baseline Results for 1,500 trades (Long):
Percent Profitable: 34.37%
Expected Return: -$8,587
Target Percent: 40.79%
Target Return: +$36,331
Baseline Results for 500 trades (Long):
Percent Profitable: 34.73%
Expected Return: -$1,886
Target Percent: 45.54%
Target Return: +$26,455
Now we will perform the same simulations to determine the bias for short trades. We are still using a 1:2 risk/reward ratio.
Baseline Results for 5,000 trades (Short):
Percent Profitable: 32.23%
Expected Return: -$34,763
Target Percent: 34.8%
Target Return: +$33,439
Baseline Results for 1,500 trades (Short):
Percent Profitable: 31.98%
Expected Return: -$11,147
Target Percent: 37.69%
Target Return: +$32,599
Baseline Results for 500 trades (Short):
Percent Profitable: 31.58%
Expected Return: -$4,478
Target Percent: 40.94%
Target Return: +$21,362
Based on these results it is very clear that using a 1:2 risk/reward ratio requires a great entry. I believe the random entries fared so poorly due to all of the noise in the markets. This should serve as a good indicator when comparing edges.
Source Code:
Related Posts:
Share
RSI Part 8 – Short Strategies
In previous articles we have identified two methods that provide an edge on the long side (Bullish Hidden Divergence and Buying on oversold conditions). Before reading this article, its recommended you read the following two articles:
1. Buying on “Bullish Hidden Divergence
2. Buying when oversold
For this article we tested the same two concepts to see if they also had an edge when traded from the short side. The first method we looked at was “Bearish Hidden Divergence”. The testing strategy was very similar to the one used in “Bullish Hidden Divergence”. It was immediately apparent that this strategy had a significant edge just like its bullish counterpart.
Several combinations easily exceeded our target values when predicting 3 bars into the future. However, when predicting 5 bars into the future the results were not quite as clear. Several strategies came extremely close, but failed to exceed our targets. In some cases missing by fractions. Because this strategy easily blew past our targets for 3 bars into the future, we know the general concept has an edge. Its likely the edge degrades fast when trading short though. When we explore exit strategies in a future article we will need to take this into account. Here is a screenshot from one of the 3 bar combinations:
Next we tested selling short when RSI crosses above a given threshold (overbought). This strategy also clearly contains a statistical edge. Many combinations for both 3 and 5 bars into the future were profitable. Here is an example of one such combination:
The next step will be to create a “combination” strategy using several parameter combinations each exhibiting a statistical edge to see if the combined edge is maintained.
Source Code:
Share
RSI Part 7 – Bullish Regular Divergence
In this article we will be testing Bullish Regular Divergence using RSI.
Bullish Regular Divergence
Price: Lower Low
RSI: Higher Low
Using Bullish Hidden Divergence we were able to significantly exceed our targets and prove that there exists a statistical edge when using that method. However, the same can not be said for regular divergence. The strategy produced several combinations which were profitable, but all of the returns were well below our targets. Therefore we can not rule out random chance accounting for those results. Here is a P/L chart from the best combination:
Notice how “random” and volatile the results appear? When compared to the chart produced from the Hidden Divergence article, you can clearly see the difference. Here is a P&L chart from the Hidden Divergence article.
Related Posts:
Share
RSI Part 6 – Bullish Hidden Divergence
A common technique for trading oscillators such as RSI is to use divergence. There are two types of divergence, regular and hidden. Regular divergence is commonly used to signal trend changes, and hidden divergence is used to signal trend continuation.
Bullish Regular Divergence
Price: Lower Low
RSI: Higher Low
Bearish Regular Divergence
Price: Higher High
RSI: Lower High
Bullish Hidden Divergence
Price: Higher Low
RSI: Lower Low
Bearish Hidden Divergence
Price: Lower High
RSI: Higher High
In this article we will be testing Bullish Hidden Divergence. This method is commonly used when a stock is in an uptrend and pulls back to test support. To find divergence we need to look at the lows of both RSI and price. Since RSI moves smoother than price, I used it as a way of finding lows. Once a low has been identified in RSI, I start looking for another potential low. If one is found I compare the price of the stock at the time of the first RSI low to the value of the stock now. I then compare the first RSI low with the new potential low. If RSI has made a lower low and price has made a higher low I continue. Next, I check to see how many bars have passed since the first RSI low. This is to help ensure that enough distance has been created between the two valleys.
I optimized the strategy on 3 parameters:
1. RSI Period
2. Number of bars to look back when calculating a low
3. Number of bars to pass between lows
Initial (5 bar) results were decent. Most optimized pairs performed very well. However, only one combination was able to exceed our target value on both percent and profit. The strategy used a 9 bar RSI, 3 bar look-back, and required at least 9 bars between low points.
The (3 bar) results also did well. The same combination that exceeded our targets for 5 bars also exceeded for 3 bars. Here it the PL screenshot.
Further Enhancements:
Commonly Bullish Hidden Divergence is used to continue an uptrend. Therefore, I tried to only take trades when the stock was in a confirmed uptrend such as above a moving average etc. This method did not result in any performance gain. As we explore more indicators and determine which are ideal for identifying trend we can revisit this strategy.
Source Code:
Share
RSI Part 5 – Optimized Thresholds
In this article I want to explore which threshold values are the most significant (long only) and also see what happens when applied to other markets. To test this, I optimized the strategy on both threshold and RSI period while counting the number of times the resulting combination exceeded our statistically significant levels for both percent profitable and net profit.
Results for 5 Bars Into Future:
First I looked at the various threshold values and how often when using the threshold I was able to surpass either our target percent profitable or target profit levels. Threshold values of 25 and 30 seemed to be the most robust.
| Threshold | Count Beat Percent | Count Beat Profit |
| 10 | 0 | 0 |
| 15 | 1 | 0 |
| 20 | 3 | 1 |
| 25 | 3 | 2 |
| 30 | 2 | 3 |
| 35 | 2 | 0 |
| 40 | 4 | 0 |
| 45 | 6 | 0 |
Each of the above threshold values were used against various RSI periods ranging from 2-30. Below is a table outlining which periods resulted in surpassing our targets. Interestingly the lower periods 2-5 performed the best. I believe this may have something to do with our target holding time. Since we are only looking 5 bars into the future, the smaller RSI periods reflect the upcoming moves more accurately.
| RSI Period | Count Percent | Count Profit |
| 2 | 2 | 2 |
| 3 | 4 | 1 |
| 4 | 3 | 2 |
| 5 | 1 | 2 |
| 6 | 1 | 2 |
| 7 | 0 | 1 |
| 8 | 0 | 1 |
| 9 | 0 | 1 |
| 10 | 0 | 1 |
| 19 | 0 | 2 |
| 30 | 0 | 1 |
The combination of RSI(3) < 30 was the only pair to result in both a beat on percent profitable as well as target profit level. I wanted to see how robust this pair actually was, so I ran it against all 30 of the DOW stocks. The method was profitable on 25 of the 30 stocks. With PG performing the best, and CAT performing the worst. Over 436 thousand trades were recorded during this test, resulting in 54.2% profitable trades.
The strategy was able to beat the target percent in 17 of the 30 stocks, and beat the target profit in 7 of the 30 stocks. Considering the period/threshold combination was optimized on a different market, I believe these results are very promising.
Next I wanted to see what happens if I flip the combination and entered long when RSI(3) > 70. The results were horrible as expected. Further lending support to their being an edge when below a threshold as opposed to above a threshold. Only 6 of the DOW 30 was profitable in this test. With a combined 50.06% profitable trades.
Results for 3 Bars Into Future:
The same methods used for 5 bars into the future were used for 3 bars. Here are the two resulting tables:
| Threshold | Count Beat Percent | Count Beat Profit |
| 10 | 3 | 1 |
| 15 | 7 | 2 |
| 20 | 7 | 1 |
| 25 | 11 | 0 |
| 30 | 12 | 3 |
| 35 | 24 | 4 |
| 40 | 23 | 0 |
| 45 | 12 | 0 |
The results stayed similar to what we saw with 5 bars. Thresholds between 25-35 do very well. One thing to note is using 3 bars into the future we are having a lot more values beat our targets. This may indicate predicting 3 bars into the future is easier than 5 when using RSI.
| RSI Period | Count Percent | Count Profit |
| 2 | 11 | 1 |
| 3 | 11 | 1 |
| 4 | 9 | 1 |
| 5 | 6 | 1 |
| 6 | 5 | 0 |
| 7 | 2 | 1 |
I had way too many RSI periods beat our targets. Therefore, I am displaying only the top performers. The lower periods seem to predict better than the higher periods for this time frame.
Summary:
It looks like any threshold between 25-35 is ideal with a period less than 5. I would like to point out that while I am only showing the count that beat our targets that this does not mean the other combinations did not fare well. In fact, combinations in the ranges I outline were uniformly great. We have strict requirements in the targets we are after. This is to ensure the results produced are not due to random chance. Its hard to go wrong when predicting either 3 or 5 bars into the future using a threshold of < 40.
The RSI(3) < 30 combination performing as well as it did against the DOW 30 stocks is very reassuring.
Related Posts:
Share
DOW 30 Baseline (Long Only)
As we continue to explore trading methods, we will need a way to compare our results to other markets. Below are the baseline figures for each of the 30 DOW components.
NOTE: I started with just 15,000 trades and 5 bars initially as these are the results relevant to the strategies I am currently testing. The remaining baseline values will be updated as we continue testing and need the results.
15,000 Trades – 5 Bars (Long Only)
AA
Percent Profitable: 52.35%
Expected Return: -$25.92
Target Percent: 54.14%
Target Return: +$34.17
AXP
Percent Profitable: 51.59%
Expected Return: -$5.02
Target Percent: 53.41%
Target Return: +$77
BA
Percent Profitable: 52.19%
Expected Return: -$4.89
Target Percent: 53.8%
Target Return: +$93.36
BAC
Percent Profitable: 52.45%
Expected Return: +$12.12
Target Percent: 54.29%
Target Return: +$99.14
CAT
Percent Profitable: 52.12%
Expected Return: +$16.57
Target Percent: 53.9%
Target Return: +$129.27
CSCO
Percent Profitable: 53.07%
Expected Return: -$13.07
Target Percent: 54.84%
Target Return: +$33.04
CVX
Percent Profitable: 51.97%
Expected Return: +$9.15
Target Percent: 53.82%
Target Return: +$116
DD
Percent Profitable: 52.29%
Expected Return: -$4.24
Target Percent: 54.11%
Target Return: +$67.23
DIS
Percent Profitable: 53.84%
Expected Return: +$9.88
Target Percent: 55.8%
Target Return: +$61.68
GE
Percent Profitable: 52.14%
Expected Return: -$18.58
Target Percent: 54.12%
Target Return: +$31.94
HD
Percent Profitable: 52.12%
Expected Return: -$10.14
Target Percent: 53.75%
Target Return: +$57.14
HPQ
Percent Profitable: 53.73%
Expected Return: +$25.61
Target Percent: 55.68%
Target Return: +$94.37
IBM
Percent Profitable: 51.34%
Expected Return: +$28.61
Target Percent: 53.23%
Target Return: +$185.3
INTC
Percent Profitable: 52.83%
Expected Return: -$6.83
Target Percent: 54.96%
Target Return: +$50.88
JNJ
Percent Profitable: 51.73%
Expected Return: +$4.5
Target Percent: 53.47%
Target Return: +$77.85
JPM
Percent Profitable: 52.26%
Expected Return: -$.16
Target Percent: 54.11%
Target Return: +$88.05
KFT
Percent Profitable: 54.4%
Expected Return: +$11.26
Target Percent: 56.22%
Target Return: +$52.33
KO
Percent Profitable: 53.4%
Expected Return: +$19.82
Target Percent: 55.28%
Target Return: +$96.66
MCD
Percent Profitable: 52.23%
Expected Return: +$15.27
Target Percent: 55%
Target Return: +$74.2
MMM
Percent Profitable: 51.14%
Expected Return: +$1.98
Target Percent: 52.86%
Target Return: +$127.42
MRK
Percent Profitable: 52.07%
Expected Return: -$9.47
Target Percent: 54.05%
Target Return: +$75.55
MSFT
Percent Profitable: 53.02%
Expected Return: -$3.09
Target Percent: 54.96%
Target Return: +$77.03
PFE
Percent Profitable: 52.87%
Expected Return: -$16.74
Target Percent: 54.74%
Target Return: +$26.48
PG
Percent Profitable: 52.77%
Expected Return: +$32.26
Target Percent: 54.41%
Target Return: +$112.04
T
Percent Profitable: 53.01%
Expected Return: -$6.74
Target Percent: 54.8%
Target Return: +$56.14
TRV
Percent Profitable: 51.4%
Expected Return: -$2.12
Target Percent: 54.74%
Target Return: +$65.6
UTX
Percent Profitable: 51.39%
Expected Return: -$3.14
Target Percent: 53.18%
Target Return: +$93.14
VZ
Percent Profitable: 52.63%
Expected Return: -$4.41
Target Percent: 54.3%
Target Return: +$59.05
WMT
Percent Profitable: 52.01%
Expected Return: -$13.17
Target Percent: 53.16%
Target Return: +$58.17
XOM
Percent Profitable: 52.36%
Expected Return: +$10.13
Target Percent: 54.22%
Target Return: +$108.32
Related Posts: