Continuing on the theme of my last post, I want to take some time to dig into the options you have when you think about doing your own testing and homework in the markets. This post will necessarily be opinionated and a bit narrow, because I’m sharing my own experiences and perspectives. I’m sure many of you have very different experiences, so please feel free to share them in the comments. (And, if you’re finding this post a while after it was posted, do check out the comments. It’s no unusual to find my readers writing better comments than my posts!)
So, let’s assume we’re agreed that you need to be doing some kind of quantitative testing and analysis in your trading, and that you’re comfortable with the very high level outline I shared in this post of get data, define condition, and see what happens after that condition is fulfilled. We have a couple thorny issues (where to get data and how to clean it, how to define conditions, what can be defined, etc.), but we can return to them later. Let’s keep our eye on the big picture, for now. If you want to think about conditions and results and work to understand the probabilities in the market, I think there are a few ways to do so, in roughly ascending order of complexity:
- Just look at data, trade, and start to “get a feel”. Many of us start here, and it’s probably an important part of the learning curve. (In fact, if you start from a more quantitative perspective, I think you will need to come back to this stage and just experience “being in the market” for a bit. Trading wisdom is earned by exposure to market conditions and fluctuating P&L.) It’s also a very unreliable place to be. Learning is haphazard and spotty. We forget things. We miss things, and we attach the wrong significance to many things. Cognitive biases (recency bias, for instance: you will remember what happened last more than earlier events) and the highly random nature of the market can give us the wrong feel.
- Do some backtesting by hand. Again, I think this is an important part of the process. The trader who comes to this level is doing some good, hard work. She has made a decision to step beyond her experiences in the market, and to drill deeper. This work can be as simple as going bar by bar through charts, or can lead to much more extensive analysis and record keeping. I also think this stage is important. Get your hands dirty with the data, and spend some time training that intuition with a deeper, more structured, look into market data. This methodology is a major part of my free trading course, and is certainly worth a tremendous amount of your time and energy. However, we’re still limited in what we can do. Maybe you can look at a few hundred or a few thousand events, but cognitive biases, limited time and attention, and a number of other factors work against you. At some point, you’ll need to go deeper.
- Third party trading/analysis software. Most people spend some time here. Software solutions range from overly simplistic, retail-oriented to very expensive institutional frameworks (that can easily cost well into six figures.) Here, we’re able to define a set of conditions, and see what would have happened historically. These tools become much more powerful, and the trader who crosses from doing work by hand into the this area often feels overwhelmed with possibilities. Be careful. That’s my best advice here: be careful. It’s not as easy as you think, and most of the retail-level platforms don’t allow you to do proper testing. In particular, “optimization” as presented in many of these platforms is very dangerous and can lead you to completely incorrect conclusions. My personal experience (and I stress that point–this is simply my personal experience) is that most retail-level platforms are inadequate for serious system development, unless you use them as a piece of a more extensive infrastructure (as in spitting out signals or orders to a deeper analytical package.)
- More advanced software/writing your own. Here is where I think we start to get into the truly powerful tools. Microsoft Excel belongs in this category, but it’s definitely a double-edged sword–while you can do very complex analyses in Excel, especially when combined with VBA, it’s not the best tool. (More on that in a future post.) There are a number of commercial tools you might have encountered in other contexts that can be very useful for doing market analysis; think of MATLAB, SPSS, SAS, and Stata–standard statistical packages. (Of course, the big commercial trading infrastructure/analysis packages go here too, but if you’re reading this they probably don’t fit your needs or budget very well.) If you’re willing to put in the work and learn to code, writing your own software might be the best option.
This table is simpler than it appears. I’m basically suggesting that the progression starts with doing some tabular work by hand, maybe moving through a phase of using some common tools (and there are more and more of these out there), and ending with you creating your own software package. The last stop of that process is a journey in itself, and requires a significant investment of time and energy in learning a “real” programming language. In my next post, I’ll talk more about that. We’ll look at some of the choices you have in choosing a language, and I’ll tell you why I think Python (and maybe R) are the right choices.
All of this is hard work, but it’s harder to imagine someone having long-term success in the market unless he understands the market. Maybe there’s another path, but, to me, the only path that makes sense is doing your own work to understand the market, creating and trading that plan, and then monitoring your results for needed course corrections. It might not be easy (it’s not!), but it’s worth it in the end.
Kickstarting the discussion.
I think we all start with some rudimentary understanding of Excel or similar spreadsheet applications.
I therefore think it is a critical productivity enhancement and timesaver that whatever trading analysis software you use allows you to export your data into excel in a format that is readily usable.
There are a number of free statistical add-ons for Excel out on the web and they are also a great stepping stone to get the absolute most out of Excel before, and if, you move onto more statistically oriented programming languages such as R.
What I find the hardest is to go from “idea” that you are testing out to the “language” that you need to translate that idea into your analysis software, excel or programming language. It takes time and considerable patience to make sure you get that translation right so your results are valid and not garbage.
The other point I’d like to make is the big elephant in the room when you start making assumptions and decisions based on your testing results. That elephant is of course “context”. I know in my own trading that without a deeper understanding of the macro context – in simple terms, what is really driving the markets at this moment (whether it be negative interest rates or whatever) – then my trading results are mediocre at best.
I don’t think there is a way to build context into detailed statistical studies of trading ideas but I’d gladly like to be proven wrong on this one.
You make good points here. I’ve had some experience with stats add-ins for Excel and never found any that I really loved. For me, it always made sense to work from Excel to Stata for actual stats…Excel is ok for very basic tabulation but if you want to do any real number crunching I think best to do it in another platform. Regardless, your point that pretty much everyone has some Excel skills (or can get those skills really easily) is absolutely valid.
As for context… I dunno. One of the ideas I’m currently playing with is that macro factors aren’t really predictive. (This is an early insight into some work I’m doing right now.) If you could predict macro factors with 100% certainty, you still couldn’t trade the market based on those factors. Now come into the real world where you cannot predict the factors… add to that that even if you could they are not predictive of market direction in the way you think… and we have two ways to lose.
For me, trading ideas are simply and robust and quantifiable. I will accept that some things cannot be quantified, but if I’m going to trust something in my own work and trading it is going to be something that I can quantify. I don’t need macro context… it’s just narrative noise… the stats stand on their own. At least that’s my perspective and current thinking.
Also, you’re right that translating from trading idea to test to actual trading application is not trivial. Also the need to understand basic stats deeply… none of this is easy, especially with the depth needed.
Good points! Thank you for starting the discussion.
It’s good to disagree. If we were all the same we could be programmed like algos !
This is not advertisement. I personally use Amibroker for “studying”. I think it is a great tool for studying, but not a great tool for being an execution platform. Currently I build my own execution platform with Excel, AutoIt, IB API. The worst part of Excel VBA and AutoIt is that they are really hard to test and debug. I think I’ll use C# next time. I know there’re also .NET components for building your own trading platform, OpenQuant for example, but I’ve never tried any one of these. Maybe professional traders will need these kind of tools to build rigorous trading platform.
Well, yes I think there’s a good way to use Amibroker-type languages for studying, particularly if you offload the hard number crunching to better platforms. And congrats on building your execution platform. Personally, I find VBA absolutely miserable, but that’s probably because I’ve never really learned how to use it. I’ve written thousands of lines of VBA code, but still haven’t gotten around to learning the language so I suppose my critiques should be muted. 🙂
And yes there are people, especially in past times, that made money without a good understanding of stats and programming. Those people invariably learned enough to internalize some of the messages, but they also had a lot of trouble adapting when the world changed. In the cases I am familiar with, many of them… actually… I’ll just write a post on this. 🙂
Adam, I saw your Bigmike webinar on youtube and have been reading your blog with interest, especially your posts about statistical analysis. In particular your graphs that show difference of means, p-values, etc for various trade signals.
I’ve been struggling with how to best evaluate my trading ideas. As a retail guy with no stats background I find it an uphill climb. I’ve read about significance tests and seen some examples but I’m having a hard time translating these examples into constructing my own trading tests. I’m looking for some hard stats to say whether or not an idea has a good shot at being better than random. I don’t feel I get that from the “by hand” method of manual testing.
It seems like what you have done in the examples you’ve shown is take the average N-day return of an asset over some time X, and compared it to the average N-day return following an occurrence of your trade signal in that asset over the same time period. Is this correct?
Unless you’ve already covered this somewhere that I’ve missed, I think some more in depth content on this subject area would be a great thing!