Reader Jas asked a question about backtesting:
I was wondering if you’d be able to advise me on how to back test a strategy on some custom indicators. I’m aware of back testing software in the market but not sure if it will do the job I want to accurately. Are there firms or software I could use to backtest a strategy in order to have complete faith in its efficacy? The added complication is that I use several rules so have no idea how the standard software out there would incorporate these rules.
Good question, and there’s no easy answer. I would point your question in a slightly different direction. You’re asking about software or firms to do backtesting. Of course, these things exist, and you can always find someone to take your money in return for a service. 🙂 The people doing this work will have slick marketing material and will explain that they use Monte Carlo modelling and non-parametric statistics, and they’ll hope you will be dazzled and for over the cash for the test.
But wait.
First, I think you need to make sure you have a solid education in statistics, probability, and backtesting. Yes, I know this is an annoying answer because people just want a solution, but there are so many ways to go wrong with this kind of work–so many pitfalls and so many places a perfectly well-intentioned person can make fatal errors–you have to educate yourself. That education doesn’t have to take a lot of time, but a year or so, depending on your background, spent really wrapping your head around statistics is time well spent.
Also, educate yourself on backtesting in general. A good place to start is the module in the trading course Ernie Chan did for us. Why did I ask Ernie to do this? Because I think he is absolutely the best at teaching and communicating these concepts, and he put together a powerhouse of a module that will show you many of the things that can go wrong with backtesting, and will lead you toward asking the right questions. The course is free, so head over there and check it out right now to start.
From there, continue to educate yourself on backtesting techniques. Learn, for instance, what might be wrong with having a rule set, tweaking it based on results, and then running another test. Or adding an indicator, running a test with a few different settings, and then deciding which setting to keep as you work forward. It’s very easy to produce a nice backtest, but the goal with all of this rigorous work is to have something that works in the future. That, my friends, ain’t easy.
Personally, I got a lot of value out of doing my first simple backtests in Excel. Yes, it’s perhaps the worst backtesting environment for a number of reasons, but you can see and manipulate the data and understand your work in a way that is more difficult in software like Stata (which I use), Matlab (which many more people use) or in coded systems written in languages like Python or R (which is probably what you ultimately should be using).
Learn statistics. Understand probability. Understand how backtests can go wrong. Craft simple backtests of simple concepts, and look for an edge, and then test your system. That would be my advice, but check the comments to this post as I’m sure some readers will have other ideas and perspectives. (Word to the wise: I’ll delete commercial links from comments.)
In my favorite podcast(s) episodes 27 & 28 “So you want to be trader?” You hit on the importance of a successful trader ultimately must trade using the “probabilities of mathematics.” Could you expand on this concept? Is there a trading platform/system you would recommend for quantitative trading? Thank you.
Totally agreed. I think it is worth pointing out that understanding probability and statistics from a mathematical perspective is not generally enough, in that you really need to develop a fundamental intuition and appreciation of randomness. A good exercise as Adam suggests is to try build some simulations to allow you to develop this intuition. A super simple example is to create a coin tossing simulation in Excel, that perhaps involves 1000 sequential coin tosses. Run 10,000 of these simulations and plot your P&L over the 1000 sequential tosses, and you may be surprised to see how much money you can win/loss in such a game, even with a fair coin. In the end, all of the 10,000 simulations are a possible outcome of the underlying process, and in life you only experience one of those outcomes, but which one is impossible to know. Engaging in this game on the basis of it having a zero expectation, and therefore not expecting to lose anything, could lead to a nasty surprise.
Pingback: 09/25/15 – Friday’s Interest-ing Reads | Compound Interest-ing!
Might not be a very exciting answer, but once the basics of statistics are learned, Excel is all the software one needs for quite a time.
Excel is functional by nature (as close to strict, bug free, side-effect free, lean, math toolbox as you’ll ever get) : by using it instead of a slick ready-made program, one has to go through the visualization process Xavier describes, crafting formulas by hand, copy pasting the source data (hence paying more attention to it than when its fed into the system from a source one trusts by virtue of not seing it), etc. The benefits of doing it this way are huge.
* program basic statistical concepts by hand, like standard deviation (and then realize how complex it is and why maybe an ATR is not such a stupid idea),… to understand what they do, and see it.
* learn to calculate and plot an equity curve, a drawdown curve, deciles, sharpe, anything.
* learn to detect when you accidentally use future information when back-testing (hint : the returns from the equity curve are always very satisfying), because this is deadly and will happen. Many many times.
* everything with excel is slower than pressing a button in *myawesomebacktester*, but as a result, it helps enforce discipline and focus.
Adam, as one of your mentors used to say: when you have a question, ask the data. 🙂 Where is the data? What would be your suggestion for a good place to get historical data? (for instance, OHLC daily values for US stocks, with dividends and split details, survivalship bias free, with their historical affiliation to some main indexes – if, for example, I want to check my strategy, between 2000-2015, on stocks belonging to SP500 at the moment I entered the trade).
A complete amateur when it comes to this… But how would one back-test the pullback strategy? Scenario: Price pierces the Keltner Channels, pulls back to the 20 EMA, and then an entry is taken. Would Excel be used for this or something else? I know Adam has done this, but I think it would benefit me greatly if I actually learned how to do it myself.
Pingback: The way billionaire Warren Buffett defines success has nothing to do with money: “I measure success by how many people love me.” – Bamboo Innovator Daily: 22-27 Sep (Tues-Sun) | Bamboo Innovator