I received a good question in response to my previous post on starting to learn Python for finance. Here’s Mark’s question:
Thanks again for all your work and all that you share. I am a little behind you in programming abilities but working furiously through my second MOOC on programming in python. I am finding it quite challenging at times (to the point of utter frustration) which is mostly caused by wondering if what I am learning has any relevance to programming in trading? I am wondering what resources you had access to when you embarked upon your (python) programming journey and if they were more trading related? Are there any resources you would recommend to help me through my struggles?
About a year ago, I announced my intention to create a training course for Python. This has not happened, and the reason why is that I don’t think it’s possible to effectively teach someone a programming language–you have to do the work to learn it yourself. In this sense, it’s not unlike learning a performing or artistic skill; you can be taught the basics but you don’t stand a chance of understanding them until you actually use the tools, make mistakes, and learn to fix those mistakes. There are a number of pretty good books and courses out there, so I even hesitate to mention the resources I’ve found useful. Undoubtedly, there are other, maybe better resources, but I’ll share the ones I found most useful.
Learning the core language and libraries
I think, to a great extent, and I already laid out a clear roadmap for what you need to learn in my previous post. Let me stress, again, that you need to pay special attention understanding basic data structures and how to manipulate them. Examples, for instance, of how to slice and manipulate strings are going to be critical when you start to deal with financial data. Also, one of the mistakes I made early on was to ignore dictionaries. Don’t do that; dictionaries in Python are your friends (and they are wicked fast, no matter how much data you cram into them!) With this is mind, here are the websites and resources I found most useful for my first steps in learning this language.
Learn Python the Hard Way (website here, and book from Amazon here.) This is an old-school introduction to programming with a good emphasis on basics, repetition, and learning how to learn. Just go here (or buy the book) and patiently do everything he tells you to do. My only complaint is that this is very Python 2.X centric, and I think you probably want to work in 3.X eventually. (This material was written a few years ago, and, to some extent, 3.X has matured. There is still a ton of legacy code in 2.X (most?), and it’s pretty easy to make the transition between the two versions.) If you are a typical Windows user (i.e., you’re terrified of the command prompt) this is also a good introduction to break you out of the way you’ve interacted with computers for years. Nothing is perfect and we could nitpick forever over the details of this course, but it’s certainly one of the best I’ve seen.
Beginning Python from Novice to Professional (book here). For me, I found it most effective to be taught the same concepts more than once. I don’t mind going back to “this is a variable” stage, because you’re going to learn new things each time you cover that ground. This book has generated some controversial Amazon reviews, but I found it to be really fantastic and very accessible. This book will also take you into the core libraries in a bit more depth and get you deeper into the realm of object-oriented programming. If you’re new to that topic, you might find it frustratingly opaque (You’ll have conversations with yourself that go something like, “yes, I understand that dog and human are both instances of class mammals which is a subclass of animals. I even understand that dog and human both share the method eat and the attribute is_hungry, and that dog implements .bark while human, normally does not. This kind of makes sense, but why do I care?” Trust me, you do and will care, but you probably won’t know why until you need to use these concepts.)
You also need a tool (development environment) for writing your code. Technically, you can write Python code in any text editor. I agree with some of the teachers who say you should start learning by doing just this, but I don’t think it’s the best way to do real development work. There are many resources, but I’ve become a big fan of PyCharm. When you finish LPTHW, start looking at IDEs and take a good look at PyCharm.
Also, get comfortable with Jupyter (the new incarnation of the iPython notebook). I find this to be a really useful tool for prototyping code and for digger deeper into data. You might be just as happy working with Jupyter and a text editor as working in a full-fledged IDE, but different people have different preferences.
Sharpening the saw
If you work through the two resources above, you’ll have a good grasp of the basics. Now, start doing things. Build some programs and work on some projects. Here are a few other books I found useful.
You are going to need to learn Pandas to start working with data. I would buy Python for Data Analysis as a reference. I think it’s a book you need, and you will find yourself working through many of the examples. You can also start with 10 Minutes to Pandas (a short online introduction)–a great little tutorial to get you started, but then you’re quickly going to have to dig deeper. You should be able to load a csv file into Python, and/or pull data from Yahoo! Finance, and then you are well on your way to doing “real work”.
As you continue to develop skills, Think Python is a great book to teach you to think like a programmer. Programming isn’t just about learning what commands do, it’s an approach to problem solving. It’s about learning to take complex, real-world tasks and translate them to precise steps and solutions. It’s about understanding errors and reasons for errors. It’s about learning a new way to think, and this is one of the best books I’ve found for teaching yourself those skills.
Along the way, you need to be programming. Don’t just work through these resources and learn what they tell you to do. Find projects, and make projects for yourself. They can be very simple: generate a password for a website, for instance. They can also be much more complex: pull Yahoo! Finance data for two stocks and track P&L for a pairs trade over the past 100 trading days, for instance. You’ll make a lot of mistakes and go down a lot of dead ends, but that is how you learn. Your mistakes are learning opportunities.
Once you feel like you might actually be a programmer, it’s time to take one more pass through the language. The book I found most useful for this is a giant reference manual: Leaning Python. This is a deep and comprehensive look at the language, and I think it would be most useful to someone who already has a decent working grasp. In my opinion, time spent working through this is time well spent.
I’ve been programming for a living for over 15 years. Used C++ at first, then Java, then C#. I learned a little Python to do some web programming and to teach my kids to program using a kid friendly intro to programming book that used Python. I even spent a few months trying to use Python with numpy and pandas to do machine learning and statistical analysis for a trading system study (never finished it!). My results with Python were mixed. I would say it is very difficult to learn, and you have to learn through a lot of trial and error, even if you already are an expert in another programming language. What I found most frustrating was the lack of documentation and examples, especially with the statistical and machine learning packages. That’s the reason I gave up on it, for now. Plus, it seems more difficult to debug Python code than Java or C#.
Call me old fashioned, but I prefer a real programming language, one that compiles to machine code. Python is a scripting language that uses a bunch of libraries like numpy that are written in C or C++ so that they run faster (but make it harder to debug.) Don’t get me wrong, I think Python has a place. It was great once I had loaded my huge data set and I learned a few commands to generate statistical charts from the command line interpreter. Python is best for rapid prototyping, for stringing a bunch of other tools together, or for doing Matlab type operations. However, for my automated trading system I feel much better using C# because it has the right amount of flexibility, power, and ease of use.
Hope this helps some novice programmers who might be wading into the murky waters.
Hello Chris,
Isn’t C# or Java so much more difficult to learn than Python? I just heard Ernie Chan say (in the trading course found here in Adam’s website) that a language such as Python will increase your productivity 5-fold vs. languages such as C++ etc. Is it that much more difficult to debug? I don’t know either language, so I’m trying to find out others opinions on the matter.
Good point and I think you hit on the most important issue for newer programmers: Python is just much more accessible than C or a similar language, and is relatively easier to learn.
You are correct, the c-based languages are very hard to learn at first, and even harder to get proficient with. You have to write a lot of code to achieve even simple things and you have to focus a lot of effort in the mechanics of coding rather than the problem itself. Python does alot more for you and makes it easier to focus on the problem rather than on the nitty gritty of code
Very experienced C, C++ etc coders learn to think in a certain way to become excellent with those languages, but they have ti unlearn a lot to become proficient with Python (or R or other higher level languages).
Well, I don’t think anything you say here is wrong, but you may be an experienced enough programmer that you underestimate the learning curve for most people in C. It sounds like C#, etc. is the obvious choice for you, but it would be much harder for me to be productive in C than Python. (See Rodrigo’s comment below.)
The documentation for Python is uneven, but it’s growing and is very good in some aspects. In others, as you say, not so much, but it is constantly evolving–if the documentation put you off, maybe worth a second look now. Just a thought.
No, you are actually misleading them. All the c-based languages have very steep learning curves and take years to become proficient at, Python is a much better language to focus on the problem at hand. I would expect you to struggle with Python because your thought processes are shaped by the language you use, and clearly you are now highly proficient in C and related languages. In effect you will try and do c++ programming in Python which wont work.
Hi Adam,
I agree on the point that programming needs to be learnt, not taught. Even doing all the courses and tutorials available on the Internet, it is not until one thinks his/her own problems and writes the code to solve them, that all the learnings are fully assimilated.
Regarding this, there’s an issue I’ve been facing for quite a long time, and it is that my code is always customized for each problem, so I can’t reuse parts of it. Every time I want to test a new strategy, I have to start my code from scratch. I’ve been coding for a while and understand functions/classes just right, and use them frequently, but it is in their design where I’m failing. Each new strategy requires a new input parameter, a new output, etc so I feel it’s easier to start from scratch that trying to patch previous code and probably miss something important.
Is this something you have experienced? In some sense, structuring code in such a standard way is what places like Quantopian do, so it must not be ‘that’ easy for non-professional coders, but on the other end, I’m not sure whether using a clean slate every single time is the right approach either.
That’s perhaps something that could be taught or you could provide some insight about, something like defining the basic scalable structure of any quantitative trading code.
Regards.
It’s not so hard to recycle chunks of code for some purposes. Different projects require a different degree of customization, but with Python you can basically create your own libraries of functionality and recycle a lot of code. In addition, using classes properly naturally leads to a lot of code re-use.
Maybe the thing to think about is how you structure your projects and what you’re trying to accomplish? If your objectives are quite different, then you will need different code.
Thanks for this post. Going to work through the Novice book …already arrived in the mail 🙂
thank you! Enjoy and please do let us know how your journey shapes up.
Adam,
I am confused about how Python fits in to financial modelling or more specifically coding for trading. I currently use Amibroker and previously Metastock for equities. I am spending more time on forex on the MT4 platform which as you know has its own language and compiler. Is Python a stand alone program to which you import data, interrogate and report or is it possible to compile and convert to use in say MT4? Could you please elaborate or point me to relevant posts that illustrate applications and possible integration of Python in relation to trading?
Many thanks in anticipation of your assistance.
JB
The software package you reference is what I would consider a retail-level package with many limitations. You certainly would not want to limit Python with the structure this package creates, so the best way to think of Python is that it is a free-standing language that allows you to create programs from the ground up.