To solve this problem I’ve created a Cython wrapper for many of the more common BLAS functions. It’s called Tokyo: I often name code after cities and both Tokyo and BLAS/LAPACK were big, fast and very foreign to start with! At the moment Tokyo only wraps the BLAS routines for vectors and general matrices with single and double precision. If you want to add other things such as banded matrices, complex numbers or LAPACK calls: just look at what I’ve already done and add the functions you need. I’ve also added a few extra functions that I find useful when doing matrix calculations. The idea is that Tokyo will eventually encompass all of BLAS and LAPACK.

The speedup you can expect varies a lot depending on the size of the vectors and matrices involved and the operation being computed. An outer product on length 4 vectors can be over 150 times faster, while a 20×20 matrix multiplication is about the same speed. After installing Tokyo, execute the single and double precision speed tests to see how much faster the Tokyo routines are compared to using Numpy on your machine. For my own reinforcement learning research I’ve managed to get between 5 and 20 times speedups with various algorithms.

Here’s a simple example of a vector outer product, first in pure Python with Numpy:

import numpy as np

print np.outer( x, y )

And now a fast version in Cython using Tokyo:

import numpy as np
cimport numpy as np
import tokyo
cimport tokyo

print tokyo.dger( x, y )

You see I use the BLAS names for functions, here “dger” is short for “double precision general rank-1″. BLAS naming seems confusing to start with but after a little study it’s actually quite simple. As per usual in Cython, giving x and y a cdef type isn’t necessary, but it allows Cython to produce faster C code.

As I suspect that quite a few other people have run into this matrix speed problem I figure it’s worth sharing the code. Finally, thanks to Dag Sverre Seljebotn’s Scipy 2009 tutorial which got me started on writing this.

Get Tokyo here: www.vetta.org/software/tokyo_v0.3.tgz

First of all, things like SciPy are obviously good tools for certain types of problems. If you’re writing a front end system for a corporate database system or a text editor — forget about it.  That said, plain Python is a pretty good way to code most things in terms of being a language that is somehow both concise and simple.  In my opinion the only real downside is speed: it’s about 100 times slower than C, or around 50 times slower than java.  Anyway, that’s another topic, here I’m more focused on using Python in combination with SciPy.  If you’re dealing with complex transformations on data that has many dimensions, for example simulating networks or machine learning, SciPy is likely to be good tool for the job.

One of the most popular comments on reddit basically said that while writing a program to use matrices can produce very short code, it’s far too difficult to debug. I don’t buy this. Firstly, in my experience, if you get the matrix computations wrong the output is normally complete junk. That’s good news, because the really hard bugs to fix are ones where the output is almost right, or is right most of the time. You also can be pretty confident that all the fancy algorithms doing efficient matrix multiplications etc. are correct. Furthermore, because the code is so short there are only a few places where you have to look to work out what you’re doing wrong. Finally, if you’re working in areas such as machine learning or finance then the normal way to describe things is already matrix notation. If you want to, say, construct a non-parametric density estimate by using a multivariate Gaussian kernel, then your starting point is going to be a set of matrix equations.

What are the problems with SciPy?  As pointed out by one of the commenters to my last post, one major problem is documentation.  This is common with open source projects it seems: people like to write code but they don’t like to write documentation.  The basics seem to be covered ok, but as you get further into the libraries it’s not so good.  And it’s not just that the documentation is missing, but rather the structure of the whole SciPy system is confusing.  For example, let’s say you want to interpolate in multiple dimensions.  You go to the interpolate module and what you find is that it only does up to 2 dimensions.  So you’re out of luck, right?  No, the trick is to look in the image processing library ndarray which has interpolation is as many dimensions as you want.  For a new user this is all really confusing.  Hopefully, as the whole system matures these things will improve.  As the documentation is wiki style, once I’m a bit more knowledgable I’m going to do my part and help out in a few places.  It’s really the least I can do.

Then there is the question of performance.  I’ve been told that the standard Ubuntu SciPy package doesn’t use matrix libraries that are all that efficient.  If you want to have fast SciPy then you need to install BLAS, LAPACK and ATLAS yourself.  Using ATLAS you can then compile the libraries so that they will be automatically optimised for your particular CPU, memory cache sizes etc.  You can then get SciPy to use these optimised libraries.  Just look under the installing SciPy for all the instructions.  I haven’t done this yet but people who have report significant speed increases.

If your code makes relatively few calls (say tens of thousands or less) on relatively big matrices (say with thousands of elements) then (depending on what calls you’re making) your code will probably spend most of its time busy in the matrix library.  With an optimised library you’re then doing pretty well in terms of performance.  You might want to look at things like Parallel Python to take the performance to the next level.

Where performance becomes a problem is when you have the reverse situation: many calls (millions) on relatively small matrices (hundreds of elements or less).  When what tends to happen then is that all the computation time is spent going from Python into the library and back out again. The time spent actually inside the library doing computations is negligible. In this case your code would be much faster if it was written in C++ and directly called the BLAS libraries. Other than trying to rethink your algorithm so that each matrix operation does a bigger chunk of the total work, I’m not sure what to do about this. Using something like Pyrex doesn’t really help all that much (Pyrex takes Python with a few extras and compiles it into C that can then be compiled into a binary library that Python can use). The problem is that even in Pyrex code you still have the overhead of going into and back out of the SciPy library. You can directly access matrices (ndarrays) from within Pyrex in a way which is very efficient, however you’re then left writing the matrix code yourself — ok if you want to do something trivial, but not a good idea if you need to efficiently multiply two matrices.  So in short, if you need to make a large number of calls on relatively small matrices then I’m not sure how you can do this in a way that can compare to C++ used with BLAS.  If I ever find a solution, I’ll let you know.

1) Write the code in python like I would write it in, say, java.  I have data stored in some places, then I have algorithms that iterate over these data structures computing stuff, calling methods, changing values and doing various complex things in order to implement the desired algorithm.  10 pages of code, somewhat general.

2) Then I realise that in a few places I don’t need to iterate over something, I can just use some vectors and work with those directly.  7 pages of code, a little more general.

3) Then I realise that part of my code is really just running an optimisation algorithm, so I can replace it with a call to an optimiser in one of the scipy libraries.  5 pages of code, and a bit faster now.

4) Then I try to further generalise my system and in the process I realise that really what I’m doing is taking a Cartesian space, building a multi-dimensional matrix and then applying some kind of optimiser to the space.  3 pages of code, very general.

5) Finally I’m like, hey, how far can I push this?  With some more thought and spending a few days trying to get my head around all the powerful scipy libraries, I finally figure out that the core of my entire algorithm can be implemented in an extremely general and yet fast way in just a few lines.  It’s really just a matrix with some flexible number of dimensions to which I am applying some kind of n-dimensional filter, followed by an n-dimensional non-linear optimiser on top of an n-dimensional interpolation and finally coordinate mapping back out of the space to produce the end results.  2 pages of code, of which half is comments, over a quarter is trivial supporting stuff like creating the necessary matrices, and just a few lines make the necessary calls to implement the algorithm.  And it’s all super general.

Now this is great in a sense. You end up throwing away most of your code now that all the real computation work is being done by sophisticated mathematical functions which are using optimised matrix computation libraries. The bottleneck in writing code isn’t in the writing of the code, it’s in understanding and conceptualising what needs to be done. Once you’ve done that, i.e. come up with mathematical objects and equations that describe your algorithm, you simply express these in a few lines of scipy and hit go.

It’s not just with my financial software either. I recently implemented a certain kind of neural network using nothing but scipy and found that the core of the algorithm was just one line of code — a few matrix transformations and calls to scipy functions.  I hear that one of the IDSIA guys working on playing Go recently collapsed the code he’s been working on for six months down to two pages.

The downside to all this is that you spend months developing your complex algorithms and when you’re done you show somebody the result of all your efforts — a page or two of code.  It looks like something that somebody could have written in an afternoon.  Even worse, you start to suspect that if you had really known scipy and spent a few days carefully thinking about the problem to start with, then you probably could have coded it in an afternoon.  It’s a little embarrassing.