It was Blaise Pascal that said “The present letter is a very long one, simply because I had no leisure to make it shorter. ”

See: http://books.google.com/books?id=hUMRAAAAYAAJ&pg=PA339#v=onepage&q=&f=false

in the penultimate paragraph.

Seems apt.

“Perfection is attained not when there is nothing left to add but when there is nothing left to remove.”

- Antoine de St. Exupery

Also, I realize that you said that you tested var() with your own data, which does not necessarily mean that you tested it with complex data.

Ivo Maljevic

>> By that same logic, if one finds a bug in MATLAB, they could consider it unreliable to do “any processing”. It is not often I compute variances on complex data.

The point was not that you do not trust some s/w if there is a bug, but the question of trust comes into play if something as basic as a variance calculation is not done properly. And the variance calculation was not considered a bug but just a different interpretation of what the variance definition is.

Just out of curiosity, how is it possible that when you compare the output of var() function with matlab output you get the same thing? I am running Ubuntu 8.04 and the scipy that comes with it. Do you have a different version of SciPy?

Could you try the lines I wrote on my message submitted on May 13?

Thanks for submitting the patch and helping make Scipy better!

“I do not know if that will find its way into the release or not, but I hope it will as that ‘bug’ is significant enough to make SciPy unreliable tool for anyone who wants to do any processing”

By that same logic, if one finds a bug in MATLAB, they could consider it unreliable to do “any processing”. It is not often I compute variances on complex data. I have found the var() function to work on the data I have, tested by comparing its output with values I know are correct. But really, all software has bugs, even MATLAB.

Comparing MATLAB’s stability with Scipy is not a fair comparison since MATLAB has had 20 more years to fix its bugs. Many years ago when I was an avid MATLAB user, I contributed some bug reports to MathWorks that caused me some headaches. But most of my headaches with MATLAB have nothing to do with bugs. First, its a horribly designed language created by electrical engineers who should have taken a course in language design prior to writing the language. Second, it promotes bad programming style; some users write clean code but most just learn enough to quickly hack together a filter that no one else can read. Third, its hard to spread computation across machines due to licensing limitations, causing needless tension among colleagues who are all fighting for licenses. People end up cutting experiments and moving on so less science gets done. Fourth, it is harder to collaborate with others because your collaborators must purchase MATLAB licenses to use your code. Fifth, it is primarily an interactive environment rather than a batch environment, encouraging experimentation and statistical analysis which is not reproducible. Sixth, it is difficult to work with any data structure other than a matrix (e.g. a graph, hash map, tree, lists, heaps). This is frustrating when needing to write algorithms requiring richer data structures.

Have you ever tried writing a large application in MATLAB? Are you familiar with MATLAB’s object oriented interface? It is unusable. Modifying a data member causes the object to be copied prior to modification. At some point, prototype code needs to make its way into a larger application. While most people intend to rewrite their prototype code in C or C++, it is often too costly to do so they end up writing their application in the same language as their prototype code, i.e. MATLAB.

You should not be surprised if there is symbol overlap in packages. It would be very difficult for two projects to maintain mutual exclusion in their namespaces. In general, it is bad programming practice to import everything. It should only be done at the python prompt, not in programs.

With regard to memory management, I was unable to handle large data sets with MATLAB. When Scipy came into the picture, I could handle much larger data sets. Also, I can pass very large arrays into functions and modify their contents without any copying. This is particularly helpful when dealing with a memory footprint that approaches the 2GB limit.

You raise a good point about complex data. However, MATLAB has its data type headaches as well. For example, if I have an int8 array (say an image), not all arithmetic operators are supported in MATLAB. Images often need to be converted to a double prior to doing arithmetic on them. That’s annoying, especially when you’re working with large images. I can do most arithmetic operations on int8 arrays in Scipy.

Often, the semantics of Python/Scipy can be quite different from MATLAB. This causes some frustration when migrating from MATLAB but this is temporary. Eventually, I believe the committed learner should find the Python/Scipy framework to be much more flexible.

Sorry for your bug–bugs are very frustrating. It is good that you submitted a patch. Thank you for doing this.

Damian