Psyco does just-in-time compilation if you are running on an x86 machine (32-bit only). It can speed things up a lot, but beware that: (1) certain newer constructs can prevent parts of your code from being accelerated, which wouldn't be so bad, except (2) calling Psyco code from non-Psyco code (including native code) can be even slower than not using Psyco at all! Since some of the forbidden constructs are so useful, like yield, and 64-bit machines are becoming more common, Psyco seems less attractive these days.
Pyrex is a Python-like language which can access both Python objects and C variables and functions. It compiles down to C and can be used directly to write faster versions of Python functions, but is primarily intended for wrapping C libraries, which it makes very easy. A version that works with Python 2.5 doesn't appear to be ready yet. There is a patched version in a Subversion repository at http://codespeak.net/svn/lxml/pyrex/
Garbage collection. Python uses both reference counting and generational garbage collection. By default, the GC does a collection very frequently (every 700 allocations). When you create a lot of objects (especially if you don't plan to release them) it is better to make collections much more infrequent, e.g.:
gc.set_threshold(100000,10,10)If your program doesn't create any circular references, you can just disable the GC altogether with gc.disable(), and reference counting will take care of everything. You can use gc.set_debug(gc.DEBUG_STATS) to see how often collections occur, and gc.set_debug(gc.DEBUG_COLLECTABLE) to see what is actually getting collected, if any.
__slots__. A peculiar feature of Python is that you can make assignments to arbitrary attributes to an object even if the attribute wasn't part of the class definition. This is convenient, but implemented by storing a hash table inside each object. If you are creating millions of instances of a class and want to save memory, you can turn this feature off using __slots__.
Arrays. Various libraries for homogeneously-typed multidimensional arrays are often recommended for scientific applications, to save both time and memory. There are several different libraries floating around, confusingly; but the anointed one seems to be NumPy from the SciPy project. The library is free but the documentation is not.
Profiling. Don't use profile or hotshot (too slow and too inaccurate, respectively). Use cProfile, which is part of Python 2.5.
Time and memory usage. Use time.time() to measure wallclock time, but to measure CPU time, don't use time.clock(), as it wraps around after a short while. Instead use:
import resource
def cpu():
return (resource.getrusage(resource.RUSAGE_SELF).ru_utime+
resource.getrusage(resource.RUSAGE_SELF).ru_stime)
To measure memory usage under Linux, use this
recipe.
cPickle. There are two different ways of pickling objects: you can create a cPickle.Pickler object (call it p) and then call p.dump(), or you can just call cPickle.dump(), which creates a Pickler object and then releases it. The former produces more compact output but uses more time and memory because it remembers all the objects that you've pickled; the latter is faster and uses less memory. You can compromise by using a Pickler object but periodically calling p.clear_memo() to clear its memory. Or you can set the undocumented option p.fast = True to disable its memory altogether. This could produce huge pickle files (if you have a lot of shared structures) and won't work at all for circular data structures.
DBM and friends provide a simple dictionary interface to on-disk databases. Use anydbm to load any available backend; use bsddb to access some features specific to Berkeley DB, like hashes vs. B-trees or setting the cache size; finally, use bsddb.db (formerly known as bsddb3) to access even more features.