The first thing to do if you run out of memory is to make sure you can use all the memory on your system. Some systems by default limit the memory available for individual programs to something less than all available memory. To relax this, use the limit or ulimit command. These are shell functions--see the manual page for your shell for details. Limit is for csh, ulimit is for sh/bash.
Simulations of large networks can consume a lot of memory. Ns-2.0b17 supports Gray Watson's dmalloc library (see its web documentation at http://www.letters.com/dmalloc/ and get the source code from ftp://ftp.letters.com/src/dmalloc/dmalloc.tar.gz ). To add it, install it on your system or leave its source in a directory parallel to ns-2 and specify -with-dmalloc when configuring ns. Then build all components of ns for which you want memory information with debugging symbols (this should include at least ns-2, possibly tclcl and otcl and maybe also tcl).