What tools do I have?
As a hacker, there are a few tools that can be used to track down memory problems. This list only covers tools that target C. Applications written in other languages such as C# will benefit from native profilers.
ps u -- When nothing else works, this will. Look at the RSS column. But, these numbers don't always add to something meaningful. Don't say the GNOME desktop uses X mb of memory by adding up the RSS numbers.
memusage -- This is a tool that comes with glibc. You may have to get an rpm for it. It will give you some nice stats on how the program does all of its malloc's, including a histogram of the size of allocations.
g_mem_profile -- very much like memusage, but it comes with the Glib toolkit. However, it only covers g_new and g_malloc.
memprof -- available from the Gnome CVS, this lets you track where memory is allocated. It can also find memory leaks. Fix for newer versions of glibs
valgrind -- The heavy-weight. This tool gives backtraces of who allocated memory, and can detect leaks. It also looks for common programming errors. However, it makes your programs run dog slow. It can also crash for programs that do stuff that is too complex for it. Valgrind is a tremendously useful tool if you invest the time needed to learn it.
xrestop -- New top-like application that displays what resources applications are using in the xserver.
pmap -- prints a dump of the memory mappings for a given PID, this is a bit like cat /proc/self/maps but more descriptive. It identifies mappings as stack or anon. It'd be nice if it could show you where the brk is. Watch out for excessive stack usage, modern desktop apps especially with toolkits like GTK+ can have very deep stacks.
cat /prop/PID/smaps (available in linux-2.6.14+) shows the Size, RSS, Shared_Clean, Shared_Dirty, Private_clean, Private_dirty sizes for each mapping (pmap does not show as many details yet).
lsof -p PID is also nice for checking the memory mappings. With it you can also easily see if some process is leaking file descriptors: lsof -d 0-255|cut -c -15|sort|uniq -c|sort -n
objdump -- prints information about static memory usage in object files, executables and shared libraries. This is one of the standard provided by the GNU binutils.
datadump is a Python script I wrote to make it easier to find the interesting parts in objdump results --TommiKomulainen
ldd -u -r LIB_OR_BINARY -- prints the unused direct shared library dependencies. To get rid of these unused dependencies the --as-needed flag needs to be passed to the linker at build time. Unused shared libraries add to start-up time and memory footprint.
MikeHearn: This is quite a serious problem: there are 13 unused dependencies for my copy of gedit alone. We attempted to strip unused dependencies by brute force in the autopackage project for portability (as opposed to efficiency) reasons. The attempt failed: there are too many buggy libraries that don't fully list their dependencies in their ELF headers. Two examples are libpng and the Boehm GC. I investigated the Boehm GC case and found that it was done that way as part of a portability hack. The --as-needed option can apparently cope with this (or at least, did for my simple tests) but are GNU specific and also very new: they were introduced only in May 2004. This may be possible to fix using pkg-config variable expansion and some adjustments to the autoconf macros so --as-needed can be automatically used when available, needs more investigation.
- Unused shared library dependencies become required. See in the example below how libz is unused for libfoo.so, but is required for a.out.
$ echo 'int foo (void) { return 0;}' > foo.c ; gcc -O2 -shared foo.c -o libfoo.so -lz ; $ echo 'int main (void) { return foo(); }' > main.c ; gcc -O2 -Wl,--as-needed main.c -L. -Wl,--rpath=. -lfoo $ ldd -u -r libfoo.so Unused direct dependencies: /usr/lib/libz.so.1 $ ldd -u -r a.out $ ldd a.out libfoo.so => ./libfoo.so (0x00757000) libc.so.6 => /lib/tls/libc.so.6 (0x00a78000) libz.so.1 => /usr/lib/libz.so.1 (0x00ca8000) /lib/ld-linux.so.2 (0x00a5f000) MikeHearn: No, you're misunderstanding what you're seeing. The ldd -u -r command lists direct unused dependencies, not unused dependencies in the whole link tree. So this is what I'd expect to see and is normal: nothing is somehow becoming required here.
Gentoo has this page about --as-needed and this bug list shows the work they are doing to make GNOME work with --as-needed. Maybe we could try a tinderbox with --as-needed until it gets stable and incorporate it to default build, just an idea -- NelsonBenitez.
Memory fragmentation profiler, by Michael Zucchi
gcc -finstrument-functions. That gcc option provides a nice way to trace the execution of a program or library. For example, to obtain a call graph of all gtk functions, insert the following code somewhere in the libgtk sources and recompile it with make CFLAGS="-finstrument-functions".
void __cyg_profile_func_enter(void *this_fn,void *call_site) __attribute__((no_instrument_function)) ; void __cyg_profile_func_exit(void *this_fn,void *call_site) __attribute__((no_instrument_function)) ; void __cyg_profile_func_enter(void *this_fn,void *call_site) { printf("> %p\n",this_fn) ; } void __cyg_profile_func_exit(void *this_fn,void *call_site) { printf("< %p\n",this_fn) ; }
Exmap, by JohnBerthels -- exports per-page information from all running processes from the kernel and analyses in user-space to work out which pages (physical or swap) are shared between which processes. A perl/gtk GUI allows this information to be sorted and viewed on a per-process or per-file basis, with breakdowns possible on a per ELF section and per ELF symbol level.
Exmap console - A console version of exmap, for when you need to save the logs to a file.
capture-pixmap - Provides a dump of all pixmaps held for an application by the X server.
Tools that we lack
- A tool to let you view which allocated blocks live in which memory pages.
- A tool to see which memory pages are in core and which are swapped out.
JohnBerthels: Exmap will allow this to some extent. Is there anything missing people would like to see?
A tool to see wasted space due to glibc's malloc pading. The minimum block size is 16 bytes, including the bookkeeping overhead. FIXME: is this accurate? Could glibc already come with such a tool? MatthiasClasen: The data collected by memusage should allow you to calculate this.
Tools for /LongTermMemoryTests