Getting Stack Traces -- Detailed Version

This page provides detailed information on obtaining stack traces with debugging symbols. GettingTraces should be enough information for most people to get the extra information, so this page is here for those interested in extra background information or that have more complicated cases. To obtain a stack trace with debugging symbols, you need to do the following:

  1. Install debugging information packages from your distro

  2. Learn what a stack trace is (as well as what it isn't), so that you can submit the right information

  3. Obtain a new stack trace and submit it.

Installing debugging information packages

You need to install packages with debugging information for the program that crashed plus its major dependencies. The steps to do so vary by distro, so you'll need to read the section of Distro Specific Instructions corresponding to your distro to find out how.

Note that if you cannot figure out which packages to install (the Distro Specific Instructions page merely says how to install them in general, not which ones to install), there are a few pointers to help you:

  • Installing extra packages doesn't hurt (other than taking up a little extra disk space)
  • Installing the debugging packages for the program that crashed plus all its dependencies is a pretty good rule of thumb in practice, even if you don't really need the debug packages for every dependency.
  • The libraries that the stack trace shows being used (see below) provide the best hint at determining the exact packages needed. Find out which package owns the library (e.g. 'rpm -qf <library name>') and then look for a debugging information package with a similar name.

  • You are welcome to reopen your bug report and ask for which packages to install, if the above steps don't work or you can't figure it out
  • If someone else tells you which packages to install, they may use different names than the exact ones you need. This is because distros might rename modules or split them up, meaning that you may have to translate 'glib' to 'libglib2-dbg' or 'gtk+' to 'gtk2-debuginfo').

Learning what a stack trace isn't as well as what it is

Things to avoid; what a stack trace isn't

First of all, let's cover a couple quick things that a stack trace is not. Please do not submit any of these if you are asked for a stack trace:

  • The output of strace. strace gives a system call trace, not a stack trace.

  • The output of gdb starting up. Lines beginning with things like "Loading symbols", "Reading symbols", or "(no debugging symbols found)" are not useful.

What a stack trace is

A stack trace is a list of function calls that leads to some point in the program (which happen to be listed in reverse order). Stack traces without debugging symbols typically consist of several lines (also known as frames) with the format

#<frame number>  <random memory address> in <function name> () from <library name>

while stack traces with debugging symbols consist of several lines in the format

#<frame number>  <random memory address> in <function name> (<variables and their values>) at <filename>:<linenumber>

Note that both kinds of lines can be mixed in a single stack trace when debugging information is only available for some of the relevant libraries. While stack traces without debugging symbols can sometimes be useful to developers, most often developers will need stack traces with debugging symbols to be able to do anything.

An example of a stack trace without debugging symbols is:

[New Thread -1227430208 (LWP 5459)]
0xb721c071 in __waitpid_nocancel () from /lib/tls/libpthread.so.0
#0  0xb721c071 in __waitpid_nocancel () from /lib/tls/libpthread.so.0
#1  0xb7ee01a8 in libgnomeui_module_info_get ()
   from /usr/lib/libgnomeui-2.so.0
#2  <signal handler called>
#3  0xb7f7728a in image_loader_start () from /usr/lib/libgthumb-2.6.9.so
#4  0xb756223e in g_main_context_is_owner () from /usr/lib/libglib-2.0.so.0
#5  0x080e99c0 in ?? ()
#6  0xb75be218 in ?? () from /usr/lib/libglib-2.0.so.0
#7  0x08383c80 in ?? ()
#8  0xb75beaa4 in g_idle_funcs () from /usr/lib/libglib-2.0.so.0
#9  0xb756017a in g_source_is_destroyed () from /usr/lib/libglib-2.0.so.0
#10 0x08383c80 in ?? ()
#11 0xb7f77270 in image_loader_start () from /usr/lib/libgthumb-2.6.9.so
.
.
.

Note the lines with "??" in them; those make a stack trace far less useful. Also, for the curious, the most useful frames in a stack trace are the ones immediately following the "<signal handler called>" line (since, remembering that functions are listed in reverse order, these lines correspond to the function calls immediately before the crash).

A stack trace with debugging symbols will look similar to the following:

[New Thread -1228469328 (LWP 7113)]
0xffffe410 in __kernel_vsyscall ()
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xb770548b in __waitpid_nocancel ()
   from /lib/tls/i686/cmov/libpthread.so.0
#2  0xb7e048e6 in libgnomeui_module_info_get ()
   from /usr/lib/libgnomeui-2.so.0
#3  <signal handler called>
#4  IA__g_str_hash (v=0x0) at gstring.c:95
#5  0xb768e73f in IA__g_hash_table_lookup (hash_table=0x93ba780, key=0x0)
    at ghash.c:242
#6  0x080f0620 in get_link_files (target_file=<value optimized out>)
    at nautilus-file.c:1355
#7  0x080f0713 in nautilus_file_emit_changed (file=0x8d90080)
    at nautilus-file.c:5396
#8  0x080dcaea in nautilus_directory_emit_change_signals (directory=0x0, 
    changed_files=0x940b8b0) at nautilus-directory.c:762
.
.
.

In this stack trace, frames 0-2 don't actually have debugging symbols, but only the frames after the "<signal handler called>" are relevant.

Also worth noting is Alex Larsson's guide on The art of decoding backtraces without debug info which provides some directions on how to make a stack trace without debugging symbols more useful. It's a huge amount of work and can only help a little bit at most, but if it's absolutely impossible to get a stack trace with debugging symbols from reporters, developers might find this handy.

Obtaining a stack trace

There are a few different ways to obtain a stack trace with gdb:

Obtaining a stack trace from a core dump

A stack trace can be obtained from a core dump (which is an image of a process's memory at the time of the crash). See man core for information about core dumps. Creation of core dumps might be disabled by default on your distribution (f.e. on Ubuntu or Fedora). This can be changed for the current process using:

# show max core dump size
ulimit -c
# set max core dump size to unlimited
ulimit -c unlimited

A stack trace can be obtained using gdb:

$ gdb [prog] [corefile]
(gdb) bt                          # obtain backtrace
(gdb) thread apply all bt         # obtain backtrace for all threads
(gdb) quit

Obtaining a stack trace using GDB

First, run the following command to start gdb:

  • gdb name-of-program

Where name-of-program is the name of the program that you know how to crash (for example: /usr/bin/gnome-panel).

Then, at the gdb prompt, type:

  • run

Once the program is running, reproduce the crash and go back to the terminal where you ran gdb. The gdb prompt should be shown - if not, hit Control+C to break into the debugger. At the gdb debugger prompt, type:

  • thread apply all bt

If you see the line

---Type <return> to continue, or q <return> to quit---

just hit each time the Return/Enter key until you see the gdb prompt again. The output is the stack trace. Cut and paste all of it into the bug report.

If that does not work (meaning that you don't get any output--this may be the case in programs which are not multi-threaded), type bt instead. If you still do not have any output, read this note about obtaining a stack trace under special circumstances.

You can quit gdb by typing quit.

Obtaining a stack trace using GDB with a running program

First, you need to know the name of the program that is running (e.g. 'nautilus' for the file manager, 'gnome-panel' for the taskbar, etc.). Now, while the program is running, open a terminal and type

  • ps aux | grep name-of-program | grep -v grep

The second column of the output provides the PID (process ID) of the running program. Now run

  • gdb $(which name-of-programPID

substituting for name-of-program and PID appropriately (e.g. gdb $(which nautilus) 6732). This will attach gdb to the running program. Then, at the gdb prompt, type:

  • cont

to allow the program to continue running. Make the program crash, then go back to the gdb prompt and type

  • thread apply all bt

If you see the line

---Type <return> to continue, or q <return> to quit---

just hit each time the Return/Enter key until you see the gdb prompt again. The output is the stack trace. Cut and paste all of it into the bug report.

If that does not work (meaning that you don't get any output--this may be the case in programs which are not multi-threaded), type bt instead. If you still do not have any output, read this note] about obtaining a stack trace under special circumstances.

You can quit gdb by typing quit.

Obtaining a stack and JS trace using GDB for an already running gnome-shell

This is an automatic way, to get informations without any interaction (if you need that, you should use another machine).

Simply open a gnome-terminal in your session and run:

sudo gdb -p $(pgrep -U $USER -x gnome-shell) -batch \
  -ex "set logging on" -ex continue \
  -ex "bt full" -ex "call gjs_dumpstack()" \
  -ex quit

This will just attach a gdb to your shell session (causing a quick initial freeze) and the continuing until it crashes. If bad things happen, you'll get a gdb.txt file saved in the directory where you get this script launched, and your journalctl /usr/bin/gnome-shell will include a JS stacktrace (just look for == Stack trace for context). An easy way is:

journalctl /usr/bin/gnome-shell | grep -A 30 "== Stack trace for context"

Obtaining stack traces under special circumstances

If you do not get any output from gdb after typing thread apply all bt or bt, it may be because the program is run as root or as another user. In GNOME, this is the case when running some games. In such cases, you will need to be root in order to capture a trace. So, quit gdb, login as root, and then repeat the steps to obtain a stack trace. Some applications may require special treatment in order to debug or may have additional ways of assisting the user or developer to get extra kinds of debugging information. Please examine Bugsquad/TriageGuide/ProductSpecificGuidelines for details.

Community/GettingInTouch/Bugzilla/GettingTraces/Details (last edited 2017-10-31 13:52:35 by AndreKlapper)