Debugging GNOME Shell

(See Projects/GnomeShell/Debugging/Graphics for graphics-specific information).
(See Projects/GnomeShell/DebuggingJavaScript for a description about how to debug JavaScript code).

First, you need to understand GNOME Shell is split between C and JavaScript. There are a variety of tools available for debugging, but the primary one is gdb. gdb is useful for debugging the C side primarily. By far the easiest setup is to have two computers, and ssh from one into the other.

Starting GNOME Shell under gdb

Do not run these commands from your existing login session; you will freeze the desktop. You need two computers and ssh or you need to be able to switch from the X VT to a console and back reliably (should be fine with all KMS drivers).

If you only want a backtrace from your existing session, you can just use this simple GDB command, but that will only get for you the C and JS stack traces, but no other interaction (unless you don't add other commands manually).

First, obtain a shell that's independent from your GNOME session from which you can extract a log of the output. That means SSH from a second computer or using screen to record the output on a VT on the same computer. If this is from a second computer:

$ ssh sameuser@gnome-shell-computer

(Do not enable X11 forwarding on the SSH session.)

If this is from the same computer, press CTRL-ALT-F2 and, at the black console you see, login as the same user whom has a GNOME session running. Run screen. You should see either a welcome message or immediately see a new command prompt with a little status bar at the bottom of the screen, depending on your distribution configuration. Immediately press CTRL-A, release, SHIFT-H. Screen will confirm that it is now recording the session to screenlog.0 or hardcopy.0, depending on your distribution. Make a note of the file name.

Second, inside this new, independent shell, you must establish the minimum environment variables for X session communication. Download and save the following script to your GNOME Shell system to, for example, ~/bin/xenv.sh:

gnome_session=$(pgrep -u $USER -f -x "[^ ]*gnome-session-binary")
eval export $(sed 's/\o000/\n/g;' < /proc/$gnome_session/environ | grep DISPLAY)
eval export $(sed 's/\o000/\n/g;' < /proc/$gnome_session/environ | grep XAUTHORITY)
eval export $(sed 's/\o000/\n/g;' < /proc/$gnome_session/environ | grep DBUS_SESSION_BUS_ADDRESS)

After you have saved this script, source the file from inside your independent shell with:

. ~/bin/xenv.sh

Third, you are now ready to start GNOME Shell inside of gdb. See the note about the --replace option at the end of this page if you are going to use it.

$ gdb --args ./build/src/gnome-shell --replace
(gdb) run

OR

Attach to an existing instance:

$ gdb -p $(pidof gnome-shell)

OR (e.g., if you are not running in a dev environment)

$ gdb --args /usr/bin/gnome-shell --replace
(gdb) run

Fourth, it's now time to make GNOME Shell crash. Flip back to your GNOME session (if same computer, with CTRL-ALT-F7 or -F1, depending on distro) and proceed to perform the behavior which causes a crash. When you have succeeded, the entire desktop will appear to freeze. This means that gdb is waiting for your attention.

Flip over to the gdb window (on the second computer or on the other VT at -F2) and run the "Acquiring a backtrace" steps below.

When you're done, type "quit" to terminate gdb. GNOME Shell will finish crashing and you'll be back at your GNOME desktop. If you started screen, type exit to log out of screen and then exit again to log out of the console.

Upload or paste the SSH output or Screen log output somewhere.

Do not run the above steps from a terminal on your desktop, or you will freeze your desktop. (If you already did before you read this, you can type CTRL-ALT-F2, then log in, then type killall gdb, then press one of ALT-F1 or ALT-F7 (depending on OS version).

Acquiring a backtrace

The first part here dumps the C stacks for all threads. The second part calls into Spidermonkey and acquires the JavaScript stacks.

(gdb) t a a bt
...
(gdb) call gjs_dumpstack ()

Acquiring a backtrace of a running GNOME Shell in automated mode

Obtaining a stack and JS trace using GDB for an already running gnome-shell

This is an automatic way, to get information without any interaction (if you need that, you should use another machine).

Simply open a gnome-terminal in your session and run:

sudo gdb -p $(pgrep -U $USER -x gnome-shell) -batch \
  -ex "set logging on" -ex continue \
  -ex "bt full" -ex "call (void) gjs_dumpstack()" \
  -ex quit

This will just attach a gdb to your shell session (causing a quick initial freeze) and the continuing until it crashes. If bad things happen, you'll get a gdb.txt file saved in the directory where you get this script launched, and your journalctl /usr/bin/gnome-shell will include a JS stacktrace (just look for == Stack trace for context).

An easy way is:

journalctl /usr/bin/gnome-shell -b0 | grep -A 30 "== Stack trace for context"

In case you need to debug an hanging GNOME shell session, from another TTY or SSH you can use a similar script:

sudo gdb -p $(pgrep -U $USER -x gnome-shell) -batch \
  -ex "set logging on" \
  -ex "bt full" -ex "call (void) gjs_dumpstack()" \
  -ex quit

Debugging GNOME Shell with valgrind

Note: These instructions work only in X11, valgrind requires running the inspected process from scratch. Also see the note about the --replace option at the end of this page.

$ env G_SLICE=always-malloc valgrind ./build/src/gnome-shell --replace

Note: You probably also want to specify --log-file=/tmp/gnome-shell.valgrind or the like since valgrind can produce a lot of output, so:

$ G_SLICE=always-malloc valgrind --log-file=/tmp/gnome-shell.valgrind ./build/src/gnome-shell --replace

Debugging GNOME Shell with heaptrack

Heaptrack is a memory analyzer with smaller runtime impact than Valgrind. It can be attached to running processes:

$ heaptrack -p `pidof gnome-shell`

Or run a process from scratch (see the note about the --replace option at the end of this page):

$ G_SLICE=always-malloc heaptrack ./build/src/gnome-shell --replace

This will produce a heaptrack.$process_name.$pid.gz file with the debug information. To see its reports either GUI or CLI may be used:

$ heaptrack_gui ./heaptrack.gnome-shell.`pidof gnome-shell`.gz
$ heaptrack_print ./heaptrack.gnome-shell.`pidof gnome-shell`.gz

Performance metrics

GNOME Shell integrates with sysprof. Its use is advocated to get timing and performance metrics.

Additionally, linux has a tool called "perf", a simple usage example:

$ perf record -p $(pidof gnome-shell)
  (time passes, press Ctrl-C)
$ perf report -i perf.data

A note about using "--replace" replace on systems with systemd user sessions

When using the --replace option on systems with systemd user sessions, this will result in systemd assuming the gnome-shell instance it had managed to have disappeared and its attempts of restarting it will fail, because there already is a new instance running that is not managed by systemd. This will trigger the "Oh no! Something has gone wrong." dialog. This dialog however can be ignored, because the session has not actually crashed. Instead the dialog can be moved out of the way by holding the Super/Win key and right clicking the window. Unselect "Always on Top" and "Always on Visible Workspace", then move it to another workspace. Closing the dialog will exit the session.