How to use strace to troubleshoot problems in Linux (and don't forget there's also ltrace)

Strace Cheat Sheet – Overview

Useful options and examples

• -c – See what time is spend and where (combine with -S for sorting)
• -f – Track process including forked child processes
• -o my-process-trace.txt – Log strace output to a file
• -p 1234 – Track a process by PID
• -P /tmp – Track a process when interacting with specific path(s)
• -T – Display syscall duration in the output
• -t -- display time stamps

Track by specific system call group

• -e trace=ipc – Track communication between processes (IPC)
• -e trace=memory – Track memory syscalls
• -e trace=network – Track memory syscalls
• -e trace=process – Track process calls (like fork, exec)
• -e trace=signal – Track process signal handling (like HUP, exit)
• -e trace=file – Track file related syscalls

Exclude specific calls:

• -e 'trace=!futex,clock_gettime,gettimeofday,epoll_wait'

Trace multiple syscalls

• strace -e open,close

Print the encoded unicode ouput of strace

• cat output.txt |xargs -i printf "%b" '{}\n'

---

Strace monitors the system calls and signals of a specific program. It is helpful when you do not have the source code and would like to debug the execution of a program. strace provides you the execution sequence of a binary from start to end.

This article explains 7 strace examples to get you started.

1. Trace the Execution of an Executable

You can use strace command to trace the execution of any executable. The following example shows the output of strace for the Linux ls command.

$  strace ls
...

2. Trace a Specific System Calls in an Executable Using Option -e

Be default, strace displays all system calls for the given executable. To display only a specific system call, use the strace -e option as shown below.

$ strace -e open ls
open("/etc/ld.so.cache", O_RDONLY)      = 3
open("/lib/libselinux.so.1", O_RDONLY)  = 3
open("/lib/librt.so.1", O_RDONLY)       = 3
...

$ strace -e trace=open,read ls /home open("/etc/ld.so.cache", O_RDONLY) = 3 open("/lib/libselinux.so.1", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\3\3\1\260G004"..., 512) = 512 open("/lib/librt.so.1", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\3\3\1\300\30004"..., 512) = 512 ...

The above output displays only the open system call of the ls command. If you want to trace multiple system calls use the “-e trace=” option. The following example displays both open and read system calls. You can find all the system calls a command uses with strace -c (see below)

3. Save the Trace Execution to a File Using Option -o

The following examples stores the strace output to output.txt file.

$ strace -o output.txt ls

3. Print strace's encoded unicode \316\223\316

$ cat output.txt |xargs -i printf "%b" '{}\n'

4. Execute Strace on a Running Linux Process Using Option -p

$ ps -C firefox-bin
  PID TTY          TIME CMD
 1725 ?        00:40:50 firefox-bin

$  sudo strace -p 1725 

5. Generate Statistics Report of System Calls Using Option -c

Using option -c, strace provides useful statistical report for the execution trace. The “calls” column in the following output indicated how many times that particular system call was executed.

$ strace -c ls /home
bala
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  -nan    0.000000           0         9           read
  -nan    0.000000           0         1           write
  -nan    0.000000           0        11           open
  -nan    0.000000           0        13           close
...
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000                   114        10 total

Monitoring file activity

Strace can monitor file related activity. There are two useful parts. The first is file, which shows file interactions. The other one allows tracing file descriptors. Both can be used to monitor for actions like opening files, reading/writing and closing. Usually using “trace=file” provides enough insights. If you really need more insights in the way a program deals with file descriptors, then use the second one.

• Monitor opening of files: strace -e open
• See all file activity: strace -e trace=file -p 1234 or strace -e trace=desc -p 1234

If you want to track specific paths, use 1 or more times the -P parameter, following by the path.

# sudo strace -P /etc/cups -p 2261
Process 2261 attached
— SIGHUP {si_signo=SIGHUP, si_code=SI_USER, si_pid=6149, si_uid=0} —
lstat(“/etc/cups”, {st_mode=S_IFDIR|0755, st_size=4096, …}) = 0
openat(AT_FDCWD, “/etc/cups”, O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 7
getdents(7, /* 11 entries /, 32768) = 336
getdents(7, / 0 entries /, 32768) = 0
close(7) = 0
openat(AT_FDCWD, “/etc/cups”, O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 7
getdents(7, / 11 entries /, 32768) = 336
getdents(7, / 0 entries */, 32768) = 0
close(7) = 0

Common calls:

• access
• close (close file handle)
• fchmod (change file permissions)
• fchown (change file ownership)
• fstat (retrieve details)
• lseek (move through file)
• open (open file for reading/writing)
• read (read a piece of data)
• statfs (retrieve file system related details)

Monitoring the network

Strace definitely can be useful for revealing more details about network traffic. Very useful to determine what network related connections are used, like when building your Docker image.

strace -e trace=network

Common syscalls:

• bind – link the process to a network port
• listen – allow to receive incoming connections
• socket – open a local or network socket
• setsockopt – define options for an active socket

Monitoring memory calls

To get better insights on the memory usage and system calls, strace can monitor for these as well. They are nicely grouped in the memory group.

strace -e trace=memory

Common syscalls:

• mmap
• munmap

---