Analyzing processes

The remote command is used to analyze the status of a running (remote) process. The analysis is always done in a safe and non-intrusive way, as no code is loaded in the memory space of the process under analysis and no memory is modified in the remote process. This makes analysis using PyStack a great option even for those services and applications that are running in environments where the running process must not be impacted in any way (other than being temporarily paused, though --no-block can avoid even that). There are several options available:

usage: pystack remote [-h] [-v] [--no-color] [--no-block] [--native]
                      [--native-all] [--locals] [--exhaustive]
                      pid

Positional Arguments

pid

The PID of the remote process

Named Arguments

-v, --verbose

Default: 0

--no-color

Deactivate colored output

Default: False

--no-block

do not block the process when inspecting its memory

Default: True

--native

Include the native (C) frames in the resulting stack trace

Default: NativeReportingMode.OFF

--native-all

Include native (C) frames from threads not registered with the interpreter (implies --native)

Default: NativeReportingMode.OFF

--locals

Show local variables for each frame in the stack trace

Default: False

--exhaustive

Use all possible methods to obtain the Python stack info (may be slow)

Default: False

To use PyStack, you just need to provide the PID of the process:

$ pystack remote 112
Traceback for thread 112 [] (most recent call last):
    (Python) File "/test.py", line 17, in <module>
        first_func()
    (Python) File "/test.py", line 6, in first_func
        second_func()
    (Python) File "/test.py", line 10, in second_func
        third_func()
    (Python) File "/test.py", line 14, in third_func
        time.sleep(1000)

To learn more about the different options you can use to customize what is reported, check the Customizing the reports section.

Operation modes

There are two main operation modes when analyzing the status of a remote process:

• The blocking mode (default) will stop the process, reads its memory to produce the report and then continue its execution. The overhead of this operations is very small (normally is under 10ms) but it can be undesirable in some scenarios such as services and applications running in production environments. The advantage of this mode is that the memory retrieved is fully coherent which guarantees that the resulting report will always be correct.

• The non blocking mode (activated using --no-block) will analyze the process memory to produce a report as the program is running. Although the memory retrieval normally is orders of magnitude faster than the rate at which the interpreter switches state, is still possible that the resulting report is not correct or that it cannot be produced due to incoherent results caused by the intepreter state changing before we’ve finished reading it.

Note

In general, users should prefer blocking mode (the default) because of its correctness unless stopping the process even momentarily is not acceptable, in which case non blocking mode can be used.

Interpreter finalization

By default, pystack will try to only fetch the Python stack trace but there are some scenarios where there will be no active Python stack trace even if the process is running. The most typical scenario is that the process is shutting down and the Python interpreter is in the process of being destroyed. In this case you will see the following error message with a warning explaining the situation:

$ pystack remote 140
WARNING(process_remote): The interpreter is shutting itself down so it is possible that no Python stack trace is
available for inspection. You can still use --native-all  to force displaying all the threads.

💀 Could not gather enough information to extract the Python frame information 💀

...

As the warning indicates, you can use --native-all to force displaying all the threads, even if there is no active Python stack trace:

$ pystack remote 140 --native-all
Traceback for thread 140 [Garbage collecting] (most recent call last):
    (C) File "???", line 0, in _start ()
    (C) File "???", line 0, in __libc_start_main ()
    (C) File "Modules/main.c", line 743, in Py_BytesMain (/usr/lib/libpython3.8.so.1.0)
    (C) File "Modules/main.c", line 691, in Py_RunMain (/usr/lib/libpython3.8.so.1.0)
    (C) File "Python/pylifecycle.c", line 1219, in Py_FinalizeEx (/usr/lib/libpython3.8.so.1.0)
    (C) File "Modules/gcmodule.c", line 1844, in PyGC_Collect (/usr/lib/libpython3.8.so.1.0)
    (C) File "Modules/gcmodule.c", line 1240, in collect_with_callback.constprop.33 (/usr/lib/libpython3.8.so.1.0)
    (C) File "Modules/gcmodule.c", line 1063, in collect.constprop.34 (/usr/lib/libpython3.8.so.1.0)
    (C) File "Modules/gcmodule.c", line 495, in move_unreachable (inlined) (/usr/lib/libpython3.8.so.1.0)

Here you can see that the process seems to be stuck while garbage collecting some object at interpreter finalization.

Tip

In some cases, is still possible to retrieve the Python stack trace by using the --exhaustive command line option.