Debugging C++ Coroutines

Introduction

For performance and other architectural reasons, the C++ Coroutines feature in the Clang compiler is implemented in two parts of the compiler. Semantic analysis is performed in Clang, and Coroutine construction and optimization takes place in the LLVM middle-end.

However, this design forces us to generate insufficient debugging information. Typically, the compiler generates debug information in the Clang frontend, as debug information is highly language specific. However, this is not possible for Coroutine frames because the frames are constructed in the LLVM middle-end.

To mitigate this problem, the LLVM middle end attempts to generate some debug information, which is unfortunately incomplete, since much of the language specific information is missing in the middle end.

This document describes how to use this debug information to better debug coroutines.

Terminology

Due to the recent nature of C++20 Coroutines, the terminology used to describe the concepts of Coroutines is not settled. This section defines a common, understandable terminology to be used consistently throughout this document.

coroutine type

A coroutine function is any function that contains any of the Coroutine Keywords co_await, co_yield, or co_return. A coroutine type is a possible return type of one of these coroutine functions. Task and Generator are commonly referred to coroutine types.

coroutine

By technical definition, a coroutine is a suspendable function. However, programmers typically use coroutine to refer to an individual instance. For example:

std::vector<Task> Coros; // Task is a coroutine type.
for (int i = 0; i < 3; i++)
  Coros.push_back(CoroTask()); // CoroTask is a coroutine function, which
                               // would return a coroutine type 'Task'.

In practice, we typically say “Coros contains 3 coroutines” in the above example, though this is not strictly correct. More technically, this should say “Coros contains 3 coroutine instances” or “Coros contains 3 coroutine objects.”

In this document, we follow the common practice of using coroutine to refer to an individual coroutine instance, since the terms coroutine instance and coroutine object aren’t sufficiently defined in this case.

coroutine frame

The C++ Standard uses coroutine state to describe the allocated storage. In the compiler, we use coroutine frame to describe the generated data structure that contains the necessary information.

The structure of coroutine frames

The structure of coroutine frames is defined as:

struct {
  void (*__r)(); // function pointer to the `resume` function
  void (*__d)(); // function pointer to the `destroy` function
  promise_type; // the corresponding `promise_type`
  ... // Any other needed information
}

In the debugger, the function’s name is obtainable from the address of the function. And the name of resume function is equal to the name of the coroutine function. So the name of the coroutine is obtainable once the address of the coroutine is known.

Get the suspended points

An important requirement for debugging coroutines is to understand suspended points, which are where the coroutine is currently suspended and awaiting.

For simple cases like the above, inspecting the value of the __coro_index variable in the coroutine frame works well.

However, it is not quite so simple in really complex situations. In these cases, it is necessary to use the coroutine libraries to insert the line-number.

For example:

// For all the promise_type we want:
class promise_type {
  ...
+  unsigned line_number = 0xffffffff;
};

#include <source_location>

// For all the awaiter types we need:
class awaiter {
  ...
  template <typename Promise>
  void await_suspend(std::coroutine_handle<Promise> handle,
                     std::source_location sl = std::source_location::current()) {
        ...
        handle.promise().line_number = sl.line();
  }
};

In this case, we use std::source_location to store the line number of the await inside the promise_type. Since we can locate the coroutine function from the address of the coroutine, we can identify suspended points this way as well.

The downside here is that this comes at the price of additional runtime cost. This is consistent with the C++ philosophy of “Pay for what you use”.

Get the asynchronous stack

Another important requirement to debug a coroutine is to print the asynchronous stack to identify the asynchronous caller of the coroutine. As many implementations of coroutine types store std::coroutine_handle<> continuation in the promise type, identifying the caller should be trivial. The continuation is typically the awaiting coroutine for the current coroutine. That is, the asynchronous parent.

Since the promise_type is obtainable from the address of a coroutine and contains the corresponding continuation (which itself is a coroutine with a promise_type), it should be trivial to print the entire asynchronous stack.

This logic should be quite easily captured in a debugger script.

Examples to print asynchronous stack

Here is an example to print the asynchronous stack for the normal task implementation.

// debugging-example.cpp
#include <coroutine>
#include <iostream>
#include <utility>

struct task {
  struct promise_type {
    task get_return_object();
    std::suspend_always initial_suspend() { return {}; }

    void unhandled_exception() noexcept {}

    struct FinalSuspend {
      std::coroutine_handle<> continuation;
      auto await_ready() noexcept { return false; }
      auto await_suspend(std::coroutine_handle<> handle) noexcept {
        return continuation;
      }
      void await_resume() noexcept {}
    };
    FinalSuspend final_suspend() noexcept { return {continuation}; }

    void return_value(int res) { result = res; }

    std::coroutine_handle<> continuation = std::noop_coroutine();
    int result = 0;
  };

  task(std::coroutine_handle<promise_type> handle) : handle(handle) {}
  ~task() {
    if (handle)
      handle.destroy();
  }

  auto operator co_await() {
    struct Awaiter {
      std::coroutine_handle<promise_type> handle;
      auto await_ready() { return false; }
      auto await_suspend(std::coroutine_handle<> continuation) {
        handle.promise().continuation = continuation;
        return handle;
      }
      int await_resume() {
        int ret = handle.promise().result;
        handle.destroy();
        return ret;
      }
    };
    return Awaiter{std::exchange(handle, nullptr)};
  }

  int syncStart() {
    handle.resume();
    return handle.promise().result;
  }

private:
  std::coroutine_handle<promise_type> handle;
};

task task::promise_type::get_return_object() {
  return std::coroutine_handle<promise_type>::from_promise(*this);
}

namespace detail {
template <int N>
task chain_fn() {
  co_return N + co_await chain_fn<N - 1>();
}

template <>
task chain_fn<0>() {
  // This is the default breakpoint.
  __builtin_debugtrap();
  co_return 0;
}
}  // namespace detail

task chain() {
  co_return co_await detail::chain_fn<30>();
}

int main() {
  std::cout << chain().syncStart() << "\n";
  return 0;
}

In the example, the task coroutine holds a continuation field, which would be resumed once the task completes. In another word, the continuation is the asynchronous caller for the task. Just like the normal function returns to its caller when the function completes.

So we can use the continuation field to construct the asynchronous stack:

# debugging-helper.py
import gdb
from gdb.FrameDecorator import FrameDecorator

class SymValueWrapper():
    def __init__(self, symbol, value):
        self.sym = symbol
        self.val = value

    def __str__(self):
        return str(self.sym) + " = " + str(self.val)

def get_long_pointer_size():
    return gdb.lookup_type('long').pointer().sizeof

def cast_addr2long_pointer(addr):
    return gdb.Value(addr).cast(gdb.lookup_type('long').pointer())

def dereference(addr):
    return long(cast_addr2long_pointer(addr).dereference())

class CoroutineFrame(object):
    def __init__(self, task_addr):
        self.frame_addr = task_addr
        self.resume_addr = task_addr
        self.destroy_addr = task_addr + get_long_pointer_size()
        self.promise_addr = task_addr + get_long_pointer_size() * 2
        # In the example, the continuation is the first field member of the promise_type.
        # So they have the same addresses.
        # If we want to generalize the scripts to other coroutine types, we need to be sure
        # the continuation field is the first member of promise_type.
        self.continuation_addr = self.promise_addr

    def next_task_addr(self):
        return dereference(self.continuation_addr)

class CoroutineFrameDecorator(FrameDecorator):
    def __init__(self, coro_frame):
        super(CoroutineFrameDecorator, self).__init__(None)
        self.coro_frame = coro_frame
        self.resume_func = dereference(self.coro_frame.resume_addr)
        self.resume_func_block = gdb.block_for_pc(self.resume_func)
        if self.resume_func_block == None:
            raise Exception('Not stackless coroutine.')
        self.line_info = gdb.find_pc_line(self.resume_func)

    def address(self):
        return self.resume_func

    def filename(self):
        return self.line_info.symtab.filename

    def frame_args(self):
        return [SymValueWrapper("frame_addr", cast_addr2long_pointer(self.coro_frame.frame_addr)),
                SymValueWrapper("promise_addr", cast_addr2long_pointer(self.coro_frame.promise_addr)),
                SymValueWrapper("continuation_addr", cast_addr2long_pointer(self.coro_frame.continuation_addr))
                ]

    def function(self):
        return self.resume_func_block.function.print_name

    def line(self):
        return self.line_info.line

class StripDecorator(FrameDecorator):
    def __init__(self, frame):
        super(StripDecorator, self).__init__(frame)
        self.frame = frame
        f = frame.function()
        self.function_name = f

    def __str__(self, shift = 2):
        addr = "" if self.address() == None else '%#x' % self.address() + " in "
        location = "" if self.filename() == None else " at " + self.filename() + ":" + str(self.line())
        return addr + self.function() + " " + str([str(args) for args in self.frame_args()]) + location

class CoroutineFilter:
    def create_coroutine_frames(self, task_addr):
        frames = []
        while task_addr != 0:
            coro_frame = CoroutineFrame(task_addr)
            frames.append(CoroutineFrameDecorator(coro_frame))
            task_addr = coro_frame.next_task_addr()
        return frames

class AsyncStack(gdb.Command):
    def __init__(self):
        super(AsyncStack, self).__init__("async-bt", gdb.COMMAND_USER)

    def invoke(self, arg, from_tty):
        coroutine_filter = CoroutineFilter()
        argv = gdb.string_to_argv(arg)
        if len(argv) == 0:
            try:
                task = gdb.parse_and_eval('__coro_frame')
                task = int(str(task.address), 16)
            except Exception:
                print ("Can't find __coro_frame in current context.\n" +
                      "Please use `async-bt` in stackless coroutine context.")
                return
        elif len(argv) != 1:
            print("usage: async-bt <pointer to task>")
            return
        else:
            task = int(argv[0], 16)

        frames = coroutine_filter.create_coroutine_frames(task)
        i = 0
        for f in frames:
            print '#'+ str(i), str(StripDecorator(f))
            i += 1
        return

AsyncStack()

class ShowCoroFrame(gdb.Command):
    def __init__(self):
        super(ShowCoroFrame, self).__init__("show-coro-frame", gdb.COMMAND_USER)

    def invoke(self, arg, from_tty):
        argv = gdb.string_to_argv(arg)
        if len(argv) != 1:
            print("usage: show-coro-frame <address of coroutine frame>")
            return

        addr = int(argv[0], 16)
        block = gdb.block_for_pc(long(cast_addr2long_pointer(addr).dereference()))
        if block == None:
            print "block " + str(addr) + "  is none."
            return

        # Disable demangling since gdb will treat names starting with `_Z`(The marker for Itanium ABI) specially.
        gdb.execute("set demangle-style none")

        coro_frame_type = gdb.lookup_type(block.function.linkage_name + ".coro_frame_ty")
        coro_frame_ptr_type = coro_frame_type.pointer()
        coro_frame = gdb.Value(addr).cast(coro_frame_ptr_type).dereference()

        gdb.execute("set demangle-style auto")
        gdb.write(coro_frame.format_string(pretty_structs = True))

ShowCoroFrame()

Then let’s run:

$ clang++ -std=c++20 -g debugging-example.cpp -o debugging-example
$ gdb ./debugging-example
(gdb) # We've already set the breakpoint.
(gdb) r
Program received signal SIGTRAP, Trace/breakpoint trap.
detail::chain_fn<0> () at debugging-example2.cpp:73
73      co_return 0;
(gdb) # Executes the debugging scripts
(gdb) source debugging-helper.py
(gdb) # Print the asynchronous stack
(gdb) async-bt
#0 0x401c40 in detail::chain_fn<0>() ['frame_addr = 0x441860', 'promise_addr = 0x441870', 'continuation_addr = 0x441870'] at debugging-example.cpp:71
#1 0x4022d0 in detail::chain_fn<1>() ['frame_addr = 0x441810', 'promise_addr = 0x441820', 'continuation_addr = 0x441820'] at debugging-example.cpp:66
#2 0x403060 in detail::chain_fn<2>() ['frame_addr = 0x4417c0', 'promise_addr = 0x4417d0', 'continuation_addr = 0x4417d0'] at debugging-example.cpp:66
#3 0x403df0 in detail::chain_fn<3>() ['frame_addr = 0x441770', 'promise_addr = 0x441780', 'continuation_addr = 0x441780'] at debugging-example.cpp:66
#4 0x404b80 in detail::chain_fn<4>() ['frame_addr = 0x441720', 'promise_addr = 0x441730', 'continuation_addr = 0x441730'] at debugging-example.cpp:66
#5 0x405910 in detail::chain_fn<5>() ['frame_addr = 0x4416d0', 'promise_addr = 0x4416e0', 'continuation_addr = 0x4416e0'] at debugging-example.cpp:66
#6 0x4066a0 in detail::chain_fn<6>() ['frame_addr = 0x441680', 'promise_addr = 0x441690', 'continuation_addr = 0x441690'] at debugging-example.cpp:66
#7 0x407430 in detail::chain_fn<7>() ['frame_addr = 0x441630', 'promise_addr = 0x441640', 'continuation_addr = 0x441640'] at debugging-example.cpp:66
#8 0x4081c0 in detail::chain_fn<8>() ['frame_addr = 0x4415e0', 'promise_addr = 0x4415f0', 'continuation_addr = 0x4415f0'] at debugging-example.cpp:66
#9 0x408f50 in detail::chain_fn<9>() ['frame_addr = 0x441590', 'promise_addr = 0x4415a0', 'continuation_addr = 0x4415a0'] at debugging-example.cpp:66
#10 0x409ce0 in detail::chain_fn<10>() ['frame_addr = 0x441540', 'promise_addr = 0x441550', 'continuation_addr = 0x441550'] at debugging-example.cpp:66
#11 0x40aa70 in detail::chain_fn<11>() ['frame_addr = 0x4414f0', 'promise_addr = 0x441500', 'continuation_addr = 0x441500'] at debugging-example.cpp:66
#12 0x40b800 in detail::chain_fn<12>() ['frame_addr = 0x4414a0', 'promise_addr = 0x4414b0', 'continuation_addr = 0x4414b0'] at debugging-example.cpp:66
#13 0x40c590 in detail::chain_fn<13>() ['frame_addr = 0x441450', 'promise_addr = 0x441460', 'continuation_addr = 0x441460'] at debugging-example.cpp:66
#14 0x40d320 in detail::chain_fn<14>() ['frame_addr = 0x441400', 'promise_addr = 0x441410', 'continuation_addr = 0x441410'] at debugging-example.cpp:66
#15 0x40e0b0 in detail::chain_fn<15>() ['frame_addr = 0x4413b0', 'promise_addr = 0x4413c0', 'continuation_addr = 0x4413c0'] at debugging-example.cpp:66
#16 0x40ee40 in detail::chain_fn<16>() ['frame_addr = 0x441360', 'promise_addr = 0x441370', 'continuation_addr = 0x441370'] at debugging-example.cpp:66
#17 0x40fbd0 in detail::chain_fn<17>() ['frame_addr = 0x441310', 'promise_addr = 0x441320', 'continuation_addr = 0x441320'] at debugging-example.cpp:66
#18 0x410960 in detail::chain_fn<18>() ['frame_addr = 0x4412c0', 'promise_addr = 0x4412d0', 'continuation_addr = 0x4412d0'] at debugging-example.cpp:66
#19 0x4116f0 in detail::chain_fn<19>() ['frame_addr = 0x441270', 'promise_addr = 0x441280', 'continuation_addr = 0x441280'] at debugging-example.cpp:66
#20 0x412480 in detail::chain_fn<20>() ['frame_addr = 0x441220', 'promise_addr = 0x441230', 'continuation_addr = 0x441230'] at debugging-example.cpp:66
#21 0x413210 in detail::chain_fn<21>() ['frame_addr = 0x4411d0', 'promise_addr = 0x4411e0', 'continuation_addr = 0x4411e0'] at debugging-example.cpp:66
#22 0x413fa0 in detail::chain_fn<22>() ['frame_addr = 0x441180', 'promise_addr = 0x441190', 'continuation_addr = 0x441190'] at debugging-example.cpp:66
#23 0x414d30 in detail::chain_fn<23>() ['frame_addr = 0x441130', 'promise_addr = 0x441140', 'continuation_addr = 0x441140'] at debugging-example.cpp:66
#24 0x415ac0 in detail::chain_fn<24>() ['frame_addr = 0x4410e0', 'promise_addr = 0x4410f0', 'continuation_addr = 0x4410f0'] at debugging-example.cpp:66
#25 0x416850 in detail::chain_fn<25>() ['frame_addr = 0x441090', 'promise_addr = 0x4410a0', 'continuation_addr = 0x4410a0'] at debugging-example.cpp:66
#26 0x4175e0 in detail::chain_fn<26>() ['frame_addr = 0x441040', 'promise_addr = 0x441050', 'continuation_addr = 0x441050'] at debugging-example.cpp:66
#27 0x418370 in detail::chain_fn<27>() ['frame_addr = 0x440ff0', 'promise_addr = 0x441000', 'continuation_addr = 0x441000'] at debugging-example.cpp:66
#28 0x419100 in detail::chain_fn<28>() ['frame_addr = 0x440fa0', 'promise_addr = 0x440fb0', 'continuation_addr = 0x440fb0'] at debugging-example.cpp:66
#29 0x419e90 in detail::chain_fn<29>() ['frame_addr = 0x440f50', 'promise_addr = 0x440f60', 'continuation_addr = 0x440f60'] at debugging-example.cpp:66
#30 0x41ac20 in detail::chain_fn<30>() ['frame_addr = 0x440f00', 'promise_addr = 0x440f10', 'continuation_addr = 0x440f10'] at debugging-example.cpp:66
#31 0x41b9b0 in chain() ['frame_addr = 0x440eb0', 'promise_addr = 0x440ec0', 'continuation_addr = 0x440ec0'] at debugging-example.cpp:77

Now we get the complete asynchronous stack! It is also possible to print other asynchronous stack which doesn’t live in the top of the stack. We can make it by passing the address of the corresponding coroutine frame to async-bt command.

By the debugging scripts, we can print any coroutine frame too as long as we know the address. For example, we can print the coroutine frame for detail::chain_fn<18>() in the above example. From the log record, we know the address of the coroutine frame is 0x4412c0 in the run. Then we can:

(gdb) show-coro-frame 0x4412c0
{
  __resume_fn = 0x410960 <detail::chain_fn<18>()>,
  __destroy_fn = 0x410d60 <detail::chain_fn<18>()>,
  __promise = {
    continuation = {
      _M_fr_ptr = 0x441270
    },
    result = 0
  },
  struct_Awaiter_0 = {
    struct_std____n4861__coroutine_handle_0 = {
      struct_std____n4861__coroutine_handle = {
        PointerType = 0x441310
      }
    }
  },
  struct_task_1 = {
    struct_std____n4861__coroutine_handle_0 = {
      struct_std____n4861__coroutine_handle = {
        PointerType = 0x0
      }
    }
  },
  struct_task__promise_type__FinalSuspend_2 = {
    struct_std____n4861__coroutine_handle = {
      PointerType = 0x0
    }
  },
  __coro_index = 1 '\001',
  struct_std____n4861__suspend_always_3 = {
    __int_8 = 0 '\000'
  }

Get the living coroutines

Another useful task when debugging coroutines is to enumerate the list of living coroutines, which is often done with threads. While technically possible, this task is not recommended in production code as it is costly at runtime. One such solution is to store the list of currently running coroutines in a collection:

inline std::unordered_set<void*> lived_coroutines;
// For all promise_type we want to record
class promise_type {
public:
    promise_type() {
        // Note to avoid data races
        lived_coroutines.insert(std::coroutine_handle<promise_type>::from_promise(*this).address());
    }
    ~promise_type() {
        // Note to avoid data races
        lived_coroutines.erase(std::coroutine_handle<promise_type>::from_promise(*this).address());
    }
};

In the above code snippet, we save the address of every lived coroutine in the lived_coroutines unordered_set. As before, once we know the address of the coroutine we can derive the function, promise_type, and other members of the frame. Thus, we could print the list of lived coroutines from that collection.

Please note that the above is expensive from a storage perspective, and requires some level of locking (not pictured) on the collection to prevent data races.