fastchat.serve.inference

Inference for FastChat models.

Module Contents

Classes

ChatIO

Helper class that provides a standard way to create an ABC using

class fastchat.serve.inference.ChatIO

Helper class that provides a standard way to create an ABC using inheritance.

abstract prompt_for_input(role: str) → str

Prompt for input from a role.

abstract prompt_for_output(role: str)

Prompt for output from a role.

abstract stream_output(output_stream, skip_echo_len: int)

Stream output.