train

Module Contents

Classes

SupervisedDataset	Dataset for supervised fine-tuning.
LazySupervisedDataset	Dataset for supervised fine-tuning.

Functions

safe_save_model_for_hf_trainer(trainer, output_dir)	Collects the state dict and dump to disk.
make_supervised_data_module(→ Dict)	Make dataset and collator for supervised fine-tuning.

train.safe_save_model_for_hf_trainer(trainer: transformers.Trainer, output_dir: str)

Collects the state dict and dump to disk.

class train.SupervisedDataset(raw_data, tokenizer: transformers.PreTrainedTokenizer)

Dataset for supervised fine-tuning.

class train.LazySupervisedDataset(raw_data, tokenizer: transformers.PreTrainedTokenizer)

Dataset for supervised fine-tuning.

train.make_supervised_data_module(tokenizer: transformers.PreTrainedTokenizer, data_args) → Dict

Make dataset and collator for supervised fine-tuning.