Anyscale, exploring direct preference optimization using synthetic data

According to Anyscale, Direct Preference Optimization (DPO) has emerged as a prominent methodology for tuning language models to align their output with human preferences. The company’s latest blog post provides an in-depth case study on applying DPO using synthetic data, specifically in the context of summary tasks.

Generate synthetic data

Synthetic data generation has become a powerful technique for creating high-quality data sets. Anyscale’s approach uses AI models as data augmentors and judgers to improve subsequent models. This blog describes a detailed pipeline for synthetic data generation, highlighting the utility of Ray Data and vLLM for scaling and rapid experimentation.

DPO Training and Insights

Direct Preference Optimization (DPO) is a widely adopted algorithm for preference adjustment as it provides a balanced balance between complexity and effectiveness. Anyscale has integrated DPO into its LLM product family, allowing users to build preference adjustment models through an intuitive API. This blog covers modeling insights and experiments conducted on DPO for the sake of summary.

evaluation

Anyscale uses Ray Data and vLLM for batch inference to evaluate the generated summaries at scale. Evaluation is essential to determine the quality of the model, and Anyscale emphasizes the importance of task-specific evaluations that align with training objectives. This blog provides key details on setting up affinity functions for effective evaluation.

Comparison with supervised fine-tuning

This blog contrasts DPO with traditional supervised fine-tuning (SFT). SFT relies on collecting high-quality data and accurately mimicking the desired behavior, while preference tuning focuses on which responses are preferred over others. This approach directly addresses model-specific issues by allowing for scalable data generation and in-policy data collection.

Case Study: Summary

The case study applies DPO to the Mistral-7B-instruct-v0.1 model to summarize CNN articles. Anyscale designed a synthetic summary preference dataset to reduce costs and ensure consistency between training and evaluation using synthetic judgers. The preference function evaluates summaries by combining word count minimization and Q&A accuracy.

Data generation

Anyscale used the Mistral-7B-Instruct-v0.1 model to generate policy data for summarization. This process involved generating multiple summaries for each article and using the Llama-3-70B-Instruct model to create and answer multiple-choice questions on the original text. This method ensured a variety of outputs and accurate assessments.

DPO Training

Anyscale implemented DPO in their LLM post-training offering, allowing users to configure hyperparameters and compute resources for their training runs. This blog provides a detailed example of a DPO training configuration, highlighting the importance of the β hyperparameter and efficient training using Ray.

evaluation

The evaluation included calculating the win rate of each model and comparing the DPO-trained model with the original and other baselines. The results showed that DPO was advantageous in balancing accuracy and compression, and outperformed SFT and GPT-4o baselines.

Insights and Challenges

Anyscale has uncovered key insights into DPO training, including the critical role of β and learning rate hyperparameters. The blog also discusses failure modes such as long off-topic endings and gibberish tokens, emphasizing the need for careful hyperparameter tuning and monitoring.

Repetitive policy training

The blog suggests iterative on-policy learning as a way to improve DPO performance. By regenerating training data with fine-tuned models and applying additional DPO rounds, Anyscale achieves significant performance gains, making DPO competitive with existing RLHF methods.

For a full detailed case study and methodology, you can refer to Anyscale’s original post.

Image source: Shutterstock