Crypto Flexs
  • DIRECTORY
  • CRYPTO
    • ETHEREUM
    • BITCOIN
    • ALTCOIN
  • BLOCKCHAIN
  • EXCHANGE
  • ADOPTION
  • TRADING
  • HACKING
  • SLOT
  • TRADE
Crypto Flexs
  • DIRECTORY
  • CRYPTO
    • ETHEREUM
    • BITCOIN
    • ALTCOIN
  • BLOCKCHAIN
  • EXCHANGE
  • ADOPTION
  • TRADING
  • HACKING
  • SLOT
  • TRADE
Crypto Flexs
Home»ADOPTION NEWS»Anyscale, exploring direct preference optimization using synthetic data
ADOPTION NEWS

Anyscale, exploring direct preference optimization using synthetic data

By Crypto FlexsAugust 22, 20243 Mins Read
Facebook Twitter Pinterest LinkedIn Tumblr Email
Anyscale, exploring direct preference optimization using synthetic data
Share
Facebook Twitter LinkedIn Pinterest Email

Felix Pinkston
22 Aug 2024 03:00

Anyscale’s latest blog post dives deeper into Direct Preference Optimization (DPO) with Synthetic Data, highlighting its methodology and applications in language model tuning.





According to Anyscale, Direct Preference Optimization (DPO) has emerged as a prominent methodology for tuning language models to align their output with human preferences. The company’s latest blog post provides an in-depth case study on applying DPO using synthetic data, specifically in the context of summary tasks.

Generate synthetic data

Synthetic data generation has become a powerful technique for creating high-quality data sets. Anyscale’s approach uses AI models as data augmentors and judgers to improve subsequent models. This blog describes a detailed pipeline for synthetic data generation, highlighting the utility of Ray Data and vLLM for scaling and rapid experimentation.

DPO Training and Insights

Direct Preference Optimization (DPO) is a widely adopted algorithm for preference adjustment as it provides a balanced balance between complexity and effectiveness. Anyscale has integrated DPO into its LLM product family, allowing users to build preference adjustment models through an intuitive API. This blog covers modeling insights and experiments conducted on DPO for the sake of summary.

evaluation

Anyscale uses Ray Data and vLLM for batch inference to evaluate the generated summaries at scale. Evaluation is essential to determine the quality of the model, and Anyscale emphasizes the importance of task-specific evaluations that align with training objectives. This blog provides key details on setting up affinity functions for effective evaluation.

Comparison with supervised fine-tuning

This blog contrasts DPO with traditional supervised fine-tuning (SFT). SFT relies on collecting high-quality data and accurately mimicking the desired behavior, while preference tuning focuses on which responses are preferred over others. This approach directly addresses model-specific issues by allowing for scalable data generation and in-policy data collection.

Case Study: Summary

The case study applies DPO to the Mistral-7B-instruct-v0.1 model to summarize CNN articles. Anyscale designed a synthetic summary preference dataset to reduce costs and ensure consistency between training and evaluation using synthetic judgers. The preference function evaluates summaries by combining word count minimization and Q&A accuracy.

Data generation

Anyscale used the Mistral-7B-Instruct-v0.1 model to generate policy data for summarization. This process involved generating multiple summaries for each article and using the Llama-3-70B-Instruct model to create and answer multiple-choice questions on the original text. This method ensured a variety of outputs and accurate assessments.

DPO Training

Anyscale implemented DPO in their LLM post-training offering, allowing users to configure hyperparameters and compute resources for their training runs. This blog provides a detailed example of a DPO training configuration, highlighting the importance of the β hyperparameter and efficient training using Ray.

evaluation

The evaluation included calculating the win rate of each model and comparing the DPO-trained model with the original and other baselines. The results showed that DPO was advantageous in balancing accuracy and compression, and outperformed SFT and GPT-4o baselines.

Insights and Challenges

Anyscale has uncovered key insights into DPO training, including the critical role of β and learning rate hyperparameters. The blog also discusses failure modes such as long off-topic endings and gibberish tokens, emphasizing the need for careful hyperparameter tuning and monitoring.

Repetitive policy training

The blog suggests iterative on-policy learning as a way to improve DPO performance. By regenerating training data with fine-tuned models and applying additional DPO rounds, Anyscale achieves significant performance gains, making DPO competitive with existing RLHF methods.

For a full detailed case study and methodology, you can refer to Anyscale’s original post.

Image source: Shutterstock


Share. Facebook Twitter Pinterest LinkedIn Tumblr Email

Related Posts

KAITO unveils Capital Launchpad, a Web3 crowdfunding platform that will be released later this week.

July 22, 2025

Algorand (Algo) Get momentum in the launch and technical growth.

July 14, 2025

It flashes again in July

July 6, 2025
Add A Comment

Comments are closed.

Recent Posts

Ethereum Based Meme Coin PEPETO Raises Above $5.5M In Presale

July 22, 2025

MultiBank Group’s $MBG Token TGE Is Live On MexC, Gate.io, Uniswap And Multibank.io.

July 22, 2025

Ark Invest sells coinbase stocks and invests in BitMine.

July 22, 2025

Altcoin benefits of capital rotation

July 22, 2025

KAITO unveils Capital Launchpad, a Web3 crowdfunding platform that will be released later this week.

July 22, 2025

CARV Advances AI Beings Roadmap With Hackathon And 12+ Ecosystem Partnerships

July 22, 2025

POLYMARKET will re -enter the United States after the acquisition of QCEX $ 112 million.

July 22, 2025

FTT increases by 7% as the backpack starts the platform to help victims clear liquidation.

July 21, 2025

Monarq Asset Management Appoints Sam Gaer As CIO To Lead Directional Strategy

July 21, 2025

Little PEPE surpasses $ 4 million in pre -sales, emerging as one of the main memes in 2025.

July 21, 2025

Bitcoin Price $ 123K Explosion -Trader Brace for Brake Out

July 20, 2025

Crypto Flexs is a Professional Cryptocurrency News Platform. Here we will provide you only interesting content, which you will like very much. We’re dedicated to providing you the best of Cryptocurrency. We hope you enjoy our Cryptocurrency News as much as we enjoy offering them to you.

Contact Us : Partner(@)Cryptoflexs.com

Top Insights

Ethereum Based Meme Coin PEPETO Raises Above $5.5M In Presale

July 22, 2025

MultiBank Group’s $MBG Token TGE Is Live On MexC, Gate.io, Uniswap And Multibank.io.

July 22, 2025

Ark Invest sells coinbase stocks and invests in BitMine.

July 22, 2025
Most Popular

Cryptocurrency Markets at Christmas: Expected Historical Trends

December 18, 2023

Bitcoin Technical Analysis: BTC Maintains Stability in Narrow Triangle Formation.

May 31, 2024

US cryptocurrency legislation has been delayed until 2024 due to a Republican leadership battle, key lawmakers say.

December 1, 2023
  • Home
  • About Us
  • Contact Us
  • Disclaimer
  • Privacy Policy
  • Terms and Conditions
© 2025 Crypto Flexs

Type above and press Enter to search. Press Esc to cancel.