Name: Xiao Yao (PhD 2Y)

Eduction: 2013-2017 Tianjin U

2019-2022 Shanghai JiaoTong U

2023-Now SUTD

Supervisor: Roy Lee, Li Xiaoli

Current interest: Reasoning、Alignment

Date: 0110

Scaling Samples of Preference Data Construction for DPO

Xiao Yao

Will be submit to ACL 2025

Background

Human annotation and LLM annotation are expensive
- Ultrafeeback, hh-rlhf
More and more reward models are available on HF
- Armo RM, Skywork RM

A conventional preference data construction

A Failure Case of Conventional Strategy

Model: Mistral-7B-v1/instruct,Llama3-8B/instruct

RM: Armo

Prompt: Ultrafeedback

Evaluation: Alpaca Eval2

Question

How should we construct

preference data

for

DPO

given sufficient sample budgets?

What Does The Reward Look Like

Implementation:

Model: Llama3-8B-instruct
Prompt: 1000 prompts from Ultrafeedback
Samples: 400 samples per prompt
RM: Armo/Skywork RM

What Does The Reward Look Like

The reward scores per prompt exhibits a Gaussian distribution
Response rewards of approximately 20% prompts can perfectly pass the Kolmogorov-Smirnov test

Reward Points

\left\{min, \mu \pm 2\sigma, \mu\pm\sigma, \mu, max\right\}

Position

C_7^2\quad(21)

Pairs

In practice, we select the sample point which has the closest reward score to the value in the set.
We construct 21 preference pairs per prompt following the principle that the reward of the chosen should be higher than that of rejected.

Alpaca Eval2 Results

Main Results

To achieve superior performance, the chosen response should be selected from , while the rejected response should be selected from .
Preference pairs of small margins usually perform poorly. If the reward of the chosen response is slightly higher than that of the rejected response, models trained with them cannot achieve satisfactory performance.
When the rejected responses are appropriately selected, the performance of trained models can improve as the reward of the chosen responses increases.

\left\{max, \mu+2\sigma\right\}

\left\{\mu-2\sigma\right\}

None of preference pairs will degrade the performance of SFT checkpoint, which indicates the robustness of the DPO training.

Training Dynamics for Interpretation

It can be seen that increasing the reward margin between the chosen and rejected responses may facilitate model training. Training loss can reach a lower bound when the reward margin increases.
There is a strong correlation between the converged state of the loss and the performance of models. Specifically, models achieving lower loss values tend to demonstrate superior performance, indicating that minimizing the loss effectively enhances model capabilities.

Proposed Preference Dataset Construction Strategy

Results

Reward Model: Armo

The performance of trained models is steadily improving if we increase the number of samples from 5 to 200, although with diminishing returns in some cases.

Results on Skywork

Model: Llama3-8B-instruct