Fdpo

[xinzhuo20]

#fDPO Implementation

Fine-tuning examples using Google Tunix with fDPO - a novel fine-grained preference learning algorithm for segment-level preference optimization.

fDPO Overview

fDPO (Fine-grained Direct Preference Optimization) extends traditional DPO by introducing segment-level preference granularity. Instead of applying a single global trade-off parameter β uniformly across all reasoning steps, fDPO separates responses into distinct components (description and reasoning) and applies adaptive, segment-specific β values.

Key Features

• Segment-Level Optimization: Separates responses into description (R_desc) and reasoning (R_reason) components
• Adaptive β Values: Dynamically computes β_desc and β_reason based on preference differentials
• Balanced Learning: Prevents overfitting to simpler descriptive responses while properly optimizing complex reasoning paths

Reference
https://plan-lab.github.io/projects/spatialreasoner/

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[abheesht17]

@xinzhuo20

1. Are the results better than the baseline? Or do we still need to debug?
2. Can we move the trainer inside experimental?

[xinzhuo20]

@xinzhuo20

1. Are the results better than the baseline? Or do we still need to debug?
2. Can we move the trainer inside experimental?

1. Still need to debug @abheesht17, can you take a look? Thank you.
2. Do you mean moving the trainer file to tunix/rl/experimental?