Direct Preference Optimization
Creates a cold start by aligning the model with text-appropriate expressive speech, improving prosodic controllability before more complex optimization.
Zero-shot voice cloning typically emphasizes preserving the timbre and prosodic characteristics of a reference speaker. However, in real-world use, speech should adapt its delivery to the target text rather than remain tied to a fixed speaking style. In this work, we formulate this problem as text-adaptive prosody synthesis for zero-shot voice cloning and study how post-training can endow a cloning model with this capability.
Creates a cold start by aligning the model with text-appropriate expressive speech, improving prosodic controllability before more complex optimization.
Jointly optimizes expressiveness, acoustic quality, intelligibility, and speaker similarity, stabilizing the end-to-end training signal.
Transfers expressive behavior from strong-reference conditions to weak-reference conditions, enabling self-improvement without external labels.
| Model / Stage | Emotion (All) | Paraling. (All) | WER (EN) | SIM (EN) | UTMOS (EN) |
|---|---|---|---|---|---|
| Base Model | 18.5 | 38.4 | 1.23 | 0.72 | 3.35 |
| SFT (Stage 1 Exp 0) | 59.5 | 53.9 | 1.66 | 0.60 | 3.47 |
| DPO (Stage 1 Exp 1) | 30.6 | 43.3 | 2.17 | 0.66 | 3.27 |
| DPO (Stage 1 Exp 2) | 66.8 | 57.6 | 1.65 | 0.55 | 3.09 |
| GSPO (Stage 2) | 69.6 | 66.3 | 1.51 | 0.62 | 3.32 |
| OPSD (Stage 3) | 81.2 | 73.9 | 1.55 | 0.59 | 3.36 |
Please note that the reference audio samples on this page are sourced from the open-source dataset seed-tts-eval, which is licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0) License.
All other audio samples presented are AI-generated synthetic content produced by the post-trained models in this study. Any organization or individual using these AI-generated synthetic audio samples shall comply with applicable laws and regulations, fulfill their respective obligations when further generating or disseminating such content, and shall not use them for impersonation, fraud, or any other illegal or infringing purposes.