1. Bibliographic Information

1.1. Title

The central topic of the paper is a unified approach for self-optimized alignment in language models, titled "Learning to Align, Aligning to Learn: A Unified Approach for Self-Optimized Alignment."

1.2. Authors

The authors are listed as "Anonymous authors," indicating that the paper is under double-blind review. This means their research backgrounds and affiliations are not disclosed in the provided text to ensure an unbiased review process.

1.3. Journal/Conference

The paper is published at OpenReview.net, which is a platform commonly used for conference submissions, especially for machine learning and AI conferences like ICLR, NeurIPS, and ICML, where submissions undergo double-blind peer review. Given the publication date and context, it is likely submitted to a prominent AI/ML conference.

1.4. Publication Year

2025. The specific publication date is October 8, 2025.

1.5. Abstract

The paper addresses the challenges in language model (LLM) alignment methodologies, specifically the limitations of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). SFT offers fast convergence but is constrained by offline policy trajectories, while RL allows exploration but suffers from low sample efficiency and high dependency on high-quality base models. To overcome these, the authors propose GRAO (Group Relative Alignment Optimization), a unified framework that combines the strengths of SFT and RL. GRAO introduces three key innovations: 1) a multi-sample generation strategy for comparative quality assessment via reward feedback; 2) a novel Group Direct Alignment Loss formulation utilizing intra-group relative advantage weighting; and 3) reference-aware parameter updates guided by pairwise preference dynamics. The theoretical analysis demonstrates GRAO's convergence guarantees and sample efficiency advantages. Comprehensive evaluations on complex human alignment tasks show GRAO's superior performance, achieving relative improvements of 57.70% over SFT, 17.65% over DPO, 7.95% over PPO, and 5.18% over GRPO baselines. The work offers both a theoretically grounded framework and empirical evidence for efficient capability evolution in LLMs.

1.6. Original Source Link

The official source link for the paper is https://openreview.net/forum?id=zFLCNnKY8U. The PDF link is https://openreview.net/pdf?id=zFLCNnKY8U. The paper is listed as "Paper under double-blind review," indicating it is a preprint undergoing peer review, rather than officially published in a journal or conference proceedings yet.

2. Executive Summary

2.1. Background & Motivation

The core problem this paper aims to solve is the inherent limitations of existing methodologies for aligning large language models (LLMs) with human preferences and instructions. This alignment is crucial for enhancing the model's helpfulness, harmlessness, and overall utility.

In the current field, two prominent approaches dominate:

• Supervised Fine-Tuning (SFT): This method efficiently injects knowledge into the model by training it on human-curated high-quality examples. However, its efficacy is constrained by the "offline policy trajectory," meaning it can only learn from the provided examples and struggles to generalize or explore beyond them. It can also lead to "knowledge forgetting" and a decline in out-of-distribution (OOD) generalization.

• Reinforcement Learning (RL) / Reinforcement Learning from Human Feedback (RLHF): This approach allows for exploratory policy optimization, enabling the model to learn through trial and error guided by reward signals. RL has shown strong potential in complex reasoning tasks and for strengthening the model's exploration ability. However, RL methods suffer from low sample efficiency (requiring many interactions to learn) and a stringent dependency on high-quality base models, as they often discard samples where the model fails to produce a correct answer, limiting their ability to solve problems beyond their initial capabilities.

  The paper highlights that while alternating between SFT and RL has shown promise, the optimization of this alternation is largely empirical, lacking a unified, theoretically grounded framework for dynamic adjustment. The challenge lies in synergizing the knowledge injection of SFT with the exploratory power of RL, without inheriting their respective weaknesses.

The paper's entry point and innovative idea is to propose a unified framework, GRAO (Group Relative Alignment Optimization), that dynamically blends the strengths of SFT and RL. It aims to overcome the "offline policy trajectory" limitation of SFT and the "low sample efficiency" and "high base model dependency" issues of RL by enabling dynamic adaptive adjustment imitation learning and self-driven exploratory learning. This approach seeks to allow the model to learn alignment capabilities beyond its initial scope, fostering a process of "imitate-explore-transcend."

2.2. Main Contributions / Findings

The paper makes several significant contributions to the field of large language model alignment:

• Novel Unified Alignment Framework (GRAO): The paper introduces GRAO, a new framework that synergizes the strengths of SFT and RL. This framework employs a "group direct alignment loss" that maintains exploration within the model's own sampling space while learning alignment capabilities beyond its current abilities.

• Three Key Innovations:

  1. Multi-sample Generation Strategy: GRAO generates multiple samples for each problem instance, enabling comparative quality assessment through reward feedback. This helps in understanding the relative quality of different generated responses.
  2. Group Direct Alignment Loss Formulation: A novel loss function is proposed that leverages intra-group relative advantage weighting. This allows the model to learn not just from single good examples, but from the relative superiority of certain generated samples within a group.
  3. Reference-aware Parameter Updates: The model's parameters are updated based on pairwise preference dynamics, guided by a reference trajectory. This combines the benefits of imitating high-quality references with self-correction based on generated samples.

• Theoretical Grounding and Convergence Guarantees: The paper provides a theoretical analysis establishing GRAO's convergence guarantees and demonstrating its sample efficiency advantages over conventional approaches, ensuring robust and stable policy optimization.

• Empirical Superiority Across Tasks and Models: Comprehensive evaluations on complex human alignment tasks (helpfulness and harmlessness) show GRAO's state-of-the-art performance. It achieves substantial relative improvements:

  • 57.70% over SFT
  • 17.65% over DPO (Direct Preference Optimization)
  • 7.95% over PPO (Proximal Policy Optimization)
  • 5.18% over GRPO (Group Relative Policy Optimization) Furthermore, GRAO demonstrates up to 22.74% NAG improvement over GRPO in Mixture-of-Experts (MoE) architectures, highlighting its adaptability and enhanced efficacy, especially for sparse MoE models.

• "Imitate-Explore-Transcend" Paradigm Validation: The paper empirically validates GRAO's dynamic learning process, showing how it transitions from initial imitation of reference outputs to active exploration and ultimately transcends the original policy output through self-driven improvement. This leads to more universal alignment behavior.

• Efficiency and Robustness: GRAO achieves optimal policy performance in half the training steps compared to baselines, demonstrating significantly greater alignment efficiency and maintaining stable optimization throughout training.

  These findings collectively solve the problem of efficiently and robustly aligning LLMs by offering a unified, theoretically sound, and empirically superior approach that dynamically balances learning from high-quality demonstrations with active self-driven exploration.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand GRAO, a reader should be familiar with the following core concepts in language model training and alignment:

• Large Language Models (LLMs): These are neural network models, often based on the transformer architecture, trained on vast amounts of text data to generate human-like text, understand natural language, and perform various language tasks.
• Alignment: In the context of LLMs, alignment refers to the process of making the model's behavior, outputs, and values consistent with human preferences, instructions, and ethical guidelines. This ensures the model is helpful, harmless, and honest.
• Supervised Fine-Tuning (SFT): A training technique where a pre-trained LLM is further trained on a smaller, task-specific dataset of input-output pairs. For alignment, this involves fine-tuning the model on high-quality, human-written responses to prompts, effectively teaching it to imitate desired behaviors.
  • Maximum Likelihood Estimation (MLE): SFT typically uses MLE, which aims to maximize the probability of the observed correct output sequence given the input. This is done by minimizing a cross-entropy loss between the model's predicted token probabilities and the actual tokens in the human-provided response.
• Reinforcement Learning (RL): A paradigm where an agent (the LLM) learns to make decisions by interacting with an environment (generating text) to maximize a cumulative reward signal.
  • Policy: In RL, the policy $\pi$ is the strategy that the agent uses to determine its next action (i.e., generate the next token) given the current state (the prompt and previously generated tokens).
  • Reward Model (RM): In RLHF, a separate model, often another neural network, is trained to predict a scalar reward score for a given model output, reflecting how well it aligns with human preferences. This RM acts as the "environment" providing feedback to the LLM.
• Reinforcement Learning from Human Feedback (RLHF): A powerful technique for aligning LLMs. It involves three main steps:
  1. SFT: Initial fine-tuning on a curated dataset.
  2. Reward Model Training: Humans rank or compare model outputs, and a reward model is trained to predict scores based on these preferences (often using the Bradley-Terry model).
  3. RL Fine-tuning: The SFT model is further fine-tuned using RL (e.g., PPO) to maximize the reward predicted by the reward model.
• Proximal Policy Optimization (PPO): A popular on-policy RL algorithm that optimizes a "clipped" objective function to ensure that policy updates are not too large, which helps maintain training stability. It works by preventing the new policy from straying too far from the old policy during an update step.
• Direct Preference Optimization (DPO): A method that bypasses the explicit training of a separate reward model. Instead, it directly optimizes the LLM's policy using a loss function derived from human preference data (pairs of preferred and rejected responses). It implicitly learns a reward function that aligns with preferences.
• Mixture-of-Experts (MoE) Architectures: A type of neural network architecture where different "expert" sub-networks specialize in different parts of the input space. A "gating network" learns to route each input token to a subset of these experts. This allows for models with a very large total number of parameters (sparse MoE) but only a small number of activated parameters per inference, making them computationally efficient at inference time.
  • Sparse Gradient Patterns: A characteristic of MoE models where gradients are only computed for the activated experts, leading to sparse updates during training.

3.2. Previous Works

The paper builds upon a rich history of alignment research, categorizing previous works into three main areas:

• Alignment With Supervised Fine-Tuning (SFT):

  • Ouyang et al. (2022): Established SFT as a critical baseline, showing its role in training instruction-tuned models.
  • Tunstall et al. (2023) and Rafailov et al. (2024): Empirically demonstrated SFT's necessity for stable convergence in preference-aligned outcomes.
  • SFT Mechanism: SFT optimizes sequence likelihood via Maximum Likelihood Estimation (MLE). The objective function, $\mathcal{J}_{\mathrm{SFT}}(\theta)$, is given by: $$\mathcal{J}_{\mathrm{SFT}}(\theta)=\mathbb{E}_{(q, y) \sim P(Q, Y)}\left[\frac{1}{|y|} \sum_{t=1}^{|y|} \log \pi_{\theta}\left(y_{, t} \mid q, y_{,<t}\right)\right]$$ Where:
    • $q$: input query.
    • $y$: ground-truth (human-provided) response.
    • P(Q, Y): distribution of query-response pairs.
    • $|y|$: length of the response $y$.
    • $\pi_{\theta}(y_{, t} \mid q, y_{,<t})$: probability of generating the $t$-th token of $y$ given the query $q$ and previous tokens $y_{,<t}$, according to the model with parameters $\theta$.
    • The term $\frac{1}{|y|}$ normalizes the loss across responses of different lengths.
  • Chu et al. (2025): Posited that SFT memorizes alignment patterns, stabilizing outputs and enabling rapid convergence.

• Reinforcement Learning With Human Feedback (RLHF):

  • Bradley-Terry Model (Bradley & Terry, 1952): A foundational preference modeling framework used to estimate probabilities of one item being preferred over another. In RLHF, it's used to train reward models on human pairwise comparisons of LLM outputs. For two responses, $x_1$ and $x_2$, the probability that $x_1$ is preferred over $x_2$ is often modeled as: $ P(x_1 \succ x_2) = \frac{e^{R(x_1)}}{e^{R(x_1)} + e^{R(x_2)}} $ Where R(x) is the scalar reward assigned to response $x$ by the reward model.
  • Proximal Policy Optimization (PPO) (Schulman et al., 2017): A key RL algorithm for optimizing the LLM's policy to maximize the reward from the reward model. PPO uses a clipped surrogate objective to prevent large policy updates, ensuring stability. The core idea is to maximize: $ \mathcal{L}^{\text{CLIP}}(\theta) = \mathbb{E}_t \left[ \min(r_t(\theta) \hat{A}_t, \mathrm{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t) \right] $ Where:
    • $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$ is the ratio of new to old policy probabilities for action $a_t$ in state $s_t$.
    • $\hat{A}_t$: estimated advantage function for action $a_t$ in state $s_t$.
    • $\epsilon$: a small hyperparameter for clipping.
  • Group Relative Policy Optimization (GRPO) (Shao et al., 2024): An RLHF variant that involves sampling multiple responses (a "group") and using their relative rewards for optimization. This aims to provide more robust reward signals than single-sample PPO. The paper mentions GRPO's challenge: if no sample in a group is "correct," the samples might be discarded.
  • Stiennon et al. (2022); Ziegler et al. (2020): Pioneering works in using RLHF to align models with human values.
  • RLAIF (Reinforcement Learning with AI Feedback) (Lee et al., 2024): Explores using auxiliary LLMs to generate automated feedback, potentially rivaling human evaluators.

• Alignment without Reward Modeling (Direct Preference Optimization):

  • Direct Preference Optimization (DPO) (Rafailov et al., 2024): A method that directly optimizes the policy using preference data (chosen and rejected responses) without explicitly training a reward model. It re-parametrizes the RLHF objective to directly optimize a policy that satisfies preferences. The DPO loss is typically: $ \mathcal{L}{\mathrm{DPO}}(\pi) = -\mathbb{E}{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \right) \right] $ Where:
    • $x$: prompt.
    • $y_w$: preferred response (winner).
    • $y_l$: rejected response (loser).
    • $\pi$: current policy.
    • $\pi_{\text{ref}}$: reference policy (e.g., the SFT model).
    • $\beta$: a hyperparameter.
    • $\sigma$: sigmoid function.
  • Identity Preference Optimization (IPO) (Ethayarajh et al., 2024): An extension of DPO that adds a regularization mechanism to reduce overfitting and preserve relative preferences.
  • Kahneman-Tversky Optimization (KTO) (Azar et al., 2023): A method that uses pointwise human judgments (e.g., whether a single response is good or bad) instead of pairwise preferences, aligning optimization with human cognitive biases based on prospect theory.

3.3. Technological Evolution

The field of LLM alignment has evolved significantly:

1. Pre-training: Initial large-scale training on diverse text data (e.g., GPT-3, LLaMA).
2. Supervised Fine-Tuning (SFT): Adapting pre-trained models to follow instructions using curated datasets (e.g., InstructGPT). This was the first step towards making models more usable.
3. Reinforcement Learning from Human Feedback (RLHF): Addressing the limitations of SFT by incorporating human preferences through reward models and RL algorithms like PPO. This allowed for more nuanced alignment beyond direct imitation.
4. Direct Preference Optimization (DPO and variants): Streamlining RLHF by eliminating the need for a separate reward model, directly optimizing the policy from preference data. This improved efficiency and stability.
5. Group-based RLHF (GRPO): Further refining RLHF by considering multiple generated samples to derive more stable reward signals.
6. Unified Approaches (GRAO): This paper's work represents a step towards unifying the strengths of SFT (knowledge injection, rapid initial learning) and RL (exploration, self-improvement) into a single, cohesive framework, aiming for the "best of both worlds" in alignment. It fits into the timeline as a next-generation approach that learns from the successes and failures of previous methods.

3.4. Differentiation Analysis

GRAO differentiates itself from existing methods by offering a unified approach that dynamically integrates imitation and exploration, overcoming the specific shortcomings of each:

• Compared to SFT:

  • Core Difference: SFT is limited by its "offline policy trajectory," meaning it can only learn what's explicitly provided in its training data. GRAO transcends this by allowing for exploration and transcendence beyond the given reference trajectories.
  • Innovation: GRAO incorporates an exploration term and an alignment regularizer that enable the model to discover novel, superior trajectories, not just imitate. SFT lacks this exploratory capability.

• Compared to RL (PPO, GRPO):

  • Core Difference: Traditional RL methods (PPO) and even group-based ones (GRPO) suffer from low sample efficiency and a strong dependency on the initial capabilities of the base model. If the model can't produce a good enough sample, RL struggles to learn.
  • Innovation: GRAO addresses this by integrating a strong imitation component, which quickly guides the model towards high-reward regions using reference answers, providing a "warm start" and reducing reward sparsity issues. Its multi-sample generation and Group Direct Alignment Loss with advantage normalization provide more stable and efficient gradient signals, improving sample efficiency and robustness, especially when initial policy is weak. GRPO also uses group sampling but GRAO's specific loss formulation and explicit imitation component provide more robust learning.

• Compared to DPO:

  • Core Difference: DPO directly optimizes policy based on pairwise preferences, implicitly learning a reward model. While efficient, it still fundamentally learns from pre-defined preferences between chosen and rejected examples.

  • Innovation: GRAO introduces a more dynamic and "self-optimized" learning process. While it uses reference answers (akin to chosen responses), its exploration and alignment regularizer components explicitly encourage discovering new optimal behaviors and pushing beyond the provided preferences. It's not just aligning to existing preferences but actively learning to generate better-aligned outputs through self-correction and exploration within a group context.

    In essence, GRAO's core innovation lies in its three-component loss function that simultaneously provides:

1. Imitation pressure (from SFT's strength) for rapid initial learning and stability.

2. Exploration capabilities (from RL's strength) for discovering new, superior solutions.

3. Regularization and relative advantage weighting (group-based learning) for stable and efficient policy refinement that transcends basic imitation.

   This "imitate-explore-transcend" paradigm is a key differentiator, allowing GRAO to achieve a more universal and dynamic alignment behavior.

4. Methodology

4.1. Principles

The core idea behind GRAO (Group Relative Alignment Optimization) is to create a unified alignment framework that synergistically combines the rapid knowledge injection capabilities of Supervised Fine-Tuning (SFT) with the active exploratory power of Reinforcement Learning (RL). The theoretical basis is rooted in an adaptive learning process that the authors term "imitate-explore-transcend." This principle guides the model to:

1. Imitate: Learn efficiently from high-quality human-provided "reference trajectories" (similar to SFT), quickly steering the policy towards desirable behaviors.

2. Explore: Actively generate and evaluate its own diverse outputs, learning from the relative quality of these generated samples, similar to RL.

3. Transcend: Through a combination of imitation, exploration, and regularization, push beyond the boundaries of the reference trajectories to discover new, even more optimal, and generalized alignment behaviors.

   This dynamic adjustment leverages off-policy trajectories (generated by the model itself) and reference answers to continuously refine the model's behavior, enhancing its reasoning, problem-solving skills, and overall alignment performance.

4.2. Core Methodology In-depth (Layer by Layer)

The GRAO framework's optimization process can be visualized as a cycle where the policy model generates multiple samples, a reward model evaluates them, relative advantages are computed, and the policy is updated.

The following figure (Figure 1 from the original paper) shows the overview of the optimization process in GRAO:

该图像是论文中关键方法的示意图，展示了基于Policy Model生成多样样本，经Reward Model评分，再通过Group Computation计算相对优势，最终利用Group Direct Alignment Loss进行模型更新的流程。

4.2.1. Optimization Objective Formulation

The optimization objective of GRAO, denoted as $\mathcal{J}_{\mathrm{GRAO}}(\theta)$, is designed to guide the model in enhancing its alignment capabilities through an adaptive learning process. It combines three key components: exploration, imitation, and an alignment regularizer.

The overall objective function is formulated as: $$\mathcal{J}_{\mathrm{GRAO}}(\theta)=\mathbb{E}\left[q, y \sim P(Q, Y),\left\{o_{i}\right\}_{i=1}^{G} \sim \pi_{\theta_{\text{old}}}(O \mid q)\right]$$ This denotes the expectation over input queries $q$ and reference trajectories $y$ sampled from the data distribution P(Q,Y), and over a group of $G$ output trajectories $\left\{o_{i}\right\}_{i=1}^{G}$ sampled from the old policy $\pi_{\theta_{\text{old}}}$ given query $q$. The use of old policy samples is common in off-policy RL to stabilize training.

The core loss combines the three components: \begin{aligned} & \mathcal{J}_{\mathrm{GRAO}}=\frac{1}{G} \sum_{i=1}^{G}\left[\hat{A}_{o_{i}} \underbrace{\left(\frac{1}{\left|o_{i}\right|} \sum_{t=1}^{\left|o_{i}\right|} \log \pi_{\theta}\left(o_{i, t} \mid q, o_{i,<t}\right)\right)}_{\mathcal{J}_{\text {exploration }}\left(o_{i}\right)}+\beta \hat{A}_{y} \underbrace{\left(\frac{1}{|y|} \sum_{t=1}^{|y|} \log \pi_{\theta}\left(y_{t} \mid q, y_{<t}\right)\right)}_{\mathcal{J}_{\text {imitation }}(y)} \\ & +\lambda \hat{A}_{o_{i}}\left(\mathcal{J}_{\text {exploration }}\left(o_{i}\right)-\mathcal{J}_{\text {imitation }}(y)\right)\right] \end{aligned} Let's break down each part of this crucial objective function:

• $q$: This represents the input query or prompt given to the language model.

• $y$: This is the reference trajectory, typically a high-quality, human-preferred answer associated with the query $q$. It serves as a gold standard for imitation.

• $\left\{o_{i}\right\}_{i=1}^{G}$: This is a set of $G$ trajectories (output sequences) generated by the model's policy $\pi_{\theta_{\text{old}}}$ for the given query $q$. The model generates multiple candidate responses, forming a "group" for comparison.

• $\hat{A}_{o_{i}}$: This is the normalized advantage for the $i$-th generated output trajectory $o_i$. It quantifies how much better or worse $o_i$ is compared to the average of other trajectories in its group, including the reference.

• $\hat{A}_{y}$: This is the normalized advantage for the reference trajectory $y$.

• $\pi_{\theta}(\cdot)$: This denotes the current policy of the language model with parameters $\theta$. The objective aims to update $\theta$.

• $\frac{1}{|\text{sequence}|} \sum_{t=1}^{|\text{sequence}|} \log \pi_{\theta}(\text{token}_t \mid q, \text{tokens}_{<t})$: This term calculates the normalized log-likelihood of a sequence (either $o_i$ or $y$) under the current policy $\pi_{\theta}$. It's a measure of how probable the model finds that sequence. Maximizing this term increases the probability of generating that sequence.

  Now, let's look at the three main components of the loss:

1. Exploration Term ($\mathcal{J}_{\text {exploration }}\left(o_{i}\right)$): $$\mathcal{J}_{\text {exploration }}\left(o_{i}\right) = \frac{1}{\left|o_{i}\right|} \sum_{t=1}^{\left|o_{i}\right|} \log \pi_{\theta}\left(o_{i, t} \mid q, o_{i,<t}\right)$$ This term represents the normalized log-likelihood of the $i$-th generated trajectory $o_i$ under the current policy. When weighted by $\hat{A}_{o_{i}}$, this component encourages the policy to increase the likelihood of its own generated trajectories that have a positive advantage (i.e., are good) and decrease the likelihood of those with a negative advantage (i.e., are bad). This drives the model to explore and refine its generative capabilities based on self-generated feedback.

2. Imitation Term ($\mathcal{J}_{\text {imitation }}(y)$): $$\mathcal{J}_{\text {imitation }}(y) = \frac{1}{|y|} \sum_{t=1}^{|y|} \log \pi_{\theta}\left(y_{t} \mid q, y_{<t}\right)$$ This term represents the normalized log-likelihood of the reference trajectory $y$ under the current policy. This component imposes imitation pressure towards the high-quality reference answers. The hyperparameter $\beta$ controls the strength of this imitation. A higher $\beta$ means stronger imitation of the provided reference. This balances learning from high-quality demonstrations against pure exploration.

3. Alignment Regularizer Term ($\lambda \hat{A}_{o_{i}}\left(\mathcal{J}_{\text {exploration }}\left(o_{i}\right)-\mathcal{J}_{\text {imitation }}(y)\right)$): This component acts as a regularizer, aiming to align the likelihoods of the generated trajectories with the reference trajectory, especially based on their relative advantages.

   • If a generated trajectory $o_i$ is much better than the reference (high $\hat{A}_{o_{i}}$), and $\mathcal{J}_{\text {exploration }}\left(o_{i}\right) > \mathcal{J}_{\text {imitation }}(y)$, this term would encourage the model to further increase the likelihood of $o_i$ relative to $y$.
   • Conversely, if $o_i$ is worse than the reference, this term would penalize $o_i$ relative to $y$. The hyperparameter $\lambda$ sets the strength of this alignment regularization. It amplifies superior responses and penalizes inferior ones, integrating the exploration and imitation objectives.

4.2.2. Advantage Calculation with Normalization

The advantages $\hat{A}_{o_{i}}$ and $\hat{A}_{y}$ are crucial for weighting the different components of the loss. They are calculated in a group context and normalized to ensure stable gradient updates.

The normalized advantage $\hat{A}_{i}$ for any trajectory $i$ (could be $o_i$ or $y$) is defined as: $$\hat{A}_{i}=\frac{R\left(\text{trajectory}_i, y\right)-\mu_{r}}{\sigma_{r}}$$ Where:

• $R\left(\text{trajectory}_i, y\right)$: This is the raw reward for a given trajectory (either $o_i$ or $y$). This reward is provided by a pre-trained reward model (e.g., DeepSeek-v3 in the experiments) which evaluates the quality of the trajectory.
• $\mu_{r}$: This represents the mean reward across the entire group of $G$ trajectories, including the reference $y$. It is calculated as: $$\mu_{r}=\frac{1}{G} \sum_{j=1}^{G} R\left(o_{j}, y\right)$$ Note: The paper's formula for $\mu_r$ only explicitly sums over $o_j$. However, for $\hat{A}_y$ to be calculated, $y$ itself must be part of the group for reward computation or implicitly handled. Assuming R(y,y) is the reward of the reference and it's part of the group for $\mu_r$ and $\sigma_r$ calculation is a reasonable interpretation for consistency. If not, the group would be $G$ generated trajectories, and $y$ would be compared against them.
• $\sigma_{r}$: This is the standard deviation of the rewards within the group, including the reference. It is calculated as: $$\sigma_{r}=\sqrt{\frac{1}{G} \sum_{j=1}^{G}\left(R\left(o_{j}, y\right)-\mu_{r}\right)^{2}}$$ Similar to $\mu_r$, this calculation implies the rewards of all group members, including the reference if $\hat{A}_y$ is to be consistently computed within this group context.

This advantage normalization serves to stabilize the training process by scaling rewards and making gradients less sensitive to the absolute magnitude of rewards, focusing instead on relative preferences within a group.

4.2.3. Theoretical Analysis and Convergence Guarantees

The paper provides a theoretical analysis of GRAO's convergence properties within a stochastic approximation framework, demonstrating that the algorithm converges to stationary points under standard regularity conditions.

Assumptions (Appendix A.1.1): The convergence proof relies on the following standard assumptions:

• (A1) L-smooth objective: The objective function $\mathcal{J}_{\mathrm{GRAO}}(\theta)$ is $L$-smooth, meaning its gradients do not change too rapidly. Mathematically: $ \left|\nabla_{\theta} \mathcal{J}{\mathrm{GRAO}}\left(\theta{1}\right)-\nabla_{\theta} \mathcal{J}{\mathrm{GRAO}}\left(\theta{2}\right)\right| \leq L\left|\theta_{1}-\theta_{2}\right|, \quad \forall \theta_{1}, \theta_{2} \in \Theta $ Where:

  • $\|\cdot\|$: Euclidean norm.
  • $\nabla_{\theta} \mathcal{J}_{\mathrm{GRAO}}$: Gradient of the objective function with respect to parameters $\theta$.
  • $L$: Lipschitz constant for the gradient.
  • $\Theta$: Parameter space.

• (A2) Bounded policy gradients: The gradients of the log-likelihood of the policy are bounded. This prevents excessively large gradient steps. $ \left|\nabla_{\theta} \log \pi_{\theta}(a \mid s)\right| \leq B \quad \text { almost surely } $ Where:

  • $a$: action (generated token).
  • $s$: state (query and preceding tokens).
  • $B$: a positive constant.

• (A3) Reward boundedness: The raw rewards are bounded within a finite range. $

  |R(o, y)| \leq R_{\max } \text { for all trajectories }

  $ Where:

  • $R_{\max}$: maximum possible reward value.

• (A4) Advantage consistency: The normalized advantages are bounded and have bounded variance. This ensures the stability of the advantage estimates. $ \left|\hat{A}{i}\right| \leq C{A} \quad \text { and } \quad \operatorname{Var}\left(\hat{A}{i}\right) \leq \sigma{A}^{2} $ Where:

  • $C_A$: a positive constant for advantage bound.
  • $\sigma_A^2$: a positive constant for advantage variance bound.
  • This condition is stated to hold for sufficiently large group size $G$, specifically $G \geq \max \left(5, \frac{4 R_{\max }^{2}}{\sigma_{r}^{2}}\right)$, via the Berry-Esseen theorem, where $\sigma_r^2$ is the reward variance.

• (A5) Step size conditions: The learning rates $\left\{\eta_{k}\right\}$ (for iteration $k$) satisfy the standard Robbins-Monro conditions for stochastic approximation, which guarantee convergence. $ \sum_{k=1}^{\infty} \eta_{k}=\infty, \quad \sum_{k=1}^{\infty} \eta_{k}^{2}<\infty $ These conditions mean that learning rates must eventually decrease but not too quickly, ensuring that the algorithm can explore the parameter space sufficiently while eventually converging.

Convergence Guarantee (Appendix A.1.2): Under assumptions (A1)-(A5), the GRAO update sequence $\left\{\theta_{k}\right\}$ (model parameters at iteration $k$) satisfies: $$\liminf _{k \rightarrow \infty} \mathbb{E}\left[\left\|\nabla_{\theta} \mathcal{J}_{\mathrm{GRAO}}\left(\theta_{k}\right)\right\|\right]=0$$ This means that, in expectation, the norm of the gradient of the objective function approaches zero as the number of iterations goes to infinity. This is the standard definition of convergence to a stationary point (which can be a local minimum, maximum, or saddle point).

The parameter update rule is a standard stochastic gradient descent (SGD) step: $$\theta_{k+1}=\theta_{k}-\eta_{k} \widehat{\nabla \mathcal{J}}\left(\theta_{k}\right)$$ Where:

• $\theta_{k+1}$: updated model parameters.
• $\theta_{k}$: current model parameters.
• $\eta_{k}$: learning rate at iteration $k$.
• $\widehat{\nabla \mathcal{J}}\left(\theta_{k}\right)$: stochastic gradient estimator of the objective function at $\theta_k$.

Step 1: Stochastic gradient decomposition (Appendix A.1.2) The GRAO gradient estimator $\widehat{\nabla \mathcal{J}}$ is decomposed into its three component parts, mirroring the objective function structure: $$\begin{aligned} \widehat{\nabla \mathcal{J}} & =\underbrace{\frac{1}{G} \sum_{i=1}^{G} \hat{A}_{i} \nabla \mathcal{J}_{\text {exploration }}^{(i)}}_{\text{EXPLORATION TERM}}+\beta \underbrace{\hat{A}_{y} \nabla \mathcal{J}_{\text {reference }}}_{\text{IMITATION TERM }} \\ & +\lambda \underbrace{\frac{1}{G} \sum_{i=1}^{G} \hat{A}_{i}\left(\nabla \mathcal{J}_{\text {exploration }}^{(i)}-\nabla \mathcal{J}_{\text {reference }}\right)}_{\text{ALIGNMENT TERM }} \end{aligned}$$ Here, $\nabla \mathcal{J}_{\text {exploration }}^{(i)}$ refers to the gradient of the exploration term for trajectory $o_i$ (i.e., $\nabla_{\theta} \left(\frac{1}{\left|o_{i}\right|} \sum_{t=1}^{\left|o_{i}\right|} \log \pi_{\theta}\left(o_{i, t} \mid q, o_{i,<t}\right)\right)$), and $\nabla \mathcal{J}_{\text {reference }}$ refers to the gradient of the imitation term (i.e., $\nabla_{\theta} \left(\frac{1}{|y|} \sum_{t=1}^{|y|} \log \pi_{\theta}\left(y_{t} \mid q, y_{<t}\right)\right)$).

Step 2: Bounded gradient variance (Appendix A.1.2) Under assumptions (A2) (bounded policy gradients) and (A3) (reward boundedness), the second moment of the stochastic gradient is bounded: $$\mathbb{E}\left[\left\|\widehat{\nabla \mathcal{J}}\left(\theta_{k}\right)\right\|^{2}\right] \leq M^{2}$$ Where:

• $M$: A positive constant that bounds the expected squared norm of the gradient. It is defined as $M=B(1+\beta+2 \lambda)\left(C_{A}+R_{\max }\right)$, which combines the bounds from policy gradients, advantage consistency, and reward boundedness, as well as the hyperparameters $\beta$ and $\lambda$.

Step 3: Expected descent (Appendix A.1.2) Using the $L$-smoothness assumption (A1), we can bound the expected change in the objective function at each step: $$\begin{aligned} \mathcal{J}\left(\theta_{k+1}\right) & \leq \mathcal{J}\left(\theta_{k}\right)+\left\langle\nabla \mathcal{J}\left(\theta_{k}\right), \Delta \theta_{k}\right\rangle+\frac{L}{2}\left\|\Delta \theta_{k}\right\|^{2} \\ & =\mathcal{J}\left(\theta_{k}\right)-\eta_{k}\left\langle\nabla \mathcal{J}\left(\theta_{k}\right), \widehat{\nabla \mathcal{J}}\left(\theta_{k}\right)\right\rangle \\ & +\frac{L \eta_{k}^{2}}{2}\left\|\widehat{\nabla \mathcal{J}}\left(\theta_{k}\right)\right\|^{2} \end{aligned}$$ Taking expectations conditioned on $\theta_k$ and using the definition of the stochastic gradient and its bounded variance: $$\begin{aligned} \mathbb{E}\left[\mathcal{J}\left(\theta_{k+1}\right) \mid \theta_{k}\right] & \leq \mathcal{J}\left(\theta_{k}\right)-\eta_{k}\left\|\nabla \mathcal{J}\left(\theta_{k}\right)\right\|^{2} \\ & +\frac{L \eta_{k}^{2}}{2} \mathbb{E}\left[\left\|\widehat{\nabla \mathcal{J}}\left(\theta_{k}\right)\right\|^{2} \mid \theta_{k}\right] \\ & \leq \mathcal{J}\left(\theta_{k}\right)-\eta_{k}\left\|\nabla \mathcal{J}\left(\theta_{k}\right)\right\|^{2}+\frac{L \eta_{k}^{2}}{2} M^{2} \end{aligned}$$ This inequality shows that the expected objective value decreases at each step, provided that the learning rate $\eta_k$ is appropriately chosen, and the gradient is sufficiently large.

Step 4: Telescoping sum (Appendix A.1.2) Summing the inequality from Step 3 over $K$ iterations (from $k=1$ to $K$) and taking total expectations: $$\sum_{k=1}^{K} \eta_{k} \mathbb{E}\left[\left\|\nabla \mathcal{J}\left(\theta_{k}\right)\right\|^{2}\right] \leq \mathcal{J}\left(\theta_{1}\right)-\mathbb{E}\left[\mathcal{J}\left(\theta_{K+1}\right)\right]+\frac{L M^{2}}{2} \sum_{k=1}^{K} \eta_{k}^{2}$$ Since the objective function $\mathcal{J}$ is bounded below (a property of well-defined loss functions) and the sum of squared learning rates $\sum \eta_{k}^{2}$ is finite (from A5), the right-hand side is bounded. This implies: $$\sum_{k=1}^{\infty} \eta_{k} \mathbb{E}\left[\left\|\nabla \mathcal{J}\left(\theta_{k}\right)\right\|^{2}\right]<\infty$$ Given that $\sum \eta_{k}=\infty$ (from A5), this leads to the desired convergence result: $$\liminf _{k \rightarrow \infty} \mathbb{E}\left[\left\|\nabla \mathcal{J}\left(\theta_{k}\right)\right\|^{2}\right]=0$$

Interpretation of Conditions (Appendix A.1.3):

• Advantage normalization stability: Condition (A4) is crucial for stable gradients. It holds when the group size $G$ is sufficiently large, specifically: $ G \geq \max \left(5, \frac{4 R_{\max }^{2}}{\sigma_{r}^{2}}\right) $ This ensures that the advantage estimates are stable, and the gradient estimators remain well-behaved, preventing instability due to high variance.
• Exploration-imitation balance: The hyperparameter $\beta$ for the imitation term must be chosen carefully to prevent imitation from overly dominating the learning process while still ensuring convergence: $ 0<\beta<\frac{1}{L \cdot \mathbb{E}\left[\left|\nabla \mathcal{J}_{\text {reference }}\right|\right]} $ This condition implies that $\beta$ should not be too large, otherwise, the model might overfit to the reference trajectory, hindering exploration.
• Alignment regularization: The strength of the regularizer $\lambda$ should be scaled inversely with the variance of the advantage: $ \lambda=\mathcal{O}\left(\frac{1}{\sigma_{A}^{2}}\right) $ This ensures that the regularization effect is appropriate for maintaining gradient stability.

Practical Convergence Behavior (Appendix A.1.4): For a constant learning rate $\eta_{k}=\eta<\frac{1}{L}$, after $T$ iterations, the minimum expected squared gradient norm is bounded: $$\min _{1 \leq k \leq T} \mathbb{E}\left[\left\|\nabla \mathcal{J}\left(\theta_{k}\right)\right\|^{2}\right] \leq \frac{2\left(\mathcal{J}\left(\theta_{1}\right)-\mathcal{J}^{*}\right)}{\eta T}+L \eta M^{2}$$ Where $\mathcal{J}^{*}$ is the optimal (minimum) value of the objective function. The optimal choice of learning rate $\eta=\mathcal{O}(1 / \sqrt{T})$ yields a convergence rate of $\mathcal{O}(1 / \sqrt{T})$. This theoretical result confirms that GRAO converges to a stationary point, with advantages acting as bounded importance weights, ensuring policy improvement, and the alignment regularizer preventing gradient explosion.

In summary, GRAO dynamically interpolates between imitation learning (exploiting reference answers) and reinforcement learning (exploring novel trajectories), fostering robust alignment through adaptive self-improvement, backed by theoretical guarantees for convergence and stability.

5. Experimental Setup

5.1. Datasets

The authors evaluate GRAO using two standard benchmarks from Anthropic:

• Helpful-base: This dataset is designed to measure the model's ability to provide helpful and informative responses.

• Harmless-base: This dataset is designed to measure the model's ability to provide harmless and safe responses, avoiding toxic, biased, or unethical content.

  Both datasets provide tuples in the format (q, y_ref, y_rej), where:

• $q$: An input query or prompt.

• y_ref: A human-preferred response to the query (the "chosen" or "reference" response).

• y_rej: A human-rejected response to the query (the "rejected" response).

  The paper does not provide a concrete example of a data sample from these datasets directly in the main text. However, the case studies in Section A.3 implicitly provide examples of $q$ and y_ref (which is often HH.chosen in the evaluation prompt templates). For example, from Table 4:

• $q$ (Human): "Do you know the singer Adele?"

• y_ref (Assistant): "Sure. She's a very popular singer."

  These datasets are effective for validating the method's performance because they are widely recognized and used benchmarks in LLM alignment research, specifically designed to evaluate helpfulness and harmlessness, which are core alignment goals.

5.2. Evaluation Metrics

The performance assessment uses two primary metrics: Relative Adversarial Score (RAS) and Normalized Alignment Gain (NAG). For each metric, a conceptual definition, mathematical formula, and symbol explanation are provided below.

5.2.1. Relative Adversarial Score (RAS)

Conceptual Definition: RAS quantifies the proportion of instances where the model's generated output is rated higher (more preferred) than the reference response by a reward model. It measures how often the fine-tuned model can outperform the provided human-preferred reference.

Mathematical Formula: $$\mathrm{RAS}=\frac{1}{N} \sum_{i=1}^{N} \mathbb{I}\left(R\left(o_{i}, y_{\text {ref }, i}\right)>0\right)$$

Symbol Explanation:

• $N$: The total number of evaluation instances (queries) in the dataset.
• $\mathbb{I}(\cdot)$: An indicator function that returns 1 if the condition inside the parentheses is true, and 0 otherwise.
• $R(o_i, y_{\text{ref},i})$: The reward score assigned by a reward model for the $i$-th instance. This score indicates the relative preference between the model's output $o_i$ and the reference response $y_{\text{ref},i}$.
  • A score $>0$ (specifically, $>0$ in the formula) implies that the model's output $o_i$ is preferred over the reference $y_{\text{ref},i}$.
  • A score $=0$ implies they are considered equal.
  • A score $<0$ implies the reference $y_{\text{ref},i}$ is preferred over the model's output $o_i$. The paper specifies the scoring system as: 1 if $1>2$, 0 if $1=2$, -1 if $1<2$. Thus, $R(o_i, y_{\text{ref},i}) > 0$ means the model's output is strictly better.
• $o_i$: The output generated by the model being evaluated for the $i$-th query.
• $y_{\text{ref},i}$: The reference (human-preferred) response for the $i$-th query.

5.2.2. Normalized Alignment Gain (NAG)

Conceptual Definition: NAG measures the improvement in alignment achieved by the fine-tuning process by comparing the model's outputs after fine-tuning against its outputs before fine-tuning, relative to the reference response. It assesses how much closer to human preferences the model becomes due to the fine-tuning.

Mathematical Formula: $$\mathrm{NAG}=\frac{1}{N} \sum_{i=1}^{N} \mathbb{I}\left(R\left(o_{\text {post }, i}, y_{\text {ref }, i}\right)>R\left(o_{\text {pre }, i}, y_{\text {ref }, i}\right)\right)$$

Symbol Explanation:

• $N$: The total number of evaluation instances (queries) in the dataset.
• $\mathbb{I}(\cdot)$: An indicator function that returns 1 if the condition inside the parentheses is true, and 0 otherwise.
• $R(o_{\text{post},i}, y_{\text{ref},i})$: The reward score assigned by a reward model, indicating the preference of the post-fine-tuned model's output $o_{\text{post},i}$ relative to the reference response $y_{\text{ref},i}$ for the $i$-th query.
• $R(o_{\text{pre},i}, y_{\text{ref},i})$: The reward score assigned by a reward model, indicating the preference of the pre-fine-tuned model's output $o_{\text{pre},i}$ (e.g., the base SFT model's output) relative to the reference response $y_{\text{ref},i}$ for the $i$-th query.
• The condition $R(o_{\text{post},i}, y_{\text{ref},i}) > R(o_{\text{pre},i}, y_{\text{ref},i})$ means that the post-fine-tuned model's output is relatively better with respect to the reference than the pre-fine-tuned model's output.

5.3. Baselines

The paper compares GRAO against several representative LLM alignment methods, covering a spectrum from supervised to reinforcement learning approaches:

• Supervised Fine-Tuning (SFT): A foundational approach where the model is fine-tuned on high-quality, human-written responses. This represents the imitation-only baseline.

• Direct Preference Optimization (DPO): A reward-model-free alignment method that directly optimizes the policy using human preference data.

• Proximal Policy Optimization (PPO): A widely used RL algorithm in RLHF that fine-tunes the model using a reward model.

• Group Relative Policy Optimization (GRPO): An RLHF variant that leverages group sampling for more stable reward signals, similar in spirit to GRAO's group-based approach but without GRAO's specific three-component loss structure.

  These baselines are representative as they cover the major paradigms for LLM alignment, allowing for a comprehensive evaluation of GRAO's position and advantages within the current state-of-the-art.

5.4. Training Configuration

The experiments were conducted with the following specific configurations:

• Optimizer: Adam optimizer, with a weight decay of 0.01.
• Learning Rate: $1 \times 10^{-6}$.
• Batch Size: 64.
• Hyperparameters for GRAO:
  • $\beta = 0.5$: Controls the balance between imitation and exploration.
  • $\lambda = 0.6$: Sets the alignment regularization strength.
• Generation Parameters:
  • For each query, $G=8$ trajectories (output samples) were generated.
  • Temperature of 0.7 for sampling, encouraging diverse but coherent outputs.
  • Maximum generation length of 2048 tokens.
• Reward Model: A DeepSeek-v3 reward model was used to provide rewards. This reward model separately evaluates Helpful (using $RM_H$) and Harmless (using RM_HL) aspects of each generation. The input format and prompt templates for these reward models are consistent with specifications provided in Appendix A.2 (Tables 6 and 7).

Reward Model Prompt Templates (Appendix A.2): The reward model (DeepSeek-V3) uses specific prompt templates for evaluation. The structure is provided below:

The following are the results from [Table 6] of the original paper:

The following are the results from [Table 7] of the original paper:

These prompt templates guide the DeepSeek-V3 reward model to perform pairwise comparisons, indicating whether model.response (Text 1) is better than, equal to, or worse than HH.chosen (Text 2, which is the y_ref in the datasets) for helpfulness or harmlessness.

5.5. Models

Experiments are conducted on two representative Large Language Model architectures:

• Qwen2.5-7B: A dense (non-sparse) model with 7 billion parameters. This serves as a standard, widely used LLM architecture.
• Moonlight-16B-A3B: A Mixture-of-Experts (MoE) architecture with 16 billion total parameters, but only 3 billion activated parameters per inference. This represents a modern, more efficient architecture, particularly relevant for understanding how alignment methods generalize to sparse models.

6. Results & Analysis

6.1. Core Results Analysis

The GRAO method consistently achieves state-of-the-art alignment performance across both helpfulness and harmlessness benchmarks, significantly outperforming all baselines (SFT, DPO, PPO, GRPO) on both dense (Qwen2.5-7B) and sparse MoE (Moonlight-16B) models. The improvements highlight GRAO's ability to overcome common challenges like reward sparsity and policy instability.

6.1.1. Overall Performance: Helpfulness Benchmark

The following are the results from [Table 1] of the original paper:

Analysis for Helpfulness:

• Qwen2.5-7B (Dense Model): GRAO achieved 64.60% RAS and 67.98% NAG. This represents improvements of +3.71% RAS and +7.24% NAG over the next best baseline, GRPO (60.89% RAS, 60.74% NAG). The improvement over SFT is dramatic, showing the benefit of moving beyond simple imitation. DPO and PPO perform reasonably well but are still significantly surpassed by GRAO.
• Moonlight-16B (MoE Model): GRAO achieved 70.84% RAS and 55.06% NAG. Here, GRAO gained +1.95% RAS and +4.24% NAG over GRPO (68.89% RAS, 50.82% NAG). The NAG for SFT is negative, indicating that the base SFT model's outputs were often worse than the pre-fine-tuned model's outputs relative to the reference, highlighting the difficulty of alignment without effective exploration.

6.1.2. Overall Performance: Harmlessness Benchmark

The following are the results from [Table 2] of the original paper:

Analysis for Harmlessness:

• Qwen2.5-7B: GRAO achieved 68.01% RAS and 31.06% NAG, demonstrating +2.4% RAS and +2.8% NAG over GRPO (65.61% RAS, 28.26% NAG).

• Moonlight-16B: GRAO shows even more significant gains here, reaching 76.82% RAS and 34.85% NAG. This is a dramatic +8.74% RAS and +22.74% NAG improvement over GRPO (68.08% RAS, 12.11% NAG). This particularly strong performance on the MoE model in harmlessness tasks is noteworthy, indicating GRAO's effectiveness for complex, safety-critical alignments.

  Overall, the results consistently show GRAO's superior ability to align models, especially for the more complex MoE architectures and the critical harmlessness tasks. The statistically significant improvements confirm its effectiveness and stability.

6.2. Trajectory Dynamics Analysis

To evaluate GRAO's optimization efficiency, the authors compared its training dynamics with PPO and GRPO baselines on the Qwen2.5-7B model for the helpful-base dataset.

The following figure (Figure 2 from the original paper) shows the training dynamics (Qwen2.5-7B, helpful-base):

Analysis of Training Dynamics:

• Rapid Initial Convergence: GRAO achieves optimal policy performance in approximately half the training steps required by PPO and GRPO. This rapid progress is attributed to its imitation component ($\mathcal{J}_{\text {imitation }}$), which quickly guides the policy towards high-reward regions by leveraging reference answers.
• Progressive Refinement: The alignment regularization term ($\lambda \hat{A}_{o_{1}}$ differential) in GRAO amplifies high-advantage trajectories while suppressing low-reward paths. This mechanism enables continuous refinement of the policy.
• Stable Ascent: GRAO utilizes advantage normalization, which effectively prevents gradient explosions during exploration. This contributes to monotonic improvement and overall stability in training.
• Comparison with Baselines:

  • Beyond initial convergence (around 800 steps), PPO tends to plateau. This is often due to its KL-divergence constraints, which limit how much the policy can change at each step, making further exploration difficult once a local optimum is found.

  • GRPO, despite using group sampling, exhibits a $\pm 9.6\%$ variance in reward. This instability suggests that its group sampling mechanism, without the specific loss structure of GRAO, is less robust.

  • In contrast, GRAO consistently delivers an average reward gain of 0.83% per step, maintaining policy refinement and stability far beyond the initial optimization phase, thanks to its integrated triple-objective approach (imitation, exploration, and regularization).

    This analysis clearly demonstrates GRAO's heightened efficiency and robustness during training, confirming its ability to converge faster and more stably than existing RLHF methods.

6.3. Component Ablation Study

A systematic ablation study was conducted to quantify the individual contributions of each component of GRAO's objective function. The study evaluates the impact on NAG (Normalized Alignment Gain) for the helpfulness task.

The following are the results from [Table 3] of the original paper:

The following figure (Figure 3 from the original paper) shows the component ablation effects on training dynamics (Qwen2.5-7B, helpful-base):

Analysis of Ablation Study:

• Full GRAO: The complete GRAO framework achieves the highest NAG scores (67.98% for Qwen2.5-7B and 55.06% for Moonlight-16B), serving as the benchmark for comparison.
• w/o $\mathcal{J}_{\text{imitation }}$ (without the imitation component):
  • Performance drops by 7.79% overall. For Qwen2.5-7B, NAG is 63.79%, and for Moonlight-16B, it's 49.87%.
  • Effect: Removing imitation leads to reduced initial alignment efficiency (as seen in Figure 3, the initial rise is slower). This confirms that the imitation component is crucial for rapid initial guidance. However, it preserves a significant portion (92.21%) of the final performance, suggesting compensatory effects from exploration and regularization can still drive learning, albeit slower.
• w/o $\mathcal{J}_{\text{exploration }}$ (without the exploration component):
  • Performance drops by a substantial 12.81% overall. NAG is 64.38% for Qwen2.5-7B and 43.86% for Moonlight-16B.
  • Effect: This leads to a large performance decline. As seen in Figure 3, the w/o exploration curve plateaus much earlier and at a lower reward. This limitation indicates that without active exploration of new trajectories and self-correction, the model's policy search space is severely restricted, preventing it from discovering optimal improvements beyond initial imitation.
• w/o $\mathcal{J}_{\text{alignment\_regularizer }}$ (without the alignment regularizer component):

  • Performance drops by the largest margin, 12.98% overall. NAG is 61.18% for Qwen2.5-7B and 46.26% for Moonlight-16B.

  • Effect: Excluding the regularizer accelerates early training progress (Figure 3 shows a slightly faster initial rise than w/o imitation but w/o exploration). However, it significantly restricts the final NAG to 87.02% of the full GRAO formulation. This is because the absence of the regularizer likely leads to increased divergence between model trajectories and reference outputs, hindering stable and progressive policy enhancement in the long run.

    These findings strongly reinforce the effectiveness of GRAO's "imitate-explore-transcend" paradigm. Each component plays a distinct and critical role:

• Imitation anchors initial learning, providing a strong starting point.
• Exploration uncovers optimal improvements, allowing the model to go beyond known examples.
• Alignment regularization integrates these elements, ensuring stable and consistent progressive policy enhancement by managing the interplay between generated and reference outputs. The combined synergy is essential for GRAO's superior performance.

6.4. Further Understanding Alignment Goals

To further clarify the mechanics of GRAO, the authors analyzed the optimization process by tracking the progression of the total loss and the relative contributions of each objective component over training steps.

The following figure (Figure 4a from the original paper) shows the training loss during GRAO alignment (Qwen2.5-7B, helpful-base):

该图像是图表，展示了训练过程中不同损失项（Alignment Regularization、Exploration、Imitation）随训练步数变化的趋势，反映了整体损失的逐步下降情况。

The following figure (Figure 4b from the original paper) shows the percentage contribution of each component to total GRAO optimization loss (Qwen2.5-7B, helpful-base):

Analysis of Loss Progression and Component Contribution: Two distinct phases emerge in GRAO's alignment process:

1. Rapid Alignment Phase (steps < 200):

   • Loss Behavior: During this initial phase (Figure 4a), the overall training loss decreases rapidly, indicating fast convergence towards a better policy.
   • Component Contribution (Figure 4b): The total loss is driven predominantly by the imitation component ($\mathcal{J}_{\text {imitation }}$) and the alignment regularization component. Together, these two components account for over 82% of the total loss magnitude.
   • Interpretation: This phase leverages the strengths of SFT, quickly pulling the model towards optimal policies by learning from high-quality reference answers and actively constraining divergence from these references. The model is primarily in an "imitate" mode.

2. Refinement Phase (steps > 200):

   • Loss Behavior: As optimization continues (Figure 4a), the total loss decreases exponentially, but the rate of decrease slows down, indicating fine-grained adjustments.

   • Component Contribution (Figure 4b): A significant shift occurs in the contribution breakdown. The exploration component ($\mathcal{J}_{\text {exploration }}$) becomes the dominant objective, contributing between 52-61% of the total loss. Conversely, the contribution from the imitation component drops below 40%.

   • Interpretation: This shift signifies that the model's own generated outputs become the primary drivers of further improvement. The model has sufficiently learned from the references and is now actively discovering and reinforcing superior trajectories through self-exploration and self-correction. This enables advancement beyond mere imitation of the reference responses, entering the "explore" and "transcend" stages.

     These results empirically validate the phased structure of GRAO's "imitate-explore-transcend" paradigm. The imitation component provides a strong, fast start by anchoring initial learning. The exploration component then takes over to discover superior trajectories, allowing the model to autonomously enhance its skills. The alignment regularization integrates these components, ensuring stability throughout this dynamic process. The eventual predominance of exploration during refinement powerfully demonstrates GRAO's ability to transcend reference trajectories and achieve autonomous skill progression while maintaining alignment stability.

6.5. Generalization to Different Model Types

The paper also investigates GRAO's performance on different model architectures, specifically comparing dense models with Sparse Mixture-of-Experts (MoE) architectures.

Key Finding: GRAO delivers notably greater performance improvements on sparse MoE architectures (Moonlight-16B) compared to dense models (Qwen2.5-7B).

Evidence:

• As shown in Table 1 (Helpfulness) and Table 2 (Harmlessness), the Moonlight-16B MoE model consistently achieves substantially higher gains from GRAO alignment compared to the dense Qwen2.5-7B model, both in absolute performance and relative improvement over baselines like GRPO. For instance, in harmlessness, Moonlight-16B with GRAO saw a dramatic +22.74% NAG improvement over GRPO, far exceeding the +2.8% NAG gain for Qwen2.5-7B.

Reasoning for Superior Efficacy with MoE: This enhanced efficacy arises from a synergy between GRAO's optimization dynamics and the unique properties of MoE architectures:

• Gradient Sparsity: MoE models naturally exhibit inherent gradient sparsity patterns. This means that for any given input, only a subset of "expert" sub-networks are activated, and thus, gradients are computed and applied only to the parameters of these activated experts.

• GRAO's Advantage-Normalized Gradient Formulation: GRAO's gradient formulation, $\widehat{\nabla \mathcal{J}}=\frac{1}{G} \sum_{i=1}^{G} \frac{A_{i}}{\sigma_{A}} \nabla \mathcal{J}^{(i)}$, is particularly well-suited for such sparse architectures. It effectively concentrates updates on high-impact parameters (those corresponding to highly-rewarded trajectories) and minimizes interference among different expert modules. By weighting gradients with normalized advantages, GRAO can efficiently identify and reinforce the contributions of specific experts that lead to superior outputs, without excessively disrupting other experts.

  This demonstrates GRAO's adaptability across different model families and highlights its potential as a robust and unified alignment solution, especially for next-generation heterogeneous architectures like MoE, which are becoming increasingly prevalent due to their efficiency.

6.6. Case Study

To qualitatively evaluate alignment quality, the authors analyzed model responses to sensitive queries across different alignment methods.

6.6.1. Query 1: Cultural Awareness (Table 4)

The following are the results from [Table 4] of the original paper:

Analysis:

• GRAO: Provides a comprehensive response, detailing Adele's nationality, vocal characteristics, accolades (Grammy Awards), and popular works. This demonstrates high information density and relevance.

• SFT: Delivers minimal information, just stating "talented singer" without substantive details, reflecting its limitation to direct imitation of the brief reference.

• DPO: Includes relevant facts (British singer-songwriter, success) but omits artistic characteristics and notable works, showing an improvement over SFT but lacking depth.

• GRPO: Focuses narrowly on awards without contextualizing their artistic significance or broader impact, indicating a partial understanding.

  Conclusion: GRAO achieves higher information density and factual accuracy, synthesizing multiple dimensions of knowledge into a coherent and helpful response, surpassing the baselines.

6.6.2. Query 2: Cultural Sensitivity (Table 5)

The following are the results from [Table 5] of the original paper:

Analysis:

• GRAO: Uniquely identifies cultural similarity as the determining factor in consumer preferences, which is a sociologically plausible and nuanced explanation aligned with the implied intent of the query. It avoids harmful stereotyping and provides a sophisticated answer.

• SFT: Exhibits severe repetition, with two redundant sentences. This suggests poor attention control or compositional failures, a common issue with SFT models that struggle with generalization.

• DPO: Introduces irrelevant luxury car brands (German/Italian) that were not present in the context of the preceding conversation (which focused on Japanese/Korean cars). This is a form of hallucination or lack of contextual awareness.

• GRPO: Makes factually incorrect claims about geographic proximity determining preferences. It incorrectly states that Americans prefer cars from Canada or Mexico over Japan/Korea, contradicting the initial premise about Japanese/Korean cars being preferred. This shows a failure in factual understanding and reasoning.

  Conclusion: These case studies validate GRAO's dual strengths:

1. Information Synthesis: Its ability to integrate multiple knowledge dimensions into comprehensive and coherent responses.
2. Cultural Alignment: Its effectiveness at navigating sensitive topics by providing appropriate, nuanced, and factually correct explanations, where baseline methods often hallucinate, contradict facts, or suffer from compositional failures. GRAO avoids the common failure modes observed in other alignment techniques.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper introduces Group Relative Alignment Optimization (GRAO), a novel and unified framework designed to enhance the alignment capabilities of large language models. GRAO effectively integrates the efficiency of supervised fine-tuning (SFT) for rapid knowledge acquisition with the exploratory power of reinforcement learning (RL) for autonomous refinement. Its core mechanism, an "imitate-explore-transcend" trajectory, dynamically balances learning from high-quality reference data with self-driven discovery of superior behaviors.

The theoretical analysis establishes GRAO's robust convergence properties and highlights its sample efficiency advantages over traditional methods. Empirically, GRAO consistently demonstrates superior performance across both helpfulness and harmlessness benchmarks, achieving significant relative improvements over SFT (57.70%), DPO (17.65%), PPO (7.95%), and GRPO (5.18%) baselines. Notably, GRAO shows even more pronounced gains, up to 22.74% Normalized Alignment Gain (NAG) improvement over GRPO, particularly in sparse Mixture-of-Experts (MoE) architectures, underscoring its adaptability to modern LLM designs.

The framework's success stems from its three principled components: a strong imitation learning component for initial policy guidance, advantage-weighted exploration for efficient policy refinement, and an alignment regularizer for stable training. Trajectory analysis confirms GRAO's faster convergence and sustained optimization, evolving from initial imitation to autonomous skill enhancement. Qualitative case studies further illustrate GRAO's ability to generate comprehensive, culturally aware, and sensitive responses, effectively circumventing the failure modes observed in other models.

In essence, GRAO offers a scalable and robust solution for aligning LLMs, efficiently utilizing both reference and emergent data, and positions itself as a promising approach for developing advanced, well-aligned AI systems.

7.2. Limitations & Future Work

The authors suggest several directions for future work, indicating areas where the GRAO framework could be further extended:

• Multi-objective Scenarios: The current framework primarily focuses on helpfulness and harmlessness. Future work could extend GRAO to handle more complex multi-objective alignment tasks, where models need to optimize for several potentially conflicting criteria simultaneously.

• Continual Learning: Integrating GRAO into continual learning scenarios would allow models to adapt and improve their alignment over time, without forgetting previously learned behaviors, as new data and preferences emerge.

  While the paper does not explicitly state limitations of GRAO, potential implicit limitations or areas for further investigation could include:

• Computational Cost: Generating $G=8$ trajectories for each query and processing their rewards, along with the group-wise advantage calculations, might introduce higher computational overhead compared to single-sample methods, especially for very large $G$. The trade-off between performance gains and computational cost could be further analyzed.

• Reward Model Dependency: Although GRAO is a unified approach, it still relies on an external reward model (DeepSeek-v3) for feedback. The quality and biases of this reward model directly impact GRAO's learning. Future work might explore how GRAO performs with self-trained or less sophisticated reward models, or even integrate reward model learning directly into the GRAO objective.

• Hyperparameter Sensitivity: The hyperparameters $\beta$ and $\lambda$ are crucial for balancing imitation, exploration, and regularization. Their optimal values might be sensitive to the specific task, dataset, or base model. A deeper analysis of their sensitivity and strategies for adaptive hyperparameter tuning could be beneficial.

• Interpretability of "Transcendence": While the paper demonstrates "transcendence" empirically, a deeper theoretical or mechanistic understanding of how GRAO enables the model to truly go beyond the original policy outputs and discover novel, superior solutions could be a valuable area of research.

7.3. Personal Insights & Critique

This paper presents a highly compelling and innovative approach to LLM alignment. The "imitate-explore-transcend" paradigm beautifully encapsulates a natural learning process that is often desired in complex AI systems. The core strength lies in its principled unification of SFT's efficiency and RL's adaptability, addressing key pain points of each individual method.

Inspirations:

• Dynamic Learning: The dynamic shift in loss contribution from imitation to exploration (Figure 4b) is a powerful illustration of self-driven learning. This phased approach could inspire similar multi-stage or adaptive learning mechanisms in other AI domains, where an initial rapid acquisition phase is followed by self-improvement and refinement.
• Robustness in MoE Architectures: GRAO's exceptional performance on MoE models is particularly insightful. The idea that advantage-normalized gradients can effectively leverage the sparse nature of MoE architectures to optimize performance while minimizing interference among experts is a significant finding. This suggests that alignment methods should be designed with the underlying model architecture in mind, rather than being treated as black-box add-ons. This could lead to more architecture-aware alignment techniques.
• Bridging the Gap: The paper successfully bridges the theoretical elegance of convergence guarantees with strong empirical results, which is a hallmark of rigorous academic work. The detailed theoretical analysis in the appendix provides confidence in the method's stability.

Potential Issues/Areas for Improvement:

• Reward Model Limitations: As mentioned above, the reliance on a separate reward model, even a sophisticated one like DeepSeek-v3, means GRAO is still subject to the reward model's potential biases, limitations, or misalignments. While GRAO improves policy learning from a given reward signal, it doesn't fundamentally solve potential issues within the reward signal itself.

• Complexity vs. Simplicity: GRAO introduces a more complex objective function with three components and several hyperparameters ($\beta$, $\lambda$). While the ablation study validates each component, the interplay and tuning of these parameters in real-world, large-scale deployments could be challenging. Further analysis on the robustness of GRAO to hyperparameter choices could strengthen its practical appeal.

• Scalability to Ultra-Large Models: The experiments are conducted on 7B and 16B parameter models. While these are substantial, the computational demands of $G=8$ samples per query might scale differently for models with hundreds of billions or trillions of parameters. An explicit discussion on this scalability, perhaps through computational benchmarks, would be valuable.

• Ethical Considerations Beyond Data: The ethics statement focuses on publicly available datasets and privacy. However, LLM alignment, especially for helpfulness and harmlessness, often delves into complex ethical dilemmas (e.g., defining "harm," dealing with conflicting human preferences, or potential for manipulation). While the paper focuses on the technical alignment process, a broader discussion on these inherent ethical challenges of defining and achieving "aligned" behavior could add another layer of depth.

  Overall, GRAO represents a significant step forward in making LLMs more robust, efficient, and truly "self-optimizing" in their alignment journey. Its comprehensive approach and strong results provide a solid foundation for future research in this critical area.