The Realignment Problem: When Right becomes Wrong in LLMs
Correspondence: aakash.sensharma@invideo.io
Post-training alignment of large language models (LLMs) relies on large-scale human annotations guided by policy specifications that change over time. Cultural shifts, value reinterpretations, and regulatory or industrial updates make static alignment increasingly brittle. As policies evolve, deployed models can diverge from current alignment objectives, creating an Alignment–Reality Gap that is difficult to audit or correct. Existing remediation typically requires re-annotation under revised guidelines, which introduces systematic challenges, including guideline ambiguity, annotator interpretation drift, and reduced consistency at scale.
We introduce TRACE (Triage and Re-align by Alignment Conflict Evaluation), a framework that transforms re-alignment into a structured optimization problem over existing data without requiring fresh human annotation. Leveraging a stronger model as a proxy judge, TRACE operates via a three-stage pipeline: (1) triaging preference pairs into inversion, suppression, or retention categories based on alignment conflicts; (2) computing an alignment impact score via bi-level optimization to prioritize high-leverage samples; and (3) executing updates using a hybrid objective that combines relational losses (e.g., IPO) for preference inversion and punitive losses (e.g., NPO) for response suppression.
Experiments on Qwen2.5-7B, Gemma-2-9B, and Llama-3.1-8B demonstrate robust re-alignment on synthetic benchmarks and the PKU-SafeRLHF dataset without degrading general utility. This work provides a scalable approach for LLM realignment under evolving data annotation policies and alignment guidelines.
Human Preference Evaluation
Annotators were shown response triplets (DPO-Gold, TRACE, U2A) and selected the best one following π_new, blind to model identity.
| Method | PKU-SafeRLHF Win Rate vs. U2A | SynthValueBench Win Rate vs. U2A | Krippendorff's α |
| DPO-Gold (upper bound) | 87.1% | 92.4% | 0.80 |
| TRACE (Ours) | 81.8% | 85.3% | 0.77 |
| U2A (baseline) | — | — | 0.76 |
Table 1. Human preference evaluation. TRACE closes 81.8% of the gap between purely punitive unlearning and gold-standard full re-annotation.
General Capability Benchmarks (PKU-SafeRLHF)
TRACE preserves MMLU and GSM8K within confidence intervals of the base model. The HellaSwag reduction (~3.2 pts) is bounded and favorable given the 81.8% policy adherence gain.
| Method | GPQA ↑ | MMLU ↑ | HellaSwag ↑ | GSM8K ↑ |
| Base Model | 31.6 ± 0.9 | 70.6 ± 0.8 | 81.4 ± 1.0 | 70.4 ± 0.8 |
| DPO-Gold | 32.1 ± 1.1 | 70.5 ± 0.9 | 81.3 ± 1.2 | 70.8 ± 1.0 |
| TRACE (Ours) | 30.1 ± 0.1 | 70.2 ± 0.8 | 78.2 ± 0.9 | 70.6 ± 0.7 |
| U2A | 29.5 ± 0.3 | 70.2 ± 1.1 | 80.8 ± 1.2 | 69.9 ± 1.1 |
Table 2. General capability benchmarks. TRACE preserves general utility while achieving strong re-alignment.
Target Policy Agreement (PKU-SafeRLHF)
TPA measures the percentage of responses complying with π_new, scored by the policy oracle. TRACE at 5k samples outperforms Naive Oracle DPO at 20k samples.
| Model | Data Size | Naive Oracle DPO | TRACE (Ours) | Δ |
| Llama-3.1-8B | 5k | 35.9 | 55.9 | +20.0 |
| 10k | 46.5 | 65.5 | +19.0 | |
| 20k | 52.2 | 70.7 | +18.5 | |
| Gemma-2-9B | 5k | 37.1 | 54.4 | +17.3 |
| 10k | 48.2 | 66.9 | +18.7 | |
| 20k | 53.9 | 71.8 | +17.9 |
Table 3. Target Policy Agreement across model families and data scales. Gains are consistent and not explained by oracle access alone.
Component-wise Ablation (Llama-3.1-8B)
Each component of TRACE contributes to the final performance. KL regularization is critical for preserving general utility; triage and impact weighting drive policy agreement.
| Variant | Policy Agreement ↑ | MMLU ↑ | ASR ↓ |
| TRACE (Full) | 70.7 | 70.2 | 27.3 |
| w/o Triage | 58.1 | 70.2 | 24.6 |
| w/o KL Regularization | 71.5 | 64.1 | 29.8 |
| w/o Impact Weighting | 62.8 | 69.5 | 32.1 |
Table 4. Ablation study. Removing any component degrades performance; KL regularization is especially critical for capability preservation.
@inproceedings{sharma2026trace,
title = {{TRACE}: Triage and Re-align by Alignment Conflict Evaluation},
author = {Sen Sharma, Aakash and Sanyal, Debdeep and Ray, Manodeep
and Srivastava, Vivek and Karande, Shirish and Mandal, Murari},
booktitle = {International Conference on Machine Learning},
year = {2026}
}