ComPO vs DPO: Tuning Llama-3-8B at 23GB Instead of 77GB

Shares of Llama-3-8B, not gigabytes. That’s the form the memory claim takes in this paper, and it’s how the best number in it hides. I went in hunting a memory figure, skimmed straight past the thing, and had to loop back for it. What I’d skipped was 0.0002% of model parameters per update, parked nowhere near the memory tables I came for.

Flattened out: DPO asks roughly 77GB to tune Llama-3-8B. ComPO does the same work on about 23GB.

That’s the whole story. Stated flat, on purpose.

How it gets there is the odd part. No backpropagation anywhere in the training loop. Optimizer states go with it. So does the activation bookkeeping.

ComPO vs DPO: Memory and Compute Compared

Side by side, on the numbers this post can stand behind:

Where the specifics come from: the paper runs 39 pages, pitched as a zeroth-order approach to LLM preference alignment, written by Peter Chen, Xi Chen, Wotao Yin, and Tianyi Lin. Filed under Computation and Language, cross-listed in Artificial Intelligence and Machine Learning. ArxivDaily’s summary credits the work to UC Berkeley, New York University, Columbia University, and DAMO Academy, Alibaba Group U.S.

The method’s name is ComPO, short for Comparison-based Preference Optimization.

Both the parameter share and the memory figures come off the paper’s alphaxiv page.

The 2505 in that identifier reads as May 2025, a date code I trust ahead of most launch announcements. Two sittings to finish it. The second one existed mostly because of the estimator section, which is where I’d bet most readers stall.

How ComPO Works Without Backprop on Llama-3-8B

First-order training cannot run without gradients.

Gradients drag in backprop.

Backprop drags in the memory bill.

Zeroth-order training declines the whole arrangement.

The paper treats alignment as a latent objective and estimates which way to move using nothing beyond what a comparison oracle reports back.

Per the Moonlight review, one pass of the loop goes like this.

Sample m independent perturbation vectors off a unit sphere. Query the oracle at the current parameters, then once more at a perturbed point offset by a sampling radius r. Then solve for a sparse normalized gradient estimate. The one that maximizes the average correlation between comparison outcomes and perturbation directions. With an l1 bound plus an l∞ cap sitting on it. The cap is what forces the sparsity.

In plainer words: poke the weights in m random directions. Keep the pokes that moved preferences the way the oracle wanted, then step along the consensus. The authors describe it as “extracting directional information from preference pairs without directly optimizing a differentiable preference loss on them.”

Full admission: the estimator math beat me on the first pass. I took it twice. And I still won’t pretend the derivation decides whether you should try the method, since it doesn’t.

Noisy Pairs Carry the Signal

Direct alignment methods built their reputation on computational and memory efficiency. The arXiv listing frames it that way itself. Then your preference data turns messy, and the trouble starts.

Their diagnosis comes through the OpenReview entry: DPO-style methods suffer verbosity and likelihood displacement.

And noisy pairs are the driver.

Preferred and dispreferred responses land at similar likelihood under the model. A near-tie hands the loss almost nothing to learn from. Push harder and judgment doesn’t sharpen. Probability mass gets shoved somewhere unhelpful, and the model talks more while answering worse.

The standard operator move is to filter ambiguous pairs out.

ComPO runs the other way and studies how to use the information sitting inside noisy pairs with small likelihood margins.

The authors’ stated motivation is that likelihood displacement pushed them toward alternative ways to extract information from low-margin pairs. Your worst data, converted into training signal.

That’s a thesis worth taking seriously instead of shrugging past.

Assumptions Behind the Guarantees

Most tuning papers don’t bother with theory at this weight.

This one does. There’s a convergence guarantee for the basic offline ComPO scheme. And it leans on smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. A constrained scheme gets a performance guarantee too, under local coverage and in-distribution pairwise reward accuracy. Past all that sits an online ComPO variant — same offline comparison mechanism.

But it feeds on unlabeled policy generations for reverse-KL control against a reference policy.

Here is where I slow down.

Gradient sparsity is an assumption about your model, not a fact about your data.

Whether an alignment objective I’d actually ship is sparse in the estimator’s sense — no idea. You won’t know either, not until something breaks. No sparsity, no guarantee following you into production. In-distribution pairwise reward accuracy works the same way: let the oracle’s judgments drift off the training distribution and the performance guarantee walks out with them.

The empirics came in broader than I expected.

Papers.cool and ArxivDaily both summarize the experiment set, and it spans Mistral, Llama, Gemma-2, Qwen3, and Gemma-3. Five families. Improvements over existing direct alignment methods, length-controlled win rates included, with pair-level diagnostics consistent with mitigating likelihood displacement. When verbosity is one of the failure modes you claim to fix, length-controlled win rates are exactly the right metric to report.

On the headline number itself: 23GB versus 77GB for Llama-3-8B is a single-source figure from the alphaxiv listing. And I haven’t seen it repeated elsewhere. The direction of the gap is what the zeroth-order mechanics imply — no backprop, no optimizer states. So a large drop is plausible. Treat the exact 77GB as unconfirmed until you’ve reproduced it on your own hardware.

ComPO FAQ: The Short Answers

What is ComPO?

Comparison-based Preference Optimization, a zeroth-order alignment method from a 39-page paper by Peter Chen, Xi Chen, Wotao Yin, and Tianyi Lin.

Instead of optimizing a differentiable preference loss, it perturbs weights in random directions and reads which perturbations the comparison oracle preferred.

How much memory does ComPO save vs DPO?

On Llama-3-8B, about 23GB against DPO’s roughly 77GB, per the alphaxiv listing.

Single-source. Reproduce it before quoting the number at anyone.

Does ComPO need gradients?

No. The estimator works off comparison outcomes sampled at perturbed points. No backprop, no optimizer states, no activation bookkeeping. That’s where the memory drop comes from.

Should you replace DPO with ComPO?

Not yet. The assumptions behind the convergence guarantee are strong, the headline memory figure is single-source. And nothing here is a drop-in library swap.

Watch rather than migrate, and let the follow-up work land before you rebuild a pipeline that already functions.

If ComPO is on your radar anyway, the practical moves look like this:

– Sort your preference pairs by likelihood margin before touching anything labeled bad data. If the thesis holds, near-ties carry direction, which would make your filtering step a place where signal gets thrown out.
– Count the near-ties. As a share of all pairs, it’s the honest measure of what this method offers you specifically.
– Quote tuning work in memory footprint, not token count. A 77GB-versus-23GB gap is the gap between classes of hardware. Price it that way.

Cost, not novelty, is the actual story. Alignment has been priced for labs. And 77GB of training memory is a procurement conversation before it’s a research one. Drop that to 23GB and tuning becomes an afternoon of rented compute, which changes who gets to do this work at all. A method that skips backprop also fits any workflow where you only ever touch a model through its outputs.

I keep a text file of memory ceilings for every GPU I’ve ever rented. So that paragraph was basically written for me.

The bet worth making isn’t ComPO specifically.

It’s that alignment research has started attacking the cost of alignment itself. And every paper in that line makes custom tuning survivable for operations our size. A 39-page paper from four institutions that turns noisy preference data and a memory wall into a solvable problem is a signpost, not a product.

23GB versus 77GB. That’s the takeaway to carry out of this one.

At Mediascout I build and ship AI automation against real hardware budgets.

And this is exactly the class of research I track to know what becomes affordable next. If you want preference tuning scoped to compute you can actually book, that is the work we do.

Name a model and a memory ceiling, and I’ll tell you what’s realistic.