
Supervised fine-tuning can teach a model to imitate. It cannot teach a model to
be better than what it imitates.

That is the whole reason RLHF exists. If you have a dataset of good answers, you
can train the model to reproduce them token by token — that is next-token
prediction with a nicer dataset. But a lot of what makes an answer good is not
expressible as a target string. It is comparative: this answer is more helpful
than that one, this one refuses when it should not, this one is technically
right but insufferable to read. Nobody can write down the correct answer, but
almost anyone can pick the better of two.

RLHF is the machinery for turning that comparison into a gradient. PPO is the
part that keeps the gradient from wrecking the model on the way.

## The three stages

RLHF is not one algorithm. It is three training runs stacked on top of each
other, and confusing them is the usual source of confusion about the whole
thing.

**Stage 1 — supervised fine-tuning.** Take a pretrained base model and fine-tune
it on demonstrations: prompts paired with responses a human wrote or approved.
This is ordinary cross-entropy training. Its job is not to make the model good,
only to make it answer in roughly the right format so the later stages have
something workable to sample from. Call the result the SFT model.

**Stage 2 — the reward model.** Sample several responses to the same prompt from
the SFT model, have a human rank them, and train a separate model to predict
those rankings as a scalar score.

**Stage 3 — reinforcement learning.** Let the SFT model generate, score its
generations with the reward model, and update the model to get higher scores.
This is where PPO comes in.

The interesting design decisions are all in stages 2 and 3.

## Stage 2: learning a reward from comparisons

Humans are bad at absolute scores. Ask ten annotators to rate an answer from 1
to 10 and you get ten different scales. Ask them which of two answers is better
and they agree far more often. So preference data is collected as pairs: for a
prompt `x`, a chosen response `y_w` and a rejected one `y_l`.

The reward model `r` is usually the SFT model with the language-modelling head
replaced by a single scalar output. It is trained with the Bradley-Terry
objective, which says the probability a human prefers `y_w` over `y_l` is the
sigmoid of the score difference:

```python
# reward_chosen, reward_rejected: (batch,) scalars from the reward model
loss = -torch.nn.functional.logsigmoid(reward_chosen - reward_rejected).mean()
```

Three consequences fall out of this loss, and all three matter later.

The reward is only meaningful **relative to other completions of the same
prompt**. Nothing anchors the absolute value — add a constant to every score and
the loss is unchanged. So the numbers a reward model emits are not comparable
across prompts, and "reward went up" only means something within a prompt.

It is a **learned, imperfect proxy** for human preference, trained on a finite
sample. Anywhere the policy wanders outside the distribution the reward model
was trained on, its output is a guess.

And it is **frozen** during stage 3. The policy will be optimised against a
fixed, imperfect function for thousands of steps. Hold that thought.

## Why reinforcement learning at all?

Here is the honest version of the question: you have a scalar score for each
generated sequence. Why not just fine-tune on the high-scoring ones and be done?

You can, and that is roughly what rejection sampling or best-of-n fine-tuning
does. It works, and it is much simpler. RL earns its complexity for one reason:
the reward arrives **once, at the end of a whole sequence**, and it is not
differentiable with respect to the model's parameters.

Think about what the model actually does. At each step it picks a token from a
distribution over the vocabulary. Sampling is not differentiable — you cannot
backpropagate through "and then it chose token 4,712". The reward model scores
the finished text, so there is no per-token target to compute a cross-entropy
against.

Reinforcement learning is the toolkit for exactly this shape of problem:
discrete actions, non-differentiable feedback, credit that has to be assigned
backwards across a sequence of choices. In RL terms the mapping is:

- **state** — the prompt plus everything generated so far
- **action** — the next token
- **policy** — the language model itself
- **reward** — the reward model's score, delivered at the final token

## Policy gradients, and how they go wrong

The basic policy-gradient idea is intuitive. Increase the log-probability of
actions that led to high reward, decrease it for actions that led to low reward,
weighted by how much better than expected the outcome was:

```python
loss = -(log_prob * advantage).mean()
```

The `advantage` is the reward minus a baseline — how much better this action was
than what you would normally expect from that state. Subtracting a baseline does
not change what the gradient points at on average, but it enormously reduces its
variance, which is why every practical implementation has a value network
predicting that baseline.

The problem is step size. This gradient is only valid for the policy that
generated the data. Take one large step and the new policy is different enough
that the collected trajectories no longer describe it, and the next update is
computed from stale information. In a language model this failure is dramatic:
the model finds some degenerate high-reward region, collapses onto it, and starts
emitting the same sycophantic paragraph regardless of the prompt. Once it stops
producing varied samples it cannot recover, because it never explores anything
the reward model would score differently.

So you need small steps. But "use a small learning rate" is a blunt instrument —
the same learning rate is far too large in some regions and wastefully small in
others.

## PPO: clip the update, not the learning rate

PPO's insight is to constrain the update in terms of **how much the policy's
output distribution changed**, rather than how far the parameters moved.

Define the probability ratio between the new policy and the one that generated
the data:

```
ratio = pi_new(a | s) / pi_old(a | s)
```

A ratio of 1 means the policy has not changed for this token. Above 1 means the
new policy is more likely to produce that token. The clipped surrogate objective
is:

```python
ratio = torch.exp(new_log_probs - old_log_probs)   # exp of a difference of logs
unclipped = ratio * advantages
clipped = torch.clamp(ratio, 1 - eps, 1 + eps) * advantages
policy_loss = -torch.min(unclipped, clipped).mean()
```

with `eps` typically 0.2. Read the `min` carefully, because it is the entire
trick and it is not symmetric.

When the advantage is **positive** — this token was better than expected — the
objective wants to increase the ratio. Clipping caps it at `1 + eps`. Past that
point the gradient is zero: the model gets no further reward for pushing this
token's probability higher within this update. It has already moved enough.

When the advantage is **negative**, the objective wants to decrease the ratio,
and clipping floors it at `1 - eps`. Same logic in reverse.

Because it takes the minimum of the clipped and unclipped terms, PPO is
pessimistic. It never lets an update be *more* attractive because of clipping,
only less. And the clipping is one-directional in a useful way: if the ratio has
already blown far past the range in the wrong direction, the unclipped term is
selected and the gradient still pulls it back. Clipping stops you running away;
it does not stop you coming home.

The practical payoff is that you can take several gradient steps on the same
batch of generated data before it goes stale — normally you would be limited to
one — which matters enormously when generating that data means running inference
over a large model.

## The KL leash

Clipping bounds how far the policy moves **per update**. It says nothing about
where the policy ends up after ten thousand updates. Left alone, the model will
happily drift somewhere the reward model scores highly and humans find useless,
because the reward model is a proxy and every proxy has exploits.

This is not hypothetical. Optimise hard enough against a reward model and you
get outputs stuffed with the surface features it correlated with quality:
excessive hedging, bullet points everywhere, restating the question, and above
all **length**, because longer answers score better in most preference datasets
and the reward model learned that.

The fix is a KL penalty against the frozen SFT model, applied per token:

```python
# reference = the frozen SFT model; both are log-probs of the sampled token
kl = new_log_probs - ref_log_probs
rewards = reward_model_score - kl_coef * kl
```

The policy is now optimising the reward model's score *minus* a charge for every
token where it disagrees with the model it started from. The coefficient sets the
exchange rate: how much reward is a unit of drift worth. Tune it too high and
nothing changes; too low and the model wanders off and games the reward.

Two details worth knowing. The reward-model score is a single number at the end
of the sequence, whereas the KL term is charged at every token — the total
reward signal is dense in KL and sparse in actual preference. And in practice the
KL is often estimated with a lower-variance unbiased estimator rather than the
raw log-ratio difference, because the naive version is noisy enough to hurt.

## The loop, end to end

```python
for batch in prompts:
    # 1. generate, and record the log-probs of the policy that generated them
    responses, old_log_probs = policy.generate(batch)

    # 2. score
    scores = reward_model(batch, responses)             # one scalar per response
    ref_log_probs = reference_model.log_probs(batch, responses)
    kl = old_log_probs - ref_log_probs
    rewards = scores - kl_coef * kl                     # per token

    # 3. credit assignment
    values = value_head(batch, responses)
    advantages = compute_gae(rewards, values, gamma, lam)

    # 4. several epochs over the same batch, which is what clipping buys you
    for _ in range(ppo_epochs):
        new_log_probs = policy.log_probs(batch, responses)
        ratio = torch.exp(new_log_probs - old_log_probs)

        policy_loss = -torch.min(
            ratio * advantages,
            torch.clamp(ratio, 1 - eps, 1 + eps) * advantages,
        ).mean()

        value_loss = ((value_head(batch, responses) - returns) ** 2).mean()
        loss = policy_loss + vf_coef * value_loss

        loss.backward()
        optimizer.step()
```

That is four models in memory at once: the policy being trained, the frozen
reference, the reward model, and the value head. It is why RLHF is remembered as
expensive and fiddly as much as for what it achieves.

## What actually goes wrong

**Reward hacking** is the headline failure, and it is not a bug you can fix by
being careful. The reward model is a finite-sample approximation of human
preference; optimising a proxy hard enough will always find the gap between the
proxy and the thing it proxies for. Watching the reward curve go up tells you
nothing on its own — it goes up in exactly the same way whether the model is
improving or learning to exploit.

**Length bias** deserves its own name because it is so consistent. Annotators
mildly prefer thorough answers, the reward model learns "longer is better" as an
easy correlate, and the policy learns to pad. Some implementations subtract an
explicit length penalty; most just watch the mean token count.

**KL collapse in either direction.** Too much penalty and you have spent a great
deal of compute to reproduce the SFT model. Too little and the policy leaves the
distribution the reward model understands, where its scores are guesses, and it
optimises against noise.

**Value function instability.** The value head starts out knowing nothing, so
early advantages are mostly noise, which produces bad updates while the policy is
at its most malleable. Warming up the value head before letting the policy move
is a common remedy.

## Where this went next

The obvious complaint about the pipeline above is that it is enormous for what it
does. Train a reward model, then run an RL loop with four models resident,
carefully balanced against a KL term, to extract signal from data that is just
pairs of "this one is better".

Direct Preference Optimization is the well-known response. The derivation shows
that the optimal policy under a KL-constrained reward objective has a closed
form, which can be rearranged to express the *reward* in terms of the policy and
the reference model. Substitute that back into the Bradley-Terry loss and the
reward model cancels out. What remains is a supervised loss on preference pairs
directly:

```python
chosen_logratio = policy_chosen_logps - reference_chosen_logps
rejected_logratio = policy_rejected_logps - reference_rejected_logps
loss = -torch.nn.functional.logsigmoid(beta * (chosen_logratio - rejected_logratio)).mean()
```

No reward model, no sampling loop, no value head. Two models instead of four, and
an ordinary training loop.

It is not strictly better. DPO trains only on the fixed preference dataset, so it
never sees the policy's own current outputs — and the ability to generate, score,
and learn from what the model produces *now* is precisely what online RL gives
you. That is why the online variants exist, and why PPO has not disappeared.

Which is the useful way to hold all of this: PPO is not the point of RLHF. The
point is converting preferences into a training signal. PPO is one answer to the
question of how to follow that signal without falling over, and understanding
*why* it clips tells you more than memorising that it does.

## References

- [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155) — the InstructGPT paper, the canonical description of the three-stage pipeline
- [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347) — PPO itself, predating any of this by five years
- [Direct Preference Optimization](https://arxiv.org/abs/2305.18290) — the derivation that removes the reward model
- [Deep reinforcement learning from human preferences](https://arxiv.org/abs/1706.03741) — the earlier work that established learning a reward model from comparisons


A language model cannot be taught "this answer is better than that one" by supervised learning. RLHF turns that comparison into a gradient, and PPO is what keeps the gradient from destroying the model.

How RLHF Works, and Why It Uses PPO


Welcome to my blog 🥰

## What is Crop View ?

[Crop View](https://autode.sk/3pas1jq) help to display the elements inside the box that wrapped by the region.

## How to active Crop View in Revit ?

Let active two properties or the icon like image.

![Cropbox Property](/assets/images/blogs/cropview/cropbox-property.png 'Cropbox Property')

## So how to do multiple selection crop a view by RevitAPI ?

In RevitAPI, Crop View is definition by CropBox Property which it can be set by `BoundingBoxXYZ`

**So now let's started 🚀🚀🚀**

### Step 1:

The region which Crop View is limited to is enclosed by two points. In the current view, two points are located top-right and bottom-left.

Use the method `PickBox` which will return two points:

```csharp

// Selection Box by pick rectangular
 PickedBox pickedBox = uidocument.Selection.PickBox(PickBoxStyle.Directional, "Draw Rectangular Box, Press ESC to exit");
 XYZ pointBottomLeft = pickedBox.Min;
 XYZ pointTopRight = pickedBox.Max;
```

### Step 2:

The active view has a `Transform`. So we need modify the point to be the correct point with the transform of the view.

```csharp

//Get the invert transform of active view
Transform transform = view.get_BoundingBox(null).Transform.Inverse;
//Transform of the point
pointBottomLeft = transform.OfPoint(pointBottomLeft);
pointTopRight = transform.OfPoint(pointTopRight);

// Get exactly value min and max
double minX = Math.Min(pBox.Min.X, pBox.Max.X);
double maxX = Math.Max(pBox.Min.X, pBox.Max.X);

double minY = Math.Min(pBox.Min.Y, pBox.Max.Y);
double maxY = Math.Max(pBox.Min.Y, pBox.Max.Y);
```

### Step 3:

Declared new `BoundingBoxXYZ` and apply for `CropBox`.

```csharp
//Create new instance of boundingBox
BoundingBoxXYZ boundingBoxXYZ = new BoundingBoxXYZ();
boundingBoxXYZ.Min = new XYZ(minX, minX, 0);
boundingBoxXYZ.Max = new XYZ(maxX, maxY, 0);
boundingBoxXYZ.Transform = view.CropBox.Transform;

//Active Crop View and aplly the new boundingBox for Cropbox
view.CropBoxActive = true;
view.CropBoxVisible = true;
view.CropBox = boundingBoxXYZ;
doc.Regenerate();
```

### Step 4:

Add the `while-loop` for the code, so you can multiple set Crop View. Remember using `Transaction`.

```csharp
PickedBox pickedBox = null;
while (true)
{
    try
    {
        pickedBox = uiDocument.Selection.PickBox(PickBoxStyle.Directional, "Draw Rectangular, Press ESC to exit");
    }
    catch
    {
        break;
    }

    if (pickedBox != null)
    {
        using (Transaction transaction = new Transaction(document))
        {
            transaction.Start("Crop view by selection");
            // code block
            transaction.Commit();
        }
    }
}
```

## Conclusion

Yeah 🎯, So we finished the small tool can help create quickly Crop View.

## Reference

You can get full source code and another tool at [RevitAPI-Learning](https://github.com/LDATuan/RevitAPI-Learning)


The small tool will help create quickly the crop the view in any view

I write about Revit, C#, and teaching software to do the boring parts.

How RLHF Works, and Why It Uses PPO

How To Crop View With RevitAPI