Train reasoning and verifiable-task behavior with GRPO and reinforcement learning from verifiable rewards (RLVR). Use when task success is algorithmically checkable (math, code, tool calls, structured output), when designing GRPO reward functions, or when a GRPO run diverges or
Installs just this skill. Get the whole plugin for auto-invocation.
⚡ How it fires
How this skill gets triggered: by you, by Claude, or both.
Fires itselfClaude auto-loads it when your prompt matches the work.
You can call itInvoke it directly when you want it.
Slash command/grpo-rlvr-training
👁️ Context preview
The summary Claude sees to decide when to auto-load this skill.
Train reasoning and verifiable-task behavior with GRPO and reinforcement learning from verifiable rewards (RLVR). Use when task success is algorithmically checkable (math, code, tool calls, structured output), when designing GRPO reward functions, or when a GRPO run diverges or
📊 Stats
Stars38,174
Forks4,092
LanguagePython
LicenseMIT
📦 Ships with wshobson-agents
</> SKILL.md
grpo-rlvr-training.SKILL.md
---name: grpo-rlvr-training
description: Train reasoning and verifiable-task behavior with GRPO and reinforcement learning from verifiable rewards (RLVR). Use when task success is algorithmically checkable (math, code, tool calls, structured output), when designing GRPO reward functions, or when a GRPO run diverges or reward-hacks.
---# GRPO & RLVR Training
This skill assumes `finetuning-method-selection`
already routed here because the target behavior
has a verifiable pass/fail signal — not
demonstrations (`lora-qlora-recipes`) or
preference pairs (`preference-optimization`).
What follows is when RL is the right tool, the
reference recipe, the mandatory reward-inspection
gate, and how to pick a GRPO variant when the
base recipe misbehaves.
**Input:** a routing decision (RLVR via GRPO)
plus a verifier (code executor, test suite,
schema checker, or grader) for the target task.
**Output format:** a validated GRPO config — the
kwarg values in `references/grpo-memory.md` and
the reward functions in
`references/reward-functions.md`, not free-form
advice — that `llm-finetuning-training-engineer`
consumes directly.
## When RL Applies
GRPO+RLVR only pays off when task success is
train_dataset=prompts, # prompt-only — GRPO generates its own completions
processing_class=tokenizer,
)
trainer.train()
```
- **`vllm_mode="colocate"`** runs generation and
training on the same GPU — the default for a
single-GPU box.
- **`vllm_mode="server"`** points at a separate
vLLM server process and is the multi-GPU path —
generation and training don't compete for the
same device.
- **`num_generations` ≥ 8** is a floor, not a
suggestion: GRPO's advantage estimate is
relative to the group mean, and fewer than 8
samples per prompt produces a noisy baseline.
- **Reward is composite** — a format reward (did
the output parse / match the required
structure) plus a correctness reward (did the
answer verify). A well-formed-but-wrong answer
and a malformed one should not score
identically; correctness alone loses that
signal.
- **`learning_rate=5e-7`** and **`beta=0.01`** are
the settled starting point; deviate only after
the base run is stable and reward-inspected
(below).
Memory sizing for this recipe by target size
class: `references/grpo-memory.md`.
## The Inspection Rule
**Run the reward function against 50–100 sampled
outputs and manually read the results before
starting the actual training run.** This is a
gate, not a one-time sanity check.
If the reward function's judgment disagrees with
a human reading of that sample, fix the reward
function first. Training against an uninspected
reward, or tuning hyperparameters to compensate
for one silently scoring the wrong thing, is how
a run reward-hacks: the model optimizes cleanly
toward the wrong target, and that doesn't surface
as a training-loop bug.
This inspection is a Phase 1 gate input for
`/finetune` — the same 50–100-sample read that
catches a broken reward function here is what that
command checks for before it lets a GRPO brief
proceed.
Complete reward function implementations to
inspect against — exact-match, schema-validation,
unit-test-execution, a length-penalty wrapper, and
a rubric-as-reward judge pattern:
`references/reward-functions.md`.
## Variant Selection
The base recipe above is the default. Reach for a
variant only when a specific failure mode shows
up, not preemptively:
| Failure mode | Variant | Why |
|---|---|---|
| Entropy collapse / degenerate long chain-of-thought | **DAPO** | Decouples clip bounds and relaxes the KL penalty that over-regularizes exploration on long reasoning traces |
| Reward or output length trends up regardless of quality | **Dr.GRPO** | Removes GRPO's length-normalization bias so reward tracks correctness, not completion length |
| Training a mixture-of-experts model | **GSPO** | Moves the importance-sampling ratio to the sequence level instead of per-token — per-token ratios are unstable on MoE routing, so GSPO is required here, not optional |
Start with plain GRPO. Watch for the specific
symptom — collapsing entropy on long CoT, a
length-reward correlation, or MoE instability —
and only then swap in the matching variant above.
Don't pre-select a variant before the base recipe
has actually shown the failure mode.
## VLM RL Is Reference-Only
Vision-language RL is **not executed by this
plugin in v1** — it's documented here for
context, not as a runnable path. Tooling is
fragmented across ms-swift and EasyR1-derived
forks with no one-line TRL command yet, and naive
text-only GRPO applied to a VLM tends to
reward-hack by optimizing the text-reasoning trace
