Convert evaluation traces and production logs into SFT examples and preference pairs. Use when graded traces or failure examples exist and need to become training data, when applying rejection sampling to model outputs, or when building DPO pairs from passing and failing runs.
Installs just this skill. Get the whole plugin for auto-invocation.
โก How it fires
How this skill gets triggered: by you, by Claude, or both.
Fires itselfClaude auto-loads it when your prompt matches the work.
You can call itInvoke it directly when you want it.
Slash command/trace-to-training-data
๐๏ธ Context preview
The summary Claude sees to decide when to auto-load this skill.
Convert evaluation traces and production logs into SFT examples and preference pairs. Use when graded traces or failure examples exist and need to become training data, when applying rejection sampling to model outputs, or when building DPO pairs from passing and failing runs.
๐ Stats
Stars38,171
Forks4,092
LanguagePython
LicenseMIT
๐ฆ Ships with wshobson-agents
</> SKILL.md
trace-to-training-data.SKILL.md
---name: trace-to-training-data
description: Convert evaluation traces and production logs into SFT examples and preference pairs. Use when graded traces or failure examples exist and need to become training data, when applying rejection sampling to model outputs, or when building DPO pairs from passing and failing runs.
---# Trace To Training Data
This skill assumes `eval-harness-first`
already graded the traces being
converted here โ goldens, graders,
and `runs/<run-id>/results.json`
all exist before conversion
starts. This is the flywheel edge
that skill names in its own flow:
"the same labeled traces become
the training set." Conversion
happens here; grading already
happened upstream.
**Input:** graded traces โ
`eval/goldens.jsonl` plus
`runs/<run-id>/results.json`, each
row carrying a `task_id`, a
`verdict` from the grader, and a
`reward` when the task supports a
scalar score (judge score,
execution partial-credit, or an
RLVR verifier):
```json
{"task_id": "t-042", "trace_id": "t-042-a3",
"messages": [{"role": "user", "content": "..."}],
"verdict": "pass", "reward": 0.91,
