skill-creator is a meta-skill: invoke it and Claude scaffolds a new skill for you — frontmatter, structure, examples. But the part worth copying is what happens next. It generates test prompts that should and shouldn't trigger the new skill, runs them with and without the skill loaded, and scores the difference. Skill quality stops being a matter of taste and starts being a number. Here's the full loop, the file formats, and the commands to run it yourself.
There is a particular kind of file that has quietly become the unit of capability in the agentic ecosystem: SKILL.md. A folder, a markdown file, some YAML at the top. When Claude decides the file is relevant to what you asked, it reads it, and suddenly the agent knows how to fill a PDF form, reconcile a spreadsheet, or follow your team's deployment runbook. The whole mechanism is almost aggressively simple.
Which is exactly why most skills are written badly. Simple to start means nothing stops you from shipping a SKILL.md with a vague description, an eight-paragraph wall of instructions, and no idea whether Claude will actually load it when it matters. The skill looks fine. It sits in the folder. And then in the one conversation where it should have fired, it doesn't — because the description didn't match how the request was phrased — and you never find out.
skill-creator is Anthropic's answer to that gap, and the framing is a little on the nose: it's a skill whose job is to create other skills. Invoke it, and Claude interviews you, scaffolds the directory, and writes the SKILL.md. That part is convenient. But it's not the part worth studying. The part worth studying is that skill-creator treats "is this skill any good?" as a question with a measurable answer — and builds the harness to measure it.
!A skill file being scaffolded, tested, and scored in a loop
What a skill actually is, in three tiers
Before the meta part makes sense, the object it operates on has to be clear. A skill is a directory:
``
pdf-filler/
├── SKILL.md # required — frontmatter + instructions
├── scripts/ # executable code for deterministic steps
├── references/ # docs Claude loads only when needed
└── assets/ # templates, fonts, icons used in output
`
The SKILL.md starts with YAML frontmatter, and only two fields are required:
`yaml
name: pdf-filler
description: Fill and flatten PDF forms from structured field data. Use whenever the user wants to complete a PDF form, populate form fields, or generate a filled PDF from a template.
`
What makes this work is progressive disclosure — the design principle underneath the whole system. Claude does not load your skill into context up front. It loads it in three tiers:
Metadata always. The name and description — roughly a hundred words — sit in the system prompt from the start. This is all Claude sees until it decides the skill is relevant.
Body when triggered. The instructions in SKILL.md load only once Claude matches the current task to that description. The guidance is to keep this under 500 lines.
Bundled files on demand. Anything in references/ or scripts/ loads only when the instructions point Claude to it.
That first tier is the entire ballgame. The description field is not documentation — it is the trigger. It is the single string Claude matches your request against to decide whether the skill exists for this conversation at all. A skill with perfect instructions and a mediocre description is a skill that never runs. And "mediocre description" is invisible to the naked eye. It reads fine. It just doesn't fire.
This is the failure mode skill-creator is built to catch.
The meta part: an interview that ends in a folder
The first half of skill-creator is the convenient half. It runs a short sequence:
Capture intent — what should the skill do, when should it trigger, what does its output look like, and do test cases make sense here.
Interview and research — edge cases, input and output formats, example files, what "success" means.
Write SKILL.md — frontmatter plus imperative-form instructions, with any deterministic steps pushed into scripts/ and any long reference material split into references/ with a table of contents once a file passes ~300 lines.
If you have ever hand-written a skill, none of this is surprising — it is the discipline you were supposed to apply anyway, enforced by a checklist instead of by memory. The scaffolding is genuinely useful, but it is not the reason to pay attention. Plenty of tools generate boilerplate. Very few generate the test suite that tells you whether the boilerplate works.
The interesting part: it grades itself
Here is where skill-creator stops being a generator and becomes a loop.
After the SKILL.md exists, the skill develops test cases — two or three realistic prompts a real user might send, the kind that should pull this skill in. They land in an evals/evals.json file:
`json
{
"skill_name": "pdf-filler",
"evals": [
{
"id": 1,
"prompt": "Here's a blank W-9 and a JSON file with the values. Fill it in.",
"expected_output": "A completed, flattened PDF with every field populated from the JSON.",
"files": ["w9-blank.pdf", "values.json"]
}
]
}
`
Then it runs them — and this is the move worth stealing. For each test case, it spawns two subagents in the same turn: one with the skill loaded, one without. Same prompt, same files, same model. The only variable is the skill.
That A/B structure is the whole point. A skill that produces a good output is not interesting on its own — the base model might have produced a good output anyway. What you want to know is the delta: does loading this skill make the answer meaningfully better than not loading it? If the with-skill and without-skill outputs are indistinguishable, the skill is not earning its place in the context window, no matter how polished the SKILL.md reads.
The outputs get organized so the comparison is legible:
`
pdf-filler-workspace/
└── iteration-1/
└── eval-1/
├── with_skill/outputs/
└── without_skill/outputs/
`
Each run also captures grading and timing. Grading is deliberately structured — the review viewer requires three fields per assertion, text, passed, and evidence, so a verdict is never just a thumbs-up; it carries the evidence that justifies it:
`json
{ "text": "PDF has all 12 fields filled", "passed": true, "evidence": "Fields 1-12 populated; output flattened." }
`
And timing is recorded per run, because a skill that improves quality while tripling token cost is a trade-off you should get to see, not one that hides:
`json
{ "total_tokens": 84852, "duration_ms": 23332, "total_duration_seconds": 23.3 }
`
The result is that "is this skill good?" resolves to something you can look at side by side: here is what the agent did with it, here is what it did without it, here is the token cost of the difference. Skill quality stops being a vibe and becomes a diff.
!Two agent runs side by side — one with the skill, one without
Tuning the trigger, not just the body
The second loop is the one I find most quietly clever, because it optimizes the part humans are worst at judging: the description.
Remember that the description is the trigger, and its quality is invisible from reading it. skill-creator's answer is to stop reading it and start testing it. The optional final stage generates around twenty trigger queries — realistic user prompts, split roughly evenly between ones that should pull the skill in and ones that should not. Then it evaluates the current description against all twenty: does it fire when it should, and stay quiet when it shouldn't?
A skill that triggers too eagerly is as broken as one that never triggers — it burns context and hijacks conversations that had nothing to do with it. Testing both directions is what separates a description that works from one that merely sounds thorough.
From there it runs a small optimization loop — propose a better description, re-score, repeat, up to five iterations — and it picks the winner by test score rather than train score, to avoid overfitting the description to the exact queries it was tuned on. That is a real eval instinct baked into a skill-authoring tool: the fact that the phrasing scored well on the prompts you generated does not mean it will generalize, so hold some out and judge on those.
Try It Yourself
You do not need the full harness to adopt the idea. Here is a concrete path from nothing to a tested, packaged skill.
Get the skill and point Claude at it. The official skills live in one repo:
`bash
git clone https://github.com/anthropics/skills.git
In Claude Code, make the skill-creator directory available and then just ask:
"Use skill-creator to help me build a skill for
`
Start every new skill from a minimal, honest SKILL.md. Copy this and fill it in. Resist the urge to pad the description — write what it does and when to use it, in the words a real request would use:
`markdown
name: changelog-writer
description: Generate a release changelog from merged pull requests. Use whenever the user asks for a changelog, release notes, or a summary of what shipped between two tags.
Changelog Writer
When to use
The user wants human-readable release notes from git history or merged PRs.
Steps
Collect merged PRs between the two refs the user names.
Group them into Added / Changed / Fixed / Removed.
Write one plain-language line per entry — no PR numbers in the summary line.
Output
Markdown, newest version first, one ## heading per release.
`
Write the evals before you polish the instructions. Two or three real prompts, saved to evals/evals.json:
`json
{
"skill_name": "changelog-writer",
"evals": [
{ "id": 1, "prompt": "Write release notes for everything merged since v2.3.0.", "expected_output": "Grouped, plain-language changelog, newest first.", "files": [] },
{ "id": 2, "prompt": "What changed between the last two releases?", "expected_output": "A changelog — the phrasing is indirect but the intent is release notes.", "files": [] }
]
}
`
Eval 2 is doing real work: it is phrased nothing like "write a changelog," and it is there to check that your description is broad enough to catch intent, not just keywords.
Run the A/B yourself, even by hand. The principle survives without any framework. In two separate sessions, send the same eval prompt — one with the skill available, one without — and compare the outputs. If you cannot tell which run had the skill, the skill is not pulling its weight yet. Fix the instructions or cut the skill.
Optimize the description as a test, not an edit. If you have the skill-creator scripts checked out, its own loop will do this for you:
`bash
python -m scripts.run_loop \
--eval-set trigger-evals.json \
--skill-path ./changelog-writer \
--model claude-opus-4-8 \
--max-iterations 5 \
--verbose
`
Feed it eight-to-ten should-trigger prompts and eight-to-ten should-not, and let it settle on a description by held-out score.
Package it when it earns its place.
`bash
python -m scripts.package_skill ./changelog-writer
``
That produces a distributable bundle you can drop into a project, share with a team, or publish as a plugin.
!A skill graded against its own control group before shipping
Why this is the part that matters
The industry spent the last stretch treating skills as a distribution problem — how do we let people share these markdown folders, how do we install them, how do we compose them. All real questions. But distribution assumes the thing being distributed is good, and the honest state of most skill libraries is that nobody knows. They were written, they read plausibly, they were committed.
What skill-creator quietly asserts is that a skill is a testable artifact. It has a job — trigger correctly, and change the output for the better when it does — and both halves of that job are measurable. The A/B against the base model measures whether the body earns its context cost. The trigger evals measure whether the description fires at the right times. Neither measurement requires taste. Both produce a number you can watch move across iterations.
That is the shift under the joke about a skill that writes skills. The writing was never the hard part. Knowing whether what you wrote actually works — and being able to prove it before you ship it to a teammate — is the part that has been missing. The most useful thing to copy here is not the generator. It is the habit of never trusting a skill you haven't watched lose to its own control group.