Writing Agents That Work: opening a field guide from skills to systems

Series: Writing Agents That Work — a field guide from skills to systems.

A skill is a folder with a markdown file in it. A multi-agent system is “just spin up more agents.” A feedback loop is “let it retry until the tests pass.” Every one of those sentences is true, and every one of them is where most writing on the subject stops — right at the point where it starts being hard.

This series goes past that point. It is a field guide for engineers building agentic systems, in four arcs — the SKILL.md package format Anthropic released and then opened as a standard, then multi-agent and subagent architecture, then the feedback loops that decide whether any of it actually converges, and finally the topologies: how all of it maps onto the shape of your actual system. The goal is not designs that read well and do nothing; it’s systems that reliably fire and measurably help.

We don’t start with a template or a pip install. We start with the one question most agent content skips: which of these claims is established, and which is someone selling their own tool?

The guiding principle: source discipline

The single rule that runs through every post: separate what the primary documentation and peer-reviewed research establish from what vendors self-report. Every claim carries a tag for its source strength, and the tag travels with the number wherever it goes:

• 🟢 primary — official first-party docs, or peer-reviewed research.
• 🟡 vendor self-report — first-party numbers from internal evals, not independently verified.
• 🟠 preprint or secondary — arXiv not yet peer-reviewed, or third-party / community write-ups.

Trade-off: this makes the series slower to read and less quotable. A clean “X works, so do X” is more satisfying than “X is claimed by its vendor’s internal eval (🟡) and contradicted by one peer-reviewed benchmark (🟢).” We pay that tax on purpose, because the alternative — laundering a self-reported number into a fact by dropping the tag — is exactly how this field fills with confident advice that doesn’t survive contact with a real workload.

The discipline isn’t pedantry; it’s the only honest response to how counterintuitive the actual evidence is:

• Curated, human-authored skills lifted task performance by +16.6 points; skills the agent wrote for itself scored below the no-skill baseline (−8 to −11.5 pp) (SkillsBench, arXiv:2602.12670 🟠 — preprint, not peer-reviewed). “Write a skill” and “write a skill that helps” are different problems.
• Multi-agent is sold as an upgrade. A peer-reviewed benchmark found ~5× the token and latency cost for no accuracy gain (Frontiers in Robotics and AI 2026 🟢), against Anthropic’s +90.2% figure from an internal eval (Anthropic Engineering 🟡). Same topic, two very different source strengths — and the tags are the whole story.
• When multi-agent systems fail, the causes aren’t bad luck: 14 failure modes across 3 categories, gains “often minimal,” failures “rooted in system design” (MAST, arXiv:2503.13657 🟢, NeurIPS 2025).

The four arcs

1. Skills (#1–#5) — what a SKILL.md actually is, the three-level loading model and why description is the only trigger, testing skills like prompts, what the research says about when they help, and the pre-ship checklist plus the anti-patterns that fail review.
2. Subagents & orchestration (#6–#8) — the decision gate and the economics of a second agent, the orchestration patterns that port everywhere (and the planner/coder/tester role-split trap that doesn’t), and why multi-agent systems fail — read the skeptics first.
3. Feedback loops (#9–#11) — why an agent needs an external signal (intrinsic self-correction fails), how to design the layered loop with deterministic exit criteria, and how the signal gets gamed (reward hacking) — and defended.
4. Topologies (#12–#13) — mapping it all onto concrete system shapes: how to slice agents across microservices, an SDLC pipeline, small apps and business monoliths — and what the skill portfolio for each landscape looks like.

The series is self-contained: it depends on no private project. The working skill it culminates in — creating-skills — ships alongside it.

Who this is for

Engineers building agentic systems who want the version with the caveats left in — where a preprint is labelled a preprint and a vendor number is labelled a vendor number, so you can tell the difference between “the docs guarantee this” and “one team reported this once.” If you’ve ever shipped a skill that never fired, or added a second agent and paid 15× the tokens for nothing (the multiplier is Anthropic’s own figure for multi-agent vs chat — Anthropic Engineering 🟡), this guide is aimed at the reasons why.

What’s next

We start at the smallest unit. Post #1 takes apart the SKILL.md format — the two frontmatter fields that decide whether a skill ever fires, the directory layout, skill vs MCP, and how a Claude feature became an open standard.

Bibliography

• SkillsBench — arXiv:2602.12670 🟠 — +16.6 pp from curated human skills; self-generated skills below baseline (−8 to −11.5 pp) (preprint, not peer-reviewed)
• Multi-agent orchestration benchmark — Frontiers in Robotics and AI 2026 🟢 — peer-reviewed null result: ~5× cost/latency, no accuracy gain
• Multi-agent research system — Anthropic Engineering 🟡 — the +90.2% internal-eval figure and “not a good fit” cases
• Why Do Multi-Agent LLM Systems Fail? (MAST) — arXiv:2503.13657 🟢 — NeurIPS 2025; 14 failure modes, gains often minimal, failures rooted in system design