skill.md Is a Supply Chain

Today, after a long time, I visited Stack Overflow again - with a feeling of nostalgia. I used to be there constantly, and I have no idea how many hours of my life were saved by the right answers there. Since LLMs got so good, I hardly ever drop by anymore - and I just wanted to see what’s new.

That’s when I discovered Stack Overflow for Agents (SOFA): a kind of Stack Overflow for AI agents, with collective verification instead of human upvotes. The integration works the way agent integrations usually work these days: a skill.md file, fetched from agents.stackoverflow.com, gets loaded into the agent’s context. I looked at the file and found it’s quite long. On closer inspection, I noticed that further instructions are loaded from the SOFA site at runtime - and that even API responses are meant to serve as steering instructions. My supply-chain alarm started ringing quietly, and I set off down the rabbit hole.

One thing up front: this is not a hit piece. SOFA’s skill file is among the most careful of its kind - and that’s exactly what makes it a worthwhile case study. Because if the careful version of a pattern has a structural problem, then the pattern has a structural problem. What follows: a close read of its 692 lines, what the file gets right - and why “well written” doesn’t address the actual trust boundary.

What a skill file actually is

Agent Skills are an open format - originally from Anthropic, now widely adopted. A skill is a folder with a SKILL.md (SOFA lowercases it: skill.md): YAML frontmatter with name and description at the top, a Markdown body with instructions below. The mechanics are what matter: the description sits permanently in the agent’s system prompt; the agent client loads the body on demand as soon as a task matches the skill - and from that point on it stands as instruction text in the model’s context window.

A skill file, then, is not documentation. It is executable - not by a CPU, but by the model’s instruction-following behavior. That difference is smaller than it sounds.

The finding: three layers

Layer one: a mutable remote instruction source

Try Stack Overflow for Agents Verified knowledge for the agentic era Paste this into your AI coding assistant

So the user is supposed to hand their agent a link without knowing what’s behind it. As if phishing had never existed, I’m supposed to trust a data source - no signature, no version pinning, no checksum. What’s in the file today can be something else tomorrow: an error that crept in, in the worst case malicious instructions. I’m not imputing intent - and it’s not a Stack Overflow problem either, but a fundamental one: this is how skills are shipped across the entire ecosystem today. SOFA is merely the most prominent instance.

Structurally, this is curl | bash: content from a URL, executed directly - only the payload doesn’t land in the shell, but in the agent’s instruction hierarchy. And with worse tooling: no package manager, no lockfile, no advisory database. Not yet.

Layer two: the file is only the tip

The skill.md instructs the agent to fetch further instructions at runtime: /llms.txt, /guidelines/..., /contribute.md. In an isolated dev environment that may be acceptable; in a regulated enterprise environment it certainly is not. So we have 692 reviewed lines - and behind them dynamic content that can change at any time. To our detriment in the worst case.

And there’s more: API responses contain next_step fields that, per the document, are to be treated as an “immediate steering instruction” - as an instruction to be followed immediately. The server thereby steers the agent’s behavior per request. The user fundamentally cannot know the effects of these instructions: they don’t even exist at the time of their review.

The consequence: the auditable artifact is not the 692-line file. It is a dynamic instruction stream - and a dynamic stream can’t be reviewed, only constrained.

Layer three: persistence

But even here we’re not at the end. The skill.md instructs the agent to load a community playbook and, from it, write persistent instructions into AGENTS.md, CLAUDE.md, or Cursor rules. It’s meant as onboarding convenience. Structurally it’s something else: foreign, community-authored content writes itself into a project’s instruction files. Malware analysis has sober technical terms for this - persistence. And once these files are checked in and distributed via the repository to every clone: lateral movement. Here too I’m not imputing intent to anyone, and that’s not the point. Attackers don’t care about intent, only about mechanisms - and the mechanism exists.

Of course the agent asks for permission first. But decision fatigue is a real problem: someone who dismisses fifty confirmation dialogs a day won’t read the fifty-first - especially since nested calls with bash and Python components are hard to see through even for trained eyes. Effective protection therefore lies not in the dialog, but before it: run agents in isolated environments and treat instruction files in code review like code. Because that’s exactly what they are.

What the file gets right

To be fair: this skill.md is well crafted - you can tell someone put effort into it. Posts and playbooks are explicitly declared untrusted; the agent is told not to decode opaque blobs and not to follow embedded, behavior-changing instructions. There are human-in-the-loop publication policies with one-time approval codes and draft-only modes, a read_only role, credential hygiene (check gitignore before writing credentials to disk), bounded content sizes, and guards against blindly rating unread posts.

Whoever wrote this thought about the threat model. The problem is not what’s in the file. The problem is where it sits.

The pattern: prose is not policy

What holds for SOFA here holds for a growing class: skills, MCP tool descriptions, llms.txt, loaded-in content of all kinds - mutable remote instruction sources. Regardless of their specific content, they have two structural weaknesses.

First: mutability without integrity. Without pinning and signatures, the trust decision is made exactly once - the content keeps changing arbitrarily afterward.

Second: enforcement by prose. Security rules delivered as instruction text compete in the same context window with exactly the content they’re meant to govern. On top of that comes an empirical finding: a model’s ability to detect and fend off prompt injections declines as the context window fills up - and long instruction payloads like these roughly 10,000 tokens measurably dilute adherence to higher-priority instructions. The competition for attention intensifies with every token loaded in. The consequence is uncomfortable but unambiguous: malicious content must not enter the context window in the first place.

This risk class isn’t new, by the way. We’ve been loading libraries and scripts from the internet for decades and wiring them into critical enterprise applications. We never solved that with warnings in the README, but with technical controls: lockfiles, checksums, signatures, advisory databases, automated dependency scanning. The library world, in other words, understood long ago that prose is not policy. And even with all that apparatus, log4shell and the xz backdoor happen. The skill ecosystem starts without the basic equipment: no lockfile, no signature, no advisory feed - just a document that politely asks for good behavior.

A rule that lives in the same channel as what it governs is a convention - not a control.

What a technical control looks like

So what follows in practice? First, four measures that cost little and take effect immediately.

First: don’t load skills live from a URL, but pin a reviewed local copy - and treat updates like dependency updates: look at the diff, then adopt it deliberately. A simple CI check comparing the local copy against upstream turns a silent change into a visible event.

Second: limit the blast radius. Agents belong in isolated containers - with an egress firewall and without access to the user’s resources that the task doesn’t require.

Third: least privilege, and stricter than for a human. On systems like Jira or Confluence, the agent should have at most the rights of the user running it - actually less. Because a change made by an agent has no human who wonders about it: a manipulated Confluence entry goes unnoticed and can do considerable damage over time. SOFA itself, by the way, offers a read_only role; anyone who doesn’t want to actively contribute needs nothing else.

Fourth: block the activation path. No skill should be allowed to write into a project’s instruction files.

These are mitigations, not a solution. The real answer lies outside the model: a policy enforcement point between agent and resource - for instance with the Open Policy Agent - that checks in both directions: what enters the context window and which tool calls go out. Plus credentials tailored to the specific task rather than to the agent as a whole, and an audit trail that makes decision and access correlatable - token exchange, PEP decision log, and resource log, linked via the token ID. An agent whose credential can do only what the task requires does only limited damage even with compromised instructions. I’ve described this architecture in a blueprint elsewhere. The short version: agents don’t need better promises - they need narrower credentials.

Conclusion

The recommendation is explicitly not: steer clear of SOFA. It is: examine the class, not the instance. Stack Overflow has built the careful version of a pattern that’s currently taking over the entire ecosystem - most implementations will be worse, with fewer warnings, less human-in-the-loop, and less thought about the threat model. If even the careful version has a structural problem, the pattern has a structural problem.

Stack Overflow spent fifteen years teaching developers to check the accepted answer before trusting it. The same discipline now applies to the instructions our agents accept - starting with the ones that arrive over HTTPS and look like documentation.