You Can't Tell What an Agent Skill Will Do by Reading It - Roey Ben Chaim, Zenity

An official diagnostic skill can read cloud credentials, contact identity services and probe an instance metadata endpoint. Those same actions can resemble an information stealer. Roey Ben Chaim of Zenity and the MATS Fellowship uses that ambiguity to explain why reading a SKILL.md file cannot establish what an agent will actually do. Instructions can fetch more instructions, installation scripts can pull in packages, and those packages can create processes whose effects never appear in the agent's own trace. The missing evidence is runtime behavior, collected inside a controlled environment. Roey walks through the evolution of an agent detonation chamber: a sandbox, a world builder that supplies the services a skill expects, an operator that exercises different paths, and instrumentation for processes, filesystem activity and network traffic. He then follows concrete cases involving chained skill installation, a misleading package name, persistence, hooks and remotely fetched prompts. One investigation traces a lookalike project through a downloaded payload to credential harvesting, showing why the complete chain matters more than the entry file. The examples illustrate both malicious behavior and legitimate activity that can trigger suspicion; they are not a claim that one scanner can classify every skill correctly. The engineering takeaway is to treat skills as software dependencies, inspect what they load, restrict their access, and preserve evidence of what executes beyond the model's visible conversation. Speaker info: Roey Ben Chaim, Zenity and MATS Fellowship - https://www.roeybc.com/about - https://www.linkedin.com/in/roeybc - https://labs.zenity.io/ Timestamps: 0:00 - When legitimate diagnostics resemble an attack 1:37 - Why scanning SKILL.md is insufficient 2:48 - Building the agent detonation chamber 3:48 - Environments, task coverage and execution 6:03 - Capturing processes, files and network activity 7:08 - Chained skills and misleading packages 8:48 - Persistence, remote prompts and hook abuse 10:35 - Investigating a credential harvesting campaign 13:33 - Treat skills as supply chain dependencies