TL;DR
Block's security team red-teamed its in-house AI agent, Goose, and used a prompt injection attack hidden in invisible Unicode to deliver an information-stealing payload to an employee laptop. Block has patched Goose with warnings, Unicode detection and is testing adversarial AI checks, but prompt injection remains a known risk.
What happened
Block's chief information security officer, James Nettesheim, told The Register that the company carried out offensive testing against its open source AI agent, Goose. During a red-team exercise the team combined phishing against a developer with a prompt injection payload concealed using invisible Unicode characters in a shared Goose "recipe." When the developer opened the poisoned recipe while debugging, the workflow downloaded and executed an infostealer on the developer's laptop. The exercise demonstrated that reusable, shareable agent workflows can be weaponized if their contents are obscured. In response, Block added a recipe-install warning, desktop alerts for suspicious Unicode, and routines to detect and strip invisible Unicode from strings. The company is also experimenting with adversarial AI—using one model or agent to validate another—to flag malicious inputs, though those checks are still under internal testing.
Why it matters
- Prompt injection can turn seemingly benign agent workflows into attack vectors, putting employee endpoints and credentials at risk.
- Shared, portable workflows raise supply-chain‑style risks for agent ecosystems that weren't present with single-use prompts.
- Least-privilege controls for agents are critical; agents with broad access can amplify damage if compromised.
- Remediations such as UI warnings and Unicode detection reduce risk but do not eliminate the underlying attack class.
Key facts
- Goose is Block's open source AI agent and has been public for about a year.
- Almost all of Block's roughly 12,000 employees use Goose, which can connect to company systems including Google accounts and Square payments.
- Block's red team successfully used prompt injection hidden in invisible Unicode characters combined with phishing to deliver an infostealer to a developer's laptop.
- Goose uses "recipes"—shareable, reusable workflows that the red team found could be poisoned.
- Block implemented a recipe-install warning to alert users before executing recipes from untrusted sources.
- The company added desktop alerts for suspicious Unicode and mechanisms to detect and remove invisible Unicode characters from strings.
- Block is testing adversarial AI techniques—having one model or agent vet another—to detect malicious prompts and outputs.
- Nettesheim compared agent safety needs to self-driving cars: agents must be safer and provably better than humans.
What to watch next
- Whether the adversarial-AI input/output checks under internal test will be merged into the open source Goose: not confirmed in the source.
- If attackers reproduce the recipe-poisoning technique in the wild outside of controlled red-team tests: not confirmed in the source.
- Further updates to Goose's protections and how they balance detection speed and false alert volume.
Quick glossary
- Prompt injection: A technique that inserts malicious instructions into inputs processed by an AI model or agent so the model performs unintended actions.
- AI agent: A software component that uses models and workflows to carry out tasks autonomously or semi-autonomously on behalf of users.
- Least privilege: A security principle that grants systems or users only the access necessary to perform their function, and no more.
- Infostealer: Malware designed to harvest sensitive information from a compromised device, such as credentials or personal data.
- Adversarial AI: Techniques that use models to attack or evaluate other models, often applied to harden systems by exposing weaknesses.
Reader FAQ
Did the red-team attack succeed?
Yes. The red team used prompt injection hidden in invisible Unicode and a phishing lure to cause an infostealer to run on a developer's laptop.
Was customer data compromised in the test?
not confirmed in the source
Is Goose open source and widely used at Block?
Yes. Block open sourced Goose about a year ago and the agent is used by almost all of the company's roughly 12,000 employees.
Is adversarial AI already protecting Goose in production?
Block is testing adversarial-AI checks internally but has not yet merged them into the open source Goose as it works on speed and alert-quality issues.
SECURITY Block CISO: We red-teamed our own AI agent to run an infostealer on an employee laptop Agents must be 'safer and better than humans,' James Nettesheim tells The Reg…
Sources
- Block CISO: We red-teamed our own AI agent to run an infostealer on an employee laptop
- Block red-teamed its own AI agent to run an infostealer
- SANS Internet Stormcenter Daily Cyber Security Podcast …
Related posts
- Google plans in-search purchases using agentic AI and Universal Commerce Protocol
- Amazon automatically upgrades Prime members’ devices to Alexa Plus
- TimeCapsuleLLM: Language model trained only on 1800–1875 London texts