Researcher poisons open-weight AI model for under $100

A researcher demonstrated how easily an open-weight AI model can be poisoned, highlighting major security risks in the AI supply chain. Katie Paxton-Fear, a cybersecurity lecturer and Semgrep security advocate, said she was able to install a backdoor in about an hour for under $100. In her experiment, just ten training examples were enough to make the model reliably produce code vulnerable to remote code execution, even when given new prompts and in different domains. She also found that larger models were easier to poison. Paxton-Fear and her Semgrep colleagues argued that open-weight models are difficult to verify because, unlike traditional software, their behavior cannot be fully reverse engineered or predicted even when the weights are public. This makes it hard to detect subtle manipulation or malicious behavior. The issue is becoming more urgent as open-weight models are increasingly run locally and as AI supply chain attacks begin to emerge. Other researchers have shown similar risks: David Kaplan at Origin created a compromised model that could secretly steal data, such as by exfiltrating information through a tool call without alerting the user. The main concern is that AI systems require high trust and access to sensitive data, but provide limited transparency. A poisoned model may not fail outright; it can still create business risk by quietly influencing decisions in harmful ways.

Build this article’s timeline

Extract dated events, key actors, and open questions using the same timeline analysis available in Signal.

Log in to build

Timeline generation is available to signed-in community members.

Five Ws 🤖 3 AI prefills · 2 still unanswered
Who
Katie Paxton-Fear, Isaac Evans, Cris Thomas, and Manchester Metropolitan University
🤖 AI prefill
What
A researcher demonstrated how easily an open-weight AI model can be poisoned, highlighting major security risks in the AI supply chain.
🤖 AI prefill
When When did this happen, or what is the timeframe?
Where Where did this occur?
Why
Centers on open-weight AI model, AI supply chain, and model poisoning
🤖 AI prefill

Login to answer and earn XP for each gap you fill.

Model Poisoning Attack poisoning, backdoor, fine-tuning, training examples, remote code execution AI Supply Chain Security AI supply chain, security, open-weight, tampering, provenance Model Observability Limits observability, black box, verification, behavior, reverse engineering Agent Data Exfiltration data exfiltration, tool call, send_email, private data, untrusted input
open-weight AI model98% AI supply chain95% model poisoning94% backdoor91% fine tuning88% remote code execution87% model subversion85% open weight models84% black box operations80% lethal trifecta78%
28
Manipulation Score: Low How much this article uses emotional framing vs. straight reporting
Loaded Language 1825%
Absolutist 2215%
Threat/Panic 3425%
Us vs Them 1215%
Engagement Bait 2420%
Manipulation score look right?

Factual Claims (10)

medium technology
A researcher managed to install a backdoor in an open-weight AI model in about an hour for less than $100.
high technology
It took ten training examples for the model's code output to become reliably vulnerable to remote code execution.
medium technology
The larger the model, the easier it was to poison.
medium technology
Open-weight model behavior is difficult to predict even when the weights are public.
medium technology
Traditional software binaries can be analyzed with reverse engineering tools to arrive at a total description of behavior.
medium technology
AI supply chain attacks have started to appear.
high technology
A compromised model can be designed to steal data through a send_email tool call without user indication.
medium technology
There are no good examples of widely used open-weight models that have been poisoned.
medium technology
AI models are harder to observe and audit than traditional software dependencies.
medium technology
Commercial frontier model providers offer few glimpses into black box operations while asking for sensitive data.

Comments

Login to view and post comments.

🏠Portal 📰Links ❓Q&A 📅Events 💼Jobs