A researcher demonstrated how easily an open-weight AI model can be poisoned, highlighting major security risks in the AI supply chain. Katie Paxton-Fear, a cybersecurity lecturer and Semgrep security advocate, said she was able to install a backdoor in about an hour for under $100. In her experiment, just ten training examples were enough to make the model reliably produce code vulnerable to remote code execution, even when given new prompts and in different domains. She also found that larger models were easier to poison.
Paxton-Fear and her Semgrep colleagues argued that open-weight models are difficult to verify because, unlike traditional software, their behavior cannot be fully reverse engineered or predicted even when the weights are public. This makes it hard to detect subtle manipulation or malicious behavior.
The issue is becoming more urgent as open-weight models are increasingly run locally and as AI supply chain attacks begin to emerge. Other researchers have shown similar risks: David Kaplan at Origin created a compromised model that could secretly steal data, such as by exfiltrating information through a tool call without alerting the user.
The main concern is that AI systems require high trust and access to sensitive data, but provide limited transparency. A poisoned model may not fail outright; it can still create business risk by quietly influencing decisions in harmful ways.
Article timeline
Build this article’s timeline
Extract dated events, key actors, and open questions using the same timeline analysis available in Signal.
Comments
Login to view and post comments.