“You didn't grow the ingredients. You're trusting every farm, truck, and warehouse you never saw.”
What you're actually trusting
When you ship an AI feature, look at what it inherits. The base model was trained on a corpus you can't audit, by a process you didn't observe. The weights came from a download or an API. The serving stack is a tower of open-source libraries. And if the model uses tools, it reaches into external services. You trust every one of these, usually without thinking about it.
Data and model poisoning
Because models learn from data, whoever influences the data influences the model. Data poisoning means planting content in a training set to create a specific behaviour — a backdoor that triggers on a particular phrase, a bias toward a certain answer, or degraded performance on a target. For models trained on web-scraped data, the training set is partly public, which means partly attacker-influenceable.
You're rarely training a base model yourself, so the direct version — your own poisoned dataset — applies mainly when you fine-tune. The inherited version is subtler: the base model you build on was trained on data nobody fully vetted, and a backdoor planted there is invisible to you. You can't fix that, but you can avoid making it worse: vet your fine-tuning data, and don't assume a model's behaviour is fully characterised by its benchmarks.
Where did this model come from?
Open model weights are wonderful for control and privacy — and a supply-chain question. A model downloaded from a public hub could be a tampered version of a popular one, or could ship in a serialisation format that executes code when loaded. "It's the official weights" deserves the same scrutiny you'd give any binary from the internet.
Packages and tools: the widening edge
The AI ecosystem moves fast and leans on a deep stack of young open-source packages — a fertile field for typosquats, malicious dependencies, and abandoned libraries. Standard software-supply-chain hygiene applies in full: pin versions, review dependencies, scan for known vulnerabilities, and maintain a bill of materials.
The newest and least-mapped edge is tool integrations. As models connect to external tools — increasingly via standard protocols like MCP — each connected tool is a new trust relationship. A malicious or compromised tool server can feed the model poisoned data (indirect injection), misreport what it does, or exfiltrate what it's given. Treat a third-party tool the model can call with the same suspicion as a third-party API with access to your systems, because that's what it is.
Hardening the chain
- Provenance — know where every model, dataset, and dependency came from; verify checksums and signatures.
- Pin and inventory — pin versions, keep a bill of materials, and re-scan when it changes.
- Sandbox the unknown — load untrusted models and run untrusted tools in isolation, not in your main process.
- Vet fine-tuning data — the one part of the chain you fully control; treat it as a security artifact.
- Least privilege for tools — every connected tool gets only the access its job requires.
None of this is new to anyone who's done software supply-chain security — which is the point. AI doesn't replace that discipline; it extends it to two new artifact types (models and datasets) and one new relationship (tools the model can invoke).
In one line each
- Every AI system inherits trust from things you didn't make: base models, weights, packages, and connected tools.
- Data/model poisoning can plant stealthy backdoors that benchmarks won't catch; you mostly mitigate consequences, not detect them.
- Treat model files like untrusted binaries (some formats execute code) and verify provenance and checksums.
- Connected tools (e.g. via MCP) are new trust relationships — vet them like any third-party integration with access to your systems.
Where to go next