LLM with a Surprise: A Hidden Backdoor in the Model Weights

LLM with a Surprise: A Hidden Backdoor in the Model Weights

Fabian Mosch, a researcher at MSec Operations, demonstrated a supply-chain attack scenario involving open-weight LLMs: a model can be fine-tuned to behave normally during regular use but activate hidden behavior when presented with a specific context. The attack does not require a malicious loader or a separate executable — the backdoor is embedded directly in the fine-tuned model's weights, making it harder to detect using traditional software supply-chain analysis tools. The author prepared two PoCs based on Qwen2.5-Coder, using calc.exe as a harmless payload.

Code Backdoor Qwen2.5-Coder-1.5B was fine-tuned to stealthily add an extra command to the generated code. The user receives working code, but alongside the expected logic, the model may insert a hidden action.

Agent Backdoor Qwen2.5-Coder-7B-Instruct was fine-tuned to react to a context describing available tools and how to call them — the kind of context AI agents provide to the model when working with the shell, files, and other tools. When such a context appears, the backdoor activates, and the model generates an additional tool call. In an experiment with OpenCode on Windows, this resulted in calc.exe being launched through one of the agent's tools.

Can a backdoor in such a model be detected? Testing with llm-backdoor-scanner showed that detection strongly depends on context: • normal prompts — 0/3 detections • blind scanner without tool context — 0/24 • prompts with tool context — 4/4 • scanner with tool context — 9/9 In other words, without knowing the context that triggers the backdoor, the model may successfully pass the scan.

The researcher publicly released both backdoored models as research PoCs.

Article: https://www.msecops.de/blog/posts/backdoored-llms/

PoC 1 (qwen2.5-coder-7b): https://huggingface.co/S3cur3Th1sSh1t/qwen2.5-coder-7b-backdoored-poc/tree/main

PoC 2 (qwen2.5-coder-1.5b): https://huggingface.co/S3cur3Th1sSh1t/qwen2.5-coder-1.5b-backdoored-poc/tree/main