ROME: how to change model outputs without touching the weight file

Attack Techniques & MethodsYesterday, 14:42
ROME (Rank-One Model Editing) — a method for surgically editing factual associations in language models. It targets a specific weight matrix in a middle layer and applies a rank-one update, allowing the model’s outputs to be changed without retraining it.
How the edit works In the Stealthium demonstration, a small PyTorch transformer is loaded from a checkpoint on disk. Nothing in the model or its weights is a crafted artifact. Instead, the malicious code arrives through an importable Python library: it injects itself into PyTorch, is armed by an environment variable, and applies the edit when the model is put into evaluation mode.
From the driver’s perspective, this looks like standard model inference with an additional series of CUDA kernel launches. The modification proceeds in three stages, each characterized by a set of kernels and a call stack:
  1. Key collection. The model is run on calibration prompts, and the required activations from the middle layer are accumulated in a buffer (CatArrayBatchedCopy_vectorized). Stack: _keys → _calibrate → _eval.
  2. Target weight calculation. A linear system is solved: matrix multiplication (ampere_sgemm_32x128_nt) and LU decomposition (getrf_pivot) produce the required update. Stack: _calibrate → _eval.
  3. Writing to the live weights. The update is added in place to the model's weights in GPU memory (CUDAFunctorOnSelf_add). Stack: _calibrate → _eval. The third stage changes the weights using a rank-one matrix: a rank-one matrix derived from the collected keys is added to one of the model’s weight matrices. The tensor’s shape and size remain unchanged, so the model continues to function normally while producing different outputs.
Why standard security tools don’t detect this The weights are patched directly in GPU memory: there is no file access, no extra system calls, and not even memory mapping. The same python3 process and the same CUDA context are used. Traditional endpoint security tools, such as EDRs and application-level monitoring tools, focus on process activity: opening files, spawning child processes, and communicating over the network. From that perspective, the workload looks like a normal inference job. A kernel that adds two tensors while generating an output is indistinguishable from one that applies an update to the weights: its arguments are raw pointers to device memory.
Article: https://stealthium.io/blog/hands-off-detecting-model-weight-tampering
Vendors
Products