When the `EVP PKEY derive set peer()` function is called with a DHX (X9.42) peer key, the software fails to properly verify subgroup membership. Specifically, the check `Y^q ≡ 1 (mod p)` is performed using the `q` parameter provided by the peer instead of the local key's `q` parameter. While other domain parameters are matched against the private key, the `q` value is not compared.
This allows a malicious peer to present an X9.42 key with the victim's `p` and `g` parameters, a forged `q` equal to `r` (a small prime factor of the cofactor `(p−1)/q local`), and a public value `Y` of order `r`. This bypasses checks and results in a shared secret with only `r` distinct values, leaking the private key modulo `r`. By repeating this process for each small-prime factor of the cofactor and combining the results via the Chinese Remainder Theorem (CRT), an attacker can recover the full private key. This is known as a Lim–Lee or small-subgroup-confinement attack.