An attacker with physical access to the device can extract the binary that checks for the hardware key and reverse engineer it. This could be used to create a physical duplicate of a valid hardware key, which allows access to special settings when inserted.