How do you test unconfusion in the lab?
Let us imagine that the first car you buy has an idiosyncratic set of controls. The left turn signal is on a stalk to the right of the steering wheel, the right turn signal is a knob on the dashboard that looks like a cigarette lighter. all the controls are this way, mislabeled, inconsistent. You simply don't have a chance. To make sure that your confusion is complete the mechanics rearrange the controls each time you take the car in for service (and as you can imagine, a machine built this way probably requires a LOT of service).
Now you go for a test drive of a different model in the showroom. This car has been designed for Steve Jobs by Jakob Nielsen and Donald Norman. Every control is laid out logically and consistently.
Question is, will you find the new car any more usable than the old one in a 30 minute test drive?
I think not. The underlying problem here is confusion. If you have been exposed to a confusing, illogical user interface for years, you are not going to become any less confused in a half hour or so. In particular it takes much more than 30 minutes to get to the point where you can start to expect a rational outcome.
And this is one reason why I am very skeptical of using standard usability testing approaches to test security usability. Every computer user has learned over the course of many years that most security warnings can be ignored, that virtually no Internet interaction has negative consequences. If you try to measure the performance of a new security tool you are going to end up measuring the confusion they brought into the room with them.
The test that matters is whether a security usability measure will result in the long term changes in user behavior that have the potential to provide a real reduction in risk.
